01HNPTZRK3EKNK1NA5GYEFSM63: helios / deploy

BasicConfig {
    output_rules: [
        "%/var/svc/log/oxide-sled-agent:default.log*",
        "%/zone/oxz_*/root/var/svc/log/oxide-*.log*",
        "%/pool/ext/*/crypt/zone/oxz_*/root/var/svc/log/oxide-*.log*",
        "%/pool/ext/*/crypt/zone/oxz_*/root/var/svc/log/system-illumos-*.log*",
        "%/pool/ext/*/crypt/zone/oxz_ntp_*/root/var/log/chrony/*.log*",
        "!/pool/ext/*/crypt/zone/oxz_propolis-server_*/root/var/svc/log/*.log*",
        "%/pool/ext/*/crypt/debug/global/oxide-sled-agent:default.log.*",
        "%/pool/ext/*/crypt/debug/oxz_*/oxide-*.log.*",
        "%/pool/ext/*/crypt/debug/oxz_*/system-illumos-*.log.*",
        "!/pool/ext/*/crypt/debug/oxz_propolis-server_*/*.log.*",
    ],
    rust_toolchain: None,
    target: Some(
        "lab-2.0-opte-0.28",
    ),
    access_repos: [],
    publish: [],
    skip_clone: true,
}

Buildomat Job: 01HNPV0EJY7V962DQRWAFR386T

Tags:

Artefacts:

Output:

SEQ GLOBAL TIME DETAILS
12024-02-03T11:06:33.926Zjob dependencies complete; ready to run (waiting for 4 h 28 m 31 s)
22024-02-03T11:12:12.936Zjob assigned to worker 01HNQACAKT6DZ3AP8Y4PZ29T4K (queued for 5 m 39 s)
32024-02-03T11:12:13.130Zhttp://172.20.0.5:9969/ipxe/sock... ok
42024-02-03T11:12:13.189Zhttp://172.20.0.5:9969/os/sock/platform/i86pc/kernel/amd64/unix... ok
52024-02-03T11:12:13.226Zhttp://172.20.0.5:9969/os/sock/platform/i86pc/amd64/boot_archive... 12%Error: No response to keepalive - Terminating session
62024-02-03T11:12:13.260ZError: No response de-activating SOL payload
72024-02-03T11:12:13.293Z[SOL Session operational. Use ~? for help]
82024-02-03T11:12:13.326Z 75%Error: No response to keepalive - Terminating session
92024-02-03T11:12:13.360ZError: No response de-activating SOL payload
102024-02-03T11:12:13.393Z[SOL Session operational. Use ~? for help]
112024-02-03T11:12:13.444ZHostname: sock
122024-02-03T11:12:13.483ZLoading smf(7) service descriptions:
132024-02-03T11:12:14.613Zdownloading input: /input/ci-tools/work/end-to-end-tests/bootstrap-ba0a15fa1ac0f64d.gz
142024-02-03T11:12:16.598Zdownloaded input: /input/ci-tools/work/end-to-end-tests/bootstrap-ba0a15fa1ac0f64d.gz
152024-02-03T11:12:16.625Zdownloading input: /input/ci-tools/work/end-to-end-tests/bootstrap.gz
162024-02-03T11:12:42.143Zdownloaded input: /input/ci-tools/work/end-to-end-tests/bootstrap.gz
172024-02-03T11:12:42.143Zdownloading input: /input/ci-tools/work/end-to-end-tests/end_to_end_tests-49aca9644d961704.gz
182024-02-03T11:13:03.749Zdownloaded input: /input/ci-tools/work/end-to-end-tests/end_to_end_tests-49aca9644d961704.gz
192024-02-03T11:13:03.749Zdownloading input: /input/ci-tools/work/caboose-util.gz
202024-02-03T11:13:04.648Zdownloaded input: /input/ci-tools/work/caboose-util.gz
212024-02-03T11:13:04.674Zdownloading input: /input/ci-tools/work/tufaceous.gz
222024-02-03T11:13:07.504Zdownloaded input: /input/ci-tools/work/tufaceous.gz
232024-02-03T11:13:07.534Zdownloading input: /input/package/work/version.txt
242024-02-03T11:13:07.558Zdownloaded input: /input/package/work/version.txt
252024-02-03T11:13:07.558Zdownloading input: /input/package/work/package.tar.gz
262024-02-03T11:13:26.971Zdownloaded input: /input/package/work/package.tar.gz
272024-02-03T11:13:26.971Zdownloading input: /input/package/work/global-zone-packages.tar.gz
282024-02-03T11:14:02.782Zdownloaded input: /input/package/work/global-zone-packages.tar.gz
292024-02-03T11:14:02.782Zdownloading input: /input/package/work/trampoline-global-zone-packages.tar.gz
302024-02-03T11:14:21.171Zdownloaded input: /input/package/work/trampoline-global-zone-packages.tar.gz
312024-02-03T11:14:21.171Zdownloading input: /input/package/work/zones/clickhouse.tar.gz
322024-02-03T11:14:50.951Zsock console login: Error: No response to keepalive - Terminating session
332024-02-03T11:15:00.898ZError: No response de-activating SOL payload
342024-02-03T11:15:02.010Z[SOL Session operational. Use ~? for help]
352024-02-03T11:15:05.265Zdownloaded input: /input/package/work/zones/clickhouse.tar.gz
362024-02-03T11:15:05.290Zdownloading input: /input/package/work/zones/clickhouse_keeper.tar.gz
372024-02-03T11:15:42.285Zdownloaded input: /input/package/work/zones/clickhouse_keeper.tar.gz
382024-02-03T11:15:42.286Zdownloading input: /input/package/work/zones/cockroachdb.tar.gz
392024-02-03T11:16:08.626Zdownloaded input: /input/package/work/zones/cockroachdb.tar.gz
402024-02-03T11:16:08.626Zdownloading input: /input/package/work/zones/crucible-pantry.tar.gz
412024-02-03T11:16:12.584Zdownloaded input: /input/package/work/zones/crucible-pantry.tar.gz
422024-02-03T11:16:12.584Zdownloading input: /input/package/work/zones/crucible.tar.gz
432024-02-03T11:16:20.649Zdownloaded input: /input/package/work/zones/crucible.tar.gz
442024-02-03T11:16:20.673Zdownloading input: /input/package/work/zones/external-dns.tar.gz
452024-02-03T11:16:26.123Zdownloaded input: /input/package/work/zones/external-dns.tar.gz
462024-02-03T11:16:26.123Zdownloading input: /input/package/work/zones/internal-dns.tar.gz
472024-02-03T11:16:31.324Zdownloaded input: /input/package/work/zones/internal-dns.tar.gz
482024-02-03T11:16:31.324Zdownloading input: /input/package/work/zones/ntp.tar.gz
492024-02-03T11:16:31.411Zdownloaded input: /input/package/work/zones/ntp.tar.gz
502024-02-03T11:16:31.436Zdownloading input: /input/package/work/zones/omicron-gateway-asic.tar.gz
512024-02-03T11:16:36.841Zdownloaded input: /input/package/work/zones/omicron-gateway-asic.tar.gz
522024-02-03T11:16:36.841Zdownloading input: /input/package/work/zones/omicron-gateway-softnpu.tar.gz
532024-02-03T11:16:41.500Zdownloaded input: /input/package/work/zones/omicron-gateway-softnpu.tar.gz
542024-02-03T11:16:41.500Zdownloading input: /input/package/work/zones/omicron-nexus-single-sled.tar.gz
552024-02-03T11:16:57.711Zdownloaded input: /input/package/work/zones/omicron-nexus-single-sled.tar.gz
562024-02-03T11:16:57.711Zdownloading input: /input/package/work/zones/omicron-nexus.tar.gz
572024-02-03T11:17:13.696Zdownloaded input: /input/package/work/zones/omicron-nexus.tar.gz
582024-02-03T11:17:13.696Zdownloading input: /input/package/work/zones/overlay.tar.gz
592024-02-03T11:17:13.795Zdownloaded input: /input/package/work/zones/overlay.tar.gz
602024-02-03T11:17:13.819Zdownloading input: /input/package/work/zones/oximeter.tar.gz
612024-02-03T11:17:32.138Zdownloaded input: /input/package/work/zones/oximeter.tar.gz
622024-02-03T11:17:32.138Zdownloading input: /input/package/work/zones/propolis-server.tar.gz
632024-02-03T11:17:49.976Zdownloaded input: /input/package/work/zones/propolis-server.tar.gz
642024-02-03T11:17:49.976Zdownloading input: /input/package/work/zones/switch-asic.tar.gz
652024-02-03T11:18:06.452ZError: No response to keepalive - Terminating session
662024-02-03T11:18:16.377ZError: No response de-activating SOL payload
672024-02-03T11:18:17.483Z[SOL Session operational. Use ~? for help]
682024-02-03T11:19:02.411Zdownloaded input: /input/package/work/zones/switch-asic.tar.gz
692024-02-03T11:19:02.411Zdownloading input: /input/package/work/zones/switch-softnpu.tar.gz
702024-02-03T11:19:51.529Zdownloaded input: /input/package/work/zones/switch-softnpu.tar.gz
 
712024-02-03T11:19:51.529Zstarting task 0: "setup"
722024-02-03T11:19:51.710Z++ uname -s
732024-02-03T11:19:51.710Z+ kern=SunOS
742024-02-03T11:19:51.710Z+ case "$kern" in
752024-02-03T11:19:51.710Z+ groupadd -g 12345 build
762024-02-03T11:19:51.710Z+ useradd -u 12345 -g build -d /home/build -s /bin/bash -c build -P 'Primary Administrator' build
772024-02-03T11:19:53.587Z+ zfs create -o mountpoint=/work rpool/work
782024-02-03T11:19:53.740Z++ awk '$2 == "/home" { print $3 }' /etc/mnttab
792024-02-03T11:19:53.741Z+ home_fs=zfs
802024-02-03T11:19:53.741Z+ [[ zfs == autofs ]]
812024-02-03T11:19:53.741Z+ mkdir -p /home/build
822024-02-03T11:19:53.741Z+ chown build:build /home/build /work
832024-02-03T11:19:54.503Z+ chmod 0700 /home/build /work
842024-02-03T11:19:54.529Zprocess exited: duration 3050 ms, exit code 0
 
852024-02-03T11:19:54.868Zstarting task 1: "authentication"
862024-02-03T11:19:54.950Zprocess exited: duration 233 ms, exit code 0
 
872024-02-03T11:19:55.122Zstarting task 2: "build"
882024-02-03T11:19:55.155Z+ trap _exit_trap EXIT
892024-02-03T11:19:55.155Z+ set +x
902024-02-03T11:20:39.038Z+ pfexec mdb -kwe 'secpolicy_fs_linkdir/v 55 48 89 e5 b8 01 00 00 00 5d c3'
912024-02-03T11:20:39.281Zsecpolicy_fs_linkdir: 0x55 = 0x55
922024-02-03T11:20:39.281Zsecpolicy_fs_linkdir+1: 0xb8 = 0x48
932024-02-03T11:20:39.281Zsecpolicy_fs_linkdir+2: 0x1 = 0x89
942024-02-03T11:20:39.281Zsecpolicy_fs_linkdir+3: 0 = 0xe5
952024-02-03T11:20:39.281Zsecpolicy_fs_linkdir+4: 0 = 0xb8
962024-02-03T11:20:39.281Zsecpolicy_fs_linkdir+5: 0 = 0x1
972024-02-03T11:20:39.307Zsecpolicy_fs_linkdir+6: 0x48 = 0x0
982024-02-03T11:20:39.307Zsecpolicy_fs_linkdir+7: 0x89 = 0x0
992024-02-03T11:20:39.307Zsecpolicy_fs_linkdir+8: 0xe5 = 0x0
1002024-02-03T11:20:39.308Zsecpolicy_fs_linkdir+9: 0x48 = 0x5d
1012024-02-03T11:20:39.308Zsecpolicy_fs_linkdir+0xa: 0x83 = 0xc3
1022024-02-03T11:20:39.308Z+ [[ -d /opt/oxide ]]
1032024-02-03T11:20:39.308Z+ pfexec mv /opt/oxide /opt/oxide-underneath
1042024-02-03T11:20:39.308Z+ pfexec mkdir /opt/oxide
1052024-02-03T11:20:39.308Z+ pfexec mount -F tmpfs -O swap /opt/oxide
1062024-02-03T11:20:39.308Z+ [[ -d /opt/oxide-underneath ]]
1072024-02-03T11:20:39.308Z+ cd /opt/oxide-underneath
1082024-02-03T11:20:39.308Z+ pfexec tar ceEp@/f - .
1092024-02-03T11:20:39.308Z+ cd /opt/oxide
1102024-02-03T11:20:39.308Z+ pfexec tar xveEp@/f -
1112024-02-03T11:20:39.343Zx ., 0 bytes, 0 tape blocks
1122024-02-03T11:20:39.343Zx ./opte, 0 bytes, 0 tape blocks
1132024-02-03T11:20:39.344Zx ./opte/bin, 0 bytes, 0 tape blocks
1142024-02-03T11:20:39.344Zx ./opte/bin/opteadm, 51799224 bytes, 101171 tape blocks
1152024-02-03T11:20:39.367Z+ pfexec rm -rf /opt/oxide-underneath
1162024-02-03T11:20:39.367Z+ pfexec /sbin/zfs create -o mountpoint=/zone rpool/zone
1172024-02-03T11:20:39.392Z+ pfexec mkdir /opt/oxide/work
1182024-02-03T11:20:39.392Z+ pfexec chown build:build /opt/oxide/work
1192024-02-03T11:20:39.392Z+ cd /opt/oxide/work
1202024-02-03T11:20:39.392Z+ ptime -m tar xvzf /input/package/work/package.tar.gz
1212024-02-03T11:20:41.059ZDecompressing '/input/package/work/package.tar.gz' with '/usr/bin/gzcat'...
1222024-02-03T11:20:41.059Zx out/mg-ddm-gz.tar, 63076864 bytes, 123197 tape blocks
1232024-02-03T11:20:41.059Zx out/omicron-sled-agent.tar, 122487808 bytes, 239234 tape blocks
1242024-02-03T11:20:41.059Zx out/oxlog.tar, 8382464 bytes, 16372 tape blocks
1252024-02-03T11:20:41.059Zx out/target/test, 75 bytes, 1 tape blocks
1262024-02-03T11:20:41.059Zx out/npuzone/npuzone, 37184104 bytes, 72626 tape blocks
1272024-02-03T11:20:41.059Zx package-manifest.toml, 23732 bytes, 47 tape blocks
1282024-02-03T11:20:41.059Zx smf/sled-agent/non-gimlet/config.toml, 3448 bytes, 7 tape blocks
1292024-02-03T11:20:41.059Zx target/release/omicron-package, 45642192 bytes, 89145 tape blocks
1302024-02-03T11:20:41.059Zx tools/create_virtual_hardware.sh, 3184 bytes, 7 tape blocks
1312024-02-03T11:20:41.060Zx tools/virtual_hardware.sh, 3416 bytes, 7 tape blocks
1322024-02-03T11:20:41.060Zx tools/scrimlet/create-softnpu-zone.sh, 627 bytes, 2 tape blocks
1332024-02-03T11:20:41.060Zx tools/scrimlet/destroy-softnpu-zone.sh, 203 bytes, 1 tape blocks
1342024-02-03T11:20:41.060Zx tools/scrimlet/softnpu-init.sh, 2157 bytes, 5 tape blocks
1352024-02-03T11:20:41.060Zx tools/scrimlet/softnpu-zone.txt, 330 bytes, 1 tape blocks
1362024-02-03T11:20:41.060Zx tools/scrimlet/softnpu.toml, 171 bytes, 1 tape blocks
1372024-02-03T11:20:41.084Z
1382024-02-03T11:20:41.084Zreal 1.676402461
1392024-02-03T11:20:41.084Zuser 1.508537946
1402024-02-03T11:20:41.084Zsys 0.291990749
1412024-02-03T11:20:41.084Ztrap 0.000066630
1422024-02-03T11:20:41.084Ztflt 0.000000000
1432024-02-03T11:20:41.084Zdflt 0.000000000
1442024-02-03T11:20:41.084Zkflt 0.000000000
1452024-02-03T11:20:41.084Zlock 0.000000000
1462024-02-03T11:20:41.084Zslp 1.508739174
1472024-02-03T11:20:41.085Zlat 0.041571157
1482024-02-03T11:20:41.085Zstop 0.000120682
1492024-02-03T11:20:41.085Z+ cp /input/package/work/zones/clickhouse_keeper.tar.gz /input/package/work/zones/clickhouse.tar.gz /input/package/work/zones/cockroachdb.tar.gz /input/package/work/zones/crucible-pantry.tar.gz /input/package/work/zones/crucible.tar.gz /input/package/work/zones/external-dns.tar.gz /input/package/work/zones/internal-dns.tar.gz /input/package/work/zones/ntp.tar.gz /input/package/work/zones/omicron-gateway-asic.tar.gz /input/package/work/zones/omicron-gateway-softnpu.tar.gz /input/package/work/zones/omicron-nexus-single-sled.tar.gz /input/package/work/zones/omicron-nexus.tar.gz /input/package/work/zones/overlay.tar.gz /input/package/work/zones/oximeter.tar.gz /input/package/work/zones/propolis-server.tar.gz /input/package/work/zones/switch-asic.tar.gz /input/package/work/zones/switch-softnpu.tar.gz out/
1502024-02-03T11:20:42.280Z+ mv out/omicron-nexus-single-sled.tar.gz out/omicron-nexus.tar.gz
1512024-02-03T11:20:42.308Z+ mkdir tests
1522024-02-03T11:20:42.309Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1532024-02-03T11:20:42.309Z+ ptime -m gunzip
1542024-02-03T11:20:42.309Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap-ba0a15fa1ac0f64d
1552024-02-03T11:20:42.528Z
1562024-02-03T11:20:42.552Zreal 0.229538803
1572024-02-03T11:20:42.552Zuser 0.214015604
1582024-02-03T11:20:42.552Zsys 0.015270953
1592024-02-03T11:20:42.553Ztrap 0.000019490
1602024-02-03T11:20:42.553Ztflt 0.000000000
1612024-02-03T11:20:42.553Zdflt 0.000000000
1622024-02-03T11:20:42.553Zkflt 0.000000000
1632024-02-03T11:20:42.553Zlock 0.000000000
1642024-02-03T11:20:42.553Zslp 0.000021130
1652024-02-03T11:20:42.553Zlat 0.000022749
1662024-02-03T11:20:42.553Zstop 0.000126893
1672024-02-03T11:20:42.553Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap-ba0a15fa1ac0f64d
1682024-02-03T11:20:42.553Z+ chmod a+x tests/bootstrap-ba0a15fa1ac0f64d
1692024-02-03T11:20:42.553Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1702024-02-03T11:20:42.554Z+ ptime -m gunzip
1712024-02-03T11:20:42.554Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap
1722024-02-03T11:20:45.062Z
1732024-02-03T11:20:45.089Zreal 2.517805119
1742024-02-03T11:20:45.089Zuser 2.369776145
1752024-02-03T11:20:45.089Zsys 0.147586035
1762024-02-03T11:20:45.090Ztrap 0.000200373
1772024-02-03T11:20:45.090Ztflt 0.000000000
1782024-02-03T11:20:45.090Zdflt 0.000000000
1792024-02-03T11:20:45.090Zkflt 0.000000000
1802024-02-03T11:20:45.090Zlock 0.000000000
1812024-02-03T11:20:45.091Zslp 0.000023740
1822024-02-03T11:20:45.091Zlat 0.000023120
1832024-02-03T11:20:45.091Zstop 0.000133242
1842024-02-03T11:20:45.092Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap
1852024-02-03T11:20:45.092Z+ chmod a+x tests/bootstrap
1862024-02-03T11:20:45.092Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1872024-02-03T11:20:45.092Z+ ptime -m gunzip
1882024-02-03T11:20:45.092Z++ basename /input/ci-tools/work/end-to-end-tests/end_to_end_tests-49aca9644d961704
1892024-02-03T11:20:47.674Z
1902024-02-03T11:20:47.700Zreal 2.600673861
1912024-02-03T11:20:47.701Zuser 2.437539908
1922024-02-03T11:20:47.701Zsys 0.162687773
1932024-02-03T11:20:47.701Ztrap 0.000196794
1942024-02-03T11:20:47.701Ztflt 0.000000000
1952024-02-03T11:20:47.701Zdflt 0.000000000
1962024-02-03T11:20:47.701Zkflt 0.000000000
1972024-02-03T11:20:47.701Zlock 0.000000000
1982024-02-03T11:20:47.701Zslp 0.000024500
1992024-02-03T11:20:47.701Zlat 0.000023340
2002024-02-03T11:20:47.702Zstop 0.000134323
2012024-02-03T11:20:47.702Z++ basename /input/ci-tools/work/end-to-end-tests/end_to_end_tests-49aca9644d961704
2022024-02-03T11:20:47.702Z+ chmod a+x tests/end_to_end_tests-49aca9644d961704
2032024-02-03T11:20:47.702Z+ bmat address ls
2042024-02-03T11:20:47.702ZNAME CIDR FIRST LAST COUNT
2052024-02-03T11:20:47.702Zextra 10.151.2.0/24 10.151.2.164 10.151.2.227 64
2062024-02-03T11:20:47.702Z++ bmat address ls -f extra -Ho first,last,count
2072024-02-03T11:20:47.702Z+ set -- 10.151.2.164 10.151.2.227 64
2082024-02-03T11:20:47.702Z+ EXTRA_IP_START=10.151.2.164
2092024-02-03T11:20:47.702Z+ EXTRA_IP_END=10.151.2.227
2102024-02-03T11:20:47.702Z+ EXTRA_IP_COUNT=64
2112024-02-03T11:20:47.702Z+ (( EXTRA_IP_COUNT >= 32 ))
2122024-02-03T11:20:47.702Z+ EXTRA_IP_BASE=10.151.2
2132024-02-03T11:20:47.702Z+ EXTRA_IP_FOCTET=164
2142024-02-03T11:20:47.702Z+ EXTRA_IP_LOCTET=227
2152024-02-03T11:20:47.702Z+ SERVICE_IP_POOL_START=10.151.2.164
2162024-02-03T11:20:47.703Z+ SERVICE_IP_POOL_END=10.151.2.173
2172024-02-03T11:20:47.703Z+ DNS_IP1=10.151.2.164
2182024-02-03T11:20:47.703Z+ DNS_IP2=10.151.2.165
2192024-02-03T11:20:47.703Z+ UPLINK_IP=10.151.2.174
2202024-02-03T11:20:47.703Z+ IPPOOL_START=10.151.2.175
2212024-02-03T11:20:47.703Z+ IPPOOL_END=10.151.2.227
2222024-02-03T11:20:47.728Z++ ipadm show-addr -po type,addr
2232024-02-03T11:20:47.728Z++ awk '-F[:/]' '$1 == "dhcp" {print $2}'
2242024-02-03T11:20:47.728Z+ GATEWAY_IP=10.151.2.100
2252024-02-03T11:20:47.728Z+ [[ -n 10.151.2.100 ]]
2262024-02-03T11:20:47.728Z+ ping -s 10.151.2.100 56 1
2272024-02-03T11:20:47.728ZPING 10.151.2.100: 56 data bytes
2282024-02-03T11:20:47.728Z64 bytes from 10.151.2.100: icmp_seq=0. time=0.012 ms
2292024-02-03T11:20:47.728Z
2302024-02-03T11:20:47.728Z----10.151.2.100 PING Statistics----
2312024-02-03T11:20:47.728Z1 packets transmitted, 1 packets received, 0% packet loss
2322024-02-03T11:20:47.728Zround-trip (ms) min/avg/max/stddev = 0.012/0.012/0.012/-nan
2332024-02-03T11:20:47.728Z++ arp -an
2342024-02-03T11:20:47.728Z++ awk -vgw=10.151.2.100 '$2 == gw {print $NF}'
2352024-02-03T11:20:47.728Z+ GATEWAY_MAC=18:c0:4d:0d:a0:2a
2362024-02-03T11:20:47.728Z+ routeadm -e ipv4-forwarding -u
2372024-02-03T11:20:47.834Z+ PXA_START=10.151.2.164
2382024-02-03T11:20:47.834Z+ PXA_END=10.151.2.227
2392024-02-03T11:20:47.857Z+ export GATEWAY_IP GATEWAY_MAC PXA_START PXA_END
2402024-02-03T11:20:47.857Z+ pfexec zpool create -f scratch c1t1d0 c2t1d0
2412024-02-03T11:20:48.033Z+ ZPOOL_VDEV_DIR=/scratch
2422024-02-03T11:20:48.033Z+ ptime -m pfexec ./tools/create_virtual_hardware.sh
2432024-02-03T11:20:48.056Z+++ dirname ./tools/create_virtual_hardware.sh
2442024-02-03T11:20:48.056Z++ cd ./tools
2452024-02-03T11:20:48.057Z++ pwd
2462024-02-03T11:20:48.057Z+ SOURCE_DIR=/opt/oxide/work/tools
2472024-02-03T11:20:48.057Z+ OMICRON_TOP=/opt/oxide/work/tools/..
2482024-02-03T11:20:48.057Z+ SOFTNPU_MODE=zone
2492024-02-03T11:20:48.057Z+ . /opt/oxide/work/tools/virtual_hardware.sh
2502024-02-03T11:20:48.057Z++ MARKER=/etc/opt/oxide/NO_INSTALL
2512024-02-03T11:20:48.057Z++ [[ -f /etc/opt/oxide/NO_INSTALL ]]
2522024-02-03T11:20:48.057Z++ dladm show-phys -p -o LINK
2532024-02-03T11:20:48.057Z++ head -1
2542024-02-03T11:20:48.081Z+ PHYSICAL_LINK=igb0
2552024-02-03T11:20:48.081Z+ echo 'Using igb0 as physical link'
2562024-02-03T11:20:48.081ZUsing igb0 as physical link
2572024-02-03T11:20:48.081Z+ ensure_run_as_root
2582024-02-03T11:20:48.082Z++ id -u
2592024-02-03T11:20:48.082Z+ [[ 0 -ne 0 ]]
2602024-02-03T11:20:48.082Z+ ensure_zpools
2612024-02-03T11:20:48.082Z+ ZPOOL_TYPES=('oxp_' 'oxi_')
2622024-02-03T11:20:48.082Z+ for ZPOOL_TYPE in "${ZPOOL_TYPES[@]}"
2632024-02-03T11:20:48.082Z+ readarray -t ZPOOLS
2642024-02-03T11:20:48.082Z++ grep '"oxp_' /opt/oxide/work/tools/../smf/sled-agent/non-gimlet/config.toml
2652024-02-03T11:20:48.082Z++ sed 's/[ ",]//g'
2662024-02-03T11:20:48.082Z+ for ZPOOL in "${ZPOOLS[@]}"
2672024-02-03T11:20:48.082Z+ echo 'Zpool: [oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b]'
2682024-02-03T11:20:48.082ZZpool: [oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b]
2692024-02-03T11:20:48.082Z+ VDEV_PATH=/scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
2702024-02-03T11:20:48.082Z+ [[ -f /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
2712024-02-03T11:20:48.082Z+ dd if=/dev/zero of=/scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
2722024-02-03T11:20:48.082Z0+0 records in
2732024-02-03T11:20:48.082Z0+0 records out
2742024-02-03T11:20:48.082Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
2752024-02-03T11:20:48.082Z+ success 'ZFS vdev /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
2762024-02-03T11:20:48.082Z+ set +x
2772024-02-03T11:20:48.082ZZFS vdev /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
2782024-02-03T11:20:48.082Z++ zpool list -o name
2792024-02-03T11:20:48.082Z++ grep oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b
2802024-02-03T11:20:48.082Z+ [[ -z '' ]]
2812024-02-03T11:20:48.082Z+ zpool create -f oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
2822024-02-03T11:20:48.112Z+ success 'ZFS zpool oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
2832024-02-03T11:20:48.137Z+ set +x
2842024-02-03T11:20:48.137ZZFS zpool oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b exists
2852024-02-03T11:20:48.137ZZpool: [oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03]
2862024-02-03T11:20:48.137Z+ for ZPOOL in "${ZPOOLS[@]}"
2872024-02-03T11:20:48.137Z+ echo 'Zpool: [oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03]'
2882024-02-03T11:20:48.137Z+ VDEV_PATH=/scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
2892024-02-03T11:20:48.137Z+ [[ -f /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
2902024-02-03T11:20:48.137Z+ dd if=/dev/zero of=/scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
2912024-02-03T11:20:48.137Z0+0 records in
2922024-02-03T11:20:48.137Z0+0 records out
2932024-02-03T11:20:48.138Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
2942024-02-03T11:20:48.138Z+ success 'ZFS vdev /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
2952024-02-03T11:20:48.138Z+ set +x
2962024-02-03T11:20:48.138ZZFS vdev /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
2972024-02-03T11:20:48.138Z++ zpool list -o name
2982024-02-03T11:20:48.138Z++ grep oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03
2992024-02-03T11:20:48.138Z+ [[ -z '' ]]
3002024-02-03T11:20:48.138Z+ zpool create -f oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3012024-02-03T11:20:48.162Z+ success 'ZFS zpool oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3022024-02-03T11:20:48.162Z+ set +x
3032024-02-03T11:20:48.162ZZFS zpool oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
3042024-02-03T11:20:48.162Z+ for ZPOOL in "${ZPOOLS[@]}"
3052024-02-03T11:20:48.162Z+ echo 'Zpool: [oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3062024-02-03T11:20:48.162ZZpool: [oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03]
3072024-02-03T11:20:48.162Z+ VDEV_PATH=/scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3082024-02-03T11:20:48.162Z+ [[ -f /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3092024-02-03T11:20:48.162Z+ dd if=/dev/zero of=/scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3102024-02-03T11:20:48.162Z0+0 records in
3112024-02-03T11:20:48.162Z0+0 records out
3122024-02-03T11:20:48.162Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
3132024-02-03T11:20:48.162Z+ success 'ZFS vdev /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3142024-02-03T11:20:48.163Z+ set +x
3152024-02-03T11:20:48.163ZZFS vdev /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
3162024-02-03T11:20:48.163Z++ zpool list -o name
3172024-02-03T11:20:48.163Z++ grep oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03
3182024-02-03T11:20:48.186Z+ [[ -z '' ]]
3192024-02-03T11:20:48.186Z+ zpool create -f oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3202024-02-03T11:20:48.210Z+ success 'ZFS zpool oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3212024-02-03T11:20:48.210Z+ set +x
3222024-02-03T11:20:48.210ZZFS zpool oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
3232024-02-03T11:20:48.210Z+ for ZPOOL in "${ZPOOLS[@]}"
3242024-02-03T11:20:48.210Z+ echo 'Zpool: [oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3252024-02-03T11:20:48.210ZZpool: [oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03]
3262024-02-03T11:20:48.210Z+ VDEV_PATH=/scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3272024-02-03T11:20:48.210Z+ [[ -f /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3282024-02-03T11:20:48.210Z+ dd if=/dev/zero of=/scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3292024-02-03T11:20:48.210Z0+0 records in
3302024-02-03T11:20:48.210Z0+0 records out
3312024-02-03T11:20:48.210Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
3322024-02-03T11:20:48.210Z+ success 'ZFS vdev /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3332024-02-03T11:20:48.211Z+ set +x
3342024-02-03T11:20:48.211ZZFS vdev /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
3352024-02-03T11:20:48.211Z++ zpool list -o name
3362024-02-03T11:20:48.211Z++ grep oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03
3372024-02-03T11:20:48.211Z+ [[ -z '' ]]
3382024-02-03T11:20:48.211Z+ zpool create -f oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3392024-02-03T11:20:48.235Z+ success 'ZFS zpool oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3402024-02-03T11:20:48.235Z+ set +x
3412024-02-03T11:20:48.235ZZFS zpool oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
3422024-02-03T11:20:48.235Z+ for ZPOOL in "${ZPOOLS[@]}"
3432024-02-03T11:20:48.235Z+ echo 'Zpool: [oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3442024-02-03T11:20:48.235ZZpool: [oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03]
3452024-02-03T11:20:48.235Z+ VDEV_PATH=/scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3462024-02-03T11:20:48.235Z+ [[ -f /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3472024-02-03T11:20:48.235Z+ dd if=/dev/zero of=/scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3482024-02-03T11:20:48.235Z0+0 records in
3492024-02-03T11:20:48.235Z0+0 records out
3502024-02-03T11:20:48.235Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3512024-02-03T11:20:48.235Z+ success 'ZFS vdev /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3522024-02-03T11:20:48.235Z+ set +x
3532024-02-03T11:20:48.235ZZFS vdev /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
3542024-02-03T11:20:48.259Z++ zpool list -o name
3552024-02-03T11:20:48.259Z++ grep oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03
3562024-02-03T11:20:48.259Z+ [[ -z '' ]]
3572024-02-03T11:20:48.259Z+ zpool create -f oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3582024-02-03T11:20:48.283Z+ success 'ZFS zpool oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3592024-02-03T11:20:48.283Z+ set +x
3602024-02-03T11:20:48.283ZZFS zpool oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
3612024-02-03T11:20:48.283Z+ for ZPOOL in "${ZPOOLS[@]}"
3622024-02-03T11:20:48.283Z+ echo 'Zpool: [oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930]'
3632024-02-03T11:20:48.283ZZpool: [oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930]
3642024-02-03T11:20:48.283Z+ VDEV_PATH=/scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev
3652024-02-03T11:20:48.283Z+ [[ -f /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev ]]
3662024-02-03T11:20:48.283Z+ dd if=/dev/zero of=/scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev bs=1 count=0 seek=20G
3672024-02-03T11:20:48.284Z0+0 records in
3682024-02-03T11:20:48.284Z0+0 records out
3692024-02-03T11:20:48.284Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3702024-02-03T11:20:48.284Z+ success 'ZFS vdev /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev exists'
3712024-02-03T11:20:48.284Z+ set +x
3722024-02-03T11:20:48.284ZZFS vdev /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev exists
3732024-02-03T11:20:48.284Z++ zpool list -o name
3742024-02-03T11:20:48.284Z++ grep oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930
3752024-02-03T11:20:48.307Z+ [[ -z '' ]]
3762024-02-03T11:20:48.308Z+ zpool create -f oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev
3772024-02-03T11:20:48.331Z+ success 'ZFS zpool oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 exists'
3782024-02-03T11:20:48.331Z+ set +x
3792024-02-03T11:20:48.331ZZFS zpool oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 exists
3802024-02-03T11:20:48.331Z+ for ZPOOL in "${ZPOOLS[@]}"
3812024-02-03T11:20:48.331Z+ echo 'Zpool: [oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60]'
3822024-02-03T11:20:48.331ZZpool: [oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60]
3832024-02-03T11:20:48.332Z+ VDEV_PATH=/scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev
3842024-02-03T11:20:48.332Z+ [[ -f /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev ]]
3852024-02-03T11:20:48.332Z+ dd if=/dev/zero of=/scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev bs=1 count=0 seek=20G
3862024-02-03T11:20:48.332Z0+0 records in
3872024-02-03T11:20:48.332Z0+0 records out
3882024-02-03T11:20:48.332Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3892024-02-03T11:20:48.332Z+ success 'ZFS vdev /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev exists'
3902024-02-03T11:20:48.332Z+ set +x
3912024-02-03T11:20:48.332ZZFS vdev /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev exists
3922024-02-03T11:20:48.332Z++ zpool list -o name
3932024-02-03T11:20:48.332Z++ grep oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60
3942024-02-03T11:20:48.332Z+ [[ -z '' ]]
3952024-02-03T11:20:48.332Z+ zpool create -f oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev
3962024-02-03T11:20:48.358Z+ success 'ZFS zpool oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 exists'
3972024-02-03T11:20:48.358Z+ set +x
3982024-02-03T11:20:48.358ZZFS zpool oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 exists
3992024-02-03T11:20:48.387Z+ for ZPOOL in "${ZPOOLS[@]}"
4002024-02-03T11:20:48.388Z+ echo 'Zpool: [oxp_31bd71cd-4736-4a12-a387-9b74b050396f]'
4012024-02-03T11:20:48.388ZZpool: [oxp_31bd71cd-4736-4a12-a387-9b74b050396f]
4022024-02-03T11:20:48.388Z+ VDEV_PATH=/scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev
4032024-02-03T11:20:48.388Z+ [[ -f /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev ]]
4042024-02-03T11:20:48.388Z+ dd if=/dev/zero of=/scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev bs=1 count=0 seek=20G
4052024-02-03T11:20:48.388Z0+0 records in
4062024-02-03T11:20:48.388Z0+0 records out
4072024-02-03T11:20:48.388Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4082024-02-03T11:20:48.388Z+ success 'ZFS vdev /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev exists'
4092024-02-03T11:20:48.388Z+ set +x
4102024-02-03T11:20:48.388ZZFS vdev /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev exists
4112024-02-03T11:20:48.388Z++ zpool list -o name
4122024-02-03T11:20:48.388Z++ grep oxp_31bd71cd-4736-4a12-a387-9b74b050396f
4132024-02-03T11:20:48.388Z+ [[ -z '' ]]
4142024-02-03T11:20:48.388Z+ zpool create -f oxp_31bd71cd-4736-4a12-a387-9b74b050396f /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev
4152024-02-03T11:20:48.417Z+ success 'ZFS zpool oxp_31bd71cd-4736-4a12-a387-9b74b050396f exists'
4162024-02-03T11:20:48.417Z+ set +x
4172024-02-03T11:20:48.418ZZFS zpool oxp_31bd71cd-4736-4a12-a387-9b74b050396f exists
4182024-02-03T11:20:48.418Z+ for ZPOOL in "${ZPOOLS[@]}"
4192024-02-03T11:20:48.418Z+ echo 'Zpool: [oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7]'
4202024-02-03T11:20:48.418ZZpool: [oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7]
4212024-02-03T11:20:48.418Z+ VDEV_PATH=/scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev
4222024-02-03T11:20:48.418Z+ [[ -f /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev ]]
4232024-02-03T11:20:48.418Z+ dd if=/dev/zero of=/scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev bs=1 count=0 seek=20G
4242024-02-03T11:20:48.418Z0+0 records in
4252024-02-03T11:20:48.418Z0+0 records out
4262024-02-03T11:20:48.418Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4272024-02-03T11:20:48.418Z+ success 'ZFS vdev /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev exists'
4282024-02-03T11:20:48.418Z+ set +x
4292024-02-03T11:20:48.418ZZFS vdev /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev exists
4302024-02-03T11:20:48.419Z++ zpool list -o name
4312024-02-03T11:20:48.419Z++ grep oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7
4322024-02-03T11:20:48.446Z+ [[ -z '' ]]
4332024-02-03T11:20:48.446Z+ zpool create -f oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev
4342024-02-03T11:20:48.470Z+ success 'ZFS zpool oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 exists'
4352024-02-03T11:20:48.470Z+ set +x
4362024-02-03T11:20:48.471ZZFS zpool oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 exists
4372024-02-03T11:20:48.471Z+ for ZPOOL_TYPE in "${ZPOOL_TYPES[@]}"
4382024-02-03T11:20:48.471Z+ readarray -t ZPOOLS
4392024-02-03T11:20:48.471Z++ grep '"oxi_' /opt/oxide/work/tools/../smf/sled-agent/non-gimlet/config.toml
4402024-02-03T11:20:48.471Z++ sed 's/[ ",]//g'
4412024-02-03T11:20:48.471Z+ for ZPOOL in "${ZPOOLS[@]}"
4422024-02-03T11:20:48.471Z+ echo 'Zpool: [oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b]'
4432024-02-03T11:20:48.471ZZpool: [oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b]
4442024-02-03T11:20:48.471Z+ VDEV_PATH=/scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4452024-02-03T11:20:48.471Z+ [[ -f /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
4462024-02-03T11:20:48.471Z+ dd if=/dev/zero of=/scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
4472024-02-03T11:20:48.471Z0+0 records in
4482024-02-03T11:20:48.471Z0+0 records out
4492024-02-03T11:20:48.471Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4502024-02-03T11:20:48.471Z+ success 'ZFS vdev /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
4512024-02-03T11:20:48.471Z+ set +x
4522024-02-03T11:20:48.471ZZFS vdev /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
4532024-02-03T11:20:48.471Z++ zpool list -o name
4542024-02-03T11:20:48.471Z++ grep oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b
4552024-02-03T11:20:48.471Z+ [[ -z '' ]]
4562024-02-03T11:20:48.471Z+ zpool create -f oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4572024-02-03T11:20:48.495Z+ success 'ZFS zpool oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
4582024-02-03T11:20:48.496Z+ set +x
4592024-02-03T11:20:48.496ZZFS zpool oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b exists
4602024-02-03T11:20:48.496Z+ for ZPOOL in "${ZPOOLS[@]}"
4612024-02-03T11:20:48.496Z+ echo 'Zpool: [oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b]'
4622024-02-03T11:20:48.496ZZpool: [oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b]
4632024-02-03T11:20:48.496Z+ VDEV_PATH=/scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4642024-02-03T11:20:48.496Z+ [[ -f /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
4652024-02-03T11:20:48.496Z+ dd if=/dev/zero of=/scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
4662024-02-03T11:20:48.521Z0+0 records in
4672024-02-03T11:20:48.521Z0+0 records out
4682024-02-03T11:20:48.521Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4692024-02-03T11:20:48.521Z+ success 'ZFS vdev /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
4702024-02-03T11:20:48.521Z+ set +x
4712024-02-03T11:20:48.521ZZFS vdev /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
4722024-02-03T11:20:48.521Z++ zpool list -o name
4732024-02-03T11:20:48.521Z++ grep oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b
4742024-02-03T11:20:48.521Z+ [[ -z '' ]]
4752024-02-03T11:20:48.522Z+ zpool create -f oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4762024-02-03T11:20:48.546Z+ success 'ZFS zpool oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
4772024-02-03T11:20:48.546Z+ set +x
4782024-02-03T11:20:48.546ZZFS zpool oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b exists
4792024-02-03T11:20:48.546Z+ [[ zone == \z\o\n\e ]]
4802024-02-03T11:20:48.547Z+ ensure_simulated_links igb0
4812024-02-03T11:20:48.547Z+ local PHYSICAL_LINK=igb0
4822024-02-03T11:20:48.547Z+ INDICES=("0" "1")
4832024-02-03T11:20:48.547Z+ for I in "${INDICES[@]}"
4842024-02-03T11:20:48.547Z++ get_simnet_name_if_exists net0
4852024-02-03T11:20:48.547Z++ dladm show-simnet -p -o LINK net0
4862024-02-03T11:20:48.547Z++ echo ''
4872024-02-03T11:20:48.547Z+ [[ -z '' ]]
4882024-02-03T11:20:48.547Z+ dladm create-simnet -t net0
4892024-02-03T11:20:48.572Z+ dladm create-simnet -t sc0_0
4902024-02-03T11:20:48.572Z+ dladm modify-simnet -t -p net0 sc0_0
4912024-02-03T11:20:48.572Z+ dladm set-linkprop -p mtu=9000 sc0_0
4922024-02-03T11:20:48.572Z+ success 'Simnet net0/sc0_0 exists'
4932024-02-03T11:20:48.572Z+ set +x
4942024-02-03T11:20:48.572ZSimnet net0/sc0_0 exists
4952024-02-03T11:20:48.573Z+ for I in "${INDICES[@]}"
4962024-02-03T11:20:48.573Z++ get_simnet_name_if_exists net1
4972024-02-03T11:20:48.573Z++ dladm show-simnet -p -o LINK net1
4982024-02-03T11:20:48.573Z++ echo ''
4992024-02-03T11:20:48.573Z+ [[ -z '' ]]
5002024-02-03T11:20:48.573Z+ dladm create-simnet -t net1
5012024-02-03T11:20:48.573Z+ dladm create-simnet -t sc1_0
5022024-02-03T11:20:48.599Z+ dladm modify-simnet -t -p net1 sc1_0
5032024-02-03T11:20:48.599Z+ dladm set-linkprop -p mtu=9000 sc1_0
5042024-02-03T11:20:48.599Z+ success 'Simnet net1/sc1_0 exists'
5052024-02-03T11:20:48.599Z+ set +x
5062024-02-03T11:20:48.599ZSimnet net1/sc1_0 exists
5072024-02-03T11:20:48.599Z++ get_vnic_name_if_exists sc0_1
5082024-02-03T11:20:48.599Z++ dladm show-vnic -p -o LINK sc0_1
5092024-02-03T11:20:48.599Z++ echo ''
5102024-02-03T11:20:48.599Z+ [[ -z '' ]]
5112024-02-03T11:20:48.599Z+ dladm create-vnic -t sc0_1 -l igb0 -m a8:e1:de:01:70:1d
5122024-02-03T11:20:48.599Z+ [[ -v PROMISC_FILT_OFF ]]
5132024-02-03T11:20:48.600Z+ success 'Vnic sc0_1 exists'
5142024-02-03T11:20:48.600Z+ set +x
5152024-02-03T11:20:48.600ZVnic sc0_1 exists
5162024-02-03T11:20:48.600Z+ warn_if_no_proxy_arp
5172024-02-03T11:20:48.600Z+ [[ -v PXA_START ]]
5182024-02-03T11:20:48.600Z+ [[ -v PXA_END ]]
5192024-02-03T11:20:48.600Z+ ensure_softnpu_zone
5202024-02-03T11:20:48.600Z+ zoneadm list
5212024-02-03T11:20:48.600Z+ grep -q sidecar_softnpu
5222024-02-03T11:20:48.629Z+ [[ -f out/npuzone/npuzone ]]
5232024-02-03T11:20:48.630Z+ out/npuzone/npuzone create sidecar --omicron-zone --ports sc0_0,tfportrear0_0 --ports sc0_1,tfportqsfp0_0 --sidecar-lite-commit e3ea4b495ba0a71801ded0776ae4bbd31df57e26 --softnpu-commit dbab082dfa89da5db5ca2325c257089d2f130092
5242024-02-03T11:20:48.630Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/softnpu/image/dbab082dfa89da5db5ca2325c257089d2f130092/softnpu.sha256.txt
5252024-02-03T11:20:55.412Zdownloading softnpu ...done
5262024-02-03T11:20:55.448Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/sidecar-lite/release/e3ea4b495ba0a71801ded0776ae4bbd31df57e26/libsidecar_lite.so.sha256.txt
5272024-02-03T11:21:02.538Zdownloading asic_program.so ...done
5282024-02-03T11:21:02.563Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/sidecar-lite/release/e3ea4b495ba0a71801ded0776ae4bbd31df57e26/scadm.sha256.txt
 
5292024-02-03T11:21:06.745ZError: No response to keepalive - Terminating session
 
5302024-02-03T11:21:16.274Zdownloading scadm ...done
5312024-02-03T11:21:16.302Zcreating port tfportrear0_0 for sc0_0
5322024-02-03T11:21:16.302Zcreating port tfportqsfp0_0 for sc0_1
5332024-02-03T11:21:16.302Zsoftnpu zone setup
5342024-02-03T11:21:16.302Zconfigure zone
5352024-02-03T11:21:16.326Zinstall zone
 
5362024-02-03T11:21:16.909ZError: No response de-activating SOL payload
 
5372024-02-03T11:21:17.364Zboot zone
 
5382024-02-03T11:21:18.123Z[SOL Session operational. Use ~? for help]
 
5392024-02-03T11:21:20.473Z[sidecar_softnpu] svcs milestone/network
5402024-02-03T11:21:20.500ZSTATE STIME FMRI
5412024-02-03T11:21:20.501Zonline 11:21:19 svc:/milestone/network:default
5422024-02-03T11:21:20.501Z[sidecar_softnpu] /softnpu/npu start
5432024-02-03T11:21:20.527Z
5442024-02-03T11:21:20.551Z+ /opt/oxide/work/tools/scrimlet/softnpu-init.sh
5452024-02-03T11:21:20.551Z+ GATEWAY_IP=10.151.2.100
5462024-02-03T11:21:20.551Z+ echo 'Using 10.151.2.100 as gateway ip'
5472024-02-03T11:21:20.552ZUsing 10.151.2.100 as gateway ip
5482024-02-03T11:21:20.552Z+ [[ ! -v GATEWAY_MAC ]]
5492024-02-03T11:21:20.552Z+ GATEWAY_MAC=18:c0:4d:0d:a0:2a
5502024-02-03T11:21:20.552Z++ grep -c -E '^([0-9a-fA-F]{2}:){5}[0-9a-fA-F]{2}$' /dev/fd/63
5512024-02-03T11:21:20.552Z+++ echo 18:c0:4d:0d:a0:2a
5522024-02-03T11:21:20.552Z+ COUNT=1
5532024-02-03T11:21:20.552Z+ [[ 1 -ne 1 ]]
5542024-02-03T11:21:20.552Z+ echo 'Using 18:c0:4d:0d:a0:2a as gateway mac'
5552024-02-03T11:21:20.552ZUsing 18:c0:4d:0d:a0:2a as gateway mac
5562024-02-03T11:21:20.552Z+ z_scadm add-arp-entry 10.151.2.100 18:c0:4d:0d:a0:2a
5572024-02-03T11:21:20.552Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone add-arp-entry 10.151.2.100 18:c0:4d:0d:a0:2a
5582024-02-03T11:21:20.590Z+ PXA_MAC=a8:e1:de:01:70:1d
5592024-02-03T11:21:20.590Z+ [[ -v PXA_START ]]
5602024-02-03T11:21:20.590Z+ z_scadm add-proxy-arp 10.151.2.164 10.151.2.227 a8:e1:de:01:70:1d
5612024-02-03T11:21:20.590Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone add-proxy-arp 10.151.2.164 10.151.2.227 a8:e1:de:01:70:1d
5622024-02-03T11:21:20.619Z+ z_scadm dump-state
5632024-02-03T11:21:20.619Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone dump-state
5642024-02-03T11:21:20.643Zlocal v6:
5652024-02-03T11:21:20.643Zlocal v4:
5662024-02-03T11:21:20.643Zrouter v6:
5672024-02-03T11:21:20.643Zrouter v4_idx:
5682024-02-03T11:21:20.643Zrouter v4_routes:
5692024-02-03T11:21:20.643Zresolver v4:
5702024-02-03T11:21:20.643Z10.151.2.100 -> 18:c0:4d:0d:a0:2a
5712024-02-03T11:21:20.643Zresolver v6:
5722024-02-03T11:21:20.643Znat v4:
5732024-02-03T11:21:20.643Znat_v6:
5742024-02-03T11:21:20.643Zport_mac:
5752024-02-03T11:21:20.643Zproxy arp:
5762024-02-03T11:21:20.643Z10.151.2.164/10.151.2.227: a8:e1:de:01:70:1d
5772024-02-03T11:21:20.643Z+ success 'softnpu zone exists'
5782024-02-03T11:21:20.643Z+ set +x
5792024-02-03T11:21:20.643Zsoftnpu zone exists
5802024-02-03T11:21:20.643Z
5812024-02-03T11:21:20.643Zreal 32.590864682
5822024-02-03T11:21:20.643Zuser 0.704365546
5832024-02-03T11:21:20.643Zsys 1.503198816
5842024-02-03T11:21:20.643Ztrap 0.000058978
5852024-02-03T11:21:20.644Ztflt 0.000124355
5862024-02-03T11:21:20.644Zdflt 0.001273688
5872024-02-03T11:21:20.644Zkflt 0.000000000
5882024-02-03T11:21:20.644Zlock 25:30.868013665
5892024-02-03T11:21:20.644Zslp 1:37.574444431
5902024-02-03T11:21:20.644Zlat 0.051114644
5912024-02-03T11:21:20.644Zstop 0.346932850
5922024-02-03T11:21:20.644Z+ tar xf out/omicron-sled-agent.tar pkg/config-rss.toml
5932024-02-03T11:21:20.668Z++ sed -n 's/silo_name = "\(.*\)"/\1/p' pkg/config-rss.toml
5942024-02-03T11:21:20.668Z+ SILO_NAME=recovery
5952024-02-03T11:21:20.668Z++ sed -n 's/external_dns_zone_name = "\(.*\)"/\1/p' pkg/config-rss.toml
5962024-02-03T11:21:20.668Z+ EXTERNAL_DNS_DOMAIN=oxide.test
5972024-02-03T11:21:20.668Z+ sed -i~ '
5982024-02-03T11:21:20.668Z /^external_dns_ips/c\
5992024-02-03T11:21:20.668Zexternal_dns_ips = [ "10.151.2.164", "10.151.2.165" ]
6002024-02-03T11:21:20.668Z /^\[\[internal_services_ip_pool_ranges/,/^$/ {
6012024-02-03T11:21:20.668Z /^first/c\
6022024-02-03T11:21:20.668Zfirst = "10.151.2.164"
6032024-02-03T11:21:20.668Z /^last/c\
6042024-02-03T11:21:20.668Zlast = "10.151.2.173"
6052024-02-03T11:21:20.668Z }
6062024-02-03T11:21:20.668Z /^\[rack_network_config/,/^$/ {
6072024-02-03T11:21:20.668Z /^infra_ip_first/c\
6082024-02-03T11:21:20.668Zinfra_ip_first = "10.151.2.174"
6092024-02-03T11:21:20.668Z /^infra_ip_last/c\
6102024-02-03T11:21:20.669Zinfra_ip_last = "10.151.2.174"
6112024-02-03T11:21:20.669Z }
6122024-02-03T11:21:20.669Z /^\[\[rack_network_config.ports/,/^$/ {
6132024-02-03T11:21:20.669Z /^routes/c\
6142024-02-03T11:21:20.669Zroutes = \[{nexthop = "10.151.2.100", destination = "0.0.0.0/0"}\]
6152024-02-03T11:21:20.669Z /^addresses/c\
6162024-02-03T11:21:20.669Zaddresses = \["10.151.2.174/24"\]
6172024-02-03T11:21:20.669Z }
6182024-02-03T11:21:20.669Z' pkg/config-rss.toml
6192024-02-03T11:21:20.704Z+ diff -u pkg/config-rss.toml~ pkg/config-rss.toml
6202024-02-03T11:21:20.704Z--- pkg/config-rss.toml~ Mon Jul 24 01:21:28 2006
6212024-02-03T11:21:20.704Z+++ pkg/config-rss.toml Sat Feb 3 11:21:21 2024
6222024-02-03T11:21:20.704Z@@ -41,7 +41,7 @@
6232024-02-03T11:21:20.704Z # the DNS domain delegated to the rack by the customer. Each of these addresses
6242024-02-03T11:21:20.704Z # must be contained in one of the "internal services" IP Pool ranges listed
6252024-02-03T11:21:20.704Z # below.
6262024-02-03T11:21:20.705Z-external_dns_ips = [ "192.168.1.20", "192.168.1.21" ]
6272024-02-03T11:21:20.705Z+external_dns_ips = [ "10.151.2.164", "10.151.2.165" ]
6282024-02-03T11:21:20.705Z
6292024-02-03T11:21:20.705Z # Initial TLS certificates for the external API
6302024-02-03T11:21:20.705Z #
6312024-02-03T11:21:20.705Z@@ -77,8 +77,8 @@
6322024-02-03T11:21:20.705Z #
6332024-02-03T11:21:20.705Z # For more on this and what to put here, see docs/how-to-run.adoc.
6342024-02-03T11:21:20.706Z [[internal_services_ip_pool_ranges]]
6352024-02-03T11:21:20.706Z-first = "192.168.1.20"
6362024-02-03T11:21:20.706Z-last = "192.168.1.29"
6372024-02-03T11:21:20.706Z+first = "10.151.2.164"
6382024-02-03T11:21:20.706Z+last = "10.151.2.173"
6392024-02-03T11:21:20.706Z
6402024-02-03T11:21:20.706Z # TODO - this configuration is subject to change going forward. Ultimately these
6412024-02-03T11:21:20.706Z # parameters should be provided to the control plane via wicket, but we need to
6422024-02-03T11:21:20.706Z@@ -92,8 +92,8 @@
6432024-02-03T11:21:20.706Z # A range of IP addresses used by Boundary Services on the external network. In
6442024-02-03T11:21:20.706Z # a real system, these would be addresses of the uplink ports on the Sidecar.
6452024-02-03T11:21:20.706Z # With softnpu, only one address is used.
6462024-02-03T11:21:20.706Z-infra_ip_first = "192.168.1.30"
6472024-02-03T11:21:20.706Z-infra_ip_last = "192.168.1.30"
6482024-02-03T11:21:20.707Z+infra_ip_first = "10.151.2.174"
6492024-02-03T11:21:20.707Z+infra_ip_last = "10.151.2.174"
6502024-02-03T11:21:20.707Z
6512024-02-03T11:21:20.707Z # Configurations for BGP routers to run on the scrimlets.
6522024-02-03T11:21:20.707Z bgp = []
6532024-02-03T11:21:20.707Z@@ -101,9 +101,9 @@
6542024-02-03T11:21:20.707Z # You can configure multiple uplinks by repeating the following stanza
6552024-02-03T11:21:20.707Z [[rack_network_config.ports]]
6562024-02-03T11:21:20.707Z # Routes associated with this port.
6572024-02-03T11:21:20.707Z-routes = [{nexthop = "192.168.1.199", destination = "0.0.0.0/0"}]
6582024-02-03T11:21:20.707Z+routes = [{nexthop = "10.151.2.100", destination = "0.0.0.0/0"}]
6592024-02-03T11:21:20.707Z # Addresses associated with this port.
6602024-02-03T11:21:20.707Z-addresses = ["192.168.1.30/24"]
6612024-02-03T11:21:20.707Z+addresses = ["10.151.2.174/24"]
6622024-02-03T11:21:20.707Z # Name of the uplink port. This should always be "qsfp0" when using softnpu.
6632024-02-03T11:21:20.708Z port = "qsfp0"
6642024-02-03T11:21:20.708Z # The speed of this port.
6652024-02-03T11:21:20.708Z+ true
6662024-02-03T11:21:20.708Z+ tar rvf out/omicron-sled-agent.tar pkg/config-rss.toml
6672024-02-03T11:21:20.708Za pkg/config-rss.toml 7K
6682024-02-03T11:21:20.708Z+ rm -f pkg/config-rss.toml pkg/config-rss.toml~
6692024-02-03T11:21:20.708Z+ TLS_NAME=recovery.sys.oxide.test
6702024-02-03T11:21:20.708Z+ openssl req -newkey rsa:4096 -x509 -sha256 -days 3 -nodes -out pkg/initial-tls-cert.pem -keyout pkg/initial-tls-key.pem -subj /CN=recovery.sys.oxide.test -addext subjectAltName=DNS:recovery.sys.oxide.test -addext basicConstraints=critical,CA:FALSE -config /dev/stdin
6712024-02-03T11:21:22.199Z...+...........+...+.+......+.....+....+...+...........+....+..+.....................+...+....+...+..+.+.....+....+..+....+............+..+.......+...+...+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*......+...+..+.........+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*....+................+..+.......+.....+...+...+.........................+.........+...+.....+..........+.....+...+......+..........+........................+..+.........+.+........+..........+..+.......+............+.....+.............+...+.........+...+............+...........+...+.......+..+............+.............+...........+......+..................+...............+.+.........+.....+...+..........+.........+........+......+....+..+......+....+......+............+...+...+..+............+.......+...+........+.........+....+..............+.+..+...............+....+...+......+.....+............+.......+...........................+...+..+.........+...............+......+................+........+......+...+..........+..+....+......+.........+......+...+...+..............+.........+....+.....+...+.......+..+.+.....+...+....+..+....+..+.........................+..+.+......+......+..+..................+.......+...+.........+...........+.......+.......................+.......+.....+.......+...+..+....+.....+.+..............+....+..+.............+...+..............+......+............+.+.........+.....................+.....................+..............+....+...........................+...+..+................+........+.....................+...+...+.........+...+..........+..+....+.........+............+..............+......................+...............+..+...+....+......+.....+....+...........+...+.......+............+.....................+..............+......+.+..............+......+...+......+.+.....+.........+.........+.......+.........+....................+.+.........+........................+...+..+......+......+.............+...+.........+...+..+......+.......+......+..+..........+....................+.........+.+........+......+.+..............+.+.................+......+.......+...+............+..+.......+.....+.........+....+........+.......+...............+............+...............+...+...........+......+....+..............+............+...+......+............+..........+.....+....+.........+......+........+.......+.....+...+......+......+.+..+...+..........+.....+...+......+......+...+.......+...+...........+.........+....+...........+.......+...+.....................+.....+.+.....+.......+........+.........+...+.......+.....+..................+.......+........+......+.......+...+......+..............+......+.+............+...+...+..+......................+...+...+.............................+..................+....+..................+..+...+...+.......+........+....+......+.....+.......+..+...............+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
6722024-02-03T11:21:22.597Z......+.+......+...+...........+.+....................+.+..+...+.......+..+.+............+.....+.......+...+..+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*..+........+....+...+..+.......+...........+....+...+........+...+...............+.+.....+...+...+...+....+........+....+...+...............+..+..........+.....+.........+.+........+.+.....+......+...+............+................+..+................+.........+............+........+.+.....+.+...+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.........+...+...+..+.............+......+......+...+............+.........+.....+......+.......+..........................+.........+...+..................+.+......+..............+..........+...........+..........+............+...+...+..+.........+...+...+............+...+.......+...........+..........+...........+..........+...+..................+.....+.......+............+...+..............................+...+...+.....+......+.+..+...............+.......+...+........+...+....+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
6732024-02-03T11:21:22.628Z-----
6742024-02-03T11:21:22.628Z+ tar rvf out/omicron-sled-agent.tar pkg/initial-tls-cert.pem pkg/initial-tls-key.pem
6752024-02-03T11:21:22.628Za pkg/initial-tls-cert.pem 2K
6762024-02-03T11:21:22.629Za pkg/initial-tls-key.pem 4K
6772024-02-03T11:21:22.629Z+ rm -f pkg/initial-tls-cert.pem pkg/initial-tls-key.pem
6782024-02-03T11:21:22.629Z+ rmdir pkg
6792024-02-03T11:21:22.629Z+ E2E_TLS_CERT=/opt/oxide/sled-agent/pkg/initial-tls-cert.pem
6802024-02-03T11:21:22.629Z+ pfexec mkdir -p /usr/oxide
6812024-02-03T11:21:22.629Z+ pfexec curl -sSfL -o /usr/oxide/oxide http://catacomb.eng.oxide.computer:12346/oxide-v0.1.0
6822024-02-03T11:21:23.536Z+ pfexec chmod +x /usr/oxide/oxide
6832024-02-03T11:21:23.559Z+ curl -sSfL -o debian-11-genericcloud-amd64.raw http://catacomb.eng.oxide.computer:12346/debian-11-genericcloud-amd64.raw
6842024-02-03T11:21:43.693Z+ OMICRON_NO_UNINSTALL=1
6852024-02-03T11:21:43.693Z+ ptime -m pfexec ./target/release/omicron-package -t test install
6862024-02-03T11:21:43.726ZLogging to: /opt/oxide/work/out/LOG
6872024-02-03T11:21:44.238Z
6882024-02-03T11:21:44.263Zreal 0.537493621
6892024-02-03T11:21:44.263Zuser 0.108081885
6902024-02-03T11:21:44.263Zsys 2.334946127
6912024-02-03T11:21:44.263Ztrap 0.000006449
6922024-02-03T11:21:44.264Ztflt 0.000076579
6932024-02-03T11:21:44.264Zdflt 0.000438525
6942024-02-03T11:21:44.264Zkflt 0.000000000
6952024-02-03T11:21:44.264Zlock 48.711704969
6962024-02-03T11:21:44.264Zslp 0.746323580
6972024-02-03T11:21:44.264Zlat 0.003095397
6982024-02-03T11:21:44.265Zstop 0.212482925
6992024-02-03T11:21:44.265Z+ retry=0
7002024-02-03T11:21:44.265Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7012024-02-03T11:21:44.265Z+ [[ 0 -gt 30 ]]
7022024-02-03T11:21:44.265Z+ sleep 1
7032024-02-03T11:21:45.246Z+ retry=1
7042024-02-03T11:21:45.246Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7052024-02-03T11:21:45.270Z+ [[ 1 -gt 30 ]]
7062024-02-03T11:21:45.270Z+ sleep 1
7072024-02-03T11:21:46.259Z+ retry=2
7082024-02-03T11:21:46.259Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7092024-02-03T11:21:46.283Z+ [[ 2 -gt 30 ]]
7102024-02-03T11:21:46.283Z+ sleep 1
7112024-02-03T11:21:47.275Z+ retry=3
7122024-02-03T11:21:47.275Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7132024-02-03T11:21:47.302Z+ [[ 3 -gt 30 ]]
7142024-02-03T11:21:47.303Z+ sleep 1
7152024-02-03T11:21:48.290Z+ retry=4
7162024-02-03T11:21:48.290Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7172024-02-03T11:22:13.464Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone dump-state
7182024-02-03T11:22:13.493Zlocal v6:
7192024-02-03T11:22:13.494Zfe80::144b:bff:fe0b:dff8
7202024-02-03T11:22:13.494Zfdd5:b536:758f:7d49::1
7212024-02-03T11:22:13.494Zlocal v4:
7222024-02-03T11:22:13.494Z10.151.2.174
7232024-02-03T11:22:13.494Zrouter v6:
7242024-02-03T11:22:13.494Zfd00:1122:3344:3::/64 -> fe80::f87d:7bff:fedb:7ab6 (1)
7252024-02-03T11:22:13.495Zfdb0:18c0:4d0d:a02a::/64 -> fe80::f87d:7bff:fedb:7ab6 (1)
7262024-02-03T11:22:13.495Zfd00:1122:3344:1::/64 -> fe80::f87d:7bff:fedb:7ab6 (1)
7272024-02-03T11:22:13.495Zfd00:1122:3344:101::/64 -> fe80::f87d:7bff:fedb:7ab6 (1)
7282024-02-03T11:22:13.495Zfd00:1122:3344:2::/64 -> fe80::f87d:7bff:fedb:7ab6 (1)
7292024-02-03T11:22:13.495Zrouter v4_idx:
7302024-02-03T11:22:13.495Z0.0.0.0/0 -> 0
7312024-02-03T11:22:13.495Zrouter v4_routes:
7322024-02-03T11:22:13.496Z0 -> 10.151.2.100 (2)
7332024-02-03T11:22:13.496Zresolver v4:
7342024-02-03T11:22:13.496Z10.151.2.100 -> 18:c0:4d:0d:a0:2a
7352024-02-03T11:22:13.496Z10.151.2.174 -> 46:e5:8a:bb:b5:db
7362024-02-03T11:22:13.496Zresolver v6:
7372024-02-03T11:22:13.496Zfe80::144b:bff:fe0b:dff8 -> 16:4b:0b:0b:df:f8
7382024-02-03T11:22:13.496Zfe80::f87d:7bff:fedb:7ab6 -> fa:7d:7b:db:7a:b6
7392024-02-03T11:22:13.496Znat v4:
7402024-02-03T11:22:13.497Z10.151.2.169 0/16383 -> fd00:1122:3344:101::1 100/a8:40:25:ff:dc:2a
7412024-02-03T11:22:13.497Znat_v6:
7422024-02-03T11:22:13.497Zport_mac:
7432024-02-03T11:22:13.497Z1000: a8:40:25:43:01:c5
7442024-02-03T11:22:13.497Z1: a8:40:25:43:01:c6
7452024-02-03T11:22:13.498Zproxy arp:
7462024-02-03T11:22:13.498Z10.151.2.164/10.151.2.227: a8:e1:de:01:70:1d
7472024-02-03T11:22:13.525Z+ retry=0
7482024-02-03T11:22:13.525Z+++ zoneadm list -n
7492024-02-03T11:22:13.526Z+++ grep oxz_ntp
7502024-02-03T11:22:13.526Z++ pfexec svcs -z oxz_ntp_e0b5cbda-123e-4edd-a832-85e3c088469a -Hostate oxide/ntp
7512024-02-03T11:22:13.526Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
7522024-02-03T11:22:13.526Z++ true
7532024-02-03T11:22:13.526Z+ [[ '' != online ]]
7542024-02-03T11:22:13.526Z+ [[ 0 -gt 60 ]]
7552024-02-03T11:22:13.526Z+ sleep 1
7562024-02-03T11:22:14.537Z+ retry=1
7572024-02-03T11:22:14.619Z+++ zoneadm list -n
7582024-02-03T11:22:14.619Z+++ grep oxz_ntp
7592024-02-03T11:22:14.648Z++ pfexec svcs -z oxz_ntp_e0b5cbda-123e-4edd-a832-85e3c088469a -Hostate oxide/ntp
7602024-02-03T11:22:14.649Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
7612024-02-03T11:22:14.649Z++ true
7622024-02-03T11:22:14.649Z+ [[ '' != online ]]
7632024-02-03T11:22:14.649Z+ [[ 1 -gt 60 ]]
7642024-02-03T11:22:14.649Z+ sleep 1
7652024-02-03T11:22:15.562Z+ retry=2
7662024-02-03T11:22:15.600Z+++ zoneadm list -n
7672024-02-03T11:22:15.600Z+++ grep oxz_ntp
7682024-02-03T11:22:15.600Z++ pfexec svcs -z oxz_ntp_e0b5cbda-123e-4edd-a832-85e3c088469a -Hostate oxide/ntp
7692024-02-03T11:22:15.600Z+ [[ disabled != online ]]
7702024-02-03T11:22:15.600Z+ [[ 2 -gt 60 ]]
7712024-02-03T11:22:15.600Z+ sleep 1
7722024-02-03T11:22:16.581Z+ retry=3
7732024-02-03T11:22:16.607Z+++ zoneadm list -n
7742024-02-03T11:22:16.607Z+++ grep oxz_ntp
7752024-02-03T11:22:16.607Z++ pfexec svcs -z oxz_ntp_e0b5cbda-123e-4edd-a832-85e3c088469a -Hostate oxide/ntp
7762024-02-03T11:22:16.607Z+ [[ disabled != online ]]
7772024-02-03T11:22:16.607Z+ [[ 3 -gt 60 ]]
7782024-02-03T11:22:16.607Z+ sleep 1
7792024-02-03T11:22:17.600Z+ retry=4
7802024-02-03T11:22:17.624Z+++ zoneadm list -n
7812024-02-03T11:22:17.625Z+++ grep oxz_ntp
7822024-02-03T11:22:17.625Z++ pfexec svcs -z oxz_ntp_e0b5cbda-123e-4edd-a832-85e3c088469a -Hostate oxide/ntp
7832024-02-03T11:22:17.625Z+ [[ disabled != online ]]
7842024-02-03T11:22:17.625Z+ [[ 4 -gt 60 ]]
7852024-02-03T11:22:17.625Z+ sleep 1
7862024-02-03T11:22:18.630Z+ retry=5
7872024-02-03T11:22:18.671Z+++ zoneadm list -n
7882024-02-03T11:22:18.671Z+++ grep oxz_ntp
7892024-02-03T11:22:18.671Z++ pfexec svcs -z oxz_ntp_e0b5cbda-123e-4edd-a832-85e3c088469a -Hostate oxide/ntp
7902024-02-03T11:22:18.671Z+ [[ disabled != online ]]
7912024-02-03T11:22:18.671Z+ [[ 5 -gt 60 ]]
7922024-02-03T11:22:18.671Z+ sleep 1
7932024-02-03T11:22:19.644Z+ retry=6
7942024-02-03T11:22:19.668Z+++ zoneadm list -n
7952024-02-03T11:22:19.668Z+++ grep oxz_ntp
7962024-02-03T11:22:19.668Z++ pfexec svcs -z oxz_ntp_e0b5cbda-123e-4edd-a832-85e3c088469a -Hostate oxide/ntp
7972024-02-03T11:22:19.668Z+ [[ disabled != online ]]
7982024-02-03T11:22:19.668Z+ [[ 6 -gt 60 ]]
7992024-02-03T11:22:19.668Z+ sleep 1
8002024-02-03T11:22:20.667Z+ retry=7
8012024-02-03T11:22:20.692Z+++ zoneadm list -n
8022024-02-03T11:22:20.692Z+++ grep oxz_ntp
8032024-02-03T11:22:20.692Z++ pfexec svcs -z oxz_ntp_e0b5cbda-123e-4edd-a832-85e3c088469a -Hostate oxide/ntp
8042024-02-03T11:22:20.692Z+ [[ disabled != online ]]
8052024-02-03T11:22:20.692Z+ [[ 7 -gt 60 ]]
8062024-02-03T11:22:20.692Z+ sleep 1
8072024-02-03T11:22:21.688Z+ retry=8
8082024-02-03T11:22:21.713Z+++ zoneadm list -n
8092024-02-03T11:22:21.713Z+++ grep oxz_ntp
8102024-02-03T11:22:21.713Z++ pfexec svcs -z oxz_ntp_e0b5cbda-123e-4edd-a832-85e3c088469a -Hostate oxide/ntp
8112024-02-03T11:22:21.713Z+ [[ disabled != online ]]
8122024-02-03T11:22:21.713Z+ [[ 8 -gt 60 ]]
8132024-02-03T11:22:21.713Z+ sleep 1
 
8142024-02-03T11:22:22.333ZFeb 3 11:22:22 sock ip: ip: joining multicasts failed (4) on vopte0 - will use link layer broadcasts for multicast
 
8152024-02-03T11:22:22.710Z+ retry=9
8162024-02-03T11:22:22.734Z+++ zoneadm list -n
8172024-02-03T11:22:22.735Z+++ grep oxz_ntp
8182024-02-03T11:22:22.735Z++ pfexec svcs -z oxz_ntp_e0b5cbda-123e-4edd-a832-85e3c088469a -Hostate oxide/ntp
8192024-02-03T11:22:22.735Z+ [[ offline* != online ]]
8202024-02-03T11:22:22.735Z+ [[ 9 -gt 60 ]]
8212024-02-03T11:22:22.735Z+ sleep 1
8222024-02-03T11:22:23.734Z+ retry=10
8232024-02-03T11:22:23.758Z+++ zoneadm list -n
8242024-02-03T11:22:23.758Z+++ grep oxz_ntp
8252024-02-03T11:22:23.758Z++ pfexec svcs -z oxz_ntp_e0b5cbda-123e-4edd-a832-85e3c088469a -Hostate oxide/ntp
8262024-02-03T11:22:23.758Z+ [[ online != online ]]
8272024-02-03T11:22:23.758Z+ echo 'Waited for chrony: 10s'
8282024-02-03T11:22:23.758Z+ export RUST_BACKTRACE=1
8292024-02-03T11:22:23.758ZWaited for chrony: 10s
8302024-02-03T11:22:23.758Z+ RUST_BACKTRACE=1
8312024-02-03T11:22:23.758Z+ export E2E_TLS_CERT IPPOOL_START IPPOOL_END
8322024-02-03T11:22:23.758Z++ ./tests/bootstrap
8332024-02-03T11:22:23.782ZUsing external DNS server at 10.151.2.164:53
8342024-02-03T11:22:23.806Z2024-02-03 11:22:24.310332762 UTC: attempting to log into API
8352024-02-03T11:22:38.822Z2024-02-03 11:22:38.894676615 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8362024-02-03T11:22:39.830Z2024-02-03 11:22:39.896510730 UTC: attempting to log into API
8372024-02-03T11:22:54.860Z2024-02-03 11:22:54.930558711 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8382024-02-03T11:22:55.860Z2024-02-03 11:22:55.932500469 UTC: attempting to log into API
8392024-02-03T11:23:10.894Z2024-02-03 11:23:10.925926138 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8402024-02-03T11:23:11.896Z2024-02-03 11:23:11.925125685 UTC: attempting to log into API
8412024-02-03T11:23:26.929Z2024-02-03 11:23:26.917758851 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8422024-02-03T11:23:27.934Z2024-02-03 11:23:27.919120520 UTC: attempting to log into API
8432024-02-03T11:23:42.966Z2024-02-03 11:23:42.952217391 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8442024-02-03T11:23:43.965Z2024-02-03 11:23:43.953909322 UTC: attempting to log into API
8452024-02-03T11:23:58.998Z2024-02-03 11:23:58.986917591 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8462024-02-03T11:24:00.002Z2024-02-03 11:23:59.987667892 UTC: attempting to log into API
8472024-02-03T11:24:15.033Z2024-02-03 11:24:15.021754407 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8482024-02-03T11:24:16.034Z2024-02-03 11:24:16.022714382 UTC: attempting to log into API
 
8492024-02-03T11:24:37.159ZError: No response to keepalive - Terminating session
8502024-02-03T11:24:37.228ZError: No response de-activating SOL payload
8512024-02-03T11:24:37.289Z[SOL Session operational. Use ~? for help]
 
8522024-02-03T11:24:37.320Z2024-02-03 11:24:31.167986824 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8532024-02-03T11:24:37.320Z2024-02-03 11:24:32.169489771 UTC: attempting to log into API
8542024-02-03T11:24:47.506Z2024-02-03 11:24:47.496040079 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8552024-02-03T11:24:48.508Z2024-02-03 11:24:48.497803773 UTC: attempting to log into API
8562024-02-03T11:25:03.544Z2024-02-03 11:25:03.533676774 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8572024-02-03T11:25:04.560Z2024-02-03 11:25:04.534381863 UTC: attempting to log into API
8582024-02-03T11:25:19.579Z2024-02-03 11:25:19.569379785 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8592024-02-03T11:25:20.581Z2024-02-03 11:25:20.570339491 UTC: attempting to log into API
8602024-02-03T11:25:35.613Z2024-02-03 11:25:35.602805457 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8612024-02-03T11:25:36.614Z2024-02-03 11:25:36.603967837 UTC: attempting to log into API
 
8622024-02-03T11:25:50.008ZFeb 3 11:25:49 sock ip: ip: joining multicasts failed (4) on vopte4 - will use link layer broadcasts for multicast
 
8632024-02-03T11:25:51.657Z2024-02-03 11:25:51.637736761 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8642024-02-03T11:25:52.648Z2024-02-03 11:25:52.638704337 UTC: attempting to log into API
 
8652024-02-03T11:25:56.661ZFeb 3 11:25:56 sock ip: ip: joining multicasts failed (4) on vopte2 - will use link layer broadcasts for multicast
 
8662024-02-03T11:26:07.682Z2024-02-03 11:26:07.672323339 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8672024-02-03T11:26:08.684Z2024-02-03 11:26:08.674293686 UTC: attempting to log into API
8682024-02-03T11:26:18.731Z2024-02-03 11:26:18.719926737 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: dns error: no record found for Query { name: Name("recovery.sys.oxide.test."), query_type: AAAA, query_class: IN }
8692024-02-03T11:26:19.731Z2024-02-03 11:26:19.721238950 UTC: attempting to log into API
8702024-02-03T11:26:20.505Z2024-02-03 11:26:20.495441345 UTC: login succeeded
8712024-02-03T11:26:20.538Zwaiting for nexus to come up...
8722024-02-03T11:26:20.613Zcreating IP pool... 10.151.2.175 - 10.151.2.227
8732024-02-03T11:26:20.801Zsetting up quotas...
8742024-02-03T11:26:20.848Zensuring datasets are ready...
8752024-02-03T11:26:23.161Zlet's roll.
8762024-02-03T11:26:23.185Z+ eval 'OXIDE_HOST=https://recovery.sys.oxide.test
8772024-02-03T11:26:23.185ZOXIDE_RESOLVE=recovery.sys.oxide.test:443:10.151.2.166
8782024-02-03T11:26:23.185ZOXIDE_TOKEN=oxide-token-21411bbf54decb5a780d179bc1e592462e28ca92'
8792024-02-03T11:26:23.185Z++ OXIDE_HOST=https://recovery.sys.oxide.test
8802024-02-03T11:26:23.185Z++ OXIDE_RESOLVE=recovery.sys.oxide.test:443:10.151.2.166
8812024-02-03T11:26:23.185Z++ OXIDE_TOKEN=oxide-token-21411bbf54decb5a780d179bc1e592462e28ca92
8822024-02-03T11:26:23.185Z+ export OXIDE_HOST OXIDE_TOKEN
8832024-02-03T11:26:23.185Z+ retry=0
8842024-02-03T11:26:23.185Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
8852024-02-03T11:26:23.252Z+ /usr/oxide/oxide --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem project create --name images --description 'some images'
8862024-02-03T11:26:23.573Z{"status":"ok"}success
8872024-02-03T11:26:23.573ZProject {
8882024-02-03T11:26:23.573Z description: "some images",
8892024-02-03T11:26:23.574Z id: 4abe9bff-a6fe-49c6-90f5-222dcfe61e00,
8902024-02-03T11:26:23.574Z name: Name(
8912024-02-03T11:26:23.574Z "images",
8922024-02-03T11:26:23.574Z ),
8932024-02-03T11:26:23.574Z time_created: 2024-02-03T11:26:23.316176Z,
8942024-02-03T11:26:23.574Z time_modified: 2024-02-03T11:26:23.316176Z,
8952024-02-03T11:26:23.574Z}
8962024-02-03T11:26:23.598Z+ /usr/oxide/oxide --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem disk import --path debian-11-genericcloud-amd64.raw --disk debian11-boot --project images --description 'debian 11 cloud image from distros' --snapshot debian11-snapshot --image debian11 --image-description 'debian 11 original base image' --image-os debian --image-version 11
 
8972024-02-03T11:27:22.514ZError: No response to keepalive - Terminating session
8982024-02-03T11:27:32.336ZError: No response de-activating SOL payload
8992024-02-03T11:27:33.694Z[SOL Session operational. Use ~? for help]
9002024-02-03T11:30:22.293ZError: No response to keepalive - Terminating session
9012024-02-03T11:30:32.358ZError: No response de-activating SOL payload
9022024-02-03T11:30:33.471Z[SOL Session operational. Use ~? for help]
 
9032024-02-03T11:31:36.568Zdone!
9042024-02-03T11:31:36.653Z+ /usr/oxide/oxide --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem image promote --project images --image debian11
9052024-02-03T11:31:37.138Zsuccess
9062024-02-03T11:31:37.138ZImage {
9072024-02-03T11:31:37.138Z block_size: ByteCount(
9082024-02-03T11:31:37.138Z 512,
9092024-02-03T11:31:37.138Z ),
9102024-02-03T11:31:37.138Z description: "debian 11 original base image",
9112024-02-03T11:31:37.138Z digest: None,
9122024-02-03T11:31:37.138Z id: aff3dc30-b68e-4e0b-82c9-f5610e01277a,
9132024-02-03T11:31:37.138Z name: Name(
9142024-02-03T11:31:37.139Z "debian11",
9152024-02-03T11:31:37.139Z ),
9162024-02-03T11:31:37.139Z os: "debian",
9172024-02-03T11:31:37.139Z project_id: None,
9182024-02-03T11:31:37.139Z size: ByteCount(
9192024-02-03T11:31:37.139Z 2147483648,
9202024-02-03T11:31:37.139Z ),
9212024-02-03T11:31:37.139Z time_created: 2024-02-03T11:31:36.508023Z,
9222024-02-03T11:31:37.139Z time_modified: 2024-02-03T11:31:36.742540Z,
9232024-02-03T11:31:37.139Z url: None,
9242024-02-03T11:31:37.139Z version: "11",
9252024-02-03T11:31:37.139Z}
9262024-02-03T11:31:37.139Z+ rm ./tests/bootstrap
9272024-02-03T11:31:37.139Z+ for test_bin in tests/*
9282024-02-03T11:31:37.139Z+ ./tests/bootstrap-ba0a15fa1ac0f64d
9292024-02-03T11:31:37.139Z
9302024-02-03T11:31:37.139Zrunning 0 tests
9312024-02-03T11:31:37.139Z
9322024-02-03T11:31:37.139Ztest result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
9332024-02-03T11:31:37.139Z
9342024-02-03T11:31:37.139Z+ for test_bin in tests/*
9352024-02-03T11:31:37.139Z+ ./tests/end_to_end_tests-49aca9644d961704
9362024-02-03T11:31:37.165Z
9372024-02-03T11:31:37.165Zrunning 2 tests
9382024-02-03T11:31:37.374Ztest no_spoof::no_spoof_here ... ok
9392024-02-03T11:32:30.071Ztest instance_launch::instance_launch ... ok
9402024-02-03T11:32:30.071Z
9412024-02-03T11:32:30.071Ztest result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 39.25s
9422024-02-03T11:32:30.071Z
9432024-02-03T11:32:30.071Z+ _exit_trap
9442024-02-03T11:32:30.071Z+ local status=0
9452024-02-03T11:32:30.071Z+ [[ 0 -eq 0 ]]
9462024-02-03T11:32:30.071Z+ exit 0
9472024-02-03T11:32:30.071Zprocess exited: duration 741262 ms, exit code 0
 
9482024-02-03T11:32:30.185Zfound 131 output files
9492024-02-03T11:32:30.185Zuploading: /var/svc/log/oxide-sled-agent:default.log (836575 bytes)
9502024-02-03T11:32:31.232Zupload warning: file "/var/svc/log/oxide-sled-agent:default.log" changed size mid upload: 836575 -> 836985
9512024-02-03T11:32:32.303Zuploaded: /var/svc/log/oxide-sled-agent:default.log
9522024-02-03T11:32:32.304Zuploading: /zone/oxz_switch/root/var/svc/log/oxide-dendrite:default.log (637512 bytes)
9532024-02-03T11:32:33.636Zupload warning: file "/zone/oxz_switch/root/var/svc/log/oxide-dendrite:default.log" changed size mid upload: 637512 -> 651620
9542024-02-03T11:32:34.715Zuploaded: /zone/oxz_switch/root/var/svc/log/oxide-dendrite:default.log
9552024-02-03T11:32:34.715Zuploading: /zone/oxz_switch/root/var/svc/log/oxide-dendrite:default.log.0 (2053223 bytes)
9562024-02-03T11:32:38.259Zuploaded: /zone/oxz_switch/root/var/svc/log/oxide-dendrite:default.log.0
9572024-02-03T11:32:38.260Zuploading: /zone/oxz_switch/root/var/svc/log/oxide-mg-ddm:default.log (20695 bytes)
9582024-02-03T11:32:38.400Zupload warning: file "/zone/oxz_switch/root/var/svc/log/oxide-mg-ddm:default.log" changed size mid upload: 20695 -> 21944
9592024-02-03T11:32:39.472Zuploaded: /zone/oxz_switch/root/var/svc/log/oxide-mg-ddm:default.log
9602024-02-03T11:32:39.497Zuploading: /zone/oxz_switch/root/var/svc/log/oxide-mg-ddm:default.log.0 (73681 bytes)
9612024-02-03T11:32:40.786Zuploaded: /zone/oxz_switch/root/var/svc/log/oxide-mg-ddm:default.log.0
9622024-02-03T11:32:40.786Zuploading: /zone/oxz_switch/root/var/svc/log/oxide-mgd:default.log (27403 bytes)
9632024-02-03T11:32:40.919Zupload warning: file "/zone/oxz_switch/root/var/svc/log/oxide-mgd:default.log" changed size mid upload: 27403 -> 29389
9642024-02-03T11:32:41.989Zuploaded: /zone/oxz_switch/root/var/svc/log/oxide-mgd:default.log
9652024-02-03T11:32:41.989Zuploading: /zone/oxz_switch/root/var/svc/log/oxide-mgd:default.log.0 (88150 bytes)
9662024-02-03T11:32:43.327Zuploaded: /zone/oxz_switch/root/var/svc/log/oxide-mgd:default.log.0
9672024-02-03T11:32:43.327Zuploading: /zone/oxz_switch/root/var/svc/log/oxide-mgs:default.log (299491 bytes)
9682024-02-03T11:32:43.962Zupload warning: file "/zone/oxz_switch/root/var/svc/log/oxide-mgs:default.log" changed size mid upload: 299491 -> 321055
9692024-02-03T11:32:45.076Zuploaded: /zone/oxz_switch/root/var/svc/log/oxide-mgs:default.log
9702024-02-03T11:32:45.076Zuploading: /zone/oxz_switch/root/var/svc/log/oxide-mgs:default.log.0 (1080774 bytes)
9712024-02-03T11:32:47.726Zuploaded: /zone/oxz_switch/root/var/svc/log/oxide-mgs:default.log.0
9722024-02-03T11:32:47.750Zuploading: /zone/oxz_switch/root/var/svc/log/oxide-sp-sim:default.log (190360 bytes)
9732024-02-03T11:32:48.208Zupload warning: file "/zone/oxz_switch/root/var/svc/log/oxide-sp-sim:default.log" changed size mid upload: 190360 -> 209042
9742024-02-03T11:32:49.282Zuploaded: /zone/oxz_switch/root/var/svc/log/oxide-sp-sim:default.log
9752024-02-03T11:32:49.282Zuploading: /zone/oxz_switch/root/var/svc/log/oxide-sp-sim:default.log.0 (649601 bytes)
9762024-02-03T11:32:51.234Zuploaded: /zone/oxz_switch/root/var/svc/log/oxide-sp-sim:default.log.0
9772024-02-03T11:32:51.234Zuploading: /zone/oxz_switch/root/var/svc/log/oxide-switch_zone_setup:default.log (0 bytes)
9782024-02-03T11:32:52.304Zuploaded: /zone/oxz_switch/root/var/svc/log/oxide-switch_zone_setup:default.log
9792024-02-03T11:32:52.304Zuploading: /zone/oxz_switch/root/var/svc/log/oxide-switch_zone_setup:default.log.0 (1388 bytes)
9802024-02-03T11:32:53.399Zuploaded: /zone/oxz_switch/root/var/svc/log/oxide-switch_zone_setup:default.log.0
9812024-02-03T11:32:53.399Zuploading: /zone/oxz_switch/root/var/svc/log/oxide-tfport:default.log (72037 bytes)
9822024-02-03T11:32:53.704Zupload warning: file "/zone/oxz_switch/root/var/svc/log/oxide-tfport:default.log" changed size mid upload: 72037 -> 84129
9832024-02-03T11:32:54.776Zuploaded: /zone/oxz_switch/root/var/svc/log/oxide-tfport:default.log
9842024-02-03T11:32:54.776Zuploading: /zone/oxz_switch/root/var/svc/log/oxide-tfport:default.log.0 (234091 bytes)
9852024-02-03T11:32:56.846Zuploaded: /zone/oxz_switch/root/var/svc/log/oxide-tfport:default.log.0
9862024-02-03T11:32:56.846Zuploading: /zone/oxz_switch/root/var/svc/log/oxide-uplink:default.log (0 bytes)
9872024-02-03T11:32:57.919Zuploaded: /zone/oxz_switch/root/var/svc/log/oxide-uplink:default.log
9882024-02-03T11:32:57.919Zuploading: /zone/oxz_switch/root/var/svc/log/oxide-uplink:default.log.0 (617 bytes)
9892024-02-03T11:32:59.013Zuploaded: /zone/oxz_switch/root/var/svc/log/oxide-uplink:default.log.0
9902024-02-03T11:32:59.013Zuploading: /zone/oxz_switch/root/var/svc/log/oxide-wicketd:default.log (3935 bytes)
9912024-02-03T11:32:59.060Zupload warning: file "/zone/oxz_switch/root/var/svc/log/oxide-wicketd:default.log" changed size mid upload: 3935 -> 4722
9922024-02-03T11:33:00.248Zuploaded: /zone/oxz_switch/root/var/svc/log/oxide-wicketd:default.log
9932024-02-03T11:33:00.248Zuploading: /zone/oxz_switch/root/var/svc/log/oxide-wicketd:default.log.0 (17982 bytes)
9942024-02-03T11:33:01.388Zuploaded: /zone/oxz_switch/root/var/svc/log/oxide-wicketd:default.log.0
9952024-02-03T11:33:01.413Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_clickhouse_ab2d803a-f891-40c3-8fb6-9aaf62d2a58f/root/var/svc/log/oxide-clickhouse:default.log (428388 bytes)
9962024-02-03T11:33:02.182Zupload warning: file "/pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_clickhouse_ab2d803a-f891-40c3-8fb6-9aaf62d2a58f/root/var/svc/log/oxide-clickhouse:default.log" changed size mid upload: 428388 -> 521932
9972024-02-03T11:33:03.257Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_clickhouse_ab2d803a-f891-40c3-8fb6-9aaf62d2a58f/root/var/svc/log/oxide-clickhouse:default.log
9982024-02-03T11:33:03.284Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_clickhouse_ab2d803a-f891-40c3-8fb6-9aaf62d2a58f/root/var/svc/log/oxide-clickhouse:default.log.0 (693044 bytes)
9992024-02-03T11:33:05.216Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_clickhouse_ab2d803a-f891-40c3-8fb6-9aaf62d2a58f/root/var/svc/log/oxide-clickhouse:default.log.0
10002024-02-03T11:33:05.243Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_clickhouse_ab2d803a-f891-40c3-8fb6-9aaf62d2a58f/root/var/svc/log/oxide-zone-network-setup:default.log (0 bytes)
10012024-02-03T11:33:06.288Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_clickhouse_ab2d803a-f891-40c3-8fb6-9aaf62d2a58f/root/var/svc/log/oxide-zone-network-setup:default.log
10022024-02-03T11:33:06.288Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_clickhouse_ab2d803a-f891-40c3-8fb6-9aaf62d2a58f/root/var/svc/log/oxide-zone-network-setup:default.log.0 (1708 bytes)
10032024-02-03T11:33:07.384Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_clickhouse_ab2d803a-f891-40c3-8fb6-9aaf62d2a58f/root/var/svc/log/oxide-zone-network-setup:default.log.0
10042024-02-03T11:33:07.384Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_e37d5382-9deb-4536-98a0-6e3fcce73e5b/root/var/svc/log/oxide-cockroachdb:default.log (0 bytes)
10052024-02-03T11:33:08.458Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_e37d5382-9deb-4536-98a0-6e3fcce73e5b/root/var/svc/log/oxide-cockroachdb:default.log
10062024-02-03T11:33:08.458Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_e37d5382-9deb-4536-98a0-6e3fcce73e5b/root/var/svc/log/oxide-cockroachdb:default.log.0 (4482 bytes)
10072024-02-03T11:33:09.556Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_e37d5382-9deb-4536-98a0-6e3fcce73e5b/root/var/svc/log/oxide-cockroachdb:default.log.0
10082024-02-03T11:33:09.556Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_e37d5382-9deb-4536-98a0-6e3fcce73e5b/root/var/svc/log/oxide-zone-network-setup:default.log (0 bytes)
10092024-02-03T11:33:10.626Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_e37d5382-9deb-4536-98a0-6e3fcce73e5b/root/var/svc/log/oxide-zone-network-setup:default.log
10102024-02-03T11:33:10.626Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_e37d5382-9deb-4536-98a0-6e3fcce73e5b/root/var/svc/log/oxide-zone-network-setup:default.log.0 (1711 bytes)
10112024-02-03T11:33:11.725Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_e37d5382-9deb-4536-98a0-6e3fcce73e5b/root/var/svc/log/oxide-zone-network-setup:default.log.0
10122024-02-03T11:33:11.725Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_41d21607-a07f-44a0-aaa9-b8777449bdbf/root/var/svc/log/oxide-crucible-agent:default.log (22670 bytes)
10132024-02-03T11:33:12.884Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_41d21607-a07f-44a0-aaa9-b8777449bdbf/root/var/svc/log/oxide-crucible-agent:default.log
10142024-02-03T11:33:12.884Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_41d21607-a07f-44a0-aaa9-b8777449bdbf/root/var/svc/log/oxide-crucible-agent:default.log.0 (3436 bytes)
10152024-02-03T11:33:13.977Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_41d21607-a07f-44a0-aaa9-b8777449bdbf/root/var/svc/log/oxide-crucible-agent:default.log.0
10162024-02-03T11:33:14.003Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_41d21607-a07f-44a0-aaa9-b8777449bdbf/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
10172024-02-03T11:33:15.053Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_41d21607-a07f-44a0-aaa9-b8777449bdbf/root/var/svc/log/oxide-crucible-downstairs:default.log
10182024-02-03T11:33:15.053Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_41d21607-a07f-44a0-aaa9-b8777449bdbf/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
10192024-02-03T11:33:16.149Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_41d21607-a07f-44a0-aaa9-b8777449bdbf/root/var/svc/log/oxide-crucible-downstairs:default.log.0
10202024-02-03T11:33:16.149Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_41d21607-a07f-44a0-aaa9-b8777449bdbf/root/var/svc/log/oxide-crucible-downstairs:downstairs-00b4d053-bb86-404f-b1fa-103097e7d63a.log (5516 bytes)
10212024-02-03T11:33:17.256Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_41d21607-a07f-44a0-aaa9-b8777449bdbf/root/var/svc/log/oxide-crucible-downstairs:downstairs-00b4d053-bb86-404f-b1fa-103097e7d63a.log
10222024-02-03T11:33:17.257Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_41d21607-a07f-44a0-aaa9-b8777449bdbf/root/var/svc/log/oxide-crucible-downstairs:downstairs-57d31acb-a070-42b1-8fc8-f9df8b91acb5.log (2933 bytes)
10232024-02-03T11:33:18.353Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_41d21607-a07f-44a0-aaa9-b8777449bdbf/root/var/svc/log/oxide-crucible-downstairs:downstairs-57d31acb-a070-42b1-8fc8-f9df8b91acb5.log
10242024-02-03T11:33:18.353Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_335aae10-5af1-4a7f-b5ba-a8099285b185/root/var/svc/log/oxide-crucible-pantry:default.log (0 bytes)
10252024-02-03T11:33:19.424Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_335aae10-5af1-4a7f-b5ba-a8099285b185/root/var/svc/log/oxide-crucible-pantry:default.log
10262024-02-03T11:33:19.424Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_335aae10-5af1-4a7f-b5ba-a8099285b185/root/var/svc/log/oxide-crucible-pantry:default.log.0 (2338 bytes)
10272024-02-03T11:33:20.520Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_335aae10-5af1-4a7f-b5ba-a8099285b185/root/var/svc/log/oxide-crucible-pantry:default.log.0
10282024-02-03T11:33:20.520Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_4c252ce7-d2d9-4c33-b567-e45974766d27/root/var/svc/log/oxide-external_dns:default.log (11331 bytes)
10292024-02-03T11:33:20.591Zupload warning: file "/pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_4c252ce7-d2d9-4c33-b567-e45974766d27/root/var/svc/log/oxide-external_dns:default.log" changed size mid upload: 11331 -> 15111
10302024-02-03T11:33:21.662Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_4c252ce7-d2d9-4c33-b567-e45974766d27/root/var/svc/log/oxide-external_dns:default.log
10312024-02-03T11:33:21.662Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_4c252ce7-d2d9-4c33-b567-e45974766d27/root/var/svc/log/oxide-external_dns:default.log.0 (18892 bytes)
10322024-02-03T11:33:22.387ZError: No response to keepalive - Terminating session
10332024-02-03T11:33:22.803Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_4c252ce7-d2d9-4c33-b567-e45974766d27/root/var/svc/log/oxide-external_dns:default.log.0
10342024-02-03T11:33:22.803Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_507d4f06-d389-4cde-b882-b5ea3b64c5c9/root/var/svc/log/oxide-internal_dns:default.log (11267 bytes)
10352024-02-03T11:33:22.873Zupload warning: file "/pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_507d4f06-d389-4cde-b882-b5ea3b64c5c9/root/var/svc/log/oxide-internal_dns:default.log" changed size mid upload: 11267 -> 15023
10362024-02-03T11:33:23.946Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_507d4f06-d389-4cde-b882-b5ea3b64c5c9/root/var/svc/log/oxide-internal_dns:default.log
10372024-02-03T11:33:23.946Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_507d4f06-d389-4cde-b882-b5ea3b64c5c9/root/var/svc/log/oxide-internal_dns:default.log.0 (108594 bytes)
10382024-02-03T11:33:25.158Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_507d4f06-d389-4cde-b882-b5ea3b64c5c9/root/var/svc/log/oxide-internal_dns:default.log.0
10392024-02-03T11:33:25.158Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_9438dc2a-a136-472a-a7fb-383743199543/root/var/svc/log/oxide-nexus:default.log (314644 bytes)
10402024-02-03T11:33:25.396Zupload warning: file "/pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_9438dc2a-a136-472a-a7fb-383743199543/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 314644 -> 424674
10412024-02-03T11:33:26.471Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_9438dc2a-a136-472a-a7fb-383743199543/root/var/svc/log/oxide-nexus:default.log
10422024-02-03T11:33:26.471Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_9438dc2a-a136-472a-a7fb-383743199543/root/var/svc/log/oxide-nexus:default.log.0 (934593 bytes)
10432024-02-03T11:33:27.811Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_9438dc2a-a136-472a-a7fb-383743199543/root/var/svc/log/oxide-nexus:default.log.0
10442024-02-03T11:33:27.811Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_e0b5cbda-123e-4edd-a832-85e3c088469a/root/var/svc/log/oxide-ntp:default.log (0 bytes)
10452024-02-03T11:33:28.882Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_e0b5cbda-123e-4edd-a832-85e3c088469a/root/var/svc/log/oxide-ntp:default.log
10462024-02-03T11:33:28.908Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_e0b5cbda-123e-4edd-a832-85e3c088469a/root/var/svc/log/oxide-ntp:default.log.0 (1045 bytes)
10472024-02-03T11:33:29.979Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_e0b5cbda-123e-4edd-a832-85e3c088469a/root/var/svc/log/oxide-ntp:default.log.0
10482024-02-03T11:33:29.980Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_oximeter_1a03d1eb-bc7d-46ef-83a9-40ee65e701ce/root/var/svc/log/oxide-oximeter:default.log (120421 bytes)
10492024-02-03T11:33:30.163Zupload warning: file "/pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_oximeter_1a03d1eb-bc7d-46ef-83a9-40ee65e701ce/root/var/svc/log/oxide-oximeter:default.log" changed size mid upload: 120421 -> 166742
10502024-02-03T11:33:31.234Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_oximeter_1a03d1eb-bc7d-46ef-83a9-40ee65e701ce/root/var/svc/log/oxide-oximeter:default.log
10512024-02-03T11:33:31.234Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_oximeter_1a03d1eb-bc7d-46ef-83a9-40ee65e701ce/root/var/svc/log/oxide-oximeter:default.log.0 (189575 bytes)
10522024-02-03T11:33:32.470ZError: No response de-activating SOL payload
10532024-02-03T11:33:32.472Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_oximeter_1a03d1eb-bc7d-46ef-83a9-40ee65e701ce/root/var/svc/log/oxide-oximeter:default.log.0
10542024-02-03T11:33:32.472Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_oximeter_1a03d1eb-bc7d-46ef-83a9-40ee65e701ce/root/var/svc/log/oxide-zone-network-setup:default.log (0 bytes)
10552024-02-03T11:33:33.548Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_oximeter_1a03d1eb-bc7d-46ef-83a9-40ee65e701ce/root/var/svc/log/oxide-zone-network-setup:default.log
10562024-02-03T11:33:33.548Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_oximeter_1a03d1eb-bc7d-46ef-83a9-40ee65e701ce/root/var/svc/log/oxide-zone-network-setup:default.log.0 (1697 bytes)
10572024-02-03T11:33:33.580Z[SOL Session operational. Use ~? for help]
10582024-02-03T11:33:34.642Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_oximeter_1a03d1eb-bc7d-46ef-83a9-40ee65e701ce/root/var/svc/log/oxide-zone-network-setup:default.log.0
10592024-02-03T11:33:34.642Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_b2ab71fa-a492-4fb8-916c-1bb4a0b9e09e/root/var/svc/log/oxide-cockroachdb:default.log (0 bytes)
10602024-02-03T11:33:35.713Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_b2ab71fa-a492-4fb8-916c-1bb4a0b9e09e/root/var/svc/log/oxide-cockroachdb:default.log
10612024-02-03T11:33:35.738Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_b2ab71fa-a492-4fb8-916c-1bb4a0b9e09e/root/var/svc/log/oxide-cockroachdb:default.log.0 (4407 bytes)
10622024-02-03T11:33:36.809Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_b2ab71fa-a492-4fb8-916c-1bb4a0b9e09e/root/var/svc/log/oxide-cockroachdb:default.log.0
10632024-02-03T11:33:36.809Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_b2ab71fa-a492-4fb8-916c-1bb4a0b9e09e/root/var/svc/log/oxide-zone-network-setup:default.log (0 bytes)
10642024-02-03T11:33:37.880Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_b2ab71fa-a492-4fb8-916c-1bb4a0b9e09e/root/var/svc/log/oxide-zone-network-setup:default.log
10652024-02-03T11:33:37.880Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_b2ab71fa-a492-4fb8-916c-1bb4a0b9e09e/root/var/svc/log/oxide-zone-network-setup:default.log.0 (1711 bytes)
10662024-02-03T11:33:38.975Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_b2ab71fa-a492-4fb8-916c-1bb4a0b9e09e/root/var/svc/log/oxide-zone-network-setup:default.log.0
10672024-02-03T11:33:38.999Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_686652a1-4ebc-4328-b8cc-fc5fbbb1748c/root/var/svc/log/oxide-crucible-agent:default.log (11570 bytes)
10682024-02-03T11:33:40.094Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_686652a1-4ebc-4328-b8cc-fc5fbbb1748c/root/var/svc/log/oxide-crucible-agent:default.log
10692024-02-03T11:33:40.094Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_686652a1-4ebc-4328-b8cc-fc5fbbb1748c/root/var/svc/log/oxide-crucible-agent:default.log.0 (3436 bytes)
10702024-02-03T11:33:41.194Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_686652a1-4ebc-4328-b8cc-fc5fbbb1748c/root/var/svc/log/oxide-crucible-agent:default.log.0
10712024-02-03T11:33:41.194Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_686652a1-4ebc-4328-b8cc-fc5fbbb1748c/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
10722024-02-03T11:33:42.268Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_686652a1-4ebc-4328-b8cc-fc5fbbb1748c/root/var/svc/log/oxide-crucible-downstairs:default.log
10732024-02-03T11:33:42.293Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_686652a1-4ebc-4328-b8cc-fc5fbbb1748c/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
10742024-02-03T11:33:43.373Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_686652a1-4ebc-4328-b8cc-fc5fbbb1748c/root/var/svc/log/oxide-crucible-downstairs:default.log.0
10752024-02-03T11:33:43.374Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_686652a1-4ebc-4328-b8cc-fc5fbbb1748c/root/var/svc/log/oxide-crucible-downstairs:downstairs-2b12888e-3eb9-4bdd-a832-65fc99323686.log (5518 bytes)
10762024-02-03T11:33:44.479Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_686652a1-4ebc-4328-b8cc-fc5fbbb1748c/root/var/svc/log/oxide-crucible-downstairs:downstairs-2b12888e-3eb9-4bdd-a832-65fc99323686.log
10772024-02-03T11:33:44.479Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_5d4f0a42-42a1-4ce4-b9ff-912d229872d5/root/var/svc/log/oxide-external_dns:default.log (11328 bytes)
10782024-02-03T11:33:44.555Zupload warning: file "/pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_5d4f0a42-42a1-4ce4-b9ff-912d229872d5/root/var/svc/log/oxide-external_dns:default.log" changed size mid upload: 11328 -> 15106
10792024-02-03T11:33:45.660Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_5d4f0a42-42a1-4ce4-b9ff-912d229872d5/root/var/svc/log/oxide-external_dns:default.log
10802024-02-03T11:33:45.661Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_5d4f0a42-42a1-4ce4-b9ff-912d229872d5/root/var/svc/log/oxide-external_dns:default.log.0 (18133 bytes)
10812024-02-03T11:33:46.804Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_5d4f0a42-42a1-4ce4-b9ff-912d229872d5/root/var/svc/log/oxide-external_dns:default.log.0
10822024-02-03T11:33:46.831Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_f0937caa-bf5c-47a2-bc1b-6b5c97fce3b9/root/var/svc/log/oxide-internal_dns:default.log (11265 bytes)
10832024-02-03T11:33:46.875Zupload warning: file "/pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_f0937caa-bf5c-47a2-bc1b-6b5c97fce3b9/root/var/svc/log/oxide-internal_dns:default.log" changed size mid upload: 11265 -> 15021
10842024-02-03T11:33:47.947Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_f0937caa-bf5c-47a2-bc1b-6b5c97fce3b9/root/var/svc/log/oxide-internal_dns:default.log
10852024-02-03T11:33:47.947Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_f0937caa-bf5c-47a2-bc1b-6b5c97fce3b9/root/var/svc/log/oxide-internal_dns:default.log.0 (103710 bytes)
10862024-02-03T11:33:49.160Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_f0937caa-bf5c-47a2-bc1b-6b5c97fce3b9/root/var/svc/log/oxide-internal_dns:default.log.0
10872024-02-03T11:33:49.160Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_cockroachdb_063cc069-1306-4877-a091-f1460619d954/root/var/svc/log/oxide-cockroachdb:default.log (0 bytes)
10882024-02-03T11:33:50.231Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_cockroachdb_063cc069-1306-4877-a091-f1460619d954/root/var/svc/log/oxide-cockroachdb:default.log
10892024-02-03T11:33:50.231Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_cockroachdb_063cc069-1306-4877-a091-f1460619d954/root/var/svc/log/oxide-cockroachdb:default.log.0 (4380 bytes)
10902024-02-03T11:33:51.328Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_cockroachdb_063cc069-1306-4877-a091-f1460619d954/root/var/svc/log/oxide-cockroachdb:default.log.0
10912024-02-03T11:33:51.328Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_cockroachdb_063cc069-1306-4877-a091-f1460619d954/root/var/svc/log/oxide-zone-network-setup:default.log (0 bytes)
10922024-02-03T11:33:52.400Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_cockroachdb_063cc069-1306-4877-a091-f1460619d954/root/var/svc/log/oxide-zone-network-setup:default.log
10932024-02-03T11:33:52.424Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_cockroachdb_063cc069-1306-4877-a091-f1460619d954/root/var/svc/log/oxide-zone-network-setup:default.log.0 (1711 bytes)
10942024-02-03T11:33:53.495Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_cockroachdb_063cc069-1306-4877-a091-f1460619d954/root/var/svc/log/oxide-zone-network-setup:default.log.0
10952024-02-03T11:33:53.496Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_f0ae5afd-78bf-4e66-876d-c203518191f9/root/var/svc/log/oxide-crucible-agent:default.log (25257 bytes)
10962024-02-03T11:33:54.636Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_f0ae5afd-78bf-4e66-876d-c203518191f9/root/var/svc/log/oxide-crucible-agent:default.log
10972024-02-03T11:33:54.636Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_f0ae5afd-78bf-4e66-876d-c203518191f9/root/var/svc/log/oxide-crucible-agent:default.log.0 (20076 bytes)
10982024-02-03T11:33:55.754Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_f0ae5afd-78bf-4e66-876d-c203518191f9/root/var/svc/log/oxide-crucible-agent:default.log.0
10992024-02-03T11:33:55.754Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_f0ae5afd-78bf-4e66-876d-c203518191f9/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
11002024-02-03T11:33:56.826Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_f0ae5afd-78bf-4e66-876d-c203518191f9/root/var/svc/log/oxide-crucible-downstairs:default.log
11012024-02-03T11:33:56.826Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_f0ae5afd-78bf-4e66-876d-c203518191f9/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
11022024-02-03T11:33:57.924Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_f0ae5afd-78bf-4e66-876d-c203518191f9/root/var/svc/log/oxide-crucible-downstairs:default.log.0
11032024-02-03T11:33:57.924Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_f0ae5afd-78bf-4e66-876d-c203518191f9/root/var/svc/log/oxide-crucible-downstairs:downstairs-008ba089-2802-421d-8cbd-81d0d01a9b8b.log (0 bytes)
11042024-02-03T11:33:58.996Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_f0ae5afd-78bf-4e66-876d-c203518191f9/root/var/svc/log/oxide-crucible-downstairs:downstairs-008ba089-2802-421d-8cbd-81d0d01a9b8b.log
11052024-02-03T11:33:58.996Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_f0ae5afd-78bf-4e66-876d-c203518191f9/root/var/svc/log/oxide-crucible-downstairs:downstairs-008ba089-2802-421d-8cbd-81d0d01a9b8b.log.0 (3048 bytes)
11062024-02-03T11:34:00.092Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_f0ae5afd-78bf-4e66-876d-c203518191f9/root/var/svc/log/oxide-crucible-downstairs:downstairs-008ba089-2802-421d-8cbd-81d0d01a9b8b.log.0
11072024-02-03T11:34:00.092Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_f0ae5afd-78bf-4e66-876d-c203518191f9/root/var/svc/log/oxide-crucible-downstairs:downstairs-6f40babd-5874-42a3-8512-9397c217aeec.log (2932 bytes)
11082024-02-03T11:34:01.189Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_f0ae5afd-78bf-4e66-876d-c203518191f9/root/var/svc/log/oxide-crucible-downstairs:downstairs-6f40babd-5874-42a3-8512-9397c217aeec.log
11092024-02-03T11:34:01.189Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_f0ae5afd-78bf-4e66-876d-c203518191f9/root/var/svc/log/oxide-crucible-downstairs:downstairs-9e65066c-91d3-42d3-a20e-e71b9ca2a7df.log (5513 bytes)
11102024-02-03T11:34:02.295Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_f0ae5afd-78bf-4e66-876d-c203518191f9/root/var/svc/log/oxide-crucible-downstairs:downstairs-9e65066c-91d3-42d3-a20e-e71b9ca2a7df.log
11112024-02-03T11:34:02.295Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_f0ae5afd-78bf-4e66-876d-c203518191f9/root/var/svc/log/oxide-crucible-downstairs:downstairs-acfa6af7-67d6-41a6-83e9-dc05c87ab264.log (1090 bytes)
11122024-02-03T11:34:03.408Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_f0ae5afd-78bf-4e66-876d-c203518191f9/root/var/svc/log/oxide-crucible-downstairs:downstairs-acfa6af7-67d6-41a6-83e9-dc05c87ab264.log
11132024-02-03T11:34:03.408Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_f0ae5afd-78bf-4e66-876d-c203518191f9/root/var/svc/log/oxide-crucible-downstairs:downstairs-acfa6af7-67d6-41a6-83e9-dc05c87ab264.log.0 (4529 bytes)
11142024-02-03T11:34:04.509Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_f0ae5afd-78bf-4e66-876d-c203518191f9/root/var/svc/log/oxide-crucible-downstairs:downstairs-acfa6af7-67d6-41a6-83e9-dc05c87ab264.log.0
11152024-02-03T11:34:04.509Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_f0ae5afd-78bf-4e66-876d-c203518191f9/root/var/svc/log/oxide-crucible-downstairs:snapshot-acfa6af7-67d6-41a6-83e9-dc05c87ab264-0c51a3be-d57e-40e1-84d8-4ca7dbd24e66.log (5157 bytes)
11162024-02-03T11:34:05.620Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_f0ae5afd-78bf-4e66-876d-c203518191f9/root/var/svc/log/oxide-crucible-downstairs:snapshot-acfa6af7-67d6-41a6-83e9-dc05c87ab264-0c51a3be-d57e-40e1-84d8-4ca7dbd24e66.log
11172024-02-03T11:34:05.645Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_internal_dns_ceac3c0d-01e9-4428-b1a7-f34e92c695b8/root/var/svc/log/oxide-internal_dns:default.log (11266 bytes)
11182024-02-03T11:34:05.711Zupload warning: file "/pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_internal_dns_ceac3c0d-01e9-4428-b1a7-f34e92c695b8/root/var/svc/log/oxide-internal_dns:default.log" changed size mid upload: 11266 -> 16274
11192024-02-03T11:34:06.781Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_internal_dns_ceac3c0d-01e9-4428-b1a7-f34e92c695b8/root/var/svc/log/oxide-internal_dns:default.log
11202024-02-03T11:34:06.781Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_internal_dns_ceac3c0d-01e9-4428-b1a7-f34e92c695b8/root/var/svc/log/oxide-internal_dns:default.log.0 (106166 bytes)
11212024-02-03T11:34:08.120Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_internal_dns_ceac3c0d-01e9-4428-b1a7-f34e92c695b8/root/var/svc/log/oxide-internal_dns:default.log.0
11222024-02-03T11:34:08.120Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_cockroachdb_7c751fcd-9cb5-4f62-a1b8-d19f04a7914b/root/var/svc/log/oxide-cockroachdb:default.log (0 bytes)
11232024-02-03T11:34:09.191Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_cockroachdb_7c751fcd-9cb5-4f62-a1b8-d19f04a7914b/root/var/svc/log/oxide-cockroachdb:default.log
11242024-02-03T11:34:09.191Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_cockroachdb_7c751fcd-9cb5-4f62-a1b8-d19f04a7914b/root/var/svc/log/oxide-cockroachdb:default.log.0 (4407 bytes)
11252024-02-03T11:34:10.831Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_cockroachdb_7c751fcd-9cb5-4f62-a1b8-d19f04a7914b/root/var/svc/log/oxide-cockroachdb:default.log.0
11262024-02-03T11:34:10.831Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_cockroachdb_7c751fcd-9cb5-4f62-a1b8-d19f04a7914b/root/var/svc/log/oxide-zone-network-setup:default.log (0 bytes)
11272024-02-03T11:34:11.898Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_cockroachdb_7c751fcd-9cb5-4f62-a1b8-d19f04a7914b/root/var/svc/log/oxide-zone-network-setup:default.log
11282024-02-03T11:34:11.898Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_cockroachdb_7c751fcd-9cb5-4f62-a1b8-d19f04a7914b/root/var/svc/log/oxide-zone-network-setup:default.log.0 (1709 bytes)
11292024-02-03T11:34:12.998Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_cockroachdb_7c751fcd-9cb5-4f62-a1b8-d19f04a7914b/root/var/svc/log/oxide-zone-network-setup:default.log.0
11302024-02-03T11:34:12.998Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_4be5724c-b0dd-47db-9b17-14101b8c4c80/root/var/svc/log/oxide-crucible-agent:default.log (0 bytes)
11312024-02-03T11:34:14.068Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_4be5724c-b0dd-47db-9b17-14101b8c4c80/root/var/svc/log/oxide-crucible-agent:default.log
11322024-02-03T11:34:14.068Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_4be5724c-b0dd-47db-9b17-14101b8c4c80/root/var/svc/log/oxide-crucible-agent:default.log.0 (14581 bytes)
11332024-02-03T11:34:15.209Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_4be5724c-b0dd-47db-9b17-14101b8c4c80/root/var/svc/log/oxide-crucible-agent:default.log.0
11342024-02-03T11:34:15.235Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_4be5724c-b0dd-47db-9b17-14101b8c4c80/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
11352024-02-03T11:34:16.285Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_4be5724c-b0dd-47db-9b17-14101b8c4c80/root/var/svc/log/oxide-crucible-downstairs:default.log
11362024-02-03T11:34:16.310Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_4be5724c-b0dd-47db-9b17-14101b8c4c80/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
11372024-02-03T11:34:17.399Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_4be5724c-b0dd-47db-9b17-14101b8c4c80/root/var/svc/log/oxide-crucible-downstairs:default.log.0
11382024-02-03T11:34:17.399Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_4be5724c-b0dd-47db-9b17-14101b8c4c80/root/var/svc/log/oxide-crucible-downstairs:downstairs-a7559a09-f89a-4888-b813-995e04b3f0f2.log (0 bytes)
11392024-02-03T11:34:18.472Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_4be5724c-b0dd-47db-9b17-14101b8c4c80/root/var/svc/log/oxide-crucible-downstairs:downstairs-a7559a09-f89a-4888-b813-995e04b3f0f2.log
11402024-02-03T11:34:18.472Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_4be5724c-b0dd-47db-9b17-14101b8c4c80/root/var/svc/log/oxide-crucible-downstairs:downstairs-a7559a09-f89a-4888-b813-995e04b3f0f2.log.0 (3048 bytes)
11412024-02-03T11:34:19.565Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_4be5724c-b0dd-47db-9b17-14101b8c4c80/root/var/svc/log/oxide-crucible-downstairs:downstairs-a7559a09-f89a-4888-b813-995e04b3f0f2.log.0
11422024-02-03T11:34:19.565Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_nexus_9e92c902-9e58-4118-a57d-70e82a490cda/root/var/svc/log/oxide-nexus:default.log (11755328 bytes)
11432024-02-03T11:34:24.757Zupload warning: file "/pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_nexus_9e92c902-9e58-4118-a57d-70e82a490cda/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 11755328 -> 11979337
11442024-02-03T11:34:25.828Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_nexus_9e92c902-9e58-4118-a57d-70e82a490cda/root/var/svc/log/oxide-nexus:default.log
11452024-02-03T11:34:25.828Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_nexus_9e92c902-9e58-4118-a57d-70e82a490cda/root/var/svc/log/oxide-nexus:default.log.0 (23346543 bytes)
11462024-02-03T11:34:35.084Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_nexus_9e92c902-9e58-4118-a57d-70e82a490cda/root/var/svc/log/oxide-nexus:default.log.0
11472024-02-03T11:34:35.085Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_cockroachdb_737d0080-75b5-4064-bcc2-1b213eba4027/root/var/svc/log/oxide-cockroachdb:default.log (0 bytes)
11482024-02-03T11:34:36.158Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_cockroachdb_737d0080-75b5-4064-bcc2-1b213eba4027/root/var/svc/log/oxide-cockroachdb:default.log
11492024-02-03T11:34:36.158Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_cockroachdb_737d0080-75b5-4064-bcc2-1b213eba4027/root/var/svc/log/oxide-cockroachdb:default.log.0 (4407 bytes)
11502024-02-03T11:34:37.267Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_cockroachdb_737d0080-75b5-4064-bcc2-1b213eba4027/root/var/svc/log/oxide-cockroachdb:default.log.0
11512024-02-03T11:34:37.267Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_cockroachdb_737d0080-75b5-4064-bcc2-1b213eba4027/root/var/svc/log/oxide-zone-network-setup:default.log (0 bytes)
11522024-02-03T11:34:38.362Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_cockroachdb_737d0080-75b5-4064-bcc2-1b213eba4027/root/var/svc/log/oxide-zone-network-setup:default.log
11532024-02-03T11:34:38.362Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_cockroachdb_737d0080-75b5-4064-bcc2-1b213eba4027/root/var/svc/log/oxide-zone-network-setup:default.log.0 (1706 bytes)
11542024-02-03T11:34:39.450Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_cockroachdb_737d0080-75b5-4064-bcc2-1b213eba4027/root/var/svc/log/oxide-zone-network-setup:default.log.0
11552024-02-03T11:34:39.450Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_42ac6ebf-db94-4e54-9029-2676a8cf3807/root/var/svc/log/oxide-crucible-agent:default.log (6788 bytes)
11562024-02-03T11:34:40.546Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_42ac6ebf-db94-4e54-9029-2676a8cf3807/root/var/svc/log/oxide-crucible-agent:default.log
11572024-02-03T11:34:40.546Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_42ac6ebf-db94-4e54-9029-2676a8cf3807/root/var/svc/log/oxide-crucible-agent:default.log.0 (3436 bytes)
11582024-02-03T11:34:41.643Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_42ac6ebf-db94-4e54-9029-2676a8cf3807/root/var/svc/log/oxide-crucible-agent:default.log.0
11592024-02-03T11:34:41.643Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_42ac6ebf-db94-4e54-9029-2676a8cf3807/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
11602024-02-03T11:34:42.725Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_42ac6ebf-db94-4e54-9029-2676a8cf3807/root/var/svc/log/oxide-crucible-downstairs:default.log
11612024-02-03T11:34:42.725Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_42ac6ebf-db94-4e54-9029-2676a8cf3807/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
11622024-02-03T11:34:43.823Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_42ac6ebf-db94-4e54-9029-2676a8cf3807/root/var/svc/log/oxide-crucible-downstairs:default.log.0
11632024-02-03T11:34:43.823Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_42ac6ebf-db94-4e54-9029-2676a8cf3807/root/var/svc/log/oxide-crucible-downstairs:downstairs-c4a7df58-7a9d-4412-a63e-4b2f2e73ffa2.log (2931 bytes)
11642024-02-03T11:34:44.919Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_42ac6ebf-db94-4e54-9029-2676a8cf3807/root/var/svc/log/oxide-crucible-downstairs:downstairs-c4a7df58-7a9d-4412-a63e-4b2f2e73ffa2.log
11652024-02-03T11:34:44.920Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_8407d211-c765-4b06-9157-135e62a9a944/root/var/svc/log/oxide-crucible-agent:default.log (0 bytes)
11662024-02-03T11:34:45.991Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_8407d211-c765-4b06-9157-135e62a9a944/root/var/svc/log/oxide-crucible-agent:default.log
11672024-02-03T11:34:45.991Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_8407d211-c765-4b06-9157-135e62a9a944/root/var/svc/log/oxide-crucible-agent:default.log.0 (3435 bytes)
11682024-02-03T11:34:47.208Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_8407d211-c765-4b06-9157-135e62a9a944/root/var/svc/log/oxide-crucible-agent:default.log.0
11692024-02-03T11:34:47.208Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_8407d211-c765-4b06-9157-135e62a9a944/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
11702024-02-03T11:34:48.282Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_8407d211-c765-4b06-9157-135e62a9a944/root/var/svc/log/oxide-crucible-downstairs:default.log
11712024-02-03T11:34:48.282Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_8407d211-c765-4b06-9157-135e62a9a944/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
11722024-02-03T11:34:49.797Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_8407d211-c765-4b06-9157-135e62a9a944/root/var/svc/log/oxide-crucible-downstairs:default.log.0
11732024-02-03T11:34:49.797Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_4eb9cd1b-06c3-49e4-bab6-29f29d4e5082/root/var/svc/log/oxide-crucible-agent:default.log (7752 bytes)
11742024-02-03T11:34:50.893Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_4eb9cd1b-06c3-49e4-bab6-29f29d4e5082/root/var/svc/log/oxide-crucible-agent:default.log
11752024-02-03T11:34:50.893Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_4eb9cd1b-06c3-49e4-bab6-29f29d4e5082/root/var/svc/log/oxide-crucible-agent:default.log.0 (20044 bytes)
11762024-02-03T11:34:52.033Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_4eb9cd1b-06c3-49e4-bab6-29f29d4e5082/root/var/svc/log/oxide-crucible-agent:default.log.0
11772024-02-03T11:34:52.033Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_4eb9cd1b-06c3-49e4-bab6-29f29d4e5082/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
11782024-02-03T11:34:53.108Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_4eb9cd1b-06c3-49e4-bab6-29f29d4e5082/root/var/svc/log/oxide-crucible-downstairs:default.log
11792024-02-03T11:34:53.108Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_4eb9cd1b-06c3-49e4-bab6-29f29d4e5082/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
11802024-02-03T11:34:54.203Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_4eb9cd1b-06c3-49e4-bab6-29f29d4e5082/root/var/svc/log/oxide-crucible-downstairs:default.log.0
11812024-02-03T11:34:54.227Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_4eb9cd1b-06c3-49e4-bab6-29f29d4e5082/root/var/svc/log/oxide-crucible-downstairs:downstairs-17f23b37-195b-4296-9bea-157f89858d6a.log (1091 bytes)
11822024-02-03T11:34:55.298Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_4eb9cd1b-06c3-49e4-bab6-29f29d4e5082/root/var/svc/log/oxide-crucible-downstairs:downstairs-17f23b37-195b-4296-9bea-157f89858d6a.log
11832024-02-03T11:34:55.298Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_4eb9cd1b-06c3-49e4-bab6-29f29d4e5082/root/var/svc/log/oxide-crucible-downstairs:downstairs-17f23b37-195b-4296-9bea-157f89858d6a.log.0 (4529 bytes)
11842024-02-03T11:34:56.397Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_4eb9cd1b-06c3-49e4-bab6-29f29d4e5082/root/var/svc/log/oxide-crucible-downstairs:downstairs-17f23b37-195b-4296-9bea-157f89858d6a.log.0
11852024-02-03T11:34:56.397Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_4eb9cd1b-06c3-49e4-bab6-29f29d4e5082/root/var/svc/log/oxide-crucible-downstairs:downstairs-56499f5c-3ea3-4244-bcbb-b85cac37645d.log (0 bytes)
11862024-02-03T11:34:57.468Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_4eb9cd1b-06c3-49e4-bab6-29f29d4e5082/root/var/svc/log/oxide-crucible-downstairs:downstairs-56499f5c-3ea3-4244-bcbb-b85cac37645d.log
11872024-02-03T11:34:57.469Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_4eb9cd1b-06c3-49e4-bab6-29f29d4e5082/root/var/svc/log/oxide-crucible-downstairs:downstairs-56499f5c-3ea3-4244-bcbb-b85cac37645d.log.0 (3050 bytes)
11882024-02-03T11:34:58.564Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_4eb9cd1b-06c3-49e4-bab6-29f29d4e5082/root/var/svc/log/oxide-crucible-downstairs:downstairs-56499f5c-3ea3-4244-bcbb-b85cac37645d.log.0
11892024-02-03T11:34:58.588Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_4eb9cd1b-06c3-49e4-bab6-29f29d4e5082/root/var/svc/log/oxide-crucible-downstairs:snapshot-17f23b37-195b-4296-9bea-157f89858d6a-0c51a3be-d57e-40e1-84d8-4ca7dbd24e66.log (5158 bytes)
11902024-02-03T11:34:59.660Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_4eb9cd1b-06c3-49e4-bab6-29f29d4e5082/root/var/svc/log/oxide-crucible-downstairs:snapshot-17f23b37-195b-4296-9bea-157f89858d6a-0c51a3be-d57e-40e1-84d8-4ca7dbd24e66.log
11912024-02-03T11:34:59.660Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_pantry_00e01828-0019-4bac-b5c2-4556a58def59/root/var/svc/log/oxide-crucible-pantry:default.log (367239 bytes)
11922024-02-03T11:35:11.560Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_pantry_00e01828-0019-4bac-b5c2-4556a58def59/root/var/svc/log/oxide-crucible-pantry:default.log
11932024-02-03T11:35:11.560Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_pantry_00e01828-0019-4bac-b5c2-4556a58def59/root/var/svc/log/oxide-crucible-pantry:default.log.0 (849011 bytes)
11942024-02-03T11:35:12.720Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_pantry_00e01828-0019-4bac-b5c2-4556a58def59/root/var/svc/log/oxide-crucible-pantry:default.log.0
11952024-02-03T11:35:12.720Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_pantry_534364e0-4a71-404e-9866-18a729e974a9/root/var/svc/log/oxide-crucible-pantry:default.log (0 bytes)
11962024-02-03T11:35:13.803Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_pantry_534364e0-4a71-404e-9866-18a729e974a9/root/var/svc/log/oxide-crucible-pantry:default.log
11972024-02-03T11:35:13.803Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_pantry_534364e0-4a71-404e-9866-18a729e974a9/root/var/svc/log/oxide-crucible-pantry:default.log.0 (2338 bytes)
11982024-02-03T11:35:14.899Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_pantry_534364e0-4a71-404e-9866-18a729e974a9/root/var/svc/log/oxide-crucible-pantry:default.log.0
11992024-02-03T11:35:14.899Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_nexus_5c8cbd50-2615-4f95-acda-dde3c9fd203e/root/var/svc/log/oxide-nexus:default.log (428110 bytes)
12002024-02-03T11:35:15.975Zupload warning: file "/pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_nexus_5c8cbd50-2615-4f95-acda-dde3c9fd203e/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 428110 -> 776847
12012024-02-03T11:35:17.045Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_nexus_5c8cbd50-2615-4f95-acda-dde3c9fd203e/root/var/svc/log/oxide-nexus:default.log
12022024-02-03T11:35:17.045Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_nexus_5c8cbd50-2615-4f95-acda-dde3c9fd203e/root/var/svc/log/oxide-nexus:default.log.0 (1465171 bytes)
12032024-02-03T11:35:19.892Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_nexus_5c8cbd50-2615-4f95-acda-dde3c9fd203e/root/var/svc/log/oxide-nexus:default.log.0
12042024-02-03T11:35:19.892Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_2492f1e5-dca6-419d-8bcc-984172a6e1d3/root/var/svc/log/oxide-crucible-agent:default.log (0 bytes)
12052024-02-03T11:35:20.961Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_2492f1e5-dca6-419d-8bcc-984172a6e1d3/root/var/svc/log/oxide-crucible-agent:default.log
12062024-02-03T11:35:20.961Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_2492f1e5-dca6-419d-8bcc-984172a6e1d3/root/var/svc/log/oxide-crucible-agent:default.log.0 (3436 bytes)
12072024-02-03T11:35:22.056Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_2492f1e5-dca6-419d-8bcc-984172a6e1d3/root/var/svc/log/oxide-crucible-agent:default.log.0
12082024-02-03T11:35:22.086Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_2492f1e5-dca6-419d-8bcc-984172a6e1d3/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
12092024-02-03T11:35:23.133Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_2492f1e5-dca6-419d-8bcc-984172a6e1d3/root/var/svc/log/oxide-crucible-downstairs:default.log
12102024-02-03T11:35:23.133Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_2492f1e5-dca6-419d-8bcc-984172a6e1d3/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
12112024-02-03T11:35:24.228Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_2492f1e5-dca6-419d-8bcc-984172a6e1d3/root/var/svc/log/oxide-crucible-downstairs:default.log.0
12122024-02-03T11:35:24.229Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_78e54c82-f2a6-4bb8-a1aa-9ff6091f0f44/root/var/svc/log/oxide-crucible-agent:default.log (10359 bytes)
12132024-02-03T11:35:25.346Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_78e54c82-f2a6-4bb8-a1aa-9ff6091f0f44/root/var/svc/log/oxide-crucible-agent:default.log
12142024-02-03T11:35:25.346Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_78e54c82-f2a6-4bb8-a1aa-9ff6091f0f44/root/var/svc/log/oxide-crucible-agent:default.log.0 (9794 bytes)
12152024-02-03T11:35:26.507Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_78e54c82-f2a6-4bb8-a1aa-9ff6091f0f44/root/var/svc/log/oxide-crucible-agent:default.log.0
12162024-02-03T11:35:26.507Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_78e54c82-f2a6-4bb8-a1aa-9ff6091f0f44/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
12172024-02-03T11:35:27.578Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_78e54c82-f2a6-4bb8-a1aa-9ff6091f0f44/root/var/svc/log/oxide-crucible-downstairs:default.log
12182024-02-03T11:35:27.578Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_78e54c82-f2a6-4bb8-a1aa-9ff6091f0f44/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
12192024-02-03T11:35:28.673Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_78e54c82-f2a6-4bb8-a1aa-9ff6091f0f44/root/var/svc/log/oxide-crucible-downstairs:default.log.0
12202024-02-03T11:35:28.673Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_78e54c82-f2a6-4bb8-a1aa-9ff6091f0f44/root/var/svc/log/oxide-crucible-downstairs:downstairs-d9648bc8-e669-4e5d-988b-8e190ab5b734.log (1091 bytes)
12212024-02-03T11:35:29.771Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_78e54c82-f2a6-4bb8-a1aa-9ff6091f0f44/root/var/svc/log/oxide-crucible-downstairs:downstairs-d9648bc8-e669-4e5d-988b-8e190ab5b734.log
12222024-02-03T11:35:29.771Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_78e54c82-f2a6-4bb8-a1aa-9ff6091f0f44/root/var/svc/log/oxide-crucible-downstairs:downstairs-d9648bc8-e669-4e5d-988b-8e190ab5b734.log.0 (4529 bytes)
12232024-02-03T11:35:30.870Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_78e54c82-f2a6-4bb8-a1aa-9ff6091f0f44/root/var/svc/log/oxide-crucible-downstairs:downstairs-d9648bc8-e669-4e5d-988b-8e190ab5b734.log.0
12242024-02-03T11:35:30.870Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_78e54c82-f2a6-4bb8-a1aa-9ff6091f0f44/root/var/svc/log/oxide-crucible-downstairs:snapshot-d9648bc8-e669-4e5d-988b-8e190ab5b734-0c51a3be-d57e-40e1-84d8-4ca7dbd24e66.log (5157 bytes)
12252024-02-03T11:35:31.969Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_78e54c82-f2a6-4bb8-a1aa-9ff6091f0f44/root/var/svc/log/oxide-crucible-downstairs:snapshot-d9648bc8-e669-4e5d-988b-8e190ab5b734-0c51a3be-d57e-40e1-84d8-4ca7dbd24e66.log
12262024-02-03T11:35:31.969Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_e0b5cbda-123e-4edd-a832-85e3c088469a/root/var/log/chrony/measurements.log (26578 bytes)
12272024-02-03T11:35:32.083Zupload warning: file "/pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_e0b5cbda-123e-4edd-a832-85e3c088469a/root/var/log/chrony/measurements.log" changed size mid upload: 26578 -> 33428
12282024-02-03T11:35:33.154Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_e0b5cbda-123e-4edd-a832-85e3c088469a/root/var/log/chrony/measurements.log
12292024-02-03T11:35:33.154Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_e0b5cbda-123e-4edd-a832-85e3c088469a/root/var/log/chrony/statistics.log (18369 bytes)
12302024-02-03T11:35:33.249Zupload warning: file "/pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_e0b5cbda-123e-4edd-a832-85e3c088469a/root/var/log/chrony/statistics.log" changed size mid upload: 18369 -> 23985
12312024-02-03T11:35:34.320Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_e0b5cbda-123e-4edd-a832-85e3c088469a/root/var/log/chrony/statistics.log
12322024-02-03T11:35:34.320Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_e0b5cbda-123e-4edd-a832-85e3c088469a/root/var/log/chrony/tracking.log (2117 bytes)
12332024-02-03T11:35:34.369Zupload warning: file "/pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_e0b5cbda-123e-4edd-a832-85e3c088469a/root/var/log/chrony/tracking.log" changed size mid upload: 2117 -> 2645
12342024-02-03T11:35:35.440Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_e0b5cbda-123e-4edd-a832-85e3c088469a/root/var/log/chrony/tracking.log