01HHGQP45QDFHENGQG5RB8YTKS: helios / deploy

BasicConfig {
    output_rules: [
        "%/var/svc/log/oxide-sled-agent:default.log*",
        "%/pool/ext/*/crypt/zone/oxz_*/root/var/svc/log/oxide-*.log*",
        "%/pool/ext/*/crypt/zone/oxz_*/root/var/svc/log/system-illumos-*.log*",
        "%/pool/ext/*/crypt/zone/oxz_ntp_*/root/var/log/chrony/*.log*",
        "!/pool/ext/*/crypt/zone/oxz_propolis-server_*/root/var/svc/log/*.log*",
        "%/pool/ext/*/crypt/debug/global/oxide-sled-agent:default.log.*",
        "%/pool/ext/*/crypt/debug/oxz_*/oxide-*.log.*",
        "%/pool/ext/*/crypt/debug/oxz_*/system-illumos-*.log.*",
        "!/pool/ext/*/crypt/debug/oxz_propolis-server_*/*.log.*",
    ],
    rust_toolchain: None,
    target: Some(
        "lab-2.0-opte-0.27",
    ),
    access_repos: [],
    publish: [],
    skip_clone: true,
}

Buildomat Job: 01HHGQQG6B1EKXJT43RFXVWWPE

Tags:

Artefacts:

Output:

SEQ GLOBAL TIME DETAILS
12023-12-13T05:27:54.251Zjob dependencies complete; ready to run (waiting for 45 m 24 s)
22023-12-13T05:53:51.910Zjob assigned to worker 01HHGVF2FEWNB61REHT5NDN7C2 (queued for 25 m 57 s)
32023-12-13T05:53:51.920Zhttp://172.20.0.5:9969/ipxe/sock... ok
42023-12-13T05:53:51.964Zhttp://172.20.0.5:9969/os/sock/platform/i86pc/kernel/amd64/unix... ok
52023-12-13T05:53:51.999Zhttp://172.20.0.5:9969/os/sock/platform/i86pc/amd64/boot_archive... ok
62023-12-13T05:53:52.033Zhttp://172.20.0.5:9969/os/sock/platform/i86pc/amd64/boot_archive.hash... ok
72023-12-13T05:53:52.216Zhttp://172.20.0.5:9969/postboot/sock... ok
82023-12-13T05:53:52.781ZwcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U
92023-12-13T05:53:52.845Zcpu0: microcode has been updated from version 0xa001133 to 0xa0011d1
102023-12-13T05:53:52.889ZOxide Helios Version helios-2.0.22333 64-bit
112023-12-13T05:53:53.338ZConfiguring devices.
122023-12-13T05:53:53.378Z##################################################
132023-12-13T05:53:53.411Z#### ############# ##
142023-12-13T05:53:53.445Z### ### ############ ##
152023-12-13T05:53:53.478Z## ### # ## ### ## Oxide Computer Company ##
162023-12-13T05:53:53.512Z## ## ## ### # ### ##
172023-12-13T05:53:53.545Z## # ### #### #### This Station Under ##
182023-12-13T05:53:53.579Z### ### #### # ### Computer Control ##
192023-12-13T05:53:53.612Z#### #### ### ## ##
202023-12-13T05:53:53.646Z##################################################
212023-12-13T05:53:53.679ZHostname: sock
222023-12-13T05:53:53.713ZLoading smf(7) service descriptions:
232023-12-13T05:53:54.627Zdownloading input: /input/ci-tools/work/end-to-end-tests/bootstrap-8aed3472db5f37d6.gz
242023-12-13T05:53:56.166Zdownloaded input: /input/ci-tools/work/end-to-end-tests/bootstrap-8aed3472db5f37d6.gz
252023-12-13T05:53:56.167Zdownloading input: /input/ci-tools/work/end-to-end-tests/bootstrap.gz
262023-12-13T05:54:17.990Zdownloaded input: /input/ci-tools/work/end-to-end-tests/bootstrap.gz
272023-12-13T05:54:17.991Zdownloading input: /input/ci-tools/work/end-to-end-tests/end_to_end_tests-fd6b950b6a51f2ac.gz
282023-12-13T05:54:38.528Zdownloaded input: /input/ci-tools/work/end-to-end-tests/end_to_end_tests-fd6b950b6a51f2ac.gz
292023-12-13T05:54:38.529Zdownloading input: /input/ci-tools/work/caboose-util.gz
302023-12-13T05:54:39.814Zdownloaded input: /input/ci-tools/work/caboose-util.gz
312023-12-13T05:54:40.300Zdownloading input: /input/ci-tools/work/tufaceous.gz
322023-12-13T05:54:42.259Zdownloaded input: /input/ci-tools/work/tufaceous.gz
332023-12-13T05:54:42.331Zdownloading input: /input/package/work/version.txt
342023-12-13T05:54:42.331Zdownloaded input: /input/package/work/version.txt
352023-12-13T05:54:42.331Zdownloading input: /input/package/work/package.tar.gz
362023-12-13T05:55:16.397Zdownloaded input: /input/package/work/package.tar.gz
372023-12-13T05:55:16.424Zdownloading input: /input/package/work/global-zone-packages.tar.gz
382023-12-13T05:55:48.506Zdownloaded input: /input/package/work/global-zone-packages.tar.gz
392023-12-13T05:55:48.506Zdownloading input: /input/package/work/trampoline-global-zone-packages.tar.gz
402023-12-13T05:55:58.934Zdownloaded input: /input/package/work/trampoline-global-zone-packages.tar.gz
412023-12-13T05:55:58.960Zdownloading input: /input/package/work/zones/clickhouse.tar.gz
422023-12-13T05:56:49.878Zdownloaded input: /input/package/work/zones/clickhouse.tar.gz
432023-12-13T05:56:49.878Zdownloading input: /input/package/work/zones/clickhouse_keeper.tar.gz
442023-12-13T05:57:37.357Zdownloaded input: /input/package/work/zones/clickhouse_keeper.tar.gz
452023-12-13T05:57:37.358Zdownloading input: /input/package/work/zones/cockroachdb.tar.gz
462023-12-13T05:58:08.439Zdownloaded input: /input/package/work/zones/cockroachdb.tar.gz
472023-12-13T05:58:08.439Zdownloading input: /input/package/work/zones/crucible-pantry.tar.gz
482023-12-13T05:58:22.202Zdownloaded input: /input/package/work/zones/crucible-pantry.tar.gz
492023-12-13T05:58:22.203Zdownloading input: /input/package/work/zones/crucible.tar.gz
502023-12-13T05:58:31.412Zdownloaded input: /input/package/work/zones/crucible.tar.gz
512023-12-13T05:58:31.412Zdownloading input: /input/package/work/zones/external-dns.tar.gz
522023-12-13T05:58:36.604Zdownloaded input: /input/package/work/zones/external-dns.tar.gz
532023-12-13T05:58:36.605Zdownloading input: /input/package/work/zones/internal-dns.tar.gz
542023-12-13T05:58:42.283Zdownloaded input: /input/package/work/zones/internal-dns.tar.gz
552023-12-13T05:58:42.317Zdownloading input: /input/package/work/zones/ntp.tar.gz
562023-12-13T05:58:42.368Zdownloaded input: /input/package/work/zones/ntp.tar.gz
572023-12-13T05:58:42.369Zdownloading input: /input/package/work/zones/omicron-gateway-asic.tar.gz
582023-12-13T05:58:48.279Zdownloaded input: /input/package/work/zones/omicron-gateway-asic.tar.gz
592023-12-13T05:58:48.279Zdownloading input: /input/package/work/zones/omicron-gateway-softnpu.tar.gz
602023-12-13T05:58:53.019Zdownloaded input: /input/package/work/zones/omicron-gateway-softnpu.tar.gz
612023-12-13T05:58:53.019Zdownloading input: /input/package/work/zones/omicron-nexus-single-sled.tar.gz
622023-12-13T05:59:10.724Zdownloaded input: /input/package/work/zones/omicron-nexus-single-sled.tar.gz
632023-12-13T05:59:10.748Zdownloading input: /input/package/work/zones/omicron-nexus.tar.gz
642023-12-13T05:59:36.167Zdownloaded input: /input/package/work/zones/omicron-nexus.tar.gz
652023-12-13T05:59:36.203Zdownloading input: /input/package/work/zones/overlay.tar.gz
662023-12-13T05:59:36.261Zdownloaded input: /input/package/work/zones/overlay.tar.gz
672023-12-13T05:59:36.261Zdownloading input: /input/package/work/zones/oximeter-collector.tar.gz
682023-12-13T05:59:43.323Zdownloaded input: /input/package/work/zones/oximeter-collector.tar.gz
692023-12-13T05:59:59.136Zdownloading input: /input/package/work/zones/propolis-server.tar.gz
702023-12-13T06:00:16.564Zdownloaded input: /input/package/work/zones/propolis-server.tar.gz
712023-12-13T06:00:16.564Zdownloading input: /input/package/work/zones/switch-asic.tar.gz
722023-12-13T06:01:24.496Zdownloaded input: /input/package/work/zones/switch-asic.tar.gz
732023-12-13T06:01:24.496Zdownloading input: /input/package/work/zones/switch-softnpu.tar.gz
742023-12-13T06:02:12.389Zdownloaded input: /input/package/work/zones/switch-softnpu.tar.gz
 
752023-12-13T06:02:12.423Zstarting task 0: "setup"
762023-12-13T06:02:12.423Z++ uname -s
772023-12-13T06:02:12.423Z+ kern=SunOS
782023-12-13T06:02:12.423Z+ case "$kern" in
792023-12-13T06:02:12.423Z+ groupadd -g 12345 build
802023-12-13T06:02:12.423Z+ useradd -u 12345 -g build -d /home/build -s /bin/bash -c build -P 'Primary Administrator' build
812023-12-13T06:02:14.405Z+ zfs create -o mountpoint=/work rpool/work
822023-12-13T06:02:14.442Z++ awk '$2 == "/home" { print $3 }' /etc/mnttab
832023-12-13T06:02:14.474Z+ home_fs=zfs
842023-12-13T06:02:14.474Z+ [[ zfs == autofs ]]
852023-12-13T06:02:14.474Z+ mkdir -p /home/build
862023-12-13T06:02:14.475Z+ chown build:build /home/build /work
872023-12-13T06:02:16.454Z+ chmod 0700 /home/build /work
882023-12-13T06:02:16.485Zprocess exited: duration 4065 ms, exit code 0
 
892023-12-13T06:02:16.538Zstarting task 1: "authentication"
902023-12-13T06:02:16.581Zprocess exited: duration 44 ms, exit code 0
 
912023-12-13T06:02:16.631Zstarting task 2: "build"
922023-12-13T06:02:16.656Z+ trap _exit_trap EXIT
932023-12-13T06:02:16.656Z+ pfexec mdb -kwe 'secpolicy_fs_linkdir/v 55 48 89 e5 b8 01 00 00 00 5d c3'
942023-12-13T06:02:16.837Zsecpolicy_fs_linkdir: 0x55 = 0x55
952023-12-13T06:02:16.838Zsecpolicy_fs_linkdir+1: 0xb8 = 0x48
962023-12-13T06:02:16.838Zsecpolicy_fs_linkdir+2: 0x1 = 0x89
972023-12-13T06:02:16.838Zsecpolicy_fs_linkdir+3: 0 = 0xe5
982023-12-13T06:02:16.839Zsecpolicy_fs_linkdir+4: 0 = 0xb8
992023-12-13T06:02:16.839Zsecpolicy_fs_linkdir+5: 0 = 0x1
1002023-12-13T06:02:16.839Zsecpolicy_fs_linkdir+6: 0x48 = 0x0
1012023-12-13T06:02:16.865Zsecpolicy_fs_linkdir+7: 0x89 = 0x0
1022023-12-13T06:02:16.866Zsecpolicy_fs_linkdir+8: 0xe5 = 0x0
1032023-12-13T06:02:16.866Zsecpolicy_fs_linkdir+9: 0x48 = 0x5d
1042023-12-13T06:02:16.866Zsecpolicy_fs_linkdir+0xa: 0x83 = 0xc3
1052023-12-13T06:02:16.866Z+ [[ -d /opt/oxide ]]
1062023-12-13T06:02:16.866Z+ pfexec mv /opt/oxide /opt/oxide-underneath
1072023-12-13T06:02:16.866Z+ pfexec mkdir /opt/oxide
1082023-12-13T06:02:16.866Z+ pfexec mount -F tmpfs -O swap /opt/oxide
1092023-12-13T06:02:16.866Z+ [[ -d /opt/oxide-underneath ]]
1102023-12-13T06:02:16.866Z+ cd /opt/oxide-underneath
1112023-12-13T06:02:16.866Z+ pfexec tar ceEp@/f - .
1122023-12-13T06:02:16.866Z+ cd /opt/oxide
1132023-12-13T06:02:16.866Z+ pfexec tar xveEp@/f -
1142023-12-13T06:02:16.965Zx ., 0 bytes, 0 tape blocks
1152023-12-13T06:02:16.965Zx ./opte, 0 bytes, 0 tape blocks
1162023-12-13T06:02:16.965Zx ./opte/bin, 0 bytes, 0 tape blocks
1172023-12-13T06:02:16.965Zx ./opte/bin/opteadm, 52567720 bytes, 102672 tape blocks
1182023-12-13T06:02:17.051Z+ pfexec rm -rf /opt/oxide-underneath
1192023-12-13T06:02:17.051Z+ pfexec /sbin/zfs create -o mountpoint=/zone rpool/zone
1202023-12-13T06:02:17.075Z+ pfexec mkdir /opt/oxide/work
1212023-12-13T06:02:17.075Z+ pfexec chown build:build /opt/oxide/work
1222023-12-13T06:02:17.075Z+ cd /opt/oxide/work
1232023-12-13T06:02:17.075Z+ ptime -m tar xvzf /input/package/work/package.tar.gz
1242023-12-13T06:02:18.548ZDecompressing '/input/package/work/package.tar.gz' with '/usr/bin/gzcat'...
1252023-12-13T06:02:18.548Zx out/mg-ddm-gz.tar, 62723072 bytes, 122506 tape blocks
1262023-12-13T06:02:18.549Zx out/omicron-sled-agent.tar, 111837696 bytes, 218433 tape blocks
1272023-12-13T06:02:18.549Zx out/target/test, 75 bytes, 1 tape blocks
1282023-12-13T06:02:18.549Zx out/npuzone/npuzone, 37184104 bytes, 72626 tape blocks
1292023-12-13T06:02:18.549Zx package-manifest.toml, 22143 bytes, 44 tape blocks
1302023-12-13T06:02:18.549Zx smf/sled-agent/non-gimlet/config.toml, 3448 bytes, 7 tape blocks
1312023-12-13T06:02:18.549Zx target/release/omicron-package, 42513104 bytes, 83034 tape blocks
1322023-12-13T06:02:18.549Zx tools/create_virtual_hardware.sh, 3176 bytes, 7 tape blocks
1332023-12-13T06:02:18.549Zx tools/virtual_hardware.sh, 3408 bytes, 7 tape blocks
1342023-12-13T06:02:18.549Zx tools/scrimlet/create-softnpu-zone.sh, 619 bytes, 2 tape blocks
1352023-12-13T06:02:18.549Zx tools/scrimlet/destroy-softnpu-zone.sh, 195 bytes, 1 tape blocks
1362023-12-13T06:02:18.549Zx tools/scrimlet/softnpu-init.sh, 2148 bytes, 5 tape blocks
1372023-12-13T06:02:18.549Zx tools/scrimlet/softnpu-zone.txt, 330 bytes, 1 tape blocks
1382023-12-13T06:02:18.549Zx tools/scrimlet/softnpu.toml, 171 bytes, 1 tape blocks
1392023-12-13T06:02:18.579Z
1402023-12-13T06:02:18.580Zreal 1.543439259
1412023-12-13T06:02:18.580Zuser 1.384189362
1422023-12-13T06:02:18.580Zsys 0.276220772
1432023-12-13T06:02:18.580Ztrap 0.000064821
1442023-12-13T06:02:18.581Ztflt 0.000000000
1452023-12-13T06:02:18.581Zdflt 0.000000000
1462023-12-13T06:02:18.581Zkflt 0.000000000
1472023-12-13T06:02:18.582Zlock 0.000000000
1482023-12-13T06:02:18.582Zslp 1.389341553
1492023-12-13T06:02:18.583Zlat 0.034782837
1502023-12-13T06:02:18.583Zstop 0.000122780
1512023-12-13T06:02:18.583Z+ cp /input/package/work/zones/clickhouse_keeper.tar.gz /input/package/work/zones/clickhouse.tar.gz /input/package/work/zones/cockroachdb.tar.gz /input/package/work/zones/crucible-pantry.tar.gz /input/package/work/zones/crucible.tar.gz /input/package/work/zones/external-dns.tar.gz /input/package/work/zones/internal-dns.tar.gz /input/package/work/zones/ntp.tar.gz /input/package/work/zones/omicron-gateway-asic.tar.gz /input/package/work/zones/omicron-gateway-softnpu.tar.gz /input/package/work/zones/omicron-nexus-single-sled.tar.gz /input/package/work/zones/omicron-nexus.tar.gz /input/package/work/zones/overlay.tar.gz /input/package/work/zones/oximeter-collector.tar.gz /input/package/work/zones/propolis-server.tar.gz /input/package/work/zones/switch-asic.tar.gz /input/package/work/zones/switch-softnpu.tar.gz out/
1522023-12-13T06:02:19.645Z+ mv out/omicron-nexus-single-sled.tar.gz out/omicron-nexus.tar.gz
1532023-12-13T06:02:19.671Z+ mkdir tests
1542023-12-13T06:02:19.671Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1552023-12-13T06:02:19.672Z+ ptime -m gunzip
1562023-12-13T06:02:19.672Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap-8aed3472db5f37d6
1572023-12-13T06:02:19.795Z
1582023-12-13T06:02:19.821Zreal 0.139895212
1592023-12-13T06:02:19.821Zuser 0.128182772
1602023-12-13T06:02:19.821Zsys 0.011470633
1612023-12-13T06:02:19.821Ztrap 0.000011660
1622023-12-13T06:02:19.821Ztflt 0.000000000
1632023-12-13T06:02:19.821Zdflt 0.000000000
1642023-12-13T06:02:19.821Zkflt 0.000000000
1652023-12-13T06:02:19.821Zlock 0.000000000
1662023-12-13T06:02:19.821Zslp 0.000017680
1672023-12-13T06:02:19.821Zlat 0.000023090
1682023-12-13T06:02:19.821Zstop 0.000125372
1692023-12-13T06:02:19.821Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap-8aed3472db5f37d6
1702023-12-13T06:02:19.821Z+ chmod a+x tests/bootstrap-8aed3472db5f37d6
1712023-12-13T06:02:19.821Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1722023-12-13T06:02:19.822Z+ ptime -m gunzip
1732023-12-13T06:02:19.822Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap
1742023-12-13T06:02:22.041Z
1752023-12-13T06:02:22.066Zreal 2.233756459
1762023-12-13T06:02:22.066Zuser 2.091403734
1772023-12-13T06:02:22.066Zsys 0.141923916
1782023-12-13T06:02:22.066Ztrap 0.000185454
1792023-12-13T06:02:22.066Ztflt 0.000000000
1802023-12-13T06:02:22.066Zdflt 0.000000000
1812023-12-13T06:02:22.066Zkflt 0.000000000
1822023-12-13T06:02:22.066Zlock 0.000000000
1832023-12-13T06:02:22.066Zslp 0.000022590
1842023-12-13T06:02:22.066Zlat 0.000028870
1852023-12-13T06:02:22.066Zstop 0.000126641
1862023-12-13T06:02:22.067Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap
1872023-12-13T06:02:22.067Z+ chmod a+x tests/bootstrap
1882023-12-13T06:02:22.068Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1892023-12-13T06:02:22.068Z+ ptime -m gunzip
1902023-12-13T06:02:22.068Z++ basename /input/ci-tools/work/end-to-end-tests/end_to_end_tests-fd6b950b6a51f2ac
1912023-12-13T06:02:24.388Z
1922023-12-13T06:02:24.415Zreal 2.327396837
1932023-12-13T06:02:24.417Zuser 2.186280810
1942023-12-13T06:02:24.418Zsys 0.140693750
1952023-12-13T06:02:24.418Ztrap 0.000185332
1962023-12-13T06:02:24.418Ztflt 0.000000000
1972023-12-13T06:02:24.418Zdflt 0.000000000
1982023-12-13T06:02:24.418Zkflt 0.000000000
1992023-12-13T06:02:24.418Zlock 0.000000000
2002023-12-13T06:02:24.418Zslp 0.000023650
2012023-12-13T06:02:24.419Zlat 0.000028840
2022023-12-13T06:02:24.419Zstop 0.000122892
2032023-12-13T06:02:24.419Z++ basename /input/ci-tools/work/end-to-end-tests/end_to_end_tests-fd6b950b6a51f2ac
2042023-12-13T06:02:24.419Z+ chmod a+x tests/end_to_end_tests-fd6b950b6a51f2ac
2052023-12-13T06:02:24.419Z+ bmat address ls
2062023-12-13T06:02:24.419ZNAME CIDR FIRST LAST COUNT
2072023-12-13T06:02:24.420Zextra 10.151.2.0/24 10.151.2.164 10.151.2.227 64
2082023-12-13T06:02:24.420Z++ bmat address ls -f extra -Ho first,last,count
2092023-12-13T06:02:24.420Z+ set -- 10.151.2.164 10.151.2.227 64
2102023-12-13T06:02:24.420Z+ EXTRA_IP_START=10.151.2.164
2112023-12-13T06:02:24.420Z+ EXTRA_IP_END=10.151.2.227
2122023-12-13T06:02:24.420Z+ EXTRA_IP_COUNT=64
2132023-12-13T06:02:24.420Z+ (( EXTRA_IP_COUNT >= 32 ))
2142023-12-13T06:02:24.421Z+ EXTRA_IP_BASE=10.151.2
2152023-12-13T06:02:24.421Z+ EXTRA_IP_FOCTET=164
2162023-12-13T06:02:24.421Z+ EXTRA_IP_LOCTET=227
2172023-12-13T06:02:24.421Z+ SERVICE_IP_POOL_START=10.151.2.164
2182023-12-13T06:02:24.421Z+ SERVICE_IP_POOL_END=10.151.2.173
2192023-12-13T06:02:24.421Z+ DNS_IP1=10.151.2.164
2202023-12-13T06:02:24.421Z+ DNS_IP2=10.151.2.165
2212023-12-13T06:02:24.422Z+ UPLINK_IP=10.151.2.174
2222023-12-13T06:02:24.422Z+ IPPOOL_START=10.151.2.175
2232023-12-13T06:02:24.423Z+ IPPOOL_END=10.151.2.227
2242023-12-13T06:02:24.423Z++ ipadm show-addr -po type,addr
2252023-12-13T06:02:24.423Z++ awk '-F[:/]' '$1 == "dhcp" {print $2}'
2262023-12-13T06:02:24.423Z+ GATEWAY_IP=10.151.2.100
2272023-12-13T06:02:24.486Z+ [[ -n 10.151.2.100 ]]
2282023-12-13T06:02:24.486Z+ ping -s 10.151.2.100 56 1
2292023-12-13T06:02:24.486ZPING 10.151.2.100: 56 data bytes
2302023-12-13T06:02:24.486Z64 bytes from 10.151.2.100: icmp_seq=0. time=0.012 ms
2312023-12-13T06:02:24.487Z
2322023-12-13T06:02:24.487Z----10.151.2.100 PING Statistics----
2332023-12-13T06:02:24.487Z1 packets transmitted, 1 packets received, 0% packet loss
2342023-12-13T06:02:24.487Zround-trip (ms) min/avg/max/stddev = 0.012/0.012/0.012/-nan
2352023-12-13T06:02:24.487Z++ arp -an
2362023-12-13T06:02:24.487Z++ awk -vgw=10.151.2.100 '$2 == gw {print $NF}'
2372023-12-13T06:02:24.487Z+ GATEWAY_MAC=18:c0:4d:0d:a0:2a
2382023-12-13T06:02:24.487Z+ routeadm -e ipv4-forwarding -u
2392023-12-13T06:02:24.609Z+ PXA_START=10.151.2.164
2402023-12-13T06:02:24.609Z+ PXA_END=10.151.2.227
2412023-12-13T06:02:24.609Z+ export GATEWAY_IP GATEWAY_MAC PXA_START PXA_END
2422023-12-13T06:02:24.639Z+ pfexec zpool create -f scratch c1t1d0 c2t1d0
2432023-12-13T06:02:24.675Z+ ZPOOL_VDEV_DIR=/scratch
2442023-12-13T06:02:24.675Z+ ptime -m pfexec ./tools/create_virtual_hardware.sh
2452023-12-13T06:02:24.706Z+++ dirname ./tools/create_virtual_hardware.sh
2462023-12-13T06:02:24.706Z++ cd ./tools
2472023-12-13T06:02:24.706Z++ pwd
2482023-12-13T06:02:24.707Z+ SOURCE_DIR=/opt/oxide/work/tools
2492023-12-13T06:02:24.708Z+ OMICRON_TOP=/opt/oxide/work/tools/..
2502023-12-13T06:02:24.708Z+ SOFTNPU_MODE=zone
2512023-12-13T06:02:24.708Z+ . /opt/oxide/work/tools/virtual_hardware.sh
2522023-12-13T06:02:24.708Z++ MARKER=/etc/opt/oxide/NO_INSTALL
2532023-12-13T06:02:24.708Z++ [[ -f /etc/opt/oxide/NO_INSTALL ]]
2542023-12-13T06:02:24.708Z++ dladm show-phys -p -o LINK
2552023-12-13T06:02:24.708Z++ head -1
2562023-12-13T06:02:24.762Z+ PHYSICAL_LINK=igb0
2572023-12-13T06:02:24.762Z+ echo 'Using igb0 as physical link'
2582023-12-13T06:02:24.790ZUsing igb0 as physical link
2592023-12-13T06:02:24.790Z+ ensure_run_as_root
2602023-12-13T06:02:24.791Z++ id -u
2612023-12-13T06:02:24.791Z+ [[ 0 -ne 0 ]]
2622023-12-13T06:02:24.791Z+ ensure_zpools
2632023-12-13T06:02:24.791Z+ ZPOOL_TYPES=('oxp_' 'oxi_')
2642023-12-13T06:02:24.791Z+ for ZPOOL_TYPE in "${ZPOOL_TYPES[@]}"
2652023-12-13T06:02:24.791Z+ readarray -t ZPOOLS
2662023-12-13T06:02:24.791Z++ grep '"oxp_' /opt/oxide/work/tools/../smf/sled-agent/non-gimlet/config.toml
2672023-12-13T06:02:24.791Z++ sed 's/[ ",]//g'
2682023-12-13T06:02:24.791Z+ for ZPOOL in "${ZPOOLS[@]}"
2692023-12-13T06:02:24.792Z+ echo 'Zpool: [oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b]'
2702023-12-13T06:02:24.792ZZpool: [oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b]
2712023-12-13T06:02:24.792Z+ VDEV_PATH=/scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
2722023-12-13T06:02:24.792Z+ [[ -f /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
2732023-12-13T06:02:24.792Z+ dd if=/dev/zero of=/scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
2742023-12-13T06:02:24.828Z0+0 records in
2752023-12-13T06:02:24.828Z0+0 records out
2762023-12-13T06:02:24.828Z0 bytes (0 iB) transferred in 0.000007 secs (0 iB/sec)
2772023-12-13T06:02:24.828Z+ success 'ZFS vdev /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
2782023-12-13T06:02:24.828Z+ set +x
2792023-12-13T06:02:24.829ZZFS vdev /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
2802023-12-13T06:02:24.829Z++ zpool list -o name
2812023-12-13T06:02:24.829Z++ grep oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b
2822023-12-13T06:02:24.829Z+ [[ -z '' ]]
2832023-12-13T06:02:24.829Z+ zpool create -f oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
2842023-12-13T06:02:24.859Z+ success 'ZFS zpool oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
2852023-12-13T06:02:24.859Z+ set +x
2862023-12-13T06:02:24.859ZZFS zpool oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b exists
2872023-12-13T06:02:24.859Z+ for ZPOOL in "${ZPOOLS[@]}"
2882023-12-13T06:02:24.859Z+ echo 'Zpool: [oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03]'
2892023-12-13T06:02:24.859ZZpool: [oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03]
2902023-12-13T06:02:24.859Z+ VDEV_PATH=/scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
2912023-12-13T06:02:24.859Z+ [[ -f /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
2922023-12-13T06:02:24.859Z+ dd if=/dev/zero of=/scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
2932023-12-13T06:02:24.859Z0+0 records in
2942023-12-13T06:02:24.860Z0+0 records out
2952023-12-13T06:02:24.860Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
2962023-12-13T06:02:24.860Z+ success 'ZFS vdev /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
2972023-12-13T06:02:24.860Z+ set +x
2982023-12-13T06:02:24.860ZZFS vdev /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
2992023-12-13T06:02:24.860Z++ zpool list -o name
3002023-12-13T06:02:24.860Z++ grep oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03
3012023-12-13T06:02:24.863Z+ [[ -z '' ]]
3022023-12-13T06:02:24.863Z+ zpool create -f oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3032023-12-13T06:02:24.894Z+ success 'ZFS zpool oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3042023-12-13T06:02:24.895Z+ set +x
3052023-12-13T06:02:24.895ZZFS zpool oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
3062023-12-13T06:02:24.895Z+ for ZPOOL in "${ZPOOLS[@]}"
3072023-12-13T06:02:24.895Z+ echo 'Zpool: [oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3082023-12-13T06:02:24.895ZZpool: [oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03]
3092023-12-13T06:02:24.895Z+ VDEV_PATH=/scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3102023-12-13T06:02:24.895Z+ [[ -f /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3112023-12-13T06:02:24.895Z+ dd if=/dev/zero of=/scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3122023-12-13T06:02:24.895Z0+0 records in
3132023-12-13T06:02:24.895Z0+0 records out
3142023-12-13T06:02:24.895Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3152023-12-13T06:02:24.895Z+ success 'ZFS vdev /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3162023-12-13T06:02:24.895Z+ set +x
3172023-12-13T06:02:24.895ZZFS vdev /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
3182023-12-13T06:02:24.895Z++ zpool list -o name
3192023-12-13T06:02:24.895Z++ grep oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03
3202023-12-13T06:02:24.920Z+ [[ -z '' ]]
3212023-12-13T06:02:24.920Z+ zpool create -f oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3222023-12-13T06:02:24.946Z+ success 'ZFS zpool oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3232023-12-13T06:02:24.946Z+ set +x
3242023-12-13T06:02:24.946ZZFS zpool oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
3252023-12-13T06:02:24.946Z+ for ZPOOL in "${ZPOOLS[@]}"
3262023-12-13T06:02:24.947Z+ echo 'Zpool: [oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3272023-12-13T06:02:24.947ZZpool: [oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03]
3282023-12-13T06:02:24.947Z+ VDEV_PATH=/scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3292023-12-13T06:02:24.947Z+ [[ -f /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3302023-12-13T06:02:24.947Z+ dd if=/dev/zero of=/scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3312023-12-13T06:02:24.947Z0+0 records in
3322023-12-13T06:02:24.947Z0+0 records out
3332023-12-13T06:02:24.947Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3342023-12-13T06:02:24.947Z+ success 'ZFS vdev /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3352023-12-13T06:02:24.947Z+ set +x
3362023-12-13T06:02:24.947ZZFS vdev /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
3372023-12-13T06:02:24.947Z++ zpool list -o name
3382023-12-13T06:02:24.947Z++ grep oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03
3392023-12-13T06:02:24.947Z+ [[ -z '' ]]
3402023-12-13T06:02:24.947Z+ zpool create -f oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3412023-12-13T06:02:24.975Z+ success 'ZFS zpool oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3422023-12-13T06:02:24.975Z+ set +x
3432023-12-13T06:02:24.975ZZFS zpool oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
3442023-12-13T06:02:24.975Z+ for ZPOOL in "${ZPOOLS[@]}"
3452023-12-13T06:02:24.975Z+ echo 'Zpool: [oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3462023-12-13T06:02:24.975ZZpool: [oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03]
3472023-12-13T06:02:24.975Z+ VDEV_PATH=/scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3482023-12-13T06:02:24.975Z+ [[ -f /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3492023-12-13T06:02:24.975Z+ dd if=/dev/zero of=/scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3502023-12-13T06:02:24.976Z0+0 records in
3512023-12-13T06:02:24.976Z0+0 records out
3522023-12-13T06:02:24.976Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3532023-12-13T06:02:24.976Z+ success 'ZFS vdev /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3542023-12-13T06:02:24.976Z+ set +x
3552023-12-13T06:02:24.976ZZFS vdev /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
3562023-12-13T06:02:25.001Z++ zpool list -o name
3572023-12-13T06:02:25.001Z++ grep oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03
3582023-12-13T06:02:25.001Z+ [[ -z '' ]]
3592023-12-13T06:02:25.001Z+ zpool create -f oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3602023-12-13T06:02:25.026Z+ success 'ZFS zpool oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3612023-12-13T06:02:25.026Z+ set +x
3622023-12-13T06:02:25.026ZZFS zpool oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
3632023-12-13T06:02:25.026Z+ for ZPOOL in "${ZPOOLS[@]}"
3642023-12-13T06:02:25.026Z+ echo 'Zpool: [oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930]'
3652023-12-13T06:02:25.026ZZpool: [oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930]
3662023-12-13T06:02:25.026Z+ VDEV_PATH=/scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev
3672023-12-13T06:02:25.026Z+ [[ -f /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev ]]
3682023-12-13T06:02:25.026Z+ dd if=/dev/zero of=/scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev bs=1 count=0 seek=20G
3692023-12-13T06:02:25.026Z0+0 records in
3702023-12-13T06:02:25.026Z0+0 records out
3712023-12-13T06:02:25.027Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3722023-12-13T06:02:25.027Z+ success 'ZFS vdev /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev exists'
3732023-12-13T06:02:25.027Z+ set +x
3742023-12-13T06:02:25.027ZZFS vdev /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev exists
3752023-12-13T06:02:25.027Z++ zpool list -o name
3762023-12-13T06:02:25.027Z++ grep oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930
3772023-12-13T06:02:25.052Z+ [[ -z '' ]]
3782023-12-13T06:02:25.052Z+ zpool create -f oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev
3792023-12-13T06:02:25.077Z+ success 'ZFS zpool oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 exists'
3802023-12-13T06:02:25.078Z+ set +x
3812023-12-13T06:02:25.078ZZFS zpool oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 exists
3822023-12-13T06:02:25.078Z+ for ZPOOL in "${ZPOOLS[@]}"
3832023-12-13T06:02:25.078Z+ echo 'Zpool: [oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60]'
3842023-12-13T06:02:25.078ZZpool: [oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60]
3852023-12-13T06:02:25.078Z+ VDEV_PATH=/scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev
3862023-12-13T06:02:25.078Z+ [[ -f /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev ]]
3872023-12-13T06:02:25.078Z+ dd if=/dev/zero of=/scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev bs=1 count=0 seek=20G
3882023-12-13T06:02:25.078Z0+0 records in
3892023-12-13T06:02:25.078Z0+0 records out
3902023-12-13T06:02:25.078Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3912023-12-13T06:02:25.078Z+ success 'ZFS vdev /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev exists'
3922023-12-13T06:02:25.078Z+ set +x
3932023-12-13T06:02:25.078ZZFS vdev /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev exists
3942023-12-13T06:02:25.078Z++ zpool list -o name
3952023-12-13T06:02:25.078Z++ grep oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60
3962023-12-13T06:02:25.078Z+ [[ -z '' ]]
3972023-12-13T06:02:25.078Z+ zpool create -f oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev
3982023-12-13T06:02:25.103Z+ success 'ZFS zpool oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 exists'
3992023-12-13T06:02:25.105Z+ set +x
4002023-12-13T06:02:25.105ZZFS zpool oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 exists
4012023-12-13T06:02:25.105Z+ for ZPOOL in "${ZPOOLS[@]}"
4022023-12-13T06:02:25.105Z+ echo 'Zpool: [oxp_31bd71cd-4736-4a12-a387-9b74b050396f]'
4032023-12-13T06:02:25.105ZZpool: [oxp_31bd71cd-4736-4a12-a387-9b74b050396f]
4042023-12-13T06:02:25.105Z+ VDEV_PATH=/scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev
4052023-12-13T06:02:25.105Z+ [[ -f /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev ]]
4062023-12-13T06:02:25.105Z+ dd if=/dev/zero of=/scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev bs=1 count=0 seek=20G
4072023-12-13T06:02:25.105Z0+0 records in
4082023-12-13T06:02:25.105Z0+0 records out
4092023-12-13T06:02:25.105Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4102023-12-13T06:02:25.105Z+ success 'ZFS vdev /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev exists'
4112023-12-13T06:02:25.105Z+ set +x
4122023-12-13T06:02:25.105ZZFS vdev /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev exists
4132023-12-13T06:02:25.105Z++ zpool list -o name
4142023-12-13T06:02:25.105Z++ grep oxp_31bd71cd-4736-4a12-a387-9b74b050396f
4152023-12-13T06:02:25.130Z+ [[ -z '' ]]
4162023-12-13T06:02:25.130Z+ zpool create -f oxp_31bd71cd-4736-4a12-a387-9b74b050396f /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev
4172023-12-13T06:02:25.158Z+ success 'ZFS zpool oxp_31bd71cd-4736-4a12-a387-9b74b050396f exists'
4182023-12-13T06:02:25.158Z+ set +x
4192023-12-13T06:02:25.158ZZFS zpool oxp_31bd71cd-4736-4a12-a387-9b74b050396f exists
4202023-12-13T06:02:25.158Z+ for ZPOOL in "${ZPOOLS[@]}"
4212023-12-13T06:02:25.158Z+ echo 'Zpool: [oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7]'
4222023-12-13T06:02:25.158ZZpool: [oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7]
4232023-12-13T06:02:25.158Z+ VDEV_PATH=/scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev
4242023-12-13T06:02:25.159Z+ [[ -f /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev ]]
4252023-12-13T06:02:25.159Z+ dd if=/dev/zero of=/scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev bs=1 count=0 seek=20G
4262023-12-13T06:02:25.159Z0+0 records in
4272023-12-13T06:02:25.159Z0+0 records out
4282023-12-13T06:02:25.159Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4292023-12-13T06:02:25.159Z+ success 'ZFS vdev /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev exists'
4302023-12-13T06:02:25.159Z+ set +x
4312023-12-13T06:02:25.159ZZFS vdev /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev exists
4322023-12-13T06:02:25.159Z++ zpool list -o name
4332023-12-13T06:02:25.159Z++ grep oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7
4342023-12-13T06:02:25.184Z+ [[ -z '' ]]
4352023-12-13T06:02:25.184Z+ zpool create -f oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev
4362023-12-13T06:02:25.209Z+ success 'ZFS zpool oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 exists'
4372023-12-13T06:02:25.209Z+ set +x
4382023-12-13T06:02:25.209ZZFS zpool oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 exists
4392023-12-13T06:02:25.209Z+ for ZPOOL_TYPE in "${ZPOOL_TYPES[@]}"
4402023-12-13T06:02:25.209Z+ readarray -t ZPOOLS
4412023-12-13T06:02:25.209Z++ grep '"oxi_' /opt/oxide/work/tools/../smf/sled-agent/non-gimlet/config.toml
4422023-12-13T06:02:25.210Z++ sed 's/[ ",]//g'
4432023-12-13T06:02:25.210Z+ for ZPOOL in "${ZPOOLS[@]}"
4442023-12-13T06:02:25.210Z+ echo 'Zpool: [oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b]'
4452023-12-13T06:02:25.210ZZpool: [oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b]
4462023-12-13T06:02:25.210Z+ VDEV_PATH=/scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4472023-12-13T06:02:25.213Z+ [[ -f /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
4482023-12-13T06:02:25.213Z+ dd if=/dev/zero of=/scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
4492023-12-13T06:02:25.215Z0+0 records in
4502023-12-13T06:02:25.215Z0+0 records out
4512023-12-13T06:02:25.215Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4522023-12-13T06:02:25.215Z+ success 'ZFS vdev /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
4532023-12-13T06:02:25.215Z+ set +x
4542023-12-13T06:02:25.215ZZFS vdev /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
4552023-12-13T06:02:25.215Z++ zpool list -o name
4562023-12-13T06:02:25.215Z++ grep oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b
4572023-12-13T06:02:25.215Z+ [[ -z '' ]]
4582023-12-13T06:02:25.215Z+ zpool create -f oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4592023-12-13T06:02:25.247Z+ success 'ZFS zpool oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
4602023-12-13T06:02:25.247Z+ set +x
4612023-12-13T06:02:25.247ZZFS zpool oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b exists
4622023-12-13T06:02:25.247Z+ for ZPOOL in "${ZPOOLS[@]}"
4632023-12-13T06:02:25.248Z+ echo 'Zpool: [oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b]'
4642023-12-13T06:02:25.248ZZpool: [oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b]
4652023-12-13T06:02:25.248Z+ VDEV_PATH=/scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4662023-12-13T06:02:25.248Z+ [[ -f /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
4672023-12-13T06:02:25.248Z+ dd if=/dev/zero of=/scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
4682023-12-13T06:02:25.248Z0+0 records in
4692023-12-13T06:02:25.249Z0+0 records out
4702023-12-13T06:02:25.249Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4712023-12-13T06:02:25.249Z+ success 'ZFS vdev /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
4722023-12-13T06:02:25.249Z+ set +x
4732023-12-13T06:02:25.249ZZFS vdev /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
4742023-12-13T06:02:25.280Z++ zpool list -o name
4752023-12-13T06:02:25.280Z++ grep oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b
4762023-12-13T06:02:25.280Z+ [[ -z '' ]]
4772023-12-13T06:02:25.280Z+ zpool create -f oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4782023-12-13T06:02:25.313Z+ success 'ZFS zpool oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
4792023-12-13T06:02:25.313Z+ set +x
4802023-12-13T06:02:25.314ZZFS zpool oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b exists
4812023-12-13T06:02:25.314Z+ [[ zone == \z\o\n\e ]]
4822023-12-13T06:02:25.314Z+ ensure_simulated_links igb0
4832023-12-13T06:02:25.314Z+ local PHYSICAL_LINK=igb0
4842023-12-13T06:02:25.314Z+ INDICES=("0" "1")
4852023-12-13T06:02:25.314Z+ for I in "${INDICES[@]}"
4862023-12-13T06:02:25.314Z++ get_simnet_name_if_exists net0
4872023-12-13T06:02:25.314Z++ dladm show-simnet -p -o LINK net0
4882023-12-13T06:02:25.314Z++ echo ''
4892023-12-13T06:02:25.314Z+ [[ -z '' ]]
4902023-12-13T06:02:25.314Z+ dladm create-simnet -t net0
4912023-12-13T06:02:25.314Z+ dladm create-simnet -t sc0_0
4922023-12-13T06:02:25.315Z+ dladm modify-simnet -t -p net0 sc0_0
4932023-12-13T06:02:25.315Z+ dladm set-linkprop -p mtu=9000 sc0_0
4942023-12-13T06:02:25.315Z+ success 'Simnet net0/sc0_0 exists'
4952023-12-13T06:02:25.315Z+ set +x
4962023-12-13T06:02:25.315ZSimnet net0/sc0_0 exists
4972023-12-13T06:02:25.315Z+ for I in "${INDICES[@]}"
4982023-12-13T06:02:25.345Z++ get_simnet_name_if_exists net1
4992023-12-13T06:02:25.345Z++ dladm show-simnet -p -o LINK net1
5002023-12-13T06:02:25.345Z++ echo ''
5012023-12-13T06:02:25.346Z+ [[ -z '' ]]
5022023-12-13T06:02:25.346Z+ dladm create-simnet -t net1
5032023-12-13T06:02:25.346Z+ dladm create-simnet -t sc1_0
5042023-12-13T06:02:25.346Z+ dladm modify-simnet -t -p net1 sc1_0
5052023-12-13T06:02:25.347Z+ dladm set-linkprop -p mtu=9000 sc1_0
5062023-12-13T06:02:25.347Z+ success 'Simnet net1/sc1_0 exists'
5072023-12-13T06:02:25.347Z+ set +x
5082023-12-13T06:02:25.348ZSimnet net1/sc1_0 exists
5092023-12-13T06:02:25.348Z++ get_vnic_name_if_exists sc0_1
5102023-12-13T06:02:25.348Z++ dladm show-vnic -p -o LINK sc0_1
5112023-12-13T06:02:25.348Z++ echo ''
5122023-12-13T06:02:25.348Z+ [[ -z '' ]]
5132023-12-13T06:02:25.349Z+ dladm create-vnic -t sc0_1 -l igb0 -m a8:e1:de:01:70:1d
5142023-12-13T06:02:25.349Z+ [[ -v PROMISC_FILT_OFF ]]
5152023-12-13T06:02:25.349Z+ success 'Vnic sc0_1 exists'
5162023-12-13T06:02:25.349Z+ set +x
5172023-12-13T06:02:25.349ZVnic sc0_1 exists
5182023-12-13T06:02:25.349Z+ warn_if_no_proxy_arp
5192023-12-13T06:02:25.350Z+ [[ -v PXA_START ]]
5202023-12-13T06:02:25.350Z+ [[ -v PXA_END ]]
5212023-12-13T06:02:25.350Z+ ensure_softnpu_zone
5222023-12-13T06:02:25.350Z+ zoneadm list
5232023-12-13T06:02:25.350Z+ grep -q sidecar_softnpu
5242023-12-13T06:02:25.350Z+ [[ -f out/npuzone/npuzone ]]
5252023-12-13T06:02:25.350Z+ out/npuzone/npuzone create sidecar --omicron-zone --ports sc0_0,tfportrear0_0 --ports sc0_1,tfportqsfp0_0 --sidecar-lite-commit 45ed98fea5824feb4d42f45bbf218e597dc9fc58 --softnpu-commit dbab082dfa89da5db5ca2325c257089d2f130092
5262023-12-13T06:02:25.383Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/softnpu/image/dbab082dfa89da5db5ca2325c257089d2f130092/softnpu.sha256.txt
5272023-12-13T06:02:40.425Zdownloading softnpu ...done
5282023-12-13T06:02:40.471Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/sidecar-lite/release/45ed98fea5824feb4d42f45bbf218e597dc9fc58/libsidecar_lite.so.sha256.txt
5292023-12-13T06:02:50.599Zdownloading asic_program.so ...done
5302023-12-13T06:02:50.639Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/sidecar-lite/release/45ed98fea5824feb4d42f45bbf218e597dc9fc58/scadm.sha256.txt
5312023-12-13T06:03:00.610Zdownloading scadm ...done
5322023-12-13T06:03:00.638Zcreating port tfportrear0_0 for sc0_0
5332023-12-13T06:03:00.638Zcreating port tfportqsfp0_0 for sc0_1
5342023-12-13T06:03:00.638Zsoftnpu zone setup
5352023-12-13T06:03:00.638Zconfigure zone
5362023-12-13T06:03:00.664Zinstall zone
5372023-12-13T06:03:01.690Zboot zone
5382023-12-13T06:03:04.799Z[sidecar_softnpu] svcs milestone/network
5392023-12-13T06:03:04.827ZSTATE STIME FMRI
5402023-12-13T06:03:04.828Zonline 6:03:02 svc:/milestone/network:default
5412023-12-13T06:03:04.828Z[sidecar_softnpu] /softnpu/npu start
5422023-12-13T06:03:04.865Z
5432023-12-13T06:03:04.898Z+ /opt/oxide/work/tools/scrimlet/softnpu-init.sh
5442023-12-13T06:03:04.898Z+ GATEWAY_IP=10.151.2.100
5452023-12-13T06:03:04.898Z+ echo 'Using 10.151.2.100 as gateway ip'
5462023-12-13T06:03:04.898ZUsing 10.151.2.100 as gateway ip
5472023-12-13T06:03:04.899Z+ [[ ! -v GATEWAY_MAC ]]
5482023-12-13T06:03:04.902Z+ GATEWAY_MAC=18:c0:4d:0d:a0:2a
5492023-12-13T06:03:04.902Z++ grep -c -E '^([0-9a-fA-F]{2}:){5}[0-9a-fA-F]{2}$' /dev/fd/63
5502023-12-13T06:03:04.902Z+++ echo 18:c0:4d:0d:a0:2a
5512023-12-13T06:03:04.902Z+ COUNT=1
5522023-12-13T06:03:04.902Z+ [[ 1 -ne 1 ]]
5532023-12-13T06:03:04.903Z+ echo 'Using 18:c0:4d:0d:a0:2a as gateway mac'
5542023-12-13T06:03:04.903ZUsing 18:c0:4d:0d:a0:2a as gateway mac
5552023-12-13T06:03:04.903Z+ z_scadm add-arp-entry 10.151.2.100 18:c0:4d:0d:a0:2a
5562023-12-13T06:03:04.903Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone add-arp-entry 10.151.2.100 18:c0:4d:0d:a0:2a
5572023-12-13T06:03:05.353Z+ PXA_MAC=a8:e1:de:01:70:1d
5582023-12-13T06:03:05.354Z+ [[ -v PXA_START ]]
5592023-12-13T06:03:05.354Z+ z_scadm add-proxy-arp 10.151.2.164 10.151.2.227 a8:e1:de:01:70:1d
5602023-12-13T06:03:05.354Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone add-proxy-arp 10.151.2.164 10.151.2.227 a8:e1:de:01:70:1d
5612023-12-13T06:03:05.355Z+ z_scadm dump-state
5622023-12-13T06:03:05.355Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone dump-state
5632023-12-13T06:03:05.407Zlocal v6:
5642023-12-13T06:03:05.407Zlocal v4:
5652023-12-13T06:03:05.408Zrouter v6:
5662023-12-13T06:03:05.408Zrouter v4_idx:
5672023-12-13T06:03:05.408Zrouter v4_routes:
5682023-12-13T06:03:05.408Zresolver v4:
5692023-12-13T06:03:05.408Z10.151.2.100 -> 18:c0:4d:0d:a0:2a
5702023-12-13T06:03:05.408Zresolver v6:
5712023-12-13T06:03:05.408Znat v4:
5722023-12-13T06:03:05.408Znat_v6:
5732023-12-13T06:03:05.408Zport_mac:
5742023-12-13T06:03:05.408Zproxy arp:
5752023-12-13T06:03:05.408Z10.151.2.164/10.151.2.227: a8:e1:de:01:70:1d
5762023-12-13T06:03:05.408Z+ success 'softnpu zone exists'
5772023-12-13T06:03:05.408Z+ set +x
5782023-12-13T06:03:05.408Zsoftnpu zone exists
5792023-12-13T06:03:05.408Z
5802023-12-13T06:03:05.408Zreal 40.273320935
5812023-12-13T06:03:05.408Zuser 0.717739271
5822023-12-13T06:03:05.408Zsys 1.617857865
5832023-12-13T06:03:05.408Ztrap 0.000092306
5842023-12-13T06:03:05.408Ztflt 0.000119639
5852023-12-13T06:03:05.408Zdflt 0.001400001
5862023-12-13T06:03:05.408Zkflt 0.000003300
5872023-12-13T06:03:05.409Zlock 31:32.954676053
5882023-12-13T06:03:05.409Zslp 2:00.741495748
5892023-12-13T06:03:05.409Zlat 0.076120360
5902023-12-13T06:03:05.409Zstop 0.352590487
5912023-12-13T06:03:05.410Z+ tar xf out/omicron-sled-agent.tar pkg/config-rss.toml
5922023-12-13T06:03:05.410Z++ sed -n 's/silo_name = "\(.*\)"/\1/p' pkg/config-rss.toml
5932023-12-13T06:03:05.410Z+ SILO_NAME=recovery
5942023-12-13T06:03:05.410Z++ sed -n 's/external_dns_zone_name = "\(.*\)"/\1/p' pkg/config-rss.toml
5952023-12-13T06:03:05.410Z+ EXTERNAL_DNS_DOMAIN=oxide.test
5962023-12-13T06:03:05.410Z+ sed -i~ '
5972023-12-13T06:03:05.410Z /^external_dns_ips/c\
5982023-12-13T06:03:05.410Zexternal_dns_ips = [ "10.151.2.164", "10.151.2.165" ]
5992023-12-13T06:03:05.410Z /^\[\[internal_services_ip_pool_ranges/,/^$/ {
6002023-12-13T06:03:05.410Z /^first/c\
6012023-12-13T06:03:05.410Zfirst = "10.151.2.164"
6022023-12-13T06:03:05.410Z /^last/c\
6032023-12-13T06:03:05.410Zlast = "10.151.2.173"
6042023-12-13T06:03:05.410Z }
6052023-12-13T06:03:05.410Z /^\[rack_network_config/,/^$/ {
6062023-12-13T06:03:05.410Z /^infra_ip_first/c\
6072023-12-13T06:03:05.410Zinfra_ip_first = "10.151.2.174"
6082023-12-13T06:03:05.410Z /^infra_ip_last/c\
6092023-12-13T06:03:05.410Zinfra_ip_last = "10.151.2.174"
6102023-12-13T06:03:05.410Z }
6112023-12-13T06:03:05.410Z /^\[\[rack_network_config.ports/,/^$/ {
6122023-12-13T06:03:05.410Z /^routes/c\
6132023-12-13T06:03:05.410Zroutes = \[{nexthop = "10.151.2.100", destination = "0.0.0.0/0"}\]
6142023-12-13T06:03:05.411Z /^addresses/c\
6152023-12-13T06:03:05.411Zaddresses = \["10.151.2.174/32"\]
6162023-12-13T06:03:05.412Z }
6172023-12-13T06:03:05.412Z' pkg/config-rss.toml
6182023-12-13T06:03:05.412Z+ diff -u pkg/config-rss.toml~ pkg/config-rss.toml
6192023-12-13T06:03:05.412Z--- pkg/config-rss.toml~ Mon Jul 24 01:21:28 2006
6202023-12-13T06:03:05.412Z+++ pkg/config-rss.toml Wed Dec 13 06:03:04 2023
6212023-12-13T06:03:05.412Z@@ -41,7 +41,7 @@
6222023-12-13T06:03:05.412Z # the DNS domain delegated to the rack by the customer. Each of these addresses
6232023-12-13T06:03:05.412Z # must be contained in one of the "internal services" IP Pool ranges listed
6242023-12-13T06:03:05.412Z # below.
6252023-12-13T06:03:05.412Z-external_dns_ips = [ "192.168.1.20", "192.168.1.21" ]
6262023-12-13T06:03:05.412Z+external_dns_ips = [ "10.151.2.164", "10.151.2.165" ]
6272023-12-13T06:03:05.412Z
6282023-12-13T06:03:05.412Z # Initial TLS certificates for the external API
6292023-12-13T06:03:05.412Z #
6302023-12-13T06:03:05.412Z@@ -77,8 +77,8 @@
6312023-12-13T06:03:05.412Z #
6322023-12-13T06:03:05.412Z # For more on this and what to put here, see docs/how-to-run.adoc.
6332023-12-13T06:03:05.412Z [[internal_services_ip_pool_ranges]]
6342023-12-13T06:03:05.412Z-first = "192.168.1.20"
6352023-12-13T06:03:05.412Z-last = "192.168.1.29"
6362023-12-13T06:03:05.413Z+first = "10.151.2.164"
6372023-12-13T06:03:05.413Z+last = "10.151.2.173"
6382023-12-13T06:03:05.413Z
6392023-12-13T06:03:05.413Z # TODO - this configuration is subject to change going forward. Ultimately these
6402023-12-13T06:03:05.413Z # parameters should be provided to the control plane via wicket, but we need to
6412023-12-13T06:03:05.413Z@@ -92,8 +92,8 @@
6422023-12-13T06:03:05.413Z # A range of IP addresses used by Boundary Services on the external network. In
6432023-12-13T06:03:05.415Z # a real system, these would be addresses of the uplink ports on the Sidecar.
6442023-12-13T06:03:05.415Z # With softnpu, only one address is used.
6452023-12-13T06:03:05.415Z-infra_ip_first = "192.168.1.30"
6462023-12-13T06:03:05.415Z-infra_ip_last = "192.168.1.30"
6472023-12-13T06:03:05.415Z+infra_ip_first = "10.151.2.174"
6482023-12-13T06:03:05.415Z+infra_ip_last = "10.151.2.174"
6492023-12-13T06:03:05.415Z
6502023-12-13T06:03:05.415Z # Configurations for BGP routers to run on the scrimlets.
6512023-12-13T06:03:05.415Z bgp = []
6522023-12-13T06:03:05.415Z@@ -101,9 +101,9 @@
6532023-12-13T06:03:05.415Z # You can configure multiple uplinks by repeating the following stanza
6542023-12-13T06:03:05.415Z [[rack_network_config.ports]]
6552023-12-13T06:03:05.415Z # Routes associated with this port.
6562023-12-13T06:03:05.415Z-routes = [{nexthop = "192.168.1.199", destination = "0.0.0.0/0"}]
6572023-12-13T06:03:05.415Z+routes = [{nexthop = "10.151.2.100", destination = "0.0.0.0/0"}]
6582023-12-13T06:03:05.415Z # Addresses associated with this port.
6592023-12-13T06:03:05.415Z-addresses = ["192.168.1.30/32"]
6602023-12-13T06:03:05.415Z+addresses = ["10.151.2.174/32"]
6612023-12-13T06:03:05.415Z # Name of the uplink port. This should always be "qsfp0" when using softnpu.
6622023-12-13T06:03:05.415Z port = "qsfp0"
6632023-12-13T06:03:05.458Z # The speed of this port.
6642023-12-13T06:03:05.458Z+ true
6652023-12-13T06:03:05.458Z+ tar rvf out/omicron-sled-agent.tar pkg/config-rss.toml
6662023-12-13T06:03:05.458Za pkg/config-rss.toml 7K
6672023-12-13T06:03:05.458Z+ rm -f pkg/config-rss.toml pkg/config-rss.toml~
6682023-12-13T06:03:05.458Z+ TLS_NAME=recovery.sys.oxide.test
6692023-12-13T06:03:05.459Z+ openssl req -newkey rsa:4096 -x509 -sha256 -days 3 -nodes -out pkg/initial-tls-cert.pem -keyout pkg/initial-tls-key.pem -subj /CN=recovery.sys.oxide.test -addext subjectAltName=DNS:recovery.sys.oxide.test -addext basicConstraints=critical,CA:FALSE -config /dev/stdin
6702023-12-13T06:03:05.517Z.+...+.........+....+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*..+...+.......+...+...+...........+.+..+...+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*..+............+...+..+.........+......+....+......+........+.......+.....+.......+..........................+....+...+..............+.........+.........+.......+...+...+.................+............+.........+.+.......................+....................................+....+...+........+........................................+......+........................+..+...................+...+..+......+.........+....+......+...+.....+...+...+............+...+.+.....+....+......+.....+....+..+.........+...+................+..............+...+............+.+..............+.+.....+......+.........+.+......+..+.....................+.+.........+...............+..+.........+...................+.........+..+...+.......+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
6712023-12-13T06:03:05.690Z...+...+..+.+..+....+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.......+..+....+.....+......+.+...+......+.....+...+...+....+...+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.................+.+.....+.......+............+......+.....+..........+.....+...............+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
6722023-12-13T06:03:05.715Z-----
6732023-12-13T06:03:05.715Z+ tar rvf out/omicron-sled-agent.tar pkg/initial-tls-cert.pem pkg/initial-tls-key.pem
6742023-12-13T06:03:05.754Za pkg/initial-tls-cert.pem 2K
6752023-12-13T06:03:05.754Za pkg/initial-tls-key.pem 4K
6762023-12-13T06:03:05.754Z+ rm -f pkg/initial-tls-cert.pem pkg/initial-tls-key.pem
6772023-12-13T06:03:05.754Z+ rmdir pkg
6782023-12-13T06:03:05.754Z+ E2E_TLS_CERT=/opt/oxide/sled-agent/pkg/initial-tls-cert.pem
6792023-12-13T06:03:05.754Z+ pfexec mkdir -p /usr/oxide
6802023-12-13T06:03:05.754Z+ pfexec curl -sSfL -o /usr/oxide/oxide http://catacomb.eng.oxide.computer:12346/oxide-v0.1.0
6812023-12-13T06:03:06.426Z+ pfexec chmod +x /usr/oxide/oxide
6822023-12-13T06:03:06.453Z+ curl -sSfL -o debian-11-genericcloud-amd64.raw http://catacomb.eng.oxide.computer:12346/debian-11-genericcloud-amd64.raw
6832023-12-13T06:03:24.722Z+ OMICRON_NO_UNINSTALL=1
6842023-12-13T06:03:24.723Z+ ptime -m pfexec ./target/release/omicron-package -t test install
6852023-12-13T06:03:24.747ZDec 13 06:03:24.619 DEBG target[test]: Target({"image": "standard", "machine": "non-gimlet", "rack-topology": "single-sled", "switch": "softnpu"})
6862023-12-13T06:03:24.747ZDec 13 06:03:24.621 INFO Installing service, dst: /opt/oxide/clickhouse.tar.gz, src: out/clickhouse.tar.gz
6872023-12-13T06:03:24.747ZDec 13 06:03:24.622 INFO Installing service, dst: /opt/oxide/crucible.tar.gz, src: out/crucible.tar.gz
6882023-12-13T06:03:24.747ZDec 13 06:03:24.622 INFO Installing service, dst: /opt/oxide/external_dns.tar.gz, src: out/external-dns.tar.gz
6892023-12-13T06:03:24.747ZDec 13 06:03:24.622 INFO Installing service, dst: /opt/oxide/propolis-server.tar.gz, src: out/propolis-server.tar.gz
6902023-12-13T06:03:24.747ZDec 13 06:03:24.622 INFO Installing service, dst: /opt/oxide/nexus.tar.gz, src: out/omicron-nexus.tar.gz
6912023-12-13T06:03:24.747ZDec 13 06:03:24.622 INFO Installing service, dst: /opt/oxide/cockroachdb.tar.gz, src: out/cockroachdb.tar.gz
6922023-12-13T06:03:24.757ZDec 13 06:03:24.622 INFO Installing service, dst: /opt/oxide/switch.tar.gz, src: out/switch-softnpu.tar.gz
6932023-12-13T06:03:24.757ZDec 13 06:03:24.622 INFO Installing service, dst: /opt/oxide/internal_dns.tar.gz, src: out/internal-dns.tar.gz
6942023-12-13T06:03:24.757ZDec 13 06:03:24.622 INFO Installing service, dst: /opt/oxide/sled-agent.tar, src: out/omicron-sled-agent.tar
6952023-12-13T06:03:24.757ZDec 13 06:03:24.622 INFO Installing service, dst: /opt/oxide/overlay.tar.gz, src: out/overlay.tar.gz
6962023-12-13T06:03:24.757ZDec 13 06:03:24.622 INFO Installing service, dst: /opt/oxide/mgs.tar.gz, src: out/omicron-gateway-softnpu.tar.gz
6972023-12-13T06:03:24.757ZDec 13 06:03:24.622 INFO Installing service, dst: /opt/oxide/crucible_pantry.tar.gz, src: out/crucible-pantry.tar.gz
6982023-12-13T06:03:24.757ZDec 13 06:03:24.622 INFO Installing service, dst: /opt/oxide/clickhouse_keeper.tar.gz, src: out/clickhouse_keeper.tar.gz
6992023-12-13T06:03:24.757ZDec 13 06:03:24.622 INFO Installing service, dst: /opt/oxide/oximeter.tar.gz, src: out/oximeter-collector.tar.gz
7002023-12-13T06:03:24.757ZDec 13 06:03:24.622 INFO Installing service, dst: /opt/oxide/ntp.tar.gz, src: out/ntp.tar.gz
7012023-12-13T06:03:24.757ZDec 13 06:03:24.622 INFO Installing service, dst: /opt/oxide/mg-ddm.tar, src: out/mg-ddm-gz.tar
7022023-12-13T06:03:24.961ZDec 13 06:03:24.842 INFO Unpacking service tarball, service_path: /opt/oxide/mg-ddm, tar_path: /opt/oxide/mg-ddm.tar
7032023-12-13T06:03:24.997ZDec 13 06:03:24.878 INFO Unpacking service tarball, service_path: /opt/oxide/sled-agent, tar_path: /opt/oxide/sled-agent.tar
7042023-12-13T06:03:25.060ZDec 13 06:03:24.938 INFO Installing bootstrap service from /opt/oxide/sled-agent/pkg/manifest.xml
7052023-12-13T06:03:25.186Z
7062023-12-13T06:03:25.226Zreal 0.457474383
7072023-12-13T06:03:25.226Zuser 0.098810672
7082023-12-13T06:03:25.226Zsys 1.868691515
7092023-12-13T06:03:25.226Ztrap 0.000000380
7102023-12-13T06:03:25.226Ztflt 0.000026630
7112023-12-13T06:03:25.229Zdflt 0.000983328
7122023-12-13T06:03:25.229Zkflt 0.000002650
7132023-12-13T06:03:25.229Zlock 40.907403453
7142023-12-13T06:03:25.229Zslp 0.665443306
7152023-12-13T06:03:25.230Zlat 0.003208286
7162023-12-13T06:03:25.230Zstop 0.196408499
7172023-12-13T06:03:25.230Z+ retry=0
7182023-12-13T06:03:25.230Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7192023-12-13T06:03:25.230Z+ [[ 0 -gt 30 ]]
7202023-12-13T06:03:25.230Z+ sleep 1
7212023-12-13T06:03:26.200Z+ retry=1
7222023-12-13T06:03:26.200Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7232023-12-13T06:03:26.227Z+ [[ 1 -gt 30 ]]
7242023-12-13T06:03:26.227Z+ sleep 1
7252023-12-13T06:03:27.216Z+ retry=2
7262023-12-13T06:03:27.217Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7272023-12-13T06:03:27.242Z+ [[ 2 -gt 30 ]]
7282023-12-13T06:03:27.242Z+ sleep 1
7292023-12-13T06:03:28.230Z+ retry=3
7302023-12-13T06:03:28.230Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7312023-12-13T06:03:28.255Z+ [[ 3 -gt 30 ]]
7322023-12-13T06:03:28.255Z+ sleep 1
7332023-12-13T06:03:29.245Z+ retry=4
7342023-12-13T06:03:29.245Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7352023-12-13T06:03:54.428Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone dump-state
7362023-12-13T06:03:54.455Zlocal v6:
7372023-12-13T06:03:54.455Zfe80::6c46:42ff:fead:9c78
7382023-12-13T06:03:54.456Zfd00:99::1
7392023-12-13T06:03:54.456Zlocal v4:
7402023-12-13T06:03:54.456Z10.151.2.174
7412023-12-13T06:03:54.456Zrouter v6:
7422023-12-13T06:03:54.456Zfd00:1122:3344:101::/64 -> fe80::38d1:73ff:fe70:b208 (1)
7432023-12-13T06:03:54.456Zfdb0:18c0:4d0d:a02a::/64 -> fe80::38d1:73ff:fe70:b208 (1)
7442023-12-13T06:03:54.456Zfd00:1122:3344:3::/64 -> fe80::38d1:73ff:fe70:b208 (1)
7452023-12-13T06:03:54.456Zrouter v4_idx:
7462023-12-13T06:03:54.456Z0.0.0.0/0 -> 0
7472023-12-13T06:03:54.456Zrouter v4_routes:
7482023-12-13T06:03:54.456Z0 -> 10.151.2.100 (2)
7492023-12-13T06:03:54.456Zresolver v4:
7502023-12-13T06:03:54.457Z10.151.2.174 -> ca:65:22:1d:e5:be
7512023-12-13T06:03:54.457Z10.151.2.100 -> 18:c0:4d:0d:a0:2a
7522023-12-13T06:03:54.457Zresolver v6:
7532023-12-13T06:03:54.457Zfe80::38d1:73ff:fe70:b208 -> 3a:d1:73:70:b2:08
7542023-12-13T06:03:54.457Zfe80::6c46:42ff:fead:9c78 -> 6e:46:42:ad:9c:78
7552023-12-13T06:03:54.457Znat v4:
7562023-12-13T06:03:54.457Znat_v6:
7572023-12-13T06:03:54.457Zport_mac:
7582023-12-13T06:03:54.457Z1000: a8:40:25:41:cd:19
7592023-12-13T06:03:54.457Z1: a8:40:25:41:cd:1a
7602023-12-13T06:03:54.457Zproxy arp:
7612023-12-13T06:03:54.457Z10.151.2.164/10.151.2.227: a8:e1:de:01:70:1d
7622023-12-13T06:03:54.457Z+ retry=0
7632023-12-13T06:03:54.457Z+++ zoneadm list -n
7642023-12-13T06:03:54.457Z+++ grep oxz_ntp
7652023-12-13T06:03:54.575Z++ pfexec svcs -z -Hostate oxide/ntp
7662023-12-13T06:03:54.575Zsvcs: zone '-Hostate': entity not found
7672023-12-13T06:03:54.575Z++ true
7682023-12-13T06:03:54.575Z+ [[ '' != online ]]
7692023-12-13T06:03:54.575Z+ [[ 0 -gt 60 ]]
7702023-12-13T06:03:54.575Z+ sleep 1
7712023-12-13T06:03:55.480Z+ retry=1
7722023-12-13T06:03:55.508Z+++ zoneadm list -n
7732023-12-13T06:03:55.508Z+++ grep oxz_ntp
7742023-12-13T06:03:55.509Z++ pfexec svcs -z -Hostate oxide/ntp
7752023-12-13T06:03:55.509Zsvcs: zone '-Hostate': entity not found
7762023-12-13T06:03:55.509Z++ true
7772023-12-13T06:03:55.509Z+ [[ '' != online ]]
7782023-12-13T06:03:55.509Z+ [[ 1 -gt 60 ]]
7792023-12-13T06:03:55.510Z+ sleep 1
7802023-12-13T06:03:56.504Z+ retry=2
7812023-12-13T06:03:56.529Z+++ zoneadm list -n
7822023-12-13T06:03:56.529Z+++ grep oxz_ntp
7832023-12-13T06:03:56.529Z++ pfexec svcs -z -Hostate oxide/ntp
7842023-12-13T06:03:56.529Zsvcs: zone '-Hostate': entity not found
7852023-12-13T06:03:56.529Z++ true
7862023-12-13T06:03:56.530Z+ [[ '' != online ]]
7872023-12-13T06:03:56.530Z+ [[ 2 -gt 60 ]]
7882023-12-13T06:03:56.530Z+ sleep 1
7892023-12-13T06:03:57.554Z+ retry=3
7902023-12-13T06:03:57.580Z+++ zoneadm list -n
7912023-12-13T06:03:57.580Z+++ grep oxz_ntp
7922023-12-13T06:03:57.580Z++ pfexec svcs -z -Hostate oxide/ntp
7932023-12-13T06:03:57.580Zsvcs: zone '-Hostate': entity not found
7942023-12-13T06:03:57.580Z++ true
7952023-12-13T06:03:57.580Z+ [[ '' != online ]]
7962023-12-13T06:03:57.580Z+ [[ 3 -gt 60 ]]
7972023-12-13T06:03:57.580Z+ sleep 1
7982023-12-13T06:03:58.534Z+ retry=4
7992023-12-13T06:03:58.592Z+++ zoneadm list -n
8002023-12-13T06:03:58.592Z+++ grep oxz_ntp
8012023-12-13T06:03:58.592Z++ pfexec svcs -z -Hostate oxide/ntp
8022023-12-13T06:03:58.592Zsvcs: zone '-Hostate': entity not found
8032023-12-13T06:03:58.592Z++ true
8042023-12-13T06:03:58.592Z+ [[ '' != online ]]
8052023-12-13T06:03:58.592Z+ [[ 4 -gt 60 ]]
8062023-12-13T06:03:58.593Z+ sleep 1
8072023-12-13T06:03:59.560Z+ retry=5
8082023-12-13T06:03:59.622Z+++ zoneadm list -n
8092023-12-13T06:03:59.622Z+++ grep oxz_ntp
8102023-12-13T06:03:59.622Z++ pfexec svcs -z oxz_ntp_22f7e918-79ab-4ea0-a1bc-53174a8cc11d -Hostate oxide/ntp
8112023-12-13T06:03:59.656Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
8122023-12-13T06:03:59.656Z++ true
8132023-12-13T06:03:59.656Z+ [[ '' != online ]]
8142023-12-13T06:03:59.656Z+ [[ 5 -gt 60 ]]
8152023-12-13T06:03:59.656Z+ sleep 1
8162023-12-13T06:04:00.618Z+ retry=6
8172023-12-13T06:04:00.659Z+++ zoneadm list -n
8182023-12-13T06:04:00.659Z+++ grep oxz_ntp
8192023-12-13T06:04:00.659Z++ pfexec svcs -z oxz_ntp_22f7e918-79ab-4ea0-a1bc-53174a8cc11d -Hostate oxide/ntp
8202023-12-13T06:04:00.659Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
8212023-12-13T06:04:00.659Z++ true
8222023-12-13T06:04:00.659Z+ [[ '' != online ]]
8232023-12-13T06:04:00.659Z+ [[ 6 -gt 60 ]]
8242023-12-13T06:04:00.659Z+ sleep 1
8252023-12-13T06:04:01.640Z+ retry=7
8262023-12-13T06:04:02.077Z+++ zoneadm list -n
8272023-12-13T06:04:02.077Z+++ grep oxz_ntp
8282023-12-13T06:04:02.077Z++ pfexec svcs -z oxz_ntp_22f7e918-79ab-4ea0-a1bc-53174a8cc11d -Hostate oxide/ntp
8292023-12-13T06:04:02.077Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
8302023-12-13T06:04:02.077Z++ true
8312023-12-13T06:04:02.077Z+ [[ '' != online ]]
8322023-12-13T06:04:02.077Z+ [[ 7 -gt 60 ]]
8332023-12-13T06:04:02.077Z+ sleep 1
8342023-12-13T06:04:02.667Z+ retry=8
8352023-12-13T06:04:02.693Z+++ zoneadm list -n
8362023-12-13T06:04:02.693Z+++ grep oxz_ntp
8372023-12-13T06:04:02.693Z++ pfexec svcs -z oxz_ntp_22f7e918-79ab-4ea0-a1bc-53174a8cc11d -Hostate oxide/ntp
8382023-12-13T06:04:02.693Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
8392023-12-13T06:04:02.693Z++ true
8402023-12-13T06:04:02.746Z+ [[ '' != online ]]
8412023-12-13T06:04:02.746Z+ [[ 8 -gt 60 ]]
8422023-12-13T06:04:02.747Z+ sleep 1
8432023-12-13T06:04:03.697Z+ retry=9
8442023-12-13T06:04:03.725Z+++ zoneadm list -n
8452023-12-13T06:04:03.725Z+++ grep oxz_ntp
8462023-12-13T06:04:03.726Z++ pfexec svcs -z oxz_ntp_22f7e918-79ab-4ea0-a1bc-53174a8cc11d -Hostate oxide/ntp
8472023-12-13T06:04:03.726Z+ [[ disabled != online ]]
8482023-12-13T06:04:03.726Z+ [[ 9 -gt 60 ]]
8492023-12-13T06:04:03.726Z+ sleep 1
8502023-12-13T06:04:04.720Z+ retry=10
8512023-12-13T06:04:04.750Z+++ zoneadm list -n
8522023-12-13T06:04:04.750Z+++ grep oxz_ntp
8532023-12-13T06:04:04.750Z++ pfexec svcs -z oxz_ntp_22f7e918-79ab-4ea0-a1bc-53174a8cc11d -Hostate oxide/ntp
8542023-12-13T06:04:04.750Z+ [[ disabled != online ]]
8552023-12-13T06:04:04.750Z+ [[ 10 -gt 60 ]]
8562023-12-13T06:04:04.750Z+ sleep 1
8572023-12-13T06:04:05.741Z+ retry=11
8582023-12-13T06:04:05.766Z+++ zoneadm list -n
8592023-12-13T06:04:05.766Z+++ grep oxz_ntp
8602023-12-13T06:04:05.766Z++ pfexec svcs -z oxz_ntp_22f7e918-79ab-4ea0-a1bc-53174a8cc11d -Hostate oxide/ntp
8612023-12-13T06:04:05.766Z+ [[ disabled != online ]]
8622023-12-13T06:04:05.766Z+ [[ 11 -gt 60 ]]
8632023-12-13T06:04:05.767Z+ sleep 1
8642023-12-13T06:04:06.763Z+ retry=12
8652023-12-13T06:04:06.795Z+++ zoneadm list -n
8662023-12-13T06:04:06.795Z+++ grep oxz_ntp
8672023-12-13T06:04:06.796Z++ pfexec svcs -z oxz_ntp_22f7e918-79ab-4ea0-a1bc-53174a8cc11d -Hostate oxide/ntp
8682023-12-13T06:04:06.796Z+ [[ disabled != online ]]
8692023-12-13T06:04:06.796Z+ [[ 12 -gt 60 ]]
8702023-12-13T06:04:06.796Z+ sleep 1
8712023-12-13T06:04:08.204Z+ retry=13
8722023-12-13T06:04:08.231Z+++ zoneadm list -n
8732023-12-13T06:04:08.231Z+++ grep oxz_ntp
8742023-12-13T06:04:08.233Z++ pfexec svcs -z oxz_ntp_22f7e918-79ab-4ea0-a1bc-53174a8cc11d -Hostate oxide/ntp
8752023-12-13T06:04:08.233Z+ [[ disabled != online ]]
8762023-12-13T06:04:08.233Z+ [[ 13 -gt 60 ]]
8772023-12-13T06:04:08.233Z+ sleep 1
8782023-12-13T06:04:09.216Z+ retry=14
8792023-12-13T06:04:09.252Z+++ zoneadm list -n
8802023-12-13T06:04:09.252Z+++ grep oxz_ntp
8812023-12-13T06:04:09.252Z++ pfexec svcs -z oxz_ntp_22f7e918-79ab-4ea0-a1bc-53174a8cc11d -Hostate oxide/ntp
8822023-12-13T06:04:09.252Z+ [[ disabled != online ]]
8832023-12-13T06:04:09.252Z+ [[ 14 -gt 60 ]]
8842023-12-13T06:04:09.252Z+ sleep 1
8852023-12-13T06:04:09.829Z+ retry=15
8862023-12-13T06:04:09.859Z+++ zoneadm list -n
8872023-12-13T06:04:09.859Z+++ grep oxz_ntp
8882023-12-13T06:04:09.859Z++ pfexec svcs -z oxz_ntp_22f7e918-79ab-4ea0-a1bc-53174a8cc11d -Hostate oxide/ntp
8892023-12-13T06:04:09.859Z+ [[ disabled != online ]]
8902023-12-13T06:04:09.860Z+ [[ 15 -gt 60 ]]
8912023-12-13T06:04:09.860Z+ sleep 1
8922023-12-13T06:04:10.852Z+ retry=16
8932023-12-13T06:04:10.891Z+++ zoneadm list -n
8942023-12-13T06:04:10.891Z+++ grep oxz_ntp
8952023-12-13T06:04:10.891Z++ pfexec svcs -z oxz_ntp_22f7e918-79ab-4ea0-a1bc-53174a8cc11d -Hostate oxide/ntp
8962023-12-13T06:04:10.891Z+ [[ disabled != online ]]
8972023-12-13T06:04:10.891Z+ [[ 16 -gt 60 ]]
8982023-12-13T06:04:10.891Z+ sleep 1
8992023-12-13T06:04:11.872Z+ retry=17
9002023-12-13T06:04:11.898Z+++ zoneadm list -n
9012023-12-13T06:04:11.898Z+++ grep oxz_ntp
9022023-12-13T06:04:11.898Z++ pfexec svcs -z oxz_ntp_22f7e918-79ab-4ea0-a1bc-53174a8cc11d -Hostate oxide/ntp
9032023-12-13T06:04:11.898Z+ [[ disabled != online ]]
9042023-12-13T06:04:11.898Z+ [[ 17 -gt 60 ]]
9052023-12-13T06:04:11.898Z+ sleep 1
 
9062023-12-13T06:04:11.926Zsock console login: Dec 13 06:04:11 sock ip: ip: joining multicasts failed (4) on vopte0 - will use link layer broadcasts for multicast
 
9072023-12-13T06:04:12.894Z+ retry=18
9082023-12-13T06:04:12.919Z+++ zoneadm list -n
9092023-12-13T06:04:12.919Z+++ grep oxz_ntp
9102023-12-13T06:04:12.919Z++ pfexec svcs -z oxz_ntp_22f7e918-79ab-4ea0-a1bc-53174a8cc11d -Hostate oxide/ntp
9112023-12-13T06:04:12.919Z+ [[ online != online ]]
9122023-12-13T06:04:12.919Z+ echo 'Waited for chrony: 18s'
9132023-12-13T06:04:12.919ZWaited for chrony: 18s
9142023-12-13T06:04:12.919Z+ export RUST_BACKTRACE=1
9152023-12-13T06:04:12.919Z+ RUST_BACKTRACE=1
9162023-12-13T06:04:12.919Z+ export E2E_TLS_CERT IPPOOL_START IPPOOL_END
9172023-12-13T06:04:12.920Z++ ./tests/bootstrap
9182023-12-13T06:04:12.944ZUsing external DNS server at 10.151.2.164:53
9192023-12-13T06:04:12.945Z2023-12-13 06:04:12.822052436 UTC: attempting to log into API
9202023-12-13T06:04:27.978Z2023-12-13 06:04:27.686203649 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9212023-12-13T06:04:28.978Z2023-12-13 06:04:28.687206695 UTC: attempting to log into API
9222023-12-13T06:04:44.012Z2023-12-13 06:04:43.720853529 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9232023-12-13T06:04:45.046Z2023-12-13 06:04:44.721554080 UTC: attempting to log into API
9242023-12-13T06:05:09.208Z2023-12-13 06:04:59.764825779 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9252023-12-13T06:05:09.208Z2023-12-13 06:05:00.765702283 UTC: attempting to log into API
9262023-12-13T06:05:16.098Z2023-12-13 06:05:15.806824653 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9272023-12-13T06:05:17.099Z2023-12-13 06:05:16.807824369 UTC: attempting to log into API
9282023-12-13T06:05:47.287Z2023-12-13 06:05:32.585946441 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
 
9292023-12-13T06:05:47.330ZDec 13 06:05:31 sock ip: ip: joining multicasts failed (4) on vopte3 - will use link layer broadcasts for multicast
 
9302023-12-13T06:05:47.369Z2023-12-13 06:05:33.584203036 UTC: attempting to log into API
 
9312023-12-13T06:05:47.442ZDec 13 06:05:36 sock ip: ip: joining multicasts failed (4) on vopte1 - will use link layer broadcasts for multicast
9322023-12-13T06:05:47.495ZDec 13 06:05:43 sock ip: ip: joining multicasts failed (4) on vopte2 - will use link layer broadcasts for multicast
 
9332023-12-13T06:05:48.205Z2023-12-13 06:05:48.576718333 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9342023-12-13T06:05:49.273Z2023-12-13 06:05:49.574939868 UTC: attempting to log into API
 
9352023-12-13T06:05:51.264ZDec 13 06:05:51 sock ip: ip: joining multicasts failed (4) on vopte5 - will use link layer broadcasts for multicast
9362023-12-13T06:05:56.872ZDec 13 06:05:56 sock ip: ip: joining multicasts failed (4) on vopte4 - will use link layer broadcasts for multicast
 
9372023-12-13T06:06:04.220Z2023-12-13 06:06:04.566770941 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9382023-12-13T06:06:05.220Z2023-12-13 06:06:05.565035967 UTC: attempting to log into API
9392023-12-13T06:06:20.252Z2023-12-13 06:06:20.556282350 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9402023-12-13T06:06:21.252Z2023-12-13 06:06:21.554568826 UTC: attempting to log into API
9412023-12-13T06:06:27.037Z2023-12-13 06:06:27.323704969 UTC: login succeeded
9422023-12-13T06:06:27.070Zwaiting for nexus to come up...
9432023-12-13T06:06:27.133Zcreating IP pool... 10.151.2.175 - 10.151.2.227
9442023-12-13T06:06:27.179Zsetting up quotas...
9452023-12-13T06:06:27.215Zensuring datasets are ready...
9462023-12-13T06:06:29.548Zlet's roll.
9472023-12-13T06:06:29.575Z+ eval 'OXIDE_HOST=https://recovery.sys.oxide.test
9482023-12-13T06:06:29.575ZOXIDE_RESOLVE=recovery.sys.oxide.test:443:10.151.2.166
9492023-12-13T06:06:29.575ZOXIDE_TOKEN=oxide-token-8df74372efb9faeeeb3f884561672badc5a8a071'
9502023-12-13T06:06:29.576Z++ OXIDE_HOST=https://recovery.sys.oxide.test
9512023-12-13T06:06:29.576Z++ OXIDE_RESOLVE=recovery.sys.oxide.test:443:10.151.2.166
9522023-12-13T06:06:29.579Z++ OXIDE_TOKEN=oxide-token-8df74372efb9faeeeb3f884561672badc5a8a071
9532023-12-13T06:06:29.579Z+ export OXIDE_HOST OXIDE_TOKEN
9542023-12-13T06:06:29.579Z+ retry=0
9552023-12-13T06:06:29.579Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
9562023-12-13T06:06:29.630Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
9572023-12-13T06:06:29.630Z+ [[ 0 -gt 60 ]]
9582023-12-13T06:06:29.630Z+ sleep 1
9592023-12-13T06:06:30.578Z+ retry=1
9602023-12-13T06:06:30.578Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
9612023-12-13T06:06:30.602Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
9622023-12-13T06:06:30.603Z+ [[ 1 -gt 60 ]]
9632023-12-13T06:06:30.603Z+ sleep 1
9642023-12-13T06:06:31.595Z+ retry=2
9652023-12-13T06:06:31.595Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
9662023-12-13T06:06:31.623Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
9672023-12-13T06:06:31.624Z+ [[ 2 -gt 60 ]]
9682023-12-13T06:06:31.624Z+ sleep 1
9692023-12-13T06:06:32.613Z+ retry=3
9702023-12-13T06:06:32.613Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
9712023-12-13T06:06:32.639Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
9722023-12-13T06:06:32.639Z+ [[ 3 -gt 60 ]]
9732023-12-13T06:06:32.639Z+ sleep 1
9742023-12-13T06:06:33.633Z+ retry=4
9752023-12-13T06:06:33.633Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
9762023-12-13T06:06:33.678Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
9772023-12-13T06:06:33.678Z+ [[ 4 -gt 60 ]]
9782023-12-13T06:06:33.678Z+ sleep 1
9792023-12-13T06:06:34.672Z+ retry=5
9802023-12-13T06:06:34.672Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
9812023-12-13T06:06:34.672Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
9822023-12-13T06:06:34.672Z+ [[ 5 -gt 60 ]]
9832023-12-13T06:06:34.672Z+ sleep 1
9842023-12-13T06:06:35.674Z+ retry=6
9852023-12-13T06:06:35.674Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
9862023-12-13T06:06:35.704Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
9872023-12-13T06:06:35.704Z+ [[ 6 -gt 60 ]]
9882023-12-13T06:06:35.704Z+ sleep 1
9892023-12-13T06:06:36.692Z+ retry=7
9902023-12-13T06:06:36.692Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
9912023-12-13T06:06:36.718Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
9922023-12-13T06:06:36.718Z+ [[ 7 -gt 60 ]]
9932023-12-13T06:06:36.719Z+ sleep 1
9942023-12-13T06:06:37.711Z+ retry=8
9952023-12-13T06:06:37.715Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
9962023-12-13T06:06:37.740Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
9972023-12-13T06:06:37.740Z+ [[ 8 -gt 60 ]]
9982023-12-13T06:06:37.740Z+ sleep 1
9992023-12-13T06:06:38.731Z+ retry=9
10002023-12-13T06:06:38.732Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
10012023-12-13T06:06:38.756Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
10022023-12-13T06:06:38.756Z+ [[ 9 -gt 60 ]]
10032023-12-13T06:06:38.756Z+ sleep 1
10042023-12-13T06:06:39.757Z+ retry=10
10052023-12-13T06:06:39.757Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
10062023-12-13T06:06:39.784Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
10072023-12-13T06:06:39.784Z+ [[ 10 -gt 60 ]]
10082023-12-13T06:06:39.784Z+ sleep 1
10092023-12-13T06:06:40.779Z+ retry=11
10102023-12-13T06:06:40.779Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
10112023-12-13T06:06:40.813Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
10122023-12-13T06:06:40.813Z+ [[ 11 -gt 60 ]]
10132023-12-13T06:06:40.813Z+ sleep 1
10142023-12-13T06:06:41.796Z+ retry=12
10152023-12-13T06:06:41.796Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
10162023-12-13T06:06:41.827Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
10172023-12-13T06:06:41.828Z+ [[ 12 -gt 60 ]]
10182023-12-13T06:06:41.828Z+ sleep 1
10192023-12-13T06:06:42.815Z+ retry=13
10202023-12-13T06:06:42.815Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
10212023-12-13T06:06:42.846Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
10222023-12-13T06:06:42.846Z+ [[ 13 -gt 60 ]]
10232023-12-13T06:06:42.846Z+ sleep 1
10242023-12-13T06:06:43.845Z+ retry=14
10252023-12-13T06:06:43.845Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
10262023-12-13T06:06:43.872Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
10272023-12-13T06:06:43.872Z+ [[ 14 -gt 60 ]]
10282023-12-13T06:06:43.873Z+ sleep 1
10292023-12-13T06:06:44.895Z+ retry=15
10302023-12-13T06:06:44.895Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
10312023-12-13T06:06:44.934Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
10322023-12-13T06:06:44.934Z+ [[ 15 -gt 60 ]]
10332023-12-13T06:06:44.934Z+ sleep 1
10342023-12-13T06:06:45.865Z+ retry=16
10352023-12-13T06:06:45.865Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
10362023-12-13T06:06:45.919Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
10372023-12-13T06:06:45.919Z+ [[ 16 -gt 60 ]]
10382023-12-13T06:06:45.919Z+ sleep 1
10392023-12-13T06:06:46.889Z+ retry=17
10402023-12-13T06:06:46.889Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
10412023-12-13T06:06:46.915Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
10422023-12-13T06:06:46.915Z+ [[ 17 -gt 60 ]]
10432023-12-13T06:06:46.915Z+ sleep 1
10442023-12-13T06:06:47.905Z+ retry=18
10452023-12-13T06:06:47.905Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
10462023-12-13T06:06:47.932Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
10472023-12-13T06:06:47.932Z+ [[ 18 -gt 60 ]]
10482023-12-13T06:06:47.932Z+ sleep 1
10492023-12-13T06:06:48.930Z+ retry=19
10502023-12-13T06:06:48.967Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
10512023-12-13T06:06:48.967Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
10522023-12-13T06:06:48.967Z+ [[ 19 -gt 60 ]]
10532023-12-13T06:06:48.969Z+ sleep 1
10542023-12-13T06:06:49.950Z+ retry=20
10552023-12-13T06:06:49.950Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
10562023-12-13T06:10:18.371Zcurl: (28) Failed to connect to recovery.sys.oxide.test port 443 after 194636 ms: Couldn't connect to server
10572023-12-13T06:10:18.371Z+ [[ 20 -gt 60 ]]
10582023-12-13T06:10:18.371Z+ sleep 1
10592023-12-13T06:10:18.371Z+ retry=21
10602023-12-13T06:10:18.371Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
10612023-12-13T06:10:18.372Z+ /usr/oxide/oxide --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem project create --name images --description 'some images'
10622023-12-13T06:10:18.372Z{"status":"ok"}success
10632023-12-13T06:10:18.372ZProject {
10642023-12-13T06:10:18.372Z description: "some images",
10652023-12-13T06:10:18.372Z id: 737e099f-1550-4f48-a4ab-cb1a2e7880b8,
10662023-12-13T06:10:18.372Z name: Name(
10672023-12-13T06:10:18.372Z "images",
10682023-12-13T06:10:18.372Z ),
10692023-12-13T06:10:18.372Z time_created: 2023-12-13T06:10:05.802348Z,
10702023-12-13T06:10:18.372Z time_modified: 2023-12-13T06:10:05.802348Z,
10712023-12-13T06:10:18.372Z}
10722023-12-13T06:10:18.372Z+ /usr/oxide/oxide --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem disk import --path debian-11-genericcloud-amd64.raw --disk debian11-boot --project images --description 'debian 11 cloud image from distros' --snapshot debian11-snapshot --image debian11 --image-description 'debian 11 original base image' --image-os debian --image-version 11
10732023-12-13T06:15:30.615Zdone!
10742023-12-13T06:15:30.716Z+ /usr/oxide/oxide --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem image promote --project images --image debian11
10752023-12-13T06:15:30.836Zsuccess
10762023-12-13T06:15:30.836ZImage {
10772023-12-13T06:15:30.836Z block_size: ByteCount(
10782023-12-13T06:15:30.896Z 512,
10792023-12-13T06:15:30.896Z ),
10802023-12-13T06:15:30.897Z description: "debian 11 original base image",
10812023-12-13T06:15:30.897Z digest: None,
10822023-12-13T06:15:30.897Z id: 5b2c5700-d3ef-4f28-8ecb-13af68c83162,
10832023-12-13T06:15:30.897Z name: Name(
10842023-12-13T06:15:30.898Z "debian11",
10852023-12-13T06:15:30.898Z ),
10862023-12-13T06:15:30.898Z os: "debian",
10872023-12-13T06:15:30.898Z project_id: None,
10882023-12-13T06:15:30.898Z size: ByteCount(
10892023-12-13T06:15:30.898Z 2147483648,
10902023-12-13T06:15:30.898Z ),
10912023-12-13T06:15:30.898Z time_created: 2023-12-13T06:15:30.550935Z,
10922023-12-13T06:15:30.898Z time_modified: 2023-12-13T06:15:30.806007Z,
10932023-12-13T06:15:30.898Z url: None,
10942023-12-13T06:15:30.898Z version: "11",
10952023-12-13T06:15:30.899Z}
10962023-12-13T06:15:30.899Z+ rm ./tests/bootstrap
10972023-12-13T06:15:30.942Z+ for test_bin in tests/*
10982023-12-13T06:15:30.942Z+ ./tests/bootstrap-8aed3472db5f37d6
10992023-12-13T06:15:30.942Z
11002023-12-13T06:15:30.943Zrunning 0 tests
11012023-12-13T06:15:30.943Z
11022023-12-13T06:15:30.943Ztest result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
11032023-12-13T06:15:30.943Z
11042023-12-13T06:15:30.943Z+ for test_bin in tests/*
11052023-12-13T06:15:30.943Z+ ./tests/end_to_end_tests-fd6b950b6a51f2ac
11062023-12-13T06:15:30.977Z
11072023-12-13T06:15:30.978Zrunning 2 tests
11082023-12-13T06:15:31.195Ztest no_spoof::no_spoof_here ... ok
11092023-12-13T06:16:05.126Ztest instance_launch::instance_launch ... ok
11102023-12-13T06:16:05.126Z
11112023-12-13T06:16:05.126Ztest result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 34.18s
11122023-12-13T06:16:05.126Z
11132023-12-13T06:16:05.152Z+ _exit_trap
11142023-12-13T06:16:05.152Z+ local status=0
11152023-12-13T06:16:05.152Z+ [[ 0 -eq 0 ]]
11162023-12-13T06:16:05.152Z+ exit 0
11172023-12-13T06:16:05.152Zprocess exited: duration 828504 ms, exit code 0
 
11182023-12-13T06:16:05.268Zfound 99 output files
11192023-12-13T06:16:05.268Zuploading: /var/svc/log/oxide-sled-agent:default.log (802230 bytes)
11202023-12-13T06:16:07.915Zuploaded: /var/svc/log/oxide-sled-agent:default.log
11212023-12-13T06:16:08.367Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_c303b3a2-6f29-460a-8139-052b23a6a319/root/var/svc/log/oxide-crucible-agent:default.log (0 bytes)
11222023-12-13T06:16:08.999Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_c303b3a2-6f29-460a-8139-052b23a6a319/root/var/svc/log/oxide-crucible-agent:default.log
11232023-12-13T06:16:08.999Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_c303b3a2-6f29-460a-8139-052b23a6a319/root/var/svc/log/oxide-crucible-agent:default.log.0 (3435 bytes)
11242023-12-13T06:16:10.103Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_c303b3a2-6f29-460a-8139-052b23a6a319/root/var/svc/log/oxide-crucible-agent:default.log.0
11252023-12-13T06:16:10.104Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_c303b3a2-6f29-460a-8139-052b23a6a319/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
11262023-12-13T06:16:11.198Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_c303b3a2-6f29-460a-8139-052b23a6a319/root/var/svc/log/oxide-crucible-downstairs:default.log
11272023-12-13T06:16:11.200Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_c303b3a2-6f29-460a-8139-052b23a6a319/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
11282023-12-13T06:16:12.365Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_c303b3a2-6f29-460a-8139-052b23a6a319/root/var/svc/log/oxide-crucible-downstairs:default.log.0
11292023-12-13T06:16:12.365Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_5bc3ac68-e2bb-436b-8959-928055e2bcd2/root/var/svc/log/oxide-crucible-pantry:default.log (89477 bytes)
11302023-12-13T06:16:13.599Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_5bc3ac68-e2bb-436b-8959-928055e2bcd2/root/var/svc/log/oxide-crucible-pantry:default.log
11312023-12-13T06:16:13.630Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_5bc3ac68-e2bb-436b-8959-928055e2bcd2/root/var/svc/log/oxide-crucible-pantry:default.log.0 (1138854 bytes)
11322023-12-13T06:16:14.979Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_5bc3ac68-e2bb-436b-8959-928055e2bcd2/root/var/svc/log/oxide-crucible-pantry:default.log.0
11332023-12-13T06:16:14.979Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_ab0c8660-3d5b-4623-a89f-c7f63f4d6ce6/root/var/svc/log/oxide-external_dns:default.log (3774 bytes)
11342023-12-13T06:16:16.102Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_ab0c8660-3d5b-4623-a89f-c7f63f4d6ce6/root/var/svc/log/oxide-external_dns:default.log
11352023-12-13T06:16:16.137Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_ab0c8660-3d5b-4623-a89f-c7f63f4d6ce6/root/var/svc/log/oxide-external_dns:default.log.0 (38266 bytes)
11362023-12-13T06:16:17.272Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_ab0c8660-3d5b-4623-a89f-c7f63f4d6ce6/root/var/svc/log/oxide-external_dns:default.log.0
11372023-12-13T06:16:17.303Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_34ed6b25-7087-4077-b0fc-3965fa7527a9/root/var/svc/log/oxide-crucible-agent:default.log (0 bytes)
11382023-12-13T06:16:18.365Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_34ed6b25-7087-4077-b0fc-3965fa7527a9/root/var/svc/log/oxide-crucible-agent:default.log
11392023-12-13T06:16:18.366Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_34ed6b25-7087-4077-b0fc-3965fa7527a9/root/var/svc/log/oxide-crucible-agent:default.log.0 (3436 bytes)
11402023-12-13T06:16:19.464Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_34ed6b25-7087-4077-b0fc-3965fa7527a9/root/var/svc/log/oxide-crucible-agent:default.log.0
11412023-12-13T06:16:19.464Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_34ed6b25-7087-4077-b0fc-3965fa7527a9/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
11422023-12-13T06:16:20.537Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_34ed6b25-7087-4077-b0fc-3965fa7527a9/root/var/svc/log/oxide-crucible-downstairs:default.log
11432023-12-13T06:16:20.537Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_34ed6b25-7087-4077-b0fc-3965fa7527a9/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
11442023-12-13T06:16:21.633Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_34ed6b25-7087-4077-b0fc-3965fa7527a9/root/var/svc/log/oxide-crucible-downstairs:default.log.0
11452023-12-13T06:16:21.633Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_46a6f302-1937-441b-9277-89a65afcfdfa/root/var/svc/log/oxide-crucible-agent:default.log (5922 bytes)
11462023-12-13T06:16:22.743Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_46a6f302-1937-441b-9277-89a65afcfdfa/root/var/svc/log/oxide-crucible-agent:default.log
11472023-12-13T06:16:22.770Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_46a6f302-1937-441b-9277-89a65afcfdfa/root/var/svc/log/oxide-crucible-agent:default.log.0 (13690 bytes)
11482023-12-13T06:16:23.862Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_46a6f302-1937-441b-9277-89a65afcfdfa/root/var/svc/log/oxide-crucible-agent:default.log.0
11492023-12-13T06:16:23.862Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_46a6f302-1937-441b-9277-89a65afcfdfa/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
11502023-12-13T06:16:24.948Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_46a6f302-1937-441b-9277-89a65afcfdfa/root/var/svc/log/oxide-crucible-downstairs:default.log
11512023-12-13T06:16:24.948Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_46a6f302-1937-441b-9277-89a65afcfdfa/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
11522023-12-13T06:16:26.104Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_46a6f302-1937-441b-9277-89a65afcfdfa/root/var/svc/log/oxide-crucible-downstairs:default.log.0
11532023-12-13T06:16:26.104Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_46a6f302-1937-441b-9277-89a65afcfdfa/root/var/svc/log/oxide-crucible-downstairs:downstairs-3c08e166-56c9-4822-a44c-9713824a9791.log (2939 bytes)
11542023-12-13T06:16:27.664Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_46a6f302-1937-441b-9277-89a65afcfdfa/root/var/svc/log/oxide-crucible-downstairs:downstairs-3c08e166-56c9-4822-a44c-9713824a9791.log
11552023-12-13T06:16:27.664Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_46a6f302-1937-441b-9277-89a65afcfdfa/root/var/svc/log/oxide-crucible-downstairs:downstairs-cf5f8390-5d94-4ed0-911a-9694f256008d.log (0 bytes)
11562023-12-13T06:16:29.161Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_46a6f302-1937-441b-9277-89a65afcfdfa/root/var/svc/log/oxide-crucible-downstairs:downstairs-cf5f8390-5d94-4ed0-911a-9694f256008d.log
11572023-12-13T06:16:29.300Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_46a6f302-1937-441b-9277-89a65afcfdfa/root/var/svc/log/oxide-crucible-downstairs:downstairs-cf5f8390-5d94-4ed0-911a-9694f256008d.log.0 (3049 bytes)
11582023-12-13T06:16:38.745Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_46a6f302-1937-441b-9277-89a65afcfdfa/root/var/svc/log/oxide-crucible-downstairs:downstairs-cf5f8390-5d94-4ed0-911a-9694f256008d.log.0
11592023-12-13T06:16:38.802Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_cockroachdb_1dc161c3-474f-4892-aafa-e1eaeab16a26/root/var/svc/log/oxide-cockroachdb:default.log (0 bytes)
11602023-12-13T06:16:40.274Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_cockroachdb_1dc161c3-474f-4892-aafa-e1eaeab16a26/root/var/svc/log/oxide-cockroachdb:default.log
11612023-12-13T06:16:40.275Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_cockroachdb_1dc161c3-474f-4892-aafa-e1eaeab16a26/root/var/svc/log/oxide-cockroachdb:default.log.0 (5386 bytes)
11622023-12-13T06:16:41.380Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_cockroachdb_1dc161c3-474f-4892-aafa-e1eaeab16a26/root/var/svc/log/oxide-cockroachdb:default.log.0
11632023-12-13T06:16:41.408Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_5394614b-e3ad-4f67-9e5b-22270bf01884/root/var/svc/log/oxide-crucible-agent:default.log (5928 bytes)
11642023-12-13T06:16:43.044Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_5394614b-e3ad-4f67-9e5b-22270bf01884/root/var/svc/log/oxide-crucible-agent:default.log
11652023-12-13T06:16:43.044Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_5394614b-e3ad-4f67-9e5b-22270bf01884/root/var/svc/log/oxide-crucible-agent:default.log.0 (13684 bytes)
11662023-12-13T06:16:44.289Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_5394614b-e3ad-4f67-9e5b-22270bf01884/root/var/svc/log/oxide-crucible-agent:default.log.0
11672023-12-13T06:16:44.289Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_5394614b-e3ad-4f67-9e5b-22270bf01884/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
11682023-12-13T06:16:45.374Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_5394614b-e3ad-4f67-9e5b-22270bf01884/root/var/svc/log/oxide-crucible-downstairs:default.log
11692023-12-13T06:16:45.374Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_5394614b-e3ad-4f67-9e5b-22270bf01884/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
11702023-12-13T06:16:46.961Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_5394614b-e3ad-4f67-9e5b-22270bf01884/root/var/svc/log/oxide-crucible-downstairs:default.log.0
11712023-12-13T06:16:46.961Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_5394614b-e3ad-4f67-9e5b-22270bf01884/root/var/svc/log/oxide-crucible-downstairs:downstairs-1fffa298-68a9-4db6-b0ba-5f76a4fc2dcc.log (0 bytes)
11722023-12-13T06:16:48.051Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_5394614b-e3ad-4f67-9e5b-22270bf01884/root/var/svc/log/oxide-crucible-downstairs:downstairs-1fffa298-68a9-4db6-b0ba-5f76a4fc2dcc.log
11732023-12-13T06:16:48.051Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_5394614b-e3ad-4f67-9e5b-22270bf01884/root/var/svc/log/oxide-crucible-downstairs:downstairs-1fffa298-68a9-4db6-b0ba-5f76a4fc2dcc.log.0 (3051 bytes)
11742023-12-13T06:16:58.072Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_5394614b-e3ad-4f67-9e5b-22270bf01884/root/var/svc/log/oxide-crucible-downstairs:downstairs-1fffa298-68a9-4db6-b0ba-5f76a4fc2dcc.log.0
11752023-12-13T06:16:58.072Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_5394614b-e3ad-4f67-9e5b-22270bf01884/root/var/svc/log/oxide-crucible-downstairs:downstairs-7b69cad2-a3e1-424f-a1a4-58c60add43f7.log (2942 bytes)
11762023-12-13T06:16:59.147Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_5394614b-e3ad-4f67-9e5b-22270bf01884/root/var/svc/log/oxide-crucible-downstairs:downstairs-7b69cad2-a3e1-424f-a1a4-58c60add43f7.log
11772023-12-13T06:16:59.147Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_58a73b11-dfe2-4c32-8174-54319f3c71b3/root/var/svc/log/oxide-crucible-agent:default.log (18002 bytes)
11782023-12-13T06:17:00.288Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_58a73b11-dfe2-4c32-8174-54319f3c71b3/root/var/svc/log/oxide-crucible-agent:default.log
11792023-12-13T06:17:00.289Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_58a73b11-dfe2-4c32-8174-54319f3c71b3/root/var/svc/log/oxide-crucible-agent:default.log.0 (9366 bytes)
11802023-12-13T06:17:01.398Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_58a73b11-dfe2-4c32-8174-54319f3c71b3/root/var/svc/log/oxide-crucible-agent:default.log.0
11812023-12-13T06:17:01.398Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_58a73b11-dfe2-4c32-8174-54319f3c71b3/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
11822023-12-13T06:17:02.475Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_58a73b11-dfe2-4c32-8174-54319f3c71b3/root/var/svc/log/oxide-crucible-downstairs:default.log
11832023-12-13T06:17:02.475Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_58a73b11-dfe2-4c32-8174-54319f3c71b3/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
11842023-12-13T06:17:03.601Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_58a73b11-dfe2-4c32-8174-54319f3c71b3/root/var/svc/log/oxide-crucible-downstairs:default.log.0
11852023-12-13T06:17:03.601Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_58a73b11-dfe2-4c32-8174-54319f3c71b3/root/var/svc/log/oxide-crucible-downstairs:downstairs-0e0b7591-5f21-44eb-8735-d12334fd114a.log (5531 bytes)
11862023-12-13T06:17:04.723Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_58a73b11-dfe2-4c32-8174-54319f3c71b3/root/var/svc/log/oxide-crucible-downstairs:downstairs-0e0b7591-5f21-44eb-8735-d12334fd114a.log
11872023-12-13T06:17:04.725Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_58a73b11-dfe2-4c32-8174-54319f3c71b3/root/var/svc/log/oxide-crucible-downstairs:downstairs-26748007-e147-460b-a7e0-6dbf60da6c98.log (1905 bytes)
11882023-12-13T06:17:05.824Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_58a73b11-dfe2-4c32-8174-54319f3c71b3/root/var/svc/log/oxide-crucible-downstairs:downstairs-26748007-e147-460b-a7e0-6dbf60da6c98.log
11892023-12-13T06:17:05.824Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_58a73b11-dfe2-4c32-8174-54319f3c71b3/root/var/svc/log/oxide-crucible-downstairs:downstairs-26748007-e147-460b-a7e0-6dbf60da6c98.log.0 (4529 bytes)
11902023-12-13T06:17:06.923Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_58a73b11-dfe2-4c32-8174-54319f3c71b3/root/var/svc/log/oxide-crucible-downstairs:downstairs-26748007-e147-460b-a7e0-6dbf60da6c98.log.0
11912023-12-13T06:17:06.923Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_58a73b11-dfe2-4c32-8174-54319f3c71b3/root/var/svc/log/oxide-crucible-downstairs:snapshot-26748007-e147-460b-a7e0-6dbf60da6c98-a376e497-e0a5-4101-9407-b0dfd3a772fc.log (5172 bytes)
11922023-12-13T06:17:08.022Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_58a73b11-dfe2-4c32-8174-54319f3c71b3/root/var/svc/log/oxide-crucible-downstairs:snapshot-26748007-e147-460b-a7e0-6dbf60da6c98-a376e497-e0a5-4101-9407-b0dfd3a772fc.log
11932023-12-13T06:17:08.022Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_pantry_6195ba09-ace6-46c2-94b0-9339b0f96fe3/root/var/svc/log/oxide-crucible-pantry:default.log (0 bytes)
11942023-12-13T06:17:09.167Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_pantry_6195ba09-ace6-46c2-94b0-9339b0f96fe3/root/var/svc/log/oxide-crucible-pantry:default.log
11952023-12-13T06:17:09.167Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_pantry_6195ba09-ace6-46c2-94b0-9339b0f96fe3/root/var/svc/log/oxide-crucible-pantry:default.log.0 (2338 bytes)
11962023-12-13T06:17:10.274Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_pantry_6195ba09-ace6-46c2-94b0-9339b0f96fe3/root/var/svc/log/oxide-crucible-pantry:default.log.0
11972023-12-13T06:17:10.274Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_14889461-c783-43ad-8ae1-179a2b05a09b/root/var/svc/log/oxide-crucible-agent:default.log (23928 bytes)
11982023-12-13T06:17:12.109Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_14889461-c783-43ad-8ae1-179a2b05a09b/root/var/svc/log/oxide-crucible-agent:default.log
11992023-12-13T06:17:12.109Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_14889461-c783-43ad-8ae1-179a2b05a09b/root/var/svc/log/oxide-crucible-agent:default.log.0 (9365 bytes)
12002023-12-13T06:17:13.248Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_14889461-c783-43ad-8ae1-179a2b05a09b/root/var/svc/log/oxide-crucible-agent:default.log.0
12012023-12-13T06:17:13.249Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_14889461-c783-43ad-8ae1-179a2b05a09b/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
12022023-12-13T06:17:14.733Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_14889461-c783-43ad-8ae1-179a2b05a09b/root/var/svc/log/oxide-crucible-downstairs:default.log
12032023-12-13T06:17:14.733Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_14889461-c783-43ad-8ae1-179a2b05a09b/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
12042023-12-13T06:17:16.258Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_14889461-c783-43ad-8ae1-179a2b05a09b/root/var/svc/log/oxide-crucible-downstairs:default.log.0
12052023-12-13T06:17:16.258Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_14889461-c783-43ad-8ae1-179a2b05a09b/root/var/svc/log/oxide-crucible-downstairs:downstairs-3308a3a7-1436-4ba4-8907-e27a974bd15f.log (2943 bytes)
12062023-12-13T06:17:17.358Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_14889461-c783-43ad-8ae1-179a2b05a09b/root/var/svc/log/oxide-crucible-downstairs:downstairs-3308a3a7-1436-4ba4-8907-e27a974bd15f.log
12072023-12-13T06:17:17.384Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_14889461-c783-43ad-8ae1-179a2b05a09b/root/var/svc/log/oxide-crucible-downstairs:downstairs-42293301-bbe2-45a6-9933-a00648890ac6.log (5534 bytes)
12082023-12-13T06:17:18.459Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_14889461-c783-43ad-8ae1-179a2b05a09b/root/var/svc/log/oxide-crucible-downstairs:downstairs-42293301-bbe2-45a6-9933-a00648890ac6.log
12092023-12-13T06:17:18.460Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_14889461-c783-43ad-8ae1-179a2b05a09b/root/var/svc/log/oxide-crucible-downstairs:downstairs-978038b3-a5f0-402e-b5da-7b21e1863412.log (1907 bytes)
12102023-12-13T06:17:19.602Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_14889461-c783-43ad-8ae1-179a2b05a09b/root/var/svc/log/oxide-crucible-downstairs:downstairs-978038b3-a5f0-402e-b5da-7b21e1863412.log
12112023-12-13T06:17:19.603Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_14889461-c783-43ad-8ae1-179a2b05a09b/root/var/svc/log/oxide-crucible-downstairs:downstairs-978038b3-a5f0-402e-b5da-7b21e1863412.log.0 (4530 bytes)
12122023-12-13T06:17:20.681Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_14889461-c783-43ad-8ae1-179a2b05a09b/root/var/svc/log/oxide-crucible-downstairs:downstairs-978038b3-a5f0-402e-b5da-7b21e1863412.log.0
12132023-12-13T06:17:20.681Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_14889461-c783-43ad-8ae1-179a2b05a09b/root/var/svc/log/oxide-crucible-downstairs:snapshot-978038b3-a5f0-402e-b5da-7b21e1863412-a376e497-e0a5-4101-9407-b0dfd3a772fc.log (5170 bytes)
12142023-12-13T06:17:21.765Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_14889461-c783-43ad-8ae1-179a2b05a09b/root/var/svc/log/oxide-crucible-downstairs:snapshot-978038b3-a5f0-402e-b5da-7b21e1863412-a376e497-e0a5-4101-9407-b0dfd3a772fc.log
12152023-12-13T06:17:21.765Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_internal_dns_a6eed8d3-9bbb-4abb-bdc9-84ce9c104710/root/var/svc/log/oxide-internal_dns:default.log (8792 bytes)
12162023-12-13T06:17:21.848Zupload warning: file "/pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_internal_dns_a6eed8d3-9bbb-4abb-bdc9-84ce9c104710/root/var/svc/log/oxide-internal_dns:default.log" changed size mid upload: 8792 -> 12548
12172023-12-13T06:17:22.954Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_internal_dns_a6eed8d3-9bbb-4abb-bdc9-84ce9c104710/root/var/svc/log/oxide-internal_dns:default.log
12182023-12-13T06:17:22.954Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_internal_dns_a6eed8d3-9bbb-4abb-bdc9-84ce9c104710/root/var/svc/log/oxide-internal_dns:default.log.0 (134298 bytes)
12192023-12-13T06:17:24.206Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_internal_dns_a6eed8d3-9bbb-4abb-bdc9-84ce9c104710/root/var/svc/log/oxide-internal_dns:default.log.0
12202023-12-13T06:17:24.207Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_ntp_22f7e918-79ab-4ea0-a1bc-53174a8cc11d/root/var/svc/log/oxide-ntp:default.log (0 bytes)
12212023-12-13T06:17:25.295Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_ntp_22f7e918-79ab-4ea0-a1bc-53174a8cc11d/root/var/svc/log/oxide-ntp:default.log
12222023-12-13T06:17:25.295Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_ntp_22f7e918-79ab-4ea0-a1bc-53174a8cc11d/root/var/svc/log/oxide-ntp:default.log.0 (1182 bytes)
12232023-12-13T06:17:26.530Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_ntp_22f7e918-79ab-4ea0-a1bc-53174a8cc11d/root/var/svc/log/oxide-ntp:default.log.0
12242023-12-13T06:17:26.580Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_external_dns_d875567c-88fd-42d8-95c6-2284a7a95f4e/root/var/svc/log/oxide-external_dns:default.log (3774 bytes)
12252023-12-13T06:17:27.000Zupload warning: file "/pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_external_dns_d875567c-88fd-42d8-95c6-2284a7a95f4e/root/var/svc/log/oxide-external_dns:default.log" changed size mid upload: 3774 -> 7551
12262023-12-13T06:17:28.085Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_external_dns_d875567c-88fd-42d8-95c6-2284a7a95f4e/root/var/svc/log/oxide-external_dns:default.log
12272023-12-13T06:17:28.280Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_external_dns_d875567c-88fd-42d8-95c6-2284a7a95f4e/root/var/svc/log/oxide-external_dns:default.log.0 (38273 bytes)
12282023-12-13T06:17:29.357Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_external_dns_d875567c-88fd-42d8-95c6-2284a7a95f4e/root/var/svc/log/oxide-external_dns:default.log.0
12292023-12-13T06:17:29.357Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_1e4bfc82-f598-47d5-b448-7a77d99690d9/root/var/svc/log/oxide-crucible-agent:default.log (10677 bytes)
12302023-12-13T06:17:30.480Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_1e4bfc82-f598-47d5-b448-7a77d99690d9/root/var/svc/log/oxide-crucible-agent:default.log
12312023-12-13T06:17:30.480Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_1e4bfc82-f598-47d5-b448-7a77d99690d9/root/var/svc/log/oxide-crucible-agent:default.log.0 (3436 bytes)
12322023-12-13T06:17:31.577Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_1e4bfc82-f598-47d5-b448-7a77d99690d9/root/var/svc/log/oxide-crucible-agent:default.log.0
12332023-12-13T06:17:31.577Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_1e4bfc82-f598-47d5-b448-7a77d99690d9/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
12342023-12-13T06:17:32.655Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_1e4bfc82-f598-47d5-b448-7a77d99690d9/root/var/svc/log/oxide-crucible-downstairs:default.log
12352023-12-13T06:17:32.687Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_1e4bfc82-f598-47d5-b448-7a77d99690d9/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
12362023-12-13T06:17:33.759Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_1e4bfc82-f598-47d5-b448-7a77d99690d9/root/var/svc/log/oxide-crucible-downstairs:default.log.0
12372023-12-13T06:17:33.760Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_1e4bfc82-f598-47d5-b448-7a77d99690d9/root/var/svc/log/oxide-crucible-downstairs:downstairs-0724910a-e482-492f-8506-172ce9bd14f6.log (5537 bytes)
12382023-12-13T06:17:34.980Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_1e4bfc82-f598-47d5-b448-7a77d99690d9/root/var/svc/log/oxide-crucible-downstairs:downstairs-0724910a-e482-492f-8506-172ce9bd14f6.log
12392023-12-13T06:17:34.981Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_cc4714a1-cfae-4238-aeb0-caab92d0b613/root/var/svc/log/oxide-crucible-agent:default.log (7755 bytes)
12402023-12-13T06:17:36.098Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_cc4714a1-cfae-4238-aeb0-caab92d0b613/root/var/svc/log/oxide-crucible-agent:default.log
12412023-12-13T06:17:36.098Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_cc4714a1-cfae-4238-aeb0-caab92d0b613/root/var/svc/log/oxide-crucible-agent:default.log.0 (19613 bytes)
12422023-12-13T06:17:37.259Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_cc4714a1-cfae-4238-aeb0-caab92d0b613/root/var/svc/log/oxide-crucible-agent:default.log.0
12432023-12-13T06:17:37.259Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_cc4714a1-cfae-4238-aeb0-caab92d0b613/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
12442023-12-13T06:17:38.333Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_cc4714a1-cfae-4238-aeb0-caab92d0b613/root/var/svc/log/oxide-crucible-downstairs:default.log
12452023-12-13T06:17:38.333Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_cc4714a1-cfae-4238-aeb0-caab92d0b613/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
12462023-12-13T06:17:39.438Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_cc4714a1-cfae-4238-aeb0-caab92d0b613/root/var/svc/log/oxide-crucible-downstairs:default.log.0
12472023-12-13T06:17:39.438Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_cc4714a1-cfae-4238-aeb0-caab92d0b613/root/var/svc/log/oxide-crucible-downstairs:downstairs-904f5b82-142e-4b2d-ba15-ab70490c9b31.log (0 bytes)
12482023-12-13T06:17:40.520Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_cc4714a1-cfae-4238-aeb0-caab92d0b613/root/var/svc/log/oxide-crucible-downstairs:downstairs-904f5b82-142e-4b2d-ba15-ab70490c9b31.log
12492023-12-13T06:17:40.520Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_cc4714a1-cfae-4238-aeb0-caab92d0b613/root/var/svc/log/oxide-crucible-downstairs:downstairs-904f5b82-142e-4b2d-ba15-ab70490c9b31.log.0 (3051 bytes)
12502023-12-13T06:17:41.622Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_cc4714a1-cfae-4238-aeb0-caab92d0b613/root/var/svc/log/oxide-crucible-downstairs:downstairs-904f5b82-142e-4b2d-ba15-ab70490c9b31.log.0
12512023-12-13T06:17:41.628Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_cc4714a1-cfae-4238-aeb0-caab92d0b613/root/var/svc/log/oxide-crucible-downstairs:downstairs-bc659d4b-e9c9-465b-b185-bea9915ea72e.log (1906 bytes)
12522023-12-13T06:17:42.748Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_cc4714a1-cfae-4238-aeb0-caab92d0b613/root/var/svc/log/oxide-crucible-downstairs:downstairs-bc659d4b-e9c9-465b-b185-bea9915ea72e.log
12532023-12-13T06:17:42.798Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_cc4714a1-cfae-4238-aeb0-caab92d0b613/root/var/svc/log/oxide-crucible-downstairs:downstairs-bc659d4b-e9c9-465b-b185-bea9915ea72e.log.0 (4530 bytes)
12542023-12-13T06:17:43.851Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_cc4714a1-cfae-4238-aeb0-caab92d0b613/root/var/svc/log/oxide-crucible-downstairs:downstairs-bc659d4b-e9c9-465b-b185-bea9915ea72e.log.0
12552023-12-13T06:17:43.898Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_cc4714a1-cfae-4238-aeb0-caab92d0b613/root/var/svc/log/oxide-crucible-downstairs:snapshot-bc659d4b-e9c9-465b-b185-bea9915ea72e-a376e497-e0a5-4101-9407-b0dfd3a772fc.log (5176 bytes)
12562023-12-13T06:17:44.975Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_cc4714a1-cfae-4238-aeb0-caab92d0b613/root/var/svc/log/oxide-crucible-downstairs:snapshot-bc659d4b-e9c9-465b-b185-bea9915ea72e-a376e497-e0a5-4101-9407-b0dfd3a772fc.log
12572023-12-13T06:17:44.976Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_f94ea710-e17a-442b-8405-647a3a3fbd7c/root/var/svc/log/oxide-crucible-agent:default.log (0 bytes)
12582023-12-13T06:17:47.608Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_f94ea710-e17a-442b-8405-647a3a3fbd7c/root/var/svc/log/oxide-crucible-agent:default.log
12592023-12-13T06:17:47.608Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_f94ea710-e17a-442b-8405-647a3a3fbd7c/root/var/svc/log/oxide-crucible-agent:default.log.0 (3436 bytes)
12602023-12-13T06:17:49.321Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_f94ea710-e17a-442b-8405-647a3a3fbd7c/root/var/svc/log/oxide-crucible-agent:default.log.0
12612023-12-13T06:17:49.369Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_f94ea710-e17a-442b-8405-647a3a3fbd7c/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
12622023-12-13T06:17:50.506Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_f94ea710-e17a-442b-8405-647a3a3fbd7c/root/var/svc/log/oxide-crucible-downstairs:default.log
12632023-12-13T06:17:50.506Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_f94ea710-e17a-442b-8405-647a3a3fbd7c/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
12642023-12-13T06:18:04.400Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_f94ea710-e17a-442b-8405-647a3a3fbd7c/root/var/svc/log/oxide-crucible-downstairs:default.log.0
12652023-12-13T06:18:04.400Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_pantry_80fd2d61-652d-4a50-bde3-fb91b9aa05ec/root/var/svc/log/oxide-crucible-pantry:default.log (0 bytes)
12662023-12-13T06:18:04.890Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_pantry_80fd2d61-652d-4a50-bde3-fb91b9aa05ec/root/var/svc/log/oxide-crucible-pantry:default.log
12672023-12-13T06:18:04.890Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_pantry_80fd2d61-652d-4a50-bde3-fb91b9aa05ec/root/var/svc/log/oxide-crucible-pantry:default.log.0 (2333 bytes)
12682023-12-13T06:18:05.998Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_pantry_80fd2d61-652d-4a50-bde3-fb91b9aa05ec/root/var/svc/log/oxide-crucible-pantry:default.log.0
12692023-12-13T06:18:05.998Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_cockroachdb_63d3a760-51e1-429a-8097-853a705a2f83/root/var/svc/log/oxide-cockroachdb:default.log (0 bytes)
12702023-12-13T06:18:07.088Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_cockroachdb_63d3a760-51e1-429a-8097-853a705a2f83/root/var/svc/log/oxide-cockroachdb:default.log
12712023-12-13T06:18:07.088Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_cockroachdb_63d3a760-51e1-429a-8097-853a705a2f83/root/var/svc/log/oxide-cockroachdb:default.log.0 (5413 bytes)
12722023-12-13T06:18:08.204Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_cockroachdb_63d3a760-51e1-429a-8097-853a705a2f83/root/var/svc/log/oxide-cockroachdb:default.log.0
12732023-12-13T06:18:08.204Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_internal_dns_2c1dde86-d4d1-44e1-997f-80a647ce190a/root/var/svc/log/oxide-internal_dns:default.log (3755 bytes)
12742023-12-13T06:18:08.283Zupload warning: file "/pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_internal_dns_2c1dde86-d4d1-44e1-997f-80a647ce190a/root/var/svc/log/oxide-internal_dns:default.log" changed size mid upload: 3755 -> 11267
12752023-12-13T06:18:09.359Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_internal_dns_2c1dde86-d4d1-44e1-997f-80a647ce190a/root/var/svc/log/oxide-internal_dns:default.log
12762023-12-13T06:18:09.359Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_internal_dns_2c1dde86-d4d1-44e1-997f-80a647ce190a/root/var/svc/log/oxide-internal_dns:default.log.0 (138743 bytes)
12772023-12-13T06:18:10.671Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_internal_dns_2c1dde86-d4d1-44e1-997f-80a647ce190a/root/var/svc/log/oxide-internal_dns:default.log.0
12782023-12-13T06:18:10.672Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_nexus_d97212e0-3544-4752-a58e-60793dae3714/root/var/svc/log/oxide-nexus:default.log (3888085 bytes)
12792023-12-13T06:18:14.057Zupload warning: file "/pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_nexus_d97212e0-3544-4752-a58e-60793dae3714/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 3888085 -> 4032620
12802023-12-13T06:18:15.146Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_nexus_d97212e0-3544-4752-a58e-60793dae3714/root/var/svc/log/oxide-nexus:default.log
12812023-12-13T06:18:15.146Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_nexus_d97212e0-3544-4752-a58e-60793dae3714/root/var/svc/log/oxide-nexus:default.log.0 (30491542 bytes)
12822023-12-13T06:18:29.628Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_nexus_d97212e0-3544-4752-a58e-60793dae3714/root/var/svc/log/oxide-nexus:default.log.0
12832023-12-13T06:18:29.628Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_clickhouse_793e2197-cf2f-4847-a5fc-a9b2cb7cc2c6/root/var/svc/log/oxide-clickhouse:default.log (123203 bytes)
12842023-12-13T06:18:29.838Zupload warning: file "/pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_clickhouse_793e2197-cf2f-4847-a5fc-a9b2cb7cc2c6/root/var/svc/log/oxide-clickhouse:default.log" changed size mid upload: 123203 -> 431347
12852023-12-13T06:18:30.909Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_clickhouse_793e2197-cf2f-4847-a5fc-a9b2cb7cc2c6/root/var/svc/log/oxide-clickhouse:default.log
12862023-12-13T06:18:30.909Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_clickhouse_793e2197-cf2f-4847-a5fc-a9b2cb7cc2c6/root/var/svc/log/oxide-clickhouse:default.log.0 (1103193 bytes)
12872023-12-13T06:18:32.792Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_clickhouse_793e2197-cf2f-4847-a5fc-a9b2cb7cc2c6/root/var/svc/log/oxide-clickhouse:default.log.0
12882023-12-13T06:18:32.792Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_27bf8893-9f3a-49b5-9786-6a339fd6393e/root/var/svc/log/oxide-cockroachdb:default.log (0 bytes)
12892023-12-13T06:18:33.863Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_27bf8893-9f3a-49b5-9786-6a339fd6393e/root/var/svc/log/oxide-cockroachdb:default.log
12902023-12-13T06:18:33.863Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_27bf8893-9f3a-49b5-9786-6a339fd6393e/root/var/svc/log/oxide-cockroachdb:default.log.0 (5411 bytes)
12912023-12-13T06:18:34.983Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_27bf8893-9f3a-49b5-9786-6a339fd6393e/root/var/svc/log/oxide-cockroachdb:default.log.0
12922023-12-13T06:18:34.983Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_a1449b9f-2132-496c-aa87-bc52b27a1c7e/root/var/svc/log/oxide-cockroachdb:default.log (0 bytes)
12932023-12-13T06:18:36.066Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_a1449b9f-2132-496c-aa87-bc52b27a1c7e/root/var/svc/log/oxide-cockroachdb:default.log
12942023-12-13T06:18:36.066Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_a1449b9f-2132-496c-aa87-bc52b27a1c7e/root/var/svc/log/oxide-cockroachdb:default.log.0 (5412 bytes)
12952023-12-13T06:18:37.164Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_a1449b9f-2132-496c-aa87-bc52b27a1c7e/root/var/svc/log/oxide-cockroachdb:default.log.0
12962023-12-13T06:18:37.164Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_f2675a50-e728-4be8-8d74-78b9c331e6e6/root/var/svc/log/oxide-cockroachdb:default.log (0 bytes)
12972023-12-13T06:18:38.235Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_f2675a50-e728-4be8-8d74-78b9c331e6e6/root/var/svc/log/oxide-cockroachdb:default.log
12982023-12-13T06:18:38.235Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_f2675a50-e728-4be8-8d74-78b9c331e6e6/root/var/svc/log/oxide-cockroachdb:default.log.0 (5413 bytes)
12992023-12-13T06:18:39.334Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_f2675a50-e728-4be8-8d74-78b9c331e6e6/root/var/svc/log/oxide-cockroachdb:default.log.0
13002023-12-13T06:18:39.334Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_60fa8bb1-acdb-4521-b6b8-93727b55e3ce/root/var/svc/log/oxide-internal_dns:default.log (16347 bytes)
13012023-12-13T06:18:39.545Zupload warning: file "/pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_60fa8bb1-acdb-4521-b6b8-93727b55e3ce/root/var/svc/log/oxide-internal_dns:default.log" changed size mid upload: 16347 -> 23860
13022023-12-13T06:18:40.623Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_60fa8bb1-acdb-4521-b6b8-93727b55e3ce/root/var/svc/log/oxide-internal_dns:default.log
13032023-12-13T06:18:40.649Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_60fa8bb1-acdb-4521-b6b8-93727b55e3ce/root/var/svc/log/oxide-internal_dns:default.log.0 (134951 bytes)
13042023-12-13T06:18:41.982Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_60fa8bb1-acdb-4521-b6b8-93727b55e3ce/root/var/svc/log/oxide-internal_dns:default.log.0
13052023-12-13T06:18:42.008Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_37704517-0274-43bb-a6ac-082b5531948c/root/var/svc/log/oxide-nexus:default.log (204412 bytes)
13062023-12-13T06:18:42.203Zupload warning: file "/pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_37704517-0274-43bb-a6ac-082b5531948c/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 204412 -> 364780
13072023-12-13T06:18:43.285Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_37704517-0274-43bb-a6ac-082b5531948c/root/var/svc/log/oxide-nexus:default.log
13082023-12-13T06:18:43.285Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_37704517-0274-43bb-a6ac-082b5531948c/root/var/svc/log/oxide-nexus:default.log.0 (2833873 bytes)
13092023-12-13T06:18:46.626Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_37704517-0274-43bb-a6ac-082b5531948c/root/var/svc/log/oxide-nexus:default.log.0
13102023-12-13T06:18:46.627Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_5bc0f2ca-46bd-4690-a0e6-c498f0d34690/root/var/svc/log/oxide-nexus:default.log (106821 bytes)
13112023-12-13T06:18:47.190Zupload warning: file "/pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_5bc0f2ca-46bd-4690-a0e6-c498f0d34690/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 106821 -> 258235
13122023-12-13T06:18:48.266Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_5bc0f2ca-46bd-4690-a0e6-c498f0d34690/root/var/svc/log/oxide-nexus:default.log
13132023-12-13T06:18:48.266Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_5bc0f2ca-46bd-4690-a0e6-c498f0d34690/root/var/svc/log/oxide-nexus:default.log.0 (975451 bytes)
13142023-12-13T06:18:50.941Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_5bc0f2ca-46bd-4690-a0e6-c498f0d34690/root/var/svc/log/oxide-nexus:default.log.0
13152023-12-13T06:18:50.941Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_oximeter_9c1ca374-ca05-4928-82be-1112d014da0b/root/var/svc/log/oxide-oximeter:default.log (40224 bytes)
13162023-12-13T06:18:51.237Zupload warning: file "/pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_oximeter_9c1ca374-ca05-4928-82be-1112d014da0b/root/var/svc/log/oxide-oximeter:default.log" changed size mid upload: 40224 -> 142865
13172023-12-13T06:18:52.310Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_oximeter_9c1ca374-ca05-4928-82be-1112d014da0b/root/var/svc/log/oxide-oximeter:default.log
13182023-12-13T06:18:52.335Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_oximeter_9c1ca374-ca05-4928-82be-1112d014da0b/root/var/svc/log/oxide-oximeter:default.log.0 (337513 bytes)
13192023-12-13T06:18:53.801Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_oximeter_9c1ca374-ca05-4928-82be-1112d014da0b/root/var/svc/log/oxide-oximeter:default.log.0
13202023-12-13T06:18:53.804Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_ntp_22f7e918-79ab-4ea0-a1bc-53174a8cc11d/root/var/log/chrony/measurements.log (23975 bytes)
13212023-12-13T06:18:53.919Zupload warning: file "/pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_ntp_22f7e918-79ab-4ea0-a1bc-53174a8cc11d/root/var/log/chrony/measurements.log" changed size mid upload: 23975 -> 30414
13222023-12-13T06:18:54.991Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_ntp_22f7e918-79ab-4ea0-a1bc-53174a8cc11d/root/var/log/chrony/measurements.log
13232023-12-13T06:18:54.991Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_ntp_22f7e918-79ab-4ea0-a1bc-53174a8cc11d/root/var/log/chrony/statistics.log (16263 bytes)
13242023-12-13T06:18:55.490Zupload warning: file "/pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_ntp_22f7e918-79ab-4ea0-a1bc-53174a8cc11d/root/var/log/chrony/statistics.log" changed size mid upload: 16263 -> 21762
13252023-12-13T06:18:59.214Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_ntp_22f7e918-79ab-4ea0-a1bc-53174a8cc11d/root/var/log/chrony/statistics.log
13262023-12-13T06:18:59.214Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_ntp_22f7e918-79ab-4ea0-a1bc-53174a8cc11d/root/var/log/chrony/tracking.log (2250 bytes)
13272023-12-13T06:18:59.435Zupload warning: file "/pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_ntp_22f7e918-79ab-4ea0-a1bc-53174a8cc11d/root/var/log/chrony/tracking.log" changed size mid upload: 2250 -> 2646
13282023-12-13T06:19:00.506Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_ntp_22f7e918-79ab-4ea0-a1bc-53174a8cc11d/root/var/log/chrony/tracking.log