01HHD88DA6CS8RFHJPWF253TQF: helios / deploy

BasicConfig {
    output_rules: [
        "%/var/svc/log/oxide-sled-agent:default.log*",
        "%/pool/ext/*/crypt/zone/oxz_*/root/var/svc/log/oxide-*.log*",
        "%/pool/ext/*/crypt/zone/oxz_*/root/var/svc/log/system-illumos-*.log*",
        "%/pool/ext/*/crypt/zone/oxz_ntp_*/root/var/log/chrony/*.log*",
        "!/pool/ext/*/crypt/zone/oxz_propolis-server_*/root/var/svc/log/*.log*",
        "%/pool/ext/*/crypt/debug/global/oxide-sled-agent:default.log.*",
        "%/pool/ext/*/crypt/debug/oxz_*/oxide-*.log.*",
        "%/pool/ext/*/crypt/debug/oxz_*/system-illumos-*.log.*",
        "!/pool/ext/*/crypt/debug/oxz_propolis-server_*/*.log.*",
    ],
    rust_toolchain: None,
    target: Some(
        "lab-2.0-opte-0.27",
    ),
    access_repos: [],
    publish: [],
    skip_clone: true,
}

Buildomat Job: 01HHD89318GPC2GH079A5GB6VZ

Tags:

Artefacts:

Output:

SEQ GLOBAL TIME DETAILS
12023-12-11T21:00:54.256Zjob dependencies complete; ready to run (waiting for 46 m 8 s)
22023-12-11T21:00:54.319Zjob assigned to worker 01HHD9NJ718PK7HD7XSVZQ2C24 (queued for 0 s)
32023-12-11T21:00:54.618Zhttp://172.20.0.5:9969/ipxe/buskin... ok
42023-12-11T21:00:54.669Zhttp://172.20.0.5:9969/os/buskin/platform/i86pc/kernel/amd64/unix... ok
52023-12-11T21:00:54.704Zhttp://172.20.0.5:9969/os/buskin/platform/i86pc/amd64/boot_archive... ok
62023-12-11T21:00:54.736Zhttp://172.20.0.5:9969/os/buskin/platform/i86pc/amd64/boot_archive.hash... ok
72023-12-11T21:00:54.794Zhttp://172.20.0.5:9969/postboot/buskin... ok
82023-12-11T21:00:54.826ZwcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U
92023-12-11T21:00:54.859Zcpu0: microcode has been updated from version 0xa001133 to 0xa0011d1
102023-12-11T21:00:54.893ZOxide Helios Version helios-2.0.22333 64-bit
112023-12-11T21:00:54.926ZConfiguring devices.
122023-12-11T21:00:54.959Z##################################################
132023-12-11T21:00:55.006Z#### ############# ##
142023-12-11T21:00:55.037Z### ### ############ ##
152023-12-11T21:00:55.071Z## ### # ## ### ## Oxide Computer Company ##
162023-12-11T21:00:55.109Z## ## ## ### # ### ##
172023-12-11T21:00:55.148Z## # ### #### #### This Station Under ##
182023-12-11T21:00:55.182Z### ### #### # ### Computer Control ##
192023-12-11T21:00:55.232Z#### #### ### ## ##
202023-12-11T21:00:55.270Z##################################################
212023-12-11T21:00:55.304ZHostname: buskin
222023-12-11T21:00:55.342ZLoading smf(7) service descriptions:
232023-12-11T21:00:56.682Zdownloading input: /input/ci-tools/work/end-to-end-tests/bootstrap-6bafd3fa6eef4151.gz
242023-12-11T21:00:57.629Zdownloaded input: /input/ci-tools/work/end-to-end-tests/bootstrap-6bafd3fa6eef4151.gz
252023-12-11T21:00:57.629Zdownloading input: /input/ci-tools/work/end-to-end-tests/bootstrap.gz
262023-12-11T21:01:14.863Zdownloaded input: /input/ci-tools/work/end-to-end-tests/bootstrap.gz
272023-12-11T21:01:14.864Zdownloading input: /input/ci-tools/work/end-to-end-tests/end_to_end_tests-d68396a0af635b1c.gz
282023-12-11T21:01:36.297Zdownloaded input: /input/ci-tools/work/end-to-end-tests/end_to_end_tests-d68396a0af635b1c.gz
292023-12-11T21:01:36.297Zdownloading input: /input/ci-tools/work/caboose-util.gz
302023-12-11T21:01:37.167Zdownloaded input: /input/ci-tools/work/caboose-util.gz
312023-12-11T21:01:37.193Zdownloading input: /input/ci-tools/work/tufaceous.gz
322023-12-11T21:01:40.336Zdownloaded input: /input/ci-tools/work/tufaceous.gz
332023-12-11T21:01:40.366Zdownloading input: /input/package/work/version.txt
342023-12-11T21:01:40.400Zdownloaded input: /input/package/work/version.txt
352023-12-11T21:01:40.400Zdownloading input: /input/package/work/package.tar.gz
362023-12-11T21:01:57.181Zdownloaded input: /input/package/work/package.tar.gz
372023-12-11T21:01:57.181Zdownloading input: /input/package/work/global-zone-packages.tar.gz
382023-12-11T21:02:25.390Zdownloaded input: /input/package/work/global-zone-packages.tar.gz
392023-12-11T21:02:25.390Zdownloading input: /input/package/work/trampoline-global-zone-packages.tar.gz
402023-12-11T21:02:35.216Zdownloaded input: /input/package/work/trampoline-global-zone-packages.tar.gz
412023-12-11T21:02:35.244Zdownloading input: /input/package/work/zones/clickhouse.tar.gz
422023-12-11T21:03:10.155Zdownloaded input: /input/package/work/zones/clickhouse.tar.gz
432023-12-11T21:03:10.155Zdownloading input: /input/package/work/zones/clickhouse_keeper.tar.gz
442023-12-11T21:03:44.651Zdownloaded input: /input/package/work/zones/clickhouse_keeper.tar.gz
452023-12-11T21:03:44.652Zdownloading input: /input/package/work/zones/cockroachdb.tar.gz
462023-12-11T21:04:07.053Zdownloaded input: /input/package/work/zones/cockroachdb.tar.gz
472023-12-11T21:04:07.053Zdownloading input: /input/package/work/zones/crucible-pantry.tar.gz
482023-12-11T21:04:11.184Zdownloaded input: /input/package/work/zones/crucible-pantry.tar.gz
492023-12-11T21:04:11.184Zdownloading input: /input/package/work/zones/crucible.tar.gz
502023-12-11T21:04:18.870Zdownloaded input: /input/package/work/zones/crucible.tar.gz
512023-12-11T21:04:18.870Zdownloading input: /input/package/work/zones/external-dns.tar.gz
522023-12-11T21:04:23.944Zdownloaded input: /input/package/work/zones/external-dns.tar.gz
532023-12-11T21:04:23.944Zdownloading input: /input/package/work/zones/internal-dns.tar.gz
542023-12-11T21:04:28.393Zdownloaded input: /input/package/work/zones/internal-dns.tar.gz
552023-12-11T21:04:28.393Zdownloading input: /input/package/work/zones/ntp.tar.gz
562023-12-11T21:04:28.430Zdownloaded input: /input/package/work/zones/ntp.tar.gz
572023-12-11T21:04:28.430Zdownloading input: /input/package/work/zones/omicron-gateway-asic.tar.gz
582023-12-11T21:04:31.795Zdownloaded input: /input/package/work/zones/omicron-gateway-asic.tar.gz
592023-12-11T21:04:31.795Zdownloading input: /input/package/work/zones/omicron-gateway-softnpu.tar.gz
602023-12-11T21:04:35.080Zdownloaded input: /input/package/work/zones/omicron-gateway-softnpu.tar.gz
612023-12-11T21:04:35.081Zdownloading input: /input/package/work/zones/omicron-nexus-single-sled.tar.gz
622023-12-11T21:04:48.513Zdownloaded input: /input/package/work/zones/omicron-nexus-single-sled.tar.gz
632023-12-11T21:04:48.513Zdownloading input: /input/package/work/zones/omicron-nexus.tar.gz
642023-12-11T21:05:03.802Zdownloaded input: /input/package/work/zones/omicron-nexus.tar.gz
652023-12-11T21:05:03.802Zdownloading input: /input/package/work/zones/overlay.tar.gz
662023-12-11T21:05:03.873Zdownloaded input: /input/package/work/zones/overlay.tar.gz
672023-12-11T21:05:03.873Zdownloading input: /input/package/work/zones/oximeter-collector.tar.gz
682023-12-11T21:05:11.768Zdownloaded input: /input/package/work/zones/oximeter-collector.tar.gz
692023-12-11T21:05:11.768Zdownloading input: /input/package/work/zones/propolis-server.tar.gz
702023-12-11T21:05:23.965Zdownloaded input: /input/package/work/zones/propolis-server.tar.gz
712023-12-11T21:05:23.965Zdownloading input: /input/package/work/zones/switch-asic.tar.gz
722023-12-11T21:06:19.164Zdownloaded input: /input/package/work/zones/switch-asic.tar.gz
732023-12-11T21:06:19.164Zdownloading input: /input/package/work/zones/switch-softnpu.tar.gz
742023-12-11T21:06:49.005Zdownloaded input: /input/package/work/zones/switch-softnpu.tar.gz
 
752023-12-11T21:06:49.029Zstarting task 0: "setup"
762023-12-11T21:06:49.030Z++ uname -s
772023-12-11T21:06:49.030Z+ kern=SunOS
782023-12-11T21:06:49.030Z+ case "$kern" in
792023-12-11T21:06:49.030Z+ groupadd -g 12345 build
802023-12-11T21:06:49.030Z+ useradd -u 12345 -g build -d /home/build -s /bin/bash -c build -P 'Primary Administrator' build
812023-12-11T21:06:51.023Z+ zfs create -o mountpoint=/work rpool/work
822023-12-11T21:06:51.073Z++ awk '$2 == "/home" { print $3 }' /etc/mnttab
832023-12-11T21:06:51.097Z+ home_fs=zfs
842023-12-11T21:06:51.097Z+ [[ zfs == autofs ]]
852023-12-11T21:06:51.098Z+ mkdir -p /home/build
862023-12-11T21:06:51.098Z+ chown build:build /home/build /work
872023-12-11T21:06:53.077Z+ chmod 0700 /home/build /work
882023-12-11T21:06:53.102Zprocess exited: duration 4075 ms, exit code 0
 
892023-12-11T21:06:53.150Zstarting task 1: "authentication"
902023-12-11T21:06:53.204Zprocess exited: duration 52 ms, exit code 0
 
912023-12-11T21:06:53.260Zstarting task 2: "build"
922023-12-11T21:06:53.285Z+ trap _exit_trap EXIT
932023-12-11T21:06:53.286Z+ pfexec mdb -kwe 'secpolicy_fs_linkdir/v 55 48 89 e5 b8 01 00 00 00 5d c3'
942023-12-11T21:06:53.472Zsecpolicy_fs_linkdir: 0x55 = 0x55
952023-12-11T21:06:53.472Zsecpolicy_fs_linkdir+1: 0xb8 = 0x48
962023-12-11T21:06:53.472Zsecpolicy_fs_linkdir+2: 0x1 = 0x89
972023-12-11T21:06:53.472Zsecpolicy_fs_linkdir+3: 0 = 0xe5
982023-12-11T21:06:53.472Zsecpolicy_fs_linkdir+4: 0 = 0xb8
992023-12-11T21:06:53.472Zsecpolicy_fs_linkdir+5: 0 = 0x1
1002023-12-11T21:06:53.472Zsecpolicy_fs_linkdir+6: 0x48 = 0x0
1012023-12-11T21:06:53.472Zsecpolicy_fs_linkdir+7: 0x89 = 0x0
1022023-12-11T21:06:53.498Zsecpolicy_fs_linkdir+8: 0xe5 = 0x0
1032023-12-11T21:06:53.498Zsecpolicy_fs_linkdir+9: 0x48 = 0x5d
1042023-12-11T21:06:53.498Zsecpolicy_fs_linkdir+0xa: 0x83 = 0xc3
1052023-12-11T21:06:53.498Z+ [[ -d /opt/oxide ]]
1062023-12-11T21:06:53.498Z+ pfexec mv /opt/oxide /opt/oxide-underneath
1072023-12-11T21:06:53.498Z+ pfexec mkdir /opt/oxide
1082023-12-11T21:06:53.498Z+ pfexec mount -F tmpfs -O swap /opt/oxide
1092023-12-11T21:06:53.498Z+ [[ -d /opt/oxide-underneath ]]
1102023-12-11T21:06:53.498Z+ cd /opt/oxide-underneath
1112023-12-11T21:06:53.498Z+ pfexec tar ceEp@/f - .
1122023-12-11T21:06:53.498Z+ cd /opt/oxide
1132023-12-11T21:06:53.499Z+ pfexec tar xveEp@/f -
1142023-12-11T21:06:53.594Zx ., 0 bytes, 0 tape blocks
1152023-12-11T21:06:53.594Zx ./opte, 0 bytes, 0 tape blocks
1162023-12-11T21:06:53.594Zx ./opte/bin, 0 bytes, 0 tape blocks
1172023-12-11T21:06:53.594Zx ./opte/bin/opteadm, 52567720 bytes, 102672 tape blocks
1182023-12-11T21:06:53.619Z+ pfexec rm -rf /opt/oxide-underneath
1192023-12-11T21:06:53.619Z+ pfexec /sbin/zfs create -o mountpoint=/zone rpool/zone
1202023-12-11T21:06:53.645Z+ pfexec mkdir /opt/oxide/work
1212023-12-11T21:06:53.645Z+ pfexec chown build:build /opt/oxide/work
1222023-12-11T21:06:53.645Z+ cd /opt/oxide/work
1232023-12-11T21:06:53.645Z+ ptime -m tar xvzf /input/package/work/package.tar.gz
1242023-12-11T21:06:55.174ZDecompressing '/input/package/work/package.tar.gz' with '/usr/bin/gzcat'...
1252023-12-11T21:06:55.174Zx out/mg-ddm-gz.tar, 62723072 bytes, 122506 tape blocks
1262023-12-11T21:06:55.174Zx out/omicron-sled-agent.tar, 111833600 bytes, 218425 tape blocks
1272023-12-11T21:06:55.175Zx out/target/test, 75 bytes, 1 tape blocks
1282023-12-11T21:06:55.175Zx out/npuzone/npuzone, 37184104 bytes, 72626 tape blocks
1292023-12-11T21:06:55.175Zx package-manifest.toml, 22143 bytes, 44 tape blocks
1302023-12-11T21:06:55.175Zx smf/sled-agent/non-gimlet/config.toml, 3448 bytes, 7 tape blocks
1312023-12-11T21:06:55.175Zx target/release/omicron-package, 42507008 bytes, 83022 tape blocks
1322023-12-11T21:06:55.175Zx tools/create_virtual_hardware.sh, 3176 bytes, 7 tape blocks
1332023-12-11T21:06:55.175Zx tools/virtual_hardware.sh, 3408 bytes, 7 tape blocks
1342023-12-11T21:06:55.175Zx tools/scrimlet/create-softnpu-zone.sh, 619 bytes, 2 tape blocks
1352023-12-11T21:06:55.175Zx tools/scrimlet/destroy-softnpu-zone.sh, 195 bytes, 1 tape blocks
1362023-12-11T21:06:55.175Zx tools/scrimlet/softnpu-init.sh, 2148 bytes, 5 tape blocks
1372023-12-11T21:06:55.175Zx tools/scrimlet/softnpu-zone.txt, 330 bytes, 1 tape blocks
1382023-12-11T21:06:55.175Zx tools/scrimlet/softnpu.toml, 171 bytes, 1 tape blocks
1392023-12-11T21:06:55.201Z
1402023-12-11T21:06:55.201Zreal 1.541938628
1412023-12-11T21:06:55.201Zuser 1.386601166
1422023-12-11T21:06:55.201Zsys 0.274562536
1432023-12-11T21:06:55.201Ztrap 0.000057672
1442023-12-11T21:06:55.201Ztflt 0.000000000
1452023-12-11T21:06:55.201Zdflt 0.000000000
1462023-12-11T21:06:55.201Zkflt 0.000000000
1472023-12-11T21:06:55.201Zlock 0.000000000
1482023-12-11T21:06:55.201Zslp 1.384739775
1492023-12-11T21:06:55.201Zlat 0.035967281
1502023-12-11T21:06:55.202Zstop 0.000120333
1512023-12-11T21:06:55.202Z+ cp /input/package/work/zones/clickhouse_keeper.tar.gz /input/package/work/zones/clickhouse.tar.gz /input/package/work/zones/cockroachdb.tar.gz /input/package/work/zones/crucible-pantry.tar.gz /input/package/work/zones/crucible.tar.gz /input/package/work/zones/external-dns.tar.gz /input/package/work/zones/internal-dns.tar.gz /input/package/work/zones/ntp.tar.gz /input/package/work/zones/omicron-gateway-asic.tar.gz /input/package/work/zones/omicron-gateway-softnpu.tar.gz /input/package/work/zones/omicron-nexus-single-sled.tar.gz /input/package/work/zones/omicron-nexus.tar.gz /input/package/work/zones/overlay.tar.gz /input/package/work/zones/oximeter-collector.tar.gz /input/package/work/zones/propolis-server.tar.gz /input/package/work/zones/switch-asic.tar.gz /input/package/work/zones/switch-softnpu.tar.gz out/
1522023-12-11T21:06:56.271Z+ mv out/omicron-nexus-single-sled.tar.gz out/omicron-nexus.tar.gz
1532023-12-11T21:06:56.296Z+ mkdir tests
1542023-12-11T21:06:56.296Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1552023-12-11T21:06:56.296Z+ ptime -m gunzip
1562023-12-11T21:06:56.296Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap-6bafd3fa6eef4151
1572023-12-11T21:06:56.430Z
1582023-12-11T21:06:56.457Zreal 0.137467349
1592023-12-11T21:06:56.457Zuser 0.126166991
1602023-12-11T21:06:56.457Zsys 0.011050561
1612023-12-11T21:06:56.457Ztrap 0.000013330
1622023-12-11T21:06:56.457Ztflt 0.000000000
1632023-12-11T21:06:56.457Zdflt 0.000000000
1642023-12-11T21:06:56.457Zkflt 0.000000000
1652023-12-11T21:06:56.457Zlock 0.000000000
1662023-12-11T21:06:56.457Zslp 0.000019670
1672023-12-11T21:06:56.457Zlat 0.000021200
1682023-12-11T21:06:56.457Zstop 0.000130742
1692023-12-11T21:06:56.457Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap-6bafd3fa6eef4151
1702023-12-11T21:06:56.457Z+ chmod a+x tests/bootstrap-6bafd3fa6eef4151
1712023-12-11T21:06:56.457Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1722023-12-11T21:06:56.457Z+ ptime -m gunzip
1732023-12-11T21:06:56.457Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap
1742023-12-11T21:06:58.670Z
1752023-12-11T21:06:58.670Zreal 2.228081583
1762023-12-11T21:06:58.670Zuser 2.086797851
1772023-12-11T21:06:58.695Zsys 0.140944006
1782023-12-11T21:06:58.695Ztrap 0.000158751
1792023-12-11T21:06:58.695Ztflt 0.000000000
1802023-12-11T21:06:58.695Zdflt 0.000000000
1812023-12-11T21:06:58.696Zkflt 0.000000000
1822023-12-11T21:06:58.696Zlock 0.000000000
1832023-12-11T21:06:58.696Zslp 0.000020120
1842023-12-11T21:06:58.696Zlat 0.000011789
1852023-12-11T21:06:58.696Zstop 0.000094432
1862023-12-11T21:06:58.696Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap
1872023-12-11T21:06:58.696Z+ chmod a+x tests/bootstrap
1882023-12-11T21:06:58.696Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1892023-12-11T21:06:58.696Z+ ptime -m gunzip
1902023-12-11T21:06:58.696Z++ basename /input/ci-tools/work/end-to-end-tests/end_to_end_tests-d68396a0af635b1c
1912023-12-11T21:07:00.975Z
1922023-12-11T21:07:01.002Zreal 2.293402175
1932023-12-11T21:07:01.002Zuser 2.157662380
1942023-12-11T21:07:01.003Zsys 0.135299752
1952023-12-11T21:07:01.003Ztrap 0.000211436
1962023-12-11T21:07:01.003Ztflt 0.000000000
1972023-12-11T21:07:01.003Zdflt 0.000000000
1982023-12-11T21:07:01.004Zkflt 0.000000000
1992023-12-11T21:07:01.004Zlock 0.000000000
2002023-12-11T21:07:01.004Zslp 0.000019960
2012023-12-11T21:07:01.004Zlat 0.000022660
2022023-12-11T21:07:01.004Zstop 0.000120882
2032023-12-11T21:07:01.004Z++ basename /input/ci-tools/work/end-to-end-tests/end_to_end_tests-d68396a0af635b1c
2042023-12-11T21:07:01.004Z+ chmod a+x tests/end_to_end_tests-d68396a0af635b1c
2052023-12-11T21:07:01.004Z+ bmat address ls
2062023-12-11T21:07:01.004ZNAME CIDR FIRST LAST COUNT
2072023-12-11T21:07:01.004Zextra 10.151.1.0/24 10.151.1.164 10.151.1.227 64
2082023-12-11T21:07:01.004Z++ bmat address ls -f extra -Ho first,last,count
2092023-12-11T21:07:01.030Z+ set -- 10.151.1.164 10.151.1.227 64
2102023-12-11T21:07:01.030Z+ EXTRA_IP_START=10.151.1.164
2112023-12-11T21:07:01.030Z+ EXTRA_IP_END=10.151.1.227
2122023-12-11T21:07:01.030Z+ EXTRA_IP_COUNT=64
2132023-12-11T21:07:01.031Z+ (( EXTRA_IP_COUNT >= 32 ))
2142023-12-11T21:07:01.031Z+ EXTRA_IP_BASE=10.151.1
2152023-12-11T21:07:01.031Z+ EXTRA_IP_FOCTET=164
2162023-12-11T21:07:01.031Z+ EXTRA_IP_LOCTET=227
2172023-12-11T21:07:01.031Z+ SERVICE_IP_POOL_START=10.151.1.164
2182023-12-11T21:07:01.031Z+ SERVICE_IP_POOL_END=10.151.1.173
2192023-12-11T21:07:01.031Z+ DNS_IP1=10.151.1.164
2202023-12-11T21:07:01.031Z+ DNS_IP2=10.151.1.165
2212023-12-11T21:07:01.031Z+ UPLINK_IP=10.151.1.174
2222023-12-11T21:07:01.031Z+ IPPOOL_START=10.151.1.175
2232023-12-11T21:07:01.031Z+ IPPOOL_END=10.151.1.227
2242023-12-11T21:07:01.031Z++ ipadm show-addr -po type,addr
2252023-12-11T21:07:01.031Z++ awk '-F[:/]' '$1 == "dhcp" {print $2}'
2262023-12-11T21:07:01.032Z+ GATEWAY_IP=10.151.1.100
2272023-12-11T21:07:01.032Z+ [[ -n 10.151.1.100 ]]
2282023-12-11T21:07:01.033Z+ ping -s 10.151.1.100 56 1
2292023-12-11T21:07:01.033ZPING 10.151.1.100: 56 data bytes
2302023-12-11T21:07:01.033Z64 bytes from 10.151.1.100: icmp_seq=0. time=0.010 ms
2312023-12-11T21:07:01.033Z
2322023-12-11T21:07:01.033Z----10.151.1.100 PING Statistics----
2332023-12-11T21:07:01.033Z1 packets transmitted, 1 packets received, 0% packet loss
2342023-12-11T21:07:01.033Zround-trip (ms) min/avg/max/stddev = 0.010/0.010/0.010/-nan
2352023-12-11T21:07:01.033Z++ arp -an
2362023-12-11T21:07:01.033Z++ awk -vgw=10.151.1.100 '$2 == gw {print $NF}'
2372023-12-11T21:07:01.033Z+ GATEWAY_MAC=18:c0:4d:0d:9f:b2
2382023-12-11T21:07:01.033Z+ routeadm -e ipv4-forwarding -u
2392023-12-11T21:07:01.141Z+ PXA_START=10.151.1.164
2402023-12-11T21:07:01.141Z+ PXA_END=10.151.1.227
2412023-12-11T21:07:01.166Z+ export GATEWAY_IP GATEWAY_MAC PXA_START PXA_END
2422023-12-11T21:07:01.166Z+ pfexec zpool create -f scratch c1t1d0 c2t1d0
2432023-12-11T21:07:01.270Z+ ZPOOL_VDEV_DIR=/scratch
2442023-12-11T21:07:01.270Z+ ptime -m pfexec ./tools/create_virtual_hardware.sh
2452023-12-11T21:07:01.297Z+++ dirname ./tools/create_virtual_hardware.sh
2462023-12-11T21:07:01.297Z++ cd ./tools
2472023-12-11T21:07:01.297Z++ pwd
2482023-12-11T21:07:01.297Z+ SOURCE_DIR=/opt/oxide/work/tools
2492023-12-11T21:07:01.297Z+ OMICRON_TOP=/opt/oxide/work/tools/..
2502023-12-11T21:07:01.298Z+ SOFTNPU_MODE=zone
2512023-12-11T21:07:01.298Z+ . /opt/oxide/work/tools/virtual_hardware.sh
2522023-12-11T21:07:01.298Z++ MARKER=/etc/opt/oxide/NO_INSTALL
2532023-12-11T21:07:01.298Z++ [[ -f /etc/opt/oxide/NO_INSTALL ]]
2542023-12-11T21:07:01.298Z++ dladm show-phys -p -o LINK
2552023-12-11T21:07:01.298Z++ head -1
2562023-12-11T21:07:01.373Z+ PHYSICAL_LINK=igb0
2572023-12-11T21:07:01.373Z+ echo 'Using igb0 as physical link'
2582023-12-11T21:07:01.373ZUsing igb0 as physical link
2592023-12-11T21:07:01.373Z+ ensure_run_as_root
2602023-12-11T21:07:01.373Z++ id -u
2612023-12-11T21:07:01.373Z+ [[ 0 -ne 0 ]]
2622023-12-11T21:07:01.373Z+ ensure_zpools
2632023-12-11T21:07:01.374Z+ ZPOOL_TYPES=('oxp_' 'oxi_')
2642023-12-11T21:07:01.374Z+ for ZPOOL_TYPE in "${ZPOOL_TYPES[@]}"
2652023-12-11T21:07:01.374Z+ readarray -t ZPOOLS
2662023-12-11T21:07:01.374Z++ grep '"oxp_' /opt/oxide/work/tools/../smf/sled-agent/non-gimlet/config.toml
2672023-12-11T21:07:01.374Z++ sed 's/[ ",]//g'
2682023-12-11T21:07:01.374Z+ for ZPOOL in "${ZPOOLS[@]}"
2692023-12-11T21:07:01.374Z+ echo 'Zpool: [oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b]'
2702023-12-11T21:07:01.374ZZpool: [oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b]
2712023-12-11T21:07:01.374Z+ VDEV_PATH=/scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
2722023-12-11T21:07:01.374Z+ [[ -f /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
2732023-12-11T21:07:01.374Z+ dd if=/dev/zero of=/scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
2742023-12-11T21:07:01.400Z0+0 records in
2752023-12-11T21:07:01.400Z0+0 records out
2762023-12-11T21:07:01.400Z0 bytes (0 iB) transferred in 0.000006 secs (0 iB/sec)
2772023-12-11T21:07:01.402Z+ success 'ZFS vdev /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
2782023-12-11T21:07:01.402Z+ set +x
2792023-12-11T21:07:01.402ZZFS vdev /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
2802023-12-11T21:07:01.402Z++ zpool list -o name
2812023-12-11T21:07:01.402Z++ grep oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b
2822023-12-11T21:07:01.402Z+ [[ -z '' ]]
2832023-12-11T21:07:01.402Z+ zpool create -f oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
2842023-12-11T21:07:01.435Z+ success 'ZFS zpool oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
2852023-12-11T21:07:01.463Z+ set +x
2862023-12-11T21:07:01.463ZZFS zpool oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b exists
2872023-12-11T21:07:01.463Z+ for ZPOOL in "${ZPOOLS[@]}"
2882023-12-11T21:07:01.464Z+ echo 'Zpool: [oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03]'
2892023-12-11T21:07:01.464ZZpool: [oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03]
2902023-12-11T21:07:01.464Z+ VDEV_PATH=/scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
2912023-12-11T21:07:01.464Z+ [[ -f /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
2922023-12-11T21:07:01.464Z+ dd if=/dev/zero of=/scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
2932023-12-11T21:07:01.468Z0+0 records in
2942023-12-11T21:07:01.468Z0+0 records out
2952023-12-11T21:07:01.468Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
2962023-12-11T21:07:01.468Z+ success 'ZFS vdev /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
2972023-12-11T21:07:01.468Z+ set +x
2982023-12-11T21:07:01.468ZZFS vdev /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
2992023-12-11T21:07:01.468Z++ zpool list -o name
3002023-12-11T21:07:01.469Z++ grep oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03
3012023-12-11T21:07:01.469Z+ [[ -z '' ]]
3022023-12-11T21:07:01.469Z+ zpool create -f oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3032023-12-11T21:07:01.495Z+ success 'ZFS zpool oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3042023-12-11T21:07:01.495Z+ set +x
3052023-12-11T21:07:01.495ZZFS zpool oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
3062023-12-11T21:07:01.495Z+ for ZPOOL in "${ZPOOLS[@]}"
3072023-12-11T21:07:01.495Z+ echo 'Zpool: [oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3082023-12-11T21:07:01.495ZZpool: [oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03]
3092023-12-11T21:07:01.495Z+ VDEV_PATH=/scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3102023-12-11T21:07:01.495Z+ [[ -f /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3112023-12-11T21:07:01.495Z+ dd if=/dev/zero of=/scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3122023-12-11T21:07:01.495Z0+0 records in
3132023-12-11T21:07:01.496Z0+0 records out
3142023-12-11T21:07:01.496Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3152023-12-11T21:07:01.496Z+ success 'ZFS vdev /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3162023-12-11T21:07:01.496Z+ set +x
3172023-12-11T21:07:01.496ZZFS vdev /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
3182023-12-11T21:07:01.496Z++ zpool list -o name
3192023-12-11T21:07:01.496Z++ grep oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03
3202023-12-11T21:07:01.496Z+ [[ -z '' ]]
3212023-12-11T21:07:01.496Z+ zpool create -f oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3222023-12-11T21:07:01.521Z+ success 'ZFS zpool oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3232023-12-11T21:07:01.521Z+ set +x
3242023-12-11T21:07:01.521ZZFS zpool oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
3252023-12-11T21:07:01.522Z+ for ZPOOL in "${ZPOOLS[@]}"
3262023-12-11T21:07:01.522Z+ echo 'Zpool: [oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3272023-12-11T21:07:01.522ZZpool: [oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03]
3282023-12-11T21:07:01.522Z+ VDEV_PATH=/scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3292023-12-11T21:07:01.522Z+ [[ -f /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3302023-12-11T21:07:01.522Z+ dd if=/dev/zero of=/scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3312023-12-11T21:07:01.522Z0+0 records in
3322023-12-11T21:07:01.522Z0+0 records out
3332023-12-11T21:07:01.522Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
3342023-12-11T21:07:01.522Z+ success 'ZFS vdev /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3352023-12-11T21:07:01.522Z+ set +x
3362023-12-11T21:07:01.522ZZFS vdev /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
3372023-12-11T21:07:01.522Z++ zpool list -o name
3382023-12-11T21:07:01.548Z++ grep oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03
3392023-12-11T21:07:01.548Z+ [[ -z '' ]]
3402023-12-11T21:07:01.548Z+ zpool create -f oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3412023-12-11T21:07:01.577Z+ success 'ZFS zpool oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3422023-12-11T21:07:01.577Z+ set +x
3432023-12-11T21:07:01.577ZZFS zpool oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
3442023-12-11T21:07:01.577Z+ for ZPOOL in "${ZPOOLS[@]}"
3452023-12-11T21:07:01.577Z+ echo 'Zpool: [oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3462023-12-11T21:07:01.577ZZpool: [oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03]
3472023-12-11T21:07:01.577Z+ VDEV_PATH=/scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3482023-12-11T21:07:01.577Z+ [[ -f /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3492023-12-11T21:07:01.577Z+ dd if=/dev/zero of=/scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3502023-12-11T21:07:01.577Z0+0 records in
3512023-12-11T21:07:01.577Z0+0 records out
3522023-12-11T21:07:01.577Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
3532023-12-11T21:07:01.577Z+ success 'ZFS vdev /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3542023-12-11T21:07:01.578Z+ set +x
3552023-12-11T21:07:01.578ZZFS vdev /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
3562023-12-11T21:07:01.578Z++ zpool list -o name
3572023-12-11T21:07:01.578Z++ grep oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03
3582023-12-11T21:07:01.578Z+ [[ -z '' ]]
3592023-12-11T21:07:01.578Z+ zpool create -f oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3602023-12-11T21:07:01.604Z+ success 'ZFS zpool oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3612023-12-11T21:07:01.604Z+ set +x
3622023-12-11T21:07:01.604ZZFS zpool oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
3632023-12-11T21:07:01.604Z+ for ZPOOL in "${ZPOOLS[@]}"
3642023-12-11T21:07:01.604Z+ echo 'Zpool: [oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930]'
3652023-12-11T21:07:01.604ZZpool: [oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930]
3662023-12-11T21:07:01.604Z+ VDEV_PATH=/scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev
3672023-12-11T21:07:01.604Z+ [[ -f /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev ]]
3682023-12-11T21:07:01.604Z+ dd if=/dev/zero of=/scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev bs=1 count=0 seek=20G
3692023-12-11T21:07:01.604Z0+0 records in
3702023-12-11T21:07:01.604Z0+0 records out
3712023-12-11T21:07:01.604Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3722023-12-11T21:07:01.604Z+ success 'ZFS vdev /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev exists'
3732023-12-11T21:07:01.604Z+ set +x
3742023-12-11T21:07:01.605ZZFS vdev /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev exists
3752023-12-11T21:07:01.653Z++ zpool list -o name
3762023-12-11T21:07:01.653Z++ grep oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930
3772023-12-11T21:07:01.653Z+ [[ -z '' ]]
3782023-12-11T21:07:01.653Z+ zpool create -f oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev
3792023-12-11T21:07:01.679Z+ success 'ZFS zpool oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 exists'
3802023-12-11T21:07:01.679Z+ set +x
3812023-12-11T21:07:01.679ZZFS zpool oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 exists
3822023-12-11T21:07:01.679Z+ for ZPOOL in "${ZPOOLS[@]}"
3832023-12-11T21:07:01.679Z+ echo 'Zpool: [oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60]'
3842023-12-11T21:07:01.679ZZpool: [oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60]
3852023-12-11T21:07:01.679Z+ VDEV_PATH=/scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev
3862023-12-11T21:07:01.679Z+ [[ -f /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev ]]
3872023-12-11T21:07:01.679Z+ dd if=/dev/zero of=/scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev bs=1 count=0 seek=20G
3882023-12-11T21:07:01.679Z0+0 records in
3892023-12-11T21:07:01.680Z0+0 records out
3902023-12-11T21:07:01.680Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
3912023-12-11T21:07:01.680Z+ success 'ZFS vdev /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev exists'
3922023-12-11T21:07:01.680Z+ set +x
3932023-12-11T21:07:01.680ZZFS vdev /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev exists
3942023-12-11T21:07:01.680Z++ zpool list -o name
3952023-12-11T21:07:01.680Z++ grep oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60
3962023-12-11T21:07:01.680Z+ [[ -z '' ]]
3972023-12-11T21:07:01.680Z+ zpool create -f oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev
3982023-12-11T21:07:01.707Z+ success 'ZFS zpool oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 exists'
3992023-12-11T21:07:01.707Z+ set +x
4002023-12-11T21:07:01.707ZZFS zpool oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 exists
4012023-12-11T21:07:01.707Z+ for ZPOOL in "${ZPOOLS[@]}"
4022023-12-11T21:07:01.707Z+ echo 'Zpool: [oxp_31bd71cd-4736-4a12-a387-9b74b050396f]'
4032023-12-11T21:07:01.707ZZpool: [oxp_31bd71cd-4736-4a12-a387-9b74b050396f]
4042023-12-11T21:07:01.707Z+ VDEV_PATH=/scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev
4052023-12-11T21:07:01.707Z+ [[ -f /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev ]]
4062023-12-11T21:07:01.708Z+ dd if=/dev/zero of=/scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev bs=1 count=0 seek=20G
4072023-12-11T21:07:01.708Z0+0 records in
4082023-12-11T21:07:01.708Z0+0 records out
4092023-12-11T21:07:01.708Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
4102023-12-11T21:07:01.708Z+ success 'ZFS vdev /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev exists'
4112023-12-11T21:07:01.708Z+ set +x
4122023-12-11T21:07:01.708ZZFS vdev /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev exists
4132023-12-11T21:07:01.708Z++ zpool list -o name
4142023-12-11T21:07:01.708Z++ grep oxp_31bd71cd-4736-4a12-a387-9b74b050396f
4152023-12-11T21:07:01.708Z+ [[ -z '' ]]
4162023-12-11T21:07:01.708Z+ zpool create -f oxp_31bd71cd-4736-4a12-a387-9b74b050396f /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev
4172023-12-11T21:07:01.733Z+ success 'ZFS zpool oxp_31bd71cd-4736-4a12-a387-9b74b050396f exists'
4182023-12-11T21:07:01.733Z+ set +x
4192023-12-11T21:07:01.734ZZFS zpool oxp_31bd71cd-4736-4a12-a387-9b74b050396f exists
4202023-12-11T21:07:01.734Z+ for ZPOOL in "${ZPOOLS[@]}"
4212023-12-11T21:07:01.734Z+ echo 'Zpool: [oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7]'
4222023-12-11T21:07:01.734ZZpool: [oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7]
4232023-12-11T21:07:01.734Z+ VDEV_PATH=/scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev
4242023-12-11T21:07:01.734Z+ [[ -f /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev ]]
4252023-12-11T21:07:01.734Z+ dd if=/dev/zero of=/scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev bs=1 count=0 seek=20G
4262023-12-11T21:07:01.763Z0+0 records in
4272023-12-11T21:07:01.763Z0+0 records out
4282023-12-11T21:07:01.763Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4292023-12-11T21:07:01.763Z+ success 'ZFS vdev /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev exists'
4302023-12-11T21:07:01.763Z+ set +x
4312023-12-11T21:07:01.763ZZFS vdev /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev exists
4322023-12-11T21:07:01.763Z++ zpool list -o name
4332023-12-11T21:07:01.763Z++ grep oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7
4342023-12-11T21:07:01.763Z+ [[ -z '' ]]
4352023-12-11T21:07:01.763Z+ zpool create -f oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev
4362023-12-11T21:07:01.789Z+ success 'ZFS zpool oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 exists'
4372023-12-11T21:07:01.789Z+ set +x
4382023-12-11T21:07:01.789ZZFS zpool oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 exists
4392023-12-11T21:07:01.789Z+ for ZPOOL_TYPE in "${ZPOOL_TYPES[@]}"
4402023-12-11T21:07:01.789Z+ readarray -t ZPOOLS
4412023-12-11T21:07:01.789Z++ grep '"oxi_' /opt/oxide/work/tools/../smf/sled-agent/non-gimlet/config.toml
4422023-12-11T21:07:01.789Z++ sed 's/[ ",]//g'
4432023-12-11T21:07:01.789Z+ for ZPOOL in "${ZPOOLS[@]}"
4442023-12-11T21:07:01.789Z+ echo 'Zpool: [oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b]'
4452023-12-11T21:07:01.790ZZpool: [oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b]
4462023-12-11T21:07:01.790Z+ VDEV_PATH=/scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4472023-12-11T21:07:01.790Z+ [[ -f /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
4482023-12-11T21:07:01.790Z+ dd if=/dev/zero of=/scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
4492023-12-11T21:07:01.790Z0+0 records in
4502023-12-11T21:07:01.790Z0+0 records out
4512023-12-11T21:07:01.790Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4522023-12-11T21:07:01.791Z+ success 'ZFS vdev /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
4532023-12-11T21:07:01.791Z+ set +x
4542023-12-11T21:07:01.791ZZFS vdev /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
4552023-12-11T21:07:01.791Z++ zpool list -o name
4562023-12-11T21:07:01.791Z++ grep oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b
4572023-12-11T21:07:01.817Z+ [[ -z '' ]]
4582023-12-11T21:07:01.817Z+ zpool create -f oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4592023-12-11T21:07:01.842Z+ success 'ZFS zpool oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
4602023-12-11T21:07:01.843Z+ set +x
4612023-12-11T21:07:01.843ZZFS zpool oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b exists
4622023-12-11T21:07:01.843Z+ for ZPOOL in "${ZPOOLS[@]}"
4632023-12-11T21:07:01.845Z+ echo 'Zpool: [oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b]'
4642023-12-11T21:07:01.846ZZpool: [oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b]
4652023-12-11T21:07:01.846Z+ VDEV_PATH=/scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4662023-12-11T21:07:01.846Z+ [[ -f /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
4672023-12-11T21:07:01.846Z+ dd if=/dev/zero of=/scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
4682023-12-11T21:07:01.846Z0+0 records in
4692023-12-11T21:07:01.846Z0+0 records out
4702023-12-11T21:07:01.846Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
4712023-12-11T21:07:01.846Z+ success 'ZFS vdev /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
4722023-12-11T21:07:01.846Z+ set +x
4732023-12-11T21:07:01.846ZZFS vdev /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
4742023-12-11T21:07:01.846Z++ zpool list -o name
4752023-12-11T21:07:01.846Z++ grep oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b
4762023-12-11T21:07:01.872Z+ [[ -z '' ]]
4772023-12-11T21:07:01.872Z+ zpool create -f oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4782023-12-11T21:07:01.897Z+ success 'ZFS zpool oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
4792023-12-11T21:07:01.897Z+ set +x
4802023-12-11T21:07:01.898ZZFS zpool oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b exists
4812023-12-11T21:07:01.898Z+ [[ zone == \z\o\n\e ]]
4822023-12-11T21:07:01.898Z+ ensure_simulated_links igb0
4832023-12-11T21:07:01.898Z+ local PHYSICAL_LINK=igb0
4842023-12-11T21:07:01.898Z+ INDICES=("0" "1")
4852023-12-11T21:07:01.898Z+ for I in "${INDICES[@]}"
4862023-12-11T21:07:01.898Z++ get_simnet_name_if_exists net0
4872023-12-11T21:07:01.898Z++ dladm show-simnet -p -o LINK net0
4882023-12-11T21:07:01.899Z++ echo ''
4892023-12-11T21:07:01.899Z+ [[ -z '' ]]
4902023-12-11T21:07:01.899Z+ dladm create-simnet -t net0
4912023-12-11T21:07:01.899Z+ dladm create-simnet -t sc0_0
4922023-12-11T21:07:01.899Z+ dladm modify-simnet -t -p net0 sc0_0
4932023-12-11T21:07:01.899Z+ dladm set-linkprop -p mtu=9000 sc0_0
4942023-12-11T21:07:01.899Z+ success 'Simnet net0/sc0_0 exists'
4952023-12-11T21:07:01.899Z+ set +x
4962023-12-11T21:07:01.899ZSimnet net0/sc0_0 exists
4972023-12-11T21:07:01.899Z+ for I in "${INDICES[@]}"
4982023-12-11T21:07:01.925Z++ get_simnet_name_if_exists net1
4992023-12-11T21:07:01.925Z++ dladm show-simnet -p -o LINK net1
5002023-12-11T21:07:01.925Z++ echo ''
5012023-12-11T21:07:01.925Z+ [[ -z '' ]]
5022023-12-11T21:07:01.925Z+ dladm create-simnet -t net1
5032023-12-11T21:07:01.926Z+ dladm create-simnet -t sc1_0
5042023-12-11T21:07:01.926Z+ dladm modify-simnet -t -p net1 sc1_0
5052023-12-11T21:07:01.926Z+ dladm set-linkprop -p mtu=9000 sc1_0
5062023-12-11T21:07:01.926Z+ success 'Simnet net1/sc1_0 exists'
5072023-12-11T21:07:01.926Z+ set +x
5082023-12-11T21:07:01.926ZSimnet net1/sc1_0 exists
5092023-12-11T21:07:01.926Z++ get_vnic_name_if_exists sc0_1
5102023-12-11T21:07:01.926Z++ dladm show-vnic -p -o LINK sc0_1
5112023-12-11T21:07:01.926Z++ echo ''
5122023-12-11T21:07:01.926Z+ [[ -z '' ]]
5132023-12-11T21:07:01.926Z+ dladm create-vnic -t sc0_1 -l igb0 -m a8:e1:de:01:70:1d
5142023-12-11T21:07:01.955Z+ [[ -v PROMISC_FILT_OFF ]]
5152023-12-11T21:07:01.955Z+ success 'Vnic sc0_1 exists'
5162023-12-11T21:07:01.955Z+ set +x
5172023-12-11T21:07:01.955ZVnic sc0_1 exists
5182023-12-11T21:07:01.955Z+ warn_if_no_proxy_arp
5192023-12-11T21:07:01.955Z+ [[ -v PXA_START ]]
5202023-12-11T21:07:01.955Z+ [[ -v PXA_END ]]
5212023-12-11T21:07:01.955Z+ ensure_softnpu_zone
5222023-12-11T21:07:01.955Z+ zoneadm list
5232023-12-11T21:07:01.955Z+ grep -q sidecar_softnpu
5242023-12-11T21:07:01.955Z+ [[ -f out/npuzone/npuzone ]]
5252023-12-11T21:07:01.955Z+ out/npuzone/npuzone create sidecar --omicron-zone --ports sc0_0,tfportrear0_0 --ports sc0_1,tfportqsfp0_0 --sidecar-lite-commit 45ed98fea5824feb4d42f45bbf218e597dc9fc58 --softnpu-commit dbab082dfa89da5db5ca2325c257089d2f130092
5262023-12-11T21:07:01.981Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/softnpu/image/dbab082dfa89da5db5ca2325c257089d2f130092/softnpu.sha256.txt
5272023-12-11T21:07:07.665Zdownloading softnpu ...done
5282023-12-11T21:07:07.691Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/sidecar-lite/release/45ed98fea5824feb4d42f45bbf218e597dc9fc58/libsidecar_lite.so.sha256.txt
5292023-12-11T21:07:13.762Zdownloading asic_program.so ...done
5302023-12-11T21:07:13.789Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/sidecar-lite/release/45ed98fea5824feb4d42f45bbf218e597dc9fc58/scadm.sha256.txt
5312023-12-11T21:07:20.054Zdownloading scadm ...done
5322023-12-11T21:07:20.080Zcreating port tfportrear0_0 for sc0_0
5332023-12-11T21:07:20.080Zcreating port tfportqsfp0_0 for sc0_1
5342023-12-11T21:07:20.080Zsoftnpu zone setup
5352023-12-11T21:07:20.080Zconfigure zone
5362023-12-11T21:07:20.105Zinstall zone
5372023-12-11T21:07:21.183Zboot zone
5382023-12-11T21:07:24.288Z[sidecar_softnpu] svcs milestone/network
5392023-12-11T21:07:24.316ZSTATE STIME FMRI
5402023-12-11T21:07:24.316Zonline 21:07:21 svc:/milestone/network:default
5412023-12-11T21:07:24.316Z[sidecar_softnpu] /softnpu/npu start
5422023-12-11T21:07:24.342Z
5432023-12-11T21:07:24.371Z+ /opt/oxide/work/tools/scrimlet/softnpu-init.sh
5442023-12-11T21:07:24.371Z+ GATEWAY_IP=10.151.1.100
5452023-12-11T21:07:24.371Z+ echo 'Using 10.151.1.100 as gateway ip'
5462023-12-11T21:07:24.371ZUsing 10.151.1.100 as gateway ip
5472023-12-11T21:07:24.372Z+ [[ ! -v GATEWAY_MAC ]]
5482023-12-11T21:07:24.372Z+ GATEWAY_MAC=18:c0:4d:0d:9f:b2
5492023-12-11T21:07:24.372Z++ grep -c -E '^([0-9a-fA-F]{2}:){5}[0-9a-fA-F]{2}$' /dev/fd/63
5502023-12-11T21:07:24.372Z+++ echo 18:c0:4d:0d:9f:b2
5512023-12-11T21:07:24.372Z+ COUNT=1
5522023-12-11T21:07:24.372Z+ [[ 1 -ne 1 ]]
5532023-12-11T21:07:24.375Z+ echo 'Using 18:c0:4d:0d:9f:b2 as gateway mac'
5542023-12-11T21:07:24.375ZUsing 18:c0:4d:0d:9f:b2 as gateway mac
5552023-12-11T21:07:24.375Z+ z_scadm add-arp-entry 10.151.1.100 18:c0:4d:0d:9f:b2
5562023-12-11T21:07:24.376Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone add-arp-entry 10.151.1.100 18:c0:4d:0d:9f:b2
5572023-12-11T21:07:24.401Z+ PXA_MAC=a8:e1:de:01:70:1d
5582023-12-11T21:07:24.402Z+ [[ -v PXA_START ]]
5592023-12-11T21:07:24.402Z+ z_scadm add-proxy-arp 10.151.1.164 10.151.1.227 a8:e1:de:01:70:1d
5602023-12-11T21:07:24.402Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone add-proxy-arp 10.151.1.164 10.151.1.227 a8:e1:de:01:70:1d
5612023-12-11T21:07:24.427Z+ z_scadm dump-state
5622023-12-11T21:07:24.428Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone dump-state
5632023-12-11T21:07:24.453Zlocal v6:
5642023-12-11T21:07:24.453Zlocal v4:
5652023-12-11T21:07:24.453Zrouter v6:
5662023-12-11T21:07:24.453Zrouter v4_idx:
5672023-12-11T21:07:24.453Zrouter v4_routes:
5682023-12-11T21:07:24.453Zresolver v4:
5692023-12-11T21:07:24.453Z10.151.1.100 -> 18:c0:4d:0d:9f:b2
5702023-12-11T21:07:24.453Zresolver v6:
5712023-12-11T21:07:24.453Znat v4:
5722023-12-11T21:07:24.453Znat_v6:
5732023-12-11T21:07:24.453Zport_mac:
5742023-12-11T21:07:24.453Zproxy arp:
5752023-12-11T21:07:24.453Z10.151.1.164/10.151.1.227: a8:e1:de:01:70:1d
5762023-12-11T21:07:24.453Z+ success 'softnpu zone exists'
5772023-12-11T21:07:24.454Z+ set +x
5782023-12-11T21:07:24.454Zsoftnpu zone exists
5792023-12-11T21:07:24.454Z
5802023-12-11T21:07:24.454Zreal 23.167360748
5812023-12-11T21:07:24.454Zuser 0.719280974
5822023-12-11T21:07:24.454Zsys 1.660035193
5832023-12-11T21:07:24.454Ztrap 0.000077948
5842023-12-11T21:07:24.454Ztflt 0.000167519
5852023-12-11T21:07:24.454Zdflt 0.001390842
5862023-12-11T21:07:24.454Zkflt 0.000000000
5872023-12-11T21:07:24.454Zlock 17:53.443708080
5882023-12-11T21:07:24.454Zslp 1:09.495649205
5892023-12-11T21:07:24.454Zlat 0.082357754
5902023-12-11T21:07:24.454Zstop 0.330910732
5912023-12-11T21:07:24.454Z+ tar xf out/omicron-sled-agent.tar pkg/config-rss.toml
5922023-12-11T21:07:24.479Z++ sed -n 's/silo_name = "\(.*\)"/\1/p' pkg/config-rss.toml
5932023-12-11T21:07:24.479Z+ SILO_NAME=recovery
5942023-12-11T21:07:24.479Z++ sed -n 's/external_dns_zone_name = "\(.*\)"/\1/p' pkg/config-rss.toml
5952023-12-11T21:07:24.505Z+ EXTERNAL_DNS_DOMAIN=oxide.test
5962023-12-11T21:07:24.506Z+ sed -i~ '
5972023-12-11T21:07:24.506Z /^external_dns_ips/c\
5982023-12-11T21:07:24.506Zexternal_dns_ips = [ "10.151.1.164", "10.151.1.165" ]
5992023-12-11T21:07:24.506Z /^\[\[internal_services_ip_pool_ranges/,/^$/ {
6002023-12-11T21:07:24.506Z /^first/c\
6012023-12-11T21:07:24.506Zfirst = "10.151.1.164"
6022023-12-11T21:07:24.506Z /^last/c\
6032023-12-11T21:07:24.506Zlast = "10.151.1.173"
6042023-12-11T21:07:24.506Z }
6052023-12-11T21:07:24.506Z /^\[rack_network_config/,/^$/ {
6062023-12-11T21:07:24.506Z /^infra_ip_first/c\
6072023-12-11T21:07:24.506Zinfra_ip_first = "10.151.1.174"
6082023-12-11T21:07:24.506Z /^infra_ip_last/c\
6092023-12-11T21:07:24.506Zinfra_ip_last = "10.151.1.174"
6102023-12-11T21:07:24.506Z }
6112023-12-11T21:07:24.506Z /^\[\[rack_network_config.ports/,/^$/ {
6122023-12-11T21:07:24.507Z /^routes/c\
6132023-12-11T21:07:24.507Zroutes = \[{nexthop = "10.151.1.100", destination = "0.0.0.0/0"}\]
6142023-12-11T21:07:24.507Z /^addresses/c\
6152023-12-11T21:07:24.507Zaddresses = \["10.151.1.174/32"\]
6162023-12-11T21:07:24.507Z }
6172023-12-11T21:07:24.507Z' pkg/config-rss.toml
6182023-12-11T21:07:24.507Z+ diff -u pkg/config-rss.toml~ pkg/config-rss.toml
6192023-12-11T21:07:24.507Z--- pkg/config-rss.toml~ Mon Jul 24 01:21:28 2006
6202023-12-11T21:07:24.507Z+++ pkg/config-rss.toml Mon Dec 11 21:07:23 2023
6212023-12-11T21:07:24.507Z@@ -41,7 +41,7 @@
6222023-12-11T21:07:24.507Z # the DNS domain delegated to the rack by the customer. Each of these addresses
6232023-12-11T21:07:24.507Z # must be contained in one of the "internal services" IP Pool ranges listed
6242023-12-11T21:07:24.507Z # below.
6252023-12-11T21:07:24.507Z-external_dns_ips = [ "192.168.1.20", "192.168.1.21" ]
6262023-12-11T21:07:24.507Z+external_dns_ips = [ "10.151.1.164", "10.151.1.165" ]
6272023-12-11T21:07:24.507Z
6282023-12-11T21:07:24.507Z # Initial TLS certificates for the external API
6292023-12-11T21:07:24.509Z #
6302023-12-11T21:07:24.509Z@@ -77,8 +77,8 @@
6312023-12-11T21:07:24.509Z #
6322023-12-11T21:07:24.509Z # For more on this and what to put here, see docs/how-to-run.adoc.
6332023-12-11T21:07:24.509Z [[internal_services_ip_pool_ranges]]
6342023-12-11T21:07:24.509Z-first = "192.168.1.20"
6352023-12-11T21:07:24.509Z-last = "192.168.1.29"
6362023-12-11T21:07:24.509Z+first = "10.151.1.164"
6372023-12-11T21:07:24.509Z+last = "10.151.1.173"
6382023-12-11T21:07:24.509Z
6392023-12-11T21:07:24.510Z # TODO - this configuration is subject to change going forward. Ultimately these
6402023-12-11T21:07:24.510Z # parameters should be provided to the control plane via wicket, but we need to
6412023-12-11T21:07:24.510Z@@ -92,8 +92,8 @@
6422023-12-11T21:07:24.510Z # A range of IP addresses used by Boundary Services on the external network. In
6432023-12-11T21:07:24.510Z # a real system, these would be addresses of the uplink ports on the Sidecar.
6442023-12-11T21:07:24.510Z # With softnpu, only one address is used.
6452023-12-11T21:07:24.510Z-infra_ip_first = "192.168.1.30"
6462023-12-11T21:07:24.510Z-infra_ip_last = "192.168.1.30"
6472023-12-11T21:07:24.510Z+infra_ip_first = "10.151.1.174"
6482023-12-11T21:07:24.510Z+infra_ip_last = "10.151.1.174"
6492023-12-11T21:07:24.510Z
6502023-12-11T21:07:24.510Z # Configurations for BGP routers to run on the scrimlets.
6512023-12-11T21:07:24.510Z bgp = []
6522023-12-11T21:07:24.510Z@@ -101,9 +101,9 @@
6532023-12-11T21:07:24.510Z # You can configure multiple uplinks by repeating the following stanza
6542023-12-11T21:07:24.511Z [[rack_network_config.ports]]
6552023-12-11T21:07:24.511Z # Routes associated with this port.
6562023-12-11T21:07:24.512Z-routes = [{nexthop = "192.168.1.199", destination = "0.0.0.0/0"}]
6572023-12-11T21:07:24.512Z+routes = [{nexthop = "10.151.1.100", destination = "0.0.0.0/0"}]
6582023-12-11T21:07:24.512Z # Addresses associated with this port.
6592023-12-11T21:07:24.512Z-addresses = ["192.168.1.30/32"]
6602023-12-11T21:07:24.512Z+addresses = ["10.151.1.174/32"]
6612023-12-11T21:07:24.512Z # Name of the uplink port. This should always be "qsfp0" when using softnpu.
6622023-12-11T21:07:24.512Z port = "qsfp0"
6632023-12-11T21:07:24.512Z # The speed of this port.
6642023-12-11T21:07:24.513Z+ true
6652023-12-11T21:07:24.513Z+ tar rvf out/omicron-sled-agent.tar pkg/config-rss.toml
6662023-12-11T21:07:24.513Za pkg/config-rss.toml 7K
6672023-12-11T21:07:24.513Z+ rm -f pkg/config-rss.toml pkg/config-rss.toml~
6682023-12-11T21:07:24.513Z+ TLS_NAME=recovery.sys.oxide.test
6692023-12-11T21:07:24.513Z+ openssl req -newkey rsa:4096 -x509 -sha256 -days 3 -nodes -out pkg/initial-tls-cert.pem -keyout pkg/initial-tls-key.pem -subj /CN=recovery.sys.oxide.test -addext subjectAltName=DNS:recovery.sys.oxide.test -addext basicConstraints=critical,CA:FALSE -config /dev/stdin
6702023-12-11T21:07:25.022Z....+...+..+.+..+............+.+...+.........+..+.+.........+.....+.........+.+......+...+.....+.........+......+.+......+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*..+.+.....+.+........+.+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*....+.+...............+........+.......+......+........+.......+....................+....+...+...+.....+..........+..+..........................................+.+............+..+....+...............+....................................+........+...............................+..+............+.......+.....+.........+.........+...+.......+...+..+.+...........+....+.....+............+.......+..+.+.....+.......+.....+...+...................+...+...+.....+...........................+.+...........+..........+.....+...................+......+...+.....+...+..........+..+......+.+...+..+................+.........+...+..+.+...............+.....+...+......+...+.+..+.........+...+.............+..+.+..+...+.+.........+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
6712023-12-11T21:07:25.723Z..+...+....+...+......+.................+.......+..+.+........+.+.....+...+......+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*..+.........+..+.+.....+....+...........+..........+..+...+...+......+....+.....+.+.....+...+.+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*...........+...+...........................+....+.........+..+....+.........+..+....+.....+......................+...+.....+......+....+.........+.....+...+.......+..+.........+............+.......+........+...+.........+.+.....+................+.........+......+...+...............+..................+...+..+....+.........+......+........+..........+...........+.......+...............+.........+......+..................+..+....+...+........+....+......+............+.....+.+...........................+.........+.........+..+.........+....+..+.........+......+...+....+...+........+.......+..+.......+......+..+..........+..+...............+.+.....+............+.......+...........+.............+..+.........+...+.+..............+......................+...+..+.........+....+...+..+.+..............+...+...+............+...+....+..+.........+......+.........+......+..........+......+...............+...+..+....+........+.+......+........+......+......................+...+..+....+........+.......+...........+.......+........+...................+......+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
6722023-12-11T21:07:25.748Z-----
6732023-12-11T21:07:25.748Z+ tar rvf out/omicron-sled-agent.tar pkg/initial-tls-cert.pem pkg/initial-tls-key.pem
6742023-12-11T21:07:25.773Za pkg/initial-tls-cert.pem 2K
6752023-12-11T21:07:25.773Za pkg/initial-tls-key.pem 4K
6762023-12-11T21:07:25.774Z+ rm -f pkg/initial-tls-cert.pem pkg/initial-tls-key.pem
6772023-12-11T21:07:25.774Z+ rmdir pkg
6782023-12-11T21:07:25.774Z+ E2E_TLS_CERT=/opt/oxide/sled-agent/pkg/initial-tls-cert.pem
6792023-12-11T21:07:25.774Z+ pfexec mkdir -p /usr/oxide
6802023-12-11T21:07:25.774Z+ pfexec curl -sSfL -o /usr/oxide/oxide http://catacomb.eng.oxide.computer:12346/oxide-v0.1.0
6812023-12-11T21:07:26.471Z+ pfexec chmod +x /usr/oxide/oxide
6822023-12-11T21:07:26.497Z+ curl -sSfL -o debian-11-genericcloud-amd64.raw http://catacomb.eng.oxide.computer:12346/debian-11-genericcloud-amd64.raw
6832023-12-11T21:07:44.758Z+ OMICRON_NO_UNINSTALL=1
6842023-12-11T21:07:44.758Z+ ptime -m pfexec ./target/release/omicron-package -t test install
6852023-12-11T21:07:44.784ZDec 11 21:07:43.867 DEBG target[test]: Target({"image": "standard", "machine": "non-gimlet", "rack-topology": "single-sled", "switch": "softnpu"})
6862023-12-11T21:07:44.787ZDec 11 21:07:43.869 INFO Installing service, dst: /opt/oxide/clickhouse.tar.gz, src: out/clickhouse.tar.gz
6872023-12-11T21:07:44.787ZDec 11 21:07:43.869 INFO Installing service, dst: /opt/oxide/ntp.tar.gz, src: out/ntp.tar.gz
6882023-12-11T21:07:44.787ZDec 11 21:07:43.869 INFO Installing service, dst: /opt/oxide/mg-ddm.tar, src: out/mg-ddm-gz.tar
6892023-12-11T21:07:44.787ZDec 11 21:07:43.869 INFO Installing service, dst: /opt/oxide/oximeter.tar.gz, src: out/oximeter-collector.tar.gz
6902023-12-11T21:07:44.787ZDec 11 21:07:43.869 INFO Installing service, dst: /opt/oxide/crucible.tar.gz, src: out/crucible.tar.gz
6912023-12-11T21:07:44.787ZDec 11 21:07:43.869 INFO Installing service, dst: /opt/oxide/crucible_pantry.tar.gz, src: out/crucible-pantry.tar.gz
6922023-12-11T21:07:44.787ZDec 11 21:07:43.869 INFO Installing service, dst: /opt/oxide/switch.tar.gz, src: out/switch-softnpu.tar.gz
6932023-12-11T21:07:44.787ZDec 11 21:07:43.869 INFO Installing service, dst: /opt/oxide/external_dns.tar.gz, src: out/external-dns.tar.gz
6942023-12-11T21:07:44.787ZDec 11 21:07:43.869 INFO Installing service, dst: /opt/oxide/propolis-server.tar.gz, src: out/propolis-server.tar.gz
6952023-12-11T21:07:44.787ZDec 11 21:07:43.869 INFO Installing service, dst: /opt/oxide/overlay.tar.gz, src: out/overlay.tar.gz
6962023-12-11T21:07:44.787ZDec 11 21:07:43.870 INFO Installing service, dst: /opt/oxide/nexus.tar.gz, src: out/omicron-nexus.tar.gz
6972023-12-11T21:07:44.787ZDec 11 21:07:43.870 INFO Installing service, dst: /opt/oxide/clickhouse_keeper.tar.gz, src: out/clickhouse_keeper.tar.gz
6982023-12-11T21:07:44.787ZDec 11 21:07:43.870 INFO Installing service, dst: /opt/oxide/sled-agent.tar, src: out/omicron-sled-agent.tar
6992023-12-11T21:07:44.787ZDec 11 21:07:43.870 INFO Installing service, dst: /opt/oxide/internal_dns.tar.gz, src: out/internal-dns.tar.gz
7002023-12-11T21:07:44.787ZDec 11 21:07:43.870 INFO Installing service, dst: /opt/oxide/cockroachdb.tar.gz, src: out/cockroachdb.tar.gz
7012023-12-11T21:07:44.787ZDec 11 21:07:43.870 INFO Installing service, dst: /opt/oxide/mgs.tar.gz, src: out/omicron-gateway-softnpu.tar.gz
7022023-12-11T21:07:45.022ZDec 11 21:07:44.116 INFO Unpacking service tarball, service_path: /opt/oxide/mg-ddm, tar_path: /opt/oxide/mg-ddm.tar
7032023-12-11T21:07:45.057ZDec 11 21:07:44.152 INFO Unpacking service tarball, service_path: /opt/oxide/sled-agent, tar_path: /opt/oxide/sled-agent.tar
7042023-12-11T21:07:45.115ZDec 11 21:07:44.210 INFO Installing bootstrap service from /opt/oxide/sled-agent/pkg/manifest.xml
7052023-12-11T21:07:45.245Z
7062023-12-11T21:07:45.270Zreal 0.481928414
7072023-12-11T21:07:45.270Zuser 0.099774008
7082023-12-11T21:07:45.270Zsys 1.927896075
7092023-12-11T21:07:45.270Ztrap 0.000011950
7102023-12-11T21:07:45.270Ztflt 0.000040390
7112023-12-11T21:07:45.270Zdflt 0.000968528
7122023-12-11T21:07:45.270Zkflt 0.000000000
7132023-12-11T21:07:45.270Zlock 43.296221278
7142023-12-11T21:07:45.270Zslp 0.693700647
7152023-12-11T21:07:45.270Zlat 0.002932222
7162023-12-11T21:07:45.270Zstop 0.201086736
7172023-12-11T21:07:45.271Z+ retry=0
7182023-12-11T21:07:45.271Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7192023-12-11T21:07:45.271Z+ [[ 0 -gt 30 ]]
7202023-12-11T21:07:45.271Z+ sleep 1
7212023-12-11T21:07:46.260Z+ retry=1
7222023-12-11T21:07:46.260Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7232023-12-11T21:07:46.285Z+ [[ 1 -gt 30 ]]
7242023-12-11T21:07:46.285Z+ sleep 1
7252023-12-11T21:07:47.281Z+ retry=2
7262023-12-11T21:07:47.281Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7272023-12-11T21:07:47.310Z+ [[ 2 -gt 30 ]]
7282023-12-11T21:07:47.310Z+ sleep 1
7292023-12-11T21:07:48.288Z+ retry=3
7302023-12-11T21:07:48.288Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7312023-12-11T21:07:48.313Z+ [[ 3 -gt 30 ]]
7322023-12-11T21:07:48.313Z+ sleep 1
7332023-12-11T21:07:49.301Z+ retry=4
7342023-12-11T21:07:49.301Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7352023-12-11T21:08:14.470Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone dump-state
7362023-12-11T21:08:14.495Zlocal v6:
7372023-12-11T21:08:14.495Zfe80::e4eb:4aff:feb0:51c2
7382023-12-11T21:08:14.495Zfd00:99::1
7392023-12-11T21:08:14.495Zlocal v4:
7402023-12-11T21:08:14.495Z10.151.1.174
7412023-12-11T21:08:14.496Zrouter v6:
7422023-12-11T21:08:14.496Zfd00:1122:3344:101::/64 -> fe80::2cde:54ff:fed9:7871 (1)
7432023-12-11T21:08:14.496Zfd00:1122:3344:2::/64 -> fe80::2cde:54ff:fed9:7871 (1)
7442023-12-11T21:08:14.496Zfd00:1122:3344:3::/64 -> fe80::2cde:54ff:fed9:7871 (1)
7452023-12-11T21:08:14.496Zfdb0:18c0:4d0d:9fb2::/64 -> fe80::2cde:54ff:fed9:7871 (1)
7462023-12-11T21:08:14.496Zrouter v4_idx:
7472023-12-11T21:08:14.521Z0.0.0.0/0 -> 0
7482023-12-11T21:08:14.521Zrouter v4_routes:
7492023-12-11T21:08:14.521Z0 -> 10.151.1.100 (2)
7502023-12-11T21:08:14.521Zresolver v4:
7512023-12-11T21:08:14.521Z10.151.1.174 -> 72:10:10:6e:ca:1d
7522023-12-11T21:08:14.521Z10.151.1.100 -> 18:c0:4d:0d:9f:b2
7532023-12-11T21:08:14.521Zresolver v6:
7542023-12-11T21:08:14.522Zfe80::e4eb:4aff:feb0:51c2 -> e6:eb:4a:b0:51:c2
7552023-12-11T21:08:14.523Zfe80::2cde:54ff:fed9:7871 -> 2e:de:54:d9:78:71
7562023-12-11T21:08:14.523Znat v4:
7572023-12-11T21:08:14.523Znat_v6:
7582023-12-11T21:08:14.523Zport_mac:
7592023-12-11T21:08:14.523Z1000: a8:40:25:65:c7:b4
7602023-12-11T21:08:14.523Z1: a8:40:25:65:c7:b5
7612023-12-11T21:08:14.523Zproxy arp:
7622023-12-11T21:08:14.523Z10.151.1.164/10.151.1.227: a8:e1:de:01:70:1d
7632023-12-11T21:08:14.524Z+ retry=0
7642023-12-11T21:08:14.524Z+++ zoneadm list -n
7652023-12-11T21:08:14.524Z+++ grep oxz_ntp
7662023-12-11T21:08:14.524Z++ pfexec svcs -z -Hostate oxide/ntp
7672023-12-11T21:08:14.524Zsvcs: zone '-Hostate': entity not found
7682023-12-11T21:08:14.524Z++ true
7692023-12-11T21:08:14.524Z+ [[ '' != online ]]
7702023-12-11T21:08:14.524Z+ [[ 0 -gt 60 ]]
7712023-12-11T21:08:14.524Z+ sleep 1
7722023-12-11T21:08:15.523Z+ retry=1
7732023-12-11T21:08:15.549Z+++ zoneadm list -n
7742023-12-11T21:08:15.549Z+++ grep oxz_ntp
7752023-12-11T21:08:15.549Z++ pfexec svcs -z -Hostate oxide/ntp
7762023-12-11T21:08:15.549Zsvcs: zone '-Hostate': entity not found
7772023-12-11T21:08:15.549Z++ true
7782023-12-11T21:08:15.549Z+ [[ '' != online ]]
7792023-12-11T21:08:15.549Z+ [[ 1 -gt 60 ]]
7802023-12-11T21:08:15.550Z+ sleep 1
7812023-12-11T21:08:16.540Z+ retry=2
7822023-12-11T21:08:16.569Z+++ zoneadm list -n
7832023-12-11T21:08:16.569Z+++ grep oxz_ntp
7842023-12-11T21:08:16.569Z++ pfexec svcs -z -Hostate oxide/ntp
7852023-12-11T21:08:16.570Zsvcs: zone '-Hostate': entity not found
7862023-12-11T21:08:16.570Z++ true
7872023-12-11T21:08:16.571Z+ [[ '' != online ]]
7882023-12-11T21:08:16.571Z+ [[ 2 -gt 60 ]]
7892023-12-11T21:08:16.572Z+ sleep 1
7902023-12-11T21:08:17.563Z+ retry=3
7912023-12-11T21:08:17.590Z+++ zoneadm list -n
7922023-12-11T21:08:17.590Z+++ grep oxz_ntp
7932023-12-11T21:08:17.590Z++ pfexec svcs -z -Hostate oxide/ntp
7942023-12-11T21:08:17.590Zsvcs: zone '-Hostate': entity not found
7952023-12-11T21:08:17.591Z++ true
7962023-12-11T21:08:17.591Z+ [[ '' != online ]]
7972023-12-11T21:08:17.591Z+ [[ 3 -gt 60 ]]
7982023-12-11T21:08:17.591Z+ sleep 1
7992023-12-11T21:08:18.587Z+ retry=4
8002023-12-11T21:08:18.619Z+++ zoneadm list -n
8012023-12-11T21:08:18.619Z+++ grep oxz_ntp
8022023-12-11T21:08:18.619Z++ pfexec svcs -z oxz_ntp_ed1f0495-ede0-49de-931d-2b633f1c2ee4 -Hostate oxide/ntp
8032023-12-11T21:08:18.645Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
8042023-12-11T21:08:18.645Z++ true
8052023-12-11T21:08:18.645Z+ [[ '' != online ]]
8062023-12-11T21:08:18.645Z+ [[ 4 -gt 60 ]]
8072023-12-11T21:08:18.645Z+ sleep 1
8082023-12-11T21:08:19.626Z+ retry=5
8092023-12-11T21:08:19.656Z+++ zoneadm list -n
8102023-12-11T21:08:19.656Z+++ grep oxz_ntp
8112023-12-11T21:08:19.656Z++ pfexec svcs -z oxz_ntp_ed1f0495-ede0-49de-931d-2b633f1c2ee4 -Hostate oxide/ntp
8122023-12-11T21:08:19.657Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
8132023-12-11T21:08:19.688Z++ true
8142023-12-11T21:08:19.688Z+ [[ '' != online ]]
8152023-12-11T21:08:19.688Z+ [[ 5 -gt 60 ]]
8162023-12-11T21:08:19.688Z+ sleep 1
8172023-12-11T21:08:20.655Z+ retry=6
8182023-12-11T21:08:20.680Z+++ zoneadm list -n
8192023-12-11T21:08:20.681Z+++ grep oxz_ntp
8202023-12-11T21:08:20.681Z++ pfexec svcs -z oxz_ntp_ed1f0495-ede0-49de-931d-2b633f1c2ee4 -Hostate oxide/ntp
8212023-12-11T21:08:20.716Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
8222023-12-11T21:08:20.716Z++ true
8232023-12-11T21:08:20.716Z+ [[ '' != online ]]
8242023-12-11T21:08:20.716Z+ [[ 6 -gt 60 ]]
8252023-12-11T21:08:20.716Z+ sleep 1
8262023-12-11T21:08:21.685Z+ retry=7
8272023-12-11T21:08:21.713Z+++ zoneadm list -n
8282023-12-11T21:08:21.713Z+++ grep oxz_ntp
8292023-12-11T21:08:21.713Z++ pfexec svcs -z oxz_ntp_ed1f0495-ede0-49de-931d-2b633f1c2ee4 -Hostate oxide/ntp
8302023-12-11T21:08:21.739Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
8312023-12-11T21:08:21.739Z++ true
8322023-12-11T21:08:21.739Z+ [[ '' != online ]]
8332023-12-11T21:08:21.739Z+ [[ 7 -gt 60 ]]
8342023-12-11T21:08:21.739Z+ sleep 1
8352023-12-11T21:08:22.715Z+ retry=8
8362023-12-11T21:08:22.740Z+++ zoneadm list -n
8372023-12-11T21:08:22.741Z+++ grep oxz_ntp
8382023-12-11T21:08:22.741Z++ pfexec svcs -z oxz_ntp_ed1f0495-ede0-49de-931d-2b633f1c2ee4 -Hostate oxide/ntp
8392023-12-11T21:08:22.741Z+ [[ disabled != online ]]
8402023-12-11T21:08:22.741Z+ [[ 8 -gt 60 ]]
8412023-12-11T21:08:22.741Z+ sleep 1
8422023-12-11T21:08:23.737Z+ retry=9
8432023-12-11T21:08:23.767Z+++ zoneadm list -n
8442023-12-11T21:08:23.767Z+++ grep oxz_ntp
8452023-12-11T21:08:23.768Z++ pfexec svcs -z oxz_ntp_ed1f0495-ede0-49de-931d-2b633f1c2ee4 -Hostate oxide/ntp
8462023-12-11T21:08:23.768Z+ [[ disabled != online ]]
8472023-12-11T21:08:23.768Z+ [[ 9 -gt 60 ]]
8482023-12-11T21:08:23.768Z+ sleep 1
8492023-12-11T21:08:24.759Z+ retry=10
8502023-12-11T21:08:24.784Z+++ zoneadm list -n
8512023-12-11T21:08:24.785Z+++ grep oxz_ntp
8522023-12-11T21:08:24.785Z++ pfexec svcs -z oxz_ntp_ed1f0495-ede0-49de-931d-2b633f1c2ee4 -Hostate oxide/ntp
8532023-12-11T21:08:24.785Z+ [[ disabled != online ]]
8542023-12-11T21:08:24.785Z+ [[ 10 -gt 60 ]]
8552023-12-11T21:08:24.785Z+ sleep 1
8562023-12-11T21:08:25.782Z+ retry=11
8572023-12-11T21:08:25.807Z+++ zoneadm list -n
8582023-12-11T21:08:25.807Z+++ grep oxz_ntp
8592023-12-11T21:08:25.807Z++ pfexec svcs -z oxz_ntp_ed1f0495-ede0-49de-931d-2b633f1c2ee4 -Hostate oxide/ntp
8602023-12-11T21:08:25.807Z+ [[ disabled != online ]]
8612023-12-11T21:08:25.807Z+ [[ 11 -gt 60 ]]
8622023-12-11T21:08:25.807Z+ sleep 1
8632023-12-11T21:08:26.801Z+ retry=12
8642023-12-11T21:08:26.826Z+++ zoneadm list -n
8652023-12-11T21:08:26.826Z+++ grep oxz_ntp
8662023-12-11T21:08:26.826Z++ pfexec svcs -z oxz_ntp_ed1f0495-ede0-49de-931d-2b633f1c2ee4 -Hostate oxide/ntp
8672023-12-11T21:08:26.826Z+ [[ disabled != online ]]
8682023-12-11T21:08:26.826Z+ [[ 12 -gt 60 ]]
8692023-12-11T21:08:26.827Z+ sleep 1
8702023-12-11T21:08:27.822Z+ retry=13
8712023-12-11T21:08:27.848Z+++ zoneadm list -n
8722023-12-11T21:08:27.848Z+++ grep oxz_ntp
8732023-12-11T21:08:27.848Z++ pfexec svcs -z oxz_ntp_ed1f0495-ede0-49de-931d-2b633f1c2ee4 -Hostate oxide/ntp
8742023-12-11T21:08:27.848Z+ [[ disabled != online ]]
8752023-12-11T21:08:27.848Z+ [[ 13 -gt 60 ]]
8762023-12-11T21:08:27.848Z+ sleep 1
8772023-12-11T21:08:28.845Z+ retry=14
8782023-12-11T21:08:28.870Z+++ zoneadm list -n
8792023-12-11T21:08:28.870Z+++ grep oxz_ntp
8802023-12-11T21:08:28.870Z++ pfexec svcs -z oxz_ntp_ed1f0495-ede0-49de-931d-2b633f1c2ee4 -Hostate oxide/ntp
8812023-12-11T21:08:28.870Z+ [[ disabled != online ]]
8822023-12-11T21:08:28.870Z+ [[ 14 -gt 60 ]]
8832023-12-11T21:08:28.870Z+ sleep 1
8842023-12-11T21:08:29.868Z+ retry=15
8852023-12-11T21:08:29.896Z+++ zoneadm list -n
8862023-12-11T21:08:29.896Z+++ grep oxz_ntp
8872023-12-11T21:08:29.896Z++ pfexec svcs -z oxz_ntp_ed1f0495-ede0-49de-931d-2b633f1c2ee4 -Hostate oxide/ntp
8882023-12-11T21:08:29.896Z+ [[ disabled != online ]]
8892023-12-11T21:08:29.896Z+ [[ 15 -gt 60 ]]
8902023-12-11T21:08:29.896Z+ sleep 1
 
8912023-12-11T21:08:30.373Zbuskin console login: Dec 11 21:08:29 buskin ip: ip: joining multicasts failed (4) on vopte0 - will use link layer broadcasts for multicast
 
8922023-12-11T21:08:30.889Z+ retry=16
8932023-12-11T21:08:30.914Z+++ zoneadm list -n
8942023-12-11T21:08:30.914Z+++ grep oxz_ntp
8952023-12-11T21:08:30.914Z++ pfexec svcs -z oxz_ntp_ed1f0495-ede0-49de-931d-2b633f1c2ee4 -Hostate oxide/ntp
8962023-12-11T21:08:30.915Z+ [[ online != online ]]
8972023-12-11T21:08:30.915Z+ echo 'Waited for chrony: 16s'
8982023-12-11T21:08:30.915ZWaited for chrony: 16s
8992023-12-11T21:08:30.915Z+ export RUST_BACKTRACE=1
9002023-12-11T21:08:30.915Z+ RUST_BACKTRACE=1
9012023-12-11T21:08:30.915Z+ export E2E_TLS_CERT IPPOOL_START IPPOOL_END
9022023-12-11T21:08:30.916Z++ ./tests/bootstrap
9032023-12-11T21:08:30.941ZUsing external DNS server at 10.151.1.164:53
9042023-12-11T21:08:30.941Z2023-12-11 21:08:30.032500745 UTC: attempting to log into API
9052023-12-11T21:08:45.976Z2023-12-11 21:08:45.949161754 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9062023-12-11T21:08:46.976Z2023-12-11 21:08:46.952573036 UTC: attempting to log into API
9072023-12-11T21:09:02.008Z2023-12-11 21:09:01.995206786 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9082023-12-11T21:09:03.009Z2023-12-11 21:09:02.996348581 UTC: attempting to log into API
9092023-12-11T21:09:18.040Z2023-12-11 21:09:18.027874232 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9102023-12-11T21:09:19.042Z2023-12-11 21:09:19.028978686 UTC: attempting to log into API
9112023-12-11T21:09:34.074Z2023-12-11 21:09:34.061694188 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9122023-12-11T21:09:35.076Z2023-12-11 21:09:35.062643648 UTC: attempting to log into API
9132023-12-11T21:09:50.112Z2023-12-11 21:09:50.095670295 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9142023-12-11T21:09:51.109Z2023-12-11 21:09:51.096572053 UTC: attempting to log into API
9152023-12-11T21:10:14.447Z2023-12-11 21:10:06.128606309 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9162023-12-11T21:10:14.939Z2023-12-11 21:10:07.129521348 UTC: attempting to log into API
9172023-12-11T21:10:22.174Z2023-12-11 21:10:22.162397256 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9182023-12-11T21:10:23.175Z2023-12-11 21:10:23.163507480 UTC: attempting to log into API
9192023-12-11T21:10:38.209Z2023-12-11 21:10:38.197010386 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9202023-12-11T21:10:39.210Z2023-12-11 21:10:39.198810618 UTC: attempting to log into API
9212023-12-11T21:10:54.242Z2023-12-11 21:10:54.230971621 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9222023-12-11T21:10:55.243Z2023-12-11 21:10:55.231789329 UTC: attempting to log into API
9232023-12-11T21:11:10.278Z2023-12-11 21:11:10.264011419 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9242023-12-11T21:11:11.279Z2023-12-11 21:11:11.265006002 UTC: attempting to log into API
9252023-12-11T21:11:26.336Z2023-12-11 21:11:26.314257070 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9262023-12-11T21:11:27.331Z2023-12-11 21:11:27.316581489 UTC: attempting to log into API
9272023-12-11T21:11:42.406Z2023-12-11 21:11:42.387818261 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9282023-12-11T21:11:43.407Z2023-12-11 21:11:43.388810034 UTC: attempting to log into API
 
9292023-12-11T21:11:52.893ZDec 11 21:11:52 buskin ip: ip: joining multicasts failed (4) on vopte5 - will use link layer broadcasts for multicast
 
9302023-12-11T21:11:58.446Z2023-12-11 21:11:58.436069268 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9312023-12-11T21:11:59.448Z2023-12-11 21:11:59.437207333 UTC: attempting to log into API
 
9322023-12-11T21:12:00.103ZDec 11 21:11:59 buskin ip: ip: joining multicasts failed (4) on vopte1 - will use link layer broadcasts for multicast
9332023-12-11T21:12:07.308ZDec 11 21:12:06 buskin ip: ip: joining multicasts failed (4) on vopte2 - will use link layer broadcasts for multicast
9342023-12-11T21:12:13.440ZDec 11 21:12:12 buskin ip: ip: joining multicasts failed (4) on vopte3 - will use link layer broadcasts for multicast
 
9352023-12-11T21:12:14.480Z2023-12-11 21:12:14.469658850 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9362023-12-11T21:12:15.481Z2023-12-11 21:12:15.470795255 UTC: attempting to log into API
 
9372023-12-11T21:12:20.377ZDec 11 21:12:19 buskin ip: ip: joining multicasts failed (4) on vopte4 - will use link layer broadcasts for multicast
 
9382023-12-11T21:12:30.515Z2023-12-11 21:12:30.504162457 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9392023-12-11T21:12:31.520Z2023-12-11 21:12:31.506224697 UTC: attempting to log into API
9402023-12-11T21:12:46.549Z2023-12-11 21:12:46.538444578 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9412023-12-11T21:12:47.556Z2023-12-11 21:12:47.539461540 UTC: attempting to log into API
9422023-12-11T21:12:48.329Z2023-12-11 21:12:48.318483721 UTC: login succeeded
9432023-12-11T21:12:48.361Zwaiting for nexus to come up...
9442023-12-11T21:12:48.436Zcreating IP pool... 10.151.1.175 - 10.151.1.227
9452023-12-11T21:12:48.489Zensuring datasets are ready...
9462023-12-11T21:12:51.179Zlet's roll.
9472023-12-11T21:12:51.205Z+ eval 'OXIDE_HOST=https://recovery.sys.oxide.test
9482023-12-11T21:12:51.205ZOXIDE_RESOLVE=recovery.sys.oxide.test:443:10.151.1.166
9492023-12-11T21:12:51.205ZOXIDE_TOKEN=oxide-token-2dafc4db219af635d1965266ca798d9408a96f98'
9502023-12-11T21:12:51.205Z++ OXIDE_HOST=https://recovery.sys.oxide.test
9512023-12-11T21:12:51.205Z++ OXIDE_RESOLVE=recovery.sys.oxide.test:443:10.151.1.166
9522023-12-11T21:12:51.205Z++ OXIDE_TOKEN=oxide-token-2dafc4db219af635d1965266ca798d9408a96f98
9532023-12-11T21:12:51.205Z+ export OXIDE_HOST OXIDE_TOKEN
9542023-12-11T21:12:51.205Z+ retry=0
9552023-12-11T21:12:51.206Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.1.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
9562023-12-11T21:12:51.271Z+ /usr/oxide/oxide --resolve recovery.sys.oxide.test:443:10.151.1.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem project create --name images --description 'some images'
9572023-12-11T21:12:51.677Z{"status":"ok"}success
9582023-12-11T21:12:51.678ZProject {
9592023-12-11T21:12:51.678Z description: "some images",
9602023-12-11T21:12:51.678Z id: 563fe63e-5e10-4ba8-8b93-486b844e53fe,
9612023-12-11T21:12:51.678Z name: Name(
9622023-12-11T21:12:51.678Z "images",
9632023-12-11T21:12:51.678Z ),
9642023-12-11T21:12:51.703Z time_created: 2023-12-11T21:12:51.344631Z,
9652023-12-11T21:12:51.703Z time_modified: 2023-12-11T21:12:51.344631Z,
9662023-12-11T21:12:51.704Z}
9672023-12-11T21:12:51.704Z+ /usr/oxide/oxide --resolve recovery.sys.oxide.test:443:10.151.1.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem disk import --path debian-11-genericcloud-amd64.raw --disk debian11-boot --project images --description 'debian 11 cloud image from distros' --snapshot debian11-snapshot --image debian11 --image-description 'debian 11 original base image' --image-os debian --image-version 11
9682023-12-11T21:18:20.969Zdone!
9692023-12-11T21:18:21.071Z+ /usr/oxide/oxide --resolve recovery.sys.oxide.test:443:10.151.1.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem image promote --project images --image debian11
9702023-12-11T21:18:21.178Zsuccess
9712023-12-11T21:18:21.179ZImage {
9722023-12-11T21:18:21.179Z block_size: ByteCount(
9732023-12-11T21:18:21.179Z 512,
9742023-12-11T21:18:21.179Z ),
9752023-12-11T21:18:21.204Z description: "debian 11 original base image",
9762023-12-11T21:18:21.205Z digest: None,
9772023-12-11T21:18:21.205Z id: 7beb3522-7605-45c2-bc56-a49c61122032,
9782023-12-11T21:18:21.205Z name: Name(
9792023-12-11T21:18:21.205Z "debian11",
9802023-12-11T21:18:21.205Z ),
9812023-12-11T21:18:21.205Z os: "debian",
9822023-12-11T21:18:21.205Z project_id: None,
9832023-12-11T21:18:21.205Z size: ByteCount(
9842023-12-11T21:18:21.205Z 2147483648,
9852023-12-11T21:18:21.205Z ),
9862023-12-11T21:18:21.205Z time_created: 2023-12-11T21:18:20.900272Z,
9872023-12-11T21:18:21.205Z time_modified: 2023-12-11T21:18:21.160983Z,
9882023-12-11T21:18:21.205Z url: None,
9892023-12-11T21:18:21.205Z version: "11",
9902023-12-11T21:18:21.206Z}
9912023-12-11T21:18:21.206Z+ rm ./tests/bootstrap
9922023-12-11T21:18:21.247Z+ for test_bin in tests/*
9932023-12-11T21:18:21.247Z+ ./tests/bootstrap-6bafd3fa6eef4151
9942023-12-11T21:18:21.273Z
9952023-12-11T21:18:21.273Zrunning 0 tests
9962023-12-11T21:18:21.273Z
9972023-12-11T21:18:21.273Ztest result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
9982023-12-11T21:18:21.273Z
9992023-12-11T21:18:21.273Z+ for test_bin in tests/*
10002023-12-11T21:18:21.273Z+ ./tests/end_to_end_tests-d68396a0af635b1c
10012023-12-11T21:18:21.381Z
10022023-12-11T21:18:21.381Zrunning 2 tests
10032023-12-11T21:18:21.719Ztest no_spoof::no_spoof_here ... ok
10042023-12-11T21:18:55.166Ztest instance_launch::instance_launch ... ok
10052023-12-11T21:18:55.166Z
10062023-12-11T21:18:55.166Ztest result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 33.78s
10072023-12-11T21:18:55.166Z
10082023-12-11T21:18:55.192Z+ _exit_trap
10092023-12-11T21:18:55.192Z+ local status=0
10102023-12-11T21:18:55.192Z+ [[ 0 -eq 0 ]]
10112023-12-11T21:18:55.192Z+ exit 0
10122023-12-11T21:18:55.217Zprocess exited: duration 721923 ms, exit code 0
 
10132023-12-11T21:18:55.360Zfound 99 output files
10142023-12-11T21:18:55.360Zuploading: /var/svc/log/oxide-sled-agent:default.log (803321 bytes)
10152023-12-11T21:18:57.877Zuploaded: /var/svc/log/oxide-sled-agent:default.log
10162023-12-11T21:18:57.878Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_da0c3bbb-2b20-4884-bac1-f07abc7ef673/root/var/svc/log/oxide-crucible-pantry:default.log (0 bytes)
10172023-12-11T21:18:58.953Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_da0c3bbb-2b20-4884-bac1-f07abc7ef673/root/var/svc/log/oxide-crucible-pantry:default.log
10182023-12-11T21:18:58.953Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_da0c3bbb-2b20-4884-bac1-f07abc7ef673/root/var/svc/log/oxide-crucible-pantry:default.log.0 (2333 bytes)
10192023-12-11T21:19:00.056Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_da0c3bbb-2b20-4884-bac1-f07abc7ef673/root/var/svc/log/oxide-crucible-pantry:default.log.0
10202023-12-11T21:19:00.081Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_bdd47b6a-ce9c-454f-a0b6-0b2ad7d9cbf3/root/var/svc/log/oxide-external_dns:default.log (15108 bytes)
10212023-12-11T21:19:01.205Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_bdd47b6a-ce9c-454f-a0b6-0b2ad7d9cbf3/root/var/svc/log/oxide-external_dns:default.log
10222023-12-11T21:19:01.206Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_bdd47b6a-ce9c-454f-a0b6-0b2ad7d9cbf3/root/var/svc/log/oxide-external_dns:default.log.0 (11837 bytes)
10232023-12-11T21:19:02.351Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_bdd47b6a-ce9c-454f-a0b6-0b2ad7d9cbf3/root/var/svc/log/oxide-external_dns:default.log.0
10242023-12-11T21:19:02.351Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_6f60cd8c-4792-4bf7-a5f1-7509857b5f0b/root/var/svc/log/oxide-internal_dns:default.log (15018 bytes)
10252023-12-11T21:19:03.499Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_6f60cd8c-4792-4bf7-a5f1-7509857b5f0b/root/var/svc/log/oxide-internal_dns:default.log
10262023-12-11T21:19:03.499Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_6f60cd8c-4792-4bf7-a5f1-7509857b5f0b/root/var/svc/log/oxide-internal_dns:default.log.0 (115794 bytes)
10272023-12-11T21:19:04.898Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_6f60cd8c-4792-4bf7-a5f1-7509857b5f0b/root/var/svc/log/oxide-internal_dns:default.log.0
10282023-12-11T21:19:04.898Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_ed1f0495-ede0-49de-931d-2b633f1c2ee4/root/var/svc/log/oxide-ntp:default.log (0 bytes)
10292023-12-11T21:19:05.974Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_ed1f0495-ede0-49de-931d-2b633f1c2ee4/root/var/svc/log/oxide-ntp:default.log
10302023-12-11T21:19:05.974Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_ed1f0495-ede0-49de-931d-2b633f1c2ee4/root/var/svc/log/oxide-ntp:default.log.0 (1121 bytes)
10312023-12-11T21:19:07.072Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_ed1f0495-ede0-49de-931d-2b633f1c2ee4/root/var/svc/log/oxide-ntp:default.log.0
10322023-12-11T21:19:07.072Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_clickhouse_cdb8aad3-f1d6-4c34-8e7d-fd5dd2522731/root/var/svc/log/oxide-clickhouse:default.log (665289 bytes)
10332023-12-11T21:19:08.595Zupload warning: file "/pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_clickhouse_cdb8aad3-f1d6-4c34-8e7d-fd5dd2522731/root/var/svc/log/oxide-clickhouse:default.log" changed size mid upload: 665289 -> 712152
10342023-12-11T21:19:09.668Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_clickhouse_cdb8aad3-f1d6-4c34-8e7d-fd5dd2522731/root/var/svc/log/oxide-clickhouse:default.log
10352023-12-11T21:19:09.668Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_clickhouse_cdb8aad3-f1d6-4c34-8e7d-fd5dd2522731/root/var/svc/log/oxide-clickhouse:default.log.0 (478559 bytes)
10362023-12-11T21:19:11.523Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_clickhouse_cdb8aad3-f1d6-4c34-8e7d-fd5dd2522731/root/var/svc/log/oxide-clickhouse:default.log.0
10372023-12-11T21:19:11.523Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_32f5950b-b60b-4134-a554-c40853705552/root/var/svc/log/oxide-external_dns:default.log (15101 bytes)
10382023-12-11T21:19:12.670Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_32f5950b-b60b-4134-a554-c40853705552/root/var/svc/log/oxide-external_dns:default.log
10392023-12-11T21:19:12.670Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_32f5950b-b60b-4134-a554-c40853705552/root/var/svc/log/oxide-external_dns:default.log.0 (11834 bytes)
10402023-12-11T21:19:13.818Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_32f5950b-b60b-4134-a554-c40853705552/root/var/svc/log/oxide-external_dns:default.log.0
10412023-12-11T21:19:13.843Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_9362c3e7-5a10-406b-bb1c-082ba71b197c/root/var/svc/log/oxide-nexus:default.log (290700 bytes)
10422023-12-11T21:19:14.439Zupload warning: file "/pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_9362c3e7-5a10-406b-bb1c-082ba71b197c/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 290700 -> 291916
10432023-12-11T21:19:15.513Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_9362c3e7-5a10-406b-bb1c-082ba71b197c/root/var/svc/log/oxide-nexus:default.log
10442023-12-11T21:19:15.513Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_9362c3e7-5a10-406b-bb1c-082ba71b197c/root/var/svc/log/oxide-nexus:default.log.0 (472519 bytes)
10452023-12-11T21:19:17.366Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_9362c3e7-5a10-406b-bb1c-082ba71b197c/root/var/svc/log/oxide-nexus:default.log.0
10462023-12-11T21:19:17.366Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_cockroachdb_12ab93d4-e03d-41ae-81bc-7e7ef5e66346/root/var/svc/log/oxide-cockroachdb:default.log (0 bytes)
10472023-12-11T21:19:18.501Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_cockroachdb_12ab93d4-e03d-41ae-81bc-7e7ef5e66346/root/var/svc/log/oxide-cockroachdb:default.log
10482023-12-11T21:19:18.502Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_cockroachdb_12ab93d4-e03d-41ae-81bc-7e7ef5e66346/root/var/svc/log/oxide-cockroachdb:default.log.0 (5413 bytes)
10492023-12-11T21:19:19.601Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_cockroachdb_12ab93d4-e03d-41ae-81bc-7e7ef5e66346/root/var/svc/log/oxide-cockroachdb:default.log.0
10502023-12-11T21:19:19.628Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_cockroachdb_5850d6ba-5562-4166-8520-6891817c7114/root/var/svc/log/oxide-cockroachdb:default.log (0 bytes)
10512023-12-11T21:19:20.675Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_cockroachdb_5850d6ba-5562-4166-8520-6891817c7114/root/var/svc/log/oxide-cockroachdb:default.log
10522023-12-11T21:19:20.675Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_cockroachdb_5850d6ba-5562-4166-8520-6891817c7114/root/var/svc/log/oxide-cockroachdb:default.log.0 (5413 bytes)
10532023-12-11T21:19:21.785Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_cockroachdb_5850d6ba-5562-4166-8520-6891817c7114/root/var/svc/log/oxide-cockroachdb:default.log.0
10542023-12-11T21:19:21.785Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_015396cf-31a4-4066-a61f-d9ec0b697661/root/var/svc/log/oxide-crucible-agent:default.log (0 bytes)
10552023-12-11T21:19:22.856Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_015396cf-31a4-4066-a61f-d9ec0b697661/root/var/svc/log/oxide-crucible-agent:default.log
10562023-12-11T21:19:22.881Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_015396cf-31a4-4066-a61f-d9ec0b697661/root/var/svc/log/oxide-crucible-agent:default.log.0 (3436 bytes)
10572023-12-11T21:19:23.954Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_015396cf-31a4-4066-a61f-d9ec0b697661/root/var/svc/log/oxide-crucible-agent:default.log.0
10582023-12-11T21:19:23.979Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_015396cf-31a4-4066-a61f-d9ec0b697661/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
10592023-12-11T21:19:25.025Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_015396cf-31a4-4066-a61f-d9ec0b697661/root/var/svc/log/oxide-crucible-downstairs:default.log
10602023-12-11T21:19:25.025Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_015396cf-31a4-4066-a61f-d9ec0b697661/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
10612023-12-11T21:19:26.126Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_015396cf-31a4-4066-a61f-d9ec0b697661/root/var/svc/log/oxide-crucible-downstairs:default.log.0
10622023-12-11T21:19:26.156Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_60eb424c-923b-4dde-a712-8c79eb19f99d/root/var/svc/log/oxide-crucible-agent:default.log (0 bytes)
10632023-12-11T21:19:27.199Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_60eb424c-923b-4dde-a712-8c79eb19f99d/root/var/svc/log/oxide-crucible-agent:default.log
10642023-12-11T21:19:27.199Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_60eb424c-923b-4dde-a712-8c79eb19f99d/root/var/svc/log/oxide-crucible-agent:default.log.0 (3436 bytes)
10652023-12-11T21:19:28.296Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_60eb424c-923b-4dde-a712-8c79eb19f99d/root/var/svc/log/oxide-crucible-agent:default.log.0
10662023-12-11T21:19:28.296Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_60eb424c-923b-4dde-a712-8c79eb19f99d/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
10672023-12-11T21:19:29.368Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_60eb424c-923b-4dde-a712-8c79eb19f99d/root/var/svc/log/oxide-crucible-downstairs:default.log
10682023-12-11T21:19:29.393Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_60eb424c-923b-4dde-a712-8c79eb19f99d/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
10692023-12-11T21:19:30.467Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_60eb424c-923b-4dde-a712-8c79eb19f99d/root/var/svc/log/oxide-crucible-downstairs:default.log.0
10702023-12-11T21:19:30.469Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_pantry_0bf4bf33-a075-4e76-96f7-1f8770d4959d/root/var/svc/log/oxide-crucible-pantry:default.log (730374 bytes)
10712023-12-11T21:19:31.799Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_pantry_0bf4bf33-a075-4e76-96f7-1f8770d4959d/root/var/svc/log/oxide-crucible-pantry:default.log
10722023-12-11T21:19:31.799Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_pantry_0bf4bf33-a075-4e76-96f7-1f8770d4959d/root/var/svc/log/oxide-crucible-pantry:default.log.0 (497840 bytes)
10732023-12-11T21:19:33.086Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_pantry_0bf4bf33-a075-4e76-96f7-1f8770d4959d/root/var/svc/log/oxide-crucible-pantry:default.log.0
10742023-12-11T21:19:33.113Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_cockroachdb_0060f6a1-3859-465d-9d7f-86f9fd24bbb9/root/var/svc/log/oxide-cockroachdb:default.log (0 bytes)
10752023-12-11T21:19:34.158Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_cockroachdb_0060f6a1-3859-465d-9d7f-86f9fd24bbb9/root/var/svc/log/oxide-cockroachdb:default.log
10762023-12-11T21:19:34.159Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_cockroachdb_0060f6a1-3859-465d-9d7f-86f9fd24bbb9/root/var/svc/log/oxide-cockroachdb:default.log.0 (5386 bytes)
10772023-12-11T21:19:35.258Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_cockroachdb_0060f6a1-3859-465d-9d7f-86f9fd24bbb9/root/var/svc/log/oxide-cockroachdb:default.log.0
10782023-12-11T21:19:35.258Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_8f8dc836-de53-4805-8c2c-4334179d6266/root/var/svc/log/oxide-crucible-agent:default.log (10358 bytes)
10792023-12-11T21:19:36.387Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_8f8dc836-de53-4805-8c2c-4334179d6266/root/var/svc/log/oxide-crucible-agent:default.log
10802023-12-11T21:19:36.387Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_8f8dc836-de53-4805-8c2c-4334179d6266/root/var/svc/log/oxide-crucible-agent:default.log.0 (9792 bytes)
10812023-12-11T21:19:37.505Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_8f8dc836-de53-4805-8c2c-4334179d6266/root/var/svc/log/oxide-crucible-agent:default.log.0
10822023-12-11T21:19:37.505Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_8f8dc836-de53-4805-8c2c-4334179d6266/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
10832023-12-11T21:19:38.577Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_8f8dc836-de53-4805-8c2c-4334179d6266/root/var/svc/log/oxide-crucible-downstairs:default.log
10842023-12-11T21:19:38.578Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_8f8dc836-de53-4805-8c2c-4334179d6266/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
10852023-12-11T21:19:39.685Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_8f8dc836-de53-4805-8c2c-4334179d6266/root/var/svc/log/oxide-crucible-downstairs:default.log.0
10862023-12-11T21:19:39.686Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_8f8dc836-de53-4805-8c2c-4334179d6266/root/var/svc/log/oxide-crucible-downstairs:downstairs-6534e688-f42c-4f26-b766-814c6e7e39ac.log (1906 bytes)
10872023-12-11T21:19:40.785Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_8f8dc836-de53-4805-8c2c-4334179d6266/root/var/svc/log/oxide-crucible-downstairs:downstairs-6534e688-f42c-4f26-b766-814c6e7e39ac.log
10882023-12-11T21:19:40.785Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_8f8dc836-de53-4805-8c2c-4334179d6266/root/var/svc/log/oxide-crucible-downstairs:downstairs-6534e688-f42c-4f26-b766-814c6e7e39ac.log.0 (4530 bytes)
10892023-12-11T21:19:41.884Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_8f8dc836-de53-4805-8c2c-4334179d6266/root/var/svc/log/oxide-crucible-downstairs:downstairs-6534e688-f42c-4f26-b766-814c6e7e39ac.log.0
10902023-12-11T21:19:41.884Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_8f8dc836-de53-4805-8c2c-4334179d6266/root/var/svc/log/oxide-crucible-downstairs:snapshot-6534e688-f42c-4f26-b766-814c6e7e39ac-cf9d3c42-58a2-471c-a9be-825ab5d395cf.log (5158 bytes)
10912023-12-11T21:19:42.977Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_8f8dc836-de53-4805-8c2c-4334179d6266/root/var/svc/log/oxide-crucible-downstairs:snapshot-6534e688-f42c-4f26-b766-814c6e7e39ac-cf9d3c42-58a2-471c-a9be-825ab5d395cf.log
10922023-12-11T21:19:42.977Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_nexus_897ea90d-b018-467b-b278-ee15c6909a59/root/var/svc/log/oxide-nexus:default.log (287189 bytes)
10932023-12-11T21:19:43.331Zupload warning: file "/pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_nexus_897ea90d-b018-467b-b278-ee15c6909a59/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 287189 -> 353536
10942023-12-11T21:19:44.404Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_nexus_897ea90d-b018-467b-b278-ee15c6909a59/root/var/svc/log/oxide-nexus:default.log
10952023-12-11T21:19:44.430Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_nexus_897ea90d-b018-467b-b278-ee15c6909a59/root/var/svc/log/oxide-nexus:default.log.0 (591342 bytes)
10962023-12-11T21:19:45.737Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_nexus_897ea90d-b018-467b-b278-ee15c6909a59/root/var/svc/log/oxide-nexus:default.log.0
10972023-12-11T21:19:45.737Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_44516da6-4c80-4524-8683-911ee33b9595/root/var/svc/log/oxide-crucible-agent:default.log (13686 bytes)
10982023-12-11T21:19:46.857Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_44516da6-4c80-4524-8683-911ee33b9595/root/var/svc/log/oxide-crucible-agent:default.log
10992023-12-11T21:19:46.857Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_44516da6-4c80-4524-8683-911ee33b9595/root/var/svc/log/oxide-crucible-agent:default.log.0 (20040 bytes)
11002023-12-11T21:19:47.996Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_44516da6-4c80-4524-8683-911ee33b9595/root/var/svc/log/oxide-crucible-agent:default.log.0
11012023-12-11T21:19:47.997Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_44516da6-4c80-4524-8683-911ee33b9595/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
11022023-12-11T21:19:49.071Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_44516da6-4c80-4524-8683-911ee33b9595/root/var/svc/log/oxide-crucible-downstairs:default.log
11032023-12-11T21:19:49.072Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_44516da6-4c80-4524-8683-911ee33b9595/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
11042023-12-11T21:19:50.169Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_44516da6-4c80-4524-8683-911ee33b9595/root/var/svc/log/oxide-crucible-downstairs:default.log.0
11052023-12-11T21:19:50.169Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_44516da6-4c80-4524-8683-911ee33b9595/root/var/svc/log/oxide-crucible-downstairs:downstairs-0a8a5b94-67c1-4777-ad41-df7eaa82b3a8.log (2931 bytes)
11062023-12-11T21:19:51.266Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_44516da6-4c80-4524-8683-911ee33b9595/root/var/svc/log/oxide-crucible-downstairs:downstairs-0a8a5b94-67c1-4777-ad41-df7eaa82b3a8.log
11072023-12-11T21:19:51.266Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_44516da6-4c80-4524-8683-911ee33b9595/root/var/svc/log/oxide-crucible-downstairs:downstairs-38a1d81b-b207-4671-9c65-c27f755fbfdd.log (1907 bytes)
11082023-12-11T21:19:52.361Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_44516da6-4c80-4524-8683-911ee33b9595/root/var/svc/log/oxide-crucible-downstairs:downstairs-38a1d81b-b207-4671-9c65-c27f755fbfdd.log
11092023-12-11T21:19:52.361Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_44516da6-4c80-4524-8683-911ee33b9595/root/var/svc/log/oxide-crucible-downstairs:downstairs-38a1d81b-b207-4671-9c65-c27f755fbfdd.log.0 (4531 bytes)
11102023-12-11T21:19:53.470Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_44516da6-4c80-4524-8683-911ee33b9595/root/var/svc/log/oxide-crucible-downstairs:downstairs-38a1d81b-b207-4671-9c65-c27f755fbfdd.log.0
11112023-12-11T21:19:53.470Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_44516da6-4c80-4524-8683-911ee33b9595/root/var/svc/log/oxide-crucible-downstairs:downstairs-6fbd914e-e2d5-446b-83e6-3f6e6bd550e7.log (0 bytes)
11122023-12-11T21:19:54.680Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_44516da6-4c80-4524-8683-911ee33b9595/root/var/svc/log/oxide-crucible-downstairs:downstairs-6fbd914e-e2d5-446b-83e6-3f6e6bd550e7.log
11132023-12-11T21:19:54.680Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_44516da6-4c80-4524-8683-911ee33b9595/root/var/svc/log/oxide-crucible-downstairs:downstairs-6fbd914e-e2d5-446b-83e6-3f6e6bd550e7.log.0 (3050 bytes)
11142023-12-11T21:19:55.781Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_44516da6-4c80-4524-8683-911ee33b9595/root/var/svc/log/oxide-crucible-downstairs:downstairs-6fbd914e-e2d5-446b-83e6-3f6e6bd550e7.log.0
11152023-12-11T21:19:55.781Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_44516da6-4c80-4524-8683-911ee33b9595/root/var/svc/log/oxide-crucible-downstairs:snapshot-38a1d81b-b207-4671-9c65-c27f755fbfdd-cf9d3c42-58a2-471c-a9be-825ab5d395cf.log (5158 bytes)
11162023-12-11T21:19:56.889Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_44516da6-4c80-4524-8683-911ee33b9595/root/var/svc/log/oxide-crucible-downstairs:snapshot-38a1d81b-b207-4671-9c65-c27f755fbfdd-cf9d3c42-58a2-471c-a9be-825ab5d395cf.log
11172023-12-11T21:19:56.889Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_6d6f808f-b6a2-4082-b5fb-d3ded267f0cb/root/var/svc/log/oxide-crucible-agent:default.log (7758 bytes)
11182023-12-11T21:19:58.020Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_6d6f808f-b6a2-4082-b5fb-d3ded267f0cb/root/var/svc/log/oxide-crucible-agent:default.log
11192023-12-11T21:19:58.020Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_6d6f808f-b6a2-4082-b5fb-d3ded267f0cb/root/var/svc/log/oxide-crucible-agent:default.log.0 (19609 bytes)
11202023-12-11T21:19:59.163Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_6d6f808f-b6a2-4082-b5fb-d3ded267f0cb/root/var/svc/log/oxide-crucible-agent:default.log.0
11212023-12-11T21:19:59.191Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_6d6f808f-b6a2-4082-b5fb-d3ded267f0cb/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
11222023-12-11T21:20:00.238Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_6d6f808f-b6a2-4082-b5fb-d3ded267f0cb/root/var/svc/log/oxide-crucible-downstairs:default.log
11232023-12-11T21:20:00.238Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_6d6f808f-b6a2-4082-b5fb-d3ded267f0cb/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
11242023-12-11T21:20:01.382Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_6d6f808f-b6a2-4082-b5fb-d3ded267f0cb/root/var/svc/log/oxide-crucible-downstairs:default.log.0
11252023-12-11T21:20:01.383Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_6d6f808f-b6a2-4082-b5fb-d3ded267f0cb/root/var/svc/log/oxide-crucible-downstairs:downstairs-887a104c-df3b-49dd-94bf-eaf676e66d9f.log (1907 bytes)
11262023-12-11T21:20:02.967Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_6d6f808f-b6a2-4082-b5fb-d3ded267f0cb/root/var/svc/log/oxide-crucible-downstairs:downstairs-887a104c-df3b-49dd-94bf-eaf676e66d9f.log
11272023-12-11T21:20:02.967Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_6d6f808f-b6a2-4082-b5fb-d3ded267f0cb/root/var/svc/log/oxide-crucible-downstairs:downstairs-887a104c-df3b-49dd-94bf-eaf676e66d9f.log.0 (4530 bytes)
11282023-12-11T21:20:04.079Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_6d6f808f-b6a2-4082-b5fb-d3ded267f0cb/root/var/svc/log/oxide-crucible-downstairs:downstairs-887a104c-df3b-49dd-94bf-eaf676e66d9f.log.0
11292023-12-11T21:20:04.105Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_6d6f808f-b6a2-4082-b5fb-d3ded267f0cb/root/var/svc/log/oxide-crucible-downstairs:downstairs-dad6a45a-41c2-422c-822c-36cb5838914e.log (0 bytes)
11302023-12-11T21:20:05.159Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_6d6f808f-b6a2-4082-b5fb-d3ded267f0cb/root/var/svc/log/oxide-crucible-downstairs:downstairs-dad6a45a-41c2-422c-822c-36cb5838914e.log
11312023-12-11T21:20:05.159Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_6d6f808f-b6a2-4082-b5fb-d3ded267f0cb/root/var/svc/log/oxide-crucible-downstairs:downstairs-dad6a45a-41c2-422c-822c-36cb5838914e.log.0 (3051 bytes)
11322023-12-11T21:20:06.263Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_6d6f808f-b6a2-4082-b5fb-d3ded267f0cb/root/var/svc/log/oxide-crucible-downstairs:downstairs-dad6a45a-41c2-422c-822c-36cb5838914e.log.0
11332023-12-11T21:20:06.288Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_6d6f808f-b6a2-4082-b5fb-d3ded267f0cb/root/var/svc/log/oxide-crucible-downstairs:snapshot-887a104c-df3b-49dd-94bf-eaf676e66d9f-cf9d3c42-58a2-471c-a9be-825ab5d395cf.log (5155 bytes)
11342023-12-11T21:20:07.365Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_6d6f808f-b6a2-4082-b5fb-d3ded267f0cb/root/var/svc/log/oxide-crucible-downstairs:snapshot-887a104c-df3b-49dd-94bf-eaf676e66d9f-cf9d3c42-58a2-471c-a9be-825ab5d395cf.log
11352023-12-11T21:20:07.365Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_pantry_b996bfc5-7ede-4759-b393-e7a3b20e783f/root/var/svc/log/oxide-crucible-pantry:default.log (0 bytes)
11362023-12-11T21:20:08.438Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_pantry_b996bfc5-7ede-4759-b393-e7a3b20e783f/root/var/svc/log/oxide-crucible-pantry:default.log
11372023-12-11T21:20:08.439Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_pantry_b996bfc5-7ede-4759-b393-e7a3b20e783f/root/var/svc/log/oxide-crucible-pantry:default.log.0 (2338 bytes)
11382023-12-11T21:20:09.540Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_pantry_b996bfc5-7ede-4759-b393-e7a3b20e783f/root/var/svc/log/oxide-crucible-pantry:default.log.0
11392023-12-11T21:20:09.540Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_oximeter_8df3ad18-234c-45c2-be30-9d1d63d0c2e4/root/var/svc/log/oxide-oximeter:default.log (187546 bytes)
11402023-12-11T21:20:09.784Zupload warning: file "/pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_oximeter_8df3ad18-234c-45c2-be30-9d1d63d0c2e4/root/var/svc/log/oxide-oximeter:default.log" changed size mid upload: 187546 -> 244642
11412023-12-11T21:20:10.867Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_oximeter_8df3ad18-234c-45c2-be30-9d1d63d0c2e4/root/var/svc/log/oxide-oximeter:default.log
11422023-12-11T21:20:10.867Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_oximeter_8df3ad18-234c-45c2-be30-9d1d63d0c2e4/root/var/svc/log/oxide-oximeter:default.log.0 (119689 bytes)
11432023-12-11T21:20:12.390Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_oximeter_8df3ad18-234c-45c2-be30-9d1d63d0c2e4/root/var/svc/log/oxide-oximeter:default.log.0
11442023-12-11T21:20:12.390Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_cockroachdb_77b945d3-997e-40ac-aa28-f37805dc8f7f/root/var/svc/log/oxide-cockroachdb:default.log (0 bytes)
11452023-12-11T21:20:13.465Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_cockroachdb_77b945d3-997e-40ac-aa28-f37805dc8f7f/root/var/svc/log/oxide-cockroachdb:default.log
11462023-12-11T21:20:13.465Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_cockroachdb_77b945d3-997e-40ac-aa28-f37805dc8f7f/root/var/svc/log/oxide-cockroachdb:default.log.0 (5412 bytes)
11472023-12-11T21:20:14.561Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_cockroachdb_77b945d3-997e-40ac-aa28-f37805dc8f7f/root/var/svc/log/oxide-cockroachdb:default.log.0
11482023-12-11T21:20:14.561Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_cockroachdb_f9a606b8-d0ca-4faa-98fa-606ab84389ce/root/var/svc/log/oxide-cockroachdb:default.log (0 bytes)
11492023-12-11T21:20:15.641Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_cockroachdb_f9a606b8-d0ca-4faa-98fa-606ab84389ce/root/var/svc/log/oxide-cockroachdb:default.log
11502023-12-11T21:20:15.641Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_cockroachdb_f9a606b8-d0ca-4faa-98fa-606ab84389ce/root/var/svc/log/oxide-cockroachdb:default.log.0 (5414 bytes)
11512023-12-11T21:20:16.739Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_cockroachdb_f9a606b8-d0ca-4faa-98fa-606ab84389ce/root/var/svc/log/oxide-cockroachdb:default.log.0
11522023-12-11T21:20:16.739Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_253b4fec-2a00-4af6-a3a5-7bdcefc7356e/root/var/svc/log/oxide-crucible-agent:default.log (22276 bytes)
11532023-12-11T21:20:17.880Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_253b4fec-2a00-4af6-a3a5-7bdcefc7356e/root/var/svc/log/oxide-crucible-agent:default.log
11542023-12-11T21:20:17.880Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_253b4fec-2a00-4af6-a3a5-7bdcefc7356e/root/var/svc/log/oxide-crucible-agent:default.log.0 (3436 bytes)
11552023-12-11T21:20:18.981Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_253b4fec-2a00-4af6-a3a5-7bdcefc7356e/root/var/svc/log/oxide-crucible-agent:default.log.0
11562023-12-11T21:20:18.981Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_253b4fec-2a00-4af6-a3a5-7bdcefc7356e/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
11572023-12-11T21:20:20.056Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_253b4fec-2a00-4af6-a3a5-7bdcefc7356e/root/var/svc/log/oxide-crucible-downstairs:default.log
11582023-12-11T21:20:20.056Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_253b4fec-2a00-4af6-a3a5-7bdcefc7356e/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
11592023-12-11T21:20:21.149Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_253b4fec-2a00-4af6-a3a5-7bdcefc7356e/root/var/svc/log/oxide-crucible-downstairs:default.log.0
11602023-12-11T21:20:21.149Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_253b4fec-2a00-4af6-a3a5-7bdcefc7356e/root/var/svc/log/oxide-crucible-downstairs:downstairs-41d30f61-6461-4198-9d60-e35c2fb17575.log (2931 bytes)
11612023-12-11T21:20:22.319Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_253b4fec-2a00-4af6-a3a5-7bdcefc7356e/root/var/svc/log/oxide-crucible-downstairs:downstairs-41d30f61-6461-4198-9d60-e35c2fb17575.log
11622023-12-11T21:20:22.319Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_253b4fec-2a00-4af6-a3a5-7bdcefc7356e/root/var/svc/log/oxide-crucible-downstairs:downstairs-78dc724f-d38b-4391-bfc9-204d9a891109.log (5513 bytes)
11632023-12-11T21:20:23.418Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_253b4fec-2a00-4af6-a3a5-7bdcefc7356e/root/var/svc/log/oxide-crucible-downstairs:downstairs-78dc724f-d38b-4391-bfc9-204d9a891109.log
11642023-12-11T21:20:23.445Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_9f2f54e0-e261-4b75-83b8-cd20e9f06b14/root/var/svc/log/oxide-crucible-agent:default.log (21811 bytes)
11652023-12-11T21:20:24.559Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_9f2f54e0-e261-4b75-83b8-cd20e9f06b14/root/var/svc/log/oxide-crucible-agent:default.log
11662023-12-11T21:20:24.559Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_9f2f54e0-e261-4b75-83b8-cd20e9f06b14/root/var/svc/log/oxide-crucible-agent:default.log.0 (3436 bytes)
11672023-12-11T21:20:25.685Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_9f2f54e0-e261-4b75-83b8-cd20e9f06b14/root/var/svc/log/oxide-crucible-agent:default.log.0
11682023-12-11T21:20:25.685Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_9f2f54e0-e261-4b75-83b8-cd20e9f06b14/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
11692023-12-11T21:20:26.756Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_9f2f54e0-e261-4b75-83b8-cd20e9f06b14/root/var/svc/log/oxide-crucible-downstairs:default.log
11702023-12-11T21:20:26.756Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_9f2f54e0-e261-4b75-83b8-cd20e9f06b14/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
11712023-12-11T21:20:27.856Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_9f2f54e0-e261-4b75-83b8-cd20e9f06b14/root/var/svc/log/oxide-crucible-downstairs:default.log.0
11722023-12-11T21:20:27.856Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_9f2f54e0-e261-4b75-83b8-cd20e9f06b14/root/var/svc/log/oxide-crucible-downstairs:downstairs-2f9e2ece-358d-428e-9249-d6b367013227.log (5515 bytes)
11732023-12-11T21:20:28.962Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_9f2f54e0-e261-4b75-83b8-cd20e9f06b14/root/var/svc/log/oxide-crucible-downstairs:downstairs-2f9e2ece-358d-428e-9249-d6b367013227.log
11742023-12-11T21:20:28.962Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_9f2f54e0-e261-4b75-83b8-cd20e9f06b14/root/var/svc/log/oxide-crucible-downstairs:downstairs-a4213a54-d0fe-4881-ab5c-a7fb7a607c13.log (2926 bytes)
11752023-12-11T21:20:30.061Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_9f2f54e0-e261-4b75-83b8-cd20e9f06b14/root/var/svc/log/oxide-crucible-downstairs:downstairs-a4213a54-d0fe-4881-ab5c-a7fb7a607c13.log
11762023-12-11T21:20:30.061Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_bbd3ea8a-292c-499c-a926-d81f0842c9f3/root/var/svc/log/oxide-crucible-agent:default.log (11147 bytes)
11772023-12-11T21:20:31.182Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_bbd3ea8a-292c-499c-a926-d81f0842c9f3/root/var/svc/log/oxide-crucible-agent:default.log
11782023-12-11T21:20:31.184Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_bbd3ea8a-292c-499c-a926-d81f0842c9f3/root/var/svc/log/oxide-crucible-agent:default.log.0 (3436 bytes)
11792023-12-11T21:20:32.297Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_bbd3ea8a-292c-499c-a926-d81f0842c9f3/root/var/svc/log/oxide-crucible-agent:default.log.0
11802023-12-11T21:20:32.297Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_bbd3ea8a-292c-499c-a926-d81f0842c9f3/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
11812023-12-11T21:20:33.373Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_bbd3ea8a-292c-499c-a926-d81f0842c9f3/root/var/svc/log/oxide-crucible-downstairs:default.log
11822023-12-11T21:20:33.373Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_bbd3ea8a-292c-499c-a926-d81f0842c9f3/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
11832023-12-11T21:20:34.479Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_bbd3ea8a-292c-499c-a926-d81f0842c9f3/root/var/svc/log/oxide-crucible-downstairs:default.log.0
11842023-12-11T21:20:34.479Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_bbd3ea8a-292c-499c-a926-d81f0842c9f3/root/var/svc/log/oxide-crucible-downstairs:downstairs-3e55602b-77ba-4540-9f69-7a57300a2b6c.log (5515 bytes)
11852023-12-11T21:20:35.582Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_bbd3ea8a-292c-499c-a926-d81f0842c9f3/root/var/svc/log/oxide-crucible-downstairs:downstairs-3e55602b-77ba-4540-9f69-7a57300a2b6c.log
11862023-12-11T21:20:35.583Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_26450dca-fd64-4c6c-ac04-8f1320d0c589/root/var/svc/log/oxide-internal_dns:default.log (15022 bytes)
11872023-12-11T21:20:35.675Zupload warning: file "/pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_26450dca-fd64-4c6c-ac04-8f1320d0c589/root/var/svc/log/oxide-internal_dns:default.log" changed size mid upload: 15022 -> 22529
11882023-12-11T21:20:36.751Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_26450dca-fd64-4c6c-ac04-8f1320d0c589/root/var/svc/log/oxide-internal_dns:default.log
11892023-12-11T21:20:36.751Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_26450dca-fd64-4c6c-ac04-8f1320d0c589/root/var/svc/log/oxide-internal_dns:default.log.0 (109621 bytes)
11902023-12-11T21:20:37.988Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_26450dca-fd64-4c6c-ac04-8f1320d0c589/root/var/svc/log/oxide-internal_dns:default.log.0
11912023-12-11T21:20:37.988Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_8fc02c99-fbdd-43d2-a8e6-b27a247d7f65/root/var/svc/log/oxide-crucible-agent:default.log (0 bytes)
11922023-12-11T21:20:39.091Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_8fc02c99-fbdd-43d2-a8e6-b27a247d7f65/root/var/svc/log/oxide-crucible-agent:default.log
11932023-12-11T21:20:39.126Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_8fc02c99-fbdd-43d2-a8e6-b27a247d7f65/root/var/svc/log/oxide-crucible-agent:default.log.0 (14582 bytes)
11942023-12-11T21:20:40.217Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_8fc02c99-fbdd-43d2-a8e6-b27a247d7f65/root/var/svc/log/oxide-crucible-agent:default.log.0
11952023-12-11T21:20:40.218Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_8fc02c99-fbdd-43d2-a8e6-b27a247d7f65/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
11962023-12-11T21:20:41.289Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_8fc02c99-fbdd-43d2-a8e6-b27a247d7f65/root/var/svc/log/oxide-crucible-downstairs:default.log
11972023-12-11T21:20:41.289Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_8fc02c99-fbdd-43d2-a8e6-b27a247d7f65/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
11982023-12-11T21:20:42.384Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_8fc02c99-fbdd-43d2-a8e6-b27a247d7f65/root/var/svc/log/oxide-crucible-downstairs:default.log.0
11992023-12-11T21:20:42.384Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_8fc02c99-fbdd-43d2-a8e6-b27a247d7f65/root/var/svc/log/oxide-crucible-downstairs:downstairs-0c8253b2-bbaa-4fcc-8cd4-bc4bb021e55b.log (0 bytes)
12002023-12-11T21:20:43.880Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_8fc02c99-fbdd-43d2-a8e6-b27a247d7f65/root/var/svc/log/oxide-crucible-downstairs:downstairs-0c8253b2-bbaa-4fcc-8cd4-bc4bb021e55b.log
12012023-12-11T21:20:43.880Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_8fc02c99-fbdd-43d2-a8e6-b27a247d7f65/root/var/svc/log/oxide-crucible-downstairs:downstairs-0c8253b2-bbaa-4fcc-8cd4-bc4bb021e55b.log.0 (3049 bytes)
12022023-12-11T21:20:44.978Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_8fc02c99-fbdd-43d2-a8e6-b27a247d7f65/root/var/svc/log/oxide-crucible-downstairs:downstairs-0c8253b2-bbaa-4fcc-8cd4-bc4bb021e55b.log.0
12032023-12-11T21:20:44.978Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_4856886c-4ef4-47aa-8e18-da03af25154c/root/var/svc/log/oxide-internal_dns:default.log (15009 bytes)
12042023-12-11T21:20:45.098Zupload warning: file "/pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_4856886c-4ef4-47aa-8e18-da03af25154c/root/var/svc/log/oxide-internal_dns:default.log" changed size mid upload: 15009 -> 22515
12052023-12-11T21:20:46.176Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_4856886c-4ef4-47aa-8e18-da03af25154c/root/var/svc/log/oxide-internal_dns:default.log
12062023-12-11T21:20:46.176Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_4856886c-4ef4-47aa-8e18-da03af25154c/root/var/svc/log/oxide-internal_dns:default.log.0 (111649 bytes)
12072023-12-11T21:20:47.579Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_4856886c-4ef4-47aa-8e18-da03af25154c/root/var/svc/log/oxide-internal_dns:default.log.0
12082023-12-11T21:20:47.579Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_635d713c-ddde-4097-8e1c-aa8497b0ac8b/root/var/svc/log/oxide-nexus:default.log (21071190 bytes)
12092023-12-11T21:20:58.119Zupload warning: file "/pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_635d713c-ddde-4097-8e1c-aa8497b0ac8b/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 21071190 -> 21222325
12102023-12-11T21:20:59.194Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_635d713c-ddde-4097-8e1c-aa8497b0ac8b/root/var/svc/log/oxide-nexus:default.log
12112023-12-11T21:20:59.194Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_635d713c-ddde-4097-8e1c-aa8497b0ac8b/root/var/svc/log/oxide-nexus:default.log.0 (14871074 bytes)
12122023-12-11T21:21:06.877Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_635d713c-ddde-4097-8e1c-aa8497b0ac8b/root/var/svc/log/oxide-nexus:default.log.0
12132023-12-11T21:21:06.877Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_ed1f0495-ede0-49de-931d-2b633f1c2ee4/root/var/log/chrony/measurements.log (22742 bytes)
12142023-12-11T21:21:07.410Zupload warning: file "/pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_ed1f0495-ede0-49de-931d-2b633f1c2ee4/root/var/log/chrony/measurements.log" changed size mid upload: 22742 -> 27126
12152023-12-11T21:21:08.085Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_ed1f0495-ede0-49de-931d-2b633f1c2ee4/root/var/log/chrony/measurements.log
12162023-12-11T21:21:08.085Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_ed1f0495-ede0-49de-931d-2b633f1c2ee4/root/var/log/chrony/statistics.log (13689 bytes)
12172023-12-11T21:21:08.177Zupload warning: file "/pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_ed1f0495-ede0-49de-931d-2b633f1c2ee4/root/var/log/chrony/statistics.log" changed size mid upload: 13689 -> 17199
12182023-12-11T21:21:09.249Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_ed1f0495-ede0-49de-931d-2b633f1c2ee4/root/var/log/chrony/statistics.log
12192023-12-11T21:21:09.249Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_ed1f0495-ede0-49de-931d-2b633f1c2ee4/root/var/log/chrony/tracking.log (2250 bytes)
12202023-12-11T21:21:09.296Zupload warning: file "/pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_ed1f0495-ede0-49de-931d-2b633f1c2ee4/root/var/log/chrony/tracking.log" changed size mid upload: 2250 -> 2514
12212023-12-11T21:21:10.368Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_ed1f0495-ede0-49de-931d-2b633f1c2ee4/root/var/log/chrony/tracking.log