01HGFAYJ74SCS4X0RYKVJQ3DSN: helios / deploy

BasicConfig {
    output_rules: [
        "%/var/svc/log/oxide-sled-agent:default.log*",
        "%/pool/ext/*/crypt/zone/oxz_*/root/var/svc/log/oxide-*.log*",
        "%/pool/ext/*/crypt/zone/oxz_*/root/var/svc/log/system-illumos-*.log*",
        "%/pool/ext/*/crypt/zone/oxz_ntp_*/root/var/log/chrony/*.log*",
        "!/pool/ext/*/crypt/zone/oxz_propolis-server_*/root/var/svc/log/*.log*",
        "%/pool/ext/*/crypt/debug/global/oxide-sled-agent:default.log.*",
        "%/pool/ext/*/crypt/debug/oxz_*/oxide-*.log.*",
        "%/pool/ext/*/crypt/debug/oxz_*/system-illumos-*.log.*",
        "!/pool/ext/*/crypt/debug/oxz_propolis-server_*/*.log.*",
    ],
    rust_toolchain: None,
    target: Some(
        "lab-2.0-opte-0.25",
    ),
    access_repos: [],
    publish: [],
    skip_clone: true,
}

Buildomat Job: 01HGFAZ5JXEQNQDE9X0BPC39VW

Tags:

Artefacts:

Output:

SEQ GLOBAL TIME DETAILS
12023-11-30T06:10:32.819Zjob dependencies complete; ready to run (waiting for 45 m 59 s)
22023-11-30T06:13:29.815Zjob assigned to worker 01HGFDFPVZPD8HB61VGTJ1GJ13 (queued for 2 m 56 s)
32023-11-30T06:13:30.015Zhttp://172.20.0.5:9969/ipxe/buskin... ok
42023-11-30T06:13:30.063Zhttp://172.20.0.5:9969/os/buskin/platform/i86pc/kernel/amd64/unix... ok
52023-11-30T06:13:30.103Zhttp://172.20.0.5:9969/os/buskin/platform/i86pc/amd64/boot_archive... ok
62023-11-30T06:13:30.140Zhttp://172.20.0.5:9969/os/buskin/platform/i86pc/amd64/boot_archive.hash... ok
72023-11-30T06:13:30.168Zhttp://172.20.0.5:9969/postboot/buskin... ok
82023-11-30T06:13:30.202ZwcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U
92023-11-30T06:13:30.236Zcpu0: microcode has been updated from version 0xa001133 to 0xa0011d1
102023-11-30T06:13:30.281ZOxide Helios Version helios-2.0.22318 64-bit
112023-11-30T06:13:30.313ZConfiguring devices.
122023-11-30T06:13:30.347Z##################################################
132023-11-30T06:13:30.380Z#### ############# ##
142023-11-30T06:13:30.414Z### ### ############ ##
152023-11-30T06:13:30.854Z## ### # ## ### ## Oxide Computer Company ##
162023-11-30T06:13:30.892Z## ## ## ### # ### ##
172023-11-30T06:13:30.926Z## # ### #### #### This Station Under ##
182023-11-30T06:13:30.958Z### ### #### # ### Computer Control ##
192023-11-30T06:13:30.991Z#### #### ### ## ##
202023-11-30T06:13:31.025Z##################################################
212023-11-30T06:13:31.058ZHostname: buskin
222023-11-30T06:13:31.092ZLoading smf(7) service descriptions:
232023-11-30T06:13:32.108Zdownloading input: /input/ci-tools/work/end-to-end-tests/bootstrap-e73dec5d2cc4791e.gz
242023-11-30T06:13:33.511Zdownloaded input: /input/ci-tools/work/end-to-end-tests/bootstrap-e73dec5d2cc4791e.gz
252023-11-30T06:13:33.511Zdownloading input: /input/ci-tools/work/end-to-end-tests/bootstrap.gz
262023-11-30T06:13:51.586Zdownloaded input: /input/ci-tools/work/end-to-end-tests/bootstrap.gz
272023-11-30T06:13:51.586Zdownloading input: /input/ci-tools/work/end-to-end-tests/end_to_end_tests-e99a7bbe87af43e0.gz
282023-11-30T06:14:11.029Zdownloaded input: /input/ci-tools/work/end-to-end-tests/end_to_end_tests-e99a7bbe87af43e0.gz
292023-11-30T06:14:11.053Zdownloading input: /input/ci-tools/work/caboose-util.gz
302023-11-30T06:14:12.235Zdownloaded input: /input/ci-tools/work/caboose-util.gz
312023-11-30T06:14:12.260Zdownloading input: /input/ci-tools/work/tufaceous.gz
322023-11-30T06:14:15.095Zdownloaded input: /input/ci-tools/work/tufaceous.gz
332023-11-30T06:14:15.125Zdownloading input: /input/package/work/version.txt
342023-11-30T06:14:15.158Zdownloaded input: /input/package/work/version.txt
352023-11-30T06:14:15.158Zdownloading input: /input/package/work/package.tar.gz
362023-11-30T06:14:33.027Zdownloaded input: /input/package/work/package.tar.gz
372023-11-30T06:14:33.056Zdownloading input: /input/package/work/global-zone-packages.tar.gz
382023-11-30T06:14:56.730Zdownloaded input: /input/package/work/global-zone-packages.tar.gz
392023-11-30T06:14:56.730Zdownloading input: /input/package/work/trampoline-global-zone-packages.tar.gz
402023-11-30T06:15:03.238Zdownloaded input: /input/package/work/trampoline-global-zone-packages.tar.gz
412023-11-30T06:15:03.263Zdownloading input: /input/package/work/zones/clickhouse.tar.gz
422023-11-30T06:15:34.342Zdownloaded input: /input/package/work/zones/clickhouse.tar.gz
432023-11-30T06:15:34.342Zdownloading input: /input/package/work/zones/clickhouse_keeper.tar.gz
442023-11-30T06:16:05.474Zdownloaded input: /input/package/work/zones/clickhouse_keeper.tar.gz
452023-11-30T06:16:05.475Zdownloading input: /input/package/work/zones/cockroachdb.tar.gz
462023-11-30T06:16:25.867Zdownloaded input: /input/package/work/zones/cockroachdb.tar.gz
472023-11-30T06:16:25.867Zdownloading input: /input/package/work/zones/crucible-pantry.tar.gz
482023-11-30T06:16:29.805Zdownloaded input: /input/package/work/zones/crucible-pantry.tar.gz
492023-11-30T06:16:29.805Zdownloading input: /input/package/work/zones/crucible.tar.gz
502023-11-30T06:16:37.197Zdownloaded input: /input/package/work/zones/crucible.tar.gz
512023-11-30T06:16:37.197Zdownloading input: /input/package/work/zones/external-dns.tar.gz
522023-11-30T06:16:41.433Zdownloaded input: /input/package/work/zones/external-dns.tar.gz
532023-11-30T06:16:41.433Zdownloading input: /input/package/work/zones/internal-dns.tar.gz
542023-11-30T06:16:45.632Zdownloaded input: /input/package/work/zones/internal-dns.tar.gz
552023-11-30T06:16:45.633Zdownloading input: /input/package/work/zones/ntp.tar.gz
562023-11-30T06:16:45.670Zdownloaded input: /input/package/work/zones/ntp.tar.gz
572023-11-30T06:16:45.670Zdownloading input: /input/package/work/zones/omicron-gateway-asic.tar.gz
582023-11-30T06:16:48.925Zdownloaded input: /input/package/work/zones/omicron-gateway-asic.tar.gz
592023-11-30T06:16:48.925Zdownloading input: /input/package/work/zones/omicron-gateway-softnpu.tar.gz
602023-11-30T06:16:52.198Zdownloaded input: /input/package/work/zones/omicron-gateway-softnpu.tar.gz
612023-11-30T06:16:52.199Zdownloading input: /input/package/work/zones/omicron-nexus-single-sled.tar.gz
622023-11-30T06:17:05.326Zdownloaded input: /input/package/work/zones/omicron-nexus-single-sled.tar.gz
632023-11-30T06:17:05.350Zdownloading input: /input/package/work/zones/omicron-nexus.tar.gz
642023-11-30T06:17:18.239Zdownloaded input: /input/package/work/zones/omicron-nexus.tar.gz
652023-11-30T06:17:18.263Zdownloading input: /input/package/work/zones/overlay.tar.gz
662023-11-30T06:17:18.318Zdownloaded input: /input/package/work/zones/overlay.tar.gz
672023-11-30T06:17:18.318Zdownloading input: /input/package/work/zones/oximeter-collector.tar.gz
682023-11-30T06:17:24.688Zdownloaded input: /input/package/work/zones/oximeter-collector.tar.gz
692023-11-30T06:17:24.688Zdownloading input: /input/package/work/zones/propolis-server.tar.gz
702023-11-30T06:17:36.096Zdownloaded input: /input/package/work/zones/propolis-server.tar.gz
712023-11-30T06:17:36.121Zdownloading input: /input/package/work/zones/switch-asic.tar.gz
722023-11-30T06:18:18.163Zdownloaded input: /input/package/work/zones/switch-asic.tar.gz
732023-11-30T06:18:18.163Zdownloading input: /input/package/work/zones/switch-softnpu.tar.gz
742023-11-30T06:18:47.743Zdownloaded input: /input/package/work/zones/switch-softnpu.tar.gz
 
752023-11-30T06:18:47.767Zstarting task 0: "setup"
762023-11-30T06:18:47.767Z++ uname -s
772023-11-30T06:18:47.767Z+ kern=SunOS
782023-11-30T06:18:47.767Z+ case "$kern" in
792023-11-30T06:18:47.767Z+ groupadd -g 12345 build
802023-11-30T06:18:47.767Z+ useradd -u 12345 -g build -d /home/build -s /bin/bash -c build -P 'Primary Administrator' build
812023-11-30T06:18:49.761Z+ zfs create -o mountpoint=/work rpool/work
822023-11-30T06:18:49.785Z++ awk '$2 == "/home" { print $3 }' /etc/mnttab
832023-11-30T06:18:49.785Z+ home_fs=zfs
842023-11-30T06:18:49.785Z+ [[ zfs == autofs ]]
852023-11-30T06:18:49.785Z+ mkdir -p /home/build
862023-11-30T06:18:49.785Z+ chown build:build /home/build /work
872023-11-30T06:18:50.786Z+ chmod 0700 /home/build /work
882023-11-30T06:18:50.810Zprocess exited: duration 3046 ms, exit code 0
 
892023-11-30T06:18:50.858Zstarting task 1: "authentication"
902023-11-30T06:18:50.903Zprocess exited: duration 44 ms, exit code 0
 
912023-11-30T06:18:50.952Zstarting task 2: "build"
922023-11-30T06:18:50.976Z+ trap _exit_trap EXIT
932023-11-30T06:18:50.976Z+ pfexec mdb -kwe 'secpolicy_fs_linkdir/v 55 48 89 e5 b8 01 00 00 00 5d c3'
942023-11-30T06:18:51.155Zsecpolicy_fs_linkdir: 0x55 = 0x55
952023-11-30T06:18:51.155Zsecpolicy_fs_linkdir+1: 0xb8 = 0x48
962023-11-30T06:18:51.155Zsecpolicy_fs_linkdir+2: 0x1 = 0x89
972023-11-30T06:18:51.155Zsecpolicy_fs_linkdir+3: 0 = 0xe5
982023-11-30T06:18:51.180Zsecpolicy_fs_linkdir+4: 0 = 0xb8
992023-11-30T06:18:51.180Zsecpolicy_fs_linkdir+5: 0 = 0x1
1002023-11-30T06:18:51.180Zsecpolicy_fs_linkdir+6: 0x48 = 0x0
1012023-11-30T06:18:51.180Zsecpolicy_fs_linkdir+7: 0x89 = 0x0
1022023-11-30T06:18:51.180Zsecpolicy_fs_linkdir+8: 0xe5 = 0x0
1032023-11-30T06:18:51.180Zsecpolicy_fs_linkdir+9: 0x48 = 0x5d
1042023-11-30T06:18:51.180Zsecpolicy_fs_linkdir+0xa: 0x83 = 0xc3
1052023-11-30T06:18:51.180Z+ [[ -d /opt/oxide ]]
1062023-11-30T06:18:51.180Z+ pfexec mv /opt/oxide /opt/oxide-underneath
1072023-11-30T06:18:51.180Z+ pfexec mkdir /opt/oxide
1082023-11-30T06:18:51.180Z+ pfexec mount -F tmpfs -O swap /opt/oxide
1092023-11-30T06:18:51.180Z+ [[ -d /opt/oxide-underneath ]]
1102023-11-30T06:18:51.180Z+ cd /opt/oxide-underneath
1112023-11-30T06:18:51.181Z+ pfexec tar ceEp@/f - .
1122023-11-30T06:18:51.181Z+ cd /opt/oxide
1132023-11-30T06:18:51.183Z+ pfexec tar xveEp@/f -
1142023-11-30T06:18:51.280Zx ., 0 bytes, 0 tape blocks
1152023-11-30T06:18:51.280Zx ./opte, 0 bytes, 0 tape blocks
1162023-11-30T06:18:51.280Zx ./opte/bin, 0 bytes, 0 tape blocks
1172023-11-30T06:18:51.280Zx ./opte/bin/opteadm, 50714832 bytes, 99053 tape blocks
1182023-11-30T06:18:51.304Z+ pfexec rm -rf /opt/oxide-underneath
1192023-11-30T06:18:51.305Z+ pfexec /sbin/zfs create -o mountpoint=/zone rpool/zone
1202023-11-30T06:18:51.329Z+ pfexec mkdir /opt/oxide/work
1212023-11-30T06:18:51.329Z+ pfexec chown build:build /opt/oxide/work
1222023-11-30T06:18:51.329Z+ cd /opt/oxide/work
1232023-11-30T06:18:51.329Z+ ptime -m tar xvzf /input/package/work/package.tar.gz
1242023-11-30T06:18:51.329Ztar: blocksize = 10
1252023-11-30T06:18:52.875ZDecompressing '/input/package/work/package.tar.gz' with '/usr/bin/gzcat'...
1262023-11-30T06:18:52.875Zx out/mg-ddm-gz.tar, 62723072 bytes, 122506 tape blocks
1272023-11-30T06:18:52.875Zx out/omicron-sled-agent.tar, 112133120 bytes, 219010 tape blocks
1282023-11-30T06:18:52.875Zx out/target/test, 75 bytes, 1 tape blocks
1292023-11-30T06:18:52.875Zx out/npuzone/npuzone, 37380856 bytes, 73010 tape blocks
1302023-11-30T06:18:52.875Zx package-manifest.toml, 22143 bytes, 44 tape blocks
1312023-11-30T06:18:52.875Zx smf/sled-agent/non-gimlet/config.toml, 3298 bytes, 7 tape blocks
1322023-11-30T06:18:52.875Zx target/release/omicron-package, 42568016 bytes, 83141 tape blocks
1332023-11-30T06:18:52.875Zx tools/create_virtual_hardware.sh, 3176 bytes, 7 tape blocks
1342023-11-30T06:18:52.875Zx tools/virtual_hardware.sh, 3408 bytes, 7 tape blocks
1352023-11-30T06:18:52.875Zx tools/scrimlet/create-softnpu-zone.sh, 619 bytes, 2 tape blocks
1362023-11-30T06:18:52.875Zx tools/scrimlet/destroy-softnpu-zone.sh, 195 bytes, 1 tape blocks
1372023-11-30T06:18:52.875Zx tools/scrimlet/softnpu-init.sh, 2148 bytes, 5 tape blocks
1382023-11-30T06:18:52.875Zx tools/scrimlet/softnpu-zone.txt, 330 bytes, 1 tape blocks
1392023-11-30T06:18:52.875Zx tools/scrimlet/softnpu.toml, 171 bytes, 1 tape blocks
1402023-11-30T06:18:52.899Z
1412023-11-30T06:18:52.900Zreal 1.554322163
1422023-11-30T06:18:52.900Zuser 1.395131607
1432023-11-30T06:18:52.900Zsys 0.300416294
1442023-11-30T06:18:52.900Ztrap 0.000069430
1452023-11-30T06:18:52.900Ztflt 0.000000000
1462023-11-30T06:18:52.900Zdflt 0.000000000
1472023-11-30T06:18:52.900Zkflt 0.000000000
1482023-11-30T06:18:52.900Zlock 0.000000000
1492023-11-30T06:18:52.900Zslp 1.370071845
1502023-11-30T06:18:52.900Zlat 0.041024668
1512023-11-30T06:18:52.900Zstop 0.000118281
1522023-11-30T06:18:52.900Z+ cp /input/package/work/zones/clickhouse_keeper.tar.gz /input/package/work/zones/clickhouse.tar.gz /input/package/work/zones/cockroachdb.tar.gz /input/package/work/zones/crucible-pantry.tar.gz /input/package/work/zones/crucible.tar.gz /input/package/work/zones/external-dns.tar.gz /input/package/work/zones/internal-dns.tar.gz /input/package/work/zones/ntp.tar.gz /input/package/work/zones/omicron-gateway-asic.tar.gz /input/package/work/zones/omicron-gateway-softnpu.tar.gz /input/package/work/zones/omicron-nexus-single-sled.tar.gz /input/package/work/zones/omicron-nexus.tar.gz /input/package/work/zones/overlay.tar.gz /input/package/work/zones/oximeter-collector.tar.gz /input/package/work/zones/propolis-server.tar.gz /input/package/work/zones/switch-asic.tar.gz /input/package/work/zones/switch-softnpu.tar.gz out/
1532023-11-30T06:18:53.942Z+ mv out/omicron-nexus-single-sled.tar.gz out/omicron-nexus.tar.gz
1542023-11-30T06:18:53.967Z+ mkdir tests
1552023-11-30T06:18:53.967Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1562023-11-30T06:18:53.967Z+ ptime -m gunzip
1572023-11-30T06:18:53.967Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap-e73dec5d2cc4791e
1582023-11-30T06:18:54.105Z
1592023-11-30T06:18:54.130Zreal 0.138362664
1602023-11-30T06:18:54.130Zuser 0.126478744
1612023-11-30T06:18:54.130Zsys 0.011631207
1622023-11-30T06:18:54.130Ztrap 0.000010660
1632023-11-30T06:18:54.130Ztflt 0.000000000
1642023-11-30T06:18:54.130Zdflt 0.000000000
1652023-11-30T06:18:54.130Zkflt 0.000000000
1662023-11-30T06:18:54.130Zlock 0.000000000
1672023-11-30T06:18:54.130Zslp 0.000022200
1682023-11-30T06:18:54.130Zlat 0.000015560
1692023-11-30T06:18:54.131Zstop 0.000138441
1702023-11-30T06:18:54.131Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap-e73dec5d2cc4791e
1712023-11-30T06:18:54.131Z+ chmod a+x tests/bootstrap-e73dec5d2cc4791e
1722023-11-30T06:18:54.131Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1732023-11-30T06:18:54.131Z+ ptime -m gunzip
1742023-11-30T06:18:54.131Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap
1752023-11-30T06:18:56.301Z
1762023-11-30T06:18:56.326Zreal 2.184594080
1772023-11-30T06:18:56.326Zuser 2.043016252
1782023-11-30T06:18:56.326Zsys 0.141197442
1792023-11-30T06:18:56.327Ztrap 0.000163553
1802023-11-30T06:18:56.327Ztflt 0.000000000
1812023-11-30T06:18:56.327Zdflt 0.000000000
1822023-11-30T06:18:56.327Zkflt 0.000000000
1832023-11-30T06:18:56.327Zlock 0.000000000
1842023-11-30T06:18:56.327Zslp 0.000019120
1852023-11-30T06:18:56.327Zlat 0.000020030
1862023-11-30T06:18:56.327Zstop 0.000113211
1872023-11-30T06:18:56.327Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap
1882023-11-30T06:18:56.328Z+ chmod a+x tests/bootstrap
1892023-11-30T06:18:56.328Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1902023-11-30T06:18:56.328Z+ ptime -m gunzip
1912023-11-30T06:18:56.328Z++ basename /input/ci-tools/work/end-to-end-tests/end_to_end_tests-e99a7bbe87af43e0
1922023-11-30T06:18:58.558Z
1932023-11-30T06:18:58.582Zreal 2.244690761
1942023-11-30T06:18:58.582Zuser 2.113243892
1952023-11-30T06:18:58.582Zsys 0.130996633
1962023-11-30T06:18:58.582Ztrap 0.000184671
1972023-11-30T06:18:58.582Ztflt 0.000000000
1982023-11-30T06:18:58.582Zdflt 0.000000000
1992023-11-30T06:18:58.582Zkflt 0.000000000
2002023-11-30T06:18:58.582Zlock 0.000000000
2012023-11-30T06:18:58.582Zslp 0.000023520
2022023-11-30T06:18:58.582Zlat 0.000034170
2032023-11-30T06:18:58.582Zstop 0.000116161
2042023-11-30T06:18:58.583Z++ basename /input/ci-tools/work/end-to-end-tests/end_to_end_tests-e99a7bbe87af43e0
2052023-11-30T06:18:58.583Z+ chmod a+x tests/end_to_end_tests-e99a7bbe87af43e0
2062023-11-30T06:18:58.583Z+ bmat address ls
2072023-11-30T06:18:58.583ZNAME CIDR FIRST LAST COUNT
2082023-11-30T06:18:58.583Zextra 10.151.1.0/24 10.151.1.164 10.151.1.227 64
2092023-11-30T06:18:58.583Z++ bmat address ls -f extra -Ho first,last,count
2102023-11-30T06:18:58.608Z+ set -- 10.151.1.164 10.151.1.227 64
2112023-11-30T06:18:58.608Z+ EXTRA_IP_START=10.151.1.164
2122023-11-30T06:18:58.608Z+ EXTRA_IP_END=10.151.1.227
2132023-11-30T06:18:58.608Z+ EXTRA_IP_COUNT=64
2142023-11-30T06:18:58.608Z+ (( EXTRA_IP_COUNT >= 32 ))
2152023-11-30T06:18:58.608Z+ EXTRA_IP_BASE=10.151.1
2162023-11-30T06:18:58.608Z+ EXTRA_IP_FOCTET=164
2172023-11-30T06:18:58.608Z+ EXTRA_IP_LOCTET=227
2182023-11-30T06:18:58.608Z+ SERVICE_IP_POOL_START=10.151.1.164
2192023-11-30T06:18:58.608Z+ SERVICE_IP_POOL_END=10.151.1.173
2202023-11-30T06:18:58.608Z+ DNS_IP1=10.151.1.164
2212023-11-30T06:18:58.608Z+ DNS_IP2=10.151.1.165
2222023-11-30T06:18:58.608Z+ UPLINK_IP=10.151.1.174
2232023-11-30T06:18:58.608Z+ IPPOOL_START=10.151.1.175
2242023-11-30T06:18:58.608Z+ IPPOOL_END=10.151.1.227
2252023-11-30T06:18:58.608Z++ ipadm show-addr -po type,addr
2262023-11-30T06:18:58.608Z++ awk '-F[:/]' '$1 == "dhcp" {print $2}'
2272023-11-30T06:18:58.608Z+ GATEWAY_IP=10.151.1.100
2282023-11-30T06:18:58.608Z+ [[ -n 10.151.1.100 ]]
2292023-11-30T06:18:58.608Z+ ping -s 10.151.1.100 56 1
2302023-11-30T06:18:58.608ZPING 10.151.1.100: 56 data bytes
2312023-11-30T06:18:58.608Z64 bytes from 10.151.1.100: icmp_seq=0. time=0.011 ms
2322023-11-30T06:18:58.608Z
2332023-11-30T06:18:58.608Z----10.151.1.100 PING Statistics----
2342023-11-30T06:18:58.608Z1 packets transmitted, 1 packets received, 0% packet loss
2352023-11-30T06:18:58.608Zround-trip (ms) min/avg/max/stddev = 0.011/0.011/0.011/-nan
2362023-11-30T06:18:58.608Z++ arp -an
2372023-11-30T06:18:58.608Z++ awk -vgw=10.151.1.100 '$2 == gw {print $NF}'
2382023-11-30T06:18:58.632Z+ GATEWAY_MAC=18:c0:4d:0d:9f:b2
2392023-11-30T06:18:58.632Z+ routeadm -e ipv4-forwarding -u
2402023-11-30T06:18:58.723Z+ PXA_START=10.151.1.164
2412023-11-30T06:18:58.723Z+ PXA_END=10.151.1.227
2422023-11-30T06:18:58.747Z+ export GATEWAY_IP GATEWAY_MAC PXA_START PXA_END
2432023-11-30T06:18:58.747Z+ pfexec zpool create -f scratch c1t1d0 c2t1d0
2442023-11-30T06:18:58.864Z+ ZPOOL_VDEV_DIR=/scratch
2452023-11-30T06:18:58.864Z+ ptime -m pfexec ./tools/create_virtual_hardware.sh
2462023-11-30T06:18:58.889Z+++ dirname ./tools/create_virtual_hardware.sh
2472023-11-30T06:18:58.889Z++ cd ./tools
2482023-11-30T06:18:58.889Z++ pwd
2492023-11-30T06:18:58.891Z+ SOURCE_DIR=/opt/oxide/work/tools
2502023-11-30T06:18:58.891Z+ OMICRON_TOP=/opt/oxide/work/tools/..
2512023-11-30T06:18:58.891Z+ SOFTNPU_MODE=zone
2522023-11-30T06:18:58.891Z+ . /opt/oxide/work/tools/virtual_hardware.sh
2532023-11-30T06:18:58.891Z++ MARKER=/etc/opt/oxide/NO_INSTALL
2542023-11-30T06:18:58.891Z++ [[ -f /etc/opt/oxide/NO_INSTALL ]]
2552023-11-30T06:18:58.891Z++ dladm show-phys -p -o LINK
2562023-11-30T06:18:58.891Z++ head -1
2572023-11-30T06:18:58.936Z+ PHYSICAL_LINK=igb0
2582023-11-30T06:18:58.960Z+ echo 'Using igb0 as physical link'
2592023-11-30T06:18:58.960ZUsing igb0 as physical link
2602023-11-30T06:18:58.960Z+ ensure_run_as_root
2612023-11-30T06:18:58.960Z++ id -u
2622023-11-30T06:18:58.960Z+ [[ 0 -ne 0 ]]
2632023-11-30T06:18:58.960Z+ ensure_zpools
2642023-11-30T06:18:58.960Z+ ZPOOL_TYPES=('oxp_' 'oxi_')
2652023-11-30T06:18:58.960Z+ for ZPOOL_TYPE in "${ZPOOL_TYPES[@]}"
2662023-11-30T06:18:58.960Z+ readarray -t ZPOOLS
2672023-11-30T06:18:58.960Z++ grep '"oxp_' /opt/oxide/work/tools/../smf/sled-agent/non-gimlet/config.toml
2682023-11-30T06:18:58.960Z++ sed 's/[ ",]//g'
2692023-11-30T06:18:58.960Z+ for ZPOOL in "${ZPOOLS[@]}"
2702023-11-30T06:18:58.960Z+ echo 'Zpool: [oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b]'
2712023-11-30T06:18:58.960ZZpool: [oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b]
2722023-11-30T06:18:58.960Z+ VDEV_PATH=/scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
2732023-11-30T06:18:58.960Z+ [[ -f /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
2742023-11-30T06:18:58.960Z+ dd if=/dev/zero of=/scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
2752023-11-30T06:18:58.985Z0+0 records in
2762023-11-30T06:18:58.985Z0+0 records out
2772023-11-30T06:18:58.986Z0 bytes (0 iB) transferred in 0.000006 secs (0 iB/sec)
2782023-11-30T06:18:58.986Z+ success 'ZFS vdev /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
2792023-11-30T06:18:58.986Z+ set +x
2802023-11-30T06:18:58.986ZZFS vdev /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
2812023-11-30T06:18:58.986Z++ zpool list -o name
2822023-11-30T06:18:58.986Z++ grep oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b
2832023-11-30T06:18:59.012Z+ [[ -z '' ]]
2842023-11-30T06:18:59.012Z+ zpool create -f oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
2852023-11-30T06:18:59.036Z+ success 'ZFS zpool oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
2862023-11-30T06:18:59.036Z+ set +x
2872023-11-30T06:18:59.036ZZFS zpool oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b exists
2882023-11-30T06:18:59.036Z+ for ZPOOL in "${ZPOOLS[@]}"
2892023-11-30T06:18:59.036Z+ echo 'Zpool: [oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03]'
2902023-11-30T06:18:59.036ZZpool: [oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03]
2912023-11-30T06:18:59.036Z+ VDEV_PATH=/scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
2922023-11-30T06:18:59.036Z+ [[ -f /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
2932023-11-30T06:18:59.036Z+ dd if=/dev/zero of=/scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
2942023-11-30T06:18:59.036Z0+0 records in
2952023-11-30T06:18:59.036Z0+0 records out
2962023-11-30T06:18:59.036Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
2972023-11-30T06:18:59.036Z+ success 'ZFS vdev /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
2982023-11-30T06:18:59.036Z+ set +x
2992023-11-30T06:18:59.036ZZFS vdev /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
3002023-11-30T06:18:59.036Z++ zpool list -o name
3012023-11-30T06:18:59.036Z++ grep oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03
3022023-11-30T06:18:59.061Z+ [[ -z '' ]]
3032023-11-30T06:18:59.061Z+ zpool create -f oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3042023-11-30T06:18:59.085Z+ success 'ZFS zpool oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3052023-11-30T06:18:59.085Z+ set +x
3062023-11-30T06:18:59.085ZZFS zpool oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
3072023-11-30T06:18:59.085Z+ for ZPOOL in "${ZPOOLS[@]}"
3082023-11-30T06:18:59.085Z+ echo 'Zpool: [oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3092023-11-30T06:18:59.085ZZpool: [oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03]
3102023-11-30T06:18:59.085Z+ VDEV_PATH=/scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3112023-11-30T06:18:59.085Z+ [[ -f /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3122023-11-30T06:18:59.085Z+ dd if=/dev/zero of=/scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3132023-11-30T06:18:59.085Z0+0 records in
3142023-11-30T06:18:59.085Z0+0 records out
3152023-11-30T06:18:59.085Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3162023-11-30T06:18:59.085Z+ success 'ZFS vdev /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3172023-11-30T06:18:59.085Z+ set +x
3182023-11-30T06:18:59.085ZZFS vdev /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
3192023-11-30T06:18:59.085Z++ zpool list -o name
3202023-11-30T06:18:59.085Z++ grep oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03
3212023-11-30T06:18:59.085Z+ [[ -z '' ]]
3222023-11-30T06:18:59.085Z+ zpool create -f oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3232023-11-30T06:18:59.110Z+ success 'ZFS zpool oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3242023-11-30T06:18:59.110Z+ set +x
3252023-11-30T06:18:59.110ZZFS zpool oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
3262023-11-30T06:18:59.110Z+ for ZPOOL in "${ZPOOLS[@]}"
3272023-11-30T06:18:59.113Z+ echo 'Zpool: [oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3282023-11-30T06:18:59.113Z+ VDEV_PATH=/scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3292023-11-30T06:18:59.113Z+ [[ -f /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3302023-11-30T06:18:59.113ZZpool: [oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03]
3312023-11-30T06:18:59.113Z+ dd if=/dev/zero of=/scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3322023-11-30T06:18:59.138Z0+0 records in
3332023-11-30T06:18:59.138Z0+0 records out
3342023-11-30T06:18:59.138Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
3352023-11-30T06:18:59.138Z+ success 'ZFS vdev /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3362023-11-30T06:18:59.138Z+ set +x
3372023-11-30T06:18:59.138ZZFS vdev /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
3382023-11-30T06:18:59.138Z++ zpool list -o name
3392023-11-30T06:18:59.138Z++ grep oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03
3402023-11-30T06:18:59.138Z+ [[ -z '' ]]
3412023-11-30T06:18:59.138Z+ zpool create -f oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3422023-11-30T06:18:59.162Z+ success 'ZFS zpool oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3432023-11-30T06:18:59.162Z+ set +x
3442023-11-30T06:18:59.162ZZFS zpool oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
3452023-11-30T06:18:59.162Z+ for ZPOOL in "${ZPOOLS[@]}"
3462023-11-30T06:18:59.162Z+ echo 'Zpool: [oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3472023-11-30T06:18:59.162ZZpool: [oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03]
3482023-11-30T06:18:59.162Z+ VDEV_PATH=/scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3492023-11-30T06:18:59.162Z+ [[ -f /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3502023-11-30T06:18:59.162Z+ dd if=/dev/zero of=/scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3512023-11-30T06:18:59.166Z0+0 records in
3522023-11-30T06:18:59.166Z0+0 records out
3532023-11-30T06:18:59.166Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3542023-11-30T06:18:59.166Z+ success 'ZFS vdev /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3552023-11-30T06:18:59.167Z+ set +x
3562023-11-30T06:18:59.167ZZFS vdev /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
3572023-11-30T06:18:59.167Z++ zpool list -o name
3582023-11-30T06:18:59.167Z++ grep oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03
3592023-11-30T06:18:59.191Z+ [[ -z '' ]]
3602023-11-30T06:18:59.191Z+ zpool create -f oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3612023-11-30T06:18:59.215Z+ success 'ZFS zpool oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3622023-11-30T06:18:59.215Z+ set +x
3632023-11-30T06:18:59.215ZZFS zpool oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
3642023-11-30T06:18:59.215Z+ for ZPOOL in "${ZPOOLS[@]}"
3652023-11-30T06:18:59.215Z+ echo 'Zpool: [oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930]'
3662023-11-30T06:18:59.215ZZpool: [oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930]
3672023-11-30T06:18:59.215Z+ VDEV_PATH=/scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev
3682023-11-30T06:18:59.215Z+ [[ -f /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev ]]
3692023-11-30T06:18:59.215Z+ dd if=/dev/zero of=/scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev bs=1 count=0 seek=20G
3702023-11-30T06:18:59.215Z0+0 records in
3712023-11-30T06:18:59.215Z0+0 records out
3722023-11-30T06:18:59.215Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3732023-11-30T06:18:59.215Z+ success 'ZFS vdev /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev exists'
3742023-11-30T06:18:59.215Z+ set +x
3752023-11-30T06:18:59.215ZZFS vdev /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev exists
3762023-11-30T06:18:59.217Z++ zpool list -o name
3772023-11-30T06:18:59.217Z++ grep oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930
3782023-11-30T06:18:59.217Z+ [[ -z '' ]]
3792023-11-30T06:18:59.217Z+ zpool create -f oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev
3802023-11-30T06:18:59.241Z+ success 'ZFS zpool oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 exists'
3812023-11-30T06:18:59.241Z+ set +x
3822023-11-30T06:18:59.241ZZFS zpool oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 exists
3832023-11-30T06:18:59.241Z+ for ZPOOL in "${ZPOOLS[@]}"
3842023-11-30T06:18:59.241Z+ echo 'Zpool: [oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60]'
3852023-11-30T06:18:59.241ZZpool: [oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60]
3862023-11-30T06:18:59.241Z+ VDEV_PATH=/scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev
3872023-11-30T06:18:59.241Z+ [[ -f /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev ]]
3882023-11-30T06:18:59.241Z+ dd if=/dev/zero of=/scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev bs=1 count=0 seek=20G
3892023-11-30T06:18:59.265Z0+0 records in
3902023-11-30T06:18:59.265Z0+0 records out
3912023-11-30T06:18:59.265Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3922023-11-30T06:18:59.265Z+ success 'ZFS vdev /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev exists'
3932023-11-30T06:18:59.265Z+ set +x
3942023-11-30T06:18:59.265ZZFS vdev /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev exists
3952023-11-30T06:18:59.266Z++ zpool list -o name
3962023-11-30T06:18:59.266Z++ grep oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60
3972023-11-30T06:18:59.266Z+ [[ -z '' ]]
3982023-11-30T06:18:59.266Z+ zpool create -f oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev
3992023-11-30T06:18:59.290Z+ success 'ZFS zpool oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 exists'
4002023-11-30T06:18:59.291Z+ set +x
4012023-11-30T06:18:59.291ZZFS zpool oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 exists
4022023-11-30T06:18:59.291Z+ for ZPOOL in "${ZPOOLS[@]}"
4032023-11-30T06:18:59.291Z+ echo 'Zpool: [oxp_31bd71cd-4736-4a12-a387-9b74b050396f]'
4042023-11-30T06:18:59.291ZZpool: [oxp_31bd71cd-4736-4a12-a387-9b74b050396f]
4052023-11-30T06:18:59.291Z+ VDEV_PATH=/scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev
4062023-11-30T06:18:59.291Z+ [[ -f /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev ]]
4072023-11-30T06:18:59.291Z+ dd if=/dev/zero of=/scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev bs=1 count=0 seek=20G
4082023-11-30T06:18:59.291Z0+0 records in
4092023-11-30T06:18:59.291Z0+0 records out
4102023-11-30T06:18:59.291Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
4112023-11-30T06:18:59.291Z+ success 'ZFS vdev /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev exists'
4122023-11-30T06:18:59.291Z+ set +x
4132023-11-30T06:18:59.291ZZFS vdev /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev exists
4142023-11-30T06:18:59.315Z++ zpool list -o name
4152023-11-30T06:18:59.315Z++ grep oxp_31bd71cd-4736-4a12-a387-9b74b050396f
4162023-11-30T06:18:59.315Z+ [[ -z '' ]]
4172023-11-30T06:18:59.315Z+ zpool create -f oxp_31bd71cd-4736-4a12-a387-9b74b050396f /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev
4182023-11-30T06:18:59.340Z+ success 'ZFS zpool oxp_31bd71cd-4736-4a12-a387-9b74b050396f exists'
4192023-11-30T06:18:59.340Z+ set +x
4202023-11-30T06:18:59.340ZZFS zpool oxp_31bd71cd-4736-4a12-a387-9b74b050396f exists
4212023-11-30T06:18:59.340Z+ for ZPOOL in "${ZPOOLS[@]}"
4222023-11-30T06:18:59.340Z+ echo 'Zpool: [oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7]'
4232023-11-30T06:18:59.340ZZpool: [oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7]
4242023-11-30T06:18:59.340Z+ VDEV_PATH=/scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev
4252023-11-30T06:18:59.340Z+ [[ -f /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev ]]
4262023-11-30T06:18:59.340Z+ dd if=/dev/zero of=/scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev bs=1 count=0 seek=20G
4272023-11-30T06:18:59.340Z0+0 records in
4282023-11-30T06:18:59.340Z0+0 records out
4292023-11-30T06:18:59.340Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4302023-11-30T06:18:59.340Z+ success 'ZFS vdev /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev exists'
4312023-11-30T06:18:59.340Z+ set +x
4322023-11-30T06:18:59.340ZZFS vdev /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev exists
4332023-11-30T06:18:59.364Z++ zpool list -o name
4342023-11-30T06:18:59.364Z++ grep oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7
4352023-11-30T06:18:59.364Z+ [[ -z '' ]]
4362023-11-30T06:18:59.364Z+ zpool create -f oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev
4372023-11-30T06:18:59.388Z+ success 'ZFS zpool oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 exists'
4382023-11-30T06:18:59.388Z+ set +x
4392023-11-30T06:18:59.388ZZFS zpool oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 exists
4402023-11-30T06:18:59.388Z+ for ZPOOL_TYPE in "${ZPOOL_TYPES[@]}"
4412023-11-30T06:18:59.388Z+ readarray -t ZPOOLS
4422023-11-30T06:18:59.388Z++ grep '"oxi_' /opt/oxide/work/tools/../smf/sled-agent/non-gimlet/config.toml
4432023-11-30T06:18:59.388Z++ sed 's/[ ",]//g'
4442023-11-30T06:18:59.388Z+ for ZPOOL in "${ZPOOLS[@]}"
4452023-11-30T06:18:59.388Z+ echo 'Zpool: [oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b]'
4462023-11-30T06:18:59.388ZZpool: [oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b]
4472023-11-30T06:18:59.388Z+ VDEV_PATH=/scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4482023-11-30T06:18:59.388Z+ [[ -f /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
4492023-11-30T06:18:59.388Z+ dd if=/dev/zero of=/scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
4502023-11-30T06:18:59.413Z0+0 records in
4512023-11-30T06:18:59.413Z0+0 records out
4522023-11-30T06:18:59.413Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
4532023-11-30T06:18:59.413Z+ success 'ZFS vdev /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
4542023-11-30T06:18:59.413Z+ set +x
4552023-11-30T06:18:59.413ZZFS vdev /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
4562023-11-30T06:18:59.413Z++ zpool list -o name
4572023-11-30T06:18:59.413Z++ grep oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b
4582023-11-30T06:18:59.413Z+ [[ -z '' ]]
4592023-11-30T06:18:59.413Z+ zpool create -f oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4602023-11-30T06:18:59.437Z+ success 'ZFS zpool oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
4612023-11-30T06:18:59.437Z+ set +x
4622023-11-30T06:18:59.437ZZFS zpool oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b exists
4632023-11-30T06:18:59.438Z+ for ZPOOL in "${ZPOOLS[@]}"
4642023-11-30T06:18:59.438Z+ echo 'Zpool: [oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b]'
4652023-11-30T06:18:59.438Z+ VDEV_PATH=/scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4662023-11-30T06:18:59.438Z+ [[ -f /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
4672023-11-30T06:18:59.438ZZpool: [oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b]
4682023-11-30T06:18:59.438Z+ dd if=/dev/zero of=/scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
4692023-11-30T06:18:59.438Z0+0 records in
4702023-11-30T06:18:59.438Z0+0 records out
4712023-11-30T06:18:59.438Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
4722023-11-30T06:18:59.438Z+ success 'ZFS vdev /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
4732023-11-30T06:18:59.438Z+ set +x
4742023-11-30T06:18:59.438ZZFS vdev /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
4752023-11-30T06:18:59.438Z++ zpool list -o name
4762023-11-30T06:18:59.438Z++ grep oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b
4772023-11-30T06:18:59.462Z+ [[ -z '' ]]
4782023-11-30T06:18:59.463Z+ zpool create -f oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4792023-11-30T06:18:59.487Z+ success 'ZFS zpool oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
4802023-11-30T06:18:59.487Z+ set +x
4812023-11-30T06:18:59.487ZZFS zpool oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b exists
4822023-11-30T06:18:59.487Z+ [[ zone == \z\o\n\e ]]
4832023-11-30T06:18:59.489Z+ ensure_simulated_links igb0
4842023-11-30T06:18:59.489Z+ local PHYSICAL_LINK=igb0
4852023-11-30T06:18:59.489Z+ INDICES=("0" "1")
4862023-11-30T06:18:59.489Z+ for I in "${INDICES[@]}"
4872023-11-30T06:18:59.489Z++ get_simnet_name_if_exists net0
4882023-11-30T06:18:59.489Z++ dladm show-simnet -p -o LINK net0
4892023-11-30T06:18:59.489Z++ echo ''
4902023-11-30T06:18:59.489Z+ [[ -z '' ]]
4912023-11-30T06:18:59.489Z+ dladm create-simnet -t net0
4922023-11-30T06:18:59.513Z+ dladm create-simnet -t sc0_0
4932023-11-30T06:18:59.513Z+ dladm modify-simnet -t -p net0 sc0_0
4942023-11-30T06:18:59.513Z+ dladm set-linkprop -p mtu=9000 sc0_0
4952023-11-30T06:18:59.513Z+ success 'Simnet net0/sc0_0 exists'
4962023-11-30T06:18:59.513Z+ set +x
4972023-11-30T06:18:59.513ZSimnet net0/sc0_0 exists
4982023-11-30T06:18:59.513Z+ for I in "${INDICES[@]}"
4992023-11-30T06:18:59.513Z++ get_simnet_name_if_exists net1
5002023-11-30T06:18:59.515Z++ dladm show-simnet -p -o LINK net1
5012023-11-30T06:18:59.515Z++ echo ''
5022023-11-30T06:18:59.515Z+ [[ -z '' ]]
5032023-11-30T06:18:59.515Z+ dladm create-simnet -t net1
5042023-11-30T06:18:59.515Z+ dladm create-simnet -t sc1_0
5052023-11-30T06:18:59.539Z+ dladm modify-simnet -t -p net1 sc1_0
5062023-11-30T06:18:59.539Z+ dladm set-linkprop -p mtu=9000 sc1_0
5072023-11-30T06:18:59.539Z+ success 'Simnet net1/sc1_0 exists'
5082023-11-30T06:18:59.539Z+ set +x
5092023-11-30T06:18:59.539ZSimnet net1/sc1_0 exists
5102023-11-30T06:18:59.539Z++ get_vnic_name_if_exists sc0_1
5112023-11-30T06:18:59.539Z++ dladm show-vnic -p -o LINK sc0_1
5122023-11-30T06:18:59.539Z++ echo ''
5132023-11-30T06:18:59.539Z+ [[ -z '' ]]
5142023-11-30T06:18:59.539Z+ dladm create-vnic -t sc0_1 -l igb0 -m a8:e1:de:01:70:1d
5152023-11-30T06:18:59.539Z+ [[ -v PROMISC_FILT_OFF ]]
5162023-11-30T06:18:59.539Z+ success 'Vnic sc0_1 exists'
5172023-11-30T06:18:59.539Z+ set +x
5182023-11-30T06:18:59.539ZVnic sc0_1 exists
5192023-11-30T06:18:59.539Z+ warn_if_no_proxy_arp
5202023-11-30T06:18:59.539Z+ [[ -v PXA_START ]]
5212023-11-30T06:18:59.539Z+ [[ -v PXA_END ]]
5222023-11-30T06:18:59.539Z+ ensure_softnpu_zone
5232023-11-30T06:18:59.539Z+ zoneadm list
5242023-11-30T06:18:59.539Z+ grep -q sidecar_softnpu
5252023-11-30T06:18:59.563Z+ [[ -f out/npuzone/npuzone ]]
5262023-11-30T06:18:59.563Z+ out/npuzone/npuzone create sidecar --omicron-zone --ports sc0_0,tfportrear0_0 --ports sc0_1,tfportqsfp0_0 --sidecar-lite-commit f0585a29fb0285f7a1220c1118856b0e5c1f75c5 --softnpu-commit dec63e67156fe6e958991bbfa090629868115ab5
5272023-11-30T06:18:59.563Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/softnpu/image/dec63e67156fe6e958991bbfa090629868115ab5/softnpu.sha256.txt
5282023-11-30T06:19:04.987Zdownloading softnpu ...done
5292023-11-30T06:19:05.012Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/sidecar-lite/release/f0585a29fb0285f7a1220c1118856b0e5c1f75c5/libsidecar_lite.so.sha256.txt
5302023-11-30T06:19:10.432Zdownloading asic_program.so ...done
5312023-11-30T06:19:10.456Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/sidecar-lite/release/f0585a29fb0285f7a1220c1118856b0e5c1f75c5/scadm.sha256.txt
5322023-11-30T06:19:15.833Zdownloading scadm ...done
5332023-11-30T06:19:15.857Zcreating port tfportrear0_0 for sc0_0
5342023-11-30T06:19:15.857Zcreating port tfportqsfp0_0 for sc0_1
5352023-11-30T06:19:15.857Zsoftnpu zone setup
5362023-11-30T06:19:15.858Zconfigure zone
5372023-11-30T06:19:15.883Zinstall zone
5382023-11-30T06:19:16.894Zboot zone
5392023-11-30T06:19:20.001Z[sidecar_softnpu] svcs milestone/network
5402023-11-30T06:19:20.028ZSTATE STIME FMRI
5412023-11-30T06:19:20.028Zonline 6:19:18 svc:/milestone/network:default
5422023-11-30T06:19:20.028Z[sidecar_softnpu] /softnpu/npu start
5432023-11-30T06:19:20.055Z
5442023-11-30T06:19:20.079Z+ /opt/oxide/work/tools/scrimlet/softnpu-init.sh
5452023-11-30T06:19:20.079Z+ GATEWAY_IP=10.151.1.100
5462023-11-30T06:19:20.079Z+ echo 'Using 10.151.1.100 as gateway ip'
5472023-11-30T06:19:20.079ZUsing 10.151.1.100 as gateway ip
5482023-11-30T06:19:20.079Z+ [[ ! -v GATEWAY_MAC ]]
5492023-11-30T06:19:20.079Z+ GATEWAY_MAC=18:c0:4d:0d:9f:b2
5502023-11-30T06:19:20.079Z++ grep -c -E '^([0-9a-fA-F]{2}:){5}[0-9a-fA-F]{2}$' /dev/fd/63
5512023-11-30T06:19:20.079Z+++ echo 18:c0:4d:0d:9f:b2
5522023-11-30T06:19:20.079Z+ COUNT=1
5532023-11-30T06:19:20.079Z+ [[ 1 -ne 1 ]]
5542023-11-30T06:19:20.079Z+ echo 'Using 18:c0:4d:0d:9f:b2 as gateway mac'
5552023-11-30T06:19:20.079ZUsing 18:c0:4d:0d:9f:b2 as gateway mac
5562023-11-30T06:19:20.079Z+ z_scadm add-arp-entry 10.151.1.100 18:c0:4d:0d:9f:b2
5572023-11-30T06:19:20.079Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone add-arp-entry 10.151.1.100 18:c0:4d:0d:9f:b2
5582023-11-30T06:19:20.104Z+ PXA_MAC=a8:e1:de:01:70:1d
5592023-11-30T06:19:20.104Z+ [[ -v PXA_START ]]
5602023-11-30T06:19:20.104Z+ z_scadm add-proxy-arp 10.151.1.164 10.151.1.227 a8:e1:de:01:70:1d
5612023-11-30T06:19:20.104Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone add-proxy-arp 10.151.1.164 10.151.1.227 a8:e1:de:01:70:1d
5622023-11-30T06:19:20.128Z+ z_scadm dump-state
5632023-11-30T06:19:20.129Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone dump-state
5642023-11-30T06:19:20.154Zlocal v6:
5652023-11-30T06:19:20.154Zlocal v4:
5662023-11-30T06:19:20.154Zrouter v6:
5672023-11-30T06:19:20.154Zrouter v4_idx:
5682023-11-30T06:19:20.154Zrouter v4_routes:
5692023-11-30T06:19:20.154Zresolver v4:
5702023-11-30T06:19:20.154Z10.151.1.100 -> 18:c0:4d:0d:9f:b2
5712023-11-30T06:19:20.154Zresolver v6:
5722023-11-30T06:19:20.154Znat v4:
5732023-11-30T06:19:20.154Znat_v6:
5742023-11-30T06:19:20.154Zport_mac:
5752023-11-30T06:19:20.154Zproxy arp:
5762023-11-30T06:19:20.156Z10.151.1.164/10.151.1.227: a8:e1:de:01:70:1d
5772023-11-30T06:19:20.156Z+ success 'softnpu zone exists'
5782023-11-30T06:19:20.156Z+ set +x
5792023-11-30T06:19:20.156Zsoftnpu zone exists
5802023-11-30T06:19:20.180Z
5812023-11-30T06:19:20.180Zreal 21.297166816
5822023-11-30T06:19:20.180Zuser 0.711118080
5832023-11-30T06:19:20.180Zsys 1.570324722
5842023-11-30T06:19:20.180Ztrap 0.000079563
5852023-11-30T06:19:20.180Ztflt 0.000100978
5862023-11-30T06:19:20.180Zdflt 0.000719628
5872023-11-30T06:19:20.180Zkflt 0.000000000
5882023-11-30T06:19:20.180Zlock 16:22.325706765
5892023-11-30T06:19:20.180Zslp 1:03.614631021
5902023-11-30T06:19:20.180Zlat 0.058620137
5912023-11-30T06:19:20.180Zstop 0.355137705
5922023-11-30T06:19:20.180Z+ tar xf out/omicron-sled-agent.tar pkg/config-rss.toml
5932023-11-30T06:19:20.206Z++ sed -n 's/silo_name = "\(.*\)"/\1/p' pkg/config-rss.toml
5942023-11-30T06:19:20.206Z+ SILO_NAME=recovery
5952023-11-30T06:19:20.206Z++ sed -n 's/external_dns_zone_name = "\(.*\)"/\1/p' pkg/config-rss.toml
5962023-11-30T06:19:20.206Z+ EXTERNAL_DNS_DOMAIN=oxide.test
5972023-11-30T06:19:20.206Z+ sed -i~ '
5982023-11-30T06:19:20.206Z /^external_dns_ips/c\
5992023-11-30T06:19:20.206Zexternal_dns_ips = [ "10.151.1.164", "10.151.1.165" ]
6002023-11-30T06:19:20.206Z /^\[\[internal_services_ip_pool_ranges/,/^$/ {
6012023-11-30T06:19:20.206Z /^first/c\
6022023-11-30T06:19:20.206Zfirst = "10.151.1.164"
6032023-11-30T06:19:20.206Z /^last/c\
6042023-11-30T06:19:20.206Zlast = "10.151.1.173"
6052023-11-30T06:19:20.206Z }
6062023-11-30T06:19:20.206Z /^\[rack_network_config/,/^$/ {
6072023-11-30T06:19:20.206Z /^infra_ip_first/c\
6082023-11-30T06:19:20.206Zinfra_ip_first = "10.151.1.174"
6092023-11-30T06:19:20.206Z /^infra_ip_last/c\
6102023-11-30T06:19:20.206Zinfra_ip_last = "10.151.1.174"
6112023-11-30T06:19:20.206Z }
6122023-11-30T06:19:20.206Z /^\[\[rack_network_config.ports/,/^$/ {
6132023-11-30T06:19:20.206Z /^routes/c\
6142023-11-30T06:19:20.206Zroutes = \[{nexthop = "10.151.1.100", destination = "0.0.0.0/0"}\]
6152023-11-30T06:19:20.206Z /^addresses/c\
6162023-11-30T06:19:20.206Zaddresses = \["10.151.1.174/32"\]
6172023-11-30T06:19:20.206Z }
6182023-11-30T06:19:20.206Z' pkg/config-rss.toml
6192023-11-30T06:19:20.206Z+ diff -u pkg/config-rss.toml~ pkg/config-rss.toml
6202023-11-30T06:19:20.206Z--- pkg/config-rss.toml~ Mon Jul 24 01:21:28 2006
6212023-11-30T06:19:20.208Z+++ pkg/config-rss.toml Thu Nov 30 06:19:20 2023
6222023-11-30T06:19:20.208Z@@ -41,7 +41,7 @@
6232023-11-30T06:19:20.208Z # the DNS domain delegated to the rack by the customer. Each of these addresses
6242023-11-30T06:19:20.208Z # must be contained in one of the "internal services" IP Pool ranges listed
6252023-11-30T06:19:20.208Z # below.
6262023-11-30T06:19:20.208Z-external_dns_ips = [ "192.168.1.20", "192.168.1.21" ]
6272023-11-30T06:19:20.208Z+external_dns_ips = [ "10.151.1.164", "10.151.1.165" ]
6282023-11-30T06:19:20.208Z
6292023-11-30T06:19:20.208Z # Initial TLS certificates for the external API
6302023-11-30T06:19:20.208Z #
6312023-11-30T06:19:20.208Z@@ -77,8 +77,8 @@
6322023-11-30T06:19:20.208Z #
6332023-11-30T06:19:20.208Z # For more on this and what to put here, see docs/how-to-run.adoc.
6342023-11-30T06:19:20.208Z [[internal_services_ip_pool_ranges]]
6352023-11-30T06:19:20.208Z-first = "192.168.1.20"
6362023-11-30T06:19:20.209Z-last = "192.168.1.29"
6372023-11-30T06:19:20.209Z+first = "10.151.1.164"
6382023-11-30T06:19:20.209Z+last = "10.151.1.173"
6392023-11-30T06:19:20.209Z
6402023-11-30T06:19:20.209Z # TODO - this configuration is subject to change going forward. Ultimately these
6412023-11-30T06:19:20.209Z # parameters should be provided to the control plane via wicket, but we need to
6422023-11-30T06:19:20.209Z@@ -92,8 +92,8 @@
6432023-11-30T06:19:20.209Z # A range of IP addresses used by Boundary Services on the external network. In
6442023-11-30T06:19:20.209Z # a real system, these would be addresses of the uplink ports on the Sidecar.
6452023-11-30T06:19:20.209Z # With softnpu, only one address is used.
6462023-11-30T06:19:20.209Z-infra_ip_first = "192.168.1.30"
6472023-11-30T06:19:20.209Z-infra_ip_last = "192.168.1.30"
6482023-11-30T06:19:20.209Z+infra_ip_first = "10.151.1.174"
6492023-11-30T06:19:20.209Z+infra_ip_last = "10.151.1.174"
6502023-11-30T06:19:20.209Z
6512023-11-30T06:19:20.209Z # Configurations for BGP routers to run on the scrimlets.
6522023-11-30T06:19:20.209Z bgp = []
6532023-11-30T06:19:20.209Z@@ -101,9 +101,9 @@
6542023-11-30T06:19:20.209Z # You can configure multiple uplinks by repeating the following stanza
6552023-11-30T06:19:20.209Z [[rack_network_config.ports]]
6562023-11-30T06:19:20.209Z # Routes associated with this port.
6572023-11-30T06:19:20.209Z-routes = [{nexthop = "192.168.1.199", destination = "0.0.0.0/0"}]
6582023-11-30T06:19:20.209Z+routes = [{nexthop = "10.151.1.100", destination = "0.0.0.0/0"}]
6592023-11-30T06:19:20.209Z # Addresses associated with this port.
6602023-11-30T06:19:20.209Z-addresses = ["192.168.1.30/32"]
6612023-11-30T06:19:20.209Z+addresses = ["10.151.1.174/32"]
6622023-11-30T06:19:20.209Z # Name of the uplink port. This should always be "qsfp0" when using softnpu.
6632023-11-30T06:19:20.209Z port = "qsfp0"
6642023-11-30T06:19:20.209Z # The speed of this port.
6652023-11-30T06:19:20.209Z+ true
6662023-11-30T06:19:20.209Z+ tar rvf out/omicron-sled-agent.tar pkg/config-rss.toml
6672023-11-30T06:19:20.233Za pkg/config-rss.toml 7K
6682023-11-30T06:19:20.233Z+ rm -f pkg/config-rss.toml pkg/config-rss.toml~
6692023-11-30T06:19:20.234Z+ TLS_NAME=recovery.sys.oxide.test
6702023-11-30T06:19:20.234Z+ openssl req -newkey rsa:4096 -x509 -sha256 -days 3 -nodes -out pkg/initial-tls-cert.pem -keyout pkg/initial-tls-key.pem -subj /CN=recovery.sys.oxide.test -addext subjectAltName=DNS:recovery.sys.oxide.test -addext basicConstraints=critical,CA:FALSE -config /dev/stdin
6712023-11-30T06:19:20.726Z.+..+....+.....+.+...+....................+......+................+...+..+.......+...+...+......+...+..+..........+......+..+......+.+...+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*...+....+..+.......+..+.........+....+....................+............+......+...+....+.....+.............+.........+.....+.+.....+.........+.+.....+....+...+..+.+.....+.......+..+....+..+....+...+..+...+.+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.+....+...+...........+.........+......+....+..+.............+.................+.+..............+..........+...+...+........+.......+..................+.........+............+.....+.........+............+..................+.+........+...+...+....+...........+.......+...+.........+...+.........+.........+........................+..+.........+.+...+......+...........+.+.................+..........+...........+.+.........+..+..........+.........+...........+..........+.........+....................+.+.........+...+.....+......+.+..................+..+...+.+..+....+...+....................+.......+...+..+....+............+.................+...+...+......+.+............+.....+.......+........+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
6722023-11-30T06:19:20.995Z..+.+........+....+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*....+.........+..+...+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.+.................+...+.......+..+.+..+...................+.........+.........+.....+.......+...+......+.....+......+.............+..+...+.............+..+...+...+.......+...+......+..............+............+...+..........+.........+............+...+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
6732023-11-30T06:19:21.019Z-----
6742023-11-30T06:19:21.043Z+ tar rvf out/omicron-sled-agent.tar pkg/initial-tls-cert.pem pkg/initial-tls-key.pem
6752023-11-30T06:19:21.043Za pkg/initial-tls-cert.pem 2K
6762023-11-30T06:19:21.043Za pkg/initial-tls-key.pem 4K
6772023-11-30T06:19:21.043Z+ rm -f pkg/initial-tls-cert.pem pkg/initial-tls-key.pem
6782023-11-30T06:19:21.043Z+ rmdir pkg
6792023-11-30T06:19:21.043Z+ E2E_TLS_CERT=/opt/oxide/sled-agent/pkg/initial-tls-cert.pem
6802023-11-30T06:19:21.043Z+ pfexec mkdir -p /usr/oxide
6812023-11-30T06:19:21.043Z+ pfexec rm -f /usr/oxide/tcpproxy
6822023-11-30T06:19:21.043Z+ pfexec curl -sSfL -o /usr/oxide/tcpproxy http://catacomb.eng.oxide.computer:12346/tcpproxy
6832023-11-30T06:19:21.256Z+ pfexec chmod +x /usr/oxide/tcpproxy
6842023-11-30T06:19:21.280Z+ pfexec rm -f /var/svc/manifest/site/tcpproxy.xml
6852023-11-30T06:19:21.280Z+ pfexec curl -sSfL -o /var/svc/manifest/site/tcpproxy.xml http://catacomb.eng.oxide.computer:12346/tcpproxy.xml
6862023-11-30T06:19:21.280Z+ pfexec svccfg import /var/svc/manifest/site/tcpproxy.xml
6872023-11-30T06:19:21.380Z+ OMICRON_NO_UNINSTALL=1
6882023-11-30T06:19:21.380Z+ ptime -m pfexec ./target/release/omicron-package -t test install
6892023-11-30T06:19:21.405ZNov 30 06:19:21.620 DEBG target[test]: Target({"image": "standard", "machine": "non-gimlet", "rack-topology": "single-sled", "switch": "softnpu"})
6902023-11-30T06:19:21.405ZNov 30 06:19:21.623 INFO Installing service, dst: /opt/oxide/clickhouse.tar.gz, src: out/clickhouse.tar.gz
6912023-11-30T06:19:21.405ZNov 30 06:19:21.623 INFO Installing service, dst: /opt/oxide/ntp.tar.gz, src: out/ntp.tar.gz
6922023-11-30T06:19:21.405ZNov 30 06:19:21.623 INFO Installing service, dst: /opt/oxide/clickhouse_keeper.tar.gz, src: out/clickhouse_keeper.tar.gz
6932023-11-30T06:19:21.405ZNov 30 06:19:21.623 INFO Installing service, dst: /opt/oxide/crucible_pantry.tar.gz, src: out/crucible-pantry.tar.gz
6942023-11-30T06:19:21.405ZNov 30 06:19:21.623 INFO Installing service, dst: /opt/oxide/nexus.tar.gz, src: out/omicron-nexus.tar.gz
6952023-11-30T06:19:21.405ZNov 30 06:19:21.623 INFO Installing service, dst: /opt/oxide/internal_dns.tar.gz, src: out/internal-dns.tar.gz
6962023-11-30T06:19:21.405ZNov 30 06:19:21.623 INFO Installing service, dst: /opt/oxide/sled-agent.tar, src: out/omicron-sled-agent.tar
6972023-11-30T06:19:21.405ZNov 30 06:19:21.623 INFO Installing service, dst: /opt/oxide/mg-ddm.tar, src: out/mg-ddm-gz.tar
6982023-11-30T06:19:21.405ZNov 30 06:19:21.623 INFO Installing service, dst: /opt/oxide/external_dns.tar.gz, src: out/external-dns.tar.gz
6992023-11-30T06:19:21.405ZNov 30 06:19:21.623 INFO Installing service, dst: /opt/oxide/cockroachdb.tar.gz, src: out/cockroachdb.tar.gz
7002023-11-30T06:19:21.405ZNov 30 06:19:21.623 INFO Installing service, dst: /opt/oxide/crucible.tar.gz, src: out/crucible.tar.gz
7012023-11-30T06:19:21.405ZNov 30 06:19:21.623 INFO Installing service, dst: /opt/oxide/propolis-server.tar.gz, src: out/propolis-server.tar.gz
7022023-11-30T06:19:21.405ZNov 30 06:19:21.623 INFO Installing service, dst: /opt/oxide/overlay.tar.gz, src: out/overlay.tar.gz
7032023-11-30T06:19:21.405ZNov 30 06:19:21.623 INFO Installing service, dst: /opt/oxide/oximeter.tar.gz, src: out/oximeter-collector.tar.gz
7042023-11-30T06:19:21.405ZNov 30 06:19:21.623 INFO Installing service, dst: /opt/oxide/switch.tar.gz, src: out/switch-softnpu.tar.gz
7052023-11-30T06:19:21.405ZNov 30 06:19:21.623 INFO Installing service, dst: /opt/oxide/mgs.tar.gz, src: out/omicron-gateway-softnpu.tar.gz
7062023-11-30T06:19:21.636ZNov 30 06:19:21.861 INFO Unpacking service tarball, service_path: /opt/oxide/mg-ddm, tar_path: /opt/oxide/mg-ddm.tar
7072023-11-30T06:19:21.672ZNov 30 06:19:21.898 INFO Unpacking service tarball, service_path: /opt/oxide/sled-agent, tar_path: /opt/oxide/sled-agent.tar
7082023-11-30T06:19:21.731ZNov 30 06:19:21.956 INFO Installing bootstrap service from /opt/oxide/sled-agent/pkg/manifest.xml
7092023-11-30T06:19:21.860Z
7102023-11-30T06:19:21.885Zreal 0.475359996
7112023-11-30T06:19:21.886Zuser 0.098750168
7122023-11-30T06:19:21.886Zsys 1.932520701
7132023-11-30T06:19:21.886Ztrap 0.000022266
7142023-11-30T06:19:21.886Ztflt 0.000017809
7152023-11-30T06:19:21.886Zdflt 0.000854350
7162023-11-30T06:19:21.886Zkflt 0.000000000
7172023-11-30T06:19:21.886Zlock 42.606723507
7182023-11-30T06:19:21.886Zslp 0.679903400
7192023-11-30T06:19:21.886Zlat 0.002781916
7202023-11-30T06:19:21.886Zstop 0.205928690
7212023-11-30T06:19:21.886Z+ retry=0
7222023-11-30T06:19:21.886Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7232023-11-30T06:19:21.886Z+ [[ 0 -gt 30 ]]
7242023-11-30T06:19:21.886Z+ sleep 1
7252023-11-30T06:19:22.875Z+ retry=1
7262023-11-30T06:19:22.875Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7272023-11-30T06:19:22.900Z+ [[ 1 -gt 30 ]]
7282023-11-30T06:19:22.900Z+ sleep 1
7292023-11-30T06:19:23.890Z+ retry=2
7302023-11-30T06:19:23.890Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7312023-11-30T06:19:23.914Z+ [[ 2 -gt 30 ]]
7322023-11-30T06:19:23.914Z+ sleep 1
7332023-11-30T06:19:24.905Z+ retry=3
7342023-11-30T06:19:24.905Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7352023-11-30T06:19:24.929Z+ [[ 3 -gt 30 ]]
7362023-11-30T06:19:24.929Z+ sleep 1
7372023-11-30T06:19:25.919Z+ retry=4
7382023-11-30T06:19:25.919Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7392023-11-30T06:19:45.658Z+ [[ 4 -gt 30 ]]
7402023-11-30T06:19:45.658Z+ sleep 1
7412023-11-30T06:19:46.661Z+ retry=5
7422023-11-30T06:19:46.661Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7432023-11-30T06:19:46.685Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone dump-state
7442023-11-30T06:19:46.710Zlocal v6:
7452023-11-30T06:19:46.710Zfe80::34a1:d5ff:fe81:4e9e
7462023-11-30T06:19:46.710Zlocal v4:
7472023-11-30T06:19:46.710Zrouter v6:
7482023-11-30T06:19:46.710Zrouter v4_idx:
7492023-11-30T06:19:46.710Zrouter v4_routes:
7502023-11-30T06:19:46.710Zresolver v4:
7512023-11-30T06:19:46.710Z10.151.1.100 -> 18:c0:4d:0d:9f:b2
7522023-11-30T06:19:46.710Zresolver v6:
7532023-11-30T06:19:46.710Zfe80::94ff:40ff:fe1c:2538 -> 96:ff:40:1c:25:38
7542023-11-30T06:19:46.712Zfe80::34a1:d5ff:fe81:4e9e -> 36:a1:d5:81:4e:9e
7552023-11-30T06:19:46.712Znat v4:
7562023-11-30T06:19:46.712Znat_v6:
7572023-11-30T06:19:46.712Zport_mac:
7582023-11-30T06:19:46.712Z1: a8:40:25:3b:22:06
7592023-11-30T06:19:46.712Z1000: a8:40:25:3b:22:05
7602023-11-30T06:19:46.712Zproxy arp:
7612023-11-30T06:19:46.712Z10.151.1.164/10.151.1.227: a8:e1:de:01:70:1d
7622023-11-30T06:19:46.712Z+ retry=0
7632023-11-30T06:19:46.712Z+++ zoneadm list -n
7642023-11-30T06:19:46.712Z+++ grep oxz_ntp
7652023-11-30T06:19:46.736Z++ pfexec svcs -z -Hostate oxide/ntp
7662023-11-30T06:19:46.736Zsvcs: zone '-Hostate': entity not found
7672023-11-30T06:19:46.736Z++ true
7682023-11-30T06:19:46.736Z+ [[ '' != online ]]
7692023-11-30T06:19:46.736Z+ [[ 0 -gt 60 ]]
7702023-11-30T06:19:46.736Z+ sleep 1
7712023-11-30T06:19:47.723Z+ retry=1
7722023-11-30T06:19:47.747Z+++ zoneadm list -n
7732023-11-30T06:19:47.747Z+++ grep oxz_ntp
7742023-11-30T06:19:47.747Z++ pfexec svcs -z -Hostate oxide/ntp
7752023-11-30T06:19:47.747Zsvcs: zone '-Hostate': entity not found
7762023-11-30T06:19:47.747Z++ true
7772023-11-30T06:19:47.747Z+ [[ '' != online ]]
7782023-11-30T06:19:47.747Z+ [[ 1 -gt 60 ]]
7792023-11-30T06:19:47.747Z+ sleep 1
7802023-11-30T06:19:48.745Z+ retry=2
7812023-11-30T06:19:48.770Z+++ zoneadm list -n
7822023-11-30T06:19:48.770Z+++ grep oxz_ntp
7832023-11-30T06:19:48.770Z++ pfexec svcs -z oxz_ntp_c74a342e-615a-400e-93e7-d0436053ed97 -Hostate oxide/ntp
7842023-11-30T06:19:48.814Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
7852023-11-30T06:19:48.838Z++ true
7862023-11-30T06:19:48.838Z+ [[ '' != online ]]
7872023-11-30T06:19:48.838Z+ [[ 2 -gt 60 ]]
7882023-11-30T06:19:48.838Z+ sleep 1
7892023-11-30T06:19:49.819Z+ retry=3
7902023-11-30T06:19:49.843Z+++ zoneadm list -n
7912023-11-30T06:19:49.843Z+++ grep oxz_ntp
7922023-11-30T06:19:49.843Z++ pfexec svcs -z oxz_ntp_c74a342e-615a-400e-93e7-d0436053ed97 -Hostate oxide/ntp
7932023-11-30T06:19:49.867Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
7942023-11-30T06:19:49.867Z++ true
7952023-11-30T06:19:49.867Z+ [[ '' != online ]]
7962023-11-30T06:19:49.867Z+ [[ 3 -gt 60 ]]
7972023-11-30T06:19:49.867Z+ sleep 1
7982023-11-30T06:19:50.850Z+ retry=4
7992023-11-30T06:19:50.874Z+++ zoneadm list -n
8002023-11-30T06:19:50.874Z+++ grep oxz_ntp
8012023-11-30T06:19:50.874Z++ pfexec svcs -z oxz_ntp_c74a342e-615a-400e-93e7-d0436053ed97 -Hostate oxide/ntp
8022023-11-30T06:19:50.898Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
8032023-11-30T06:19:50.898Z++ true
8042023-11-30T06:19:50.898Z+ [[ '' != online ]]
8052023-11-30T06:19:50.898Z+ [[ 4 -gt 60 ]]
8062023-11-30T06:19:50.898Z+ sleep 1
8072023-11-30T06:19:51.880Z+ retry=5
8082023-11-30T06:19:51.904Z+++ zoneadm list -n
8092023-11-30T06:19:51.904Z+++ grep oxz_ntp
8102023-11-30T06:19:51.904Z++ pfexec svcs -z oxz_ntp_c74a342e-615a-400e-93e7-d0436053ed97 -Hostate oxide/ntp
8112023-11-30T06:19:51.929Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
8122023-11-30T06:19:51.929Z++ true
8132023-11-30T06:19:51.929Z+ [[ '' != online ]]
8142023-11-30T06:19:51.929Z+ [[ 5 -gt 60 ]]
8152023-11-30T06:19:51.929Z+ sleep 1
8162023-11-30T06:19:52.910Z+ retry=6
8172023-11-30T06:19:52.934Z+++ zoneadm list -n
8182023-11-30T06:19:52.934Z+++ grep oxz_ntp
8192023-11-30T06:19:52.934Z++ pfexec svcs -z oxz_ntp_c74a342e-615a-400e-93e7-d0436053ed97 -Hostate oxide/ntp
8202023-11-30T06:19:52.934Z+ [[ disabled != online ]]
8212023-11-30T06:19:52.934Z+ [[ 6 -gt 60 ]]
8222023-11-30T06:19:52.934Z+ sleep 1
8232023-11-30T06:19:53.932Z+ retry=7
8242023-11-30T06:19:53.957Z+++ zoneadm list -n
8252023-11-30T06:19:53.957Z+++ grep oxz_ntp
8262023-11-30T06:19:53.957Z++ pfexec svcs -z oxz_ntp_c74a342e-615a-400e-93e7-d0436053ed97 -Hostate oxide/ntp
8272023-11-30T06:19:53.957Z+ [[ disabled != online ]]
8282023-11-30T06:19:53.957Z+ [[ 7 -gt 60 ]]
8292023-11-30T06:19:53.957Z+ sleep 1
8302023-11-30T06:19:54.954Z+ retry=8
8312023-11-30T06:19:54.980Z+++ zoneadm list -n
8322023-11-30T06:19:54.980Z+++ grep oxz_ntp
8332023-11-30T06:19:54.980Z++ pfexec svcs -z oxz_ntp_c74a342e-615a-400e-93e7-d0436053ed97 -Hostate oxide/ntp
8342023-11-30T06:19:54.980Z+ [[ disabled != online ]]
8352023-11-30T06:19:54.980Z+ [[ 8 -gt 60 ]]
8362023-11-30T06:19:54.980Z+ sleep 1
8372023-11-30T06:19:55.976Z+ retry=9
8382023-11-30T06:19:56.000Z+++ zoneadm list -n
8392023-11-30T06:19:56.001Z+++ grep oxz_ntp
8402023-11-30T06:19:56.001Z++ pfexec svcs -z oxz_ntp_c74a342e-615a-400e-93e7-d0436053ed97 -Hostate oxide/ntp
8412023-11-30T06:19:56.001Z+ [[ disabled != online ]]
8422023-11-30T06:19:56.001Z+ [[ 9 -gt 60 ]]
8432023-11-30T06:19:56.001Z+ sleep 1
8442023-11-30T06:19:56.998Z+ retry=10
8452023-11-30T06:19:57.023Z+++ zoneadm list -n
8462023-11-30T06:19:57.023Z+++ grep oxz_ntp
8472023-11-30T06:19:57.023Z++ pfexec svcs -z oxz_ntp_c74a342e-615a-400e-93e7-d0436053ed97 -Hostate oxide/ntp
8482023-11-30T06:19:57.023Z+ [[ disabled != online ]]
8492023-11-30T06:19:57.023Z+ [[ 10 -gt 60 ]]
8502023-11-30T06:19:57.023Z+ sleep 1
8512023-11-30T06:19:58.021Z+ retry=11
8522023-11-30T06:19:58.045Z+++ zoneadm list -n
8532023-11-30T06:19:58.045Z+++ grep oxz_ntp
8542023-11-30T06:19:58.045Z++ pfexec svcs -z oxz_ntp_c74a342e-615a-400e-93e7-d0436053ed97 -Hostate oxide/ntp
8552023-11-30T06:19:58.045Z+ [[ disabled != online ]]
8562023-11-30T06:19:58.045Z+ [[ 11 -gt 60 ]]
8572023-11-30T06:19:58.045Z+ sleep 1
8582023-11-30T06:19:59.043Z+ retry=12
8592023-11-30T06:19:59.067Z+++ zoneadm list -n
8602023-11-30T06:19:59.067Z+++ grep oxz_ntp
8612023-11-30T06:19:59.067Z++ pfexec svcs -z oxz_ntp_c74a342e-615a-400e-93e7-d0436053ed97 -Hostate oxide/ntp
8622023-11-30T06:19:59.067Z+ [[ disabled != online ]]
8632023-11-30T06:19:59.067Z+ [[ 12 -gt 60 ]]
8642023-11-30T06:19:59.067Z+ sleep 1
 
8652023-11-30T06:19:59.319Zbuskin console login: Nov 30 06:19:59 buskin ip: ip: joining multicasts failed (4) on vopte0 - will use link layer broadcasts for multicast
 
8662023-11-30T06:20:00.065Z+ retry=13
8672023-11-30T06:20:00.089Z+++ zoneadm list -n
8682023-11-30T06:20:00.089Z+++ grep oxz_ntp
8692023-11-30T06:20:00.089Z++ pfexec svcs -z oxz_ntp_c74a342e-615a-400e-93e7-d0436053ed97 -Hostate oxide/ntp
8702023-11-30T06:20:00.089Z+ [[ online != online ]]
8712023-11-30T06:20:00.089Z+ echo 'Waited for chrony: 13s'
8722023-11-30T06:20:00.089Z+ export RUST_BACKTRACE=1
8732023-11-30T06:20:00.089Z+ RUST_BACKTRACE=1
8742023-11-30T06:20:00.089ZWaited for chrony: 13s
8752023-11-30T06:20:00.089Z+ export E2E_TLS_CERT IPPOOL_START IPPOOL_END
8762023-11-30T06:20:00.089Z+ ./tests/bootstrap
8772023-11-30T06:20:00.113ZUsing external DNS server at 10.151.1.164:53
8782023-11-30T06:20:00.113Z2023-11-30 06:20:00.337535029 UTC: attempting to log into API
8792023-11-30T06:20:15.177Z2023-11-30 06:20:14.809922523 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8802023-11-30T06:20:16.150Z2023-11-30 06:20:15.810992960 UTC: attempting to log into API
8812023-11-30T06:20:31.183Z2023-11-30 06:20:30.844121317 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8822023-11-30T06:20:32.184Z2023-11-30 06:20:31.845167034 UTC: attempting to log into API
8832023-11-30T06:20:47.222Z2023-11-30 06:20:46.882356514 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8842023-11-30T06:20:48.223Z2023-11-30 06:20:47.883349211 UTC: attempting to log into API
8852023-11-30T06:21:03.260Z2023-11-30 06:21:02.920079354 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8862023-11-30T06:21:04.261Z2023-11-30 06:21:03.920797339 UTC: attempting to log into API
 
8872023-11-30T06:21:16.354ZNov 30 06:21:15 buskin ip: ip: joining multicasts failed (4) on vopte3 - will use link layer broadcasts for multicast
 
8882023-11-30T06:21:19.295Z2023-11-30 06:21:18.976575522 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8892023-11-30T06:21:20.296Z2023-11-30 06:21:19.980310333 UTC: attempting to log into API
 
8902023-11-30T06:21:23.045ZNov 30 06:21:22 buskin ip: ip: joining multicasts failed (4) on vopte2 - will use link layer broadcasts for multicast
8912023-11-30T06:21:30.257ZNov 30 06:21:29 buskin ip: ip: joining multicasts failed (4) on vopte1 - will use link layer broadcasts for multicast
 
8922023-11-30T06:21:35.328Z2023-11-30 06:21:35.052510899 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8932023-11-30T06:21:36.329Z2023-11-30 06:21:36.056178655 UTC: attempting to log into API
 
8942023-11-30T06:21:38.542ZNov 30 06:21:37 bukin ip: ip: joining multicasts failed (4) on vopte5 - will use link layer broadcasts for multicast
8952023-11-30T06:21:44.967ZNov 30 06:21:44 bukin ip: ip: joining multicasts failed (4) on vopte4 - will use link layer broadcasts for multicast
 
8962023-11-30T06:21:51.361Z2023-11-30 06:21:51.128711279 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8972023-11-30T06:21:52.362Z2023-11-30 06:21:52.132521327 UTC: attempting to log into API
8982023-11-30T06:22:03.157Z2023-11-30 06:22:02.956186684 UTC: login succeeded
8992023-11-30T06:22:03.189Zwaiting for nexus to come up...
9002023-11-30T06:22:03.259Zcreating IP pool... 10.151.1.175 - 10.151.1.227
9012023-11-30T06:22:03.326Zensuring datasets are ready...
9022023-11-30T06:22:05.468Zlet's roll.
9032023-11-30T06:22:05.492Z+ rm ./tests/bootstrap
9042023-11-30T06:22:05.544Z+ for test_bin in tests/*
9052023-11-30T06:22:05.544Z+ ./tests/bootstrap-e73dec5d2cc4791e
9062023-11-30T06:22:05.568Z
9072023-11-30T06:22:05.568Zrunning 0 tests
9082023-11-30T06:22:05.568Z
9092023-11-30T06:22:05.568Ztest result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
9102023-11-30T06:22:05.568Z
9112023-11-30T06:22:05.568Z+ for test_bin in tests/*
9122023-11-30T06:22:05.568Z+ ./tests/end_to_end_tests-e99a7bbe87af43e0
9132023-11-30T06:22:05.568Z
9142023-11-30T06:22:05.568Zrunning 2 tests
9152023-11-30T06:22:05.781Ztest no_spoof::no_spoof_here ... ok
9162023-11-30T06:23:05.568Ztest instance_launch::instance_launch has been running for over 60 seconds
9172023-11-30T06:24:58.357Ztest instance_launch::instance_launch ... ok
9182023-11-30T06:24:58.357Z
9192023-11-30T06:24:58.357Ztest result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 172.79s
9202023-11-30T06:24:58.358Z
9212023-11-30T06:24:58.383Z+ _exit_trap
9222023-11-30T06:24:58.383Z+ local status=0
9232023-11-30T06:24:58.383Z+ [[ 0 -eq 0 ]]
9242023-11-30T06:24:58.383Z+ exit 0
9252023-11-30T06:24:58.383Zprocess exited: duration 367407 ms, exit code 0
 
9262023-11-30T06:24:58.519Zfound 47 output files
9272023-11-30T06:24:58.519Zuploading: /var/svc/log/oxide-sled-agent:default.log (704828 bytes)
9282023-11-30T06:24:59.800Zuploaded: /var/svc/log/oxide-sled-agent:default.log
9292023-11-30T06:24:59.800Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_dfcbe5e9-a89e-4475-af5f-9ae3b1a62d26/root/var/svc/log/oxide-cockroachdb:default.log (5367 bytes)
9302023-11-30T06:25:00.898Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_dfcbe5e9-a89e-4475-af5f-9ae3b1a62d26/root/var/svc/log/oxide-cockroachdb:default.log
9312023-11-30T06:25:00.898Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_2f330fea-db40-4407-bc57-f00393c55fe1/root/var/svc/log/oxide-crucible-agent:default.log (13686 bytes)
9322023-11-30T06:25:02.017Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_2f330fea-db40-4407-bc57-f00393c55fe1/root/var/svc/log/oxide-crucible-agent:default.log
9332023-11-30T06:25:02.017Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_2f330fea-db40-4407-bc57-f00393c55fe1/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
9342023-11-30T06:25:03.112Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_2f330fea-db40-4407-bc57-f00393c55fe1/root/var/svc/log/oxide-crucible-downstairs:default.log
9352023-11-30T06:25:03.112Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_2f330fea-db40-4407-bc57-f00393c55fe1/root/var/svc/log/oxide-crucible-downstairs:downstairs-643c6372-797d-42e9-9197-61bfa06fc305.log (3050 bytes)
9362023-11-30T06:25:04.206Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_2f330fea-db40-4407-bc57-f00393c55fe1/root/var/svc/log/oxide-crucible-downstairs:downstairs-643c6372-797d-42e9-9197-61bfa06fc305.log
9372023-11-30T06:25:04.206Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_3cc9e73f-9fba-4715-8a9d-6de3e606ffcd/root/var/svc/log/oxide-crucible-agent:default.log (24363 bytes)
9382023-11-30T06:25:05.345Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_3cc9e73f-9fba-4715-8a9d-6de3e606ffcd/root/var/svc/log/oxide-crucible-agent:default.log
9392023-11-30T06:25:05.345Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_3cc9e73f-9fba-4715-8a9d-6de3e606ffcd/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
9402023-11-30T06:25:06.438Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_3cc9e73f-9fba-4715-8a9d-6de3e606ffcd/root/var/svc/log/oxide-crucible-downstairs:default.log
9412023-11-30T06:25:06.438Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_3cc9e73f-9fba-4715-8a9d-6de3e606ffcd/root/var/svc/log/oxide-crucible-downstairs:downstairs-1dd78810-a4aa-4bdc-b23b-d71a878148bd.log (5536 bytes)
9422023-11-30T06:25:07.530Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_3cc9e73f-9fba-4715-8a9d-6de3e606ffcd/root/var/svc/log/oxide-crucible-downstairs:downstairs-1dd78810-a4aa-4bdc-b23b-d71a878148bd.log
9432023-11-30T06:25:07.530Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_3cc9e73f-9fba-4715-8a9d-6de3e606ffcd/root/var/svc/log/oxide-crucible-downstairs:downstairs-49a2ff01-0d0f-408d-a43f-ef417a1eca8a.log (3050 bytes)
9442023-11-30T06:25:08.623Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_3cc9e73f-9fba-4715-8a9d-6de3e606ffcd/root/var/svc/log/oxide-crucible-downstairs:downstairs-49a2ff01-0d0f-408d-a43f-ef417a1eca8a.log
9452023-11-30T06:25:08.623Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_72730721-be32-42d2-9320-fc57402d8b17/root/var/svc/log/oxide-crucible-agent:default.log (3436 bytes)
9462023-11-30T06:25:09.715Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_72730721-be32-42d2-9320-fc57402d8b17/root/var/svc/log/oxide-crucible-agent:default.log
9472023-11-30T06:25:09.715Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_72730721-be32-42d2-9320-fc57402d8b17/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
9482023-11-30T06:25:10.834Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_72730721-be32-42d2-9320-fc57402d8b17/root/var/svc/log/oxide-crucible-downstairs:default.log
9492023-11-30T06:25:10.834Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_bd0e5e36-6c7e-4f6b-9aee-9b37a2a6a08a/root/var/svc/log/oxide-crucible-agent:default.log (13689 bytes)
9502023-11-30T06:25:11.949Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_bd0e5e36-6c7e-4f6b-9aee-9b37a2a6a08a/root/var/svc/log/oxide-crucible-agent:default.log
9512023-11-30T06:25:11.949Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_bd0e5e36-6c7e-4f6b-9aee-9b37a2a6a08a/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
9522023-11-30T06:25:13.041Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_bd0e5e36-6c7e-4f6b-9aee-9b37a2a6a08a/root/var/svc/log/oxide-crucible-downstairs:default.log
9532023-11-30T06:25:13.041Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_bd0e5e36-6c7e-4f6b-9aee-9b37a2a6a08a/root/var/svc/log/oxide-crucible-downstairs:downstairs-15fbf0af-6042-438d-94e7-36ace567e14a.log (5536 bytes)
9542023-11-30T06:25:14.132Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_bd0e5e36-6c7e-4f6b-9aee-9b37a2a6a08a/root/var/svc/log/oxide-crucible-downstairs:downstairs-15fbf0af-6042-438d-94e7-36ace567e14a.log
9552023-11-30T06:25:14.132Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_62654947-4076-499a-9870-1ecec0cdf334/root/var/svc/log/oxide-crucible-pantry:default.log (2338 bytes)
9562023-11-30T06:25:15.223Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_62654947-4076-499a-9870-1ecec0cdf334/root/var/svc/log/oxide-crucible-pantry:default.log
9572023-11-30T06:25:15.223Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_oximeter_a0d7cf4e-b58c-42e7-b984-b05345f137a2/root/var/svc/log/oxide-oximeter:default.log (174410 bytes)
9582023-11-30T06:25:15.399Zupload warning: file "/pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_oximeter_a0d7cf4e-b58c-42e7-b984-b05345f137a2/root/var/svc/log/oxide-oximeter:default.log" changed size mid upload: 174410 -> 185953
9592023-11-30T06:25:16.467Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_oximeter_a0d7cf4e-b58c-42e7-b984-b05345f137a2/root/var/svc/log/oxide-oximeter:default.log
9602023-11-30T06:25:16.467Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_0f64dddc-25ed-4dcf-ab29-423b84220b5d/root/var/svc/log/oxide-crucible-agent:default.log (13684 bytes)
9612023-11-30T06:25:17.582Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_0f64dddc-25ed-4dcf-ab29-423b84220b5d/root/var/svc/log/oxide-crucible-agent:default.log
9622023-11-30T06:25:17.582Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_0f64dddc-25ed-4dcf-ab29-423b84220b5d/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
9632023-11-30T06:25:18.673Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_0f64dddc-25ed-4dcf-ab29-423b84220b5d/root/var/svc/log/oxide-crucible-downstairs:default.log
9642023-11-30T06:25:18.673Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_0f64dddc-25ed-4dcf-ab29-423b84220b5d/root/var/svc/log/oxide-crucible-downstairs:downstairs-4668c1fd-cfe8-4f50-8075-d8fbfcb3704c.log (3050 bytes)
9652023-11-30T06:25:19.765Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_0f64dddc-25ed-4dcf-ab29-423b84220b5d/root/var/svc/log/oxide-crucible-downstairs:downstairs-4668c1fd-cfe8-4f50-8075-d8fbfcb3704c.log
9662023-11-30T06:25:19.765Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_9164513d-4b30-4953-8eb0-eda3387069c1/root/var/svc/log/oxide-nexus:default.log (4650015 bytes)
9672023-11-30T06:25:21.123Zupload warning: file "/pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_9164513d-4b30-4953-8eb0-eda3387069c1/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 4650015 -> 4655394
9682023-11-30T06:25:22.192Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_9164513d-4b30-4953-8eb0-eda3387069c1/root/var/svc/log/oxide-nexus:default.log
9692023-11-30T06:25:22.192Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_internal_dns_d481eca6-a76b-49b9-ad2c-70ec16cf919f/root/var/svc/log/oxide-internal_dns:default.log (103216 bytes)
9702023-11-30T06:25:23.371Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_internal_dns_d481eca6-a76b-49b9-ad2c-70ec16cf919f/root/var/svc/log/oxide-internal_dns:default.log
9712023-11-30T06:25:23.371Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_ntp_c74a342e-615a-400e-93e7-d0436053ed97/root/var/svc/log/oxide-ntp:default.log (1182 bytes)
9722023-11-30T06:25:24.462Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_ntp_c74a342e-615a-400e-93e7-d0436053ed97/root/var/svc/log/oxide-ntp:default.log
9732023-11-30T06:25:24.462Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_7f0d7d18-2f53-4939-a885-bff4553dd91e/root/var/svc/log/oxide-crucible-agent:default.log (3436 bytes)
9742023-11-30T06:25:25.556Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_7f0d7d18-2f53-4939-a885-bff4553dd91e/root/var/svc/log/oxide-crucible-agent:default.log
9752023-11-30T06:25:25.556Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_7f0d7d18-2f53-4939-a885-bff4553dd91e/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
9762023-11-30T06:25:26.647Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_7f0d7d18-2f53-4939-a885-bff4553dd91e/root/var/svc/log/oxide-crucible-downstairs:default.log
9772023-11-30T06:25:26.647Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_pantry_4e7c74b9-b3ae-4b38-a3e4-1e6482c515fa/root/var/svc/log/oxide-crucible-pantry:default.log (2338 bytes)
9782023-11-30T06:25:27.737Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_pantry_4e7c74b9-b3ae-4b38-a3e4-1e6482c515fa/root/var/svc/log/oxide-crucible-pantry:default.log
9792023-11-30T06:25:27.737Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_cockroachdb_e128f891-0308-45a6-8130-e66aa3735a24/root/var/svc/log/oxide-cockroachdb:default.log (5369 bytes)
9802023-11-30T06:25:28.829Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_cockroachdb_e128f891-0308-45a6-8130-e66aa3735a24/root/var/svc/log/oxide-cockroachdb:default.log
9812023-11-30T06:25:28.829Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_cockroachdb_f30da864-101b-4cc8-94e4-9a4c31d50fdb/root/var/svc/log/oxide-cockroachdb:default.log (5367 bytes)
9822023-11-30T06:25:29.920Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_cockroachdb_f30da864-101b-4cc8-94e4-9a4c31d50fdb/root/var/svc/log/oxide-cockroachdb:default.log
9832023-11-30T06:25:29.920Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_external_dns_65d7d531-d866-43a9-8e16-7fb5bd4fd365/root/var/svc/log/oxide-external_dns:default.log (15612 bytes)
9842023-11-30T06:25:31.035Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_external_dns_65d7d531-d866-43a9-8e16-7fb5bd4fd365/root/var/svc/log/oxide-external_dns:default.log
9852023-11-30T06:25:31.035Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_cockroachdb_4023f2be-cc6a-46bc-90b1-0a643c886ffb/root/var/svc/log/oxide-cockroachdb:default.log (5342 bytes)
9862023-11-30T06:25:32.127Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_cockroachdb_4023f2be-cc6a-46bc-90b1-0a643c886ffb/root/var/svc/log/oxide-cockroachdb:default.log
9872023-11-30T06:25:32.127Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_1c762b6a-6ff7-4c49-b9a6-e1b25c0a259c/root/var/svc/log/oxide-crucible-agent:default.log (14117 bytes)
9882023-11-30T06:25:33.241Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_1c762b6a-6ff7-4c49-b9a6-e1b25c0a259c/root/var/svc/log/oxide-crucible-agent:default.log
9892023-11-30T06:25:33.241Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_1c762b6a-6ff7-4c49-b9a6-e1b25c0a259c/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
9902023-11-30T06:25:34.333Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_1c762b6a-6ff7-4c49-b9a6-e1b25c0a259c/root/var/svc/log/oxide-crucible-downstairs:default.log
9912023-11-30T06:25:34.333Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_1c762b6a-6ff7-4c49-b9a6-e1b25c0a259c/root/var/svc/log/oxide-crucible-downstairs:downstairs-43d1b48a-00de-403e-b38a-29c3bf71cc0d.log (5536 bytes)
9922023-11-30T06:25:35.425Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_1c762b6a-6ff7-4c49-b9a6-e1b25c0a259c/root/var/svc/log/oxide-crucible-downstairs:downstairs-43d1b48a-00de-403e-b38a-29c3bf71cc0d.log
9932023-11-30T06:25:35.434Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_6a0e6478-1881-4d77-988a-e4ba5539771d/root/var/svc/log/oxide-crucible-agent:default.log (3436 bytes)
9942023-11-30T06:25:36.532Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_6a0e6478-1881-4d77-988a-e4ba5539771d/root/var/svc/log/oxide-crucible-agent:default.log
9952023-11-30T06:25:36.532Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_6a0e6478-1881-4d77-988a-e4ba5539771d/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
9962023-11-30T06:25:37.625Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_6a0e6478-1881-4d77-988a-e4ba5539771d/root/var/svc/log/oxide-crucible-downstairs:default.log
9972023-11-30T06:25:37.625Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_pantry_50bbc5c2-166d-485d-8f77-dde773b220c2/root/var/svc/log/oxide-crucible-pantry:default.log (2333 bytes)
9982023-11-30T06:25:38.717Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_pantry_50bbc5c2-166d-485d-8f77-dde773b220c2/root/var/svc/log/oxide-crucible-pantry:default.log
9992023-11-30T06:25:38.719Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_internal_dns_7ff161c9-3db3-4cda-92bd-32f52b4aa8d1/root/var/svc/log/oxide-internal_dns:default.log (99477 bytes)
10002023-11-30T06:25:39.917Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_internal_dns_7ff161c9-3db3-4cda-92bd-32f52b4aa8d1/root/var/svc/log/oxide-internal_dns:default.log
10012023-11-30T06:25:39.917Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_internal_dns_af26ef38-0766-4b15-b67b-f474cee55891/root/var/svc/log/oxide-internal_dns:default.log (99573 bytes)
10022023-11-30T06:25:41.116Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_internal_dns_af26ef38-0766-4b15-b67b-f474cee55891/root/var/svc/log/oxide-internal_dns:default.log
10032023-11-30T06:25:41.116Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_9c976434-1209-4067-aca9-a894f0aebecc/root/var/svc/log/oxide-nexus:default.log (507431 bytes)
10042023-11-30T06:25:41.361Zupload warning: file "/pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_9c976434-1209-4067-aca9-a894f0aebecc/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 507431 -> 513415
10052023-11-30T06:25:42.430Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_9c976434-1209-4067-aca9-a894f0aebecc/root/var/svc/log/oxide-nexus:default.log
10062023-11-30T06:25:42.430Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_aab47d47-fb93-4274-9417-ce24a6c7802a/root/var/svc/log/oxide-nexus:default.log (532965 bytes)
10072023-11-30T06:25:42.653Zupload warning: file "/pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_aab47d47-fb93-4274-9417-ce24a6c7802a/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 532965 -> 538947
10082023-11-30T06:25:43.722Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_aab47d47-fb93-4274-9417-ce24a6c7802a/root/var/svc/log/oxide-nexus:default.log
10092023-11-30T06:25:43.722Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_clickhouse_046f1f61-ded5-4bcc-96af-62665ccc6b44/root/var/svc/log/oxide-clickhouse:default.log (660453 bytes)
10102023-11-30T06:25:43.991Zupload warning: file "/pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_clickhouse_046f1f61-ded5-4bcc-96af-62665ccc6b44/root/var/svc/log/oxide-clickhouse:default.log" changed size mid upload: 660453 -> 787047
10112023-11-30T06:25:45.059Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_clickhouse_046f1f61-ded5-4bcc-96af-62665ccc6b44/root/var/svc/log/oxide-clickhouse:default.log
10122023-11-30T06:25:45.059Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_c57c5379-e538-4b8f-9142-b11b8b242aeb/root/var/svc/log/oxide-cockroachdb:default.log (5369 bytes)
10132023-11-30T06:25:46.151Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_c57c5379-e538-4b8f-9142-b11b8b242aeb/root/var/svc/log/oxide-cockroachdb:default.log
10142023-11-30T06:25:46.152Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_f3f22b5b-4c5f-41b6-8a8f-eb9f8405ff80/root/var/svc/log/oxide-crucible-agent:default.log (3436 bytes)
10152023-11-30T06:25:47.242Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_f3f22b5b-4c5f-41b6-8a8f-eb9f8405ff80/root/var/svc/log/oxide-crucible-agent:default.log
10162023-11-30T06:25:47.242Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_f3f22b5b-4c5f-41b6-8a8f-eb9f8405ff80/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10172023-11-30T06:25:48.333Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_f3f22b5b-4c5f-41b6-8a8f-eb9f8405ff80/root/var/svc/log/oxide-crucible-downstairs:default.log
10182023-11-30T06:25:48.333Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_8d79e9d2-da7a-4db1-9f61-0a34b00e9181/root/var/svc/log/oxide-external_dns:default.log (15613 bytes)
10192023-11-30T06:25:48.421Zupload warning: file "/pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_8d79e9d2-da7a-4db1-9f61-0a34b00e9181/root/var/svc/log/oxide-external_dns:default.log" changed size mid upload: 15613 -> 19390
10202023-11-30T06:25:49.490Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_8d79e9d2-da7a-4db1-9f61-0a34b00e9181/root/var/svc/log/oxide-external_dns:default.log
10212023-11-30T06:25:49.490Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_ntp_c74a342e-615a-400e-93e7-d0436053ed97/root/var/log/chrony/measurements.log (6713 bytes)
10222023-11-30T06:25:49.557Zupload warning: file "/pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_ntp_c74a342e-615a-400e-93e7-d0436053ed97/root/var/log/chrony/measurements.log" changed size mid upload: 6713 -> 7672
10232023-11-30T06:25:50.626Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_ntp_c74a342e-615a-400e-93e7-d0436053ed97/root/var/log/chrony/measurements.log
10242023-11-30T06:25:50.626Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_ntp_c74a342e-615a-400e-93e7-d0436053ed97/root/var/log/chrony/statistics.log (3744 bytes)
10252023-11-30T06:25:50.671Zupload warning: file "/pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_ntp_c74a342e-615a-400e-93e7-d0436053ed97/root/var/log/chrony/statistics.log" changed size mid upload: 3744 -> 4914
10262023-11-30T06:25:51.740Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_ntp_c74a342e-615a-400e-93e7-d0436053ed97/root/var/log/chrony/statistics.log
10272023-11-30T06:25:51.740Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_ntp_c74a342e-615a-400e-93e7-d0436053ed97/root/var/log/chrony/tracking.log (1590 bytes)
10282023-11-30T06:25:51.785Zupload warning: file "/pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_ntp_c74a342e-615a-400e-93e7-d0436053ed97/root/var/log/chrony/tracking.log" changed size mid upload: 1590 -> 1722
10292023-11-30T06:25:52.852Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_ntp_c74a342e-615a-400e-93e7-d0436053ed97/root/var/log/chrony/tracking.log