01HFNQZ514MT5DE021PGXVYWV5: helios / deploy

BasicConfig {
    output_rules: [
        "%/var/svc/log/oxide-sled-agent:default.log*",
        "%/pool/ext/*/crypt/zone/oxz_*/root/var/svc/log/oxide-*.log*",
        "%/pool/ext/*/crypt/zone/oxz_*/root/var/svc/log/system-illumos-*.log*",
        "%/pool/ext/*/crypt/zone/oxz_ntp_*/root/var/log/chrony/*.log*",
        "!/pool/ext/*/crypt/zone/oxz_propolis-server_*/root/var/svc/log/*.log*",
        "%/pool/ext/*/crypt/debug/global/oxide-sled-agent:default.log.*",
        "%/pool/ext/*/crypt/debug/oxz_*/oxide-*.log.*",
        "%/pool/ext/*/crypt/debug/oxz_*/system-illumos-*.log.*",
        "!/pool/ext/*/crypt/debug/oxz_propolis-server_*/*.log.*",
    ],
    rust_toolchain: None,
    target: Some(
        "lab-2.0-opte-0.25",
    ),
    access_repos: [],
    publish: [],
    skip_clone: true,
}

Buildomat Job: 01HFNR099TY5PD356VYT8NBDAK

Tags:

Artefacts:

Output:

SEQ GLOBAL TIME DETAILS
12023-11-20T07:39:54.100Zjob dependencies complete; ready to run (waiting for 47 m 48 s)
22023-11-20T07:41:11.584Zjob assigned to worker 01HFNTE75YFYJ57NSV6QCVS7ZK (queued for 1 m 17 s)
32023-11-20T07:41:11.642Zhttp://172.20.0.5:9969/ipxe/sock... ok
42023-11-20T07:41:11.679Zhttp://172.20.0.5:9969/os/sock/platform/i86pc/kernel/amd64/unix... ok
52023-11-20T07:41:12.117Zhttp://172.20.0.5:9969/os/sock/platform/i86pc/amd64/boot_archive... ok
62023-11-20T07:41:12.157Zhttp://172.20.0.5:9969/os/sock/platform/i86pc/amd64/boot_archive.hash... ok
72023-11-20T07:41:12.191Zhttp://172.20.0.5:9969/postboot/sock... ok
82023-11-20T07:41:12.224ZwcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U
92023-11-20T07:41:12.262Zcpu0: microcode has been updated from version 0xa001133 to 0xa0011d1
102023-11-20T07:41:12.299ZOxide Helios Version helios-2.0.22241 64-bit
112023-11-20T07:41:12.336ZConfiguring devices.
122023-11-20T07:41:12.369Z##################################################
132023-11-20T07:41:12.403Z#### ############# ##
142023-11-20T07:41:12.436Z### ### ############ ##
152023-11-20T07:41:12.469Z## ### # ## ### ## Oxide Computer Company ##
162023-11-20T07:41:12.502Z## ## ## ### # ### ##
172023-11-20T07:41:12.536Z## # ### #### #### This Station Under ##
182023-11-20T07:41:12.569Z### ### #### # ### Computer Control ##
192023-11-20T07:41:12.603Z#### #### ### ## ##
202023-11-20T07:41:12.636Z##################################################
212023-11-20T07:41:12.673ZHostname: sock
222023-11-20T07:41:12.710ZLoading smf(7) service descriptions:
232023-11-20T07:41:13.208Zdownloading input: /input/ci-tools/work/end-to-end-tests/bootstrap-7746be776f1918b3.gz
242023-11-20T07:41:15.324Zdownloaded input: /input/ci-tools/work/end-to-end-tests/bootstrap-7746be776f1918b3.gz
252023-11-20T07:41:15.776Zdownloading input: /input/ci-tools/work/end-to-end-tests/bootstrap.gz
262023-11-20T07:41:37.059Zdownloaded input: /input/ci-tools/work/end-to-end-tests/bootstrap.gz
272023-11-20T07:41:37.095Zdownloading input: /input/ci-tools/work/end-to-end-tests/end_to_end_tests-cb4cc51c87b08c71.gz
282023-11-20T07:42:08.920Zdownloaded input: /input/ci-tools/work/end-to-end-tests/end_to_end_tests-cb4cc51c87b08c71.gz
292023-11-20T07:42:08.960Zdownloading input: /input/ci-tools/work/caboose-util.gz
302023-11-20T07:42:10.702Zdownloaded input: /input/ci-tools/work/caboose-util.gz
312023-11-20T07:42:10.734Zdownloading input: /input/ci-tools/work/tufaceous.gz
322023-11-20T07:42:19.733Zdownloaded input: /input/ci-tools/work/tufaceous.gz
332023-11-20T07:42:20.468Zdownloading input: /input/package/work/version.txt
342023-11-20T07:42:21.596Zdownloaded input: /input/package/work/version.txt
352023-11-20T07:42:22.088Zdownloading input: /input/package/work/package.tar.gz
362023-11-20T07:42:46.343Zdownloaded input: /input/package/work/package.tar.gz
372023-11-20T07:42:46.382Zdownloading input: /input/package/work/global-zone-packages.tar.gz
382023-11-20T07:43:21.818Zdownloaded input: /input/package/work/global-zone-packages.tar.gz
392023-11-20T07:43:21.855Zdownloading input: /input/package/work/trampoline-global-zone-packages.tar.gz
402023-11-20T07:43:33.759Zdownloaded input: /input/package/work/trampoline-global-zone-packages.tar.gz
412023-11-20T07:43:33.798Zdownloading input: /input/package/work/zones/clickhouse.tar.gz
422023-11-20T07:44:30.926Zdownloaded input: /input/package/work/zones/clickhouse.tar.gz
432023-11-20T07:44:30.959Zdownloading input: /input/package/work/zones/clickhouse_keeper.tar.gz
442023-11-20T07:45:33.151Zdownloaded input: /input/package/work/zones/clickhouse_keeper.tar.gz
452023-11-20T07:45:33.191Zdownloading input: /input/package/work/zones/cockroachdb.tar.gz
462023-11-20T07:46:06.166Zdownloaded input: /input/package/work/zones/cockroachdb.tar.gz
472023-11-20T07:46:06.644Zdownloading input: /input/package/work/zones/crucible-pantry.tar.gz
482023-11-20T07:46:13.205Zdownloaded input: /input/package/work/zones/crucible-pantry.tar.gz
492023-11-20T07:46:13.299Zdownloading input: /input/package/work/zones/crucible.tar.gz
502023-11-20T07:46:26.221Zdownloaded input: /input/package/work/zones/crucible.tar.gz
512023-11-20T07:46:26.252Zdownloading input: /input/package/work/zones/external-dns.tar.gz
522023-11-20T07:46:33.459Zdownloaded input: /input/package/work/zones/external-dns.tar.gz
532023-11-20T07:46:33.497Zdownloading input: /input/package/work/zones/internal-dns.tar.gz
542023-11-20T07:46:39.545Zdownloaded input: /input/package/work/zones/internal-dns.tar.gz
552023-11-20T07:46:39.577Zdownloading input: /input/package/work/zones/ntp.tar.gz
562023-11-20T07:46:39.614Zdownloaded input: /input/package/work/zones/ntp.tar.gz
572023-11-20T07:46:39.651Zdownloading input: /input/package/work/zones/omicron-gateway-asic.tar.gz
582023-11-20T07:46:45.190Zdownloaded input: /input/package/work/zones/omicron-gateway-asic.tar.gz
592023-11-20T07:46:45.229Zdownloading input: /input/package/work/zones/omicron-gateway-softnpu.tar.gz
602023-11-20T07:46:50.008Zdownloaded input: /input/package/work/zones/omicron-gateway-softnpu.tar.gz
612023-11-20T07:46:50.042Zdownloading input: /input/package/work/zones/omicron-nexus-single-sled.tar.gz
622023-11-20T07:47:16.646Zdownloaded input: /input/package/work/zones/omicron-nexus-single-sled.tar.gz
632023-11-20T07:47:16.684Zdownloading input: /input/package/work/zones/omicron-nexus.tar.gz
642023-11-20T07:47:33.585Zdownloaded input: /input/package/work/zones/omicron-nexus.tar.gz
652023-11-20T07:47:33.616Zdownloading input: /input/package/work/zones/overlay.tar.gz
662023-11-20T07:47:33.670Zdownloaded input: /input/package/work/zones/overlay.tar.gz
672023-11-20T07:47:33.709Zdownloading input: /input/package/work/zones/oximeter-collector.tar.gz
682023-11-20T07:47:41.403Zdownloaded input: /input/package/work/zones/oximeter-collector.tar.gz
692023-11-20T07:47:41.441Zdownloading input: /input/package/work/zones/propolis-server.tar.gz
702023-11-20T07:47:56.367Zdownloaded input: /input/package/work/zones/propolis-server.tar.gz
712023-11-20T07:47:56.412Zdownloading input: /input/package/work/zones/switch-asic.tar.gz
722023-11-20T07:48:52.015Zdownloaded input: /input/package/work/zones/switch-asic.tar.gz
732023-11-20T07:48:52.054Zdownloading input: /input/package/work/zones/switch-softnpu.tar.gz
742023-11-20T07:49:32.961Zdownloaded input: /input/package/work/zones/switch-softnpu.tar.gz
 
752023-11-20T07:49:32.992Zstarting task 0: "setup"
762023-11-20T07:49:33.033Z++ uname -s
772023-11-20T07:49:33.064Z+ kern=SunOS
782023-11-20T07:49:33.095Z+ case "$kern" in
792023-11-20T07:49:33.126Z+ groupadd -g 12345 build
802023-11-20T07:49:33.165Z+ useradd -u 12345 -g build -d /home/build -s /bin/bash -c build -P 'Primary Administrator' build
812023-11-20T07:49:35.042Z+ zfs create -o mountpoint=/work rpool/work
822023-11-20T07:49:35.080Z++ awk '$2 == "/home" { print $3 }' /etc/mnttab
832023-11-20T07:49:35.111Z+ home_fs=zfs
842023-11-20T07:49:35.143Z+ [[ zfs == autofs ]]
852023-11-20T07:49:35.174Z+ mkdir -p /home/build
862023-11-20T07:49:35.205Z+ chown build:build /home/build /work
872023-11-20T07:49:37.092Z+ chmod 0700 /home/build /work
882023-11-20T07:49:37.128Zprocess exited: duration 4067 ms, exit code 0
 
892023-11-20T07:49:37.197Zstarting task 1: "authentication"
902023-11-20T07:49:37.279Zprocess exited: duration 50 ms, exit code 0
 
912023-11-20T07:49:37.343Zstarting task 2: "build"
922023-11-20T07:49:37.381Z+ trap _exit_trap EXIT
932023-11-20T07:49:37.412Z+ pfexec mdb -kwe 'secpolicy_fs_linkdir/v 55 48 89 e5 b8 01 00 00 00 5d c3'
942023-11-20T07:49:37.620Zsecpolicy_fs_linkdir: 0x55 = 0x55
952023-11-20T07:49:37.654Zsecpolicy_fs_linkdir+1: 0xb8 = 0x48
962023-11-20T07:49:37.686Zsecpolicy_fs_linkdir+2: 0x1 = 0x89
972023-11-20T07:49:37.716Zsecpolicy_fs_linkdir+3: 0 = 0xe5
982023-11-20T07:49:37.748Zsecpolicy_fs_linkdir+4: 0 = 0xb8
992023-11-20T07:49:37.779Zsecpolicy_fs_linkdir+5: 0 = 0x1
1002023-11-20T07:49:37.810Zsecpolicy_fs_linkdir+6: 0x48 = 0x0
1012023-11-20T07:49:37.842Zsecpolicy_fs_linkdir+7: 0x89 = 0x0
1022023-11-20T07:49:37.874Zsecpolicy_fs_linkdir+8: 0xe5 = 0x0
1032023-11-20T07:49:37.905Zsecpolicy_fs_linkdir+9: 0x48 = 0x5d
1042023-11-20T07:49:37.936Zsecpolicy_fs_linkdir+0xa: 0x83 = 0xc3
1052023-11-20T07:49:37.976Z+ [[ -d /opt/oxide ]]
1062023-11-20T07:49:38.008Z+ pfexec mv /opt/oxide /opt/oxide-underneath
1072023-11-20T07:49:38.039Z+ pfexec mkdir /opt/oxide
1082023-11-20T07:49:38.077Z+ pfexec mount -F tmpfs -O swap /opt/oxide
1092023-11-20T07:49:38.117Z+ [[ -d /opt/oxide-underneath ]]
1102023-11-20T07:49:38.163Z+ cd /opt/oxide-underneath
1112023-11-20T07:49:38.207Z+ pfexec tar ceEp@/f - .
1122023-11-20T07:49:38.255Z+ cd /opt/oxide
1132023-11-20T07:49:38.331Z+ pfexec tar xveEp@/f -
1142023-11-20T07:49:38.362Zx ., 0 bytes, 0 tape blocks
1152023-11-20T07:49:38.394Zx ./opte, 0 bytes, 0 tape blocks
1162023-11-20T07:49:38.457Zx ./opte/bin, 0 bytes, 0 tape blocks
1172023-11-20T07:49:38.488Zx ./opte/bin/opteadm, 50714832 bytes, 99053 tape blocks
1182023-11-20T07:49:38.522Z+ pfexec rm -rf /opt/oxide-underneath
1192023-11-20T07:49:38.556Z+ pfexec /sbin/zfs create -o mountpoint=/zone rpool/zone
1202023-11-20T07:49:38.591Z+ pfexec mkdir /opt/oxide/work
1212023-11-20T07:49:38.649Z+ pfexec chown build:build /opt/oxide/work
1222023-11-20T07:49:38.680Z+ cd /opt/oxide/work
1232023-11-20T07:49:38.711Z+ ptime -m tar xvzf /input/package/work/package.tar.gz
1242023-11-20T07:49:39.286ZDecompressing '/input/package/work/package.tar.gz' with '/usr/bin/gzcat'...
1252023-11-20T07:49:39.317Zx out/mg-ddm-gz.tar, 62723072 bytes, 122506 tape blocks
1262023-11-20T07:49:39.348Zx out/omicron-sled-agent.tar, 111626752 bytes, 218021 tape blocks
1272023-11-20T07:49:39.379Zx out/target/test, 75 bytes, 1 tape blocks
1282023-11-20T07:49:39.410Zx out/npuzone/npuzone, 37380856 bytes, 73010 tape blocks
1292023-11-20T07:49:39.441Zx package-manifest.toml, 22143 bytes, 44 tape blocks
1302023-11-20T07:49:39.472Zx smf/sled-agent/non-gimlet/config.toml, 3298 bytes, 7 tape blocks
1312023-11-20T07:49:39.503Zx target/release/omicron-package, 42522496 bytes, 83052 tape blocks
1322023-11-20T07:49:39.534Zx tools/create_virtual_hardware.sh, 3176 bytes, 7 tape blocks
1332023-11-20T07:49:39.565Zx tools/virtual_hardware.sh, 3408 bytes, 7 tape blocks
1342023-11-20T07:49:39.596Zx tools/scrimlet/create-softnpu-zone.sh, 619 bytes, 2 tape blocks
1352023-11-20T07:49:39.627Zx tools/scrimlet/destroy-softnpu-zone.sh, 195 bytes, 1 tape blocks
1362023-11-20T07:49:39.679Zx tools/scrimlet/softnpu-init.sh, 2148 bytes, 5 tape blocks
1372023-11-20T07:49:39.710Zx tools/scrimlet/softnpu-zone.txt, 330 bytes, 1 tape blocks
1382023-11-20T07:49:39.741Zx tools/scrimlet/softnpu.toml, 171 bytes, 1 tape blocks
1392023-11-20T07:49:39.772Z
1402023-11-20T07:49:39.803Zreal 1.548969966
1412023-11-20T07:49:39.834Zuser 1.392580043
1422023-11-20T07:49:39.865Zsys 0.275068885
1432023-11-20T07:49:39.895Ztrap 0.000060040
1442023-11-20T07:49:39.926Ztflt 0.000000000
1452023-11-20T07:49:39.957Zdflt 0.000000000
1462023-11-20T07:49:39.988Zkflt 0.000000000
1472023-11-20T07:49:40.019Zlock 0.000000000
1482023-11-20T07:49:40.051Zslp 1.393456544
1492023-11-20T07:49:40.082Zlat 0.034732388
1502023-11-20T07:49:40.113Zstop 0.000117371
1512023-11-20T07:49:40.144Z+ cp /input/package/work/zones/clickhouse_keeper.tar.gz /input/package/work/zones/clickhouse.tar.gz /input/package/work/zones/cockroachdb.tar.gz /input/package/work/zones/crucible-pantry.tar.gz /input/package/work/zones/crucible.tar.gz /input/package/work/zones/external-dns.tar.gz /input/package/work/zones/internal-dns.tar.gz /input/package/work/zones/ntp.tar.gz /input/package/work/zones/omicron-gateway-asic.tar.gz /input/package/work/zones/omicron-gateway-softnpu.tar.gz /input/package/work/zones/omicron-nexus-single-sled.tar.gz /input/package/work/zones/omicron-nexus.tar.gz /input/package/work/zones/overlay.tar.gz /input/package/work/zones/oximeter-collector.tar.gz /input/package/work/zones/propolis-server.tar.gz /input/package/work/zones/switch-asic.tar.gz /input/package/work/zones/switch-softnpu.tar.gz out/
1522023-11-20T07:49:40.365Z+ mv out/omicron-nexus-single-sled.tar.gz out/omicron-nexus.tar.gz
1532023-11-20T07:49:40.396Z+ mkdir tests
1542023-11-20T07:49:40.427Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1552023-11-20T07:49:40.458Z+ ptime -m gunzip
1562023-11-20T07:49:40.489Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap-7746be776f1918b3
1572023-11-20T07:49:40.521Z
1582023-11-20T07:49:40.553Zreal 0.137935685
1592023-11-20T07:49:40.584Zuser 0.127218781
1602023-11-20T07:49:40.615Zsys 0.010494198
1612023-11-20T07:49:40.646Ztrap 0.000011230
1622023-11-20T07:49:40.677Ztflt 0.000000000
1632023-11-20T07:49:40.709Zdflt 0.000000000
1642023-11-20T07:49:40.741Zkflt 0.000000000
1652023-11-20T07:49:40.772Zlock 0.000000000
1662023-11-20T07:49:40.806Zslp 0.000018610
1672023-11-20T07:49:40.839Zlat 0.000015720
1682023-11-20T07:49:40.871Zstop 0.000116192
1692023-11-20T07:49:40.902Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap-7746be776f1918b3
1702023-11-20T07:49:40.933Z+ chmod a+x tests/bootstrap-7746be776f1918b3
1712023-11-20T07:49:40.964Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1722023-11-20T07:49:40.995Z+ ptime -m gunzip
1732023-11-20T07:49:41.026Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap
1742023-11-20T07:49:42.517Z
1752023-11-20T07:49:42.549Zreal 1.983664586
1762023-11-20T07:49:42.580Zuser 1.860215705
1772023-11-20T07:49:42.616Zsys 0.123029134
1782023-11-20T07:49:42.647Ztrap 0.000160201
1792023-11-20T07:49:42.679Ztflt 0.000000000
1802023-11-20T07:49:42.710Zdflt 0.000000000
1812023-11-20T07:49:42.741Zkflt 0.000000000
1822023-11-20T07:49:42.774Zlock 0.000000000
1832023-11-20T07:49:42.806Zslp 0.000023190
1842023-11-20T07:49:42.837Zlat 0.000033210
1852023-11-20T07:49:42.868Zstop 0.000133722
1862023-11-20T07:49:42.899Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap
1872023-11-20T07:49:42.930Z+ chmod a+x tests/bootstrap
1882023-11-20T07:49:42.961Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1892023-11-20T07:49:42.993Z+ ptime -m gunzip
1902023-11-20T07:49:43.024Z++ basename /input/ci-tools/work/end-to-end-tests/end_to_end_tests-cb4cc51c87b08c71
1912023-11-20T07:49:44.833Z
1922023-11-20T07:49:44.864Zreal 2.275605238
1932023-11-20T07:49:44.895Zuser 2.126083122
1942023-11-20T07:49:44.926Zsys 0.149139296
1952023-11-20T07:49:44.957Ztrap 0.000148865
1962023-11-20T07:49:44.988Ztflt 0.000000000
1972023-11-20T07:49:45.020Zdflt 0.000000000
1982023-11-20T07:49:45.051Zkflt 0.000000000
1992023-11-20T07:49:45.082Zlock 0.000000000
2002023-11-20T07:49:45.113Zslp 0.000022020
2012023-11-20T07:49:45.144Zlat 0.000022720
2022023-11-20T07:49:45.175Zstop 0.000117952
2032023-11-20T07:49:45.206Z++ basename /input/ci-tools/work/end-to-end-tests/end_to_end_tests-cb4cc51c87b08c71
2042023-11-20T07:49:45.238Z+ chmod a+x tests/end_to_end_tests-cb4cc51c87b08c71
2052023-11-20T07:49:45.269Z+ bmat address ls
2062023-11-20T07:49:45.300ZNAME CIDR FIRST LAST COUNT
2072023-11-20T07:49:45.331Zextra 10.151.2.0/24 10.151.2.164 10.151.2.227 64
2082023-11-20T07:49:45.362Z++ bmat address ls -f extra -Ho first,last,count
2092023-11-20T07:49:45.393Z+ set -- 10.151.2.164 10.151.2.227 64
2102023-11-20T07:49:45.424Z+ EXTRA_IP_START=10.151.2.164
2112023-11-20T07:49:45.455Z+ EXTRA_IP_END=10.151.2.227
2122023-11-20T07:49:45.486Z+ EXTRA_IP_COUNT=64
2132023-11-20T07:49:45.517Z+ (( EXTRA_IP_COUNT >= 32 ))
2142023-11-20T07:49:45.549Z+ EXTRA_IP_BASE=10.151.2
2152023-11-20T07:49:45.579Z+ EXTRA_IP_FOCTET=164
2162023-11-20T07:49:45.610Z+ EXTRA_IP_LOCTET=227
2172023-11-20T07:49:45.641Z+ SERVICE_IP_POOL_START=10.151.2.164
2182023-11-20T07:49:45.673Z+ SERVICE_IP_POOL_END=10.151.2.173
2192023-11-20T07:49:45.703Z+ DNS_IP1=10.151.2.164
2202023-11-20T07:49:45.734Z+ DNS_IP2=10.151.2.165
2212023-11-20T07:49:45.765Z+ UPLINK_IP=10.151.2.174
2222023-11-20T07:49:45.796Z+ IPPOOL_START=10.151.2.175
2232023-11-20T07:49:45.827Z+ IPPOOL_END=10.151.2.227
2242023-11-20T07:49:45.863Z++ ipadm show-addr -po type,addr
2252023-11-20T07:49:45.894Z++ awk '-F[:/]' '$1 == "dhcp" {print $2}'
2262023-11-20T07:49:45.926Z+ GATEWAY_IP=10.151.2.100
2272023-11-20T07:49:45.956Z+ [[ -n 10.151.2.100 ]]
2282023-11-20T07:49:45.987Z+ ping -s 10.151.2.100 56 1
2292023-11-20T07:49:46.019ZPING 10.151.2.100: 56 data bytes
2302023-11-20T07:49:46.056Z64 bytes from 10.151.2.100: icmp_seq=0. time=0.011 ms
2312023-11-20T07:49:46.087Z
2322023-11-20T07:49:46.118Z----10.151.2.100 PING Statistics----
2332023-11-20T07:49:46.149Z1 packets transmitted, 1 packets received, 0% packet loss
2342023-11-20T07:49:46.180Zround-trip (ms) min/avg/max/stddev = 0.011/0.011/0.011/-nan
2352023-11-20T07:49:46.211Z++ arp -an
2362023-11-20T07:49:46.244Z++ awk -vgw=10.151.2.100 '$2 == gw {print $NF}'
2372023-11-20T07:49:46.275Z+ GATEWAY_MAC=18:c0:4d:0d:a0:2a
2382023-11-20T07:49:46.306Z+ routeadm -e ipv4-forwarding -u
2392023-11-20T07:49:46.338Z+ PXA_START=10.151.2.164
2402023-11-20T07:49:46.376Z+ PXA_END=10.151.2.227
2412023-11-20T07:49:46.411Z+ export GATEWAY_IP GATEWAY_MAC PXA_START PXA_END
2422023-11-20T07:49:46.445Z+ pfexec zpool create -f scratch c1t1d0 c2t1d0
2432023-11-20T07:49:46.480Z+ ZPOOL_VDEV_DIR=/scratch
2442023-11-20T07:49:46.515Z+ ptime -m pfexec ./tools/create_virtual_hardware.sh
2452023-11-20T07:49:46.549Z+++ dirname ./tools/create_virtual_hardware.sh
2462023-11-20T07:49:46.583Z++ cd ./tools
2472023-11-20T07:49:46.619Z++ pwd
2482023-11-20T07:49:46.656Z+ SOURCE_DIR=/opt/oxide/work/tools
2492023-11-20T07:49:46.693Z+ OMICRON_TOP=/opt/oxide/work/tools/..
2502023-11-20T07:49:46.731Z+ SOFTNPU_MODE=zone
2512023-11-20T07:49:46.768Z+ . /opt/oxide/work/tools/virtual_hardware.sh
2522023-11-20T07:49:46.808Z++ MARKER=/etc/opt/oxide/NO_INSTALL
2532023-11-20T07:49:46.852Z++ [[ -f /etc/opt/oxide/NO_INSTALL ]]
2542023-11-20T07:49:46.918Z++ dladm show-phys -p -o LINK
2552023-11-20T07:49:46.951Z++ head -1
2562023-11-20T07:49:46.983Z+ PHYSICAL_LINK=igb0
2572023-11-20T07:49:47.015Z+ echo 'Using igb0 as physical link'
2582023-11-20T07:49:47.047ZUsing igb0 as physical link
2592023-11-20T07:49:47.079Z+ ensure_run_as_root
2602023-11-20T07:49:47.110Z++ id -u
2612023-11-20T07:49:47.141Z+ [[ 0 -ne 0 ]]
2622023-11-20T07:49:47.172Z+ ensure_zpools
2632023-11-20T07:49:47.203Z+ ZPOOL_TYPES=('oxp_' 'oxi_')
2642023-11-20T07:49:47.234Z+ for ZPOOL_TYPE in "${ZPOOL_TYPES[@]}"
2652023-11-20T07:49:47.265Z+ readarray -t ZPOOLS
2662023-11-20T07:49:47.712Z++ grep '"oxp_' /opt/oxide/work/tools/../smf/sled-agent/non-gimlet/config.toml
2672023-11-20T07:49:47.745Z++ sed 's/[ ",]//g'
2682023-11-20T07:49:47.776Z+ for ZPOOL in "${ZPOOLS[@]}"
2692023-11-20T07:49:47.808Z+ echo 'Zpool: [oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b]'
2702023-11-20T07:49:47.839ZZpool: [oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b]
2712023-11-20T07:49:47.870Z+ VDEV_PATH=/scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
2722023-11-20T07:49:47.901ZZFS vdev /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
2732023-11-20T07:49:47.946Z+ [[ -f /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
2742023-11-20T07:49:47.977ZZFS zpool oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b exists
2752023-11-20T07:49:48.011Z+ dd if=/dev/zero of=/scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
2762023-11-20T07:49:48.043ZZpool: [oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03]
2772023-11-20T07:49:48.075Z0+0 records in
2782023-11-20T07:49:48.106ZZFS vdev /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
2792023-11-20T07:49:48.138Z0+0 records out
2802023-11-20T07:49:48.171ZZFS zpool oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
2812023-11-20T07:49:48.203Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
2822023-11-20T07:49:48.267ZZpool: [oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03]
2832023-11-20T07:49:48.299Z+ success 'ZFS vdev /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
2842023-11-20T07:49:48.330ZZFS vdev /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
2852023-11-20T07:49:48.363Z+ set +x
2862023-11-20T07:49:48.394ZZFS zpool oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
2872023-11-20T07:49:48.425Z++ zpool list -o name
2882023-11-20T07:49:48.456ZZpool: [oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03]
2892023-11-20T07:49:48.487Z++ grep oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b
2902023-11-20T07:49:48.519ZZFS vdev /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
2912023-11-20T07:49:48.551Z+ [[ -z '' ]]
2922023-11-20T07:49:48.582ZZFS zpool oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
2932023-11-20T07:49:48.613Z+ zpool create -f oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
2942023-11-20T07:49:48.644ZZpool: [oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03]
2952023-11-20T07:49:48.675Z+ success 'ZFS zpool oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
2962023-11-20T07:49:48.715ZZFS vdev /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
2972023-11-20T07:49:48.746Z+ set +x
2982023-11-20T07:49:48.777ZZFS zpool oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
2992023-11-20T07:49:48.808Z+ for ZPOOL in "${ZPOOLS[@]}"
3002023-11-20T07:49:48.839ZZpool: [oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930]
3012023-11-20T07:49:48.870Z+ echo 'Zpool: [oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3022023-11-20T07:49:48.901ZZFS vdev /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev exists
3032023-11-20T07:49:48.933Z+ VDEV_PATH=/scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3042023-11-20T07:49:48.975ZZFS zpool oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 exists
3052023-11-20T07:49:49.007Z+ [[ -f /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3062023-11-20T07:49:49.038ZZpool: [oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60]
3072023-11-20T07:49:49.069Z+ dd if=/dev/zero of=/scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3082023-11-20T07:49:49.100ZZFS vdev /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev exists
3092023-11-20T07:49:49.134Z0+0 records in
3102023-11-20T07:49:49.165ZZFS zpool oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 exists
3112023-11-20T07:49:49.196Z0+0 records out
3122023-11-20T07:49:49.227ZZpool: [oxp_31bd71cd-4736-4a12-a387-9b74b050396f]
3132023-11-20T07:49:49.258Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3142023-11-20T07:49:49.289ZZFS vdev /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev exists
3152023-11-20T07:49:49.320Z+ success 'ZFS vdev /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3162023-11-20T07:49:49.351ZZFS zpool oxp_31bd71cd-4736-4a12-a387-9b74b050396f exists
3172023-11-20T07:49:49.382Z+ set +x
3182023-11-20T07:49:49.413ZZpool: [oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7]
3192023-11-20T07:49:49.444Z++ zpool list -o name
3202023-11-20T07:49:49.476ZZFS vdev /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev exists
3212023-11-20T07:49:49.507Z++ grep oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03
3222023-11-20T07:49:49.538ZZFS zpool oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 exists
3232023-11-20T07:49:49.569Z+ [[ -z '' ]]
3242023-11-20T07:49:49.600ZZpool: [oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b]
3252023-11-20T07:49:49.630Z+ zpool create -f oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3262023-11-20T07:49:49.662ZZFS vdev /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
3272023-11-20T07:49:49.693Z+ success 'ZFS zpool oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3282023-11-20T07:49:49.724ZZFS zpool oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b exists
3292023-11-20T07:49:49.756Z+ set +x
3302023-11-20T07:49:49.787ZZpool: [oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b]
3312023-11-20T07:49:49.820Z+ for ZPOOL in "${ZPOOLS[@]}"
3322023-11-20T07:49:49.851ZZFS vdev /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
3332023-11-20T07:49:49.882Z+ echo 'Zpool: [oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3342023-11-20T07:49:49.913ZZFS zpool oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b exists
3352023-11-20T07:49:49.944Z+ VDEV_PATH=/scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3362023-11-20T07:49:50.005ZSimnet net0/sc0_0 exists
3372023-11-20T07:49:50.036Z+ [[ -f /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3382023-11-20T07:49:50.067ZSimnet net1/sc1_0 exists
3392023-11-20T07:49:50.098Z+ dd if=/dev/zero of=/scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3402023-11-20T07:49:50.133ZVnic sc0_1 exists
3412023-11-20T07:49:50.170Z0+0 records in
3422023-11-20T07:49:50.204Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/softnpu/image/dec63e67156fe6e958991bbfa090629868115ab5/softnpu.sha256.txt
3432023-11-20T07:49:50.236Z0+0 records out
3442023-11-20T07:49:50.267Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3452023-11-20T07:49:50.298Z+ success 'ZFS vdev /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3462023-11-20T07:49:50.330Z+ set +x
3472023-11-20T07:49:50.362Z++ zpool list -o name
3482023-11-20T07:49:50.393Z++ grep oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03
3492023-11-20T07:49:50.424Z+ [[ -z '' ]]
3502023-11-20T07:49:50.455Z+ zpool create -f oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3512023-11-20T07:49:50.486Z+ success 'ZFS zpool oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3522023-11-20T07:49:50.517Z+ set +x
3532023-11-20T07:49:50.549Z+ for ZPOOL in "${ZPOOLS[@]}"
3542023-11-20T07:49:50.586Z+ echo 'Zpool: [oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3552023-11-20T07:49:50.621Z+ VDEV_PATH=/scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3562023-11-20T07:49:50.653Z+ [[ -f /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3572023-11-20T07:49:50.684Z+ dd if=/dev/zero of=/scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3582023-11-20T07:49:50.715Z0+0 records in
3592023-11-20T07:49:50.747Z0+0 records out
3602023-11-20T07:49:50.778Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3612023-11-20T07:49:50.809Z+ success 'ZFS vdev /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3622023-11-20T07:49:50.840Z+ set +x
3632023-11-20T07:49:50.871Z++ zpool list -o name
3642023-11-20T07:49:50.902Z++ grep oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03
3652023-11-20T07:49:50.934Z+ [[ -z '' ]]
3662023-11-20T07:49:50.965Z+ zpool create -f oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3672023-11-20T07:49:50.999Z+ success 'ZFS zpool oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3682023-11-20T07:49:51.045Z+ set +x
3692023-11-20T07:49:51.079Z+ for ZPOOL in "${ZPOOLS[@]}"
3702023-11-20T07:49:51.112Z+ echo 'Zpool: [oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3712023-11-20T07:49:51.143Z+ VDEV_PATH=/scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3722023-11-20T07:49:51.174Z+ [[ -f /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3732023-11-20T07:49:51.205Z+ dd if=/dev/zero of=/scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3742023-11-20T07:49:51.236Z0+0 records in
3752023-11-20T07:49:51.279Z0+0 records out
3762023-11-20T07:49:51.310Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3772023-11-20T07:49:51.341Z+ success 'ZFS vdev /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3782023-11-20T07:49:51.372Z+ set +x
3792023-11-20T07:49:51.403Z++ zpool list -o name
3802023-11-20T07:49:51.434Z++ grep oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03
3812023-11-20T07:49:51.469Z+ [[ -z '' ]]
3822023-11-20T07:49:51.515Z+ zpool create -f oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3832023-11-20T07:49:51.560Z+ success 'ZFS zpool oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3842023-11-20T07:49:51.591Z+ set +x
3852023-11-20T07:49:51.624Z+ for ZPOOL in "${ZPOOLS[@]}"
3862023-11-20T07:49:51.655Z+ echo 'Zpool: [oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930]'
3872023-11-20T07:49:51.687Z+ VDEV_PATH=/scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev
3882023-11-20T07:49:51.718Z+ [[ -f /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev ]]
3892023-11-20T07:49:51.749Z+ dd if=/dev/zero of=/scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev bs=1 count=0 seek=20G
3902023-11-20T07:49:51.781Z0+0 records in
3912023-11-20T07:49:51.812Z0+0 records out
3922023-11-20T07:49:51.843Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3932023-11-20T07:49:51.874Z+ success 'ZFS vdev /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev exists'
3942023-11-20T07:49:51.905Z+ set +x
3952023-11-20T07:49:51.944Z++ zpool list -o name
3962023-11-20T07:49:51.977Z++ grep oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930
3972023-11-20T07:49:52.009Z+ [[ -z '' ]]
3982023-11-20T07:49:52.040Z+ zpool create -f oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev
3992023-11-20T07:49:52.075Z+ success 'ZFS zpool oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 exists'
4002023-11-20T07:49:52.107Z+ set +x
4012023-11-20T07:49:52.138Z+ for ZPOOL in "${ZPOOLS[@]}"
4022023-11-20T07:49:52.169Z+ echo 'Zpool: [oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60]'
4032023-11-20T07:49:52.201Z+ VDEV_PATH=/scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev
4042023-11-20T07:49:52.231Z+ [[ -f /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev ]]
4052023-11-20T07:49:52.262Z+ dd if=/dev/zero of=/scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev bs=1 count=0 seek=20G
4062023-11-20T07:49:52.294Z0+0 records in
4072023-11-20T07:49:52.325Z0+0 records out
4082023-11-20T07:49:52.356Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4092023-11-20T07:49:52.387Z+ success 'ZFS vdev /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev exists'
4102023-11-20T07:49:52.418Z+ set +x
4112023-11-20T07:49:52.449Z++ zpool list -o name
4122023-11-20T07:49:52.480Z++ grep oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60
4132023-11-20T07:49:52.511Z+ [[ -z '' ]]
4142023-11-20T07:49:52.542Z+ zpool create -f oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev
4152023-11-20T07:49:52.574Z+ success 'ZFS zpool oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 exists'
4162023-11-20T07:49:52.605Z+ set +x
4172023-11-20T07:49:52.635Z+ for ZPOOL in "${ZPOOLS[@]}"
4182023-11-20T07:49:52.666Z+ echo 'Zpool: [oxp_31bd71cd-4736-4a12-a387-9b74b050396f]'
4192023-11-20T07:49:52.697Z+ VDEV_PATH=/scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev
4202023-11-20T07:49:52.729Z+ [[ -f /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev ]]
4212023-11-20T07:49:52.760Z+ dd if=/dev/zero of=/scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev bs=1 count=0 seek=20G
4222023-11-20T07:49:52.791Z0+0 records in
4232023-11-20T07:49:52.822Z0+0 records out
4242023-11-20T07:49:52.853Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4252023-11-20T07:49:52.884Z+ success 'ZFS vdev /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev exists'
4262023-11-20T07:49:52.915Z+ set +x
4272023-11-20T07:49:52.946Z++ zpool list -o name
4282023-11-20T07:49:52.977Z++ grep oxp_31bd71cd-4736-4a12-a387-9b74b050396f
4292023-11-20T07:49:53.009Z+ [[ -z '' ]]
4302023-11-20T07:49:53.039Z+ zpool create -f oxp_31bd71cd-4736-4a12-a387-9b74b050396f /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev
4312023-11-20T07:49:53.070Z+ success 'ZFS zpool oxp_31bd71cd-4736-4a12-a387-9b74b050396f exists'
4322023-11-20T07:49:53.104Z+ set +x
4332023-11-20T07:49:53.135Z+ for ZPOOL in "${ZPOOLS[@]}"
4342023-11-20T07:49:53.166Z+ echo 'Zpool: [oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7]'
4352023-11-20T07:49:53.197Z+ VDEV_PATH=/scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev
4362023-11-20T07:49:53.228Z+ [[ -f /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev ]]
4372023-11-20T07:49:53.259Z+ dd if=/dev/zero of=/scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev bs=1 count=0 seek=20G
4382023-11-20T07:49:53.323Z0+0 records in
4392023-11-20T07:49:53.354Z0+0 records out
4402023-11-20T07:49:53.385Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4412023-11-20T07:49:53.416Z+ success 'ZFS vdev /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev exists'
4422023-11-20T07:49:53.447Z+ set +x
4432023-11-20T07:49:53.478Z++ zpool list -o name
4442023-11-20T07:49:53.508Z++ grep oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7
4452023-11-20T07:49:53.539Z+ [[ -z '' ]]
4462023-11-20T07:49:53.572Z+ zpool create -f oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev
4472023-11-20T07:49:53.613Z+ success 'ZFS zpool oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 exists'
4482023-11-20T07:49:53.650Z+ set +x
4492023-11-20T07:49:53.687Z+ for ZPOOL_TYPE in "${ZPOOL_TYPES[@]}"
4502023-11-20T07:49:53.719Z+ readarray -t ZPOOLS
4512023-11-20T07:49:53.751Z++ grep '"oxi_' /opt/oxide/work/tools/../smf/sled-agent/non-gimlet/config.toml
4522023-11-20T07:49:53.781Z++ sed 's/[ ",]//g'
4532023-11-20T07:49:53.812Z+ for ZPOOL in "${ZPOOLS[@]}"
4542023-11-20T07:49:53.844Z+ echo 'Zpool: [oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b]'
4552023-11-20T07:49:53.875Z+ VDEV_PATH=/scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4562023-11-20T07:49:53.905Z+ [[ -f /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
4572023-11-20T07:49:53.937Z+ dd if=/dev/zero of=/scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
4582023-11-20T07:49:53.968Z0+0 records in
4592023-11-20T07:49:54.001Z0+0 records out
4602023-11-20T07:49:54.036Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
4612023-11-20T07:49:54.079Z+ success 'ZFS vdev /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
4622023-11-20T07:49:54.146Z+ set +x
4632023-11-20T07:49:54.183Z++ zpool list -o name
4642023-11-20T07:49:54.215Z++ grep oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b
4652023-11-20T07:49:54.246Z+ [[ -z '' ]]
4662023-11-20T07:49:54.277Z+ zpool create -f oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4672023-11-20T07:49:54.308Z+ success 'ZFS zpool oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
4682023-11-20T07:49:54.339Z+ set +x
4692023-11-20T07:49:54.372Z+ for ZPOOL in "${ZPOOLS[@]}"
4702023-11-20T07:49:54.413Z+ echo 'Zpool: [oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b]'
4712023-11-20T07:49:54.445Z+ VDEV_PATH=/scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4722023-11-20T07:49:54.476Z+ [[ -f /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
4732023-11-20T07:49:54.507Z+ dd if=/dev/zero of=/scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
4742023-11-20T07:49:54.538Z0+0 records in
4752023-11-20T07:49:54.568Z0+0 records out
4762023-11-20T07:49:54.608Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
4772023-11-20T07:49:54.647Z+ success 'ZFS vdev /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
4782023-11-20T07:49:54.681Z+ set +x
4792023-11-20T07:49:54.712Z++ zpool list -o name
4802023-11-20T07:49:54.744Z++ grep oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b
4812023-11-20T07:49:54.775Z+ [[ -z '' ]]
4822023-11-20T07:49:54.806Z+ zpool create -f oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4832023-11-20T07:49:54.837Z+ success 'ZFS zpool oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
4842023-11-20T07:49:54.868Z+ set +x
4852023-11-20T07:49:54.899Z+ [[ zone == \z\o\n\e ]]
4862023-11-20T07:49:54.930Z+ ensure_simulated_links igb0
4872023-11-20T07:49:54.961Z+ local PHYSICAL_LINK=igb0
4882023-11-20T07:49:54.992Z+ INDICES=("0" "1")
4892023-11-20T07:49:55.023Z+ for I in "${INDICES[@]}"
4902023-11-20T07:49:55.061Z++ get_simnet_name_if_exists net0
4912023-11-20T07:49:55.097Z++ dladm show-simnet -p -o LINK net0
4922023-11-20T07:49:55.129Z++ echo ''
4932023-11-20T07:49:55.170Z+ [[ -z '' ]]
4942023-11-20T07:49:55.201Z+ dladm create-simnet -t net0
4952023-11-20T07:49:55.232Z+ dladm create-simnet -t sc0_0
4962023-11-20T07:49:55.263Z+ dladm modify-simnet -t -p net0 sc0_0
4972023-11-20T07:49:55.295Z+ dladm set-linkprop -p mtu=9000 sc0_0
4982023-11-20T07:49:55.325Z+ success 'Simnet net0/sc0_0 exists'
4992023-11-20T07:49:55.356Z+ set +x
5002023-11-20T07:49:55.387Z+ for I in "${INDICES[@]}"
5012023-11-20T07:49:55.418Z++ get_simnet_name_if_exists net1
5022023-11-20T07:49:55.450Zdownloading softnpu ...done
5032023-11-20T07:49:55.486Z++ dladm show-simnet -p -o LINK net1
5042023-11-20T07:49:55.523Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/sidecar-lite/release/f0585a29fb0285f7a1220c1118856b0e5c1f75c5/libsidecar_lite.so.sha256.txt
5052023-11-20T07:49:55.555Z++ echo ''
5062023-11-20T07:49:55.587Z+ [[ -z '' ]]
5072023-11-20T07:49:55.618Z+ dladm create-simnet -t net1
5082023-11-20T07:49:55.649Z+ dladm create-simnet -t sc1_0
5092023-11-20T07:49:55.680Z+ dladm modify-simnet -t -p net1 sc1_0
5102023-11-20T07:49:55.711Z+ dladm set-linkprop -p mtu=9000 sc1_0
5112023-11-20T07:49:55.742Z+ success 'Simnet net1/sc1_0 exists'
5122023-11-20T07:49:55.777Z+ set +x
5132023-11-20T07:49:55.809Z++ get_vnic_name_if_exists sc0_1
5142023-11-20T07:49:55.841Z++ dladm show-vnic -p -o LINK sc0_1
5152023-11-20T07:49:55.872Z++ echo ''
5162023-11-20T07:49:55.903Z+ [[ -z '' ]]
5172023-11-20T07:49:55.939Z+ dladm create-vnic -t sc0_1 -l igb0 -m a8:e1:de:01:70:1d
5182023-11-20T07:49:55.979Z+ [[ -v PROMISC_FILT_OFF ]]
5192023-11-20T07:49:56.012Z+ success 'Vnic sc0_1 exists'
5202023-11-20T07:49:56.043Z+ set +x
5212023-11-20T07:49:56.075Z+ warn_if_no_proxy_arp
5222023-11-20T07:49:56.106Z+ [[ -v PXA_START ]]
5232023-11-20T07:49:56.137Z+ [[ -v PXA_END ]]
5242023-11-20T07:49:56.197Z+ ensure_softnpu_zone
5252023-11-20T07:49:56.288Z+ zoneadm list
5262023-11-20T07:49:56.320Z+ grep -q sidecar_softnpu
5272023-11-20T07:49:56.350Z+ [[ -f out/npuzone/npuzone ]]
5282023-11-20T07:49:56.386Z+ out/npuzone/npuzone create sidecar --omicron-zone --ports sc0_0,tfportrear0_0 --ports sc0_1,tfportqsfp0_0 --sidecar-lite-commit f0585a29fb0285f7a1220c1118856b0e5c1f75c5 --softnpu-commit dec63e67156fe6e958991bbfa090629868115ab5
5292023-11-20T07:49:57.960Zdownloading asic_program.so ...done
5302023-11-20T07:49:57.991Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/sidecar-lite/release/f0585a29fb0285f7a1220c1118856b0e5c1f75c5/scadm.sha256.txt
5312023-11-20T07:50:04.863Zdownloading scadm ...done
5322023-11-20T07:50:04.894Zcreating port tfportrear0_0 for sc0_0
5332023-11-20T07:50:04.926Zcreating port tfportqsfp0_0 for sc0_1
5342023-11-20T07:50:04.958Zsoftnpu zone setup
5352023-11-20T07:50:04.990Zconfigure zone
5362023-11-20T07:50:05.022Zinstall zone
5372023-11-20T07:50:05.963Zboot zone
5382023-11-20T07:50:09.071Z[sidecar_softnpu] svcs milestone/network
5392023-11-20T07:50:09.102ZSTATE STIME FMRI
5402023-11-20T07:50:09.133Zonline 7:50:07 svc:/milestone/network:default
5412023-11-20T07:50:09.165Z[sidecar_softnpu] /softnpu/npu start
5422023-11-20T07:50:09.196Z
5432023-11-20T07:50:09.227Z+ /opt/oxide/work/tools/scrimlet/softnpu-init.sh
5442023-11-20T07:50:09.258Z+ GATEWAY_IP=10.151.2.100
5452023-11-20T07:50:09.290Z+ echo 'Using 10.151.2.100 as gateway ip'
5462023-11-20T07:50:09.321ZUsing 10.151.2.100 as gateway ip
5472023-11-20T07:50:09.352Z+ [[ ! -v GATEWAY_MAC ]]
5482023-11-20T07:50:09.384Z+ GATEWAY_MAC=18:c0:4d:0d:a0:2a
5492023-11-20T07:50:09.415Z++ grep -c -E '^([0-9a-fA-F]{2}:){5}[0-9a-fA-F]{2}$' /dev/fd/63
5502023-11-20T07:50:09.446Z+++ echo 18:c0:4d:0d:a0:2a
5512023-11-20T07:50:09.478Z+ COUNT=1
5522023-11-20T07:50:09.509Z+ [[ 1 -ne 1 ]]
5532023-11-20T07:50:09.540Z+ echo 'Using 18:c0:4d:0d:a0:2a as gateway mac'
5542023-11-20T07:50:09.571ZUsing 18:c0:4d:0d:a0:2a as gateway mac
5552023-11-20T07:50:09.611Z+ z_scadm add-arp-entry 10.151.2.100 18:c0:4d:0d:a0:2a
5562023-11-20T07:50:09.643Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone add-arp-entry 10.151.2.100 18:c0:4d:0d:a0:2a
5572023-11-20T07:50:09.674Z+ PXA_MAC=a8:e1:de:01:70:1d
5582023-11-20T07:50:09.706Z+ [[ -v PXA_START ]]
5592023-11-20T07:50:09.738Z+ z_scadm add-proxy-arp 10.151.2.164 10.151.2.227 a8:e1:de:01:70:1d
5602023-11-20T07:50:09.769Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone add-proxy-arp 10.151.2.164 10.151.2.227 a8:e1:de:01:70:1d
5612023-11-20T07:50:09.800Z+ z_scadm dump-state
5622023-11-20T07:50:09.832Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone dump-state
5632023-11-20T07:50:09.863Zlocal v6:
5642023-11-20T07:50:09.894Zlocal v4:
5652023-11-20T07:50:09.926Zrouter v6:
5662023-11-20T07:50:09.958Zrouter v4_idx:
5672023-11-20T07:50:09.990Zrouter v4_routes:
5682023-11-20T07:50:10.021Zresolver v4:
5692023-11-20T07:50:10.052Z10.151.2.100 -> 18:c0:4d:0d:a0:2a
5702023-11-20T07:50:10.083Zresolver v6:
5712023-11-20T07:50:10.115Znat v4:
5722023-11-20T07:50:10.146Znat_v6:
5732023-11-20T07:50:10.177Zport_mac:
5742023-11-20T07:50:10.209Zproxy arp:
5752023-11-20T07:50:10.240Z10.151.2.164/10.151.2.227: a8:e1:de:01:70:1d
5762023-11-20T07:50:10.271Z+ success 'softnpu zone exists'
5772023-11-20T07:50:10.303Z+ set +x
5782023-11-20T07:50:10.335Zsoftnpu zone exists
5792023-11-20T07:50:10.368Z
5802023-11-20T07:50:10.399Zreal 24.025657578
5812023-11-20T07:50:10.430Zuser 0.725295845
5822023-11-20T07:50:10.461Zsys 1.603046024
5832023-11-20T07:50:10.493Ztrap 0.000072795
5842023-11-20T07:50:10.525Ztflt 0.000100976
5852023-11-20T07:50:10.556Zdflt 0.001295307
5862023-11-20T07:50:10.587Zkflt 0.000000000
5872023-11-20T07:50:10.641Zlock 18:33.397868965
5882023-11-20T07:50:10.672Zslp 1:11.947761914
5892023-11-20T07:50:10.704Zlat 0.058177535
5902023-11-20T07:50:10.735Zstop 0.365835391
5912023-11-20T07:50:10.766Z+ tar xf out/omicron-sled-agent.tar pkg/config-rss.toml
5922023-11-20T07:50:10.797Z++ sed -n 's/silo_name = "\(.*\)"/\1/p' pkg/config-rss.toml
5932023-11-20T07:50:10.828Z+ SILO_NAME=recovery
5942023-11-20T07:50:10.859Z++ sed -n 's/external_dns_zone_name = "\(.*\)"/\1/p' pkg/config-rss.toml
5952023-11-20T07:50:10.890Z+ EXTERNAL_DNS_DOMAIN=oxide.test
5962023-11-20T07:50:10.920Z+ sed -i~ '
5972023-11-20T07:50:10.951Z /^external_dns_ips/c\
5982023-11-20T07:50:10.983Zexternal_dns_ips = [ "10.151.2.164", "10.151.2.165" ]
5992023-11-20T07:50:11.014Z /^\[\[internal_services_ip_pool_ranges/,/^$/ {
6002023-11-20T07:50:11.045Z /^first/c\
6012023-11-20T07:50:11.076Zfirst = "10.151.2.164"
6022023-11-20T07:50:11.107Z /^last/c\
6032023-11-20T07:50:11.137Zlast = "10.151.2.173"
6042023-11-20T07:50:11.169Z }
6052023-11-20T07:50:11.199Z /^\[rack_network_config/,/^$/ {
6062023-11-20T07:50:11.231Z /^infra_ip_first/c\
6072023-11-20T07:50:11.323Zinfra_ip_first = "10.151.2.174"
6082023-11-20T07:50:11.354Z /^infra_ip_last/c\
6092023-11-20T07:50:11.386Zinfra_ip_last = "10.151.2.174"
6102023-11-20T07:50:11.417Z--- pkg/config-rss.toml~ Mon Jul 24 01:21:28 2006
6112023-11-20T07:50:11.448Z }
6122023-11-20T07:50:11.479Z+++ pkg/config-rss.toml Mon Nov 20 07:50:09 2023
6132023-11-20T07:50:11.511Z /^\[\[rack_network_config.ports/,/^$/ {
6142023-11-20T07:50:11.543Z@@ -41,7 +41,7 @@
6152023-11-20T07:50:11.574Z /^routes/c\
6162023-11-20T07:50:11.605Z # the DNS domain delegated to the rack by the customer. Each of these addresses
6172023-11-20T07:50:11.636Zroutes = \[{nexthop = "10.151.2.100", destination = "0.0.0.0/0"}\]
6182023-11-20T07:50:11.671Z # must be contained in one of the "internal services" IP Pool ranges listed
6192023-11-20T07:50:11.702Z /^addresses/c\
6202023-11-20T07:50:11.733Z # below.
6212023-11-20T07:50:11.764Zaddresses = \["10.151.2.174/32"\]
6222023-11-20T07:50:11.796Z-external_dns_ips = [ "192.168.1.20", "192.168.1.21" ]
6232023-11-20T07:50:11.827Z }
6242023-11-20T07:50:11.858Z+external_dns_ips = [ "10.151.2.164", "10.151.2.165" ]
6252023-11-20T07:50:11.889Z' pkg/config-rss.toml
6262023-11-20T07:50:11.920Z
6272023-11-20T07:50:11.951Z+ diff -u pkg/config-rss.toml~ pkg/config-rss.toml
6282023-11-20T07:50:11.982Z # Initial TLS certificates for the external API
6292023-11-20T07:50:12.013Z+ true
6302023-11-20T07:50:12.044Z #
6312023-11-20T07:50:12.076Z+ tar rvf out/omicron-sled-agent.tar pkg/config-rss.toml
6322023-11-20T07:50:12.107Z@@ -77,8 +77,8 @@
6332023-11-20T07:50:12.138Z+ rm -f pkg/config-rss.toml pkg/config-rss.toml~
6342023-11-20T07:50:12.169Z #
6352023-11-20T07:50:12.200Z+ TLS_NAME=recovery.sys.oxide.test
6362023-11-20T07:50:12.231Z # For more on this and what to put here, see docs/how-to-run.adoc.
6372023-11-20T07:50:12.262Z+ openssl req -newkey rsa:4096 -x509 -sha256 -days 3 -nodes -out pkg/initial-tls-cert.pem -keyout pkg/initial-tls-key.pem -subj /CN=recovery.sys.oxide.test -addext subjectAltName=DNS:recovery.sys.oxide.test -addext basicConstraints=critical,CA:FALSE -config /dev/stdin
6382023-11-20T07:50:12.294Z [[internal_services_ip_pool_ranges]]
6392023-11-20T07:50:12.325Z.+...+..+............+.+..+....+.....+....+.....+.......+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*...+..+...+......+.............+.....+.+.....+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.+......+...+..+...+................+......+..+..........+...+..+.....................+....+......+..+.........+.........+.........+...............+............+.+..+......+.+........+......+.+.......................+..........+.....+.+..+....+......+...+.........+.................+....+..+...+....+...+............+...+..............+...+...+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
6402023-11-20T07:50:12.356Z-first = "192.168.1.20"
6412023-11-20T07:50:12.387Z.+..+......+...+.+...+..+.+...+............+..+...............+......+............+....+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.....+.+..+...+................+.....+.+..+......+....+...+.....+....+.....+....+.....+............+....+........+.+.........+...+.........+...+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.+.....+....+...........+.......+......+..............+.+..+.......+........+...+.........+......+..........+..+....+.....+...+......+..........+...........+.............+..+...+..........+...+...+..+.............+............+...+..+....+...+.....+.........................+...+..+......................+...........+....+...+..+.........+......+.........+.........+..........+..+..................+...+............+...+...+....+.....+...+....+.....+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
6422023-11-20T07:50:12.418Z-last = "192.168.1.29"
6432023-11-20T07:50:12.449Z-----
6442023-11-20T07:50:12.481Z+first = "10.151.2.164"
6452023-11-20T07:50:12.512Z+ tar rvf out/omicron-sled-agent.tar pkg/initial-tls-cert.pem pkg/initial-tls-key.pem
6462023-11-20T07:50:12.543Z+last = "10.151.2.173"
6472023-11-20T07:50:12.574Z+ rm -f pkg/initial-tls-cert.pem pkg/initial-tls-key.pem
6482023-11-20T07:50:12.606Z
6492023-11-20T07:50:12.637Z+ rmdir pkg
6502023-11-20T07:50:12.668Z # TODO - this configuration is subject to change going forward. Ultimately these
6512023-11-20T07:50:12.701Z+ E2E_TLS_CERT=/opt/oxide/sled-agent/pkg/initial-tls-cert.pem
6522023-11-20T07:50:12.732Z # parameters should be provided to the control plane via wicket, but we need to
6532023-11-20T07:50:12.763Z+ pfexec mkdir -p /usr/oxide
6542023-11-20T07:50:12.794Z@@ -92,8 +92,8 @@
6552023-11-20T07:50:12.825Z+ pfexec rm -f /usr/oxide/tcpproxy
6562023-11-20T07:50:12.856Z # A range of IP addresses used by Boundary Services on the external network. In
6572023-11-20T07:50:12.887Z+ pfexec curl -sSfL -o /usr/oxide/tcpproxy http://catacomb.eng.oxide.computer:12346/tcpproxy
6582023-11-20T07:50:12.918Z # a real system, these would be addresses of the uplink ports on the Sidecar.
6592023-11-20T07:50:12.949Z+ pfexec chmod +x /usr/oxide/tcpproxy
6602023-11-20T07:50:12.980Z # With softnpu, only one address is used.
6612023-11-20T07:50:13.011Z+ pfexec rm -f /var/svc/manifest/site/tcpproxy.xml
6622023-11-20T07:50:13.042Z-infra_ip_first = "192.168.1.30"
6632023-11-20T07:50:13.073Z+ pfexec curl -sSfL -o /var/svc/manifest/site/tcpproxy.xml http://catacomb.eng.oxide.computer:12346/tcpproxy.xml
6642023-11-20T07:50:13.104Z-infra_ip_last = "192.168.1.30"
6652023-11-20T07:50:13.135Z+ pfexec svccfg import /var/svc/manifest/site/tcpproxy.xml
6662023-11-20T07:50:13.166Z+infra_ip_first = "10.151.2.174"
6672023-11-20T07:50:13.197Z+ OMICRON_NO_UNINSTALL=1
6682023-11-20T07:50:13.229Z+infra_ip_last = "10.151.2.174"
6692023-11-20T07:50:13.292Z+ ptime -m pfexec ./target/release/omicron-package -t test install
6702023-11-20T07:50:13.323Z
6712023-11-20T07:50:13.428ZNov 20 07:50:10.502 DEBG target[test]: Target({"image": "standard", "machine": "non-gimlet", "rack-topology": "single-sled", "switch": "softnpu"})
6722023-11-20T07:50:13.459Z # Configurations for BGP routers to run on the scrimlets.
6732023-11-20T07:50:13.490ZNov 20 07:50:10.505 INFO Installing service, dst: /opt/oxide/clickhouse.tar.gz, src: out/clickhouse.tar.gz
6742023-11-20T07:50:13.521Z bgp = []
6752023-11-20T07:50:13.552ZNov 20 07:50:10.505 INFO Installing service, dst: /opt/oxide/mg-ddm.tar, src: out/mg-ddm-gz.tar
6762023-11-20T07:50:13.584Z@@ -101,9 +101,9 @@
6772023-11-20T07:50:13.614ZNov 20 07:50:10.505 INFO Installing service, dst: /opt/oxide/oximeter.tar.gz, src: out/oximeter-collector.tar.gz
6782023-11-20T07:50:13.645Z # You can configure multiple uplinks by repeating the following stanza
6792023-11-20T07:50:13.677ZNov 20 07:50:10.505 INFO Installing service, dst: /opt/oxide/crucible.tar.gz, src: out/crucible.tar.gz
6802023-11-20T07:50:13.731Z [[rack_network_config.ports]]
6812023-11-20T07:50:13.762ZNov 20 07:50:10.505 INFO Installing service, dst: /opt/oxide/sled-agent.tar, src: out/omicron-sled-agent.tar
6822023-11-20T07:50:13.793Z # Routes associated with this port.
6832023-11-20T07:50:13.825ZNov 20 07:50:10.505 INFO Installing service, dst: /opt/oxide/internal_dns.tar.gz, src: out/internal-dns.tar.gz
6842023-11-20T07:50:13.856Z-routes = [{nexthop = "192.168.1.199", destination = "0.0.0.0/0"}]
6852023-11-20T07:50:13.887ZNov 20 07:50:10.505 INFO Installing service, dst: /opt/oxide/mgs.tar.gz, src: out/omicron-gateway-softnpu.tar.gz
6862023-11-20T07:50:13.918Z+routes = [{nexthop = "10.151.2.100", destination = "0.0.0.0/0"}]
6872023-11-20T07:50:13.950ZNov 20 07:50:10.505 INFO Installing service, dst: /opt/oxide/cockroachdb.tar.gz, src: out/cockroachdb.tar.gz
6882023-11-20T07:50:13.981Z # Addresses associated with this port.
6892023-11-20T07:50:14.012ZNov 20 07:50:10.505 INFO Installing service, dst: /opt/oxide/clickhouse_keeper.tar.gz, src: out/clickhouse_keeper.tar.gz
6902023-11-20T07:50:14.043Z-addresses = ["192.168.1.30/32"]
6912023-11-20T07:50:14.074ZNov 20 07:50:10.505 INFO Installing service, dst: /opt/oxide/propolis-server.tar.gz, src: out/propolis-server.tar.gz
6922023-11-20T07:50:14.105Z+addresses = ["10.151.2.174/32"]
6932023-11-20T07:50:14.136ZNov 20 07:50:10.505 INFO Installing service, dst: /opt/oxide/external_dns.tar.gz, src: out/external-dns.tar.gz
6942023-11-20T07:50:14.167Z # Name of the uplink port. This should always be "qsfp0" when using softnpu.
6952023-11-20T07:50:14.198ZNov 20 07:50:10.505 INFO Installing service, dst: /opt/oxide/crucible_pantry.tar.gz, src: out/crucible-pantry.tar.gz
6962023-11-20T07:50:14.229Z port = "qsfp0"
6972023-11-20T07:50:14.261ZNov 20 07:50:10.505 INFO Installing service, dst: /opt/oxide/switch.tar.gz, src: out/switch-softnpu.tar.gz
6982023-11-20T07:50:14.293Z # The speed of this port.
6992023-11-20T07:50:14.324ZNov 20 07:50:10.505 INFO Installing service, dst: /opt/oxide/overlay.tar.gz, src: out/overlay.tar.gz
7002023-11-20T07:50:14.355Za pkg/config-rss.toml 7K
7012023-11-20T07:50:14.386ZNov 20 07:50:10.505 INFO Installing service, dst: /opt/oxide/ntp.tar.gz, src: out/ntp.tar.gz
7022023-11-20T07:50:14.417Za pkg/initial-tls-cert.pem 2K
7032023-11-20T07:50:14.449ZNov 20 07:50:10.505 INFO Installing service, dst: /opt/oxide/nexus.tar.gz, src: out/omicron-nexus.tar.gz
7042023-11-20T07:50:14.480Za pkg/initial-tls-key.pem 4K
7052023-11-20T07:50:14.511ZNov 20 07:50:10.742 INFO Unpacking service tarball, service_path: /opt/oxide/mg-ddm, tar_path: /opt/oxide/mg-ddm.tar
7062023-11-20T07:50:14.542ZNov 20 07:50:10.776 INFO Unpacking service tarball, service_path: /opt/oxide/sled-agent, tar_path: /opt/oxide/sled-agent.tar
7072023-11-20T07:50:14.573ZNov 20 07:50:10.837 INFO Installing bootstrap service from /opt/oxide/sled-agent/pkg/manifest.xml
7082023-11-20T07:50:14.604Z
7092023-11-20T07:50:14.635Zreal 0.469286654
7102023-11-20T07:50:14.666Zuser 0.096605449
7112023-11-20T07:50:14.698Zsys 1.918389054
7122023-11-20T07:50:14.729Ztrap 0.000000650
7132023-11-20T07:50:14.761Ztflt 0.000579428
7142023-11-20T07:50:14.792Zdflt 0.000254209
7152023-11-20T07:50:14.823Zkflt 0.000000000
7162023-11-20T07:50:14.854Zlock 42.136990454
7172023-11-20T07:50:14.885Zslp 0.670596767
7182023-11-20T07:50:14.916Zlat 0.002806116
7192023-11-20T07:50:14.947Zstop 0.203937942
7202023-11-20T07:50:14.978Z+ retry=0
7212023-11-20T07:50:15.011Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7222023-11-20T07:50:15.042Z+ [[ 0 -gt 30 ]]
7232023-11-20T07:50:15.073Z+ sleep 1
7242023-11-20T07:50:15.104Z+ retry=1
7252023-11-20T07:50:15.135Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7262023-11-20T07:50:15.166Z+ [[ 1 -gt 30 ]]
7272023-11-20T07:50:15.197Z+ sleep 1
7282023-11-20T07:50:15.228Z+ retry=2
7292023-11-20T07:50:15.259Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7302023-11-20T07:50:15.290Z+ [[ 2 -gt 30 ]]
7312023-11-20T07:50:15.321Z+ sleep 1
7322023-11-20T07:50:15.352Z+ retry=3
7332023-11-20T07:50:15.383Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7342023-11-20T07:50:15.417Z+ [[ 3 -gt 30 ]]
7352023-11-20T07:50:15.448Z+ sleep 1
7362023-11-20T07:50:15.479Z+ retry=4
7372023-11-20T07:50:15.513Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7382023-11-20T07:50:34.871Z+ [[ 4 -gt 30 ]]
7392023-11-20T07:50:34.903Z+ sleep 1
7402023-11-20T07:50:35.875Z+ retry=5
7412023-11-20T07:50:35.906Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7422023-11-20T07:50:35.939Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone dump-state
7432023-11-20T07:50:35.970Zlocal v6:
7442023-11-20T07:50:36.001Zfe80::f858:c2ff:fe1d:5103
7452023-11-20T07:50:36.032Zfd00:99::1
7462023-11-20T07:50:36.064Zlocal v4:
7472023-11-20T07:50:36.095Z10.151.2.174
7482023-11-20T07:50:36.126Zrouter v6:
7492023-11-20T07:50:36.159Zrouter v4_idx:
7502023-11-20T07:50:36.190Z0.0.0.0/0 -> 0
7512023-11-20T07:50:36.222Zrouter v4_routes:
7522023-11-20T07:50:36.253Z0 -> 10.151.2.100 (2)
7532023-11-20T07:50:36.284Zresolver v4:
7542023-11-20T07:50:36.317Z10.151.2.100 -> 18:c0:4d:0d:a0:2a
7552023-11-20T07:50:36.348Zresolver v6:
7562023-11-20T07:50:36.387Zfe80::f858:c2ff:fe1d:5103 -> fa:58:c2:1d:51:03
7572023-11-20T07:50:36.418Zfe80::704d:2ff:fe20:a090 -> 72:4d:02:20:a0:90
7582023-11-20T07:50:36.449Znat v4:
7592023-11-20T07:50:36.480Znat_v6:
7602023-11-20T07:50:36.512Zport_mac:
7612023-11-20T07:50:36.543Z1000: a8:40:25:1d:2e:e9
7622023-11-20T07:50:36.574Z1: a8:40:25:1d:2e:ea
7632023-11-20T07:50:36.605Zproxy arp:
7642023-11-20T07:50:36.636Z10.151.2.164/10.151.2.227: a8:e1:de:01:70:1d
7652023-11-20T07:50:36.667Z+ retry=0
7662023-11-20T07:50:36.699Z+++ zoneadm list -n
7672023-11-20T07:50:36.730Z+++ grep oxz_ntp
7682023-11-20T07:50:36.761Z++ pfexec svcs -z -Hostate oxide/ntp
7692023-11-20T07:50:36.792Zsvcs: zone '-Hostate': entity not found
7702023-11-20T07:50:36.823Z++ true
7712023-11-20T07:50:36.854Z+ [[ '' != online ]]
7722023-11-20T07:50:36.885Z+ [[ 0 -gt 60 ]]
7732023-11-20T07:50:36.916Z+ sleep 1
7742023-11-20T07:50:36.948Z+ retry=1
7752023-11-20T07:50:36.979Z+++ zoneadm list -n
7762023-11-20T07:50:37.010Z+++ grep oxz_ntp
7772023-11-20T07:50:37.041Z++ pfexec svcs -z -Hostate oxide/ntp
7782023-11-20T07:50:37.072Zsvcs: zone '-Hostate': entity not found
7792023-11-20T07:50:37.103Z++ true
7802023-11-20T07:50:37.136Z+ [[ '' != online ]]
7812023-11-20T07:50:37.167Z+ [[ 1 -gt 60 ]]
7822023-11-20T07:50:37.198Z+ sleep 1
7832023-11-20T07:50:37.960Z+ retry=2
7842023-11-20T07:50:37.993Z+++ zoneadm list -n
7852023-11-20T07:50:38.025Z+++ grep oxz_ntp
7862023-11-20T07:50:38.057Z++ pfexec svcs -z oxz_ntp_505fd4f2-4a1e-4353-87c4-9b9b02e962ea -Hostate oxide/ntp
7872023-11-20T07:50:38.090Zsvcs: Could not bind to repository server for zone oxz_ntp_505fd4f2-4a1e-4353-87c4-9b9b02e962ea: repository server unavailable
7882023-11-20T07:50:38.122Z++ true
7892023-11-20T07:50:38.153Z+ [[ '' != online ]]
7902023-11-20T07:50:38.184Z+ [[ 2 -gt 60 ]]
7912023-11-20T07:50:38.215Z+ sleep 1
7922023-11-20T07:50:38.982Z+ retry=3
7932023-11-20T07:50:39.014Z+++ zoneadm list -n
7942023-11-20T07:50:39.045Z+++ grep oxz_ntp
7952023-11-20T07:50:39.077Z++ pfexec svcs -z oxz_ntp_505fd4f2-4a1e-4353-87c4-9b9b02e962ea -Hostate oxide/ntp
7962023-11-20T07:50:39.108Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
7972023-11-20T07:50:39.139Z++ true
7982023-11-20T07:50:39.171Z+ [[ '' != online ]]
7992023-11-20T07:50:39.202Z+ [[ 3 -gt 60 ]]
8002023-11-20T07:50:39.233Z+ sleep 1
8012023-11-20T07:50:40.010Z+ retry=4
8022023-11-20T07:50:40.041Z+++ zoneadm list -n
8032023-11-20T07:50:40.072Z+++ grep oxz_ntp
8042023-11-20T07:50:40.103Z++ pfexec svcs -z oxz_ntp_505fd4f2-4a1e-4353-87c4-9b9b02e962ea -Hostate oxide/ntp
8052023-11-20T07:50:40.134Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
8062023-11-20T07:50:40.165Z++ true
8072023-11-20T07:50:40.196Z+ [[ '' != online ]]
8082023-11-20T07:50:40.227Z+ [[ 4 -gt 60 ]]
8092023-11-20T07:50:40.259Z+ sleep 1
8102023-11-20T07:50:41.040Z+ retry=5
8112023-11-20T07:50:41.072Z+++ zoneadm list -n
8122023-11-20T07:50:41.104Z+++ grep oxz_ntp
8132023-11-20T07:50:41.135Z++ pfexec svcs -z oxz_ntp_505fd4f2-4a1e-4353-87c4-9b9b02e962ea -Hostate oxide/ntp
8142023-11-20T07:50:41.166Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
8152023-11-20T07:50:41.197Z++ true
8162023-11-20T07:50:41.228Z+ [[ '' != online ]]
8172023-11-20T07:50:41.260Z+ [[ 5 -gt 60 ]]
8182023-11-20T07:50:41.291Z+ sleep 1
8192023-11-20T07:50:42.069Z+ retry=6
8202023-11-20T07:50:42.100Z+++ zoneadm list -n
8212023-11-20T07:50:42.132Z+++ grep oxz_ntp
8222023-11-20T07:50:42.166Z++ pfexec svcs -z oxz_ntp_505fd4f2-4a1e-4353-87c4-9b9b02e962ea -Hostate oxide/ntp
8232023-11-20T07:50:42.197Z+ [[ disabled != online ]]
8242023-11-20T07:50:42.228Z+ [[ 6 -gt 60 ]]
8252023-11-20T07:50:42.259Z+ sleep 1
8262023-11-20T07:50:43.093Z+ retry=7
8272023-11-20T07:50:43.125Z+++ zoneadm list -n
8282023-11-20T07:50:43.156Z+++ grep oxz_ntp
8292023-11-20T07:50:43.187Z++ pfexec svcs -z oxz_ntp_505fd4f2-4a1e-4353-87c4-9b9b02e962ea -Hostate oxide/ntp
8302023-11-20T07:50:43.218Z+ [[ disabled != online ]]
8312023-11-20T07:50:43.249Z+ [[ 7 -gt 60 ]]
8322023-11-20T07:50:43.280Z+ sleep 1
8332023-11-20T07:50:44.116Z+ retry=8
8342023-11-20T07:50:44.147Z+++ zoneadm list -n
8352023-11-20T07:50:44.179Z+++ grep oxz_ntp
8362023-11-20T07:50:44.210Z++ pfexec svcs -z oxz_ntp_505fd4f2-4a1e-4353-87c4-9b9b02e962ea -Hostate oxide/ntp
8372023-11-20T07:50:44.242Z+ [[ disabled != online ]]
8382023-11-20T07:50:44.273Z+ [[ 8 -gt 60 ]]
8392023-11-20T07:50:44.304Z+ sleep 1
8402023-11-20T07:50:45.139Z+ retry=9
8412023-11-20T07:50:45.170Z+++ zoneadm list -n
8422023-11-20T07:50:45.201Z+++ grep oxz_ntp
8432023-11-20T07:50:45.232Z++ pfexec svcs -z oxz_ntp_505fd4f2-4a1e-4353-87c4-9b9b02e962ea -Hostate oxide/ntp
8442023-11-20T07:50:45.263Z+ [[ disabled != online ]]
8452023-11-20T07:50:45.294Z+ [[ 9 -gt 60 ]]
8462023-11-20T07:50:45.325Z+ sleep 1
8472023-11-20T07:50:46.162Z+ retry=10
8482023-11-20T07:50:46.193Z+++ zoneadm list -n
8492023-11-20T07:50:46.224Z+++ grep oxz_ntp
8502023-11-20T07:50:46.255Z++ pfexec svcs -z oxz_ntp_505fd4f2-4a1e-4353-87c4-9b9b02e962ea -Hostate oxide/ntp
8512023-11-20T07:50:46.287Z+ [[ disabled != online ]]
8522023-11-20T07:50:46.318Z+ [[ 10 -gt 60 ]]
8532023-11-20T07:50:46.350Z+ sleep 1
8542023-11-20T07:50:47.183Z+ retry=11
8552023-11-20T07:50:47.214Z+++ zoneadm list -n
8562023-11-20T07:50:47.245Z+++ grep oxz_ntp
8572023-11-20T07:50:47.276Z++ pfexec svcs -z oxz_ntp_505fd4f2-4a1e-4353-87c4-9b9b02e962ea -Hostate oxide/ntp
8582023-11-20T07:50:47.307Z+ [[ disabled != online ]]
8592023-11-20T07:50:47.339Z+ [[ 11 -gt 60 ]]
8602023-11-20T07:50:47.370Z+ sleep 1
8612023-11-20T07:50:48.205Z+ retry=12
8622023-11-20T07:50:48.236Z+++ zoneadm list -n
8632023-11-20T07:50:48.300Z+++ grep oxz_ntp
8642023-11-20T07:50:48.331Z++ pfexec svcs -z oxz_ntp_505fd4f2-4a1e-4353-87c4-9b9b02e962ea -Hostate oxide/ntp
8652023-11-20T07:50:48.362Z+ [[ disabled != online ]]
8662023-11-20T07:50:48.393Z+ [[ 12 -gt 60 ]]
8672023-11-20T07:50:48.424Z+ sleep 1
8682023-11-20T07:50:49.227Z+ retry=13
8692023-11-20T07:50:49.258Z+++ zoneadm list -n
8702023-11-20T07:50:49.289Z+++ grep oxz_ntp
8712023-11-20T07:50:49.320Z++ pfexec svcs -z oxz_ntp_505fd4f2-4a1e-4353-87c4-9b9b02e962ea -Hostate oxide/ntp
8722023-11-20T07:50:49.352Z+ [[ disabled != online ]]
8732023-11-20T07:50:49.383Z+ [[ 13 -gt 60 ]]
 
8742023-11-20T07:50:49.399Zsock console login: Nov 20 07:50:49 sock ip: ip: joining multicasts failed (4) on vopte0 - will use link layer broadcasts for multicast
 
8752023-11-20T07:50:49.414Z+ sleep 1
8762023-11-20T07:50:50.248Z+ retry=14
8772023-11-20T07:50:50.279Z+++ zoneadm list -n
8782023-11-20T07:50:50.310Z+++ grep oxz_ntp
8792023-11-20T07:50:50.341Z++ pfexec svcs -z oxz_ntp_505fd4f2-4a1e-4353-87c4-9b9b02e962ea -Hostate oxide/ntp
8802023-11-20T07:50:50.372Z+ [[ online != online ]]
8812023-11-20T07:50:50.403Z+ echo 'Waited for chrony: 14s'
8822023-11-20T07:50:50.434ZWaited for chrony: 14s
8832023-11-20T07:50:50.465Z+ export RUST_BACKTRACE=1
8842023-11-20T07:50:50.496Z+ RUST_BACKTRACE=1
8852023-11-20T07:50:50.527Z+ export E2E_TLS_CERT IPPOOL_START IPPOOL_END
8862023-11-20T07:50:50.558Z+ ./tests/bootstrap
8872023-11-20T07:50:50.589ZUsing external DNS server at 10.151.2.164:53
8882023-11-20T07:50:50.621Z2023-11-20 07:50:50.264185370 UTC: attempting to log into API
8892023-11-20T07:51:05.335Z2023-11-20 07:51:05.320166386 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8902023-11-20T07:51:06.336Z2023-11-20 07:51:06.321219387 UTC: attempting to log into API
8912023-11-20T07:51:21.369Z2023-11-20 07:51:21.355437153 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8922023-11-20T07:51:22.370Z2023-11-20 07:51:22.356664339 UTC: attempting to log into API
8932023-11-20T07:51:37.403Z2023-11-20 07:51:37.390776675 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8942023-11-20T07:51:38.404Z2023-11-20 07:51:38.391981638 UTC: attempting to log into API
8952023-11-20T07:51:53.437Z2023-11-20 07:51:53.427247647 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8962023-11-20T07:51:54.438Z2023-11-20 07:51:54.427860733 UTC: attempting to log into API
 
8972023-11-20T07:52:05.440ZNov 20 07:52:04 sock ip: ip: joining multicasts failed (4) on vopte2 - will use link layer broadcasts for multicast
 
8982023-11-20T07:52:09.495Z2023-11-20 07:52:09.478173020 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8992023-11-20T07:52:10.496Z2023-11-20 07:52:10.479336315 UTC: attempting to log into API
 
9002023-11-20T07:52:12.007ZNov 20 07:52:11 sock ip: ip: joining multicasts failed (4) on vopte4 - will use link layer broadcasts for multicast
9012023-11-20T07:52:19.158ZNov 20 07:52:18 sok ip: ip: joining multicasts failed (4) on vopte1 - will use link layer broadcasts for multicast
9022023-11-20T07:52:25.890ZNov 20 07:52:24 sok ip: ip: joining multicasts failed (4) on vopte5 - will use link layer broadcasts for multicast
 
9032023-11-20T07:52:25.900Z2023-11-20 07:52:25.511421555 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9042023-11-20T07:52:26.529Z2023-11-20 07:52:26.512345345 UTC: attempting to log into API
 
9052023-11-20T07:52:31.436ZNov 20 07:52:30 sock ip: ip: joining multicasts failed (4) on vopte3 - will use link layer broadcasts for multicast
 
9062023-11-20T07:52:41.562Z2023-11-20 07:52:41.545632268 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9072023-11-20T07:52:42.564Z2023-11-20 07:52:42.547732641 UTC: attempting to log into API
9082023-11-20T07:52:57.599Z2023-11-20 07:52:57.581476093 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9092023-11-20T07:52:58.601Z2023-11-20 07:52:58.583398912 UTC: attempting to log into API
9102023-11-20T07:53:04.389Z2023-11-20 07:53:04.371050374 UTC: login succeeded
9112023-11-20T07:53:04.425Zwaiting for nexus to come up...
9122023-11-20T07:53:04.492Zcreating IP pool... 10.151.2.175 - 10.151.2.227
9132023-11-20T07:53:04.539Zensuring datasets are ready...
9142023-11-20T07:53:07.012Zlet's roll.
9152023-11-20T07:53:07.048Z+ rm ./tests/bootstrap
9162023-11-20T07:53:07.549Z+ for test_bin in tests/*
9172023-11-20T07:53:07.580Z+ ./tests/bootstrap-7746be776f1918b3
9182023-11-20T07:53:07.611Z
9192023-11-20T07:53:07.644Zrunning 0 tests
9202023-11-20T07:53:07.676Z
9212023-11-20T07:53:07.706Ztest result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
9222023-11-20T07:53:07.737Z
9232023-11-20T07:53:07.768Z+ for test_bin in tests/*
9242023-11-20T07:53:07.799Z+ ./tests/end_to_end_tests-cb4cc51c87b08c71
9252023-11-20T07:53:07.830Z
9262023-11-20T07:53:07.861Zrunning 2 tests
9272023-11-20T07:53:07.892Ztest no_spoof::no_spoof_here ... ok
9282023-11-20T07:54:07.523Ztest instance_launch::instance_launch has been running for over 60 seconds
9292023-11-20T07:56:05.863Ztest instance_launch::instance_launch ... ok
9302023-11-20T07:56:05.895Z
9312023-11-20T07:56:05.926Ztest result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 178.75s
9322023-11-20T07:56:05.957Z
9332023-11-20T07:56:05.989Z+ _exit_trap
9342023-11-20T07:56:06.020Z+ local status=0
9352023-11-20T07:56:06.051Z+ [[ 0 -eq 0 ]]
9362023-11-20T07:56:06.083Z+ exit 0
9372023-11-20T07:56:06.125Zprocess exited: duration 388499 ms, exit code 0
 
9382023-11-20T07:56:06.228Zfound 47 output files
9392023-11-20T07:56:06.288Zuploading: /var/svc/log/oxide-sled-agent:default.log (656763 bytes)
9402023-11-20T07:56:08.076Zuploaded: /var/svc/log/oxide-sled-agent:default.log
9412023-11-20T07:56:08.099Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_a09c9ef2-b44c-46b9-8f18-372e76dd8249/root/var/svc/log/oxide-cockroachdb:default.log (5369 bytes)
9422023-11-20T07:56:09.198Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_a09c9ef2-b44c-46b9-8f18-372e76dd8249/root/var/svc/log/oxide-cockroachdb:default.log
9432023-11-20T07:56:09.222Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_07e03e9a-beed-4e41-b732-48a7946b5a69/root/var/svc/log/oxide-crucible-agent:default.log (3436 bytes)
9442023-11-20T07:56:10.351Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_07e03e9a-beed-4e41-b732-48a7946b5a69/root/var/svc/log/oxide-crucible-agent:default.log
9452023-11-20T07:56:10.375Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_07e03e9a-beed-4e41-b732-48a7946b5a69/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
9462023-11-20T07:56:11.478Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_07e03e9a-beed-4e41-b732-48a7946b5a69/root/var/svc/log/oxide-crucible-downstairs:default.log
9472023-11-20T07:56:11.502Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_84037cc2-9a57-4b92-aa79-1df7e38b4b09/root/var/svc/log/oxide-external_dns:default.log (15610 bytes)
9482023-11-20T07:56:12.634Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_84037cc2-9a57-4b92-aa79-1df7e38b4b09/root/var/svc/log/oxide-external_dns:default.log
9492023-11-20T07:56:12.659Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_33c7324a-3441-45ea-8d79-212b82858480/root/var/svc/log/oxide-cockroachdb:default.log (5343 bytes)
9502023-11-20T07:56:13.758Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_33c7324a-3441-45ea-8d79-212b82858480/root/var/svc/log/oxide-cockroachdb:default.log
9512023-11-20T07:56:13.782Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_05afc584-303a-494e-958d-48a2486cccde/root/var/svc/log/oxide-crucible-agent:default.log (23931 bytes)
9522023-11-20T07:56:14.939Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_05afc584-303a-494e-958d-48a2486cccde/root/var/svc/log/oxide-crucible-agent:default.log
9532023-11-20T07:56:14.962Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_05afc584-303a-494e-958d-48a2486cccde/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
9542023-11-20T07:56:16.061Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_05afc584-303a-494e-958d-48a2486cccde/root/var/svc/log/oxide-crucible-downstairs:default.log
9552023-11-20T07:56:16.085Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_05afc584-303a-494e-958d-48a2486cccde/root/var/svc/log/oxide-crucible-downstairs:downstairs-ad502828-b7c9-43bc-9a58-e7635035bdb3.log (5537 bytes)
9562023-11-20T07:56:17.186Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_05afc584-303a-494e-958d-48a2486cccde/root/var/svc/log/oxide-crucible-downstairs:downstairs-ad502828-b7c9-43bc-9a58-e7635035bdb3.log
9572023-11-20T07:56:17.209Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_05afc584-303a-494e-958d-48a2486cccde/root/var/svc/log/oxide-crucible-downstairs:downstairs-dff278fc-227d-4f3d-acd4-20a62eeca5ee.log (3047 bytes)
9582023-11-20T07:56:18.308Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_05afc584-303a-494e-958d-48a2486cccde/root/var/svc/log/oxide-crucible-downstairs:downstairs-dff278fc-227d-4f3d-acd4-20a62eeca5ee.log
9592023-11-20T07:56:18.350Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_7ac38bb8-58b9-46d6-877a-58d68deadc2c/root/var/svc/log/oxide-crucible-agent:default.log (3436 bytes)
9602023-11-20T07:56:19.454Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_7ac38bb8-58b9-46d6-877a-58d68deadc2c/root/var/svc/log/oxide-crucible-agent:default.log
9612023-11-20T07:56:19.480Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_7ac38bb8-58b9-46d6-877a-58d68deadc2c/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
9622023-11-20T07:56:20.576Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_7ac38bb8-58b9-46d6-877a-58d68deadc2c/root/var/svc/log/oxide-crucible-downstairs:default.log
9632023-11-20T07:56:20.600Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_8a5fa6e3-3703-4684-a6e8-b41d55048601/root/var/svc/log/oxide-nexus:default.log (539351 bytes)
9642023-11-20T07:56:20.911Zupload warning: file "/pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_8a5fa6e3-3703-4684-a6e8-b41d55048601/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 539351 -> 539959
9652023-11-20T07:56:22.002Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_8a5fa6e3-3703-4684-a6e8-b41d55048601/root/var/svc/log/oxide-nexus:default.log
9662023-11-20T07:56:22.026Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_cockroachdb_a3323bbb-65d3-472d-a293-6c510636cffb/root/var/svc/log/oxide-cockroachdb:default.log (5368 bytes)
9672023-11-20T07:56:23.126Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_cockroachdb_a3323bbb-65d3-472d-a293-6c510636cffb/root/var/svc/log/oxide-cockroachdb:default.log
9682023-11-20T07:56:23.150Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_internal_dns_306c7298-6fe3-4323-b3ea-7a7f9075ced3/root/var/svc/log/oxide-internal_dns:default.log (102809 bytes)
9692023-11-20T07:56:24.407Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_internal_dns_306c7298-6fe3-4323-b3ea-7a7f9075ced3/root/var/svc/log/oxide-internal_dns:default.log
9702023-11-20T07:56:24.431Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_internal_dns_defece7e-1f41-48b6-a56f-7f3b68d5a311/root/var/svc/log/oxide-internal_dns:default.log (99568 bytes)
9712023-11-20T07:56:25.675Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_internal_dns_defece7e-1f41-48b6-a56f-7f3b68d5a311/root/var/svc/log/oxide-internal_dns:default.log
9722023-11-20T07:56:25.699Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_clickhouse_847cd99c-7101-48eb-bf5f-29a882b481fe/root/var/svc/log/oxide-clickhouse:default.log (706128 bytes)
9732023-11-20T07:56:26.036Zupload warning: file "/pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_clickhouse_847cd99c-7101-48eb-bf5f-29a882b481fe/root/var/svc/log/oxide-clickhouse:default.log" changed size mid upload: 706128 -> 764995
9742023-11-20T07:56:27.541Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_clickhouse_847cd99c-7101-48eb-bf5f-29a882b481fe/root/var/svc/log/oxide-clickhouse:default.log
9752023-11-20T07:56:27.566Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_cockroachdb_f561426b-04e6-4e63-b481-1b397d5231d4/root/var/svc/log/oxide-cockroachdb:default.log (5368 bytes)
9762023-11-20T07:56:28.665Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_cockroachdb_f561426b-04e6-4e63-b481-1b397d5231d4/root/var/svc/log/oxide-cockroachdb:default.log
9772023-11-20T07:56:28.689Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_a9743596-df71-421e-b6dc-00d0636ce520/root/var/svc/log/oxide-crucible-agent:default.log (3434 bytes)
9782023-11-20T07:56:29.791Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_a9743596-df71-421e-b6dc-00d0636ce520/root/var/svc/log/oxide-crucible-agent:default.log
9792023-11-20T07:56:29.815Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_a9743596-df71-421e-b6dc-00d0636ce520/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
9802023-11-20T07:56:30.940Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_a9743596-df71-421e-b6dc-00d0636ce520/root/var/svc/log/oxide-crucible-downstairs:default.log
9812023-11-20T07:56:30.964Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_f8c559b1-5e66-4e93-9478-2feb1a69060f/root/var/svc/log/oxide-crucible-agent:default.log (14119 bytes)
9822023-11-20T07:56:32.084Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_f8c559b1-5e66-4e93-9478-2feb1a69060f/root/var/svc/log/oxide-crucible-agent:default.log
9832023-11-20T07:56:32.107Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_f8c559b1-5e66-4e93-9478-2feb1a69060f/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
9842023-11-20T07:56:33.224Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_f8c559b1-5e66-4e93-9478-2feb1a69060f/root/var/svc/log/oxide-crucible-downstairs:default.log
9852023-11-20T07:56:33.248Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_f8c559b1-5e66-4e93-9478-2feb1a69060f/root/var/svc/log/oxide-crucible-downstairs:downstairs-2ff2e5af-a714-4aa7-a938-e718275eccda.log (5535 bytes)
9862023-11-20T07:56:34.330Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_f8c559b1-5e66-4e93-9478-2feb1a69060f/root/var/svc/log/oxide-crucible-downstairs:downstairs-2ff2e5af-a714-4aa7-a938-e718275eccda.log
9872023-11-20T07:56:34.356Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_nexus_381f1240-83d8-4fb0-9bf7-1e162996eca3/root/var/svc/log/oxide-nexus:default.log (4667531 bytes)
9882023-11-20T07:56:36.295Zupload warning: file "/pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_nexus_381f1240-83d8-4fb0-9bf7-1e162996eca3/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 4667531 -> 4732884
9892023-11-20T07:56:37.386Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_nexus_381f1240-83d8-4fb0-9bf7-1e162996eca3/root/var/svc/log/oxide-nexus:default.log
9902023-11-20T07:56:37.411Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_249188b9-13b4-4193-9568-84db6fc17529/root/var/svc/log/oxide-crucible-agent:default.log (3436 bytes)
9912023-11-20T07:56:38.509Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_249188b9-13b4-4193-9568-84db6fc17529/root/var/svc/log/oxide-crucible-agent:default.log
9922023-11-20T07:56:38.536Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_249188b9-13b4-4193-9568-84db6fc17529/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
9932023-11-20T07:56:39.650Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_249188b9-13b4-4193-9568-84db6fc17529/root/var/svc/log/oxide-crucible-downstairs:default.log
9942023-11-20T07:56:39.675Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_nexus_dc808808-c29f-4e50-89dc-eb71dee374d3/root/var/svc/log/oxide-nexus:default.log (544168 bytes)
9952023-11-20T07:56:39.981Zupload warning: file "/pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_nexus_dc808808-c29f-4e50-89dc-eb71dee374d3/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 544168 -> 607723
9962023-11-20T07:56:41.073Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_nexus_dc808808-c29f-4e50-89dc-eb71dee374d3/root/var/svc/log/oxide-nexus:default.log
9972023-11-20T07:56:41.097Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_pantry_f17d1012-45a4-4e32-a4cd-f647739a92e6/root/var/svc/log/oxide-crucible-pantry:default.log (2338 bytes)
9982023-11-20T07:56:42.208Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_pantry_f17d1012-45a4-4e32-a4cd-f647739a92e6/root/var/svc/log/oxide-crucible-pantry:default.log
9992023-11-20T07:56:42.232Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_external_dns_5dda65cf-da71-4547-b51e-e1cf5d91d287/root/var/svc/log/oxide-external_dns:default.log (15606 bytes)
10002023-11-20T07:56:42.327Zupload warning: file "/pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_external_dns_5dda65cf-da71-4547-b51e-e1cf5d91d287/root/var/svc/log/oxide-external_dns:default.log" changed size mid upload: 15606 -> 19385
10012023-11-20T07:56:43.419Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_external_dns_5dda65cf-da71-4547-b51e-e1cf5d91d287/root/var/svc/log/oxide-external_dns:default.log
10022023-11-20T07:56:43.443Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_da996e81-243f-4d6a-9a19-416683cfc273/root/var/svc/log/oxide-cockroachdb:default.log (5369 bytes)
10032023-11-20T07:56:44.542Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_da996e81-243f-4d6a-9a19-416683cfc273/root/var/svc/log/oxide-cockroachdb:default.log
10042023-11-20T07:56:44.569Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_201086a7-56f2-4de0-8aee-210e5cfefed7/root/var/svc/log/oxide-crucible-agent:default.log (13686 bytes)
10052023-11-20T07:56:45.694Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_201086a7-56f2-4de0-8aee-210e5cfefed7/root/var/svc/log/oxide-crucible-agent:default.log
10062023-11-20T07:56:45.718Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_201086a7-56f2-4de0-8aee-210e5cfefed7/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10072023-11-20T07:56:46.816Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_201086a7-56f2-4de0-8aee-210e5cfefed7/root/var/svc/log/oxide-crucible-downstairs:default.log
10082023-11-20T07:56:46.840Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_201086a7-56f2-4de0-8aee-210e5cfefed7/root/var/svc/log/oxide-crucible-downstairs:downstairs-e829f193-6fe3-43be-8952-4d99e7bd16fc.log (3051 bytes)
10092023-11-20T07:56:47.938Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_201086a7-56f2-4de0-8aee-210e5cfefed7/root/var/svc/log/oxide-crucible-downstairs:downstairs-e829f193-6fe3-43be-8952-4d99e7bd16fc.log
10102023-11-20T07:56:47.969Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_3abd98fd-69cc-4f10-9866-db07b595b88e/root/var/svc/log/oxide-crucible-agent:default.log (14116 bytes)
10112023-11-20T07:56:49.096Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_3abd98fd-69cc-4f10-9866-db07b595b88e/root/var/svc/log/oxide-crucible-agent:default.log
10122023-11-20T07:56:49.128Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_3abd98fd-69cc-4f10-9866-db07b595b88e/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10132023-11-20T07:56:50.189Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_3abd98fd-69cc-4f10-9866-db07b595b88e/root/var/svc/log/oxide-crucible-downstairs:default.log
10142023-11-20T07:56:50.220Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_3abd98fd-69cc-4f10-9866-db07b595b88e/root/var/svc/log/oxide-crucible-downstairs:downstairs-1bf8e323-d6ee-422c-afa1-b46461e71acb.log (5537 bytes)
10152023-11-20T07:56:51.303Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_3abd98fd-69cc-4f10-9866-db07b595b88e/root/var/svc/log/oxide-crucible-downstairs:downstairs-1bf8e323-d6ee-422c-afa1-b46461e71acb.log
10162023-11-20T07:56:51.335Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_8bbe15f8-a976-42cd-ae90-1c7c4c5d3fea/root/var/svc/log/oxide-crucible-agent:default.log (14116 bytes)
10172023-11-20T07:56:52.442Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_8bbe15f8-a976-42cd-ae90-1c7c4c5d3fea/root/var/svc/log/oxide-crucible-agent:default.log
10182023-11-20T07:56:52.473Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_8bbe15f8-a976-42cd-ae90-1c7c4c5d3fea/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10192023-11-20T07:56:53.536Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_8bbe15f8-a976-42cd-ae90-1c7c4c5d3fea/root/var/svc/log/oxide-crucible-downstairs:default.log
10202023-11-20T07:56:53.599Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_8bbe15f8-a976-42cd-ae90-1c7c4c5d3fea/root/var/svc/log/oxide-crucible-downstairs:downstairs-55c1f2db-a195-4b53-a72d-b98be0a62107.log (3050 bytes)
10212023-11-20T07:56:54.652Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_8bbe15f8-a976-42cd-ae90-1c7c4c5d3fea/root/var/svc/log/oxide-crucible-downstairs:downstairs-55c1f2db-a195-4b53-a72d-b98be0a62107.log
10222023-11-20T07:56:54.676Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_088f787e-f34a-4c96-b5de-12e3e6780164/root/var/svc/log/oxide-crucible-pantry:default.log (2333 bytes)
10232023-11-20T07:56:55.776Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_088f787e-f34a-4c96-b5de-12e3e6780164/root/var/svc/log/oxide-crucible-pantry:default.log
10242023-11-20T07:56:55.799Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_505fd4f2-4a1e-4353-87c4-9b9b02e962ea/root/var/svc/log/oxide-ntp:default.log (783 bytes)
10252023-11-20T07:56:56.904Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_505fd4f2-4a1e-4353-87c4-9b9b02e962ea/root/var/svc/log/oxide-ntp:default.log
10262023-11-20T07:56:56.928Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_oximeter_8cb95d51-90ee-4878-942d-5d274a6755b3/root/var/svc/log/oxide-oximeter:default.log (186013 bytes)
10272023-11-20T07:56:57.172Zupload warning: file "/pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_oximeter_8cb95d51-90ee-4878-942d-5d274a6755b3/root/var/svc/log/oxide-oximeter:default.log" changed size mid upload: 186013 -> 224916
10282023-11-20T07:56:58.265Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_oximeter_8cb95d51-90ee-4878-942d-5d274a6755b3/root/var/svc/log/oxide-oximeter:default.log
10292023-11-20T07:56:58.289Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_16b5abb9-d189-4e4e-96c0-108193957b3d/root/var/svc/log/oxide-crucible-pantry:default.log (2338 bytes)
10302023-11-20T07:56:59.389Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_16b5abb9-d189-4e4e-96c0-108193957b3d/root/var/svc/log/oxide-crucible-pantry:default.log
10312023-11-20T07:56:59.413Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_c9a9027a-c2d6-437c-b4f4-de4f53331cd0/root/var/svc/log/oxide-internal_dns:default.log (104346 bytes)
10322023-11-20T07:56:59.602Zupload warning: file "/pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_c9a9027a-c2d6-437c-b4f4-de4f53331cd0/root/var/svc/log/oxide-internal_dns:default.log" changed size mid upload: 104346 -> 108099
10332023-11-20T07:57:00.694Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_c9a9027a-c2d6-437c-b4f4-de4f53331cd0/root/var/svc/log/oxide-internal_dns:default.log
10342023-11-20T07:57:00.717Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_505fd4f2-4a1e-4353-87c4-9b9b02e962ea/root/var/log/chrony/measurements.log (7946 bytes)
10352023-11-20T07:57:00.800Zupload warning: file "/pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_505fd4f2-4a1e-4353-87c4-9b9b02e962ea/root/var/log/chrony/measurements.log" changed size mid upload: 7946 -> 11645
10362023-11-20T07:57:01.885Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_505fd4f2-4a1e-4353-87c4-9b9b02e962ea/root/var/log/chrony/measurements.log
10372023-11-20T07:57:01.909Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_505fd4f2-4a1e-4353-87c4-9b9b02e962ea/root/var/log/chrony/statistics.log (4914 bytes)
10382023-11-20T07:57:01.947Zupload warning: file "/pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_505fd4f2-4a1e-4353-87c4-9b9b02e962ea/root/var/log/chrony/statistics.log" changed size mid upload: 4914 -> 6786
10392023-11-20T07:57:03.039Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_505fd4f2-4a1e-4353-87c4-9b9b02e962ea/root/var/log/chrony/statistics.log
10402023-11-20T07:57:03.064Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_505fd4f2-4a1e-4353-87c4-9b9b02e962ea/root/var/log/chrony/tracking.log (1322 bytes)
10412023-11-20T07:57:03.104Zupload warning: file "/pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_505fd4f2-4a1e-4353-87c4-9b9b02e962ea/root/var/log/chrony/tracking.log" changed size mid upload: 1322 -> 1454
10422023-11-20T07:57:04.205Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_505fd4f2-4a1e-4353-87c4-9b9b02e962ea/root/var/log/chrony/tracking.log