01HH005WSZF0FANY5WTSCDYBX3: helios / deploy

BasicConfig {
    output_rules: [
        "%/var/svc/log/oxide-sled-agent:default.log*",
        "%/pool/ext/*/crypt/zone/oxz_*/root/var/svc/log/oxide-*.log*",
        "%/pool/ext/*/crypt/zone/oxz_*/root/var/svc/log/system-illumos-*.log*",
        "%/pool/ext/*/crypt/zone/oxz_ntp_*/root/var/log/chrony/*.log*",
        "!/pool/ext/*/crypt/zone/oxz_propolis-server_*/root/var/svc/log/*.log*",
        "%/pool/ext/*/crypt/debug/global/oxide-sled-agent:default.log.*",
        "%/pool/ext/*/crypt/debug/oxz_*/oxide-*.log.*",
        "%/pool/ext/*/crypt/debug/oxz_*/system-illumos-*.log.*",
        "!/pool/ext/*/crypt/debug/oxz_propolis-server_*/*.log.*",
    ],
    rust_toolchain: None,
    target: Some(
        "lab-2.0-opte-0.25",
    ),
    access_repos: [],
    publish: [],
    skip_clone: true,
}

Buildomat Job: 01HH00725K69P6DYYVC7S50HQQ

Tags:

Artefacts:

Output:

SEQ GLOBAL TIME DETAILS
12023-12-06T17:28:53.815Zjob dependencies complete; ready to run (waiting for 45 m 11 s)
22023-12-06T17:34:31.070Zjob assigned to worker 01HH02T0CCDPB3QZ6W41W44CNY (queued for 5 m 37 s)
32023-12-06T17:34:31.200Zhttp://172.20.0.5:9969/ipxe/sock... ok
42023-12-06T17:34:31.241Zhttp://172.20.0.5:9969/os/sock/platform/i86pc/kernel/amd64/unix... ok
52023-12-06T17:34:31.274Zhttp://172.20.0.5:9969/os/sock/platform/i86pc/amd64/boot_archive... ok
62023-12-06T17:34:31.308Zhttp://172.20.0.5:9969/os/sock/platform/i86pc/amd64/boot_archive.hash... ok
72023-12-06T17:34:31.341Zhttp://172.20.0.5:9969/postboot/sock... ok
82023-12-06T17:34:31.375ZwcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U
92023-12-06T17:34:31.408Zcpu0: microcode has been updated from version 0xa001133 to 0xa0011d1
102023-12-06T17:34:31.441ZOxide Helios Version helios-2.0.22318 64-bit
112023-12-06T17:34:31.475ZConfiguring devices.
122023-12-06T17:34:31.510Z##################################################
132023-12-06T17:34:31.542Z#### ############# ##
142023-12-06T17:34:31.575Z### ### ############ ##
152023-12-06T17:34:31.608Z## ### # ## ### ## Oxide Computer Company ##
162023-12-06T17:34:31.642Z## ## ## ### # ### ##
172023-12-06T17:34:31.675Z## # ### #### #### This Station Under ##
182023-12-06T17:34:31.709Z### ### #### # ### Computer Control ##
192023-12-06T17:34:31.774Z#### #### ### ## ##
202023-12-06T17:34:31.809Z##################################################
212023-12-06T17:34:31.843ZHostname: sock
222023-12-06T17:34:31.879ZLoading smf(7) service descriptions:
232023-12-06T17:34:32.130Zdownloading input: /input/ci-tools/work/end-to-end-tests/bootstrap-be9c817b94af4e8a.gz
242023-12-06T17:34:33.923Zdownloaded input: /input/ci-tools/work/end-to-end-tests/bootstrap-be9c817b94af4e8a.gz
252023-12-06T17:34:33.924Zdownloading input: /input/ci-tools/work/end-to-end-tests/bootstrap.gz
262023-12-06T17:34:50.744Zdownloaded input: /input/ci-tools/work/end-to-end-tests/bootstrap.gz
272023-12-06T17:34:50.745Zdownloading input: /input/ci-tools/work/end-to-end-tests/end_to_end_tests-ff923d3f5c6228d4.gz
282023-12-06T17:35:08.088Zdownloaded input: /input/ci-tools/work/end-to-end-tests/end_to_end_tests-ff923d3f5c6228d4.gz
292023-12-06T17:35:08.115Zdownloading input: /input/ci-tools/work/caboose-util.gz
302023-12-06T17:35:09.314Zdownloaded input: /input/ci-tools/work/caboose-util.gz
312023-12-06T17:35:09.314Zdownloading input: /input/ci-tools/work/tufaceous.gz
322023-12-06T17:35:12.099Zdownloaded input: /input/ci-tools/work/tufaceous.gz
332023-12-06T17:35:12.127Zdownloading input: /input/package/work/version.txt
342023-12-06T17:35:12.127Zdownloaded input: /input/package/work/version.txt
352023-12-06T17:35:12.127Zdownloading input: /input/package/work/package.tar.gz
362023-12-06T17:35:26.938Zdownloaded input: /input/package/work/package.tar.gz
372023-12-06T17:35:26.938Zdownloading input: /input/package/work/global-zone-packages.tar.gz
382023-12-06T17:36:01.944Zdownloaded input: /input/package/work/global-zone-packages.tar.gz
392023-12-06T17:36:01.944Zdownloading input: /input/package/work/trampoline-global-zone-packages.tar.gz
402023-12-06T17:36:09.927Zdownloaded input: /input/package/work/trampoline-global-zone-packages.tar.gz
412023-12-06T17:36:09.959Zdownloading input: /input/package/work/zones/clickhouse.tar.gz
422023-12-06T17:36:42.401Zdownloaded input: /input/package/work/zones/clickhouse.tar.gz
432023-12-06T17:36:42.402Zdownloading input: /input/package/work/zones/clickhouse_keeper.tar.gz
442023-12-06T17:37:13.481Zdownloaded input: /input/package/work/zones/clickhouse_keeper.tar.gz
452023-12-06T17:37:13.482Zdownloading input: /input/package/work/zones/cockroachdb.tar.gz
462023-12-06T17:37:35.167Zdownloaded input: /input/package/work/zones/cockroachdb.tar.gz
472023-12-06T17:37:35.167Zdownloading input: /input/package/work/zones/crucible-pantry.tar.gz
482023-12-06T17:37:39.123Zdownloaded input: /input/package/work/zones/crucible-pantry.tar.gz
492023-12-06T17:37:39.123Zdownloading input: /input/package/work/zones/crucible.tar.gz
502023-12-06T17:37:47.164Zdownloaded input: /input/package/work/zones/crucible.tar.gz
512023-12-06T17:37:47.164Zdownloading input: /input/package/work/zones/external-dns.tar.gz
522023-12-06T17:37:52.623Zdownloaded input: /input/package/work/zones/external-dns.tar.gz
532023-12-06T17:37:52.624Zdownloading input: /input/package/work/zones/internal-dns.tar.gz
542023-12-06T17:37:56.897Zdownloaded input: /input/package/work/zones/internal-dns.tar.gz
552023-12-06T17:37:56.897Zdownloading input: /input/package/work/zones/ntp.tar.gz
562023-12-06T17:37:56.938Zdownloaded input: /input/package/work/zones/ntp.tar.gz
572023-12-06T17:37:56.963Zdownloading input: /input/package/work/zones/omicron-gateway-asic.tar.gz
582023-12-06T17:38:00.584Zdownloaded input: /input/package/work/zones/omicron-gateway-asic.tar.gz
592023-12-06T17:38:00.584Zdownloading input: /input/package/work/zones/omicron-gateway-softnpu.tar.gz
602023-12-06T17:38:05.677Zdownloaded input: /input/package/work/zones/omicron-gateway-softnpu.tar.gz
612023-12-06T17:38:05.677Zdownloading input: /input/package/work/zones/omicron-nexus-single-sled.tar.gz
622023-12-06T17:38:19.062Zdownloaded input: /input/package/work/zones/omicron-nexus-single-sled.tar.gz
632023-12-06T17:38:19.062Zdownloading input: /input/package/work/zones/omicron-nexus.tar.gz
642023-12-06T17:38:31.835Zdownloaded input: /input/package/work/zones/omicron-nexus.tar.gz
652023-12-06T17:38:31.835Zdownloading input: /input/package/work/zones/overlay.tar.gz
662023-12-06T17:38:31.873Zdownloaded input: /input/package/work/zones/overlay.tar.gz
672023-12-06T17:38:31.898Zdownloading input: /input/package/work/zones/oximeter-collector.tar.gz
682023-12-06T17:38:37.813Zdownloaded input: /input/package/work/zones/oximeter-collector.tar.gz
692023-12-06T17:38:37.813Zdownloading input: /input/package/work/zones/propolis-server.tar.gz
702023-12-06T17:38:49.177Zdownloaded input: /input/package/work/zones/propolis-server.tar.gz
712023-12-06T17:38:49.177Zdownloading input: /input/package/work/zones/switch-asic.tar.gz
722023-12-06T17:39:31.242Zdownloaded input: /input/package/work/zones/switch-asic.tar.gz
732023-12-06T17:39:31.242Zdownloading input: /input/package/work/zones/switch-softnpu.tar.gz
742023-12-06T17:40:01.575Zdownloaded input: /input/package/work/zones/switch-softnpu.tar.gz
 
752023-12-06T17:40:01.602Zstarting task 0: "setup"
762023-12-06T17:40:01.602Z++ uname -s
772023-12-06T17:40:01.602Z+ kern=SunOS
782023-12-06T17:40:01.602Z+ case "$kern" in
792023-12-06T17:40:01.602Z+ groupadd -g 12345 build
802023-12-06T17:40:01.603Z+ useradd -u 12345 -g build -d /home/build -s /bin/bash -c build -P 'Primary Administrator' build
812023-12-06T17:40:03.586Z+ zfs create -o mountpoint=/work rpool/work
822023-12-06T17:40:03.614Z++ awk '$2 == "/home" { print $3 }' /etc/mnttab
832023-12-06T17:40:03.640Z+ home_fs=zfs
842023-12-06T17:40:03.640Z+ [[ zfs == autofs ]]
852023-12-06T17:40:03.640Z+ mkdir -p /home/build
862023-12-06T17:40:03.640Z+ chown build:build /home/build /work
872023-12-06T17:40:05.622Z+ chmod 0700 /home/build /work
882023-12-06T17:40:05.647Zprocess exited: duration 4057 ms, exit code 0
 
892023-12-06T17:40:05.698Zstarting task 1: "authentication"
902023-12-06T17:40:05.746Zprocess exited: duration 46 ms, exit code 0
 
912023-12-06T17:40:05.801Zstarting task 2: "build"
922023-12-06T17:40:05.827Z+ trap _exit_trap EXIT
932023-12-06T17:40:05.827Z+ pfexec mdb -kwe 'secpolicy_fs_linkdir/v 55 48 89 e5 b8 01 00 00 00 5d c3'
942023-12-06T17:40:06.005Zsecpolicy_fs_linkdir: 0x55 = 0x55
952023-12-06T17:40:06.005Zsecpolicy_fs_linkdir+1: 0xb8 = 0x48
962023-12-06T17:40:06.005Zsecpolicy_fs_linkdir+2: 0x1 = 0x89
972023-12-06T17:40:06.005Zsecpolicy_fs_linkdir+3: 0 = 0xe5
982023-12-06T17:40:06.005Zsecpolicy_fs_linkdir+4: 0 = 0xb8
992023-12-06T17:40:06.005Zsecpolicy_fs_linkdir+5: 0 = 0x1
1002023-12-06T17:40:06.005Zsecpolicy_fs_linkdir+6: 0x48 = 0x0
1012023-12-06T17:40:06.005Zsecpolicy_fs_linkdir+7: 0x89 = 0x0
1022023-12-06T17:40:06.005Zsecpolicy_fs_linkdir+8: 0xe5 = 0x0
1032023-12-06T17:40:06.006Zsecpolicy_fs_linkdir+9: 0x48 = 0x5d
1042023-12-06T17:40:06.006Zsecpolicy_fs_linkdir+0xa: 0x83 = 0xc3
1052023-12-06T17:40:06.032Z+ [[ -d /opt/oxide ]]
1062023-12-06T17:40:06.032Z+ pfexec mv /opt/oxide /opt/oxide-underneath
1072023-12-06T17:40:06.032Z+ pfexec mkdir /opt/oxide
1082023-12-06T17:40:06.032Z+ pfexec mount -F tmpfs -O swap /opt/oxide
1092023-12-06T17:40:06.032Z+ [[ -d /opt/oxide-underneath ]]
1102023-12-06T17:40:06.032Z+ cd /opt/oxide-underneath
1112023-12-06T17:40:06.032Z+ pfexec tar ceEp@/f - .
1122023-12-06T17:40:06.032Z+ cd /opt/oxide
1132023-12-06T17:40:06.032Z+ pfexec tar xveEp@/f -
1142023-12-06T17:40:06.131Zx ., 0 bytes, 0 tape blocks
1152023-12-06T17:40:06.131Zx ./opte, 0 bytes, 0 tape blocks
1162023-12-06T17:40:06.131Zx ./opte/bin, 0 bytes, 0 tape blocks
1172023-12-06T17:40:06.131Zx ./opte/bin/opteadm, 50714832 bytes, 99053 tape blocks
1182023-12-06T17:40:06.156Z+ pfexec rm -rf /opt/oxide-underneath
1192023-12-06T17:40:06.156Z+ pfexec /sbin/zfs create -o mountpoint=/zone rpool/zone
1202023-12-06T17:40:06.182Z+ pfexec mkdir /opt/oxide/work
1212023-12-06T17:40:06.182Z+ pfexec chown build:build /opt/oxide/work
1222023-12-06T17:40:06.182Z+ cd /opt/oxide/work
1232023-12-06T17:40:06.182Z+ ptime -m tar xvzf /input/package/work/package.tar.gz
1242023-12-06T17:40:07.702ZDecompressing '/input/package/work/package.tar.gz' with '/usr/bin/gzcat'...
1252023-12-06T17:40:07.702Zx out/mg-ddm-gz.tar, 62723072 bytes, 122506 tape blocks
1262023-12-06T17:40:07.702Zx out/omicron-sled-agent.tar, 111105536 bytes, 217003 tape blocks
1272023-12-06T17:40:07.702Zx out/target/test, 75 bytes, 1 tape blocks
1282023-12-06T17:40:07.702Zx out/npuzone/npuzone, 37380856 bytes, 73010 tape blocks
1292023-12-06T17:40:07.702Zx package-manifest.toml, 22143 bytes, 44 tape blocks
1302023-12-06T17:40:07.702Zx smf/sled-agent/non-gimlet/config.toml, 3298 bytes, 7 tape blocks
1312023-12-06T17:40:07.740Zx target/release/omicron-package, 42582768 bytes, 83170 tape blocks
1322023-12-06T17:40:07.740Zx tools/create_virtual_hardware.sh, 3176 bytes, 7 tape blocks
1332023-12-06T17:40:07.740Zx tools/virtual_hardware.sh, 3408 bytes, 7 tape blocks
1342023-12-06T17:40:07.740Zx tools/scrimlet/create-softnpu-zone.sh, 619 bytes, 2 tape blocks
1352023-12-06T17:40:07.740Zx tools/scrimlet/destroy-softnpu-zone.sh, 195 bytes, 1 tape blocks
1362023-12-06T17:40:07.740Zx tools/scrimlet/softnpu-init.sh, 2148 bytes, 5 tape blocks
1372023-12-06T17:40:07.740Zx tools/scrimlet/softnpu-zone.txt, 330 bytes, 1 tape blocks
1382023-12-06T17:40:07.741Zx tools/scrimlet/softnpu.toml, 171 bytes, 1 tape blocks
1392023-12-06T17:40:07.741Z
1402023-12-06T17:40:07.741Zreal 1.536547443
1412023-12-06T17:40:07.741Zuser 1.387207162
1422023-12-06T17:40:07.741Zsys 0.266258725
1432023-12-06T17:40:07.741Ztrap 0.000053832
1442023-12-06T17:40:07.741Ztflt 0.000000000
1452023-12-06T17:40:07.741Zdflt 0.000000000
1462023-12-06T17:40:07.741Zkflt 0.000000000
1472023-12-06T17:40:07.741Zlock 0.000000000
1482023-12-06T17:40:07.741Zslp 1.385343102
1492023-12-06T17:40:07.741Zlat 0.032224691
1502023-12-06T17:40:07.741Zstop 0.000120001
1512023-12-06T17:40:07.742Z+ cp /input/package/work/zones/clickhouse_keeper.tar.gz /input/package/work/zones/clickhouse.tar.gz /input/package/work/zones/cockroachdb.tar.gz /input/package/work/zones/crucible-pantry.tar.gz /input/package/work/zones/crucible.tar.gz /input/package/work/zones/external-dns.tar.gz /input/package/work/zones/internal-dns.tar.gz /input/package/work/zones/ntp.tar.gz /input/package/work/zones/omicron-gateway-asic.tar.gz /input/package/work/zones/omicron-gateway-softnpu.tar.gz /input/package/work/zones/omicron-nexus-single-sled.tar.gz /input/package/work/zones/omicron-nexus.tar.gz /input/package/work/zones/overlay.tar.gz /input/package/work/zones/oximeter-collector.tar.gz /input/package/work/zones/propolis-server.tar.gz /input/package/work/zones/switch-asic.tar.gz /input/package/work/zones/switch-softnpu.tar.gz out/
1522023-12-06T17:40:08.816Z+ mv out/omicron-nexus-single-sled.tar.gz out/omicron-nexus.tar.gz
1532023-12-06T17:40:08.843Z+ mkdir tests
1542023-12-06T17:40:08.843Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1552023-12-06T17:40:08.843Z+ ptime -m gunzip
1562023-12-06T17:40:08.843Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap-be9c817b94af4e8a
1572023-12-06T17:40:08.968Z
1582023-12-06T17:40:08.994Zreal 0.137260596
1592023-12-06T17:40:08.994Zuser 0.126281751
1602023-12-06T17:40:08.994Zsys 0.010745530
1612023-12-06T17:40:08.994Ztrap 0.000015050
1622023-12-06T17:40:08.994Ztflt 0.000000000
1632023-12-06T17:40:08.994Zdflt 0.000000000
1642023-12-06T17:40:08.994Zkflt 0.000000000
1652023-12-06T17:40:08.994Zlock 0.000000000
1662023-12-06T17:40:08.994Zslp 0.000021470
1672023-12-06T17:40:08.994Zlat 0.000016930
1682023-12-06T17:40:08.994Zstop 0.000119891
1692023-12-06T17:40:08.994Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap-be9c817b94af4e8a
1702023-12-06T17:40:08.994Z+ chmod a+x tests/bootstrap-be9c817b94af4e8a
1712023-12-06T17:40:08.994Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1722023-12-06T17:40:08.994Z+ ptime -m gunzip
1732023-12-06T17:40:08.994Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap
1742023-12-06T17:40:11.156Z
1752023-12-06T17:40:11.181Zreal 2.176564462
1762023-12-06T17:40:11.181Zuser 2.051263417
1772023-12-06T17:40:11.182Zsys 0.124905035
1782023-12-06T17:40:11.182Ztrap 0.000181214
1792023-12-06T17:40:11.182Ztflt 0.000000000
1802023-12-06T17:40:11.182Zdflt 0.000000000
1812023-12-06T17:40:11.182Zkflt 0.000000000
1822023-12-06T17:40:11.182Zlock 0.000000000
1832023-12-06T17:40:11.182Zslp 0.000019060
1842023-12-06T17:40:11.182Zlat 0.000016230
1852023-12-06T17:40:11.182Zstop 0.000111912
1862023-12-06T17:40:11.182Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap
1872023-12-06T17:40:11.182Z+ chmod a+x tests/bootstrap
1882023-12-06T17:40:11.182Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1892023-12-06T17:40:11.182Z+ ptime -m gunzip
1902023-12-06T17:40:11.182Z++ basename /input/ci-tools/work/end-to-end-tests/end_to_end_tests-ff923d3f5c6228d4
1912023-12-06T17:40:13.424Z
1922023-12-06T17:40:13.449Zreal 2.256085598
1932023-12-06T17:40:13.449Zuser 2.117602890
1942023-12-06T17:40:13.449Zsys 0.138117110
1952023-12-06T17:40:13.449Ztrap 0.000136151
1962023-12-06T17:40:13.449Ztflt 0.000000000
1972023-12-06T17:40:13.449Zdflt 0.000000000
1982023-12-06T17:40:13.449Zkflt 0.000000000
1992023-12-06T17:40:13.449Zlock 0.000000000
2002023-12-06T17:40:13.449Zslp 0.000020320
2012023-12-06T17:40:13.449Zlat 0.000023310
2022023-12-06T17:40:13.449Zstop 0.000118082
2032023-12-06T17:40:13.449Z++ basename /input/ci-tools/work/end-to-end-tests/end_to_end_tests-ff923d3f5c6228d4
2042023-12-06T17:40:13.449Z+ chmod a+x tests/end_to_end_tests-ff923d3f5c6228d4
2052023-12-06T17:40:13.449Z+ bmat address ls
2062023-12-06T17:40:13.449ZNAME CIDR FIRST LAST COUNT
2072023-12-06T17:40:13.449Zextra 10.151.2.0/24 10.151.2.164 10.151.2.227 64
2082023-12-06T17:40:13.450Z++ bmat address ls -f extra -Ho first,last,count
2092023-12-06T17:40:13.477Z+ set -- 10.151.2.164 10.151.2.227 64
2102023-12-06T17:40:13.478Z+ EXTRA_IP_START=10.151.2.164
2112023-12-06T17:40:13.478Z+ EXTRA_IP_END=10.151.2.227
2122023-12-06T17:40:13.478Z+ EXTRA_IP_COUNT=64
2132023-12-06T17:40:13.478Z+ (( EXTRA_IP_COUNT >= 32 ))
2142023-12-06T17:40:13.478Z+ EXTRA_IP_BASE=10.151.2
2152023-12-06T17:40:13.478Z+ EXTRA_IP_FOCTET=164
2162023-12-06T17:40:13.478Z+ EXTRA_IP_LOCTET=227
2172023-12-06T17:40:13.478Z+ SERVICE_IP_POOL_START=10.151.2.164
2182023-12-06T17:40:13.478Z+ SERVICE_IP_POOL_END=10.151.2.173
2192023-12-06T17:40:13.479Z+ DNS_IP1=10.151.2.164
2202023-12-06T17:40:13.479Z+ DNS_IP2=10.151.2.165
2212023-12-06T17:40:13.479Z+ UPLINK_IP=10.151.2.174
2222023-12-06T17:40:13.479Z+ IPPOOL_START=10.151.2.175
2232023-12-06T17:40:13.479Z+ IPPOOL_END=10.151.2.227
2242023-12-06T17:40:13.479Z++ ipadm show-addr -po type,addr
2252023-12-06T17:40:13.479Z++ awk '-F[:/]' '$1 == "dhcp" {print $2}'
2262023-12-06T17:40:13.479Z+ GATEWAY_IP=10.151.2.100
2272023-12-06T17:40:13.479Z+ [[ -n 10.151.2.100 ]]
2282023-12-06T17:40:13.479Z+ ping -s 10.151.2.100 56 1
2292023-12-06T17:40:13.479ZPING 10.151.2.100: 56 data bytes
2302023-12-06T17:40:13.479Z64 bytes from 10.151.2.100: icmp_seq=0. time=0.010 ms
2312023-12-06T17:40:13.479Z
2322023-12-06T17:40:13.479Z----10.151.2.100 PING Statistics----
2332023-12-06T17:40:13.479Z1 packets transmitted, 1 packets received, 0% packet loss
2342023-12-06T17:40:13.479Zround-trip (ms) min/avg/max/stddev = 0.010/0.010/0.010/-nan
2352023-12-06T17:40:13.479Z++ arp -an
2362023-12-06T17:40:13.479Z++ awk -vgw=10.151.2.100 '$2 == gw {print $NF}'
2372023-12-06T17:40:13.479Z+ GATEWAY_MAC=18:c0:4d:0d:a0:2a
2382023-12-06T17:40:13.479Z+ routeadm -e ipv4-forwarding -u
2392023-12-06T17:40:13.584Z+ PXA_START=10.151.2.164
2402023-12-06T17:40:13.584Z+ PXA_END=10.151.2.227
2412023-12-06T17:40:13.608Z+ export GATEWAY_IP GATEWAY_MAC PXA_START PXA_END
2422023-12-06T17:40:13.608Z+ pfexec zpool create -f scratch c1t1d0 c2t1d0
2432023-12-06T17:40:13.716Z+ ZPOOL_VDEV_DIR=/scratch
2442023-12-06T17:40:13.716Z+ ptime -m pfexec ./tools/create_virtual_hardware.sh
2452023-12-06T17:40:13.744Z+++ dirname ./tools/create_virtual_hardware.sh
2462023-12-06T17:40:13.744Z++ cd ./tools
2472023-12-06T17:40:13.744Z++ pwd
2482023-12-06T17:40:13.744Z+ SOURCE_DIR=/opt/oxide/work/tools
2492023-12-06T17:40:13.744Z+ OMICRON_TOP=/opt/oxide/work/tools/..
2502023-12-06T17:40:13.745Z+ SOFTNPU_MODE=zone
2512023-12-06T17:40:13.745Z+ . /opt/oxide/work/tools/virtual_hardware.sh
2522023-12-06T17:40:13.745Z++ MARKER=/etc/opt/oxide/NO_INSTALL
2532023-12-06T17:40:13.745Z++ [[ -f /etc/opt/oxide/NO_INSTALL ]]
2542023-12-06T17:40:13.745Z++ dladm show-phys -p -o LINK
2552023-12-06T17:40:13.745Z++ head -1
2562023-12-06T17:40:13.804Z+ PHYSICAL_LINK=igb0
2572023-12-06T17:40:13.831Z+ echo 'Using igb0 as physical link'
2582023-12-06T17:40:13.832ZUsing igb0 as physical link
2592023-12-06T17:40:13.832Z+ ensure_run_as_root
2602023-12-06T17:40:13.832Z++ id -u
2612023-12-06T17:40:13.832Z+ [[ 0 -ne 0 ]]
2622023-12-06T17:40:13.832Z+ ensure_zpools
2632023-12-06T17:40:13.832Z+ ZPOOL_TYPES=('oxp_' 'oxi_')
2642023-12-06T17:40:13.832Z+ for ZPOOL_TYPE in "${ZPOOL_TYPES[@]}"
2652023-12-06T17:40:13.832Z+ readarray -t ZPOOLS
2662023-12-06T17:40:13.832Z++ grep '"oxp_' /opt/oxide/work/tools/../smf/sled-agent/non-gimlet/config.toml
2672023-12-06T17:40:13.832Z++ sed 's/[ ",]//g'
2682023-12-06T17:40:13.832Z+ for ZPOOL in "${ZPOOLS[@]}"
2692023-12-06T17:40:13.832Z+ echo 'Zpool: [oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b]'
2702023-12-06T17:40:13.832ZZpool: [oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b]
2712023-12-06T17:40:13.832Z+ VDEV_PATH=/scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
2722023-12-06T17:40:13.832Z+ [[ -f /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
2732023-12-06T17:40:13.832Z+ dd if=/dev/zero of=/scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
2742023-12-06T17:40:13.859Z0+0 records in
2752023-12-06T17:40:13.859Z0+0 records out
2762023-12-06T17:40:13.859Z0 bytes (0 iB) transferred in 0.000006 secs (0 iB/sec)
2772023-12-06T17:40:13.859Z+ success 'ZFS vdev /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
2782023-12-06T17:40:13.859Z+ set +x
2792023-12-06T17:40:13.860ZZFS vdev /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
2802023-12-06T17:40:13.860Z++ zpool list -o name
2812023-12-06T17:40:13.860Z++ grep oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b
2822023-12-06T17:40:13.860Z+ [[ -z '' ]]
2832023-12-06T17:40:13.860Z+ zpool create -f oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
2842023-12-06T17:40:13.890Z+ success 'ZFS zpool oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
2852023-12-06T17:40:13.918Z+ set +x
2862023-12-06T17:40:13.918ZZFS zpool oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b exists
2872023-12-06T17:40:13.918Z+ for ZPOOL in "${ZPOOLS[@]}"
2882023-12-06T17:40:13.918Z+ echo 'Zpool: [oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03]'
2892023-12-06T17:40:13.918Z+ VDEV_PATH=/scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
2902023-12-06T17:40:13.919ZZpool: [oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03]
2912023-12-06T17:40:13.919Z+ [[ -f /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
2922023-12-06T17:40:13.919Z+ dd if=/dev/zero of=/scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
2932023-12-06T17:40:13.919Z0+0 records in
2942023-12-06T17:40:13.919Z0+0 records out
2952023-12-06T17:40:13.919Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
2962023-12-06T17:40:13.919Z+ success 'ZFS vdev /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
2972023-12-06T17:40:13.919Z+ set +x
2982023-12-06T17:40:13.919ZZFS vdev /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
2992023-12-06T17:40:13.919Z++ zpool list -o name
3002023-12-06T17:40:13.919Z++ grep oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03
3012023-12-06T17:40:13.919Z+ [[ -z '' ]]
3022023-12-06T17:40:13.919Z+ zpool create -f oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3032023-12-06T17:40:13.944Z+ success 'ZFS zpool oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3042023-12-06T17:40:13.944Z+ set +x
3052023-12-06T17:40:13.944ZZFS zpool oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
3062023-12-06T17:40:13.944Z+ for ZPOOL in "${ZPOOLS[@]}"
3072023-12-06T17:40:13.944Z+ echo 'Zpool: [oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3082023-12-06T17:40:13.944ZZpool: [oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03]
3092023-12-06T17:40:13.944Z+ VDEV_PATH=/scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3102023-12-06T17:40:13.944Z+ [[ -f /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3112023-12-06T17:40:13.944Z+ dd if=/dev/zero of=/scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3122023-12-06T17:40:13.944Z0+0 records in
3132023-12-06T17:40:13.944Z0+0 records out
3142023-12-06T17:40:13.945Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3152023-12-06T17:40:13.945Z+ success 'ZFS vdev /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3162023-12-06T17:40:13.945Z+ set +x
3172023-12-06T17:40:13.945ZZFS vdev /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
3182023-12-06T17:40:13.945Z++ zpool list -o name
3192023-12-06T17:40:13.945Z++ grep oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03
3202023-12-06T17:40:13.970Z+ [[ -z '' ]]
3212023-12-06T17:40:13.970Z+ zpool create -f oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3222023-12-06T17:40:13.995Z+ success 'ZFS zpool oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3232023-12-06T17:40:13.995Z+ set +x
3242023-12-06T17:40:13.995ZZFS zpool oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
3252023-12-06T17:40:13.995Z+ for ZPOOL in "${ZPOOLS[@]}"
3262023-12-06T17:40:13.995Z+ echo 'Zpool: [oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3272023-12-06T17:40:13.995ZZpool: [oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03]
3282023-12-06T17:40:13.995Z+ VDEV_PATH=/scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3292023-12-06T17:40:13.995Z+ [[ -f /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3302023-12-06T17:40:13.995Z+ dd if=/dev/zero of=/scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3312023-12-06T17:40:13.995Z0+0 records in
3322023-12-06T17:40:13.995Z0+0 records out
3332023-12-06T17:40:13.995Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
3342023-12-06T17:40:13.996Z+ success 'ZFS vdev /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3352023-12-06T17:40:13.996Z+ set +x
3362023-12-06T17:40:13.996ZZFS vdev /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
3372023-12-06T17:40:13.996Z++ zpool list -o name
3382023-12-06T17:40:13.996Z++ grep oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03
3392023-12-06T17:40:13.996Z+ [[ -z '' ]]
3402023-12-06T17:40:13.996Z+ zpool create -f oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3412023-12-06T17:40:14.021Z+ success 'ZFS zpool oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3422023-12-06T17:40:14.021Z+ set +x
3432023-12-06T17:40:14.021ZZFS zpool oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
3442023-12-06T17:40:14.021Z+ for ZPOOL in "${ZPOOLS[@]}"
3452023-12-06T17:40:14.021Z+ echo 'Zpool: [oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3462023-12-06T17:40:14.021ZZpool: [oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03]
3472023-12-06T17:40:14.021Z+ VDEV_PATH=/scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3482023-12-06T17:40:14.021Z+ [[ -f /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3492023-12-06T17:40:14.021Z+ dd if=/dev/zero of=/scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3502023-12-06T17:40:14.049Z0+0 records in
3512023-12-06T17:40:14.049Z0+0 records out
3522023-12-06T17:40:14.049Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
3532023-12-06T17:40:14.049Z+ success 'ZFS vdev /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3542023-12-06T17:40:14.049Z+ set +x
3552023-12-06T17:40:14.049ZZFS vdev /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
3562023-12-06T17:40:14.050Z++ zpool list -o name
3572023-12-06T17:40:14.050Z++ grep oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03
3582023-12-06T17:40:14.050Z+ [[ -z '' ]]
3592023-12-06T17:40:14.050Z+ zpool create -f oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3602023-12-06T17:40:14.082Z+ success 'ZFS zpool oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3612023-12-06T17:40:14.082Z+ set +x
3622023-12-06T17:40:14.083ZZFS zpool oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
3632023-12-06T17:40:14.083Z+ for ZPOOL in "${ZPOOLS[@]}"
3642023-12-06T17:40:14.083Z+ echo 'Zpool: [oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930]'
3652023-12-06T17:40:14.083ZZpool: [oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930]
3662023-12-06T17:40:14.083Z+ VDEV_PATH=/scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev
3672023-12-06T17:40:14.083Z+ [[ -f /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev ]]
3682023-12-06T17:40:14.083Z+ dd if=/dev/zero of=/scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev bs=1 count=0 seek=20G
3692023-12-06T17:40:14.083Z0+0 records in
3702023-12-06T17:40:14.083Z0+0 records out
3712023-12-06T17:40:14.083Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3722023-12-06T17:40:14.083Z+ success 'ZFS vdev /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev exists'
3732023-12-06T17:40:14.083Z+ set +x
3742023-12-06T17:40:14.083ZZFS vdev /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev exists
3752023-12-06T17:40:14.083Z++ zpool list -o name
3762023-12-06T17:40:14.083Z++ grep oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930
3772023-12-06T17:40:14.083Z+ [[ -z '' ]]
3782023-12-06T17:40:14.083Z+ zpool create -f oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev
3792023-12-06T17:40:14.109Z+ success 'ZFS zpool oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 exists'
3802023-12-06T17:40:14.109Z+ set +x
3812023-12-06T17:40:14.109ZZFS zpool oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 exists
3822023-12-06T17:40:14.109Z+ for ZPOOL in "${ZPOOLS[@]}"
3832023-12-06T17:40:14.109Z+ echo 'Zpool: [oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60]'
3842023-12-06T17:40:14.109ZZpool: [oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60]
3852023-12-06T17:40:14.109Z+ VDEV_PATH=/scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev
3862023-12-06T17:40:14.110Z+ [[ -f /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev ]]
3872023-12-06T17:40:14.110Z+ dd if=/dev/zero of=/scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev bs=1 count=0 seek=20G
3882023-12-06T17:40:14.110Z0+0 records in
3892023-12-06T17:40:14.110Z0+0 records out
3902023-12-06T17:40:14.110Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3912023-12-06T17:40:14.110Z+ success 'ZFS vdev /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev exists'
3922023-12-06T17:40:14.110Z+ set +x
3932023-12-06T17:40:14.110ZZFS vdev /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev exists
3942023-12-06T17:40:14.110Z++ zpool list -o name
3952023-12-06T17:40:14.135Z++ grep oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60
3962023-12-06T17:40:14.135Z+ [[ -z '' ]]
3972023-12-06T17:40:14.135Z+ zpool create -f oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev
3982023-12-06T17:40:14.160Z+ success 'ZFS zpool oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 exists'
3992023-12-06T17:40:14.160Z+ set +x
4002023-12-06T17:40:14.160ZZFS zpool oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 exists
4012023-12-06T17:40:14.160Z+ for ZPOOL in "${ZPOOLS[@]}"
4022023-12-06T17:40:14.160Z+ echo 'Zpool: [oxp_31bd71cd-4736-4a12-a387-9b74b050396f]'
4032023-12-06T17:40:14.160Z+ VDEV_PATH=/scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev
4042023-12-06T17:40:14.160Z+ [[ -f /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev ]]
4052023-12-06T17:40:14.160Z+ dd if=/dev/zero of=/scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev bs=1 count=0 seek=20G
4062023-12-06T17:40:14.160ZZpool: [oxp_31bd71cd-4736-4a12-a387-9b74b050396f]
4072023-12-06T17:40:14.160Z0+0 records in
4082023-12-06T17:40:14.160Z0+0 records out
4092023-12-06T17:40:14.161Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4102023-12-06T17:40:14.161Z+ success 'ZFS vdev /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev exists'
4112023-12-06T17:40:14.161Z+ set +x
4122023-12-06T17:40:14.161ZZFS vdev /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev exists
4132023-12-06T17:40:14.161Z++ zpool list -o name
4142023-12-06T17:40:14.161Z++ grep oxp_31bd71cd-4736-4a12-a387-9b74b050396f
4152023-12-06T17:40:14.161Z+ [[ -z '' ]]
4162023-12-06T17:40:14.161Z+ zpool create -f oxp_31bd71cd-4736-4a12-a387-9b74b050396f /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev
4172023-12-06T17:40:14.187Z+ success 'ZFS zpool oxp_31bd71cd-4736-4a12-a387-9b74b050396f exists'
4182023-12-06T17:40:14.187Z+ set +x
4192023-12-06T17:40:14.213ZZFS zpool oxp_31bd71cd-4736-4a12-a387-9b74b050396f exists
4202023-12-06T17:40:14.213Z+ for ZPOOL in "${ZPOOLS[@]}"
4212023-12-06T17:40:14.213ZZpool: [oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7]
4222023-12-06T17:40:14.213Z+ echo 'Zpool: [oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7]'
4232023-12-06T17:40:14.213Z+ VDEV_PATH=/scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev
4242023-12-06T17:40:14.213Z+ [[ -f /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev ]]
4252023-12-06T17:40:14.214Z+ dd if=/dev/zero of=/scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev bs=1 count=0 seek=20G
4262023-12-06T17:40:14.214Z0+0 records in
4272023-12-06T17:40:14.214Z0+0 records out
4282023-12-06T17:40:14.214Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4292023-12-06T17:40:14.214Z+ success 'ZFS vdev /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev exists'
4302023-12-06T17:40:14.214Z+ set +x
4312023-12-06T17:40:14.214ZZFS vdev /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev exists
4322023-12-06T17:40:14.214Z++ zpool list -o name
4332023-12-06T17:40:14.214Z++ grep oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7
4342023-12-06T17:40:14.214Z+ [[ -z '' ]]
4352023-12-06T17:40:14.214Z+ zpool create -f oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev
4362023-12-06T17:40:14.247Z+ success 'ZFS zpool oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 exists'
4372023-12-06T17:40:14.247Z+ set +x
4382023-12-06T17:40:14.247ZZFS zpool oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 exists
4392023-12-06T17:40:14.248Z+ for ZPOOL_TYPE in "${ZPOOL_TYPES[@]}"
4402023-12-06T17:40:14.248Z+ readarray -t ZPOOLS
4412023-12-06T17:40:14.248Z++ grep '"oxi_' /opt/oxide/work/tools/../smf/sled-agent/non-gimlet/config.toml
4422023-12-06T17:40:14.248Z++ sed 's/[ ",]//g'
4432023-12-06T17:40:14.248Z+ for ZPOOL in "${ZPOOLS[@]}"
4442023-12-06T17:40:14.248Z+ echo 'Zpool: [oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b]'
4452023-12-06T17:40:14.248ZZpool: [oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b]
4462023-12-06T17:40:14.248Z+ VDEV_PATH=/scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4472023-12-06T17:40:14.248Z+ [[ -f /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
4482023-12-06T17:40:14.248Z+ dd if=/dev/zero of=/scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
4492023-12-06T17:40:14.274Z0+0 records in
4502023-12-06T17:40:14.275Z0+0 records out
4512023-12-06T17:40:14.275Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4522023-12-06T17:40:14.275Z+ success 'ZFS vdev /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
4532023-12-06T17:40:14.275Z+ set +x
4542023-12-06T17:40:14.275ZZFS vdev /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
4552023-12-06T17:40:14.275Z++ zpool list -o name
4562023-12-06T17:40:14.275Z++ grep oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b
4572023-12-06T17:40:14.275Z+ [[ -z '' ]]
4582023-12-06T17:40:14.275Z+ zpool create -f oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4592023-12-06T17:40:14.300Z+ success 'ZFS zpool oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
4602023-12-06T17:40:14.300Z+ set +x
4612023-12-06T17:40:14.300ZZFS zpool oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b exists
4622023-12-06T17:40:14.301Z+ for ZPOOL in "${ZPOOLS[@]}"
4632023-12-06T17:40:14.301Z+ echo 'Zpool: [oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b]'
4642023-12-06T17:40:14.301ZZpool: [oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b]
4652023-12-06T17:40:14.301Z+ VDEV_PATH=/scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4662023-12-06T17:40:14.301Z+ [[ -f /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
4672023-12-06T17:40:14.301Z+ dd if=/dev/zero of=/scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
4682023-12-06T17:40:14.301Z0+0 records in
4692023-12-06T17:40:14.301Z0+0 records out
4702023-12-06T17:40:14.301Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
4712023-12-06T17:40:14.301Z+ success 'ZFS vdev /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
4722023-12-06T17:40:14.301Z+ set +x
4732023-12-06T17:40:14.301ZZFS vdev /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
4742023-12-06T17:40:14.301Z++ zpool list -o name
4752023-12-06T17:40:14.302Z++ grep oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b
4762023-12-06T17:40:14.302Z+ [[ -z '' ]]
4772023-12-06T17:40:14.302Z+ zpool create -f oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4782023-12-06T17:40:14.333Z+ success 'ZFS zpool oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
4792023-12-06T17:40:14.334Z+ set +x
4802023-12-06T17:40:14.359ZZFS zpool oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b exists
4812023-12-06T17:40:14.359Z+ [[ zone == \z\o\n\e ]]
4822023-12-06T17:40:14.359Z+ ensure_simulated_links igb0
4832023-12-06T17:40:14.359Z+ local PHYSICAL_LINK=igb0
4842023-12-06T17:40:14.359Z+ INDICES=("0" "1")
4852023-12-06T17:40:14.359Z+ for I in "${INDICES[@]}"
4862023-12-06T17:40:14.359Z++ get_simnet_name_if_exists net0
4872023-12-06T17:40:14.360Z++ dladm show-simnet -p -o LINK net0
4882023-12-06T17:40:14.360Z++ echo ''
4892023-12-06T17:40:14.360Z+ [[ -z '' ]]
4902023-12-06T17:40:14.360Z+ dladm create-simnet -t net0
4912023-12-06T17:40:14.360Z+ dladm create-simnet -t sc0_0
4922023-12-06T17:40:14.360Z+ dladm modify-simnet -t -p net0 sc0_0
4932023-12-06T17:40:14.360Z+ dladm set-linkprop -p mtu=9000 sc0_0
4942023-12-06T17:40:14.360Z+ success 'Simnet net0/sc0_0 exists'
4952023-12-06T17:40:14.360Z+ set +x
4962023-12-06T17:40:14.360ZSimnet net0/sc0_0 exists
4972023-12-06T17:40:14.360Z+ for I in "${INDICES[@]}"
4982023-12-06T17:40:14.360Z++ get_simnet_name_if_exists net1
4992023-12-06T17:40:14.360Z++ dladm show-simnet -p -o LINK net1
5002023-12-06T17:40:14.360Z++ echo ''
5012023-12-06T17:40:14.387Z+ [[ -z '' ]]
5022023-12-06T17:40:14.387Z+ dladm create-simnet -t net1
5032023-12-06T17:40:14.387Z+ dladm create-simnet -t sc1_0
5042023-12-06T17:40:14.388Z+ dladm modify-simnet -t -p net1 sc1_0
5052023-12-06T17:40:14.388Z+ dladm set-linkprop -p mtu=9000 sc1_0
5062023-12-06T17:40:14.388Z+ success 'Simnet net1/sc1_0 exists'
5072023-12-06T17:40:14.388Z+ set +x
5082023-12-06T17:40:14.388ZSimnet net1/sc1_0 exists
5092023-12-06T17:40:14.388Z++ get_vnic_name_if_exists sc0_1
5102023-12-06T17:40:14.388Z++ dladm show-vnic -p -o LINK sc0_1
5112023-12-06T17:40:14.388Z++ echo ''
5122023-12-06T17:40:14.388Z+ [[ -z '' ]]
5132023-12-06T17:40:14.388Z+ dladm create-vnic -t sc0_1 -l igb0 -m a8:e1:de:01:70:1d
5142023-12-06T17:40:14.416Z+ [[ -v PROMISC_FILT_OFF ]]
5152023-12-06T17:40:14.416Z+ success 'Vnic sc0_1 exists'
5162023-12-06T17:40:14.416Z+ set +x
5172023-12-06T17:40:14.416ZVnic sc0_1 exists
5182023-12-06T17:40:14.416Z+ warn_if_no_proxy_arp
5192023-12-06T17:40:14.416Z+ [[ -v PXA_START ]]
5202023-12-06T17:40:14.416Z+ [[ -v PXA_END ]]
5212023-12-06T17:40:14.416Z+ ensure_softnpu_zone
5222023-12-06T17:40:14.416Z+ zoneadm list
5232023-12-06T17:40:14.416Z+ grep -q sidecar_softnpu
5242023-12-06T17:40:14.416Z+ [[ -f out/npuzone/npuzone ]]
5252023-12-06T17:40:14.416Z+ out/npuzone/npuzone create sidecar --omicron-zone --ports sc0_0,tfportrear0_0 --ports sc0_1,tfportqsfp0_0 --sidecar-lite-commit f0585a29fb0285f7a1220c1118856b0e5c1f75c5 --softnpu-commit dec63e67156fe6e958991bbfa090629868115ab5
5262023-12-06T17:40:14.416Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/softnpu/image/dec63e67156fe6e958991bbfa090629868115ab5/softnpu.sha256.txt
5272023-12-06T17:40:20.413Zdownloading softnpu ...done
5282023-12-06T17:40:20.438Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/sidecar-lite/release/f0585a29fb0285f7a1220c1118856b0e5c1f75c5/libsidecar_lite.so.sha256.txt
5292023-12-06T17:40:25.924Zdownloading asic_program.so ...done
5302023-12-06T17:40:25.960Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/sidecar-lite/release/f0585a29fb0285f7a1220c1118856b0e5c1f75c5/scadm.sha256.txt
5312023-12-06T17:40:32.277Zdownloading scadm ...done
5322023-12-06T17:40:32.302Zcreating port tfportrear0_0 for sc0_0
5332023-12-06T17:40:32.302Zcreating port tfportqsfp0_0 for sc0_1
5342023-12-06T17:40:32.327Zsoftnpu zone setup
5352023-12-06T17:40:32.327Zconfigure zone
5362023-12-06T17:40:32.327Zinstall zone
5372023-12-06T17:40:33.353Zboot zone
5382023-12-06T17:40:36.585Z[sidecar_softnpu] svcs milestone/network
5392023-12-06T17:40:36.611ZSTATE STIME FMRI
5402023-12-06T17:40:36.611Zonline 17:40:34 svc:/milestone/network:default
5412023-12-06T17:40:36.611Z[sidecar_softnpu] /softnpu/npu start
5422023-12-06T17:40:36.611Z
5432023-12-06T17:40:36.611Z+ /opt/oxide/work/tools/scrimlet/softnpu-init.sh
5442023-12-06T17:40:36.611Z+ GATEWAY_IP=10.151.2.100
5452023-12-06T17:40:36.611Z+ echo 'Using 10.151.2.100 as gateway ip'
5462023-12-06T17:40:36.611ZUsing 10.151.2.100 as gateway ip
5472023-12-06T17:40:36.611Z+ [[ ! -v GATEWAY_MAC ]]
5482023-12-06T17:40:36.612Z+ GATEWAY_MAC=18:c0:4d:0d:a0:2a
5492023-12-06T17:40:36.612Z++ grep -c -E '^([0-9a-fA-F]{2}:){5}[0-9a-fA-F]{2}$' /dev/fd/63
5502023-12-06T17:40:36.612Z+++ echo 18:c0:4d:0d:a0:2a
5512023-12-06T17:40:36.612Z+ COUNT=1
5522023-12-06T17:40:36.612Z+ [[ 1 -ne 1 ]]
5532023-12-06T17:40:36.612Z+ echo 'Using 18:c0:4d:0d:a0:2a as gateway mac'
5542023-12-06T17:40:36.612ZUsing 18:c0:4d:0d:a0:2a as gateway mac
5552023-12-06T17:40:36.612Z+ z_scadm add-arp-entry 10.151.2.100 18:c0:4d:0d:a0:2a
5562023-12-06T17:40:36.612Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone add-arp-entry 10.151.2.100 18:c0:4d:0d:a0:2a
5572023-12-06T17:40:36.612Z+ PXA_MAC=a8:e1:de:01:70:1d
5582023-12-06T17:40:36.612Z+ [[ -v PXA_START ]]
5592023-12-06T17:40:36.612Z+ z_scadm add-proxy-arp 10.151.2.164 10.151.2.227 a8:e1:de:01:70:1d
5602023-12-06T17:40:36.613Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone add-proxy-arp 10.151.2.164 10.151.2.227 a8:e1:de:01:70:1d
5612023-12-06T17:40:36.613Z+ z_scadm dump-state
5622023-12-06T17:40:36.613Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone dump-state
5632023-12-06T17:40:36.638Zlocal v6:
5642023-12-06T17:40:36.638Zlocal v4:
5652023-12-06T17:40:36.638Zrouter v6:
5662023-12-06T17:40:36.638Zrouter v4_idx:
5672023-12-06T17:40:36.639Zrouter v4_routes:
5682023-12-06T17:40:36.639Zresolver v4:
5692023-12-06T17:40:36.639Z10.151.2.100 -> 18:c0:4d:0d:a0:2a
5702023-12-06T17:40:36.639Zresolver v6:
5712023-12-06T17:40:36.639Znat v4:
5722023-12-06T17:40:36.639Znat_v6:
5732023-12-06T17:40:36.639Zport_mac:
5742023-12-06T17:40:36.639Zproxy arp:
5752023-12-06T17:40:36.639Z10.151.2.164/10.151.2.227: a8:e1:de:01:70:1d
5762023-12-06T17:40:36.639Z+ success 'softnpu zone exists'
5772023-12-06T17:40:36.639Z+ set +x
5782023-12-06T17:40:36.639Zsoftnpu zone exists
5792023-12-06T17:40:36.639Z
5802023-12-06T17:40:36.639Zreal 22.899636537
5812023-12-06T17:40:36.639Zuser 0.683870040
5822023-12-06T17:40:36.639Zsys 1.571312316
5832023-12-06T17:40:36.639Ztrap 0.000059857
5842023-12-06T17:40:36.639Ztflt 0.000124832
5852023-12-06T17:40:36.640Zdflt 0.000785397
5862023-12-06T17:40:36.640Zkflt 0.000000000
5872023-12-06T17:40:36.640Zlock 17:38.159356309
5882023-12-06T17:40:36.640Zslp 1:08.616945448
5892023-12-06T17:40:36.640Zlat 0.069209499
5902023-12-06T17:40:36.640Zstop 0.365279145
5912023-12-06T17:40:36.640Z+ tar xf out/omicron-sled-agent.tar pkg/config-rss.toml
5922023-12-06T17:40:36.668Z++ sed -n 's/silo_name = "\(.*\)"/\1/p' pkg/config-rss.toml
5932023-12-06T17:40:36.668Z+ SILO_NAME=recovery
5942023-12-06T17:40:36.668Z++ sed -n 's/external_dns_zone_name = "\(.*\)"/\1/p' pkg/config-rss.toml
5952023-12-06T17:40:36.668Z+ EXTERNAL_DNS_DOMAIN=oxide.test
5962023-12-06T17:40:36.669Z+ sed -i~ '
5972023-12-06T17:40:36.669Z /^external_dns_ips/c\
5982023-12-06T17:40:36.669Zexternal_dns_ips = [ "10.151.2.164", "10.151.2.165" ]
5992023-12-06T17:40:36.669Z /^\[\[internal_services_ip_pool_ranges/,/^$/ {
6002023-12-06T17:40:36.669Z /^first/c\
6012023-12-06T17:40:36.669Zfirst = "10.151.2.164"
6022023-12-06T17:40:36.669Z /^last/c\
6032023-12-06T17:40:36.669Zlast = "10.151.2.173"
6042023-12-06T17:40:36.669Z }
6052023-12-06T17:40:36.669Z /^\[rack_network_config/,/^$/ {
6062023-12-06T17:40:36.669Z /^infra_ip_first/c\
6072023-12-06T17:40:36.669Zinfra_ip_first = "10.151.2.174"
6082023-12-06T17:40:36.669Z /^infra_ip_last/c\
6092023-12-06T17:40:36.669Zinfra_ip_last = "10.151.2.174"
6102023-12-06T17:40:36.669Z }
6112023-12-06T17:40:36.670Z /^\[\[rack_network_config.ports/,/^$/ {
6122023-12-06T17:40:36.670Z /^routes/c\
6132023-12-06T17:40:36.670Zroutes = \[{nexthop = "10.151.2.100", destination = "0.0.0.0/0"}\]
6142023-12-06T17:40:36.670Z /^addresses/c\
6152023-12-06T17:40:36.670Zaddresses = \["10.151.2.174/32"\]
6162023-12-06T17:40:36.670Z }
6172023-12-06T17:40:36.670Z' pkg/config-rss.toml
6182023-12-06T17:40:36.670Z+ diff -u pkg/config-rss.toml~ pkg/config-rss.toml
6192023-12-06T17:40:36.670Z--- pkg/config-rss.toml~ Mon Jul 24 01:21:28 2006
6202023-12-06T17:40:36.670Z+++ pkg/config-rss.toml Wed Dec 6 17:40:36 2023
6212023-12-06T17:40:36.670Z@@ -41,7 +41,7 @@
6222023-12-06T17:40:36.670Z # the DNS domain delegated to the rack by the customer. Each of these addresses
6232023-12-06T17:40:36.670Z # must be contained in one of the "internal services" IP Pool ranges listed
6242023-12-06T17:40:36.670Z # below.
6252023-12-06T17:40:36.670Z-external_dns_ips = [ "192.168.1.20", "192.168.1.21" ]
6262023-12-06T17:40:36.671Z+external_dns_ips = [ "10.151.2.164", "10.151.2.165" ]
6272023-12-06T17:40:36.671Z
6282023-12-06T17:40:36.671Z # Initial TLS certificates for the external API
6292023-12-06T17:40:36.671Z #
6302023-12-06T17:40:36.671Z@@ -77,8 +77,8 @@
6312023-12-06T17:40:36.671Z #
6322023-12-06T17:40:36.671Z # For more on this and what to put here, see docs/how-to-run.adoc.
6332023-12-06T17:40:36.671Z [[internal_services_ip_pool_ranges]]
6342023-12-06T17:40:36.671Z-first = "192.168.1.20"
6352023-12-06T17:40:36.671Z-last = "192.168.1.29"
6362023-12-06T17:40:36.671Z+first = "10.151.2.164"
6372023-12-06T17:40:36.671Z+last = "10.151.2.173"
6382023-12-06T17:40:36.671Z
6392023-12-06T17:40:36.671Z # TODO - this configuration is subject to change going forward. Ultimately these
6402023-12-06T17:40:36.671Z # parameters should be provided to the control plane via wicket, but we need to
6412023-12-06T17:40:36.671Z@@ -92,8 +92,8 @@
6422023-12-06T17:40:36.671Z # A range of IP addresses used by Boundary Services on the external network. In
6432023-12-06T17:40:36.672Z # a real system, these would be addresses of the uplink ports on the Sidecar.
6442023-12-06T17:40:36.672Z # With softnpu, only one address is used.
6452023-12-06T17:40:36.672Z-infra_ip_first = "192.168.1.30"
6462023-12-06T17:40:36.672Z-infra_ip_last = "192.168.1.30"
6472023-12-06T17:40:36.672Z+infra_ip_first = "10.151.2.174"
6482023-12-06T17:40:36.672Z+infra_ip_last = "10.151.2.174"
6492023-12-06T17:40:36.672Z
6502023-12-06T17:40:36.672Z # Configurations for BGP routers to run on the scrimlets.
6512023-12-06T17:40:36.672Z bgp = []
6522023-12-06T17:40:36.672Z@@ -101,9 +101,9 @@
6532023-12-06T17:40:36.673Z # You can configure multiple uplinks by repeating the following stanza
6542023-12-06T17:40:36.673Z [[rack_network_config.ports]]
6552023-12-06T17:40:36.673Z # Routes associated with this port.
6562023-12-06T17:40:36.673Z-routes = [{nexthop = "192.168.1.199", destination = "0.0.0.0/0"}]
6572023-12-06T17:40:36.673Z+routes = [{nexthop = "10.151.2.100", destination = "0.0.0.0/0"}]
6582023-12-06T17:40:36.673Z # Addresses associated with this port.
6592023-12-06T17:40:36.673Z-addresses = ["192.168.1.30/32"]
6602023-12-06T17:40:36.673Z+addresses = ["10.151.2.174/32"]
6612023-12-06T17:40:36.673Z # Name of the uplink port. This should always be "qsfp0" when using softnpu.
6622023-12-06T17:40:36.673Z port = "qsfp0"
6632023-12-06T17:40:36.673Z # The speed of this port.
6642023-12-06T17:40:36.673Z+ true
6652023-12-06T17:40:36.673Z+ tar rvf out/omicron-sled-agent.tar pkg/config-rss.toml
6662023-12-06T17:40:36.673Za pkg/config-rss.toml 7K
6672023-12-06T17:40:36.673Z+ rm -f pkg/config-rss.toml pkg/config-rss.toml~
6682023-12-06T17:40:36.698Z+ TLS_NAME=recovery.sys.oxide.test
6692023-12-06T17:40:36.699Z+ openssl req -newkey rsa:4096 -x509 -sha256 -days 3 -nodes -out pkg/initial-tls-cert.pem -keyout pkg/initial-tls-key.pem -subj /CN=recovery.sys.oxide.test -addext subjectAltName=DNS:recovery.sys.oxide.test -addext basicConstraints=critical,CA:FALSE -config /dev/stdin
6702023-12-06T17:40:36.889Z.+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*......+......+....+.....+.+.....+......+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*............+......+...+.+..+.......+......+.....+.........+......+...............................+........+......+............+..........+..+..........+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
6712023-12-06T17:40:37.196Z............+.....+.+...+......+...+.....+...+.........+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*..+............+..................+...+......+......+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.+........+.............+.....+.+..............+......+...+...+.........................+.....+.......+............+.....+.........+...............+.+...+......+...........+.+..+...+..........+............+......+.................+.........+.........+.+...............+......+........+..........+.................+...+......+.........+......+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
6722023-12-06T17:40:37.220Z-----
6732023-12-06T17:40:37.220Z+ tar rvf out/omicron-sled-agent.tar pkg/initial-tls-cert.pem pkg/initial-tls-key.pem
6742023-12-06T17:40:37.245Za pkg/initial-tls-cert.pem 2K
6752023-12-06T17:40:37.245Za pkg/initial-tls-key.pem 4K
6762023-12-06T17:40:37.245Z+ rm -f pkg/initial-tls-cert.pem pkg/initial-tls-key.pem
6772023-12-06T17:40:37.245Z+ rmdir pkg
6782023-12-06T17:40:37.245Z+ E2E_TLS_CERT=/opt/oxide/sled-agent/pkg/initial-tls-cert.pem
6792023-12-06T17:40:37.246Z+ pfexec mkdir -p /usr/oxide
6802023-12-06T17:40:37.246Z+ pfexec rm -f /usr/oxide/tcpproxy
6812023-12-06T17:40:37.246Z+ pfexec curl -sSfL -o /usr/oxide/tcpproxy http://catacomb.eng.oxide.computer:12346/tcpproxy
6822023-12-06T17:40:37.458Z+ pfexec chmod +x /usr/oxide/tcpproxy
6832023-12-06T17:40:37.486Z+ pfexec rm -f /var/svc/manifest/site/tcpproxy.xml
6842023-12-06T17:40:37.486Z+ pfexec curl -sSfL -o /var/svc/manifest/site/tcpproxy.xml http://catacomb.eng.oxide.computer:12346/tcpproxy.xml
6852023-12-06T17:40:37.486Z+ pfexec svccfg import /var/svc/manifest/site/tcpproxy.xml
6862023-12-06T17:40:37.585Z+ OMICRON_NO_UNINSTALL=1
6872023-12-06T17:40:37.585Z+ ptime -m pfexec ./target/release/omicron-package -t test install
6882023-12-06T17:40:37.610ZDec 06 17:40:37.549 DEBG target[test]: Target({"image": "standard", "machine": "non-gimlet", "rack-topology": "single-sled", "switch": "softnpu"})
6892023-12-06T17:40:37.610ZDec 06 17:40:37.551 INFO Installing service, dst: /opt/oxide/ntp.tar.gz, src: out/ntp.tar.gz
6902023-12-06T17:40:37.610ZDec 06 17:40:37.551 INFO Installing service, dst: /opt/oxide/clickhouse_keeper.tar.gz, src: out/clickhouse_keeper.tar.gz
6912023-12-06T17:40:37.610ZDec 06 17:40:37.551 INFO Installing service, dst: /opt/oxide/propolis-server.tar.gz, src: out/propolis-server.tar.gz
6922023-12-06T17:40:37.610ZDec 06 17:40:37.551 INFO Installing service, dst: /opt/oxide/internal_dns.tar.gz, src: out/internal-dns.tar.gz
6932023-12-06T17:40:37.610ZDec 06 17:40:37.551 INFO Installing service, dst: /opt/oxide/nexus.tar.gz, src: out/omicron-nexus.tar.gz
6942023-12-06T17:40:37.610ZDec 06 17:40:37.551 INFO Installing service, dst: /opt/oxide/switch.tar.gz, src: out/switch-softnpu.tar.gz
6952023-12-06T17:40:37.611ZDec 06 17:40:37.551 INFO Installing service, dst: /opt/oxide/mg-ddm.tar, src: out/mg-ddm-gz.tar
6962023-12-06T17:40:37.611ZDec 06 17:40:37.551 INFO Installing service, dst: /opt/oxide/clickhouse.tar.gz, src: out/clickhouse.tar.gz
6972023-12-06T17:40:37.611ZDec 06 17:40:37.551 INFO Installing service, dst: /opt/oxide/crucible_pantry.tar.gz, src: out/crucible-pantry.tar.gz
6982023-12-06T17:40:37.611ZDec 06 17:40:37.551 INFO Installing service, dst: /opt/oxide/overlay.tar.gz, src: out/overlay.tar.gz
6992023-12-06T17:40:37.611ZDec 06 17:40:37.551 INFO Installing service, dst: /opt/oxide/crucible.tar.gz, src: out/crucible.tar.gz
7002023-12-06T17:40:37.611ZDec 06 17:40:37.551 INFO Installing service, dst: /opt/oxide/cockroachdb.tar.gz, src: out/cockroachdb.tar.gz
7012023-12-06T17:40:37.611ZDec 06 17:40:37.551 INFO Installing service, dst: /opt/oxide/external_dns.tar.gz, src: out/external-dns.tar.gz
7022023-12-06T17:40:37.611ZDec 06 17:40:37.551 INFO Installing service, dst: /opt/oxide/mgs.tar.gz, src: out/omicron-gateway-softnpu.tar.gz
7032023-12-06T17:40:37.611ZDec 06 17:40:37.551 INFO Installing service, dst: /opt/oxide/oximeter.tar.gz, src: out/oximeter-collector.tar.gz
7042023-12-06T17:40:37.611ZDec 06 17:40:37.551 INFO Installing service, dst: /opt/oxide/sled-agent.tar, src: out/omicron-sled-agent.tar
7052023-12-06T17:40:37.856ZDec 06 17:40:37.804 INFO Unpacking service tarball, service_path: /opt/oxide/mg-ddm, tar_path: /opt/oxide/mg-ddm.tar
7062023-12-06T17:40:37.927ZDec 06 17:40:37.838 INFO Unpacking service tarball, service_path: /opt/oxide/sled-agent, tar_path: /opt/oxide/sled-agent.tar
7072023-12-06T17:40:37.953ZDec 06 17:40:37.896 INFO Installing bootstrap service from /opt/oxide/sled-agent/pkg/manifest.xml
7082023-12-06T17:40:38.080Z
7092023-12-06T17:40:38.106Zreal 0.486016489
7102023-12-06T17:40:38.106Zuser 0.098653555
7112023-12-06T17:40:38.106Zsys 2.053520725
7122023-12-06T17:40:38.106Ztrap 0.000003607
7132023-12-06T17:40:38.106Ztflt 0.000042289
7142023-12-06T17:40:38.106Zdflt 0.000734174
7152023-12-06T17:40:38.107Zkflt 0.000008570
7162023-12-06T17:40:38.107Zlock 43.472743299
7172023-12-06T17:40:38.107Zslp 0.687227288
7182023-12-06T17:40:38.107Zlat 0.002853151
7192023-12-06T17:40:38.107Zstop 0.204157454
7202023-12-06T17:40:38.107Z+ retry=0
7212023-12-06T17:40:38.107Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7222023-12-06T17:40:38.107Z+ [[ 0 -gt 30 ]]
7232023-12-06T17:40:38.107Z+ sleep 1
7242023-12-06T17:40:39.091Z+ retry=1
7252023-12-06T17:40:39.091Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7262023-12-06T17:40:39.116Z+ [[ 1 -gt 30 ]]
7272023-12-06T17:40:39.116Z+ sleep 1
7282023-12-06T17:40:40.107Z+ retry=2
7292023-12-06T17:40:40.107Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7302023-12-06T17:40:40.133Z+ [[ 2 -gt 30 ]]
7312023-12-06T17:40:40.133Z+ sleep 1
7322023-12-06T17:40:41.121Z+ retry=3
7332023-12-06T17:40:41.121Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7342023-12-06T17:40:41.151Z+ [[ 3 -gt 30 ]]
7352023-12-06T17:40:41.151Z+ sleep 1
7362023-12-06T17:40:42.135Z+ retry=4
7372023-12-06T17:40:42.135Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7382023-12-06T17:41:02.047Z+ [[ 4 -gt 30 ]]
7392023-12-06T17:41:02.048Z+ sleep 1
7402023-12-06T17:41:03.051Z+ retry=5
7412023-12-06T17:41:03.052Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7422023-12-06T17:41:03.077Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone dump-state
7432023-12-06T17:41:03.102Zlocal v6:
7442023-12-06T17:41:03.103Zfd00:99::1
7452023-12-06T17:41:03.103Zfe80::60fc:8ff:fe95:bcc8
7462023-12-06T17:41:03.103Zlocal v4:
7472023-12-06T17:41:03.103Z10.151.2.174
7482023-12-06T17:41:03.103Zrouter v6:
7492023-12-06T17:41:03.103Zrouter v4_idx:
7502023-12-06T17:41:03.103Z0.0.0.0/0 -> 0
7512023-12-06T17:41:03.103Zrouter v4_routes:
7522023-12-06T17:41:03.103Z0 -> 10.151.2.100 (2)
7532023-12-06T17:41:03.103Zresolver v4:
7542023-12-06T17:41:03.103Z10.151.2.100 -> 18:c0:4d:0d:a0:2a
7552023-12-06T17:41:03.103Zresolver v6:
7562023-12-06T17:41:03.104Zfe80::8c48:cfff:feaf:7def -> 8e:48:cf:af:7d:ef
7572023-12-06T17:41:03.104Zfe80::60fc:8ff:fe95:bcc8 -> 62:fc:08:95:bc:c8
7582023-12-06T17:41:03.104Znat v4:
7592023-12-06T17:41:03.104Znat_v6:
7602023-12-06T17:41:03.104Zport_mac:
7612023-12-06T17:41:03.104Z1000: a8:40:25:7f:8b:6b
7622023-12-06T17:41:03.104Z1: a8:40:25:7f:8b:6c
7632023-12-06T17:41:03.104Zproxy arp:
7642023-12-06T17:41:03.104Z10.151.2.164/10.151.2.227: a8:e1:de:01:70:1d
7652023-12-06T17:41:03.104Z+ retry=0
7662023-12-06T17:41:03.104Z+++ zoneadm list -n
7672023-12-06T17:41:03.104Z+++ grep oxz_ntp
7682023-12-06T17:41:03.129Z++ pfexec svcs -z -Hostate oxide/ntp
7692023-12-06T17:41:03.129Zsvcs: zone '-Hostate': entity not found
7702023-12-06T17:41:03.130Z++ true
7712023-12-06T17:41:03.130Z+ [[ '' != online ]]
7722023-12-06T17:41:03.130Z+ [[ 0 -gt 60 ]]
7732023-12-06T17:41:03.130Z+ sleep 1
7742023-12-06T17:41:04.114Z+ retry=1
7752023-12-06T17:41:04.140Z+++ zoneadm list -n
7762023-12-06T17:41:04.140Z+++ grep oxz_ntp
7772023-12-06T17:41:04.140Z++ pfexec svcs -z -Hostate oxide/ntp
7782023-12-06T17:41:04.140Zsvcs: zone '-Hostate': entity not found
7792023-12-06T17:41:04.140Z++ true
7802023-12-06T17:41:04.141Z+ [[ '' != online ]]
7812023-12-06T17:41:04.141Z+ [[ 1 -gt 60 ]]
7822023-12-06T17:41:04.141Z+ sleep 1
7832023-12-06T17:41:05.136Z+ retry=2
7842023-12-06T17:41:05.165Z+++ zoneadm list -n
7852023-12-06T17:41:05.165Z+++ grep oxz_ntp
7862023-12-06T17:41:05.165Z++ pfexec svcs -z -Hostate oxide/ntp
7872023-12-06T17:41:05.165Zsvcs: zone '-Hostate': entity not found
7882023-12-06T17:41:05.165Z++ true
7892023-12-06T17:41:05.165Z+ [[ '' != online ]]
7902023-12-06T17:41:05.165Z+ [[ 2 -gt 60 ]]
7912023-12-06T17:41:05.166Z+ sleep 1
7922023-12-06T17:41:06.158Z+ retry=3
7932023-12-06T17:41:06.185Z+++ zoneadm list -n
7942023-12-06T17:41:06.185Z+++ grep oxz_ntp
7952023-12-06T17:41:06.185Z++ pfexec svcs -z oxz_ntp_0e075832-9eaa-40c5-a328-dcde3812ee47 -Hostate oxide/ntp
7962023-12-06T17:41:06.225Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
7972023-12-06T17:41:06.251Z++ true
7982023-12-06T17:41:06.251Z+ [[ '' != online ]]
7992023-12-06T17:41:06.251Z+ [[ 3 -gt 60 ]]
8002023-12-06T17:41:06.251Z+ sleep 1
8012023-12-06T17:41:07.230Z+ retry=4
8022023-12-06T17:41:07.257Z+++ zoneadm list -n
8032023-12-06T17:41:07.257Z+++ grep oxz_ntp
8042023-12-06T17:41:07.257Z++ pfexec svcs -z oxz_ntp_0e075832-9eaa-40c5-a328-dcde3812ee47 -Hostate oxide/ntp
8052023-12-06T17:41:07.282Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
8062023-12-06T17:41:07.282Z++ true
8072023-12-06T17:41:07.282Z+ [[ '' != online ]]
8082023-12-06T17:41:07.282Z+ [[ 4 -gt 60 ]]
8092023-12-06T17:41:07.283Z+ sleep 1
8102023-12-06T17:41:08.262Z+ retry=5
8112023-12-06T17:41:08.287Z+++ zoneadm list -n
8122023-12-06T17:41:08.287Z+++ grep oxz_ntp
8132023-12-06T17:41:08.287Z++ pfexec svcs -z oxz_ntp_0e075832-9eaa-40c5-a328-dcde3812ee47 -Hostate oxide/ntp
8142023-12-06T17:41:08.312Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
8152023-12-06T17:41:08.312Z++ true
8162023-12-06T17:41:08.312Z+ [[ '' != online ]]
8172023-12-06T17:41:08.312Z+ [[ 5 -gt 60 ]]
8182023-12-06T17:41:08.312Z+ sleep 1
8192023-12-06T17:41:09.301Z+ retry=6
8202023-12-06T17:41:09.329Z+++ zoneadm list -n
8212023-12-06T17:41:09.329Z+++ grep oxz_ntp
8222023-12-06T17:41:09.329Z++ pfexec svcs -z oxz_ntp_0e075832-9eaa-40c5-a328-dcde3812ee47 -Hostate oxide/ntp
8232023-12-06T17:41:09.330Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
8242023-12-06T17:41:09.330Z++ true
8252023-12-06T17:41:09.330Z+ [[ '' != online ]]
8262023-12-06T17:41:09.330Z+ [[ 6 -gt 60 ]]
8272023-12-06T17:41:09.330Z+ sleep 1
8282023-12-06T17:41:10.326Z+ retry=7
8292023-12-06T17:41:10.352Z+++ zoneadm list -n
8302023-12-06T17:41:10.352Z+++ grep oxz_ntp
8312023-12-06T17:41:10.353Z++ pfexec svcs -z oxz_ntp_0e075832-9eaa-40c5-a328-dcde3812ee47 -Hostate oxide/ntp
8322023-12-06T17:41:10.353Z+ [[ disabled != online ]]
8332023-12-06T17:41:10.353Z+ [[ 7 -gt 60 ]]
8342023-12-06T17:41:10.353Z+ sleep 1
8352023-12-06T17:41:11.351Z+ retry=8
8362023-12-06T17:41:11.380Z+++ zoneadm list -n
8372023-12-06T17:41:11.381Z+++ grep oxz_ntp
8382023-12-06T17:41:11.381Z++ pfexec svcs -z oxz_ntp_0e075832-9eaa-40c5-a328-dcde3812ee47 -Hostate oxide/ntp
8392023-12-06T17:41:11.381Z+ [[ disabled != online ]]
8402023-12-06T17:41:11.381Z+ [[ 8 -gt 60 ]]
8412023-12-06T17:41:11.381Z+ sleep 1
8422023-12-06T17:41:12.369Z+ retry=9
8432023-12-06T17:41:12.395Z+++ zoneadm list -n
8442023-12-06T17:41:12.395Z+++ grep oxz_ntp
8452023-12-06T17:41:12.395Z++ pfexec svcs -z oxz_ntp_0e075832-9eaa-40c5-a328-dcde3812ee47 -Hostate oxide/ntp
8462023-12-06T17:41:12.396Z+ [[ disabled != online ]]
8472023-12-06T17:41:12.396Z+ [[ 9 -gt 60 ]]
8482023-12-06T17:41:12.396Z+ sleep 1
8492023-12-06T17:41:13.392Z+ retry=10
8502023-12-06T17:41:13.418Z+++ zoneadm list -n
8512023-12-06T17:41:13.418Z+++ grep oxz_ntp
8522023-12-06T17:41:13.418Z++ pfexec svcs -z oxz_ntp_0e075832-9eaa-40c5-a328-dcde3812ee47 -Hostate oxide/ntp
8532023-12-06T17:41:13.418Z+ [[ disabled != online ]]
8542023-12-06T17:41:13.418Z+ [[ 10 -gt 60 ]]
8552023-12-06T17:41:13.418Z+ sleep 1
8562023-12-06T17:41:14.415Z+ retry=11
8572023-12-06T17:41:14.440Z+++ zoneadm list -n
8582023-12-06T17:41:14.440Z+++ grep oxz_ntp
8592023-12-06T17:41:14.440Z++ pfexec svcs -z oxz_ntp_0e075832-9eaa-40c5-a328-dcde3812ee47 -Hostate oxide/ntp
8602023-12-06T17:41:14.440Z+ [[ disabled != online ]]
8612023-12-06T17:41:14.440Z+ [[ 11 -gt 60 ]]
8622023-12-06T17:41:14.440Z+ sleep 1
8632023-12-06T17:41:15.438Z+ retry=12
8642023-12-06T17:41:15.464Z+++ zoneadm list -n
8652023-12-06T17:41:15.464Z+++ grep oxz_ntp
8662023-12-06T17:41:15.464Z++ pfexec svcs -z oxz_ntp_0e075832-9eaa-40c5-a328-dcde3812ee47 -Hostate oxide/ntp
8672023-12-06T17:41:15.464Z+ [[ disabled != online ]]
8682023-12-06T17:41:15.464Z+ [[ 12 -gt 60 ]]
8692023-12-06T17:41:15.464Z+ sleep 1
8702023-12-06T17:41:16.460Z+ retry=13
8712023-12-06T17:41:16.485Z+++ zoneadm list -n
8722023-12-06T17:41:16.485Z+++ grep oxz_ntp
8732023-12-06T17:41:16.485Z++ pfexec svcs -z oxz_ntp_0e075832-9eaa-40c5-a328-dcde3812ee47 -Hostate oxide/ntp
8742023-12-06T17:41:16.485Z+ [[ disabled != online ]]
8752023-12-06T17:41:16.485Z+ [[ 13 -gt 60 ]]
8762023-12-06T17:41:16.485Z+ sleep 1
8772023-12-06T17:41:17.482Z+ retry=14
8782023-12-06T17:41:17.507Z+++ zoneadm list -n
8792023-12-06T17:41:17.507Z+++ grep oxz_ntp
8802023-12-06T17:41:17.507Z++ pfexec svcs -z oxz_ntp_0e075832-9eaa-40c5-a328-dcde3812ee47 -Hostate oxide/ntp
8812023-12-06T17:41:17.507Z+ [[ disabled != online ]]
8822023-12-06T17:41:17.507Z+ [[ 14 -gt 60 ]]
8832023-12-06T17:41:17.507Z+ sleep 1
8842023-12-06T17:41:18.509Z+ retry=15
8852023-12-06T17:41:18.534Z+++ zoneadm list -n
8862023-12-06T17:41:18.534Z+++ grep oxz_ntp
8872023-12-06T17:41:18.534Z++ pfexec svcs -z oxz_ntp_0e075832-9eaa-40c5-a328-dcde3812ee47 -Hostate oxide/ntp
8882023-12-06T17:41:18.534Z+ [[ disabled != online ]]
8892023-12-06T17:41:18.534Z+ [[ 15 -gt 60 ]]
8902023-12-06T17:41:18.534Z+ sleep 1
 
8912023-12-06T17:41:18.724Zsock console login: Dec 6 17:41:18 sock ip: ip: joining multicasts failed (4) on vopte0 - will use link layer broadcasts for multicast
 
8922023-12-06T17:41:19.529Z+ retry=16
8932023-12-06T17:41:19.554Z+++ zoneadm list -n
8942023-12-06T17:41:19.554Z+++ grep oxz_ntp
8952023-12-06T17:41:19.554Z++ pfexec svcs -z oxz_ntp_0e075832-9eaa-40c5-a328-dcde3812ee47 -Hostate oxide/ntp
8962023-12-06T17:41:19.554Z+ [[ online != online ]]
8972023-12-06T17:41:19.554Z+ echo 'Waited for chrony: 16s'
8982023-12-06T17:41:19.555Z+ export RUST_BACKTRACE=1
8992023-12-06T17:41:19.555ZWaited for chrony: 16s
9002023-12-06T17:41:19.555Z+ RUST_BACKTRACE=1
9012023-12-06T17:41:19.555Z+ export E2E_TLS_CERT IPPOOL_START IPPOOL_END
9022023-12-06T17:41:19.555Z+ ./tests/bootstrap
9032023-12-06T17:41:19.589ZUsing external DNS server at 10.151.2.164:53
9042023-12-06T17:41:19.589Z2023-12-06 17:41:19.523640990 UTC: attempting to log into API
9052023-12-06T17:41:34.613Z2023-12-06 17:41:34.586127613 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9062023-12-06T17:41:35.613Z2023-12-06 17:41:35.589411724 UTC: attempting to log into API
9072023-12-06T17:41:50.646Z2023-12-06 17:41:50.634467263 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9082023-12-06T17:41:51.647Z2023-12-06 17:41:51.635104982 UTC: attempting to log into API
9092023-12-06T17:42:06.681Z2023-12-06 17:42:06.669371271 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9102023-12-06T17:42:07.682Z2023-12-06 17:42:07.670744894 UTC: attempting to log into API
 
9112023-12-06T17:42:19.464ZDec 6 17:42:18 sock ip: ip: joining multicasts failed (4) on vopte4 - will use link layer broadcasts for multicast
 
9122023-12-06T17:42:22.740Z2023-12-06 17:42:22.728672800 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9132023-12-06T17:42:23.742Z2023-12-06 17:42:23.730627815 UTC: attempting to log into API
 
9142023-12-06T17:42:26.154ZDec 6 17:42:25 sock ip: ip: joining multicasts failed (4) on vopte1 - will use link layer broadcasts for multicast
9152023-12-06T17:42:32.805ZDec 6 17:42:32 sock ip: ip: joining multicasts failed (4) on vopte2 - will use link layer broadcasts for multicast
 
9162023-12-06T17:42:38.775Z2023-12-06 17:42:38.763161775 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9172023-12-06T17:42:39.788Z2023-12-06 17:42:39.763909786 UTC: attempting to log into API
 
9182023-12-06T17:42:43.429ZDec 6 17:42:39 sock ip: ip: joining multicasts failed (4) on vopte3 - will use link layer broadcasts for multicast
9192023-12-06T17:42:45.594ZDec 6 17:42:44 sock ip: ip: joining multicasts failed (4) on vopte5 - will use link layer broadcasts for multicast
 
9202023-12-06T17:42:54.836Z2023-12-06 17:42:54.796987367 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9212023-12-06T17:42:55.813Z2023-12-06 17:42:55.797988283 UTC: attempting to log into API
9222023-12-06T17:43:10.844Z2023-12-06 17:43:10.832224646 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9232023-12-06T17:43:11.845Z2023-12-06 17:43:11.833204742 UTC: attempting to log into API
9242023-12-06T17:43:26.876Z2023-12-06 17:43:26.865203570 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9252023-12-06T17:43:27.878Z2023-12-06 17:43:27.866232707 UTC: attempting to log into API
9262023-12-06T17:43:37.930Z2023-12-06 17:43:37.918681483 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: dns error: no record found for Query { name: Name("recovery.sys.oxide.test."), query_type: AAAA, query_class: IN }
9272023-12-06T17:43:38.931Z2023-12-06 17:43:38.919821122 UTC: attempting to log into API
9282023-12-06T17:43:39.718Z2023-12-06 17:43:39.706245890 UTC: login succeeded
9292023-12-06T17:43:39.750Zwaiting for nexus to come up...
9302023-12-06T17:43:39.821Zcreating IP pool... 10.151.2.175 - 10.151.2.227
9312023-12-06T17:43:39.900Zensuring datasets are ready...
9322023-12-06T17:43:44.587Zlet's roll.
9332023-12-06T17:43:44.612Z+ rm ./tests/bootstrap
9342023-12-06T17:43:44.659Z+ for test_bin in tests/*
9352023-12-06T17:43:44.659Z+ ./tests/bootstrap-be9c817b94af4e8a
9362023-12-06T17:43:44.684Z
9372023-12-06T17:43:44.684Zrunning 0 tests
9382023-12-06T17:43:44.684Z
9392023-12-06T17:43:44.684Ztest result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
9402023-12-06T17:43:44.684Z
9412023-12-06T17:43:44.684Z+ for test_bin in tests/*
9422023-12-06T17:43:44.684Z+ ./tests/end_to_end_tests-ff923d3f5c6228d4
9432023-12-06T17:43:44.684Z
9442023-12-06T17:43:44.684Zrunning 2 tests
9452023-12-06T17:43:44.903Ztest no_spoof::no_spoof_here ... ok
9462023-12-06T17:44:44.684Ztest instance_launch::instance_launch has been running for over 60 seconds
9472023-12-06T17:46:43.714Ztest instance_launch::instance_launch ... ok
9482023-12-06T17:46:43.714Z
9492023-12-06T17:46:43.714Ztest result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 179.03s
9502023-12-06T17:46:43.714Z
9512023-12-06T17:46:43.739Z+ _exit_trap
9522023-12-06T17:46:43.739Z+ local status=0
9532023-12-06T17:46:43.739Z+ [[ 0 -eq 0 ]]
9542023-12-06T17:46:43.739Z+ exit 0
9552023-12-06T17:46:43.739Zprocess exited: duration 397921 ms, exit code 0
 
9562023-12-06T17:46:43.840Zfound 90 output files
9572023-12-06T17:46:43.840Zuploading: /var/svc/log/oxide-sled-agent:default.log (644309 bytes)
9582023-12-06T17:46:45.133Zuploaded: /var/svc/log/oxide-sled-agent:default.log
9592023-12-06T17:46:45.160Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_65a5c94d-7dd3-419e-b0c4-23533a58135d/root/var/svc/log/oxide-crucible-agent:default.log (4324 bytes)
9602023-12-06T17:46:46.231Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_65a5c94d-7dd3-419e-b0c4-23533a58135d/root/var/svc/log/oxide-crucible-agent:default.log
9612023-12-06T17:46:46.231Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_65a5c94d-7dd3-419e-b0c4-23533a58135d/root/var/svc/log/oxide-crucible-agent:default.log.0 (9794 bytes)
9622023-12-06T17:46:47.359Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_65a5c94d-7dd3-419e-b0c4-23533a58135d/root/var/svc/log/oxide-crucible-agent:default.log.0
9632023-12-06T17:46:47.359Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_65a5c94d-7dd3-419e-b0c4-23533a58135d/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
9642023-12-06T17:46:48.434Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_65a5c94d-7dd3-419e-b0c4-23533a58135d/root/var/svc/log/oxide-crucible-downstairs:default.log
9652023-12-06T17:46:48.434Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_65a5c94d-7dd3-419e-b0c4-23533a58135d/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
9662023-12-06T17:46:49.536Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_65a5c94d-7dd3-419e-b0c4-23533a58135d/root/var/svc/log/oxide-crucible-downstairs:default.log.0
9672023-12-06T17:46:49.536Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_65a5c94d-7dd3-419e-b0c4-23533a58135d/root/var/svc/log/oxide-crucible-downstairs:downstairs-3566aab8-f46e-477f-afef-b12bd7a042ce.log (1006 bytes)
9682023-12-06T17:46:50.631Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_65a5c94d-7dd3-419e-b0c4-23533a58135d/root/var/svc/log/oxide-crucible-downstairs:downstairs-3566aab8-f46e-477f-afef-b12bd7a042ce.log
9692023-12-06T17:46:50.632Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_65a5c94d-7dd3-419e-b0c4-23533a58135d/root/var/svc/log/oxide-crucible-downstairs:downstairs-3566aab8-f46e-477f-afef-b12bd7a042ce.log.0 (4529 bytes)
9702023-12-06T17:46:51.741Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_65a5c94d-7dd3-419e-b0c4-23533a58135d/root/var/svc/log/oxide-crucible-downstairs:downstairs-3566aab8-f46e-477f-afef-b12bd7a042ce.log.0
9712023-12-06T17:46:51.741Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_f92ea1e7-347c-41d1-bd44-a15ef90d284c/root/var/svc/log/oxide-crucible-agent:default.log (0 bytes)
9722023-12-06T17:46:52.825Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_f92ea1e7-347c-41d1-bd44-a15ef90d284c/root/var/svc/log/oxide-crucible-agent:default.log
9732023-12-06T17:46:52.825Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_f92ea1e7-347c-41d1-bd44-a15ef90d284c/root/var/svc/log/oxide-crucible-agent:default.log.0 (14543 bytes)
9742023-12-06T17:46:56.863Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_f92ea1e7-347c-41d1-bd44-a15ef90d284c/root/var/svc/log/oxide-crucible-agent:default.log.0
9752023-12-06T17:46:56.863Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_f92ea1e7-347c-41d1-bd44-a15ef90d284c/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
9762023-12-06T17:46:57.956Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_f92ea1e7-347c-41d1-bd44-a15ef90d284c/root/var/svc/log/oxide-crucible-downstairs:default.log
9772023-12-06T17:46:57.956Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_f92ea1e7-347c-41d1-bd44-a15ef90d284c/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
9782023-12-06T17:46:59.155Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_f92ea1e7-347c-41d1-bd44-a15ef90d284c/root/var/svc/log/oxide-crucible-downstairs:default.log.0
9792023-12-06T17:46:59.155Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_f92ea1e7-347c-41d1-bd44-a15ef90d284c/root/var/svc/log/oxide-crucible-downstairs:downstairs-67c384bf-4e8a-470a-b59e-0dc3477b6543.log (0 bytes)
9802023-12-06T17:47:00.287Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_f92ea1e7-347c-41d1-bd44-a15ef90d284c/root/var/svc/log/oxide-crucible-downstairs:downstairs-67c384bf-4e8a-470a-b59e-0dc3477b6543.log
9812023-12-06T17:47:00.315Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_f92ea1e7-347c-41d1-bd44-a15ef90d284c/root/var/svc/log/oxide-crucible-downstairs:downstairs-67c384bf-4e8a-470a-b59e-0dc3477b6543.log.0 (3051 bytes)
9822023-12-06T17:47:01.390Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_f92ea1e7-347c-41d1-bd44-a15ef90d284c/root/var/svc/log/oxide-crucible-downstairs:downstairs-67c384bf-4e8a-470a-b59e-0dc3477b6543.log.0
9832023-12-06T17:47:01.390Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_0bfeea29-a47a-4323-893a-53a06601ad8e/root/var/svc/log/oxide-crucible-pantry:default.log (0 bytes)
9842023-12-06T17:47:02.465Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_0bfeea29-a47a-4323-893a-53a06601ad8e/root/var/svc/log/oxide-crucible-pantry:default.log
9852023-12-06T17:47:02.465Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_0bfeea29-a47a-4323-893a-53a06601ad8e/root/var/svc/log/oxide-crucible-pantry:default.log.0 (2338 bytes)
9862023-12-06T17:47:03.571Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_0bfeea29-a47a-4323-893a-53a06601ad8e/root/var/svc/log/oxide-crucible-pantry:default.log.0
9872023-12-06T17:47:03.571Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_d2016ddd-0efc-4439-89aa-2f166d289e6f/root/var/svc/log/oxide-nexus:default.log (79431 bytes)
9882023-12-06T17:47:03.751Zupload warning: file "/pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_d2016ddd-0efc-4439-89aa-2f166d289e6f/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 79431 -> 143884
9892023-12-06T17:47:04.824Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_d2016ddd-0efc-4439-89aa-2f166d289e6f/root/var/svc/log/oxide-nexus:default.log
9902023-12-06T17:47:04.824Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_d2016ddd-0efc-4439-89aa-2f166d289e6f/root/var/svc/log/oxide-nexus:default.log.0 (508124 bytes)
9912023-12-06T17:47:06.127Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_d2016ddd-0efc-4439-89aa-2f166d289e6f/root/var/svc/log/oxide-nexus:default.log.0
9922023-12-06T17:47:06.128Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_c9428da0-985f-4421-8de9-d60f987ba8ab/root/var/svc/log/oxide-cockroachdb:default.log (0 bytes)
9932023-12-06T17:47:07.238Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_c9428da0-985f-4421-8de9-d60f987ba8ab/root/var/svc/log/oxide-cockroachdb:default.log
9942023-12-06T17:47:07.238Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_c9428da0-985f-4421-8de9-d60f987ba8ab/root/var/svc/log/oxide-cockroachdb:default.log.0 (5368 bytes)
9952023-12-06T17:47:08.333Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_c9428da0-985f-4421-8de9-d60f987ba8ab/root/var/svc/log/oxide-cockroachdb:default.log.0
9962023-12-06T17:47:08.333Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_51de6a17-3c29-4573-9303-268fbfac8ee2/root/var/svc/log/oxide-crucible-agent:default.log (0 bytes)
9972023-12-06T17:47:09.409Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_51de6a17-3c29-4573-9303-268fbfac8ee2/root/var/svc/log/oxide-crucible-agent:default.log
9982023-12-06T17:47:09.409Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_51de6a17-3c29-4573-9303-268fbfac8ee2/root/var/svc/log/oxide-crucible-agent:default.log.0 (3435 bytes)
9992023-12-06T17:47:10.512Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_51de6a17-3c29-4573-9303-268fbfac8ee2/root/var/svc/log/oxide-crucible-agent:default.log.0
10002023-12-06T17:47:10.513Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_51de6a17-3c29-4573-9303-268fbfac8ee2/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
10012023-12-06T17:47:11.589Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_51de6a17-3c29-4573-9303-268fbfac8ee2/root/var/svc/log/oxide-crucible-downstairs:default.log
10022023-12-06T17:47:11.589Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_51de6a17-3c29-4573-9303-268fbfac8ee2/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
10032023-12-06T17:47:12.764Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_51de6a17-3c29-4573-9303-268fbfac8ee2/root/var/svc/log/oxide-crucible-downstairs:default.log.0
10042023-12-06T17:47:12.764Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_f4bd4239-a995-46ad-bd81-8c0c7f2f9049/root/var/svc/log/oxide-external_dns:default.log (3777 bytes)
10052023-12-06T17:47:12.836Zupload warning: file "/pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_f4bd4239-a995-46ad-bd81-8c0c7f2f9049/root/var/svc/log/oxide-external_dns:default.log" changed size mid upload: 3777 -> 7551
10062023-12-06T17:47:13.911Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_f4bd4239-a995-46ad-bd81-8c0c7f2f9049/root/var/svc/log/oxide-external_dns:default.log
10072023-12-06T17:47:13.944Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_f4bd4239-a995-46ad-bd81-8c0c7f2f9049/root/var/svc/log/oxide-external_dns:default.log.0 (11831 bytes)
10082023-12-06T17:47:15.033Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_f4bd4239-a995-46ad-bd81-8c0c7f2f9049/root/var/svc/log/oxide-external_dns:default.log.0
10092023-12-06T17:47:15.033Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_cockroachdb_5c732813-f138-434d-bcc4-6ed72087dd45/root/var/svc/log/oxide-cockroachdb:default.log (0 bytes)
10102023-12-06T17:47:16.108Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_cockroachdb_5c732813-f138-434d-bcc4-6ed72087dd45/root/var/svc/log/oxide-cockroachdb:default.log
10112023-12-06T17:47:16.108Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_cockroachdb_5c732813-f138-434d-bcc4-6ed72087dd45/root/var/svc/log/oxide-cockroachdb:default.log.0 (5342 bytes)
10122023-12-06T17:47:17.223Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_cockroachdb_5c732813-f138-434d-bcc4-6ed72087dd45/root/var/svc/log/oxide-cockroachdb:default.log.0
10132023-12-06T17:47:17.223Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_9bbb87d1-f50a-41e0-85ed-19784220ea16/root/var/svc/log/oxide-crucible-agent:default.log (0 bytes)
10142023-12-06T17:47:18.300Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_9bbb87d1-f50a-41e0-85ed-19784220ea16/root/var/svc/log/oxide-crucible-agent:default.log
10152023-12-06T17:47:18.300Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_9bbb87d1-f50a-41e0-85ed-19784220ea16/root/var/svc/log/oxide-crucible-agent:default.log.0 (3434 bytes)
10162023-12-06T17:47:19.400Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_9bbb87d1-f50a-41e0-85ed-19784220ea16/root/var/svc/log/oxide-crucible-agent:default.log.0
10172023-12-06T17:47:19.400Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_9bbb87d1-f50a-41e0-85ed-19784220ea16/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
10182023-12-06T17:47:20.481Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_9bbb87d1-f50a-41e0-85ed-19784220ea16/root/var/svc/log/oxide-crucible-downstairs:default.log
10192023-12-06T17:47:20.520Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_9bbb87d1-f50a-41e0-85ed-19784220ea16/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
10202023-12-06T17:47:21.583Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_9bbb87d1-f50a-41e0-85ed-19784220ea16/root/var/svc/log/oxide-crucible-downstairs:default.log.0
10212023-12-06T17:47:21.584Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_pantry_f6447f87-90dd-43a9-ac51-9e1831c10c3d/root/var/svc/log/oxide-crucible-pantry:default.log (0 bytes)
10222023-12-06T17:47:22.658Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_pantry_f6447f87-90dd-43a9-ac51-9e1831c10c3d/root/var/svc/log/oxide-crucible-pantry:default.log
10232023-12-06T17:47:22.658Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_pantry_f6447f87-90dd-43a9-ac51-9e1831c10c3d/root/var/svc/log/oxide-crucible-pantry:default.log.0 (2333 bytes)
10242023-12-06T17:47:23.756Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_pantry_f6447f87-90dd-43a9-ac51-9e1831c10c3d/root/var/svc/log/oxide-crucible-pantry:default.log.0
10252023-12-06T17:47:23.756Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_cockroachdb_8f3a1d19-d2a3-45ba-b935-68f0390bcdf8/root/var/svc/log/oxide-cockroachdb:default.log (0 bytes)
10262023-12-06T17:47:24.831Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_cockroachdb_8f3a1d19-d2a3-45ba-b935-68f0390bcdf8/root/var/svc/log/oxide-cockroachdb:default.log
10272023-12-06T17:47:24.831Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_cockroachdb_8f3a1d19-d2a3-45ba-b935-68f0390bcdf8/root/var/svc/log/oxide-cockroachdb:default.log.0 (5368 bytes)
10282023-12-06T17:47:25.932Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_cockroachdb_8f3a1d19-d2a3-45ba-b935-68f0390bcdf8/root/var/svc/log/oxide-cockroachdb:default.log.0
10292023-12-06T17:47:25.932Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_cockroachdb_ccbe62b8-1461-4dec-a1a5-93b3a022681d/root/var/svc/log/oxide-cockroachdb:default.log (0 bytes)
10302023-12-06T17:47:27.007Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_cockroachdb_ccbe62b8-1461-4dec-a1a5-93b3a022681d/root/var/svc/log/oxide-cockroachdb:default.log
10312023-12-06T17:47:27.007Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_cockroachdb_ccbe62b8-1461-4dec-a1a5-93b3a022681d/root/var/svc/log/oxide-cockroachdb:default.log.0 (5367 bytes)
10322023-12-06T17:47:28.104Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_cockroachdb_ccbe62b8-1461-4dec-a1a5-93b3a022681d/root/var/svc/log/oxide-cockroachdb:default.log.0
10332023-12-06T17:47:28.104Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_internal_dns_ac3d1180-e1c3-477d-a802-3f8bd92e2528/root/var/svc/log/oxide-internal_dns:default.log (13835 bytes)
10342023-12-06T17:47:28.213Zupload warning: file "/pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_internal_dns_ac3d1180-e1c3-477d-a802-3f8bd92e2528/root/var/svc/log/oxide-internal_dns:default.log" changed size mid upload: 13835 -> 17588
10352023-12-06T17:47:29.293Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_internal_dns_ac3d1180-e1c3-477d-a802-3f8bd92e2528/root/var/svc/log/oxide-internal_dns:default.log
10362023-12-06T17:47:29.293Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_internal_dns_ac3d1180-e1c3-477d-a802-3f8bd92e2528/root/var/svc/log/oxide-internal_dns:default.log.0 (106042 bytes)
10372023-12-06T17:47:30.550Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_internal_dns_ac3d1180-e1c3-477d-a802-3f8bd92e2528/root/var/svc/log/oxide-internal_dns:default.log.0
10382023-12-06T17:47:30.550Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_internal_dns_be8d8c76-3f0a-4bd9-8466-47f0b72402ea/root/var/svc/log/oxide-internal_dns:default.log (3756 bytes)
10392023-12-06T17:47:30.624Zupload warning: file "/pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_internal_dns_be8d8c76-3f0a-4bd9-8466-47f0b72402ea/root/var/svc/log/oxide-internal_dns:default.log" changed size mid upload: 3756 -> 7511
10402023-12-06T17:47:31.700Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_internal_dns_be8d8c76-3f0a-4bd9-8466-47f0b72402ea/root/var/svc/log/oxide-internal_dns:default.log
10412023-12-06T17:47:31.700Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_internal_dns_be8d8c76-3f0a-4bd9-8466-47f0b72402ea/root/var/svc/log/oxide-internal_dns:default.log.0 (112460 bytes)
10422023-12-06T17:47:32.923Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_internal_dns_be8d8c76-3f0a-4bd9-8466-47f0b72402ea/root/var/svc/log/oxide-internal_dns:default.log.0
10432023-12-06T17:47:32.949Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_993ea282-1eee-42e3-b3d8-ea8aa4c06162/root/var/svc/log/oxide-crucible-agent:default.log (4322 bytes)
10442023-12-06T17:47:34.031Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_993ea282-1eee-42e3-b3d8-ea8aa4c06162/root/var/svc/log/oxide-crucible-agent:default.log
10452023-12-06T17:47:34.031Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_993ea282-1eee-42e3-b3d8-ea8aa4c06162/root/var/svc/log/oxide-crucible-agent:default.log.0 (20471 bytes)
10462023-12-06T17:47:35.180Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_993ea282-1eee-42e3-b3d8-ea8aa4c06162/root/var/svc/log/oxide-crucible-agent:default.log.0
10472023-12-06T17:47:35.180Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_993ea282-1eee-42e3-b3d8-ea8aa4c06162/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
10482023-12-06T17:47:36.257Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_993ea282-1eee-42e3-b3d8-ea8aa4c06162/root/var/svc/log/oxide-crucible-downstairs:default.log
10492023-12-06T17:47:36.258Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_993ea282-1eee-42e3-b3d8-ea8aa4c06162/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
10502023-12-06T17:47:37.369Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_993ea282-1eee-42e3-b3d8-ea8aa4c06162/root/var/svc/log/oxide-crucible-downstairs:default.log.0
10512023-12-06T17:47:37.369Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_993ea282-1eee-42e3-b3d8-ea8aa4c06162/root/var/svc/log/oxide-crucible-downstairs:downstairs-1d304be3-4f03-4335-afb2-dbce24245753.log (0 bytes)
10522023-12-06T17:47:38.460Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_993ea282-1eee-42e3-b3d8-ea8aa4c06162/root/var/svc/log/oxide-crucible-downstairs:downstairs-1d304be3-4f03-4335-afb2-dbce24245753.log
10532023-12-06T17:47:38.461Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_993ea282-1eee-42e3-b3d8-ea8aa4c06162/root/var/svc/log/oxide-crucible-downstairs:downstairs-1d304be3-4f03-4335-afb2-dbce24245753.log.0 (3051 bytes)
10542023-12-06T17:47:39.565Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_993ea282-1eee-42e3-b3d8-ea8aa4c06162/root/var/svc/log/oxide-crucible-downstairs:downstairs-1d304be3-4f03-4335-afb2-dbce24245753.log.0
10552023-12-06T17:47:39.590Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_993ea282-1eee-42e3-b3d8-ea8aa4c06162/root/var/svc/log/oxide-crucible-downstairs:downstairs-2f84b89e-b4ab-4f2c-8874-0175b85a3b2a.log (1006 bytes)
10562023-12-06T17:47:40.664Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_993ea282-1eee-42e3-b3d8-ea8aa4c06162/root/var/svc/log/oxide-crucible-downstairs:downstairs-2f84b89e-b4ab-4f2c-8874-0175b85a3b2a.log
10572023-12-06T17:47:40.664Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_993ea282-1eee-42e3-b3d8-ea8aa4c06162/root/var/svc/log/oxide-crucible-downstairs:downstairs-2f84b89e-b4ab-4f2c-8874-0175b85a3b2a.log.0 (4529 bytes)
10582023-12-06T17:47:41.763Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_993ea282-1eee-42e3-b3d8-ea8aa4c06162/root/var/svc/log/oxide-crucible-downstairs:downstairs-2f84b89e-b4ab-4f2c-8874-0175b85a3b2a.log.0
10592023-12-06T17:47:41.763Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_ntp_0e075832-9eaa-40c5-a328-dcde3812ee47/root/var/svc/log/oxide-ntp:default.log (0 bytes)
10602023-12-06T17:47:41.813Zupload warning: file "/pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_ntp_0e075832-9eaa-40c5-a328-dcde3812ee47/root/var/svc/log/oxide-ntp:default.log" changed size mid upload: 0 -> 65
10612023-12-06T17:47:42.897Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_ntp_0e075832-9eaa-40c5-a328-dcde3812ee47/root/var/svc/log/oxide-ntp:default.log
10622023-12-06T17:47:42.897Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_ntp_0e075832-9eaa-40c5-a328-dcde3812ee47/root/var/svc/log/oxide-ntp:default.log.0 (847 bytes)
10632023-12-06T17:47:43.998Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_ntp_0e075832-9eaa-40c5-a328-dcde3812ee47/root/var/svc/log/oxide-ntp:default.log.0
10642023-12-06T17:47:43.998Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_nexus_1dfcdbef-d660-4977-8363-0facffed5dd7/root/var/svc/log/oxide-nexus:default.log (1232459 bytes)
10652023-12-06T17:47:44.315Zupload warning: file "/pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_nexus_1dfcdbef-d660-4977-8363-0facffed5dd7/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 1232459 -> 1307993
10662023-12-06T17:47:45.393Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_nexus_1dfcdbef-d660-4977-8363-0facffed5dd7/root/var/svc/log/oxide-nexus:default.log
10672023-12-06T17:47:45.393Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_nexus_1dfcdbef-d660-4977-8363-0facffed5dd7/root/var/svc/log/oxide-nexus:default.log.0 (3604266 bytes)
10682023-12-06T17:47:48.070Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_nexus_1dfcdbef-d660-4977-8363-0facffed5dd7/root/var/svc/log/oxide-nexus:default.log.0
10692023-12-06T17:47:48.070Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_879f5844-21f5-4128-accd-2686d45ad2ab/root/var/svc/log/oxide-crucible-agent:default.log (0 bytes)
10702023-12-06T17:47:49.148Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_879f5844-21f5-4128-accd-2686d45ad2ab/root/var/svc/log/oxide-crucible-agent:default.log
10712023-12-06T17:47:49.174Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_879f5844-21f5-4128-accd-2686d45ad2ab/root/var/svc/log/oxide-crucible-agent:default.log.0 (3435 bytes)
10722023-12-06T17:47:50.248Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_879f5844-21f5-4128-accd-2686d45ad2ab/root/var/svc/log/oxide-crucible-agent:default.log.0
10732023-12-06T17:47:50.275Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_879f5844-21f5-4128-accd-2686d45ad2ab/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
10742023-12-06T17:47:51.322Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_879f5844-21f5-4128-accd-2686d45ad2ab/root/var/svc/log/oxide-crucible-downstairs:default.log
10752023-12-06T17:47:51.348Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_879f5844-21f5-4128-accd-2686d45ad2ab/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
10762023-12-06T17:47:52.430Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_879f5844-21f5-4128-accd-2686d45ad2ab/root/var/svc/log/oxide-crucible-downstairs:default.log.0
10772023-12-06T17:47:52.430Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_f148960a-0913-4686-9975-ff2972d55de0/root/var/svc/log/oxide-crucible-agent:default.log (4321 bytes)
10782023-12-06T17:47:53.532Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_f148960a-0913-4686-9975-ff2972d55de0/root/var/svc/log/oxide-crucible-agent:default.log
10792023-12-06T17:47:53.532Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_f148960a-0913-4686-9975-ff2972d55de0/root/var/svc/log/oxide-crucible-agent:default.log.0 (20470 bytes)
10802023-12-06T17:47:54.672Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_f148960a-0913-4686-9975-ff2972d55de0/root/var/svc/log/oxide-crucible-agent:default.log.0
10812023-12-06T17:47:54.672Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_f148960a-0913-4686-9975-ff2972d55de0/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
10822023-12-06T17:47:55.747Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_f148960a-0913-4686-9975-ff2972d55de0/root/var/svc/log/oxide-crucible-downstairs:default.log
10832023-12-06T17:47:55.747Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_f148960a-0913-4686-9975-ff2972d55de0/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
10842023-12-06T17:47:56.845Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_f148960a-0913-4686-9975-ff2972d55de0/root/var/svc/log/oxide-crucible-downstairs:default.log.0
10852023-12-06T17:47:56.874Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_f148960a-0913-4686-9975-ff2972d55de0/root/var/svc/log/oxide-crucible-downstairs:downstairs-ab5c680a-4edc-4be9-b427-256326ec9ad6.log (0 bytes)
10862023-12-06T17:47:57.922Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_f148960a-0913-4686-9975-ff2972d55de0/root/var/svc/log/oxide-crucible-downstairs:downstairs-ab5c680a-4edc-4be9-b427-256326ec9ad6.log
10872023-12-06T17:47:57.922Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_f148960a-0913-4686-9975-ff2972d55de0/root/var/svc/log/oxide-crucible-downstairs:downstairs-ab5c680a-4edc-4be9-b427-256326ec9ad6.log.0 (3051 bytes)
10882023-12-06T17:47:59.017Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_f148960a-0913-4686-9975-ff2972d55de0/root/var/svc/log/oxide-crucible-downstairs:downstairs-ab5c680a-4edc-4be9-b427-256326ec9ad6.log.0
10892023-12-06T17:47:59.041Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_f148960a-0913-4686-9975-ff2972d55de0/root/var/svc/log/oxide-crucible-downstairs:downstairs-d54beced-792a-4bb4-be5d-5c1abf7eee9c.log (1006 bytes)
10902023-12-06T17:48:00.115Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_f148960a-0913-4686-9975-ff2972d55de0/root/var/svc/log/oxide-crucible-downstairs:downstairs-d54beced-792a-4bb4-be5d-5c1abf7eee9c.log
10912023-12-06T17:48:00.115Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_f148960a-0913-4686-9975-ff2972d55de0/root/var/svc/log/oxide-crucible-downstairs:downstairs-d54beced-792a-4bb4-be5d-5c1abf7eee9c.log.0 (4527 bytes)
10922023-12-06T17:48:01.215Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_f148960a-0913-4686-9975-ff2972d55de0/root/var/svc/log/oxide-crucible-downstairs:downstairs-d54beced-792a-4bb4-be5d-5c1abf7eee9c.log.0
10932023-12-06T17:48:01.215Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_5f32b190-8672-494f-90a3-7cd7719c5987/root/var/svc/log/oxide-crucible-pantry:default.log (0 bytes)
10942023-12-06T17:48:02.290Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_5f32b190-8672-494f-90a3-7cd7719c5987/root/var/svc/log/oxide-crucible-pantry:default.log
10952023-12-06T17:48:02.290Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_5f32b190-8672-494f-90a3-7cd7719c5987/root/var/svc/log/oxide-crucible-pantry:default.log.0 (2338 bytes)
10962023-12-06T17:48:03.395Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_5f32b190-8672-494f-90a3-7cd7719c5987/root/var/svc/log/oxide-crucible-pantry:default.log.0
10972023-12-06T17:48:03.395Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_2cfabf1e-5b74-40a8-8d00-5f984444bf51/root/var/svc/log/oxide-external_dns:default.log (3778 bytes)
10982023-12-06T17:48:03.468Zupload warning: file "/pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_2cfabf1e-5b74-40a8-8d00-5f984444bf51/root/var/svc/log/oxide-external_dns:default.log" changed size mid upload: 3778 -> 11328
10992023-12-06T17:48:04.543Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_2cfabf1e-5b74-40a8-8d00-5f984444bf51/root/var/svc/log/oxide-external_dns:default.log
11002023-12-06T17:48:04.543Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_2cfabf1e-5b74-40a8-8d00-5f984444bf51/root/var/svc/log/oxide-external_dns:default.log.0 (12596 bytes)
11012023-12-06T17:48:05.671Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_2cfabf1e-5b74-40a8-8d00-5f984444bf51/root/var/svc/log/oxide-external_dns:default.log.0
11022023-12-06T17:48:05.671Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_oximeter_358b59f5-3c51-4c63-92f1-1dcada1dc798/root/var/svc/log/oxide-oximeter:default.log (66019 bytes)
11032023-12-06T17:48:05.862Zupload warning: file "/pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_oximeter_358b59f5-3c51-4c63-92f1-1dcada1dc798/root/var/svc/log/oxide-oximeter:default.log" changed size mid upload: 66019 -> 115911
11042023-12-06T17:48:06.937Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_oximeter_358b59f5-3c51-4c63-92f1-1dcada1dc798/root/var/svc/log/oxide-oximeter:default.log
11052023-12-06T17:48:06.937Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_oximeter_358b59f5-3c51-4c63-92f1-1dcada1dc798/root/var/svc/log/oxide-oximeter:default.log.0 (96236 bytes)
11062023-12-06T17:48:08.160Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_oximeter_358b59f5-3c51-4c63-92f1-1dcada1dc798/root/var/svc/log/oxide-oximeter:default.log.0
11072023-12-06T17:48:08.160Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_clickhouse_345211c8-e540-44f1-be03-4dfe81553f49/root/var/svc/log/oxide-clickhouse:default.log (200770 bytes)
11082023-12-06T17:48:08.388Zupload warning: file "/pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_clickhouse_345211c8-e540-44f1-be03-4dfe81553f49/root/var/svc/log/oxide-clickhouse:default.log" changed size mid upload: 200770 -> 356677
11092023-12-06T17:48:09.464Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_clickhouse_345211c8-e540-44f1-be03-4dfe81553f49/root/var/svc/log/oxide-clickhouse:default.log
11102023-12-06T17:48:09.464Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_clickhouse_345211c8-e540-44f1-be03-4dfe81553f49/root/var/svc/log/oxide-clickhouse:default.log.0 (367956 bytes)
11112023-12-06T17:48:10.748Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_clickhouse_345211c8-e540-44f1-be03-4dfe81553f49/root/var/svc/log/oxide-clickhouse:default.log.0
11122023-12-06T17:48:10.748Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_8caca53a-45ee-4bbf-9f3b-bf2a9c9243ef/root/var/svc/log/oxide-cockroachdb:default.log (0 bytes)
11132023-12-06T17:48:11.823Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_8caca53a-45ee-4bbf-9f3b-bf2a9c9243ef/root/var/svc/log/oxide-cockroachdb:default.log
11142023-12-06T17:48:11.823Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_8caca53a-45ee-4bbf-9f3b-bf2a9c9243ef/root/var/svc/log/oxide-cockroachdb:default.log.0 (5368 bytes)
11152023-12-06T17:48:12.922Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_8caca53a-45ee-4bbf-9f3b-bf2a9c9243ef/root/var/svc/log/oxide-cockroachdb:default.log.0
11162023-12-06T17:48:12.922Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_873ec7ab-6024-4863-b381-a4fcad4788ec/root/var/svc/log/oxide-crucible-agent:default.log (0 bytes)
11172023-12-06T17:48:13.996Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_873ec7ab-6024-4863-b381-a4fcad4788ec/root/var/svc/log/oxide-crucible-agent:default.log
11182023-12-06T17:48:13.997Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_873ec7ab-6024-4863-b381-a4fcad4788ec/root/var/svc/log/oxide-crucible-agent:default.log.0 (3436 bytes)
11192023-12-06T17:48:15.095Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_873ec7ab-6024-4863-b381-a4fcad4788ec/root/var/svc/log/oxide-crucible-agent:default.log.0
11202023-12-06T17:48:15.095Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_873ec7ab-6024-4863-b381-a4fcad4788ec/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
11212023-12-06T17:48:16.194Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_873ec7ab-6024-4863-b381-a4fcad4788ec/root/var/svc/log/oxide-crucible-downstairs:default.log
11222023-12-06T17:48:16.194Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_873ec7ab-6024-4863-b381-a4fcad4788ec/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
11232023-12-06T17:48:17.296Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_873ec7ab-6024-4863-b381-a4fcad4788ec/root/var/svc/log/oxide-crucible-downstairs:default.log.0
11242023-12-06T17:48:17.334Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_f70a9b4a-82f3-4f7e-9b2e-db724f71ab50/root/var/svc/log/oxide-crucible-agent:default.log (0 bytes)
11252023-12-06T17:48:18.388Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_f70a9b4a-82f3-4f7e-9b2e-db724f71ab50/root/var/svc/log/oxide-crucible-agent:default.log
11262023-12-06T17:48:18.414Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_f70a9b4a-82f3-4f7e-9b2e-db724f71ab50/root/var/svc/log/oxide-crucible-agent:default.log.0 (3436 bytes)
11272023-12-06T17:48:19.489Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_f70a9b4a-82f3-4f7e-9b2e-db724f71ab50/root/var/svc/log/oxide-crucible-agent:default.log.0
11282023-12-06T17:48:19.490Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_f70a9b4a-82f3-4f7e-9b2e-db724f71ab50/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
11292023-12-06T17:48:20.565Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_f70a9b4a-82f3-4f7e-9b2e-db724f71ab50/root/var/svc/log/oxide-crucible-downstairs:default.log
11302023-12-06T17:48:20.590Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_f70a9b4a-82f3-4f7e-9b2e-db724f71ab50/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
11312023-12-06T17:48:21.698Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_f70a9b4a-82f3-4f7e-9b2e-db724f71ab50/root/var/svc/log/oxide-crucible-downstairs:default.log.0
11322023-12-06T17:48:21.699Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_00f82355-ab6a-44f8-adeb-55a0298e7676/root/var/svc/log/oxide-internal_dns:default.log (13833 bytes)
11332023-12-06T17:48:21.804Zupload warning: file "/pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_00f82355-ab6a-44f8-adeb-55a0298e7676/root/var/svc/log/oxide-internal_dns:default.log" changed size mid upload: 13833 -> 21344
11342023-12-06T17:48:22.881Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_00f82355-ab6a-44f8-adeb-55a0298e7676/root/var/svc/log/oxide-internal_dns:default.log
11352023-12-06T17:48:22.881Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_00f82355-ab6a-44f8-adeb-55a0298e7676/root/var/svc/log/oxide-internal_dns:default.log.0 (100762 bytes)
11362023-12-06T17:48:24.096Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_00f82355-ab6a-44f8-adeb-55a0298e7676/root/var/svc/log/oxide-internal_dns:default.log.0
11372023-12-06T17:48:24.097Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_9a25727f-e844-4dd1-b1b4-978eca7f5c90/root/var/svc/log/oxide-nexus:default.log (79417 bytes)
11382023-12-06T17:48:24.308Zupload warning: file "/pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_9a25727f-e844-4dd1-b1b4-978eca7f5c90/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 79417 -> 222100
11392023-12-06T17:48:25.389Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_9a25727f-e844-4dd1-b1b4-978eca7f5c90/root/var/svc/log/oxide-nexus:default.log
11402023-12-06T17:48:25.389Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_9a25727f-e844-4dd1-b1b4-978eca7f5c90/root/var/svc/log/oxide-nexus:default.log.0 (505751 bytes)
11412023-12-06T17:48:26.715Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_9a25727f-e844-4dd1-b1b4-978eca7f5c90/root/var/svc/log/oxide-nexus:default.log.0
11422023-12-06T17:48:26.744Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_ntp_0e075832-9eaa-40c5-a328-dcde3812ee47/root/var/log/chrony/measurements.log (16851 bytes)
11432023-12-06T17:48:26.827Zupload warning: file "/pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_ntp_0e075832-9eaa-40c5-a328-dcde3812ee47/root/var/log/chrony/measurements.log" changed size mid upload: 16851 -> 21646
11442023-12-06T17:48:27.944Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_ntp_0e075832-9eaa-40c5-a328-dcde3812ee47/root/var/log/chrony/measurements.log
11452023-12-06T17:48:27.944Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_ntp_0e075832-9eaa-40c5-a328-dcde3812ee47/root/var/log/chrony/statistics.log (10296 bytes)
11462023-12-06T17:48:28.018Zupload warning: file "/pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_ntp_0e075832-9eaa-40c5-a328-dcde3812ee47/root/var/log/chrony/statistics.log" changed size mid upload: 10296 -> 14157
11472023-12-06T17:48:29.095Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_ntp_0e075832-9eaa-40c5-a328-dcde3812ee47/root/var/log/chrony/statistics.log
11482023-12-06T17:48:29.096Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_ntp_0e075832-9eaa-40c5-a328-dcde3812ee47/root/var/log/chrony/tracking.log (1453 bytes)
11492023-12-06T17:48:29.147Zupload warning: file "/pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_ntp_0e075832-9eaa-40c5-a328-dcde3812ee47/root/var/log/chrony/tracking.log" changed size mid upload: 1453 -> 1717
11502023-12-06T17:48:30.222Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_ntp_0e075832-9eaa-40c5-a328-dcde3812ee47/root/var/log/chrony/tracking.log