01HH18HGY9XG2QQFB0AV6ETAC1: helios / deploy

BasicConfig {
    output_rules: [
        "%/var/svc/log/oxide-sled-agent:default.log*",
        "%/pool/ext/*/crypt/zone/oxz_*/root/var/svc/log/oxide-*.log*",
        "%/pool/ext/*/crypt/zone/oxz_*/root/var/svc/log/system-illumos-*.log*",
        "%/pool/ext/*/crypt/zone/oxz_ntp_*/root/var/log/chrony/*.log*",
        "!/pool/ext/*/crypt/zone/oxz_propolis-server_*/root/var/svc/log/*.log*",
        "%/pool/ext/*/crypt/debug/global/oxide-sled-agent:default.log.*",
        "%/pool/ext/*/crypt/debug/oxz_*/oxide-*.log.*",
        "%/pool/ext/*/crypt/debug/oxz_*/system-illumos-*.log.*",
        "!/pool/ext/*/crypt/debug/oxz_propolis-server_*/*.log.*",
    ],
    rust_toolchain: None,
    target: Some(
        "lab-2.0-opte-0.25",
    ),
    access_repos: [],
    publish: [],
    skip_clone: true,
}

Buildomat Job: 01HH18KBVYYHF9DH0412T6NB82

Tags:

Artefacts:

Output:

SEQ GLOBAL TIME DETAILS
12023-12-07T05:18:02.590Zjob dependencies complete; ready to run (waiting for 48 m 33 s)
22023-12-07T05:30:48.516Zjob assigned to worker 01HH1BV67KZR3J134K9JFG1FF6 (queued for 12 m 45 s)
32023-12-07T05:30:48.646Zhttp://172.20.0.5:9969/ipxe/buskin... ok
42023-12-07T05:30:48.702Zhttp://172.20.0.5:9969/os/buskin/platform/i86pc/kernel/amd64/unix... ok
52023-12-07T05:30:48.747Zhttp://172.20.0.5:9969/os/buskin/platform/i86pc/amd64/boot_archive... ok
62023-12-07T05:30:48.800Zhttp://172.20.0.5:9969/os/buskin/platform/i86pc/amd64/boot_archive.hash... ok
72023-12-07T05:30:48.837Zhttp://172.20.0.5:9969/postboot/buskin... ok
82023-12-07T05:30:48.871ZwcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U
92023-12-07T05:30:48.912Zcpu0: microcode has been updated from version 0xa001133 to 0xa0011d1
102023-12-07T05:30:48.945ZOxide Helios Version helios-2.0.22318 64-bit
112023-12-07T05:30:48.979ZConfiguring devices.
122023-12-07T05:30:49.012Z##################################################
132023-12-07T05:30:49.046Z#### ############# ##
142023-12-07T05:30:49.079Z### ### ############ ##
152023-12-07T05:30:49.112Z## ### # ## ### ## Oxide Computer Company ##
162023-12-07T05:30:49.146Z## ## ## ### # ### ##
172023-12-07T05:30:49.179Z## # ### #### #### This Station Under ##
182023-12-07T05:30:49.212Z### ### #### # ### Computer Control ##
192023-12-07T05:30:49.246Z#### #### ### ## ##
202023-12-07T05:30:49.300Z##################################################
212023-12-07T05:30:49.335ZHostname: buskin
222023-12-07T05:30:49.368ZLoading smf(7) serice descriptions:
232023-12-07T05:30:50.480Zdownloading input: /input/ci-tools/work/end-to-end-tests/bootstrap-be9c817b94af4e8a.gz
242023-12-07T05:30:52.276Zdownloaded input: /input/ci-tools/work/end-to-end-tests/bootstrap-be9c817b94af4e8a.gz
252023-12-07T05:30:52.276Zdownloading input: /input/ci-tools/work/end-to-end-tests/bootstrap.gz
262023-12-07T05:31:14.808Zdownloaded input: /input/ci-tools/work/end-to-end-tests/bootstrap.gz
272023-12-07T05:31:14.808Zdownloading input: /input/ci-tools/work/end-to-end-tests/end_to_end_tests-ff923d3f5c6228d4.gz
282023-12-07T05:31:34.036Zdownloaded input: /input/ci-tools/work/end-to-end-tests/end_to_end_tests-ff923d3f5c6228d4.gz
292023-12-07T05:31:34.036Zdownloading input: /input/ci-tools/work/caboose-util.gz
302023-12-07T05:31:34.928Zdownloaded input: /input/ci-tools/work/caboose-util.gz
312023-12-07T05:31:34.954Zdownloading input: /input/ci-tools/work/tufaceous.gz
322023-12-07T05:31:38.138Zdownloaded input: /input/ci-tools/work/tufaceous.gz
332023-12-07T05:31:38.162Zdownloading input: /input/package/work/version.txt
342023-12-07T05:31:38.187Zdownloaded input: /input/package/work/version.txt
352023-12-07T05:31:38.187Zdownloading input: /input/package/work/package.tar.gz
362023-12-07T05:31:58.641Zdownloaded input: /input/package/work/package.tar.gz
372023-12-07T05:31:58.641Zdownloading input: /input/package/work/global-zone-packages.tar.gz
382023-12-07T05:32:28.581Zdownloaded input: /input/package/work/global-zone-packages.tar.gz
392023-12-07T05:32:28.581Zdownloading input: /input/package/work/trampoline-global-zone-packages.tar.gz
402023-12-07T05:32:35.553Zdownloaded input: /input/package/work/trampoline-global-zone-packages.tar.gz
412023-12-07T05:32:35.580Zdownloading input: /input/package/work/zones/clickhouse.tar.gz
422023-12-07T05:33:11.891Zdownloaded input: /input/package/work/zones/clickhouse.tar.gz
432023-12-07T05:33:11.891Zdownloading input: /input/package/work/zones/clickhouse_keeper.tar.gz
442023-12-07T05:33:53.565Zdownloaded input: /input/package/work/zones/clickhouse_keeper.tar.gz
452023-12-07T05:33:53.565Zdownloading input: /input/package/work/zones/cockroachdb.tar.gz
462023-12-07T05:34:16.193Zdownloaded input: /input/package/work/zones/cockroachdb.tar.gz
472023-12-07T05:34:16.194Zdownloading input: /input/package/work/zones/crucible-pantry.tar.gz
482023-12-07T05:34:21.529Zdownloaded input: /input/package/work/zones/crucible-pantry.tar.gz
492023-12-07T05:34:21.571Zdownloading input: /input/package/work/zones/crucible.tar.gz
502023-12-07T05:34:45.604Zdownloaded input: /input/package/work/zones/crucible.tar.gz
512023-12-07T05:34:45.605Zdownloading input: /input/package/work/zones/external-dns.tar.gz
522023-12-07T05:34:51.132Zdownloaded input: /input/package/work/zones/external-dns.tar.gz
532023-12-07T05:34:51.132Zdownloading input: /input/package/work/zones/internal-dns.tar.gz
542023-12-07T05:34:56.252Zdownloaded input: /input/package/work/zones/internal-dns.tar.gz
552023-12-07T05:34:56.252Zdownloading input: /input/package/work/zones/ntp.tar.gz
562023-12-07T05:34:56.310Zdownloaded input: /input/package/work/zones/ntp.tar.gz
572023-12-07T05:34:56.334Zdownloading input: /input/package/work/zones/omicron-gateway-asic.tar.gz
582023-12-07T05:34:59.810Zdownloaded input: /input/package/work/zones/omicron-gateway-asic.tar.gz
592023-12-07T05:34:59.810Zdownloading input: /input/package/work/zones/omicron-gateway-softnpu.tar.gz
602023-12-07T05:35:03.384Zdownloaded input: /input/package/work/zones/omicron-gateway-softnpu.tar.gz
612023-12-07T05:35:03.384Zdownloading input: /input/package/work/zones/omicron-nexus-single-sled.tar.gz
622023-12-07T05:35:16.796Zdownloaded input: /input/package/work/zones/omicron-nexus-single-sled.tar.gz
632023-12-07T05:35:16.796Zdownloading input: /input/package/work/zones/omicron-nexus.tar.gz
642023-12-07T05:35:35.265Zdownloaded input: /input/package/work/zones/omicron-nexus.tar.gz
652023-12-07T05:35:35.265Zdownloading input: /input/package/work/zones/overlay.tar.gz
662023-12-07T05:35:35.481Zdownloaded input: /input/package/work/zones/overlay.tar.gz
672023-12-07T05:35:35.481Zdownloading input: /input/package/work/zones/oximeter-collector.tar.gz
682023-12-07T05:35:42.009Zdownloaded input: /input/package/work/zones/oximeter-collector.tar.gz
692023-12-07T05:35:42.035Zdownloading input: /input/package/work/zones/propolis-server.tar.gz
702023-12-07T05:35:55.698Zdownloaded input: /input/package/work/zones/propolis-server.tar.gz
712023-12-07T05:35:55.725Zdownloading input: /input/package/work/zones/switch-asic.tar.gz
722023-12-07T05:36:43.848Zdownloaded input: /input/package/work/zones/switch-asic.tar.gz
732023-12-07T05:36:43.848Zdownloading input: /input/package/work/zones/switch-softnpu.tar.gz
742023-12-07T05:37:23.566Zdownloaded input: /input/package/work/zones/switch-softnpu.tar.gz
 
752023-12-07T05:37:23.596Zstarting task 0: "setup"
762023-12-07T05:37:23.596Z++ uname -s
772023-12-07T05:37:23.596Z+ kern=SunOS
782023-12-07T05:37:23.596Z+ case "$kern" in
792023-12-07T05:37:23.596Z+ groupadd -g 12345 build
802023-12-07T05:37:23.596Z+ useradd -u 12345 -g build -d /home/build -s /bin/bash -c build -P 'Primary Administrator' build
812023-12-07T05:37:25.584Z+ zfs create -o mountpoint=/work rpool/work
822023-12-07T05:37:25.610Z++ awk '$2 == "/home" { print $3 }' /etc/mnttab
832023-12-07T05:37:25.610Z+ home_fs=zfs
842023-12-07T05:37:25.610Z+ [[ zfs == autofs ]]
852023-12-07T05:37:25.610Z+ mkdir -p /home/build
862023-12-07T05:37:25.636Z+ chown build:build /home/build /work
872023-12-07T05:37:27.625Z+ chmod 0700 /home/build /work
882023-12-07T05:37:27.668Zprocess exited: duration 4051 ms, exit code 0
 
892023-12-07T05:37:27.742Zstarting task 1: "authentication"
902023-12-07T05:37:27.799Zprocess exited: duration 57 ms, exit code 0
 
912023-12-07T05:37:27.870Zstarting task 2: "build"
922023-12-07T05:37:27.903Z+ trap _exit_trap EXIT
932023-12-07T05:37:27.903Z+ pfexec mdb -kwe 'secpolicy_fs_linkdir/v 55 48 89 e5 b8 01 00 00 00 5d c3'
942023-12-07T05:37:28.073Zsecpolicy_fs_linkdir: 0x55 = 0x55
952023-12-07T05:37:28.073Zsecpolicy_fs_linkdir+1: 0xb8 = 0x48
962023-12-07T05:37:28.073Zsecpolicy_fs_linkdir+2: 0x1 = 0x89
972023-12-07T05:37:28.074Zsecpolicy_fs_linkdir+3: 0 = 0xe5
982023-12-07T05:37:28.074Zsecpolicy_fs_linkdir+4: 0 = 0xb8
992023-12-07T05:37:28.074Zsecpolicy_fs_linkdir+5: 0 = 0x1
1002023-12-07T05:37:28.074Zsecpolicy_fs_linkdir+6: 0x48 = 0x0
1012023-12-07T05:37:28.113Zsecpolicy_fs_linkdir+7: 0x89 = 0x0
1022023-12-07T05:37:28.113Zsecpolicy_fs_linkdir+8: 0xe5 = 0x0
1032023-12-07T05:37:28.113Zsecpolicy_fs_linkdir+9: 0x48 = 0x5d
1042023-12-07T05:37:28.113Zsecpolicy_fs_linkdir+0xa: 0x83 = 0xc3
1052023-12-07T05:37:28.113Z+ [[ -d /opt/oxide ]]
1062023-12-07T05:37:28.114Z+ pfexec mv /opt/oxide /opt/oxide-underneath
1072023-12-07T05:37:28.114Z+ pfexec mkdir /opt/oxide
1082023-12-07T05:37:28.114Z+ pfexec mount -F tmpfs -O swap /opt/oxide
1092023-12-07T05:37:28.114Z+ [[ -d /opt/oxide-underneath ]]
1102023-12-07T05:37:28.114Z+ cd /opt/oxide-underneath
1112023-12-07T05:37:28.114Z+ pfexec tar ceEp@/f - .
1122023-12-07T05:37:28.115Z+ cd /opt/oxide
1132023-12-07T05:37:28.115Z+ pfexec tar xveEp@/f -
1142023-12-07T05:37:28.191Zx ., 0 bytes, 0 tape blocks
1152023-12-07T05:37:28.191Zx ./opte, 0 bytes, 0 tape blocks
1162023-12-07T05:37:28.191Zx ./opte/bin, 0 bytes, 0 tape blocks
1172023-12-07T05:37:28.192Zx ./opte/bin/opteadm, 50714832 bytes, 99053 tape blocks
1182023-12-07T05:37:28.218Z+ pfexec rm -rf /opt/oxide-underneath
1192023-12-07T05:37:28.218Z+ pfexec /sbin/zfs create -o mountpoint=/zone rpool/zone
1202023-12-07T05:37:28.218Z+ pfexec mkdir /opt/oxide/work
1212023-12-07T05:37:28.247Z+ pfexec chown build:build /opt/oxide/work
1222023-12-07T05:37:28.248Z+ cd /opt/oxide/work
1232023-12-07T05:37:28.248Z+ ptime -m tar xvzf /input/package/work/package.tar.gz
1242023-12-07T05:37:30.162ZDecompressing '/input/package/work/package.tar.gz' with '/usr/bin/gzcat'...
1252023-12-07T05:37:30.163Zx out/mg-ddm-gz.tar, 62723072 bytes, 122506 tape blocks
1262023-12-07T05:37:30.163Zx out/omicron-sled-agent.tar, 111105536 bytes, 217003 tape blocks
1272023-12-07T05:37:30.163Zx out/target/test, 75 bytes, 1 tape blocks
1282023-12-07T05:37:30.163Zx out/npuzone/npuzone, 37380856 bytes, 73010 tape blocks
1292023-12-07T05:37:30.163Zx package-manifest.toml, 22143 bytes, 44 tape blocks
1302023-12-07T05:37:30.163Zx smf/sled-agent/non-gimlet/config.toml, 3298 bytes, 7 tape blocks
1312023-12-07T05:37:30.163Zx target/release/omicron-package, 42582768 bytes, 83170 tape blocks
1322023-12-07T05:37:30.163Zx tools/create_virtual_hardware.sh, 3176 bytes, 7 tape blocks
1332023-12-07T05:37:30.163Zx tools/virtual_hardware.sh, 3408 bytes, 7 tape blocks
1342023-12-07T05:37:30.163Zx tools/scrimlet/create-softnpu-zone.sh, 619 bytes, 2 tape blocks
1352023-12-07T05:37:30.164Zx tools/scrimlet/destroy-softnpu-zone.sh, 195 bytes, 1 tape blocks
1362023-12-07T05:37:30.165Zx tools/scrimlet/softnpu-init.sh, 2148 bytes, 5 tape blocks
1372023-12-07T05:37:30.165Zx tools/scrimlet/softnpu-zone.txt, 330 bytes, 1 tape blocks
1382023-12-07T05:37:30.165Zx tools/scrimlet/softnpu.toml, 171 bytes, 1 tape blocks
1392023-12-07T05:37:30.190Z
1402023-12-07T05:37:30.190Zreal 1.527396316
1412023-12-07T05:37:30.190Zuser 1.379821394
1422023-12-07T05:37:30.190Zsys 0.259393394
1432023-12-07T05:37:30.190Ztrap 0.000061819
1442023-12-07T05:37:30.191Ztflt 0.000000000
1452023-12-07T05:37:30.191Zdflt 0.000000000
1462023-12-07T05:37:30.191Zkflt 0.000000000
1472023-12-07T05:37:30.191Zlock 0.000000000
1482023-12-07T05:37:30.191Zslp 1.380099090
1492023-12-07T05:37:30.191Zlat 0.033396217
1502023-12-07T05:37:30.191Zstop 0.000123332
1512023-12-07T05:37:30.191Z+ cp /input/package/work/zones/clickhouse_keeper.tar.gz /input/package/work/zones/clickhouse.tar.gz /input/package/work/zones/cockroachdb.tar.gz /input/package/work/zones/crucible-pantry.tar.gz /input/package/work/zones/crucible.tar.gz /input/package/work/zones/external-dns.tar.gz /input/package/work/zones/internal-dns.tar.gz /input/package/work/zones/ntp.tar.gz /input/package/work/zones/omicron-gateway-asic.tar.gz /input/package/work/zones/omicron-gateway-softnpu.tar.gz /input/package/work/zones/omicron-nexus-single-sled.tar.gz /input/package/work/zones/omicron-nexus.tar.gz /input/package/work/zones/overlay.tar.gz /input/package/work/zones/oximeter-collector.tar.gz /input/package/work/zones/propolis-server.tar.gz /input/package/work/zones/switch-asic.tar.gz /input/package/work/zones/switch-softnpu.tar.gz out/
1522023-12-07T05:37:30.820Z+ mv out/omicron-nexus-single-sled.tar.gz out/omicron-nexus.tar.gz
1532023-12-07T05:37:30.847Z+ mkdir tests
1542023-12-07T05:37:30.848Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1552023-12-07T05:37:30.848Z+ ptime -m gunzip
1562023-12-07T05:37:30.848Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap-be9c817b94af4e8a
1572023-12-07T05:37:30.988Z
1582023-12-07T05:37:31.015Zreal 0.137473041
1592023-12-07T05:37:31.016Zuser 0.126425107
1602023-12-07T05:37:31.016Zsys 0.010795697
1612023-12-07T05:37:31.016Ztrap 0.000014580
1622023-12-07T05:37:31.016Ztflt 0.000000000
1632023-12-07T05:37:31.016Zdflt 0.000000000
1642023-12-07T05:37:31.016Zkflt 0.000000000
1652023-12-07T05:37:31.016Zlock 0.000000000
1662023-12-07T05:37:31.016Zslp 0.000020900
1672023-12-07T05:37:31.016Zlat 0.000022870
1682023-12-07T05:37:31.016Zstop 0.000125933
1692023-12-07T05:37:31.016Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap-be9c817b94af4e8a
1702023-12-07T05:37:31.017Z+ chmod a+x tests/bootstrap-be9c817b94af4e8a
1712023-12-07T05:37:31.017Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1722023-12-07T05:37:31.017Z+ ptime -m gunzip
1732023-12-07T05:37:31.017Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap
1742023-12-07T05:37:33.192Z
1752023-12-07T05:37:33.221Zreal 2.199821491
1762023-12-07T05:37:33.221Zuser 2.069613748
1772023-12-07T05:37:33.221Zsys 0.129816557
1782023-12-07T05:37:33.221Ztrap 0.000158242
1792023-12-07T05:37:33.221Ztflt 0.000000000
1802023-12-07T05:37:33.222Zdflt 0.000000000
1812023-12-07T05:37:33.222Zkflt 0.000000000
1822023-12-07T05:37:33.222Zlock 0.000000000
1832023-12-07T05:37:33.222Zslp 0.000028190
1842023-12-07T05:37:33.222Zlat 0.000023570
1852023-12-07T05:37:33.222Zstop 0.000115863
1862023-12-07T05:37:33.222Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap
1872023-12-07T05:37:33.222Z+ chmod a+x tests/bootstrap
1882023-12-07T05:37:33.222Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1892023-12-07T05:37:33.222Z+ ptime -m gunzip
1902023-12-07T05:37:33.222Z++ basename /input/ci-tools/work/end-to-end-tests/end_to_end_tests-ff923d3f5c6228d4
1912023-12-07T05:37:35.461Z
1922023-12-07T05:37:35.487Zreal 2.254412951
1932023-12-07T05:37:35.487Zuser 2.116009555
1942023-12-07T05:37:35.487Zsys 0.138009197
1952023-12-07T05:37:35.487Ztrap 0.000161873
1962023-12-07T05:37:35.487Ztflt 0.000000000
1972023-12-07T05:37:35.487Zdflt 0.000000000
1982023-12-07T05:37:35.487Zkflt 0.000000000
1992023-12-07T05:37:35.487Zlock 0.000000000
2002023-12-07T05:37:35.487Zslp 0.000018590
2012023-12-07T05:37:35.487Zlat 0.000023040
2022023-12-07T05:37:35.487Zstop 0.000125393
2032023-12-07T05:37:35.488Z++ basename /input/ci-tools/work/end-to-end-tests/end_to_end_tests-ff923d3f5c6228d4
2042023-12-07T05:37:35.488Z+ chmod a+x tests/end_to_end_tests-ff923d3f5c6228d4
2052023-12-07T05:37:35.488Z+ bmat address ls
2062023-12-07T05:37:35.488ZNAME CIDR FIRST LAST COUNT
2072023-12-07T05:37:35.489Zextra 10.151.1.0/24 10.151.1.164 10.151.1.227 64
2082023-12-07T05:37:35.489Z++ bmat address ls -f extra -Ho first,last,count
2092023-12-07T05:37:35.489Z+ set -- 10.151.1.164 10.151.1.227 64
2102023-12-07T05:37:35.489Z+ EXTRA_IP_START=10.151.1.164
2112023-12-07T05:37:35.489Z+ EXTRA_IP_END=10.151.1.227
2122023-12-07T05:37:35.489Z+ EXTRA_IP_COUNT=64
2132023-12-07T05:37:35.489Z+ (( EXTRA_IP_COUNT >= 32 ))
2142023-12-07T05:37:35.489Z+ EXTRA_IP_BASE=10.151.1
2152023-12-07T05:37:35.489Z+ EXTRA_IP_FOCTET=164
2162023-12-07T05:37:35.489Z+ EXTRA_IP_LOCTET=227
2172023-12-07T05:37:35.489Z+ SERVICE_IP_POOL_START=10.151.1.164
2182023-12-07T05:37:35.489Z+ SERVICE_IP_POOL_END=10.151.1.173
2192023-12-07T05:37:35.490Z+ DNS_IP1=10.151.1.164
2202023-12-07T05:37:35.490Z+ DNS_IP2=10.151.1.165
2212023-12-07T05:37:35.490Z+ UPLINK_IP=10.151.1.174
2222023-12-07T05:37:35.490Z+ IPPOOL_START=10.151.1.175
2232023-12-07T05:37:35.490Z+ IPPOOL_END=10.151.1.227
2242023-12-07T05:37:35.515Z++ ipadm show-addr -po type,addr
2252023-12-07T05:37:35.515Z++ awk '-F[:/]' '$1 == "dhcp" {print $2}'
2262023-12-07T05:37:35.515Z+ GATEWAY_IP=10.151.1.100
2272023-12-07T05:37:35.515Z+ [[ -n 10.151.1.100 ]]
2282023-12-07T05:37:35.515Z+ ping -s 10.151.1.100 56 1
2292023-12-07T05:37:35.515ZPING 10.151.1.100: 56 data bytes
2302023-12-07T05:37:35.516Z64 bytes from 10.151.1.100: icmp_seq=0. time=0.010 ms
2312023-12-07T05:37:35.516Z
2322023-12-07T05:37:35.516Z----10.151.1.100 PING Statistics----
2332023-12-07T05:37:35.516Z1 packets transmitted, 1 packets received, 0% packet loss
2342023-12-07T05:37:35.516Zround-trip (ms) min/avg/max/stddev = 0.010/0.010/0.010/-nan
2352023-12-07T05:37:35.516Z++ arp -an
2362023-12-07T05:37:35.517Z++ awk -vgw=10.151.1.100 '$2 == gw {print $NF}'
2372023-12-07T05:37:35.517Z+ GATEWAY_MAC=18:c0:4d:0d:9f:b2
2382023-12-07T05:37:35.517Z+ routeadm -e ipv4-forwarding -u
2392023-12-07T05:37:35.625Z+ PXA_START=10.151.1.164
2402023-12-07T05:37:35.625Z+ PXA_END=10.151.1.227
2412023-12-07T05:37:35.650Z+ export GATEWAY_IP GATEWAY_MAC PXA_START PXA_END
2422023-12-07T05:37:35.651Z+ pfexec zpool create -f scratch c1t1d0 c2t1d0
2432023-12-07T05:37:36.164Z+ ZPOOL_VDEV_DIR=/scratch
2442023-12-07T05:37:36.164Z+ ptime -m pfexec ./tools/create_virtual_hardware.sh
2452023-12-07T05:37:36.191Z+++ dirname ./tools/create_virtual_hardware.sh
2462023-12-07T05:37:36.192Z++ cd ./tools
2472023-12-07T05:37:36.192Z++ pwd
2482023-12-07T05:37:36.192Z+ SOURCE_DIR=/opt/oxide/work/tools
2492023-12-07T05:37:36.192Z+ OMICRON_TOP=/opt/oxide/work/tools/..
2502023-12-07T05:37:36.192Z+ SOFTNPU_MODE=zone
2512023-12-07T05:37:36.192Z+ . /opt/oxide/work/tools/virtual_hardware.sh
2522023-12-07T05:37:36.192Z++ MARKER=/etc/opt/oxide/NO_INSTALL
2532023-12-07T05:37:36.192Z++ [[ -f /etc/opt/oxide/NO_INSTALL ]]
2542023-12-07T05:37:36.193Z++ dladm show-phys -p -o LINK
2552023-12-07T05:37:36.193Z++ head -1
2562023-12-07T05:37:36.193Z+ PHYSICAL_LINK=igb0
2572023-12-07T05:37:36.193Z+ echo 'Using igb0 as physical link'
2582023-12-07T05:37:36.193ZUsing igb0 as physical link
2592023-12-07T05:37:36.193Z+ ensure_run_as_root
2602023-12-07T05:37:36.194Z++ id -u
2612023-12-07T05:37:36.194Z+ [[ 0 -ne 0 ]]
2622023-12-07T05:37:36.194Z+ ensure_zpools
2632023-12-07T05:37:36.194Z+ ZPOOL_TYPES=('oxp_' 'oxi_')
2642023-12-07T05:37:36.194Z+ for ZPOOL_TYPE in "${ZPOOL_TYPES[@]}"
2652023-12-07T05:37:36.194Z+ readarray -t ZPOOLS
2662023-12-07T05:37:36.194Z++ grep '"oxp_' /opt/oxide/work/tools/../smf/sled-agent/non-gimlet/config.toml
2672023-12-07T05:37:36.194Z++ sed 's/[ ",]//g'
2682023-12-07T05:37:36.194Z+ for ZPOOL in "${ZPOOLS[@]}"
2692023-12-07T05:37:36.194Z+ echo 'Zpool: [oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b]'
2702023-12-07T05:37:36.194ZZpool: [oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b]
2712023-12-07T05:37:36.194Z+ VDEV_PATH=/scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
2722023-12-07T05:37:36.194Z+ [[ -f /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
2732023-12-07T05:37:36.194Z+ dd if=/dev/zero of=/scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
2742023-12-07T05:37:36.194Z0+0 records in
2752023-12-07T05:37:36.194Z0+0 records out
2762023-12-07T05:37:36.194Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
2772023-12-07T05:37:36.195Z+ success 'ZFS vdev /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
2782023-12-07T05:37:36.195Z+ set +x
2792023-12-07T05:37:36.195ZZFS vdev /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
2802023-12-07T05:37:36.195Z++ zpool list -o name
2812023-12-07T05:37:36.195Z++ grep oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b
2822023-12-07T05:37:36.195Z+ [[ -z '' ]]
2832023-12-07T05:37:36.195Z+ zpool create -f oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
2842023-12-07T05:37:36.195Z+ success 'ZFS zpool oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
2852023-12-07T05:37:36.195Z+ set +x
2862023-12-07T05:37:36.195ZZFS zpool oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b exists
2872023-12-07T05:37:36.195Z+ for ZPOOL in "${ZPOOLS[@]}"
2882023-12-07T05:37:36.195Z+ echo 'Zpool: [oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03]'
2892023-12-07T05:37:36.195ZZpool: [oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03]
2902023-12-07T05:37:36.195Z+ VDEV_PATH=/scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
2912023-12-07T05:37:36.195Z+ [[ -f /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
2922023-12-07T05:37:36.195Z+ dd if=/dev/zero of=/scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
2932023-12-07T05:37:36.195Z0+0 records in
2942023-12-07T05:37:36.195Z0+0 records out
2952023-12-07T05:37:36.196Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
2962023-12-07T05:37:36.196Z+ success 'ZFS vdev /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
2972023-12-07T05:37:36.196Z+ set +x
2982023-12-07T05:37:36.196ZZFS vdev /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
2992023-12-07T05:37:36.196Z++ zpool list -o name
3002023-12-07T05:37:36.196Z++ grep oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03
3012023-12-07T05:37:36.196Z+ [[ -z '' ]]
3022023-12-07T05:37:36.196Z+ zpool create -f oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3032023-12-07T05:37:36.196Z+ success 'ZFS zpool oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3042023-12-07T05:37:36.196Z+ set +x
3052023-12-07T05:37:36.196ZZFS zpool oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
3062023-12-07T05:37:36.197Z+ for ZPOOL in "${ZPOOLS[@]}"
3072023-12-07T05:37:36.197Z+ echo 'Zpool: [oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3082023-12-07T05:37:36.197ZZpool: [oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03]
3092023-12-07T05:37:36.197Z+ VDEV_PATH=/scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3102023-12-07T05:37:36.197Z+ [[ -f /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3112023-12-07T05:37:36.197Z+ dd if=/dev/zero of=/scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3122023-12-07T05:37:36.197Z0+0 records in
3132023-12-07T05:37:36.199Z0+0 records out
3142023-12-07T05:37:36.199Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
3152023-12-07T05:37:36.199Z+ success 'ZFS vdev /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3162023-12-07T05:37:36.199Z+ set +x
3172023-12-07T05:37:36.199ZZFS vdev /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
3182023-12-07T05:37:36.200Z++ zpool list -o name
3192023-12-07T05:37:36.200Z++ grep oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03
3202023-12-07T05:37:36.200Z+ [[ -z '' ]]
3212023-12-07T05:37:36.200Z+ zpool create -f oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3222023-12-07T05:37:36.200Z+ success 'ZFS zpool oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3232023-12-07T05:37:36.200Z+ set +x
3242023-12-07T05:37:36.200ZZFS zpool oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
3252023-12-07T05:37:36.200Z+ for ZPOOL in "${ZPOOLS[@]}"
3262023-12-07T05:37:36.200Z+ echo 'Zpool: [oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3272023-12-07T05:37:36.200ZZpool: [oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03]
3282023-12-07T05:37:36.200Z+ VDEV_PATH=/scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3292023-12-07T05:37:36.200Z+ [[ -f /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3302023-12-07T05:37:36.202Z+ dd if=/dev/zero of=/scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3312023-12-07T05:37:36.202Z0+0 records in
3322023-12-07T05:37:36.202Z0+0 records out
3332023-12-07T05:37:36.202Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3342023-12-07T05:37:36.202Z+ success 'ZFS vdev /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3352023-12-07T05:37:36.202Z+ set +x
3362023-12-07T05:37:36.202ZZFS vdev /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
3372023-12-07T05:37:36.202Z++ zpool list -o name
3382023-12-07T05:37:36.202Z++ grep oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03
3392023-12-07T05:37:36.202Z+ [[ -z '' ]]
3402023-12-07T05:37:36.203Z+ zpool create -f oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3412023-12-07T05:37:36.203Z+ success 'ZFS zpool oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3422023-12-07T05:37:36.203Z+ set +x
3432023-12-07T05:37:36.203ZZFS zpool oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
3442023-12-07T05:37:36.203Z+ for ZPOOL in "${ZPOOLS[@]}"
3452023-12-07T05:37:36.229Z+ echo 'Zpool: [oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3462023-12-07T05:37:36.229ZZpool: [oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03]
3472023-12-07T05:37:36.229Z+ VDEV_PATH=/scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3482023-12-07T05:37:36.229Z+ [[ -f /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3492023-12-07T05:37:36.229Z+ dd if=/dev/zero of=/scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3502023-12-07T05:37:36.230Z0+0 records in
3512023-12-07T05:37:36.230Z0+0 records out
3522023-12-07T05:37:36.230Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3532023-12-07T05:37:36.232Z+ success 'ZFS vdev /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3542023-12-07T05:37:36.232Z+ set +x
3552023-12-07T05:37:36.232ZZFS vdev /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
3562023-12-07T05:37:36.232Z++ zpool list -o name
3572023-12-07T05:37:36.232Z++ grep oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03
3582023-12-07T05:37:36.232Z+ [[ -z '' ]]
3592023-12-07T05:37:36.233Z+ zpool create -f oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3602023-12-07T05:37:36.233Z+ success 'ZFS zpool oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3612023-12-07T05:37:36.233Z+ set +x
3622023-12-07T05:37:36.233ZZFS zpool oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
3632023-12-07T05:37:36.233Z+ for ZPOOL in "${ZPOOLS[@]}"
3642023-12-07T05:37:36.233ZZpool: [oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930]
3652023-12-07T05:37:36.233Z+ echo 'Zpool: [oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930]'
3662023-12-07T05:37:36.233Z+ VDEV_PATH=/scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev
3672023-12-07T05:37:36.233Z+ [[ -f /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev ]]
3682023-12-07T05:37:36.233Z+ dd if=/dev/zero of=/scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev bs=1 count=0 seek=20G
3692023-12-07T05:37:36.234Z0+0 records in
3702023-12-07T05:37:36.234Z0+0 records out
3712023-12-07T05:37:36.234Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3722023-12-07T05:37:36.234Z+ success 'ZFS vdev /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev exists'
3732023-12-07T05:37:36.234Z+ set +x
3742023-12-07T05:37:36.234ZZFS vdev /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev exists
3752023-12-07T05:37:36.234Z++ zpool list -o name
3762023-12-07T05:37:36.234Z++ grep oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930
3772023-12-07T05:37:36.234Z+ [[ -z '' ]]
3782023-12-07T05:37:36.234Z+ zpool create -f oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev
3792023-12-07T05:37:36.235Z+ success 'ZFS zpool oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 exists'
3802023-12-07T05:37:36.235Z+ set +x
3812023-12-07T05:37:36.235ZZFS zpool oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 exists
3822023-12-07T05:37:36.235Z+ for ZPOOL in "${ZPOOLS[@]}"
3832023-12-07T05:37:36.235Z+ echo 'Zpool: [oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60]'
3842023-12-07T05:37:36.235ZZpool: [oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60]
3852023-12-07T05:37:36.235Z+ VDEV_PATH=/scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev
3862023-12-07T05:37:36.235Z+ [[ -f /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev ]]
3872023-12-07T05:37:36.235Z+ dd if=/dev/zero of=/scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev bs=1 count=0 seek=20G
3882023-12-07T05:37:36.235Z0+0 records in
3892023-12-07T05:37:36.235Z0+0 records out
3902023-12-07T05:37:36.235Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
3912023-12-07T05:37:36.236Z+ success 'ZFS vdev /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev exists'
3922023-12-07T05:37:36.236Z+ set +x
3932023-12-07T05:37:36.236ZZFS vdev /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev exists
3942023-12-07T05:37:36.236Z++ zpool list -o name
3952023-12-07T05:37:36.236Z++ grep oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60
3962023-12-07T05:37:36.236Z+ [[ -z '' ]]
3972023-12-07T05:37:36.236Z+ zpool create -f oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev
3982023-12-07T05:37:36.236Z+ success 'ZFS zpool oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 exists'
3992023-12-07T05:37:36.236Z+ set +x
4002023-12-07T05:37:36.236ZZFS zpool oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 exists
4012023-12-07T05:37:36.236Z+ for ZPOOL in "${ZPOOLS[@]}"
4022023-12-07T05:37:36.236Z+ echo 'Zpool: [oxp_31bd71cd-4736-4a12-a387-9b74b050396f]'
4032023-12-07T05:37:36.237ZZpool: [oxp_31bd71cd-4736-4a12-a387-9b74b050396f]
4042023-12-07T05:37:36.237Z+ VDEV_PATH=/scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev
4052023-12-07T05:37:36.237Z+ [[ -f /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev ]]
4062023-12-07T05:37:36.237Z+ dd if=/dev/zero of=/scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev bs=1 count=0 seek=20G
4072023-12-07T05:37:36.237Z0+0 records in
4082023-12-07T05:37:36.237Z0+0 records out
4092023-12-07T05:37:36.237Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
4102023-12-07T05:37:36.237Z+ success 'ZFS vdev /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev exists'
4112023-12-07T05:37:36.237Z+ set +x
4122023-12-07T05:37:36.237ZZFS vdev /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev exists
4132023-12-07T05:37:36.238Z++ zpool list -o name
4142023-12-07T05:37:36.238Z++ grep oxp_31bd71cd-4736-4a12-a387-9b74b050396f
4152023-12-07T05:37:36.238Z+ [[ -z '' ]]
4162023-12-07T05:37:36.238Z+ zpool create -f oxp_31bd71cd-4736-4a12-a387-9b74b050396f /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev
4172023-12-07T05:37:36.238Z+ success 'ZFS zpool oxp_31bd71cd-4736-4a12-a387-9b74b050396f exists'
4182023-12-07T05:37:36.238Z+ set +x
4192023-12-07T05:37:36.238ZZFS zpool oxp_31bd71cd-4736-4a12-a387-9b74b050396f exists
4202023-12-07T05:37:36.238Z+ for ZPOOL in "${ZPOOLS[@]}"
4212023-12-07T05:37:36.238Z+ echo 'Zpool: [oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7]'
4222023-12-07T05:37:36.238ZZpool: [oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7]
4232023-12-07T05:37:36.238Z+ VDEV_PATH=/scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev
4242023-12-07T05:37:36.239Z+ [[ -f /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev ]]
4252023-12-07T05:37:36.239Z+ dd if=/dev/zero of=/scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev bs=1 count=0 seek=20G
4262023-12-07T05:37:36.269Z0+0 records in
4272023-12-07T05:37:36.271Z0+0 records out
4282023-12-07T05:37:36.271Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4292023-12-07T05:37:36.271Z+ success 'ZFS vdev /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev exists'
4302023-12-07T05:37:36.271Z+ set +x
4312023-12-07T05:37:36.271ZZFS vdev /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev exists
4322023-12-07T05:37:36.271Z++ zpool list -o name
4332023-12-07T05:37:36.271Z++ grep oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7
4342023-12-07T05:37:36.272Z+ [[ -z '' ]]
4352023-12-07T05:37:36.272Z+ zpool create -f oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev
4362023-12-07T05:37:36.297Z+ success 'ZFS zpool oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 exists'
4372023-12-07T05:37:36.297Z+ set +x
4382023-12-07T05:37:36.298ZZFS zpool oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 exists
4392023-12-07T05:37:36.298Z+ for ZPOOL_TYPE in "${ZPOOL_TYPES[@]}"
4402023-12-07T05:37:36.298Z+ readarray -t ZPOOLS
4412023-12-07T05:37:36.298Z++ grep '"oxi_' /opt/oxide/work/tools/../smf/sled-agent/non-gimlet/config.toml
4422023-12-07T05:37:36.298Z++ sed 's/[ ",]//g'
4432023-12-07T05:37:36.298Z+ for ZPOOL in "${ZPOOLS[@]}"
4442023-12-07T05:37:36.298Z+ echo 'Zpool: [oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b]'
4452023-12-07T05:37:36.299ZZpool: [oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b]
4462023-12-07T05:37:36.299Z+ VDEV_PATH=/scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4472023-12-07T05:37:36.299Z+ [[ -f /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
4482023-12-07T05:37:36.299Z+ dd if=/dev/zero of=/scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
4492023-12-07T05:37:36.299Z0+0 records in
4502023-12-07T05:37:36.299Z0+0 records out
4512023-12-07T05:37:36.299Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
4522023-12-07T05:37:36.312Z+ success 'ZFS vdev /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
4532023-12-07T05:37:36.312Z+ set +x
4542023-12-07T05:37:36.312ZZFS vdev /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
4552023-12-07T05:37:36.313Z++ zpool list -o name
4562023-12-07T05:37:36.313Z++ grep oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b
4572023-12-07T05:37:36.313Z+ [[ -z '' ]]
4582023-12-07T05:37:36.313Z+ zpool create -f oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4592023-12-07T05:37:36.339Z+ success 'ZFS zpool oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
4602023-12-07T05:37:36.339Z+ set +x
4612023-12-07T05:37:36.339ZZFS zpool oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b exists
4622023-12-07T05:37:36.339Z+ for ZPOOL in "${ZPOOLS[@]}"
4632023-12-07T05:37:36.339Z+ echo 'Zpool: [oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b]'
4642023-12-07T05:37:36.339ZZpool: [oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b]
4652023-12-07T05:37:36.339Z+ VDEV_PATH=/scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4662023-12-07T05:37:36.340Z+ [[ -f /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
4672023-12-07T05:37:36.340Z+ dd if=/dev/zero of=/scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
4682023-12-07T05:37:36.340Z0+0 records in
4692023-12-07T05:37:36.340Z0+0 records out
4702023-12-07T05:37:36.340Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
4712023-12-07T05:37:36.340Z+ success 'ZFS vdev /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
4722023-12-07T05:37:36.340Z+ set +x
4732023-12-07T05:37:36.340ZZFS vdev /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
4742023-12-07T05:37:36.340Z++ zpool list -o name
4752023-12-07T05:37:36.340Z++ grep oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b
4762023-12-07T05:37:36.365Z+ [[ -z '' ]]
4772023-12-07T05:37:36.375Z+ zpool create -f oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4782023-12-07T05:37:36.412Z+ success 'ZFS zpool oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
4792023-12-07T05:37:36.412Z+ set +x
4802023-12-07T05:37:36.413ZZFS zpool oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b exists
4812023-12-07T05:37:36.413Z+ [[ zone == \z\o\n\e ]]
4822023-12-07T05:37:36.413Z+ ensure_simulated_links igb0
4832023-12-07T05:37:36.413Z+ local PHYSICAL_LINK=igb0
4842023-12-07T05:37:36.413Z+ INDICES=("0" "1")
4852023-12-07T05:37:36.413Z+ for I in "${INDICES[@]}"
4862023-12-07T05:37:36.413Z++ get_simnet_name_if_exists net0
4872023-12-07T05:37:36.413Z++ dladm show-simnet -p -o LINK net0
4882023-12-07T05:37:36.413Z++ echo ''
4892023-12-07T05:37:36.414Z+ [[ -z '' ]]
4902023-12-07T05:37:36.414Z+ dladm create-simnet -t net0
4912023-12-07T05:37:36.414Z+ dladm create-simnet -t sc0_0
4922023-12-07T05:37:36.414Z+ dladm modify-simnet -t -p net0 sc0_0
4932023-12-07T05:37:36.414Z+ dladm set-linkprop -p mtu=9000 sc0_0
4942023-12-07T05:37:36.414Z+ success 'Simnet net0/sc0_0 exists'
4952023-12-07T05:37:36.414Z+ set +x
4962023-12-07T05:37:36.414Z+ for I in "${INDICES[@]}"
4972023-12-07T05:37:36.414ZSimnet net0/sc0_0 exists
4982023-12-07T05:37:36.414Z++ get_simnet_name_if_exists net1
4992023-12-07T05:37:36.416Z++ dladm show-simnet -p -o LINK net1
5002023-12-07T05:37:36.416Z++ echo ''
5012023-12-07T05:37:36.445Z+ [[ -z '' ]]
5022023-12-07T05:37:36.445Z+ dladm create-simnet -t net1
5032023-12-07T05:37:36.445Z+ dladm create-simnet -t sc1_0
5042023-12-07T05:37:36.446Z+ dladm modify-simnet -t -p net1 sc1_0
5052023-12-07T05:37:36.446Z+ dladm set-linkprop -p mtu=9000 sc1_0
5062023-12-07T05:37:36.446Z+ success 'Simnet net1/sc1_0 exists'
5072023-12-07T05:37:36.446Z+ set +x
5082023-12-07T05:37:36.446ZSimnet net1/sc1_0 exists
5092023-12-07T05:37:36.446Z++ get_vnic_name_if_exists sc0_1
5102023-12-07T05:37:36.446Z++ dladm show-vnic -p -o LINK sc0_1
5112023-12-07T05:37:36.446Z++ echo ''
5122023-12-07T05:37:36.446Z+ [[ -z '' ]]
5132023-12-07T05:37:36.446Z+ dladm create-vnic -t sc0_1 -l igb0 -m a8:e1:de:01:70:1d
5142023-12-07T05:37:36.447Z+ [[ -v PROMISC_FILT_OFF ]]
5152023-12-07T05:37:36.447Z+ success 'Vnic sc0_1 exists'
5162023-12-07T05:37:36.447Z+ set +x
5172023-12-07T05:37:36.447ZVnic sc0_1 exists
5182023-12-07T05:37:36.447Z+ warn_if_no_proxy_arp
5192023-12-07T05:37:36.447Z+ [[ -v PXA_START ]]
5202023-12-07T05:37:36.447Z+ [[ -v PXA_END ]]
5212023-12-07T05:37:36.447Z+ ensure_softnpu_zone
5222023-12-07T05:37:36.447Z+ zoneadm list
5232023-12-07T05:37:36.447Z+ grep -q sidecar_softnpu
5242023-12-07T05:37:36.448Z+ [[ -f out/npuzone/npuzone ]]
5252023-12-07T05:37:36.448Z+ out/npuzone/npuzone create sidecar --omicron-zone --ports sc0_0,tfportrear0_0 --ports sc0_1,tfportqsfp0_0 --sidecar-lite-commit f0585a29fb0285f7a1220c1118856b0e5c1f75c5 --softnpu-commit dec63e67156fe6e958991bbfa090629868115ab5
5262023-12-07T05:37:36.473Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/softnpu/image/dec63e67156fe6e958991bbfa090629868115ab5/softnpu.sha256.txt
5272023-12-07T05:37:44.982Zdownloading softnpu ...done
5282023-12-07T05:37:45.030Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/sidecar-lite/release/f0585a29fb0285f7a1220c1118856b0e5c1f75c5/libsidecar_lite.so.sha256.txt
5292023-12-07T05:37:54.353Zdownloading asic_program.so ...done
5302023-12-07T05:37:54.384Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/sidecar-lite/release/f0585a29fb0285f7a1220c1118856b0e5c1f75c5/scadm.sha256.txt
5312023-12-07T05:38:00.677Zdownloading scadm ...done
5322023-12-07T05:38:00.807Zcreating port tfportrear0_0 for sc0_0
5332023-12-07T05:38:00.807Zcreating port tfportqsfp0_0 for sc0_1
5342023-12-07T05:38:00.832Zsoftnpu zone setup
5352023-12-07T05:38:00.833Zconfigure zone
5362023-12-07T05:38:00.833Zinstall zone
5372023-12-07T05:38:01.771Zboot zone
5382023-12-07T05:38:04.874Z[sidecar_softnpu] svcs milestone/network
5392023-12-07T05:38:04.953ZSTATE STIME FMRI
5402023-12-07T05:38:04.953Zonline 5:38:03 svc:/milestone/network:default
5412023-12-07T05:38:04.953Z[sidecar_softnpu] /softnpu/npu start
5422023-12-07T05:38:05.393Z
5432023-12-07T05:38:05.393Z+ /opt/oxide/work/tools/scrimlet/softnpu-init.sh
5442023-12-07T05:38:05.393Z+ GATEWAY_IP=10.151.1.100
5452023-12-07T05:38:05.393Z+ echo 'Using 10.151.1.100 as gateway ip'
5462023-12-07T05:38:05.393ZUsing 10.151.1.100 as gateway ip
5472023-12-07T05:38:05.393Z+ [[ ! -v GATEWAY_MAC ]]
5482023-12-07T05:38:05.393Z+ GATEWAY_MAC=18:c0:4d:0d:9f:b2
5492023-12-07T05:38:05.393Z++ grep -c -E '^([0-9a-fA-F]{2}:){5}[0-9a-fA-F]{2}$' /dev/fd/63
5502023-12-07T05:38:05.393Z+++ echo 18:c0:4d:0d:9f:b2
5512023-12-07T05:38:05.393Z+ COUNT=1
5522023-12-07T05:38:05.393Z+ [[ 1 -ne 1 ]]
5532023-12-07T05:38:05.394Z+ echo 'Using 18:c0:4d:0d:9f:b2 as gateway mac'
5542023-12-07T05:38:05.394ZUsing 18:c0:4d:0d:9f:b2 as gateway mac
5552023-12-07T05:38:05.394Z+ z_scadm add-arp-entry 10.151.1.100 18:c0:4d:0d:9f:b2
5562023-12-07T05:38:05.394Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone add-arp-entry 10.151.1.100 18:c0:4d:0d:9f:b2
5572023-12-07T05:38:05.394Z+ PXA_MAC=a8:e1:de:01:70:1d
5582023-12-07T05:38:05.394Z+ [[ -v PXA_START ]]
5592023-12-07T05:38:05.394Z+ z_scadm add-proxy-arp 10.151.1.164 10.151.1.227 a8:e1:de:01:70:1d
5602023-12-07T05:38:05.394Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone add-proxy-arp 10.151.1.164 10.151.1.227 a8:e1:de:01:70:1d
5612023-12-07T05:38:05.442Z+ z_scadm dump-state
5622023-12-07T05:38:05.443Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone dump-state
5632023-12-07T05:38:05.443Zlocal v6:
5642023-12-07T05:38:05.445Zlocal v4:
5652023-12-07T05:38:05.445Zrouter v6:
5662023-12-07T05:38:05.445Zrouter v4_idx:
5672023-12-07T05:38:05.445Zrouter v4_routes:
5682023-12-07T05:38:05.446Zresolver v4:
5692023-12-07T05:38:05.446Z10.151.1.100 -> 18:c0:4d:0d:9f:b2
5702023-12-07T05:38:05.446Zresolver v6:
5712023-12-07T05:38:05.446Znat v4:
5722023-12-07T05:38:05.446Znat_v6:
5732023-12-07T05:38:05.446Zport_mac:
5742023-12-07T05:38:05.446Zproxy arp:
5752023-12-07T05:38:05.446Z10.151.1.164/10.151.1.227: a8:e1:de:01:70:1d
5762023-12-07T05:38:05.446Z+ success 'softnpu zone exists'
5772023-12-07T05:38:05.446Z+ set +x
5782023-12-07T05:38:05.447Zsoftnpu zone exists
5792023-12-07T05:38:05.447Z
5802023-12-07T05:38:05.447Zreal 29.264536568
5812023-12-07T05:38:05.447Zuser 0.712671523
5822023-12-07T05:38:05.447Zsys 1.576404634
5832023-12-07T05:38:05.447Ztrap 0.000080376
5842023-12-07T05:38:05.447Ztflt 0.000139525
5852023-12-07T05:38:05.447Zdflt 0.001036522
5862023-12-07T05:38:05.447Zkflt 0.000000000
5872023-12-07T05:38:05.447Zlock 22:45.171197319
5882023-12-07T05:38:05.448Zslp 1:27.585788058
5892023-12-07T05:38:05.448Zlat 0.055715975
5902023-12-07T05:38:05.448Zstop 0.351749427
5912023-12-07T05:38:05.448Z+ tar xf out/omicron-sled-agent.tar pkg/config-rss.toml
5922023-12-07T05:38:05.448Z++ sed -n 's/silo_name = "\(.*\)"/\1/p' pkg/config-rss.toml
5932023-12-07T05:38:05.448Z+ SILO_NAME=recovery
5942023-12-07T05:38:05.448Z++ sed -n 's/external_dns_zone_name = "\(.*\)"/\1/p' pkg/config-rss.toml
5952023-12-07T05:38:05.448Z+ EXTERNAL_DNS_DOMAIN=oxide.test
5962023-12-07T05:38:05.448Z+ sed -i~ '
5972023-12-07T05:38:05.449Z /^external_dns_ips/c\
5982023-12-07T05:38:05.449Zexternal_dns_ips = [ "10.151.1.164", "10.151.1.165" ]
5992023-12-07T05:38:05.449Z /^\[\[internal_services_ip_pool_ranges/,/^$/ {
6002023-12-07T05:38:05.449Z /^first/c\
6012023-12-07T05:38:05.449Zfirst = "10.151.1.164"
6022023-12-07T05:38:05.449Z /^last/c\
6032023-12-07T05:38:05.449Zlast = "10.151.1.173"
6042023-12-07T05:38:05.449Z }
6052023-12-07T05:38:05.449Z /^\[rack_network_config/,/^$/ {
6062023-12-07T05:38:05.450Z /^infra_ip_first/c\
6072023-12-07T05:38:05.450Zinfra_ip_first = "10.151.1.174"
6082023-12-07T05:38:05.450Z /^infra_ip_last/c\
6092023-12-07T05:38:05.450Zinfra_ip_last = "10.151.1.174"
6102023-12-07T05:38:05.450Z }
6112023-12-07T05:38:05.450Z /^\[\[rack_network_config.ports/,/^$/ {
6122023-12-07T05:38:05.450Z /^routes/c\
6132023-12-07T05:38:05.450Zroutes = \[{nexthop = "10.151.1.100", destination = "0.0.0.0/0"}\]
6142023-12-07T05:38:05.450Z /^addresses/c\
6152023-12-07T05:38:05.450Zaddresses = \["10.151.1.174/32"\]
6162023-12-07T05:38:05.451Z }
6172023-12-07T05:38:05.451Z' pkg/config-rss.toml
6182023-12-07T05:38:05.451Z+ diff -u pkg/config-rss.toml~ pkg/config-rss.toml
6192023-12-07T05:38:05.451Z--- pkg/config-rss.toml~ Mon Jul 24 01:21:28 2006
6202023-12-07T05:38:05.451Z+++ pkg/config-rss.toml Thu Dec 7 05:38:05 2023
6212023-12-07T05:38:05.451Z@@ -41,7 +41,7 @@
6222023-12-07T05:38:05.451Z # the DNS domain delegated to the rack by the customer. Each of these addresses
6232023-12-07T05:38:05.469Z # must be contained in one of the "internal services" IP Pool ranges listed
6242023-12-07T05:38:05.469Z # below.
6252023-12-07T05:38:05.469Z-external_dns_ips = [ "192.168.1.20", "192.168.1.21" ]
6262023-12-07T05:38:05.469Z+external_dns_ips = [ "10.151.1.164", "10.151.1.165" ]
6272023-12-07T05:38:05.469Z
6282023-12-07T05:38:05.470Z # Initial TLS certificates for the external API
6292023-12-07T05:38:05.470Z #
6302023-12-07T05:38:05.470Z@@ -77,8 +77,8 @@
6312023-12-07T05:38:05.470Z #
6322023-12-07T05:38:05.470Z # For more on this and what to put here, see docs/how-to-run.adoc.
6332023-12-07T05:38:05.470Z [[internal_services_ip_pool_ranges]]
6342023-12-07T05:38:05.470Z-first = "192.168.1.20"
6352023-12-07T05:38:05.470Z-last = "192.168.1.29"
6362023-12-07T05:38:05.470Z+first = "10.151.1.164"
6372023-12-07T05:38:05.470Z+last = "10.151.1.173"
6382023-12-07T05:38:05.470Z
6392023-12-07T05:38:05.470Z # TODO - this configuration is subject to change going forward. Ultimately these
6402023-12-07T05:38:05.470Z # parameters should be provided to the control plane via wicket, but we need to
6412023-12-07T05:38:05.470Z@@ -92,8 +92,8 @@
6422023-12-07T05:38:05.470Z # A range of IP addresses used by Boundary Services on the external network. In
6432023-12-07T05:38:05.470Z # a real system, these would be addresses of the uplink ports on the Sidecar.
6442023-12-07T05:38:05.470Z # With softnpu, only one address is used.
6452023-12-07T05:38:05.470Z-infra_ip_first = "192.168.1.30"
6462023-12-07T05:38:05.471Z-infra_ip_last = "192.168.1.30"
6472023-12-07T05:38:05.471Z+infra_ip_first = "10.151.1.174"
6482023-12-07T05:38:05.471Z+infra_ip_last = "10.151.1.174"
6492023-12-07T05:38:05.471Z
6502023-12-07T05:38:05.472Z # Configurations for BGP routers to run on the scrimlets.
6512023-12-07T05:38:05.472Z bgp = []
6522023-12-07T05:38:05.472Z@@ -101,9 +101,9 @@
6532023-12-07T05:38:05.472Z # You can configure multiple uplinks by repeating the following stanza
6542023-12-07T05:38:05.472Z [[rack_network_config.ports]]
6552023-12-07T05:38:05.472Z # Routes associated with this port.
6562023-12-07T05:38:05.472Z-routes = [{nexthop = "192.168.1.199", destination = "0.0.0.0/0"}]
6572023-12-07T05:38:05.472Z+routes = [{nexthop = "10.151.1.100", destination = "0.0.0.0/0"}]
6582023-12-07T05:38:05.472Z # Addresses associated with this port.
6592023-12-07T05:38:05.472Z-addresses = ["192.168.1.30/32"]
6602023-12-07T05:38:05.472Z+addresses = ["10.151.1.174/32"]
6612023-12-07T05:38:05.539Z # Name of the uplink port. This should always be "qsfp0" when using softnpu.
6622023-12-07T05:38:05.540Z port = "qsfp0"
6632023-12-07T05:38:05.541Z # The speed of this port.
6642023-12-07T05:38:05.541Z+ true
6652023-12-07T05:38:05.541Z+ tar rvf out/omicron-sled-agent.tar pkg/config-rss.toml
6662023-12-07T05:38:05.541Za pkg/config-rss.toml 7K
6672023-12-07T05:38:05.541Z+ rm -f pkg/config-rss.toml pkg/config-rss.toml~
6682023-12-07T05:38:05.541Z+ TLS_NAME=recovery.sys.oxide.test
6692023-12-07T05:38:05.541Z+ openssl req -newkey rsa:4096 -x509 -sha256 -days 3 -nodes -out pkg/initial-tls-cert.pem -keyout pkg/initial-tls-key.pem -subj /CN=recovery.sys.oxide.test -addext subjectAltName=DNS:recovery.sys.oxide.test -addext basicConstraints=critical,CA:FALSE -config /dev/stdin
6702023-12-07T05:38:05.918Z.............+....+........+......+.........+......+....+...+..+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*........+...+......+....+..+.........+.......+...+.....+......+.+..+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*...........+.......+..+....+.....+..........+.................+...+..........+......+..............+.+........+.+......+...+..+...+...+.........+......+...+....+.....+...+...............+.......+......+..+...+....+...........+..........+..............+......+.+.....+.+...............+.....+.............+.....+.+...+...+...+............+..+.......+......+..............+......+....+.....+...............+...............+...+.+.....+.+......+...............+.......................+...............+.+...........+.........+......+.............+..+..........+..+...+.........+.+...+...............+..+............+......+......+.+.................+..........+..+...+......+.+..............+....+...+...+.....+......+.+..+.......+......+..+...+.......+......+.........+.................+......+...+..........+.........+............+........+............+.........+....+..+......+............+.......+.....+...+...............+.+..+..........+...+.....+.+......+............+..+...+...+.......+...+........+..........+..+.........+..................+................+......+.....+.+...+.....+.+..............+.......+.....+.........+...+.+..+.........+....+.....+....+...+..+.+...........+...............+....+.....+............+....+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
6712023-12-07T05:38:06.297Z.+..............+.......+........+.+......+...+..+............+...+.......+........+....+...............+...+.....+......+............+...+.......+.....+.........+......+...+......+.+...........................+...+..+.+.....+.......+...+.....+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*...+....+...........+....+......+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.............+.+............+........+.......+.....+............+............+.+............+..+......+.............+.....+.+........+.......+.........+..............+.+..+....+..............+.+..................+.....+.......+...+.....+......+.........+............+.+..+.......+..............+.......+.....+...+.+...+...+............+........+....+...+..................+.....+.........+...+.............+............+..................+..................+......+........+...+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
6722023-12-07T05:38:06.322Z-----
6732023-12-07T05:38:06.322Z+ tar rvf out/omicron-sled-agent.tar pkg/initial-tls-cert.pem pkg/initial-tls-key.pem
6742023-12-07T05:38:06.758Za pkg/initial-tls-cert.pem 2K
6752023-12-07T05:38:06.758Za pkg/initial-tls-key.pem 4K
6762023-12-07T05:38:06.758Z+ rm -f pkg/initial-tls-cert.pem pkg/initial-tls-key.pem
6772023-12-07T05:38:06.759Z+ rmdir pkg
6782023-12-07T05:38:06.759Z+ E2E_TLS_CERT=/opt/oxide/sled-agent/pkg/initial-tls-cert.pem
6792023-12-07T05:38:06.759Z+ pfexec mkdir -p /usr/oxide
6802023-12-07T05:38:06.759Z+ pfexec rm -f /usr/oxide/tcpproxy
6812023-12-07T05:38:06.759Z+ pfexec curl -sSfL -o /usr/oxide/tcpproxy http://catacomb.eng.oxide.computer:12346/tcpproxy
6822023-12-07T05:38:06.784Z+ pfexec chmod +x /usr/oxide/tcpproxy
6832023-12-07T05:38:06.784Z+ pfexec rm -f /var/svc/manifest/site/tcpproxy.xml
6842023-12-07T05:38:06.784Z+ pfexec curl -sSfL -o /var/svc/manifest/site/tcpproxy.xml http://catacomb.eng.oxide.computer:12346/tcpproxy.xml
6852023-12-07T05:38:06.785Z+ pfexec svccfg import /var/svc/manifest/site/tcpproxy.xml
6862023-12-07T05:38:06.785Z+ OMICRON_NO_UNINSTALL=1
6872023-12-07T05:38:06.785Z+ ptime -m pfexec ./target/release/omicron-package -t test install
6882023-12-07T05:38:06.785ZDec 07 05:38:07.465 DEBG target[test]: Target({"image": "standard", "machine": "non-gimlet", "rack-topology": "single-sled", "switch": "softnpu"})
6892023-12-07T05:38:06.785ZDec 07 05:38:07.467 INFO Installing service, dst: /opt/oxide/clickhouse.tar.gz, src: out/clickhouse.tar.gz
6902023-12-07T05:38:06.785ZDec 07 05:38:07.467 INFO Installing service, dst: /opt/oxide/internal_dns.tar.gz, src: out/internal-dns.tar.gz
6912023-12-07T05:38:06.785ZDec 07 05:38:07.467 INFO Installing service, dst: /opt/oxide/ntp.tar.gz, src: out/ntp.tar.gz
6922023-12-07T05:38:06.785ZDec 07 05:38:07.467 INFO Installing service, dst: /opt/oxide/cockroachdb.tar.gz, src: out/cockroachdb.tar.gz
6932023-12-07T05:38:06.785ZDec 07 05:38:07.467 INFO Installing service, dst: /opt/oxide/propolis-server.tar.gz, src: out/propolis-server.tar.gz
6942023-12-07T05:38:06.785ZDec 07 05:38:07.467 INFO Installing service, dst: /opt/oxide/switch.tar.gz, src: out/switch-softnpu.tar.gz
6952023-12-07T05:38:06.785ZDec 07 05:38:07.467 INFO Installing service, dst: /opt/oxide/mg-ddm.tar, src: out/mg-ddm-gz.tar
6962023-12-07T05:38:06.786ZDec 07 05:38:07.467 INFO Installing service, dst: /opt/oxide/crucible_pantry.tar.gz, src: out/crucible-pantry.tar.gz
6972023-12-07T05:38:06.786ZDec 07 05:38:07.467 INFO Installing service, dst: /opt/oxide/external_dns.tar.gz, src: out/external-dns.tar.gz
6982023-12-07T05:38:06.786ZDec 07 05:38:07.467 INFO Installing service, dst: /opt/oxide/crucible.tar.gz, src: out/crucible.tar.gz
6992023-12-07T05:38:06.786ZDec 07 05:38:07.467 INFO Installing service, dst: /opt/oxide/overlay.tar.gz, src: out/overlay.tar.gz
7002023-12-07T05:38:06.786ZDec 07 05:38:07.467 INFO Installing service, dst: /opt/oxide/clickhouse_keeper.tar.gz, src: out/clickhouse_keeper.tar.gz
7012023-12-07T05:38:06.786ZDec 07 05:38:07.467 INFO Installing service, dst: /opt/oxide/mgs.tar.gz, src: out/omicron-gateway-softnpu.tar.gz
7022023-12-07T05:38:06.786ZDec 07 05:38:07.467 INFO Installing service, dst: /opt/oxide/sled-agent.tar, src: out/omicron-sled-agent.tar
7032023-12-07T05:38:06.786ZDec 07 05:38:07.467 INFO Installing service, dst: /opt/oxide/oximeter.tar.gz, src: out/oximeter-collector.tar.gz
7042023-12-07T05:38:06.786ZDec 07 05:38:07.467 INFO Installing service, dst: /opt/oxide/nexus.tar.gz, src: out/omicron-nexus.tar.gz
7052023-12-07T05:38:07.029ZDec 07 05:38:07.713 INFO Unpacking service tarball, service_path: /opt/oxide/mg-ddm, tar_path: /opt/oxide/mg-ddm.tar
7062023-12-07T05:38:07.066ZDec 07 05:38:07.749 INFO Unpacking service tarball, service_path: /opt/oxide/sled-agent, tar_path: /opt/oxide/sled-agent.tar
7072023-12-07T05:38:07.127ZDec 07 05:38:07.810 INFO Installing bootstrap service from /opt/oxide/sled-agent/pkg/manifest.xml
7082023-12-07T05:38:07.267Z
7092023-12-07T05:38:07.300Zreal 0.482980027
7102023-12-07T05:38:07.300Zuser 0.098882380
7112023-12-07T05:38:07.300Zsys 1.975658956
7122023-12-07T05:38:07.300Ztrap 0.000006389
7132023-12-07T05:38:07.300Ztflt 0.001428811
7142023-12-07T05:38:07.301Zdflt 0.000443301
7152023-12-07T05:38:07.301Zkflt 0.000000000
7162023-12-07T05:38:07.301Zlock 43.423231587
7172023-12-07T05:38:07.301Zslp 0.699801044
7182023-12-07T05:38:07.301Zlat 0.003869740
7192023-12-07T05:38:07.302Zstop 0.214953234
7202023-12-07T05:38:07.302Z+ retry=0
7212023-12-07T05:38:07.302Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7222023-12-07T05:38:07.303Z+ [[ 0 -gt 30 ]]
7232023-12-07T05:38:07.303Z+ sleep 1
7242023-12-07T05:38:08.273Z+ retry=1
7252023-12-07T05:38:08.273Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7262023-12-07T05:38:08.298Z+ [[ 1 -gt 30 ]]
7272023-12-07T05:38:08.298Z+ sleep 1
7282023-12-07T05:38:09.289Z+ retry=2
7292023-12-07T05:38:09.289Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7302023-12-07T05:38:09.315Z+ [[ 2 -gt 30 ]]
7312023-12-07T05:38:09.315Z+ sleep 1
7322023-12-07T05:38:10.308Z+ retry=3
7332023-12-07T05:38:10.308Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7342023-12-07T05:38:10.336Z+ [[ 3 -gt 30 ]]
7352023-12-07T05:38:10.336Z+ sleep 1
7362023-12-07T05:38:11.316Z+ retry=4
7372023-12-07T05:38:11.316Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7382023-12-07T05:38:31.177Z+ [[ 4 -gt 30 ]]
7392023-12-07T05:38:31.177Z+ sleep 1
7402023-12-07T05:38:32.181Z+ retry=5
7412023-12-07T05:38:32.181Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7422023-12-07T05:38:32.208Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone dump-state
7432023-12-07T05:38:32.238Zlocal v6:
7442023-12-07T05:38:32.238Zfe80::583f:8dff:fedb:1f92
7452023-12-07T05:38:32.238Zfd00:99::1
7462023-12-07T05:38:32.239Zlocal v4:
7472023-12-07T05:38:32.239Z10.151.1.174
7482023-12-07T05:38:32.239Zrouter v6:
7492023-12-07T05:38:32.239Zrouter v4_idx:
7502023-12-07T05:38:32.239Z0.0.0.0/0 -> 0
7512023-12-07T05:38:32.239Zrouter v4_routes:
7522023-12-07T05:38:32.239Z0 -> 10.151.1.100 (2)
7532023-12-07T05:38:32.240Zresolver v4:
7542023-12-07T05:38:32.240Z10.151.1.100 -> 18:c0:4d:0d:9f:b2
7552023-12-07T05:38:32.240Zresolver v6:
7562023-12-07T05:38:32.240Zfe80::583f:8dff:fedb:1f92 -> 5a:3f:8d:db:1f:92
7572023-12-07T05:38:32.240Zfe80::54ec:ecff:fe2f:3230 -> 56:ec:ec:2f:32:30
7582023-12-07T05:38:32.240Znat v4:
7592023-12-07T05:38:32.240Znat_v6:
7602023-12-07T05:38:32.240Zport_mac:
7612023-12-07T05:38:32.240Z1000: a8:40:25:20:33:e6
7622023-12-07T05:38:32.241Z1: a8:40:25:20:33:e7
7632023-12-07T05:38:32.241Zproxy arp:
7642023-12-07T05:38:32.241Z10.151.1.164/10.151.1.227: a8:e1:de:01:70:1d
7652023-12-07T05:38:32.241Z+ retry=0
7662023-12-07T05:38:32.241Z+++ zoneadm list -n
7672023-12-07T05:38:32.241Z+++ grep oxz_ntp
7682023-12-07T05:38:32.241Z++ pfexec svcs -z -Hostate oxide/ntp
7692023-12-07T05:38:32.266Zsvcs: zone '-Hostate': entity not found
7702023-12-07T05:38:32.266Z++ true
7712023-12-07T05:38:32.266Z+ [[ '' != online ]]
7722023-12-07T05:38:32.266Z+ [[ 0 -gt 60 ]]
7732023-12-07T05:38:32.266Z+ sleep 1
7742023-12-07T05:38:33.256Z+ retry=1
7752023-12-07T05:38:33.284Z+++ zoneadm list -n
7762023-12-07T05:38:33.284Z+++ grep oxz_ntp
7772023-12-07T05:38:33.284Z++ pfexec svcs -z -Hostate oxide/ntp
7782023-12-07T05:38:33.285Zsvcs: zone '-Hostate': entity not found
7792023-12-07T05:38:33.285Z++ true
7802023-12-07T05:38:33.285Z+ [[ '' != online ]]
7812023-12-07T05:38:33.285Z+ [[ 1 -gt 60 ]]
7822023-12-07T05:38:33.285Z+ sleep 1
7832023-12-07T05:38:34.266Z+ retry=2
7842023-12-07T05:38:34.291Z+++ zoneadm list -n
7852023-12-07T05:38:34.291Z+++ grep oxz_ntp
7862023-12-07T05:38:34.291Z++ pfexec svcs -z -Hostate oxide/ntp
7872023-12-07T05:38:34.291Zsvcs: zone '-Hostate': entity not found
7882023-12-07T05:38:34.291Z++ true
7892023-12-07T05:38:34.291Z+ [[ '' != online ]]
7902023-12-07T05:38:34.291Z+ [[ 2 -gt 60 ]]
7912023-12-07T05:38:34.291Z+ sleep 1
7922023-12-07T05:38:35.288Z+ retry=3
7932023-12-07T05:38:35.371Z+++ zoneadm list -n
7942023-12-07T05:38:35.371Z+++ grep oxz_ntp
7952023-12-07T05:38:35.371Z++ pfexec svcs -z oxz_ntp_b9a0957f-237b-41c7-8e2a-526915db846f -Hostate oxide/ntp
7962023-12-07T05:38:35.396Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
7972023-12-07T05:38:35.396Z++ true
7982023-12-07T05:38:35.396Z+ [[ '' != online ]]
7992023-12-07T05:38:35.396Z+ [[ 3 -gt 60 ]]
8002023-12-07T05:38:35.396Z+ sleep 1
8012023-12-07T05:38:36.358Z+ retry=4
8022023-12-07T05:38:36.385Z+++ zoneadm list -n
8032023-12-07T05:38:36.385Z+++ grep oxz_ntp
8042023-12-07T05:38:36.385Z++ pfexec svcs -z oxz_ntp_b9a0957f-237b-41c7-8e2a-526915db846f -Hostate oxide/ntp
8052023-12-07T05:38:36.385Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
8062023-12-07T05:38:36.385Z++ true
8072023-12-07T05:38:36.385Z+ [[ '' != online ]]
8082023-12-07T05:38:36.385Z+ [[ 4 -gt 60 ]]
8092023-12-07T05:38:36.385Z+ sleep 1
8102023-12-07T05:38:37.386Z+ retry=5
8112023-12-07T05:38:37.412Z+++ zoneadm list -n
8122023-12-07T05:38:37.412Z+++ grep oxz_ntp
8132023-12-07T05:38:37.413Z++ pfexec svcs -z oxz_ntp_b9a0957f-237b-41c7-8e2a-526915db846f -Hostate oxide/ntp
8142023-12-07T05:38:37.413Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
8152023-12-07T05:38:37.413Z++ true
8162023-12-07T05:38:37.413Z+ [[ '' != online ]]
8172023-12-07T05:38:37.413Z+ [[ 5 -gt 60 ]]
8182023-12-07T05:38:37.413Z+ sleep 1
8192023-12-07T05:38:38.414Z+ retry=6
8202023-12-07T05:38:38.440Z+++ zoneadm list -n
8212023-12-07T05:38:38.440Z+++ grep oxz_ntp
8222023-12-07T05:38:38.440Z++ pfexec svcs -z oxz_ntp_b9a0957f-237b-41c7-8e2a-526915db846f -Hostate oxide/ntp
8232023-12-07T05:38:38.441Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
8242023-12-07T05:38:38.441Z++ true
8252023-12-07T05:38:38.467Z+ [[ '' != online ]]
8262023-12-07T05:38:38.467Z+ [[ 6 -gt 60 ]]
8272023-12-07T05:38:38.467Z+ sleep 1
8282023-12-07T05:38:39.444Z+ retry=7
8292023-12-07T05:38:39.470Z+++ zoneadm list -n
8302023-12-07T05:38:39.470Z+++ grep oxz_ntp
8312023-12-07T05:38:39.470Z++ pfexec svcs -z oxz_ntp_b9a0957f-237b-41c7-8e2a-526915db846f -Hostate oxide/ntp
8322023-12-07T05:38:39.470Z+ [[ disabled != online ]]
8332023-12-07T05:38:39.470Z+ [[ 7 -gt 60 ]]
8342023-12-07T05:38:39.470Z+ sleep 1
8352023-12-07T05:38:40.466Z+ retry=8
8362023-12-07T05:38:40.491Z+++ zoneadm list -n
8372023-12-07T05:38:40.491Z+++ grep oxz_ntp
8382023-12-07T05:38:40.491Z++ pfexec svcs -z oxz_ntp_b9a0957f-237b-41c7-8e2a-526915db846f -Hostate oxide/ntp
8392023-12-07T05:38:40.492Z+ [[ disabled != online ]]
8402023-12-07T05:38:40.492Z+ [[ 8 -gt 60 ]]
8412023-12-07T05:38:40.492Z+ sleep 1
8422023-12-07T05:38:41.489Z+ retry=9
8432023-12-07T05:38:41.515Z+++ zoneadm list -n
8442023-12-07T05:38:41.515Z+++ grep oxz_ntp
8452023-12-07T05:38:41.516Z++ pfexec svcs -z oxz_ntp_b9a0957f-237b-41c7-8e2a-526915db846f -Hostate oxide/ntp
8462023-12-07T05:38:41.516Z+ [[ disabled != online ]]
8472023-12-07T05:38:41.516Z+ [[ 9 -gt 60 ]]
8482023-12-07T05:38:41.516Z+ sleep 1
8492023-12-07T05:38:42.523Z+ retry=10
8502023-12-07T05:38:42.552Z+++ zoneadm list -n
8512023-12-07T05:38:42.552Z+++ grep oxz_ntp
8522023-12-07T05:38:42.552Z++ pfexec svcs -z oxz_ntp_b9a0957f-237b-41c7-8e2a-526915db846f -Hostate oxide/ntp
8532023-12-07T05:38:42.552Z+ [[ disabled != online ]]
8542023-12-07T05:38:42.552Z+ [[ 10 -gt 60 ]]
8552023-12-07T05:38:42.553Z+ sleep 1
8562023-12-07T05:38:43.539Z+ retry=11
8572023-12-07T05:38:43.570Z+++ zoneadm list -n
8582023-12-07T05:38:43.570Z+++ grep oxz_ntp
8592023-12-07T05:38:43.570Z++ pfexec svcs -z oxz_ntp_b9a0957f-237b-41c7-8e2a-526915db846f -Hostate oxide/ntp
8602023-12-07T05:38:43.571Z+ [[ disabled != online ]]
8612023-12-07T05:38:43.571Z+ [[ 11 -gt 60 ]]
8622023-12-07T05:38:43.571Z+ sleep 1
8632023-12-07T05:38:44.569Z+ retry=12
8642023-12-07T05:38:44.602Z+++ zoneadm list -n
8652023-12-07T05:38:44.603Z+++ grep oxz_ntp
8662023-12-07T05:38:44.604Z++ pfexec svcs -z oxz_ntp_b9a0957f-237b-41c7-8e2a-526915db846f -Hostate oxide/ntp
8672023-12-07T05:38:44.604Z+ [[ disabled != online ]]
8682023-12-07T05:38:44.604Z+ [[ 12 -gt 60 ]]
8692023-12-07T05:38:44.604Z+ sleep 1
8702023-12-07T05:38:45.586Z+ retry=13
8712023-12-07T05:38:45.622Z+++ zoneadm list -n
8722023-12-07T05:38:45.622Z+++ grep oxz_ntp
8732023-12-07T05:38:45.622Z++ pfexec svcs -z oxz_ntp_b9a0957f-237b-41c7-8e2a-526915db846f -Hostate oxide/ntp
8742023-12-07T05:38:45.622Z+ [[ disabled != online ]]
8752023-12-07T05:38:45.622Z+ [[ 13 -gt 60 ]]
8762023-12-07T05:38:45.622Z+ sleep 1
8772023-12-07T05:38:46.603Z+ retry=14
8782023-12-07T05:38:46.631Z+++ zoneadm list -n
8792023-12-07T05:38:46.631Z+++ grep oxz_ntp
8802023-12-07T05:38:46.631Z++ pfexec svcs -z oxz_ntp_b9a0957f-237b-41c7-8e2a-526915db846f -Hostate oxide/ntp
8812023-12-07T05:38:46.631Z+ [[ disabled != online ]]
8822023-12-07T05:38:46.631Z+ [[ 14 -gt 60 ]]
8832023-12-07T05:38:46.631Z+ sleep 1
 
8842023-12-07T05:38:47.315Zbuskin console login: Dec 7 05:38:47 buskin ip: ip: joining multicasts failed (4) on vopte0 - will use link layer broadcasts for multicast
 
8852023-12-07T05:38:47.632Z+ retry=15
8862023-12-07T05:38:47.677Z+++ zoneadm list -n
8872023-12-07T05:38:47.677Z+++ grep oxz_ntp
8882023-12-07T05:38:47.678Z++ pfexec svcs -z oxz_ntp_b9a0957f-237b-41c7-8e2a-526915db846f -Hostate oxide/ntp
8892023-12-07T05:38:47.678Z+ [[ online != online ]]
8902023-12-07T05:38:47.679Z+ echo 'Waited for chrony: 15s'
8912023-12-07T05:38:47.679Z+ export RUST_BACKTRACE=1
8922023-12-07T05:38:47.679Z+ RUST_BACKTRACE=1
8932023-12-07T05:38:47.679ZWaited for chrony: 15s
8942023-12-07T05:38:47.680Z+ export E2E_TLS_CERT IPPOOL_START IPPOOL_END
8952023-12-07T05:38:47.680Z+ ./tests/bootstrap
8962023-12-07T05:38:47.680ZUsing external DNS server at 10.151.1.164:53
8972023-12-07T05:38:47.710Z2023-12-07 05:38:48.354127148 UTC: attempting to log into API
8982023-12-07T05:39:02.706Z2023-12-07 05:39:02.919742360 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8992023-12-07T05:39:03.709Z2023-12-07 05:39:03.920936268 UTC: attempting to log into API
9002023-12-07T05:39:18.755Z2023-12-07 05:39:18.953020809 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9012023-12-07T05:39:19.740Z2023-12-07 05:39:19.953945271 UTC: attempting to log into API
9022023-12-07T05:39:34.776Z2023-12-07 05:39:34.985869739 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9032023-12-07T05:39:35.773Z2023-12-07 05:39:35.986950225 UTC: attempting to log into API
9042023-12-07T05:39:50.809Z2023-12-07 05:39:51.019111898 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9052023-12-07T05:39:51.814Z2023-12-07 05:39:52.019983348 UTC: attempting to log into API
9062023-12-07T05:40:06.839Z2023-12-07 05:40:06.430724914 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9072023-12-07T05:40:07.862Z2023-12-07 05:40:07.434543047 UTC: attempting to log into API
9082023-12-07T05:40:22.876Z2023-12-07 05:40:22.508074969 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9092023-12-07T05:40:23.874Z2023-12-07 05:40:23.511819006 UTC: attempting to log into API
9102023-12-07T05:40:38.912Z2023-12-07 05:40:38.586116599 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9112023-12-07T05:40:39.914Z2023-12-07 05:40:39.589627553 UTC: attempting to log into API
9122023-12-07T05:40:54.946Z2023-12-07 05:40:54.661978660 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9132023-12-07T05:40:56.024Z2023-12-07 05:40:55.665526900 UTC: attempting to log into API
9142023-12-07T05:41:10.974Z2023-12-07 05:41:10.738664565 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9152023-12-07T05:41:11.975Z2023-12-07 05:41:11.743290177 UTC: attempting to log into API
9162023-12-07T05:41:27.010Z2023-12-07 05:41:26.817054990 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9172023-12-07T05:41:28.054Z2023-12-07 05:41:27.821559685 UTC: attempting to log into API
9182023-12-07T05:41:43.049Z2023-12-07 05:41:42.895202727 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9192023-12-07T05:41:44.044Z2023-12-07 05:41:43.899005552 UTC: attempting to log into API
9202023-12-07T05:41:59.078Z2023-12-07 05:41:58.971044613 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9212023-12-07T05:42:00.077Z2023-12-07 05:41:59.974801745 UTC: attempting to log into API
9222023-12-07T05:42:15.110Z2023-12-07 05:42:15.048531580 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9232023-12-07T05:42:16.111Z2023-12-07 05:42:16.052192141 UTC: attempting to log into API
9242023-12-07T05:42:31.166Z2023-12-07 05:42:31.125088250 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9252023-12-07T05:42:32.148Z2023-12-07 05:42:32.128943189 UTC: attempting to log into API
9262023-12-07T05:42:47.186Z2023-12-07 05:42:47.164600592 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9272023-12-07T05:42:48.178Z2023-12-07 05:42:48.165346500 UTC: attempting to log into API
9282023-12-07T05:43:03.222Z2023-12-07 05:43:03.199421916 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9292023-12-07T05:43:04.221Z2023-12-07 05:43:04.200485161 UTC: attempting to log into API
9302023-12-07T05:43:19.247Z2023-12-07 05:43:19.234359111 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9312023-12-07T05:43:20.249Z2023-12-07 05:43:20.236246436 UTC: attempting to log into API
9322023-12-07T05:43:35.432Z2023-12-07 05:43:35.363775939 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9332023-12-07T05:43:36.382Z2023-12-07 05:43:36.364986077 UTC: attempting to log into API
 
9342023-12-07T05:43:38.460ZDec 7 05:43:38 buskin ip: ip: joining multicasts failed (4) on vopte2 - will use link layer broadcasts for multicast
9352023-12-07T05:43:46.115ZDec 7 05:43:45 buskin ip: ip: joining multicasts failed (4) on vopte4 - will use link layer broadcasts for multicast
9362023-12-07T05:43:51.277ZDec 7 05:43:50 buskin ip: ip: joining multicasts failed (4) on vopte3 - will use link layer broadcasts for multicast
 
9372023-12-07T05:43:51.411Z2023-12-07 05:43:51.398675863 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9382023-12-07T05:43:52.412Z2023-12-07 05:43:52.399766308 UTC: attempting to log into API
 
9392023-12-07T05:43:58.444ZDec 7 05:43:58 buskin ip: ip: joining multicasts failed (4) on vopte5 - will use link layer broadcasts for multicast
9402023-12-07T05:44:05.603ZDec 7 05:44:05 bukin ip: ip: joining multicasts failed (4) on vopte1 - will use link layer broadcasts for multicast
 
9412023-12-07T05:44:07.446Z2023-12-07 05:44:07.433060894 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9422023-12-07T05:44:08.447Z2023-12-07 05:44:08.434203652 UTC: attempting to log into API
9432023-12-07T05:44:23.480Z2023-12-07 05:44:23.467495392 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9442023-12-07T05:44:24.481Z2023-12-07 05:44:24.468551407 UTC: attempting to log into API
9452023-12-07T05:44:25.698Z2023-12-07 05:44:25.271205794 UTC: login succeeded
9462023-12-07T05:44:25.723Zwaiting for nexus to come up...
9472023-12-07T05:44:25.724Zcreating IP pool... 10.151.1.175 - 10.151.1.227
9482023-12-07T05:44:25.724Zensuring datasets are ready...
9492023-12-07T05:44:29.287Zlet's roll.
9502023-12-07T05:44:29.313Z+ rm ./tests/bootstrap
9512023-12-07T05:44:29.341Z+ for test_bin in tests/*
9522023-12-07T05:44:29.341Z+ ./tests/bootstrap-be9c817b94af4e8a
9532023-12-07T05:44:29.341Z
9542023-12-07T05:44:29.341Zrunning 0 tests
9552023-12-07T05:44:29.342Z
9562023-12-07T05:44:29.342Ztest result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
9572023-12-07T05:44:29.342Z
9582023-12-07T05:44:29.342Z+ for test_bin in tests/*
9592023-12-07T05:44:29.342Z+ ./tests/end_to_end_tests-ff923d3f5c6228d4
9602023-12-07T05:44:29.367Z
9612023-12-07T05:44:29.367Zrunning 2 tests
9622023-12-07T05:44:29.554Ztest no_spoof::no_spoof_here ... ok
9632023-12-07T05:45:29.353Ztest instance_launch::instance_launch has been running for over 60 seconds
9642023-12-07T05:47:30.527Ztest instance_launch::instance_launch ... ok
9652023-12-07T05:47:30.527Z
9662023-12-07T05:47:30.556Ztest result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 181.18s
9672023-12-07T05:47:30.557Z
9682023-12-07T05:47:30.557Z+ _exit_trap
9692023-12-07T05:47:30.557Z+ local status=0
9702023-12-07T05:47:30.557Z+ [[ 0 -eq 0 ]]
9712023-12-07T05:47:30.557Z+ exit 0
9722023-12-07T05:47:30.557Zprocess exited: duration 602666 ms, exit code 0
 
9732023-12-07T05:47:30.665Zfound 90 output files
9742023-12-07T05:47:30.665Zuploading: /var/svc/log/oxide-sled-agent:default.log (720599 bytes)
9752023-12-07T05:47:31.959Zuploaded: /var/svc/log/oxide-sled-agent:default.log
9762023-12-07T05:47:31.960Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_88b78c70-a8bf-423f-ba5a-1131ec9728b5/root/var/svc/log/oxide-cockroachdb:default.log (0 bytes)
9772023-12-07T05:47:33.084Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_88b78c70-a8bf-423f-ba5a-1131ec9728b5/root/var/svc/log/oxide-cockroachdb:default.log
9782023-12-07T05:47:33.084Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_88b78c70-a8bf-423f-ba5a-1131ec9728b5/root/var/svc/log/oxide-cockroachdb:default.log.0 (5369 bytes)
9792023-12-07T05:47:34.183Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_88b78c70-a8bf-423f-ba5a-1131ec9728b5/root/var/svc/log/oxide-cockroachdb:default.log.0
9802023-12-07T05:47:34.184Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_6ca23218-fa9e-45df-865e-d8ef539ee836/root/var/svc/log/oxide-external_dns:default.log (11327 bytes)
9812023-12-07T05:47:35.309Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_6ca23218-fa9e-45df-865e-d8ef539ee836/root/var/svc/log/oxide-external_dns:default.log
9822023-12-07T05:47:35.309Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_6ca23218-fa9e-45df-865e-d8ef539ee836/root/var/svc/log/oxide-external_dns:default.log.0 (4278 bytes)
9832023-12-07T05:47:36.414Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_6ca23218-fa9e-45df-865e-d8ef539ee836/root/var/svc/log/oxide-external_dns:default.log.0
9842023-12-07T05:47:36.414Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_9662baa0-af1b-4748-b7a3-2756377245b6/root/var/svc/log/oxide-nexus:default.log (347825 bytes)
9852023-12-07T05:47:37.834Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_9662baa0-af1b-4748-b7a3-2756377245b6/root/var/svc/log/oxide-nexus:default.log
9862023-12-07T05:47:37.835Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_9662baa0-af1b-4748-b7a3-2756377245b6/root/var/svc/log/oxide-nexus:default.log.0 (212534 bytes)
9872023-12-07T05:47:39.165Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_9662baa0-af1b-4748-b7a3-2756377245b6/root/var/svc/log/oxide-nexus:default.log.0
9882023-12-07T05:47:39.190Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_372e49ef-af27-48f5-b905-60b06d369a97/root/var/svc/log/oxide-crucible-agent:default.log (4323 bytes)
9892023-12-07T05:47:40.272Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_372e49ef-af27-48f5-b905-60b06d369a97/root/var/svc/log/oxide-crucible-agent:default.log
9902023-12-07T05:47:40.298Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_372e49ef-af27-48f5-b905-60b06d369a97/root/var/svc/log/oxide-crucible-agent:default.log.0 (9792 bytes)
9912023-12-07T05:47:41.396Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_372e49ef-af27-48f5-b905-60b06d369a97/root/var/svc/log/oxide-crucible-agent:default.log.0
9922023-12-07T05:47:41.396Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_372e49ef-af27-48f5-b905-60b06d369a97/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
9932023-12-07T05:47:42.483Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_372e49ef-af27-48f5-b905-60b06d369a97/root/var/svc/log/oxide-crucible-downstairs:default.log
9942023-12-07T05:47:42.483Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_372e49ef-af27-48f5-b905-60b06d369a97/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
9952023-12-07T05:47:43.583Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_372e49ef-af27-48f5-b905-60b06d369a97/root/var/svc/log/oxide-crucible-downstairs:default.log.0
9962023-12-07T05:47:43.583Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_372e49ef-af27-48f5-b905-60b06d369a97/root/var/svc/log/oxide-crucible-downstairs:downstairs-0ac784aa-d620-4839-a175-be66d89d9865.log (1005 bytes)
9972023-12-07T05:47:44.683Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_372e49ef-af27-48f5-b905-60b06d369a97/root/var/svc/log/oxide-crucible-downstairs:downstairs-0ac784aa-d620-4839-a175-be66d89d9865.log
9982023-12-07T05:47:44.683Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_372e49ef-af27-48f5-b905-60b06d369a97/root/var/svc/log/oxide-crucible-downstairs:downstairs-0ac784aa-d620-4839-a175-be66d89d9865.log.0 (4528 bytes)
9992023-12-07T05:47:45.783Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_372e49ef-af27-48f5-b905-60b06d369a97/root/var/svc/log/oxide-crucible-downstairs:downstairs-0ac784aa-d620-4839-a175-be66d89d9865.log.0
10002023-12-07T05:47:45.783Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_d15af560-6d9f-40d1-825d-f1322080019a/root/var/svc/log/oxide-crucible-agent:default.log (0 bytes)
10012023-12-07T05:47:46.860Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_d15af560-6d9f-40d1-825d-f1322080019a/root/var/svc/log/oxide-crucible-agent:default.log
10022023-12-07T05:47:46.860Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_d15af560-6d9f-40d1-825d-f1322080019a/root/var/svc/log/oxide-crucible-agent:default.log.0 (3436 bytes)
10032023-12-07T05:47:47.961Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_d15af560-6d9f-40d1-825d-f1322080019a/root/var/svc/log/oxide-crucible-agent:default.log.0
10042023-12-07T05:47:47.961Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_d15af560-6d9f-40d1-825d-f1322080019a/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
10052023-12-07T05:47:49.040Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_d15af560-6d9f-40d1-825d-f1322080019a/root/var/svc/log/oxide-crucible-downstairs:default.log
10062023-12-07T05:47:49.040Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_d15af560-6d9f-40d1-825d-f1322080019a/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
10072023-12-07T05:47:50.172Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_d15af560-6d9f-40d1-825d-f1322080019a/root/var/svc/log/oxide-crucible-downstairs:default.log.0
10082023-12-07T05:47:50.172Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_f992ba59-2f58-4b79-9056-a354ec18772d/root/var/svc/log/oxide-crucible-agent:default.log (0 bytes)
10092023-12-07T05:47:51.246Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_f992ba59-2f58-4b79-9056-a354ec18772d/root/var/svc/log/oxide-crucible-agent:default.log
10102023-12-07T05:47:51.246Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_f992ba59-2f58-4b79-9056-a354ec18772d/root/var/svc/log/oxide-crucible-agent:default.log.0 (14581 bytes)
10112023-12-07T05:47:52.399Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_f992ba59-2f58-4b79-9056-a354ec18772d/root/var/svc/log/oxide-crucible-agent:default.log.0
10122023-12-07T05:47:52.399Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_f992ba59-2f58-4b79-9056-a354ec18772d/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
10132023-12-07T05:47:53.474Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_f992ba59-2f58-4b79-9056-a354ec18772d/root/var/svc/log/oxide-crucible-downstairs:default.log
10142023-12-07T05:47:53.474Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_f992ba59-2f58-4b79-9056-a354ec18772d/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
10152023-12-07T05:47:54.574Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_f992ba59-2f58-4b79-9056-a354ec18772d/root/var/svc/log/oxide-crucible-downstairs:default.log.0
10162023-12-07T05:47:54.605Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_f992ba59-2f58-4b79-9056-a354ec18772d/root/var/svc/log/oxide-crucible-downstairs:downstairs-b5c094f9-f2b7-4073-b752-1a426b40f7a3.log (0 bytes)
10172023-12-07T05:47:56.711Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_f992ba59-2f58-4b79-9056-a354ec18772d/root/var/svc/log/oxide-crucible-downstairs:downstairs-b5c094f9-f2b7-4073-b752-1a426b40f7a3.log
10182023-12-07T05:47:56.711Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_f992ba59-2f58-4b79-9056-a354ec18772d/root/var/svc/log/oxide-crucible-downstairs:downstairs-b5c094f9-f2b7-4073-b752-1a426b40f7a3.log.0 (3050 bytes)
10192023-12-07T05:47:56.776Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_f992ba59-2f58-4b79-9056-a354ec18772d/root/var/svc/log/oxide-crucible-downstairs:downstairs-b5c094f9-f2b7-4073-b752-1a426b40f7a3.log.0
10202023-12-07T05:47:56.777Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_faf821dd-dc01-4af1-a09c-01ca53d934a3/root/var/svc/log/oxide-crucible-agent:default.log (0 bytes)
10212023-12-07T05:47:57.861Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_faf821dd-dc01-4af1-a09c-01ca53d934a3/root/var/svc/log/oxide-crucible-agent:default.log
10222023-12-07T05:47:57.861Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_faf821dd-dc01-4af1-a09c-01ca53d934a3/root/var/svc/log/oxide-crucible-agent:default.log.0 (14121 bytes)
10232023-12-07T05:47:58.981Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_faf821dd-dc01-4af1-a09c-01ca53d934a3/root/var/svc/log/oxide-crucible-agent:default.log.0
10242023-12-07T05:47:58.981Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_faf821dd-dc01-4af1-a09c-01ca53d934a3/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
10252023-12-07T05:48:00.063Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_faf821dd-dc01-4af1-a09c-01ca53d934a3/root/var/svc/log/oxide-crucible-downstairs:default.log
10262023-12-07T05:48:00.063Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_faf821dd-dc01-4af1-a09c-01ca53d934a3/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
10272023-12-07T05:48:01.161Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_faf821dd-dc01-4af1-a09c-01ca53d934a3/root/var/svc/log/oxide-crucible-downstairs:default.log.0
10282023-12-07T05:48:01.161Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_faf821dd-dc01-4af1-a09c-01ca53d934a3/root/var/svc/log/oxide-crucible-downstairs:downstairs-aa044fb6-7cd7-415d-a251-899a4bc7bc2e.log (0 bytes)
10292023-12-07T05:48:02.237Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_faf821dd-dc01-4af1-a09c-01ca53d934a3/root/var/svc/log/oxide-crucible-downstairs:downstairs-aa044fb6-7cd7-415d-a251-899a4bc7bc2e.log
10302023-12-07T05:48:02.237Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_faf821dd-dc01-4af1-a09c-01ca53d934a3/root/var/svc/log/oxide-crucible-downstairs:downstairs-aa044fb6-7cd7-415d-a251-899a4bc7bc2e.log.0 (3050 bytes)
10312023-12-07T05:48:03.751Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_faf821dd-dc01-4af1-a09c-01ca53d934a3/root/var/svc/log/oxide-crucible-downstairs:downstairs-aa044fb6-7cd7-415d-a251-899a4bc7bc2e.log.0
10322023-12-07T05:48:03.752Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_ntp_b9a0957f-237b-41c7-8e2a-526915db846f/root/var/svc/log/oxide-ntp:default.log (0 bytes)
10332023-12-07T05:48:04.825Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_ntp_b9a0957f-237b-41c7-8e2a-526915db846f/root/var/svc/log/oxide-ntp:default.log
10342023-12-07T05:48:04.825Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_ntp_b9a0957f-237b-41c7-8e2a-526915db846f/root/var/svc/log/oxide-ntp:default.log.0 (1182 bytes)
10352023-12-07T05:48:05.923Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_ntp_b9a0957f-237b-41c7-8e2a-526915db846f/root/var/svc/log/oxide-ntp:default.log.0
10362023-12-07T05:48:05.924Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_clickhouse_1dfaf60b-ef64-4814-b755-d3c94253083f/root/var/svc/log/oxide-clickhouse:default.log (341791 bytes)
10372023-12-07T05:48:06.178Zupload warning: file "/pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_clickhouse_1dfaf60b-ef64-4814-b755-d3c94253083f/root/var/svc/log/oxide-clickhouse:default.log" changed size mid upload: 341791 -> 405502
10382023-12-07T05:48:07.252Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_clickhouse_1dfaf60b-ef64-4814-b755-d3c94253083f/root/var/svc/log/oxide-clickhouse:default.log
10392023-12-07T05:48:07.252Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_clickhouse_1dfaf60b-ef64-4814-b755-d3c94253083f/root/var/svc/log/oxide-clickhouse:default.log.0 (212239 bytes)
10402023-12-07T05:48:08.534Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_clickhouse_1dfaf60b-ef64-4814-b755-d3c94253083f/root/var/svc/log/oxide-clickhouse:default.log.0
10412023-12-07T05:48:08.534Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_152b19a5-bee0-43e8-92d7-a89842ecc8cb/root/var/svc/log/oxide-crucible-agent:default.log (0 bytes)
10422023-12-07T05:48:09.608Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_152b19a5-bee0-43e8-92d7-a89842ecc8cb/root/var/svc/log/oxide-crucible-agent:default.log
10432023-12-07T05:48:09.608Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_152b19a5-bee0-43e8-92d7-a89842ecc8cb/root/var/svc/log/oxide-crucible-agent:default.log.0 (14113 bytes)
10442023-12-07T05:48:10.733Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_152b19a5-bee0-43e8-92d7-a89842ecc8cb/root/var/svc/log/oxide-crucible-agent:default.log.0
10452023-12-07T05:48:10.733Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_152b19a5-bee0-43e8-92d7-a89842ecc8cb/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
10462023-12-07T05:48:11.807Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_152b19a5-bee0-43e8-92d7-a89842ecc8cb/root/var/svc/log/oxide-crucible-downstairs:default.log
10472023-12-07T05:48:11.807Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_152b19a5-bee0-43e8-92d7-a89842ecc8cb/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
10482023-12-07T05:48:12.958Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_152b19a5-bee0-43e8-92d7-a89842ecc8cb/root/var/svc/log/oxide-crucible-downstairs:default.log.0
10492023-12-07T05:48:12.958Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_152b19a5-bee0-43e8-92d7-a89842ecc8cb/root/var/svc/log/oxide-crucible-downstairs:downstairs-abf90425-0c19-4e74-8be9-ad5537ec40ce.log (0 bytes)
10502023-12-07T05:48:14.032Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_152b19a5-bee0-43e8-92d7-a89842ecc8cb/root/var/svc/log/oxide-crucible-downstairs:downstairs-abf90425-0c19-4e74-8be9-ad5537ec40ce.log
10512023-12-07T05:48:14.032Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_152b19a5-bee0-43e8-92d7-a89842ecc8cb/root/var/svc/log/oxide-crucible-downstairs:downstairs-abf90425-0c19-4e74-8be9-ad5537ec40ce.log.0 (3051 bytes)
10522023-12-07T05:48:15.133Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_152b19a5-bee0-43e8-92d7-a89842ecc8cb/root/var/svc/log/oxide-crucible-downstairs:downstairs-abf90425-0c19-4e74-8be9-ad5537ec40ce.log.0
10532023-12-07T05:48:15.160Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_b297fb51-17bd-4cb8-8e07-6e86c7f5115e/root/var/svc/log/oxide-crucible-agent:default.log (0 bytes)
10542023-12-07T05:48:16.207Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_b297fb51-17bd-4cb8-8e07-6e86c7f5115e/root/var/svc/log/oxide-crucible-agent:default.log
10552023-12-07T05:48:16.207Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_b297fb51-17bd-4cb8-8e07-6e86c7f5115e/root/var/svc/log/oxide-crucible-agent:default.log.0 (3436 bytes)
10562023-12-07T05:48:17.307Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_b297fb51-17bd-4cb8-8e07-6e86c7f5115e/root/var/svc/log/oxide-crucible-agent:default.log.0
10572023-12-07T05:48:17.309Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_b297fb51-17bd-4cb8-8e07-6e86c7f5115e/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
10582023-12-07T05:48:18.386Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_b297fb51-17bd-4cb8-8e07-6e86c7f5115e/root/var/svc/log/oxide-crucible-downstairs:default.log
10592023-12-07T05:48:18.386Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_b297fb51-17bd-4cb8-8e07-6e86c7f5115e/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
10602023-12-07T05:48:19.489Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_b297fb51-17bd-4cb8-8e07-6e86c7f5115e/root/var/svc/log/oxide-crucible-downstairs:default.log.0
10612023-12-07T05:48:19.489Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_internal_dns_10f15f5f-fbfa-48cc-b92a-4af9261609ca/root/var/svc/log/oxide-internal_dns:default.log (11262 bytes)
10622023-12-07T05:48:19.570Zupload warning: file "/pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_internal_dns_10f15f5f-fbfa-48cc-b92a-4af9261609ca/root/var/svc/log/oxide-internal_dns:default.log" changed size mid upload: 11262 -> 15017
10632023-12-07T05:48:20.645Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_internal_dns_10f15f5f-fbfa-48cc-b92a-4af9261609ca/root/var/svc/log/oxide-internal_dns:default.log
10642023-12-07T05:48:20.645Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_internal_dns_10f15f5f-fbfa-48cc-b92a-4af9261609ca/root/var/svc/log/oxide-internal_dns:default.log.0 (102924 bytes)
10652023-12-07T05:48:21.861Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_internal_dns_10f15f5f-fbfa-48cc-b92a-4af9261609ca/root/var/svc/log/oxide-internal_dns:default.log.0
10662023-12-07T05:48:21.861Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_internal_dns_f8983fb8-70bc-42fc-8916-99f37d54c0ea/root/var/svc/log/oxide-internal_dns:default.log (11256 bytes)
10672023-12-07T05:48:21.933Zupload warning: file "/pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_internal_dns_f8983fb8-70bc-42fc-8916-99f37d54c0ea/root/var/svc/log/oxide-internal_dns:default.log" changed size mid upload: 11256 -> 15012
10682023-12-07T05:48:23.009Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_internal_dns_f8983fb8-70bc-42fc-8916-99f37d54c0ea/root/var/svc/log/oxide-internal_dns:default.log
10692023-12-07T05:48:23.009Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_internal_dns_f8983fb8-70bc-42fc-8916-99f37d54c0ea/root/var/svc/log/oxide-internal_dns:default.log.0 (104951 bytes)
10702023-12-07T05:48:24.351Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_internal_dns_f8983fb8-70bc-42fc-8916-99f37d54c0ea/root/var/svc/log/oxide-internal_dns:default.log.0
10712023-12-07T05:48:24.351Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_pantry_3ab38c46-f3eb-4c89-805b-304df193e43f/root/var/svc/log/oxide-crucible-pantry:default.log (0 bytes)
10722023-12-07T05:48:38.494Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_pantry_3ab38c46-f3eb-4c89-805b-304df193e43f/root/var/svc/log/oxide-crucible-pantry:default.log
10732023-12-07T05:48:38.494Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_pantry_3ab38c46-f3eb-4c89-805b-304df193e43f/root/var/svc/log/oxide-crucible-pantry:default.log.0 (2338 bytes)
10742023-12-07T05:48:39.750Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_pantry_3ab38c46-f3eb-4c89-805b-304df193e43f/root/var/svc/log/oxide-crucible-pantry:default.log.0
10752023-12-07T05:48:39.751Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_internal_dns_3f972cb4-a4b1-4964-b4cb-c0820f5679cb/root/var/svc/log/oxide-internal_dns:default.log (11259 bytes)
10762023-12-07T05:48:39.823Zupload warning: file "/pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_internal_dns_3f972cb4-a4b1-4964-b4cb-c0820f5679cb/root/var/svc/log/oxide-internal_dns:default.log" changed size mid upload: 11259 -> 15013
10772023-12-07T05:48:40.899Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_internal_dns_3f972cb4-a4b1-4964-b4cb-c0820f5679cb/root/var/svc/log/oxide-internal_dns:default.log
10782023-12-07T05:48:40.899Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_internal_dns_3f972cb4-a4b1-4964-b4cb-c0820f5679cb/root/var/svc/log/oxide-internal_dns:default.log.0 (108602 bytes)
10792023-12-07T05:48:42.126Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_internal_dns_3f972cb4-a4b1-4964-b4cb-c0820f5679cb/root/var/svc/log/oxide-internal_dns:default.log.0
10802023-12-07T05:48:42.126Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_oximeter_72c33d85-0e56-4a11-ab89-caca9e25a9f5/root/var/svc/log/oxide-oximeter:default.log (113275 bytes)
10812023-12-07T05:48:42.359Zupload warning: file "/pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_oximeter_72c33d85-0e56-4a11-ab89-caca9e25a9f5/root/var/svc/log/oxide-oximeter:default.log" changed size mid upload: 113275 -> 156508
10822023-12-07T05:48:43.454Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_oximeter_72c33d85-0e56-4a11-ab89-caca9e25a9f5/root/var/svc/log/oxide-oximeter:default.log
10832023-12-07T05:48:43.454Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_oximeter_72c33d85-0e56-4a11-ab89-caca9e25a9f5/root/var/svc/log/oxide-oximeter:default.log.0 (40242 bytes)
10842023-12-07T05:48:44.672Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_oximeter_72c33d85-0e56-4a11-ab89-caca9e25a9f5/root/var/svc/log/oxide-oximeter:default.log.0
10852023-12-07T05:48:44.675Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_cockroachdb_feef012b-d339-46ea-b67e-b97f180bf03e/root/var/svc/log/oxide-cockroachdb:default.log (0 bytes)
10862023-12-07T05:48:45.754Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_cockroachdb_feef012b-d339-46ea-b67e-b97f180bf03e/root/var/svc/log/oxide-cockroachdb:default.log
10872023-12-07T05:48:45.754Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_cockroachdb_feef012b-d339-46ea-b67e-b97f180bf03e/root/var/svc/log/oxide-cockroachdb:default.log.0 (5368 bytes)
10882023-12-07T05:48:46.861Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_cockroachdb_feef012b-d339-46ea-b67e-b97f180bf03e/root/var/svc/log/oxide-cockroachdb:default.log.0
10892023-12-07T05:48:46.861Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_18907228-9926-4813-af3e-579c7c251c82/root/var/svc/log/oxide-crucible-agent:default.log (0 bytes)
10902023-12-07T05:48:47.945Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_18907228-9926-4813-af3e-579c7c251c82/root/var/svc/log/oxide-crucible-agent:default.log
10912023-12-07T05:48:47.945Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_18907228-9926-4813-af3e-579c7c251c82/root/var/svc/log/oxide-crucible-agent:default.log.0 (3436 bytes)
10922023-12-07T05:48:49.074Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_18907228-9926-4813-af3e-579c7c251c82/root/var/svc/log/oxide-crucible-agent:default.log.0
10932023-12-07T05:48:49.075Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_18907228-9926-4813-af3e-579c7c251c82/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
10942023-12-07T05:48:50.174Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_18907228-9926-4813-af3e-579c7c251c82/root/var/svc/log/oxide-crucible-downstairs:default.log
10952023-12-07T05:48:50.175Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_18907228-9926-4813-af3e-579c7c251c82/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
10962023-12-07T05:48:51.793Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_18907228-9926-4813-af3e-579c7c251c82/root/var/svc/log/oxide-crucible-downstairs:default.log.0
10972023-12-07T05:48:51.793Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_pantry_6dce58d5-8efb-45b7-93cf-417d5e71067e/root/var/svc/log/oxide-crucible-pantry:default.log (0 bytes)
10982023-12-07T05:48:53.083Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_pantry_6dce58d5-8efb-45b7-93cf-417d5e71067e/root/var/svc/log/oxide-crucible-pantry:default.log
10992023-12-07T05:48:53.084Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_pantry_6dce58d5-8efb-45b7-93cf-417d5e71067e/root/var/svc/log/oxide-crucible-pantry:default.log.0 (2333 bytes)
11002023-12-07T05:48:54.212Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_pantry_6dce58d5-8efb-45b7-93cf-417d5e71067e/root/var/svc/log/oxide-crucible-pantry:default.log.0
11012023-12-07T05:48:54.212Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_nexus_147e496d-179a-409b-afaa-86b9bcf434bb/root/var/svc/log/oxide-nexus:default.log (360354 bytes)
11022023-12-07T05:48:55.172Zupload warning: file "/pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_nexus_147e496d-179a-409b-afaa-86b9bcf434bb/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 360354 -> 442153
11032023-12-07T05:48:56.278Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_nexus_147e496d-179a-409b-afaa-86b9bcf434bb/root/var/svc/log/oxide-nexus:default.log
11042023-12-07T05:48:56.279Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_nexus_147e496d-179a-409b-afaa-86b9bcf434bb/root/var/svc/log/oxide-nexus:default.log.0 (508148 bytes)
11052023-12-07T05:48:58.234Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_nexus_147e496d-179a-409b-afaa-86b9bcf434bb/root/var/svc/log/oxide-nexus:default.log.0
11062023-12-07T05:48:58.236Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_cockroachdb_924955aa-23e1-495c-a53e-eaeddc43ab8c/root/var/svc/log/oxide-cockroachdb:default.log (0 bytes)
11072023-12-07T05:48:59.290Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_cockroachdb_924955aa-23e1-495c-a53e-eaeddc43ab8c/root/var/svc/log/oxide-cockroachdb:default.log
11082023-12-07T05:48:59.290Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_cockroachdb_924955aa-23e1-495c-a53e-eaeddc43ab8c/root/var/svc/log/oxide-cockroachdb:default.log.0 (5369 bytes)
11092023-12-07T05:49:00.409Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_cockroachdb_924955aa-23e1-495c-a53e-eaeddc43ab8c/root/var/svc/log/oxide-cockroachdb:default.log.0
11102023-12-07T05:49:00.409Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_41b13540-12c9-4173-88a4-556b8d3929e9/root/var/svc/log/oxide-crucible-agent:default.log (4788 bytes)
11112023-12-07T05:49:01.521Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_41b13540-12c9-4173-88a4-556b8d3929e9/root/var/svc/log/oxide-crucible-agent:default.log
11122023-12-07T05:49:01.521Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_41b13540-12c9-4173-88a4-556b8d3929e9/root/var/svc/log/oxide-crucible-agent:default.log.0 (9795 bytes)
11132023-12-07T05:49:02.646Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_41b13540-12c9-4173-88a4-556b8d3929e9/root/var/svc/log/oxide-crucible-agent:default.log.0
11142023-12-07T05:49:02.671Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_41b13540-12c9-4173-88a4-556b8d3929e9/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
11152023-12-07T05:49:03.722Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_41b13540-12c9-4173-88a4-556b8d3929e9/root/var/svc/log/oxide-crucible-downstairs:default.log
11162023-12-07T05:49:03.722Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_41b13540-12c9-4173-88a4-556b8d3929e9/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
11172023-12-07T05:49:04.824Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_41b13540-12c9-4173-88a4-556b8d3929e9/root/var/svc/log/oxide-crucible-downstairs:default.log.0
11182023-12-07T05:49:04.825Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_41b13540-12c9-4173-88a4-556b8d3929e9/root/var/svc/log/oxide-crucible-downstairs:downstairs-43b9cb61-faad-4487-a983-7fb3a50070e7.log (1005 bytes)
11192023-12-07T05:49:05.929Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_41b13540-12c9-4173-88a4-556b8d3929e9/root/var/svc/log/oxide-crucible-downstairs:downstairs-43b9cb61-faad-4487-a983-7fb3a50070e7.log
11202023-12-07T05:49:05.929Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_41b13540-12c9-4173-88a4-556b8d3929e9/root/var/svc/log/oxide-crucible-downstairs:downstairs-43b9cb61-faad-4487-a983-7fb3a50070e7.log.0 (4531 bytes)
11212023-12-07T05:49:07.033Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_41b13540-12c9-4173-88a4-556b8d3929e9/root/var/svc/log/oxide-crucible-downstairs:downstairs-43b9cb61-faad-4487-a983-7fb3a50070e7.log.0
11222023-12-07T05:49:07.033Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_nexus_07453083-f90c-4424-8f4f-48811c2192e4/root/var/svc/log/oxide-nexus:default.log (1684095 bytes)
11232023-12-07T05:49:08.686Zupload warning: file "/pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_nexus_07453083-f90c-4424-8f4f-48811c2192e4/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 1684095 -> 1826167
11242023-12-07T05:49:09.789Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_nexus_07453083-f90c-4424-8f4f-48811c2192e4/root/var/svc/log/oxide-nexus:default.log
11252023-12-07T05:49:09.789Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_nexus_07453083-f90c-4424-8f4f-48811c2192e4/root/var/svc/log/oxide-nexus:default.log.0 (2817518 bytes)
11262023-12-07T05:49:12.868Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_nexus_07453083-f90c-4424-8f4f-48811c2192e4/root/var/svc/log/oxide-nexus:default.log.0
11272023-12-07T05:49:12.868Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_3ad5ba46-d825-47d4-bd18-8bbf69c06d03/root/var/svc/log/oxide-cockroachdb:default.log (0 bytes)
11282023-12-07T05:49:13.953Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_3ad5ba46-d825-47d4-bd18-8bbf69c06d03/root/var/svc/log/oxide-cockroachdb:default.log
11292023-12-07T05:49:13.954Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_3ad5ba46-d825-47d4-bd18-8bbf69c06d03/root/var/svc/log/oxide-cockroachdb:default.log.0 (5343 bytes)
11302023-12-07T05:49:15.078Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_3ad5ba46-d825-47d4-bd18-8bbf69c06d03/root/var/svc/log/oxide-cockroachdb:default.log.0
11312023-12-07T05:49:15.078Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_a071a8d7-ba84-46ad-a945-ae7b05cb64e4/root/var/svc/log/oxide-cockroachdb:default.log (0 bytes)
11322023-12-07T05:49:16.154Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_a071a8d7-ba84-46ad-a945-ae7b05cb64e4/root/var/svc/log/oxide-cockroachdb:default.log
11332023-12-07T05:49:16.154Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_a071a8d7-ba84-46ad-a945-ae7b05cb64e4/root/var/svc/log/oxide-cockroachdb:default.log.0 (5369 bytes)
11342023-12-07T05:49:17.257Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_a071a8d7-ba84-46ad-a945-ae7b05cb64e4/root/var/svc/log/oxide-cockroachdb:default.log.0
11352023-12-07T05:49:17.257Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_734fbddd-0684-40a4-9cc7-b8c92bab038d/root/var/svc/log/oxide-crucible-agent:default.log (4789 bytes)
11362023-12-07T05:49:18.359Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_734fbddd-0684-40a4-9cc7-b8c92bab038d/root/var/svc/log/oxide-crucible-agent:default.log
11372023-12-07T05:49:18.359Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_734fbddd-0684-40a4-9cc7-b8c92bab038d/root/var/svc/log/oxide-crucible-agent:default.log.0 (9796 bytes)
11382023-12-07T05:49:19.488Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_734fbddd-0684-40a4-9cc7-b8c92bab038d/root/var/svc/log/oxide-crucible-agent:default.log.0
11392023-12-07T05:49:19.489Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_734fbddd-0684-40a4-9cc7-b8c92bab038d/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
11402023-12-07T05:49:20.596Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_734fbddd-0684-40a4-9cc7-b8c92bab038d/root/var/svc/log/oxide-crucible-downstairs:default.log
11412023-12-07T05:49:20.596Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_734fbddd-0684-40a4-9cc7-b8c92bab038d/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
11422023-12-07T05:49:21.705Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_734fbddd-0684-40a4-9cc7-b8c92bab038d/root/var/svc/log/oxide-crucible-downstairs:default.log.0
11432023-12-07T05:49:21.705Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_734fbddd-0684-40a4-9cc7-b8c92bab038d/root/var/svc/log/oxide-crucible-downstairs:downstairs-f2d43a0b-ea6d-4c52-bee4-acd17d3923ea.log (1006 bytes)
11442023-12-07T05:49:22.807Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_734fbddd-0684-40a4-9cc7-b8c92bab038d/root/var/svc/log/oxide-crucible-downstairs:downstairs-f2d43a0b-ea6d-4c52-bee4-acd17d3923ea.log
11452023-12-07T05:49:22.807Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_734fbddd-0684-40a4-9cc7-b8c92bab038d/root/var/svc/log/oxide-crucible-downstairs:downstairs-f2d43a0b-ea6d-4c52-bee4-acd17d3923ea.log.0 (4530 bytes)
11462023-12-07T05:49:23.918Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_734fbddd-0684-40a4-9cc7-b8c92bab038d/root/var/svc/log/oxide-crucible-downstairs:downstairs-f2d43a0b-ea6d-4c52-bee4-acd17d3923ea.log.0
11472023-12-07T05:49:23.918Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_723fd453-6a4c-4514-a706-143398c6c124/root/var/svc/log/oxide-crucible-pantry:default.log (0 bytes)
11482023-12-07T05:49:25.001Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_723fd453-6a4c-4514-a706-143398c6c124/root/var/svc/log/oxide-crucible-pantry:default.log
11492023-12-07T05:49:25.042Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_723fd453-6a4c-4514-a706-143398c6c124/root/var/svc/log/oxide-crucible-pantry:default.log.0 (2338 bytes)
11502023-12-07T05:49:26.121Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_723fd453-6a4c-4514-a706-143398c6c124/root/var/svc/log/oxide-crucible-pantry:default.log.0
11512023-12-07T05:49:26.122Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_ad6519df-80fc-40a2-9022-da2983b428f5/root/var/svc/log/oxide-external_dns:default.log (11333 bytes)
11522023-12-07T05:49:26.216Zupload warning: file "/pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_ad6519df-80fc-40a2-9022-da2983b428f5/root/var/svc/log/oxide-external_dns:default.log" changed size mid upload: 11333 -> 18886
11532023-12-07T05:49:27.297Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_ad6519df-80fc-40a2-9022-da2983b428f5/root/var/svc/log/oxide-external_dns:default.log
11542023-12-07T05:49:27.297Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_ad6519df-80fc-40a2-9022-da2983b428f5/root/var/svc/log/oxide-external_dns:default.log.0 (4281 bytes)
11552023-12-07T05:49:28.395Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_ad6519df-80fc-40a2-9022-da2983b428f5/root/var/svc/log/oxide-external_dns:default.log.0
11562023-12-07T05:49:28.395Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_ntp_b9a0957f-237b-41c7-8e2a-526915db846f/root/var/log/chrony/measurements.log (23153 bytes)
11572023-12-07T05:49:28.494Zupload warning: file "/pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_ntp_b9a0957f-237b-41c7-8e2a-526915db846f/root/var/log/chrony/measurements.log" changed size mid upload: 23153 -> 26852
11582023-12-07T05:49:29.586Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_ntp_b9a0957f-237b-41c7-8e2a-526915db846f/root/var/log/chrony/measurements.log
11592023-12-07T05:49:29.586Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_ntp_b9a0957f-237b-41c7-8e2a-526915db846f/root/var/log/chrony/statistics.log (12987 bytes)
11602023-12-07T05:49:30.317Zupload warning: file "/pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_ntp_b9a0957f-237b-41c7-8e2a-526915db846f/root/var/log/chrony/statistics.log" changed size mid upload: 12987 -> 15444
11612023-12-07T05:49:31.818Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_ntp_b9a0957f-237b-41c7-8e2a-526915db846f/root/var/log/chrony/statistics.log
11622023-12-07T05:49:31.818Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_ntp_b9a0957f-237b-41c7-8e2a-526915db846f/root/var/log/chrony/tracking.log (2118 bytes)
11632023-12-07T05:49:31.890Zupload warning: file "/pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_ntp_b9a0957f-237b-41c7-8e2a-526915db846f/root/var/log/chrony/tracking.log" changed size mid upload: 2118 -> 2250
11642023-12-07T05:49:32.973Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_ntp_b9a0957f-237b-41c7-8e2a-526915db846f/root/var/log/chrony/tracking.log