01HGEWFY8W12J5C464BKGKYM5Y: helios / deploy

BasicConfig {
    output_rules: [
        "%/var/svc/log/oxide-sled-agent:default.log*",
        "%/pool/ext/*/crypt/zone/oxz_*/root/var/svc/log/oxide-*.log*",
        "%/pool/ext/*/crypt/zone/oxz_*/root/var/svc/log/system-illumos-*.log*",
        "%/pool/ext/*/crypt/zone/oxz_ntp_*/root/var/log/chrony/*.log*",
        "!/pool/ext/*/crypt/zone/oxz_propolis-server_*/root/var/svc/log/*.log*",
        "%/pool/ext/*/crypt/debug/global/oxide-sled-agent:default.log.*",
        "%/pool/ext/*/crypt/debug/oxz_*/oxide-*.log.*",
        "%/pool/ext/*/crypt/debug/oxz_*/system-illumos-*.log.*",
        "!/pool/ext/*/crypt/debug/oxz_propolis-server_*/*.log.*",
    ],
    rust_toolchain: None,
    target: Some(
        "lab-2.0-opte-0.25",
    ),
    access_repos: [],
    publish: [],
    skip_clone: true,
}

Buildomat Job: 01HGEWGG0C6SFDG828GXK5JMN6

Tags:

Artefacts:

Output:

SEQ GLOBAL TIME DETAILS
12023-11-30T01:58:37.797Zjob dependencies complete; ready to run (waiting for 46 m 45 s)
22023-11-30T02:00:26.711Zjob assigned to worker 01HGEZ88CHG4E46TZ071AV1EDG (queued for 1 m 48 s)
32023-11-30T02:00:26.927Zhttp://172.20.0.5:9969/ipxe/buskin... ok
42023-11-30T02:00:26.974Zhttp://172.20.0.5:9969/os/buskin/platform/i86pc/kernel/amd64/unix... ok
52023-11-30T02:00:27.008Zhttp://172.20.0.5:9969/os/buskin/platform/i86pc/amd64/boot_archive... ok
62023-11-30T02:00:27.042Zhttp://172.20.0.5:9969/os/buskin/platform/i86pc/amd64/boot_archive.hash... ok
72023-11-30T02:00:27.075Zhttp://172.20.0.5:9969/postboot/buskin... ok
82023-11-30T02:00:27.110ZwcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U
92023-11-30T02:00:27.146Zcpu0: microcode has been updated from version 0xa001133 to 0xa0011d1
102023-11-30T02:00:27.185ZOxide Helios Version helios-2.0.22318 64-bit
112023-11-30T02:00:27.221ZConfiguring devices.
122023-11-30T02:00:27.256Z#################################################
132023-11-30T02:00:27.289Z#### ############# ##
142023-11-30T02:00:27.325Z### ### ############ ##
152023-11-30T02:00:27.355Z## ### # ## ### ## Oxide Computer Company ##
162023-11-30T02:00:27.389Z## ## ## ### # ### ##
172023-11-30T02:00:27.422Z## # ### #### #### This Station Under ##
182023-11-30T02:00:27.456Z### ### #### # ### Computer Control ##
192023-11-30T02:00:27.489Z#### #### ### ## ##
202023-11-30T02:00:27.522Z##################################################
212023-11-30T02:00:27.556ZHostname: buskin
222023-11-30T02:00:27.593ZLoading smf(7) serice descriptions:
232023-11-30T02:00:28.695Zdownloading input: /input/ci-tools/work/end-to-end-tests/bootstrap-8d69322849e0fa66.gz
242023-11-30T02:00:30.229Zdownloaded input: /input/ci-tools/work/end-to-end-tests/bootstrap-8d69322849e0fa66.gz
252023-11-30T02:00:30.229Zdownloading input: /input/ci-tools/work/end-to-end-tests/bootstrap.gz
262023-11-30T02:00:51.755Zdownloaded input: /input/ci-tools/work/end-to-end-tests/bootstrap.gz
272023-11-30T02:00:51.755Zdownloading input: /input/ci-tools/work/end-to-end-tests/end_to_end_tests-7d7c590276acd289.gz
282023-11-30T02:01:13.196Zdownloaded input: /input/ci-tools/work/end-to-end-tests/end_to_end_tests-7d7c590276acd289.gz
292023-11-30T02:01:13.196Zdownloading input: /input/ci-tools/work/caboose-util.gz
302023-11-30T02:01:14.209Zdownloaded input: /input/ci-tools/work/caboose-util.gz
312023-11-30T02:01:14.233Zdownloading input: /input/ci-tools/work/tufaceous.gz
322023-11-30T02:01:17.736Zdownloaded input: /input/ci-tools/work/tufaceous.gz
332023-11-30T02:01:17.761Zdownloading input: /input/package/work/version.txt
342023-11-30T02:01:17.785Zdownloaded input: /input/package/work/version.txt
352023-11-30T02:01:17.785Zdownloading input: /input/package/work/package.tar.gz
362023-11-30T02:01:34.686Zdownloaded input: /input/package/work/package.tar.gz
372023-11-30T02:01:34.710Zdownloading input: /input/package/work/global-zone-packages.tar.gz
382023-11-30T02:02:06.792Zdownloaded input: /input/package/work/global-zone-packages.tar.gz
392023-11-30T02:02:06.792Zdownloading input: /input/package/work/trampoline-global-zone-packages.tar.gz
402023-11-30T02:02:15.284Zdownloaded input: /input/package/work/trampoline-global-zone-packages.tar.gz
412023-11-30T02:02:15.309Zdownloading input: /input/package/work/zones/clickhouse.tar.gz
422023-11-30T02:02:55.504Zdownloaded input: /input/package/work/zones/clickhouse.tar.gz
432023-11-30T02:02:55.504Zdownloading input: /input/package/work/zones/clickhouse_keeper.tar.gz
442023-11-30T02:03:29.941Zdownloaded input: /input/package/work/zones/clickhouse_keeper.tar.gz
452023-11-30T02:03:29.965Zdownloading input: /input/package/work/zones/cockroachdb.tar.gz
462023-11-30T02:03:52.742Zdownloaded input: /input/package/work/zones/cockroachdb.tar.gz
472023-11-30T02:03:52.742Zdownloading input: /input/package/work/zones/crucible-pantry.tar.gz
482023-11-30T02:03:56.958Zdownloaded input: /input/package/work/zones/crucible-pantry.tar.gz
492023-11-30T02:03:56.958Zdownloading input: /input/package/work/zones/crucible.tar.gz
502023-11-30T02:04:04.662Zdownloaded input: /input/package/work/zones/crucible.tar.gz
512023-11-30T02:04:04.662Zdownloading input: /input/package/work/zones/external-dns.tar.gz
522023-11-30T02:04:09.754Zdownloaded input: /input/package/work/zones/external-dns.tar.gz
532023-11-30T02:04:09.754Zdownloading input: /input/package/work/zones/internal-dns.tar.gz
542023-11-30T02:04:14.566Zdownloaded input: /input/package/work/zones/internal-dns.tar.gz
552023-11-30T02:04:14.590Zdownloading input: /input/package/work/zones/ntp.tar.gz
562023-11-30T02:04:14.590Zdownloaded input: /input/package/work/zones/ntp.tar.gz
572023-11-30T02:04:14.590Zdownloading input: /input/package/work/zones/omicron-gateway-asic.tar.gz
582023-11-30T02:04:18.607Zdownloaded input: /input/package/work/zones/omicron-gateway-asic.tar.gz
592023-11-30T02:04:18.607Zdownloading input: /input/package/work/zones/omicron-gateway-softnpu.tar.gz
602023-11-30T02:04:21.927Zdownloaded input: /input/package/work/zones/omicron-gateway-softnpu.tar.gz
612023-11-30T02:04:21.927Zdownloading input: /input/package/work/zones/omicron-nexus-single-sled.tar.gz
622023-11-30T02:04:36.055Zdownloaded input: /input/package/work/zones/omicron-nexus-single-sled.tar.gz
632023-11-30T02:04:36.055Zdownloading input: /input/package/work/zones/omicron-nexus.tar.gz
642023-11-30T02:04:49.742Zdownloaded input: /input/package/work/zones/omicron-nexus.tar.gz
652023-11-30T02:04:49.742Zdownloading input: /input/package/work/zones/overlay.tar.gz
662023-11-30T02:04:49.783Zdownloaded input: /input/package/work/zones/overlay.tar.gz
672023-11-30T02:04:49.783Zdownloading input: /input/package/work/zones/oximeter-collector.tar.gz
682023-11-30T02:04:55.926Zdownloaded input: /input/package/work/zones/oximeter-collector.tar.gz
692023-11-30T02:04:55.926Zdownloading input: /input/package/work/zones/propolis-server.tar.gz
702023-11-30T02:05:07.914Zdownloaded input: /input/package/work/zones/propolis-server.tar.gz
712023-11-30T02:05:07.914Zdownloading input: /input/package/work/zones/switch-asic.tar.gz
722023-11-30T02:05:53.765Zdownloaded input: /input/package/work/zones/switch-asic.tar.gz
732023-11-30T02:05:53.765Zdownloading input: /input/package/work/zones/switch-softnpu.tar.gz
742023-11-30T02:06:43.744Zdownloaded input: /input/package/work/zones/switch-softnpu.tar.gz
 
752023-11-30T02:06:43.744Zstarting task 0: "setup"
762023-11-30T02:06:43.776Z++ uname -s
772023-11-30T02:06:43.776Z+ kern=SunOS
782023-11-30T02:06:43.776Z+ case "$kern" in
792023-11-30T02:06:43.776Z+ groupadd -g 12345 build
802023-11-30T02:06:43.776Z+ useradd -u 12345 -g build -d /home/build -s /bin/bash -c build -P 'Primary Administrator' build
812023-11-30T02:06:45.758Z+ zfs create -o mountpoint=/work rpool/work
822023-11-30T02:06:45.801Z++ awk '$2 == "/home" { print $3 }' /etc/mnttab
832023-11-30T02:06:45.825Z+ home_fs=zfs
842023-11-30T02:06:45.825Z+ [[ zfs == autofs ]]
852023-11-30T02:06:45.825Z+ mkdir -p /home/build
862023-11-30T02:06:45.825Z+ chown build:build /home/build /work
872023-11-30T02:06:47.808Z+ chmod 0700 /home/build /work
882023-11-30T02:06:47.834Zprocess exited: duration 4071 ms, exit code 0
 
892023-11-30T02:06:47.882Zstarting task 1: "authentication"
902023-11-30T02:06:47.925Zprocess exited: duration 43 ms, exit code 0
 
912023-11-30T02:06:47.973Zstarting task 2: "build"
922023-11-30T02:06:47.997Z+ trap _exit_trap EXIT
932023-11-30T02:06:47.997Z+ pfexec mdb -kwe 'secpolicy_fs_linkdir/v 55 48 89 e5 b8 01 00 00 00 5d c3'
942023-11-30T02:06:48.181Zsecpolicy_fs_linkdir: 0x55 = 0x55
952023-11-30T02:06:48.181Zsecpolicy_fs_linkdir+1: 0xb8 = 0x48
962023-11-30T02:06:48.181Zsecpolicy_fs_linkdir+2: 0x1 = 0x89
972023-11-30T02:06:48.181Zsecpolicy_fs_linkdir+3: 0 = 0xe5
982023-11-30T02:06:48.181Zsecpolicy_fs_linkdir+4: 0 = 0xb8
992023-11-30T02:06:48.181Zsecpolicy_fs_linkdir+5: 0 = 0x1
1002023-11-30T02:06:48.181Zsecpolicy_fs_linkdir+6: 0x48 = 0x0
1012023-11-30T02:06:48.206Zsecpolicy_fs_linkdir+7: 0x89 = 0x0
1022023-11-30T02:06:48.206Zsecpolicy_fs_linkdir+8: 0xe5 = 0x0
1032023-11-30T02:06:48.206Zsecpolicy_fs_linkdir+9: 0x48 = 0x5d
1042023-11-30T02:06:48.206Zsecpolicy_fs_linkdir+0xa: 0x83 = 0xc3
1052023-11-30T02:06:48.206Z+ [[ -d /opt/oxide ]]
1062023-11-30T02:06:48.206Z+ pfexec mv /opt/oxide /opt/oxide-underneath
1072023-11-30T02:06:48.206Z+ pfexec mkdir /opt/oxide
1082023-11-30T02:06:48.206Z+ pfexec mount -F tmpfs -O swap /opt/oxide
1092023-11-30T02:06:48.206Z+ [[ -d /opt/oxide-underneath ]]
1102023-11-30T02:06:48.206Z+ cd /opt/oxide-underneath
1112023-11-30T02:06:48.206Z+ pfexec tar ceEp@/f - .
1122023-11-30T02:06:48.208Z+ cd /opt/oxide
1132023-11-30T02:06:48.208Z+ pfexec tar xveEp@/f -
1142023-11-30T02:06:48.302Zx ., 0 bytes, 0 tape blocks
1152023-11-30T02:06:48.302Zx ./opte, 0 bytes, 0 tape blocks
1162023-11-30T02:06:48.302Zx ./opte/bin, 0 bytes, 0 tape blocks
1172023-11-30T02:06:48.302Zx ./opte/bin/opteadm, 50714832 bytes, 99053 tape blocks
1182023-11-30T02:06:48.326Z+ pfexec rm -rf /opt/oxide-underneath
1192023-11-30T02:06:48.326Z+ pfexec /sbin/zfs create -o mountpoint=/zone rpool/zone
1202023-11-30T02:06:48.350Z+ pfexec mkdir /opt/oxide/work
1212023-11-30T02:06:48.350Z+ pfexec chown build:build /opt/oxide/work
1222023-11-30T02:06:48.350Z+ cd /opt/oxide/work
1232023-11-30T02:06:48.350Z+ ptime -m tar xvzf /input/package/work/package.tar.gz
1242023-11-30T02:06:49.890ZDecompressing '/input/package/work/package.tar.gz' with '/usr/bin/gzcat'...
1252023-11-30T02:06:49.891Zx out/mg-ddm-gz.tar, 62723072 bytes, 122506 tape blocks
1262023-11-30T02:06:49.891Zx out/omicron-sled-agent.tar, 112139264 bytes, 219022 tape blocks
1272023-11-30T02:06:49.891Zx out/target/test, 75 bytes, 1 tape blocks
1282023-11-30T02:06:49.891Zx out/npuzone/npuzone, 37380856 bytes, 73010 tape blocks
1292023-11-30T02:06:49.891Zx package-manifest.toml, 22143 bytes, 44 tape blocks
1302023-11-30T02:06:49.891Zx smf/sled-agent/non-gimlet/config.toml, 3298 bytes, 7 tape blocks
1312023-11-30T02:06:49.891Zx target/release/omicron-package, 42567616 bytes, 83140 tape blocks
1322023-11-30T02:06:49.891Zx tools/create_virtual_hardware.sh, 3176 bytes, 7 tape blocks
1332023-11-30T02:06:49.891Zx tools/virtual_hardware.sh, 3408 bytes, 7 tape blocks
1342023-11-30T02:06:49.891Zx tools/scrimlet/create-softnpu-zone.sh, 619 bytes, 2 tape blocks
1352023-11-30T02:06:49.891Zx tools/scrimlet/destroy-softnpu-zone.sh, 195 bytes, 1 tape blocks
1362023-11-30T02:06:49.897Zx tools/scrimlet/softnpu-init.sh, 2148 bytes, 5 tape blocks
1372023-11-30T02:06:49.897Zx tools/scrimlet/softnpu-zone.txt, 330 bytes, 1 tape blocks
1382023-11-30T02:06:49.897Zx tools/scrimlet/softnpu.toml, 171 bytes, 1 tape blocks
1392023-11-30T02:06:49.921Z
1402023-11-30T02:06:49.921Zreal 1.551240101
1412023-11-30T02:06:49.921Zuser 1.394132912
1422023-11-30T02:06:49.921Zsys 0.275076041
1432023-11-30T02:06:49.921Ztrap 0.000047690
1442023-11-30T02:06:49.921Ztflt 0.000000000
1452023-11-30T02:06:49.921Zdflt 0.000000000
1462023-11-30T02:06:49.921Zkflt 0.000000000
1472023-11-30T02:06:49.921Zlock 0.000000000
1482023-11-30T02:06:49.921Zslp 1.395216757
1492023-11-30T02:06:49.921Zlat 0.035856997
1502023-11-30T02:06:49.921Zstop 0.000122563
1512023-11-30T02:06:49.921Z+ cp /input/package/work/zones/clickhouse_keeper.tar.gz /input/package/work/zones/clickhouse.tar.gz /input/package/work/zones/cockroachdb.tar.gz /input/package/work/zones/crucible-pantry.tar.gz /input/package/work/zones/crucible.tar.gz /input/package/work/zones/external-dns.tar.gz /input/package/work/zones/internal-dns.tar.gz /input/package/work/zones/ntp.tar.gz /input/package/work/zones/omicron-gateway-asic.tar.gz /input/package/work/zones/omicron-gateway-softnpu.tar.gz /input/package/work/zones/omicron-nexus-single-sled.tar.gz /input/package/work/zones/omicron-nexus.tar.gz /input/package/work/zones/overlay.tar.gz /input/package/work/zones/oximeter-collector.tar.gz /input/package/work/zones/propolis-server.tar.gz /input/package/work/zones/switch-asic.tar.gz /input/package/work/zones/switch-softnpu.tar.gz out/
1522023-11-30T02:06:50.962Z+ mv out/omicron-nexus-single-sled.tar.gz out/omicron-nexus.tar.gz
1532023-11-30T02:06:50.994Z+ mkdir tests
1542023-11-30T02:06:50.994Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1552023-11-30T02:06:50.994Z+ ptime -m gunzip
1562023-11-30T02:06:50.994Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap-8d69322849e0fa66
1572023-11-30T02:06:51.121Z
1582023-11-30T02:06:51.145Zreal 0.138096873
1592023-11-30T02:06:51.145Zuser 0.127081232
1602023-11-30T02:06:51.145Zsys 0.010726121
1612023-11-30T02:06:51.145Ztrap 0.000014371
1622023-11-30T02:06:51.145Ztflt 0.000000000
1632023-11-30T02:06:51.145Zdflt 0.000000000
1642023-11-30T02:06:51.145Zkflt 0.000000000
1652023-11-30T02:06:51.146Zlock 0.000000000
1662023-11-30T02:06:51.146Zslp 0.000023880
1672023-11-30T02:06:51.146Zlat 0.000031540
1682023-11-30T02:06:51.146Zstop 0.000125503
1692023-11-30T02:06:51.146Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap-8d69322849e0fa66
1702023-11-30T02:06:51.146Z+ chmod a+x tests/bootstrap-8d69322849e0fa66
1712023-11-30T02:06:51.146Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1722023-11-30T02:06:51.146Z+ ptime -m gunzip
1732023-11-30T02:06:51.146Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap
1742023-11-30T02:06:53.330Z
1752023-11-30T02:06:53.354Zreal 2.199408606
1762023-11-30T02:06:53.354Zuser 2.040575122
1772023-11-30T02:06:53.354Zsys 0.158423112
1782023-11-30T02:06:53.354Ztrap 0.000179345
1792023-11-30T02:06:53.354Ztflt 0.000000000
1802023-11-30T02:06:53.354Zdflt 0.000000000
1812023-11-30T02:06:53.354Zkflt 0.000000000
1822023-11-30T02:06:53.354Zlock 0.000000000
1832023-11-30T02:06:53.354Zslp 0.000019920
1842023-11-30T02:06:53.354Zlat 0.000021710
1852023-11-30T02:06:53.354Zstop 0.000125103
1862023-11-30T02:06:53.354Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap
1872023-11-30T02:06:53.354Z+ chmod a+x tests/bootstrap
1882023-11-30T02:06:53.354Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1892023-11-30T02:06:53.354Z+ ptime -m gunzip
1902023-11-30T02:06:53.354Z++ basename /input/ci-tools/work/end-to-end-tests/end_to_end_tests-7d7c590276acd289
1912023-11-30T02:06:55.640Z
1922023-11-30T02:06:55.665Zreal 2.299528056
1932023-11-30T02:06:55.665Zuser 2.110258588
1942023-11-30T02:06:55.665Zsys 0.188898237
1952023-11-30T02:06:55.665Ztrap 0.000155446
1962023-11-30T02:06:55.665Ztflt 0.000000000
1972023-11-30T02:06:55.665Zdflt 0.000000000
1982023-11-30T02:06:55.665Zkflt 0.000000000
1992023-11-30T02:06:55.665Zlock 0.000000000
2002023-11-30T02:06:55.665Zslp 0.000022970
2012023-11-30T02:06:55.665Zlat 0.000017669
2022023-11-30T02:06:55.665Zstop 0.000111693
2032023-11-30T02:06:55.665Z++ basename /input/ci-tools/work/end-to-end-tests/end_to_end_tests-7d7c590276acd289
2042023-11-30T02:06:55.665Z+ chmod a+x tests/end_to_end_tests-7d7c590276acd289
2052023-11-30T02:06:55.665Z+ bmat address ls
2062023-11-30T02:06:55.665ZNAME CIDR FIRST LAST COUNT
2072023-11-30T02:06:55.665Zextra 10.151.1.0/24 10.151.1.164 10.151.1.227 64
2082023-11-30T02:06:55.665Z++ bmat address ls -f extra -Ho first,last,count
2092023-11-30T02:06:55.689Z+ set -- 10.151.1.164 10.151.1.227 64
2102023-11-30T02:06:55.689Z+ EXTRA_IP_START=10.151.1.164
2112023-11-30T02:06:55.689Z+ EXTRA_IP_END=10.151.1.227
2122023-11-30T02:06:55.689Z+ EXTRA_IP_COUNT=64
2132023-11-30T02:06:55.689Z+ (( EXTRA_IP_COUNT >= 32 ))
2142023-11-30T02:06:55.689Z+ EXTRA_IP_BASE=10.151.1
2152023-11-30T02:06:55.689Z+ EXTRA_IP_FOCTET=164
2162023-11-30T02:06:55.690Z+ EXTRA_IP_LOCTET=227
2172023-11-30T02:06:55.690Z+ SERVICE_IP_POOL_START=10.151.1.164
2182023-11-30T02:06:55.690Z+ SERVICE_IP_POOL_END=10.151.1.173
2192023-11-30T02:06:55.690Z+ DNS_IP1=10.151.1.164
2202023-11-30T02:06:55.690Z+ DNS_IP2=10.151.1.165
2212023-11-30T02:06:55.690Z+ UPLINK_IP=10.151.1.174
2222023-11-30T02:06:55.690Z+ IPPOOL_START=10.151.1.175
2232023-11-30T02:06:55.690Z+ IPPOOL_END=10.151.1.227
2242023-11-30T02:06:55.690Z++ ipadm show-addr -po type,addr
2252023-11-30T02:06:55.690Z++ awk '-F[:/]' '$1 == "dhcp" {print $2}'
2262023-11-30T02:06:55.690Z+ GATEWAY_IP=10.151.1.100
2272023-11-30T02:06:55.690Z+ [[ -n 10.151.1.100 ]]
2282023-11-30T02:06:55.690Z+ ping -s 10.151.1.100 56 1
2292023-11-30T02:06:55.690ZPING 10.151.1.100: 56 data bytes
2302023-11-30T02:06:55.691Z64 bytes from 10.151.1.100: icmp_seq=0. time=0.011 ms
2312023-11-30T02:06:55.691Z
2322023-11-30T02:06:55.691Z----10.151.1.100 PING Statistics----
2332023-11-30T02:06:55.691Z1 packets transmitted, 1 packets received, 0% packet loss
2342023-11-30T02:06:55.691Zround-trip (ms) min/avg/max/stddev = 0.011/0.011/0.011/-nan
2352023-11-30T02:06:55.691Z++ arp -an
2362023-11-30T02:06:55.715Z++ awk -vgw=10.151.1.100 '$2 == gw {print $NF}'
2372023-11-30T02:06:55.715Z+ GATEWAY_MAC=18:c0:4d:0d:9f:b2
2382023-11-30T02:06:55.715Z+ routeadm -e ipv4-forwarding -u
2392023-11-30T02:06:55.808Z+ PXA_START=10.151.1.164
2402023-11-30T02:06:55.808Z+ PXA_END=10.151.1.227
2412023-11-30T02:06:55.808Z+ export GATEWAY_IP GATEWAY_MAC PXA_START PXA_END
2422023-11-30T02:06:55.832Z+ pfexec zpool create -f scratch c1t1d0 c2t1d0
2432023-11-30T02:06:55.943Z+ ZPOOL_VDEV_DIR=/scratch
2442023-11-30T02:06:55.943Z+ ptime -m pfexec ./tools/create_virtual_hardware.sh
2452023-11-30T02:06:55.969Z+++ dirname ./tools/create_virtual_hardware.sh
2462023-11-30T02:06:55.969Z++ cd ./tools
2472023-11-30T02:06:55.969Z++ pwd
2482023-11-30T02:06:55.969Z+ SOURCE_DIR=/opt/oxide/work/tools
2492023-11-30T02:06:55.969Z+ OMICRON_TOP=/opt/oxide/work/tools/..
2502023-11-30T02:06:55.969Z+ SOFTNPU_MODE=zone
2512023-11-30T02:06:55.969Z+ . /opt/oxide/work/tools/virtual_hardware.sh
2522023-11-30T02:06:55.969Z++ MARKER=/etc/opt/oxide/NO_INSTALL
2532023-11-30T02:06:55.969Z++ [[ -f /etc/opt/oxide/NO_INSTALL ]]
2542023-11-30T02:06:55.969Z++ dladm show-phys -p -o LINK
2552023-11-30T02:06:55.969Z++ head -1
2562023-11-30T02:06:56.020Z+ PHYSICAL_LINK=igb0
2572023-11-30T02:06:56.020Z+ echo 'Using igb0 as physical link'
2582023-11-30T02:06:56.048ZUsing igb0 as physical link
2592023-11-30T02:06:56.048Z+ ensure_run_as_root
2602023-11-30T02:06:56.048Z++ id -u
2612023-11-30T02:06:56.048Z+ [[ 0 -ne 0 ]]
2622023-11-30T02:06:56.048Z+ ensure_zpools
2632023-11-30T02:06:56.048Z+ ZPOOL_TYPES=('oxp_' 'oxi_')
2642023-11-30T02:06:56.048Z+ for ZPOOL_TYPE in "${ZPOOL_TYPES[@]}"
2652023-11-30T02:06:56.048Z+ readarray -t ZPOOLS
2662023-11-30T02:06:56.048Z++ grep '"oxp_' /opt/oxide/work/tools/../smf/sled-agent/non-gimlet/config.toml
2672023-11-30T02:06:56.048Z++ sed 's/[ ",]//g'
2682023-11-30T02:06:56.048Z+ for ZPOOL in "${ZPOOLS[@]}"
2692023-11-30T02:06:56.049Z+ echo 'Zpool: [oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b]'
2702023-11-30T02:06:56.049ZZpool: [oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b]
2712023-11-30T02:06:56.050Z+ VDEV_PATH=/scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
2722023-11-30T02:06:56.050Z+ [[ -f /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
2732023-11-30T02:06:56.050Z+ dd if=/dev/zero of=/scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
2742023-11-30T02:06:56.074Z0+0 records in
2752023-11-30T02:06:56.074Z0+0 records out
2762023-11-30T02:06:56.074Z0 bytes (0 iB) transferred in 0.000006 secs (0 iB/sec)
2772023-11-30T02:06:56.074Z+ success 'ZFS vdev /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
2782023-11-30T02:06:56.074Z+ set +x
2792023-11-30T02:06:56.074ZZFS vdev /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
2802023-11-30T02:06:56.074Z++ zpool list -o name
2812023-11-30T02:06:56.074Z++ grep oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b
2822023-11-30T02:06:56.074Z+ [[ -z '' ]]
2832023-11-30T02:06:56.074Z+ zpool create -f oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
2842023-11-30T02:06:56.109Z+ success 'ZFS zpool oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
2852023-11-30T02:06:56.133Z+ set +x
2862023-11-30T02:06:56.133ZZFS zpool oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b exists
2872023-11-30T02:06:56.133Z+ for ZPOOL in "${ZPOOLS[@]}"
2882023-11-30T02:06:56.133Z+ echo 'Zpool: [oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03]'
2892023-11-30T02:06:56.133ZZpool: [oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03]
2902023-11-30T02:06:56.133Z+ VDEV_PATH=/scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
2912023-11-30T02:06:56.133Z+ [[ -f /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
2922023-11-30T02:06:56.133Z+ dd if=/dev/zero of=/scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
2932023-11-30T02:06:56.133Z0+0 records in
2942023-11-30T02:06:56.134Z0+0 records out
2952023-11-30T02:06:56.134Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
2962023-11-30T02:06:56.134Z+ success 'ZFS vdev /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
2972023-11-30T02:06:56.134Z+ set +x
2982023-11-30T02:06:56.134ZZFS vdev /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
2992023-11-30T02:06:56.134Z++ zpool list -o name
3002023-11-30T02:06:56.134Z++ grep oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03
3012023-11-30T02:06:56.134Z+ [[ -z '' ]]
3022023-11-30T02:06:56.134Z+ zpool create -f oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3032023-11-30T02:06:56.158Z+ success 'ZFS zpool oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3042023-11-30T02:06:56.158Z+ set +x
3052023-11-30T02:06:56.158ZZFS zpool oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
3062023-11-30T02:06:56.158Z+ for ZPOOL in "${ZPOOLS[@]}"
3072023-11-30T02:06:56.158Z+ echo 'Zpool: [oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3082023-11-30T02:06:56.158Z+ VDEV_PATH=/scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3092023-11-30T02:06:56.158ZZpool: [oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03]
3102023-11-30T02:06:56.158Z+ [[ -f /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3112023-11-30T02:06:56.158Z+ dd if=/dev/zero of=/scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3122023-11-30T02:06:56.158Z0+0 records in
3132023-11-30T02:06:56.158Z0+0 records out
3142023-11-30T02:06:56.158Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3152023-11-30T02:06:56.158Z+ success 'ZFS vdev /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3162023-11-30T02:06:56.158Z+ set +x
3172023-11-30T02:06:56.158ZZFS vdev /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
3182023-11-30T02:06:56.159Z++ zpool list -o name
3192023-11-30T02:06:56.159Z++ grep oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03
3202023-11-30T02:06:56.183Z+ [[ -z '' ]]
3212023-11-30T02:06:56.183Z+ zpool create -f oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3222023-11-30T02:06:56.207Z+ success 'ZFS zpool oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3232023-11-30T02:06:56.207Z+ set +x
3242023-11-30T02:06:56.207ZZFS zpool oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
3252023-11-30T02:06:56.207Z+ for ZPOOL in "${ZPOOLS[@]}"
3262023-11-30T02:06:56.207Z+ echo 'Zpool: [oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3272023-11-30T02:06:56.207ZZpool: [oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03]
3282023-11-30T02:06:56.207Z+ VDEV_PATH=/scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3292023-11-30T02:06:56.207Z+ [[ -f /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3302023-11-30T02:06:56.207Z+ dd if=/dev/zero of=/scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3312023-11-30T02:06:56.207Z0+0 records in
3322023-11-30T02:06:56.207Z0+0 records out
3332023-11-30T02:06:56.207Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3342023-11-30T02:06:56.207Z+ success 'ZFS vdev /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3352023-11-30T02:06:56.207Z+ set +x
3362023-11-30T02:06:56.208ZZFS vdev /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
3372023-11-30T02:06:56.208Z++ zpool list -o name
3382023-11-30T02:06:56.208Z++ grep oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03
3392023-11-30T02:06:56.208Z+ [[ -z '' ]]
3402023-11-30T02:06:56.208Z+ zpool create -f oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3412023-11-30T02:06:56.236Z+ success 'ZFS zpool oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3422023-11-30T02:06:56.239Z+ set +x
3432023-11-30T02:06:56.265ZZFS zpool oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
3442023-11-30T02:06:56.265Z+ for ZPOOL in "${ZPOOLS[@]}"
3452023-11-30T02:06:56.265Z+ echo 'Zpool: [oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3462023-11-30T02:06:56.265ZZpool: [oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03]
3472023-11-30T02:06:56.265Z+ VDEV_PATH=/scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3482023-11-30T02:06:56.265Z+ [[ -f /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3492023-11-30T02:06:56.265Z+ dd if=/dev/zero of=/scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3502023-11-30T02:06:56.265Z0+0 records in
3512023-11-30T02:06:56.266Z0+0 records out
3522023-11-30T02:06:56.266Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
3532023-11-30T02:06:56.266Z+ success 'ZFS vdev /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3542023-11-30T02:06:56.266Z+ set +x
3552023-11-30T02:06:56.266ZZFS vdev /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
3562023-11-30T02:06:56.266Z++ zpool list -o name
3572023-11-30T02:06:56.266Z++ grep oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03
3582023-11-30T02:06:56.266Z+ [[ -z '' ]]
3592023-11-30T02:06:56.266Z+ zpool create -f oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3602023-11-30T02:06:56.292Z+ success 'ZFS zpool oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3612023-11-30T02:06:56.292Z+ set +x
3622023-11-30T02:06:56.292ZZFS zpool oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
3632023-11-30T02:06:56.292Z+ for ZPOOL in "${ZPOOLS[@]}"
3642023-11-30T02:06:56.292Z+ echo 'Zpool: [oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930]'
3652023-11-30T02:06:56.292ZZpool: [oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930]
3662023-11-30T02:06:56.292Z+ VDEV_PATH=/scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev
3672023-11-30T02:06:56.294Z+ [[ -f /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev ]]
3682023-11-30T02:06:56.294Z+ dd if=/dev/zero of=/scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev bs=1 count=0 seek=20G
3692023-11-30T02:06:56.294Z0+0 records in
3702023-11-30T02:06:56.294Z0+0 records out
3712023-11-30T02:06:56.294Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3722023-11-30T02:06:56.294Z+ success 'ZFS vdev /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev exists'
3732023-11-30T02:06:56.294Z+ set +x
3742023-11-30T02:06:56.294ZZFS vdev /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev exists
3752023-11-30T02:06:56.294Z++ zpool list -o name
3762023-11-30T02:06:56.294Z++ grep oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930
3772023-11-30T02:06:56.321Z+ [[ -z '' ]]
3782023-11-30T02:06:56.321Z+ zpool create -f oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev
3792023-11-30T02:06:56.345Z+ success 'ZFS zpool oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 exists'
3802023-11-30T02:06:56.345Z+ set +x
3812023-11-30T02:06:56.345ZZFS zpool oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 exists
3822023-11-30T02:06:56.345Z+ for ZPOOL in "${ZPOOLS[@]}"
3832023-11-30T02:06:56.345Z+ echo 'Zpool: [oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60]'
3842023-11-30T02:06:56.345ZZpool: [oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60]
3852023-11-30T02:06:56.345Z+ VDEV_PATH=/scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev
3862023-11-30T02:06:56.345Z+ [[ -f /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev ]]
3872023-11-30T02:06:56.345Z+ dd if=/dev/zero of=/scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev bs=1 count=0 seek=20G
3882023-11-30T02:06:56.345Z0+0 records in
3892023-11-30T02:06:56.345Z0+0 records out
3902023-11-30T02:06:56.345Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
3912023-11-30T02:06:56.345Z+ success 'ZFS vdev /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev exists'
3922023-11-30T02:06:56.346Z+ set +x
3932023-11-30T02:06:56.346ZZFS vdev /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev exists
3942023-11-30T02:06:56.346Z++ zpool list -o name
3952023-11-30T02:06:56.346Z++ grep oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60
3962023-11-30T02:06:56.346Z+ [[ -z '' ]]
3972023-11-30T02:06:56.346Z+ zpool create -f oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev
3982023-11-30T02:06:56.370Z+ success 'ZFS zpool oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 exists'
3992023-11-30T02:06:56.370Z+ set +x
4002023-11-30T02:06:56.370ZZFS zpool oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 exists
4012023-11-30T02:06:56.370Z+ for ZPOOL in "${ZPOOLS[@]}"
4022023-11-30T02:06:56.370Z+ echo 'Zpool: [oxp_31bd71cd-4736-4a12-a387-9b74b050396f]'
4032023-11-30T02:06:56.370ZZpool: [oxp_31bd71cd-4736-4a12-a387-9b74b050396f]
4042023-11-30T02:06:56.370Z+ VDEV_PATH=/scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev
4052023-11-30T02:06:56.370Z+ [[ -f /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev ]]
4062023-11-30T02:06:56.370Z+ dd if=/dev/zero of=/scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev bs=1 count=0 seek=20G
4072023-11-30T02:06:56.370Z0+0 records in
4082023-11-30T02:06:56.370Z0+0 records out
4092023-11-30T02:06:56.370Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4102023-11-30T02:06:56.370Z+ success 'ZFS vdev /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev exists'
4112023-11-30T02:06:56.370Z+ set +x
4122023-11-30T02:06:56.370ZZFS vdev /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev exists
4132023-11-30T02:06:56.370Z++ zpool list -o name
4142023-11-30T02:06:56.370Z++ grep oxp_31bd71cd-4736-4a12-a387-9b74b050396f
4152023-11-30T02:06:56.394Z+ [[ -z '' ]]
4162023-11-30T02:06:56.395Z+ zpool create -f oxp_31bd71cd-4736-4a12-a387-9b74b050396f /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev
4172023-11-30T02:06:56.418Z+ success 'ZFS zpool oxp_31bd71cd-4736-4a12-a387-9b74b050396f exists'
4182023-11-30T02:06:56.418Z+ set +x
4192023-11-30T02:06:56.418ZZFS zpool oxp_31bd71cd-4736-4a12-a387-9b74b050396f exists
4202023-11-30T02:06:56.418Z+ for ZPOOL in "${ZPOOLS[@]}"
4212023-11-30T02:06:56.419Z+ echo 'Zpool: [oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7]'
4222023-11-30T02:06:56.419ZZpool: [oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7]
4232023-11-30T02:06:56.419Z+ VDEV_PATH=/scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev
4242023-11-30T02:06:56.419Z+ [[ -f /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev ]]
4252023-11-30T02:06:56.419Z+ dd if=/dev/zero of=/scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev bs=1 count=0 seek=20G
4262023-11-30T02:06:56.419Z0+0 records in
4272023-11-30T02:06:56.419Z0+0 records out
4282023-11-30T02:06:56.419Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
4292023-11-30T02:06:56.419Z+ success 'ZFS vdev /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev exists'
4302023-11-30T02:06:56.419Z+ set +x
4312023-11-30T02:06:56.419ZZFS vdev /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev exists
4322023-11-30T02:06:56.419Z++ zpool list -o name
4332023-11-30T02:06:56.419Z++ grep oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7
4342023-11-30T02:06:56.451Z+ [[ -z '' ]]
4352023-11-30T02:06:56.451Z+ zpool create -f oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev
4362023-11-30T02:06:56.477Z+ success 'ZFS zpool oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 exists'
4372023-11-30T02:06:56.477Z+ set +x
4382023-11-30T02:06:56.477ZZFS zpool oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 exists
4392023-11-30T02:06:56.477Z+ for ZPOOL_TYPE in "${ZPOOL_TYPES[@]}"
4402023-11-30T02:06:56.477Z+ readarray -t ZPOOLS
4412023-11-30T02:06:56.477Z++ grep '"oxi_' /opt/oxide/work/tools/../smf/sled-agent/non-gimlet/config.toml
4422023-11-30T02:06:56.477Z++ sed 's/[ ",]//g'
4432023-11-30T02:06:56.477Z+ for ZPOOL in "${ZPOOLS[@]}"
4442023-11-30T02:06:56.477Z+ echo 'Zpool: [oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b]'
4452023-11-30T02:06:56.477ZZpool: [oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b]
4462023-11-30T02:06:56.477Z+ VDEV_PATH=/scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4472023-11-30T02:06:56.477Z+ [[ -f /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
4482023-11-30T02:06:56.477Z+ dd if=/dev/zero of=/scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
4492023-11-30T02:06:56.477Z0+0 records in
4502023-11-30T02:06:56.477Z0+0 records out
4512023-11-30T02:06:56.477Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4522023-11-30T02:06:56.477Z+ success 'ZFS vdev /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
4532023-11-30T02:06:56.477Z+ set +x
4542023-11-30T02:06:56.477ZZFS vdev /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
4552023-11-30T02:06:56.479Z++ zpool list -o name
4562023-11-30T02:06:56.479Z++ grep oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b
4572023-11-30T02:06:56.479Z+ [[ -z '' ]]
4582023-11-30T02:06:56.479Z+ zpool create -f oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4592023-11-30T02:06:56.506Z+ success 'ZFS zpool oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
4602023-11-30T02:06:56.506Z+ set +x
4612023-11-30T02:06:56.530ZZFS zpool oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b exists
4622023-11-30T02:06:56.530Z+ for ZPOOL in "${ZPOOLS[@]}"
4632023-11-30T02:06:56.530Z+ echo 'Zpool: [oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b]'
4642023-11-30T02:06:56.530Z+ VDEV_PATH=/scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4652023-11-30T02:06:56.530Z+ [[ -f /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
4662023-11-30T02:06:56.531ZZpool: [oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b]
4672023-11-30T02:06:56.531Z+ dd if=/dev/zero of=/scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
4682023-11-30T02:06:56.531Z0+0 records in
4692023-11-30T02:06:56.531Z0+0 records out
4702023-11-30T02:06:56.531Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
4712023-11-30T02:06:56.531Z+ success 'ZFS vdev /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
4722023-11-30T02:06:56.531Z+ set +x
4732023-11-30T02:06:56.531ZZFS vdev /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
4742023-11-30T02:06:56.531Z++ zpool list -o name
4752023-11-30T02:06:56.531Z++ grep oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b
4762023-11-30T02:06:56.531Z+ [[ -z '' ]]
4772023-11-30T02:06:56.531Z+ zpool create -f oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4782023-11-30T02:06:56.555Z+ success 'ZFS zpool oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
4792023-11-30T02:06:56.555Z+ set +x
4802023-11-30T02:06:56.555ZZFS zpool oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b exists
4812023-11-30T02:06:56.555Z+ [[ zone == \z\o\n\e ]]
4822023-11-30T02:06:56.555Z+ ensure_simulated_links igb0
4832023-11-30T02:06:56.555Z+ local PHYSICAL_LINK=igb0
4842023-11-30T02:06:56.555Z+ INDICES=("0" "1")
4852023-11-30T02:06:56.555Z+ for I in "${INDICES[@]}"
4862023-11-30T02:06:56.555Z++ get_simnet_name_if_exists net0
4872023-11-30T02:06:56.555Z++ dladm show-simnet -p -o LINK net0
4882023-11-30T02:06:56.579Z++ echo ''
4892023-11-30T02:06:56.579Z+ [[ -z '' ]]
4902023-11-30T02:06:56.579Z+ dladm create-simnet -t net0
4912023-11-30T02:06:56.579Z+ dladm create-simnet -t sc0_0
4922023-11-30T02:06:56.580Z+ dladm modify-simnet -t -p net0 sc0_0
4932023-11-30T02:06:56.580Z+ dladm set-linkprop -p mtu=9000 sc0_0
4942023-11-30T02:06:56.580Z+ success 'Simnet net0/sc0_0 exists'
4952023-11-30T02:06:56.580Z+ set +x
4962023-11-30T02:06:56.580ZSimnet net0/sc0_0 exists
4972023-11-30T02:06:56.580Z+ for I in "${INDICES[@]}"
4982023-11-30T02:06:56.580Z++ get_simnet_name_if_exists net1
4992023-11-30T02:06:56.580Z++ dladm show-simnet -p -o LINK net1
5002023-11-30T02:06:56.604Z++ echo ''
5012023-11-30T02:06:56.604Z+ [[ -z '' ]]
5022023-11-30T02:06:56.604Z+ dladm create-simnet -t net1
5032023-11-30T02:06:56.604Z+ dladm create-simnet -t sc1_0
5042023-11-30T02:06:56.604Z+ dladm modify-simnet -t -p net1 sc1_0
5052023-11-30T02:06:56.604Z+ dladm set-linkprop -p mtu=9000 sc1_0
5062023-11-30T02:06:56.604Z+ success 'Simnet net1/sc1_0 exists'
5072023-11-30T02:06:56.604Z+ set +x
5082023-11-30T02:06:56.604ZSimnet net1/sc1_0 exists
5092023-11-30T02:06:56.604Z++ get_vnic_name_if_exists sc0_1
5102023-11-30T02:06:56.604Z++ dladm show-vnic -p -o LINK sc0_1
5112023-11-30T02:06:56.604Z++ echo ''
5122023-11-30T02:06:56.604Z+ [[ -z '' ]]
5132023-11-30T02:06:56.604Z+ dladm create-vnic -t sc0_1 -l igb0 -m a8:e1:de:01:70:1d
5142023-11-30T02:06:56.634Z+ [[ -v PROMISC_FILT_OFF ]]
5152023-11-30T02:06:56.634Z+ success 'Vnic sc0_1 exists'
5162023-11-30T02:06:56.634Z+ set +x
5172023-11-30T02:06:56.634ZVnic sc0_1 exists
5182023-11-30T02:06:56.634Z+ warn_if_no_proxy_arp
5192023-11-30T02:06:56.634Z+ [[ -v PXA_START ]]
5202023-11-30T02:06:56.634Z+ [[ -v PXA_END ]]
5212023-11-30T02:06:56.636Z+ ensure_softnpu_zone
5222023-11-30T02:06:56.636Z+ zoneadm list
5232023-11-30T02:06:56.636Z+ grep -q sidecar_softnpu
5242023-11-30T02:06:56.636Z+ [[ -f out/npuzone/npuzone ]]
5252023-11-30T02:06:56.636Z+ out/npuzone/npuzone create sidecar --omicron-zone --ports sc0_0,tfportrear0_0 --ports sc0_1,tfportqsfp0_0 --sidecar-lite-commit f0585a29fb0285f7a1220c1118856b0e5c1f75c5 --softnpu-commit dec63e67156fe6e958991bbfa090629868115ab5
5262023-11-30T02:06:56.660Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/softnpu/image/dec63e67156fe6e958991bbfa090629868115ab5/softnpu.sha256.txt
5272023-11-30T02:07:05.218Zdownloading softnpu ...done
5282023-11-30T02:07:05.242Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/sidecar-lite/release/f0585a29fb0285f7a1220c1118856b0e5c1f75c5/libsidecar_lite.so.sha256.txt
5292023-11-30T02:07:10.883Zdownloading asic_program.so ...done
5302023-11-30T02:07:10.907Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/sidecar-lite/release/f0585a29fb0285f7a1220c1118856b0e5c1f75c5/scadm.sha256.txt
5312023-11-30T02:07:15.496Zdownloading scadm ...done
5322023-11-30T02:07:15.520Zcreating port tfportrear0_0 for sc0_0
5332023-11-30T02:07:15.521Zcreating port tfportqsfp0_0 for sc0_1
5342023-11-30T02:07:15.544Zsoftnpu zone setup
5352023-11-30T02:07:15.544Zconfigure zone
5362023-11-30T02:07:15.544Zinstall zone
5372023-11-30T02:07:16.580Zboot zone
5382023-11-30T02:07:19.695Z[sidecar_softnpu] svcs milestone/network
5392023-11-30T02:07:19.719ZSTATE STIME FMRI
5402023-11-30T02:07:19.721Zonline 2:07:18 svc:/milestone/network:default
5412023-11-30T02:07:19.721Z[sidecar_softnpu] /softnpu/npu start
5422023-11-30T02:07:19.745Z
5432023-11-30T02:07:19.769Z+ /opt/oxide/work/tools/scrimlet/softnpu-init.sh
5442023-11-30T02:07:19.769Z+ GATEWAY_IP=10.151.1.100
5452023-11-30T02:07:19.769Z+ echo 'Using 10.151.1.100 as gateway ip'
5462023-11-30T02:07:19.769ZUsing 10.151.1.100 as gateway ip
5472023-11-30T02:07:19.769Z+ [[ ! -v GATEWAY_MAC ]]
5482023-11-30T02:07:19.769Z+ GATEWAY_MAC=18:c0:4d:0d:9f:b2
5492023-11-30T02:07:19.769Z++ grep -c -E '^([0-9a-fA-F]{2}:){5}[0-9a-fA-F]{2}$' /dev/fd/63
5502023-11-30T02:07:19.769Z+++ echo 18:c0:4d:0d:9f:b2
5512023-11-30T02:07:19.769Z+ COUNT=1
5522023-11-30T02:07:19.769Z+ [[ 1 -ne 1 ]]
5532023-11-30T02:07:19.770Z+ echo 'Using 18:c0:4d:0d:9f:b2 as gateway mac'
5542023-11-30T02:07:19.770ZUsing 18:c0:4d:0d:9f:b2 as gateway mac
5552023-11-30T02:07:19.770Z+ z_scadm add-arp-entry 10.151.1.100 18:c0:4d:0d:9f:b2
5562023-11-30T02:07:19.770Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone add-arp-entry 10.151.1.100 18:c0:4d:0d:9f:b2
5572023-11-30T02:07:19.794Z+ PXA_MAC=a8:e1:de:01:70:1d
5582023-11-30T02:07:19.794Z+ [[ -v PXA_START ]]
5592023-11-30T02:07:19.794Z+ z_scadm add-proxy-arp 10.151.1.164 10.151.1.227 a8:e1:de:01:70:1d
5602023-11-30T02:07:19.794Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone add-proxy-arp 10.151.1.164 10.151.1.227 a8:e1:de:01:70:1d
5612023-11-30T02:07:19.818Z+ z_scadm dump-state
5622023-11-30T02:07:19.818Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone dump-state
5632023-11-30T02:07:19.842Zlocal v6:
5642023-11-30T02:07:19.842Zlocal v4:
5652023-11-30T02:07:19.842Zrouter v6:
5662023-11-30T02:07:19.844Zrouter v4_idx:
5672023-11-30T02:07:19.844Zrouter v4_routes:
5682023-11-30T02:07:19.844Zresolver v4:
5692023-11-30T02:07:19.844Z10.151.1.100 -> 18:c0:4d:0d:9f:b2
5702023-11-30T02:07:19.844Zresolver v6:
5712023-11-30T02:07:19.844Znat v4:
5722023-11-30T02:07:19.844Znat_v6:
5732023-11-30T02:07:19.844Zport_mac:
5742023-11-30T02:07:19.844Zproxy arp:
5752023-11-30T02:07:19.844Z10.151.1.164/10.151.1.227: a8:e1:de:01:70:1d
5762023-11-30T02:07:19.844Z+ success 'softnpu zone exists'
5772023-11-30T02:07:19.844Z+ set +x
5782023-11-30T02:07:19.844Zsoftnpu zone exists
5792023-11-30T02:07:19.844Z
5802023-11-30T02:07:19.844Zreal 23.901966276
5812023-11-30T02:07:19.844Zuser 0.716774620
5822023-11-30T02:07:19.844Zsys 1.620432297
5832023-11-30T02:07:19.844Ztrap 0.000079872
5842023-11-30T02:07:19.844Ztflt 0.000103909
5852023-11-30T02:07:19.844Zdflt 0.001198653
5862023-11-30T02:07:19.844Zkflt 0.000000000
5872023-11-30T02:07:19.844Zlock 18:25.684011255
5882023-11-30T02:07:19.844Zslp 1:11.542732800
5892023-11-30T02:07:19.844Zlat 0.082824537
5902023-11-30T02:07:19.844Zstop 0.358801548
5912023-11-30T02:07:19.844Z+ tar xf out/omicron-sled-agent.tar pkg/config-rss.toml
5922023-11-30T02:07:19.876Z++ sed -n 's/silo_name = "\(.*\)"/\1/p' pkg/config-rss.toml
5932023-11-30T02:07:19.900Z+ SILO_NAME=recovery
5942023-11-30T02:07:19.900Z++ sed -n 's/external_dns_zone_name = "\(.*\)"/\1/p' pkg/config-rss.toml
5952023-11-30T02:07:19.901Z+ EXTERNAL_DNS_DOMAIN=oxide.test
5962023-11-30T02:07:19.901Z+ sed -i~ '
5972023-11-30T02:07:19.901Z /^external_dns_ips/c\
5982023-11-30T02:07:19.901Zexternal_dns_ips = [ "10.151.1.164", "10.151.1.165" ]
5992023-11-30T02:07:19.901Z /^\[\[internal_services_ip_pool_ranges/,/^$/ {
6002023-11-30T02:07:19.901Z /^first/c\
6012023-11-30T02:07:19.901Zfirst = "10.151.1.164"
6022023-11-30T02:07:19.901Z /^last/c\
6032023-11-30T02:07:19.902Zlast = "10.151.1.173"
6042023-11-30T02:07:19.902Z }
6052023-11-30T02:07:19.902Z /^\[rack_network_config/,/^$/ {
6062023-11-30T02:07:19.902Z /^infra_ip_first/c\
6072023-11-30T02:07:19.902Zinfra_ip_first = "10.151.1.174"
6082023-11-30T02:07:19.902Z /^infra_ip_last/c\
6092023-11-30T02:07:19.902Zinfra_ip_last = "10.151.1.174"
6102023-11-30T02:07:19.902Z }
6112023-11-30T02:07:19.902Z /^\[\[rack_network_config.ports/,/^$/ {
6122023-11-30T02:07:19.902Z /^routes/c\
6132023-11-30T02:07:19.902Zroutes = \[{nexthop = "10.151.1.100", destination = "0.0.0.0/0"}\]
6142023-11-30T02:07:19.902Z /^addresses/c\
6152023-11-30T02:07:19.902Zaddresses = \["10.151.1.174/32"\]
6162023-11-30T02:07:19.902Z }
6172023-11-30T02:07:19.902Z' pkg/config-rss.toml
6182023-11-30T02:07:19.902Z+ diff -u pkg/config-rss.toml~ pkg/config-rss.toml
6192023-11-30T02:07:19.902Z--- pkg/config-rss.toml~ Mon Jul 24 01:21:28 2006
6202023-11-30T02:07:19.902Z+++ pkg/config-rss.toml Thu Nov 30 02:07:20 2023
6212023-11-30T02:07:19.902Z@@ -41,7 +41,7 @@
6222023-11-30T02:07:19.902Z # the DNS domain delegated to the rack by the customer. Each of these addresses
6232023-11-30T02:07:19.902Z # must be contained in one of the "internal services" IP Pool ranges listed
6242023-11-30T02:07:19.902Z # below.
6252023-11-30T02:07:19.902Z-external_dns_ips = [ "192.168.1.20", "192.168.1.21" ]
6262023-11-30T02:07:19.902Z+external_dns_ips = [ "10.151.1.164", "10.151.1.165" ]
6272023-11-30T02:07:19.902Z
6282023-11-30T02:07:19.902Z # Initial TLS certificates for the external API
6292023-11-30T02:07:19.902Z #
6302023-11-30T02:07:19.902Z@@ -77,8 +77,8 @@
6312023-11-30T02:07:19.902Z #
6322023-11-30T02:07:19.902Z # For more on this and what to put here, see docs/how-to-run.adoc.
6332023-11-30T02:07:19.902Z [[internal_services_ip_pool_ranges]]
6342023-11-30T02:07:19.902Z-first = "192.168.1.20"
6352023-11-30T02:07:19.902Z-last = "192.168.1.29"
6362023-11-30T02:07:19.902Z+first = "10.151.1.164"
6372023-11-30T02:07:19.902Z+last = "10.151.1.173"
6382023-11-30T02:07:19.902Z
6392023-11-30T02:07:19.902Z # TODO - this configuration is subject to change going forward. Ultimately these
6402023-11-30T02:07:19.902Z # parameters should be provided to the control plane via wicket, but we need to
6412023-11-30T02:07:19.903Z@@ -92,8 +92,8 @@
6422023-11-30T02:07:19.903Z # A range of IP addresses used by Boundary Services on the external network. In
6432023-11-30T02:07:19.903Z # a real system, these would be addresses of the uplink ports on the Sidecar.
6442023-11-30T02:07:19.903Z # With softnpu, only one address is used.
6452023-11-30T02:07:19.903Z-infra_ip_first = "192.168.1.30"
6462023-11-30T02:07:19.903Z-infra_ip_last = "192.168.1.30"
6472023-11-30T02:07:19.903Z+infra_ip_first = "10.151.1.174"
6482023-11-30T02:07:19.903Z+infra_ip_last = "10.151.1.174"
6492023-11-30T02:07:19.903Z
6502023-11-30T02:07:19.905Z # Configurations for BGP routers to run on the scrimlets.
6512023-11-30T02:07:19.905Z bgp = []
6522023-11-30T02:07:19.905Z@@ -101,9 +101,9 @@
6532023-11-30T02:07:19.905Z # You can configure multiple uplinks by repeating the following stanza
6542023-11-30T02:07:19.905Z [[rack_network_config.ports]]
6552023-11-30T02:07:19.905Z # Routes associated with this port.
6562023-11-30T02:07:19.905Z-routes = [{nexthop = "192.168.1.199", destination = "0.0.0.0/0"}]
6572023-11-30T02:07:19.905Z+routes = [{nexthop = "10.151.1.100", destination = "0.0.0.0/0"}]
6582023-11-30T02:07:19.905Z+ true
6592023-11-30T02:07:19.905Z # Addresses associated with this port.
6602023-11-30T02:07:19.905Z-addresses = ["192.168.1.30/32"]
6612023-11-30T02:07:19.905Z+addresses = ["10.151.1.174/32"]
6622023-11-30T02:07:19.905Z # Name of the uplink port. This should always be "qsfp0" when using softnpu.
6632023-11-30T02:07:19.905Z port = "qsfp0"
6642023-11-30T02:07:19.905Z # The speed of this port.
6652023-11-30T02:07:19.905Z+ tar rvf out/omicron-sled-agent.tar pkg/config-rss.toml
6662023-11-30T02:07:19.905Za pkg/config-rss.toml 7K
6672023-11-30T02:07:19.905Z+ rm -f pkg/config-rss.toml pkg/config-rss.toml~
6682023-11-30T02:07:19.905Z+ TLS_NAME=recovery.sys.oxide.test
6692023-11-30T02:07:19.905Z+ openssl req -newkey rsa:4096 -x509 -sha256 -days 3 -nodes -out pkg/initial-tls-cert.pem -keyout pkg/initial-tls-key.pem -subj /CN=recovery.sys.oxide.test -addext subjectAltName=DNS:recovery.sys.oxide.test -addext basicConstraints=critical,CA:FALSE -config /dev/stdin
6702023-11-30T02:07:20.360Z..............+....+..+.........+......+.+..+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.......+..+...+.+......+...+...+..+...+...+.........+.+............+.....+....+.....+...+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.+...........+.+.....+.+........+.......+...+............+........+......+.+.............................+..................+...+.........+...+.+............+.....+............+.......+....................+.+...+......+.........+..+............+.+.....+.........+.......+.....+.+..+.+......+........+...+....+.........+..+.......+..+...+............+...+......+............+..................+...............+.+...........+...+....+...........+..........+..............+...+.............+......+........+......+.+..................+........+...+.+.....+......+.......+............+........+......+..........+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
6712023-11-30T02:07:20.622Z.............+.+..+.......+.....+.+......+........+...............+...+...+......+.........+....+...+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*...+........+.......+..+...+....+.........+..+..........+.....+.......+..+...+.............+...............+.........+.....+....+...........+.+..+.+..+....+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.......+.....+.+.........+...+...+.....+.+.....+..........+........+......+....+...+.....+...............+...+....+...+............+..+..........+......+..+...+..........+..+............+....+.........+..+...............+....+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
6722023-11-30T02:07:20.650Z-----
6732023-11-30T02:07:20.650Z+ tar rvf out/omicron-sled-agent.tar pkg/initial-tls-cert.pem pkg/initial-tls-key.pem
6742023-11-30T02:07:20.675Za pkg/initial-tls-cert.pem 2K
6752023-11-30T02:07:20.675Za pkg/initial-tls-key.pem 4K
6762023-11-30T02:07:20.675Z+ rm -f pkg/initial-tls-cert.pem pkg/initial-tls-key.pem
6772023-11-30T02:07:20.675Z+ rmdir pkg
6782023-11-30T02:07:20.675Z+ E2E_TLS_CERT=/opt/oxide/sled-agent/pkg/initial-tls-cert.pem
6792023-11-30T02:07:20.675Z+ pfexec mkdir -p /usr/oxide
6802023-11-30T02:07:20.675Z+ pfexec rm -f /usr/oxide/tcpproxy
6812023-11-30T02:07:20.675Z+ pfexec curl -sSfL -o /usr/oxide/tcpproxy http://catacomb.eng.oxide.computer:12346/tcpproxy
6822023-11-30T02:07:20.965Z+ pfexec chmod +x /usr/oxide/tcpproxy
6832023-11-30T02:07:20.989Z+ pfexec rm -f /var/svc/manifest/site/tcpproxy.xml
6842023-11-30T02:07:20.989Z+ pfexec curl -sSfL -o /var/svc/manifest/site/tcpproxy.xml http://catacomb.eng.oxide.computer:12346/tcpproxy.xml
6852023-11-30T02:07:20.998Z+ pfexec svccfg import /var/svc/manifest/site/tcpproxy.xml
6862023-11-30T02:07:21.088Z+ OMICRON_NO_UNINSTALL=1
6872023-11-30T02:07:21.088Z+ ptime -m pfexec ./target/release/omicron-package -t test install
6882023-11-30T02:07:21.112ZNov 30 02:07:21.610 DEBG target[test]: Target({"image": "standard", "machine": "non-gimlet", "rack-topology": "single-sled", "switch": "softnpu"})
6892023-11-30T02:07:21.112ZNov 30 02:07:21.612 INFO Installing service, dst: /opt/oxide/cockroachdb.tar.gz, src: out/cockroachdb.tar.gz
6902023-11-30T02:07:21.112ZNov 30 02:07:21.612 INFO Installing service, dst: /opt/oxide/clickhouse.tar.gz, src: out/clickhouse.tar.gz
6912023-11-30T02:07:21.112ZNov 30 02:07:21.612 INFO Installing service, dst: /opt/oxide/clickhouse_keeper.tar.gz, src: out/clickhouse_keeper.tar.gz
6922023-11-30T02:07:21.112ZNov 30 02:07:21.612 INFO Installing service, dst: /opt/oxide/crucible_pantry.tar.gz, src: out/crucible-pantry.tar.gz
6932023-11-30T02:07:21.112ZNov 30 02:07:21.612 INFO Installing service, dst: /opt/oxide/nexus.tar.gz, src: out/omicron-nexus.tar.gz
6942023-11-30T02:07:21.112ZNov 30 02:07:21.612 INFO Installing service, dst: /opt/oxide/ntp.tar.gz, src: out/ntp.tar.gz
6952023-11-30T02:07:21.112ZNov 30 02:07:21.612 INFO Installing service, dst: /opt/oxide/mgs.tar.gz, src: out/omicron-gateway-softnpu.tar.gz
6962023-11-30T02:07:21.112ZNov 30 02:07:21.613 INFO Installing service, dst: /opt/oxide/crucible.tar.gz, src: out/crucible.tar.gz
6972023-11-30T02:07:21.112ZNov 30 02:07:21.613 INFO Installing service, dst: /opt/oxide/internal_dns.tar.gz, src: out/internal-dns.tar.gz
6982023-11-30T02:07:21.112ZNov 30 02:07:21.613 INFO Installing service, dst: /opt/oxide/propolis-server.tar.gz, src: out/propolis-server.tar.gz
6992023-11-30T02:07:21.112ZNov 30 02:07:21.613 INFO Installing service, dst: /opt/oxide/switch.tar.gz, src: out/switch-softnpu.tar.gz
7002023-11-30T02:07:21.112ZNov 30 02:07:21.613 INFO Installing service, dst: /opt/oxide/overlay.tar.gz, src: out/overlay.tar.gz
7012023-11-30T02:07:21.112ZNov 30 02:07:21.613 INFO Installing service, dst: /opt/oxide/sled-agent.tar, src: out/omicron-sled-agent.tar
7022023-11-30T02:07:21.112ZNov 30 02:07:21.613 INFO Installing service, dst: /opt/oxide/mg-ddm.tar, src: out/mg-ddm-gz.tar
7032023-11-30T02:07:21.113ZNov 30 02:07:21.613 INFO Installing service, dst: /opt/oxide/oximeter.tar.gz, src: out/oximeter-collector.tar.gz
7042023-11-30T02:07:21.113ZNov 30 02:07:21.613 INFO Installing service, dst: /opt/oxide/external_dns.tar.gz, src: out/external-dns.tar.gz
7052023-11-30T02:07:21.346ZNov 30 02:07:21.854 INFO Unpacking service tarball, service_path: /opt/oxide/mg-ddm, tar_path: /opt/oxide/mg-ddm.tar
7062023-11-30T02:07:21.380ZNov 30 02:07:21.888 INFO Unpacking service tarball, service_path: /opt/oxide/sled-agent, tar_path: /opt/oxide/sled-agent.tar
7072023-11-30T02:07:21.449ZNov 30 02:07:21.953 INFO Installing bootstrap service from /opt/oxide/sled-agent/pkg/manifest.xml
7082023-11-30T02:07:21.573Z
7092023-11-30T02:07:21.597Zreal 0.480003651
7102023-11-30T02:07:21.597Zuser 0.098415487
7112023-11-30T02:07:21.597Zsys 1.949950715
7122023-11-30T02:07:21.597Ztrap 0.000017500
7132023-11-30T02:07:21.597Ztflt 0.000061339
7142023-11-30T02:07:21.597Zdflt 0.001047101
7152023-11-30T02:07:21.597Zkflt 0.000002229
7162023-11-30T02:07:21.597Zlock 43.179493625
7172023-11-30T02:07:21.597Zslp 0.692023700
7182023-11-30T02:07:21.597Zlat 0.002973977
7192023-11-30T02:07:21.597Zstop 0.197128725
7202023-11-30T02:07:21.597Z+ retry=0
7212023-11-30T02:07:21.597Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7222023-11-30T02:07:21.597Z+ [[ 0 -gt 30 ]]
7232023-11-30T02:07:21.597Z+ sleep 1
7242023-11-30T02:07:22.588Z+ retry=1
7252023-11-30T02:07:22.588Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7262023-11-30T02:07:22.612Z+ [[ 1 -gt 30 ]]
7272023-11-30T02:07:22.612Z+ sleep 1
7282023-11-30T02:07:23.602Z+ retry=2
7292023-11-30T02:07:23.603Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7302023-11-30T02:07:23.626Z+ [[ 2 -gt 30 ]]
7312023-11-30T02:07:23.626Z+ sleep 1
7322023-11-30T02:07:24.616Z+ retry=3
7332023-11-30T02:07:24.616Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7342023-11-30T02:07:24.657Z+ [[ 3 -gt 30 ]]
7352023-11-30T02:07:24.657Z+ sleep 1
7362023-11-30T02:07:25.630Z+ retry=4
7372023-11-30T02:07:25.630Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7382023-11-30T02:07:45.452Z+ [[ 4 -gt 30 ]]
7392023-11-30T02:07:45.452Z+ sleep 1
7402023-11-30T02:07:46.456Z+ retry=5
7412023-11-30T02:07:46.456Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7422023-11-30T02:07:46.480Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone dump-state
7432023-11-30T02:07:46.504Zlocal v6:
7442023-11-30T02:07:46.504Zfd00:99::1
7452023-11-30T02:07:46.504Zfe80::9052:f5ff:fe65:147a
7462023-11-30T02:07:46.505Zlocal v4:
7472023-11-30T02:07:46.505Z10.151.1.174
7482023-11-30T02:07:46.505Zrouter v6:
7492023-11-30T02:07:46.505Zrouter v4_idx:
7502023-11-30T02:07:46.505Z0.0.0.0/0 -> 0
7512023-11-30T02:07:46.505Zrouter v4_routes:
7522023-11-30T02:07:46.505Z0 -> 10.151.1.100 (2)
7532023-11-30T02:07:46.505Zresolver v4:
7542023-11-30T02:07:46.505Z10.151.1.100 -> 18:c0:4d:0d:9f:b2
7552023-11-30T02:07:46.505Zresolver v6:
7562023-11-30T02:07:46.505Zfe80::5823:26ff:fe78:3198 -> 5a:23:26:78:31:98
7572023-11-30T02:07:46.505Zfe80::9052:f5ff:fe65:147a -> 92:52:f5:65:14:7a
7582023-11-30T02:07:46.505Znat v4:
7592023-11-30T02:07:46.505Znat_v6:
7602023-11-30T02:07:46.505Zport_mac:
7612023-11-30T02:07:46.505Z1000: a8:40:25:3e:55:14
7622023-11-30T02:07:46.505Z1: a8:40:25:3e:55:15
7632023-11-30T02:07:46.505Zproxy arp:
7642023-11-30T02:07:46.505Z10.151.1.164/10.151.1.227: a8:e1:de:01:70:1d
7652023-11-30T02:07:46.505Z+ retry=0
7662023-11-30T02:07:46.505Z+++ zoneadm list -n
7672023-11-30T02:07:46.505Z+++ grep oxz_ntp
7682023-11-30T02:07:46.530Z++ pfexec svcs -z -Hostate oxide/ntp
7692023-11-30T02:07:46.530Zsvcs: zone '-Hostate': entity not found
7702023-11-30T02:07:46.530Z++ true
7712023-11-30T02:07:46.530Z+ [[ '' != online ]]
7722023-11-30T02:07:46.530Z+ [[ 0 -gt 60 ]]
7732023-11-30T02:07:46.530Z+ sleep 1
7742023-11-30T02:07:47.518Z+ retry=1
7752023-11-30T02:07:47.544Z+++ zoneadm list -n
7762023-11-30T02:07:47.544Z+++ grep oxz_ntp
7772023-11-30T02:07:47.544Z++ pfexec svcs -z -Hostate oxide/ntp
7782023-11-30T02:07:47.544Zsvcs: zone '-Hostate': entity not found
7792023-11-30T02:07:47.544Z++ true
7802023-11-30T02:07:47.544Z+ [[ '' != online ]]
7812023-11-30T02:07:47.544Z+ [[ 1 -gt 60 ]]
7822023-11-30T02:07:47.544Z+ sleep 1
7832023-11-30T02:07:48.540Z+ retry=2
7842023-11-30T02:07:48.572Z+++ zoneadm list -n
7852023-11-30T02:07:48.572Z+++ grep oxz_ntp
7862023-11-30T02:07:48.573Z++ pfexec svcs -z -Hostate oxide/ntp
7872023-11-30T02:07:48.573Zsvcs: zone '-Hostate': entity not found
7882023-11-30T02:07:48.573Z++ true
7892023-11-30T02:07:48.573Z+ [[ '' != online ]]
7902023-11-30T02:07:48.573Z+ [[ 2 -gt 60 ]]
7912023-11-30T02:07:48.573Z+ sleep 1
7922023-11-30T02:07:49.560Z+ retry=3
7932023-11-30T02:07:49.584Z+++ zoneadm list -n
7942023-11-30T02:07:49.584Z+++ grep oxz_ntp
7952023-11-30T02:07:49.584Z++ pfexec svcs -z -Hostate oxide/ntp
7962023-11-30T02:07:49.584Zsvcs: zone '-Hostate': entity not found
7972023-11-30T02:07:49.584Z++ true
7982023-11-30T02:07:49.584Z+ [[ '' != online ]]
7992023-11-30T02:07:49.585Z+ [[ 3 -gt 60 ]]
8002023-11-30T02:07:49.585Z+ sleep 1
8012023-11-30T02:07:50.595Z+ retry=4
8022023-11-30T02:07:50.622Z+++ zoneadm list -n
8032023-11-30T02:07:50.622Z+++ grep oxz_ntp
8042023-11-30T02:07:50.622Z++ pfexec svcs -z oxz_ntp_38b10f0a-4607-4870-a143-d157c535ffd4 -Hostate oxide/ntp
8052023-11-30T02:07:50.622Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
8062023-11-30T02:07:50.622Z++ true
8072023-11-30T02:07:50.622Z+ [[ '' != online ]]
8082023-11-30T02:07:50.622Z+ [[ 4 -gt 60 ]]
8092023-11-30T02:07:50.622Z+ sleep 1
8102023-11-30T02:07:51.614Z+ retry=5
8112023-11-30T02:07:51.688Z+++ zoneadm list -n
8122023-11-30T02:07:51.688Z+++ grep oxz_ntp
8132023-11-30T02:07:51.688Z++ pfexec svcs -z oxz_ntp_38b10f0a-4607-4870-a143-d157c535ffd4 -Hostate oxide/ntp
8142023-11-30T02:07:51.712Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
8152023-11-30T02:07:51.712Z++ true
8162023-11-30T02:07:51.712Z+ [[ '' != online ]]
8172023-11-30T02:07:51.712Z+ [[ 5 -gt 60 ]]
8182023-11-30T02:07:51.712Z+ sleep 1
8192023-11-30T02:07:52.648Z+ retry=6
8202023-11-30T02:07:52.673Z+++ zoneadm list -n
8212023-11-30T02:07:52.673Z+++ grep oxz_ntp
8222023-11-30T02:07:52.673Z++ pfexec svcs -z oxz_ntp_38b10f0a-4607-4870-a143-d157c535ffd4 -Hostate oxide/ntp
8232023-11-30T02:07:52.697Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
8242023-11-30T02:07:52.697Z++ true
8252023-11-30T02:07:52.697Z+ [[ '' != online ]]
8262023-11-30T02:07:52.697Z+ [[ 6 -gt 60 ]]
8272023-11-30T02:07:52.698Z+ sleep 1
8282023-11-30T02:07:53.677Z+ retry=7
8292023-11-30T02:07:53.701Z+++ zoneadm list -n
8302023-11-30T02:07:53.701Z+++ grep oxz_ntp
8312023-11-30T02:07:53.701Z++ pfexec svcs -z oxz_ntp_38b10f0a-4607-4870-a143-d157c535ffd4 -Hostate oxide/ntp
8322023-11-30T02:07:53.725Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
8332023-11-30T02:07:53.725Z++ true
8342023-11-30T02:07:53.725Z+ [[ '' != online ]]
8352023-11-30T02:07:53.725Z+ [[ 7 -gt 60 ]]
8362023-11-30T02:07:53.725Z+ sleep 1
8372023-11-30T02:07:54.707Z+ retry=8
8382023-11-30T02:07:54.732Z+++ zoneadm list -n
8392023-11-30T02:07:54.732Z+++ grep oxz_ntp
8402023-11-30T02:07:54.732Z++ pfexec svcs -z oxz_ntp_38b10f0a-4607-4870-a143-d157c535ffd4 -Hostate oxide/ntp
8412023-11-30T02:07:54.732Z+ [[ disabled != online ]]
8422023-11-30T02:07:54.732Z+ [[ 8 -gt 60 ]]
8432023-11-30T02:07:54.732Z+ sleep 1
8442023-11-30T02:07:55.729Z+ retry=9
8452023-11-30T02:07:55.753Z+++ zoneadm list -n
8462023-11-30T02:07:55.753Z+++ grep oxz_ntp
8472023-11-30T02:07:55.753Z++ pfexec svcs -z oxz_ntp_38b10f0a-4607-4870-a143-d157c535ffd4 -Hostate oxide/ntp
8482023-11-30T02:07:55.753Z+ [[ disabled != online ]]
8492023-11-30T02:07:55.753Z+ [[ 9 -gt 60 ]]
8502023-11-30T02:07:55.753Z+ sleep 1
8512023-11-30T02:07:56.751Z+ retry=10
8522023-11-30T02:07:56.776Z+++ zoneadm list -n
8532023-11-30T02:07:56.776Z+++ grep oxz_ntp
8542023-11-30T02:07:56.776Z++ pfexec svcs -z oxz_ntp_38b10f0a-4607-4870-a143-d157c535ffd4 -Hostate oxide/ntp
8552023-11-30T02:07:56.776Z+ [[ disabled != online ]]
8562023-11-30T02:07:56.776Z+ [[ 10 -gt 60 ]]
8572023-11-30T02:07:56.776Z+ sleep 1
8582023-11-30T02:07:57.773Z+ retry=11
8592023-11-30T02:07:57.798Z+++ zoneadm list -n
8602023-11-30T02:07:57.799Z+++ grep oxz_ntp
8612023-11-30T02:07:57.799Z++ pfexec svcs -z oxz_ntp_38b10f0a-4607-4870-a143-d157c535ffd4 -Hostate oxide/ntp
8622023-11-30T02:07:57.799Z+ [[ disabled != online ]]
8632023-11-30T02:07:57.799Z+ [[ 11 -gt 60 ]]
8642023-11-30T02:07:57.799Z+ sleep 1
8652023-11-30T02:07:58.795Z+ retry=12
8662023-11-30T02:07:58.819Z+++ zoneadm list -n
8672023-11-30T02:07:58.819Z+++ grep oxz_ntp
8682023-11-30T02:07:58.819Z++ pfexec svcs -z oxz_ntp_38b10f0a-4607-4870-a143-d157c535ffd4 -Hostate oxide/ntp
8692023-11-30T02:07:58.819Z+ [[ disabled != online ]]
8702023-11-30T02:07:58.819Z+ [[ 12 -gt 60 ]]
8712023-11-30T02:07:58.819Z+ sleep 1
8722023-11-30T02:07:59.817Z+ retry=13
8732023-11-30T02:07:59.840Z+++ zoneadm list -n
8742023-11-30T02:07:59.840Z+++ grep oxz_ntp
8752023-11-30T02:07:59.840Z++ pfexec svcs -z oxz_ntp_38b10f0a-4607-4870-a143-d157c535ffd4 -Hostate oxide/ntp
8762023-11-30T02:07:59.840Z+ [[ disabled != online ]]
8772023-11-30T02:07:59.840Z+ [[ 13 -gt 60 ]]
8782023-11-30T02:07:59.840Z+ sleep 1
8792023-11-30T02:08:00.839Z+ retry=14
8802023-11-30T02:08:00.863Z+++ zoneadm list -n
8812023-11-30T02:08:00.863Z+++ grep oxz_ntp
8822023-11-30T02:08:00.863Z++ pfexec svcs -z oxz_ntp_38b10f0a-4607-4870-a143-d157c535ffd4 -Hostate oxide/ntp
8832023-11-30T02:08:00.863Z+ [[ disabled != online ]]
8842023-11-30T02:08:00.863Z+ [[ 14 -gt 60 ]]
8852023-11-30T02:08:00.864Z+ sleep 1
8862023-11-30T02:08:01.864Z+ retry=15
8872023-11-30T02:08:01.892Z+++ zoneadm list -n
8882023-11-30T02:08:01.892Z+++ grep oxz_ntp
8892023-11-30T02:08:01.892Z++ pfexec svcs -z oxz_ntp_38b10f0a-4607-4870-a143-d157c535ffd4 -Hostate oxide/ntp
8902023-11-30T02:08:01.892Z+ [[ disabled != online ]]
8912023-11-30T02:08:01.892Z+ [[ 15 -gt 60 ]]
8922023-11-30T02:08:01.892Z+ sleep 1
8932023-11-30T02:08:02.884Z+ retry=16
8942023-11-30T02:08:02.908Z+++ zoneadm list -n
8952023-11-30T02:08:02.909Z+++ grep oxz_ntp
8962023-11-30T02:08:02.909Z++ pfexec svcs -z oxz_ntp_38b10f0a-4607-4870-a143-d157c535ffd4 -Hostate oxide/ntp
8972023-11-30T02:08:02.909Z+ [[ disabled != online ]]
8982023-11-30T02:08:02.909Z+ [[ 16 -gt 60 ]]
8992023-11-30T02:08:02.909Z+ sleep 1
 
9002023-11-30T02:08:03.079Zbuskin console login: Nov 30 02:08:02 bukin ip: ip: joining multicasts failed (4) on vopte0 - will use link layer broadcasts for multicast
 
9012023-11-30T02:08:03.906Z+ retry=17
9022023-11-30T02:08:03.930Z+++ zoneadm list -n
9032023-11-30T02:08:03.930Z+++ grep oxz_ntp
9042023-11-30T02:08:03.930Z++ pfexec svcs -z oxz_ntp_38b10f0a-4607-4870-a143-d157c535ffd4 -Hostate oxide/ntp
9052023-11-30T02:08:03.932Z+ [[ online != online ]]
9062023-11-30T02:08:03.932Z+ echo 'Waited for chrony: 17s'
9072023-11-30T02:08:03.932ZWaited for chrony: 17s
9082023-11-30T02:08:03.932Z+ export RUST_BACKTRACE=1
9092023-11-30T02:08:03.932Z+ RUST_BACKTRACE=1
9102023-11-30T02:08:03.932Z+ export E2E_TLS_CERT IPPOOL_START IPPOOL_END
9112023-11-30T02:08:03.932Z+ ./tests/bootstrap
9122023-11-30T02:08:03.956ZUsing external DNS server at 10.151.1.164:53
9132023-11-30T02:08:03.957Z2023-11-30 02:08:04.462039516 UTC: attempting to log into API
9142023-11-30T02:08:25.065Z2023-11-30 02:08:19.436259553 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9152023-11-30T02:08:25.065Z2023-11-30 02:08:20.437171023 UTC: attempting to log into API
9162023-11-30T02:08:35.023Z2023-11-30 02:08:35.468804230 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9172023-11-30T02:08:36.024Z2023-11-30 02:08:36.469867205 UTC: attempting to log into API
9182023-11-30T02:08:51.058Z2023-11-30 02:08:51.502188721 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9192023-11-30T02:08:52.057Z2023-11-30 02:08:52.502748671 UTC: attempting to log into API
9202023-11-30T02:09:07.091Z2023-11-30 02:09:07.535269183 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9212023-11-30T02:09:08.094Z2023-11-30 02:09:08.535996609 UTC: attempting to log into API
9222023-11-30T02:09:23.122Z2023-11-30 02:09:22.853941090 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9232023-11-30T02:09:24.123Z2023-11-30 02:09:23.854811899 UTC: attempting to log into API
9242023-11-30T02:09:39.155Z2023-11-30 02:09:38.887325451 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9252023-11-30T02:09:40.232Z2023-11-30 02:09:39.887957244 UTC: attempting to log into API
9262023-11-30T02:09:55.190Z2023-11-30 02:09:54.921869457 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9272023-11-30T02:09:56.190Z2023-11-30 02:09:55.922511389 UTC: attempting to log into API
9282023-11-30T02:10:11.223Z2023-11-30 02:10:10.954933889 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9292023-11-30T02:10:12.223Z2023-11-30 02:10:11.955696575 UTC: attempting to log into API
9302023-11-30T02:10:27.256Z2023-11-30 02:10:27.017957892 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9312023-11-30T02:10:28.257Z2023-11-30 02:10:28.021310215 UTC: attempting to log into API
9322023-11-30T02:10:43.290Z2023-11-30 02:10:43.095981815 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9332023-11-30T02:10:44.291Z2023-11-30 02:10:44.099378384 UTC: attempting to log into API
9342023-11-30T02:10:59.324Z2023-11-30 02:10:59.172675267 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9352023-11-30T02:11:00.324Z2023-11-30 02:11:00.176000907 UTC: attempting to log into API
9362023-11-30T02:11:15.357Z2023-11-30 02:11:15.249210746 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9372023-11-30T02:11:16.358Z2023-11-30 02:11:16.252765797 UTC: attempting to log into API
9382023-11-30T02:11:31.390Z2023-11-30 02:11:31.326581662 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9392023-11-30T02:11:32.393Z2023-11-30 02:11:32.330024785 UTC: attempting to log into API
9402023-11-30T02:11:47.424Z2023-11-30 02:11:47.403738722 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9412023-11-30T02:11:48.424Z2023-11-30 02:11:48.407165193 UTC: attempting to log into API
9422023-11-30T02:12:03.457Z2023-11-30 02:12:03.449850808 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9432023-11-30T02:12:04.458Z2023-11-30 02:12:04.450692815 UTC: attempting to log into API
9442023-11-30T02:12:19.491Z2023-11-30 02:12:19.484561993 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9452023-11-30T02:12:20.493Z2023-11-30 02:12:20.486541634 UTC: attempting to log into API
9462023-11-30T02:12:35.531Z2023-11-30 02:12:35.518776437 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9472023-11-30T02:12:36.526Z2023-11-30 02:12:36.519710219 UTC: attempting to log into API
9482023-11-30T02:12:51.592Z2023-11-30 02:12:51.553698631 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9492023-11-30T02:12:52.583Z2023-11-30 02:12:52.554369125 UTC: attempting to log into API
 
9502023-11-30T02:13:13.735ZNov 30 02:13:09 bukin ip: ip: joining multicasts failed (4) on vopte4 - will use link layer broadcasts for multicast
 
9512023-11-30T02:13:13.796Z2023-11-30 02:13:07.593503848 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9522023-11-30T02:13:14.253Z2023-11-30 02:13:08.594569054 UTC: attempting to log into API
 
9532023-11-30T02:13:17.129ZNov 30 02:13:16 bukin ip: ip: joining multicasts failed (4) on vopte5 - will use link layer broadcasts for multicast
 
9542023-11-30T02:13:23.646Z2023-11-30 02:13:23.626331561 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
 
9552023-11-30T02:13:24.541ZNov 30 02:13:23 buskin ip: ip: joining multicasts failed (4) on vopte1 - will use link layer broadcasts for multicast
 
9562023-11-30T02:13:24.634Z2023-11-30 02:13:24.627364195 UTC: attempting to log into API
 
9572023-11-30T02:13:30.941ZNov 30 02:13:30 buskin ip: ip: joining multicasts failed (4) on vopte2 - will use link layer broadcasts for multicast
9582023-11-30T02:13:38.197ZNov 30 02:13:37 buskin ip: ip: joining multicasts failed (4) on vopte3 - will use link layer broadcasts for multicast
 
9592023-11-30T02:13:39.668Z2023-11-30 02:13:39.658029037 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9602023-11-30T02:13:40.669Z2023-11-30 02:13:40.658933737 UTC: attempting to log into API
9612023-11-30T02:13:55.702Z2023-11-30 02:13:55.692184331 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9622023-11-30T02:13:56.704Z2023-11-30 02:13:56.693183705 UTC: attempting to log into API
9632023-11-30T02:13:57.485Z2023-11-30 02:13:57.474827666 UTC: login succeeded
9642023-11-30T02:13:57.517Zwaiting for nexus to come up...
9652023-11-30T02:13:57.587Zcreating IP pool... 10.151.1.175 - 10.151.1.227
9662023-11-30T02:13:57.630Zensuring datasets are ready...
9672023-11-30T02:14:00.297Zlet's roll.
9682023-11-30T02:14:00.322Z+ rm ./tests/bootstrap
9692023-11-30T02:14:00.347Z+ for test_bin in tests/*
9702023-11-30T02:14:00.347Z+ ./tests/bootstrap-8d69322849e0fa66
9712023-11-30T02:14:00.347Z
9722023-11-30T02:14:00.347Zrunning 0 tests
9732023-11-30T02:14:00.347Z
9742023-11-30T02:14:00.347Ztest result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
9752023-11-30T02:14:00.347Z
9762023-11-30T02:14:00.347Z+ for test_bin in tests/*
9772023-11-30T02:14:00.347Z+ ./tests/end_to_end_tests-7d7c590276acd289
9782023-11-30T02:14:00.372Z
9792023-11-30T02:14:00.372Zrunning 2 tests
9802023-11-30T02:14:00.573Ztest no_spoof::no_spoof_here ... ok
9812023-11-30T02:15:01.012Ztest instance_launch::instance_launch has been running for over 60 seconds
9822023-11-30T02:16:57.247Ztest instance_launch::instance_launch ... ok
9832023-11-30T02:16:57.247Z
9842023-11-30T02:16:57.247Ztest result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 176.89s
9852023-11-30T02:16:57.247Z
9862023-11-30T02:16:57.289Z+ _exit_trap
9872023-11-30T02:16:57.289Z+ local status=0
9882023-11-30T02:16:57.289Z+ [[ 0 -eq 0 ]]
9892023-11-30T02:16:57.289Z+ exit 0
9902023-11-30T02:16:57.289Zprocess exited: duration 609281 ms, exit code 0
 
9912023-11-30T02:16:57.717Zfound 90 output files
9922023-11-30T02:16:57.717Zuploading: /var/svc/log/oxide-sled-agent:default.log (739710 bytes)
9932023-11-30T02:16:59.085Zuploaded: /var/svc/log/oxide-sled-agent:default.log
9942023-11-30T02:16:59.085Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_bb8ea64f-fe85-41af-80f4-c4fd258b2e57/root/var/svc/log/oxide-cockroachdb:default.log (0 bytes)
9952023-11-30T02:17:00.171Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_bb8ea64f-fe85-41af-80f4-c4fd258b2e57/root/var/svc/log/oxide-cockroachdb:default.log
9962023-11-30T02:17:00.171Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_bb8ea64f-fe85-41af-80f4-c4fd258b2e57/root/var/svc/log/oxide-cockroachdb:default.log.0 (5368 bytes)
9972023-11-30T02:17:01.273Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_bb8ea64f-fe85-41af-80f4-c4fd258b2e57/root/var/svc/log/oxide-cockroachdb:default.log.0
9982023-11-30T02:17:01.273Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_f6ce8b79-f9da-41bd-acd2-b4dd18a6f33d/root/var/svc/log/oxide-internal_dns:default.log (7511 bytes)
9992023-11-30T02:17:02.464Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_f6ce8b79-f9da-41bd-acd2-b4dd18a6f33d/root/var/svc/log/oxide-internal_dns:default.log
10002023-11-30T02:17:02.490Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_f6ce8b79-f9da-41bd-acd2-b4dd18a6f33d/root/var/svc/log/oxide-internal_dns:default.log.0 (111946 bytes)
10012023-11-30T02:17:03.680Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_f6ce8b79-f9da-41bd-acd2-b4dd18a6f33d/root/var/svc/log/oxide-internal_dns:default.log.0
10022023-11-30T02:17:03.680Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_684c8e87-3775-4df3-9a18-4d5e217d87b9/root/var/svc/log/oxide-cockroachdb:default.log (0 bytes)
10032023-11-30T02:17:04.753Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_684c8e87-3775-4df3-9a18-4d5e217d87b9/root/var/svc/log/oxide-cockroachdb:default.log
10042023-11-30T02:17:04.753Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_684c8e87-3775-4df3-9a18-4d5e217d87b9/root/var/svc/log/oxide-cockroachdb:default.log.0 (5370 bytes)
10052023-11-30T02:17:05.857Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_684c8e87-3775-4df3-9a18-4d5e217d87b9/root/var/svc/log/oxide-cockroachdb:default.log.0
10062023-11-30T02:17:05.857Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_60aaebf6-c299-451f-b222-8836faa6e402/root/var/svc/log/oxide-crucible-agent:default.log (4321 bytes)
10072023-11-30T02:17:06.961Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_60aaebf6-c299-451f-b222-8836faa6e402/root/var/svc/log/oxide-crucible-agent:default.log
10082023-11-30T02:17:06.961Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_60aaebf6-c299-451f-b222-8836faa6e402/root/var/svc/log/oxide-crucible-agent:default.log.0 (19616 bytes)
10092023-11-30T02:17:08.108Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_60aaebf6-c299-451f-b222-8836faa6e402/root/var/svc/log/oxide-crucible-agent:default.log.0
10102023-11-30T02:17:08.108Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_60aaebf6-c299-451f-b222-8836faa6e402/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
10112023-11-30T02:17:09.180Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_60aaebf6-c299-451f-b222-8836faa6e402/root/var/svc/log/oxide-crucible-downstairs:default.log
10122023-11-30T02:17:09.180Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_60aaebf6-c299-451f-b222-8836faa6e402/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
10132023-11-30T02:17:10.277Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_60aaebf6-c299-451f-b222-8836faa6e402/root/var/svc/log/oxide-crucible-downstairs:default.log.0
10142023-11-30T02:17:10.277Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_60aaebf6-c299-451f-b222-8836faa6e402/root/var/svc/log/oxide-crucible-downstairs:downstairs-78ed061e-d42f-4f71-a86a-e11713d884f6.log (1006 bytes)
10152023-11-30T02:17:11.379Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_60aaebf6-c299-451f-b222-8836faa6e402/root/var/svc/log/oxide-crucible-downstairs:downstairs-78ed061e-d42f-4f71-a86a-e11713d884f6.log
10162023-11-30T02:17:11.404Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_60aaebf6-c299-451f-b222-8836faa6e402/root/var/svc/log/oxide-crucible-downstairs:downstairs-78ed061e-d42f-4f71-a86a-e11713d884f6.log.0 (4529 bytes)
10172023-11-30T02:17:12.477Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_60aaebf6-c299-451f-b222-8836faa6e402/root/var/svc/log/oxide-crucible-downstairs:downstairs-78ed061e-d42f-4f71-a86a-e11713d884f6.log.0
10182023-11-30T02:17:12.477Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_60aaebf6-c299-451f-b222-8836faa6e402/root/var/svc/log/oxide-crucible-downstairs:downstairs-bac590fc-6ecc-4714-8cfa-1cb84a02667e.log (0 bytes)
10192023-11-30T02:17:13.550Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_60aaebf6-c299-451f-b222-8836faa6e402/root/var/svc/log/oxide-crucible-downstairs:downstairs-bac590fc-6ecc-4714-8cfa-1cb84a02667e.log
10202023-11-30T02:17:13.550Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_60aaebf6-c299-451f-b222-8836faa6e402/root/var/svc/log/oxide-crucible-downstairs:downstairs-bac590fc-6ecc-4714-8cfa-1cb84a02667e.log.0 (3051 bytes)
10212023-11-30T02:17:14.648Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_60aaebf6-c299-451f-b222-8836faa6e402/root/var/svc/log/oxide-crucible-downstairs:downstairs-bac590fc-6ecc-4714-8cfa-1cb84a02667e.log.0
10222023-11-30T02:17:14.673Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_745d7582-8c6c-4bd7-996c-e840d668fbb6/root/var/svc/log/oxide-crucible-agent:default.log (4787 bytes)
10232023-11-30T02:17:15.745Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_745d7582-8c6c-4bd7-996c-e840d668fbb6/root/var/svc/log/oxide-crucible-agent:default.log
10242023-11-30T02:17:15.745Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_745d7582-8c6c-4bd7-996c-e840d668fbb6/root/var/svc/log/oxide-crucible-agent:default.log.0 (19614 bytes)
10252023-11-30T02:17:16.903Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_745d7582-8c6c-4bd7-996c-e840d668fbb6/root/var/svc/log/oxide-crucible-agent:default.log.0
10262023-11-30T02:17:16.928Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_745d7582-8c6c-4bd7-996c-e840d668fbb6/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
10272023-11-30T02:17:17.979Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_745d7582-8c6c-4bd7-996c-e840d668fbb6/root/var/svc/log/oxide-crucible-downstairs:default.log
10282023-11-30T02:17:17.979Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_745d7582-8c6c-4bd7-996c-e840d668fbb6/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
10292023-11-30T02:17:19.076Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_745d7582-8c6c-4bd7-996c-e840d668fbb6/root/var/svc/log/oxide-crucible-downstairs:default.log.0
10302023-11-30T02:17:19.076Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_745d7582-8c6c-4bd7-996c-e840d668fbb6/root/var/svc/log/oxide-crucible-downstairs:downstairs-112673fb-1c1b-4158-8f78-76325da3e598.log (1006 bytes)
10312023-11-30T02:17:20.174Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_745d7582-8c6c-4bd7-996c-e840d668fbb6/root/var/svc/log/oxide-crucible-downstairs:downstairs-112673fb-1c1b-4158-8f78-76325da3e598.log
10322023-11-30T02:17:20.174Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_745d7582-8c6c-4bd7-996c-e840d668fbb6/root/var/svc/log/oxide-crucible-downstairs:downstairs-112673fb-1c1b-4158-8f78-76325da3e598.log.0 (4528 bytes)
10332023-11-30T02:17:21.271Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_745d7582-8c6c-4bd7-996c-e840d668fbb6/root/var/svc/log/oxide-crucible-downstairs:downstairs-112673fb-1c1b-4158-8f78-76325da3e598.log.0
10342023-11-30T02:17:21.271Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_745d7582-8c6c-4bd7-996c-e840d668fbb6/root/var/svc/log/oxide-crucible-downstairs:downstairs-d04796d9-29c7-4b49-a18d-67c67f0b0944.log (0 bytes)
10352023-11-30T02:17:22.345Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_745d7582-8c6c-4bd7-996c-e840d668fbb6/root/var/svc/log/oxide-crucible-downstairs:downstairs-d04796d9-29c7-4b49-a18d-67c67f0b0944.log
10362023-11-30T02:17:22.345Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_745d7582-8c6c-4bd7-996c-e840d668fbb6/root/var/svc/log/oxide-crucible-downstairs:downstairs-d04796d9-29c7-4b49-a18d-67c67f0b0944.log.0 (3051 bytes)
10372023-11-30T02:17:23.443Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_745d7582-8c6c-4bd7-996c-e840d668fbb6/root/var/svc/log/oxide-crucible-downstairs:downstairs-d04796d9-29c7-4b49-a18d-67c67f0b0944.log.0
10382023-11-30T02:17:23.443Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_a893abd5-87b7-4c3a-b16e-59fffd03946b/root/var/svc/log/oxide-crucible-agent:default.log (0 bytes)
10392023-11-30T02:17:24.516Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_a893abd5-87b7-4c3a-b16e-59fffd03946b/root/var/svc/log/oxide-crucible-agent:default.log
10402023-11-30T02:17:24.516Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_a893abd5-87b7-4c3a-b16e-59fffd03946b/root/var/svc/log/oxide-crucible-agent:default.log.0 (3436 bytes)
10412023-11-30T02:17:25.613Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_a893abd5-87b7-4c3a-b16e-59fffd03946b/root/var/svc/log/oxide-crucible-agent:default.log.0
10422023-11-30T02:17:25.613Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_a893abd5-87b7-4c3a-b16e-59fffd03946b/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
10432023-11-30T02:17:26.686Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_a893abd5-87b7-4c3a-b16e-59fffd03946b/root/var/svc/log/oxide-crucible-downstairs:default.log
10442023-11-30T02:17:26.710Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_a893abd5-87b7-4c3a-b16e-59fffd03946b/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
10452023-11-30T02:17:27.784Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_a893abd5-87b7-4c3a-b16e-59fffd03946b/root/var/svc/log/oxide-crucible-downstairs:default.log.0
10462023-11-30T02:17:27.784Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_90d09110-8766-4c74-ab90-22845880fcb6/root/var/svc/log/oxide-nexus:default.log (137257 bytes)
10472023-11-30T02:17:27.970Zupload warning: file "/pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_90d09110-8766-4c74-ab90-22845880fcb6/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 137257 -> 142630
10482023-11-30T02:17:29.043Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_90d09110-8766-4c74-ab90-22845880fcb6/root/var/svc/log/oxide-nexus:default.log
10492023-11-30T02:17:29.043Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_90d09110-8766-4c74-ab90-22845880fcb6/root/var/svc/log/oxide-nexus:default.log.0 (385961 bytes)
10502023-11-30T02:17:30.316Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_90d09110-8766-4c74-ab90-22845880fcb6/root/var/svc/log/oxide-nexus:default.log.0
10512023-11-30T02:17:30.316Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_clickhouse_d3dd5f84-8ffa-434f-973a-e7551aa7a1da/root/var/svc/log/oxide-clickhouse:default.log (322550 bytes)
10522023-11-30T02:17:30.554Zupload warning: file "/pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_clickhouse_d3dd5f84-8ffa-434f-973a-e7551aa7a1da/root/var/svc/log/oxide-clickhouse:default.log" changed size mid upload: 322550 -> 422304
10532023-11-30T02:17:31.626Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_clickhouse_d3dd5f84-8ffa-434f-973a-e7551aa7a1da/root/var/svc/log/oxide-clickhouse:default.log
10542023-11-30T02:17:31.627Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_clickhouse_d3dd5f84-8ffa-434f-973a-e7551aa7a1da/root/var/svc/log/oxide-clickhouse:default.log.0 (320401 bytes)
10552023-11-30T02:17:32.908Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_clickhouse_d3dd5f84-8ffa-434f-973a-e7551aa7a1da/root/var/svc/log/oxide-clickhouse:default.log.0
10562023-11-30T02:17:32.931Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_internal_dns_72883b6a-c0d8-473d-a6d8-bfe6130c5b25/root/var/svc/log/oxide-internal_dns:default.log (12551 bytes)
10572023-11-30T02:17:34.441Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_internal_dns_72883b6a-c0d8-473d-a6d8-bfe6130c5b25/root/var/svc/log/oxide-internal_dns:default.log
10582023-11-30T02:17:34.441Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_internal_dns_72883b6a-c0d8-473d-a6d8-bfe6130c5b25/root/var/svc/log/oxide-internal_dns:default.log.0 (101630 bytes)
10592023-11-30T02:17:35.656Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_internal_dns_72883b6a-c0d8-473d-a6d8-bfe6130c5b25/root/var/svc/log/oxide-internal_dns:default.log.0
10602023-11-30T02:17:35.692Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_pantry_0b918206-3dc1-40b7-a725-dcfc4fcce8d4/root/var/svc/log/oxide-crucible-pantry:default.log (0 bytes)
10612023-11-30T02:17:36.741Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_pantry_0b918206-3dc1-40b7-a725-dcfc4fcce8d4/root/var/svc/log/oxide-crucible-pantry:default.log
10622023-11-30T02:17:36.771Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_pantry_0b918206-3dc1-40b7-a725-dcfc4fcce8d4/root/var/svc/log/oxide-crucible-pantry:default.log.0 (2338 bytes)
10632023-11-30T02:17:37.849Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_pantry_0b918206-3dc1-40b7-a725-dcfc4fcce8d4/root/var/svc/log/oxide-crucible-pantry:default.log.0
10642023-11-30T02:17:37.849Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_nexus_39f17bc0-ce39-490a-a333-8c48d29a9610/root/var/svc/log/oxide-nexus:default.log (1356256 bytes)
10652023-11-30T02:17:38.163Zupload warning: file "/pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_nexus_39f17bc0-ce39-490a-a333-8c48d29a9610/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 1356256 -> 1362241
10662023-11-30T02:17:39.232Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_nexus_39f17bc0-ce39-490a-a333-8c48d29a9610/root/var/svc/log/oxide-nexus:default.log
10672023-11-30T02:17:39.232Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_nexus_39f17bc0-ce39-490a-a333-8c48d29a9610/root/var/svc/log/oxide-nexus:default.log.0 (3385386 bytes)
10682023-11-30T02:17:41.977Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_nexus_39f17bc0-ce39-490a-a333-8c48d29a9610/root/var/svc/log/oxide-nexus:default.log.0
10692023-11-30T02:17:42.002Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_oximeter_83ddb671-4afb-48de-87d5-6fa1cfee8ea7/root/var/svc/log/oxide-oximeter:default.log (94948 bytes)
10702023-11-30T02:17:42.156Zupload warning: file "/pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_oximeter_83ddb671-4afb-48de-87d5-6fa1cfee8ea7/root/var/svc/log/oxide-oximeter:default.log" changed size mid upload: 94948 -> 130104
10712023-11-30T02:17:43.225Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_oximeter_83ddb671-4afb-48de-87d5-6fa1cfee8ea7/root/var/svc/log/oxide-oximeter:default.log
10722023-11-30T02:17:43.250Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_oximeter_83ddb671-4afb-48de-87d5-6fa1cfee8ea7/root/var/svc/log/oxide-oximeter:default.log.0 (73443 bytes)
10732023-11-30T02:17:44.407Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_oximeter_83ddb671-4afb-48de-87d5-6fa1cfee8ea7/root/var/svc/log/oxide-oximeter:default.log.0
10742023-11-30T02:17:44.407Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_3dc23f99-3428-44c0-9c35-e18c35053b3e/root/var/svc/log/oxide-crucible-agent:default.log (0 bytes)
10752023-11-30T02:17:45.477Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_3dc23f99-3428-44c0-9c35-e18c35053b3e/root/var/svc/log/oxide-crucible-agent:default.log
10762023-11-30T02:17:45.477Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_3dc23f99-3428-44c0-9c35-e18c35053b3e/root/var/svc/log/oxide-crucible-agent:default.log.0 (3435 bytes)
10772023-11-30T02:17:46.584Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_3dc23f99-3428-44c0-9c35-e18c35053b3e/root/var/svc/log/oxide-crucible-agent:default.log.0
10782023-11-30T02:17:46.584Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_3dc23f99-3428-44c0-9c35-e18c35053b3e/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
10792023-11-30T02:17:47.655Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_3dc23f99-3428-44c0-9c35-e18c35053b3e/root/var/svc/log/oxide-crucible-downstairs:default.log
10802023-11-30T02:17:47.655Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_3dc23f99-3428-44c0-9c35-e18c35053b3e/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
10812023-11-30T02:17:48.749Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_3dc23f99-3428-44c0-9c35-e18c35053b3e/root/var/svc/log/oxide-crucible-downstairs:default.log.0
10822023-11-30T02:17:48.749Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_5afb4dca-e7aa-436e-9de9-dacfe7ec84b4/root/var/svc/log/oxide-crucible-agent:default.log (0 bytes)
10832023-11-30T02:17:49.818Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_5afb4dca-e7aa-436e-9de9-dacfe7ec84b4/root/var/svc/log/oxide-crucible-agent:default.log
10842023-11-30T02:17:49.854Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_5afb4dca-e7aa-436e-9de9-dacfe7ec84b4/root/var/svc/log/oxide-crucible-agent:default.log.0 (3436 bytes)
10852023-11-30T02:17:50.923Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_5afb4dca-e7aa-436e-9de9-dacfe7ec84b4/root/var/svc/log/oxide-crucible-agent:default.log.0
10862023-11-30T02:17:50.948Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_5afb4dca-e7aa-436e-9de9-dacfe7ec84b4/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
10872023-11-30T02:17:51.993Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_5afb4dca-e7aa-436e-9de9-dacfe7ec84b4/root/var/svc/log/oxide-crucible-downstairs:default.log
10882023-11-30T02:17:51.993Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_5afb4dca-e7aa-436e-9de9-dacfe7ec84b4/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
10892023-11-30T02:17:53.092Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_5afb4dca-e7aa-436e-9de9-dacfe7ec84b4/root/var/svc/log/oxide-crucible-downstairs:default.log.0
10902023-11-30T02:17:53.117Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_internal_dns_adc4268d-8f00-4216-beff-a70ffb352bfe/root/var/svc/log/oxide-internal_dns:default.log (7510 bytes)
10912023-11-30T02:17:53.163Zupload warning: file "/pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_internal_dns_adc4268d-8f00-4216-beff-a70ffb352bfe/root/var/svc/log/oxide-internal_dns:default.log" changed size mid upload: 7510 -> 11265
10922023-11-30T02:17:54.236Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_internal_dns_adc4268d-8f00-4216-beff-a70ffb352bfe/root/var/svc/log/oxide-internal_dns:default.log
10932023-11-30T02:17:54.236Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_internal_dns_adc4268d-8f00-4216-beff-a70ffb352bfe/root/var/svc/log/oxide-internal_dns:default.log.0 (109913 bytes)
10942023-11-30T02:17:55.469Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_internal_dns_adc4268d-8f00-4216-beff-a70ffb352bfe/root/var/svc/log/oxide-internal_dns:default.log.0
10952023-11-30T02:17:55.493Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_cockroachdb_2da54815-b23e-48cb-a757-30d5e295abbb/root/var/svc/log/oxide-cockroachdb:default.log (0 bytes)
10962023-11-30T02:17:56.542Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_cockroachdb_2da54815-b23e-48cb-a757-30d5e295abbb/root/var/svc/log/oxide-cockroachdb:default.log
10972023-11-30T02:17:56.542Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_cockroachdb_2da54815-b23e-48cb-a757-30d5e295abbb/root/var/svc/log/oxide-cockroachdb:default.log.0 (5343 bytes)
10982023-11-30T02:17:57.643Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_cockroachdb_2da54815-b23e-48cb-a757-30d5e295abbb/root/var/svc/log/oxide-cockroachdb:default.log.0
10992023-11-30T02:17:57.643Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_cockroachdb_ce2c88c6-5627-4a58-9d48-a861118ef27d/root/var/svc/log/oxide-cockroachdb:default.log (0 bytes)
11002023-11-30T02:17:58.715Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_cockroachdb_ce2c88c6-5627-4a58-9d48-a861118ef27d/root/var/svc/log/oxide-cockroachdb:default.log
11012023-11-30T02:17:58.715Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_cockroachdb_ce2c88c6-5627-4a58-9d48-a861118ef27d/root/var/svc/log/oxide-cockroachdb:default.log.0 (5370 bytes)
11022023-11-30T02:17:59.812Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_cockroachdb_ce2c88c6-5627-4a58-9d48-a861118ef27d/root/var/svc/log/oxide-cockroachdb:default.log.0
11032023-11-30T02:17:59.812Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_03e50aef-02a7-4d28-aaa7-7ffbcb9caa4e/root/var/svc/log/oxide-crucible-agent:default.log (0 bytes)
11042023-11-30T02:18:00.886Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_03e50aef-02a7-4d28-aaa7-7ffbcb9caa4e/root/var/svc/log/oxide-crucible-agent:default.log
11052023-11-30T02:18:00.886Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_03e50aef-02a7-4d28-aaa7-7ffbcb9caa4e/root/var/svc/log/oxide-crucible-agent:default.log.0 (3436 bytes)
11062023-11-30T02:18:01.987Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_03e50aef-02a7-4d28-aaa7-7ffbcb9caa4e/root/var/svc/log/oxide-crucible-agent:default.log.0
11072023-11-30T02:18:01.987Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_03e50aef-02a7-4d28-aaa7-7ffbcb9caa4e/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
11082023-11-30T02:18:03.062Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_03e50aef-02a7-4d28-aaa7-7ffbcb9caa4e/root/var/svc/log/oxide-crucible-downstairs:default.log
11092023-11-30T02:18:03.062Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_03e50aef-02a7-4d28-aaa7-7ffbcb9caa4e/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
11102023-11-30T02:18:04.160Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_03e50aef-02a7-4d28-aaa7-7ffbcb9caa4e/root/var/svc/log/oxide-crucible-downstairs:default.log.0
11112023-11-30T02:18:04.160Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_1797e703-1973-41e1-b972-c226d7711c1b/root/var/svc/log/oxide-crucible-agent:default.log (0 bytes)
11122023-11-30T02:18:05.234Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_1797e703-1973-41e1-b972-c226d7711c1b/root/var/svc/log/oxide-crucible-agent:default.log
11132023-11-30T02:18:05.234Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_1797e703-1973-41e1-b972-c226d7711c1b/root/var/svc/log/oxide-crucible-agent:default.log.0 (3436 bytes)
11142023-11-30T02:18:06.806Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_1797e703-1973-41e1-b972-c226d7711c1b/root/var/svc/log/oxide-crucible-agent:default.log.0
11152023-11-30T02:18:06.806Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_1797e703-1973-41e1-b972-c226d7711c1b/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
11162023-11-30T02:18:07.879Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_1797e703-1973-41e1-b972-c226d7711c1b/root/var/svc/log/oxide-crucible-downstairs:default.log
11172023-11-30T02:18:07.879Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_1797e703-1973-41e1-b972-c226d7711c1b/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
11182023-11-30T02:18:08.975Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_1797e703-1973-41e1-b972-c226d7711c1b/root/var/svc/log/oxide-crucible-downstairs:default.log.0
11192023-11-30T02:18:08.975Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_905df8f1-ba2f-44d0-999b-98b9d0111b4a/root/var/svc/log/oxide-crucible-agent:default.log (4789 bytes)
11202023-11-30T02:18:10.074Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_905df8f1-ba2f-44d0-999b-98b9d0111b4a/root/var/svc/log/oxide-crucible-agent:default.log
11212023-11-30T02:18:10.098Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_905df8f1-ba2f-44d0-999b-98b9d0111b4a/root/var/svc/log/oxide-crucible-agent:default.log.0 (9366 bytes)
11222023-11-30T02:18:11.193Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_905df8f1-ba2f-44d0-999b-98b9d0111b4a/root/var/svc/log/oxide-crucible-agent:default.log.0
11232023-11-30T02:18:11.217Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_905df8f1-ba2f-44d0-999b-98b9d0111b4a/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
11242023-11-30T02:18:12.266Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_905df8f1-ba2f-44d0-999b-98b9d0111b4a/root/var/svc/log/oxide-crucible-downstairs:default.log
11252023-11-30T02:18:12.290Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_905df8f1-ba2f-44d0-999b-98b9d0111b4a/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
11262023-11-30T02:18:13.363Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_905df8f1-ba2f-44d0-999b-98b9d0111b4a/root/var/svc/log/oxide-crucible-downstairs:default.log.0
11272023-11-30T02:18:13.387Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_905df8f1-ba2f-44d0-999b-98b9d0111b4a/root/var/svc/log/oxide-crucible-downstairs:downstairs-1a2854ab-7297-4b35-abef-5fc55638d3c7.log (1006 bytes)
11282023-11-30T02:18:14.460Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_905df8f1-ba2f-44d0-999b-98b9d0111b4a/root/var/svc/log/oxide-crucible-downstairs:downstairs-1a2854ab-7297-4b35-abef-5fc55638d3c7.log
11292023-11-30T02:18:14.460Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_905df8f1-ba2f-44d0-999b-98b9d0111b4a/root/var/svc/log/oxide-crucible-downstairs:downstairs-1a2854ab-7297-4b35-abef-5fc55638d3c7.log.0 (4529 bytes)
11302023-11-30T02:18:15.556Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_905df8f1-ba2f-44d0-999b-98b9d0111b4a/root/var/svc/log/oxide-crucible-downstairs:downstairs-1a2854ab-7297-4b35-abef-5fc55638d3c7.log.0
11312023-11-30T02:18:15.582Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_pantry_24a9b919-8742-409a-93ab-76721bc39e56/root/var/svc/log/oxide-crucible-pantry:default.log (0 bytes)
11322023-11-30T02:18:16.722Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_pantry_24a9b919-8742-409a-93ab-76721bc39e56/root/var/svc/log/oxide-crucible-pantry:default.log
11332023-11-30T02:18:16.748Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_pantry_24a9b919-8742-409a-93ab-76721bc39e56/root/var/svc/log/oxide-crucible-pantry:default.log.0 (2338 bytes)
11342023-11-30T02:18:17.829Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_pantry_24a9b919-8742-409a-93ab-76721bc39e56/root/var/svc/log/oxide-crucible-pantry:default.log.0
11352023-11-30T02:18:17.856Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_external_dns_938efc1d-6f3e-4294-916b-2a14f7ec303f/root/var/svc/log/oxide-external_dns:default.log (7553 bytes)
11362023-11-30T02:18:17.924Zupload warning: file "/pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_external_dns_938efc1d-6f3e-4294-916b-2a14f7ec303f/root/var/svc/log/oxide-external_dns:default.log" changed size mid upload: 7553 -> 11331
11372023-11-30T02:18:18.998Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_external_dns_938efc1d-6f3e-4294-916b-2a14f7ec303f/root/var/svc/log/oxide-external_dns:default.log
11382023-11-30T02:18:18.998Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_external_dns_938efc1d-6f3e-4294-916b-2a14f7ec303f/root/var/svc/log/oxide-external_dns:default.log.0 (8060 bytes)
11392023-11-30T02:18:20.119Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_external_dns_938efc1d-6f3e-4294-916b-2a14f7ec303f/root/var/svc/log/oxide-external_dns:default.log.0
11402023-11-30T02:18:20.143Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_613fed58-7436-4453-aab0-d83f9d304062/root/var/svc/log/oxide-cockroachdb:default.log (0 bytes)
11412023-11-30T02:18:21.192Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_613fed58-7436-4453-aab0-d83f9d304062/root/var/svc/log/oxide-cockroachdb:default.log
11422023-11-30T02:18:21.218Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_613fed58-7436-4453-aab0-d83f9d304062/root/var/svc/log/oxide-cockroachdb:default.log.0 (5368 bytes)
11432023-11-30T02:18:22.297Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_613fed58-7436-4453-aab0-d83f9d304062/root/var/svc/log/oxide-cockroachdb:default.log.0
11442023-11-30T02:18:22.297Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_92358cf0-a99a-4f22-b386-4987883ab4d8/root/var/svc/log/oxide-external_dns:default.log (7554 bytes)
11452023-11-30T02:18:22.391Zupload warning: file "/pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_92358cf0-a99a-4f22-b386-4987883ab4d8/root/var/svc/log/oxide-external_dns:default.log" changed size mid upload: 7554 -> 11332
11462023-11-30T02:18:23.463Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_92358cf0-a99a-4f22-b386-4987883ab4d8/root/var/svc/log/oxide-external_dns:default.log
11472023-11-30T02:18:23.463Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_92358cf0-a99a-4f22-b386-4987883ab4d8/root/var/svc/log/oxide-external_dns:default.log.0 (8061 bytes)
11482023-11-30T02:18:24.596Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_92358cf0-a99a-4f22-b386-4987883ab4d8/root/var/svc/log/oxide-external_dns:default.log.0
11492023-11-30T02:18:24.596Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_44e366d2-5b45-468f-95ec-0531a83b28fa/root/var/svc/log/oxide-nexus:default.log (147977 bytes)
11502023-11-30T02:18:25.121Zupload warning: file "/pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_44e366d2-5b45-468f-95ec-0531a83b28fa/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 147977 -> 225872
11512023-11-30T02:18:26.194Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_44e366d2-5b45-468f-95ec-0531a83b28fa/root/var/svc/log/oxide-nexus:default.log
11522023-11-30T02:18:26.194Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_44e366d2-5b45-468f-95ec-0531a83b28fa/root/var/svc/log/oxide-nexus:default.log.0 (402861 bytes)
11532023-11-30T02:18:27.983Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_44e366d2-5b45-468f-95ec-0531a83b28fa/root/var/svc/log/oxide-nexus:default.log.0
11542023-11-30T02:18:27.983Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_38b10f0a-4607-4870-a143-d157c535ffd4/root/var/svc/log/oxide-ntp:default.log (0 bytes)
11552023-11-30T02:18:29.057Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_38b10f0a-4607-4870-a143-d157c535ffd4/root/var/svc/log/oxide-ntp:default.log
11562023-11-30T02:18:29.057Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_38b10f0a-4607-4870-a143-d157c535ffd4/root/var/svc/log/oxide-ntp:default.log.0 (1185 bytes)
11572023-11-30T02:18:30.166Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_38b10f0a-4607-4870-a143-d157c535ffd4/root/var/svc/log/oxide-ntp:default.log.0
11582023-11-30T02:18:30.166Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_11bdfaaa-5ee5-4910-801e-a6710a47d9b4/root/var/svc/log/oxide-crucible-agent:default.log (0 bytes)
11592023-11-30T02:18:31.240Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_11bdfaaa-5ee5-4910-801e-a6710a47d9b4/root/var/svc/log/oxide-crucible-agent:default.log
11602023-11-30T02:18:31.240Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_11bdfaaa-5ee5-4910-801e-a6710a47d9b4/root/var/svc/log/oxide-crucible-agent:default.log.0 (14116 bytes)
11612023-11-30T02:18:32.383Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_11bdfaaa-5ee5-4910-801e-a6710a47d9b4/root/var/svc/log/oxide-crucible-agent:default.log.0
11622023-11-30T02:18:32.383Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_11bdfaaa-5ee5-4910-801e-a6710a47d9b4/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
11632023-11-30T02:18:33.456Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_11bdfaaa-5ee5-4910-801e-a6710a47d9b4/root/var/svc/log/oxide-crucible-downstairs:default.log
11642023-11-30T02:18:33.456Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_11bdfaaa-5ee5-4910-801e-a6710a47d9b4/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
11652023-11-30T02:18:34.565Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_11bdfaaa-5ee5-4910-801e-a6710a47d9b4/root/var/svc/log/oxide-crucible-downstairs:default.log.0
11662023-11-30T02:18:34.565Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_11bdfaaa-5ee5-4910-801e-a6710a47d9b4/root/var/svc/log/oxide-crucible-downstairs:downstairs-902d1e32-0a26-46fb-be5e-d5bf9093f6e9.log (0 bytes)
11672023-11-30T02:18:35.637Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_11bdfaaa-5ee5-4910-801e-a6710a47d9b4/root/var/svc/log/oxide-crucible-downstairs:downstairs-902d1e32-0a26-46fb-be5e-d5bf9093f6e9.log
11682023-11-30T02:18:35.637Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_11bdfaaa-5ee5-4910-801e-a6710a47d9b4/root/var/svc/log/oxide-crucible-downstairs:downstairs-902d1e32-0a26-46fb-be5e-d5bf9093f6e9.log.0 (3051 bytes)
11692023-11-30T02:18:36.731Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_11bdfaaa-5ee5-4910-801e-a6710a47d9b4/root/var/svc/log/oxide-crucible-downstairs:downstairs-902d1e32-0a26-46fb-be5e-d5bf9093f6e9.log.0
11702023-11-30T02:18:36.756Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_af688b74-7c0e-4a8c-8d6c-6862488cf881/root/var/svc/log/oxide-crucible-pantry:default.log (0 bytes)
11712023-11-30T02:18:37.800Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_af688b74-7c0e-4a8c-8d6c-6862488cf881/root/var/svc/log/oxide-crucible-pantry:default.log
11722023-11-30T02:18:37.800Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_af688b74-7c0e-4a8c-8d6c-6862488cf881/root/var/svc/log/oxide-crucible-pantry:default.log.0 (2332 bytes)
11732023-11-30T02:18:38.898Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_af688b74-7c0e-4a8c-8d6c-6862488cf881/root/var/svc/log/oxide-crucible-pantry:default.log.0
11742023-11-30T02:18:38.898Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_38b10f0a-4607-4870-a143-d157c535ffd4/root/var/log/chrony/measurements.log (20139 bytes)
11752023-11-30T02:18:39.017Zupload warning: file "/pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_38b10f0a-4607-4870-a143-d157c535ffd4/root/var/log/chrony/measurements.log" changed size mid upload: 20139 -> 23153
11762023-11-30T02:18:40.089Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_38b10f0a-4607-4870-a143-d157c535ffd4/root/var/log/chrony/measurements.log
11772023-11-30T02:18:40.091Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_38b10f0a-4607-4870-a143-d157c535ffd4/root/var/log/chrony/statistics.log (13338 bytes)
11782023-11-30T02:18:40.185Zupload warning: file "/pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_38b10f0a-4607-4870-a143-d157c535ffd4/root/var/log/chrony/statistics.log" changed size mid upload: 13338 -> 15912
11792023-11-30T02:18:41.259Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_38b10f0a-4607-4870-a143-d157c535ffd4/root/var/log/chrony/statistics.log
11802023-11-30T02:18:41.259Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_38b10f0a-4607-4870-a143-d157c535ffd4/root/var/log/chrony/tracking.log (2118 bytes)
11812023-11-30T02:18:41.308Zupload warning: file "/pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_38b10f0a-4607-4870-a143-d157c535ffd4/root/var/log/chrony/tracking.log" changed size mid upload: 2118 -> 2250
11822023-11-30T02:18:42.381Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_38b10f0a-4607-4870-a143-d157c535ffd4/root/var/log/chrony/tracking.log