01HERK171QVJ2XM8SVJ4B1ZYE2: helios / deploy

BasicConfig {
    output_rules: [
        "%/var/svc/log/oxide-sled-agent:default.log*",
        "%/pool/ext/*/crypt/zone/oxz_*/root/var/svc/log/oxide-*.log*",
        "%/pool/ext/*/crypt/zone/oxz_*/root/var/svc/log/system-illumos-*.log*",
        "%/pool/ext/*/crypt/zone/oxz_ntp_*/root/var/log/chrony/*.log*",
        "!/pool/ext/*/crypt/zone/oxz_propolis-server_*/root/var/svc/log/*.log*",
        "%/pool/ext/*/crypt/debug/global/oxide-sled-agent:default.log.*",
        "%/pool/ext/*/crypt/debug/oxz_*/oxide-*.log.*",
        "%/pool/ext/*/crypt/debug/oxz_*/system-illumos-*.log.*",
        "!/pool/ext/*/crypt/debug/oxz_propolis-server_*/*.log.*",
    ],
    rust_toolchain: None,
    target: Some(
        "lab-2.0-opte-0.25",
    ),
    access_repos: [],
    publish: [],
    skip_clone: true,
}

Buildomat Job: 01HERK3RE9VG0HGZKPPEB1507D

Tags:

Artefacts:

Output:

SEQ GLOBAL TIME DETAILS
12023-11-09T00:03:54.943Zjob dependencies complete; ready to run (waiting for 55 m 16 s)
22023-11-09T00:20:55.538Zjob assigned to worker 01HERQ71GY9SNYXQ2BAC4N75DZ (queued for 17 m 0 s)
32023-11-09T00:20:55.809Zhttp://172.20.0.5:9969/ipxe/sock... ok
42023-11-09T00:20:55.852Zhttp://172.20.0.5:9969/os/sock/platform/i86pc/kernel/amd64/unix... ok
52023-11-09T00:20:55.895Zhttp://172.20.0.5:9969/os/sock/platform/i86pc/amd64/boot_archive... ok
62023-11-09T00:20:55.930Zhttp://172.20.0.5:9969/os/sock/platform/i86pc/amd64/boot_archive.hash... ok
72023-11-09T00:20:55.964Zhttp://172.20.0.5:9969/postboot/sock... ok
82023-11-09T00:20:55.997ZwcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U
92023-11-09T00:20:56.031Zcpu0: microcode has been updated from version 0xa001133 to 0xa0011d1
102023-11-09T00:20:56.071ZOxide Helios Version helios-2.0.22241 64-bit
112023-11-09T00:20:56.108ZConfiguring devices.
122023-11-09T00:20:56.142Z##################################################
132023-11-09T00:20:56.176Z#### ############# ##
142023-11-09T00:20:56.209Z### ### ############ ##
152023-11-09T00:20:56.242Z## ### # ## ### ## Oxide Computer Company ##
162023-11-09T00:20:56.278Z## ## ## ### # ### ##
172023-11-09T00:20:56.316Z## # ### #### #### This Station Under ##
182023-11-09T00:20:56.353Z### ### #### # ### Computer Control ##
192023-11-09T00:20:56.387Z#### #### ### ## ##
202023-11-09T00:20:56.420Z##################################################
212023-11-09T00:20:56.454ZHostname: sock
222023-11-09T00:20:56.487ZLoading smf(7) serice descriptions:
232023-11-09T00:20:56.805Zdownloading input: /input/ci-tools/work/end-to-end-tests/bootstrap-b24d35b7ab856b37.gz
242023-11-09T00:20:58.313Zdownloaded input: /input/ci-tools/work/end-to-end-tests/bootstrap-b24d35b7ab856b37.gz
252023-11-09T00:20:58.337Zdownloading input: /input/ci-tools/work/end-to-end-tests/bootstrap.gz
262023-11-09T00:21:16.502Zdownloaded input: /input/ci-tools/work/end-to-end-tests/bootstrap.gz
272023-11-09T00:21:16.527Zdownloading input: /input/ci-tools/work/end-to-end-tests/end_to_end_tests-45b3c6eea9aa5f9f.gz
282023-11-09T00:21:39.717Zdownloaded input: /input/ci-tools/work/end-to-end-tests/end_to_end_tests-45b3c6eea9aa5f9f.gz
292023-11-09T00:21:39.742Zdownloading input: /input/ci-tools/work/caboose-util.gz
302023-11-09T00:21:40.764Zdownloaded input: /input/ci-tools/work/caboose-util.gz
312023-11-09T00:21:40.789Zdownloading input: /input/ci-tools/work/tufaceous.gz
322023-11-09T00:21:43.979Zdownloaded input: /input/ci-tools/work/tufaceous.gz
332023-11-09T00:21:44.011Zdownloading input: /input/package/work/version.txt
342023-11-09T00:21:44.039Zdownloaded input: /input/package/work/version.txt
352023-11-09T00:21:44.063Zdownloading input: /input/package/work/package.tar.gz
362023-11-09T00:22:00.356Zdownloaded input: /input/package/work/package.tar.gz
372023-11-09T00:22:00.380Zdownloading input: /input/package/work/global-zone-packages.tar.gz
382023-11-09T00:22:23.690Zdownloaded input: /input/package/work/global-zone-packages.tar.gz
392023-11-09T00:22:23.714Zdownloading input: /input/package/work/trampoline-global-zone-packages.tar.gz
402023-11-09T00:22:30.505Zdownloaded input: /input/package/work/trampoline-global-zone-packages.tar.gz
412023-11-09T00:22:30.531Zdownloading input: /input/package/work/zones/clickhouse.tar.gz
422023-11-09T00:23:05.732Zdownloaded input: /input/package/work/zones/clickhouse.tar.gz
432023-11-09T00:23:05.757Zdownloading input: /input/package/work/zones/clickhouse_keeper.tar.gz
442023-11-09T00:23:39.514Zdownloaded input: /input/package/work/zones/clickhouse_keeper.tar.gz
452023-11-09T00:23:39.538Zdownloading input: /input/package/work/zones/cockroachdb.tar.gz
462023-11-09T00:24:00.546Zdownloaded input: /input/package/work/zones/cockroachdb.tar.gz
472023-11-09T00:24:00.571Zdownloading input: /input/package/work/zones/crucible-pantry.tar.gz
482023-11-09T00:24:05.437Zdownloaded input: /input/package/work/zones/crucible-pantry.tar.gz
492023-11-09T00:24:05.462Zdownloading input: /input/package/work/zones/crucible.tar.gz
502023-11-09T00:24:13.112Zdownloaded input: /input/package/work/zones/crucible.tar.gz
512023-11-09T00:24:13.136Zdownloading input: /input/package/work/zones/external-dns.tar.gz
522023-11-09T00:24:17.542Zdownloaded input: /input/package/work/zones/external-dns.tar.gz
532023-11-09T00:24:17.566Zdownloading input: /input/package/work/zones/internal-dns.tar.gz
542023-11-09T00:24:23.008Zdownloaded input: /input/package/work/zones/internal-dns.tar.gz
552023-11-09T00:24:23.032Zdownloading input: /input/package/work/zones/ntp.tar.gz
562023-11-09T00:24:23.066Zdownloaded input: /input/package/work/zones/ntp.tar.gz
572023-11-09T00:24:23.096Zdownloading input: /input/package/work/zones/omicron-gateway-asic.tar.gz
582023-11-09T00:24:27.180Zdownloaded input: /input/package/work/zones/omicron-gateway-asic.tar.gz
592023-11-09T00:24:27.204Zdownloading input: /input/package/work/zones/omicron-gateway-softnpu.tar.gz
602023-11-09T00:24:31.478Zdownloaded input: /input/package/work/zones/omicron-gateway-softnpu.tar.gz
612023-11-09T00:24:31.503Zdownloading input: /input/package/work/zones/omicron-nexus-single-sled.tar.gz
622023-11-09T00:24:44.470Zdownloaded input: /input/package/work/zones/omicron-nexus-single-sled.tar.gz
632023-11-09T00:24:44.494Zdownloading input: /input/package/work/zones/omicron-nexus.tar.gz
642023-11-09T00:24:57.947Zdownloaded input: /input/package/work/zones/omicron-nexus.tar.gz
652023-11-09T00:24:57.971Zdownloading input: /input/package/work/zones/overlay.tar.gz
662023-11-09T00:24:58.011Zdownloaded input: /input/package/work/zones/overlay.tar.gz
672023-11-09T00:24:58.035Zdownloading input: /input/package/work/zones/oximeter-collector.tar.gz
682023-11-09T00:25:01.671Zdownloaded input: /input/package/work/zones/oximeter-collector.tar.gz
692023-11-09T00:25:01.697Zdownloading input: /input/package/work/zones/propolis-server.tar.gz
702023-11-09T00:25:15.261Zdownloaded input: /input/package/work/zones/propolis-server.tar.gz
712023-11-09T00:25:15.285Zdownloading input: /input/package/work/zones/switch-asic.tar.gz
722023-11-09T00:26:04.251Zdownloaded input: /input/package/work/zones/switch-asic.tar.gz
732023-11-09T00:26:04.280Zdownloading input: /input/package/work/zones/switch-softnpu.tar.gz
742023-11-09T00:26:45.241Zdownloaded input: /input/package/work/zones/switch-softnpu.tar.gz
 
752023-11-09T00:26:45.270Zstarting task 0: "setup"
762023-11-09T00:26:45.314Z++ uname -s
772023-11-09T00:26:45.345Z+ kern=SunOS
782023-11-09T00:26:45.370Z+ case "$kern" in
792023-11-09T00:26:45.394Z+ groupadd -g 12345 build
802023-11-09T00:26:45.418Z+ useradd -u 12345 -g build -d /home/build -s /bin/bash -c build -P 'Primary Administrator' build
812023-11-09T00:26:47.317Z+ zfs create -o mountpoint=/work rpool/work
822023-11-09T00:26:47.341Z++ awk '$2 == "/home" { print $3 }' /etc/mnttab
832023-11-09T00:26:47.366Z+ home_fs=zfs
842023-11-09T00:26:47.390Z+ [[ zfs == autofs ]]
852023-11-09T00:26:47.414Z+ mkdir -p /home/build
862023-11-09T00:26:47.438Z+ chown build:build /home/build /work
872023-11-09T00:26:49.350Z+ chmod 0700 /home/build /work
882023-11-09T00:26:49.375Zprocess exited: duration 4054 ms, exit code 0
 
892023-11-09T00:26:49.425Zstarting task 1: "authentication"
902023-11-09T00:26:49.494Zprocess exited: duration 44 ms, exit code 0
 
912023-11-09T00:26:49.543Zstarting task 2: "build"
922023-11-09T00:26:49.577Z+ trap _exit_trap EXIT
932023-11-09T00:26:49.603Z+ pfexec mdb -kwe 'secpolicy_fs_linkdir/v 55 48 89 e5 b8 01 00 00 00 5d c3'
942023-11-09T00:26:49.771Zsecpolicy_fs_linkdir: 0x55 = 0x55
952023-11-09T00:26:49.801Zsecpolicy_fs_linkdir+1: 0xb8 = 0x48
962023-11-09T00:26:49.830Zsecpolicy_fs_linkdir+2: 0x1 = 0x89
972023-11-09T00:26:49.860Zsecpolicy_fs_linkdir+3: 0 = 0xe5
982023-11-09T00:26:49.889Zsecpolicy_fs_linkdir+4: 0 = 0xb8
992023-11-09T00:26:49.921Zsecpolicy_fs_linkdir+5: 0 = 0x1
1002023-11-09T00:26:49.952Zsecpolicy_fs_linkdir+6: 0x48 = 0x0
1012023-11-09T00:26:49.978Zsecpolicy_fs_linkdir+7: 0x89 = 0x0
1022023-11-09T00:26:50.006Zsecpolicy_fs_linkdir+8: 0xe5 = 0x0
1032023-11-09T00:26:50.034Zsecpolicy_fs_linkdir+9: 0x48 = 0x5d
1042023-11-09T00:26:50.062Zsecpolicy_fs_linkdir+0xa: 0x83 = 0xc3
1052023-11-09T00:26:50.092Z+ [[ -d /opt/oxide ]]
1062023-11-09T00:26:50.130Z+ pfexec mv /opt/oxide /opt/oxide-underneath
1072023-11-09T00:26:50.161Z+ pfexec mkdir /opt/oxide
1082023-11-09T00:26:50.213Z+ pfexec mount -F tmpfs -O swap /opt/oxide
1092023-11-09T00:26:50.238Z+ [[ -d /opt/oxide-underneath ]]
1102023-11-09T00:26:50.264Z+ cd /opt/oxide-underneath
1112023-11-09T00:26:50.288Z+ pfexec tar ceEp@/f - .
1122023-11-09T00:26:50.313Z+ cd /opt/oxide
1132023-11-09T00:26:50.339Z+ pfexec tar xveEp@/f -
1142023-11-09T00:26:50.363Zx ., 0 bytes, 0 tape blocks
1152023-11-09T00:26:50.387Zx ./opte, 0 bytes, 0 tape blocks
1162023-11-09T00:26:50.411Zx ./opte/bin, 0 bytes, 0 tape blocks
1172023-11-09T00:26:50.435Zx ./opte/bin/opteadm, 50714832 bytes, 99053 tape blocks
1182023-11-09T00:26:50.459Z+ pfexec rm -rf /opt/oxide-underneath
1192023-11-09T00:26:50.482Z+ pfexec /sbin/zfs create -o mountpoint=/zone rpool/zone
1202023-11-09T00:26:50.506Z+ pfexec mkdir /opt/oxide/work
1212023-11-09T00:26:50.530Z+ pfexec chown build:build /opt/oxide/work
1222023-11-09T00:26:50.559Z+ cd /opt/oxide/work
1232023-11-09T00:26:50.589Z+ ptime -m tar xvzf /input/package/work/package.tar.gz
1242023-11-09T00:26:51.546ZDecompressing '/input/package/work/package.tar.gz' with '/usr/bin/gzcat'...
1252023-11-09T00:26:51.571Zx out/mg-ddm-gz.tar, 62974464 bytes, 122997 tape blocks
1262023-11-09T00:26:51.595Zx out/omicron-sled-agent.tar, 123651584 bytes, 241507 tape blocks
1272023-11-09T00:26:51.619Zx out/target/test, 75 bytes, 1 tape blocks
1282023-11-09T00:26:51.643Zx out/npuzone/npuzone, 37381232 bytes, 73011 tape blocks
1292023-11-09T00:26:51.667Zx package-manifest.toml, 22037 bytes, 44 tape blocks
1302023-11-09T00:26:51.691Zx smf/sled-agent/non-gimlet/config.toml, 3102 bytes, 7 tape blocks
1312023-11-09T00:26:51.717Zx target/release/omicron-package, 42419472 bytes, 82851 tape blocks
1322023-11-09T00:26:51.743Zx tools/create_virtual_hardware.sh, 3067 bytes, 6 tape blocks
1332023-11-09T00:26:51.772Zx tools/virtual_hardware.sh, 3408 bytes, 7 tape blocks
1342023-11-09T00:26:51.806Zx tools/scrimlet/create-softnpu-zone.sh, 619 bytes, 2 tape blocks
1352023-11-09T00:26:51.837Zx tools/scrimlet/destroy-softnpu-zone.sh, 195 bytes, 1 tape blocks
1362023-11-09T00:26:51.903Zx tools/scrimlet/softnpu-init.sh, 2148 bytes, 5 tape blocks
1372023-11-09T00:26:51.933Zx tools/scrimlet/softnpu-zone.txt, 330 bytes, 1 tape blocks
1382023-11-09T00:26:51.965Zx tools/scrimlet/softnpu.toml, 171 bytes, 1 tape blocks
1392023-11-09T00:26:51.995Z
1402023-11-09T00:26:52.029Zreal 1.611276416
1412023-11-09T00:26:52.058Zuser 1.451851343
1422023-11-09T00:26:52.085Zsys 0.281419743
1432023-11-09T00:26:52.114Ztrap 0.000064302
1442023-11-09T00:26:52.144Ztflt 0.000000000
1452023-11-09T00:26:52.173Zdflt 0.000000000
1462023-11-09T00:26:52.204Zkflt 0.000000000
1472023-11-09T00:26:52.233Zlock 0.000000000
1482023-11-09T00:26:52.292Zslp 1.449314526
1492023-11-09T00:26:52.323Zlat 0.038001032
1502023-11-09T00:26:52.356Zstop 0.000118743
1512023-11-09T00:26:52.391Z+ cp /input/package/work/zones/clickhouse_keeper.tar.gz /input/package/work/zones/clickhouse.tar.gz /input/package/work/zones/cockroachdb.tar.gz /input/package/work/zones/crucible-pantry.tar.gz /input/package/work/zones/crucible.tar.gz /input/package/work/zones/external-dns.tar.gz /input/package/work/zones/internal-dns.tar.gz /input/package/work/zones/ntp.tar.gz /input/package/work/zones/omicron-gateway-asic.tar.gz /input/package/work/zones/omicron-gateway-softnpu.tar.gz /input/package/work/zones/omicron-nexus-single-sled.tar.gz /input/package/work/zones/omicron-nexus.tar.gz /input/package/work/zones/overlay.tar.gz /input/package/work/zones/oximeter-collector.tar.gz /input/package/work/zones/propolis-server.tar.gz /input/package/work/zones/switch-asic.tar.gz /input/package/work/zones/switch-softnpu.tar.gz out/
1522023-11-09T00:26:52.609Z+ mv out/omicron-nexus-single-sled.tar.gz out/omicron-nexus.tar.gz
1532023-11-09T00:26:52.633Z+ mkdir tests
1542023-11-09T00:26:52.660Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1552023-11-09T00:26:52.684Z+ ptime -m gunzip
1562023-11-09T00:26:52.709Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap-b24d35b7ab856b37
1572023-11-09T00:26:52.766Z
1582023-11-09T00:26:52.794Zreal 0.139699802
1592023-11-09T00:26:52.821Zuser 0.127118912
1602023-11-09T00:26:52.845Zsys 0.012337762
1612023-11-09T00:26:52.869Ztrap 0.000016290
1622023-11-09T00:26:52.893Ztflt 0.000000000
1632023-11-09T00:26:52.918Zdflt 0.000000000
1642023-11-09T00:26:52.948Zkflt 0.000000000
1652023-11-09T00:26:52.978Zlock 0.000000000
1662023-11-09T00:26:53.012Zslp 0.000020210
1672023-11-09T00:26:53.041Zlat 0.000022051
1682023-11-09T00:26:53.065Zstop 0.000127533
1692023-11-09T00:26:53.089Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap-b24d35b7ab856b37
1702023-11-09T00:26:53.117Z+ chmod a+x tests/bootstrap-b24d35b7ab856b37
1712023-11-09T00:26:53.141Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1722023-11-09T00:26:53.165Z+ ptime -m gunzip
1732023-11-09T00:26:53.190Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap
1742023-11-09T00:26:55.096Z
1752023-11-09T00:26:55.122Zreal 2.318355139
1762023-11-09T00:26:55.146Zuser 2.178326574
1772023-11-09T00:26:55.174Zsys 0.139638484
1782023-11-09T00:26:55.207Ztrap 0.000172735
1792023-11-09T00:26:55.233Ztflt 0.000000000
1802023-11-09T00:26:55.259Zdflt 0.000000000
1812023-11-09T00:26:55.284Zkflt 0.000000000
1822023-11-09T00:26:55.308Zlock 0.000000000
1832023-11-09T00:26:55.332Zslp 0.000022260
1842023-11-09T00:26:55.457Zlat 0.000025250
1852023-11-09T00:26:55.491Zstop 0.000108143
1862023-11-09T00:26:55.526Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap
1872023-11-09T00:26:55.562Z+ chmod a+x tests/bootstrap
1882023-11-09T00:26:55.599Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1892023-11-09T00:26:55.631Z+ ptime -m gunzip
1902023-11-09T00:26:55.660Z++ basename /input/ci-tools/work/end-to-end-tests/end_to_end_tests-45b3c6eea9aa5f9f
1912023-11-09T00:26:57.759Z
1922023-11-09T00:26:57.783Zreal 2.650783277
1932023-11-09T00:26:57.809Zuser 2.485390748
1942023-11-09T00:26:57.838Zsys 0.164968535
1952023-11-09T00:26:57.887Ztrap 0.000194616
1962023-11-09T00:26:57.911Ztflt 0.000000000
1972023-11-09T00:26:57.936Zdflt 0.000000000
1982023-11-09T00:26:57.961Zkflt 0.000000000
1992023-11-09T00:26:57.986Zlock 0.000000000
2002023-11-09T00:26:58.011Zslp 0.000024770
2012023-11-09T00:26:58.035Zlat 0.000018621
2022023-11-09T00:26:58.059Zstop 0.000121743
2032023-11-09T00:26:58.084Z++ basename /input/ci-tools/work/end-to-end-tests/end_to_end_tests-45b3c6eea9aa5f9f
2042023-11-09T00:26:58.529Z+ chmod a+x tests/end_to_end_tests-45b3c6eea9aa5f9f
2052023-11-09T00:26:58.557Z+ bmat address ls
2062023-11-09T00:26:58.583ZNAME CIDR FIRST LAST COUNT
2072023-11-09T00:26:58.608Zextra 10.151.2.0/24 10.151.2.164 10.151.2.227 64
2082023-11-09T00:26:58.633Z++ bmat address ls -f extra -Ho first,last,count
2092023-11-09T00:26:58.658Z+ set -- 10.151.2.164 10.151.2.227 64
2102023-11-09T00:26:58.683Z+ EXTRA_IP_START=10.151.2.164
2112023-11-09T00:26:58.721Z+ EXTRA_IP_END=10.151.2.227
2122023-11-09T00:26:58.752Z+ EXTRA_IP_COUNT=64
2132023-11-09T00:26:58.776Z+ (( EXTRA_IP_COUNT >= 32 ))
2142023-11-09T00:26:59.210Z+ EXTRA_IP_BASE=10.151.2
2152023-11-09T00:26:59.234Z+ EXTRA_IP_FOCTET=164
2162023-11-09T00:26:59.259Z+ EXTRA_IP_LOCTET=227
2172023-11-09T00:26:59.284Z+ SERVICE_IP_POOL_START=10.151.2.164
2182023-11-09T00:26:59.308Z+ SERVICE_IP_POOL_END=10.151.2.173
2192023-11-09T00:26:59.332Z+ DNS_IP1=10.151.2.164
2202023-11-09T00:26:59.358Z+ DNS_IP2=10.151.2.165
2212023-11-09T00:26:59.384Z+ UPLINK_IP=10.151.2.174
2222023-11-09T00:26:59.411Z+ IPPOOL_START=10.151.2.175
2232023-11-09T00:26:59.439Z+ IPPOOL_END=10.151.2.227
2242023-11-09T00:26:59.472Z++ ipadm show-addr -po type,addr
2252023-11-09T00:26:59.497Z++ awk '-F[:/]' '$1 == "dhcp" {print $2}'
2262023-11-09T00:26:59.523Z+ GATEWAY_IP=10.151.2.100
2272023-11-09T00:26:59.548Z+ [[ -n 10.151.2.100 ]]
2282023-11-09T00:26:59.576Z+ ping -s 10.151.2.100 56 1
2292023-11-09T00:26:59.601ZPING 10.151.2.100: 56 data bytes
2302023-11-09T00:26:59.626Z64 bytes from 10.151.2.100: icmp_seq=0. time=0.011 ms
2312023-11-09T00:26:59.651Z
2322023-11-09T00:26:59.675Z----10.151.2.100 PING Statistics----
2332023-11-09T00:26:59.700Z1 packets transmitted, 1 packets received, 0% packet loss
2342023-11-09T00:26:59.725Zround-trip (ms) min/avg/max/stddev = 0.011/0.011/0.011/-nan
2352023-11-09T00:26:59.751Z++ arp -an
2362023-11-09T00:26:59.786Z++ awk -vgw=10.151.2.100 '$2 == gw {print $NF}'
2372023-11-09T00:26:59.816Z+ GATEWAY_MAC=18:c0:4d:0d:a0:2a
2382023-11-09T00:26:59.843Z+ routeadm -e ipv4-forwarding -u
2392023-11-09T00:26:59.880Z+ PXA_START=10.151.2.164
2402023-11-09T00:26:59.997Z+ PXA_END=10.151.2.227
2412023-11-09T00:27:00.025Z+ export GATEWAY_IP GATEWAY_MAC PXA_START PXA_END
2422023-11-09T00:27:00.059Z+ pfexec zpool create -f scratch c1t1d0 c2t1d0
2432023-11-09T00:27:00.084Z+ ZPOOL_VDEV_DIR=/scratch
2442023-11-09T00:27:00.109Z+ ptime -m pfexec ./tools/create_virtual_hardware.sh
2452023-11-09T00:27:00.135Z+++ dirname ./tools/create_virtual_hardware.sh
2462023-11-09T00:27:00.160Z++ cd ./tools
2472023-11-09T00:27:00.185Z++ pwd
2482023-11-09T00:27:00.209Z+ SOURCE_DIR=/opt/oxide/work/tools
2492023-11-09T00:27:00.233Z+ OMICRON_TOP=/opt/oxide/work/tools/..
2502023-11-09T00:27:00.257Z+ SOFTNPU_MODE=zone
2512023-11-09T00:27:00.282Z+ . /opt/oxide/work/tools/virtual_hardware.sh
2522023-11-09T00:27:00.317Z++ MARKER=/etc/opt/oxide/NO_INSTALL
2532023-11-09T00:27:00.353Z++ [[ -f /etc/opt/oxide/NO_INSTALL ]]
2542023-11-09T00:27:00.389Z++ dladm show-phys -p -o LINK
2552023-11-09T00:27:00.421Z++ head -1
2562023-11-09T00:27:00.456Z+ PHYSICAL_LINK=igb0
2572023-11-09T00:27:00.492Z+ echo 'Using igb0 as physical link'
2582023-11-09T00:27:00.524ZUsing igb0 as physical link
2592023-11-09T00:27:00.561Z+ ensure_run_as_root
2602023-11-09T00:27:00.588Z++ id -u
2612023-11-09T00:27:00.613Z+ [[ 0 -ne 0 ]]
2622023-11-09T00:27:00.637Z+ ensure_zpools
2632023-11-09T00:27:00.662Z+ ZPOOL_TYPES=('oxp_' 'oxi_')
2642023-11-09T00:27:00.686Z+ for ZPOOL_TYPE in "${ZPOOL_TYPES[@]}"
2652023-11-09T00:27:00.710Z+ readarray -t ZPOOLS
2662023-11-09T00:27:00.734Z++ grep '"oxp_' /opt/oxide/work/tools/../smf/sled-agent/non-gimlet/config.toml
2672023-11-09T00:27:00.761Z++ sed 's/[ ",]//g'
2682023-11-09T00:27:00.788Z+ for ZPOOL in "${ZPOOLS[@]}"
2692023-11-09T00:27:00.818Z+ echo 'Zpool: [oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b]'
2702023-11-09T00:27:00.842ZZpool: [oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b]
2712023-11-09T00:27:00.866Z+ VDEV_PATH=/scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
2722023-11-09T00:27:00.890ZZFS vdev /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
2732023-11-09T00:27:00.914Z+ [[ -f /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
2742023-11-09T00:27:00.938ZZFS zpool oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b exists
2752023-11-09T00:27:00.963Z+ dd if=/dev/zero of=/scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
2762023-11-09T00:27:00.995ZZpool: [oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03]
2772023-11-09T00:27:01.034Z0+0 records in
2782023-11-09T00:27:01.066ZZFS vdev /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
2792023-11-09T00:27:01.094Z0+0 records out
2802023-11-09T00:27:01.123ZZFS zpool oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
2812023-11-09T00:27:01.152Z0 bytes (0 iB) transferred in 0.000005 secs (0 iB/sec)
2822023-11-09T00:27:01.181ZZpool: [oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03]
2832023-11-09T00:27:01.212Z+ success 'ZFS vdev /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
2842023-11-09T00:27:01.239ZZFS vdev /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
2852023-11-09T00:27:01.265Z+ set +x
2862023-11-09T00:27:01.290ZZFS zpool oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
2872023-11-09T00:27:01.315Z++ zpool list -o name
2882023-11-09T00:27:01.344ZZpool: [oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03]
2892023-11-09T00:27:01.372Z++ grep oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b
2902023-11-09T00:27:01.400ZZFS vdev /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
2912023-11-09T00:27:01.428Z+ [[ -z '' ]]
2922023-11-09T00:27:01.463ZZFS zpool oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
2932023-11-09T00:27:01.504Z+ zpool create -f oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
2942023-11-09T00:27:01.539ZZpool: [oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03]
2952023-11-09T00:27:01.564Z+ success 'ZFS zpool oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
2962023-11-09T00:27:01.589ZZFS vdev /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
2972023-11-09T00:27:01.613Z+ set +x
2982023-11-09T00:27:01.651ZZFS zpool oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
2992023-11-09T00:27:02.234Z+ for ZPOOL in "${ZPOOLS[@]}"
3002023-11-09T00:27:02.263ZZpool: [oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930]
3012023-11-09T00:27:02.289Z+ echo 'Zpool: [oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3022023-11-09T00:27:02.314ZZFS vdev /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev exists
3032023-11-09T00:27:02.342Z+ VDEV_PATH=/scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3042023-11-09T00:27:02.368ZZFS zpool oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 exists
3052023-11-09T00:27:02.401Z+ [[ -f /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3062023-11-09T00:27:02.437ZZpool: [oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60]
3072023-11-09T00:27:02.470Z+ dd if=/dev/zero of=/scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3082023-11-09T00:27:02.497ZZFS vdev /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev exists
3092023-11-09T00:27:02.522Z0+0 records in
3102023-11-09T00:27:02.547ZZFS zpool oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 exists
3112023-11-09T00:27:02.572Z0+0 records out
3122023-11-09T00:27:02.597ZZpool: [oxp_31bd71cd-4736-4a12-a387-9b74b050396f]
3132023-11-09T00:27:02.622Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
3142023-11-09T00:27:02.647ZZFS vdev /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev exists
3152023-11-09T00:27:02.671Z+ success 'ZFS vdev /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3162023-11-09T00:27:02.696ZZFS zpool oxp_31bd71cd-4736-4a12-a387-9b74b050396f exists
3172023-11-09T00:27:02.721Z+ set +x
3182023-11-09T00:27:02.747ZZpool: [oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7]
3192023-11-09T00:27:02.771Z++ zpool list -o name
3202023-11-09T00:27:02.796ZZFS vdev /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev exists
3212023-11-09T00:27:02.821Z++ grep oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03
3222023-11-09T00:27:02.846ZZFS zpool oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 exists
3232023-11-09T00:27:02.873Z+ [[ -z '' ]]
3242023-11-09T00:27:02.902ZZpool: [oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b]
3252023-11-09T00:27:02.931Z+ zpool create -f oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3262023-11-09T00:27:02.959ZZFS vdev /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
3272023-11-09T00:27:02.992Z+ success 'ZFS zpool oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3282023-11-09T00:27:03.465ZZFS zpool oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b exists
3292023-11-09T00:27:03.503Z+ set +x
3302023-11-09T00:27:03.528ZZpool: [oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b]
3312023-11-09T00:27:03.554Z+ for ZPOOL in "${ZPOOLS[@]}"
3322023-11-09T00:27:03.579ZZFS vdev /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
3332023-11-09T00:27:03.604Z+ echo 'Zpool: [oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3342023-11-09T00:27:03.629ZZFS zpool oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b exists
3352023-11-09T00:27:03.654Z+ VDEV_PATH=/scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3362023-11-09T00:27:03.683ZSimnet net0/sc0_0 exists
3372023-11-09T00:27:03.713Z+ [[ -f /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3382023-11-09T00:27:03.739ZSimnet net1/sc1_0 exists
3392023-11-09T00:27:03.765Z+ dd if=/dev/zero of=/scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3402023-11-09T00:27:03.794ZVnic sc0_1 exists
3412023-11-09T00:27:03.828Z0+0 records in
3422023-11-09T00:27:03.863Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/softnpu/image/c1c42398c82b0220c8b5fa3bfba9c7a3bcaa0943/softnpu.sha256.txt
3432023-11-09T00:27:03.890Z0+0 records out
3442023-11-09T00:27:03.920Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
3452023-11-09T00:27:03.949Z+ success 'ZFS vdev /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3462023-11-09T00:27:03.978Z+ set +x
3472023-11-09T00:27:04.009Z++ zpool list -o name
3482023-11-09T00:27:04.039Z++ grep oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03
3492023-11-09T00:27:04.069Z+ [[ -z '' ]]
3502023-11-09T00:27:04.104Z+ zpool create -f oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3512023-11-09T00:27:04.160Z+ success 'ZFS zpool oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3522023-11-09T00:27:04.206Z+ set +x
3532023-11-09T00:27:04.243Z+ for ZPOOL in "${ZPOOLS[@]}"
3542023-11-09T00:27:04.703Z+ echo 'Zpool: [oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3552023-11-09T00:27:04.727Z+ VDEV_PATH=/scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3562023-11-09T00:27:04.751Z+ [[ -f /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3572023-11-09T00:27:04.775Z+ dd if=/dev/zero of=/scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3582023-11-09T00:27:04.800Z0+0 records in
3592023-11-09T00:27:04.825Z0+0 records out
3602023-11-09T00:27:04.851Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
3612023-11-09T00:27:04.876Z+ success 'ZFS vdev /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3622023-11-09T00:27:04.901Z+ set +x
3632023-11-09T00:27:04.927Z++ zpool list -o name
3642023-11-09T00:27:05.367Z++ grep oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03
3652023-11-09T00:27:05.391Z+ [[ -z '' ]]
3662023-11-09T00:27:05.417Z+ zpool create -f oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3672023-11-09T00:27:05.459Z+ success 'ZFS zpool oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3682023-11-09T00:27:05.488Z+ set +x
3692023-11-09T00:27:05.515Z+ for ZPOOL in "${ZPOOLS[@]}"
3702023-11-09T00:27:05.540Z+ echo 'Zpool: [oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3712023-11-09T00:27:05.566Z+ VDEV_PATH=/scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3722023-11-09T00:27:05.593Z+ [[ -f /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3732023-11-09T00:27:05.618Z+ dd if=/dev/zero of=/scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3742023-11-09T00:27:05.644Z0+0 records in
3752023-11-09T00:27:05.668Z0+0 records out
3762023-11-09T00:27:05.695Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3772023-11-09T00:27:05.719Z+ success 'ZFS vdev /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3782023-11-09T00:27:05.744Z+ set +x
3792023-11-09T00:27:05.768Z++ zpool list -o name
3802023-11-09T00:27:05.793Z++ grep oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03
3812023-11-09T00:27:05.817Z+ [[ -z '' ]]
3822023-11-09T00:27:05.842Z+ zpool create -f oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3832023-11-09T00:27:05.866Z+ success 'ZFS zpool oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3842023-11-09T00:27:05.891Z+ set +x
3852023-11-09T00:27:05.919Z+ for ZPOOL in "${ZPOOLS[@]}"
3862023-11-09T00:27:05.945Z+ echo 'Zpool: [oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930]'
3872023-11-09T00:27:05.971Z+ VDEV_PATH=/scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev
3882023-11-09T00:27:05.996Z+ [[ -f /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev ]]
3892023-11-09T00:27:06.021Z+ dd if=/dev/zero of=/scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev bs=1 count=0 seek=20G
3902023-11-09T00:27:06.046Z0+0 records in
3912023-11-09T00:27:06.070Z0+0 records out
3922023-11-09T00:27:06.108Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
3932023-11-09T00:27:06.139Z+ success 'ZFS vdev /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev exists'
3942023-11-09T00:27:06.173Z+ set +x
3952023-11-09T00:27:06.223Z++ zpool list -o name
3962023-11-09T00:27:06.255Z++ grep oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930
3972023-11-09T00:27:06.285Z+ [[ -z '' ]]
3982023-11-09T00:27:06.312Z+ zpool create -f oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev
3992023-11-09T00:27:06.347Z+ success 'ZFS zpool oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 exists'
4002023-11-09T00:27:06.383Z+ set +x
4012023-11-09T00:27:06.421Z+ for ZPOOL in "${ZPOOLS[@]}"
4022023-11-09T00:27:06.450Z+ echo 'Zpool: [oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60]'
4032023-11-09T00:27:06.477Z+ VDEV_PATH=/scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev
4042023-11-09T00:27:06.503Z+ [[ -f /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev ]]
4052023-11-09T00:27:06.528Z+ dd if=/dev/zero of=/scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev bs=1 count=0 seek=20G
4062023-11-09T00:27:06.552Z0+0 records in
4072023-11-09T00:27:06.576Z0+0 records out
4082023-11-09T00:27:06.600Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
4092023-11-09T00:27:06.625Z+ success 'ZFS vdev /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev exists'
4102023-11-09T00:27:06.649Z+ set +x
4112023-11-09T00:27:06.673Z++ zpool list -o name
4122023-11-09T00:27:06.697Z++ grep oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60
4132023-11-09T00:27:06.730Z+ [[ -z '' ]]
4142023-11-09T00:27:06.758Z+ zpool create -f oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev
4152023-11-09T00:27:06.784Z+ success 'ZFS zpool oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 exists'
4162023-11-09T00:27:06.848Z+ set +x
4172023-11-09T00:27:06.877Z+ for ZPOOL in "${ZPOOLS[@]}"
4182023-11-09T00:27:06.905Z+ echo 'Zpool: [oxp_31bd71cd-4736-4a12-a387-9b74b050396f]'
4192023-11-09T00:27:06.933Z+ VDEV_PATH=/scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev
4202023-11-09T00:27:06.961Z+ [[ -f /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev ]]
4212023-11-09T00:27:06.990Z+ dd if=/dev/zero of=/scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev bs=1 count=0 seek=20G
4222023-11-09T00:27:07.020Z0+0 records in
4232023-11-09T00:27:07.056Z0+0 records out
4242023-11-09T00:27:07.086Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4252023-11-09T00:27:07.125Z+ success 'ZFS vdev /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev exists'
4262023-11-09T00:27:07.156Z+ set +x
4272023-11-09T00:27:07.180Z++ zpool list -o name
4282023-11-09T00:27:07.204Z++ grep oxp_31bd71cd-4736-4a12-a387-9b74b050396f
4292023-11-09T00:27:07.248Z+ [[ -z '' ]]
4302023-11-09T00:27:07.274Z+ zpool create -f oxp_31bd71cd-4736-4a12-a387-9b74b050396f /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev
4312023-11-09T00:27:07.299Z+ success 'ZFS zpool oxp_31bd71cd-4736-4a12-a387-9b74b050396f exists'
4322023-11-09T00:27:07.324Z+ set +x
4332023-11-09T00:27:07.348Z+ for ZPOOL in "${ZPOOLS[@]}"
4342023-11-09T00:27:07.372Z+ echo 'Zpool: [oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7]'
4352023-11-09T00:27:07.396Z+ VDEV_PATH=/scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev
4362023-11-09T00:27:07.420Z+ [[ -f /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev ]]
4372023-11-09T00:27:07.446Z+ dd if=/dev/zero of=/scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev bs=1 count=0 seek=20G
4382023-11-09T00:27:07.473Z0+0 records in
4392023-11-09T00:27:07.498Z0+0 records out
4402023-11-09T00:27:07.523Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4412023-11-09T00:27:07.554Z+ success 'ZFS vdev /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev exists'
4422023-11-09T00:27:07.579Z+ set +x
4432023-11-09T00:27:07.604Z++ zpool list -o name
4442023-11-09T00:27:07.631Z++ grep oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7
4452023-11-09T00:27:07.655Z+ [[ -z '' ]]
4462023-11-09T00:27:07.679Z+ zpool create -f oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev
4472023-11-09T00:27:07.709Z+ success 'ZFS zpool oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 exists'
4482023-11-09T00:27:07.734Z+ set +x
4492023-11-09T00:27:07.758Z+ for ZPOOL_TYPE in "${ZPOOL_TYPES[@]}"
4502023-11-09T00:27:07.782Z+ readarray -t ZPOOLS
4512023-11-09T00:27:07.808Z++ grep '"oxi_' /opt/oxide/work/tools/../smf/sled-agent/non-gimlet/config.toml
4522023-11-09T00:27:07.832Z++ sed 's/[ ",]//g'
4532023-11-09T00:27:07.857Z+ for ZPOOL in "${ZPOOLS[@]}"
4542023-11-09T00:27:07.881Z+ echo 'Zpool: [oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b]'
4552023-11-09T00:27:07.905Z+ VDEV_PATH=/scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4562023-11-09T00:27:07.930Z+ [[ -f /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
4572023-11-09T00:27:07.954Z+ dd if=/dev/zero of=/scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
4582023-11-09T00:27:07.978Z0+0 records in
4592023-11-09T00:27:08.002Z0+0 records out
4602023-11-09T00:27:08.027Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
4612023-11-09T00:27:08.052Z+ success 'ZFS vdev /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
4622023-11-09T00:27:08.077Z+ set +x
4632023-11-09T00:27:08.101Z++ zpool list -o name
4642023-11-09T00:27:08.131Z++ grep oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b
4652023-11-09T00:27:08.606Z+ [[ -z '' ]]
4662023-11-09T00:27:08.637Z+ zpool create -f oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4672023-11-09T00:27:08.663Z+ success 'ZFS zpool oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
4682023-11-09T00:27:08.688Z+ set +x
4692023-11-09T00:27:08.713Z+ for ZPOOL in "${ZPOOLS[@]}"
4702023-11-09T00:27:08.737Z+ echo 'Zpool: [oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b]'
4712023-11-09T00:27:08.762Z+ VDEV_PATH=/scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4722023-11-09T00:27:08.786Z+ [[ -f /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
4732023-11-09T00:27:09.218Z+ dd if=/dev/zero of=/scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
4742023-11-09T00:27:09.250Z0+0 records in
4752023-11-09T00:27:09.299Z0+0 records out
4762023-11-09T00:27:09.323Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
4772023-11-09T00:27:09.348Z+ success 'ZFS vdev /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
4782023-11-09T00:27:09.372Z+ set +x
4792023-11-09T00:27:09.397Z++ zpool list -o name
4802023-11-09T00:27:09.422Z++ grep oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b
4812023-11-09T00:27:09.447Z+ [[ -z '' ]]
4822023-11-09T00:27:09.471Z+ zpool create -f oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4832023-11-09T00:27:09.495Z+ success 'ZFS zpool oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
4842023-11-09T00:27:09.519Z+ set +x
4852023-11-09T00:27:09.543Z+ [[ zone == \z\o\n\e ]]
4862023-11-09T00:27:09.567Z+ ensure_simulated_links igb0
4872023-11-09T00:27:09.592Z+ local PHYSICAL_LINK=igb0
4882023-11-09T00:27:09.616Z+ INDICES=("0" "1")
4892023-11-09T00:27:09.641Z+ for I in "${INDICES[@]}"
4902023-11-09T00:27:09.666Z++ get_simnet_name_if_exists net0
4912023-11-09T00:27:09.691Z++ dladm show-simnet -p -o LINK net0
4922023-11-09T00:27:09.715Z++ echo ''
4932023-11-09T00:27:09.739Z+ [[ -z '' ]]
4942023-11-09T00:27:09.763Z+ dladm create-simnet -t net0
4952023-11-09T00:27:09.788Z+ dladm create-simnet -t sc0_0
4962023-11-09T00:27:09.813Z+ dladm modify-simnet -t -p net0 sc0_0
4972023-11-09T00:27:09.837Z+ dladm set-linkprop -p mtu=9000 sc0_0
4982023-11-09T00:27:09.861Z+ success 'Simnet net0/sc0_0 exists'
4992023-11-09T00:27:09.885Z+ set +x
5002023-11-09T00:27:09.910Z+ for I in "${INDICES[@]}"
5012023-11-09T00:27:09.934Z++ get_simnet_name_if_exists net1
5022023-11-09T00:27:09.958Z++ dladm show-simnet -p -o LINK net1
5032023-11-09T00:27:09.983Z++ echo ''
5042023-11-09T00:27:10.015Z+ [[ -z '' ]]
5052023-11-09T00:27:10.046Z+ dladm create-simnet -t net1
5062023-11-09T00:27:10.081Z+ dladm create-simnet -t sc1_0
5072023-11-09T00:27:10.107Z+ dladm modify-simnet -t -p net1 sc1_0
5082023-11-09T00:27:10.137Z+ dladm set-linkprop -p mtu=9000 sc1_0
5092023-11-09T00:27:10.583Z+ success 'Simnet net1/sc1_0 exists'
5102023-11-09T00:27:10.614Z+ set +x
5112023-11-09T00:27:10.639Z++ get_vnic_name_if_exists sc0_1
5122023-11-09T00:27:10.663Z++ dladm show-vnic -p -o LINK sc0_1
5132023-11-09T00:27:10.688Z++ echo ''
5142023-11-09T00:27:10.713Z+ [[ -z '' ]]
5152023-11-09T00:27:10.738Z+ dladm create-vnic -t sc0_1 -l igb0 -m a8:e1:de:01:70:1d
5162023-11-09T00:27:10.762Z+ [[ -v PROMISC_FILT_OFF ]]
5172023-11-09T00:27:10.788Z+ success 'Vnic sc0_1 exists'
5182023-11-09T00:27:10.820Z+ set +x
5192023-11-09T00:27:10.845Z+ warn_if_no_proxy_arp
5202023-11-09T00:27:10.870Z+ [[ -v PXA_START ]]
5212023-11-09T00:27:10.894Z+ [[ -v PXA_END ]]
5222023-11-09T00:27:10.919Z+ ensure_softnpu_zone
5232023-11-09T00:27:10.943Z+ zoneadm list
5242023-11-09T00:27:10.967Z+ grep -q sidecar_softnpu
5252023-11-09T00:27:10.992Z+ [[ -f out/npuzone/npuzone ]]
5262023-11-09T00:27:11.025Z+ out/npuzone/npuzone create sidecar --omicron-zone --ports sc0_0,tfportrear0_0 --ports sc0_1,tfportqsfp0_0 --sidecar-lite-branch main
5272023-11-09T00:27:11.480Zdownloading softnpu ...done
5282023-11-09T00:27:11.505Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/sidecar-lite/release/f0585a29fb0285f7a1220c1118856b0e5c1f75c5/libsidecar_lite.so.sha256.txt
5292023-11-09T00:27:23.353Zdownloading asic_program.so ...done
5302023-11-09T00:27:23.397Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/sidecar-lite/release/f0585a29fb0285f7a1220c1118856b0e5c1f75c5/scadm.sha256.txt
5312023-11-09T00:27:31.007Zdownloading scadm ...done
5322023-11-09T00:27:31.031Zcreating port tfportrear0_0 for sc0_0
5332023-11-09T00:27:31.056Zcreating port tfportqsfp0_0 for sc0_1
5342023-11-09T00:27:31.107Zsoftnpu zone setup
5352023-11-09T00:27:31.131Zconfigure zone
5362023-11-09T00:27:31.156Zinstall zone
5372023-11-09T00:27:32.091Zboot zone
5382023-11-09T00:27:35.199Z[sidecar_softnpu] svcs milestone/network
5392023-11-09T00:27:35.229ZSTATE STIME FMRI
5402023-11-09T00:27:35.254Zonline 0:27:34 svc:/milestone/network:default
5412023-11-09T00:27:35.280Z[sidecar_softnpu] /softnpu/npu start
5422023-11-09T00:27:35.304Z
5432023-11-09T00:27:35.329Z+ /opt/oxide/work/tools/scrimlet/softnpu-init.sh
5442023-11-09T00:27:35.353Z+ GATEWAY_IP=10.151.2.100
5452023-11-09T00:27:35.377Z+ echo 'Using 10.151.2.100 as gateway ip'
5462023-11-09T00:27:35.401ZUsing 10.151.2.100 as gateway ip
5472023-11-09T00:27:35.425Z+ [[ ! -v GATEWAY_MAC ]]
5482023-11-09T00:27:35.451Z+ GATEWAY_MAC=18:c0:4d:0d:a0:2a
5492023-11-09T00:27:35.477Z++ grep -c -E '^([0-9a-fA-F]{2}:){5}[0-9a-fA-F]{2}$' /dev/fd/63
5502023-11-09T00:27:35.506Z+++ echo 18:c0:4d:0d:a0:2a
5512023-11-09T00:27:35.533Z+ COUNT=1
5522023-11-09T00:27:35.979Z+ [[ 1 -ne 1 ]]
5532023-11-09T00:27:36.011Z+ echo 'Using 18:c0:4d:0d:a0:2a as gateway mac'
5542023-11-09T00:27:36.038ZUsing 18:c0:4d:0d:a0:2a as gateway mac
5552023-11-09T00:27:36.065Z+ z_scadm add-arp-entry 10.151.2.100 18:c0:4d:0d:a0:2a
5562023-11-09T00:27:36.089Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone add-arp-entry 10.151.2.100 18:c0:4d:0d:a0:2a
5572023-11-09T00:27:36.113Z+ PXA_MAC=a8:e1:de:01:70:1d
5582023-11-09T00:27:36.138Z+ [[ -v PXA_START ]]
5592023-11-09T00:27:36.163Z+ z_scadm add-proxy-arp 10.151.2.164 10.151.2.227 a8:e1:de:01:70:1d
5602023-11-09T00:27:36.188Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone add-proxy-arp 10.151.2.164 10.151.2.227 a8:e1:de:01:70:1d
5612023-11-09T00:27:36.213Z+ z_scadm dump-state
5622023-11-09T00:27:36.254Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone dump-state
5632023-11-09T00:27:36.278Zlocal v6:
5642023-11-09T00:27:36.304Zlocal v4:
5652023-11-09T00:27:36.329Zrouter v6:
5662023-11-09T00:27:36.357Zrouter v4_idx:
5672023-11-09T00:27:36.383Zrouter v4_routes:
5682023-11-09T00:27:36.411Zresolver v4:
5692023-11-09T00:27:36.437Z10.151.2.100 -> 18:c0:4d:0d:a0:2a
5702023-11-09T00:27:36.467Zresolver v6:
5712023-11-09T00:27:36.501Znat v4:
5722023-11-09T00:27:36.530Znat_v6:
5732023-11-09T00:27:36.558Zport_mac:
5742023-11-09T00:27:36.587Zproxy arp:
5752023-11-09T00:27:36.615Z10.151.2.164/10.151.2.227: a8:e1:de:01:70:1d
5762023-11-09T00:27:36.643Z+ success 'softnpu zone exists'
5772023-11-09T00:27:36.671Z+ set +x
5782023-11-09T00:27:36.699Zsoftnpu zone exists
5792023-11-09T00:27:36.727Z
5802023-11-09T00:27:36.753Zreal 37.238170467
5812023-11-09T00:27:36.783Zuser 0.697987129
5822023-11-09T00:27:36.811Zsys 1.598174037
5832023-11-09T00:27:36.840Ztrap 0.000109185
5842023-11-09T00:27:36.870Ztflt 0.000163685
5852023-11-09T00:27:36.901Zdflt 0.001187449
5862023-11-09T00:27:36.936Zkflt 0.000004220
5872023-11-09T00:27:36.968Zlock 29:18.720048172
5882023-11-09T00:27:36.994Zslp 1:51.034911217
5892023-11-09T00:27:37.020Zlat 0.074619457
5902023-11-09T00:27:37.086Zstop 0.361733270
5912023-11-09T00:27:37.114Z+ tar xf out/omicron-sled-agent.tar pkg/config-rss.toml
5922023-11-09T00:27:37.142Z++ sed -n 's/silo_name = "\(.*\)"/\1/p' pkg/config-rss.toml
5932023-11-09T00:27:37.189Z+ SILO_NAME=recovery
5942023-11-09T00:27:37.652Z++ sed -n 's/external_dns_zone_name = "\(.*\)"/\1/p' pkg/config-rss.toml
5952023-11-09T00:27:37.679Z+ EXTERNAL_DNS_DOMAIN=oxide.test
5962023-11-09T00:27:37.703Z+ sed -i~ '
5972023-11-09T00:27:37.727Z /^external_dns_ips/c\
5982023-11-09T00:27:37.751Zexternal_dns_ips = [ "10.151.2.164", "10.151.2.165" ]
5992023-11-09T00:27:37.775Z /^\[\[internal_services_ip_pool_ranges/,/^$/ {
6002023-11-09T00:27:37.800Z /^first/c\
6012023-11-09T00:27:37.828Zfirst = "10.151.2.164"
6022023-11-09T00:27:37.853Z /^last/c\
6032023-11-09T00:27:37.885Zlast = "10.151.2.173"
6042023-11-09T00:27:37.916Z }
6052023-11-09T00:27:37.947Z /^\[rack_network_config/,/^$/ {
6062023-11-09T00:27:37.978Z /^infra_ip_first/c\
6072023-11-09T00:27:38.013Zinfra_ip_first = "10.151.2.174"
6082023-11-09T00:27:38.039Z /^infra_ip_last/c\
6092023-11-09T00:27:38.069Zinfra_ip_last = "10.151.2.174"
6102023-11-09T00:27:38.094Z }
6112023-11-09T00:27:38.119Z--- pkg/config-rss.toml~ Mon Jul 24 01:21:28 2006
6122023-11-09T00:27:38.144Z /^\[\[rack_network_config.ports/,/^$/ {
6132023-11-09T00:27:38.168Z+++ pkg/config-rss.toml Thu Nov 9 00:27:36 2023
6142023-11-09T00:27:38.192Z /^routes/c\
6152023-11-09T00:27:38.216Z@@ -41,7 +41,7 @@
6162023-11-09T00:27:38.243Zroutes = \[{nexthop = "10.151.2.100", destination = "0.0.0.0/0"}\]
6172023-11-09T00:27:38.269Z # the DNS domain delegated to the rack by the customer. Each of these addresses
6182023-11-09T00:27:38.310Z /^addresses/c\
6192023-11-09T00:27:38.336Z # must be contained in one of the "internal services" IP Pool ranges listed
6202023-11-09T00:27:38.365Zaddresses = \["10.151.2.174/32"\]
6212023-11-09T00:27:38.394Z # below.
6222023-11-09T00:27:38.420Z }
6232023-11-09T00:27:38.446Z-external_dns_ips = [ "192.168.1.20", "192.168.1.21" ]
6242023-11-09T00:27:38.477Z' pkg/config-rss.toml
6252023-11-09T00:27:38.506Z+external_dns_ips = [ "10.151.2.164", "10.151.2.165" ]
6262023-11-09T00:27:38.540Z+ diff -u pkg/config-rss.toml~ pkg/config-rss.toml
6272023-11-09T00:27:38.565Z
6282023-11-09T00:27:38.590Z+ true
6292023-11-09T00:27:38.615Z # Initial TLS certificates for the external API
6302023-11-09T00:27:38.639Z+ tar rvf out/omicron-sled-agent.tar pkg/config-rss.toml
6312023-11-09T00:27:38.667Z #
6322023-11-09T00:27:38.695Z+ rm -f pkg/config-rss.toml pkg/config-rss.toml~
6332023-11-09T00:27:38.720Z@@ -77,8 +77,8 @@
6342023-11-09T00:27:38.744Z+ TLS_NAME=recovery.sys.oxide.test
6352023-11-09T00:27:38.774Z #
6362023-11-09T00:27:38.811Z+ openssl req -newkey rsa:4096 -x509 -sha256 -days 3 -nodes -out pkg/initial-tls-cert.pem -keyout pkg/initial-tls-key.pem -subj /CN=recovery.sys.oxide.test -addext subjectAltName=DNS:recovery.sys.oxide.test -addext basicConstraints=critical,CA:FALSE -config /dev/stdin
6372023-11-09T00:27:38.838Z # For more on this and what to put here, see docs/how-to-run.adoc.
6382023-11-09T00:27:38.872Z..+......+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.+......+.....+.........+.+......+.....+..........+........+...+...+....+..+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*........+.+...+.....+.......+..................+.....................+.....+.........+...................+...........+...............+.+..............+..........+............+..............+...+.+.....+.......+........+.+......+..+...............+.............+.....+....+...+..+....+..+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
6392023-11-09T00:27:38.904Z [[internal_services_ip_pool_ranges]]
6402023-11-09T00:27:38.931Z.+......+.+...+............+..+...+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*....+.....+.........+...+.............+............+..+.+......+...+....................+.......+...+...+.....+......................+...+..+...+....+.....+....+..+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.......+..+...+......+.........+......+......+....+............+.......................+....+......+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
6412023-11-09T00:27:38.956Z-first = "192.168.1.20"
6422023-11-09T00:27:38.987Z-----
6432023-11-09T00:27:39.014Z-last = "192.168.1.29"
6442023-11-09T00:27:39.038Z+ tar rvf out/omicron-sled-agent.tar pkg/initial-tls-cert.pem pkg/initial-tls-key.pem
6452023-11-09T00:27:39.063Z+first = "10.151.2.164"
6462023-11-09T00:27:39.088Z+ rm -f pkg/initial-tls-cert.pem pkg/initial-tls-key.pem
6472023-11-09T00:27:39.121Z+last = "10.151.2.173"
6482023-11-09T00:27:39.147Z+ rmdir pkg
6492023-11-09T00:27:39.171Z
6502023-11-09T00:27:39.197Z+ E2E_TLS_CERT=/opt/oxide/sled-agent/pkg/initial-tls-cert.pem
6512023-11-09T00:27:39.227Z # TODO - this configuration is subject to change going forward. Ultimately these
6522023-11-09T00:27:39.257Z+ pfexec mkdir -p /usr/oxide
6532023-11-09T00:27:39.292Z # parameters should be provided to the control plane via wicket, but we need to
6542023-11-09T00:27:39.346Z+ pfexec rm -f /usr/oxide/tcpproxy
6552023-11-09T00:27:39.810Z@@ -92,8 +92,8 @@
6562023-11-09T00:27:39.842Z+ pfexec curl -sSfL -o /usr/oxide/tcpproxy http://catacomb.eng.oxide.computer:12346/tcpproxy
6572023-11-09T00:27:39.871Z # A range of IP addresses used by Boundary Services on the external network. In
6582023-11-09T00:27:39.901Z+ pfexec chmod +x /usr/oxide/tcpproxy
6592023-11-09T00:27:40.335Z # a real system, these would be addresses of the uplink ports on the Sidecar.
6602023-11-09T00:27:40.374Z+ pfexec rm -f /var/svc/manifest/site/tcpproxy.xml
6612023-11-09T00:27:40.408Z # With softnpu, only one address is used.
6622023-11-09T00:27:40.449Z+ pfexec curl -sSfL -o /var/svc/manifest/site/tcpproxy.xml http://catacomb.eng.oxide.computer:12346/tcpproxy.xml
6632023-11-09T00:27:40.482Z-infra_ip_first = "192.168.1.30"
6642023-11-09T00:27:40.506Z+ pfexec svccfg import /var/svc/manifest/site/tcpproxy.xml
6652023-11-09T00:27:40.531Z-infra_ip_last = "192.168.1.30"
6662023-11-09T00:27:40.558Z+ OMICRON_NO_UNINSTALL=1
6672023-11-09T00:27:40.585Z+infra_ip_first = "10.151.2.174"
6682023-11-09T00:27:40.610Z+ ptime -m pfexec ./target/release/omicron-package -t test install
6692023-11-09T00:27:40.634Z+infra_ip_last = "10.151.2.174"
6702023-11-09T00:27:40.659ZNov 09 00:27:37.000 DEBG target[test]: Target({"image": "standard", "machine": "non-gimlet", "rack-topology": "single-sled", "switch": "softnpu"})
6712023-11-09T00:27:40.692Z
6722023-11-09T00:27:42.021ZNov 09 00:27:37.002 INFO Installing service, dst: /opt/oxide/cockroachdb.tar.gz, src: out/cockroachdb.tar.gz
6732023-11-09T00:27:42.053Z # Configurations for BGP routers to run on the scrimlets.
6742023-11-09T00:27:42.093ZNov 09 00:27:37.002 INFO Installing service, dst: /opt/oxide/ntp.tar.gz, src: out/ntp.tar.gz
6752023-11-09T00:27:42.320Z bgp = []
6762023-11-09T00:27:42.349ZNov 09 00:27:37.002 INFO Installing service, dst: /opt/oxide/mgs.tar.gz, src: out/omicron-gateway-softnpu.tar.gz
6772023-11-09T00:27:42.379Z@@ -101,9 +101,9 @@
6782023-11-09T00:27:42.407ZNov 09 00:27:37.002 INFO Installing service, dst: /opt/oxide/clickhouse.tar.gz, src: out/clickhouse.tar.gz
6792023-11-09T00:27:42.468Z # You can configure multiple uplinks by repeating the following stanza
6802023-11-09T00:27:42.501ZNov 09 00:27:37.002 INFO Installing service, dst: /opt/oxide/internal_dns.tar.gz, src: out/internal-dns.tar.gz
6812023-11-09T00:27:42.532Z [[rack_network_config.ports]]
6822023-11-09T00:27:42.560ZNov 09 00:27:37.002 INFO Installing service, dst: /opt/oxide/clickhouse_keeper.tar.gz, src: out/clickhouse_keeper.tar.gz
6832023-11-09T00:27:42.584Z # Routes associated with this port.
6842023-11-09T00:27:42.609ZNov 09 00:27:37.002 INFO Installing service, dst: /opt/oxide/nexus.tar.gz, src: out/omicron-nexus.tar.gz
6852023-11-09T00:27:42.634Z-routes = [{nexthop = "192.168.1.199", destination = "0.0.0.0/0"}]
6862023-11-09T00:27:42.668ZNov 09 00:27:37.002 INFO Installing service, dst: /opt/oxide/external_dns.tar.gz, src: out/external-dns.tar.gz
6872023-11-09T00:27:43.112Z+routes = [{nexthop = "10.151.2.100", destination = "0.0.0.0/0"}]
6882023-11-09T00:27:43.137ZNov 09 00:27:37.002 INFO Installing service, dst: /opt/oxide/oximeter.tar.gz, src: out/oximeter-collector.tar.gz
6892023-11-09T00:27:43.163Z # Addresses associated with this port.
6902023-11-09T00:27:43.187ZNov 09 00:27:37.003 INFO Installing service, dst: /opt/oxide/sled-agent.tar, src: out/omicron-sled-agent.tar
6912023-11-09T00:27:43.211Z-addresses = ["192.168.1.30/32"]
6922023-11-09T00:27:43.241ZNov 09 00:27:37.003 INFO Installing service, dst: /opt/oxide/mg-ddm.tar, src: out/mg-ddm-gz.tar
6932023-11-09T00:27:43.265Z+addresses = ["10.151.2.174/32"]
6942023-11-09T00:27:43.289ZNov 09 00:27:37.003 INFO Installing service, dst: /opt/oxide/overlay.tar.gz, src: out/overlay.tar.gz
6952023-11-09T00:27:43.314Z # Name of the uplink port. This should always be "qsfp0" when using softnpu.
6962023-11-09T00:27:43.338ZNov 09 00:27:37.003 INFO Installing service, dst: /opt/oxide/propolis-server.tar.gz, src: out/propolis-server.tar.gz
6972023-11-09T00:27:43.363Z port = "qsfp0"
6982023-11-09T00:27:43.387ZNov 09 00:27:37.003 INFO Installing service, dst: /opt/oxide/crucible.tar.gz, src: out/crucible.tar.gz
6992023-11-09T00:27:43.412Z # The speed of this port.
7002023-11-09T00:27:43.438ZNov 09 00:27:37.003 INFO Installing service, dst: /opt/oxide/crucible_pantry.tar.gz, src: out/crucible-pantry.tar.gz
7012023-11-09T00:27:43.463Za pkg/config-rss.toml 7K
7022023-11-09T00:27:43.494ZNov 09 00:27:37.003 INFO Installing service, dst: /opt/oxide/switch.tar.gz, src: out/switch-softnpu.tar.gz
7032023-11-09T00:27:43.518Za pkg/initial-tls-cert.pem 2K
7042023-11-09T00:27:43.542ZNov 09 00:27:37.250 INFO Unpacking service tarball, service_path: /opt/oxide/mg-ddm, tar_path: /opt/oxide/mg-ddm.tar
7052023-11-09T00:27:43.566Za pkg/initial-tls-key.pem 4K
7062023-11-09T00:27:43.590ZNov 09 00:27:37.286 INFO Unpacking service tarball, service_path: /opt/oxide/sled-agent, tar_path: /opt/oxide/sled-agent.tar
7072023-11-09T00:27:43.615ZNov 09 00:27:37.354 INFO Installing bootstrap service from /opt/oxide/sled-agent/pkg/manifest.xml
7082023-11-09T00:27:43.639Z
7092023-11-09T00:27:43.663Zreal 0.492982377
7102023-11-09T00:27:43.688Zuser 0.103596356
7112023-11-09T00:27:43.713Zsys 2.089780888
7122023-11-09T00:27:43.737Ztrap 0.000000410
7132023-11-09T00:27:43.761Ztflt 0.000044240
7142023-11-09T00:27:43.786Zdflt 0.000468011
7152023-11-09T00:27:43.811Zkflt 0.000000000
7162023-11-09T00:27:43.836Zlock 44.242549600
7172023-11-09T00:27:43.860Zslp 0.708788135
7182023-11-09T00:27:43.885Zlat 0.003298998
7192023-11-09T00:27:43.912Zstop 0.211914498
7202023-11-09T00:27:44.367Z+ retry=0
7212023-11-09T00:27:44.392Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7222023-11-09T00:27:44.417Z+ [[ 0 -gt 30 ]]
7232023-11-09T00:27:44.444Z+ sleep 1
7242023-11-09T00:27:44.475Z+ retry=1
7252023-11-09T00:27:44.520Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7262023-11-09T00:27:44.545Z+ [[ 1 -gt 30 ]]
7272023-11-09T00:27:44.570Z+ sleep 1
7282023-11-09T00:27:44.594Z+ retry=2
7292023-11-09T00:27:44.619Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7302023-11-09T00:28:04.055Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone dump-state
7312023-11-09T00:28:04.093Zlocal v6:
7322023-11-09T00:28:04.124Zfe80::f4c1:cdff:feb8:b2d4
7332023-11-09T00:28:04.149Zfd00:99::1
7342023-11-09T00:28:04.176Zlocal v4:
7352023-11-09T00:28:04.201Z10.151.2.174
7362023-11-09T00:28:04.226Zrouter v6:
7372023-11-09T00:28:04.251Zfd00:1122:3344:101::/64 -> fe80::ccef:c6ff:fe81:896c (1)
7382023-11-09T00:28:04.282Zfdb0:18c0:4d0d:a02a::/64 -> fe80::ccef:c6ff:fe81:896c (1)
7392023-11-09T00:28:04.307Zrouter v4_idx:
7402023-11-09T00:28:04.333Z0.0.0.0/0 -> 0
7412023-11-09T00:28:04.357Zrouter v4_routes:
7422023-11-09T00:28:04.381Z0 -> 10.151.2.100 (2)
7432023-11-09T00:28:04.414Zresolver v4:
7442023-11-09T00:28:04.443Z10.151.2.174 -> 46:22:15:d1:4b:47
7452023-11-09T00:28:04.472Z10.151.2.100 -> 18:c0:4d:0d:a0:2a
7462023-11-09T00:28:04.514Zresolver v6:
7472023-11-09T00:28:04.560Zfe80::f4c1:cdff:feb8:b2d4 -> f6:c1:cd:b8:b2:d4
7482023-11-09T00:28:04.590Zfe80::ccef:c6ff:fe81:896c -> ce:ef:c6:81:89:6c
7492023-11-09T00:28:04.619Znat v4:
7502023-11-09T00:28:04.649Znat_v6:
7512023-11-09T00:28:04.681Zport_mac:
7522023-11-09T00:28:04.713Z1000: a8:40:25:71:d6:b4
7532023-11-09T00:28:04.744Z1: a8:40:25:71:d6:b5
7542023-11-09T00:28:04.779Zproxy arp:
7552023-11-09T00:28:04.811Z10.151.2.164/10.151.2.227: a8:e1:de:01:70:1d
7562023-11-09T00:28:04.836Z+ retry=0
7572023-11-09T00:28:04.861Z+++ zoneadm list -n
7582023-11-09T00:28:04.888Z+++ grep oxz_ntp
7592023-11-09T00:28:04.912Z++ pfexec svcs -z -Hostate oxide/ntp
7602023-11-09T00:28:04.939Zsvcs: zone '-Hostate': entity not found
7612023-11-09T00:28:04.973Z++ true
7622023-11-09T00:28:05.019Z+ [[ '' != online ]]
7632023-11-09T00:28:05.061Z+ [[ 0 -gt 60 ]]
7642023-11-09T00:28:05.086Z+ sleep 1
7652023-11-09T00:28:05.111Z+ retry=1
7662023-11-09T00:28:05.137Z+++ zoneadm list -n
7672023-11-09T00:28:05.162Z+++ grep oxz_ntp
7682023-11-09T00:28:05.186Z++ pfexec svcs -z -Hostate oxide/ntp
7692023-11-09T00:28:05.216Zsvcs: zone '-Hostate': entity not found
7702023-11-09T00:28:05.259Z++ true
7712023-11-09T00:28:05.324Z+ [[ '' != online ]]
7722023-11-09T00:28:05.350Z+ [[ 1 -gt 60 ]]
7732023-11-09T00:28:05.375Z+ sleep 1
7742023-11-09T00:28:06.126Z+ retry=2
7752023-11-09T00:28:06.152Z+++ zoneadm list -n
7762023-11-09T00:28:06.176Z+++ grep oxz_ntp
7772023-11-09T00:28:06.202Z++ pfexec svcs -z -Hostate oxide/ntp
7782023-11-09T00:28:06.226Zsvcs: zone '-Hostate': entity not found
7792023-11-09T00:28:06.251Z++ true
7802023-11-09T00:28:06.274Z+ [[ '' != online ]]
7812023-11-09T00:28:06.299Z+ [[ 2 -gt 60 ]]
7822023-11-09T00:28:06.347Z+ sleep 1
7832023-11-09T00:28:07.147Z+ retry=3
7842023-11-09T00:28:07.172Z+++ zoneadm list -n
7852023-11-09T00:28:07.196Z+++ grep oxz_ntp
7862023-11-09T00:28:07.220Z++ pfexec svcs -z -Hostate oxide/ntp
7872023-11-09T00:28:07.244Zsvcs: zone '-Hostate': entity not found
7882023-11-09T00:28:07.269Z++ true
7892023-11-09T00:28:07.294Z+ [[ '' != online ]]
7902023-11-09T00:28:07.319Z+ [[ 3 -gt 60 ]]
7912023-11-09T00:28:07.344Z+ sleep 1
7922023-11-09T00:28:08.171Z+ retry=4
7932023-11-09T00:28:08.195Z+++ zoneadm list -n
7942023-11-09T00:28:08.222Z+++ grep oxz_ntp
7952023-11-09T00:28:08.252Z++ pfexec svcs -z -Hostate oxide/ntp
7962023-11-09T00:28:08.276Zsvcs: zone '-Hostate': entity not found
7972023-11-09T00:28:08.301Z++ true
7982023-11-09T00:28:08.325Z+ [[ '' != online ]]
7992023-11-09T00:28:08.351Z+ [[ 4 -gt 60 ]]
8002023-11-09T00:28:08.400Z+ sleep 1
8012023-11-09T00:28:09.198Z+ retry=5
8022023-11-09T00:28:09.225Z+++ zoneadm list -n
8032023-11-09T00:28:09.254Z+++ grep oxz_ntp
8042023-11-09T00:28:09.280Z++ pfexec svcs -z -Hostate oxide/ntp
8052023-11-09T00:28:09.307Zsvcs: zone '-Hostate': entity not found
8062023-11-09T00:28:09.339Z++ true
8072023-11-09T00:28:09.369Z+ [[ '' != online ]]
8082023-11-09T00:28:09.445Z+ [[ 5 -gt 60 ]]
8092023-11-09T00:28:09.470Z+ sleep 1
8102023-11-09T00:28:10.223Z+ retry=6
8112023-11-09T00:28:10.249Z+++ zoneadm list -n
8122023-11-09T00:28:10.273Z+++ grep oxz_ntp
8132023-11-09T00:28:10.298Z++ pfexec svcs -z -Hostate oxide/ntp
8142023-11-09T00:28:10.322Zsvcs: zone '-Hostate': entity not found
8152023-11-09T00:28:10.353Z++ true
8162023-11-09T00:28:10.389Z+ [[ '' != online ]]
8172023-11-09T00:28:10.419Z+ [[ 6 -gt 60 ]]
8182023-11-09T00:28:10.471Z+ sleep 1
8192023-11-09T00:28:11.240Z+ retry=7
8202023-11-09T00:28:11.265Z+++ zoneadm list -n
8212023-11-09T00:28:11.289Z+++ grep oxz_ntp
8222023-11-09T00:28:11.314Z++ pfexec svcs -z -Hostate oxide/ntp
8232023-11-09T00:28:11.338Zsvcs: zone '-Hostate': entity not found
8242023-11-09T00:28:11.363Z++ true
8252023-11-09T00:28:11.387Z+ [[ '' != online ]]
8262023-11-09T00:28:11.411Z+ [[ 7 -gt 60 ]]
8272023-11-09T00:28:11.436Z+ sleep 1
8282023-11-09T00:28:12.261Z+ retry=8
8292023-11-09T00:28:12.288Z+++ zoneadm list -n
8302023-11-09T00:28:12.314Z+++ grep oxz_ntp
8312023-11-09T00:28:12.339Z++ pfexec svcs -z -Hostate oxide/ntp
8322023-11-09T00:28:12.364Zsvcs: zone '-Hostate': entity not found
8332023-11-09T00:28:12.388Z++ true
8342023-11-09T00:28:12.423Z+ [[ '' != online ]]
8352023-11-09T00:28:12.862Z+ [[ 8 -gt 60 ]]
8362023-11-09T00:28:12.898Z+ sleep 1
8372023-11-09T00:28:13.281Z+ retry=9
8382023-11-09T00:28:13.309Z+++ zoneadm list -n
8392023-11-09T00:28:13.341Z+++ grep oxz_ntp
8402023-11-09T00:28:13.366Z++ pfexec svcs -z oxz_ntp_1c2c4826-d978-4371-bc32-047fe7244a8b -Hostate oxide/ntp
8412023-11-09T00:28:13.395Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
8422023-11-09T00:28:13.421Z++ true
8432023-11-09T00:28:13.445Z+ [[ '' != online ]]
8442023-11-09T00:28:13.475Z+ [[ 9 -gt 60 ]]
8452023-11-09T00:28:13.504Z+ sleep 1
8462023-11-09T00:28:14.332Z+ retry=10
8472023-11-09T00:28:14.367Z+++ zoneadm list -n
8482023-11-09T00:28:14.822Z+++ grep oxz_ntp
8492023-11-09T00:28:14.855Z++ pfexec svcs -z oxz_ntp_1c2c4826-d978-4371-bc32-047fe7244a8b -Hostate oxide/ntp
8502023-11-09T00:28:14.883Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
8512023-11-09T00:28:14.914Z++ true
8522023-11-09T00:28:14.952Z+ [[ '' != online ]]
8532023-11-09T00:28:15.428Z+ [[ 10 -gt 60 ]]
8542023-11-09T00:28:15.486Z+ sleep 1
8552023-11-09T00:28:15.523Z+ retry=11
8562023-11-09T00:28:15.556Z+++ zoneadm list -n
8572023-11-09T00:28:15.589Z+++ grep oxz_ntp
8582023-11-09T00:28:15.625Z++ pfexec svcs -z oxz_ntp_1c2c4826-d978-4371-bc32-047fe7244a8b -Hostate oxide/ntp
8592023-11-09T00:28:15.697Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
8602023-11-09T00:28:15.733Z++ true
8612023-11-09T00:28:15.769Z+ [[ '' != online ]]
8622023-11-09T00:28:15.808Z+ [[ 11 -gt 60 ]]
8632023-11-09T00:28:15.855Z+ sleep 1
8642023-11-09T00:28:16.379Z+ retry=12
8652023-11-09T00:28:16.410Z+++ zoneadm list -n
8662023-11-09T00:28:16.439Z+++ grep oxz_ntp
8672023-11-09T00:28:16.473Z++ pfexec svcs -z oxz_ntp_1c2c4826-d978-4371-bc32-047fe7244a8b -Hostate oxide/ntp
8682023-11-09T00:28:16.504Z+ [[ disabled != online ]]
8692023-11-09T00:28:16.535Z+ [[ 12 -gt 60 ]]
8702023-11-09T00:28:16.572Z+ sleep 1
8712023-11-09T00:28:17.420Z+ retry=13
8722023-11-09T00:28:17.456Z+++ zoneadm list -n
8732023-11-09T00:28:17.483Z+++ grep oxz_ntp
8742023-11-09T00:28:17.512Z++ pfexec svcs -z oxz_ntp_1c2c4826-d978-4371-bc32-047fe7244a8b -Hostate oxide/ntp
8752023-11-09T00:28:17.549Z+ [[ disabled != online ]]
8762023-11-09T00:28:17.581Z+ [[ 13 -gt 60 ]]
8772023-11-09T00:28:17.615Z+ sleep 1
8782023-11-09T00:28:18.435Z+ retry=14
8792023-11-09T00:28:18.469Z+++ zoneadm list -n
8802023-11-09T00:28:18.515Z+++ grep oxz_ntp
8812023-11-09T00:28:18.553Z++ pfexec svcs -z oxz_ntp_1c2c4826-d978-4371-bc32-047fe7244a8b -Hostate oxide/ntp
8822023-11-09T00:28:18.584Z+ [[ disabled != online ]]
8832023-11-09T00:28:19.023Z+ [[ 14 -gt 60 ]]
8842023-11-09T00:28:19.048Z+ sleep 1
8852023-11-09T00:28:19.452Z+ retry=15
8862023-11-09T00:28:19.479Z+++ zoneadm list -n
8872023-11-09T00:28:19.509Z+++ grep oxz_ntp
8882023-11-09T00:28:19.536Z++ pfexec svcs -z oxz_ntp_1c2c4826-d978-4371-bc32-047fe7244a8b -Hostate oxide/ntp
8892023-11-09T00:28:19.563Z+ [[ disabled != online ]]
8902023-11-09T00:28:19.595Z+ [[ 15 -gt 60 ]]
8912023-11-09T00:28:19.627Z+ sleep 1
8922023-11-09T00:28:20.469Z+ retry=16
8932023-11-09T00:28:20.496Z+++ zoneadm list -n
8942023-11-09T00:28:20.521Z+++ grep oxz_ntp
8952023-11-09T00:28:20.545Z++ pfexec svcs -z oxz_ntp_1c2c4826-d978-4371-bc32-047fe7244a8b -Hostate oxide/ntp
8962023-11-09T00:28:20.570Z+ [[ disabled != online ]]
8972023-11-09T00:28:20.595Z+ [[ 16 -gt 60 ]]
8982023-11-09T00:28:20.625Z+ sleep 1
8992023-11-09T00:28:21.491Z+ retry=17
9002023-11-09T00:28:21.516Z+++ zoneadm list -n
9012023-11-09T00:28:21.543Z+++ grep oxz_ntp
9022023-11-09T00:28:21.581Z++ pfexec svcs -z oxz_ntp_1c2c4826-d978-4371-bc32-047fe7244a8b -Hostate oxide/ntp
9032023-11-09T00:28:21.614Z+ [[ disabled != online ]]
9042023-11-09T00:28:21.644Z+ [[ 17 -gt 60 ]]
9052023-11-09T00:28:21.673Z+ sleep 1
9062023-11-09T00:28:22.519Z+ retry=18
9072023-11-09T00:28:22.554Z+++ zoneadm list -n
9082023-11-09T00:28:22.585Z+++ grep oxz_ntp
9092023-11-09T00:28:22.616Z++ pfexec svcs -z oxz_ntp_1c2c4826-d978-4371-bc32-047fe7244a8b -Hostate oxide/ntp
9102023-11-09T00:28:22.654Z+ [[ disabled != online ]]
9112023-11-09T00:28:22.682Z+ [[ 18 -gt 60 ]]
9122023-11-09T00:28:22.707Z+ sleep 1
 
9132023-11-09T00:28:23.531Zsock console login: Nov 9 00:28:23 sok ip: ip: joining multicasts failed (4) on vopte0 - will use link layer broadcasts for multicast
 
9142023-11-09T00:28:23.538Z+ retry=19
9152023-11-09T00:28:23.566Z+++ zoneadm list -n
9162023-11-09T00:28:23.592Z+++ grep oxz_ntp
9172023-11-09T00:28:23.616Z++ pfexec svcs -z oxz_ntp_1c2c4826-d978-4371-bc32-047fe7244a8b -Hostate oxide/ntp
9182023-11-09T00:28:23.641Z+ [[ disabled != online ]]
9192023-11-09T00:28:23.665Z+ [[ 19 -gt 60 ]]
9202023-11-09T00:28:23.689Z+ sleep 1
9212023-11-09T00:28:24.563Z+ retry=20
9222023-11-09T00:28:24.995Z+++ zoneadm list -n
9232023-11-09T00:28:25.035Z+++ grep oxz_ntp
9242023-11-09T00:28:25.059Z++ pfexec svcs -z oxz_ntp_1c2c4826-d978-4371-bc32-047fe7244a8b -Hostate oxide/ntp
9252023-11-09T00:28:25.083Z+ [[ online != online ]]
9262023-11-09T00:28:25.107Z+ echo 'Waited for chrony: 20s'
9272023-11-09T00:28:25.131Z+ export RUST_BACKTRACE=1
9282023-11-09T00:28:25.159ZWaited for chrony: 20s
9292023-11-09T00:28:25.601Z+ RUST_BACKTRACE=1
9302023-11-09T00:28:25.625Z+ export E2E_TLS_CERT IPPOOL_START IPPOOL_END
9312023-11-09T00:28:25.649Z+ ./tests/bootstrap
9322023-11-09T00:28:25.676ZUsing external DNS server at 10.151.2.164:53
9332023-11-09T00:28:25.708Z2023-11-09 00:28:25.249543793 UTC: attempting to log into API
9342023-11-09T00:28:39.648Z2023-11-09 00:28:39.634597544 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9352023-11-09T00:28:40.649Z2023-11-09 00:28:40.638333819 UTC: attempting to log into API
9362023-11-09T00:28:55.742Z2023-11-09 00:28:55.675035980 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9372023-11-09T00:28:56.684Z2023-11-09 00:28:56.675862633 UTC: attempting to log into API
9382023-11-09T00:29:11.732Z2023-11-09 00:29:11.725109455 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9392023-11-09T00:29:12.733Z2023-11-09 00:29:12.726077851 UTC: attempting to log into API
9402023-11-09T00:29:27.766Z2023-11-09 00:29:27.759447235 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9412023-11-09T00:29:28.767Z2023-11-09 00:29:28.760776972 UTC: attempting to log into API
9422023-11-09T00:29:43.811Z2023-11-09 00:29:43.802380616 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
 
9432023-11-09T00:29:44.113ZNov 9 00:29:43 sock ip: ip: joining multicasts failed (4) on vopte3 - will use link layer broadcasts for multicast
 
9442023-11-09T00:29:44.822Z2023-11-09 00:29:44.804030131 UTC: attempting to log into API
 
9452023-11-09T00:29:51.535ZNov 9 00:29:50 sock ip: ip: joining multicasts failed (4) on vopte1 - will use link layer broadcasts for multicast
9462023-11-09T00:29:58.409ZNov 9 00:29:57 sock ip: ip: joining multicasts failed (4) on vopte4 - will use link layer broadcasts for multicast
 
9472023-11-09T00:29:59.975Z2023-11-09 00:29:59.836851081 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9482023-11-09T00:30:00.848Z2023-11-09 00:30:00.838783055 UTC: attempting to log into API
 
9492023-11-09T00:30:05.797ZNov 9 00:30:05 sock ip: ip: joining multicasts failed (4) on vopte2 - will use link layer broadcasts for multicast
9502023-11-09T00:30:12.431ZNov 9 00:30:11 sock ip: ip: joining multicasts failed (4) on vopte5 - will use link layer broadcasts for multicast
 
9512023-11-09T00:30:15.883Z2023-11-09 00:30:15.873211050 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9522023-11-09T00:30:16.884Z2023-11-09 00:30:16.874143636 UTC: attempting to log into API
9532023-11-09T00:30:31.916Z2023-11-09 00:30:31.906976126 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9542023-11-09T00:30:33.028Z2023-11-09 00:30:32.908902650 UTC: attempting to log into API
9552023-11-09T00:30:37.959Z2023-11-09 00:30:37.949801029 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: dns error: no record found for Query { name: Name("recovery.sys.oxide.test."), query_type: AAAA, query_class: IN }
9562023-11-09T00:30:39.367Z2023-11-09 00:30:38.951043234 UTC: attempting to log into API
9572023-11-09T00:30:39.737Z2023-11-09 00:30:39.722230643 UTC: login succeeded
9582023-11-09T00:30:39.764Zwaiting for nexus to come up...
9592023-11-09T00:30:39.834Zcreating IP pool... 10.151.2.175 - 10.151.2.227
9602023-11-09T00:30:39.879Zensuring datasets are ready...
9612023-11-09T00:30:41.890Zlet's roll.
9622023-11-09T00:30:41.915Z+ rm ./tests/bootstrap
9632023-11-09T00:30:41.986Z+ for test_bin in tests/*
9642023-11-09T00:30:42.011Z+ ./tests/bootstrap-b24d35b7ab856b37
9652023-11-09T00:30:42.036Z
9662023-11-09T00:30:42.063Zrunning 0 tests
9672023-11-09T00:30:42.089Z
9682023-11-09T00:30:42.122Ztest result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
9692023-11-09T00:30:42.151Z
9702023-11-09T00:30:42.596Z+ for test_bin in tests/*
9712023-11-09T00:30:42.622Z+ ./tests/end_to_end_tests-45b3c6eea9aa5f9f
9722023-11-09T00:30:42.648Z
9732023-11-09T00:30:42.687Zrunning 2 tests
9742023-11-09T00:30:42.716Ztest no_spoof::no_spoof_here ... ok
9752023-11-09T00:31:42.027Ztest instance_launch::instance_launch has been running for over 60 seconds
9762023-11-09T00:33:38.375Ztest instance_launch::instance_launch ... ok
9772023-11-09T00:33:38.401Z
9782023-11-09T00:33:38.425Ztest result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 176.36s
9792023-11-09T00:33:38.451Z
9802023-11-09T00:33:38.475Z+ _exit_trap
9812023-11-09T00:33:38.501Z+ local status=0
9822023-11-09T00:33:38.528Z+ [[ 0 -eq 0 ]]
9832023-11-09T00:33:38.554Z+ exit 0
9842023-11-09T00:33:38.581Zprocess exited: duration 408819 ms, exit code 0
 
9852023-11-09T00:33:38.670Zfound 84 output files
9862023-11-09T00:33:38.731Zuploading: /var/svc/log/oxide-sled-agent:default.log (594090 bytes)
9872023-11-09T00:33:40.138Zupload warning: file "/var/svc/log/oxide-sled-agent:default.log" changed size mid upload: 594090 -> 594487
9882023-11-09T00:33:41.215Zuploaded: /var/svc/log/oxide-sled-agent:default.log
9892023-11-09T00:33:41.646Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_clickhouse_71948cb0-eb4d-4b3b-ac22-deebae988ef2/root/var/svc/log/oxide-clickhouse:default.log (720263 bytes)
9902023-11-09T00:33:42.092Zupload warning: file "/pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_clickhouse_71948cb0-eb4d-4b3b-ac22-deebae988ef2/root/var/svc/log/oxide-clickhouse:default.log" changed size mid upload: 720263 -> 732146
9912023-11-09T00:33:43.176Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_clickhouse_71948cb0-eb4d-4b3b-ac22-deebae988ef2/root/var/svc/log/oxide-clickhouse:default.log
9922023-11-09T00:33:43.201Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_clickhouse_71948cb0-eb4d-4b3b-ac22-deebae988ef2/root/var/svc/log/oxide-clickhouse:default.log.0 (115264 bytes)
9932023-11-09T00:33:44.493Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_clickhouse_71948cb0-eb4d-4b3b-ac22-deebae988ef2/root/var/svc/log/oxide-clickhouse:default.log.0
9942023-11-09T00:33:44.518Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_0ee9db5f-8495-4f07-8a13-7051cc719c33/root/var/svc/log/oxide-cockroachdb:default.log (0 bytes)
9952023-11-09T00:33:45.569Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_0ee9db5f-8495-4f07-8a13-7051cc719c33/root/var/svc/log/oxide-cockroachdb:default.log
9962023-11-09T00:33:45.598Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_0ee9db5f-8495-4f07-8a13-7051cc719c33/root/var/svc/log/oxide-cockroachdb:default.log.0 (5342 bytes)
9972023-11-09T00:33:47.099Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_0ee9db5f-8495-4f07-8a13-7051cc719c33/root/var/svc/log/oxide-cockroachdb:default.log.0
9982023-11-09T00:33:47.124Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_f947c6fb-d684-4bf6-9b80-08124cb82651/root/var/svc/log/oxide-cockroachdb:default.log (0 bytes)
9992023-11-09T00:33:48.171Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_f947c6fb-d684-4bf6-9b80-08124cb82651/root/var/svc/log/oxide-cockroachdb:default.log
10002023-11-09T00:33:48.200Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_f947c6fb-d684-4bf6-9b80-08124cb82651/root/var/svc/log/oxide-cockroachdb:default.log.0 (5369 bytes)
10012023-11-09T00:33:49.275Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_f947c6fb-d684-4bf6-9b80-08124cb82651/root/var/svc/log/oxide-cockroachdb:default.log.0
10022023-11-09T00:33:49.300Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_108625ec-5d84-4962-bfab-83e40b7d990c/root/var/svc/log/oxide-crucible-pantry:default.log (0 bytes)
10032023-11-09T00:33:50.350Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_108625ec-5d84-4962-bfab-83e40b7d990c/root/var/svc/log/oxide-crucible-pantry:default.log
10042023-11-09T00:33:50.376Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_108625ec-5d84-4962-bfab-83e40b7d990c/root/var/svc/log/oxide-crucible-pantry:default.log.0 (2338 bytes)
10052023-11-09T00:33:51.456Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_108625ec-5d84-4962-bfab-83e40b7d990c/root/var/svc/log/oxide-crucible-pantry:default.log.0
10062023-11-09T00:33:51.482Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_a46a381d-a111-40e6-a317-af4d11cd26cd/root/var/svc/log/oxide-internal_dns:default.log (20078 bytes)
10072023-11-09T00:33:53.008Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_a46a381d-a111-40e6-a317-af4d11cd26cd/root/var/svc/log/oxide-internal_dns:default.log
10082023-11-09T00:33:53.036Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_a46a381d-a111-40e6-a317-af4d11cd26cd/root/var/svc/log/oxide-internal_dns:default.log.0 (92889 bytes)
10092023-11-09T00:33:54.232Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_a46a381d-a111-40e6-a317-af4d11cd26cd/root/var/svc/log/oxide-internal_dns:default.log.0
10102023-11-09T00:33:54.258Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_11a88192-74dc-449b-a5b7-3fec8f75c6fa/root/var/svc/log/oxide-external_dns:default.log (16205 bytes)
10112023-11-09T00:33:55.382Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_11a88192-74dc-449b-a5b7-3fec8f75c6fa/root/var/svc/log/oxide-external_dns:default.log
10122023-11-09T00:33:55.406Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_11a88192-74dc-449b-a5b7-3fec8f75c6fa/root/var/svc/log/oxide-external_dns:default.log.0 (120 bytes)
10132023-11-09T00:33:56.488Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_11a88192-74dc-449b-a5b7-3fec8f75c6fa/root/var/svc/log/oxide-external_dns:default.log.0
10142023-11-09T00:33:56.542Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_bc4fa146-8747-4e63-8af3-da3c92c2b78e/root/var/svc/log/oxide-nexus:default.log (385327 bytes)
10152023-11-09T00:33:56.757Zupload warning: file "/pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_bc4fa146-8747-4e63-8af3-da3c92c2b78e/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 385327 -> 386543
10162023-11-09T00:33:57.832Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_bc4fa146-8747-4e63-8af3-da3c92c2b78e/root/var/svc/log/oxide-nexus:default.log
10172023-11-09T00:33:57.863Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_bc4fa146-8747-4e63-8af3-da3c92c2b78e/root/var/svc/log/oxide-nexus:default.log.0 (244008 bytes)
10182023-11-09T00:33:59.336Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_bc4fa146-8747-4e63-8af3-da3c92c2b78e/root/var/svc/log/oxide-nexus:default.log.0
10192023-11-09T00:33:59.361Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_1c2c4826-d978-4371-bc32-047fe7244a8b/root/var/svc/log/oxide-ntp:default.log (0 bytes)
10202023-11-09T00:34:00.409Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_1c2c4826-d978-4371-bc32-047fe7244a8b/root/var/svc/log/oxide-ntp:default.log
10212023-11-09T00:34:00.434Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_1c2c4826-d978-4371-bc32-047fe7244a8b/root/var/svc/log/oxide-ntp:default.log.0 (986 bytes)
10222023-11-09T00:34:01.507Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_1c2c4826-d978-4371-bc32-047fe7244a8b/root/var/svc/log/oxide-ntp:default.log.0
10232023-11-09T00:34:01.533Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_cockroachdb_dbe43835-1982-4725-9a5b-78fdd55db4ab/root/var/svc/log/oxide-cockroachdb:default.log (0 bytes)
10242023-11-09T00:34:02.585Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_cockroachdb_dbe43835-1982-4725-9a5b-78fdd55db4ab/root/var/svc/log/oxide-cockroachdb:default.log
10252023-11-09T00:34:02.612Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_cockroachdb_dbe43835-1982-4725-9a5b-78fdd55db4ab/root/var/svc/log/oxide-cockroachdb:default.log.0 (5368 bytes)
10262023-11-09T00:34:03.690Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_cockroachdb_dbe43835-1982-4725-9a5b-78fdd55db4ab/root/var/svc/log/oxide-cockroachdb:default.log.0
10272023-11-09T00:34:03.715Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_6a8caf8f-1c05-4557-9392-341dc2d33644/root/var/svc/log/oxide-crucible-agent:default.log (11147 bytes)
10282023-11-09T00:34:04.812Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_6a8caf8f-1c05-4557-9392-341dc2d33644/root/var/svc/log/oxide-crucible-agent:default.log
10292023-11-09T00:34:04.838Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_6a8caf8f-1c05-4557-9392-341dc2d33644/root/var/svc/log/oxide-crucible-agent:default.log.0 (3436 bytes)
10302023-11-09T00:34:05.911Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_6a8caf8f-1c05-4557-9392-341dc2d33644/root/var/svc/log/oxide-crucible-agent:default.log.0
10312023-11-09T00:34:05.935Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_6a8caf8f-1c05-4557-9392-341dc2d33644/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
10322023-11-09T00:34:06.989Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_6a8caf8f-1c05-4557-9392-341dc2d33644/root/var/svc/log/oxide-crucible-downstairs:default.log
10332023-11-09T00:34:07.015Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_6a8caf8f-1c05-4557-9392-341dc2d33644/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
10342023-11-09T00:34:08.086Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_6a8caf8f-1c05-4557-9392-341dc2d33644/root/var/svc/log/oxide-crucible-downstairs:default.log.0
10352023-11-09T00:34:08.122Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_6a8caf8f-1c05-4557-9392-341dc2d33644/root/var/svc/log/oxide-crucible-downstairs:downstairs-78cec65d-4e10-4758-9f67-12d988d6e5e4.log (5513 bytes)
10362023-11-09T00:34:09.215Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_6a8caf8f-1c05-4557-9392-341dc2d33644/root/var/svc/log/oxide-crucible-downstairs:downstairs-78cec65d-4e10-4758-9f67-12d988d6e5e4.log
10372023-11-09T00:34:09.240Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_b53a09ac-da02-44b9-bc9f-24b5da4c7935/root/var/svc/log/oxide-crucible-agent:default.log (11147 bytes)
10382023-11-09T00:34:10.335Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_b53a09ac-da02-44b9-bc9f-24b5da4c7935/root/var/svc/log/oxide-crucible-agent:default.log
10392023-11-09T00:34:10.359Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_b53a09ac-da02-44b9-bc9f-24b5da4c7935/root/var/svc/log/oxide-crucible-agent:default.log.0 (3435 bytes)
10402023-11-09T00:34:11.436Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_b53a09ac-da02-44b9-bc9f-24b5da4c7935/root/var/svc/log/oxide-crucible-agent:default.log.0
10412023-11-09T00:34:11.461Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_b53a09ac-da02-44b9-bc9f-24b5da4c7935/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
10422023-11-09T00:34:12.510Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_b53a09ac-da02-44b9-bc9f-24b5da4c7935/root/var/svc/log/oxide-crucible-downstairs:default.log
10432023-11-09T00:34:12.535Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_b53a09ac-da02-44b9-bc9f-24b5da4c7935/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
10442023-11-09T00:34:13.607Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_b53a09ac-da02-44b9-bc9f-24b5da4c7935/root/var/svc/log/oxide-crucible-downstairs:default.log.0
10452023-11-09T00:34:13.631Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_b53a09ac-da02-44b9-bc9f-24b5da4c7935/root/var/svc/log/oxide-crucible-downstairs:downstairs-1108280c-2723-4129-b19b-ca1a39d332a8.log (5515 bytes)
10462023-11-09T00:34:14.742Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_b53a09ac-da02-44b9-bc9f-24b5da4c7935/root/var/svc/log/oxide-crucible-downstairs:downstairs-1108280c-2723-4129-b19b-ca1a39d332a8.log
10472023-11-09T00:34:14.770Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_oximeter_0161227e-d7c8-4a75-92ee-835bd0095a06/root/var/svc/log/oxide-oximeter:default.log (208228 bytes)
10482023-11-09T00:34:15.154Zupload warning: file "/pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_oximeter_0161227e-d7c8-4a75-92ee-835bd0095a06/root/var/svc/log/oxide-oximeter:default.log" changed size mid upload: 208228 -> 240566
10492023-11-09T00:34:16.235Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_oximeter_0161227e-d7c8-4a75-92ee-835bd0095a06/root/var/svc/log/oxide-oximeter:default.log
10502023-11-09T00:34:16.261Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_oximeter_0161227e-d7c8-4a75-92ee-835bd0095a06/root/var/svc/log/oxide-oximeter:default.log.0 (12300 bytes)
10512023-11-09T00:34:17.354Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_oximeter_0161227e-d7c8-4a75-92ee-835bd0095a06/root/var/svc/log/oxide-oximeter:default.log.0
10522023-11-09T00:34:17.383Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_05c9f7d6-e991-4b81-a1b2-12c1de86c109/root/var/svc/log/oxide-crucible-agent:default.log (0 bytes)
10532023-11-09T00:34:18.855Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_05c9f7d6-e991-4b81-a1b2-12c1de86c109/root/var/svc/log/oxide-crucible-agent:default.log
10542023-11-09T00:34:18.880Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_05c9f7d6-e991-4b81-a1b2-12c1de86c109/root/var/svc/log/oxide-crucible-agent:default.log.0 (3436 bytes)
10552023-11-09T00:34:19.951Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_05c9f7d6-e991-4b81-a1b2-12c1de86c109/root/var/svc/log/oxide-crucible-agent:default.log.0
10562023-11-09T00:34:19.977Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_05c9f7d6-e991-4b81-a1b2-12c1de86c109/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
10572023-11-09T00:34:21.033Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_05c9f7d6-e991-4b81-a1b2-12c1de86c109/root/var/svc/log/oxide-crucible-downstairs:default.log
10582023-11-09T00:34:21.546Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_05c9f7d6-e991-4b81-a1b2-12c1de86c109/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
10592023-11-09T00:34:22.587Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_05c9f7d6-e991-4b81-a1b2-12c1de86c109/root/var/svc/log/oxide-crucible-downstairs:default.log.0
10602023-11-09T00:34:22.612Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_dfadc581-3055-42bd-be19-7078059e4768/root/var/svc/log/oxide-crucible-agent:default.log (10250 bytes)
10612023-11-09T00:34:23.709Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_dfadc581-3055-42bd-be19-7078059e4768/root/var/svc/log/oxide-crucible-agent:default.log
10622023-11-09T00:34:23.733Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_dfadc581-3055-42bd-be19-7078059e4768/root/var/svc/log/oxide-crucible-agent:default.log.0 (3436 bytes)
10632023-11-09T00:34:24.806Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_dfadc581-3055-42bd-be19-7078059e4768/root/var/svc/log/oxide-crucible-agent:default.log.0
10642023-11-09T00:34:24.840Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_dfadc581-3055-42bd-be19-7078059e4768/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
10652023-11-09T00:34:26.330Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_dfadc581-3055-42bd-be19-7078059e4768/root/var/svc/log/oxide-crucible-downstairs:default.log
10662023-11-09T00:34:26.354Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_dfadc581-3055-42bd-be19-7078059e4768/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
10672023-11-09T00:34:27.837Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_dfadc581-3055-42bd-be19-7078059e4768/root/var/svc/log/oxide-crucible-downstairs:default.log.0
10682023-11-09T00:34:27.865Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_dfadc581-3055-42bd-be19-7078059e4768/root/var/svc/log/oxide-crucible-downstairs:downstairs-d4149bb3-0cdb-42da-978c-cfc66c021870.log (3037 bytes)
10692023-11-09T00:34:28.534Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_dfadc581-3055-42bd-be19-7078059e4768/root/var/svc/log/oxide-crucible-downstairs:downstairs-d4149bb3-0cdb-42da-978c-cfc66c021870.log
10702023-11-09T00:34:28.567Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_pantry_1ecfdbeb-43fe-4879-bcad-3d3f5b65e961/root/var/svc/log/oxide-crucible-pantry:default.log (0 bytes)
10712023-11-09T00:34:29.624Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_pantry_1ecfdbeb-43fe-4879-bcad-3d3f5b65e961/root/var/svc/log/oxide-crucible-pantry:default.log
10722023-11-09T00:34:29.653Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_pantry_1ecfdbeb-43fe-4879-bcad-3d3f5b65e961/root/var/svc/log/oxide-crucible-pantry:default.log.0 (2333 bytes)
10732023-11-09T00:34:31.135Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_pantry_1ecfdbeb-43fe-4879-bcad-3d3f5b65e961/root/var/svc/log/oxide-crucible-pantry:default.log.0
10742023-11-09T00:34:31.171Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_external_dns_0f2d8f91-f628-4ce0-a8c4-36585dd43340/root/var/svc/log/oxide-external_dns:default.log (13059 bytes)
10752023-11-09T00:34:31.231Zupload warning: file "/pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_external_dns_0f2d8f91-f628-4ce0-a8c4-36585dd43340/root/var/svc/log/oxide-external_dns:default.log" changed size mid upload: 13059 -> 16838
10762023-11-09T00:34:32.312Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_external_dns_0f2d8f91-f628-4ce0-a8c4-36585dd43340/root/var/svc/log/oxide-external_dns:default.log
10772023-11-09T00:34:32.338Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_external_dns_0f2d8f91-f628-4ce0-a8c4-36585dd43340/root/var/svc/log/oxide-external_dns:default.log.0 (2554 bytes)
10782023-11-09T00:34:33.409Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_external_dns_0f2d8f91-f628-4ce0-a8c4-36585dd43340/root/var/svc/log/oxide-external_dns:default.log.0
10792023-11-09T00:34:33.434Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_internal_dns_ce64dfc5-d8a6-4731-afcb-823ff2324f56/root/var/svc/log/oxide-internal_dns:default.log (12519 bytes)
10802023-11-09T00:34:33.503Zupload warning: file "/pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_internal_dns_ce64dfc5-d8a6-4731-afcb-823ff2324f56/root/var/svc/log/oxide-internal_dns:default.log" changed size mid upload: 12519 -> 16275
10812023-11-09T00:34:34.578Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_internal_dns_ce64dfc5-d8a6-4731-afcb-823ff2324f56/root/var/svc/log/oxide-internal_dns:default.log
10822023-11-09T00:34:34.618Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_internal_dns_ce64dfc5-d8a6-4731-afcb-823ff2324f56/root/var/svc/log/oxide-internal_dns:default.log.0 (102471 bytes)
10832023-11-09T00:34:36.364Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_internal_dns_ce64dfc5-d8a6-4731-afcb-823ff2324f56/root/var/svc/log/oxide-internal_dns:default.log.0
10842023-11-09T00:34:36.389Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_0f95bbf7-9a67-4ba0-894b-521ede553d09/root/var/svc/log/oxide-crucible-agent:default.log (0 bytes)
10852023-11-09T00:34:37.437Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_0f95bbf7-9a67-4ba0-894b-521ede553d09/root/var/svc/log/oxide-crucible-agent:default.log
10862023-11-09T00:34:37.461Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_0f95bbf7-9a67-4ba0-894b-521ede553d09/root/var/svc/log/oxide-crucible-agent:default.log.0 (3436 bytes)
10872023-11-09T00:34:38.534Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_0f95bbf7-9a67-4ba0-894b-521ede553d09/root/var/svc/log/oxide-crucible-agent:default.log.0
10882023-11-09T00:34:38.559Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_0f95bbf7-9a67-4ba0-894b-521ede553d09/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
10892023-11-09T00:34:39.622Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_0f95bbf7-9a67-4ba0-894b-521ede553d09/root/var/svc/log/oxide-crucible-downstairs:default.log
10902023-11-09T00:34:39.647Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_0f95bbf7-9a67-4ba0-894b-521ede553d09/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
10912023-11-09T00:34:40.719Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_0f95bbf7-9a67-4ba0-894b-521ede553d09/root/var/svc/log/oxide-crucible-downstairs:default.log.0
10922023-11-09T00:34:40.744Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_ceac9411-a7c4-4c50-89bb-6fc10b1fbd34/root/var/svc/log/oxide-crucible-agent:default.log (10680 bytes)
10932023-11-09T00:34:42.249Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_ceac9411-a7c4-4c50-89bb-6fc10b1fbd34/root/var/svc/log/oxide-crucible-agent:default.log
10942023-11-09T00:34:42.273Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_ceac9411-a7c4-4c50-89bb-6fc10b1fbd34/root/var/svc/log/oxide-crucible-agent:default.log.0 (3433 bytes)
10952023-11-09T00:34:43.756Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_ceac9411-a7c4-4c50-89bb-6fc10b1fbd34/root/var/svc/log/oxide-crucible-agent:default.log.0
10962023-11-09T00:34:43.781Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_ceac9411-a7c4-4c50-89bb-6fc10b1fbd34/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
10972023-11-09T00:34:44.828Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_ceac9411-a7c4-4c50-89bb-6fc10b1fbd34/root/var/svc/log/oxide-crucible-downstairs:default.log
10982023-11-09T00:34:44.853Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_ceac9411-a7c4-4c50-89bb-6fc10b1fbd34/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
10992023-11-09T00:34:45.927Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_ceac9411-a7c4-4c50-89bb-6fc10b1fbd34/root/var/svc/log/oxide-crucible-downstairs:default.log.0
11002023-11-09T00:34:45.951Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_ceac9411-a7c4-4c50-89bb-6fc10b1fbd34/root/var/svc/log/oxide-crucible-downstairs:downstairs-b7b97ca3-daf4-4a18-b86f-e393af34b2b7.log (5510 bytes)
11012023-11-09T00:34:47.070Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_ceac9411-a7c4-4c50-89bb-6fc10b1fbd34/root/var/svc/log/oxide-crucible-downstairs:downstairs-b7b97ca3-daf4-4a18-b86f-e393af34b2b7.log
11022023-11-09T00:34:47.095Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_e81a1588-a912-4f05-b561-b5522ce91a34/root/var/svc/log/oxide-crucible-agent:default.log (10246 bytes)
11032023-11-09T00:34:48.194Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_e81a1588-a912-4f05-b561-b5522ce91a34/root/var/svc/log/oxide-crucible-agent:default.log
11042023-11-09T00:34:48.219Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_e81a1588-a912-4f05-b561-b5522ce91a34/root/var/svc/log/oxide-crucible-agent:default.log.0 (3436 bytes)
11052023-11-09T00:34:49.735Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_e81a1588-a912-4f05-b561-b5522ce91a34/root/var/svc/log/oxide-crucible-agent:default.log.0
11062023-11-09T00:34:49.763Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_e81a1588-a912-4f05-b561-b5522ce91a34/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
11072023-11-09T00:34:50.387Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_e81a1588-a912-4f05-b561-b5522ce91a34/root/var/svc/log/oxide-crucible-downstairs:default.log
11082023-11-09T00:34:50.416Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_e81a1588-a912-4f05-b561-b5522ce91a34/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
11092023-11-09T00:34:51.502Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_e81a1588-a912-4f05-b561-b5522ce91a34/root/var/svc/log/oxide-crucible-downstairs:default.log.0
11102023-11-09T00:34:51.527Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_e81a1588-a912-4f05-b561-b5522ce91a34/root/var/svc/log/oxide-crucible-downstairs:downstairs-da4725b9-69bd-4877-be74-a37bfbfa34c7.log (3039 bytes)
11112023-11-09T00:34:52.616Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_e81a1588-a912-4f05-b561-b5522ce91a34/root/var/svc/log/oxide-crucible-downstairs:downstairs-da4725b9-69bd-4877-be74-a37bfbfa34c7.log
11122023-11-09T00:34:52.657Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_internal_dns_aea45032-6817-4a0a-9c7c-5b4242786b26/root/var/svc/log/oxide-internal_dns:default.log (20071 bytes)
11132023-11-09T00:34:52.740Zupload warning: file "/pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_internal_dns_aea45032-6817-4a0a-9c7c-5b4242786b26/root/var/svc/log/oxide-internal_dns:default.log" changed size mid upload: 20071 -> 23827
11142023-11-09T00:34:53.814Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_internal_dns_aea45032-6817-4a0a-9c7c-5b4242786b26/root/var/svc/log/oxide-internal_dns:default.log
11152023-11-09T00:34:53.840Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_internal_dns_aea45032-6817-4a0a-9c7c-5b4242786b26/root/var/svc/log/oxide-internal_dns:default.log.0 (97756 bytes)
11162023-11-09T00:34:55.189Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_internal_dns_aea45032-6817-4a0a-9c7c-5b4242786b26/root/var/svc/log/oxide-internal_dns:default.log.0
11172023-11-09T00:34:55.220Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_nexus_8b30688c-aaaf-46e2-af4c-70534610b253/root/var/svc/log/oxide-nexus:default.log (642229 bytes)
11182023-11-09T00:34:56.185Zupload warning: file "/pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_nexus_8b30688c-aaaf-46e2-af4c-70534610b253/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 642229 -> 705148
11192023-11-09T00:34:57.258Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_nexus_8b30688c-aaaf-46e2-af4c-70534610b253/root/var/svc/log/oxide-nexus:default.log
11202023-11-09T00:34:57.282Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_nexus_8b30688c-aaaf-46e2-af4c-70534610b253/root/var/svc/log/oxide-nexus:default.log.0 (120 bytes)
11212023-11-09T00:34:58.355Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_nexus_8b30688c-aaaf-46e2-af4c-70534610b253/root/var/svc/log/oxide-nexus:default.log.0
11222023-11-09T00:34:58.379Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_cockroachdb_ebe5b952-ab39-4df9-ab4f-e47413c39a38/root/var/svc/log/oxide-cockroachdb:default.log (0 bytes)
11232023-11-09T00:34:59.427Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_cockroachdb_ebe5b952-ab39-4df9-ab4f-e47413c39a38/root/var/svc/log/oxide-cockroachdb:default.log
11242023-11-09T00:34:59.452Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_cockroachdb_ebe5b952-ab39-4df9-ab4f-e47413c39a38/root/var/svc/log/oxide-cockroachdb:default.log.0 (5368 bytes)
11252023-11-09T00:35:00.534Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_cockroachdb_ebe5b952-ab39-4df9-ab4f-e47413c39a38/root/var/svc/log/oxide-cockroachdb:default.log.0
11262023-11-09T00:35:00.559Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_293b6057-1361-4049-bc8b-900444b3977d/root/var/svc/log/oxide-crucible-agent:default.log (0 bytes)
11272023-11-09T00:35:01.608Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_293b6057-1361-4049-bc8b-900444b3977d/root/var/svc/log/oxide-crucible-agent:default.log
11282023-11-09T00:35:01.635Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_293b6057-1361-4049-bc8b-900444b3977d/root/var/svc/log/oxide-crucible-agent:default.log.0 (3435 bytes)
11292023-11-09T00:35:02.713Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_293b6057-1361-4049-bc8b-900444b3977d/root/var/svc/log/oxide-crucible-agent:default.log.0
11302023-11-09T00:35:02.738Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_293b6057-1361-4049-bc8b-900444b3977d/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
11312023-11-09T00:35:03.787Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_293b6057-1361-4049-bc8b-900444b3977d/root/var/svc/log/oxide-crucible-downstairs:default.log
11322023-11-09T00:35:03.811Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_293b6057-1361-4049-bc8b-900444b3977d/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
11332023-11-09T00:35:04.885Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_293b6057-1361-4049-bc8b-900444b3977d/root/var/svc/log/oxide-crucible-downstairs:default.log.0
11342023-11-09T00:35:04.928Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_nexus_55e5637c-cc33-48f0-9ca8-3a01cac92b10/root/var/svc/log/oxide-nexus:default.log (4706153 bytes)
11352023-11-09T00:35:08.961Zupload warning: file "/pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_nexus_55e5637c-cc33-48f0-9ca8-3a01cac92b10/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 4706153 -> 4769678
11362023-11-09T00:35:10.041Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_nexus_55e5637c-cc33-48f0-9ca8-3a01cac92b10/root/var/svc/log/oxide-nexus:default.log
11372023-11-09T00:35:10.066Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_nexus_55e5637c-cc33-48f0-9ca8-3a01cac92b10/root/var/svc/log/oxide-nexus:default.log.0 (75 bytes)
11382023-11-09T00:35:11.139Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_nexus_55e5637c-cc33-48f0-9ca8-3a01cac92b10/root/var/svc/log/oxide-nexus:default.log.0
11392023-11-09T00:35:11.169Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_74a81a37-7a41-4825-8981-fa89abc82e76/root/var/svc/log/oxide-cockroachdb:default.log (0 bytes)
11402023-11-09T00:35:12.216Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_74a81a37-7a41-4825-8981-fa89abc82e76/root/var/svc/log/oxide-cockroachdb:default.log
11412023-11-09T00:35:12.661Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_74a81a37-7a41-4825-8981-fa89abc82e76/root/var/svc/log/oxide-cockroachdb:default.log.0 (5369 bytes)
11422023-11-09T00:35:13.403Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_74a81a37-7a41-4825-8981-fa89abc82e76/root/var/svc/log/oxide-cockroachdb:default.log.0
11432023-11-09T00:35:13.429Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_9d46e17f-d4bc-460b-a80b-11168877173c/root/var/svc/log/oxide-crucible-agent:default.log (10250 bytes)
11442023-11-09T00:35:14.629Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_9d46e17f-d4bc-460b-a80b-11168877173c/root/var/svc/log/oxide-crucible-agent:default.log
11452023-11-09T00:35:14.698Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_9d46e17f-d4bc-460b-a80b-11168877173c/root/var/svc/log/oxide-crucible-agent:default.log.0 (3435 bytes)
11462023-11-09T00:35:29.372Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_9d46e17f-d4bc-460b-a80b-11168877173c/root/var/svc/log/oxide-crucible-agent:default.log.0
11472023-11-09T00:35:29.684Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_9d46e17f-d4bc-460b-a80b-11168877173c/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
11482023-11-09T00:35:30.844Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_9d46e17f-d4bc-460b-a80b-11168877173c/root/var/svc/log/oxide-crucible-downstairs:default.log
11492023-11-09T00:35:30.925Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_9d46e17f-d4bc-460b-a80b-11168877173c/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
11502023-11-09T00:35:32.099Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_9d46e17f-d4bc-460b-a80b-11168877173c/root/var/svc/log/oxide-crucible-downstairs:default.log.0
11512023-11-09T00:35:32.166Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_9d46e17f-d4bc-460b-a80b-11168877173c/root/var/svc/log/oxide-crucible-downstairs:downstairs-95a715b6-574a-4676-aa31-4a52052c0b74.log (3039 bytes)
11522023-11-09T00:35:33.263Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_9d46e17f-d4bc-460b-a80b-11168877173c/root/var/svc/log/oxide-crucible-downstairs:downstairs-95a715b6-574a-4676-aa31-4a52052c0b74.log
11532023-11-09T00:35:33.302Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_9d1207a6-0bed-46a9-b703-e6a253b27257/root/var/svc/log/oxide-crucible-pantry:default.log (0 bytes)
11542023-11-09T00:35:34.355Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_9d1207a6-0bed-46a9-b703-e6a253b27257/root/var/svc/log/oxide-crucible-pantry:default.log
11552023-11-09T00:35:34.389Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_9d1207a6-0bed-46a9-b703-e6a253b27257/root/var/svc/log/oxide-crucible-pantry:default.log.0 (2337 bytes)
11562023-11-09T00:35:35.513Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_9d1207a6-0bed-46a9-b703-e6a253b27257/root/var/svc/log/oxide-crucible-pantry:default.log.0
11572023-11-09T00:35:35.544Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_1c2c4826-d978-4371-bc32-047fe7244a8b/root/var/log/chrony/measurements.log (9042 bytes)
11582023-11-09T00:35:35.590Zupload warning: file "/pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_1c2c4826-d978-4371-bc32-047fe7244a8b/root/var/log/chrony/measurements.log" changed size mid upload: 9042 -> 11645
11592023-11-09T00:35:36.664Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_1c2c4826-d978-4371-bc32-047fe7244a8b/root/var/log/chrony/measurements.log
11602023-11-09T00:35:36.694Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_1c2c4826-d978-4371-bc32-047fe7244a8b/root/var/log/chrony/statistics.log (5616 bytes)
11612023-11-09T00:35:36.741Zupload warning: file "/pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_1c2c4826-d978-4371-bc32-047fe7244a8b/root/var/log/chrony/statistics.log" changed size mid upload: 5616 -> 7488
11622023-11-09T00:35:37.850Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_1c2c4826-d978-4371-bc32-047fe7244a8b/root/var/log/chrony/statistics.log
11632023-11-09T00:35:37.918Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_1c2c4826-d978-4371-bc32-047fe7244a8b/root/var/log/chrony/tracking.log (1455 bytes)
11642023-11-09T00:35:37.956Zupload warning: file "/pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_1c2c4826-d978-4371-bc32-047fe7244a8b/root/var/log/chrony/tracking.log" changed size mid upload: 1455 -> 1719
11652023-11-09T00:35:39.087Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_1c2c4826-d978-4371-bc32-047fe7244a8b/root/var/log/chrony/tracking.log