01HEPK0M3Z25XX3A880PS6JGAR: helios / deploy

BasicConfig {
    output_rules: [
        "%/var/svc/log/oxide-sled-agent:default.log*",
        "%/pool/ext/*/crypt/zone/oxz_*/root/var/svc/log/oxide-*.log*",
        "%/pool/ext/*/crypt/zone/oxz_*/root/var/svc/log/system-illumos-*.log*",
        "%/pool/ext/*/crypt/zone/oxz_ntp_*/root/var/log/chrony/*.log*",
        "!/pool/ext/*/crypt/zone/oxz_propolis-server_*/root/var/svc/log/*.log*",
        "%/pool/ext/*/crypt/debug/global/oxide-sled-agent:default.log.*",
        "%/pool/ext/*/crypt/debug/oxz_*/oxide-*.log.*",
        "%/pool/ext/*/crypt/debug/oxz_*/system-illumos-*.log.*",
        "!/pool/ext/*/crypt/debug/oxz_propolis-server_*/*.log.*",
    ],
    rust_toolchain: None,
    target: Some(
        "lab-2.0-opte-0.25",
    ),
    access_repos: [],
    publish: [],
    skip_clone: true,
}

Buildomat Job: 01HEPK602HZDRWJH1HWQHZH1R4

Tags:

Artefacts:

Output:

SEQ GLOBAL TIME DETAILS
12023-11-08T05:37:50.428Zjob dependencies complete; ready to run (waiting for 1 h 6 m 27 s)
22023-11-08T06:39:33.410Zjob assigned to worker 01HEPTFAT9D5V53VHDSY2A971Y (queued for 1 h 1 m 42 s)
32023-11-08T06:39:33.570Zhttp://172.20.0.5:9969/ipxe/buskin... ok
42023-11-08T06:39:33.631Zhttp://172.20.0.5:9969/os/buskin/platform/i86pc/kernel/amd64/unix... ok
52023-11-08T06:39:33.727Zhttp://172.20.0.5:9969/os/buskin/platform/i86pc/amd64/boot_archive... ok
62023-11-08T06:39:33.771Zhttp://172.20.0.5:9969/os/buskin/platform/i86pc/amd64/boot_archive.hash... ok
72023-11-08T06:39:33.818Zhttp://172.20.0.5:9969/postboot/buskin... ok
82023-11-08T06:39:33.858ZwcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U
92023-11-08T06:39:34.405Zcpu0: microcode has been updated from version 0xa001133 to 0xa0011d1
102023-11-08T06:39:34.445ZOxide Helios Version helios-2.0.22241 64-bit
112023-11-08T06:39:34.481ZConfiguring devices.
122023-11-08T06:39:34.521Z##################################################
132023-11-08T06:39:34.568Z#### ############# ##
142023-11-08T06:39:34.603Z### ### ############ ##
152023-11-08T06:39:34.641Z## ### # ## ### ## Oxide Computer Company ##
162023-11-08T06:39:34.678Z## ## ## ### # ### ##
172023-11-08T06:39:34.716Z## # ### #### #### This Station Under ##
182023-11-08T06:39:34.757Z### ### #### # ### Computer Control ##
192023-11-08T06:39:34.811Z#### #### ### ## ##
202023-11-08T06:39:34.856Z##################################################
212023-11-08T06:39:34.894ZHostname: buskin
222023-11-08T06:39:34.937ZLoading smf(7) serice descriptions:
232023-11-08T06:39:35.233Zdownloading input: /input/ci-tools/work/end-to-end-tests/bootstrap-a6f7429594f412b4.gz
242023-11-08T06:39:37.708Zdownloaded input: /input/ci-tools/work/end-to-end-tests/bootstrap-a6f7429594f412b4.gz
252023-11-08T06:39:37.738Zdownloading input: /input/ci-tools/work/end-to-end-tests/bootstrap.gz
262023-11-08T06:40:37.058Zdownloaded input: /input/ci-tools/work/end-to-end-tests/bootstrap.gz
272023-11-08T06:40:37.090Zdownloading input: /input/ci-tools/work/end-to-end-tests/end_to_end_tests-294235f008612b3a.gz
282023-11-08T06:41:41.649Zdownloaded input: /input/ci-tools/work/end-to-end-tests/end_to_end_tests-294235f008612b3a.gz
292023-11-08T06:41:41.710Zdownloading input: /input/ci-tools/work/caboose-util.gz
302023-11-08T06:41:43.517Zdownloaded input: /input/ci-tools/work/caboose-util.gz
312023-11-08T06:41:43.555Zdownloading input: /input/ci-tools/work/tufaceous.gz
322023-11-08T06:41:52.014Zdownloaded input: /input/ci-tools/work/tufaceous.gz
332023-11-08T06:41:52.092Zdownloading input: /input/package/work/version.txt
342023-11-08T06:41:52.178Zdownloaded input: /input/package/work/version.txt
352023-11-08T06:41:52.239Zdownloading input: /input/package/work/package.tar.gz
362023-11-08T06:42:48.966Zdownloaded input: /input/package/work/package.tar.gz
372023-11-08T06:42:50.250Zdownloading input: /input/package/work/global-zone-packages.tar.gz
382023-11-08T06:43:45.851Zdownloaded input: /input/package/work/global-zone-packages.tar.gz
392023-11-08T06:43:45.887Zdownloading input: /input/package/work/trampoline-global-zone-packages.tar.gz
402023-11-08T06:44:11.410Zdownloaded input: /input/package/work/trampoline-global-zone-packages.tar.gz
412023-11-08T06:44:11.454Zdownloading input: /input/package/work/zones/clickhouse.tar.gz
422023-11-08T06:45:31.495Zdownloaded input: /input/package/work/zones/clickhouse.tar.gz
432023-11-08T06:45:31.551Zdownloading input: /input/package/work/zones/clickhouse_keeper.tar.gz
442023-11-08T06:46:55.545Zdownloaded input: /input/package/work/zones/clickhouse_keeper.tar.gz
452023-11-08T06:46:55.800Zdownloading input: /input/package/work/zones/cockroachdb.tar.gz
462023-11-08T06:47:45.129Zdownloaded input: /input/package/work/zones/cockroachdb.tar.gz
472023-11-08T06:47:45.156Zdownloading input: /input/package/work/zones/crucible-pantry.tar.gz
482023-11-08T06:48:03.277Zdownloaded input: /input/package/work/zones/crucible-pantry.tar.gz
492023-11-08T06:48:03.303Zdownloading input: /input/package/work/zones/crucible.tar.gz
502023-11-08T06:48:23.391Zdownloaded input: /input/package/work/zones/crucible.tar.gz
512023-11-08T06:48:23.484Zdownloading input: /input/package/work/zones/external-dns.tar.gz
522023-11-08T06:48:37.529Zdownloaded input: /input/package/work/zones/external-dns.tar.gz
532023-11-08T06:48:37.556Zdownloading input: /input/package/work/zones/internal-dns.tar.gz
542023-11-08T06:48:48.089Zdownloaded input: /input/package/work/zones/internal-dns.tar.gz
552023-11-08T06:48:48.115Zdownloading input: /input/package/work/zones/ntp.tar.gz
562023-11-08T06:48:48.139Zdownloaded input: /input/package/work/zones/ntp.tar.gz
572023-11-08T06:48:48.165Zdownloading input: /input/package/work/zones/omicron-gateway-asic.tar.gz
582023-11-08T06:48:53.115Zdownloaded input: /input/package/work/zones/omicron-gateway-asic.tar.gz
592023-11-08T06:48:53.149Zdownloading input: /input/package/work/zones/omicron-gateway-softnpu.tar.gz
602023-11-08T06:49:01.777Zdownloaded input: /input/package/work/zones/omicron-gateway-softnpu.tar.gz
612023-11-08T06:49:01.813Zdownloading input: /input/package/work/zones/omicron-nexus-single-sled.tar.gz
622023-11-08T06:49:32.679Zdownloaded input: /input/package/work/zones/omicron-nexus-single-sled.tar.gz
632023-11-08T06:49:33.128Zdownloading input: /input/package/work/zones/omicron-nexus.tar.gz
642023-11-08T06:49:56.251Zdownloaded input: /input/package/work/zones/omicron-nexus.tar.gz
652023-11-08T06:49:56.282Zdownloading input: /input/package/work/zones/overlay.tar.gz
662023-11-08T06:49:56.329Zdownloaded input: /input/package/work/zones/overlay.tar.gz
672023-11-08T06:49:56.359Zdownloading input: /input/package/work/zones/oximeter-collector.tar.gz
682023-11-08T06:50:02.383Zdownloaded input: /input/package/work/zones/oximeter-collector.tar.gz
692023-11-08T06:50:02.408Zdownloading input: /input/package/work/zones/propolis-server.tar.gz
702023-11-08T06:50:46.833Zdownloaded input: /input/package/work/zones/propolis-server.tar.gz
712023-11-08T06:50:46.874Zdownloading input: /input/package/work/zones/switch-asic.tar.gz
722023-11-08T06:52:14.988Zdownloaded input: /input/package/work/zones/switch-asic.tar.gz
732023-11-08T06:52:15.085Zdownloading input: /input/package/work/zones/switch-softnpu.tar.gz
742023-11-08T06:53:23.718Zdownloaded input: /input/package/work/zones/switch-softnpu.tar.gz
 
752023-11-08T06:53:23.760Zstarting task 0: "setup"
762023-11-08T06:53:23.796Z++ uname -s
772023-11-08T06:53:23.821Z+ kern=SunOS
782023-11-08T06:53:23.887Z+ case "$kern" in
792023-11-08T06:53:23.915Z+ groupadd -g 12345 build
802023-11-08T06:53:23.940Z+ useradd -u 12345 -g build -d /home/build -s /bin/bash -c build -P 'Primary Administrator' build
812023-11-08T06:53:25.813Z+ zfs create -o mountpoint=/work rpool/work
822023-11-08T06:53:25.855Z++ awk '$2 == "/home" { print $3 }' /etc/mnttab
832023-11-08T06:53:25.917Z+ home_fs=zfs
842023-11-08T06:53:25.973Z+ [[ zfs == autofs ]]
852023-11-08T06:53:26.038Z+ mkdir -p /home/build
862023-11-08T06:53:26.070Z+ chown build:build /home/build /work
872023-11-08T06:53:27.842Z+ chmod 0700 /home/build /work
882023-11-08T06:53:27.898Zprocess exited: duration 4058 ms, exit code 0
 
892023-11-08T06:53:28.040Zstarting task 1: "authentication"
902023-11-08T06:53:28.175Zprocess exited: duration 45 ms, exit code 0
 
912023-11-08T06:53:28.246Zstarting task 2: "build"
922023-11-08T06:53:28.285Z+ trap _exit_trap EXIT
932023-11-08T06:53:28.315Z+ pfexec mdb -kwe 'secpolicy_fs_linkdir/v 55 48 89 e5 b8 01 00 00 00 5d c3'
942023-11-08T06:53:28.484Zsecpolicy_fs_linkdir: 0x55 = 0x55
952023-11-08T06:53:28.526Zsecpolicy_fs_linkdir+1: 0xb8 = 0x48
962023-11-08T06:53:28.561Zsecpolicy_fs_linkdir+2: 0x1 = 0x89
972023-11-08T06:53:29.010Zsecpolicy_fs_linkdir+3: 0 = 0xe5
982023-11-08T06:53:29.095Zsecpolicy_fs_linkdir+4: 0 = 0xb8
992023-11-08T06:53:29.147Zsecpolicy_fs_linkdir+5: 0 = 0x1
1002023-11-08T06:53:29.225Zsecpolicy_fs_linkdir+6: 0x48 = 0x0
1012023-11-08T06:53:29.265Zsecpolicy_fs_linkdir+7: 0x89 = 0x0
1022023-11-08T06:53:29.294Zsecpolicy_fs_linkdir+8: 0xe5 = 0x0
1032023-11-08T06:53:29.320Zsecpolicy_fs_linkdir+9: 0x48 = 0x5d
1042023-11-08T06:53:29.387Zsecpolicy_fs_linkdir+0xa: 0x83 = 0xc3
1052023-11-08T06:53:29.412Z+ [[ -d /opt/oxide ]]
1062023-11-08T06:53:29.443Z+ pfexec mv /opt/oxide /opt/oxide-underneath
1072023-11-08T06:53:29.473Z+ pfexec mkdir /opt/oxide
1082023-11-08T06:53:29.498Z+ pfexec mount -F tmpfs -O swap /opt/oxide
1092023-11-08T06:53:29.529Z+ [[ -d /opt/oxide-underneath ]]
1102023-11-08T06:53:29.968Z+ cd /opt/oxide-underneath
1112023-11-08T06:53:30.042Z+ pfexec tar ceEp@/f - .
1122023-11-08T06:53:30.085Z+ cd /opt/oxide
1132023-11-08T06:53:30.123Z+ pfexec tar xveEp@/f -
1142023-11-08T06:53:30.213Zx ., 0 bytes, 0 tape blocks
1152023-11-08T06:53:30.286Zx ./opte, 0 bytes, 0 tape blocks
1162023-11-08T06:53:30.326Zx ./opte/bin, 0 bytes, 0 tape blocks
1172023-11-08T06:53:30.388Zx ./opte/bin/opteadm, 50714832 bytes, 99053 tape blocks
1182023-11-08T06:53:30.424Z+ pfexec rm -rf /opt/oxide-underneath
1192023-11-08T06:53:30.470Z+ pfexec /sbin/zfs create -o mountpoint=/zone rpool/zone
1202023-11-08T06:53:30.517Z+ pfexec mkdir /opt/oxide/work
1212023-11-08T06:53:30.578Z+ pfexec chown build:build /opt/oxide/work
1222023-11-08T06:53:30.649Z+ cd /opt/oxide/work
1232023-11-08T06:53:30.686Z+ ptime -m tar xvzf /input/package/work/package.tar.gz
1242023-11-08T06:53:30.718ZDecompressing '/input/package/work/package.tar.gz' with '/usr/bin/gzcat'...
1252023-11-08T06:53:30.780Zx out/mg-ddm-gz.tar, 62974464 bytes, 122997 tape blocks
1262023-11-08T06:53:30.829Zx out/omicron-sled-agent.tar, 122453504 bytes, 239167 tape blocks
1272023-11-08T06:53:30.885Zx out/target/test, 75 bytes, 1 tape blocks
1282023-11-08T06:53:30.927Zx out/npuzone/npuzone, 37381232 bytes, 73011 tape blocks
1292023-11-08T06:53:30.969Zx package-manifest.toml, 22037 bytes, 44 tape blocks
1302023-11-08T06:53:31.012Zx smf/sled-agent/non-gimlet/config.toml, 3102 bytes, 7 tape blocks
1312023-11-08T06:53:31.049Zx target/release/omicron-package, 42419680 bytes, 82851 tape blocks
1322023-11-08T06:53:31.100Zx tools/create_virtual_hardware.sh, 3067 bytes, 6 tape blocks
1332023-11-08T06:53:31.145Zx tools/virtual_hardware.sh, 3408 bytes, 7 tape blocks
1342023-11-08T06:53:31.185Zx tools/scrimlet/create-softnpu-zone.sh, 619 bytes, 2 tape blocks
1352023-11-08T06:53:31.223Zx tools/scrimlet/destroy-softnpu-zone.sh, 195 bytes, 1 tape blocks
1362023-11-08T06:53:31.434Zx tools/scrimlet/softnpu-init.sh, 2148 bytes, 5 tape blocks
1372023-11-08T06:53:31.465Zx tools/scrimlet/softnpu-zone.txt, 330 bytes, 1 tape blocks
1382023-11-08T06:53:31.501Zx tools/scrimlet/softnpu.toml, 171 bytes, 1 tape blocks
1392023-11-08T06:53:31.528Z
1402023-11-08T06:53:31.554Zreal 1.601133811
1412023-11-08T06:53:31.595Zuser 1.444123026
1422023-11-08T06:53:31.666Zsys 0.271208522
1432023-11-08T06:53:31.756Ztrap 0.000072601
1442023-11-08T06:53:31.806Ztflt 0.000000000
1452023-11-08T06:53:31.843Zdflt 0.000000000
1462023-11-08T06:53:31.973Zkflt 0.000000000
1472023-11-08T06:53:32.013Zlock 0.000000000
1482023-11-08T06:53:32.059Zslp 1.447217970
1492023-11-08T06:53:32.103Zlat 0.037742213
1502023-11-08T06:53:32.583Zstop 0.000116113
1512023-11-08T06:53:32.655Z+ cp /input/package/work/zones/clickhouse_keeper.tar.gz /input/package/work/zones/clickhouse.tar.gz /input/package/work/zones/cockroachdb.tar.gz /input/package/work/zones/crucible-pantry.tar.gz /input/package/work/zones/crucible.tar.gz /input/package/work/zones/external-dns.tar.gz /input/package/work/zones/internal-dns.tar.gz /input/package/work/zones/ntp.tar.gz /input/package/work/zones/omicron-gateway-asic.tar.gz /input/package/work/zones/omicron-gateway-softnpu.tar.gz /input/package/work/zones/omicron-nexus-single-sled.tar.gz /input/package/work/zones/omicron-nexus.tar.gz /input/package/work/zones/overlay.tar.gz /input/package/work/zones/oximeter-collector.tar.gz /input/package/work/zones/propolis-server.tar.gz /input/package/work/zones/switch-asic.tar.gz /input/package/work/zones/switch-softnpu.tar.gz out/
1522023-11-08T06:53:32.695Z+ mv out/omicron-nexus-single-sled.tar.gz out/omicron-nexus.tar.gz
1532023-11-08T06:53:32.723Z+ mkdir tests
1542023-11-08T06:53:32.763Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1552023-11-08T06:53:32.798Z+ ptime -m gunzip
1562023-11-08T06:53:32.851Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap-a6f7429594f412b4
1572023-11-08T06:53:32.894Z
1582023-11-08T06:53:32.943Zreal 0.141327241
1592023-11-08T06:53:32.999Zuser 0.129916911
1602023-11-08T06:53:33.029Zsys 0.011160242
1612023-11-08T06:53:33.079Ztrap 0.000015010
1622023-11-08T06:53:33.117Ztflt 0.000000000
1632023-11-08T06:53:33.159Zdflt 0.000000000
1642023-11-08T06:53:33.195Zkflt 0.000000000
1652023-11-08T06:53:33.231Zlock 0.000000000
1662023-11-08T06:53:33.681Zslp 0.000020660
1672023-11-08T06:53:33.715Zlat 0.000028701
1682023-11-08T06:53:33.749Zstop 0.000122552
1692023-11-08T06:53:33.822Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap-a6f7429594f412b4
1702023-11-08T06:53:33.871Z+ chmod a+x tests/bootstrap-a6f7429594f412b4
1712023-11-08T06:53:33.901Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1722023-11-08T06:53:33.930Z+ ptime -m gunzip
1732023-11-08T06:53:33.957Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap
1742023-11-08T06:53:34.001Z
1752023-11-08T06:53:34.038Zreal 2.283719100
1762023-11-08T06:53:34.070Zuser 2.152207477
1772023-11-08T06:53:34.111Zsys 0.131140633
1782023-11-08T06:53:34.146Ztrap 0.000144504
1792023-11-08T06:53:34.184Ztflt 0.000000000
1802023-11-08T06:53:34.224Zdflt 0.000000000
1812023-11-08T06:53:34.263Zkflt 0.000000000
1822023-11-08T06:53:34.334Zlock 0.000000000
1832023-11-08T06:53:34.391Zslp 0.000020370
1842023-11-08T06:53:34.438Zlat 0.000027970
1852023-11-08T06:53:34.478Zstop 0.000109233
1862023-11-08T06:53:34.516Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap
1872023-11-08T06:53:34.570Z+ chmod a+x tests/bootstrap
1882023-11-08T06:53:34.643Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1892023-11-08T06:53:34.699Z+ ptime -m gunzip
1902023-11-08T06:53:34.749Z++ basename /input/ci-tools/work/end-to-end-tests/end_to_end_tests-294235f008612b3a
1912023-11-08T06:53:36.393Z
1922023-11-08T06:53:36.448Zreal 2.629431832
1932023-11-08T06:53:36.499Zuser 2.451469668
1942023-11-08T06:53:36.542Zsys 0.177499022
1952023-11-08T06:53:36.581Ztrap 0.000226165
1962023-11-08T06:53:36.614Ztflt 0.000000000
1972023-11-08T06:53:36.651Zdflt 0.000000000
1982023-11-08T06:53:36.685Zkflt 0.000000000
1992023-11-08T06:53:36.724Zlock 0.000000000
2002023-11-08T06:53:36.811Zslp 0.000023540
2012023-11-08T06:53:36.847Zlat 0.000022199
2022023-11-08T06:53:36.882Zstop 0.000126953
2032023-11-08T06:53:36.935Z++ basename /input/ci-tools/work/end-to-end-tests/end_to_end_tests-294235f008612b3a
2042023-11-08T06:53:37.424Z+ chmod a+x tests/end_to_end_tests-294235f008612b3a
2052023-11-08T06:53:37.454Z+ bmat address ls
2062023-11-08T06:53:37.491ZNAME CIDR FIRST LAST COUNT
2072023-11-08T06:53:37.543Zextra 10.151.1.0/24 10.151.1.164 10.151.1.227 64
2082023-11-08T06:53:38.015Z++ bmat address ls -f extra -Ho first,last,count
2092023-11-08T06:53:38.044Z+ set -- 10.151.1.164 10.151.1.227 64
2102023-11-08T06:53:38.101Z+ EXTRA_IP_START=10.151.1.164
2112023-11-08T06:53:38.182Z+ EXTRA_IP_END=10.151.1.227
2122023-11-08T06:53:38.241Z+ EXTRA_IP_COUNT=64
2132023-11-08T06:53:38.298Z+ (( EXTRA_IP_COUNT >= 32 ))
2142023-11-08T06:53:38.334Z+ EXTRA_IP_BASE=10.151.1
2152023-11-08T06:53:38.368Z+ EXTRA_IP_FOCTET=164
2162023-11-08T06:53:38.405Z+ EXTRA_IP_LOCTET=227
2172023-11-08T06:53:38.435Z+ SERVICE_IP_POOL_START=10.151.1.164
2182023-11-08T06:53:38.461Z+ SERVICE_IP_POOL_END=10.151.1.173
2192023-11-08T06:53:38.488Z+ DNS_IP1=10.151.1.164
2202023-11-08T06:53:38.524Z+ DNS_IP2=10.151.1.165
2212023-11-08T06:53:38.552Z+ UPLINK_IP=10.151.1.174
2222023-11-08T06:53:38.578Z+ IPPOOL_START=10.151.1.175
2232023-11-08T06:53:38.625Z+ IPPOOL_END=10.151.1.227
2242023-11-08T06:53:38.660Z++ ipadm show-addr -po type,addr
2252023-11-08T06:53:38.798Z++ awk '-F[:/]' '$1 == "dhcp" {print $2}'
2262023-11-08T06:53:38.835Z+ GATEWAY_IP=10.151.1.100
2272023-11-08T06:53:38.910Z+ [[ -n 10.151.1.100 ]]
2282023-11-08T06:53:38.965Z+ ping -s 10.151.1.100 56 1
2292023-11-08T06:53:39.011ZPING 10.151.1.100: 56 data bytes
2302023-11-08T06:53:39.071Z64 bytes from 10.151.1.100: icmp_seq=0. time=0.011 ms
2312023-11-08T06:53:39.099Z
2322023-11-08T06:53:39.130Z----10.151.1.100 PING Statistics----
2332023-11-08T06:53:39.181Z1 packets transmitted, 1 packets received, 0% packet loss
2342023-11-08T06:53:39.212Zround-trip (ms) min/avg/max/stddev = 0.011/0.011/0.011/-nan
2352023-11-08T06:53:39.263Z++ arp -an
2362023-11-08T06:53:39.294Z++ awk -vgw=10.151.1.100 '$2 == gw {print $NF}'
2372023-11-08T06:53:39.341Z+ GATEWAY_MAC=18:c0:4d:0d:9f:b2
2382023-11-08T06:53:39.411Z+ routeadm -e ipv4-forwarding -u
2392023-11-08T06:53:39.473Z+ PXA_START=10.151.1.164
2402023-11-08T06:53:39.505Z+ PXA_END=10.151.1.227
2412023-11-08T06:53:39.552Z+ export GATEWAY_IP GATEWAY_MAC PXA_START PXA_END
2422023-11-08T06:53:39.591Z+ pfexec zpool create -f scratch c1t1d0 c2t1d0
2432023-11-08T06:53:39.638Z+ ZPOOL_VDEV_DIR=/scratch
2442023-11-08T06:53:39.678Z+ ptime -m pfexec ./tools/create_virtual_hardware.sh
2452023-11-08T06:53:39.724Z+++ dirname ./tools/create_virtual_hardware.sh
2462023-11-08T06:53:39.754Z++ cd ./tools
2472023-11-08T06:53:39.789Z++ pwd
2482023-11-08T06:53:39.824Z+ SOURCE_DIR=/opt/oxide/work/tools
2492023-11-08T06:53:39.883Z+ OMICRON_TOP=/opt/oxide/work/tools/..
2502023-11-08T06:53:39.957Z+ SOFTNPU_MODE=zone
2512023-11-08T06:53:39.994Z+ . /opt/oxide/work/tools/virtual_hardware.sh
2522023-11-08T06:53:40.042Z++ MARKER=/etc/opt/oxide/NO_INSTALL
2532023-11-08T06:53:40.099Z++ [[ -f /etc/opt/oxide/NO_INSTALL ]]
2542023-11-08T06:53:40.149Z++ dladm show-phys -p -o LINK
2552023-11-08T06:53:40.273Z++ head -1
2562023-11-08T06:53:40.318Z+ PHYSICAL_LINK=igb0
2572023-11-08T06:53:40.372Z+ echo 'Using igb0 as physical link'
2582023-11-08T06:53:40.813ZUsing igb0 as physical link
2592023-11-08T06:53:40.852Z+ ensure_run_as_root
2602023-11-08T06:53:40.889Z++ id -u
2612023-11-08T06:53:40.938Z+ [[ 0 -ne 0 ]]
2622023-11-08T06:53:41.032Z+ ensure_zpools
2632023-11-08T06:53:41.501Z+ ZPOOL_TYPES=('oxp_' 'oxi_')
2642023-11-08T06:53:41.571Z+ for ZPOOL_TYPE in "${ZPOOL_TYPES[@]}"
2652023-11-08T06:53:41.599Z+ readarray -t ZPOOLS
2662023-11-08T06:53:41.632Z++ grep '"oxp_' /opt/oxide/work/tools/../smf/sled-agent/non-gimlet/config.toml
2672023-11-08T06:53:41.699Z++ sed 's/[ ",]//g'
2682023-11-08T06:53:41.734ZZpool: [oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b]
2692023-11-08T06:53:41.887Z+ for ZPOOL in "${ZPOOLS[@]}"
2702023-11-08T06:53:41.951ZZFS vdev /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
2712023-11-08T06:53:41.992Z+ echo 'Zpool: [oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b]'
2722023-11-08T06:53:42.055ZZFS zpool oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b exists
2732023-11-08T06:53:42.090Z+ VDEV_PATH=/scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
2742023-11-08T06:53:42.117ZZpool: [oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03]
2752023-11-08T06:53:42.144Z+ [[ -f /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
2762023-11-08T06:53:42.592ZZFS vdev /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
2772023-11-08T06:53:42.626Z+ dd if=/dev/zero of=/scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
2782023-11-08T06:53:42.658ZZFS zpool oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
2792023-11-08T06:53:42.687Z0+0 records in
2802023-11-08T06:53:42.721ZZpool: [oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03]
2812023-11-08T06:53:42.753Z0+0 records out
2822023-11-08T06:53:42.783ZZFS vdev /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
2832023-11-08T06:53:42.822Z0 bytes (0 iB) transferred in 0.000008 secs (0 iB/sec)
2842023-11-08T06:53:42.861ZZFS zpool oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
2852023-11-08T06:53:42.924Z+ success 'ZFS vdev /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
2862023-11-08T06:53:42.987ZZpool: [oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03]
2872023-11-08T06:53:43.038Z+ set +x
2882023-11-08T06:53:43.107ZZFS vdev /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
2892023-11-08T06:53:43.133Z++ zpool list -o name
2902023-11-08T06:53:43.165ZZFS zpool oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
2912023-11-08T06:53:43.217Z++ grep oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b
2922023-11-08T06:53:43.247ZZpool: [oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03]
2932023-11-08T06:53:43.290Z+ [[ -z '' ]]
2942023-11-08T06:53:43.323ZZFS vdev /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
2952023-11-08T06:53:43.360Z+ zpool create -f oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
2962023-11-08T06:53:43.389ZZFS zpool oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
2972023-11-08T06:53:43.419Z+ success 'ZFS zpool oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
2982023-11-08T06:53:43.452ZZpool: [oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930]
2992023-11-08T06:53:43.489Z+ set +x
3002023-11-08T06:53:43.522ZZFS vdev /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev exists
3012023-11-08T06:53:43.547Z+ for ZPOOL in "${ZPOOLS[@]}"
3022023-11-08T06:53:43.578ZZFS zpool oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 exists
3032023-11-08T06:53:43.611Z+ echo 'Zpool: [oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3042023-11-08T06:53:43.651ZZpool: [oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60]
3052023-11-08T06:53:43.687Z+ VDEV_PATH=/scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3062023-11-08T06:53:43.719ZZFS vdev /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev exists
3072023-11-08T06:53:43.769Z+ [[ -f /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3082023-11-08T06:53:43.820ZZFS zpool oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 exists
3092023-11-08T06:53:43.877Z+ dd if=/dev/zero of=/scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3102023-11-08T06:53:43.918ZZpool: [oxp_31bd71cd-4736-4a12-a387-9b74b050396f]
3112023-11-08T06:53:43.966Z0+0 records in
3122023-11-08T06:53:44.017ZZFS vdev /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev exists
3132023-11-08T06:53:44.045Z0+0 records out
3142023-11-08T06:53:44.088ZZFS zpool oxp_31bd71cd-4736-4a12-a387-9b74b050396f exists
3152023-11-08T06:53:44.207Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
3162023-11-08T06:53:44.250ZZpool: [oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7]
3172023-11-08T06:53:44.288Z+ success 'ZFS vdev /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3182023-11-08T06:53:44.315ZZFS vdev /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev exists
3192023-11-08T06:53:44.348Z+ set +x
3202023-11-08T06:53:44.399ZZFS zpool oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 exists
3212023-11-08T06:53:44.452Z++ zpool list -o name
3222023-11-08T06:53:44.507ZZpool: [oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b]
3232023-11-08T06:53:44.579Z++ grep oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03
3242023-11-08T06:53:44.627ZZFS vdev /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
3252023-11-08T06:53:44.656Z+ [[ -z '' ]]
3262023-11-08T06:53:44.685ZZFS zpool oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b exists
3272023-11-08T06:53:44.717Z+ zpool create -f oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3282023-11-08T06:53:44.832ZZpool: [oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b]
3292023-11-08T06:53:44.909Z+ success 'ZFS zpool oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3302023-11-08T06:53:44.947ZZFS vdev /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
3312023-11-08T06:53:44.979Z+ set +x
3322023-11-08T06:53:45.048ZZFS zpool oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b exists
3332023-11-08T06:53:45.094Z+ for ZPOOL in "${ZPOOLS[@]}"
3342023-11-08T06:53:45.128ZSimnet net0/sc0_0 exists
3352023-11-08T06:53:45.269Z+ echo 'Zpool: [oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3362023-11-08T06:53:45.334ZSimnet net1/sc1_0 exists
3372023-11-08T06:53:45.364Z+ VDEV_PATH=/scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3382023-11-08T06:53:45.416ZVnic sc0_1 exists
3392023-11-08T06:53:45.482Z+ [[ -f /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3402023-11-08T06:53:45.508Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/softnpu/image/c1c42398c82b0220c8b5fa3bfba9c7a3bcaa0943/softnpu.sha256.txt
3412023-11-08T06:53:45.543Z+ dd if=/dev/zero of=/scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3422023-11-08T06:53:45.572Z0+0 records in
3432023-11-08T06:53:46.051Z0+0 records out
3442023-11-08T06:53:46.108Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3452023-11-08T06:53:46.147Z+ success 'ZFS vdev /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3462023-11-08T06:53:46.196Z+ set +x
3472023-11-08T06:53:46.276Z++ zpool list -o name
3482023-11-08T06:53:46.370Z++ grep oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03
3492023-11-08T06:53:46.665Z+ [[ -z '' ]]
3502023-11-08T06:53:47.552Z+ zpool create -f oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3512023-11-08T06:53:55.997Z+ success 'ZFS zpool oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3522023-11-08T06:53:56.046Z+ set +x
3532023-11-08T06:53:56.212Z+ for ZPOOL in "${ZPOOLS[@]}"
3542023-11-08T06:53:56.243Z+ echo 'Zpool: [oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3552023-11-08T06:53:56.272Z+ VDEV_PATH=/scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3562023-11-08T06:53:56.307Z+ [[ -f /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3572023-11-08T06:53:56.348Z+ dd if=/dev/zero of=/scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3582023-11-08T06:53:56.394Z0+0 records in
3592023-11-08T06:53:56.422Z0+0 records out
3602023-11-08T06:53:56.455Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3612023-11-08T06:53:56.493Z+ success 'ZFS vdev /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3622023-11-08T06:53:56.539Z+ set +x
3632023-11-08T06:53:56.579Z++ zpool list -o name
3642023-11-08T06:53:56.622Z++ grep oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03
3652023-11-08T06:53:56.662Z+ [[ -z '' ]]
3662023-11-08T06:53:56.722Z+ zpool create -f oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3672023-11-08T06:53:56.769Z+ success 'ZFS zpool oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3682023-11-08T06:53:56.801Z+ set +x
3692023-11-08T06:53:56.842Z+ for ZPOOL in "${ZPOOLS[@]}"
3702023-11-08T06:53:56.878Z+ echo 'Zpool: [oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3712023-11-08T06:53:56.908Z+ VDEV_PATH=/scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3722023-11-08T06:53:56.949Z+ [[ -f /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3732023-11-08T06:53:57.019Z+ dd if=/dev/zero of=/scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3742023-11-08T06:53:57.062Z0+0 records in
3752023-11-08T06:53:57.093Z0+0 records out
3762023-11-08T06:53:57.126Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
3772023-11-08T06:53:57.568Z+ success 'ZFS vdev /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3782023-11-08T06:53:57.606Z+ set +x
3792023-11-08T06:53:57.639Z++ zpool list -o name
3802023-11-08T06:53:57.667Z++ grep oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03
3812023-11-08T06:53:58.131Z+ [[ -z '' ]]
3822023-11-08T06:53:58.180Z+ zpool create -f oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3832023-11-08T06:53:58.226Z+ success 'ZFS zpool oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3842023-11-08T06:53:58.266Z+ set +x
3852023-11-08T06:53:58.345Z+ for ZPOOL in "${ZPOOLS[@]}"
3862023-11-08T06:53:58.375Z+ echo 'Zpool: [oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930]'
3872023-11-08T06:53:58.416Z+ VDEV_PATH=/scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev
3882023-11-08T06:53:58.448Z+ [[ -f /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev ]]
3892023-11-08T06:53:58.485Z+ dd if=/dev/zero of=/scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev bs=1 count=0 seek=20G
3902023-11-08T06:53:58.513Z0+0 records in
3912023-11-08T06:53:58.541Z0+0 records out
3922023-11-08T06:53:58.586Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3932023-11-08T06:53:58.622Z+ success 'ZFS vdev /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev exists'
3942023-11-08T06:53:58.651Z+ set +x
3952023-11-08T06:53:58.676Z++ zpool list -o name
3962023-11-08T06:53:58.701Z++ grep oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930
3972023-11-08T06:53:58.732Z+ [[ -z '' ]]
3982023-11-08T06:53:58.761Z+ zpool create -f oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev
3992023-11-08T06:53:58.800Z+ success 'ZFS zpool oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 exists'
4002023-11-08T06:53:58.828Z+ set +x
4012023-11-08T06:53:58.870Z+ for ZPOOL in "${ZPOOLS[@]}"
4022023-11-08T06:53:58.909Z+ echo 'Zpool: [oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60]'
4032023-11-08T06:53:58.939Z+ VDEV_PATH=/scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev
4042023-11-08T06:53:58.979Z+ [[ -f /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev ]]
4052023-11-08T06:53:59.682Z+ dd if=/dev/zero of=/scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev bs=1 count=0 seek=20G
4062023-11-08T06:53:59.730Z0+0 records in
4072023-11-08T06:53:59.764Z0+0 records out
4082023-11-08T06:53:59.844Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4092023-11-08T06:53:59.894Z+ success 'ZFS vdev /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev exists'
4102023-11-08T06:53:59.925Z+ set +x
4112023-11-08T06:53:59.960Z++ zpool list -o name
4122023-11-08T06:54:00.003Z++ grep oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60
4132023-11-08T06:54:00.045Z+ [[ -z '' ]]
4142023-11-08T06:54:00.096Z+ zpool create -f oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev
4152023-11-08T06:54:00.136Z+ success 'ZFS zpool oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 exists'
4162023-11-08T06:54:00.610Zdownloading softnpu ...done
4172023-11-08T06:54:00.656Z+ set +x
4182023-11-08T06:54:00.686Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/sidecar-lite/release/f0585a29fb0285f7a1220c1118856b0e5c1f75c5/libsidecar_lite.so.sha256.txt
4192023-11-08T06:54:00.715Z+ for ZPOOL in "${ZPOOLS[@]}"
4202023-11-08T06:54:00.749Z+ echo 'Zpool: [oxp_31bd71cd-4736-4a12-a387-9b74b050396f]'
4212023-11-08T06:54:00.831Z+ VDEV_PATH=/scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev
4222023-11-08T06:54:00.856Z+ [[ -f /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev ]]
4232023-11-08T06:54:00.887Z+ dd if=/dev/zero of=/scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev bs=1 count=0 seek=20G
4242023-11-08T06:54:00.934Z0+0 records in
4252023-11-08T06:54:00.989Z0+0 records out
4262023-11-08T06:54:01.017Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4272023-11-08T06:54:01.042Z+ success 'ZFS vdev /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev exists'
4282023-11-08T06:54:01.068Z+ set +x
4292023-11-08T06:54:01.094Z++ zpool list -o name
4302023-11-08T06:54:01.126Z++ grep oxp_31bd71cd-4736-4a12-a387-9b74b050396f
4312023-11-08T06:54:01.163Z+ [[ -z '' ]]
4322023-11-08T06:54:01.191Z+ zpool create -f oxp_31bd71cd-4736-4a12-a387-9b74b050396f /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev
4332023-11-08T06:54:01.218Z+ success 'ZFS zpool oxp_31bd71cd-4736-4a12-a387-9b74b050396f exists'
4342023-11-08T06:54:01.243Z+ set +x
4352023-11-08T06:54:01.268Z+ for ZPOOL in "${ZPOOLS[@]}"
4362023-11-08T06:54:01.294Z+ echo 'Zpool: [oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7]'
4372023-11-08T06:54:01.323Z+ VDEV_PATH=/scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev
4382023-11-08T06:54:01.769Z+ [[ -f /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev ]]
4392023-11-08T06:54:01.806Z+ dd if=/dev/zero of=/scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev bs=1 count=0 seek=20G
4402023-11-08T06:54:01.840Z0+0 records in
4412023-11-08T06:54:01.865Z0+0 records out
4422023-11-08T06:54:01.891Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
4432023-11-08T06:54:01.916Z+ success 'ZFS vdev /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev exists'
4442023-11-08T06:54:01.942Z+ set +x
4452023-11-08T06:54:01.970Z++ zpool list -o name
4462023-11-08T06:54:02.004Z++ grep oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7
4472023-11-08T06:54:02.044Z+ [[ -z '' ]]
4482023-11-08T06:54:02.082Z+ zpool create -f oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev
4492023-11-08T06:54:02.603Z+ success 'ZFS zpool oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 exists'
4502023-11-08T06:54:02.642Z+ set +x
4512023-11-08T06:54:02.723Z+ for ZPOOL_TYPE in "${ZPOOL_TYPES[@]}"
4522023-11-08T06:54:02.748Z+ readarray -t ZPOOLS
4532023-11-08T06:54:02.773Z++ grep '"oxi_' /opt/oxide/work/tools/../smf/sled-agent/non-gimlet/config.toml
4542023-11-08T06:54:02.801Z++ sed 's/[ ",]//g'
4552023-11-08T06:54:02.826Z+ for ZPOOL in "${ZPOOLS[@]}"
4562023-11-08T06:54:02.852Z+ echo 'Zpool: [oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b]'
4572023-11-08T06:54:02.881Z+ VDEV_PATH=/scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4582023-11-08T06:54:02.908Z+ [[ -f /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
4592023-11-08T06:54:02.934Z+ dd if=/dev/zero of=/scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
4602023-11-08T06:54:02.963Z0+0 records in
4612023-11-08T06:54:03.398Z0+0 records out
4622023-11-08T06:54:03.430Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4632023-11-08T06:54:03.466Z+ success 'ZFS vdev /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
4642023-11-08T06:54:03.500Z+ set +x
4652023-11-08T06:54:03.548Z++ zpool list -o name
4662023-11-08T06:54:03.578Z++ grep oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b
4672023-11-08T06:54:03.616Z+ [[ -z '' ]]
4682023-11-08T06:54:03.654Z+ zpool create -f oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4692023-11-08T06:54:03.683Z+ success 'ZFS zpool oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
4702023-11-08T06:54:03.707Z+ set +x
4712023-11-08T06:54:03.763Z+ for ZPOOL in "${ZPOOLS[@]}"
4722023-11-08T06:54:03.789Z+ echo 'Zpool: [oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b]'
4732023-11-08T06:54:03.814Z+ VDEV_PATH=/scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4742023-11-08T06:54:03.846Z+ [[ -f /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
4752023-11-08T06:54:03.874Z+ dd if=/dev/zero of=/scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
4762023-11-08T06:54:03.903Z0+0 records in
4772023-11-08T06:54:03.936Z0+0 records out
4782023-11-08T06:54:03.969Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4792023-11-08T06:54:04.010Z+ success 'ZFS vdev /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
4802023-11-08T06:54:04.043Z+ set +x
4812023-11-08T06:54:04.074Z++ zpool list -o name
4822023-11-08T06:54:04.108Z++ grep oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b
4832023-11-08T06:54:04.135Z+ [[ -z '' ]]
4842023-11-08T06:54:04.161Z+ zpool create -f oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4852023-11-08T06:54:04.202Z+ success 'ZFS zpool oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
4862023-11-08T06:54:04.230Z+ set +x
4872023-11-08T06:54:04.258Z+ [[ zone == \z\o\n\e ]]
4882023-11-08T06:54:04.703Z+ ensure_simulated_links igb0
4892023-11-08T06:54:04.742Z+ local PHYSICAL_LINK=igb0
4902023-11-08T06:54:04.838Z+ INDICES=("0" "1")
4912023-11-08T06:54:04.872Z+ for I in "${INDICES[@]}"
4922023-11-08T06:54:04.908Z++ get_simnet_name_if_exists net0
4932023-11-08T06:54:04.945Z++ dladm show-simnet -p -o LINK net0
4942023-11-08T06:54:04.984Z++ echo ''
4952023-11-08T06:54:05.028Z+ [[ -z '' ]]
4962023-11-08T06:54:05.072Z+ dladm create-simnet -t net0
4972023-11-08T06:54:05.959Z+ dladm create-simnet -t sc0_0
4982023-11-08T06:54:06.692Z+ dladm modify-simnet -t -p net0 sc0_0
4992023-11-08T06:54:06.736Z+ dladm set-linkprop -p mtu=9000 sc0_0
5002023-11-08T06:54:06.784Z+ success 'Simnet net0/sc0_0 exists'
5012023-11-08T06:54:06.826Z+ set +x
5022023-11-08T06:54:06.863Z+ for I in "${INDICES[@]}"
5032023-11-08T06:54:06.939Z++ get_simnet_name_if_exists net1
5042023-11-08T06:54:06.968Z++ dladm show-simnet -p -o LINK net1
5052023-11-08T06:54:07.003Z++ echo ''
5062023-11-08T06:54:07.032Z+ [[ -z '' ]]
5072023-11-08T06:54:07.058Z+ dladm create-simnet -t net1
5082023-11-08T06:54:07.088Z+ dladm create-simnet -t sc1_0
5092023-11-08T06:54:07.127Z+ dladm modify-simnet -t -p net1 sc1_0
5102023-11-08T06:54:07.157Z+ dladm set-linkprop -p mtu=9000 sc1_0
5112023-11-08T06:54:07.603Z+ success 'Simnet net1/sc1_0 exists'
5122023-11-08T06:54:07.650Z+ set +x
5132023-11-08T06:54:07.696Z++ get_vnic_name_if_exists sc0_1
5142023-11-08T06:54:07.738Z++ dladm show-vnic -p -o LINK sc0_1
5152023-11-08T06:54:07.780Z++ echo ''
5162023-11-08T06:54:07.827Z+ [[ -z '' ]]
5172023-11-08T06:54:07.884Z+ dladm create-vnic -t sc0_1 -l igb0 -m a8:e1:de:01:70:1d
5182023-11-08T06:54:07.916Z+ [[ -v PROMISC_FILT_OFF ]]
5192023-11-08T06:54:07.945Z+ success 'Vnic sc0_1 exists'
5202023-11-08T06:54:08.024Z+ set +x
5212023-11-08T06:54:08.056Z+ warn_if_no_proxy_arp
5222023-11-08T06:54:08.084Z+ [[ -v PXA_START ]]
5232023-11-08T06:54:08.109Z+ [[ -v PXA_END ]]
5242023-11-08T06:54:08.133Z+ ensure_softnpu_zone
5252023-11-08T06:54:08.567Z+ zoneadm list
5262023-11-08T06:54:08.594Z+ grep -q sidecar_softnpu
5272023-11-08T06:54:08.622Z+ [[ -f out/npuzone/npuzone ]]
5282023-11-08T06:54:08.646Z+ out/npuzone/npuzone create sidecar --omicron-zone --ports sc0_0,tfportrear0_0 --ports sc0_1,tfportqsfp0_0 --sidecar-lite-branch main
5292023-11-08T06:54:08.671Zdownloading asic_program.so ...done
5302023-11-08T06:54:08.697Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/sidecar-lite/release/f0585a29fb0285f7a1220c1118856b0e5c1f75c5/scadm.sha256.txt
5312023-11-08T06:54:12.759Zdownloading scadm ...done
5322023-11-08T06:54:12.784Zcreating port tfportrear0_0 for sc0_0
5332023-11-08T06:54:12.809Zcreating port tfportqsfp0_0 for sc0_1
5342023-11-08T06:54:12.851Zsoftnpu zone setup
5352023-11-08T06:54:12.885Zconfigure zone
5362023-11-08T06:54:12.914Zinstall zone
5372023-11-08T06:54:13.842Zboot zone
5382023-11-08T06:54:16.954Z[sidecar_softnpu] svcs milestone/network
5392023-11-08T06:54:16.989ZSTATE STIME FMRI
5402023-11-08T06:54:17.016Zonline 6:54:15 svc:/milestone/network:default
5412023-11-08T06:54:17.040Z[sidecar_softnpu] /softnpu/npu start
5422023-11-08T06:54:17.065Z
5432023-11-08T06:54:17.090Z+ /opt/oxide/work/tools/scrimlet/softnpu-init.sh
5442023-11-08T06:54:17.114Z+ GATEWAY_IP=10.151.1.100
5452023-11-08T06:54:17.139Z+ echo 'Using 10.151.1.100 as gateway ip'
5462023-11-08T06:54:17.583ZUsing 10.151.1.100 as gateway ip
5472023-11-08T06:54:17.615Z+ [[ ! -v GATEWAY_MAC ]]
5482023-11-08T06:54:17.647Z+ GATEWAY_MAC=18:c0:4d:0d:9f:b2
5492023-11-08T06:54:17.673Z++ grep -c -E '^([0-9a-fA-F]{2}:){5}[0-9a-fA-F]{2}$' /dev/fd/63
5502023-11-08T06:54:17.704Z+++ echo 18:c0:4d:0d:9f:b2
5512023-11-08T06:54:17.737Z+ COUNT=1
5522023-11-08T06:54:18.203Z+ [[ 1 -ne 1 ]]
5532023-11-08T06:54:18.241Z+ echo 'Using 18:c0:4d:0d:9f:b2 as gateway mac'
5542023-11-08T06:54:18.272ZUsing 18:c0:4d:0d:9f:b2 as gateway mac
5552023-11-08T06:54:18.312Z+ z_scadm add-arp-entry 10.151.1.100 18:c0:4d:0d:9f:b2
5562023-11-08T06:54:18.344Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone add-arp-entry 10.151.1.100 18:c0:4d:0d:9f:b2
5572023-11-08T06:54:18.381Z+ PXA_MAC=a8:e1:de:01:70:1d
5582023-11-08T06:54:18.413Z+ [[ -v PXA_START ]]
5592023-11-08T06:54:18.456Z+ z_scadm add-proxy-arp 10.151.1.164 10.151.1.227 a8:e1:de:01:70:1d
5602023-11-08T06:54:18.487Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone add-proxy-arp 10.151.1.164 10.151.1.227 a8:e1:de:01:70:1d
5612023-11-08T06:54:18.525Z+ z_scadm dump-state
5622023-11-08T06:54:18.560Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone dump-state
5632023-11-08T06:54:18.676Zlocal v6:
5642023-11-08T06:54:18.702Zlocal v4:
5652023-11-08T06:54:18.729Zrouter v6:
5662023-11-08T06:54:18.761Zrouter v4_idx:
5672023-11-08T06:54:18.798Zrouter v4_routes:
5682023-11-08T06:54:18.822Zresolver v4:
5692023-11-08T06:54:18.850Z10.151.1.100 -> 18:c0:4d:0d:9f:b2
5702023-11-08T06:54:18.879Zresolver v6:
5712023-11-08T06:54:18.907Znat v4:
5722023-11-08T06:54:18.939Znat_v6:
5732023-11-08T06:54:18.968Zport_mac:
5742023-11-08T06:54:19.007Zproxy arp:
5752023-11-08T06:54:19.050Z10.151.1.164/10.151.1.227: a8:e1:de:01:70:1d
5762023-11-08T06:54:19.079Z+ success 'softnpu zone exists'
5772023-11-08T06:54:19.114Z+ set +x
5782023-11-08T06:54:19.150Zsoftnpu zone exists
5792023-11-08T06:54:19.181Z
5802023-11-08T06:54:19.219Zreal 40.294739328
5812023-11-08T06:54:19.253Zuser 0.701279985
5822023-11-08T06:54:19.283Zsys 1.601142295
5832023-11-08T06:54:19.315Ztrap 0.000094948
5842023-11-08T06:54:19.372Ztflt 0.000111809
5852023-11-08T06:54:19.408Zdflt 0.000808521
5862023-11-08T06:54:19.446Zkflt 0.000000000
5872023-11-08T06:54:19.490Zlock 31:44.458419149
5882023-11-08T06:54:19.522Zslp 2:00.028930022
5892023-11-08T06:54:19.570Zlat 0.069870863
5902023-11-08T06:54:19.596Zstop 0.347426682
5912023-11-08T06:54:19.622Z+ tar xf out/omicron-sled-agent.tar pkg/config-rss.toml
5922023-11-08T06:54:19.700Z++ sed -n 's/silo_name = "\(.*\)"/\1/p' pkg/config-rss.toml
5932023-11-08T06:54:19.735Z+ SILO_NAME=recovery
5942023-11-08T06:54:19.769Z++ sed -n 's/external_dns_zone_name = "\(.*\)"/\1/p' pkg/config-rss.toml
5952023-11-08T06:54:19.794Z+ EXTERNAL_DNS_DOMAIN=oxide.test
5962023-11-08T06:54:19.821Z+ sed -i~ '
5972023-11-08T06:54:19.855Z /^external_dns_ips/c\
5982023-11-08T06:54:19.883Zexternal_dns_ips = [ "10.151.1.164", "10.151.1.165" ]
5992023-11-08T06:54:19.921Z /^\[\[internal_services_ip_pool_ranges/,/^$/ {
6002023-11-08T06:54:19.957Z /^first/c\
6012023-11-08T06:54:19.987Zfirst = "10.151.1.164"
6022023-11-08T06:54:20.017Z /^last/c\
6032023-11-08T06:54:20.046Zlast = "10.151.1.173"
6042023-11-08T06:54:20.079Z }
6052023-11-08T06:54:20.112Z /^\[rack_network_config/,/^$/ {
6062023-11-08T06:54:20.143Z /^infra_ip_first/c\
6072023-11-08T06:54:20.167Zinfra_ip_first = "10.151.1.174"
6082023-11-08T06:54:20.210Z /^infra_ip_last/c\
6092023-11-08T06:54:20.303Zinfra_ip_last = "10.151.1.174"
6102023-11-08T06:54:20.335Z }
6112023-11-08T06:54:20.367Z--- pkg/config-rss.toml~ Mon Jul 24 01:21:28 2006
6122023-11-08T06:54:20.397Z /^\[\[rack_network_config.ports/,/^$/ {
6132023-11-08T06:54:20.426Z+++ pkg/config-rss.toml Wed Nov 8 06:54:17 2023
6142023-11-08T06:54:20.452Z /^routes/c\
6152023-11-08T06:54:20.477Z@@ -41,7 +41,7 @@
6162023-11-08T06:54:20.922Zroutes = \[{nexthop = "10.151.1.100", destination = "0.0.0.0/0"}\]
6172023-11-08T06:54:20.953Z # the DNS domain delegated to the rack by the customer. Each of these addresses
6182023-11-08T06:54:20.978Z /^addresses/c\
6192023-11-08T06:54:21.003Z # must be contained in one of the "internal services" IP Pool ranges listed
6202023-11-08T06:54:21.446Zaddresses = \["10.151.1.174/32"\]
6212023-11-08T06:54:21.473Z # below.
6222023-11-08T06:54:21.500Z }
6232023-11-08T06:54:21.529Z-external_dns_ips = [ "192.168.1.20", "192.168.1.21" ]
6242023-11-08T06:54:21.557Z' pkg/config-rss.toml
6252023-11-08T06:54:21.597Z+external_dns_ips = [ "10.151.1.164", "10.151.1.165" ]
6262023-11-08T06:54:22.037Z+ diff -u pkg/config-rss.toml~ pkg/config-rss.toml
6272023-11-08T06:54:22.062Z
6282023-11-08T06:54:22.087Z+ true
6292023-11-08T06:54:22.112Z # Initial TLS certificates for the external API
6302023-11-08T06:54:22.138Z+ tar rvf out/omicron-sled-agent.tar pkg/config-rss.toml
6312023-11-08T06:54:22.183Z #
6322023-11-08T06:54:23.041Z+ rm -f pkg/config-rss.toml pkg/config-rss.toml~
6332023-11-08T06:54:23.082Z@@ -77,8 +77,8 @@
6342023-11-08T06:54:23.112Z+ TLS_NAME=recovery.sys.oxide.test
6352023-11-08T06:54:23.143Z #
6362023-11-08T06:54:23.185Z+ openssl req -newkey rsa:4096 -x509 -sha256 -days 3 -nodes -out pkg/initial-tls-cert.pem -keyout pkg/initial-tls-key.pem -subj /CN=recovery.sys.oxide.test -addext subjectAltName=DNS:recovery.sys.oxide.test -addext basicConstraints=critical,CA:FALSE -config /dev/stdin
6372023-11-08T06:54:23.221Z # For more on this and what to put here, see docs/how-to-run.adoc.
6382023-11-08T06:54:23.804Z.....+............+.....+....+......+...+...........+.+........+.......+.....+...+.+...+......+..+.........+.+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*....+...+......+.....+.+...+...+..+...+...+.+.........+......+......+..+.+..+.........+..........+...+..................+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.......+....+......+..............+....+..+.........+.+...+..+.+.....+.......+..+.............+.....+......+.+....................+....+...........+.+..+.+.....+......+.+............+...........+..........+..+....+.....+....+...............+..+.......+..+.+...+............+........+....+............+..............+...................+.....+...+............+.+.........+.....+.+.....+.........+...+............................+...........+...+.......+........+.......+......+.........+..............+...............+....+...........+....+.................+...+................+..............+......+.+...+.....+.............+.....+...+.........+...+...+.......+........+....+..+......+....+...+........+...+.+...+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
6392023-11-08T06:54:23.872Z [[internal_services_ip_pool_ranges]]
6402023-11-08T06:54:23.902Z.+...+...+.................+.+...........+.+.....+....+......+.....+.........+.+..............+....+...............+..+..........+..+......+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*...+.......+..+......+.......+..+.........+............+...+.......+.................+.+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.+.....+....+...+........+....+..+...................+.........+..+...+...+.............+.........+.....+.+..............+.+.........+.................+.+.........+......+..+.+...........+..........+.....................+.....+....+........+.+.................+.............+....................+....+...+..+......+.+......+...+.....+.........+...+....+..................+......+.....+.+...+...+...............+............+.....+..........+......+..+........................+...+.+.....+....+......+...........+...+...............+.+............+..+......+.....................+.+.........+.....+.........+....+...+...+......+...+...+...+.....+...............+...............+....+...+...+.........+........+.................................................+......+..+.+.................+....+...+........+.......+.....+.+..+....+.........+..+....+.........+.....+......+...+......+............+....+......+..............+......+...+....+.....+.+...........+.........+....+........+...+.......+...+.....+.+.........+.....+.+.....+.+............+..+......+..........+.........+...+..+..........+...........+....+...+...+........+...............+.......+..............+.........+....+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
6412023-11-08T06:54:23.927Z-first = "192.168.1.20"
6422023-11-08T06:54:23.951Z-----
6432023-11-08T06:54:23.975Z-last = "192.168.1.29"
6442023-11-08T06:54:24.000Z+ tar rvf out/omicron-sled-agent.tar pkg/initial-tls-cert.pem pkg/initial-tls-key.pem
6452023-11-08T06:54:24.025Z+first = "10.151.1.164"
6462023-11-08T06:54:24.050Z+ rm -f pkg/initial-tls-cert.pem pkg/initial-tls-key.pem
6472023-11-08T06:54:24.075Z+last = "10.151.1.173"
6482023-11-08T06:54:24.113Z+ rmdir pkg
6492023-11-08T06:54:24.577Z
6502023-11-08T06:54:24.613Z+ E2E_TLS_CERT=/opt/oxide/sled-agent/pkg/initial-tls-cert.pem
6512023-11-08T06:54:24.642Z # TODO - this configuration is subject to change going forward. Ultimately these
6522023-11-08T06:54:25.070Z+ pfexec mkdir -p /usr/oxide
6532023-11-08T06:54:25.099Z # parameters should be provided to the control plane via wicket, but we need to
6542023-11-08T06:54:25.157Z+ pfexec rm -f /usr/oxide/tcpproxy
6552023-11-08T06:54:25.197Z@@ -92,8 +92,8 @@
6562023-11-08T06:54:25.226Z+ pfexec curl -sSfL -o /usr/oxide/tcpproxy http://catacomb.eng.oxide.computer:12346/tcpproxy
6572023-11-08T06:54:25.253Z # A range of IP addresses used by Boundary Services on the external network. In
6582023-11-08T06:54:25.786Z+ pfexec chmod +x /usr/oxide/tcpproxy
6592023-11-08T06:54:25.818Z # a real system, these would be addresses of the uplink ports on the Sidecar.
6602023-11-08T06:54:25.858Z+ pfexec rm -f /var/svc/manifest/site/tcpproxy.xml
6612023-11-08T06:54:25.885Z # With softnpu, only one address is used.
6622023-11-08T06:54:25.951Z+ pfexec curl -sSfL -o /var/svc/manifest/site/tcpproxy.xml http://catacomb.eng.oxide.computer:12346/tcpproxy.xml
6632023-11-08T06:54:25.982Z-infra_ip_first = "192.168.1.30"
6642023-11-08T06:54:26.009Z+ pfexec svccfg import /var/svc/manifest/site/tcpproxy.xml
6652023-11-08T06:54:26.042Z-infra_ip_last = "192.168.1.30"
6662023-11-08T06:54:26.070Z+ OMICRON_NO_UNINSTALL=1
6672023-11-08T06:54:26.104Z+infra_ip_first = "10.151.1.174"
6682023-11-08T06:54:26.134Z+ ptime -m pfexec ./target/release/omicron-package -t test install
6692023-11-08T06:54:26.168Z+infra_ip_last = "10.151.1.174"
6702023-11-08T06:54:26.198ZNov 08 06:54:19.272 DEBG target[test]: Target({"image": "standard", "machine": "non-gimlet", "rack-topology": "single-sled", "switch": "softnpu"})
6712023-11-08T06:54:27.525Z
6722023-11-08T06:54:27.552ZNov 08 06:54:19.274 INFO Installing service, dst: /opt/oxide/clickhouse.tar.gz, src: out/clickhouse.tar.gz
6732023-11-08T06:54:27.583Z # Configurations for BGP routers to run on the scrimlets.
6742023-11-08T06:54:27.608ZNov 08 06:54:19.274 INFO Installing service, dst: /opt/oxide/ntp.tar.gz, src: out/ntp.tar.gz
6752023-11-08T06:54:27.639Z bgp = []
6762023-11-08T06:54:28.087ZNov 08 06:54:19.274 INFO Installing service, dst: /opt/oxide/cockroachdb.tar.gz, src: out/cockroachdb.tar.gz
6772023-11-08T06:54:28.121Z@@ -101,9 +101,9 @@
6782023-11-08T06:54:28.146ZNov 08 06:54:19.274 INFO Installing service, dst: /opt/oxide/mgs.tar.gz, src: out/omicron-gateway-softnpu.tar.gz
6792023-11-08T06:54:28.173Z # You can configure multiple uplinks by repeating the following stanza
6802023-11-08T06:54:28.201ZNov 08 06:54:19.274 INFO Installing service, dst: /opt/oxide/clickhouse_keeper.tar.gz, src: out/clickhouse_keeper.tar.gz
6812023-11-08T06:54:28.227Z [[rack_network_config.ports]]
6822023-11-08T06:54:28.259ZNov 08 06:54:19.274 INFO Installing service, dst: /opt/oxide/external_dns.tar.gz, src: out/external-dns.tar.gz
6832023-11-08T06:54:28.289Z # Routes associated with this port.
6842023-11-08T06:54:28.330ZNov 08 06:54:19.274 INFO Installing service, dst: /opt/oxide/internal_dns.tar.gz, src: out/internal-dns.tar.gz
6852023-11-08T06:54:28.362Z-routes = [{nexthop = "192.168.1.199", destination = "0.0.0.0/0"}]
6862023-11-08T06:54:28.389ZNov 08 06:54:19.274 INFO Installing service, dst: /opt/oxide/overlay.tar.gz, src: out/overlay.tar.gz
6872023-11-08T06:54:28.416Z+routes = [{nexthop = "10.151.1.100", destination = "0.0.0.0/0"}]
6882023-11-08T06:54:28.442ZNov 08 06:54:19.274 INFO Installing service, dst: /opt/oxide/mg-ddm.tar, src: out/mg-ddm-gz.tar
6892023-11-08T06:54:28.478Z # Addresses associated with this port.
6902023-11-08T06:54:28.508ZNov 08 06:54:19.274 INFO Installing service, dst: /opt/oxide/propolis-server.tar.gz, src: out/propolis-server.tar.gz
6912023-11-08T06:54:28.535Z-addresses = ["192.168.1.30/32"]
6922023-11-08T06:54:28.560ZNov 08 06:54:19.274 INFO Installing service, dst: /opt/oxide/nexus.tar.gz, src: out/omicron-nexus.tar.gz
6932023-11-08T06:54:28.591Z+addresses = ["10.151.1.174/32"]
6942023-11-08T06:54:28.616ZNov 08 06:54:19.274 INFO Installing service, dst: /opt/oxide/oximeter.tar.gz, src: out/oximeter-collector.tar.gz
6952023-11-08T06:54:28.650Z # Name of the uplink port. This should always be "qsfp0" when using softnpu.
6962023-11-08T06:54:28.678ZNov 08 06:54:19.274 INFO Installing service, dst: /opt/oxide/switch.tar.gz, src: out/switch-softnpu.tar.gz
6972023-11-08T06:54:28.703Z port = "qsfp0"
6982023-11-08T06:54:28.727ZNov 08 06:54:19.274 INFO Installing service, dst: /opt/oxide/crucible.tar.gz, src: out/crucible.tar.gz
6992023-11-08T06:54:28.755Z # The speed of this port.
7002023-11-08T06:54:28.783ZNov 08 06:54:19.274 INFO Installing service, dst: /opt/oxide/sled-agent.tar, src: out/omicron-sled-agent.tar
7012023-11-08T06:54:28.811Za pkg/config-rss.toml 7K
7022023-11-08T06:54:28.836ZNov 08 06:54:19.274 INFO Installing service, dst: /opt/oxide/crucible_pantry.tar.gz, src: out/crucible-pantry.tar.gz
7032023-11-08T06:54:28.860Za pkg/initial-tls-cert.pem 2K
7042023-11-08T06:54:28.891ZNov 08 06:54:19.509 INFO Unpacking service tarball, service_path: /opt/oxide/mg-ddm, tar_path: /opt/oxide/mg-ddm.tar
7052023-11-08T06:54:28.926Za pkg/initial-tls-key.pem 4K
7062023-11-08T06:54:28.962ZNov 08 06:54:19.542 INFO Unpacking service tarball, service_path: /opt/oxide/sled-agent, tar_path: /opt/oxide/sled-agent.tar
7072023-11-08T06:54:28.993ZNov 08 06:54:19.603 INFO Installing bootstrap service from /opt/oxide/sled-agent/pkg/manifest.xml
7082023-11-08T06:54:29.431Z
7092023-11-08T06:54:29.473Zreal 0.469362708
7102023-11-08T06:54:29.930Zuser 0.100021732
7112023-11-08T06:54:29.959Zsys 1.927439936
7122023-11-08T06:54:29.984Ztrap 0.000000240
7132023-11-08T06:54:30.010Ztflt 0.000039928
7142023-11-08T06:54:30.035Zdflt 0.001042371
7152023-11-08T06:54:30.059Zkflt 0.000000000
7162023-11-08T06:54:30.092Zlock 42.083813002
7172023-11-08T06:54:30.118Zslp 0.680511503
7182023-11-08T06:54:30.145Zlat 0.003712205
7192023-11-08T06:54:30.181Zstop 0.199181317
7202023-11-08T06:54:30.374Z+ retry=0
7212023-11-08T06:54:30.424Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7222023-11-08T06:54:30.463Z+ [[ 0 -gt 30 ]]
7232023-11-08T06:54:30.555Z+ sleep 1
7242023-11-08T06:54:30.589Z+ retry=1
7252023-11-08T06:54:30.620Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7262023-11-08T06:54:30.646Z+ [[ 1 -gt 30 ]]
7272023-11-08T06:54:30.677Z+ sleep 1
7282023-11-08T06:54:30.707Z+ retry=2
7292023-11-08T06:54:30.735Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7302023-11-08T06:54:46.615Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone dump-state
7312023-11-08T06:54:46.641Zlocal v6:
7322023-11-08T06:54:46.667Zfe80::707f:9eff:feb7:88e1
7332023-11-08T06:54:46.703Zfd00:99::1
7342023-11-08T06:54:46.741Zlocal v4:
7352023-11-08T06:54:46.800Z10.151.1.174
7362023-11-08T06:54:46.828Zrouter v6:
7372023-11-08T06:54:46.862Zfd00:1122:3344:101::/64 -> fe80::2c9f:25ff:fea2:f8b1 (1)
7382023-11-08T06:54:46.892Zfdb0:18c0:4d0d:9fb2::/64 -> fe80::2c9f:25ff:fea2:f8b1 (1)
7392023-11-08T06:54:46.918Zrouter v4_idx:
7402023-11-08T06:54:46.949Z0.0.0.0/0 -> 0
7412023-11-08T06:54:46.982Zrouter v4_routes:
7422023-11-08T06:54:47.022Z0 -> 10.151.1.100 (2)
7432023-11-08T06:54:47.054Zresolver v4:
7442023-11-08T06:54:47.084Z10.151.1.174 -> 1e:11:43:95:f7:8e
7452023-11-08T06:54:47.115Z10.151.1.100 -> 18:c0:4d:0d:9f:b2
7462023-11-08T06:54:47.174Zresolver v6:
7472023-11-08T06:54:47.275Zfe80::2c9f:25ff:fea2:f8b1 -> 2e:9f:25:a2:f8:b1
7482023-11-08T06:54:47.302Zfe80::707f:9eff:feb7:88e1 -> 72:7f:9e:b7:88:e1
7492023-11-08T06:54:47.332Znat v4:
7502023-11-08T06:54:47.362Znat_v6:
7512023-11-08T06:54:47.387Zport_mac:
7522023-11-08T06:54:47.416Z1000: a8:40:25:1c:3c:bc
7532023-11-08T06:54:47.449Z1: a8:40:25:1c:3c:bd
7542023-11-08T06:54:47.478Zproxy arp:
7552023-11-08T06:54:47.511Z10.151.1.164/10.151.1.227: a8:e1:de:01:70:1d
7562023-11-08T06:54:47.546Z+ retry=0
7572023-11-08T06:54:47.570Z+++ zoneadm list -n
7582023-11-08T06:54:47.599Z+++ grep oxz_ntp
7592023-11-08T06:54:47.626Z++ pfexec svcs -z -Hostate oxide/ntp
7602023-11-08T06:54:47.657Zsvcs: zone '-Hostate': entity not found
7612023-11-08T06:54:48.107Z++ true
7622023-11-08T06:54:48.143Z+ [[ '' != online ]]
7632023-11-08T06:54:48.241Z+ [[ 0 -gt 60 ]]
7642023-11-08T06:54:48.270Z+ sleep 1
7652023-11-08T06:54:48.299Z+ retry=1
7662023-11-08T06:54:48.325Z+++ zoneadm list -n
7672023-11-08T06:54:48.349Z+++ grep oxz_ntp
7682023-11-08T06:54:48.374Z++ pfexec svcs -z -Hostate oxide/ntp
7692023-11-08T06:54:48.399Zsvcs: zone '-Hostate': entity not found
7702023-11-08T06:54:48.424Z++ true
7712023-11-08T06:54:48.449Z+ [[ '' != online ]]
7722023-11-08T06:54:48.475Z+ [[ 1 -gt 60 ]]
7732023-11-08T06:54:48.500Z+ sleep 1
7742023-11-08T06:54:48.698Z+ retry=2
7752023-11-08T06:54:48.730Z+++ zoneadm list -n
7762023-11-08T06:54:48.755Z+++ grep oxz_ntp
7772023-11-08T06:54:48.783Z++ pfexec svcs -z -Hostate oxide/ntp
7782023-11-08T06:54:48.808Zsvcs: zone '-Hostate': entity not found
7792023-11-08T06:54:48.834Z++ true
7802023-11-08T06:54:48.863Z+ [[ '' != online ]]
7812023-11-08T06:54:48.888Z+ [[ 2 -gt 60 ]]
7822023-11-08T06:54:48.912Z+ sleep 1
7832023-11-08T06:54:49.708Z+ retry=3
7842023-11-08T06:54:49.739Z+++ zoneadm list -n
7852023-11-08T06:54:49.770Z+++ grep oxz_ntp
7862023-11-08T06:54:49.803Z++ pfexec svcs -z -Hostate oxide/ntp
7872023-11-08T06:54:49.828Zsvcs: zone '-Hostate': entity not found
7882023-11-08T06:54:49.855Z++ true
7892023-11-08T06:54:49.891Z+ [[ '' != online ]]
7902023-11-08T06:54:50.461Z+ [[ 3 -gt 60 ]]
7912023-11-08T06:54:50.661Z+ sleep 1
7922023-11-08T06:54:50.730Z+ retry=4
7932023-11-08T06:54:50.758Z+++ zoneadm list -n
7942023-11-08T06:54:50.784Z+++ grep oxz_ntp
7952023-11-08T06:54:50.809Z++ pfexec svcs -z -Hostate oxide/ntp
7962023-11-08T06:54:50.836Zsvcs: zone '-Hostate': entity not found
7972023-11-08T06:54:50.868Z++ true
7982023-11-08T06:54:51.303Z+ [[ '' != online ]]
7992023-11-08T06:54:51.331Z+ [[ 4 -gt 60 ]]
8002023-11-08T06:54:51.357Z+ sleep 1
8012023-11-08T06:54:52.163Z+ retry=5
8022023-11-08T06:54:52.200Z+++ zoneadm list -n
8032023-11-08T06:54:52.241Z+++ grep oxz_ntp
8042023-11-08T06:54:52.282Z++ pfexec svcs -z -Hostate oxide/ntp
8052023-11-08T06:54:52.318Zsvcs: zone '-Hostate': entity not found
8062023-11-08T06:54:52.352Z++ true
8072023-11-08T06:54:52.385Z+ [[ '' != online ]]
8082023-11-08T06:54:52.427Z+ [[ 5 -gt 60 ]]
8092023-11-08T06:54:52.458Z+ sleep 1
8102023-11-08T06:54:52.789Z+ retry=6
8112023-11-08T06:54:52.823Z+++ zoneadm list -n
8122023-11-08T06:54:52.863Z+++ grep oxz_ntp
8132023-11-08T06:54:52.892Z++ pfexec svcs -z -Hostate oxide/ntp
8142023-11-08T06:54:52.922Zsvcs: zone '-Hostate': entity not found
8152023-11-08T06:54:52.953Z++ true
8162023-11-08T06:54:52.995Z+ [[ '' != online ]]
8172023-11-08T06:54:53.024Z+ [[ 6 -gt 60 ]]
8182023-11-08T06:54:53.057Z+ sleep 1
8192023-11-08T06:54:53.794Z+ retry=7
8202023-11-08T06:54:53.818Z+++ zoneadm list -n
8212023-11-08T06:54:53.843Z+++ grep oxz_ntp
8222023-11-08T06:54:53.869Z++ pfexec svcs -z -Hostate oxide/ntp
8232023-11-08T06:54:53.894Zsvcs: zone '-Hostate': entity not found
8242023-11-08T06:54:53.919Z++ true
8252023-11-08T06:54:53.946Z+ [[ '' != online ]]
8262023-11-08T06:54:53.975Z+ [[ 7 -gt 60 ]]
8272023-11-08T06:54:54.016Z+ sleep 1
8282023-11-08T06:54:54.814Z+ retry=8
8292023-11-08T06:54:54.839Z+++ zoneadm list -n
8302023-11-08T06:54:54.865Z+++ grep oxz_ntp
8312023-11-08T06:54:54.893Z++ pfexec svcs -z -Hostate oxide/ntp
8322023-11-08T06:54:54.918Zsvcs: zone '-Hostate': entity not found
8332023-11-08T06:54:55.402Z++ true
8342023-11-08T06:54:55.432Z+ [[ '' != online ]]
8352023-11-08T06:54:55.459Z+ [[ 8 -gt 60 ]]
8362023-11-08T06:54:55.484Z+ sleep 1
8372023-11-08T06:54:55.849Z+ retry=9
8382023-11-08T06:54:55.898Z+++ zoneadm list -n
8392023-11-08T06:54:55.930Z+++ grep oxz_ntp
8402023-11-08T06:54:55.970Z++ pfexec svcs -z -Hostate oxide/ntp
8412023-11-08T06:54:56.006Zsvcs: zone '-Hostate': entity not found
8422023-11-08T06:54:56.038Z++ true
8432023-11-08T06:54:56.065Z+ [[ '' != online ]]
8442023-11-08T06:54:56.506Z+ [[ 9 -gt 60 ]]
8452023-11-08T06:54:56.944Z+ sleep 1
8462023-11-08T06:54:56.974Z+ retry=10
8472023-11-08T06:54:57.005Z+++ zoneadm list -n
8482023-11-08T06:54:57.030Z+++ grep oxz_ntp
8492023-11-08T06:54:57.060Z++ pfexec svcs -z -Hostate oxide/ntp
8502023-11-08T06:54:57.090Zsvcs: zone '-Hostate': entity not found
8512023-11-08T06:54:57.115Z++ true
8522023-11-08T06:54:57.142Z+ [[ '' != online ]]
8532023-11-08T06:54:57.168Z+ [[ 10 -gt 60 ]]
8542023-11-08T06:54:57.195Z+ sleep 1
8552023-11-08T06:54:57.923Z+ retry=11
8562023-11-08T06:54:57.955Z+++ zoneadm list -n
8572023-11-08T06:54:57.997Z+++ grep oxz_ntp
8582023-11-08T06:54:58.026Z++ pfexec svcs -z -Hostate oxide/ntp
8592023-11-08T06:54:58.057Zsvcs: zone '-Hostate': entity not found
8602023-11-08T06:54:58.086Z++ true
8612023-11-08T06:54:58.115Z+ [[ '' != online ]]
8622023-11-08T06:54:58.143Z+ [[ 11 -gt 60 ]]
8632023-11-08T06:54:58.180Z+ sleep 1
8642023-11-08T06:54:58.917Z+ retry=12
8652023-11-08T06:54:58.963Z+++ zoneadm list -n
8662023-11-08T06:54:58.997Z+++ grep oxz_ntp
8672023-11-08T06:54:59.030Z++ pfexec svcs -z -Hostate oxide/ntp
8682023-11-08T06:54:59.065Zsvcs: zone '-Hostate': entity not found
8692023-11-08T06:54:59.099Z++ true
8702023-11-08T06:54:59.126Z+ [[ '' != online ]]
8712023-11-08T06:54:59.160Z+ [[ 12 -gt 60 ]]
8722023-11-08T06:54:59.190Z+ sleep 1
8732023-11-08T06:54:59.950Z+ retry=13
8742023-11-08T06:55:00.043Z+++ zoneadm list -n
8752023-11-08T06:55:00.072Z+++ grep oxz_ntp
8762023-11-08T06:55:00.105Z++ pfexec svcs -z -Hostate oxide/ntp
8772023-11-08T06:55:00.134Zsvcs: zone '-Hostate': entity not found
8782023-11-08T06:55:00.169Z++ true
8792023-11-08T06:55:00.228Z+ [[ '' != online ]]
8802023-11-08T06:55:00.686Z+ [[ 13 -gt 60 ]]
8812023-11-08T06:55:00.711Z+ sleep 1
8822023-11-08T06:55:00.958Z+ retry=14
8832023-11-08T06:55:00.989Z+++ zoneadm list -n
8842023-11-08T06:55:01.017Z+++ grep oxz_ntp
8852023-11-08T06:55:01.087Z++ pfexec svcs -z oxz_ntp_e6549cd5-2475-4abb-bc6b-53355bcf4542 -Hostate oxide/ntp
8862023-11-08T06:55:01.116Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
8872023-11-08T06:55:01.146Z++ true
8882023-11-08T06:55:01.179Z+ [[ '' != online ]]
8892023-11-08T06:55:01.208Z+ [[ 14 -gt 60 ]]
8902023-11-08T06:55:01.240Z+ sleep 1
8912023-11-08T06:55:01.987Z+ retry=15
8922023-11-08T06:55:02.013Z+++ zoneadm list -n
8932023-11-08T06:55:02.038Z+++ grep oxz_ntp
8942023-11-08T06:55:02.069Z++ pfexec svcs -z oxz_ntp_e6549cd5-2475-4abb-bc6b-53355bcf4542 -Hostate oxide/ntp
8952023-11-08T06:55:02.124Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
8962023-11-08T06:55:02.158Z++ true
8972023-11-08T06:55:02.186Z+ [[ '' != online ]]
8982023-11-08T06:55:02.211Z+ [[ 15 -gt 60 ]]
8992023-11-08T06:55:02.237Z+ sleep 1
9002023-11-08T06:55:03.024Z+ retry=16
9012023-11-08T06:55:03.053Z+++ zoneadm list -n
9022023-11-08T06:55:03.080Z+++ grep oxz_ntp
9032023-11-08T06:55:03.105Z++ pfexec svcs -z oxz_ntp_e6549cd5-2475-4abb-bc6b-53355bcf4542 -Hostate oxide/ntp
9042023-11-08T06:55:03.154Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
9052023-11-08T06:55:03.183Z++ true
9062023-11-08T06:55:03.218Z+ [[ '' != online ]]
9072023-11-08T06:55:03.688Z+ [[ 16 -gt 60 ]]
9082023-11-08T06:55:03.716Z+ sleep 1
9092023-11-08T06:55:04.057Z+ retry=17
9102023-11-08T06:55:04.091Z+++ zoneadm list -n
9112023-11-08T06:55:04.534Z+++ grep oxz_ntp
9122023-11-08T06:55:04.567Z++ pfexec svcs -z oxz_ntp_e6549cd5-2475-4abb-bc6b-53355bcf4542 -Hostate oxide/ntp
9132023-11-08T06:55:04.612Z+ [[ disabled != online ]]
9142023-11-08T06:55:04.653Z+ [[ 17 -gt 60 ]]
9152023-11-08T06:55:04.745Z+ sleep 1
9162023-11-08T06:55:05.076Z+ retry=18
9172023-11-08T06:55:05.101Z+++ zoneadm list -n
9182023-11-08T06:55:05.132Z+++ grep oxz_ntp
9192023-11-08T06:55:05.158Z++ pfexec svcs -z oxz_ntp_e6549cd5-2475-4abb-bc6b-53355bcf4542 -Hostate oxide/ntp
9202023-11-08T06:55:05.302Z+ [[ disabled != online ]]
9212023-11-08T06:55:05.332Z+ [[ 18 -gt 60 ]]
9222023-11-08T06:55:05.370Z+ sleep 1
9232023-11-08T06:55:06.101Z+ retry=19
9242023-11-08T06:55:06.127Z+++ zoneadm list -n
9252023-11-08T06:55:06.153Z+++ grep oxz_ntp
9262023-11-08T06:55:06.179Z++ pfexec svcs -z oxz_ntp_e6549cd5-2475-4abb-bc6b-53355bcf4542 -Hostate oxide/ntp
9272023-11-08T06:55:06.211Z+ [[ disabled != online ]]
9282023-11-08T06:55:06.242Z+ [[ 19 -gt 60 ]]
9292023-11-08T06:55:06.317Z+ sleep 1
9302023-11-08T06:55:07.125Z+ retry=20
9312023-11-08T06:55:07.156Z+++ zoneadm list -n
9322023-11-08T06:55:07.181Z+++ grep oxz_ntp
9332023-11-08T06:55:07.208Z++ pfexec svcs -z oxz_ntp_e6549cd5-2475-4abb-bc6b-53355bcf4542 -Hostate oxide/ntp
9342023-11-08T06:55:07.242Z+ [[ disabled != online ]]
9352023-11-08T06:55:07.274Z+ [[ 20 -gt 60 ]]
9362023-11-08T06:55:07.308Z+ sleep 1
9372023-11-08T06:55:08.147Z+ retry=21
9382023-11-08T06:55:08.172Z+++ zoneadm list -n
9392023-11-08T06:55:08.196Z+++ grep oxz_ntp
9402023-11-08T06:55:08.632Z++ pfexec svcs -z oxz_ntp_e6549cd5-2475-4abb-bc6b-53355bcf4542 -Hostate oxide/ntp
9412023-11-08T06:55:08.664Z+ [[ disabled != online ]]
9422023-11-08T06:55:08.697Z+ [[ 21 -gt 60 ]]
9432023-11-08T06:55:08.725Z+ sleep 1
9442023-11-08T06:55:09.172Z+ retry=22
9452023-11-08T06:55:09.207Z+++ zoneadm list -n
9462023-11-08T06:55:09.237Z+++ grep oxz_ntp
9472023-11-08T06:55:09.265Z++ pfexec svcs -z oxz_ntp_e6549cd5-2475-4abb-bc6b-53355bcf4542 -Hostate oxide/ntp
9482023-11-08T06:55:09.294Z+ [[ disabled != online ]]
9492023-11-08T06:55:09.326Z+ [[ 22 -gt 60 ]]
9502023-11-08T06:55:09.360Z+ sleep 1
 
9512023-11-08T06:55:10.602Zbuskin console login: Nov 8 06:55:10 buskin ip: ip: joining multicasts failed (4) on vopte0 - will use link layer broadcasts for multicast
 
9522023-11-08T06:55:10.715Z+ retry=23
9532023-11-08T06:55:10.747Z+++ zoneadm list -n
9542023-11-08T06:55:10.775Z+++ grep oxz_ntp
9552023-11-08T06:55:10.809Z++ pfexec svcs -z oxz_ntp_e6549cd5-2475-4abb-bc6b-53355bcf4542 -Hostate oxide/ntp
9562023-11-08T06:55:10.838Z+ [[ disabled != online ]]
9572023-11-08T06:55:10.863Z+ [[ 23 -gt 60 ]]
9582023-11-08T06:55:10.891Z+ sleep 1
9592023-11-08T06:55:11.223Z+ retry=24
9602023-11-08T06:55:11.531Z+++ zoneadm list -n
9612023-11-08T06:55:11.815Z+++ grep oxz_ntp
9622023-11-08T06:55:12.515Z++ pfexec svcs -z oxz_ntp_e6549cd5-2475-4abb-bc6b-53355bcf4542 -Hostate oxide/ntp
9632023-11-08T06:55:12.566Z+ [[ online != online ]]
9642023-11-08T06:55:12.654Z+ echo 'Waited for chrony: 24s'
9652023-11-08T06:55:12.782ZWaited for chrony: 24s
9662023-11-08T06:55:13.081Z+ export RUST_BACKTRACE=1
9672023-11-08T06:55:13.559Z+ RUST_BACKTRACE=1
9682023-11-08T06:55:13.692Z+ export E2E_TLS_CERT IPPOOL_START IPPOOL_END
9692023-11-08T06:55:13.766Z+ ./tests/bootstrap
9702023-11-08T06:55:13.804ZUsing external DNS server at 10.151.1.164:53
9712023-11-08T06:55:13.892Z2023-11-08 06:55:11.581167484 UTC: attempting to log into API
9722023-11-08T06:55:26.299Z2023-11-08 06:55:26.699598503 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9732023-11-08T06:55:27.307Z2023-11-08 06:55:27.697512545 UTC: attempting to log into API
9742023-11-08T06:55:42.334Z2023-11-08 06:55:42.690783424 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9752023-11-08T06:55:43.337Z2023-11-08 06:55:43.688791559 UTC: attempting to log into API
9762023-11-08T06:55:58.375Z2023-11-08 06:55:58.680919346 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9772023-11-08T06:55:59.371Z2023-11-08 06:55:59.678895750 UTC: attempting to log into API
9782023-11-08T06:56:14.406Z2023-11-08 06:56:14.671346776 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9792023-11-08T06:56:15.402Z2023-11-08 06:56:15.669226856 UTC: attempting to log into API
9802023-11-08T06:56:30.439Z2023-11-08 06:56:30.661860957 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9812023-11-08T06:56:31.449Z2023-11-08 06:56:31.660143399 UTC: attempting to log into API
9822023-11-08T06:56:46.883Z2023-11-08 06:56:46.653095809 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9832023-11-08T06:56:47.474Z2023-11-08 06:56:47.652135031 UTC: attempting to log into API
9842023-11-08T06:57:02.909Z2023-11-08 06:57:02.644120790 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9852023-11-08T06:57:03.513Z2023-11-08 06:57:03.642223667 UTC: attempting to log into API
9862023-11-08T06:57:18.574Z2023-11-08 06:57:18.635603546 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9872023-11-08T06:57:19.990Z2023-11-08 06:57:19.633592020 UTC: attempting to log into API
9882023-11-08T06:57:36.225Z2023-11-08 06:57:34.625789477 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9892023-11-08T06:57:36.341Z2023-11-08 06:57:35.623773951 UTC: attempting to log into API
9902023-11-08T06:57:50.606Z2023-11-08 06:57:50.616440021 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9912023-11-08T06:57:51.608Z2023-11-08 06:57:51.615830445 UTC: attempting to log into API
9922023-11-08T06:58:06.641Z2023-11-08 06:58:06.628202769 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9932023-11-08T06:58:07.643Z2023-11-08 06:58:07.629072026 UTC: attempting to log into API
9942023-11-08T06:58:22.679Z2023-11-08 06:58:22.663383914 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9952023-11-08T06:58:23.689Z2023-11-08 06:58:23.666181258 UTC: attempting to log into API
9962023-11-08T06:58:42.690Z2023-11-08 06:58:38.733136612 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9972023-11-08T06:58:42.891Z2023-11-08 06:58:39.733696910 UTC: attempting to log into API
9982023-11-08T06:58:54.783Z2023-11-08 06:58:54.772164943 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
 
9992023-11-08T06:58:55.531ZNov 8 06:58:54 buskin ip: ip: joining multicasts failed (4) on vopte5 - will use link layer broadcasts for multicast
 
10002023-11-08T06:58:55.929Z2023-11-08 06:58:55.773314470 UTC: attempting to log into API
 
10012023-11-08T06:59:03.253ZNov 8 06:59:02 buskin ip: ip: joining multicasts failed (4) on vopte1 - will use link layer broadcasts for multicast
10022023-11-08T06:59:10.331ZNov 8 06:59:09 buskin ip: ip: joining multicasts failed (4) on vopte2 - will use link layer broadcasts for multicast
 
10032023-11-08T06:59:10.819Z2023-11-08 06:59:10.806897761 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
10042023-11-08T06:59:11.824Z2023-11-08 06:59:11.807969596 UTC: attempting to log into API
 
10052023-11-08T06:59:17.681ZNov 8 06:59:17 buskin ip: ip: joining multicasts failed (4) on vopte3 - will use link layer broadcasts for multicast
10062023-11-08T06:59:24.261ZNov 8 06:59:23 buskin ip: ip: joining multicasts failed (4) on vopte4 - will use link layer broadcasts for multicast
 
10072023-11-08T06:59:27.233Z2023-11-08 06:59:26.841244469 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
10082023-11-08T06:59:27.852Z2023-11-08 06:59:27.842276212 UTC: attempting to log into API
10092023-11-08T06:59:42.885Z2023-11-08 06:59:42.874368934 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
10102023-11-08T06:59:43.902Z2023-11-08 06:59:43.875315267 UTC: attempting to log into API
10112023-11-08T06:59:44.301Z2023-11-08 06:59:43.912420226 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: dns error: no record found for Query { name: Name("recovery.sys.oxide.test."), query_type: AAAA, query_class: IN }
10122023-11-08T06:59:44.961Z2023-11-08 06:59:44.913844542 UTC: attempting to log into API
10132023-11-08T06:59:45.703Z2023-11-08 06:59:45.692844943 UTC: login succeeded
10142023-11-08T06:59:45.815Zwaiting for nexus to come up...
10152023-11-08T06:59:46.119Zcreating IP pool... 10.151.1.175 - 10.151.1.227
10162023-11-08T06:59:46.252Zensuring datasets are ready...
10172023-11-08T06:59:47.835Zlet's roll.
10182023-11-08T06:59:47.996Z+ rm ./tests/bootstrap
10192023-11-08T06:59:48.134Z+ for test_bin in tests/*
10202023-11-08T06:59:48.415Z+ ./tests/bootstrap-a6f7429594f412b4
10212023-11-08T06:59:48.513Z
10222023-11-08T06:59:48.640Zrunning 0 tests
10232023-11-08T06:59:48.745Z
10242023-11-08T06:59:48.948Ztest result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
10252023-11-08T06:59:49.113Z
10262023-11-08T06:59:49.200Z+ for test_bin in tests/*
10272023-11-08T06:59:49.286Z+ ./tests/end_to_end_tests-294235f008612b3a
10282023-11-08T06:59:49.404Z
10292023-11-08T06:59:49.500Zrunning 2 tests
10302023-11-08T06:59:49.594Ztest no_spoof::no_spoof_here ... ok
10312023-11-08T07:00:54.456Ztest instance_launch::instance_launch has been running for over 60 seconds
10322023-11-08T07:02:45.700Ztest instance_launch::instance_launch ... ok
10332023-11-08T07:02:45.729Z
10342023-11-08T07:02:45.757Ztest result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 177.74s
10352023-11-08T07:02:45.785Z
10362023-11-08T07:02:45.810Z+ _exit_trap
10372023-11-08T07:02:45.836Z+ local status=0
10382023-11-08T07:02:45.875Z+ [[ 0 -eq 0 ]]
10392023-11-08T07:02:45.903Z+ exit 0
10402023-11-08T07:02:45.930Zprocess exited: duration 557430 ms, exit code 0
 
10412023-11-08T07:02:46.013Zfound 90 output files
10422023-11-08T07:02:46.073Zuploading: /var/svc/log/oxide-sled-agent:default.log (664926 bytes)
10432023-11-08T07:02:47.342Zuploaded: /var/svc/log/oxide-sled-agent:default.log
10442023-11-08T07:02:47.373Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_6216fa3e-8985-4cef-a593-0f1e3188c9d6/root/var/svc/log/oxide-cockroachdb:default.log (0 bytes)
10452023-11-08T07:02:48.419Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_6216fa3e-8985-4cef-a593-0f1e3188c9d6/root/var/svc/log/oxide-cockroachdb:default.log
10462023-11-08T07:02:48.444Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_6216fa3e-8985-4cef-a593-0f1e3188c9d6/root/var/svc/log/oxide-cockroachdb:default.log.0 (5369 bytes)
10472023-11-08T07:02:49.547Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_6216fa3e-8985-4cef-a593-0f1e3188c9d6/root/var/svc/log/oxide-cockroachdb:default.log.0
10482023-11-08T07:02:49.580Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_21d55b92-e14d-4780-ad13-52bb4dfc4a7a/root/var/svc/log/oxide-crucible-agent:default.log (0 bytes)
10492023-11-08T07:02:50.636Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_21d55b92-e14d-4780-ad13-52bb4dfc4a7a/root/var/svc/log/oxide-crucible-agent:default.log
10502023-11-08T07:02:50.663Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_21d55b92-e14d-4780-ad13-52bb4dfc4a7a/root/var/svc/log/oxide-crucible-agent:default.log.0 (14110 bytes)
10512023-11-08T07:02:52.174Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_21d55b92-e14d-4780-ad13-52bb4dfc4a7a/root/var/svc/log/oxide-crucible-agent:default.log.0
10522023-11-08T07:02:52.200Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_21d55b92-e14d-4780-ad13-52bb4dfc4a7a/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
10532023-11-08T07:02:53.246Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_21d55b92-e14d-4780-ad13-52bb4dfc4a7a/root/var/svc/log/oxide-crucible-downstairs:default.log
10542023-11-08T07:02:53.270Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_21d55b92-e14d-4780-ad13-52bb4dfc4a7a/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
10552023-11-08T07:02:54.357Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_21d55b92-e14d-4780-ad13-52bb4dfc4a7a/root/var/svc/log/oxide-crucible-downstairs:default.log.0
10562023-11-08T07:02:54.416Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_21d55b92-e14d-4780-ad13-52bb4dfc4a7a/root/var/svc/log/oxide-crucible-downstairs:downstairs-35227d2c-427b-4de8-8dcd-896ffc43ae00.log (0 bytes)
10572023-11-08T07:02:55.430Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_21d55b92-e14d-4780-ad13-52bb4dfc4a7a/root/var/svc/log/oxide-crucible-downstairs:downstairs-35227d2c-427b-4de8-8dcd-896ffc43ae00.log
10582023-11-08T07:02:55.456Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_21d55b92-e14d-4780-ad13-52bb4dfc4a7a/root/var/svc/log/oxide-crucible-downstairs:downstairs-35227d2c-427b-4de8-8dcd-896ffc43ae00.log.0 (3051 bytes)
10592023-11-08T07:02:56.529Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_21d55b92-e14d-4780-ad13-52bb4dfc4a7a/root/var/svc/log/oxide-crucible-downstairs:downstairs-35227d2c-427b-4de8-8dcd-896ffc43ae00.log.0
10602023-11-08T07:02:56.575Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_oximeter_2dd53c9f-e723-4f6d-9546-e7826ca557b4/root/var/svc/log/oxide-oximeter:default.log (148145 bytes)
10612023-11-08T07:02:57.430Zupload warning: file "/pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_oximeter_2dd53c9f-e723-4f6d-9546-e7826ca557b4/root/var/svc/log/oxide-oximeter:default.log" changed size mid upload: 148145 -> 156922
10622023-11-08T07:02:58.510Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_oximeter_2dd53c9f-e723-4f6d-9546-e7826ca557b4/root/var/svc/log/oxide-oximeter:default.log
10632023-11-08T07:02:58.536Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_oximeter_2dd53c9f-e723-4f6d-9546-e7826ca557b4/root/var/svc/log/oxide-oximeter:default.log.0 (46425 bytes)
10642023-11-08T07:02:59.727Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_oximeter_2dd53c9f-e723-4f6d-9546-e7826ca557b4/root/var/svc/log/oxide-oximeter:default.log.0
10652023-11-08T07:02:59.756Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_8a2002db-d0d4-463a-a435-aad9cbe764cf/root/var/svc/log/oxide-crucible-agent:default.log (0 bytes)
10662023-11-08T07:03:00.819Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_8a2002db-d0d4-463a-a435-aad9cbe764cf/root/var/svc/log/oxide-crucible-agent:default.log
10672023-11-08T07:03:00.846Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_8a2002db-d0d4-463a-a435-aad9cbe764cf/root/var/svc/log/oxide-crucible-agent:default.log.0 (3436 bytes)
10682023-11-08T07:03:01.935Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_8a2002db-d0d4-463a-a435-aad9cbe764cf/root/var/svc/log/oxide-crucible-agent:default.log.0
10692023-11-08T07:03:01.966Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_8a2002db-d0d4-463a-a435-aad9cbe764cf/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
10702023-11-08T07:03:03.013Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_8a2002db-d0d4-463a-a435-aad9cbe764cf/root/var/svc/log/oxide-crucible-downstairs:default.log
10712023-11-08T07:03:03.040Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_8a2002db-d0d4-463a-a435-aad9cbe764cf/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
10722023-11-08T07:03:04.138Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_8a2002db-d0d4-463a-a435-aad9cbe764cf/root/var/svc/log/oxide-crucible-downstairs:default.log.0
10732023-11-08T07:03:04.167Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_9ec408c4-6ce4-4b2c-8e43-377a6eccbaac/root/var/svc/log/oxide-crucible-agent:default.log (0 bytes)
10742023-11-08T07:03:06.080Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_9ec408c4-6ce4-4b2c-8e43-377a6eccbaac/root/var/svc/log/oxide-crucible-agent:default.log
10752023-11-08T07:03:06.116Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_9ec408c4-6ce4-4b2c-8e43-377a6eccbaac/root/var/svc/log/oxide-crucible-agent:default.log.0 (13685 bytes)
10762023-11-08T07:03:07.227Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_9ec408c4-6ce4-4b2c-8e43-377a6eccbaac/root/var/svc/log/oxide-crucible-agent:default.log.0
10772023-11-08T07:03:07.254Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_9ec408c4-6ce4-4b2c-8e43-377a6eccbaac/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
10782023-11-08T07:03:08.317Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_9ec408c4-6ce4-4b2c-8e43-377a6eccbaac/root/var/svc/log/oxide-crucible-downstairs:default.log
10792023-11-08T07:03:08.348Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_9ec408c4-6ce4-4b2c-8e43-377a6eccbaac/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
10802023-11-08T07:03:09.457Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_9ec408c4-6ce4-4b2c-8e43-377a6eccbaac/root/var/svc/log/oxide-crucible-downstairs:default.log.0
10812023-11-08T07:03:09.488Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_9ec408c4-6ce4-4b2c-8e43-377a6eccbaac/root/var/svc/log/oxide-crucible-downstairs:downstairs-5f363b05-e97d-4338-b8e9-c89d6198382d.log (0 bytes)
10822023-11-08T07:03:10.540Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_9ec408c4-6ce4-4b2c-8e43-377a6eccbaac/root/var/svc/log/oxide-crucible-downstairs:downstairs-5f363b05-e97d-4338-b8e9-c89d6198382d.log
10832023-11-08T07:03:10.572Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_9ec408c4-6ce4-4b2c-8e43-377a6eccbaac/root/var/svc/log/oxide-crucible-downstairs:downstairs-5f363b05-e97d-4338-b8e9-c89d6198382d.log.0 (3050 bytes)
10842023-11-08T07:03:11.639Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_9ec408c4-6ce4-4b2c-8e43-377a6eccbaac/root/var/svc/log/oxide-crucible-downstairs:downstairs-5f363b05-e97d-4338-b8e9-c89d6198382d.log.0
10852023-11-08T07:03:11.665Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_33a38aaf-1469-4af0-bc90-be202273759d/root/var/svc/log/oxide-nexus:default.log (320312 bytes)
10862023-11-08T07:03:12.467Zupload warning: file "/pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_33a38aaf-1469-4af0-bc90-be202273759d/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 320312 -> 321528
10872023-11-08T07:03:13.540Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_33a38aaf-1469-4af0-bc90-be202273759d/root/var/svc/log/oxide-nexus:default.log
10882023-11-08T07:03:13.569Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_33a38aaf-1469-4af0-bc90-be202273759d/root/var/svc/log/oxide-nexus:default.log.0 (487469 bytes)
10892023-11-08T07:03:15.883Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_33a38aaf-1469-4af0-bc90-be202273759d/root/var/svc/log/oxide-nexus:default.log.0
10902023-11-08T07:03:15.909Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_ec378940-3587-4f60-90e2-7be002dd08ca/root/var/svc/log/oxide-nexus:default.log (320319 bytes)
10912023-11-08T07:03:17.025Zupload warning: file "/pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_ec378940-3587-4f60-90e2-7be002dd08ca/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 320319 -> 322142
10922023-11-08T07:03:18.515Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_ec378940-3587-4f60-90e2-7be002dd08ca/root/var/svc/log/oxide-nexus:default.log
10932023-11-08T07:03:18.543Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_ec378940-3587-4f60-90e2-7be002dd08ca/root/var/svc/log/oxide-nexus:default.log.0 (194544 bytes)
10942023-11-08T07:03:20.006Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_ec378940-3587-4f60-90e2-7be002dd08ca/root/var/svc/log/oxide-nexus:default.log.0
10952023-11-08T07:03:20.038Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_clickhouse_43572699-e0ef-4658-ac27-481e6c16b76c/root/var/svc/log/oxide-clickhouse:default.log (544636 bytes)
10962023-11-08T07:03:21.334Zupload warning: file "/pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_clickhouse_43572699-e0ef-4658-ac27-481e6c16b76c/root/var/svc/log/oxide-clickhouse:default.log" changed size mid upload: 544636 -> 646495
10972023-11-08T07:03:22.419Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_clickhouse_43572699-e0ef-4658-ac27-481e6c16b76c/root/var/svc/log/oxide-clickhouse:default.log
10982023-11-08T07:03:22.447Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_clickhouse_43572699-e0ef-4658-ac27-481e6c16b76c/root/var/svc/log/oxide-clickhouse:default.log.0 (184961 bytes)
10992023-11-08T07:03:23.930Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_clickhouse_43572699-e0ef-4658-ac27-481e6c16b76c/root/var/svc/log/oxide-clickhouse:default.log.0
11002023-11-08T07:03:23.956Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_nexus_160d4bcd-6207-4380-88e5-278cacf938b4/root/var/svc/log/oxide-nexus:default.log (1768487 bytes)
11012023-11-08T07:03:25.915Zupload warning: file "/pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_nexus_160d4bcd-6207-4380-88e5-278cacf938b4/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 1768487 -> 1829101
11022023-11-08T07:03:27.476Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_nexus_160d4bcd-6207-4380-88e5-278cacf938b4/root/var/svc/log/oxide-nexus:default.log
11032023-11-08T07:03:27.934Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_nexus_160d4bcd-6207-4380-88e5-278cacf938b4/root/var/svc/log/oxide-nexus:default.log.0 (2586176 bytes)
11042023-11-08T07:03:33.238Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_nexus_160d4bcd-6207-4380-88e5-278cacf938b4/root/var/svc/log/oxide-nexus:default.log.0
11052023-11-08T07:03:33.280Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_ntp_e6549cd5-2475-4abb-bc6b-53355bcf4542/root/var/svc/log/oxide-ntp:default.log (0 bytes)
11062023-11-08T07:03:34.410Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_ntp_e6549cd5-2475-4abb-bc6b-53355bcf4542/root/var/svc/log/oxide-ntp:default.log
11072023-11-08T07:03:35.783Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_ntp_e6549cd5-2475-4abb-bc6b-53355bcf4542/root/var/svc/log/oxide-ntp:default.log.0 (1386 bytes)
11082023-11-08T07:03:44.573Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_ntp_e6549cd5-2475-4abb-bc6b-53355bcf4542/root/var/svc/log/oxide-ntp:default.log.0
11092023-11-08T07:03:44.607Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_pantry_c0fdba41-8255-49a6-a8dd-c997557d213f/root/var/svc/log/oxide-crucible-pantry:default.log (0 bytes)
11102023-11-08T07:03:45.648Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_pantry_c0fdba41-8255-49a6-a8dd-c997557d213f/root/var/svc/log/oxide-crucible-pantry:default.log
11112023-11-08T07:03:45.674Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_pantry_c0fdba41-8255-49a6-a8dd-c997557d213f/root/var/svc/log/oxide-crucible-pantry:default.log.0 (2338 bytes)
11122023-11-08T07:03:46.749Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_pantry_c0fdba41-8255-49a6-a8dd-c997557d213f/root/var/svc/log/oxide-crucible-pantry:default.log.0
11132023-11-08T07:03:46.790Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_external_dns_3e641d6c-445d-4905-9139-22836bf7f7b6/root/var/svc/log/oxide-external_dns:default.log (11330 bytes)
11142023-11-08T07:03:46.866Zupload warning: file "/pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_external_dns_3e641d6c-445d-4905-9139-22836bf7f7b6/root/var/svc/log/oxide-external_dns:default.log" changed size mid upload: 11330 -> 15107
11152023-11-08T07:03:47.945Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_external_dns_3e641d6c-445d-4905-9139-22836bf7f7b6/root/var/svc/log/oxide-external_dns:default.log
11162023-11-08T07:03:47.981Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_external_dns_3e641d6c-445d-4905-9139-22836bf7f7b6/root/var/svc/log/oxide-external_dns:default.log.0 (5045 bytes)
11172023-11-08T07:03:49.042Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_external_dns_3e641d6c-445d-4905-9139-22836bf7f7b6/root/var/svc/log/oxide-external_dns:default.log.0
11182023-11-08T07:03:49.083Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_external_dns_b29f4dad-cc2f-48a1-b92b-489040c5df15/root/var/svc/log/oxide-external_dns:default.log (11330 bytes)
11192023-11-08T07:03:49.159Zupload warning: file "/pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_external_dns_b29f4dad-cc2f-48a1-b92b-489040c5df15/root/var/svc/log/oxide-external_dns:default.log" changed size mid upload: 11330 -> 15103
11202023-11-08T07:03:50.246Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_external_dns_b29f4dad-cc2f-48a1-b92b-489040c5df15/root/var/svc/log/oxide-external_dns:default.log
11212023-11-08T07:03:50.287Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_external_dns_b29f4dad-cc2f-48a1-b92b-489040c5df15/root/var/svc/log/oxide-external_dns:default.log.0 (4283 bytes)
11222023-11-08T07:03:51.366Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_external_dns_b29f4dad-cc2f-48a1-b92b-489040c5df15/root/var/svc/log/oxide-external_dns:default.log.0
11232023-11-08T07:03:51.401Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_internal_dns_7e5ab4ae-8602-4d4b-b0d1-a6a35f1f0f94/root/var/svc/log/oxide-internal_dns:default.log (11262 bytes)
11242023-11-08T07:03:51.464Zupload warning: file "/pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_internal_dns_7e5ab4ae-8602-4d4b-b0d1-a6a35f1f0f94/root/var/svc/log/oxide-internal_dns:default.log" changed size mid upload: 11262 -> 15016
11252023-11-08T07:03:52.546Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_internal_dns_7e5ab4ae-8602-4d4b-b0d1-a6a35f1f0f94/root/var/svc/log/oxide-internal_dns:default.log
11262023-11-08T07:03:52.584Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_internal_dns_7e5ab4ae-8602-4d4b-b0d1-a6a35f1f0f94/root/var/svc/log/oxide-internal_dns:default.log.0 (101697 bytes)
11272023-11-08T07:03:53.980Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_internal_dns_7e5ab4ae-8602-4d4b-b0d1-a6a35f1f0f94/root/var/svc/log/oxide-internal_dns:default.log.0
11282023-11-08T07:03:54.009Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_5b088b6f-d1d0-4107-9778-723a53157f9b/root/var/svc/log/oxide-crucible-agent:default.log (0 bytes)
11292023-11-08T07:03:55.597Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_5b088b6f-d1d0-4107-9778-723a53157f9b/root/var/svc/log/oxide-crucible-agent:default.log
11302023-11-08T07:03:55.671Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_5b088b6f-d1d0-4107-9778-723a53157f9b/root/var/svc/log/oxide-crucible-agent:default.log.0 (3436 bytes)
11312023-11-08T07:03:56.727Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_5b088b6f-d1d0-4107-9778-723a53157f9b/root/var/svc/log/oxide-crucible-agent:default.log.0
11322023-11-08T07:03:56.784Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_5b088b6f-d1d0-4107-9778-723a53157f9b/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
11332023-11-08T07:03:57.847Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_5b088b6f-d1d0-4107-9778-723a53157f9b/root/var/svc/log/oxide-crucible-downstairs:default.log
11342023-11-08T07:03:57.919Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_5b088b6f-d1d0-4107-9778-723a53157f9b/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
11352023-11-08T07:03:59.011Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_5b088b6f-d1d0-4107-9778-723a53157f9b/root/var/svc/log/oxide-crucible-downstairs:default.log.0
11362023-11-08T07:03:59.085Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_e55025a9-becd-47d8-a303-caefb4d254ee/root/var/svc/log/oxide-crucible-agent:default.log (4787 bytes)
11372023-11-08T07:04:00.363Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_e55025a9-becd-47d8-a303-caefb4d254ee/root/var/svc/log/oxide-crucible-agent:default.log
11382023-11-08T07:04:12.608Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_e55025a9-becd-47d8-a303-caefb4d254ee/root/var/svc/log/oxide-crucible-agent:default.log.0 (20038 bytes)
11392023-11-08T07:04:14.301Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_e55025a9-becd-47d8-a303-caefb4d254ee/root/var/svc/log/oxide-crucible-agent:default.log.0
11402023-11-08T07:04:14.500Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_e55025a9-becd-47d8-a303-caefb4d254ee/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
11412023-11-08T07:04:28.254Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_e55025a9-becd-47d8-a303-caefb4d254ee/root/var/svc/log/oxide-crucible-downstairs:default.log
11422023-11-08T07:04:28.553Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_e55025a9-becd-47d8-a303-caefb4d254ee/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
11432023-11-08T07:04:29.992Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_e55025a9-becd-47d8-a303-caefb4d254ee/root/var/svc/log/oxide-crucible-downstairs:default.log.0
11442023-11-08T07:04:30.141Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_e55025a9-becd-47d8-a303-caefb4d254ee/root/var/svc/log/oxide-crucible-downstairs:downstairs-ac0b58bd-5bc0-482b-a959-f73d1d488594.log (1006 bytes)
11452023-11-08T07:04:31.684Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_e55025a9-becd-47d8-a303-caefb4d254ee/root/var/svc/log/oxide-crucible-downstairs:downstairs-ac0b58bd-5bc0-482b-a959-f73d1d488594.log
11462023-11-08T07:04:31.923Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_e55025a9-becd-47d8-a303-caefb4d254ee/root/var/svc/log/oxide-crucible-downstairs:downstairs-ac0b58bd-5bc0-482b-a959-f73d1d488594.log.0 (4529 bytes)
11472023-11-08T07:04:33.279Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_e55025a9-becd-47d8-a303-caefb4d254ee/root/var/svc/log/oxide-crucible-downstairs:downstairs-ac0b58bd-5bc0-482b-a959-f73d1d488594.log.0
11482023-11-08T07:04:33.369Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_e55025a9-becd-47d8-a303-caefb4d254ee/root/var/svc/log/oxide-crucible-downstairs:downstairs-b74de7d0-a61a-4144-977c-f623378e191a.log (0 bytes)
11492023-11-08T07:04:34.992Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_e55025a9-becd-47d8-a303-caefb4d254ee/root/var/svc/log/oxide-crucible-downstairs:downstairs-b74de7d0-a61a-4144-977c-f623378e191a.log
11502023-11-08T07:04:35.651Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_e55025a9-becd-47d8-a303-caefb4d254ee/root/var/svc/log/oxide-crucible-downstairs:downstairs-b74de7d0-a61a-4144-977c-f623378e191a.log.0 (3048 bytes)
11512023-11-08T07:04:49.084Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_e55025a9-becd-47d8-a303-caefb4d254ee/root/var/svc/log/oxide-crucible-downstairs:downstairs-b74de7d0-a61a-4144-977c-f623378e191a.log.0
11522023-11-08T07:04:49.625Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_pantry_3c82b59c-1bb8-44c1-9d80-6c4c875303ef/root/var/svc/log/oxide-crucible-pantry:default.log (0 bytes)
11532023-11-08T07:05:05.655Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_pantry_3c82b59c-1bb8-44c1-9d80-6c4c875303ef/root/var/svc/log/oxide-crucible-pantry:default.log
11542023-11-08T07:05:06.107Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_pantry_3c82b59c-1bb8-44c1-9d80-6c4c875303ef/root/var/svc/log/oxide-crucible-pantry:default.log.0 (2338 bytes)
11552023-11-08T07:05:07.348Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_pantry_3c82b59c-1bb8-44c1-9d80-6c4c875303ef/root/var/svc/log/oxide-crucible-pantry:default.log.0
11562023-11-08T07:05:07.901Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_cockroachdb_72cd1a3c-388a-4567-9bee-b2e774044585/root/var/svc/log/oxide-cockroachdb:default.log (0 bytes)
11572023-11-08T07:05:07.958Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_cockroachdb_72cd1a3c-388a-4567-9bee-b2e774044585/root/var/svc/log/oxide-cockroachdb:default.log
11582023-11-08T07:05:07.999Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_cockroachdb_72cd1a3c-388a-4567-9bee-b2e774044585/root/var/svc/log/oxide-cockroachdb:default.log.0 (5370 bytes)
11592023-11-08T07:05:08.286Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_cockroachdb_72cd1a3c-388a-4567-9bee-b2e774044585/root/var/svc/log/oxide-cockroachdb:default.log.0
11602023-11-08T07:05:08.349Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_cockroachdb_e61db6d2-be53-4a13-afcd-e88a9466dc80/root/var/svc/log/oxide-cockroachdb:default.log (0 bytes)
11612023-11-08T07:05:09.405Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_cockroachdb_e61db6d2-be53-4a13-afcd-e88a9466dc80/root/var/svc/log/oxide-cockroachdb:default.log
11622023-11-08T07:05:09.455Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_cockroachdb_e61db6d2-be53-4a13-afcd-e88a9466dc80/root/var/svc/log/oxide-cockroachdb:default.log.0 (5367 bytes)
11632023-11-08T07:05:10.516Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_cockroachdb_e61db6d2-be53-4a13-afcd-e88a9466dc80/root/var/svc/log/oxide-cockroachdb:default.log.0
11642023-11-08T07:05:10.544Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_20c855df-4088-4df4-bdb1-33c82aa783f2/root/var/svc/log/oxide-crucible-agent:default.log (0 bytes)
11652023-11-08T07:05:11.598Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_20c855df-4088-4df4-bdb1-33c82aa783f2/root/var/svc/log/oxide-crucible-agent:default.log
11662023-11-08T07:05:11.627Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_20c855df-4088-4df4-bdb1-33c82aa783f2/root/var/svc/log/oxide-crucible-agent:default.log.0 (3436 bytes)
11672023-11-08T07:05:13.125Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_20c855df-4088-4df4-bdb1-33c82aa783f2/root/var/svc/log/oxide-crucible-agent:default.log.0
11682023-11-08T07:05:13.177Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_20c855df-4088-4df4-bdb1-33c82aa783f2/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
11692023-11-08T07:05:14.242Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_20c855df-4088-4df4-bdb1-33c82aa783f2/root/var/svc/log/oxide-crucible-downstairs:default.log
11702023-11-08T07:05:14.269Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_20c855df-4088-4df4-bdb1-33c82aa783f2/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
11712023-11-08T07:05:15.351Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_20c855df-4088-4df4-bdb1-33c82aa783f2/root/var/svc/log/oxide-crucible-downstairs:default.log.0
11722023-11-08T07:05:15.379Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_pantry_0f01d014-15bd-4f16-9217-33fe32b41827/root/var/svc/log/oxide-crucible-pantry:default.log (0 bytes)
11732023-11-08T07:05:16.429Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_pantry_0f01d014-15bd-4f16-9217-33fe32b41827/root/var/svc/log/oxide-crucible-pantry:default.log
11742023-11-08T07:05:16.461Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_pantry_0f01d014-15bd-4f16-9217-33fe32b41827/root/var/svc/log/oxide-crucible-pantry:default.log.0 (2333 bytes)
11752023-11-08T07:05:17.535Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_pantry_0f01d014-15bd-4f16-9217-33fe32b41827/root/var/svc/log/oxide-crucible-pantry:default.log.0
11762023-11-08T07:05:17.564Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_internal_dns_36a40fae-90de-4548-be54-11f562cd3520/root/var/svc/log/oxide-internal_dns:default.log (23862 bytes)
11772023-11-08T07:05:17.754Zupload warning: file "/pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_internal_dns_36a40fae-90de-4548-be54-11f562cd3520/root/var/svc/log/oxide-internal_dns:default.log" changed size mid upload: 23862 -> 31374
11782023-11-08T07:05:18.831Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_internal_dns_36a40fae-90de-4548-be54-11f562cd3520/root/var/svc/log/oxide-internal_dns:default.log
11792023-11-08T07:05:18.859Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_internal_dns_36a40fae-90de-4548-be54-11f562cd3520/root/var/svc/log/oxide-internal_dns:default.log.0 (90323 bytes)
11802023-11-08T07:05:20.614Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_internal_dns_36a40fae-90de-4548-be54-11f562cd3520/root/var/svc/log/oxide-internal_dns:default.log.0
11812023-11-08T07:05:20.648Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_cockroachdb_c1f7da38-bdd6-4773-93aa-f891f8bb05aa/root/var/svc/log/oxide-cockroachdb:default.log (0 bytes)
11822023-11-08T07:05:21.697Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_cockroachdb_c1f7da38-bdd6-4773-93aa-f891f8bb05aa/root/var/svc/log/oxide-cockroachdb:default.log
11832023-11-08T07:05:21.724Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_cockroachdb_c1f7da38-bdd6-4773-93aa-f891f8bb05aa/root/var/svc/log/oxide-cockroachdb:default.log.0 (5368 bytes)
11842023-11-08T07:05:22.829Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_cockroachdb_c1f7da38-bdd6-4773-93aa-f891f8bb05aa/root/var/svc/log/oxide-cockroachdb:default.log.0
11852023-11-08T07:05:22.861Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_ff392c69-0b98-40da-9978-d0bb98fda171/root/var/svc/log/oxide-crucible-agent:default.log (0 bytes)
11862023-11-08T07:05:23.934Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_ff392c69-0b98-40da-9978-d0bb98fda171/root/var/svc/log/oxide-crucible-agent:default.log
11872023-11-08T07:05:23.962Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_ff392c69-0b98-40da-9978-d0bb98fda171/root/var/svc/log/oxide-crucible-agent:default.log.0 (3436 bytes)
11882023-11-08T07:05:25.451Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_ff392c69-0b98-40da-9978-d0bb98fda171/root/var/svc/log/oxide-crucible-agent:default.log.0
11892023-11-08T07:05:25.481Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_ff392c69-0b98-40da-9978-d0bb98fda171/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
11902023-11-08T07:05:26.544Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_ff392c69-0b98-40da-9978-d0bb98fda171/root/var/svc/log/oxide-crucible-downstairs:default.log
11912023-11-08T07:05:26.570Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_ff392c69-0b98-40da-9978-d0bb98fda171/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
11922023-11-08T07:05:27.652Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_ff392c69-0b98-40da-9978-d0bb98fda171/root/var/svc/log/oxide-crucible-downstairs:default.log.0
11932023-11-08T07:05:27.689Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_internal_dns_0d838878-5ae6-4c44-a600-9c44dfe6614f/root/var/svc/log/oxide-internal_dns:default.log (13785 bytes)
11942023-11-08T07:05:28.646Zupload warning: file "/pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_internal_dns_0d838878-5ae6-4c44-a600-9c44dfe6614f/root/var/svc/log/oxide-internal_dns:default.log" changed size mid upload: 13785 -> 25051
11952023-11-08T07:05:29.736Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_internal_dns_0d838878-5ae6-4c44-a600-9c44dfe6614f/root/var/svc/log/oxide-internal_dns:default.log
11962023-11-08T07:05:29.774Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_internal_dns_0d838878-5ae6-4c44-a600-9c44dfe6614f/root/var/svc/log/oxide-internal_dns:default.log.0 (99981 bytes)
11972023-11-08T07:05:31.144Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_internal_dns_0d838878-5ae6-4c44-a600-9c44dfe6614f/root/var/svc/log/oxide-internal_dns:default.log.0
11982023-11-08T07:05:31.178Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_3c68f5f6-7104-4b77-b9af-c5db8d07df9e/root/var/svc/log/oxide-cockroachdb:default.log (0 bytes)
11992023-11-08T07:05:32.245Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_3c68f5f6-7104-4b77-b9af-c5db8d07df9e/root/var/svc/log/oxide-cockroachdb:default.log
12002023-11-08T07:05:32.284Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_3c68f5f6-7104-4b77-b9af-c5db8d07df9e/root/var/svc/log/oxide-cockroachdb:default.log.0 (5342 bytes)
12012023-11-08T07:05:33.386Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_3c68f5f6-7104-4b77-b9af-c5db8d07df9e/root/var/svc/log/oxide-cockroachdb:default.log.0
12022023-11-08T07:05:33.422Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_232e483d-b34d-4841-97b5-b2609d353f43/root/var/svc/log/oxide-crucible-agent:default.log (4787 bytes)
12032023-11-08T07:05:34.526Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_232e483d-b34d-4841-97b5-b2609d353f43/root/var/svc/log/oxide-crucible-agent:default.log
12042023-11-08T07:05:34.567Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_232e483d-b34d-4841-97b5-b2609d353f43/root/var/svc/log/oxide-crucible-agent:default.log.0 (9796 bytes)
12052023-11-08T07:05:35.679Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_232e483d-b34d-4841-97b5-b2609d353f43/root/var/svc/log/oxide-crucible-agent:default.log.0
12062023-11-08T07:05:35.707Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_232e483d-b34d-4841-97b5-b2609d353f43/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
12072023-11-08T07:05:37.196Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_232e483d-b34d-4841-97b5-b2609d353f43/root/var/svc/log/oxide-crucible-downstairs:default.log
12082023-11-08T07:05:37.241Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_232e483d-b34d-4841-97b5-b2609d353f43/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
12092023-11-08T07:05:38.332Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_232e483d-b34d-4841-97b5-b2609d353f43/root/var/svc/log/oxide-crucible-downstairs:default.log.0
12102023-11-08T07:05:38.357Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_232e483d-b34d-4841-97b5-b2609d353f43/root/var/svc/log/oxide-crucible-downstairs:downstairs-da0d3798-2442-4a74-a53f-e54032dda1ee.log (1006 bytes)
12112023-11-08T07:05:39.435Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_232e483d-b34d-4841-97b5-b2609d353f43/root/var/svc/log/oxide-crucible-downstairs:downstairs-da0d3798-2442-4a74-a53f-e54032dda1ee.log
12122023-11-08T07:05:39.467Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_232e483d-b34d-4841-97b5-b2609d353f43/root/var/svc/log/oxide-crucible-downstairs:downstairs-da0d3798-2442-4a74-a53f-e54032dda1ee.log.0 (4530 bytes)
12132023-11-08T07:05:40.962Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_232e483d-b34d-4841-97b5-b2609d353f43/root/var/svc/log/oxide-crucible-downstairs:downstairs-da0d3798-2442-4a74-a53f-e54032dda1ee.log.0
12142023-11-08T07:05:40.999Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_54dc2251-a1b0-4e97-b389-42277c44d08d/root/var/svc/log/oxide-crucible-agent:default.log (4321 bytes)
12152023-11-08T07:05:42.764Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_54dc2251-a1b0-4e97-b389-42277c44d08d/root/var/svc/log/oxide-crucible-agent:default.log
12162023-11-08T07:05:42.857Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_54dc2251-a1b0-4e97-b389-42277c44d08d/root/var/svc/log/oxide-crucible-agent:default.log.0 (9363 bytes)
12172023-11-08T07:05:44.014Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_54dc2251-a1b0-4e97-b389-42277c44d08d/root/var/svc/log/oxide-crucible-agent:default.log.0
12182023-11-08T07:05:44.041Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_54dc2251-a1b0-4e97-b389-42277c44d08d/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
12192023-11-08T07:05:45.499Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_54dc2251-a1b0-4e97-b389-42277c44d08d/root/var/svc/log/oxide-crucible-downstairs:default.log
12202023-11-08T07:05:45.528Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_54dc2251-a1b0-4e97-b389-42277c44d08d/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
12212023-11-08T07:05:46.608Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_54dc2251-a1b0-4e97-b389-42277c44d08d/root/var/svc/log/oxide-crucible-downstairs:default.log.0
12222023-11-08T07:05:46.636Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_54dc2251-a1b0-4e97-b389-42277c44d08d/root/var/svc/log/oxide-crucible-downstairs:downstairs-c329f39d-dd10-4d73-8406-963662cc33e8.log (1006 bytes)
12232023-11-08T07:05:48.143Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_54dc2251-a1b0-4e97-b389-42277c44d08d/root/var/svc/log/oxide-crucible-downstairs:downstairs-c329f39d-dd10-4d73-8406-963662cc33e8.log
12242023-11-08T07:05:48.175Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_54dc2251-a1b0-4e97-b389-42277c44d08d/root/var/svc/log/oxide-crucible-downstairs:downstairs-c329f39d-dd10-4d73-8406-963662cc33e8.log.0 (4527 bytes)
12252023-11-08T07:05:49.256Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_54dc2251-a1b0-4e97-b389-42277c44d08d/root/var/svc/log/oxide-crucible-downstairs:downstairs-c329f39d-dd10-4d73-8406-963662cc33e8.log.0
12262023-11-08T07:05:49.282Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_ntp_e6549cd5-2475-4abb-bc6b-53355bcf4542/root/var/log/chrony/measurements.log (16440 bytes)
12272023-11-08T07:05:49.403Zupload warning: file "/pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_ntp_e6549cd5-2475-4abb-bc6b-53355bcf4542/root/var/log/chrony/measurements.log" changed size mid upload: 16440 -> 26030
12282023-11-08T07:05:50.482Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_ntp_e6549cd5-2475-4abb-bc6b-53355bcf4542/root/var/log/chrony/measurements.log
12292023-11-08T07:05:50.507Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_ntp_e6549cd5-2475-4abb-bc6b-53355bcf4542/root/var/log/chrony/statistics.log (10881 bytes)
12302023-11-08T07:05:50.995Zupload warning: file "/pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_ntp_e6549cd5-2475-4abb-bc6b-53355bcf4542/root/var/log/chrony/statistics.log" changed size mid upload: 10881 -> 17199
12312023-11-08T07:05:52.074Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_ntp_e6549cd5-2475-4abb-bc6b-53355bcf4542/root/var/log/chrony/statistics.log
12322023-11-08T07:05:52.611Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_ntp_e6549cd5-2475-4abb-bc6b-53355bcf4542/root/var/log/chrony/tracking.log (2251 bytes)
12332023-11-08T07:05:52.648Zupload warning: file "/pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_ntp_e6549cd5-2475-4abb-bc6b-53355bcf4542/root/var/log/chrony/tracking.log" changed size mid upload: 2251 -> 2647
12342023-11-08T07:05:53.267Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_ntp_e6549cd5-2475-4abb-bc6b-53355bcf4542/root/var/log/chrony/tracking.log