01HFJZM5QYQQJJQQRZMCYMS3VS: helios / deploy

BasicConfig {
    output_rules: [
        "%/var/svc/log/oxide-sled-agent:default.log*",
        "%/pool/ext/*/crypt/zone/oxz_*/root/var/svc/log/oxide-*.log*",
        "%/pool/ext/*/crypt/zone/oxz_*/root/var/svc/log/system-illumos-*.log*",
        "%/pool/ext/*/crypt/zone/oxz_ntp_*/root/var/log/chrony/*.log*",
        "!/pool/ext/*/crypt/zone/oxz_propolis-server_*/root/var/svc/log/*.log*",
        "%/pool/ext/*/crypt/debug/global/oxide-sled-agent:default.log.*",
        "%/pool/ext/*/crypt/debug/oxz_*/oxide-*.log.*",
        "%/pool/ext/*/crypt/debug/oxz_*/system-illumos-*.log.*",
        "!/pool/ext/*/crypt/debug/oxz_propolis-server_*/*.log.*",
    ],
    rust_toolchain: None,
    target: Some(
        "lab-2.0-opte-0.25",
    ),
    access_repos: [],
    publish: [],
    skip_clone: true,
}

Buildomat Job: 01HFJZN2XTQZE3CXAGX8GRQNWP

Tags:

Artefacts:

Output:

SEQ GLOBAL TIME DETAILS
12023-11-19T05:59:48.395Zjob dependencies complete; ready to run (waiting for 51 m 44 s)
22023-11-19T06:00:22.029Zjob assigned to worker 01HFK2KGQFRE042C5BDPT925T2 (queued for 33 s)
32023-11-19T06:00:22.118Zhttp://172.20.0.5:9969/ipxe/buskin... ok
42023-11-19T06:00:22.170Zhttp://172.20.0.5:9969/os/buskin/platform/i86pc/kernel/amd64/unix... ok
52023-11-19T06:00:22.204Zhttp://172.20.0.5:9969/os/buskin/platform/i86pc/amd64/boot_archive... ok
62023-11-19T06:00:22.238Zhttp://172.20.0.5:9969/os/buskin/platform/i86pc/amd64/boot_archive.hash... ok
72023-11-19T06:00:22.271Zhttp://172.20.0.5:9969/postboot/buskin... ok
82023-11-19T06:00:22.323ZwcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U
92023-11-19T06:00:22.367Zcpu0: microcode has been updated from version 0xa001133 to 0xa0011d1
102023-11-19T06:00:22.408ZOxide Helios Version helios-2.0.22241 64-bit
112023-11-19T06:00:22.506ZConfiguring devices.
122023-11-19T06:00:22.544Z##################################################
132023-11-19T06:00:22.584Z#### ############# ##
142023-11-19T06:00:22.618Z### ### ############ ##
152023-11-19T06:00:22.652Z## ### # ## ### ## Oxide Computer Company ##
162023-11-19T06:00:22.685Z## ## ## ### # ### ##
172023-11-19T06:00:22.719Z## # ### #### #### This Station Under ##
182023-11-19T06:00:22.753Z### ### #### # ### Computer Control ##
192023-11-19T06:00:22.806Z#### #### ### ## ##
202023-11-19T06:00:22.843Z##################################################
212023-11-19T06:00:22.915ZHostname: buskin
222023-11-19T06:00:23.104ZLoading smf(7) service descriptions:
232023-11-19T06:00:23.849Zdownloading input: /input/ci-tools/work/end-to-end-tests/bootstrap-2ed5f6fd6904f562.gz
242023-11-19T06:00:25.834Zdownloaded input: /input/ci-tools/work/end-to-end-tests/bootstrap-2ed5f6fd6904f562.gz
252023-11-19T06:00:25.866Zdownloading input: /input/ci-tools/work/end-to-end-tests/bootstrap.gz
262023-11-19T06:00:44.131Zdownloaded input: /input/ci-tools/work/end-to-end-tests/bootstrap.gz
272023-11-19T06:00:44.156Zdownloading input: /input/ci-tools/work/end-to-end-tests/end_to_end_tests-03815e4bc8ad474c.gz
282023-11-19T06:01:14.763Zdownloaded input: /input/ci-tools/work/end-to-end-tests/end_to_end_tests-03815e4bc8ad474c.gz
292023-11-19T06:01:14.807Zdownloading input: /input/ci-tools/work/caboose-util.gz
302023-11-19T06:01:15.769Zdownloaded input: /input/ci-tools/work/caboose-util.gz
312023-11-19T06:01:15.800Zdownloading input: /input/ci-tools/work/tufaceous.gz
322023-11-19T06:01:21.458Zdownloaded input: /input/ci-tools/work/tufaceous.gz
332023-11-19T06:01:21.493Zdownloading input: /input/package/work/version.txt
342023-11-19T06:01:21.532Zdownloaded input: /input/package/work/version.txt
352023-11-19T06:01:21.567Zdownloading input: /input/package/work/package.tar.gz
362023-11-19T06:01:40.715Zdownloaded input: /input/package/work/package.tar.gz
372023-11-19T06:01:41.153Zdownloading input: /input/package/work/global-zone-packages.tar.gz
382023-11-19T06:02:22.543Zdownloaded input: /input/package/work/global-zone-packages.tar.gz
392023-11-19T06:02:22.584Zdownloading input: /input/package/work/trampoline-global-zone-packages.tar.gz
402023-11-19T06:02:32.491Zdownloaded input: /input/package/work/trampoline-global-zone-packages.tar.gz
412023-11-19T06:02:32.531Zdownloading input: /input/package/work/zones/clickhouse.tar.gz
422023-11-19T06:03:33.766Zdownloaded input: /input/package/work/zones/clickhouse.tar.gz
432023-11-19T06:03:33.803Zdownloading input: /input/package/work/zones/clickhouse_keeper.tar.gz
442023-11-19T06:04:25.200Zdownloaded input: /input/package/work/zones/clickhouse_keeper.tar.gz
452023-11-19T06:04:25.233Zdownloading input: /input/package/work/zones/cockroachdb.tar.gz
462023-11-19T06:05:05.474Zdownloaded input: /input/package/work/zones/cockroachdb.tar.gz
472023-11-19T06:05:05.523Zdownloading input: /input/package/work/zones/crucible-pantry.tar.gz
482023-11-19T06:05:10.815Zdownloaded input: /input/package/work/zones/crucible-pantry.tar.gz
492023-11-19T06:05:10.839Zdownloading input: /input/package/work/zones/crucible.tar.gz
502023-11-19T06:05:23.664Zdownloaded input: /input/package/work/zones/crucible.tar.gz
512023-11-19T06:05:23.696Zdownloading input: /input/package/work/zones/external-dns.tar.gz
522023-11-19T06:05:28.434Zdownloaded input: /input/package/work/zones/external-dns.tar.gz
532023-11-19T06:05:28.458Zdownloading input: /input/package/work/zones/internal-dns.tar.gz
542023-11-19T06:05:35.876Zdownloaded input: /input/package/work/zones/internal-dns.tar.gz
552023-11-19T06:05:35.927Zdownloading input: /input/package/work/zones/ntp.tar.gz
562023-11-19T06:05:35.960Zdownloaded input: /input/package/work/zones/ntp.tar.gz
572023-11-19T06:05:35.993Zdownloading input: /input/package/work/zones/omicron-gateway-asic.tar.gz
582023-11-19T06:05:39.410Zdownloaded input: /input/package/work/zones/omicron-gateway-asic.tar.gz
592023-11-19T06:05:39.435Zdownloading input: /input/package/work/zones/omicron-gateway-softnpu.tar.gz
602023-11-19T06:05:44.940Zdownloaded input: /input/package/work/zones/omicron-gateway-softnpu.tar.gz
612023-11-19T06:05:44.994Zdownloading input: /input/package/work/zones/omicron-nexus-single-sled.tar.gz
622023-11-19T06:06:00.292Zdownloaded input: /input/package/work/zones/omicron-nexus-single-sled.tar.gz
632023-11-19T06:06:00.318Zdownloading input: /input/package/work/zones/omicron-nexus.tar.gz
642023-11-19T06:06:20.681Zdownloaded input: /input/package/work/zones/omicron-nexus.tar.gz
652023-11-19T06:06:20.715Zdownloading input: /input/package/work/zones/overlay.tar.gz
662023-11-19T06:06:20.739Zdownloaded input: /input/package/work/zones/overlay.tar.gz
672023-11-19T06:06:20.801Zdownloading input: /input/package/work/zones/oximeter-collector.tar.gz
682023-11-19T06:06:27.422Zdownloaded input: /input/package/work/zones/oximeter-collector.tar.gz
692023-11-19T06:06:27.455Zdownloading input: /input/package/work/zones/propolis-server.tar.gz
702023-11-19T06:06:42.003Zdownloaded input: /input/package/work/zones/propolis-server.tar.gz
712023-11-19T06:06:42.041Zdownloading input: /input/package/work/zones/switch-asic.tar.gz
722023-11-19T06:07:40.081Zdownloaded input: /input/package/work/zones/switch-asic.tar.gz
732023-11-19T06:07:40.175Zdownloading input: /input/package/work/zones/switch-softnpu.tar.gz
742023-11-19T06:08:28.708Zdownloaded input: /input/package/work/zones/switch-softnpu.tar.gz
 
752023-11-19T06:08:28.746Zstarting task 0: "setup"
762023-11-19T06:08:28.791Z++ uname -s
772023-11-19T06:08:28.823Z+ kern=SunOS
782023-11-19T06:08:29.441Z+ case "$kern" in
792023-11-19T06:08:29.476Z+ groupadd -g 12345 build
802023-11-19T06:08:29.509Z+ useradd -u 12345 -g build -d /home/build -s /bin/bash -c build -P 'Primary Administrator' build
812023-11-19T06:08:30.800Z+ zfs create -o mountpoint=/work rpool/work
822023-11-19T06:08:30.833Z++ awk '$2 == "/home" { print $3 }' /etc/mnttab
832023-11-19T06:08:30.865Z+ home_fs=zfs
842023-11-19T06:08:30.896Z+ [[ zfs == autofs ]]
852023-11-19T06:08:30.927Z+ mkdir -p /home/build
862023-11-19T06:08:30.958Z+ chown build:build /home/build /work
872023-11-19T06:08:32.842Z+ chmod 0700 /home/build /work
882023-11-19T06:08:32.873Zprocess exited: duration 4063 ms, exit code 0
 
892023-11-19T06:08:32.941Zstarting task 1: "authentication"
902023-11-19T06:08:33.039Zprocess exited: duration 60 ms, exit code 0
 
912023-11-19T06:08:33.102Zstarting task 2: "build"
922023-11-19T06:08:33.140Z+ trap _exit_trap EXIT
932023-11-19T06:08:33.171Z+ pfexec mdb -kwe 'secpolicy_fs_linkdir/v 55 48 89 e5 b8 01 00 00 00 5d c3'
942023-11-19T06:08:33.341Zsecpolicy_fs_linkdir: 0x55 = 0x55
952023-11-19T06:08:33.378Zsecpolicy_fs_linkdir+1: 0xb8 = 0x48
962023-11-19T06:08:33.413Zsecpolicy_fs_linkdir+2: 0x1 = 0x89
972023-11-19T06:08:33.450Zsecpolicy_fs_linkdir+3: 0 = 0xe5
982023-11-19T06:08:33.500Zsecpolicy_fs_linkdir+4: 0 = 0xb8
992023-11-19T06:08:33.535Zsecpolicy_fs_linkdir+5: 0 = 0x1
1002023-11-19T06:08:33.567Zsecpolicy_fs_linkdir+6: 0x48 = 0x0
1012023-11-19T06:08:33.598Zsecpolicy_fs_linkdir+7: 0x89 = 0x0
1022023-11-19T06:08:33.634Zsecpolicy_fs_linkdir+8: 0xe5 = 0x0
1032023-11-19T06:08:33.671Zsecpolicy_fs_linkdir+9: 0x48 = 0x5d
1042023-11-19T06:08:33.706Zsecpolicy_fs_linkdir+0xa: 0x83 = 0xc3
1052023-11-19T06:08:33.738Z+ [[ -d /opt/oxide ]]
1062023-11-19T06:08:33.774Z+ pfexec mv /opt/oxide /opt/oxide-underneath
1072023-11-19T06:08:33.805Z+ pfexec mkdir /opt/oxide
1082023-11-19T06:08:33.836Z+ pfexec mount -F tmpfs -O swap /opt/oxide
1092023-11-19T06:08:33.869Z+ [[ -d /opt/oxide-underneath ]]
1102023-11-19T06:08:33.968Z+ cd /opt/oxide-underneath
1112023-11-19T06:08:34.000Z+ pfexec tar ceEp@/f - .
1122023-11-19T06:08:34.030Z+ cd /opt/oxide
1132023-11-19T06:08:34.061Z+ pfexec tar xveEp@/f -
1142023-11-19T06:08:34.093Zx ., 0 bytes, 0 tape blocks
1152023-11-19T06:08:34.126Zx ./opte, 0 bytes, 0 tape blocks
1162023-11-19T06:08:34.158Zx ./opte/bin, 0 bytes, 0 tape blocks
1172023-11-19T06:08:34.190Zx ./opte/bin/opteadm, 50714832 bytes, 99053 tape blocks
1182023-11-19T06:08:34.221Z+ pfexec rm -rf /opt/oxide-underneath
1192023-11-19T06:08:34.255Z+ pfexec /sbin/zfs create -o mountpoint=/zone rpool/zone
1202023-11-19T06:08:34.297Z+ pfexec mkdir /opt/oxide/work
1212023-11-19T06:08:34.335Z+ pfexec chown build:build /opt/oxide/work
1222023-11-19T06:08:34.372Z+ cd /opt/oxide/work
1232023-11-19T06:08:34.404Z+ ptime -m tar xvzf /input/package/work/package.tar.gz
1242023-11-19T06:08:35.044ZDecompressing '/input/package/work/package.tar.gz' with '/usr/bin/gzcat'...
1252023-11-19T06:08:35.076Zx out/mg-ddm-gz.tar, 62723072 bytes, 122506 tape blocks
1262023-11-19T06:08:35.107Zx out/omicron-sled-agent.tar, 111619584 bytes, 218007 tape blocks
1272023-11-19T06:08:35.138Zx out/target/test, 75 bytes, 1 tape blocks
1282023-11-19T06:08:35.169Zx out/npuzone/npuzone, 37380856 bytes, 73010 tape blocks
1292023-11-19T06:08:35.200Zx package-manifest.toml, 22143 bytes, 44 tape blocks
1302023-11-19T06:08:35.231Zx smf/sled-agent/non-gimlet/config.toml, 3298 bytes, 7 tape blocks
1312023-11-19T06:08:35.262Zx target/release/omicron-package, 42517744 bytes, 83043 tape blocks
1322023-11-19T06:08:35.295Zx tools/create_virtual_hardware.sh, 3176 bytes, 7 tape blocks
1332023-11-19T06:08:35.327Zx tools/virtual_hardware.sh, 3408 bytes, 7 tape blocks
1342023-11-19T06:08:35.357Zx tools/scrimlet/create-softnpu-zone.sh, 619 bytes, 2 tape blocks
1352023-11-19T06:08:35.388Zx tools/scrimlet/destroy-softnpu-zone.sh, 195 bytes, 1 tape blocks
1362023-11-19T06:08:35.433Zx tools/scrimlet/softnpu-init.sh, 2148 bytes, 5 tape blocks
1372023-11-19T06:08:35.471Zx tools/scrimlet/softnpu-zone.txt, 330 bytes, 1 tape blocks
1382023-11-19T06:08:35.506Zx tools/scrimlet/softnpu.toml, 171 bytes, 1 tape blocks
1392023-11-19T06:08:35.541Z
1402023-11-19T06:08:35.573Zreal 1.539315803
1412023-11-19T06:08:35.604Zuser 1.386296855
1422023-11-19T06:08:35.635Zsys 0.269020781
1432023-11-19T06:08:35.666Ztrap 0.000063450
1442023-11-19T06:08:35.697Ztflt 0.000000000
1452023-11-19T06:08:35.728Zdflt 0.000000000
1462023-11-19T06:08:35.759Zkflt 0.000000000
1472023-11-19T06:08:35.790Zlock 0.000000000
1482023-11-19T06:08:35.820Zslp 1.387738198
1492023-11-19T06:08:35.851Zlat 0.033430264
1502023-11-19T06:08:35.894Zstop 0.000120922
1512023-11-19T06:08:35.927Z+ cp /input/package/work/zones/clickhouse_keeper.tar.gz /input/package/work/zones/clickhouse.tar.gz /input/package/work/zones/cockroachdb.tar.gz /input/package/work/zones/crucible-pantry.tar.gz /input/package/work/zones/crucible.tar.gz /input/package/work/zones/external-dns.tar.gz /input/package/work/zones/internal-dns.tar.gz /input/package/work/zones/ntp.tar.gz /input/package/work/zones/omicron-gateway-asic.tar.gz /input/package/work/zones/omicron-gateway-softnpu.tar.gz /input/package/work/zones/omicron-nexus-single-sled.tar.gz /input/package/work/zones/omicron-nexus.tar.gz /input/package/work/zones/overlay.tar.gz /input/package/work/zones/oximeter-collector.tar.gz /input/package/work/zones/propolis-server.tar.gz /input/package/work/zones/switch-asic.tar.gz /input/package/work/zones/switch-softnpu.tar.gz out/
1522023-11-19T06:08:36.148Z+ mv out/omicron-nexus-single-sled.tar.gz out/omicron-nexus.tar.gz
1532023-11-19T06:08:36.183Z+ mkdir tests
1542023-11-19T06:08:36.219Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1552023-11-19T06:08:36.251Z+ ptime -m gunzip
1562023-11-19T06:08:36.289Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap-2ed5f6fd6904f562
1572023-11-19T06:08:36.321Z
1582023-11-19T06:08:36.361Zreal 0.138411730
1592023-11-19T06:08:36.398Zuser 0.128323943
1602023-11-19T06:08:36.432Zsys 0.009848990
1612023-11-19T06:08:36.466Ztrap 0.000013240
1622023-11-19T06:08:36.502Ztflt 0.000000000
1632023-11-19T06:08:36.570Zdflt 0.000000000
1642023-11-19T06:08:36.614Zkflt 0.000000000
1652023-11-19T06:08:36.653Zlock 0.000000000
1662023-11-19T06:08:36.692Zslp 0.000019640
1672023-11-19T06:08:36.732Zlat 0.000017380
1682023-11-19T06:08:36.772Zstop 0.000123382
1692023-11-19T06:08:36.820Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap-2ed5f6fd6904f562
1702023-11-19T06:08:36.855Z+ chmod a+x tests/bootstrap-2ed5f6fd6904f562
1712023-11-19T06:08:36.889Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1722023-11-19T06:08:36.927Z+ ptime -m gunzip
1732023-11-19T06:08:36.968Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap
1742023-11-19T06:08:38.336Z
1752023-11-19T06:08:38.454Zreal 2.013751040
1762023-11-19T06:08:38.489Zuser 1.884348444
1772023-11-19T06:08:38.522Zsys 0.128988360
1782023-11-19T06:08:38.555Ztrap 0.000188181
1792023-11-19T06:08:38.590Ztflt 0.000000000
1802023-11-19T06:08:38.630Zdflt 0.000000000
1812023-11-19T06:08:38.738Zkflt 0.000000000
1822023-11-19T06:08:38.773Zlock 0.000000000
1832023-11-19T06:08:38.814Zslp 0.000013670
1842023-11-19T06:08:38.847Zlat 0.000023569
1852023-11-19T06:08:38.879Zstop 0.000127093
1862023-11-19T06:08:38.914Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap
1872023-11-19T06:08:38.988Z+ chmod a+x tests/bootstrap
1882023-11-19T06:08:39.442Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1892023-11-19T06:08:39.491Z+ ptime -m gunzip
1902023-11-19T06:08:39.526Z++ basename /input/ci-tools/work/end-to-end-tests/end_to_end_tests-03815e4bc8ad474c
1912023-11-19T06:08:40.629Z
1922023-11-19T06:08:40.669Zreal 2.282405256
1932023-11-19T06:08:40.723Zuser 2.123954330
1942023-11-19T06:08:40.767Zsys 0.158022168
1952023-11-19T06:08:40.872Ztrap 0.000183811
1962023-11-19T06:08:40.908Ztflt 0.000000000
1972023-11-19T06:08:41.361Zdflt 0.000000000
1982023-11-19T06:08:41.399Zkflt 0.000000000
1992023-11-19T06:08:41.440Zlock 0.000000000
2002023-11-19T06:08:41.474Zslp 0.000023430
2012023-11-19T06:08:41.507Zlat 0.000023671
2022023-11-19T06:08:41.541Zstop 0.000129382
2032023-11-19T06:08:41.576Z++ basename /input/ci-tools/work/end-to-end-tests/end_to_end_tests-03815e4bc8ad474c
2042023-11-19T06:08:41.609Z+ chmod a+x tests/end_to_end_tests-03815e4bc8ad474c
2052023-11-19T06:08:41.642Z+ bmat address ls
2062023-11-19T06:08:41.677ZNAME CIDR FIRST LAST COUNT
2072023-11-19T06:08:41.715Zextra 10.151.1.0/24 10.151.1.164 10.151.1.227 64
2082023-11-19T06:08:41.751Z++ bmat address ls -f extra -Ho first,last,count
2092023-11-19T06:08:41.783Z+ set -- 10.151.1.164 10.151.1.227 64
2102023-11-19T06:08:41.818Z+ EXTRA_IP_START=10.151.1.164
2112023-11-19T06:08:41.851Z+ EXTRA_IP_END=10.151.1.227
2122023-11-19T06:08:41.904Z+ EXTRA_IP_COUNT=64
2132023-11-19T06:08:41.936Z+ (( EXTRA_IP_COUNT >= 32 ))
2142023-11-19T06:08:41.971Z+ EXTRA_IP_BASE=10.151.1
2152023-11-19T06:08:42.019Z+ EXTRA_IP_FOCTET=164
2162023-11-19T06:08:42.053Z+ EXTRA_IP_LOCTET=227
2172023-11-19T06:08:42.091Z+ SERVICE_IP_POOL_START=10.151.1.164
2182023-11-19T06:08:42.132Z+ SERVICE_IP_POOL_END=10.151.1.173
2192023-11-19T06:08:42.176Z+ DNS_IP1=10.151.1.164
2202023-11-19T06:08:42.219Z+ DNS_IP2=10.151.1.165
2212023-11-19T06:08:42.662Z+ UPLINK_IP=10.151.1.174
2222023-11-19T06:08:42.693Z+ IPPOOL_START=10.151.1.175
2232023-11-19T06:08:42.743Z+ IPPOOL_END=10.151.1.227
2242023-11-19T06:08:42.781Z++ ipadm show-addr -po type,addr
2252023-11-19T06:08:42.822Z++ awk '-F[:/]' '$1 == "dhcp" {print $2}'
2262023-11-19T06:08:43.268Z+ GATEWAY_IP=10.151.1.100
2272023-11-19T06:08:43.299Z+ [[ -n 10.151.1.100 ]]
2282023-11-19T06:08:43.334Z+ ping -s 10.151.1.100 56 1
2292023-11-19T06:08:43.368ZPING 10.151.1.100: 56 data bytes
2302023-11-19T06:08:43.399Z64 bytes from 10.151.1.100: icmp_seq=0. time=0.012 ms
2312023-11-19T06:08:43.434Z
2322023-11-19T06:08:43.475Z----10.151.1.100 PING Statistics----
2332023-11-19T06:08:43.509Z1 packets transmitted, 1 packets received, 0% packet loss
2342023-11-19T06:08:43.546Zround-trip (ms) min/avg/max/stddev = 0.012/0.012/0.012/-nan
2352023-11-19T06:08:43.583Z++ arp -an
2362023-11-19T06:08:43.633Z++ awk -vgw=10.151.1.100 '$2 == gw {print $NF}'
2372023-11-19T06:08:43.670Z+ GATEWAY_MAC=18:c0:4d:0d:9f:b2
2382023-11-19T06:08:43.707Z+ routeadm -e ipv4-forwarding -u
2392023-11-19T06:08:43.742Z+ PXA_START=10.151.1.164
2402023-11-19T06:08:43.777Z+ PXA_END=10.151.1.227
2412023-11-19T06:08:43.817Z+ export GATEWAY_IP GATEWAY_MAC PXA_START PXA_END
2422023-11-19T06:08:43.861Z+ pfexec zpool create -f scratch c1t1d0 c2t1d0
2432023-11-19T06:08:44.383Z+ ZPOOL_VDEV_DIR=/scratch
2442023-11-19T06:08:44.421Z+ ptime -m pfexec ./tools/create_virtual_hardware.sh
2452023-11-19T06:08:44.457Z+++ dirname ./tools/create_virtual_hardware.sh
2462023-11-19T06:08:44.493Z++ cd ./tools
2472023-11-19T06:08:44.535Z++ pwd
2482023-11-19T06:08:44.596Z+ SOURCE_DIR=/opt/oxide/work/tools
2492023-11-19T06:08:44.658Z+ OMICRON_TOP=/opt/oxide/work/tools/..
2502023-11-19T06:08:44.703Z+ SOFTNPU_MODE=zone
2512023-11-19T06:08:44.745Z+ . /opt/oxide/work/tools/virtual_hardware.sh
2522023-11-19T06:08:44.786Z++ MARKER=/etc/opt/oxide/NO_INSTALL
2532023-11-19T06:08:45.279Z++ [[ -f /etc/opt/oxide/NO_INSTALL ]]
2542023-11-19T06:08:45.314Z++ dladm show-phys -p -o LINK
2552023-11-19T06:08:45.353Z++ head -1
2562023-11-19T06:08:45.389Z+ PHYSICAL_LINK=igb0
2572023-11-19T06:08:45.429Z+ echo 'Using igb0 as physical link'
2582023-11-19T06:08:45.470ZUsing igb0 as physical link
2592023-11-19T06:08:45.918Z+ ensure_run_as_root
2602023-11-19T06:08:45.968Z++ id -u
2612023-11-19T06:08:46.414Z+ [[ 0 -ne 0 ]]
2622023-11-19T06:08:46.452Z+ ensure_zpools
2632023-11-19T06:08:46.491ZZpool: [oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b]
2642023-11-19T06:08:46.532Z+ ZPOOL_TYPES=('oxp_' 'oxi_')
2652023-11-19T06:08:46.567ZZFS vdev /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
2662023-11-19T06:08:46.610Z+ for ZPOOL_TYPE in "${ZPOOL_TYPES[@]}"
2672023-11-19T06:08:46.655ZZFS zpool oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b exists
2682023-11-19T06:08:46.704Z+ readarray -t ZPOOLS
2692023-11-19T06:08:46.751ZZpool: [oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03]
2702023-11-19T06:08:46.785Z++ grep '"oxp_' /opt/oxide/work/tools/../smf/sled-agent/non-gimlet/config.toml
2712023-11-19T06:08:46.833ZZFS vdev /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
2722023-11-19T06:08:46.876Z++ sed 's/[ ",]//g'
2732023-11-19T06:08:46.915ZZFS zpool oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
2742023-11-19T06:08:46.956Z+ for ZPOOL in "${ZPOOLS[@]}"
2752023-11-19T06:08:47.007ZZpool: [oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03]
2762023-11-19T06:08:47.047Z+ echo 'Zpool: [oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b]'
2772023-11-19T06:08:47.087ZZFS vdev /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
2782023-11-19T06:08:47.127Z+ VDEV_PATH=/scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
2792023-11-19T06:08:47.222ZZFS zpool oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
2802023-11-19T06:08:47.269Z+ [[ -f /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
2812023-11-19T06:08:47.312ZZpool: [oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03]
2822023-11-19T06:08:47.780Z+ dd if=/dev/zero of=/scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
2832023-11-19T06:08:48.290ZZFS vdev /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
2842023-11-19T06:08:48.328Z0+0 records in
2852023-11-19T06:08:48.365ZZFS zpool oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
2862023-11-19T06:08:48.408Z0+0 records out
2872023-11-19T06:08:48.448ZZpool: [oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03]
2882023-11-19T06:08:48.488Z0 bytes (0 iB) transferred in 0.000005 secs (0 iB/sec)
2892023-11-19T06:08:48.528ZZFS vdev /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
2902023-11-19T06:08:48.563Z+ success 'ZFS vdev /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
2912023-11-19T06:08:48.610ZZFS zpool oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
2922023-11-19T06:08:48.661Z+ set +x
2932023-11-19T06:08:48.706ZZpool: [oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930]
2942023-11-19T06:08:49.158Z++ zpool list -o name
2952023-11-19T06:08:49.360ZZFS vdev /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev exists
2962023-11-19T06:08:49.428Z++ grep oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b
2972023-11-19T06:08:49.476ZZFS zpool oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 exists
2982023-11-19T06:08:49.920Z+ [[ -z '' ]]
2992023-11-19T06:08:49.957ZZpool: [oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60]
3002023-11-19T06:08:49.991Z+ zpool create -f oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
3012023-11-19T06:08:50.026ZZFS vdev /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev exists
3022023-11-19T06:08:50.066Z+ success 'ZFS zpool oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
3032023-11-19T06:08:50.513ZZFS zpool oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 exists
3042023-11-19T06:08:50.568Z+ set +x
3052023-11-19T06:08:50.610ZZpool: [oxp_31bd71cd-4736-4a12-a387-9b74b050396f]
3062023-11-19T06:08:50.643Z+ for ZPOOL in "${ZPOOLS[@]}"
3072023-11-19T06:08:50.679ZZFS vdev /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev exists
3082023-11-19T06:08:50.715Z+ echo 'Zpool: [oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3092023-11-19T06:08:50.747ZZFS zpool oxp_31bd71cd-4736-4a12-a387-9b74b050396f exists
3102023-11-19T06:08:50.780Z+ VDEV_PATH=/scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3112023-11-19T06:08:50.813ZZpool: [oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7]
3122023-11-19T06:08:50.848Z+ [[ -f /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3132023-11-19T06:08:50.884ZZFS vdev /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev exists
3142023-11-19T06:08:50.931Z+ dd if=/dev/zero of=/scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3152023-11-19T06:08:50.978ZZFS zpool oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 exists
3162023-11-19T06:08:51.020Z0+0 records in
3172023-11-19T06:08:51.055ZZpool: [oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b]
3182023-11-19T06:08:51.093Z0+0 records out
3192023-11-19T06:08:51.125ZZFS vdev /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
3202023-11-19T06:08:51.158Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3212023-11-19T06:08:51.190ZZFS zpool oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b exists
3222023-11-19T06:08:51.227Z+ success 'ZFS vdev /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3232023-11-19T06:08:51.259ZZpool: [oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b]
3242023-11-19T06:08:51.291Z+ set +x
3252023-11-19T06:08:51.324ZZFS vdev /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
3262023-11-19T06:08:51.359Z++ zpool list -o name
3272023-11-19T06:08:51.403ZZFS zpool oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b exists
3282023-11-19T06:08:51.469Z++ grep oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03
3292023-11-19T06:08:51.504ZSimnet net0/sc0_0 exists
3302023-11-19T06:08:51.536Z+ [[ -z '' ]]
3312023-11-19T06:08:51.567ZSimnet net1/sc1_0 exists
3322023-11-19T06:08:51.598Z+ zpool create -f oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3332023-11-19T06:08:51.630ZVnic sc0_1 exists
3342023-11-19T06:08:51.661Z+ success 'ZFS zpool oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3352023-11-19T06:08:51.693Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/softnpu/image/dec63e67156fe6e958991bbfa090629868115ab5/softnpu.sha256.txt
3362023-11-19T06:08:51.727Z+ set +x
3372023-11-19T06:08:52.167Z+ for ZPOOL in "${ZPOOLS[@]}"
3382023-11-19T06:08:52.207Z+ echo 'Zpool: [oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3392023-11-19T06:08:52.246Z+ VDEV_PATH=/scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3402023-11-19T06:08:52.292Z+ [[ -f /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3412023-11-19T06:08:52.339Z+ dd if=/dev/zero of=/scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3422023-11-19T06:08:52.784Z0+0 records in
3432023-11-19T06:08:52.827Z0+0 records out
3442023-11-19T06:08:52.860Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
3452023-11-19T06:08:52.896Z+ success 'ZFS vdev /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3462023-11-19T06:08:52.931Z+ set +x
3472023-11-19T06:08:52.970Z++ zpool list -o name
3482023-11-19T06:08:53.005Z++ grep oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03
3492023-11-19T06:08:53.044Z+ [[ -z '' ]]
3502023-11-19T06:08:53.078Z+ zpool create -f oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3512023-11-19T06:08:53.110Z+ success 'ZFS zpool oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3522023-11-19T06:08:53.144Z+ set +x
3532023-11-19T06:08:53.178Z+ for ZPOOL in "${ZPOOLS[@]}"
3542023-11-19T06:08:53.224Z+ echo 'Zpool: [oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3552023-11-19T06:08:53.270Z+ VDEV_PATH=/scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3562023-11-19T06:08:53.301Z+ [[ -f /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3572023-11-19T06:08:53.332Z+ dd if=/dev/zero of=/scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3582023-11-19T06:08:53.363Z0+0 records in
3592023-11-19T06:08:53.396Z0+0 records out
3602023-11-19T06:08:53.430Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
3612023-11-19T06:08:53.596Z+ success 'ZFS vdev /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3622023-11-19T06:08:53.645Z+ set +x
3632023-11-19T06:08:54.304Z++ zpool list -o name
3642023-11-19T06:08:54.355Z++ grep oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03
3652023-11-19T06:08:54.397Z+ [[ -z '' ]]
3662023-11-19T06:08:54.437Z+ zpool create -f oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3672023-11-19T06:08:54.477Z+ success 'ZFS zpool oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3682023-11-19T06:08:54.523Z+ set +x
3692023-11-19T06:08:54.560Z+ for ZPOOL in "${ZPOOLS[@]}"
3702023-11-19T06:08:54.662Z+ echo 'Zpool: [oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3712023-11-19T06:08:54.702Z+ VDEV_PATH=/scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3722023-11-19T06:08:54.742Z+ [[ -f /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3732023-11-19T06:08:54.779Z+ dd if=/dev/zero of=/scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3742023-11-19T06:08:54.830Z0+0 records in
3752023-11-19T06:08:54.874Z0+0 records out
3762023-11-19T06:08:54.915Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
3772023-11-19T06:08:54.948Z+ success 'ZFS vdev /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3782023-11-19T06:08:54.992Z+ set +x
3792023-11-19T06:08:55.028Z++ zpool list -o name
3802023-11-19T06:08:55.063Z++ grep oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03
3812023-11-19T06:08:55.100Z+ [[ -z '' ]]
3822023-11-19T06:08:55.141Z+ zpool create -f oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3832023-11-19T06:08:55.173Z+ success 'ZFS zpool oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3842023-11-19T06:08:55.216Z+ set +x
3852023-11-19T06:08:55.249Z+ for ZPOOL in "${ZPOOLS[@]}"
3862023-11-19T06:08:55.283Z+ echo 'Zpool: [oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930]'
3872023-11-19T06:08:55.730Z+ VDEV_PATH=/scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev
3882023-11-19T06:08:55.766Z+ [[ -f /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev ]]
3892023-11-19T06:08:55.798Z+ dd if=/dev/zero of=/scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev bs=1 count=0 seek=20G
3902023-11-19T06:08:55.837Z0+0 records in
3912023-11-19T06:08:55.873Z0+0 records out
3922023-11-19T06:08:55.916Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3932023-11-19T06:08:56.365Z+ success 'ZFS vdev /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev exists'
3942023-11-19T06:08:56.402Z+ set +x
3952023-11-19T06:08:56.440Z++ zpool list -o name
3962023-11-19T06:08:56.485Z++ grep oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930
3972023-11-19T06:08:56.528Z+ [[ -z '' ]]
3982023-11-19T06:08:56.567Z+ zpool create -f oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev
3992023-11-19T06:08:56.602Z+ success 'ZFS zpool oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 exists'
4002023-11-19T06:08:56.636Z+ set +x
4012023-11-19T06:08:56.668Z+ for ZPOOL in "${ZPOOLS[@]}"
4022023-11-19T06:08:56.702Z+ echo 'Zpool: [oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60]'
4032023-11-19T06:08:56.766Z+ VDEV_PATH=/scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev
4042023-11-19T06:08:56.808Z+ [[ -f /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev ]]
4052023-11-19T06:08:56.848Z+ dd if=/dev/zero of=/scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev bs=1 count=0 seek=20G
4062023-11-19T06:08:56.882Z0+0 records in
4072023-11-19T06:08:56.917Z0+0 records out
4082023-11-19T06:08:56.948Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4092023-11-19T06:08:56.979Z+ success 'ZFS vdev /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev exists'
4102023-11-19T06:08:57.020Z+ set +x
4112023-11-19T06:08:57.056Z++ zpool list -o name
4122023-11-19T06:08:57.086Z++ grep oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60
4132023-11-19T06:08:57.117Z+ [[ -z '' ]]
4142023-11-19T06:08:57.148Z+ zpool create -f oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev
4152023-11-19T06:08:57.179Z+ success 'ZFS zpool oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 exists'
4162023-11-19T06:08:57.210Z+ set +x
4172023-11-19T06:08:57.253Z+ for ZPOOL in "${ZPOOLS[@]}"
4182023-11-19T06:08:57.293Z+ echo 'Zpool: [oxp_31bd71cd-4736-4a12-a387-9b74b050396f]'
4192023-11-19T06:08:57.331Z+ VDEV_PATH=/scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev
4202023-11-19T06:08:57.362Zdownloading softnpu ...done
4212023-11-19T06:08:57.393Z+ [[ -f /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev ]]
4222023-11-19T06:08:57.424Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/sidecar-lite/release/f0585a29fb0285f7a1220c1118856b0e5c1f75c5/libsidecar_lite.so.sha256.txt
4232023-11-19T06:08:57.474Z+ dd if=/dev/zero of=/scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev bs=1 count=0 seek=20G
4242023-11-19T06:08:57.510Z0+0 records in
4252023-11-19T06:08:57.543Z0+0 records out
4262023-11-19T06:08:57.575Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
4272023-11-19T06:08:57.608Z+ success 'ZFS vdev /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev exists'
4282023-11-19T06:08:57.639Z+ set +x
4292023-11-19T06:08:57.679Z++ zpool list -o name
4302023-11-19T06:08:57.717Z++ grep oxp_31bd71cd-4736-4a12-a387-9b74b050396f
4312023-11-19T06:08:57.750Z+ [[ -z '' ]]
4322023-11-19T06:08:57.792Z+ zpool create -f oxp_31bd71cd-4736-4a12-a387-9b74b050396f /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev
4332023-11-19T06:08:57.823Z+ success 'ZFS zpool oxp_31bd71cd-4736-4a12-a387-9b74b050396f exists'
4342023-11-19T06:08:57.855Z+ set +x
4352023-11-19T06:08:57.893Z+ for ZPOOL in "${ZPOOLS[@]}"
4362023-11-19T06:08:58.380Z+ echo 'Zpool: [oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7]'
4372023-11-19T06:08:58.419Z+ VDEV_PATH=/scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev
4382023-11-19T06:08:58.457Z+ [[ -f /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev ]]
4392023-11-19T06:08:58.900Z+ dd if=/dev/zero of=/scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev bs=1 count=0 seek=20G
4402023-11-19T06:08:58.931Z0+0 records in
4412023-11-19T06:08:59.094Z0+0 records out
4422023-11-19T06:08:59.134Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
4432023-11-19T06:08:59.188Z+ success 'ZFS vdev /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev exists'
4442023-11-19T06:08:59.230Z+ set +x
4452023-11-19T06:08:59.265Z++ zpool list -o name
4462023-11-19T06:08:59.760Z++ grep oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7
4472023-11-19T06:08:59.793Z+ [[ -z '' ]]
4482023-11-19T06:08:59.873Z+ zpool create -f oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev
4492023-11-19T06:08:59.905Z+ success 'ZFS zpool oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 exists'
4502023-11-19T06:09:00.347Z+ set +x
4512023-11-19T06:09:00.380Z+ for ZPOOL_TYPE in "${ZPOOL_TYPES[@]}"
4522023-11-19T06:09:00.417Z+ readarray -t ZPOOLS
4532023-11-19T06:09:00.454Z++ grep '"oxi_' /opt/oxide/work/tools/../smf/sled-agent/non-gimlet/config.toml
4542023-11-19T06:09:00.487Z++ sed 's/[ ",]//g'
4552023-11-19T06:09:00.520Z+ for ZPOOL in "${ZPOOLS[@]}"
4562023-11-19T06:09:00.554Z+ echo 'Zpool: [oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b]'
4572023-11-19T06:09:00.588Z+ VDEV_PATH=/scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4582023-11-19T06:09:00.625Z+ [[ -f /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
4592023-11-19T06:09:00.658Z+ dd if=/dev/zero of=/scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
4602023-11-19T06:09:00.689Z0+0 records in
4612023-11-19T06:09:00.721Z0+0 records out
4622023-11-19T06:09:00.753Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4632023-11-19T06:09:00.785Z+ success 'ZFS vdev /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
4642023-11-19T06:09:00.818Z+ set +x
4652023-11-19T06:09:00.856Z++ zpool list -o name
4662023-11-19T06:09:00.899Z++ grep oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b
4672023-11-19T06:09:00.930Z+ [[ -z '' ]]
4682023-11-19T06:09:00.961Z+ zpool create -f oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4692023-11-19T06:09:00.992Z+ success 'ZFS zpool oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
4702023-11-19T06:09:01.024Z+ set +x
4712023-11-19T06:09:01.055Z+ for ZPOOL in "${ZPOOLS[@]}"
4722023-11-19T06:09:01.089Z+ echo 'Zpool: [oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b]'
4732023-11-19T06:09:01.126Z+ VDEV_PATH=/scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4742023-11-19T06:09:01.174Z+ [[ -f /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
4752023-11-19T06:09:01.208Z+ dd if=/dev/zero of=/scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
4762023-11-19T06:09:01.239Z0+0 records in
4772023-11-19T06:09:01.272Z0+0 records out
4782023-11-19T06:09:01.314Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
4792023-11-19T06:09:01.349Z+ success 'ZFS vdev /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
4802023-11-19T06:09:01.384Z+ set +x
4812023-11-19T06:09:01.422Z++ zpool list -o name
4822023-11-19T06:09:01.458Z++ grep oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b
4832023-11-19T06:09:01.498Z+ [[ -z '' ]]
4842023-11-19T06:09:01.533Z+ zpool create -f oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4852023-11-19T06:09:01.571Z+ success 'ZFS zpool oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
4862023-11-19T06:09:01.613Z+ set +x
4872023-11-19T06:09:01.651Z+ [[ zone == \z\o\n\e ]]
4882023-11-19T06:09:01.689Z+ ensure_simulated_links igb0
4892023-11-19T06:09:01.732Z+ local PHYSICAL_LINK=igb0
4902023-11-19T06:09:02.186Z+ INDICES=("0" "1")
4912023-11-19T06:09:02.225Z+ for I in "${INDICES[@]}"
4922023-11-19T06:09:02.666Z++ get_simnet_name_if_exists net0
4932023-11-19T06:09:03.116Z++ dladm show-simnet -p -o LINK net0
4942023-11-19T06:09:03.152Z++ echo ''
4952023-11-19T06:09:03.195Z+ [[ -z '' ]]
4962023-11-19T06:09:03.251Z+ dladm create-simnet -t net0
4972023-11-19T06:09:03.293Z+ dladm create-simnet -t sc0_0
4982023-11-19T06:09:03.327Z+ dladm modify-simnet -t -p net0 sc0_0
4992023-11-19T06:09:03.358Z+ dladm set-linkprop -p mtu=9000 sc0_0
5002023-11-19T06:09:03.390Z+ success 'Simnet net0/sc0_0 exists'
5012023-11-19T06:09:03.422Z+ set +x
5022023-11-19T06:09:03.458Z+ for I in "${INDICES[@]}"
5032023-11-19T06:09:03.490Z++ get_simnet_name_if_exists net1
5042023-11-19T06:09:03.522Z++ dladm show-simnet -p -o LINK net1
5052023-11-19T06:09:03.553Z++ echo ''
5062023-11-19T06:09:03.587Z+ [[ -z '' ]]
5072023-11-19T06:09:03.620Z+ dladm create-simnet -t net1
5082023-11-19T06:09:03.653Z+ dladm create-simnet -t sc1_0
5092023-11-19T06:09:03.690Z+ dladm modify-simnet -t -p net1 sc1_0
5102023-11-19T06:09:03.744Z+ dladm set-linkprop -p mtu=9000 sc1_0
5112023-11-19T06:09:04.209Z+ success 'Simnet net1/sc1_0 exists'
5122023-11-19T06:09:04.249Z+ set +x
5132023-11-19T06:09:04.304Z++ get_vnic_name_if_exists sc0_1
5142023-11-19T06:09:04.424Z++ dladm show-vnic -p -o LINK sc0_1
5152023-11-19T06:09:04.458Z++ echo ''
5162023-11-19T06:09:04.491Z+ [[ -z '' ]]
5172023-11-19T06:09:04.526Z+ dladm create-vnic -t sc0_1 -l igb0 -m a8:e1:de:01:70:1d
5182023-11-19T06:09:04.558Z+ [[ -v PROMISC_FILT_OFF ]]
5192023-11-19T06:09:04.592Z+ success 'Vnic sc0_1 exists'
5202023-11-19T06:09:04.625Z+ set +x
5212023-11-19T06:09:04.656Z+ warn_if_no_proxy_arp
5222023-11-19T06:09:04.688Z+ [[ -v PXA_START ]]
5232023-11-19T06:09:04.719Z+ [[ -v PXA_END ]]
5242023-11-19T06:09:04.757Z+ ensure_softnpu_zone
5252023-11-19T06:09:04.793Z+ zoneadm list
5262023-11-19T06:09:04.824Z+ grep -q sidecar_softnpu
5272023-11-19T06:09:04.856Z+ [[ -f out/npuzone/npuzone ]]
5282023-11-19T06:09:04.889Z+ out/npuzone/npuzone create sidecar --omicron-zone --ports sc0_0,tfportrear0_0 --ports sc0_1,tfportqsfp0_0 --sidecar-lite-commit f0585a29fb0285f7a1220c1118856b0e5c1f75c5 --softnpu-commit dec63e67156fe6e958991bbfa090629868115ab5
5292023-11-19T06:09:04.936Zdownloading asic_program.so ...done
5302023-11-19T06:09:04.989Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/sidecar-lite/release/f0585a29fb0285f7a1220c1118856b0e5c1f75c5/scadm.sha256.txt
5312023-11-19T06:09:12.886Zdownloading scadm ...done
5322023-11-19T06:09:12.924Zcreating port tfportrear0_0 for sc0_0
5332023-11-19T06:09:12.967Zcreating port tfportqsfp0_0 for sc0_1
5342023-11-19T06:09:13.415Zsoftnpu zone setup
5352023-11-19T06:09:13.465Zconfigure zone
5362023-11-19T06:09:13.504Zinstall zone
5372023-11-19T06:09:13.548Zboot zone
5382023-11-19T06:09:13.590Z[sidecar_softnpu] svcs milestone/network
5392023-11-19T06:09:13.629ZSTATE STIME FMRI
5402023-11-19T06:09:13.668Zonline 6:09:11 svc:/milestone/network:default
5412023-11-19T06:09:13.703Z[sidecar_softnpu] /softnpu/npu start
5422023-11-19T06:09:13.759Z
5432023-11-19T06:09:13.814Z+ /opt/oxide/work/tools/scrimlet/softnpu-init.sh
5442023-11-19T06:09:13.858Z+ GATEWAY_IP=10.151.1.100
5452023-11-19T06:09:13.892Z+ echo 'Using 10.151.1.100 as gateway ip'
5462023-11-19T06:09:13.927ZUsing 10.151.1.100 as gateway ip
5472023-11-19T06:09:13.964Z+ [[ ! -v GATEWAY_MAC ]]
5482023-11-19T06:09:14.096Z+ GATEWAY_MAC=18:c0:4d:0d:9f:b2
5492023-11-19T06:09:14.142Z++ grep -c -E '^([0-9a-fA-F]{2}:){5}[0-9a-fA-F]{2}$' /dev/fd/63
5502023-11-19T06:09:14.176Z+++ echo 18:c0:4d:0d:9f:b2
5512023-11-19T06:09:14.220Z+ COUNT=1
5522023-11-19T06:09:14.270Z+ [[ 1 -ne 1 ]]
5532023-11-19T06:09:14.303Z+ echo 'Using 18:c0:4d:0d:9f:b2 as gateway mac'
5542023-11-19T06:09:14.334ZUsing 18:c0:4d:0d:9f:b2 as gateway mac
5552023-11-19T06:09:14.372Z+ z_scadm add-arp-entry 10.151.1.100 18:c0:4d:0d:9f:b2
5562023-11-19T06:09:14.409Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone add-arp-entry 10.151.1.100 18:c0:4d:0d:9f:b2
5572023-11-19T06:09:14.443Z+ PXA_MAC=a8:e1:de:01:70:1d
5582023-11-19T06:09:14.477Z+ [[ -v PXA_START ]]
5592023-11-19T06:09:14.511Z+ z_scadm add-proxy-arp 10.151.1.164 10.151.1.227 a8:e1:de:01:70:1d
5602023-11-19T06:09:14.545Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone add-proxy-arp 10.151.1.164 10.151.1.227 a8:e1:de:01:70:1d
5612023-11-19T06:09:14.578Z+ z_scadm dump-state
5622023-11-19T06:09:14.613Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone dump-state
5632023-11-19T06:09:14.649Zlocal v6:
5642023-11-19T06:09:14.681Zlocal v4:
5652023-11-19T06:09:14.714Zrouter v6:
5662023-11-19T06:09:14.748Zrouter v4_idx:
5672023-11-19T06:09:14.780Zrouter v4_routes:
5682023-11-19T06:09:15.219Zresolver v4:
5692023-11-19T06:09:15.265Z10.151.1.100 -> 18:c0:4d:0d:9f:b2
5702023-11-19T06:09:15.303Zresolver v6:
5712023-11-19T06:09:15.334Znat v4:
5722023-11-19T06:09:15.366Znat_v6:
5732023-11-19T06:09:15.397Zport_mac:
5742023-11-19T06:09:15.428Zproxy arp:
5752023-11-19T06:09:15.459Z10.151.1.164/10.151.1.227: a8:e1:de:01:70:1d
5762023-11-19T06:09:15.490Z+ success 'softnpu zone exists'
5772023-11-19T06:09:15.524Z+ set +x
5782023-11-19T06:09:15.571Zsoftnpu zone exists
5792023-11-19T06:09:15.605Z
5802023-11-19T06:09:15.638Zreal 32.432839911
5812023-11-19T06:09:15.677Zuser 0.699301692
5822023-11-19T06:09:15.717Zsys 1.611097537
5832023-11-19T06:09:15.750Ztrap 0.000080407
5842023-11-19T06:09:15.781Ztflt 0.000127797
5852023-11-19T06:09:15.813Zdflt 0.000916554
5862023-11-19T06:09:15.843Zkflt 0.000000000
5872023-11-19T06:09:15.874Zlock 25:16.864688061
5882023-11-19T06:09:15.906Zslp 1:37.211803505
5892023-11-19T06:09:15.937Zlat 0.074085808
5902023-11-19T06:09:15.969Zstop 0.356005570
5912023-11-19T06:09:16.007Z+ tar xf out/omicron-sled-agent.tar pkg/config-rss.toml
5922023-11-19T06:09:16.042Z++ sed -n 's/silo_name = "\(.*\)"/\1/p' pkg/config-rss.toml
5932023-11-19T06:09:16.076Z+ SILO_NAME=recovery
5942023-11-19T06:09:16.110Z++ sed -n 's/external_dns_zone_name = "\(.*\)"/\1/p' pkg/config-rss.toml
5952023-11-19T06:09:16.144Z+ EXTERNAL_DNS_DOMAIN=oxide.test
5962023-11-19T06:09:16.621Z+ sed -i~ '
5972023-11-19T06:09:16.655Z /^external_dns_ips/c\
5982023-11-19T06:09:16.690Zexternal_dns_ips = [ "10.151.1.164", "10.151.1.165" ]
5992023-11-19T06:09:16.730Z /^\[\[internal_services_ip_pool_ranges/,/^$/ {
6002023-11-19T06:09:16.778Z /^first/c\
6012023-11-19T06:09:16.820Zfirst = "10.151.1.164"
6022023-11-19T06:09:16.860Z /^last/c\
6032023-11-19T06:09:16.903Z--- pkg/config-rss.toml~ Mon Jul 24 01:21:28 2006
6042023-11-19T06:09:16.956Zlast = "10.151.1.173"
6052023-11-19T06:09:17.406Z+++ pkg/config-rss.toml Sun Nov 19 06:09:13 2023
6062023-11-19T06:09:17.442Z }
6072023-11-19T06:09:17.476Z@@ -41,7 +41,7 @@
6082023-11-19T06:09:17.511Z /^\[rack_network_config/,/^$/ {
6092023-11-19T06:09:17.547Z # the DNS domain delegated to the rack by the customer. Each of these addresses
6102023-11-19T06:09:17.583Z /^infra_ip_first/c\
6112023-11-19T06:09:17.619Z # must be contained in one of the "internal services" IP Pool ranges listed
6122023-11-19T06:09:17.656Zinfra_ip_first = "10.151.1.174"
6132023-11-19T06:09:18.097Z # below.
6142023-11-19T06:09:18.129Z /^infra_ip_last/c\
6152023-11-19T06:09:18.174Z-external_dns_ips = [ "192.168.1.20", "192.168.1.21" ]
6162023-11-19T06:09:18.213Zinfra_ip_last = "10.151.1.174"
6172023-11-19T06:09:18.274Z+external_dns_ips = [ "10.151.1.164", "10.151.1.165" ]
6182023-11-19T06:09:18.309Z }
6192023-11-19T06:09:18.368Z
6202023-11-19T06:09:18.400Z /^\[\[rack_network_config.ports/,/^$/ {
6212023-11-19T06:09:18.434Z # Initial TLS certificates for the external API
6222023-11-19T06:09:18.466Z /^routes/c\
6232023-11-19T06:09:18.498Z #
6242023-11-19T06:09:18.533Zroutes = \[{nexthop = "10.151.1.100", destination = "0.0.0.0/0"}\]
6252023-11-19T06:09:18.568Z@@ -77,8 +77,8 @@
6262023-11-19T06:09:18.609Z /^addresses/c\
6272023-11-19T06:09:18.646Z #
6282023-11-19T06:09:18.684Zaddresses = \["10.151.1.174/32"\]
6292023-11-19T06:09:18.722Z # For more on this and what to put here, see docs/how-to-run.adoc.
6302023-11-19T06:09:18.763Z }
6312023-11-19T06:09:18.798Z [[internal_services_ip_pool_ranges]]
6322023-11-19T06:09:18.838Z' pkg/config-rss.toml
6332023-11-19T06:09:18.888Z-first = "192.168.1.20"
6342023-11-19T06:09:18.923Z+ diff -u pkg/config-rss.toml~ pkg/config-rss.toml
6352023-11-19T06:09:18.984Z-last = "192.168.1.29"
6362023-11-19T06:09:19.052Z+ true
6372023-11-19T06:09:19.088Z+first = "10.151.1.164"
6382023-11-19T06:09:19.126Z+ tar rvf out/omicron-sled-agent.tar pkg/config-rss.toml
6392023-11-19T06:09:19.159Z+last = "10.151.1.173"
6402023-11-19T06:09:19.259Z+ rm -f pkg/config-rss.toml pkg/config-rss.toml~
6412023-11-19T06:09:19.325Z
6422023-11-19T06:09:19.360Z+ TLS_NAME=recovery.sys.oxide.test
6432023-11-19T06:09:19.394Z # TODO - this configuration is subject to change going forward. Ultimately these
6442023-11-19T06:09:19.427Z+ openssl req -newkey rsa:4096 -x509 -sha256 -days 3 -nodes -out pkg/initial-tls-cert.pem -keyout pkg/initial-tls-key.pem -subj /CN=recovery.sys.oxide.test -addext subjectAltName=DNS:recovery.sys.oxide.test -addext basicConstraints=critical,CA:FALSE -config /dev/stdin
6452023-11-19T06:09:19.462Z # parameters should be provided to the control plane via wicket, but we need to
6462023-11-19T06:09:19.507Z..+..........+..+....+...........+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.....+................+..+....+...+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*......................+.......+.....+..........+..............+......+...................+......+........+.........+.+..+.........+...+.+.................+..........+..+.+............+........+.+...+.....+......+....+...............+.................+....+..+.............+...........+.........+.......+..+...+..........+..+...+....+..........................+....+.....+..........+........+.......+......+.........+...........+...+.+...+..+......+.........+.............+...+...+..+....+.........+..............+...+...+..........+.....+.+...............+...........+...+....+...+...+.........+...+.....+...............+...+.+...+...............+.....+......+.......+.........+......+.......................+....+..+..................+...+...+.+......+..+.......+........+.+.....+......+.........+......+.+.....+.......+...+.....+.........+.............+...........+.+.........+............+..............+...+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
6472023-11-19T06:09:19.542Z@@ -92,8 +92,8 @@
6482023-11-19T06:09:19.577Z..+.+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.......+......+...+......+....+...+...+.....+......+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*..........+..+.............+..............+.......+.........+.....+......+.+........+...................+..+......+.......+...............+...........+......+....+............+..+............+......+.......+......+............+..+...+....+.........+.........+...............+...+.........+.....+.+.....+.........+..........+.....+....+.....+......+...+............+.............+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
6492023-11-19T06:09:19.610Z # A range of IP addresses used by Boundary Services on the external network. In
6502023-11-19T06:09:20.064Z-----
6512023-11-19T06:09:20.106Z # a real system, these would be addresses of the uplink ports on the Sidecar.
6522023-11-19T06:09:20.148Z+ tar rvf out/omicron-sled-agent.tar pkg/initial-tls-cert.pem pkg/initial-tls-key.pem
6532023-11-19T06:09:20.179Z # With softnpu, only one address is used.
6542023-11-19T06:09:20.211Z+ rm -f pkg/initial-tls-cert.pem pkg/initial-tls-key.pem
6552023-11-19T06:09:20.242Z-infra_ip_first = "192.168.1.30"
6562023-11-19T06:09:20.273Z+ rmdir pkg
6572023-11-19T06:09:20.332Z-infra_ip_last = "192.168.1.30"
6582023-11-19T06:09:20.366Z+ E2E_TLS_CERT=/opt/oxide/sled-agent/pkg/initial-tls-cert.pem
6592023-11-19T06:09:20.397Z+infra_ip_first = "10.151.1.174"
6602023-11-19T06:09:20.443Z+ pfexec mkdir -p /usr/oxide
6612023-11-19T06:09:20.476Z+infra_ip_last = "10.151.1.174"
6622023-11-19T06:09:20.507Z+ pfexec rm -f /usr/oxide/tcpproxy
6632023-11-19T06:09:20.538Z
6642023-11-19T06:09:20.569Z+ pfexec curl -sSfL -o /usr/oxide/tcpproxy http://catacomb.eng.oxide.computer:12346/tcpproxy
6652023-11-19T06:09:20.600Z # Configurations for BGP routers to run on the scrimlets.
6662023-11-19T06:09:20.631Z+ pfexec chmod +x /usr/oxide/tcpproxy
6672023-11-19T06:09:20.664Z bgp = []
6682023-11-19T06:09:20.698Z+ pfexec rm -f /var/svc/manifest/site/tcpproxy.xml
6692023-11-19T06:09:20.730Z@@ -101,9 +101,9 @@
6702023-11-19T06:09:20.769Z+ pfexec curl -sSfL -o /var/svc/manifest/site/tcpproxy.xml http://catacomb.eng.oxide.computer:12346/tcpproxy.xml
6712023-11-19T06:09:20.804Z # You can configure multiple uplinks by repeating the following stanza
6722023-11-19T06:09:20.835Z+ pfexec svccfg import /var/svc/manifest/site/tcpproxy.xml
6732023-11-19T06:09:20.878Z [[rack_network_config.ports]]
6742023-11-19T06:09:20.918Z+ OMICRON_NO_UNINSTALL=1
6752023-11-19T06:09:20.955Z # Routes associated with this port.
6762023-11-19T06:09:20.986Z+ ptime -m pfexec ./target/release/omicron-package -t test install
6772023-11-19T06:09:21.019Z-routes = [{nexthop = "192.168.1.199", destination = "0.0.0.0/0"}]
6782023-11-19T06:09:21.052ZNov 19 06:09:14.975 DEBG target[test]: Target({"image": "standard", "machine": "non-gimlet", "rack-topology": "single-sled", "switch": "softnpu"})
6792023-11-19T06:09:21.083Z+routes = [{nexthop = "10.151.1.100", destination = "0.0.0.0/0"}]
6802023-11-19T06:09:21.114ZNov 19 06:09:14.977 INFO Installing service, dst: /opt/oxide/clickhouse.tar.gz, src: out/clickhouse.tar.gz
6812023-11-19T06:09:21.145Z # Addresses associated with this port.
6822023-11-19T06:09:21.177ZNov 19 06:09:14.977 INFO Installing service, dst: /opt/oxide/cockroachdb.tar.gz, src: out/cockroachdb.tar.gz
6832023-11-19T06:09:21.214Z-addresses = ["192.168.1.30/32"]
6842023-11-19T06:09:21.246ZNov 19 06:09:14.977 INFO Installing service, dst: /opt/oxide/mgs.tar.gz, src: out/omicron-gateway-softnpu.tar.gz
6852023-11-19T06:09:21.277Z+addresses = ["10.151.1.174/32"]
6862023-11-19T06:09:21.311ZNov 19 06:09:14.977 INFO Installing service, dst: /opt/oxide/nexus.tar.gz, src: out/omicron-nexus.tar.gz
6872023-11-19T06:09:21.374Z # Name of the uplink port. This should always be "qsfp0" when using softnpu.
6882023-11-19T06:09:21.405ZNov 19 06:09:14.977 INFO Installing service, dst: /opt/oxide/crucible.tar.gz, src: out/crucible.tar.gz
6892023-11-19T06:09:21.439Z port = "qsfp0"
6902023-11-19T06:09:21.470ZNov 19 06:09:14.977 INFO Installing service, dst: /opt/oxide/sled-agent.tar, src: out/omicron-sled-agent.tar
6912023-11-19T06:09:21.509Z # The speed of this port.
6922023-11-19T06:09:21.540ZNov 19 06:09:14.977 INFO Installing service, dst: /opt/oxide/clickhouse_keeper.tar.gz, src: out/clickhouse_keeper.tar.gz
6932023-11-19T06:09:21.571Za pkg/config-rss.toml 7K
6942023-11-19T06:09:21.602ZNov 19 06:09:14.977 INFO Installing service, dst: /opt/oxide/propolis-server.tar.gz, src: out/propolis-server.tar.gz
6952023-11-19T06:09:21.633Za pkg/initial-tls-cert.pem 2K
6962023-11-19T06:09:21.666ZNov 19 06:09:14.977 INFO Installing service, dst: /opt/oxide/overlay.tar.gz, src: out/overlay.tar.gz
6972023-11-19T06:09:21.698Za pkg/initial-tls-key.pem 4K
6982023-11-19T06:09:21.729ZNov 19 06:09:14.977 INFO Installing service, dst: /opt/oxide/mg-ddm.tar, src: out/mg-ddm-gz.tar
6992023-11-19T06:09:21.762ZNov 19 06:09:14.977 INFO Installing service, dst: /opt/oxide/internal_dns.tar.gz, src: out/internal-dns.tar.gz
7002023-11-19T06:09:22.204ZNov 19 06:09:14.977 INFO Installing service, dst: /opt/oxide/ntp.tar.gz, src: out/ntp.tar.gz
7012023-11-19T06:09:22.240ZNov 19 06:09:14.977 INFO Installing service, dst: /opt/oxide/crucible_pantry.tar.gz, src: out/crucible-pantry.tar.gz
7022023-11-19T06:09:22.278ZNov 19 06:09:14.977 INFO Installing service, dst: /opt/oxide/switch.tar.gz, src: out/switch-softnpu.tar.gz
7032023-11-19T06:09:22.315ZNov 19 06:09:14.977 INFO Installing service, dst: /opt/oxide/external_dns.tar.gz, src: out/external-dns.tar.gz
7042023-11-19T06:09:22.352ZNov 19 06:09:14.977 INFO Installing service, dst: /opt/oxide/oximeter.tar.gz, src: out/oximeter-collector.tar.gz
7052023-11-19T06:09:22.426ZNov 19 06:09:15.199 INFO Unpacking service tarball, service_path: /opt/oxide/mg-ddm, tar_path: /opt/oxide/mg-ddm.tar
7062023-11-19T06:09:22.470ZNov 19 06:09:15.235 INFO Unpacking service tarball, service_path: /opt/oxide/sled-agent, tar_path: /opt/oxide/sled-agent.tar
7072023-11-19T06:09:22.512ZNov 19 06:09:15.294 INFO Installing bootstrap service from /opt/oxide/sled-agent/pkg/manifest.xml
7082023-11-19T06:09:22.547Z
7092023-11-19T06:09:22.585Zreal 0.458032508
7102023-11-19T06:09:22.622Zuser 0.099001575
7112023-11-19T06:09:22.661Zsys 1.840276090
7122023-11-19T06:09:22.701Ztrap 0.000013280
7132023-11-19T06:09:22.740Ztflt 0.000069050
7142023-11-19T06:09:22.779Zdflt 0.000582066
7152023-11-19T06:09:22.817Zkflt 0.000000000
7162023-11-19T06:09:22.857Zlock 41.133047799
7172023-11-19T06:09:22.894Zslp 0.675515596
7182023-11-19T06:09:22.933Zlat 0.002794196
7192023-11-19T06:09:22.974Zstop 0.215998098
7202023-11-19T06:09:23.007Z+ retry=0
7212023-11-19T06:09:23.042Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7222023-11-19T06:09:23.087Z+ [[ 0 -gt 30 ]]
7232023-11-19T06:09:23.124Z+ sleep 1
7242023-11-19T06:09:23.155Z+ retry=1
7252023-11-19T06:09:23.186Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7262023-11-19T06:09:23.219Z+ [[ 1 -gt 30 ]]
7272023-11-19T06:09:23.252Z+ sleep 1
7282023-11-19T06:09:23.287Z+ retry=2
7292023-11-19T06:09:23.322Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7302023-11-19T06:09:23.353Z+ [[ 2 -gt 30 ]]
7312023-11-19T06:09:23.388Z+ sleep 1
7322023-11-19T06:09:23.874Z+ retry=3
7332023-11-19T06:09:23.920Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7342023-11-19T06:09:23.964Z+ [[ 3 -gt 30 ]]
7352023-11-19T06:09:24.150Z+ sleep 1
7362023-11-19T06:09:24.182Z+ retry=4
7372023-11-19T06:09:24.216Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7382023-11-19T06:09:44.730Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone dump-state
7392023-11-19T06:09:44.764Zlocal v6:
7402023-11-19T06:09:44.795Zfd00:99::1
7412023-11-19T06:09:44.832Zfe80::c02b:62ff:fecb:abe8
7422023-11-19T06:09:44.871Zlocal v4:
7432023-11-19T06:09:44.913Z10.151.1.174
7442023-11-19T06:09:44.949Zrouter v6:
7452023-11-19T06:09:44.992Zfd00:1122:3344:3::/64 -> fe80::444a:85ff:fedf:e262 (1)
7462023-11-19T06:09:45.028Zfd00:1122:3344:2::/64 -> fe80::444a:85ff:fedf:e262 (1)
7472023-11-19T06:09:45.062Zfdb0:18c0:4d0d:9fb2::/64 -> fe80::444a:85ff:fedf:e262 (1)
7482023-11-19T06:09:45.094Zfd00:1122:3344:101::/64 -> fe80::444a:85ff:fedf:e262 (1)
7492023-11-19T06:09:45.127Zfd00:1122:3344:1::/64 -> fe80::444a:85ff:fedf:e262 (1)
7502023-11-19T06:09:45.159Zrouter v4_idx:
7512023-11-19T06:09:45.193Z0.0.0.0/0 -> 0
7522023-11-19T06:09:45.227Zrouter v4_routes:
7532023-11-19T06:09:45.681Z0 -> 10.151.1.100 (2)
7542023-11-19T06:09:45.722Zresolver v4:
7552023-11-19T06:09:45.765Z10.151.1.100 -> 18:c0:4d:0d:9f:b2
7562023-11-19T06:09:45.799Z10.151.1.174 -> 42:55:8c:67:42:f7
7572023-11-19T06:09:45.831Zresolver v6:
7582023-11-19T06:09:45.872Zfe80::c02b:62ff:fecb:abe8 -> c2:2b:62:cb:ab:e8
7592023-11-19T06:09:45.908Zfe80::444a:85ff:fedf:e262 -> 46:4a:85:df:e2:62
7602023-11-19T06:09:45.940Znat v4:
7612023-11-19T06:09:45.971Z10.151.1.169 0/16383 -> fd00:1122:3344:101::1 100/a8:40:25:ff:c3:ef
7622023-11-19T06:09:46.002Znat_v6:
7632023-11-19T06:09:46.033Zport_mac:
7642023-11-19T06:09:46.064Z1000: a8:40:25:7d:28:33
7652023-11-19T06:09:46.100Z1: a8:40:25:7d:28:34
7662023-11-19T06:09:46.136Zproxy arp:
7672023-11-19T06:09:46.180Z10.151.1.164/10.151.1.227: a8:e1:de:01:70:1d
7682023-11-19T06:09:46.631Z+ retry=0
7692023-11-19T06:09:46.665Z+++ zoneadm list -n
7702023-11-19T06:09:46.697Z+++ grep oxz_ntp
7712023-11-19T06:09:46.732Z++ pfexec svcs -z oxz_ntp_433037e4-b4a0-439a-a8b0-2af8fce793f4 -Hostate oxide/ntp
7722023-11-19T06:09:46.777Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
7732023-11-19T06:09:46.810Z++ true
7742023-11-19T06:09:46.841Z+ [[ '' != online ]]
7752023-11-19T06:09:46.872Z+ [[ 0 -gt 60 ]]
7762023-11-19T06:09:46.904Z+ sleep 1
7772023-11-19T06:09:46.935Z+ retry=1
7782023-11-19T06:09:46.968Z+++ zoneadm list -n
7792023-11-19T06:09:47.001Z+++ grep oxz_ntp
7802023-11-19T06:09:47.033Z++ pfexec svcs -z oxz_ntp_433037e4-b4a0-439a-a8b0-2af8fce793f4 -Hostate oxide/ntp
7812023-11-19T06:09:47.071Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
7822023-11-19T06:09:47.103Z++ true
7832023-11-19T06:09:47.134Z+ [[ '' != online ]]
7842023-11-19T06:09:47.165Z+ [[ 1 -gt 60 ]]
7852023-11-19T06:09:47.201Z+ sleep 1
7862023-11-19T06:09:47.240Z+ retry=2
7872023-11-19T06:09:47.279Z+++ zoneadm list -n
7882023-11-19T06:09:47.314Z+++ grep oxz_ntp
7892023-11-19T06:09:47.345Z++ pfexec svcs -z oxz_ntp_433037e4-b4a0-439a-a8b0-2af8fce793f4 -Hostate oxide/ntp
7902023-11-19T06:09:47.415Z+ [[ disabled != online ]]
7912023-11-19T06:09:47.459Z+ [[ 2 -gt 60 ]]
7922023-11-19T06:09:47.492Z+ sleep 1
7932023-11-19T06:09:47.842Z+ retry=3
7942023-11-19T06:09:47.873Z+++ zoneadm list -n
7952023-11-19T06:09:47.904Z+++ grep oxz_ntp
7962023-11-19T06:09:47.935Z++ pfexec svcs -z oxz_ntp_433037e4-b4a0-439a-a8b0-2af8fce793f4 -Hostate oxide/ntp
7972023-11-19T06:09:47.966Z+ [[ disabled != online ]]
7982023-11-19T06:09:47.998Z+ [[ 3 -gt 60 ]]
7992023-11-19T06:09:48.044Z+ sleep 1
8002023-11-19T06:09:48.904Z+ retry=4
8012023-11-19T06:09:48.937Z+++ zoneadm list -n
8022023-11-19T06:09:48.969Z+++ grep oxz_ntp
8032023-11-19T06:09:49.002Z++ pfexec svcs -z oxz_ntp_433037e4-b4a0-439a-a8b0-2af8fce793f4 -Hostate oxide/ntp
8042023-11-19T06:09:49.034Z+ [[ disabled != online ]]
8052023-11-19T06:09:49.066Z+ [[ 4 -gt 60 ]]
8062023-11-19T06:09:49.098Z+ sleep 1
8072023-11-19T06:09:49.891Z+ retry=5
8082023-11-19T06:09:49.930Z+++ zoneadm list -n
8092023-11-19T06:09:49.963Z+++ grep oxz_ntp
8102023-11-19T06:09:49.997Z++ pfexec svcs -z oxz_ntp_433037e4-b4a0-439a-a8b0-2af8fce793f4 -Hostate oxide/ntp
8112023-11-19T06:09:50.033Z+ [[ disabled != online ]]
8122023-11-19T06:09:50.073Z+ [[ 5 -gt 60 ]]
8132023-11-19T06:09:50.114Z+ sleep 1
8142023-11-19T06:09:50.912Z+ retry=6
8152023-11-19T06:09:50.944Z+++ zoneadm list -n
8162023-11-19T06:09:51.072Z+++ grep oxz_ntp
8172023-11-19T06:09:51.114Z++ pfexec svcs -z oxz_ntp_433037e4-b4a0-439a-a8b0-2af8fce793f4 -Hostate oxide/ntp
8182023-11-19T06:09:51.145Z+ [[ disabled != online ]]
8192023-11-19T06:09:51.176Z+ [[ 6 -gt 60 ]]
8202023-11-19T06:09:51.207Z+ sleep 1
8212023-11-19T06:09:51.934Z+ retry=7
8222023-11-19T06:09:51.973Z+++ zoneadm list -n
8232023-11-19T06:09:52.006Z+++ grep oxz_ntp
8242023-11-19T06:09:52.038Z++ pfexec svcs -z oxz_ntp_433037e4-b4a0-439a-a8b0-2af8fce793f4 -Hostate oxide/ntp
8252023-11-19T06:09:52.086Z+ [[ disabled != online ]]
8262023-11-19T06:09:52.118Z+ [[ 7 -gt 60 ]]
8272023-11-19T06:09:52.150Z+ sleep 1
 
8282023-11-19T06:09:52.810Zbuskin console login: Nov 19 06:09:52 buskin ip: ip: joining multicasts failed (4) on vopte0 - will use link layer broadcasts for multicast
 
8292023-11-19T06:09:52.956Z+ retry=8
8302023-11-19T06:09:52.989Z+++ zoneadm list -n
8312023-11-19T06:09:53.026Z+++ grep oxz_ntp
8322023-11-19T06:09:53.062Z++ pfexec svcs -z oxz_ntp_433037e4-b4a0-439a-a8b0-2af8fce793f4 -Hostate oxide/ntp
8332023-11-19T06:09:53.095Z+ [[ offline* != online ]]
8342023-11-19T06:09:53.126Z+ [[ 8 -gt 60 ]]
8352023-11-19T06:09:53.157Z+ sleep 1
8362023-11-19T06:09:53.983Z+ retry=9
8372023-11-19T06:09:54.026Z+++ zoneadm list -n
8382023-11-19T06:09:54.058Z+++ grep oxz_ntp
8392023-11-19T06:09:54.089Z++ pfexec svcs -z oxz_ntp_433037e4-b4a0-439a-a8b0-2af8fce793f4 -Hostate oxide/ntp
8402023-11-19T06:09:54.120Z+ [[ online != online ]]
8412023-11-19T06:09:54.151Z+ echo 'Waited for chrony: 9s'
8422023-11-19T06:09:54.187Z+ export RUST_BACKTRACE=1
8432023-11-19T06:09:54.233Z+ RUST_BACKTRACE=1
8442023-11-19T06:09:54.265ZWaited for chrony: 9s
8452023-11-19T06:09:54.296Z+ export E2E_TLS_CERT IPPOOL_START IPPOOL_END
8462023-11-19T06:09:54.327Z+ ./tests/bootstrap
8472023-11-19T06:09:54.359ZUsing external DNS server at 10.151.1.164:53
8482023-11-19T06:09:54.390Z2023-11-19 06:09:53.961812299 UTC: attempting to log into API
8492023-11-19T06:10:09.066Z2023-11-19 06:10:09.027995229 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8502023-11-19T06:10:10.067Z2023-11-19 06:10:10.031658316 UTC: attempting to log into API
8512023-11-19T06:10:25.101Z2023-11-19 06:10:25.091388380 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8522023-11-19T06:10:33.701Z2023-11-19 06:10:26.092392830 UTC: attempting to log into API
8532023-11-19T06:10:41.133Z2023-11-19 06:10:41.125329990 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8542023-11-19T06:10:42.134Z2023-11-19 06:10:42.126431952 UTC: attempting to log into API
8552023-11-19T06:10:57.172Z2023-11-19 06:10:57.160553007 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8562023-11-19T06:10:58.175Z2023-11-19 06:10:58.165434889 UTC: attempting to log into API
8572023-11-19T06:11:13.425Z2023-11-19 06:11:13.412345684 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8582023-11-19T06:11:14.425Z2023-11-19 06:11:14.414301074 UTC: attempting to log into API
 
8592023-11-19T06:11:17.959ZNov 19 06:11:17 bukin ip: ip: joining multicasts failed (4) on vopte2 - will use link layer broadcasts for multicast
8602023-11-19T06:11:26.132ZNov 19 06:11:25 buskin ip: ip: joining multicasts failed (4) on vopte1 - will use link layer broadcasts for multicast
 
8612023-11-19T06:11:29.477Z2023-11-19 06:11:29.469067049 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8622023-11-19T06:11:30.479Z2023-11-19 06:11:30.470056689 UTC: attempting to log into API
 
8632023-11-19T06:11:33.276ZNov 19 06:11:32 buskin ip: ip: joining multicasts failed (4) on vopte3 - will use link layer broadcasts for multicast
8642023-11-19T06:11:39.077ZNov 19 06:11:38 buskin ip: ip: joining multicasts failed (4) on vopte4 - will use link layer broadcasts for multicast
8652023-11-19T06:11:45.130ZNov 19 06:11:44 bukin ip: ip: joining multicasts failed (4) on vopte5 - will use link layer broadcasts for multicast
 
8662023-11-19T06:11:45.511Z2023-11-19 06:11:45.502906418 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8672023-11-19T06:11:46.516Z2023-11-19 06:11:46.504168894 UTC: attempting to log into API
8682023-11-19T06:12:01.545Z2023-11-19 06:12:01.537638876 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8692023-11-19T06:12:02.548Z2023-11-19 06:12:02.538438002 UTC: attempting to log into API
8702023-11-19T06:12:02.599Z2023-11-19 06:12:02.590854845 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: tcp connect error: Connection refused (os error 146)
8712023-11-19T06:12:03.600Z2023-11-19 06:12:03.592173142 UTC: attempting to log into API
8722023-11-19T06:12:03.633Z2023-11-19 06:12:03.625390167 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: tcp connect error: Connection refused (os error 146)
8732023-11-19T06:12:04.638Z2023-11-19 06:12:04.626647753 UTC: attempting to log into API
8742023-11-19T06:12:05.417Z2023-11-19 06:12:05.409539450 UTC: login succeeded
8752023-11-19T06:12:05.452Zwaiting for nexus to come up...
8762023-11-19T06:12:05.521Zcreating IP pool... 10.151.1.175 - 10.151.1.227
8772023-11-19T06:12:05.580Zensuring datasets are ready...
8782023-11-19T06:12:09.747Zlet's roll.
8792023-11-19T06:12:09.779Z+ rm ./tests/bootstrap
8802023-11-19T06:12:09.821Z+ for test_bin in tests/*
8812023-11-19T06:12:09.853Z+ ./tests/bootstrap-2ed5f6fd6904f562
8822023-11-19T06:12:09.884Z
8832023-11-19T06:12:09.916Zrunning 0 tests
8842023-11-19T06:12:09.947Z
8852023-11-19T06:12:09.986Ztest result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
8862023-11-19T06:12:10.018Z
8872023-11-19T06:12:10.049Z+ for test_bin in tests/*
8882023-11-19T06:12:10.081Z+ ./tests/end_to_end_tests-03815e4bc8ad474c
8892023-11-19T06:12:10.114Z
8902023-11-19T06:12:10.556Zrunning 2 tests
8912023-11-19T06:12:10.595Ztest no_spoof::no_spoof_here ... ok
8922023-11-19T06:13:09.844Ztest instance_launch::instance_launch has been running for over 60 seconds
8932023-11-19T06:15:10.507Ztest instance_launch::instance_launch ... ok
8942023-11-19T06:15:10.538Z
8952023-11-19T06:15:10.569Ztest result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 180.67s
8962023-11-19T06:15:10.601Z
8972023-11-19T06:15:10.632Z+ _exit_trap
8982023-11-19T06:15:10.664Z+ local status=0
8992023-11-19T06:15:10.697Z+ [[ 0 -eq 0 ]]
9002023-11-19T06:15:10.780Z+ exit 0
9012023-11-19T06:15:10.812Zprocess exited: duration 397378 ms, exit code 0
 
9022023-11-19T06:15:10.958Zfound 90 output files
9032023-11-19T06:15:11.083Zuploading: /var/svc/log/oxide-sled-agent:default.log (668190 bytes)
9042023-11-19T06:15:13.132Zuploaded: /var/svc/log/oxide-sled-agent:default.log
9052023-11-19T06:15:13.156Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_a89820e2-72e6-46f6-9120-cb71d21d5ec1/root/var/svc/log/oxide-internal_dns:default.log (9651 bytes)
9062023-11-19T06:15:14.313Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_a89820e2-72e6-46f6-9120-cb71d21d5ec1/root/var/svc/log/oxide-internal_dns:default.log
9072023-11-19T06:15:14.337Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_a89820e2-72e6-46f6-9120-cb71d21d5ec1/root/var/svc/log/oxide-internal_dns:default.log.0 (105339 bytes)
9082023-11-19T06:15:15.582Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_a89820e2-72e6-46f6-9120-cb71d21d5ec1/root/var/svc/log/oxide-internal_dns:default.log.0
9092023-11-19T06:15:15.607Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_77c7e8cf-beb4-4504-9e73-73fb8321d373/root/var/svc/log/oxide-crucible-agent:default.log (0 bytes)
9102023-11-19T06:15:16.674Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_77c7e8cf-beb4-4504-9e73-73fb8321d373/root/var/svc/log/oxide-crucible-agent:default.log
9112023-11-19T06:15:16.698Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_77c7e8cf-beb4-4504-9e73-73fb8321d373/root/var/svc/log/oxide-crucible-agent:default.log.0 (14582 bytes)
9122023-11-19T06:15:17.840Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_77c7e8cf-beb4-4504-9e73-73fb8321d373/root/var/svc/log/oxide-crucible-agent:default.log.0
9132023-11-19T06:15:17.864Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_77c7e8cf-beb4-4504-9e73-73fb8321d373/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
9142023-11-19T06:15:18.936Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_77c7e8cf-beb4-4504-9e73-73fb8321d373/root/var/svc/log/oxide-crucible-downstairs:default.log
9152023-11-19T06:15:18.960Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_77c7e8cf-beb4-4504-9e73-73fb8321d373/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
9162023-11-19T06:15:20.058Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_77c7e8cf-beb4-4504-9e73-73fb8321d373/root/var/svc/log/oxide-crucible-downstairs:default.log.0
9172023-11-19T06:15:20.082Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_77c7e8cf-beb4-4504-9e73-73fb8321d373/root/var/svc/log/oxide-crucible-downstairs:downstairs-b2a29e05-a5dd-4abb-a3fc-5f0b1c2fe600.log (0 bytes)
9182023-11-19T06:15:21.233Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_77c7e8cf-beb4-4504-9e73-73fb8321d373/root/var/svc/log/oxide-crucible-downstairs:downstairs-b2a29e05-a5dd-4abb-a3fc-5f0b1c2fe600.log
9192023-11-19T06:15:21.258Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_77c7e8cf-beb4-4504-9e73-73fb8321d373/root/var/svc/log/oxide-crucible-downstairs:downstairs-b2a29e05-a5dd-4abb-a3fc-5f0b1c2fe600.log.0 (3051 bytes)
9202023-11-19T06:15:22.356Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_77c7e8cf-beb4-4504-9e73-73fb8321d373/root/var/svc/log/oxide-crucible-downstairs:downstairs-b2a29e05-a5dd-4abb-a3fc-5f0b1c2fe600.log.0
9212023-11-19T06:15:22.380Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_cf3c7630-01d7-4a58-bb59-bf274619dccc/root/var/svc/log/oxide-external_dns:default.log (0 bytes)
9222023-11-19T06:15:23.453Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_cf3c7630-01d7-4a58-bb59-bf274619dccc/root/var/svc/log/oxide-external_dns:default.log
9232023-11-19T06:15:23.477Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_cf3c7630-01d7-4a58-bb59-bf274619dccc/root/var/svc/log/oxide-external_dns:default.log.0 (15610 bytes)
9242023-11-19T06:15:24.629Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_cf3c7630-01d7-4a58-bb59-bf274619dccc/root/var/svc/log/oxide-external_dns:default.log.0
9252023-11-19T06:15:24.668Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_nexus_ac0caecc-fb3c-401c-9b83-f22e31be4bce/root/var/svc/log/oxide-nexus:default.log (608 bytes)
9262023-11-19T06:15:24.718Zupload warning: file "/pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_nexus_ac0caecc-fb3c-401c-9b83-f22e31be4bce/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 608 -> 4766
9272023-11-19T06:15:25.810Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_nexus_ac0caecc-fb3c-401c-9b83-f22e31be4bce/root/var/svc/log/oxide-nexus:default.log
9282023-11-19T06:15:25.834Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_nexus_ac0caecc-fb3c-401c-9b83-f22e31be4bce/root/var/svc/log/oxide-nexus:default.log.0 (519431 bytes)
9292023-11-19T06:15:27.204Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_nexus_ac0caecc-fb3c-401c-9b83-f22e31be4bce/root/var/svc/log/oxide-nexus:default.log.0
9302023-11-19T06:15:27.228Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_nexus_bb01605d-8b0f-435c-94fd-a2ae123d4057/root/var/svc/log/oxide-nexus:default.log (608 bytes)
9312023-11-19T06:15:27.265Zupload warning: file "/pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_nexus_bb01605d-8b0f-435c-94fd-a2ae123d4057/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 608 -> 5375
9322023-11-19T06:15:28.362Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_nexus_bb01605d-8b0f-435c-94fd-a2ae123d4057/root/var/svc/log/oxide-nexus:default.log
9332023-11-19T06:15:28.808Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_nexus_bb01605d-8b0f-435c-94fd-a2ae123d4057/root/var/svc/log/oxide-nexus:default.log.0 (649113 bytes)
9342023-11-19T06:15:29.765Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_nexus_bb01605d-8b0f-435c-94fd-a2ae123d4057/root/var/svc/log/oxide-nexus:default.log.0
9352023-11-19T06:15:29.796Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_3259d953-4e29-4b58-97af-70f2798c63d4/root/var/svc/log/oxide-crucible-agent:default.log (0 bytes)
9362023-11-19T06:15:30.848Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_3259d953-4e29-4b58-97af-70f2798c63d4/root/var/svc/log/oxide-crucible-agent:default.log
9372023-11-19T06:15:30.876Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_3259d953-4e29-4b58-97af-70f2798c63d4/root/var/svc/log/oxide-crucible-agent:default.log.0 (3436 bytes)
9382023-11-19T06:15:31.996Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_3259d953-4e29-4b58-97af-70f2798c63d4/root/var/svc/log/oxide-crucible-agent:default.log.0
9392023-11-19T06:15:32.033Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_3259d953-4e29-4b58-97af-70f2798c63d4/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
9402023-11-19T06:15:33.107Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_3259d953-4e29-4b58-97af-70f2798c63d4/root/var/svc/log/oxide-crucible-downstairs:default.log
9412023-11-19T06:15:33.144Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_3259d953-4e29-4b58-97af-70f2798c63d4/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
9422023-11-19T06:15:34.216Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_3259d953-4e29-4b58-97af-70f2798c63d4/root/var/svc/log/oxide-crucible-downstairs:default.log.0
9432023-11-19T06:15:34.255Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_6e75aacd-e692-474b-8dd1-8b7584d36372/root/var/svc/log/oxide-crucible-agent:default.log (0 bytes)
9442023-11-19T06:15:35.305Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_6e75aacd-e692-474b-8dd1-8b7584d36372/root/var/svc/log/oxide-crucible-agent:default.log
9452023-11-19T06:15:35.340Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_6e75aacd-e692-474b-8dd1-8b7584d36372/root/var/svc/log/oxide-crucible-agent:default.log.0 (3436 bytes)
9462023-11-19T06:15:36.857Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_6e75aacd-e692-474b-8dd1-8b7584d36372/root/var/svc/log/oxide-crucible-agent:default.log.0
9472023-11-19T06:15:36.882Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_6e75aacd-e692-474b-8dd1-8b7584d36372/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
9482023-11-19T06:15:37.949Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_6e75aacd-e692-474b-8dd1-8b7584d36372/root/var/svc/log/oxide-crucible-downstairs:default.log
9492023-11-19T06:15:37.973Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_6e75aacd-e692-474b-8dd1-8b7584d36372/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
9502023-11-19T06:15:39.077Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_6e75aacd-e692-474b-8dd1-8b7584d36372/root/var/svc/log/oxide-crucible-downstairs:default.log.0
9512023-11-19T06:15:39.102Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_e1c3f6c1-365a-4f2d-980f-5e7a625849cc/root/var/svc/log/oxide-crucible-agent:default.log (4323 bytes)
9522023-11-19T06:15:40.204Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_e1c3f6c1-365a-4f2d-980f-5e7a625849cc/root/var/svc/log/oxide-crucible-agent:default.log
9532023-11-19T06:15:40.228Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_e1c3f6c1-365a-4f2d-980f-5e7a625849cc/root/var/svc/log/oxide-crucible-agent:default.log.0 (9793 bytes)
9542023-11-19T06:15:41.364Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_e1c3f6c1-365a-4f2d-980f-5e7a625849cc/root/var/svc/log/oxide-crucible-agent:default.log.0
9552023-11-19T06:15:41.412Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_e1c3f6c1-365a-4f2d-980f-5e7a625849cc/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
9562023-11-19T06:15:42.484Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_e1c3f6c1-365a-4f2d-980f-5e7a625849cc/root/var/svc/log/oxide-crucible-downstairs:default.log
9572023-11-19T06:15:42.511Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_e1c3f6c1-365a-4f2d-980f-5e7a625849cc/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
9582023-11-19T06:15:44.645Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_e1c3f6c1-365a-4f2d-980f-5e7a625849cc/root/var/svc/log/oxide-crucible-downstairs:default.log.0
9592023-11-19T06:15:44.670Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_e1c3f6c1-365a-4f2d-980f-5e7a625849cc/root/var/svc/log/oxide-crucible-downstairs:downstairs-10be010f-ccbb-4582-b9c5-e6ded04fc215.log (1006 bytes)
9602023-11-19T06:15:45.781Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_e1c3f6c1-365a-4f2d-980f-5e7a625849cc/root/var/svc/log/oxide-crucible-downstairs:downstairs-10be010f-ccbb-4582-b9c5-e6ded04fc215.log
9612023-11-19T06:15:45.815Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_e1c3f6c1-365a-4f2d-980f-5e7a625849cc/root/var/svc/log/oxide-crucible-downstairs:downstairs-10be010f-ccbb-4582-b9c5-e6ded04fc215.log.0 (4530 bytes)
9622023-11-19T06:15:46.913Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_e1c3f6c1-365a-4f2d-980f-5e7a625849cc/root/var/svc/log/oxide-crucible-downstairs:downstairs-10be010f-ccbb-4582-b9c5-e6ded04fc215.log.0
9632023-11-19T06:15:46.938Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_pantry_58348c6f-2a0b-465f-853a-291e4fe0701d/root/var/svc/log/oxide-crucible-pantry:default.log (0 bytes)
9642023-11-19T06:15:48.011Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_pantry_58348c6f-2a0b-465f-853a-291e4fe0701d/root/var/svc/log/oxide-crucible-pantry:default.log
9652023-11-19T06:15:48.049Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_pantry_58348c6f-2a0b-465f-853a-291e4fe0701d/root/var/svc/log/oxide-crucible-pantry:default.log.0 (2338 bytes)
9662023-11-19T06:15:49.137Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_pantry_58348c6f-2a0b-465f-853a-291e4fe0701d/root/var/svc/log/oxide-crucible-pantry:default.log.0
9672023-11-19T06:15:49.161Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_internal_dns_7ae1d9f5-178f-4fde-a8d3-377f89a7f9fe/root/var/svc/log/oxide-internal_dns:default.log (0 bytes)
9682023-11-19T06:15:49.197Zupload warning: file "/pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_internal_dns_7ae1d9f5-178f-4fde-a8d3-377f89a7f9fe/root/var/svc/log/oxide-internal_dns:default.log" changed size mid upload: 0 -> 3755
9692023-11-19T06:15:50.299Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_internal_dns_7ae1d9f5-178f-4fde-a8d3-377f89a7f9fe/root/var/svc/log/oxide-internal_dns:default.log
9702023-11-19T06:15:50.739Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_internal_dns_7ae1d9f5-178f-4fde-a8d3-377f89a7f9fe/root/var/svc/log/oxide-internal_dns:default.log.0 (112949 bytes)
9712023-11-19T06:15:51.588Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_internal_dns_7ae1d9f5-178f-4fde-a8d3-377f89a7f9fe/root/var/svc/log/oxide-internal_dns:default.log.0
9722023-11-19T06:15:51.622Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_cockroachdb_94691b48-a97f-40b5-b082-d6c9165b59e6/root/var/svc/log/oxide-cockroachdb:default.log (0 bytes)
9732023-11-19T06:15:53.077Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_cockroachdb_94691b48-a97f-40b5-b082-d6c9165b59e6/root/var/svc/log/oxide-cockroachdb:default.log
9742023-11-19T06:15:53.113Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_cockroachdb_94691b48-a97f-40b5-b082-d6c9165b59e6/root/var/svc/log/oxide-cockroachdb:default.log.0 (5369 bytes)
9752023-11-19T06:15:54.198Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_cockroachdb_94691b48-a97f-40b5-b082-d6c9165b59e6/root/var/svc/log/oxide-cockroachdb:default.log.0
9762023-11-19T06:15:54.235Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_4ec7f9e6-d6a5-4c11-86f9-106d49d290ba/root/var/svc/log/oxide-crucible-agent:default.log (4786 bytes)
9772023-11-19T06:15:55.302Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_4ec7f9e6-d6a5-4c11-86f9-106d49d290ba/root/var/svc/log/oxide-crucible-agent:default.log
9782023-11-19T06:15:55.344Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_4ec7f9e6-d6a5-4c11-86f9-106d49d290ba/root/var/svc/log/oxide-crucible-agent:default.log.0 (9792 bytes)
9792023-11-19T06:15:56.420Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_4ec7f9e6-d6a5-4c11-86f9-106d49d290ba/root/var/svc/log/oxide-crucible-agent:default.log.0
9802023-11-19T06:15:56.451Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_4ec7f9e6-d6a5-4c11-86f9-106d49d290ba/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
9812023-11-19T06:15:57.511Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_4ec7f9e6-d6a5-4c11-86f9-106d49d290ba/root/var/svc/log/oxide-crucible-downstairs:default.log
9822023-11-19T06:15:57.545Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_4ec7f9e6-d6a5-4c11-86f9-106d49d290ba/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
9832023-11-19T06:15:58.673Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_4ec7f9e6-d6a5-4c11-86f9-106d49d290ba/root/var/svc/log/oxide-crucible-downstairs:default.log.0
9842023-11-19T06:15:58.702Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_4ec7f9e6-d6a5-4c11-86f9-106d49d290ba/root/var/svc/log/oxide-crucible-downstairs:downstairs-25ff7d8c-6862-4ad3-9825-f4f667eb999a.log (1006 bytes)
9852023-11-19T06:15:59.805Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_4ec7f9e6-d6a5-4c11-86f9-106d49d290ba/root/var/svc/log/oxide-crucible-downstairs:downstairs-25ff7d8c-6862-4ad3-9825-f4f667eb999a.log
9862023-11-19T06:15:59.829Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_4ec7f9e6-d6a5-4c11-86f9-106d49d290ba/root/var/svc/log/oxide-crucible-downstairs:downstairs-25ff7d8c-6862-4ad3-9825-f4f667eb999a.log.0 (4528 bytes)
9872023-11-19T06:16:00.932Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_4ec7f9e6-d6a5-4c11-86f9-106d49d290ba/root/var/svc/log/oxide-crucible-downstairs:downstairs-25ff7d8c-6862-4ad3-9825-f4f667eb999a.log.0
9882023-11-19T06:16:00.956Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_fadd7932-2c90-43cc-aa79-5eff8a701280/root/var/svc/log/oxide-crucible-agent:default.log (0 bytes)
9892023-11-19T06:16:02.042Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_fadd7932-2c90-43cc-aa79-5eff8a701280/root/var/svc/log/oxide-crucible-agent:default.log
9902023-11-19T06:16:02.067Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_fadd7932-2c90-43cc-aa79-5eff8a701280/root/var/svc/log/oxide-crucible-agent:default.log.0 (14580 bytes)
9912023-11-19T06:16:03.196Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_fadd7932-2c90-43cc-aa79-5eff8a701280/root/var/svc/log/oxide-crucible-agent:default.log.0
9922023-11-19T06:16:03.222Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_fadd7932-2c90-43cc-aa79-5eff8a701280/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
9932023-11-19T06:16:04.288Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_fadd7932-2c90-43cc-aa79-5eff8a701280/root/var/svc/log/oxide-crucible-downstairs:default.log
9942023-11-19T06:16:04.313Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_fadd7932-2c90-43cc-aa79-5eff8a701280/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
9952023-11-19T06:16:05.410Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_fadd7932-2c90-43cc-aa79-5eff8a701280/root/var/svc/log/oxide-crucible-downstairs:default.log.0
9962023-11-19T06:16:05.435Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_fadd7932-2c90-43cc-aa79-5eff8a701280/root/var/svc/log/oxide-crucible-downstairs:downstairs-2b03f291-c35c-4850-94ab-03911ed18043.log (0 bytes)
9972023-11-19T06:16:06.506Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_fadd7932-2c90-43cc-aa79-5eff8a701280/root/var/svc/log/oxide-crucible-downstairs:downstairs-2b03f291-c35c-4850-94ab-03911ed18043.log
9982023-11-19T06:16:06.540Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_fadd7932-2c90-43cc-aa79-5eff8a701280/root/var/svc/log/oxide-crucible-downstairs:downstairs-2b03f291-c35c-4850-94ab-03911ed18043.log.0 (3046 bytes)
9992023-11-19T06:16:07.636Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_fadd7932-2c90-43cc-aa79-5eff8a701280/root/var/svc/log/oxide-crucible-downstairs:downstairs-2b03f291-c35c-4850-94ab-03911ed18043.log.0
10002023-11-19T06:16:07.660Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_cockroachdb_84136bc6-1e6e-48b4-8780-ce5129376649/root/var/svc/log/oxide-cockroachdb:default.log (0 bytes)
10012023-11-19T06:16:08.728Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_cockroachdb_84136bc6-1e6e-48b4-8780-ce5129376649/root/var/svc/log/oxide-cockroachdb:default.log
10022023-11-19T06:16:08.752Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_cockroachdb_84136bc6-1e6e-48b4-8780-ce5129376649/root/var/svc/log/oxide-cockroachdb:default.log.0 (5368 bytes)
10032023-11-19T06:16:09.859Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_cockroachdb_84136bc6-1e6e-48b4-8780-ce5129376649/root/var/svc/log/oxide-cockroachdb:default.log.0
10042023-11-19T06:16:09.884Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_cockroachdb_95dcf1da-239c-4151-89d1-77db4c011b59/root/var/svc/log/oxide-cockroachdb:default.log (0 bytes)
10052023-11-19T06:16:10.952Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_cockroachdb_95dcf1da-239c-4151-89d1-77db4c011b59/root/var/svc/log/oxide-cockroachdb:default.log
10062023-11-19T06:16:10.976Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_cockroachdb_95dcf1da-239c-4151-89d1-77db4c011b59/root/var/svc/log/oxide-cockroachdb:default.log.0 (5368 bytes)
10072023-11-19T06:16:12.079Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_cockroachdb_95dcf1da-239c-4151-89d1-77db4c011b59/root/var/svc/log/oxide-cockroachdb:default.log.0
10082023-11-19T06:16:12.103Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_60aafcd7-d0c0-4ba8-a321-90aa29af4b7b/root/var/svc/log/oxide-crucible-agent:default.log (0 bytes)
10092023-11-19T06:16:13.174Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_60aafcd7-d0c0-4ba8-a321-90aa29af4b7b/root/var/svc/log/oxide-crucible-agent:default.log
10102023-11-19T06:16:13.198Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_60aafcd7-d0c0-4ba8-a321-90aa29af4b7b/root/var/svc/log/oxide-crucible-agent:default.log.0 (14584 bytes)
10112023-11-19T06:16:14.354Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_60aafcd7-d0c0-4ba8-a321-90aa29af4b7b/root/var/svc/log/oxide-crucible-agent:default.log.0
10122023-11-19T06:16:14.382Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_60aafcd7-d0c0-4ba8-a321-90aa29af4b7b/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
10132023-11-19T06:16:15.453Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_60aafcd7-d0c0-4ba8-a321-90aa29af4b7b/root/var/svc/log/oxide-crucible-downstairs:default.log
10142023-11-19T06:16:15.477Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_60aafcd7-d0c0-4ba8-a321-90aa29af4b7b/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
10152023-11-19T06:16:16.574Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_60aafcd7-d0c0-4ba8-a321-90aa29af4b7b/root/var/svc/log/oxide-crucible-downstairs:default.log.0
10162023-11-19T06:16:16.598Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_60aafcd7-d0c0-4ba8-a321-90aa29af4b7b/root/var/svc/log/oxide-crucible-downstairs:downstairs-3518c3b4-9c39-4ffb-a5b6-324ed4c7dda7.log (0 bytes)
10172023-11-19T06:16:17.666Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_60aafcd7-d0c0-4ba8-a321-90aa29af4b7b/root/var/svc/log/oxide-crucible-downstairs:downstairs-3518c3b4-9c39-4ffb-a5b6-324ed4c7dda7.log
10182023-11-19T06:16:17.690Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_60aafcd7-d0c0-4ba8-a321-90aa29af4b7b/root/var/svc/log/oxide-crucible-downstairs:downstairs-3518c3b4-9c39-4ffb-a5b6-324ed4c7dda7.log.0 (3051 bytes)
10192023-11-19T06:16:18.788Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_60aafcd7-d0c0-4ba8-a321-90aa29af4b7b/root/var/svc/log/oxide-crucible-downstairs:downstairs-3518c3b4-9c39-4ffb-a5b6-324ed4c7dda7.log.0
10202023-11-19T06:16:18.813Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_6f634528-4a3a-401b-9c82-87be1438b198/root/var/svc/log/oxide-crucible-agent:default.log (4323 bytes)
10212023-11-19T06:16:19.902Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_6f634528-4a3a-401b-9c82-87be1438b198/root/var/svc/log/oxide-crucible-agent:default.log
10222023-11-19T06:16:19.939Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_6f634528-4a3a-401b-9c82-87be1438b198/root/var/svc/log/oxide-crucible-agent:default.log.0 (10226 bytes)
10232023-11-19T06:16:21.039Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_6f634528-4a3a-401b-9c82-87be1438b198/root/var/svc/log/oxide-crucible-agent:default.log.0
10242023-11-19T06:16:21.077Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_6f634528-4a3a-401b-9c82-87be1438b198/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
10252023-11-19T06:16:22.123Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_6f634528-4a3a-401b-9c82-87be1438b198/root/var/svc/log/oxide-crucible-downstairs:default.log
10262023-11-19T06:16:22.147Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_6f634528-4a3a-401b-9c82-87be1438b198/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
10272023-11-19T06:16:23.256Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_6f634528-4a3a-401b-9c82-87be1438b198/root/var/svc/log/oxide-crucible-downstairs:default.log.0
10282023-11-19T06:16:23.280Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_6f634528-4a3a-401b-9c82-87be1438b198/root/var/svc/log/oxide-crucible-downstairs:downstairs-e7a55953-5eab-4916-8998-7e672cdc1a34.log (1006 bytes)
10292023-11-19T06:16:24.378Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_6f634528-4a3a-401b-9c82-87be1438b198/root/var/svc/log/oxide-crucible-downstairs:downstairs-e7a55953-5eab-4916-8998-7e672cdc1a34.log
10302023-11-19T06:16:24.402Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_6f634528-4a3a-401b-9c82-87be1438b198/root/var/svc/log/oxide-crucible-downstairs:downstairs-e7a55953-5eab-4916-8998-7e672cdc1a34.log.0 (4529 bytes)
10312023-11-19T06:16:25.510Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_6f634528-4a3a-401b-9c82-87be1438b198/root/var/svc/log/oxide-crucible-downstairs:downstairs-e7a55953-5eab-4916-8998-7e672cdc1a34.log.0
10322023-11-19T06:16:25.543Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_external_dns_4d123e84-bf18-4ba8-aef8-34cac2ffd7fd/root/var/svc/log/oxide-external_dns:default.log (0 bytes)
10332023-11-19T06:16:25.583Zupload warning: file "/pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_external_dns_4d123e84-bf18-4ba8-aef8-34cac2ffd7fd/root/var/svc/log/oxide-external_dns:default.log" changed size mid upload: 0 -> 3779
10342023-11-19T06:16:26.700Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_external_dns_4d123e84-bf18-4ba8-aef8-34cac2ffd7fd/root/var/svc/log/oxide-external_dns:default.log
10352023-11-19T06:16:26.724Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_external_dns_4d123e84-bf18-4ba8-aef8-34cac2ffd7fd/root/var/svc/log/oxide-external_dns:default.log.0 (15609 bytes)
10362023-11-19T06:16:27.859Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_external_dns_4d123e84-bf18-4ba8-aef8-34cac2ffd7fd/root/var/svc/log/oxide-external_dns:default.log.0
10372023-11-19T06:16:27.890Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_ntp_433037e4-b4a0-439a-a8b0-2af8fce793f4/root/var/svc/log/oxide-ntp:default.log (0 bytes)
10382023-11-19T06:16:27.929Zupload warning: file "/pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_ntp_433037e4-b4a0-439a-a8b0-2af8fce793f4/root/var/svc/log/oxide-ntp:default.log" changed size mid upload: 0 -> 68
10392023-11-19T06:16:29.020Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_ntp_433037e4-b4a0-439a-a8b0-2af8fce793f4/root/var/svc/log/oxide-ntp:default.log
10402023-11-19T06:16:29.047Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_ntp_433037e4-b4a0-439a-a8b0-2af8fce793f4/root/var/svc/log/oxide-ntp:default.log.0 (848 bytes)
10412023-11-19T06:16:30.141Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_ntp_433037e4-b4a0-439a-a8b0-2af8fce793f4/root/var/svc/log/oxide-ntp:default.log.0
10422023-11-19T06:16:30.166Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_oximeter_431199b5-5482-47d3-abe2-38c29cf17103/root/var/svc/log/oxide-oximeter:default.log (9008 bytes)
10432023-11-19T06:16:30.321Zupload warning: file "/pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_oximeter_431199b5-5482-47d3-abe2-38c29cf17103/root/var/svc/log/oxide-oximeter:default.log" changed size mid upload: 9008 -> 70921
10442023-11-19T06:16:31.413Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_oximeter_431199b5-5482-47d3-abe2-38c29cf17103/root/var/svc/log/oxide-oximeter:default.log
10452023-11-19T06:16:31.437Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_oximeter_431199b5-5482-47d3-abe2-38c29cf17103/root/var/svc/log/oxide-oximeter:default.log.0 (161077 bytes)
10462023-11-19T06:16:32.747Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_oximeter_431199b5-5482-47d3-abe2-38c29cf17103/root/var/svc/log/oxide-oximeter:default.log.0
10472023-11-19T06:16:32.785Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_f508fbc2-f635-42e0-9493-4a745b4e8907/root/var/svc/log/oxide-cockroachdb:default.log (0 bytes)
10482023-11-19T06:16:33.898Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_f508fbc2-f635-42e0-9493-4a745b4e8907/root/var/svc/log/oxide-cockroachdb:default.log
10492023-11-19T06:16:33.925Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_f508fbc2-f635-42e0-9493-4a745b4e8907/root/var/svc/log/oxide-cockroachdb:default.log.0 (5366 bytes)
10502023-11-19T06:16:35.399Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_f508fbc2-f635-42e0-9493-4a745b4e8907/root/var/svc/log/oxide-cockroachdb:default.log.0
10512023-11-19T06:16:35.426Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_12b5af5e-c1d4-4622-b722-071ec350f3c1/root/var/svc/log/oxide-crucible-agent:default.log (0 bytes)
10522023-11-19T06:16:36.498Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_12b5af5e-c1d4-4622-b722-071ec350f3c1/root/var/svc/log/oxide-crucible-agent:default.log
10532023-11-19T06:16:36.522Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_12b5af5e-c1d4-4622-b722-071ec350f3c1/root/var/svc/log/oxide-crucible-agent:default.log.0 (3436 bytes)
10542023-11-19T06:16:37.630Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_12b5af5e-c1d4-4622-b722-071ec350f3c1/root/var/svc/log/oxide-crucible-agent:default.log.0
10552023-11-19T06:16:37.667Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_12b5af5e-c1d4-4622-b722-071ec350f3c1/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
10562023-11-19T06:16:38.737Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_12b5af5e-c1d4-4622-b722-071ec350f3c1/root/var/svc/log/oxide-crucible-downstairs:default.log
10572023-11-19T06:16:38.762Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_12b5af5e-c1d4-4622-b722-071ec350f3c1/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
10582023-11-19T06:16:39.864Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_12b5af5e-c1d4-4622-b722-071ec350f3c1/root/var/svc/log/oxide-crucible-downstairs:default.log.0
10592023-11-19T06:16:39.888Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_3aef9bed-34d2-469e-9781-33abbbb5a23e/root/var/svc/log/oxide-crucible-pantry:default.log (0 bytes)
10602023-11-19T06:16:40.955Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_3aef9bed-34d2-469e-9781-33abbbb5a23e/root/var/svc/log/oxide-crucible-pantry:default.log
10612023-11-19T06:16:40.979Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_3aef9bed-34d2-469e-9781-33abbbb5a23e/root/var/svc/log/oxide-crucible-pantry:default.log.0 (2333 bytes)
10622023-11-19T06:16:42.083Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_3aef9bed-34d2-469e-9781-33abbbb5a23e/root/var/svc/log/oxide-crucible-pantry:default.log.0
10632023-11-19T06:16:42.114Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_9f695549-8ad1-4798-bd61-cd5d08287dbc/root/var/svc/log/oxide-nexus:default.log (561374 bytes)
10642023-11-19T06:16:42.481Zupload warning: file "/pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_9f695549-8ad1-4798-bd61-cd5d08287dbc/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 561374 -> 641657
10652023-11-19T06:16:43.572Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_9f695549-8ad1-4798-bd61-cd5d08287dbc/root/var/svc/log/oxide-nexus:default.log
10662023-11-19T06:16:43.596Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_9f695549-8ad1-4798-bd61-cd5d08287dbc/root/var/svc/log/oxide-nexus:default.log.0 (3944963 bytes)
10672023-11-19T06:16:47.425Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_9f695549-8ad1-4798-bd61-cd5d08287dbc/root/var/svc/log/oxide-nexus:default.log.0
10682023-11-19T06:16:47.449Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_clickhouse_acecbb9d-34b5-4519-8332-c57fd9f0ad1e/root/var/svc/log/oxide-clickhouse:default.log (23696 bytes)
10692023-11-19T06:16:47.784Zupload warning: file "/pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_clickhouse_acecbb9d-34b5-4519-8332-c57fd9f0ad1e/root/var/svc/log/oxide-clickhouse:default.log" changed size mid upload: 23696 -> 298403
10702023-11-19T06:16:48.881Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_clickhouse_acecbb9d-34b5-4519-8332-c57fd9f0ad1e/root/var/svc/log/oxide-clickhouse:default.log
10712023-11-19T06:16:48.906Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_clickhouse_acecbb9d-34b5-4519-8332-c57fd9f0ad1e/root/var/svc/log/oxide-clickhouse:default.log.0 (602566 bytes)
10722023-11-19T06:16:50.415Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_clickhouse_acecbb9d-34b5-4519-8332-c57fd9f0ad1e/root/var/svc/log/oxide-clickhouse:default.log.0
10732023-11-19T06:16:50.439Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_7a287c31-8ced-4000-860f-abc38b9e160e/root/var/svc/log/oxide-cockroachdb:default.log (0 bytes)
10742023-11-19T06:16:51.514Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_7a287c31-8ced-4000-860f-abc38b9e160e/root/var/svc/log/oxide-cockroachdb:default.log
10752023-11-19T06:16:51.542Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_7a287c31-8ced-4000-860f-abc38b9e160e/root/var/svc/log/oxide-cockroachdb:default.log.0 (5343 bytes)
10762023-11-19T06:16:52.655Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_7a287c31-8ced-4000-860f-abc38b9e160e/root/var/svc/log/oxide-cockroachdb:default.log.0
10772023-11-19T06:16:52.703Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_6294b18b-a9a3-4cc5-81fb-5aa6fd57c70d/root/var/svc/log/oxide-crucible-pantry:default.log (0 bytes)
10782023-11-19T06:16:53.767Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_6294b18b-a9a3-4cc5-81fb-5aa6fd57c70d/root/var/svc/log/oxide-crucible-pantry:default.log
10792023-11-19T06:16:53.791Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_6294b18b-a9a3-4cc5-81fb-5aa6fd57c70d/root/var/svc/log/oxide-crucible-pantry:default.log.0 (2338 bytes)
10802023-11-19T06:16:54.882Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_6294b18b-a9a3-4cc5-81fb-5aa6fd57c70d/root/var/svc/log/oxide-crucible-pantry:default.log.0
10812023-11-19T06:16:54.914Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_ddb6ca49-1c81-4102-9dca-81a3f0374031/root/var/svc/log/oxide-internal_dns:default.log (0 bytes)
10822023-11-19T06:16:54.951Zupload warning: file "/pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_ddb6ca49-1c81-4102-9dca-81a3f0374031/root/var/svc/log/oxide-internal_dns:default.log" changed size mid upload: 0 -> 7510
10832023-11-19T06:16:56.021Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_ddb6ca49-1c81-4102-9dca-81a3f0374031/root/var/svc/log/oxide-internal_dns:default.log
10842023-11-19T06:16:56.052Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_ddb6ca49-1c81-4102-9dca-81a3f0374031/root/var/svc/log/oxide-internal_dns:default.log.0 (118642 bytes)
10852023-11-19T06:16:57.255Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_ddb6ca49-1c81-4102-9dca-81a3f0374031/root/var/svc/log/oxide-internal_dns:default.log.0
10862023-11-19T06:16:57.286Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_ntp_433037e4-b4a0-439a-a8b0-2af8fce793f4/root/var/log/chrony/measurements.log (9590 bytes)
10872023-11-19T06:16:57.325Zupload warning: file "/pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_ntp_433037e4-b4a0-439a-a8b0-2af8fce793f4/root/var/log/chrony/measurements.log" changed size mid upload: 9590 -> 12741
10882023-11-19T06:16:58.394Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_ntp_433037e4-b4a0-439a-a8b0-2af8fce793f4/root/var/log/chrony/measurements.log
10892023-11-19T06:16:58.425Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_ntp_433037e4-b4a0-439a-a8b0-2af8fce793f4/root/var/log/chrony/statistics.log (4797 bytes)
10902023-11-19T06:16:58.456Zupload warning: file "/pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_ntp_433037e4-b4a0-439a-a8b0-2af8fce793f4/root/var/log/chrony/statistics.log" changed size mid upload: 4797 -> 6903
10912023-11-19T06:16:59.517Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_ntp_433037e4-b4a0-439a-a8b0-2af8fce793f4/root/var/log/chrony/statistics.log
10922023-11-19T06:16:59.542Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_ntp_433037e4-b4a0-439a-a8b0-2af8fce793f4/root/var/log/chrony/tracking.log (1454 bytes)
10932023-11-19T06:16:59.585Zupload warning: file "/pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_ntp_433037e4-b4a0-439a-a8b0-2af8fce793f4/root/var/log/chrony/tracking.log" changed size mid upload: 1454 -> 1718
10942023-11-19T06:17:00.675Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_ntp_433037e4-b4a0-439a-a8b0-2af8fce793f4/root/var/log/chrony/tracking.log