01HH0A04MV8BAA76TBF70J7ECX: helios / deploy

BasicConfig {
    output_rules: [
        "%/var/svc/log/oxide-sled-agent:default.log*",
        "%/pool/ext/*/crypt/zone/oxz_*/root/var/svc/log/oxide-*.log*",
        "%/pool/ext/*/crypt/zone/oxz_*/root/var/svc/log/system-illumos-*.log*",
        "%/pool/ext/*/crypt/zone/oxz_ntp_*/root/var/log/chrony/*.log*",
        "!/pool/ext/*/crypt/zone/oxz_propolis-server_*/root/var/svc/log/*.log*",
        "%/pool/ext/*/crypt/debug/global/oxide-sled-agent:default.log.*",
        "%/pool/ext/*/crypt/debug/oxz_*/oxide-*.log.*",
        "%/pool/ext/*/crypt/debug/oxz_*/system-illumos-*.log.*",
        "!/pool/ext/*/crypt/debug/oxz_propolis-server_*/*.log.*",
    ],
    rust_toolchain: None,
    target: Some(
        "lab-2.0-opte-0.25",
    ),
    access_repos: [],
    publish: [],
    skip_clone: true,
}

Buildomat Job: 01HH0A1R3N6BHP244HQSR2GYH7

Tags:

Artefacts:

Output:

SEQ GLOBAL TIME DETAILS
12023-12-06T20:24:22.190Zjob dependencies complete; ready to run (waiting for 48 m 47 s)
22023-12-06T20:30:22.487Zjob assigned to worker 01HH0CV57P3JNBAZE8NM9B30SB (queued for 6 m 0 s)
32023-12-06T20:30:22.700Zhttp://172.20.0.5:9969/ipxe/sock... ok
42023-12-06T20:30:22.748Zhttp://172.20.0.5:9969/os/sock/platform/i86pc/kernel/amd64/unix... ok
52023-12-06T20:30:22.784Zhttp://172.20.0.5:9969/os/sock/platform/i86pc/amd64/boot_archive... ok
62023-12-06T20:30:22.824Zhttp://172.20.0.5:9969/os/sock/platform/i86pc/amd64/boot_archive.hash... ok
72023-12-06T20:30:22.910Zhttp://172.20.0.5:9969/postboot/sock... ok
82023-12-06T20:30:22.949ZwcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U
92023-12-06T20:30:23.047Zcpu0: microcode has been updated from version 0xa001133 to 0xa0011d1
102023-12-06T20:30:23.085ZOxide Helios Version helios-2.0.22318 64-bit
112023-12-06T20:30:23.121ZConfiguring devices.
122023-12-06T20:30:23.212Z#################################################
132023-12-06T20:30:23.318Z#### ############# ##
142023-12-06T20:30:23.357Z### ### ############ ##
152023-12-06T20:30:23.394Z## ### # ## ### ## Oxide Computer Company ##
162023-12-06T20:30:23.430Z## ## ## ### # ### ##
172023-12-06T20:30:23.470Z## # ### #### #### This Station Under ##
182023-12-06T20:30:23.605Z### ### #### # ### Computer Control ##
192023-12-06T20:30:23.641Z#### #### ### ## ##
202023-12-06T20:30:23.675Z##################################################
212023-12-06T20:30:23.785ZHostname: sock
222023-12-06T20:30:23.839ZLoading smf(7) serice descriptions:
232023-12-06T20:30:24.795Zdownloading input: /input/ci-tools/work/end-to-end-tests/bootstrap-83d41f1b41d1fb89.gz
242023-12-06T20:30:26.047Zdownloaded input: /input/ci-tools/work/end-to-end-tests/bootstrap-83d41f1b41d1fb89.gz
252023-12-06T20:30:26.088Zdownloading input: /input/ci-tools/work/end-to-end-tests/bootstrap.gz
262023-12-06T20:30:49.553Zdownloaded input: /input/ci-tools/work/end-to-end-tests/bootstrap.gz
272023-12-06T20:30:49.553Zdownloading input: /input/ci-tools/work/end-to-end-tests/end_to_end_tests-8c93950b566420c7.gz
282023-12-06T20:31:13.756Zdownloaded input: /input/ci-tools/work/end-to-end-tests/end_to_end_tests-8c93950b566420c7.gz
292023-12-06T20:31:13.756Zdownloading input: /input/ci-tools/work/caboose-util.gz
302023-12-06T20:31:15.149Zdownloaded input: /input/ci-tools/work/caboose-util.gz
312023-12-06T20:31:15.149Zdownloading input: /input/ci-tools/work/tufaceous.gz
322023-12-06T20:31:18.851Zdownloaded input: /input/ci-tools/work/tufaceous.gz
332023-12-06T20:31:18.880Zdownloading input: /input/package/work/version.txt
342023-12-06T20:31:18.880Zdownloaded input: /input/package/work/version.txt
352023-12-06T20:31:18.880Zdownloading input: /input/package/work/package.tar.gz
362023-12-06T20:31:33.994Zdownloaded input: /input/package/work/package.tar.gz
372023-12-06T20:31:33.994Zdownloading input: /input/package/work/global-zone-packages.tar.gz
382023-12-06T20:31:56.963Zdownloaded input: /input/package/work/global-zone-packages.tar.gz
392023-12-06T20:31:56.963Zdownloading input: /input/package/work/trampoline-global-zone-packages.tar.gz
402023-12-06T20:32:08.500Zdownloaded input: /input/package/work/trampoline-global-zone-packages.tar.gz
412023-12-06T20:32:08.527Zdownloading input: /input/package/work/zones/clickhouse.tar.gz
422023-12-06T20:32:44.567Zdownloaded input: /input/package/work/zones/clickhouse.tar.gz
432023-12-06T20:32:44.567Zdownloading input: /input/package/work/zones/clickhouse_keeper.tar.gz
442023-12-06T20:33:18.786Zdownloaded input: /input/package/work/zones/clickhouse_keeper.tar.gz
452023-12-06T20:33:18.786Zdownloading input: /input/package/work/zones/cockroachdb.tar.gz
462023-12-06T20:33:40.065Zdownloaded input: /input/package/work/zones/cockroachdb.tar.gz
472023-12-06T20:33:40.092Zdownloading input: /input/package/work/zones/crucible-pantry.tar.gz
482023-12-06T20:33:44.155Zdownloaded input: /input/package/work/zones/crucible-pantry.tar.gz
492023-12-06T20:33:44.155Zdownloading input: /input/package/work/zones/crucible.tar.gz
502023-12-06T20:33:52.006Zdownloaded input: /input/package/work/zones/crucible.tar.gz
512023-12-06T20:33:52.007Zdownloading input: /input/package/work/zones/external-dns.tar.gz
522023-12-06T20:33:56.916Zdownloaded input: /input/package/work/zones/external-dns.tar.gz
532023-12-06T20:33:56.916Zdownloading input: /input/package/work/zones/internal-dns.tar.gz
542023-12-06T20:34:01.278Zdownloaded input: /input/package/work/zones/internal-dns.tar.gz
552023-12-06T20:34:01.278Zdownloading input: /input/package/work/zones/ntp.tar.gz
562023-12-06T20:34:01.332Zdownloaded input: /input/package/work/zones/ntp.tar.gz
572023-12-06T20:34:01.332Zdownloading input: /input/package/work/zones/omicron-gateway-asic.tar.gz
582023-12-06T20:34:05.375Zdownloaded input: /input/package/work/zones/omicron-gateway-asic.tar.gz
592023-12-06T20:34:05.401Zdownloading input: /input/package/work/zones/omicron-gateway-softnpu.tar.gz
602023-12-06T20:34:08.784Zdownloaded input: /input/package/work/zones/omicron-gateway-softnpu.tar.gz
612023-12-06T20:34:08.797Zdownloading input: /input/package/work/zones/omicron-nexus-single-sled.tar.gz
622023-12-06T20:34:28.841Zdownloaded input: /input/package/work/zones/omicron-nexus-single-sled.tar.gz
632023-12-06T20:34:28.841Zdownloading input: /input/package/work/zones/omicron-nexus.tar.gz
642023-12-06T20:34:42.008Zdownloaded input: /input/package/work/zones/omicron-nexus.tar.gz
652023-12-06T20:34:42.008Zdownloading input: /input/package/work/zones/overlay.tar.gz
662023-12-06T20:34:42.081Zdownloaded input: /input/package/work/zones/overlay.tar.gz
672023-12-06T20:34:42.081Zdownloading input: /input/package/work/zones/oximeter-collector.tar.gz
682023-12-06T20:34:48.314Zdownloaded input: /input/package/work/zones/oximeter-collector.tar.gz
692023-12-06T20:34:48.314Zdownloading input: /input/package/work/zones/propolis-server.tar.gz
702023-12-06T20:35:04.234Zdownloaded input: /input/package/work/zones/propolis-server.tar.gz
712023-12-06T20:35:04.234Zdownloading input: /input/package/work/zones/switch-asic.tar.gz
722023-12-06T20:35:51.294Zdownloaded input: /input/package/work/zones/switch-asic.tar.gz
732023-12-06T20:35:51.294Zdownloading input: /input/package/work/zones/switch-softnpu.tar.gz
742023-12-06T20:36:28.712Zdownloaded input: /input/package/work/zones/switch-softnpu.tar.gz
 
752023-12-06T20:36:28.739Zstarting task 0: "setup"
762023-12-06T20:36:28.739Z++ uname -s
772023-12-06T20:36:28.739Z+ kern=SunOS
782023-12-06T20:36:28.739Z+ case "$kern" in
792023-12-06T20:36:28.739Z+ groupadd -g 12345 build
802023-12-06T20:36:28.740Z+ useradd -u 12345 -g build -d /home/build -s /bin/bash -c build -P 'Primary Administrator' build
812023-12-06T20:36:30.731Z+ zfs create -o mountpoint=/work rpool/work
822023-12-06T20:36:30.770Z++ awk '$2 == "/home" { print $3 }' /etc/mnttab
832023-12-06T20:36:30.795Z+ home_fs=zfs
842023-12-06T20:36:30.795Z+ [[ zfs == autofs ]]
852023-12-06T20:36:30.795Z+ mkdir -p /home/build
862023-12-06T20:36:30.795Z+ chown build:build /home/build /work
872023-12-06T20:36:32.777Z+ chmod 0700 /home/build /work
882023-12-06T20:36:32.809Zprocess exited: duration 4067 ms, exit code 0
 
892023-12-06T20:36:32.871Zstarting task 1: "authentication"
902023-12-06T20:36:32.911Zprocess exited: duration 45 ms, exit code 0
 
912023-12-06T20:36:32.962Zstarting task 2: "build"
922023-12-06T20:36:32.987Z+ trap _exit_trap EXIT
932023-12-06T20:36:32.987Z+ pfexec mdb -kwe 'secpolicy_fs_linkdir/v 55 48 89 e5 b8 01 00 00 00 5d c3'
942023-12-06T20:36:33.168Zsecpolicy_fs_linkdir: 0x55 = 0x55
952023-12-06T20:36:33.169Zsecpolicy_fs_linkdir+1: 0xb8 = 0x48
962023-12-06T20:36:33.169Zsecpolicy_fs_linkdir+2: 0x1 = 0x89
972023-12-06T20:36:33.169Zsecpolicy_fs_linkdir+3: 0 = 0xe5
982023-12-06T20:36:33.169Zsecpolicy_fs_linkdir+4: 0 = 0xb8
992023-12-06T20:36:33.169Zsecpolicy_fs_linkdir+5: 0 = 0x1
1002023-12-06T20:36:33.169Zsecpolicy_fs_linkdir+6: 0x48 = 0x0
1012023-12-06T20:36:33.169Zsecpolicy_fs_linkdir+7: 0x89 = 0x0
1022023-12-06T20:36:33.169Zsecpolicy_fs_linkdir+8: 0xe5 = 0x0
1032023-12-06T20:36:33.169Zsecpolicy_fs_linkdir+9: 0x48 = 0x5d
1042023-12-06T20:36:33.195Zsecpolicy_fs_linkdir+0xa: 0x83 = 0xc3
1052023-12-06T20:36:33.195Z+ [[ -d /opt/oxide ]]
1062023-12-06T20:36:33.195Z+ pfexec mv /opt/oxide /opt/oxide-underneath
1072023-12-06T20:36:33.195Z+ pfexec mkdir /opt/oxide
1082023-12-06T20:36:33.195Z+ pfexec mount -F tmpfs -O swap /opt/oxide
1092023-12-06T20:36:33.195Z+ [[ -d /opt/oxide-underneath ]]
1102023-12-06T20:36:33.195Z+ cd /opt/oxide-underneath
1112023-12-06T20:36:33.197Z+ pfexec tar ceEp@/f - .
1122023-12-06T20:36:33.197Z+ cd /opt/oxide
1132023-12-06T20:36:33.197Z+ pfexec tar xveEp@/f -
1142023-12-06T20:36:33.292Zx ., 0 bytes, 0 tape blocks
1152023-12-06T20:36:33.292Zx ./opte, 0 bytes, 0 tape blocks
1162023-12-06T20:36:33.293Zx ./opte/bin, 0 bytes, 0 tape blocks
1172023-12-06T20:36:33.293Zx ./opte/bin/opteadm, 50714832 bytes, 99053 tape blocks
1182023-12-06T20:36:33.352Z+ pfexec rm -rf /opt/oxide-underneath
1192023-12-06T20:36:33.352Z+ pfexec /sbin/zfs create -o mountpoint=/zone rpool/zone
1202023-12-06T20:36:33.377Z+ pfexec mkdir /opt/oxide/work
1212023-12-06T20:36:33.377Z+ pfexec chown build:build /opt/oxide/work
1222023-12-06T20:36:33.377Z+ cd /opt/oxide/work
1232023-12-06T20:36:33.377Z+ ptime -m tar xvzf /input/package/work/package.tar.gz
1242023-12-06T20:36:39.383ZDecompressing '/input/package/work/package.tar.gz' with '/usr/bin/gzcat'...
1252023-12-06T20:36:39.383Zx out/mg-ddm-gz.tar, 62723072 bytes, 122506 tape blocks
1262023-12-06T20:36:39.383Zx out/omicron-sled-agent.tar, 111094784 bytes, 216982 tape blocks
1272023-12-06T20:36:39.383Zx out/target/test, 75 bytes, 1 tape blocks
1282023-12-06T20:36:39.383Zx out/npuzone/npuzone, 37380856 bytes, 73010 tape blocks
1292023-12-06T20:36:39.383Zx package-manifest.toml, 22143 bytes, 44 tape blocks
1302023-12-06T20:36:39.383Zx smf/sled-agent/non-gimlet/config.toml, 3298 bytes, 7 tape blocks
1312023-12-06T20:36:39.383Zx target/release/omicron-package, 42582368 bytes, 83169 tape blocks
1322023-12-06T20:36:39.393Zx tools/create_virtual_hardware.sh, 3176 bytes, 7 tape blocks
1332023-12-06T20:36:39.393Zx tools/virtual_hardware.sh, 3408 bytes, 7 tape blocks
1342023-12-06T20:36:39.393Zx tools/scrimlet/create-softnpu-zone.sh, 619 bytes, 2 tape blocks
1352023-12-06T20:36:39.393Zx tools/scrimlet/destroy-softnpu-zone.sh, 195 bytes, 1 tape blocks
1362023-12-06T20:36:39.393Zx tools/scrimlet/softnpu-init.sh, 2148 bytes, 5 tape blocks
1372023-12-06T20:36:39.393Zx tools/scrimlet/softnpu-zone.txt, 330 bytes, 1 tape blocks
1382023-12-06T20:36:39.393Zx tools/scrimlet/softnpu.toml, 171 bytes, 1 tape blocks
1392023-12-06T20:36:39.393Z
1402023-12-06T20:36:39.393Zreal 1.538665976
1412023-12-06T20:36:39.394Zuser 1.380241239
1422023-12-06T20:36:39.394Zsys 0.278255281
1432023-12-06T20:36:39.430Ztrap 0.000064309
1442023-12-06T20:36:39.430Ztflt 0.000000000
1452023-12-06T20:36:39.430Zdflt 0.000000000
1462023-12-06T20:36:39.431Zkflt 0.000000000
1472023-12-06T20:36:39.431Zlock 0.000000000
1482023-12-06T20:36:39.431Zslp 1.379234674
1492023-12-06T20:36:39.431Zlat 0.037485174
1502023-12-06T20:36:39.431Zstop 0.000116040
1512023-12-06T20:36:39.431Z+ cp /input/package/work/zones/clickhouse_keeper.tar.gz /input/package/work/zones/clickhouse.tar.gz /input/package/work/zones/cockroachdb.tar.gz /input/package/work/zones/crucible-pantry.tar.gz /input/package/work/zones/crucible.tar.gz /input/package/work/zones/external-dns.tar.gz /input/package/work/zones/internal-dns.tar.gz /input/package/work/zones/ntp.tar.gz /input/package/work/zones/omicron-gateway-asic.tar.gz /input/package/work/zones/omicron-gateway-softnpu.tar.gz /input/package/work/zones/omicron-nexus-single-sled.tar.gz /input/package/work/zones/omicron-nexus.tar.gz /input/package/work/zones/overlay.tar.gz /input/package/work/zones/oximeter-collector.tar.gz /input/package/work/zones/propolis-server.tar.gz /input/package/work/zones/switch-asic.tar.gz /input/package/work/zones/switch-softnpu.tar.gz out/
1522023-12-06T20:36:39.431Z+ mv out/omicron-nexus-single-sled.tar.gz out/omicron-nexus.tar.gz
1532023-12-06T20:36:39.431Z+ mkdir tests
1542023-12-06T20:36:39.480Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1552023-12-06T20:36:39.480Z+ ptime -m gunzip
1562023-12-06T20:36:39.480Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap-83d41f1b41d1fb89
1572023-12-06T20:36:39.480Z
1582023-12-06T20:36:39.481Zreal 0.138798368
1592023-12-06T20:36:39.481Zuser 0.127746909
1602023-12-06T20:36:39.481Zsys 0.010797745
1612023-12-06T20:36:39.481Ztrap 0.000015620
1622023-12-06T20:36:39.481Ztflt 0.000000000
1632023-12-06T20:36:39.481Zdflt 0.000000000
1642023-12-06T20:36:39.481Zkflt 0.000000000
1652023-12-06T20:36:39.481Zlock 0.000000000
1662023-12-06T20:36:39.482Zslp 0.000020160
1672023-12-06T20:36:39.482Zlat 0.000022690
1682023-12-06T20:36:39.482Zstop 0.000127731
1692023-12-06T20:36:39.482Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap-83d41f1b41d1fb89
1702023-12-06T20:36:39.482Z+ chmod a+x tests/bootstrap-83d41f1b41d1fb89
1712023-12-06T20:36:39.482Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1722023-12-06T20:36:39.482Z+ ptime -m gunzip
1732023-12-06T20:36:39.482Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap
1742023-12-06T20:36:42.646Z
1752023-12-06T20:36:42.647Zreal 2.210389046
1762023-12-06T20:36:42.647Zuser 2.076461419
1772023-12-06T20:36:42.647Zsys 0.133537564
1782023-12-06T20:36:42.647Ztrap 0.000164701
1792023-12-06T20:36:42.647Ztflt 0.000000000
1802023-12-06T20:36:42.647Zdflt 0.000000000
1812023-12-06T20:36:42.647Zkflt 0.000000000
1822023-12-06T20:36:42.647Zlock 0.000000000
1832023-12-06T20:36:42.647Zslp 0.000011920
1842023-12-06T20:36:42.647Zlat 0.000022899
1852023-12-06T20:36:42.647Zstop 0.000135952
1862023-12-06T20:36:42.647Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap
1872023-12-06T20:36:42.647Z+ chmod a+x tests/bootstrap
1882023-12-06T20:36:42.647Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1892023-12-06T20:36:42.647Z+ ptime -m gunzip
1902023-12-06T20:36:42.647Z++ basename /input/ci-tools/work/end-to-end-tests/end_to_end_tests-8c93950b566420c7
1912023-12-06T20:37:04.127Z
1922023-12-06T20:37:04.127Zreal 2.255090614
1932023-12-06T20:37:04.127Zuser 2.116239828
1942023-12-06T20:37:04.127Zsys 0.138481453
1952023-12-06T20:37:04.127Ztrap 0.000145680
1962023-12-06T20:37:04.127Ztflt 0.000000000
1972023-12-06T20:37:04.127Zdflt 0.000000000
1982023-12-06T20:37:04.127Zkflt 0.000000000
1992023-12-06T20:37:04.127Zlock 0.000000000
2002023-12-06T20:37:04.127Zslp 0.000021800
2012023-12-06T20:37:04.127Zlat 0.000017800
2022023-12-06T20:37:04.127Zstop 0.000115601
2032023-12-06T20:37:04.127Z++ basename /input/ci-tools/work/end-to-end-tests/end_to_end_tests-8c93950b566420c7
2042023-12-06T20:37:04.127Z+ chmod a+x tests/end_to_end_tests-8c93950b566420c7
2052023-12-06T20:37:04.132Z+ bmat address ls
2062023-12-06T20:37:04.293ZNAME CIDR FIRST LAST COUNT
2072023-12-06T20:37:04.293Zextra 10.151.2.0/24 10.151.2.164 10.151.2.227 64
2082023-12-06T20:37:04.293Z++ bmat address ls -f extra -Ho first,last,count
2092023-12-06T20:37:04.293Z+ set -- 10.151.2.164 10.151.2.227 64
2102023-12-06T20:37:04.293Z+ EXTRA_IP_START=10.151.2.164
2112023-12-06T20:37:04.293Z+ EXTRA_IP_END=10.151.2.227
2122023-12-06T20:37:04.293Z+ EXTRA_IP_COUNT=64
2132023-12-06T20:37:04.294Z+ (( EXTRA_IP_COUNT >= 32 ))
2142023-12-06T20:37:04.294Z+ EXTRA_IP_BASE=10.151.2
2152023-12-06T20:37:04.294Z+ EXTRA_IP_FOCTET=164
2162023-12-06T20:37:04.294Z+ EXTRA_IP_LOCTET=227
2172023-12-06T20:37:04.294Z+ SERVICE_IP_POOL_START=10.151.2.164
2182023-12-06T20:37:04.294Z+ SERVICE_IP_POOL_END=10.151.2.173
2192023-12-06T20:37:04.294Z+ DNS_IP1=10.151.2.164
2202023-12-06T20:37:04.294Z+ DNS_IP2=10.151.2.165
2212023-12-06T20:37:04.294Z+ UPLINK_IP=10.151.2.174
2222023-12-06T20:37:04.294Z+ IPPOOL_START=10.151.2.175
2232023-12-06T20:37:04.294Z+ IPPOOL_END=10.151.2.227
2242023-12-06T20:37:04.294Z++ ipadm show-addr -po type,addr
2252023-12-06T20:37:04.294Z++ awk '-F[:/]' '$1 == "dhcp" {print $2}'
2262023-12-06T20:37:04.294Z+ GATEWAY_IP=10.151.2.100
2272023-12-06T20:37:04.294Z+ [[ -n 10.151.2.100 ]]
2282023-12-06T20:37:04.294Z+ ping -s 10.151.2.100 56 1
2292023-12-06T20:37:04.294ZPING 10.151.2.100: 56 data bytes
2302023-12-06T20:37:04.294Z64 bytes from 10.151.2.100: icmp_seq=0. time=0.011 ms
2312023-12-06T20:37:04.294Z
2322023-12-06T20:37:04.295Z----10.151.2.100 PING Statistics----
2332023-12-06T20:37:04.295Z1 packets transmitted, 1 packets received, 0% packet loss
2342023-12-06T20:37:04.295Zround-trip (ms) min/avg/max/stddev = 0.011/0.011/0.011/-nan
2352023-12-06T20:37:04.295Z++ arp -an
2362023-12-06T20:37:04.295Z++ awk -vgw=10.151.2.100 '$2 == gw {print $NF}'
2372023-12-06T20:37:04.295Z+ GATEWAY_MAC=18:c0:4d:0d:a0:2a
2382023-12-06T20:37:04.295Z+ routeadm -e ipv4-forwarding -u
2392023-12-06T20:37:04.295Z+ PXA_START=10.151.2.164
2402023-12-06T20:37:04.295Z+ PXA_END=10.151.2.227
2412023-12-06T20:37:04.295Z+ export GATEWAY_IP GATEWAY_MAC PXA_START PXA_END
2422023-12-06T20:37:04.295Z+ pfexec zpool create -f scratch c1t1d0 c2t1d0
2432023-12-06T20:37:04.295Z+ ZPOOL_VDEV_DIR=/scratch
2442023-12-06T20:37:04.295Z+ ptime -m pfexec ./tools/create_virtual_hardware.sh
2452023-12-06T20:37:04.295Z+++ dirname ./tools/create_virtual_hardware.sh
2462023-12-06T20:37:04.296Z++ cd ./tools
2472023-12-06T20:37:04.296Z++ pwd
2482023-12-06T20:37:04.296Z+ SOURCE_DIR=/opt/oxide/work/tools
2492023-12-06T20:37:04.296Z+ OMICRON_TOP=/opt/oxide/work/tools/..
2502023-12-06T20:37:04.296Z+ SOFTNPU_MODE=zone
2512023-12-06T20:37:04.296Z+ . /opt/oxide/work/tools/virtual_hardware.sh
2522023-12-06T20:37:04.296Z++ MARKER=/etc/opt/oxide/NO_INSTALL
2532023-12-06T20:37:04.296Z++ [[ -f /etc/opt/oxide/NO_INSTALL ]]
2542023-12-06T20:37:04.296Z++ dladm show-phys -p -o LINK
2552023-12-06T20:37:04.297Z++ head -1
2562023-12-06T20:37:04.297Z+ PHYSICAL_LINK=igb0
2572023-12-06T20:37:04.297Z+ echo 'Using igb0 as physical link'
2582023-12-06T20:37:04.297ZUsing igb0 as physical link
2592023-12-06T20:37:04.297Z+ ensure_run_as_root
2602023-12-06T20:37:04.297Z++ id -u
2612023-12-06T20:37:04.297Z+ [[ 0 -ne 0 ]]
2622023-12-06T20:37:04.297Z+ ensure_zpools
2632023-12-06T20:37:04.297Z+ ZPOOL_TYPES=('oxp_' 'oxi_')
2642023-12-06T20:37:04.297Z+ for ZPOOL_TYPE in "${ZPOOL_TYPES[@]}"
2652023-12-06T20:37:04.297Z+ readarray -t ZPOOLS
2662023-12-06T20:37:04.297Z++ grep '"oxp_' /opt/oxide/work/tools/../smf/sled-agent/non-gimlet/config.toml
2672023-12-06T20:37:04.297Z++ sed 's/[ ",]//g'
2682023-12-06T20:37:04.297Z+ for ZPOOL in "${ZPOOLS[@]}"
2692023-12-06T20:37:04.297Z+ echo 'Zpool: [oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b]'
2702023-12-06T20:37:04.297ZZpool: [oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b]
2712023-12-06T20:37:04.297Z+ VDEV_PATH=/scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
2722023-12-06T20:37:04.297Z+ [[ -f /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
2732023-12-06T20:37:04.297Z+ dd if=/dev/zero of=/scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
2742023-12-06T20:37:04.545Z0+0 records in
2752023-12-06T20:37:04.545Z0+0 records out
2762023-12-06T20:37:04.545Z0 bytes (0 iB) transferred in 0.000007 secs (0 iB/sec)
2772023-12-06T20:37:04.546Z+ success 'ZFS vdev /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
2782023-12-06T20:37:04.546Z+ set +x
2792023-12-06T20:37:04.546ZZFS vdev /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
2802023-12-06T20:37:04.546Z++ zpool list -o name
2812023-12-06T20:37:04.546Z++ grep oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b
2822023-12-06T20:37:04.546Z+ [[ -z '' ]]
2832023-12-06T20:37:04.546Z+ zpool create -f oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
2842023-12-06T20:37:04.546Z+ success 'ZFS zpool oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
2852023-12-06T20:37:04.546Z+ set +x
2862023-12-06T20:37:04.546ZZFS zpool oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b exists
2872023-12-06T20:37:04.546Z+ for ZPOOL in "${ZPOOLS[@]}"
2882023-12-06T20:37:04.546Z+ echo 'Zpool: [oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03]'
2892023-12-06T20:37:04.546ZZpool: [oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03]
2902023-12-06T20:37:04.546Z+ VDEV_PATH=/scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
2912023-12-06T20:37:04.546Z+ [[ -f /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
2922023-12-06T20:37:04.546Z+ dd if=/dev/zero of=/scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
2932023-12-06T20:37:04.546Z0+0 records in
2942023-12-06T20:37:04.546Z0+0 records out
2952023-12-06T20:37:04.546Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
2962023-12-06T20:37:04.547Z+ success 'ZFS vdev /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
2972023-12-06T20:37:04.547Z+ set +x
2982023-12-06T20:37:04.547ZZFS vdev /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
2992023-12-06T20:37:04.547Z++ zpool list -o name
3002023-12-06T20:37:04.547Z++ grep oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03
3012023-12-06T20:37:04.547Z+ [[ -z '' ]]
3022023-12-06T20:37:04.547Z+ zpool create -f oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3032023-12-06T20:37:04.547Z+ success 'ZFS zpool oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3042023-12-06T20:37:04.547Z+ set +x
3052023-12-06T20:37:04.547ZZFS zpool oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
3062023-12-06T20:37:04.547Z+ for ZPOOL in "${ZPOOLS[@]}"
3072023-12-06T20:37:04.547Z+ echo 'Zpool: [oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3082023-12-06T20:37:04.547ZZpool: [oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03]
3092023-12-06T20:37:04.547Z+ VDEV_PATH=/scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3102023-12-06T20:37:04.547Z+ [[ -f /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3112023-12-06T20:37:04.547Z+ dd if=/dev/zero of=/scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3122023-12-06T20:37:04.547Z0+0 records in
3132023-12-06T20:37:04.547Z0+0 records out
3142023-12-06T20:37:04.547Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3152023-12-06T20:37:04.547Z+ success 'ZFS vdev /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3162023-12-06T20:37:04.547Z+ set +x
3172023-12-06T20:37:04.547ZZFS vdev /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
3182023-12-06T20:37:04.549Z++ zpool list -o name
3192023-12-06T20:37:04.549Z++ grep oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03
3202023-12-06T20:37:04.549Z+ [[ -z '' ]]
3212023-12-06T20:37:04.549Z+ zpool create -f oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3222023-12-06T20:37:04.549Z+ success 'ZFS zpool oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3232023-12-06T20:37:04.549Z+ set +x
3242023-12-06T20:37:04.550ZZFS zpool oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
3252023-12-06T20:37:04.550Z+ for ZPOOL in "${ZPOOLS[@]}"
3262023-12-06T20:37:04.550ZZpool: [oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03]
3272023-12-06T20:37:04.550Z+ echo 'Zpool: [oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3282023-12-06T20:37:04.550Z+ VDEV_PATH=/scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3292023-12-06T20:37:04.550Z+ [[ -f /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3302023-12-06T20:37:04.550Z+ dd if=/dev/zero of=/scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3312023-12-06T20:37:04.551Z0+0 records in
3322023-12-06T20:37:04.551Z0+0 records out
3332023-12-06T20:37:04.551Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3342023-12-06T20:37:04.551Z+ success 'ZFS vdev /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3352023-12-06T20:37:04.551Z+ set +x
3362023-12-06T20:37:04.551ZZFS vdev /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
3372023-12-06T20:37:04.551Z++ zpool list -o name
3382023-12-06T20:37:04.551Z++ grep oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03
3392023-12-06T20:37:04.551Z+ [[ -z '' ]]
3402023-12-06T20:37:04.551Z+ zpool create -f oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3412023-12-06T20:37:04.551Z+ success 'ZFS zpool oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3422023-12-06T20:37:04.552Z+ set +x
3432023-12-06T20:37:04.552ZZFS zpool oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
3442023-12-06T20:37:04.552Z+ for ZPOOL in "${ZPOOLS[@]}"
3452023-12-06T20:37:04.552Z+ echo 'Zpool: [oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3462023-12-06T20:37:04.552ZZpool: [oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03]
3472023-12-06T20:37:04.552Z+ VDEV_PATH=/scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3482023-12-06T20:37:04.552Z+ [[ -f /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3492023-12-06T20:37:04.552Z+ dd if=/dev/zero of=/scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3502023-12-06T20:37:04.552Z0+0 records in
3512023-12-06T20:37:04.552Z0+0 records out
3522023-12-06T20:37:04.552Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3532023-12-06T20:37:04.553Z+ success 'ZFS vdev /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3542023-12-06T20:37:04.553Z+ set +x
3552023-12-06T20:37:04.553ZZFS vdev /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
3562023-12-06T20:37:04.553Z++ zpool list -o name
3572023-12-06T20:37:04.553Z++ grep oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03
3582023-12-06T20:37:04.789Z+ [[ -z '' ]]
3592023-12-06T20:37:04.789Z+ zpool create -f oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3602023-12-06T20:37:04.790Z+ success 'ZFS zpool oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3612023-12-06T20:37:04.790Z+ set +x
3622023-12-06T20:37:04.790ZZFS zpool oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
3632023-12-06T20:37:04.790Z+ for ZPOOL in "${ZPOOLS[@]}"
3642023-12-06T20:37:04.790Z+ echo 'Zpool: [oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930]'
3652023-12-06T20:37:04.790ZZpool: [oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930]
3662023-12-06T20:37:04.790Z+ VDEV_PATH=/scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev
3672023-12-06T20:37:04.790Z+ [[ -f /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev ]]
3682023-12-06T20:37:04.790Z+ dd if=/dev/zero of=/scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev bs=1 count=0 seek=20G
3692023-12-06T20:37:04.790Z0+0 records in
3702023-12-06T20:37:04.790Z0+0 records out
3712023-12-06T20:37:04.791Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
3722023-12-06T20:37:04.791Z+ success 'ZFS vdev /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev exists'
3732023-12-06T20:37:04.791Z+ set +x
3742023-12-06T20:37:04.791ZZFS vdev /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev exists
3752023-12-06T20:37:04.791Z++ zpool list -o name
3762023-12-06T20:37:04.791Z++ grep oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930
3772023-12-06T20:37:04.791Z+ [[ -z '' ]]
3782023-12-06T20:37:04.791Z+ zpool create -f oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev
3792023-12-06T20:37:04.791Z+ success 'ZFS zpool oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 exists'
3802023-12-06T20:37:04.791Z+ set +x
3812023-12-06T20:37:04.791ZZFS zpool oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 exists
3822023-12-06T20:37:04.791Z+ for ZPOOL in "${ZPOOLS[@]}"
3832023-12-06T20:37:04.791Z+ echo 'Zpool: [oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60]'
3842023-12-06T20:37:04.791ZZpool: [oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60]
3852023-12-06T20:37:04.791Z+ VDEV_PATH=/scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev
3862023-12-06T20:37:04.791Z+ [[ -f /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev ]]
3872023-12-06T20:37:04.791Z+ dd if=/dev/zero of=/scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev bs=1 count=0 seek=20G
3882023-12-06T20:37:04.791Z0+0 records in
3892023-12-06T20:37:04.791Z0+0 records out
3902023-12-06T20:37:04.792Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3912023-12-06T20:37:04.792Z+ success 'ZFS vdev /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev exists'
3922023-12-06T20:37:04.792Z+ set +x
3932023-12-06T20:37:04.792ZZFS vdev /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev exists
3942023-12-06T20:37:04.792Z++ zpool list -o name
3952023-12-06T20:37:04.792Z++ grep oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60
3962023-12-06T20:37:04.792Z+ [[ -z '' ]]
3972023-12-06T20:37:04.792Z+ zpool create -f oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev
3982023-12-06T20:37:04.792Z+ success 'ZFS zpool oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 exists'
3992023-12-06T20:37:04.792Z+ set +x
4002023-12-06T20:37:04.792ZZFS zpool oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 exists
4012023-12-06T20:37:04.792Z+ for ZPOOL in "${ZPOOLS[@]}"
4022023-12-06T20:37:04.792Z+ echo 'Zpool: [oxp_31bd71cd-4736-4a12-a387-9b74b050396f]'
4032023-12-06T20:37:04.792ZZpool: [oxp_31bd71cd-4736-4a12-a387-9b74b050396f]
4042023-12-06T20:37:04.792Z+ VDEV_PATH=/scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev
4052023-12-06T20:37:04.792Z+ [[ -f /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev ]]
4062023-12-06T20:37:04.792Z+ dd if=/dev/zero of=/scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev bs=1 count=0 seek=20G
4072023-12-06T20:37:04.792Z0+0 records in
4082023-12-06T20:37:04.792Z0+0 records out
4092023-12-06T20:37:04.792Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4102023-12-06T20:37:04.792Z+ success 'ZFS vdev /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev exists'
4112023-12-06T20:37:04.792Z+ set +x
4122023-12-06T20:37:04.792ZZFS vdev /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev exists
4132023-12-06T20:37:04.793Z++ zpool list -o name
4142023-12-06T20:37:04.793Z++ grep oxp_31bd71cd-4736-4a12-a387-9b74b050396f
4152023-12-06T20:37:04.793Z+ [[ -z '' ]]
4162023-12-06T20:37:04.793Z+ zpool create -f oxp_31bd71cd-4736-4a12-a387-9b74b050396f /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev
4172023-12-06T20:37:04.793Z+ success 'ZFS zpool oxp_31bd71cd-4736-4a12-a387-9b74b050396f exists'
4182023-12-06T20:37:04.793Z+ set +x
4192023-12-06T20:37:04.793ZZFS zpool oxp_31bd71cd-4736-4a12-a387-9b74b050396f exists
4202023-12-06T20:37:04.793Z+ for ZPOOL in "${ZPOOLS[@]}"
4212023-12-06T20:37:04.793Z+ echo 'Zpool: [oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7]'
4222023-12-06T20:37:04.794ZZpool: [oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7]
4232023-12-06T20:37:04.794Z+ VDEV_PATH=/scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev
4242023-12-06T20:37:04.795Z+ [[ -f /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev ]]
4252023-12-06T20:37:04.795Z+ dd if=/dev/zero of=/scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev bs=1 count=0 seek=20G
4262023-12-06T20:37:04.795Z0+0 records in
4272023-12-06T20:37:04.795Z0+0 records out
4282023-12-06T20:37:04.795Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
4292023-12-06T20:37:04.795Z+ success 'ZFS vdev /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev exists'
4302023-12-06T20:37:04.795Z+ set +x
4312023-12-06T20:37:04.795ZZFS vdev /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev exists
4322023-12-06T20:37:04.795Z++ zpool list -o name
4332023-12-06T20:37:04.795Z++ grep oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7
4342023-12-06T20:37:04.795Z+ [[ -z '' ]]
4352023-12-06T20:37:04.795Z+ zpool create -f oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev
4362023-12-06T20:37:04.795Z+ success 'ZFS zpool oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 exists'
4372023-12-06T20:37:04.795Z+ set +x
4382023-12-06T20:37:04.795ZZFS zpool oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 exists
4392023-12-06T20:37:04.795Z+ for ZPOOL_TYPE in "${ZPOOL_TYPES[@]}"
4402023-12-06T20:37:04.795Z+ readarray -t ZPOOLS
4412023-12-06T20:37:04.795Z++ grep '"oxi_' /opt/oxide/work/tools/../smf/sled-agent/non-gimlet/config.toml
4422023-12-06T20:37:04.795Z++ sed 's/[ ",]//g'
4432023-12-06T20:37:04.795Z+ for ZPOOL in "${ZPOOLS[@]}"
4442023-12-06T20:37:04.795Z+ echo 'Zpool: [oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b]'
4452023-12-06T20:37:04.795ZZpool: [oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b]
4462023-12-06T20:37:04.795Z+ VDEV_PATH=/scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4472023-12-06T20:37:04.796Z+ [[ -f /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
4482023-12-06T20:37:04.796Z+ dd if=/dev/zero of=/scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
4492023-12-06T20:37:04.796Z0+0 records in
4502023-12-06T20:37:04.796Z0+0 records out
4512023-12-06T20:37:04.796Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
4522023-12-06T20:37:04.796Z+ success 'ZFS vdev /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
4532023-12-06T20:37:04.796Z+ set +x
4542023-12-06T20:37:04.796ZZFS vdev /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
4552023-12-06T20:37:04.796Z++ zpool list -o name
4562023-12-06T20:37:04.797Z++ grep oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b
4572023-12-06T20:37:04.797Z+ [[ -z '' ]]
4582023-12-06T20:37:04.926Z+ zpool create -f oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4592023-12-06T20:37:04.926Z+ success 'ZFS zpool oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
4602023-12-06T20:37:04.927Z+ set +x
4612023-12-06T20:37:04.927ZZFS zpool oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b exists
4622023-12-06T20:37:04.927ZZpool: [oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b]
4632023-12-06T20:37:04.927Z+ for ZPOOL in "${ZPOOLS[@]}"
4642023-12-06T20:37:04.927Z+ echo 'Zpool: [oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b]'
4652023-12-06T20:37:04.927Z+ VDEV_PATH=/scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4662023-12-06T20:37:04.927Z+ [[ -f /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
4672023-12-06T20:37:04.927Z+ dd if=/dev/zero of=/scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
4682023-12-06T20:37:04.927Z0+0 records in
4692023-12-06T20:37:04.927Z0+0 records out
4702023-12-06T20:37:04.927Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4712023-12-06T20:37:04.927Z+ success 'ZFS vdev /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
4722023-12-06T20:37:04.928Z+ set +x
4732023-12-06T20:37:04.928ZZFS vdev /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
4742023-12-06T20:37:04.928Z++ zpool list -o name
4752023-12-06T20:37:04.928Z++ grep oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b
4762023-12-06T20:37:04.928Z+ [[ -z '' ]]
4772023-12-06T20:37:04.928Z+ zpool create -f oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4782023-12-06T20:37:04.928Z+ success 'ZFS zpool oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
4792023-12-06T20:37:04.928Z+ set +x
4802023-12-06T20:37:04.928ZZFS zpool oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b exists
4812023-12-06T20:37:04.928Z+ [[ zone == \z\o\n\e ]]
4822023-12-06T20:37:04.928Z+ ensure_simulated_links igb0
4832023-12-06T20:37:04.928Z+ local PHYSICAL_LINK=igb0
4842023-12-06T20:37:04.928Z+ INDICES=("0" "1")
4852023-12-06T20:37:04.928Z+ for I in "${INDICES[@]}"
4862023-12-06T20:37:04.928Z++ get_simnet_name_if_exists net0
4872023-12-06T20:37:04.928Z++ dladm show-simnet -p -o LINK net0
4882023-12-06T20:37:04.928Z++ echo ''
4892023-12-06T20:37:04.928Z+ [[ -z '' ]]
4902023-12-06T20:37:04.928Z+ dladm create-simnet -t net0
4912023-12-06T20:37:04.928Z+ dladm create-simnet -t sc0_0
4922023-12-06T20:37:04.928Z+ dladm modify-simnet -t -p net0 sc0_0
4932023-12-06T20:37:04.928Z+ dladm set-linkprop -p mtu=9000 sc0_0
4942023-12-06T20:37:04.928Z+ success 'Simnet net0/sc0_0 exists'
4952023-12-06T20:37:04.929Z+ set +x
4962023-12-06T20:37:04.929Z+ for I in "${INDICES[@]}"
4972023-12-06T20:37:04.929ZSimnet net0/sc0_0 exists
4982023-12-06T20:37:04.929Z++ get_simnet_name_if_exists net1
4992023-12-06T20:37:04.930Z++ dladm show-simnet -p -o LINK net1
5002023-12-06T20:37:04.930Z++ echo ''
5012023-12-06T20:37:04.930Z+ [[ -z '' ]]
5022023-12-06T20:37:04.930Z+ dladm create-simnet -t net1
5032023-12-06T20:37:04.930Z+ dladm create-simnet -t sc1_0
5042023-12-06T20:37:04.930Z+ dladm modify-simnet -t -p net1 sc1_0
5052023-12-06T20:37:04.930Z+ dladm set-linkprop -p mtu=9000 sc1_0
5062023-12-06T20:37:04.930Z+ success 'Simnet net1/sc1_0 exists'
5072023-12-06T20:37:04.930Z+ set +x
5082023-12-06T20:37:04.930ZSimnet net1/sc1_0 exists
5092023-12-06T20:37:04.930Z++ get_vnic_name_if_exists sc0_1
5102023-12-06T20:37:04.931Z++ dladm show-vnic -p -o LINK sc0_1
5112023-12-06T20:37:04.931Z++ echo ''
5122023-12-06T20:37:04.931Z+ [[ -z '' ]]
5132023-12-06T20:37:04.931Z+ dladm create-vnic -t sc0_1 -l igb0 -m a8:e1:de:01:70:1d
5142023-12-06T20:37:05.045Z+ [[ -v PROMISC_FILT_OFF ]]
5152023-12-06T20:37:05.046Z+ success 'Vnic sc0_1 exists'
5162023-12-06T20:37:05.046Z+ set +x
5172023-12-06T20:37:05.047ZVnic sc0_1 exists
5182023-12-06T20:37:05.047Z+ warn_if_no_proxy_arp
5192023-12-06T20:37:05.047Z+ [[ -v PXA_START ]]
5202023-12-06T20:37:05.047Z+ [[ -v PXA_END ]]
5212023-12-06T20:37:05.047Z+ ensure_softnpu_zone
5222023-12-06T20:37:05.047Z+ zoneadm list
5232023-12-06T20:37:05.047Z+ grep -q sidecar_softnpu
5242023-12-06T20:37:05.047Z+ [[ -f out/npuzone/npuzone ]]
5252023-12-06T20:37:05.047Z+ out/npuzone/npuzone create sidecar --omicron-zone --ports sc0_0,tfportrear0_0 --ports sc0_1,tfportqsfp0_0 --sidecar-lite-commit f0585a29fb0285f7a1220c1118856b0e5c1f75c5 --softnpu-commit dec63e67156fe6e958991bbfa090629868115ab5
5262023-12-06T20:37:05.047Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/softnpu/image/dec63e67156fe6e958991bbfa090629868115ab5/softnpu.sha256.txt
5272023-12-06T20:37:13.426Zdownloading softnpu ...done
5282023-12-06T20:37:13.452Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/sidecar-lite/release/f0585a29fb0285f7a1220c1118856b0e5c1f75c5/libsidecar_lite.so.sha256.txt
5292023-12-06T20:37:20.140Zdownloading asic_program.so ...done
5302023-12-06T20:37:20.166Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/sidecar-lite/release/f0585a29fb0285f7a1220c1118856b0e5c1f75c5/scadm.sha256.txt
5312023-12-06T20:37:26.994Zdownloading scadm ...done
5322023-12-06T20:37:27.020Zcreating port tfportrear0_0 for sc0_0
5332023-12-06T20:37:27.020Zcreating port tfportqsfp0_0 for sc0_1
5342023-12-06T20:37:27.045Zsoftnpu zone setup
5352023-12-06T20:37:27.045Zconfigure zone
5362023-12-06T20:37:27.045Zinstall zone
5372023-12-06T20:37:28.124Zboot zone
5382023-12-06T20:37:31.247Z[sidecar_softnpu] svcs milestone/network
5392023-12-06T20:37:31.274ZSTATE STIME FMRI
5402023-12-06T20:37:31.274Zonline 20:37:29 svc:/milestone/network:default
5412023-12-06T20:37:31.274Z[sidecar_softnpu] /softnpu/npu start
5422023-12-06T20:37:31.300Z
5432023-12-06T20:37:31.300Z+ /opt/oxide/work/tools/scrimlet/softnpu-init.sh
5442023-12-06T20:37:31.300Z+ GATEWAY_IP=10.151.2.100
5452023-12-06T20:37:31.300Z+ echo 'Using 10.151.2.100 as gateway ip'
5462023-12-06T20:37:31.300ZUsing 10.151.2.100 as gateway ip
5472023-12-06T20:37:31.300Z+ [[ ! -v GATEWAY_MAC ]]
5482023-12-06T20:37:31.300Z+ GATEWAY_MAC=18:c0:4d:0d:a0:2a
5492023-12-06T20:37:31.300Z++ grep -c -E '^([0-9a-fA-F]{2}:){5}[0-9a-fA-F]{2}$' /dev/fd/63
5502023-12-06T20:37:31.300Z+++ echo 18:c0:4d:0d:a0:2a
5512023-12-06T20:37:31.300Z+ COUNT=1
5522023-12-06T20:37:31.300Z+ [[ 1 -ne 1 ]]
5532023-12-06T20:37:31.301Z+ echo 'Using 18:c0:4d:0d:a0:2a as gateway mac'
5542023-12-06T20:37:31.301ZUsing 18:c0:4d:0d:a0:2a as gateway mac
5552023-12-06T20:37:31.301Z+ z_scadm add-arp-entry 10.151.2.100 18:c0:4d:0d:a0:2a
5562023-12-06T20:37:31.301Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone add-arp-entry 10.151.2.100 18:c0:4d:0d:a0:2a
5572023-12-06T20:37:31.326Z+ PXA_MAC=a8:e1:de:01:70:1d
5582023-12-06T20:37:31.326Z+ [[ -v PXA_START ]]
5592023-12-06T20:37:31.327Z+ z_scadm add-proxy-arp 10.151.2.164 10.151.2.227 a8:e1:de:01:70:1d
5602023-12-06T20:37:31.327Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone add-proxy-arp 10.151.2.164 10.151.2.227 a8:e1:de:01:70:1d
5612023-12-06T20:37:31.353Z+ z_scadm dump-state
5622023-12-06T20:37:31.379Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone dump-state
5632023-12-06T20:37:31.379Zlocal v6:
5642023-12-06T20:37:31.379Zlocal v4:
5652023-12-06T20:37:31.379Zrouter v6:
5662023-12-06T20:37:31.379Zrouter v4_idx:
5672023-12-06T20:37:31.379Zrouter v4_routes:
5682023-12-06T20:37:31.379Zresolver v4:
5692023-12-06T20:37:31.379Z10.151.2.100 -> 18:c0:4d:0d:a0:2a
5702023-12-06T20:37:31.379Zresolver v6:
5712023-12-06T20:37:31.379Znat v4:
5722023-12-06T20:37:31.379Znat_v6:
5732023-12-06T20:37:31.379Zport_mac:
5742023-12-06T20:37:31.379Zproxy arp:
5752023-12-06T20:37:31.379Z10.151.2.164/10.151.2.227: a8:e1:de:01:70:1d
5762023-12-06T20:37:31.405Z+ success 'softnpu zone exists'
5772023-12-06T20:37:31.405Z+ set +x
5782023-12-06T20:37:31.405Zsoftnpu zone exists
5792023-12-06T20:37:31.405Z
5802023-12-06T20:37:31.405Zreal 27.332035382
5812023-12-06T20:37:31.405Zuser 0.715544907
5822023-12-06T20:37:31.406Zsys 1.612017191
5832023-12-06T20:37:31.406Ztrap 0.000082189
5842023-12-06T20:37:31.406Ztflt 0.000088188
5852023-12-06T20:37:31.406Zdflt 0.001170677
5862023-12-06T20:37:31.406Zkflt 0.000000000
5872023-12-06T20:37:31.406Zlock 21:13.502812260
5882023-12-06T20:37:31.406Zslp 1:21.818518064
5892023-12-06T20:37:31.406Zlat 0.053436519
5902023-12-06T20:37:31.406Zstop 0.366420970
5912023-12-06T20:37:31.406Z+ tar xf out/omicron-sled-agent.tar pkg/config-rss.toml
5922023-12-06T20:37:31.436Z++ sed -n 's/silo_name = "\(.*\)"/\1/p' pkg/config-rss.toml
5932023-12-06T20:37:31.437Z+ SILO_NAME=recovery
5942023-12-06T20:37:31.437Z++ sed -n 's/external_dns_zone_name = "\(.*\)"/\1/p' pkg/config-rss.toml
5952023-12-06T20:37:31.437Z+ EXTERNAL_DNS_DOMAIN=oxide.test
5962023-12-06T20:37:31.437Z+ sed -i~ '
5972023-12-06T20:37:31.437Z /^external_dns_ips/c\
5982023-12-06T20:37:31.437Zexternal_dns_ips = [ "10.151.2.164", "10.151.2.165" ]
5992023-12-06T20:37:31.438Z /^\[\[internal_services_ip_pool_ranges/,/^$/ {
6002023-12-06T20:37:31.438Z /^first/c\
6012023-12-06T20:37:31.438Zfirst = "10.151.2.164"
6022023-12-06T20:37:31.438Z /^last/c\
6032023-12-06T20:37:31.438Zlast = "10.151.2.173"
6042023-12-06T20:37:31.438Z }
6052023-12-06T20:37:31.438Z /^\[rack_network_config/,/^$/ {
6062023-12-06T20:37:31.438Z /^infra_ip_first/c\
6072023-12-06T20:37:31.438Zinfra_ip_first = "10.151.2.174"
6082023-12-06T20:37:31.439Z /^infra_ip_last/c\
6092023-12-06T20:37:31.439Zinfra_ip_last = "10.151.2.174"
6102023-12-06T20:37:31.439Z }
6112023-12-06T20:37:31.440Z /^\[\[rack_network_config.ports/,/^$/ {
6122023-12-06T20:37:31.441Z /^routes/c\
6132023-12-06T20:37:31.441Zroutes = \[{nexthop = "10.151.2.100", destination = "0.0.0.0/0"}\]
6142023-12-06T20:37:31.441Z /^addresses/c\
6152023-12-06T20:37:31.441Zaddresses = \["10.151.2.174/32"\]
6162023-12-06T20:37:31.441Z }
6172023-12-06T20:37:31.441Z' pkg/config-rss.toml
6182023-12-06T20:37:31.442Z+ diff -u pkg/config-rss.toml~ pkg/config-rss.toml
6192023-12-06T20:37:31.442Z--- pkg/config-rss.toml~ Mon Jul 24 01:21:28 2006
6202023-12-06T20:37:31.442Z+++ pkg/config-rss.toml Wed Dec 6 20:37:31 2023
6212023-12-06T20:37:31.442Z@@ -41,7 +41,7 @@
6222023-12-06T20:37:31.443Z # the DNS domain delegated to the rack by the customer. Each of these addresses
6232023-12-06T20:37:31.443Z # must be contained in one of the "internal services" IP Pool ranges listed
6242023-12-06T20:37:31.443Z # below.
6252023-12-06T20:37:31.443Z-external_dns_ips = [ "192.168.1.20", "192.168.1.21" ]
6262023-12-06T20:37:31.443Z+external_dns_ips = [ "10.151.2.164", "10.151.2.165" ]
6272023-12-06T20:37:31.443Z
6282023-12-06T20:37:31.443Z # Initial TLS certificates for the external API
6292023-12-06T20:37:31.443Z #
6302023-12-06T20:37:31.444Z@@ -77,8 +77,8 @@
6312023-12-06T20:37:31.444Z #
6322023-12-06T20:37:31.444Z # For more on this and what to put here, see docs/how-to-run.adoc.
6332023-12-06T20:37:31.444Z [[internal_services_ip_pool_ranges]]
6342023-12-06T20:37:31.445Z-first = "192.168.1.20"
6352023-12-06T20:37:31.445Z-last = "192.168.1.29"
6362023-12-06T20:37:31.445Z+first = "10.151.2.164"
6372023-12-06T20:37:31.445Z+last = "10.151.2.173"
6382023-12-06T20:37:31.445Z
6392023-12-06T20:37:31.445Z # TODO - this configuration is subject to change going forward. Ultimately these
6402023-12-06T20:37:31.446Z # parameters should be provided to the control plane via wicket, but we need to
6412023-12-06T20:37:31.447Z@@ -92,8 +92,8 @@
6422023-12-06T20:37:31.447Z # A range of IP addresses used by Boundary Services on the external network. In
6432023-12-06T20:37:31.447Z # a real system, these would be addresses of the uplink ports on the Sidecar.
6442023-12-06T20:37:31.447Z # With softnpu, only one address is used.
6452023-12-06T20:37:31.447Z-infra_ip_first = "192.168.1.30"
6462023-12-06T20:37:31.447Z-infra_ip_last = "192.168.1.30"
6472023-12-06T20:37:31.448Z+infra_ip_first = "10.151.2.174"
6482023-12-06T20:37:31.448Z+infra_ip_last = "10.151.2.174"
6492023-12-06T20:37:31.448Z
6502023-12-06T20:37:31.448Z # Configurations for BGP routers to run on the scrimlets.
6512023-12-06T20:37:31.448Z bgp = []
6522023-12-06T20:37:31.448Z@@ -101,9 +101,9 @@
6532023-12-06T20:37:31.449Z # You can configure multiple uplinks by repeating the following stanza
6542023-12-06T20:37:31.449Z [[rack_network_config.ports]]
6552023-12-06T20:37:31.449Z # Routes associated with this port.
6562023-12-06T20:37:31.449Z-routes = [{nexthop = "192.168.1.199", destination = "0.0.0.0/0"}]
6572023-12-06T20:37:31.449Z+routes = [{nexthop = "10.151.2.100", destination = "0.0.0.0/0"}]
6582023-12-06T20:37:31.449Z # Addresses associated with this port.
6592023-12-06T20:37:31.449Z-addresses = ["192.168.1.30/32"]
6602023-12-06T20:37:31.449Z+addresses = ["10.151.2.174/32"]
6612023-12-06T20:37:31.450Z # Name of the uplink port. This should always be "qsfp0" when using softnpu.
6622023-12-06T20:37:31.450Z port = "qsfp0"
6632023-12-06T20:37:31.450Z # The speed of this port.
6642023-12-06T20:37:31.450Z+ true
6652023-12-06T20:37:31.450Z+ tar rvf out/omicron-sled-agent.tar pkg/config-rss.toml
6662023-12-06T20:37:31.450Za pkg/config-rss.toml 7K
6672023-12-06T20:37:31.451Z+ rm -f pkg/config-rss.toml pkg/config-rss.toml~
6682023-12-06T20:37:31.451Z+ TLS_NAME=recovery.sys.oxide.test
6692023-12-06T20:37:31.451Z+ openssl req -newkey rsa:4096 -x509 -sha256 -days 3 -nodes -out pkg/initial-tls-cert.pem -keyout pkg/initial-tls-key.pem -subj /CN=recovery.sys.oxide.test -addext subjectAltName=DNS:recovery.sys.oxide.test -addext basicConstraints=critical,CA:FALSE -config /dev/stdin
6702023-12-06T20:37:31.615Z....+.+.....+............+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.+.....+.+...............+...........+...+.+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.....+....+..+..........+..+..........+............+...........+...+.........+.......+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
6712023-12-06T20:37:32.785Z.+.+.....+......+.......+.....+.+..+....+...........+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.+.....+..........+.........+.....+.......+......+..+...+....+.....+...............+.+...+.....+.+...+..+...+...+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.....+.....+..........+.....+.+..+...+.+..............+...............+.+........+...+............................+............+.........+........+......+................+..............+......+.+........................+...+......+..+..........+....................+............+.+.........+..+.+........+......+...+.+.........+...........+....+...............+..+.+.....+....+...............+......+..+...+...+......+.+........+.......+...+..+.........................+........+.......+........................+...+......+..........................+......+....+........+..........+...........+.........+...+....+...+......+......+..................+.....+.............+.....+....+.....+.+..+...+..........+...+.....+......+....+..+....+...+......+......+...........+.......+............+.....+.+............+...+.....+.........+...................+..............+...+..........+............+.....+...........................+......+.......+..+................+............+..+....+...............+..+..........+..+.....................+.+...........+...+.+......+.....+.........+....+..............+...............+..........+..+...+.......+...............+...+............+...+..+....+.....+......+.+.....+.........+..................+......+.+.......................+......+.+...+............+.........+..+..........+..+.......+...........+...+.+......+........+....+............+.....+......+.......+..+.+........+..........+.....+.......+.........+...+....................+......+....+..+....+.....+.........+...+...+.+.....................+............+........+.............+..+..........+.........+.........+......+....................+......+...+....+..+.........+......+.+...+....................+.......+...+.........+...+..+......+.+..............+.....................+.+........+....+..+..........+.........+.........+.....+...+....+........+.+......+......+..+...+............+.......+..................+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
6722023-12-06T20:37:32.821Z-----
6732023-12-06T20:37:32.821Z+ tar rvf out/omicron-sled-agent.tar pkg/initial-tls-cert.pem pkg/initial-tls-key.pem
6742023-12-06T20:37:33.333Za pkg/initial-tls-cert.pem 2K
6752023-12-06T20:37:33.333Za pkg/initial-tls-key.pem 4K
6762023-12-06T20:37:33.333Z+ rm -f pkg/initial-tls-cert.pem pkg/initial-tls-key.pem
6772023-12-06T20:37:33.333Z+ rmdir pkg
6782023-12-06T20:37:33.333Z+ E2E_TLS_CERT=/opt/oxide/sled-agent/pkg/initial-tls-cert.pem
6792023-12-06T20:37:33.333Z+ pfexec mkdir -p /usr/oxide
6802023-12-06T20:37:33.333Z+ pfexec rm -f /usr/oxide/tcpproxy
6812023-12-06T20:37:33.333Z+ pfexec curl -sSfL -o /usr/oxide/tcpproxy http://catacomb.eng.oxide.computer:12346/tcpproxy
6822023-12-06T20:37:33.335Z+ pfexec chmod +x /usr/oxide/tcpproxy
6832023-12-06T20:37:33.335Z+ pfexec rm -f /var/svc/manifest/site/tcpproxy.xml
6842023-12-06T20:37:33.335Z+ pfexec curl -sSfL -o /var/svc/manifest/site/tcpproxy.xml http://catacomb.eng.oxide.computer:12346/tcpproxy.xml
6852023-12-06T20:37:33.335Z+ pfexec svccfg import /var/svc/manifest/site/tcpproxy.xml
6862023-12-06T20:37:33.335Z+ OMICRON_NO_UNINSTALL=1
6872023-12-06T20:37:33.335Z+ ptime -m pfexec ./target/release/omicron-package -t test install
6882023-12-06T20:37:33.335ZDec 06 20:37:33.518 DEBG target[test]: Target({"image": "standard", "machine": "non-gimlet", "rack-topology": "single-sled", "switch": "softnpu"})
6892023-12-06T20:37:33.335ZDec 06 20:37:33.520 INFO Installing service, dst: /opt/oxide/crucible_pantry.tar.gz, src: out/crucible-pantry.tar.gz
6902023-12-06T20:37:33.335ZDec 06 20:37:33.520 INFO Installing service, dst: /opt/oxide/clickhouse.tar.gz, src: out/clickhouse.tar.gz
6912023-12-06T20:37:33.335ZDec 06 20:37:33.520 INFO Installing service, dst: /opt/oxide/internal_dns.tar.gz, src: out/internal-dns.tar.gz
6922023-12-06T20:37:33.335ZDec 06 20:37:33.520 INFO Installing service, dst: /opt/oxide/overlay.tar.gz, src: out/overlay.tar.gz
6932023-12-06T20:37:33.335ZDec 06 20:37:33.520 INFO Installing service, dst: /opt/oxide/propolis-server.tar.gz, src: out/propolis-server.tar.gz
6942023-12-06T20:37:33.335ZDec 06 20:37:33.520 INFO Installing service, dst: /opt/oxide/ntp.tar.gz, src: out/ntp.tar.gz
6952023-12-06T20:37:33.335ZDec 06 20:37:33.520 INFO Installing service, dst: /opt/oxide/oximeter.tar.gz, src: out/oximeter-collector.tar.gz
6962023-12-06T20:37:33.335ZDec 06 20:37:33.520 INFO Installing service, dst: /opt/oxide/crucible.tar.gz, src: out/crucible.tar.gz
6972023-12-06T20:37:33.336ZDec 06 20:37:33.520 INFO Installing service, dst: /opt/oxide/external_dns.tar.gz, src: out/external-dns.tar.gz
6982023-12-06T20:37:33.336ZDec 06 20:37:33.520 INFO Installing service, dst: /opt/oxide/clickhouse_keeper.tar.gz, src: out/clickhouse_keeper.tar.gz
6992023-12-06T20:37:33.336ZDec 06 20:37:33.520 INFO Installing service, dst: /opt/oxide/cockroachdb.tar.gz, src: out/cockroachdb.tar.gz
7002023-12-06T20:37:33.336ZDec 06 20:37:33.520 INFO Installing service, dst: /opt/oxide/switch.tar.gz, src: out/switch-softnpu.tar.gz
7012023-12-06T20:37:33.337ZDec 06 20:37:33.521 INFO Installing service, dst: /opt/oxide/nexus.tar.gz, src: out/omicron-nexus.tar.gz
7022023-12-06T20:37:33.337ZDec 06 20:37:33.521 INFO Installing service, dst: /opt/oxide/mg-ddm.tar, src: out/mg-ddm-gz.tar
7032023-12-06T20:37:33.337ZDec 06 20:37:33.521 INFO Installing service, dst: /opt/oxide/mgs.tar.gz, src: out/omicron-gateway-softnpu.tar.gz
7042023-12-06T20:37:33.337ZDec 06 20:37:33.521 INFO Installing service, dst: /opt/oxide/sled-agent.tar, src: out/omicron-sled-agent.tar
7052023-12-06T20:37:33.383ZDec 06 20:37:33.756 INFO Unpacking service tarball, service_path: /opt/oxide/mg-ddm, tar_path: /opt/oxide/mg-ddm.tar
7062023-12-06T20:37:33.421ZDec 06 20:37:33.795 INFO Unpacking service tarball, service_path: /opt/oxide/sled-agent, tar_path: /opt/oxide/sled-agent.tar
7072023-12-06T20:37:33.484ZDec 06 20:37:33.855 INFO Installing bootstrap service from /opt/oxide/sled-agent/pkg/manifest.xml
7082023-12-06T20:37:33.612Z
7092023-12-06T20:37:33.639Zreal 0.476295130
7102023-12-06T20:37:33.639Zuser 0.098424564
7112023-12-06T20:37:33.639Zsys 1.892767938
7122023-12-06T20:37:33.639Ztrap 0.000002609
7132023-12-06T20:37:33.639Ztflt 0.000058408
7142023-12-06T20:37:33.639Zdflt 0.000479921
7152023-12-06T20:37:33.639Zkflt 0.000000000
7162023-12-06T20:37:33.639Zlock 42.809513121
7172023-12-06T20:37:33.639Zslp 0.689816883
7182023-12-06T20:37:33.639Zlat 0.003180327
7192023-12-06T20:37:33.639Zstop 0.219553809
7202023-12-06T20:37:33.639Z+ retry=0
7212023-12-06T20:37:33.639Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7222023-12-06T20:37:33.639Z+ [[ 0 -gt 30 ]]
7232023-12-06T20:37:33.639Z+ sleep 1
7242023-12-06T20:37:34.627Z+ retry=1
7252023-12-06T20:37:34.627Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7262023-12-06T20:37:34.652Z+ [[ 1 -gt 30 ]]
7272023-12-06T20:37:34.652Z+ sleep 1
7282023-12-06T20:37:35.641Z+ retry=2
7292023-12-06T20:37:35.641Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7302023-12-06T20:37:35.667Z+ [[ 2 -gt 30 ]]
7312023-12-06T20:37:35.667Z+ sleep 1
7322023-12-06T20:37:36.659Z+ retry=3
7332023-12-06T20:37:36.659Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7342023-12-06T20:37:36.686Z+ [[ 3 -gt 30 ]]
7352023-12-06T20:37:36.687Z+ sleep 1
7362023-12-06T20:37:37.669Z+ retry=4
7372023-12-06T20:37:37.669Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7382023-12-06T20:37:57.528Z+ [[ 4 -gt 30 ]]
7392023-12-06T20:37:57.528Z+ sleep 1
7402023-12-06T20:37:58.531Z+ retry=5
7412023-12-06T20:37:58.531Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7422023-12-06T20:37:58.561Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone dump-state
7432023-12-06T20:37:58.587Zlocal v6:
7442023-12-06T20:37:58.587Zfd00:99::1
7452023-12-06T20:37:58.587Zfe80::dcfd:25ff:fe33:3d9f
7462023-12-06T20:37:58.587Zlocal v4:
7472023-12-06T20:37:58.588Z10.151.2.174
7482023-12-06T20:37:58.588Zrouter v6:
7492023-12-06T20:37:58.588Zrouter v4_idx:
7502023-12-06T20:37:58.588Z0.0.0.0/0 -> 0
7512023-12-06T20:37:58.588Zrouter v4_routes:
7522023-12-06T20:37:58.588Z0 -> 10.151.2.100 (2)
7532023-12-06T20:37:58.588Zresolver v4:
7542023-12-06T20:37:58.588Z10.151.2.100 -> 18:c0:4d:0d:a0:2a
7552023-12-06T20:37:58.588Zresolver v6:
7562023-12-06T20:37:58.588Zfe80::f402:45ff:febe:44b3 -> f6:02:45:be:44:b3
7572023-12-06T20:37:58.588Zfe80::dcfd:25ff:fe33:3d9f -> de:fd:25:33:3d:9f
7582023-12-06T20:37:58.589Znat v4:
7592023-12-06T20:37:58.589Znat_v6:
7602023-12-06T20:37:58.589Zport_mac:
7612023-12-06T20:37:58.590Z1000: a8:40:25:6a:9e:12
7622023-12-06T20:37:58.590Z1: a8:40:25:6a:9e:13
7632023-12-06T20:37:58.590Zproxy arp:
7642023-12-06T20:37:58.590Z10.151.2.164/10.151.2.227: a8:e1:de:01:70:1d
7652023-12-06T20:37:58.590Z+ retry=0
7662023-12-06T20:37:58.590Z+++ zoneadm list -n
7672023-12-06T20:37:58.590Z+++ grep oxz_ntp
7682023-12-06T20:37:58.590Z++ pfexec svcs -z -Hostate oxide/ntp
7692023-12-06T20:37:58.616Zsvcs: zone '-Hostate': entity not found
7702023-12-06T20:37:58.616Z++ true
7712023-12-06T20:37:58.616Z+ [[ '' != online ]]
7722023-12-06T20:37:58.616Z+ [[ 0 -gt 60 ]]
7732023-12-06T20:37:58.617Z+ sleep 1
7742023-12-06T20:37:59.594Z+ retry=1
7752023-12-06T20:37:59.620Z+++ zoneadm list -n
7762023-12-06T20:37:59.620Z+++ grep oxz_ntp
7772023-12-06T20:37:59.620Z++ pfexec svcs -z -Hostate oxide/ntp
7782023-12-06T20:37:59.620Zsvcs: zone '-Hostate': entity not found
7792023-12-06T20:37:59.620Z++ true
7802023-12-06T20:37:59.620Z+ [[ '' != online ]]
7812023-12-06T20:37:59.620Z+ [[ 1 -gt 60 ]]
7822023-12-06T20:37:59.620Z+ sleep 1
7832023-12-06T20:38:00.621Z+ retry=2
7842023-12-06T20:38:00.654Z+++ zoneadm list -n
7852023-12-06T20:38:00.654Z+++ grep oxz_ntp
7862023-12-06T20:38:00.654Z++ pfexec svcs -z -Hostate oxide/ntp
7872023-12-06T20:38:00.654Zsvcs: zone '-Hostate': entity not found
7882023-12-06T20:38:00.655Z++ true
7892023-12-06T20:38:00.655Z+ [[ '' != online ]]
7902023-12-06T20:38:00.655Z+ [[ 2 -gt 60 ]]
7912023-12-06T20:38:00.655Z+ sleep 1
7922023-12-06T20:38:01.640Z+ retry=3
7932023-12-06T20:38:01.667Z+++ zoneadm list -n
7942023-12-06T20:38:01.669Z+++ grep oxz_ntp
7952023-12-06T20:38:01.669Z++ pfexec svcs -z oxz_ntp_70c55001-ca75-4aaf-b438-7ef62dacb383 -Hostate oxide/ntp
7962023-12-06T20:38:01.695Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
7972023-12-06T20:38:01.695Z++ true
7982023-12-06T20:38:01.695Z+ [[ '' != online ]]
7992023-12-06T20:38:01.695Z+ [[ 3 -gt 60 ]]
8002023-12-06T20:38:01.695Z+ sleep 1
8012023-12-06T20:38:02.676Z+ retry=4
8022023-12-06T20:38:02.707Z+++ zoneadm list -n
8032023-12-06T20:38:02.707Z+++ grep oxz_ntp
8042023-12-06T20:38:02.707Z++ pfexec svcs -z oxz_ntp_70c55001-ca75-4aaf-b438-7ef62dacb383 -Hostate oxide/ntp
8052023-12-06T20:38:02.707Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
8062023-12-06T20:38:02.708Z++ true
8072023-12-06T20:38:02.712Z+ [[ '' != online ]]
8082023-12-06T20:38:02.712Z+ [[ 4 -gt 60 ]]
8092023-12-06T20:38:02.712Z+ sleep 1
8102023-12-06T20:38:03.703Z+ retry=5
8112023-12-06T20:38:03.730Z+++ zoneadm list -n
8122023-12-06T20:38:03.730Z+++ grep oxz_ntp
8132023-12-06T20:38:03.730Z++ pfexec svcs -z oxz_ntp_70c55001-ca75-4aaf-b438-7ef62dacb383 -Hostate oxide/ntp
8142023-12-06T20:38:03.730Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
8152023-12-06T20:38:03.731Z++ true
8162023-12-06T20:38:03.731Z+ [[ '' != online ]]
8172023-12-06T20:38:03.731Z+ [[ 5 -gt 60 ]]
8182023-12-06T20:38:03.731Z+ sleep 1
8192023-12-06T20:38:04.985Z+ retry=6
8202023-12-06T20:38:04.985Z+++ zoneadm list -n
8212023-12-06T20:38:04.985Z+++ grep oxz_ntp
8222023-12-06T20:38:04.985Z++ pfexec svcs -z oxz_ntp_70c55001-ca75-4aaf-b438-7ef62dacb383 -Hostate oxide/ntp
8232023-12-06T20:38:04.986Z+ [[ disabled != online ]]
8242023-12-06T20:38:04.986Z+ [[ 6 -gt 60 ]]
8252023-12-06T20:38:04.986Z+ sleep 1
8262023-12-06T20:38:05.754Z+ retry=7
8272023-12-06T20:38:05.781Z+++ zoneadm list -n
8282023-12-06T20:38:05.781Z+++ grep oxz_ntp
8292023-12-06T20:38:05.781Z++ pfexec svcs -z oxz_ntp_70c55001-ca75-4aaf-b438-7ef62dacb383 -Hostate oxide/ntp
8302023-12-06T20:38:05.781Z+ [[ disabled != online ]]
8312023-12-06T20:38:05.781Z+ [[ 7 -gt 60 ]]
8322023-12-06T20:38:05.781Z+ sleep 1
8332023-12-06T20:38:06.779Z+ retry=8
8342023-12-06T20:38:06.813Z+++ zoneadm list -n
8352023-12-06T20:38:06.813Z+++ grep oxz_ntp
8362023-12-06T20:38:06.813Z++ pfexec svcs -z oxz_ntp_70c55001-ca75-4aaf-b438-7ef62dacb383 -Hostate oxide/ntp
8372023-12-06T20:38:06.813Z+ [[ disabled != online ]]
8382023-12-06T20:38:06.813Z+ [[ 8 -gt 60 ]]
8392023-12-06T20:38:06.813Z+ sleep 1
8402023-12-06T20:38:07.799Z+ retry=9
8412023-12-06T20:38:07.827Z+++ zoneadm list -n
8422023-12-06T20:38:07.828Z+++ grep oxz_ntp
8432023-12-06T20:38:07.828Z++ pfexec svcs -z oxz_ntp_70c55001-ca75-4aaf-b438-7ef62dacb383 -Hostate oxide/ntp
8442023-12-06T20:38:07.831Z+ [[ disabled != online ]]
8452023-12-06T20:38:07.831Z+ [[ 9 -gt 60 ]]
8462023-12-06T20:38:07.831Z+ sleep 1
8472023-12-06T20:38:08.822Z+ retry=10
8482023-12-06T20:38:08.848Z+++ zoneadm list -n
8492023-12-06T20:38:08.848Z+++ grep oxz_ntp
8502023-12-06T20:38:08.848Z++ pfexec svcs -z oxz_ntp_70c55001-ca75-4aaf-b438-7ef62dacb383 -Hostate oxide/ntp
8512023-12-06T20:38:08.849Z+ [[ disabled != online ]]
8522023-12-06T20:38:08.849Z+ [[ 10 -gt 60 ]]
8532023-12-06T20:38:08.849Z+ sleep 1
8542023-12-06T20:38:09.844Z+ retry=11
8552023-12-06T20:38:09.870Z+++ zoneadm list -n
8562023-12-06T20:38:09.870Z+++ grep oxz_ntp
8572023-12-06T20:38:09.870Z++ pfexec svcs -z oxz_ntp_70c55001-ca75-4aaf-b438-7ef62dacb383 -Hostate oxide/ntp
8582023-12-06T20:38:09.870Z+ [[ disabled != online ]]
8592023-12-06T20:38:09.870Z+ [[ 11 -gt 60 ]]
8602023-12-06T20:38:09.870Z+ sleep 1
8612023-12-06T20:38:10.870Z+ retry=12
8622023-12-06T20:38:10.898Z+++ zoneadm list -n
8632023-12-06T20:38:10.898Z+++ grep oxz_ntp
8642023-12-06T20:38:10.898Z++ pfexec svcs -z oxz_ntp_70c55001-ca75-4aaf-b438-7ef62dacb383 -Hostate oxide/ntp
8652023-12-06T20:38:10.898Z+ [[ disabled != online ]]
8662023-12-06T20:38:10.898Z+ [[ 12 -gt 60 ]]
8672023-12-06T20:38:10.898Z+ sleep 1
8682023-12-06T20:38:11.887Z+ retry=13
8692023-12-06T20:38:11.913Z+++ zoneadm list -n
8702023-12-06T20:38:11.913Z+++ grep oxz_ntp
8712023-12-06T20:38:11.913Z++ pfexec svcs -z oxz_ntp_70c55001-ca75-4aaf-b438-7ef62dacb383 -Hostate oxide/ntp
8722023-12-06T20:38:11.913Z+ [[ disabled != online ]]
8732023-12-06T20:38:11.913Z+ [[ 13 -gt 60 ]]
8742023-12-06T20:38:11.913Z+ sleep 1
 
8752023-12-06T20:38:12.600Zsock console login: Dec 6 20:38:12 sock ip: ip: joining multicasts failed (4) on vopte0 - will use link layer broadcasts for multicast
 
8762023-12-06T20:38:12.907Z+ retry=14
8772023-12-06T20:38:12.933Z+++ zoneadm list -n
8782023-12-06T20:38:12.933Z+++ grep oxz_ntp
8792023-12-06T20:38:12.933Z++ pfexec svcs -z oxz_ntp_70c55001-ca75-4aaf-b438-7ef62dacb383 -Hostate oxide/ntp
8802023-12-06T20:38:12.933Z+ [[ online != online ]]
8812023-12-06T20:38:12.933Z+ echo 'Waited for chrony: 14s'
8822023-12-06T20:38:12.933Z+ export RUST_BACKTRACE=1
8832023-12-06T20:38:12.933ZWaited for chrony: 14s
8842023-12-06T20:38:12.933Z+ RUST_BACKTRACE=1
8852023-12-06T20:38:12.933Z+ export E2E_TLS_CERT IPPOOL_START IPPOOL_END
8862023-12-06T20:38:12.933Z+ ./tests/bootstrap
8872023-12-06T20:38:12.959ZUsing external DNS server at 10.151.2.164:53
8882023-12-06T20:38:12.959Z2023-12-06 20:38:13.331230274 UTC: attempting to log into API
8892023-12-06T20:38:28.006Z2023-12-06 20:38:27.922978714 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8902023-12-06T20:38:28.995Z2023-12-06 20:38:28.926816679 UTC: attempting to log into API
8912023-12-06T20:38:44.028Z2023-12-06 20:38:44.000140820 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8922023-12-06T20:38:45.029Z2023-12-06 20:38:45.004066750 UTC: attempting to log into API
8932023-12-06T20:39:00.064Z2023-12-06 20:39:00.046487836 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8942023-12-06T20:39:01.062Z2023-12-06 20:39:01.047246894 UTC: attempting to log into API
8952023-12-06T20:39:16.094Z2023-12-06 20:39:16.079760057 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8962023-12-06T20:39:17.097Z2023-12-06 20:39:17.081843040 UTC: attempting to log into API
8972023-12-06T20:39:32.128Z2023-12-06 20:39:32.113349332 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8982023-12-06T20:39:33.129Z2023-12-06 20:39:33.114474834 UTC: attempting to log into API
8992023-12-06T20:39:48.161Z2023-12-06 20:39:48.146496131 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9002023-12-06T20:39:49.163Z2023-12-06 20:39:49.147261829 UTC: attempting to log into API
9012023-12-06T20:40:04.194Z2023-12-06 20:40:04.178928613 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9022023-12-06T20:40:05.196Z2023-12-06 20:40:05.179992405 UTC: attempting to log into API
9032023-12-06T20:40:20.227Z2023-12-06 20:40:20.211664088 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9042023-12-06T20:40:21.227Z2023-12-06 20:40:21.212566918 UTC: attempting to log into API
9052023-12-06T20:40:36.260Z2023-12-06 20:40:36.247379496 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9062023-12-06T20:40:37.261Z2023-12-06 20:40:37.248480490 UTC: attempting to log into API
9072023-12-06T20:40:52.296Z2023-12-06 20:40:52.283240926 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9082023-12-06T20:40:53.297Z2023-12-06 20:40:53.284315118 UTC: attempting to log into API
9092023-12-06T20:41:08.384Z2023-12-06 20:41:08.325037381 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9102023-12-06T20:41:09.482Z2023-12-06 20:41:09.325834320 UTC: attempting to log into API
9112023-12-06T20:41:24.388Z2023-12-06 20:41:24.375881707 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9122023-12-06T20:41:25.443Z2023-12-06 20:41:25.376980831 UTC: attempting to log into API
9132023-12-06T20:41:40.502Z2023-12-06 20:41:40.425593165 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
 
9142023-12-06T20:41:41.224ZDec 6 20:41:40 sock ip: ip: joining multicasts failed (4) on vopte1 - will use link layer broadcasts for multicast
 
9152023-12-06T20:41:41.497Z2023-12-06 20:41:41.426698918 UTC: attempting to log into API
 
9162023-12-06T20:41:48.437ZDec 6 20:41:47 sok ip: ip: joining multicasts failed (4) on vopte2 - will use link layer broadcasts for multicast
9172023-12-06T20:41:54.290ZDec 6 20:41:53 sok ip: ip: joining multicasts failed (4) on vopte3 - will use link layer broadcasts for multicast
 
9182023-12-06T20:41:56.473Z2023-12-06 20:41:56.458666571 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9192023-12-06T20:41:57.474Z2023-12-06 20:41:57.460015677 UTC: attempting to log into API
 
9202023-12-06T20:42:00.422ZDec 6 20:42:00 sok ip: ip: joining multicasts failed (4) on vopte4 - will use link layer broadcasts for multicast
9212023-12-06T20:42:06.546ZDec 6 20:42:05 sock ip: ip: joining multicasts failed (4) on vopte5 - will use link layer broadcasts for multicast
 
9222023-12-06T20:42:12.528Z2023-12-06 20:42:12.491153082 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9232023-12-06T20:42:13.507Z2023-12-06 20:42:13.492223314 UTC: attempting to log into API
9242023-12-06T20:42:19.308Z2023-12-06 20:42:19.293194347 UTC: login succeeded
9252023-12-06T20:42:19.341Zwaiting for nexus to come up...
9262023-12-06T20:42:19.410Zcreating IP pool... 10.151.2.175 - 10.151.2.227
9272023-12-06T20:42:19.456Zensuring datasets are ready...
9282023-12-06T20:42:22.089Zlet's roll.
9292023-12-06T20:42:22.114Z+ rm ./tests/bootstrap
9302023-12-06T20:42:22.175Z+ for test_bin in tests/*
9312023-12-06T20:42:22.175Z+ ./tests/bootstrap-83d41f1b41d1fb89
9322023-12-06T20:42:22.200Z
9332023-12-06T20:42:22.200Zrunning 0 tests
9342023-12-06T20:42:22.200Z
9352023-12-06T20:42:22.200Ztest result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
9362023-12-06T20:42:22.200Z
9372023-12-06T20:42:22.200Z+ for test_bin in tests/*
9382023-12-06T20:42:22.200Z+ ./tests/end_to_end_tests-8c93950b566420c7
9392023-12-06T20:42:22.200Z
9402023-12-06T20:42:22.200Zrunning 2 tests
9412023-12-06T20:42:22.413Ztest no_spoof::no_spoof_here ... ok
9422023-12-06T20:43:22.198Ztest instance_launch::instance_launch has been running for over 60 seconds
9432023-12-06T20:45:19.468Ztest instance_launch::instance_launch ... ok
9442023-12-06T20:45:19.469Z
9452023-12-06T20:45:19.469Ztest result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 177.27s
9462023-12-06T20:45:19.469Z
9472023-12-06T20:45:19.495Z+ _exit_trap
9482023-12-06T20:45:19.496Z+ local status=0
9492023-12-06T20:45:19.496Z+ [[ 0 -eq 0 ]]
9502023-12-06T20:45:19.496Z+ exit 0
9512023-12-06T20:45:19.496Zprocess exited: duration 526509 ms, exit code 0
 
9522023-12-06T20:45:19.631Zfound 90 output files
9532023-12-06T20:45:19.631Zuploading: /var/svc/log/oxide-sled-agent:default.log (747753 bytes)
9542023-12-06T20:45:21.724Zuploaded: /var/svc/log/oxide-sled-agent:default.log
9552023-12-06T20:45:21.725Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_d05fad10-fc2f-4439-af5c-a5dd445dd0c9/root/var/svc/log/oxide-crucible-agent:default.log (0 bytes)
9562023-12-06T20:45:22.804Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_d05fad10-fc2f-4439-af5c-a5dd445dd0c9/root/var/svc/log/oxide-crucible-agent:default.log
9572023-12-06T20:45:22.842Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_d05fad10-fc2f-4439-af5c-a5dd445dd0c9/root/var/svc/log/oxide-crucible-agent:default.log.0 (3436 bytes)
9582023-12-06T20:45:23.914Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_d05fad10-fc2f-4439-af5c-a5dd445dd0c9/root/var/svc/log/oxide-crucible-agent:default.log.0
9592023-12-06T20:45:23.914Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_d05fad10-fc2f-4439-af5c-a5dd445dd0c9/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
9602023-12-06T20:45:25.397Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_d05fad10-fc2f-4439-af5c-a5dd445dd0c9/root/var/svc/log/oxide-crucible-downstairs:default.log
9612023-12-06T20:45:25.397Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_d05fad10-fc2f-4439-af5c-a5dd445dd0c9/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
9622023-12-06T20:45:26.505Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_d05fad10-fc2f-4439-af5c-a5dd445dd0c9/root/var/svc/log/oxide-crucible-downstairs:default.log.0
9632023-12-06T20:45:26.505Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_fc8f6c9d-231b-4745-996c-2cf631a32873/root/var/svc/log/oxide-cockroachdb:default.log (0 bytes)
9642023-12-06T20:45:27.578Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_fc8f6c9d-231b-4745-996c-2cf631a32873/root/var/svc/log/oxide-cockroachdb:default.log
9652023-12-06T20:45:27.578Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_fc8f6c9d-231b-4745-996c-2cf631a32873/root/var/svc/log/oxide-cockroachdb:default.log.0 (5368 bytes)
9662023-12-06T20:45:28.678Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_fc8f6c9d-231b-4745-996c-2cf631a32873/root/var/svc/log/oxide-cockroachdb:default.log.0
9672023-12-06T20:45:28.678Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_84139306-e363-4e00-bf44-36af8e06a140/root/var/svc/log/oxide-crucible-pantry:default.log (0 bytes)
9682023-12-06T20:45:29.749Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_84139306-e363-4e00-bf44-36af8e06a140/root/var/svc/log/oxide-crucible-pantry:default.log
9692023-12-06T20:45:29.749Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_84139306-e363-4e00-bf44-36af8e06a140/root/var/svc/log/oxide-crucible-pantry:default.log.0 (2333 bytes)
9702023-12-06T20:45:30.847Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_84139306-e363-4e00-bf44-36af8e06a140/root/var/svc/log/oxide-crucible-pantry:default.log.0
9712023-12-06T20:45:30.847Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_06ec1ba1-d5df-486d-aeae-8b941bdd08bb/root/var/svc/log/oxide-external_dns:default.log (3778 bytes)
9722023-12-06T20:45:31.972Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_06ec1ba1-d5df-486d-aeae-8b941bdd08bb/root/var/svc/log/oxide-external_dns:default.log
9732023-12-06T20:45:31.997Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_06ec1ba1-d5df-486d-aeae-8b941bdd08bb/root/var/svc/log/oxide-external_dns:default.log.0 (11838 bytes)
9742023-12-06T20:45:33.108Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_06ec1ba1-d5df-486d-aeae-8b941bdd08bb/root/var/svc/log/oxide-external_dns:default.log.0
9752023-12-06T20:45:33.108Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_75fedf4d-07d7-49c3-bafe-cb2a10b4db43/root/var/svc/log/oxide-internal_dns:default.log (11307 bytes)
9762023-12-06T20:45:34.225Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_75fedf4d-07d7-49c3-bafe-cb2a10b4db43/root/var/svc/log/oxide-internal_dns:default.log
9772023-12-06T20:45:34.251Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_75fedf4d-07d7-49c3-bafe-cb2a10b4db43/root/var/svc/log/oxide-internal_dns:default.log.0 (103273 bytes)
9782023-12-06T20:45:35.446Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_75fedf4d-07d7-49c3-bafe-cb2a10b4db43/root/var/svc/log/oxide-internal_dns:default.log.0
9792023-12-06T20:45:35.446Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_112ba3df-2a10-4d32-b6f0-56b24abc680a/root/var/svc/log/oxide-nexus:default.log (757343 bytes)
9802023-12-06T20:45:35.719Zupload warning: file "/pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_112ba3df-2a10-4d32-b6f0-56b24abc680a/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 757343 -> 757952
9812023-12-06T20:45:36.792Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_112ba3df-2a10-4d32-b6f0-56b24abc680a/root/var/svc/log/oxide-nexus:default.log
9822023-12-06T20:45:36.792Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_112ba3df-2a10-4d32-b6f0-56b24abc680a/root/var/svc/log/oxide-nexus:default.log.0 (3931627 bytes)
9832023-12-06T20:45:38.685Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_112ba3df-2a10-4d32-b6f0-56b24abc680a/root/var/svc/log/oxide-nexus:default.log.0
9842023-12-06T20:45:38.685Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_70c55001-ca75-4aaf-b438-7ef62dacb383/root/var/svc/log/oxide-ntp:default.log (0 bytes)
9852023-12-06T20:45:40.163Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_70c55001-ca75-4aaf-b438-7ef62dacb383/root/var/svc/log/oxide-ntp:default.log
9862023-12-06T20:45:40.163Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_70c55001-ca75-4aaf-b438-7ef62dacb383/root/var/svc/log/oxide-ntp:default.log.0 (1186 bytes)
9872023-12-06T20:45:41.378Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_70c55001-ca75-4aaf-b438-7ef62dacb383/root/var/svc/log/oxide-ntp:default.log.0
9882023-12-06T20:45:41.378Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_cockroachdb_ce4eae82-ccd1-4061-8202-b73f942f5896/root/var/svc/log/oxide-cockroachdb:default.log (0 bytes)
9892023-12-06T20:45:42.561Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_cockroachdb_ce4eae82-ccd1-4061-8202-b73f942f5896/root/var/svc/log/oxide-cockroachdb:default.log
9902023-12-06T20:45:42.619Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_cockroachdb_ce4eae82-ccd1-4061-8202-b73f942f5896/root/var/svc/log/oxide-cockroachdb:default.log.0 (5369 bytes)
9912023-12-06T20:45:44.008Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_cockroachdb_ce4eae82-ccd1-4061-8202-b73f942f5896/root/var/svc/log/oxide-cockroachdb:default.log.0
9922023-12-06T20:45:44.008Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_cockroachdb_f96e482e-99d8-4eb1-b611-57fcdcc00f7f/root/var/svc/log/oxide-cockroachdb:default.log (0 bytes)
9932023-12-06T20:45:52.685Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_cockroachdb_f96e482e-99d8-4eb1-b611-57fcdcc00f7f/root/var/svc/log/oxide-cockroachdb:default.log
9942023-12-06T20:45:52.685Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_cockroachdb_f96e482e-99d8-4eb1-b611-57fcdcc00f7f/root/var/svc/log/oxide-cockroachdb:default.log.0 (5370 bytes)
9952023-12-06T20:45:53.820Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_cockroachdb_f96e482e-99d8-4eb1-b611-57fcdcc00f7f/root/var/svc/log/oxide-cockroachdb:default.log.0
9962023-12-06T20:45:53.820Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_2aed5484-4ab2-4482-a38a-aad9f4217256/root/var/svc/log/oxide-crucible-agent:default.log (0 bytes)
9972023-12-06T20:45:55.575Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_2aed5484-4ab2-4482-a38a-aad9f4217256/root/var/svc/log/oxide-crucible-agent:default.log
9982023-12-06T20:45:55.620Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_2aed5484-4ab2-4482-a38a-aad9f4217256/root/var/svc/log/oxide-crucible-agent:default.log.0 (14113 bytes)
9992023-12-06T20:45:57.613Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_2aed5484-4ab2-4482-a38a-aad9f4217256/root/var/svc/log/oxide-crucible-agent:default.log.0
10002023-12-06T20:45:57.613Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_2aed5484-4ab2-4482-a38a-aad9f4217256/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
10012023-12-06T20:45:58.696Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_2aed5484-4ab2-4482-a38a-aad9f4217256/root/var/svc/log/oxide-crucible-downstairs:default.log
10022023-12-06T20:45:58.722Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_2aed5484-4ab2-4482-a38a-aad9f4217256/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
10032023-12-06T20:45:59.819Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_2aed5484-4ab2-4482-a38a-aad9f4217256/root/var/svc/log/oxide-crucible-downstairs:default.log.0
10042023-12-06T20:45:59.819Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_2aed5484-4ab2-4482-a38a-aad9f4217256/root/var/svc/log/oxide-crucible-downstairs:downstairs-26844f53-9594-4af0-86e9-c2e7edd14e69.log (0 bytes)
10052023-12-06T20:46:00.901Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_2aed5484-4ab2-4482-a38a-aad9f4217256/root/var/svc/log/oxide-crucible-downstairs:downstairs-26844f53-9594-4af0-86e9-c2e7edd14e69.log
10062023-12-06T20:46:00.901Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_2aed5484-4ab2-4482-a38a-aad9f4217256/root/var/svc/log/oxide-crucible-downstairs:downstairs-26844f53-9594-4af0-86e9-c2e7edd14e69.log.0 (3051 bytes)
10072023-12-06T20:46:02.430Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_2aed5484-4ab2-4482-a38a-aad9f4217256/root/var/svc/log/oxide-crucible-downstairs:downstairs-26844f53-9594-4af0-86e9-c2e7edd14e69.log.0
10082023-12-06T20:46:02.430Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_nexus_e31d2871-4382-4d4e-acff-e681cede1597/root/var/svc/log/oxide-nexus:default.log (64447 bytes)
10092023-12-06T20:46:02.888Zupload warning: file "/pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_nexus_e31d2871-4382-4d4e-acff-e681cede1597/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 64447 -> 71932
10102023-12-06T20:46:04.316Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_nexus_e31d2871-4382-4d4e-acff-e681cede1597/root/var/svc/log/oxide-nexus:default.log
10112023-12-06T20:46:04.316Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_nexus_e31d2871-4382-4d4e-acff-e681cede1597/root/var/svc/log/oxide-nexus:default.log.0 (452228 bytes)
10122023-12-06T20:46:07.641Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_nexus_e31d2871-4382-4d4e-acff-e681cede1597/root/var/svc/log/oxide-nexus:default.log.0
10132023-12-06T20:46:07.641Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_2a63ad12-6c90-4cd9-aa6f-26bd101ac138/root/var/svc/log/oxide-crucible-agent:default.log (4323 bytes)
10142023-12-06T20:46:09.214Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_2a63ad12-6c90-4cd9-aa6f-26bd101ac138/root/var/svc/log/oxide-crucible-agent:default.log
10152023-12-06T20:46:09.214Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_2a63ad12-6c90-4cd9-aa6f-26bd101ac138/root/var/svc/log/oxide-crucible-agent:default.log.0 (9793 bytes)
10162023-12-06T20:46:10.500Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_2a63ad12-6c90-4cd9-aa6f-26bd101ac138/root/var/svc/log/oxide-crucible-agent:default.log.0
10172023-12-06T20:46:10.501Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_2a63ad12-6c90-4cd9-aa6f-26bd101ac138/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
10182023-12-06T20:46:11.581Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_2a63ad12-6c90-4cd9-aa6f-26bd101ac138/root/var/svc/log/oxide-crucible-downstairs:default.log
10192023-12-06T20:46:11.582Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_2a63ad12-6c90-4cd9-aa6f-26bd101ac138/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
10202023-12-06T20:46:12.692Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_2a63ad12-6c90-4cd9-aa6f-26bd101ac138/root/var/svc/log/oxide-crucible-downstairs:default.log.0
10212023-12-06T20:46:12.791Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_2a63ad12-6c90-4cd9-aa6f-26bd101ac138/root/var/svc/log/oxide-crucible-downstairs:downstairs-fec9d407-a77b-401a-9a82-39f437ab00a7.log (1005 bytes)
10222023-12-06T20:46:14.469Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_2a63ad12-6c90-4cd9-aa6f-26bd101ac138/root/var/svc/log/oxide-crucible-downstairs:downstairs-fec9d407-a77b-401a-9a82-39f437ab00a7.log
10232023-12-06T20:46:14.469Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_2a63ad12-6c90-4cd9-aa6f-26bd101ac138/root/var/svc/log/oxide-crucible-downstairs:downstairs-fec9d407-a77b-401a-9a82-39f437ab00a7.log.0 (4528 bytes)
10242023-12-06T20:46:15.625Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_2a63ad12-6c90-4cd9-aa6f-26bd101ac138/root/var/svc/log/oxide-crucible-downstairs:downstairs-fec9d407-a77b-401a-9a82-39f437ab00a7.log.0
10252023-12-06T20:46:15.626Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_b056c2f4-fd35-4071-bb67-5f0e9641c1bc/root/var/svc/log/oxide-crucible-agent:default.log (0 bytes)
10262023-12-06T20:46:16.699Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_b056c2f4-fd35-4071-bb67-5f0e9641c1bc/root/var/svc/log/oxide-crucible-agent:default.log
10272023-12-06T20:46:16.699Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_b056c2f4-fd35-4071-bb67-5f0e9641c1bc/root/var/svc/log/oxide-crucible-agent:default.log.0 (3434 bytes)
10282023-12-06T20:46:17.798Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_b056c2f4-fd35-4071-bb67-5f0e9641c1bc/root/var/svc/log/oxide-crucible-agent:default.log.0
10292023-12-06T20:46:17.799Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_b056c2f4-fd35-4071-bb67-5f0e9641c1bc/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
10302023-12-06T20:46:18.951Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_b056c2f4-fd35-4071-bb67-5f0e9641c1bc/root/var/svc/log/oxide-crucible-downstairs:default.log
10312023-12-06T20:46:18.952Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_b056c2f4-fd35-4071-bb67-5f0e9641c1bc/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
10322023-12-06T20:46:20.049Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_b056c2f4-fd35-4071-bb67-5f0e9641c1bc/root/var/svc/log/oxide-crucible-downstairs:default.log.0
10332023-12-06T20:46:20.049Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_nexus_4d87892e-81ab-403d-a70a-055891e0a0b5/root/var/svc/log/oxide-nexus:default.log (64431 bytes)
10342023-12-06T20:46:20.429Zupload warning: file "/pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_nexus_4d87892e-81ab-403d-a70a-055891e0a0b5/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 64431 -> 136369
10352023-12-06T20:46:21.506Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_nexus_4d87892e-81ab-403d-a70a-055891e0a0b5/root/var/svc/log/oxide-nexus:default.log
10362023-12-06T20:46:21.532Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_nexus_4d87892e-81ab-403d-a70a-055891e0a0b5/root/var/svc/log/oxide-nexus:default.log.0 (559329 bytes)
10372023-12-06T20:46:23.400Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_nexus_4d87892e-81ab-403d-a70a-055891e0a0b5/root/var/svc/log/oxide-nexus:default.log.0
10382023-12-06T20:46:23.426Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_clickhouse_d1459886-0502-404a-bc4f-b719a1d2b786/root/var/svc/log/oxide-clickhouse:default.log (29383 bytes)
10392023-12-06T20:46:23.796Zupload warning: file "/pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_clickhouse_d1459886-0502-404a-bc4f-b719a1d2b786/root/var/svc/log/oxide-clickhouse:default.log" changed size mid upload: 29383 -> 155676
10402023-12-06T20:46:24.871Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_clickhouse_d1459886-0502-404a-bc4f-b719a1d2b786/root/var/svc/log/oxide-clickhouse:default.log
10412023-12-06T20:46:24.871Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_clickhouse_d1459886-0502-404a-bc4f-b719a1d2b786/root/var/svc/log/oxide-clickhouse:default.log.0 (492023 bytes)
10422023-12-06T20:46:26.707Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_clickhouse_d1459886-0502-404a-bc4f-b719a1d2b786/root/var/svc/log/oxide-clickhouse:default.log.0
10432023-12-06T20:46:26.707Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_1342bc75-a2c2-48cc-99e7-afe8089fb64e/root/var/svc/log/oxide-crucible-agent:default.log (4785 bytes)
10442023-12-06T20:46:28.221Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_1342bc75-a2c2-48cc-99e7-afe8089fb64e/root/var/svc/log/oxide-crucible-agent:default.log
10452023-12-06T20:46:28.221Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_1342bc75-a2c2-48cc-99e7-afe8089fb64e/root/var/svc/log/oxide-crucible-agent:default.log.0 (9795 bytes)
10462023-12-06T20:46:29.348Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_1342bc75-a2c2-48cc-99e7-afe8089fb64e/root/var/svc/log/oxide-crucible-agent:default.log.0
10472023-12-06T20:46:29.385Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_1342bc75-a2c2-48cc-99e7-afe8089fb64e/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
10482023-12-06T20:46:30.433Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_1342bc75-a2c2-48cc-99e7-afe8089fb64e/root/var/svc/log/oxide-crucible-downstairs:default.log
10492023-12-06T20:46:30.433Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_1342bc75-a2c2-48cc-99e7-afe8089fb64e/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
10502023-12-06T20:46:31.530Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_1342bc75-a2c2-48cc-99e7-afe8089fb64e/root/var/svc/log/oxide-crucible-downstairs:default.log.0
10512023-12-06T20:46:31.530Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_1342bc75-a2c2-48cc-99e7-afe8089fb64e/root/var/svc/log/oxide-crucible-downstairs:downstairs-73a54903-2951-4c33-844e-eabb2cfaabbe.log (1003 bytes)
10522023-12-06T20:46:32.652Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_1342bc75-a2c2-48cc-99e7-afe8089fb64e/root/var/svc/log/oxide-crucible-downstairs:downstairs-73a54903-2951-4c33-844e-eabb2cfaabbe.log
10532023-12-06T20:46:32.652Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_1342bc75-a2c2-48cc-99e7-afe8089fb64e/root/var/svc/log/oxide-crucible-downstairs:downstairs-73a54903-2951-4c33-844e-eabb2cfaabbe.log.0 (4530 bytes)
10542023-12-06T20:46:33.761Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_1342bc75-a2c2-48cc-99e7-afe8089fb64e/root/var/svc/log/oxide-crucible-downstairs:downstairs-73a54903-2951-4c33-844e-eabb2cfaabbe.log.0
10552023-12-06T20:46:33.761Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_eee6f940-39b0-4792-9dda-b67e49b48dca/root/var/svc/log/oxide-crucible-agent:default.log (0 bytes)
10562023-12-06T20:46:34.909Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_eee6f940-39b0-4792-9dda-b67e49b48dca/root/var/svc/log/oxide-crucible-agent:default.log
10572023-12-06T20:46:34.909Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_eee6f940-39b0-4792-9dda-b67e49b48dca/root/var/svc/log/oxide-crucible-agent:default.log.0 (3435 bytes)
10582023-12-06T20:46:36.035Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_eee6f940-39b0-4792-9dda-b67e49b48dca/root/var/svc/log/oxide-crucible-agent:default.log.0
10592023-12-06T20:46:36.035Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_eee6f940-39b0-4792-9dda-b67e49b48dca/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
10602023-12-06T20:46:37.118Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_eee6f940-39b0-4792-9dda-b67e49b48dca/root/var/svc/log/oxide-crucible-downstairs:default.log
10612023-12-06T20:46:37.119Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_eee6f940-39b0-4792-9dda-b67e49b48dca/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
10622023-12-06T20:46:38.227Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_eee6f940-39b0-4792-9dda-b67e49b48dca/root/var/svc/log/oxide-crucible-downstairs:default.log.0
10632023-12-06T20:46:38.228Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_cockroachdb_e249e966-2143-43a0-ab95-238c20d161a8/root/var/svc/log/oxide-cockroachdb:default.log (0 bytes)
10642023-12-06T20:46:39.318Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_cockroachdb_e249e966-2143-43a0-ab95-238c20d161a8/root/var/svc/log/oxide-cockroachdb:default.log
10652023-12-06T20:46:39.319Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_cockroachdb_e249e966-2143-43a0-ab95-238c20d161a8/root/var/svc/log/oxide-cockroachdb:default.log.0 (5369 bytes)
10662023-12-06T20:46:40.444Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_cockroachdb_e249e966-2143-43a0-ab95-238c20d161a8/root/var/svc/log/oxide-cockroachdb:default.log.0
10672023-12-06T20:46:40.445Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_oximeter_9bfbe319-6319-4bf7-a94e-7c11511b5e83/root/var/svc/log/oxide-oximeter:default.log (13017 bytes)
10682023-12-06T20:46:40.605Zupload warning: file "/pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_oximeter_9bfbe319-6319-4bf7-a94e-7c11511b5e83/root/var/svc/log/oxide-oximeter:default.log" changed size mid upload: 13017 -> 61642
10692023-12-06T20:46:42.080Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_oximeter_9bfbe319-6319-4bf7-a94e-7c11511b5e83/root/var/svc/log/oxide-oximeter:default.log
10702023-12-06T20:46:42.081Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_oximeter_9bfbe319-6319-4bf7-a94e-7c11511b5e83/root/var/svc/log/oxide-oximeter:default.log.0 (138227 bytes)
10712023-12-06T20:46:43.416Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_oximeter_9bfbe319-6319-4bf7-a94e-7c11511b5e83/root/var/svc/log/oxide-oximeter:default.log.0
10722023-12-06T20:46:43.449Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_a4867f7d-b013-4aca-8ccd-8d29ecf5654b/root/var/svc/log/oxide-cockroachdb:default.log (0 bytes)
10732023-12-06T20:46:44.518Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_a4867f7d-b013-4aca-8ccd-8d29ecf5654b/root/var/svc/log/oxide-cockroachdb:default.log
10742023-12-06T20:46:44.518Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_a4867f7d-b013-4aca-8ccd-8d29ecf5654b/root/var/svc/log/oxide-cockroachdb:default.log.0 (5418 bytes)
10752023-12-06T20:46:46.074Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_a4867f7d-b013-4aca-8ccd-8d29ecf5654b/root/var/svc/log/oxide-cockroachdb:default.log.0
10762023-12-06T20:46:46.074Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_21931def-f303-457b-bb8d-00fae6c61273/root/var/svc/log/oxide-crucible-agent:default.log (0 bytes)
10772023-12-06T20:46:47.151Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_21931def-f303-457b-bb8d-00fae6c61273/root/var/svc/log/oxide-crucible-agent:default.log
10782023-12-06T20:46:47.151Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_21931def-f303-457b-bb8d-00fae6c61273/root/var/svc/log/oxide-crucible-agent:default.log.0 (13688 bytes)
10792023-12-06T20:46:48.336Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_21931def-f303-457b-bb8d-00fae6c61273/root/var/svc/log/oxide-crucible-agent:default.log.0
10802023-12-06T20:46:48.336Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_21931def-f303-457b-bb8d-00fae6c61273/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
10812023-12-06T20:46:49.421Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_21931def-f303-457b-bb8d-00fae6c61273/root/var/svc/log/oxide-crucible-downstairs:default.log
10822023-12-06T20:46:49.421Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_21931def-f303-457b-bb8d-00fae6c61273/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
10832023-12-06T20:46:50.575Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_21931def-f303-457b-bb8d-00fae6c61273/root/var/svc/log/oxide-crucible-downstairs:default.log.0
10842023-12-06T20:46:50.575Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_21931def-f303-457b-bb8d-00fae6c61273/root/var/svc/log/oxide-crucible-downstairs:downstairs-5dcae389-6e13-437c-8ede-eb1453fe50a6.log (0 bytes)
10852023-12-06T20:46:51.648Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_21931def-f303-457b-bb8d-00fae6c61273/root/var/svc/log/oxide-crucible-downstairs:downstairs-5dcae389-6e13-437c-8ede-eb1453fe50a6.log
10862023-12-06T20:46:51.649Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_21931def-f303-457b-bb8d-00fae6c61273/root/var/svc/log/oxide-crucible-downstairs:downstairs-5dcae389-6e13-437c-8ede-eb1453fe50a6.log.0 (3051 bytes)
10872023-12-06T20:46:52.746Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_21931def-f303-457b-bb8d-00fae6c61273/root/var/svc/log/oxide-crucible-downstairs:downstairs-5dcae389-6e13-437c-8ede-eb1453fe50a6.log.0
10882023-12-06T20:46:52.746Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_b950a9ad-c2ab-4709-8982-dfe023bf9403/root/var/svc/log/oxide-crucible-agent:default.log (0 bytes)
10892023-12-06T20:46:53.837Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_b950a9ad-c2ab-4709-8982-dfe023bf9403/root/var/svc/log/oxide-crucible-agent:default.log
10902023-12-06T20:46:53.837Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_b950a9ad-c2ab-4709-8982-dfe023bf9403/root/var/svc/log/oxide-crucible-agent:default.log.0 (14114 bytes)
10912023-12-06T20:46:55.404Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_b950a9ad-c2ab-4709-8982-dfe023bf9403/root/var/svc/log/oxide-crucible-agent:default.log.0
10922023-12-06T20:46:55.404Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_b950a9ad-c2ab-4709-8982-dfe023bf9403/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
10932023-12-06T20:46:56.480Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_b950a9ad-c2ab-4709-8982-dfe023bf9403/root/var/svc/log/oxide-crucible-downstairs:default.log
10942023-12-06T20:46:56.480Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_b950a9ad-c2ab-4709-8982-dfe023bf9403/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
10952023-12-06T20:46:57.587Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_b950a9ad-c2ab-4709-8982-dfe023bf9403/root/var/svc/log/oxide-crucible-downstairs:default.log.0
10962023-12-06T20:46:57.587Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_b950a9ad-c2ab-4709-8982-dfe023bf9403/root/var/svc/log/oxide-crucible-downstairs:downstairs-3da51f9d-e4a6-4183-a8d5-3b975fc85eea.log (0 bytes)
10972023-12-06T20:46:58.660Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_b950a9ad-c2ab-4709-8982-dfe023bf9403/root/var/svc/log/oxide-crucible-downstairs:downstairs-3da51f9d-e4a6-4183-a8d5-3b975fc85eea.log
10982023-12-06T20:46:58.660Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_b950a9ad-c2ab-4709-8982-dfe023bf9403/root/var/svc/log/oxide-crucible-downstairs:downstairs-3da51f9d-e4a6-4183-a8d5-3b975fc85eea.log.0 (3051 bytes)
10992023-12-06T20:46:59.760Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_b950a9ad-c2ab-4709-8982-dfe023bf9403/root/var/svc/log/oxide-crucible-downstairs:downstairs-3da51f9d-e4a6-4183-a8d5-3b975fc85eea.log.0
11002023-12-06T20:46:59.761Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_fbac8a30-eb45-4407-ac11-af7dccfe134c/root/var/svc/log/oxide-crucible-pantry:default.log (0 bytes)
11012023-12-06T20:47:00.850Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_fbac8a30-eb45-4407-ac11-af7dccfe134c/root/var/svc/log/oxide-crucible-pantry:default.log
11022023-12-06T20:47:00.850Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_fbac8a30-eb45-4407-ac11-af7dccfe134c/root/var/svc/log/oxide-crucible-pantry:default.log.0 (2338 bytes)
11032023-12-06T20:47:01.953Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_fbac8a30-eb45-4407-ac11-af7dccfe134c/root/var/svc/log/oxide-crucible-pantry:default.log.0
11042023-12-06T20:47:01.953Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_e2579bde-8075-4079-8924-df6a7ba6f0ff/root/var/svc/log/oxide-external_dns:default.log (3777 bytes)
11052023-12-06T20:47:02.027Zupload warning: file "/pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_e2579bde-8075-4079-8924-df6a7ba6f0ff/root/var/svc/log/oxide-external_dns:default.log" changed size mid upload: 3777 -> 7552
11062023-12-06T20:47:03.101Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_e2579bde-8075-4079-8924-df6a7ba6f0ff/root/var/svc/log/oxide-external_dns:default.log
11072023-12-06T20:47:03.101Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_e2579bde-8075-4079-8924-df6a7ba6f0ff/root/var/svc/log/oxide-external_dns:default.log.0 (11837 bytes)
11082023-12-06T20:47:04.241Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_e2579bde-8075-4079-8924-df6a7ba6f0ff/root/var/svc/log/oxide-external_dns:default.log.0
11092023-12-06T20:47:04.241Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_35730255-92c9-4cd0-b18f-7af3787d2586/root/var/svc/log/oxide-internal_dns:default.log (3756 bytes)
11102023-12-06T20:47:04.316Zupload warning: file "/pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_35730255-92c9-4cd0-b18f-7af3787d2586/root/var/svc/log/oxide-internal_dns:default.log" changed size mid upload: 3756 -> 7513
11112023-12-06T20:47:05.389Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_35730255-92c9-4cd0-b18f-7af3787d2586/root/var/svc/log/oxide-internal_dns:default.log
11122023-12-06T20:47:05.389Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_35730255-92c9-4cd0-b18f-7af3787d2586/root/var/svc/log/oxide-internal_dns:default.log.0 (115306 bytes)
11132023-12-06T20:47:06.774Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_35730255-92c9-4cd0-b18f-7af3787d2586/root/var/svc/log/oxide-internal_dns:default.log.0
11142023-12-06T20:47:06.774Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_b80664d8-3012-4680-81b0-63376c037106/root/var/svc/log/oxide-crucible-agent:default.log (4324 bytes)
11152023-12-06T20:47:07.874Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_b80664d8-3012-4680-81b0-63376c037106/root/var/svc/log/oxide-crucible-agent:default.log
11162023-12-06T20:47:07.874Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_b80664d8-3012-4680-81b0-63376c037106/root/var/svc/log/oxide-crucible-agent:default.log.0 (9363 bytes)
11172023-12-06T20:47:08.999Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_b80664d8-3012-4680-81b0-63376c037106/root/var/svc/log/oxide-crucible-agent:default.log.0
11182023-12-06T20:47:09.000Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_b80664d8-3012-4680-81b0-63376c037106/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
11192023-12-06T20:47:10.073Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_b80664d8-3012-4680-81b0-63376c037106/root/var/svc/log/oxide-crucible-downstairs:default.log
11202023-12-06T20:47:10.074Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_b80664d8-3012-4680-81b0-63376c037106/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
11212023-12-06T20:47:11.187Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_b80664d8-3012-4680-81b0-63376c037106/root/var/svc/log/oxide-crucible-downstairs:default.log.0
11222023-12-06T20:47:11.187Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_b80664d8-3012-4680-81b0-63376c037106/root/var/svc/log/oxide-crucible-downstairs:downstairs-8183c672-e7de-42b1-a776-6d3c74d5e91b.log (1004 bytes)
11232023-12-06T20:47:12.321Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_b80664d8-3012-4680-81b0-63376c037106/root/var/svc/log/oxide-crucible-downstairs:downstairs-8183c672-e7de-42b1-a776-6d3c74d5e91b.log
11242023-12-06T20:47:12.321Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_b80664d8-3012-4680-81b0-63376c037106/root/var/svc/log/oxide-crucible-downstairs:downstairs-8183c672-e7de-42b1-a776-6d3c74d5e91b.log.0 (4528 bytes)
11252023-12-06T20:47:13.446Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_b80664d8-3012-4680-81b0-63376c037106/root/var/svc/log/oxide-crucible-downstairs:downstairs-8183c672-e7de-42b1-a776-6d3c74d5e91b.log.0
11262023-12-06T20:47:13.446Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_47b23ba7-185c-4f91-afe5-9b20853bff23/root/var/svc/log/oxide-crucible-pantry:default.log (0 bytes)
11272023-12-06T20:47:14.530Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_47b23ba7-185c-4f91-afe5-9b20853bff23/root/var/svc/log/oxide-crucible-pantry:default.log
11282023-12-06T20:47:14.530Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_47b23ba7-185c-4f91-afe5-9b20853bff23/root/var/svc/log/oxide-crucible-pantry:default.log.0 (2338 bytes)
11292023-12-06T20:47:15.644Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_47b23ba7-185c-4f91-afe5-9b20853bff23/root/var/svc/log/oxide-crucible-pantry:default.log.0
11302023-12-06T20:47:15.644Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_7433ba0d-16f9-489d-8967-07a1e09402ff/root/var/svc/log/oxide-internal_dns:default.log (3755 bytes)
11312023-12-06T20:47:15.715Zupload warning: file "/pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_7433ba0d-16f9-489d-8967-07a1e09402ff/root/var/svc/log/oxide-internal_dns:default.log" changed size mid upload: 3755 -> 11268
11322023-12-06T20:47:16.792Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_7433ba0d-16f9-489d-8967-07a1e09402ff/root/var/svc/log/oxide-internal_dns:default.log
11332023-12-06T20:47:16.793Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_7433ba0d-16f9-489d-8967-07a1e09402ff/root/var/svc/log/oxide-internal_dns:default.log.0 (113266 bytes)
11342023-12-06T20:47:18.175Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_7433ba0d-16f9-489d-8967-07a1e09402ff/root/var/svc/log/oxide-internal_dns:default.log.0
11352023-12-06T20:47:18.176Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_70c55001-ca75-4aaf-b438-7ef62dacb383/root/var/log/chrony/measurements.log (13563 bytes)
11362023-12-06T20:47:18.282Zupload warning: file "/pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_70c55001-ca75-4aaf-b438-7ef62dacb383/root/var/log/chrony/measurements.log" changed size mid upload: 13563 -> 18358
11372023-12-06T20:47:19.362Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_70c55001-ca75-4aaf-b438-7ef62dacb383/root/var/log/chrony/measurements.log
11382023-12-06T20:47:19.362Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_70c55001-ca75-4aaf-b438-7ef62dacb383/root/var/log/chrony/statistics.log (8775 bytes)
11392023-12-06T20:47:19.454Zupload warning: file "/pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_70c55001-ca75-4aaf-b438-7ef62dacb383/root/var/log/chrony/statistics.log" changed size mid upload: 8775 -> 11934
11402023-12-06T20:47:20.526Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_70c55001-ca75-4aaf-b438-7ef62dacb383/root/var/log/chrony/statistics.log
11412023-12-06T20:47:20.526Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_70c55001-ca75-4aaf-b438-7ef62dacb383/root/var/log/chrony/tracking.log (1854 bytes)
11422023-12-06T20:47:20.573Zupload warning: file "/pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_70c55001-ca75-4aaf-b438-7ef62dacb383/root/var/log/chrony/tracking.log" changed size mid upload: 1854 -> 2118
11432023-12-06T20:47:21.647Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_70c55001-ca75-4aaf-b438-7ef62dacb383/root/var/log/chrony/tracking.log