01HHGNPT22X81TNY74C0E9R2MJ: helios / deploy

BasicConfig {
    output_rules: [
        "%/var/svc/log/oxide-sled-agent:default.log*",
        "%/pool/ext/*/crypt/zone/oxz_*/root/var/svc/log/oxide-*.log*",
        "%/pool/ext/*/crypt/zone/oxz_*/root/var/svc/log/system-illumos-*.log*",
        "%/pool/ext/*/crypt/zone/oxz_ntp_*/root/var/log/chrony/*.log*",
        "!/pool/ext/*/crypt/zone/oxz_propolis-server_*/root/var/svc/log/*.log*",
        "%/pool/ext/*/crypt/debug/global/oxide-sled-agent:default.log.*",
        "%/pool/ext/*/crypt/debug/oxz_*/oxide-*.log.*",
        "%/pool/ext/*/crypt/debug/oxz_*/system-illumos-*.log.*",
        "!/pool/ext/*/crypt/debug/oxz_propolis-server_*/*.log.*",
    ],
    rust_toolchain: None,
    target: Some(
        "lab-2.0-opte-0.27",
    ),
    access_repos: [],
    publish: [],
    skip_clone: true,
}

Buildomat Job: 01HHGNQFWVSRS8ETAJRJHEEBBY

Tags:

Artefacts:

Output:

SEQ GLOBAL TIME DETAILS
12023-12-13T05:00:48.015Zjob dependencies complete; ready to run (waiting for 53 m 16 s)
22023-12-13T05:04:58.233Zjob assigned to worker 01HHGRNG6BV4146TAM6DWWKEBT (queued for 4 m 10 s)
32023-12-13T05:04:58.524Zhttp://172.20.0.5:9969/ipxe/sock... ok
42023-12-13T05:04:58.600Zhttp://172.20.0.5:9969/os/sock/platform/i86pc/kernel/amd64/unix... ok
52023-12-13T05:04:59.044Zhttp://172.20.0.5:9969/os/sock/platform/i86pc/amd64/boot_archive... ok
62023-12-13T05:04:59.077Zhttp://172.20.0.5:9969/os/sock/platform/i86pc/amd64/boot_archive.hash... ok
72023-12-13T05:04:59.111Zhttp://172.20.0.5:9969/postboot/sock... ok
82023-12-13T05:04:59.175ZwcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U
92023-12-13T05:04:59.222Zcpu0: microcode has been updated from version 0xa001133 to 0xa0011d1
102023-12-13T05:04:59.273ZOxide Helios Version helios-2.0.22333 64-bit
112023-12-13T05:04:59.314ZConfiguring devices.
122023-12-13T05:04:59.347Z##################################################
132023-12-13T05:04:59.418Z#### ############# ##
142023-12-13T05:04:59.505Z### ### ############ ##
152023-12-13T05:04:59.548Z## ### # ## ### ## Oxide Computer Company ##
162023-12-13T05:04:59.581Z## ## ## ### # ### ##
172023-12-13T05:04:59.625Z## # ### #### #### This Station Under ##
182023-12-13T05:04:59.658Z### ### #### # ### Computer Control ##
192023-12-13T05:04:59.713Z#### #### ### ## ##
202023-12-13T05:04:59.752Z##################################################
212023-12-13T05:04:59.848ZHostname: sock
222023-12-13T05:04:59.896ZLoading smf(7) service descriptions:
232023-12-13T05:05:03.038Zdownloading input: /input/ci-tools/work/end-to-end-tests/bootstrap-cc02af252c51050b.gz
242023-12-13T05:05:05.319Zdownloaded input: /input/ci-tools/work/end-to-end-tests/bootstrap-cc02af252c51050b.gz
252023-12-13T05:05:05.619Zdownloading input: /input/ci-tools/work/end-to-end-tests/bootstrap.gz
262023-12-13T05:05:56.850Zdownloaded input: /input/ci-tools/work/end-to-end-tests/bootstrap.gz
272023-12-13T05:05:56.875Zdownloading input: /input/ci-tools/work/end-to-end-tests/end_to_end_tests-c0f56f9124e0823c.gz
282023-12-13T05:06:34.090Zdownloaded input: /input/ci-tools/work/end-to-end-tests/end_to_end_tests-c0f56f9124e0823c.gz
292023-12-13T05:06:34.123Zdownloading input: /input/ci-tools/work/caboose-util.gz
302023-12-13T05:06:35.236Zdownloaded input: /input/ci-tools/work/caboose-util.gz
312023-12-13T05:06:35.263Zdownloading input: /input/ci-tools/work/tufaceous.gz
322023-12-13T05:06:38.323Zdownloaded input: /input/ci-tools/work/tufaceous.gz
332023-12-13T05:06:38.349Zdownloading input: /input/package/work/version.txt
342023-12-13T05:06:38.349Zdownloaded input: /input/package/work/version.txt
352023-12-13T05:06:38.349Zdownloading input: /input/package/work/package.tar.gz
362023-12-13T05:06:57.641Zdownloaded input: /input/package/work/package.tar.gz
372023-12-13T05:06:57.641Zdownloading input: /input/package/work/global-zone-packages.tar.gz
382023-12-13T05:07:25.904Zdownloaded input: /input/package/work/global-zone-packages.tar.gz
392023-12-13T05:07:25.935Zdownloading input: /input/package/work/trampoline-global-zone-packages.tar.gz
402023-12-13T05:07:33.766Zdownloaded input: /input/package/work/trampoline-global-zone-packages.tar.gz
412023-12-13T05:07:33.791Zdownloading input: /input/package/work/zones/clickhouse.tar.gz
422023-12-13T05:08:09.798Zdownloaded input: /input/package/work/zones/clickhouse.tar.gz
432023-12-13T05:08:09.825Zdownloading input: /input/package/work/zones/clickhouse_keeper.tar.gz
442023-12-13T05:08:43.350Zdownloaded input: /input/package/work/zones/clickhouse_keeper.tar.gz
452023-12-13T05:08:43.375Zdownloading input: /input/package/work/zones/cockroachdb.tar.gz
462023-12-13T05:09:05.595Zdownloaded input: /input/package/work/zones/cockroachdb.tar.gz
472023-12-13T05:09:05.622Zdownloading input: /input/package/work/zones/crucible-pantry.tar.gz
482023-12-13T05:09:10.442Zdownloaded input: /input/package/work/zones/crucible-pantry.tar.gz
492023-12-13T05:09:10.470Zdownloading input: /input/package/work/zones/crucible.tar.gz
502023-12-13T05:09:19.164Zdownloaded input: /input/package/work/zones/crucible.tar.gz
512023-12-13T05:09:19.190Zdownloading input: /input/package/work/zones/external-dns.tar.gz
522023-12-13T05:09:25.781Zdownloaded input: /input/package/work/zones/external-dns.tar.gz
532023-12-13T05:09:25.781Zdownloading input: /input/package/work/zones/internal-dns.tar.gz
542023-12-13T05:09:30.840Zdownloaded input: /input/package/work/zones/internal-dns.tar.gz
552023-12-13T05:09:30.840Zdownloading input: /input/package/work/zones/ntp.tar.gz
562023-12-13T05:09:30.879Zdownloaded input: /input/package/work/zones/ntp.tar.gz
572023-12-13T05:09:30.904Zdownloading input: /input/package/work/zones/omicron-gateway-asic.tar.gz
582023-12-13T05:09:34.434Zdownloaded input: /input/package/work/zones/omicron-gateway-asic.tar.gz
592023-12-13T05:09:34.442Zdownloading input: /input/package/work/zones/omicron-gateway-softnpu.tar.gz
602023-12-13T05:09:38.466Zdownloaded input: /input/package/work/zones/omicron-gateway-softnpu.tar.gz
612023-12-13T05:09:38.491Zdownloading input: /input/package/work/zones/omicron-nexus-single-sled.tar.gz
622023-12-13T05:09:52.536Zdownloaded input: /input/package/work/zones/omicron-nexus-single-sled.tar.gz
632023-12-13T05:09:52.563Zdownloading input: /input/package/work/zones/omicron-nexus.tar.gz
642023-12-13T05:10:06.614Zdownloaded input: /input/package/work/zones/omicron-nexus.tar.gz
652023-12-13T05:10:06.615Zdownloading input: /input/package/work/zones/overlay.tar.gz
662023-12-13T05:10:06.683Zdownloaded input: /input/package/work/zones/overlay.tar.gz
672023-12-13T05:10:06.722Zdownloading input: /input/package/work/zones/oximeter-collector.tar.gz
682023-12-13T05:10:13.978Zdownloaded input: /input/package/work/zones/oximeter-collector.tar.gz
692023-12-13T05:10:13.980Zdownloading input: /input/package/work/zones/propolis-server.tar.gz
702023-12-13T05:10:26.355Zdownloaded input: /input/package/work/zones/propolis-server.tar.gz
712023-12-13T05:10:26.356Zdownloading input: /input/package/work/zones/switch-asic.tar.gz
722023-12-13T05:11:19.661Zdownloaded input: /input/package/work/zones/switch-asic.tar.gz
732023-12-13T05:11:19.697Zdownloading input: /input/package/work/zones/switch-softnpu.tar.gz
742023-12-13T05:11:50.687Zdownloaded input: /input/package/work/zones/switch-softnpu.tar.gz
 
752023-12-13T05:11:50.733Zstarting task 0: "setup"
762023-12-13T05:11:50.733Z++ uname -s
772023-12-13T05:11:50.734Z+ kern=SunOS
782023-12-13T05:11:50.734Z+ case "$kern" in
792023-12-13T05:11:50.735Z+ groupadd -g 12345 build
802023-12-13T05:11:50.735Z+ useradd -u 12345 -g build -d /home/build -s /bin/bash -c build -P 'Primary Administrator' build
812023-12-13T05:11:52.704Z+ zfs create -o mountpoint=/work rpool/work
822023-12-13T05:11:52.730Z++ awk '$2 == "/home" { print $3 }' /etc/mnttab
832023-12-13T05:11:52.730Z+ home_fs=zfs
842023-12-13T05:11:52.730Z+ [[ zfs == autofs ]]
852023-12-13T05:11:52.731Z+ mkdir -p /home/build
862023-12-13T05:11:52.731Z+ chown build:build /home/build /work
872023-12-13T05:11:54.734Z+ chmod 0700 /home/build /work
882023-12-13T05:11:54.764Zprocess exited: duration 4048 ms, exit code 0
 
892023-12-13T05:11:54.821Zstarting task 1: "authentication"
902023-12-13T05:11:54.865Zprocess exited: duration 46 ms, exit code 0
 
912023-12-13T05:11:54.929Zstarting task 2: "build"
922023-12-13T05:11:54.955Z+ trap _exit_trap EXIT
932023-12-13T05:11:54.955Z+ pfexec mdb -kwe 'secpolicy_fs_linkdir/v 55 48 89 e5 b8 01 00 00 00 5d c3'
942023-12-13T05:11:55.137Zsecpolicy_fs_linkdir: 0x55 = 0x55
952023-12-13T05:11:55.137Zsecpolicy_fs_linkdir+1: 0xb8 = 0x48
962023-12-13T05:11:55.137Zsecpolicy_fs_linkdir+2: 0x1 = 0x89
972023-12-13T05:11:55.137Zsecpolicy_fs_linkdir+3: 0 = 0xe5
982023-12-13T05:11:55.137Zsecpolicy_fs_linkdir+4: 0 = 0xb8
992023-12-13T05:11:55.138Zsecpolicy_fs_linkdir+5: 0 = 0x1
1002023-12-13T05:11:55.138Zsecpolicy_fs_linkdir+6: 0x48 = 0x0
1012023-12-13T05:11:55.188Zsecpolicy_fs_linkdir+7: 0x89 = 0x0
1022023-12-13T05:11:55.188Zsecpolicy_fs_linkdir+8: 0xe5 = 0x0
1032023-12-13T05:11:55.188Zsecpolicy_fs_linkdir+9: 0x48 = 0x5d
1042023-12-13T05:11:55.188Zsecpolicy_fs_linkdir+0xa: 0x83 = 0xc3
1052023-12-13T05:11:55.188Z+ [[ -d /opt/oxide ]]
1062023-12-13T05:11:55.188Z+ pfexec mv /opt/oxide /opt/oxide-underneath
1072023-12-13T05:11:55.189Z+ pfexec mkdir /opt/oxide
1082023-12-13T05:11:55.189Z+ pfexec mount -F tmpfs -O swap /opt/oxide
1092023-12-13T05:11:55.189Z+ [[ -d /opt/oxide-underneath ]]
1102023-12-13T05:11:55.189Z+ cd /opt/oxide-underneath
1112023-12-13T05:11:55.189Z+ pfexec tar ceEp@/f - .
1122023-12-13T05:11:55.189Z+ cd /opt/oxide
1132023-12-13T05:11:55.189Z+ pfexec tar xveEp@/f -
1142023-12-13T05:11:55.265Zx ., 0 bytes, 0 tape blocks
1152023-12-13T05:11:55.265Zx ./opte, 0 bytes, 0 tape blocks
1162023-12-13T05:11:55.266Zx ./opte/bin, 0 bytes, 0 tape blocks
1172023-12-13T05:11:55.266Zx ./opte/bin/opteadm, 52567720 bytes, 102672 tape blocks
1182023-12-13T05:11:55.292Z+ pfexec rm -rf /opt/oxide-underneath
1192023-12-13T05:11:55.292Z+ pfexec /sbin/zfs create -o mountpoint=/zone rpool/zone
1202023-12-13T05:11:55.325Z+ pfexec mkdir /opt/oxide/work
1212023-12-13T05:11:55.325Z+ pfexec chown build:build /opt/oxide/work
1222023-12-13T05:11:55.325Z+ cd /opt/oxide/work
1232023-12-13T05:11:55.325Z+ ptime -m tar xvzf /input/package/work/package.tar.gz
1242023-12-13T05:11:55.325Ztar: blocksize = 10
1252023-12-13T05:11:56.854ZDecompressing '/input/package/work/package.tar.gz' with '/usr/bin/gzcat'...
1262023-12-13T05:11:56.854Zx out/mg-ddm-gz.tar, 62723072 bytes, 122506 tape blocks
1272023-12-13T05:11:56.880Zx out/omicron-sled-agent.tar, 111883264 bytes, 218522 tape blocks
1282023-12-13T05:11:56.880Zx out/target/test, 75 bytes, 1 tape blocks
1292023-12-13T05:11:56.880Zx out/npuzone/npuzone, 37184104 bytes, 72626 tape blocks
1302023-12-13T05:11:56.880Zx package-manifest.toml, 22143 bytes, 44 tape blocks
1312023-12-13T05:11:56.881Zx smf/sled-agent/non-gimlet/config.toml, 3448 bytes, 7 tape blocks
1322023-12-13T05:11:56.881Zx target/release/omicron-package, 42517232 bytes, 83042 tape blocks
1332023-12-13T05:11:56.881Zx tools/create_virtual_hardware.sh, 3176 bytes, 7 tape blocks
1342023-12-13T05:11:56.881Zx tools/virtual_hardware.sh, 3408 bytes, 7 tape blocks
1352023-12-13T05:11:56.881Zx tools/scrimlet/create-softnpu-zone.sh, 619 bytes, 2 tape blocks
1362023-12-13T05:11:56.881Zx tools/scrimlet/destroy-softnpu-zone.sh, 195 bytes, 1 tape blocks
1372023-12-13T05:11:56.881Zx tools/scrimlet/softnpu-init.sh, 2148 bytes, 5 tape blocks
1382023-12-13T05:11:56.881Zx tools/scrimlet/softnpu-zone.txt, 330 bytes, 1 tape blocks
1392023-12-13T05:11:56.881Zx tools/scrimlet/softnpu.toml, 171 bytes, 1 tape blocks
1402023-12-13T05:11:56.881Z
1412023-12-13T05:11:56.882Zreal 1.550054973
1422023-12-13T05:11:56.882Zuser 1.388559777
1432023-12-13T05:11:56.882Zsys 0.303208025
1442023-12-13T05:11:56.882Ztrap 0.000058801
1452023-12-13T05:11:56.882Ztflt 0.000000000
1462023-12-13T05:11:56.882Zdflt 0.000000000
1472023-12-13T05:11:56.883Zkflt 0.000000000
1482023-12-13T05:11:56.883Zlock 0.000000000
1492023-12-13T05:11:56.883Zslp 1.363754303
1502023-12-13T05:11:56.883Zlat 0.042633808
1512023-12-13T05:11:56.883Zstop 0.000119690
1522023-12-13T05:11:56.884Z+ cp /input/package/work/zones/clickhouse_keeper.tar.gz /input/package/work/zones/clickhouse.tar.gz /input/package/work/zones/cockroachdb.tar.gz /input/package/work/zones/crucible-pantry.tar.gz /input/package/work/zones/crucible.tar.gz /input/package/work/zones/external-dns.tar.gz /input/package/work/zones/internal-dns.tar.gz /input/package/work/zones/ntp.tar.gz /input/package/work/zones/omicron-gateway-asic.tar.gz /input/package/work/zones/omicron-gateway-softnpu.tar.gz /input/package/work/zones/omicron-nexus-single-sled.tar.gz /input/package/work/zones/omicron-nexus.tar.gz /input/package/work/zones/overlay.tar.gz /input/package/work/zones/oximeter-collector.tar.gz /input/package/work/zones/propolis-server.tar.gz /input/package/work/zones/switch-asic.tar.gz /input/package/work/zones/switch-softnpu.tar.gz out/
1532023-12-13T05:11:57.953Z+ mv out/omicron-nexus-single-sled.tar.gz out/omicron-nexus.tar.gz
1542023-12-13T05:11:57.980Z+ mkdir tests
1552023-12-13T05:11:57.980Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1562023-12-13T05:11:57.981Z+ ptime -m gunzip
1572023-12-13T05:11:57.981Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap-cc02af252c51050b
1582023-12-13T05:11:58.109Z
1592023-12-13T05:11:58.135Zreal 0.139298415
1602023-12-13T05:11:58.136Zuser 0.128555968
1612023-12-13T05:11:58.136Zsys 0.010496972
1622023-12-13T05:11:58.136Ztrap 0.000012169
1632023-12-13T05:11:58.136Ztflt 0.000000000
1642023-12-13T05:11:58.136Zdflt 0.000000000
1652023-12-13T05:11:58.137Zkflt 0.000000000
1662023-12-13T05:11:58.137Zlock 0.000000000
1672023-12-13T05:11:58.137Zslp 0.000017950
1682023-12-13T05:11:58.137Zlat 0.000029350
1692023-12-13T05:11:58.138Zstop 0.000116991
1702023-12-13T05:11:58.138Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap-cc02af252c51050b
1712023-12-13T05:11:58.138Z+ chmod a+x tests/bootstrap-cc02af252c51050b
1722023-12-13T05:11:58.138Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1732023-12-13T05:11:58.138Z+ ptime -m gunzip
1742023-12-13T05:11:58.138Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap
1752023-12-13T05:12:00.348Z
1762023-12-13T05:12:00.374Zreal 2.218231299
1772023-12-13T05:12:00.374Zuser 2.090006562
1782023-12-13T05:12:00.374Zsys 0.127839040
1792023-12-13T05:12:00.374Ztrap 0.000170573
1802023-12-13T05:12:00.374Ztflt 0.000000000
1812023-12-13T05:12:00.374Zdflt 0.000000000
1822023-12-13T05:12:00.374Zkflt 0.000000000
1832023-12-13T05:12:00.375Zlock 0.000000000
1842023-12-13T05:12:00.375Zslp 0.000019030
1852023-12-13T05:12:00.375Zlat 0.000024340
1862023-12-13T05:12:00.375Zstop 0.000111662
1872023-12-13T05:12:00.376Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap
1882023-12-13T05:12:00.376Z+ chmod a+x tests/bootstrap
1892023-12-13T05:12:00.376Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1902023-12-13T05:12:00.376Z+ ptime -m gunzip
1912023-12-13T05:12:00.376Z++ basename /input/ci-tools/work/end-to-end-tests/end_to_end_tests-c0f56f9124e0823c
1922023-12-13T05:12:02.666Z
1932023-12-13T05:12:02.772Zreal 2.308621441
1942023-12-13T05:12:02.772Zuser 2.157044225
1952023-12-13T05:12:02.773Zsys 0.151148238
1962023-12-13T05:12:02.773Ztrap 0.000176194
1972023-12-13T05:12:02.773Ztflt 0.000000000
1982023-12-13T05:12:02.773Zdflt 0.000000000
1992023-12-13T05:12:02.773Zkflt 0.000000000
2002023-12-13T05:12:02.773Zlock 0.000000000
2012023-12-13T05:12:02.773Zslp 0.000021330
2022023-12-13T05:12:02.773Zlat 0.000033090
2032023-12-13T05:12:02.774Zstop 0.000115871
2042023-12-13T05:12:02.774Z++ basename /input/ci-tools/work/end-to-end-tests/end_to_end_tests-c0f56f9124e0823c
2052023-12-13T05:12:02.774Z+ chmod a+x tests/end_to_end_tests-c0f56f9124e0823c
2062023-12-13T05:12:02.774Z+ bmat address ls
2072023-12-13T05:12:02.774ZNAME CIDR FIRST LAST COUNT
2082023-12-13T05:12:02.774Zextra 10.151.2.0/24 10.151.2.164 10.151.2.227 64
2092023-12-13T05:12:02.775Z++ bmat address ls -f extra -Ho first,last,count
2102023-12-13T05:12:02.775Z+ set -- 10.151.2.164 10.151.2.227 64
2112023-12-13T05:12:02.776Z+ EXTRA_IP_START=10.151.2.164
2122023-12-13T05:12:02.776Z+ EXTRA_IP_END=10.151.2.227
2132023-12-13T05:12:02.776Z+ EXTRA_IP_COUNT=64
2142023-12-13T05:12:02.776Z+ (( EXTRA_IP_COUNT >= 32 ))
2152023-12-13T05:12:02.776Z+ EXTRA_IP_BASE=10.151.2
2162023-12-13T05:12:02.776Z+ EXTRA_IP_FOCTET=164
2172023-12-13T05:12:02.776Z+ EXTRA_IP_LOCTET=227
2182023-12-13T05:12:02.777Z+ SERVICE_IP_POOL_START=10.151.2.164
2192023-12-13T05:12:02.777Z+ SERVICE_IP_POOL_END=10.151.2.173
2202023-12-13T05:12:02.777Z+ DNS_IP1=10.151.2.164
2212023-12-13T05:12:02.777Z+ DNS_IP2=10.151.2.165
2222023-12-13T05:12:02.777Z+ UPLINK_IP=10.151.2.174
2232023-12-13T05:12:02.777Z+ IPPOOL_START=10.151.2.175
2242023-12-13T05:12:02.778Z+ IPPOOL_END=10.151.2.227
2252023-12-13T05:12:02.778Z++ ipadm show-addr -po type,addr
2262023-12-13T05:12:02.778Z++ awk '-F[:/]' '$1 == "dhcp" {print $2}'
2272023-12-13T05:12:02.778Z+ GATEWAY_IP=10.151.2.100
2282023-12-13T05:12:02.778Z+ [[ -n 10.151.2.100 ]]
2292023-12-13T05:12:02.778Z+ ping -s 10.151.2.100 56 1
2302023-12-13T05:12:02.779ZPING 10.151.2.100: 56 data bytes
2312023-12-13T05:12:02.779Z64 bytes from 10.151.2.100: icmp_seq=0. time=0.010 ms
2322023-12-13T05:12:02.779Z
2332023-12-13T05:12:02.779Z----10.151.2.100 PING Statistics----
2342023-12-13T05:12:02.779Z1 packets transmitted, 1 packets received, 0% packet loss
2352023-12-13T05:12:02.779Zround-trip (ms) min/avg/max/stddev = 0.010/0.010/0.010/-nan
2362023-12-13T05:12:02.780Z++ arp -an
2372023-12-13T05:12:02.781Z++ awk -vgw=10.151.2.100 '$2 == gw {print $NF}'
2382023-12-13T05:12:02.808Z+ GATEWAY_MAC=18:c0:4d:0d:a0:2a
2392023-12-13T05:12:02.808Z+ routeadm -e ipv4-forwarding -u
2402023-12-13T05:12:02.836Z+ PXA_START=10.151.2.164
2412023-12-13T05:12:02.836Z+ PXA_END=10.151.2.227
2422023-12-13T05:12:02.836Z+ export GATEWAY_IP GATEWAY_MAC PXA_START PXA_END
2432023-12-13T05:12:02.836Z+ pfexec zpool create -f scratch c1t1d0 c2t1d0
2442023-12-13T05:12:02.964Z+ ZPOOL_VDEV_DIR=/scratch
2452023-12-13T05:12:02.964Z+ ptime -m pfexec ./tools/create_virtual_hardware.sh
2462023-12-13T05:12:02.991Z+++ dirname ./tools/create_virtual_hardware.sh
2472023-12-13T05:12:02.991Z++ cd ./tools
2482023-12-13T05:12:02.991Z++ pwd
2492023-12-13T05:12:02.991Z+ SOURCE_DIR=/opt/oxide/work/tools
2502023-12-13T05:12:02.992Z+ OMICRON_TOP=/opt/oxide/work/tools/..
2512023-12-13T05:12:02.992Z+ SOFTNPU_MODE=zone
2522023-12-13T05:12:02.992Z+ . /opt/oxide/work/tools/virtual_hardware.sh
2532023-12-13T05:12:02.992Z++ MARKER=/etc/opt/oxide/NO_INSTALL
2542023-12-13T05:12:02.992Z++ [[ -f /etc/opt/oxide/NO_INSTALL ]]
2552023-12-13T05:12:02.992Z++ dladm show-phys -p -o LINK
2562023-12-13T05:12:02.992Z++ head -1
2572023-12-13T05:12:03.048Z+ PHYSICAL_LINK=igb0
2582023-12-13T05:12:03.075Z+ echo 'Using igb0 as physical link'
2592023-12-13T05:12:03.076ZUsing igb0 as physical link
2602023-12-13T05:12:03.076Z+ ensure_run_as_root
2612023-12-13T05:12:03.076Z++ id -u
2622023-12-13T05:12:03.077Z+ [[ 0 -ne 0 ]]
2632023-12-13T05:12:03.077Z+ ensure_zpools
2642023-12-13T05:12:03.077Z+ ZPOOL_TYPES=('oxp_' 'oxi_')
2652023-12-13T05:12:03.077Z+ for ZPOOL_TYPE in "${ZPOOL_TYPES[@]}"
2662023-12-13T05:12:03.078Z+ readarray -t ZPOOLS
2672023-12-13T05:12:03.078Z++ grep '"oxp_' /opt/oxide/work/tools/../smf/sled-agent/non-gimlet/config.toml
2682023-12-13T05:12:03.078Z++ sed 's/[ ",]//g'
2692023-12-13T05:12:03.078Z+ for ZPOOL in "${ZPOOLS[@]}"
2702023-12-13T05:12:03.078Z+ echo 'Zpool: [oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b]'
2712023-12-13T05:12:03.078ZZpool: [oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b]
2722023-12-13T05:12:03.078Z+ VDEV_PATH=/scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
2732023-12-13T05:12:03.078Z+ [[ -f /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
2742023-12-13T05:12:03.078Z+ dd if=/dev/zero of=/scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
2752023-12-13T05:12:03.105Z0+0 records in
2762023-12-13T05:12:03.105Z0+0 records out
2772023-12-13T05:12:03.105Z0 bytes (0 iB) transferred in 0.000005 secs (0 iB/sec)
2782023-12-13T05:12:03.106Z+ success 'ZFS vdev /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
2792023-12-13T05:12:03.106Z+ set +x
2802023-12-13T05:12:03.106ZZFS vdev /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
2812023-12-13T05:12:03.106Z++ zpool list -o name
2822023-12-13T05:12:03.106Z++ grep oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b
2832023-12-13T05:12:03.106Z+ [[ -z '' ]]
2842023-12-13T05:12:03.106Z+ zpool create -f oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
2852023-12-13T05:12:03.137Z+ success 'ZFS zpool oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
2862023-12-13T05:12:03.163Z+ set +x
2872023-12-13T05:12:03.164ZZFS zpool oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b exists
2882023-12-13T05:12:03.165Z+ for ZPOOL in "${ZPOOLS[@]}"
2892023-12-13T05:12:03.166Z+ echo 'Zpool: [oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03]'
2902023-12-13T05:12:03.166ZZpool: [oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03]
2912023-12-13T05:12:03.166Z+ VDEV_PATH=/scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
2922023-12-13T05:12:03.166Z+ [[ -f /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
2932023-12-13T05:12:03.166Z+ dd if=/dev/zero of=/scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
2942023-12-13T05:12:03.166Z0+0 records in
2952023-12-13T05:12:03.166Z0+0 records out
2962023-12-13T05:12:03.166Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
2972023-12-13T05:12:03.166Z+ success 'ZFS vdev /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
2982023-12-13T05:12:03.166Z+ set +x
2992023-12-13T05:12:03.166ZZFS vdev /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
3002023-12-13T05:12:03.166Z++ zpool list -o name
3012023-12-13T05:12:03.166Z++ grep oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03
3022023-12-13T05:12:03.167Z+ [[ -z '' ]]
3032023-12-13T05:12:03.167Z+ zpool create -f oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3042023-12-13T05:12:03.199Z+ success 'ZFS zpool oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3052023-12-13T05:12:03.200Z+ set +x
3062023-12-13T05:12:03.200ZZFS zpool oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
3072023-12-13T05:12:03.200Z+ for ZPOOL in "${ZPOOLS[@]}"
3082023-12-13T05:12:03.200Z+ echo 'Zpool: [oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3092023-12-13T05:12:03.200ZZpool: [oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03]
3102023-12-13T05:12:03.200Z+ VDEV_PATH=/scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3112023-12-13T05:12:03.200Z+ [[ -f /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3122023-12-13T05:12:03.201Z+ dd if=/dev/zero of=/scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3132023-12-13T05:12:03.201Z0+0 records in
3142023-12-13T05:12:03.201Z0+0 records out
3152023-12-13T05:12:03.201Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3162023-12-13T05:12:03.202Z+ success 'ZFS vdev /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3172023-12-13T05:12:03.202Z+ set +x
3182023-12-13T05:12:03.202ZZFS vdev /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
3192023-12-13T05:12:03.202Z++ zpool list -o name
3202023-12-13T05:12:03.202Z++ grep oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03
3212023-12-13T05:12:03.202Z+ [[ -z '' ]]
3222023-12-13T05:12:03.202Z+ zpool create -f oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3232023-12-13T05:12:03.233Z+ success 'ZFS zpool oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3242023-12-13T05:12:03.233Z+ set +x
3252023-12-13T05:12:03.267ZZFS zpool oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
3262023-12-13T05:12:03.267Z+ for ZPOOL in "${ZPOOLS[@]}"
3272023-12-13T05:12:03.268Z+ echo 'Zpool: [oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3282023-12-13T05:12:03.268ZZpool: [oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03]
3292023-12-13T05:12:03.268Z+ VDEV_PATH=/scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3302023-12-13T05:12:03.268Z+ [[ -f /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3312023-12-13T05:12:03.268Z+ dd if=/dev/zero of=/scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3322023-12-13T05:12:03.268Z0+0 records in
3332023-12-13T05:12:03.268Z0+0 records out
3342023-12-13T05:12:03.268Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3352023-12-13T05:12:03.268Z+ success 'ZFS vdev /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3362023-12-13T05:12:03.269Z+ set +x
3372023-12-13T05:12:03.269ZZFS vdev /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
3382023-12-13T05:12:03.269Z++ zpool list -o name
3392023-12-13T05:12:03.269Z++ grep oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03
3402023-12-13T05:12:03.270Z+ [[ -z '' ]]
3412023-12-13T05:12:03.270Z+ zpool create -f oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3422023-12-13T05:12:03.297Z+ success 'ZFS zpool oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3432023-12-13T05:12:03.297Z+ set +x
3442023-12-13T05:12:03.297ZZFS zpool oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
3452023-12-13T05:12:03.297Z+ for ZPOOL in "${ZPOOLS[@]}"
3462023-12-13T05:12:03.297Z+ echo 'Zpool: [oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3472023-12-13T05:12:03.297ZZpool: [oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03]
3482023-12-13T05:12:03.298Z+ VDEV_PATH=/scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3492023-12-13T05:12:03.298Z+ [[ -f /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3502023-12-13T05:12:03.298Z+ dd if=/dev/zero of=/scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3512023-12-13T05:12:03.298Z0+0 records in
3522023-12-13T05:12:03.300Z0+0 records out
3532023-12-13T05:12:03.300Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
3542023-12-13T05:12:03.300Z+ success 'ZFS vdev /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3552023-12-13T05:12:03.300Z+ set +x
3562023-12-13T05:12:03.300ZZFS vdev /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
3572023-12-13T05:12:03.300Z++ zpool list -o name
3582023-12-13T05:12:03.300Z++ grep oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03
3592023-12-13T05:12:03.301Z+ [[ -z '' ]]
3602023-12-13T05:12:03.301Z+ zpool create -f oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3612023-12-13T05:12:03.329Z+ success 'ZFS zpool oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3622023-12-13T05:12:03.329Z+ set +x
3632023-12-13T05:12:03.329ZZFS zpool oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
3642023-12-13T05:12:03.330Z+ for ZPOOL in "${ZPOOLS[@]}"
3652023-12-13T05:12:03.330Z+ echo 'Zpool: [oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930]'
3662023-12-13T05:12:03.330ZZpool: [oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930]
3672023-12-13T05:12:03.330Z+ VDEV_PATH=/scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev
3682023-12-13T05:12:03.330Z+ [[ -f /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev ]]
3692023-12-13T05:12:03.330Z+ dd if=/dev/zero of=/scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev bs=1 count=0 seek=20G
3702023-12-13T05:12:03.357Z0+0 records in
3712023-12-13T05:12:03.357Z0+0 records out
3722023-12-13T05:12:03.357Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3732023-12-13T05:12:03.357Z+ success 'ZFS vdev /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev exists'
3742023-12-13T05:12:03.357Z+ set +x
3752023-12-13T05:12:03.357ZZFS vdev /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev exists
3762023-12-13T05:12:03.357Z++ zpool list -o name
3772023-12-13T05:12:03.357Z++ grep oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930
3782023-12-13T05:12:03.358Z+ [[ -z '' ]]
3792023-12-13T05:12:03.358Z+ zpool create -f oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev
3802023-12-13T05:12:03.384Z+ success 'ZFS zpool oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 exists'
3812023-12-13T05:12:03.384Z+ set +x
3822023-12-13T05:12:03.384ZZFS zpool oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 exists
3832023-12-13T05:12:03.384Z+ for ZPOOL in "${ZPOOLS[@]}"
3842023-12-13T05:12:03.384Z+ echo 'Zpool: [oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60]'
3852023-12-13T05:12:03.384ZZpool: [oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60]
3862023-12-13T05:12:03.385Z+ VDEV_PATH=/scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev
3872023-12-13T05:12:03.385Z+ [[ -f /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev ]]
3882023-12-13T05:12:03.385Z+ dd if=/dev/zero of=/scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev bs=1 count=0 seek=20G
3892023-12-13T05:12:03.385Z0+0 records in
3902023-12-13T05:12:03.385Z0+0 records out
3912023-12-13T05:12:03.385Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3922023-12-13T05:12:03.385Z+ success 'ZFS vdev /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev exists'
3932023-12-13T05:12:03.385Z+ set +x
3942023-12-13T05:12:03.385ZZFS vdev /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev exists
3952023-12-13T05:12:03.385Z++ zpool list -o name
3962023-12-13T05:12:03.385Z++ grep oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60
3972023-12-13T05:12:03.412Z+ [[ -z '' ]]
3982023-12-13T05:12:03.413Z+ zpool create -f oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev
3992023-12-13T05:12:03.439Z+ success 'ZFS zpool oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 exists'
4002023-12-13T05:12:03.439Z+ set +x
4012023-12-13T05:12:03.439ZZFS zpool oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 exists
4022023-12-13T05:12:03.439Z+ for ZPOOL in "${ZPOOLS[@]}"
4032023-12-13T05:12:03.439Z+ echo 'Zpool: [oxp_31bd71cd-4736-4a12-a387-9b74b050396f]'
4042023-12-13T05:12:03.440ZZpool: [oxp_31bd71cd-4736-4a12-a387-9b74b050396f]
4052023-12-13T05:12:03.440Z+ VDEV_PATH=/scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev
4062023-12-13T05:12:03.440Z+ [[ -f /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev ]]
4072023-12-13T05:12:03.440Z+ dd if=/dev/zero of=/scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev bs=1 count=0 seek=20G
4082023-12-13T05:12:03.440Z0+0 records in
4092023-12-13T05:12:03.441Z0+0 records out
4102023-12-13T05:12:03.441Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4112023-12-13T05:12:03.441Z+ success 'ZFS vdev /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev exists'
4122023-12-13T05:12:03.442Z+ set +x
4132023-12-13T05:12:03.442ZZFS vdev /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev exists
4142023-12-13T05:12:03.442Z++ zpool list -o name
4152023-12-13T05:12:03.442Z++ grep oxp_31bd71cd-4736-4a12-a387-9b74b050396f
4162023-12-13T05:12:03.469Z+ [[ -z '' ]]
4172023-12-13T05:12:03.469Z+ zpool create -f oxp_31bd71cd-4736-4a12-a387-9b74b050396f /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev
4182023-12-13T05:12:03.496Z+ success 'ZFS zpool oxp_31bd71cd-4736-4a12-a387-9b74b050396f exists'
4192023-12-13T05:12:03.496Z+ set +x
4202023-12-13T05:12:03.496ZZFS zpool oxp_31bd71cd-4736-4a12-a387-9b74b050396f exists
4212023-12-13T05:12:03.496Z+ for ZPOOL in "${ZPOOLS[@]}"
4222023-12-13T05:12:03.496Z+ echo 'Zpool: [oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7]'
4232023-12-13T05:12:03.496ZZpool: [oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7]
4242023-12-13T05:12:03.496Z+ VDEV_PATH=/scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev
4252023-12-13T05:12:03.497Z+ [[ -f /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev ]]
4262023-12-13T05:12:03.497Z+ dd if=/dev/zero of=/scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev bs=1 count=0 seek=20G
4272023-12-13T05:12:03.497Z0+0 records in
4282023-12-13T05:12:03.497Z0+0 records out
4292023-12-13T05:12:03.497Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4302023-12-13T05:12:03.497Z+ success 'ZFS vdev /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev exists'
4312023-12-13T05:12:03.497Z+ set +x
4322023-12-13T05:12:03.498ZZFS vdev /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev exists
4332023-12-13T05:12:03.498Z++ zpool list -o name
4342023-12-13T05:12:03.499Z++ grep oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7
4352023-12-13T05:12:03.499Z+ [[ -z '' ]]
4362023-12-13T05:12:03.499Z+ zpool create -f oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev
4372023-12-13T05:12:03.552Z+ success 'ZFS zpool oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 exists'
4382023-12-13T05:12:03.552Z+ set +x
4392023-12-13T05:12:03.580ZZFS zpool oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 exists
4402023-12-13T05:12:03.580Z+ for ZPOOL_TYPE in "${ZPOOL_TYPES[@]}"
4412023-12-13T05:12:03.580Z+ readarray -t ZPOOLS
4422023-12-13T05:12:03.580Z++ grep '"oxi_' /opt/oxide/work/tools/../smf/sled-agent/non-gimlet/config.toml
4432023-12-13T05:12:03.580Z++ sed 's/[ ",]//g'
4442023-12-13T05:12:03.580Z+ for ZPOOL in "${ZPOOLS[@]}"
4452023-12-13T05:12:03.581Z+ echo 'Zpool: [oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b]'
4462023-12-13T05:12:03.582ZZpool: [oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b]
4472023-12-13T05:12:03.582Z+ VDEV_PATH=/scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4482023-12-13T05:12:03.582Z+ [[ -f /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
4492023-12-13T05:12:03.582Z+ dd if=/dev/zero of=/scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
4502023-12-13T05:12:03.582Z0+0 records in
4512023-12-13T05:12:03.582Z0+0 records out
4522023-12-13T05:12:03.582Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4532023-12-13T05:12:03.582Z+ success 'ZFS vdev /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
4542023-12-13T05:12:03.582Z+ set +x
4552023-12-13T05:12:03.582ZZFS vdev /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
4562023-12-13T05:12:03.582Z++ zpool list -o name
4572023-12-13T05:12:03.583Z++ grep oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b
4582023-12-13T05:12:03.583Z+ [[ -z '' ]]
4592023-12-13T05:12:03.584Z+ zpool create -f oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4602023-12-13T05:12:03.666Z+ success 'ZFS zpool oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
4612023-12-13T05:12:03.666Z+ set +x
4622023-12-13T05:12:03.666ZZFS zpool oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b exists
4632023-12-13T05:12:03.666Z+ for ZPOOL in "${ZPOOLS[@]}"
4642023-12-13T05:12:03.667Z+ echo 'Zpool: [oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b]'
4652023-12-13T05:12:03.667ZZpool: [oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b]
4662023-12-13T05:12:03.667Z+ VDEV_PATH=/scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4672023-12-13T05:12:03.667Z+ [[ -f /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
4682023-12-13T05:12:03.667Z+ dd if=/dev/zero of=/scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
4692023-12-13T05:12:03.667Z0+0 records in
4702023-12-13T05:12:03.667Z0+0 records out
4712023-12-13T05:12:03.667Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4722023-12-13T05:12:03.667Z+ success 'ZFS vdev /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
4732023-12-13T05:12:03.667Z+ set +x
4742023-12-13T05:12:03.667ZZFS vdev /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
4752023-12-13T05:12:03.667Z++ zpool list -o name
4762023-12-13T05:12:03.668Z++ grep oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b
4772023-12-13T05:12:03.668Z+ [[ -z '' ]]
4782023-12-13T05:12:03.668Z+ zpool create -f oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4792023-12-13T05:12:03.668Z+ success 'ZFS zpool oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
4802023-12-13T05:12:03.668Z+ set +x
4812023-12-13T05:12:03.668ZZFS zpool oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b exists
4822023-12-13T05:12:03.668Z+ [[ zone == \z\o\n\e ]]
4832023-12-13T05:12:03.668Z+ ensure_simulated_links igb0
4842023-12-13T05:12:03.691Z+ local PHYSICAL_LINK=igb0
4852023-12-13T05:12:03.692Z+ INDICES=("0" "1")
4862023-12-13T05:12:03.692Z+ for I in "${INDICES[@]}"
4872023-12-13T05:12:03.692Z++ get_simnet_name_if_exists net0
4882023-12-13T05:12:03.692Z++ dladm show-simnet -p -o LINK net0
4892023-12-13T05:12:03.692Z++ echo ''
4902023-12-13T05:12:03.692Z+ [[ -z '' ]]
4912023-12-13T05:12:03.692Z+ dladm create-simnet -t net0
4922023-12-13T05:12:03.692Z+ dladm create-simnet -t sc0_0
4932023-12-13T05:12:03.725Z+ dladm modify-simnet -t -p net0 sc0_0
4942023-12-13T05:12:03.725Z+ dladm set-linkprop -p mtu=9000 sc0_0
4952023-12-13T05:12:03.726Z+ success 'Simnet net0/sc0_0 exists'
4962023-12-13T05:12:03.726Z+ set +x
4972023-12-13T05:12:03.726ZSimnet net0/sc0_0 exists
4982023-12-13T05:12:03.726Z+ for I in "${INDICES[@]}"
4992023-12-13T05:12:03.726Z++ get_simnet_name_if_exists net1
5002023-12-13T05:12:03.726Z++ dladm show-simnet -p -o LINK net1
5012023-12-13T05:12:03.726Z++ echo ''
5022023-12-13T05:12:03.726Z+ [[ -z '' ]]
5032023-12-13T05:12:03.726Z+ dladm create-simnet -t net1
5042023-12-13T05:12:03.726Z+ dladm create-simnet -t sc1_0
5052023-12-13T05:12:03.726Z+ dladm modify-simnet -t -p net1 sc1_0
5062023-12-13T05:12:03.726Z+ dladm set-linkprop -p mtu=9000 sc1_0
5072023-12-13T05:12:03.726Z+ success 'Simnet net1/sc1_0 exists'
5082023-12-13T05:12:03.727Z+ set +x
5092023-12-13T05:12:03.727ZSimnet net1/sc1_0 exists
5102023-12-13T05:12:03.728Z++ get_vnic_name_if_exists sc0_1
5112023-12-13T05:12:03.728Z++ dladm show-vnic -p -o LINK sc0_1
5122023-12-13T05:12:03.728Z++ echo ''
5132023-12-13T05:12:03.728Z+ [[ -z '' ]]
5142023-12-13T05:12:03.728Z+ dladm create-vnic -t sc0_1 -l igb0 -m a8:e1:de:01:70:1d
5152023-12-13T05:12:03.728Z+ [[ -v PROMISC_FILT_OFF ]]
5162023-12-13T05:12:03.728Z+ success 'Vnic sc0_1 exists'
5172023-12-13T05:12:03.729Z+ set +x
5182023-12-13T05:12:03.729ZVnic sc0_1 exists
5192023-12-13T05:12:03.729Z+ warn_if_no_proxy_arp
5202023-12-13T05:12:03.729Z+ [[ -v PXA_START ]]
5212023-12-13T05:12:03.729Z+ [[ -v PXA_END ]]
5222023-12-13T05:12:03.729Z+ ensure_softnpu_zone
5232023-12-13T05:12:03.729Z+ zoneadm list
5242023-12-13T05:12:03.729Z+ grep -q sidecar_softnpu
5252023-12-13T05:12:03.729Z+ [[ -f out/npuzone/npuzone ]]
5262023-12-13T05:12:03.729Z+ out/npuzone/npuzone create sidecar --omicron-zone --ports sc0_0,tfportrear0_0 --ports sc0_1,tfportqsfp0_0 --sidecar-lite-commit 45ed98fea5824feb4d42f45bbf218e597dc9fc58 --softnpu-commit dbab082dfa89da5db5ca2325c257089d2f130092
5272023-12-13T05:12:03.763Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/softnpu/image/dbab082dfa89da5db5ca2325c257089d2f130092/softnpu.sha256.txt
5282023-12-13T05:12:13.125Zdownloading softnpu ...done
5292023-12-13T05:12:13.163Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/sidecar-lite/release/45ed98fea5824feb4d42f45bbf218e597dc9fc58/libsidecar_lite.so.sha256.txt
5302023-12-13T05:12:21.001Zdownloading asic_program.so ...done
5312023-12-13T05:12:21.030Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/sidecar-lite/release/45ed98fea5824feb4d42f45bbf218e597dc9fc58/scadm.sha256.txt
5322023-12-13T05:12:28.792Zdownloading scadm ...done
5332023-12-13T05:12:28.820Zcreating port tfportrear0_0 for sc0_0
5342023-12-13T05:12:28.820Zcreating port tfportqsfp0_0 for sc0_1
5352023-12-13T05:12:28.846Zsoftnpu zone setup
5362023-12-13T05:12:28.846Zconfigure zone
5372023-12-13T05:12:28.846Zinstall zone
5382023-12-13T05:12:29.902Zboot zone
5392023-12-13T05:12:33.008Z[sidecar_softnpu] svcs milestone/network
5402023-12-13T05:12:33.035ZSTATE STIME FMRI
5412023-12-13T05:12:33.035Zonline 5:12:32 svc:/milestone/network:default
5422023-12-13T05:12:33.035Z[sidecar_softnpu] /softnpu/npu start
5432023-12-13T05:12:33.060Z
5442023-12-13T05:12:33.086Z+ /opt/oxide/work/tools/scrimlet/softnpu-init.sh
5452023-12-13T05:12:33.086Z+ GATEWAY_IP=10.151.2.100
5462023-12-13T05:12:33.086Z+ echo 'Using 10.151.2.100 as gateway ip'
5472023-12-13T05:12:33.086ZUsing 10.151.2.100 as gateway ip
5482023-12-13T05:12:33.086Z+ [[ ! -v GATEWAY_MAC ]]
5492023-12-13T05:12:33.086Z+ GATEWAY_MAC=18:c0:4d:0d:a0:2a
5502023-12-13T05:12:33.088Z++ grep -c -E '^([0-9a-fA-F]{2}:){5}[0-9a-fA-F]{2}$' /dev/fd/63
5512023-12-13T05:12:33.088Z+++ echo 18:c0:4d:0d:a0:2a
5522023-12-13T05:12:33.088Z+ COUNT=1
5532023-12-13T05:12:33.088Z+ [[ 1 -ne 1 ]]
5542023-12-13T05:12:33.088Z+ echo 'Using 18:c0:4d:0d:a0:2a as gateway mac'
5552023-12-13T05:12:33.088ZUsing 18:c0:4d:0d:a0:2a as gateway mac
5562023-12-13T05:12:33.088Z+ z_scadm add-arp-entry 10.151.2.100 18:c0:4d:0d:a0:2a
5572023-12-13T05:12:33.088Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone add-arp-entry 10.151.2.100 18:c0:4d:0d:a0:2a
5582023-12-13T05:12:33.140Z+ PXA_MAC=a8:e1:de:01:70:1d
5592023-12-13T05:12:33.140Z+ [[ -v PXA_START ]]
5602023-12-13T05:12:33.140Z+ z_scadm add-proxy-arp 10.151.2.164 10.151.2.227 a8:e1:de:01:70:1d
5612023-12-13T05:12:33.140Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone add-proxy-arp 10.151.2.164 10.151.2.227 a8:e1:de:01:70:1d
5622023-12-13T05:12:33.165Z+ z_scadm dump-state
5632023-12-13T05:12:33.165Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone dump-state
5642023-12-13T05:12:33.165Zlocal v6:
5652023-12-13T05:12:33.165Zlocal v4:
5662023-12-13T05:12:33.166Zrouter v6:
5672023-12-13T05:12:33.166Zrouter v4_idx:
5682023-12-13T05:12:33.166Zrouter v4_routes:
5692023-12-13T05:12:33.166Zresolver v4:
5702023-12-13T05:12:33.166Z10.151.2.100 -> 18:c0:4d:0d:a0:2a
5712023-12-13T05:12:33.166Zresolver v6:
5722023-12-13T05:12:33.166Znat v4:
5732023-12-13T05:12:33.166Znat_v6:
5742023-12-13T05:12:33.166Zport_mac:
5752023-12-13T05:12:33.166Zproxy arp:
5762023-12-13T05:12:33.166Z10.151.2.164/10.151.2.227: a8:e1:de:01:70:1d
5772023-12-13T05:12:33.167Z+ success 'softnpu zone exists'
5782023-12-13T05:12:33.167Z+ set +x
5792023-12-13T05:12:33.167Zsoftnpu zone exists
5802023-12-13T05:12:33.167Z
5812023-12-13T05:12:33.167Zreal 30.193974037
5822023-12-13T05:12:33.167Zuser 0.708860456
5832023-12-13T05:12:33.167Zsys 1.642786569
5842023-12-13T05:12:33.167Ztrap 0.000079369
5852023-12-13T05:12:33.167Ztflt 0.000292456
5862023-12-13T05:12:33.167Zdflt 0.001104524
5872023-12-13T05:12:33.167Zkflt 0.000000000
5882023-12-13T05:12:33.167Zlock 23:26.225892175
5892023-12-13T05:12:33.167Zslp 1:30.377067533
5902023-12-13T05:12:33.167Zlat 0.080876347
5912023-12-13T05:12:33.167Zstop 0.345375179
5922023-12-13T05:12:33.167Z+ tar xf out/omicron-sled-agent.tar pkg/config-rss.toml
5932023-12-13T05:12:33.203Z++ sed -n 's/silo_name = "\(.*\)"/\1/p' pkg/config-rss.toml
5942023-12-13T05:12:33.236Z+ SILO_NAME=recovery
5952023-12-13T05:12:33.237Z++ sed -n 's/external_dns_zone_name = "\(.*\)"/\1/p' pkg/config-rss.toml
5962023-12-13T05:12:33.237Z+ EXTERNAL_DNS_DOMAIN=oxide.test
5972023-12-13T05:12:33.237Z+ sed -i~ '
5982023-12-13T05:12:33.237Z /^external_dns_ips/c\
5992023-12-13T05:12:33.237Zexternal_dns_ips = [ "10.151.2.164", "10.151.2.165" ]
6002023-12-13T05:12:33.237Z /^\[\[internal_services_ip_pool_ranges/,/^$/ {
6012023-12-13T05:12:33.237Z /^first/c\
6022023-12-13T05:12:33.237Zfirst = "10.151.2.164"
6032023-12-13T05:12:33.237Z /^last/c\
6042023-12-13T05:12:33.237Zlast = "10.151.2.173"
6052023-12-13T05:12:33.238Z }
6062023-12-13T05:12:33.238Z /^\[rack_network_config/,/^$/ {
6072023-12-13T05:12:33.238Z /^infra_ip_first/c\
6082023-12-13T05:12:33.238Zinfra_ip_first = "10.151.2.174"
6092023-12-13T05:12:33.238Z /^infra_ip_last/c\
6102023-12-13T05:12:33.238Zinfra_ip_last = "10.151.2.174"
6112023-12-13T05:12:33.238Z }
6122023-12-13T05:12:33.238Z /^\[\[rack_network_config.ports/,/^$/ {
6132023-12-13T05:12:33.238Z /^routes/c\
6142023-12-13T05:12:33.238Zroutes = \[{nexthop = "10.151.2.100", destination = "0.0.0.0/0"}\]
6152023-12-13T05:12:33.239Z /^addresses/c\
6162023-12-13T05:12:33.239Zaddresses = \["10.151.2.174/32"\]
6172023-12-13T05:12:33.239Z }
6182023-12-13T05:12:33.239Z' pkg/config-rss.toml
6192023-12-13T05:12:33.239Z+ diff -u pkg/config-rss.toml~ pkg/config-rss.toml
6202023-12-13T05:12:33.239Z--- pkg/config-rss.toml~ Mon Jul 24 01:21:28 2006
6212023-12-13T05:12:33.239Z+++ pkg/config-rss.toml Wed Dec 13 05:12:33 2023
6222023-12-13T05:12:33.239Z@@ -41,7 +41,7 @@
6232023-12-13T05:12:33.239Z # the DNS domain delegated to the rack by the customer. Each of these addresses
6242023-12-13T05:12:33.239Z # must be contained in one of the "internal services" IP Pool ranges listed
6252023-12-13T05:12:33.239Z # below.
6262023-12-13T05:12:33.239Z-external_dns_ips = [ "192.168.1.20", "192.168.1.21" ]
6272023-12-13T05:12:33.239Z+external_dns_ips = [ "10.151.2.164", "10.151.2.165" ]
6282023-12-13T05:12:33.239Z
6292023-12-13T05:12:33.239Z # Initial TLS certificates for the external API
6302023-12-13T05:12:33.239Z #
6312023-12-13T05:12:33.241Z@@ -77,8 +77,8 @@
6322023-12-13T05:12:33.241Z #
6332023-12-13T05:12:33.241Z # For more on this and what to put here, see docs/how-to-run.adoc.
6342023-12-13T05:12:33.242Z [[internal_services_ip_pool_ranges]]
6352023-12-13T05:12:33.242Z-first = "192.168.1.20"
6362023-12-13T05:12:33.242Z-last = "192.168.1.29"
6372023-12-13T05:12:33.242Z+first = "10.151.2.164"
6382023-12-13T05:12:33.242Z+last = "10.151.2.173"
6392023-12-13T05:12:33.242Z
6402023-12-13T05:12:33.242Z # TODO - this configuration is subject to change going forward. Ultimately these
6412023-12-13T05:12:33.242Z # parameters should be provided to the control plane via wicket, but we need to
6422023-12-13T05:12:33.242Z@@ -92,8 +92,8 @@
6432023-12-13T05:12:33.242Z # A range of IP addresses used by Boundary Services on the external network. In
6442023-12-13T05:12:33.242Z # a real system, these would be addresses of the uplink ports on the Sidecar.
6452023-12-13T05:12:33.242Z # With softnpu, only one address is used.
6462023-12-13T05:12:33.242Z-infra_ip_first = "192.168.1.30"
6472023-12-13T05:12:33.242Z-infra_ip_last = "192.168.1.30"
6482023-12-13T05:12:33.242Z+infra_ip_first = "10.151.2.174"
6492023-12-13T05:12:33.242Z+infra_ip_last = "10.151.2.174"
6502023-12-13T05:12:33.243Z
6512023-12-13T05:12:33.243Z # Configurations for BGP routers to run on the scrimlets.
6522023-12-13T05:12:33.243Z bgp = []
6532023-12-13T05:12:33.243Z@@ -101,9 +101,9 @@
6542023-12-13T05:12:33.243Z # You can configure multiple uplinks by repeating the following stanza
6552023-12-13T05:12:33.243Z [[rack_network_config.ports]]
6562023-12-13T05:12:33.243Z # Routes associated with this port.
6572023-12-13T05:12:33.243Z-routes = [{nexthop = "192.168.1.199", destination = "0.0.0.0/0"}]
6582023-12-13T05:12:33.243Z+routes = [{nexthop = "10.151.2.100", destination = "0.0.0.0/0"}]
6592023-12-13T05:12:33.243Z # Addresses associated with this port.
6602023-12-13T05:12:33.243Z-addresses = ["192.168.1.30/32"]
6612023-12-13T05:12:33.243Z+addresses = ["10.151.2.174/32"]
6622023-12-13T05:12:33.243Z # Name of the uplink port. This should always be "qsfp0" when using softnpu.
6632023-12-13T05:12:33.243Z port = "qsfp0"
6642023-12-13T05:12:33.243Z # The speed of this port.
6652023-12-13T05:12:33.243Z+ true
6662023-12-13T05:12:33.243Z+ tar rvf out/omicron-sled-agent.tar pkg/config-rss.toml
6672023-12-13T05:12:33.243Za pkg/config-rss.toml 7K
6682023-12-13T05:12:33.243Z+ rm -f pkg/config-rss.toml pkg/config-rss.toml~
6692023-12-13T05:12:33.243Z+ TLS_NAME=recovery.sys.oxide.test
6702023-12-13T05:12:33.243Z+ openssl req -newkey rsa:4096 -x509 -sha256 -days 3 -nodes -out pkg/initial-tls-cert.pem -keyout pkg/initial-tls-key.pem -subj /CN=recovery.sys.oxide.test -addext subjectAltName=DNS:recovery.sys.oxide.test -addext basicConstraints=critical,CA:FALSE -config /dev/stdin
6712023-12-13T05:12:34.173Z......+...+...+...............+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*......+.........................+......+...+...+...+..+....+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*......+.+..+............+....+.................+...+....+...+...+...........+.+..+.......+...+......+..+....+...+........+....+.....+.+.........+........+..........+............+...+...+.....+...+.........................+..............+.+......+.....+...............+......+...+.+..............+.+..............+.......+...+.........+......+...............+......+.....+....+..............+.+..+....+.....+...............+.............+.....+.+...........+......+...+....+...+.....+...+....+..+.+...+...........+.........+......+......+.......+..+.........+....+...+.....+...................+......+............+..+....+............+..+.+.................+...+............+...+.........+...+...............+..................+....+..+.+..+.......+..............+............+..........+.....+...+........................+..........+........+..................+..........+..+......................+..+...............+.+..+.+.........+.....+......+.........+.......+......+..............+...+.+..+...+...............+.+..................+......+..+....+.....+.........+.......+........+.......+........+...............+....+...........+...........................+....+...+.....+.......+......+..+...+.......................................+.........+....+...+.....+.........+............................+...........+.+...............+.....+.........+.........+......+...............+......+.+.......................+...............+.+.....................+...+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
6722023-12-13T05:12:35.285Z......+...+........+....+.....+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.......+....+...+..+...+....+...............+...........+...+..........+..............+...............+......+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*......+.+..+.+.....+.........+............+.+..+......+.....................+....+..+....+...+............+........+.+.........+...+..................+.........+........+...........................+...+....+.....+....+..+....+...+........+...+......+...............+...............+.............+...+...+..+..........+..............+.............+......+..........................+.+........+......+..........+...+..+....+..+.........+..........+.....+......+............+.............+........+......+.+..+...+....+...........+.............+...........+.+...+......+...+....................+.+..+....+........+.+...........+....+.....+......+...+...+.......+........+.......+.....+............+....+..+.......+...+............+...+.......................+...............+.......+...............+.....+...........................+......+.......+.........+.........+..............+.......+...+........+.......+..+.+..+.+.........+...............+..................+.........+.........+..+...+......+.............+..+......+.........+...+.......+.........+.....................+.....+...............+.+..+...+..........+............+......+..+.........+...+...............+......+.+...+.....+.+..+...+.+...+............+...........+.......+...+..............+.+...+.........+............+...........+.......+.....+.+............+.........+........+.+.........+...+.....+....+.........+.....+.......+............+..+...+......+.........+.+..+.........+.........+.......+...+..+..........+..+...+.......+.............................................+...........+....+...........+......+.+..+.+......+.....+.+........+............+.........+.+...+.....................+......+..+......+......+...............+...+.+......+......+.....+.......+..+.............+.....+..........+......+.........+...+..+......+.........+............+.+.........+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
6732023-12-13T05:12:35.311Z-----
6742023-12-13T05:12:35.311Z+ tar rvf out/omicron-sled-agent.tar pkg/initial-tls-cert.pem pkg/initial-tls-key.pem
6752023-12-13T05:12:35.336Za pkg/initial-tls-cert.pem 2K
6762023-12-13T05:12:35.337Za pkg/initial-tls-key.pem 4K
6772023-12-13T05:12:35.337Z+ rm -f pkg/initial-tls-cert.pem pkg/initial-tls-key.pem
6782023-12-13T05:12:35.337Z+ rmdir pkg
6792023-12-13T05:12:35.337Z+ E2E_TLS_CERT=/opt/oxide/sled-agent/pkg/initial-tls-cert.pem
6802023-12-13T05:12:35.337Z+ pfexec mkdir -p /usr/oxide
6812023-12-13T05:12:35.337Z+ pfexec curl -sSfL -o /usr/oxide/oxide http://catacomb.eng.oxide.computer:12346/oxide-v0.1.0
6822023-12-13T05:12:36.020Z+ pfexec chmod +x /usr/oxide/oxide
6832023-12-13T05:12:36.046Z+ curl -sSfL -o debian-11-genericcloud-amd64.raw http://catacomb.eng.oxide.computer:12346/debian-11-genericcloud-amd64.raw
6842023-12-13T05:12:54.328Z+ OMICRON_NO_UNINSTALL=1
6852023-12-13T05:12:54.328Z+ ptime -m pfexec ./target/release/omicron-package -t test install
6862023-12-13T05:12:54.353ZDec 13 05:12:55.017 DEBG target[test]: Target({"image": "standard", "machine": "non-gimlet", "rack-topology": "single-sled", "switch": "softnpu"})
6872023-12-13T05:12:54.353ZDec 13 05:12:55.020 INFO Installing service, dst: /opt/oxide/clickhouse.tar.gz, src: out/clickhouse.tar.gz
6882023-12-13T05:12:54.353ZDec 13 05:12:55.020 INFO Installing service, dst: /opt/oxide/switch.tar.gz, src: out/switch-softnpu.tar.gz
6892023-12-13T05:12:54.353ZDec 13 05:12:55.020 INFO Installing service, dst: /opt/oxide/sled-agent.tar, src: out/omicron-sled-agent.tar
6902023-12-13T05:12:54.353ZDec 13 05:12:55.020 INFO Installing service, dst: /opt/oxide/internal_dns.tar.gz, src: out/internal-dns.tar.gz
6912023-12-13T05:12:54.353ZDec 13 05:12:55.020 INFO Installing service, dst: /opt/oxide/clickhouse_keeper.tar.gz, src: out/clickhouse_keeper.tar.gz
6922023-12-13T05:12:54.353ZDec 13 05:12:55.020 INFO Installing service, dst: /opt/oxide/mg-ddm.tar, src: out/mg-ddm-gz.tar
6932023-12-13T05:12:54.353ZDec 13 05:12:55.020 INFO Installing service, dst: /opt/oxide/mgs.tar.gz, src: out/omicron-gateway-softnpu.tar.gz
6942023-12-13T05:12:54.353ZDec 13 05:12:55.020 INFO Installing service, dst: /opt/oxide/propolis-server.tar.gz, src: out/propolis-server.tar.gz
6952023-12-13T05:12:54.353ZDec 13 05:12:55.020 INFO Installing service, dst: /opt/oxide/ntp.tar.gz, src: out/ntp.tar.gz
6962023-12-13T05:12:54.354ZDec 13 05:12:55.020 INFO Installing service, dst: /opt/oxide/crucible.tar.gz, src: out/crucible.tar.gz
6972023-12-13T05:12:54.355ZDec 13 05:12:55.020 INFO Installing service, dst: /opt/oxide/nexus.tar.gz, src: out/omicron-nexus.tar.gz
6982023-12-13T05:12:54.355ZDec 13 05:12:55.020 INFO Installing service, dst: /opt/oxide/overlay.tar.gz, src: out/overlay.tar.gz
6992023-12-13T05:12:54.355ZDec 13 05:12:55.020 INFO Installing service, dst: /opt/oxide/crucible_pantry.tar.gz, src: out/crucible-pantry.tar.gz
7002023-12-13T05:12:54.355ZDec 13 05:12:55.020 INFO Installing service, dst: /opt/oxide/oximeter.tar.gz, src: out/oximeter-collector.tar.gz
7012023-12-13T05:12:54.355ZDec 13 05:12:55.020 INFO Installing service, dst: /opt/oxide/external_dns.tar.gz, src: out/external-dns.tar.gz
7022023-12-13T05:12:54.355ZDec 13 05:12:55.020 INFO Installing service, dst: /opt/oxide/cockroachdb.tar.gz, src: out/cockroachdb.tar.gz
7032023-12-13T05:12:54.557ZDec 13 05:12:55.232 INFO Unpacking service tarball, service_path: /opt/oxide/mg-ddm, tar_path: /opt/oxide/mg-ddm.tar
7042023-12-13T05:12:54.590ZDec 13 05:12:55.266 INFO Unpacking service tarball, service_path: /opt/oxide/sled-agent, tar_path: /opt/oxide/sled-agent.tar
7052023-12-13T05:12:54.652ZDec 13 05:12:55.324 INFO Installing bootstrap service from /opt/oxide/sled-agent/pkg/manifest.xml
7062023-12-13T05:12:54.779Z
7072023-12-13T05:12:54.847Zreal 0.447458579
7082023-12-13T05:12:54.847Zuser 0.100179777
7092023-12-13T05:12:54.847Zsys 1.762529179
7102023-12-13T05:12:54.847Ztrap 0.000007050
7112023-12-13T05:12:54.847Ztflt 0.000051498
7122023-12-13T05:12:54.847Zdflt 0.000946868
7132023-12-13T05:12:54.847Zkflt 0.000001929
7142023-12-13T05:12:54.847Zlock 40.067924156
7152023-12-13T05:12:54.847Zslp 0.659577898
7162023-12-13T05:12:54.848Zlat 0.003119203
7172023-12-13T05:12:54.848Zstop 0.212059590
7182023-12-13T05:12:54.848Z+ retry=0
7192023-12-13T05:12:54.848Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7202023-12-13T05:12:54.848Z+ [[ 0 -gt 30 ]]
7212023-12-13T05:12:54.848Z+ sleep 1
7222023-12-13T05:12:55.793Z+ retry=1
7232023-12-13T05:12:55.795Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7242023-12-13T05:12:55.820Z+ [[ 1 -gt 30 ]]
7252023-12-13T05:12:55.820Z+ sleep 1
7262023-12-13T05:12:56.807Z+ retry=2
7272023-12-13T05:12:56.807Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7282023-12-13T05:12:56.832Z+ [[ 2 -gt 30 ]]
7292023-12-13T05:12:56.832Z+ sleep 1
7302023-12-13T05:12:57.829Z+ retry=3
7312023-12-13T05:12:57.829Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7322023-12-13T05:12:57.858Z+ [[ 3 -gt 30 ]]
7332023-12-13T05:12:57.858Z+ sleep 1
7342023-12-13T05:12:58.837Z+ retry=4
7352023-12-13T05:12:58.837Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7362023-12-13T05:13:24.228Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone dump-state
7372023-12-13T05:13:24.228Zlocal v6:
7382023-12-13T05:13:24.228Zfe80::c07f:f5ff:fe43:8887
7392023-12-13T05:13:24.228Zfd00:99::1
7402023-12-13T05:13:24.228Zlocal v4:
7412023-12-13T05:13:24.228Z10.151.2.174
7422023-12-13T05:13:24.228Zrouter v6:
7432023-12-13T05:13:24.228Zfdb0:18c0:4d0d:a02a::/64 -> fe80::88e0:97ff:fe00:cbc6 (1)
7442023-12-13T05:13:24.228Zfd00:1122:3344:1::/64 -> fe80::88e0:97ff:fe00:cbc6 (1)
7452023-12-13T05:13:24.228Zfd00:1122:3344:3::/64 -> fe80::88e0:97ff:fe00:cbc6 (1)
7462023-12-13T05:13:24.228Zfd00:1122:3344:101::/64 -> fe80::88e0:97ff:fe00:cbc6 (1)
7472023-12-13T05:13:24.229Zfd00:1122:3344:2::/64 -> fe80::88e0:97ff:fe00:cbc6 (1)
7482023-12-13T05:13:24.229Zrouter v4_idx:
7492023-12-13T05:13:24.230Z0.0.0.0/0 -> 0
7502023-12-13T05:13:24.230Zrouter v4_routes:
7512023-12-13T05:13:24.230Z0 -> 10.151.2.100 (2)
7522023-12-13T05:13:24.230Zresolver v4:
7532023-12-13T05:13:24.230Z10.151.2.174 -> 96:a8:0e:3c:02:72
7542023-12-13T05:13:24.231Z10.151.2.100 -> 18:c0:4d:0d:a0:2a
7552023-12-13T05:13:24.231Zresolver v6:
7562023-12-13T05:13:24.231Zfe80::88e0:97ff:fe00:cbc6 -> 8a:e0:97:00:cb:c6
7572023-12-13T05:13:24.231Zfe80::c07f:f5ff:fe43:8887 -> c2:7f:f5:43:88:87
7582023-12-13T05:13:24.231Znat v4:
7592023-12-13T05:13:24.231Z10.151.2.169 0/16383 -> fd00:1122:3344:101::1 100/a8:40:25:ff:a8:2d
7602023-12-13T05:13:24.231Znat_v6:
7612023-12-13T05:13:24.231Zport_mac:
7622023-12-13T05:13:24.262Z1000: a8:40:25:51:5e:6f
7632023-12-13T05:13:24.262Z1: a8:40:25:51:5e:70
7642023-12-13T05:13:24.262Zproxy arp:
7652023-12-13T05:13:24.262Z10.151.2.164/10.151.2.227: a8:e1:de:01:70:1d
7662023-12-13T05:13:24.262Z+ retry=0
7672023-12-13T05:13:24.262Z+++ zoneadm list -n
7682023-12-13T05:13:24.262Z+++ grep oxz_ntp
7692023-12-13T05:13:24.262Z++ pfexec svcs -z oxz_ntp_12d9fff4-405b-4a0c-a4f4-b3d2413b256e -Hostate oxide/ntp
7702023-12-13T05:13:24.262Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
7712023-12-13T05:13:24.262Z++ true
7722023-12-13T05:13:24.262Z+ [[ '' != online ]]
7732023-12-13T05:13:24.263Z+ [[ 0 -gt 60 ]]
7742023-12-13T05:13:24.263Z+ sleep 1
7752023-12-13T05:13:25.086Z+ retry=1
7762023-12-13T05:13:25.124Z+++ zoneadm list -n
7772023-12-13T05:13:25.124Z+++ grep oxz_ntp
7782023-12-13T05:13:25.124Z++ pfexec svcs -z oxz_ntp_12d9fff4-405b-4a0c-a4f4-b3d2413b256e -Hostate oxide/ntp
7792023-12-13T05:13:25.124Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
7802023-12-13T05:13:25.124Z++ true
7812023-12-13T05:13:25.124Z+ [[ '' != online ]]
7822023-12-13T05:13:25.124Z+ [[ 1 -gt 60 ]]
7832023-12-13T05:13:25.124Z+ sleep 1
7842023-12-13T05:13:26.116Z+ retry=2
7852023-12-13T05:13:26.148Z+++ zoneadm list -n
7862023-12-13T05:13:26.149Z+++ grep oxz_ntp
7872023-12-13T05:13:26.149Z++ pfexec svcs -z oxz_ntp_12d9fff4-405b-4a0c-a4f4-b3d2413b256e -Hostate oxide/ntp
7882023-12-13T05:13:26.149Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
7892023-12-13T05:13:26.149Z++ true
7902023-12-13T05:13:26.149Z+ [[ '' != online ]]
7912023-12-13T05:13:26.150Z+ [[ 2 -gt 60 ]]
7922023-12-13T05:13:26.150Z+ sleep 1
7932023-12-13T05:13:27.147Z+ retry=3
7942023-12-13T05:13:27.174Z+++ zoneadm list -n
7952023-12-13T05:13:27.174Z+++ grep oxz_ntp
7962023-12-13T05:13:27.175Z++ pfexec svcs -z oxz_ntp_12d9fff4-405b-4a0c-a4f4-b3d2413b256e -Hostate oxide/ntp
7972023-12-13T05:13:27.175Z+ [[ disabled != online ]]
7982023-12-13T05:13:27.175Z+ [[ 3 -gt 60 ]]
7992023-12-13T05:13:27.175Z+ sleep 1
8002023-12-13T05:13:28.178Z+ retry=4
8012023-12-13T05:13:28.216Z+++ zoneadm list -n
8022023-12-13T05:13:28.217Z+++ grep oxz_ntp
8032023-12-13T05:13:28.217Z++ pfexec svcs -z oxz_ntp_12d9fff4-405b-4a0c-a4f4-b3d2413b256e -Hostate oxide/ntp
8042023-12-13T05:13:28.217Z+ [[ disabled != online ]]
8052023-12-13T05:13:28.217Z+ [[ 4 -gt 60 ]]
8062023-12-13T05:13:28.218Z+ sleep 1
8072023-12-13T05:13:29.198Z+ retry=5
8082023-12-13T05:13:29.647Z+++ zoneadm list -n
8092023-12-13T05:13:29.647Z+++ grep oxz_ntp
8102023-12-13T05:13:29.649Z++ pfexec svcs -z oxz_ntp_12d9fff4-405b-4a0c-a4f4-b3d2413b256e -Hostate oxide/ntp
8112023-12-13T05:13:29.649Z+ [[ disabled != online ]]
8122023-12-13T05:13:29.649Z+ [[ 5 -gt 60 ]]
8132023-12-13T05:13:29.649Z+ sleep 1
8142023-12-13T05:13:30.229Z+ retry=6
8152023-12-13T05:13:30.724Z+++ zoneadm list -n
8162023-12-13T05:13:30.724Z+++ grep oxz_ntp
8172023-12-13T05:13:30.725Z++ pfexec svcs -z oxz_ntp_12d9fff4-405b-4a0c-a4f4-b3d2413b256e -Hostate oxide/ntp
8182023-12-13T05:13:30.725Z+ [[ disabled != online ]]
8192023-12-13T05:13:30.725Z+ [[ 6 -gt 60 ]]
8202023-12-13T05:13:30.725Z+ sleep 1
8212023-12-13T05:13:31.304Z+ retry=7
8222023-12-13T05:13:31.341Z+++ zoneadm list -n
8232023-12-13T05:13:31.341Z+++ grep oxz_ntp
8242023-12-13T05:13:31.341Z++ pfexec svcs -z oxz_ntp_12d9fff4-405b-4a0c-a4f4-b3d2413b256e -Hostate oxide/ntp
8252023-12-13T05:13:31.341Z+ [[ disabled != online ]]
8262023-12-13T05:13:31.342Z+ [[ 7 -gt 60 ]]
8272023-12-13T05:13:31.342Z+ sleep 1
8282023-12-13T05:13:32.262Z+ retry=8
8292023-12-13T05:13:32.389Z+++ zoneadm list -n
8302023-12-13T05:13:32.389Z+++ grep oxz_ntp
8312023-12-13T05:13:32.390Z++ pfexec svcs -z oxz_ntp_12d9fff4-405b-4a0c-a4f4-b3d2413b256e -Hostate oxide/ntp
8322023-12-13T05:13:32.391Z+ [[ disabled != online ]]
8332023-12-13T05:13:32.391Z+ [[ 8 -gt 60 ]]
8342023-12-13T05:13:32.391Z+ sleep 1
8352023-12-13T05:13:33.281Z+ retry=9
8362023-12-13T05:13:33.310Z+++ zoneadm list -n
8372023-12-13T05:13:33.311Z+++ grep oxz_ntp
8382023-12-13T05:13:33.311Z++ pfexec svcs -z oxz_ntp_12d9fff4-405b-4a0c-a4f4-b3d2413b256e -Hostate oxide/ntp
8392023-12-13T05:13:33.312Z+ [[ offline* != online ]]
8402023-12-13T05:13:33.312Z+ [[ 9 -gt 60 ]]
8412023-12-13T05:13:33.312Z+ sleep 1
 
8422023-12-13T05:13:33.463Zsock console login: Dec 13 05:13:33 sock ip: ip: joining multicasts failed (4) on vopte0 - will use link layer broadcasts for multicast
 
8432023-12-13T05:13:34.304Z+ retry=10
8442023-12-13T05:13:34.333Z+++ zoneadm list -n
8452023-12-13T05:13:34.334Z+++ grep oxz_ntp
8462023-12-13T05:13:34.334Z++ pfexec svcs -z oxz_ntp_12d9fff4-405b-4a0c-a4f4-b3d2413b256e -Hostate oxide/ntp
8472023-12-13T05:13:34.334Z+ [[ online != online ]]
8482023-12-13T05:13:34.334Z+ echo 'Waited for chrony: 10s'
8492023-12-13T05:13:34.334ZWaited for chrony: 10s
8502023-12-13T05:13:34.334Z+ export RUST_BACKTRACE=1
8512023-12-13T05:13:34.335Z+ RUST_BACKTRACE=1
8522023-12-13T05:13:34.335Z+ export E2E_TLS_CERT IPPOOL_START IPPOOL_END
8532023-12-13T05:13:34.335Z++ ./tests/bootstrap
8542023-12-13T05:13:34.360ZUsing external DNS server at 10.151.2.164:53
8552023-12-13T05:13:34.360Z2023-12-13 05:13:35.025480252 UTC: attempting to log into API
8562023-12-13T05:13:49.386Z2023-12-13 05:13:49.287756074 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8572023-12-13T05:13:50.387Z2023-12-13 05:13:50.288833136 UTC: attempting to log into API
8582023-12-13T05:14:05.421Z2023-12-13 05:14:05.322416408 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8592023-12-13T05:14:06.422Z2023-12-13 05:14:06.323921156 UTC: attempting to log into API
8602023-12-13T05:14:21.475Z2023-12-13 05:14:21.376598462 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8612023-12-13T05:14:22.485Z2023-12-13 05:14:22.378014849 UTC: attempting to log into API
8622023-12-13T05:14:37.605Z2023-12-13 05:14:37.507327845 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8632023-12-13T05:14:38.610Z2023-12-13 05:14:38.508465938 UTC: attempting to log into API
 
8642023-12-13T05:14:50.470ZDec 13 05:14:50 sock ip: ip: joining multicasts failed (4) on vopte1 - will use link layer broadcasts for multicast
 
8652023-12-13T05:14:53.638Z2023-12-13 05:14:53.563809554 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8662023-12-13T05:14:54.640Z2023-12-13 05:14:54.567567278 UTC: attempting to log into API
 
8672023-12-13T05:14:57.693ZDec 13 05:14:57 sock ip: ip: joining multicasts failed (4) on vopte2 - will use link layer broadcasts for multicast
8682023-12-13T05:15:04.375ZDec 13 05:15:03 sock ip: ip: joining multicasts failed (4) on vopte4 - will use link layer broadcasts for multicast
 
8692023-12-13T05:15:09.673Z2023-12-13 05:15:09.641009286 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
 
8702023-12-13T05:15:10.460ZDec 13 05:15:10 sock ip: ip: joining multicasts failed (4) on vopte5 - will use link layer broadcasts for multicast
 
8712023-12-13T05:15:10.675Z2023-12-13 05:15:10.645762524 UTC: attempting to log into API
 
8722023-12-13T05:15:18.109ZDec 13 05:15:17 sock ip: ip: joining multicasts failed (4) on vopte3 - will use link layer broadcasts for multicast
 
8732023-12-13T05:15:25.708Z2023-12-13 05:15:25.695967367 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8742023-12-13T05:15:26.710Z2023-12-13 05:15:26.697961752 UTC: attempting to log into API
8752023-12-13T05:15:32.523Z2023-12-13 05:15:32.492533339 UTC: login succeeded
8762023-12-13T05:15:32.549Zwaiting for nexus to come up...
8772023-12-13T05:15:32.604Zcreating IP pool... 10.151.2.175 - 10.151.2.227
8782023-12-13T05:15:32.648Zensuring datasets are ready...
8792023-12-13T05:15:34.965Zlet's roll.
8802023-12-13T05:15:34.992Z+ eval 'OXIDE_HOST=https://recovery.sys.oxide.test
8812023-12-13T05:15:34.992ZOXIDE_RESOLVE=recovery.sys.oxide.test:443:10.151.2.166
8822023-12-13T05:15:34.992ZOXIDE_TOKEN=oxide-token-c759151e6f172dd7e8e0ba637ed2ea36a18060f2'
8832023-12-13T05:15:34.992Z++ OXIDE_HOST=https://recovery.sys.oxide.test
8842023-12-13T05:15:34.992Z++ OXIDE_RESOLVE=recovery.sys.oxide.test:443:10.151.2.166
8852023-12-13T05:15:34.992Z++ OXIDE_TOKEN=oxide-token-c759151e6f172dd7e8e0ba637ed2ea36a18060f2
8862023-12-13T05:15:34.993Z+ export OXIDE_HOST OXIDE_TOKEN
8872023-12-13T05:15:34.993Z+ retry=0
8882023-12-13T05:15:34.993Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
8892023-12-13T05:15:34.993Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
8902023-12-13T05:15:34.993Z+ [[ 0 -gt 60 ]]
8912023-12-13T05:15:34.993Z+ sleep 1
8922023-12-13T05:15:35.993Z+ retry=1
8932023-12-13T05:15:35.993Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
8942023-12-13T05:15:36.020Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
8952023-12-13T05:15:36.020Z+ [[ 1 -gt 60 ]]
8962023-12-13T05:15:36.021Z+ sleep 1
8972023-12-13T05:15:37.010Z+ retry=2
8982023-12-13T05:15:37.010Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
8992023-12-13T05:15:37.036Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
9002023-12-13T05:15:37.036Z+ [[ 2 -gt 60 ]]
9012023-12-13T05:15:37.036Z+ sleep 1
9022023-12-13T05:15:38.025Z+ retry=3
9032023-12-13T05:15:38.025Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
9042023-12-13T05:15:38.051Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
9052023-12-13T05:15:38.051Z+ [[ 3 -gt 60 ]]
9062023-12-13T05:15:38.051Z+ sleep 1
9072023-12-13T05:15:39.041Z+ retry=4
9082023-12-13T05:15:39.042Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
9092023-12-13T05:15:39.067Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
9102023-12-13T05:15:39.068Z+ [[ 4 -gt 60 ]]
9112023-12-13T05:15:39.068Z+ sleep 1
9122023-12-13T05:15:40.059Z+ retry=5
9132023-12-13T05:15:40.059Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
9142023-12-13T05:15:40.090Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
9152023-12-13T05:15:40.090Z+ [[ 5 -gt 60 ]]
9162023-12-13T05:15:40.090Z+ sleep 1
9172023-12-13T05:15:41.075Z+ retry=6
9182023-12-13T05:15:41.075Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
9192023-12-13T05:15:41.100Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
9202023-12-13T05:15:41.100Z+ [[ 6 -gt 60 ]]
9212023-12-13T05:15:41.101Z+ sleep 1
9222023-12-13T05:15:42.089Z+ retry=7
9232023-12-13T05:15:42.089Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
9242023-12-13T05:15:42.114Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
9252023-12-13T05:15:42.114Z+ [[ 7 -gt 60 ]]
9262023-12-13T05:15:42.114Z+ sleep 1
9272023-12-13T05:15:43.105Z+ retry=8
9282023-12-13T05:15:43.105Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
9292023-12-13T05:15:43.130Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
9302023-12-13T05:15:43.131Z+ [[ 8 -gt 60 ]]
9312023-12-13T05:15:43.131Z+ sleep 1
9322023-12-13T05:15:44.121Z+ retry=9
9332023-12-13T05:15:44.121Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
9342023-12-13T05:15:44.146Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
9352023-12-13T05:15:44.146Z+ [[ 9 -gt 60 ]]
9362023-12-13T05:15:44.147Z+ sleep 1
9372023-12-13T05:15:45.137Z+ retry=10
9382023-12-13T05:15:45.137Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
9392023-12-13T05:15:45.162Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
9402023-12-13T05:15:45.162Z+ [[ 10 -gt 60 ]]
9412023-12-13T05:15:45.162Z+ sleep 1
9422023-12-13T05:15:46.152Z+ retry=11
9432023-12-13T05:15:46.152Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
9442023-12-13T05:15:46.177Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
9452023-12-13T05:15:46.177Z+ [[ 11 -gt 60 ]]
9462023-12-13T05:15:46.177Z+ sleep 1
9472023-12-13T05:15:47.169Z+ retry=12
9482023-12-13T05:15:47.169Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
9492023-12-13T05:15:47.195Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
9502023-12-13T05:15:47.195Z+ [[ 12 -gt 60 ]]
9512023-12-13T05:15:47.195Z+ sleep 1
9522023-12-13T05:15:48.185Z+ retry=13
9532023-12-13T05:15:48.186Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
9542023-12-13T05:15:48.211Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
9552023-12-13T05:15:48.212Z+ [[ 13 -gt 60 ]]
9562023-12-13T05:15:48.212Z+ sleep 1
9572023-12-13T05:15:49.216Z+ retry=14
9582023-12-13T05:15:49.217Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
9592023-12-13T05:15:49.243Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
9602023-12-13T05:15:49.243Z+ [[ 14 -gt 60 ]]
9612023-12-13T05:15:49.243Z+ sleep 1
9622023-12-13T05:15:50.245Z+ retry=15
9632023-12-13T05:15:50.245Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
9642023-12-13T05:15:50.273Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
9652023-12-13T05:15:50.273Z+ [[ 15 -gt 60 ]]
9662023-12-13T05:15:50.273Z+ sleep 1
9672023-12-13T05:15:51.233Z+ retry=16
9682023-12-13T05:15:51.233Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
9692023-12-13T05:15:51.278Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
9702023-12-13T05:15:51.279Z+ [[ 16 -gt 60 ]]
9712023-12-13T05:15:51.279Z+ sleep 1
9722023-12-13T05:15:52.249Z+ retry=17
9732023-12-13T05:15:52.249Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
9742023-12-13T05:15:52.275Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
9752023-12-13T05:15:52.275Z+ [[ 17 -gt 60 ]]
9762023-12-13T05:15:52.275Z+ sleep 1
9772023-12-13T05:15:53.264Z+ retry=18
9782023-12-13T05:15:53.289Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
9792023-12-13T05:15:53.289Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
9802023-12-13T05:15:53.289Z+ [[ 18 -gt 60 ]]
9812023-12-13T05:15:53.289Z+ sleep 1
9822023-12-13T05:15:54.281Z+ retry=19
9832023-12-13T05:15:54.281Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
9842023-12-13T05:15:54.308Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
9852023-12-13T05:15:54.308Z+ [[ 19 -gt 60 ]]
9862023-12-13T05:15:54.308Z+ sleep 1
9872023-12-13T05:15:55.297Z+ retry=20
9882023-12-13T05:15:55.297Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
9892023-12-13T05:15:55.323Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
9902023-12-13T05:15:55.323Z+ [[ 20 -gt 60 ]]
9912023-12-13T05:15:55.323Z+ sleep 1
9922023-12-13T05:15:56.326Z+ retry=21
9932023-12-13T05:15:56.326Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
9942023-12-13T05:15:56.351Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
9952023-12-13T05:15:56.351Z+ [[ 21 -gt 60 ]]
9962023-12-13T05:15:56.352Z+ sleep 1
9972023-12-13T05:15:57.329Z+ retry=22
9982023-12-13T05:15:57.329Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
9992023-12-13T05:15:57.407Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
10002023-12-13T05:15:57.407Z+ [[ 22 -gt 60 ]]
10012023-12-13T05:15:57.407Z+ sleep 1
10022023-12-13T05:15:58.346Z+ retry=23
10032023-12-13T05:15:58.346Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
10042023-12-13T05:15:58.371Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
10052023-12-13T05:15:58.371Z+ [[ 23 -gt 60 ]]
10062023-12-13T05:15:58.371Z+ sleep 1
10072023-12-13T05:15:59.363Z+ retry=24
10082023-12-13T05:15:59.363Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
10092023-12-13T05:15:59.388Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
10102023-12-13T05:15:59.388Z+ [[ 24 -gt 60 ]]
10112023-12-13T05:15:59.388Z+ sleep 1
10122023-12-13T05:16:00.381Z+ retry=25
10132023-12-13T05:16:00.381Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
10142023-12-13T05:16:00.410Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
10152023-12-13T05:16:00.410Z+ [[ 25 -gt 60 ]]
10162023-12-13T05:16:00.410Z+ sleep 1
10172023-12-13T05:16:01.397Z+ retry=26
10182023-12-13T05:16:01.397Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
10192023-12-13T05:16:01.422Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
10202023-12-13T05:16:01.422Z+ [[ 26 -gt 60 ]]
10212023-12-13T05:16:01.422Z+ sleep 1
10222023-12-13T05:16:02.414Z+ retry=27
10232023-12-13T05:16:02.414Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
10242023-12-13T05:16:02.439Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
10252023-12-13T05:16:02.439Z+ [[ 27 -gt 60 ]]
10262023-12-13T05:16:02.439Z+ sleep 1
10272023-12-13T05:16:03.432Z+ retry=28
10282023-12-13T05:16:03.432Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
10292023-12-13T05:16:03.457Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
10302023-12-13T05:16:03.457Z+ [[ 28 -gt 60 ]]
10312023-12-13T05:16:03.458Z+ sleep 1
10322023-12-13T05:16:04.448Z+ retry=29
10332023-12-13T05:16:04.449Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
10342023-12-13T05:16:04.473Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
10352023-12-13T05:16:04.473Z+ [[ 29 -gt 60 ]]
10362023-12-13T05:16:04.474Z+ sleep 1
10372023-12-13T05:16:05.472Z+ retry=30
10382023-12-13T05:16:05.472Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
10392023-12-13T05:16:05.500Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
10402023-12-13T05:16:05.500Z+ [[ 30 -gt 60 ]]
10412023-12-13T05:16:05.500Z+ sleep 1
10422023-12-13T05:16:06.481Z+ retry=31
10432023-12-13T05:16:06.481Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
10442023-12-13T05:16:06.506Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
10452023-12-13T05:16:06.506Z+ [[ 31 -gt 60 ]]
10462023-12-13T05:16:06.506Z+ sleep 1
10472023-12-13T05:16:07.497Z+ retry=32
10482023-12-13T05:16:07.521Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
10492023-12-13T05:16:07.522Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
10502023-12-13T05:16:07.522Z+ [[ 32 -gt 60 ]]
10512023-12-13T05:16:07.522Z+ sleep 1
10522023-12-13T05:16:08.514Z+ retry=33
10532023-12-13T05:16:08.514Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
10542023-12-13T05:16:08.539Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
10552023-12-13T05:16:08.539Z+ [[ 33 -gt 60 ]]
10562023-12-13T05:16:08.539Z+ sleep 1
10572023-12-13T05:16:09.530Z+ retry=34
10582023-12-13T05:16:09.530Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
10592023-12-13T05:16:09.557Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
10602023-12-13T05:16:09.557Z+ [[ 34 -gt 60 ]]
10612023-12-13T05:16:09.557Z+ sleep 1
10622023-12-13T05:16:10.549Z+ retry=35
10632023-12-13T05:16:10.550Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
10642023-12-13T05:16:10.577Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
10652023-12-13T05:16:10.577Z+ [[ 35 -gt 60 ]]
10662023-12-13T05:16:10.577Z+ sleep 1
10672023-12-13T05:16:11.564Z+ retry=36
10682023-12-13T05:16:11.564Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
10692023-12-13T05:16:11.588Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
10702023-12-13T05:16:11.589Z+ [[ 36 -gt 60 ]]
10712023-12-13T05:16:11.589Z+ sleep 1
10722023-12-13T05:16:12.580Z+ retry=37
10732023-12-13T05:16:12.581Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
10742023-12-13T05:16:12.673Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
10752023-12-13T05:16:12.673Z+ [[ 37 -gt 60 ]]
10762023-12-13T05:16:12.673Z+ sleep 1
10772023-12-13T05:16:13.598Z+ retry=38
10782023-12-13T05:16:13.626Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
10792023-12-13T05:16:13.626Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
10802023-12-13T05:16:13.626Z+ [[ 38 -gt 60 ]]
10812023-12-13T05:16:13.626Z+ sleep 1
10822023-12-13T05:16:14.614Z+ retry=39
10832023-12-13T05:16:14.614Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
10842023-12-13T05:16:14.638Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
10852023-12-13T05:16:14.639Z+ [[ 39 -gt 60 ]]
10862023-12-13T05:16:14.639Z+ sleep 1
10872023-12-13T05:16:15.630Z+ retry=40
10882023-12-13T05:16:15.630Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
10892023-12-13T05:16:15.655Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
10902023-12-13T05:16:15.656Z+ [[ 40 -gt 60 ]]
10912023-12-13T05:16:15.656Z+ sleep 1
10922023-12-13T05:16:16.648Z+ retry=41
10932023-12-13T05:16:16.648Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
10942023-12-13T05:16:16.674Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
10952023-12-13T05:16:16.674Z+ [[ 41 -gt 60 ]]
10962023-12-13T05:16:16.676Z+ sleep 1
10972023-12-13T05:16:17.666Z+ retry=42
10982023-12-13T05:16:17.668Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
10992023-12-13T05:16:17.692Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
11002023-12-13T05:16:17.692Z+ [[ 42 -gt 60 ]]
11012023-12-13T05:16:17.692Z+ sleep 1
11022023-12-13T05:16:18.683Z+ retry=43
11032023-12-13T05:16:18.683Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
11042023-12-13T05:16:18.707Zcurl: (7) Failed to connect to recovery.sys.oxide.test port 443 after 0 ms: Couldn't connect to server
11052023-12-13T05:16:18.707Z+ [[ 43 -gt 60 ]]
11062023-12-13T05:16:18.707Z+ sleep 1
11072023-12-13T05:16:19.700Z+ retry=44
11082023-12-13T05:16:19.700Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
11092023-12-13T05:16:19.785Z+ /usr/oxide/oxide --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem project create --name images --description 'some images'
11102023-12-13T05:16:20.268Z{"status":"ok"}success
11112023-12-13T05:16:20.268ZProject {
11122023-12-13T05:16:20.268Z description: "some images",
11132023-12-13T05:16:20.268Z id: ff72cde7-b9a7-435a-b620-dea2af28a6bf,
11142023-12-13T05:16:20.293Z name: Name(
11152023-12-13T05:16:20.293Z "images",
11162023-12-13T05:16:20.293Z ),
11172023-12-13T05:16:20.293Z time_created: 2023-12-13T05:16:19.885023Z,
11182023-12-13T05:16:20.293Z time_modified: 2023-12-13T05:16:19.885023Z,
11192023-12-13T05:16:20.293Z}
11202023-12-13T05:16:20.293Z+ /usr/oxide/oxide --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem disk import --path debian-11-genericcloud-amd64.raw --disk debian11-boot --project images --description 'debian 11 cloud image from distros' --snapshot debian11-snapshot --image debian11 --image-description 'debian 11 original base image' --image-os debian --image-version 11
11212023-12-13T05:21:45.883Zdone!
11222023-12-13T05:21:45.988Z+ /usr/oxide/oxide --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem image promote --project images --image debian11
11232023-12-13T05:21:46.096Zsuccess
11242023-12-13T05:21:46.096ZImage {
11252023-12-13T05:21:46.120Z block_size: ByteCount(
11262023-12-13T05:21:46.121Z 512,
11272023-12-13T05:21:46.121Z ),
11282023-12-13T05:21:46.121Z description: "debian 11 original base image",
11292023-12-13T05:21:46.121Z digest: None,
11302023-12-13T05:21:46.121Z id: 59eca61b-2d5b-4c36-91aa-a79a385ce4b5,
11312023-12-13T05:21:46.121Z name: Name(
11322023-12-13T05:21:46.121Z "debian11",
11332023-12-13T05:21:46.121Z ),
11342023-12-13T05:21:46.121Z os: "debian",
11352023-12-13T05:21:46.121Z project_id: None,
11362023-12-13T05:21:46.121Z size: ByteCount(
11372023-12-13T05:21:46.121Z 2147483648,
11382023-12-13T05:21:46.121Z ),
11392023-12-13T05:21:46.121Z time_created: 2023-12-13T05:21:45.809041Z,
11402023-12-13T05:21:46.121Z time_modified: 2023-12-13T05:21:46.067328Z,
11412023-12-13T05:21:46.121Z url: None,
11422023-12-13T05:21:46.121Z version: "11",
11432023-12-13T05:21:46.121Z}
11442023-12-13T05:21:46.121Z+ rm ./tests/bootstrap
11452023-12-13T05:21:46.156Z+ for test_bin in tests/*
11462023-12-13T05:21:46.157Z+ ./tests/bootstrap-cc02af252c51050b
11472023-12-13T05:21:46.183Z
11482023-12-13T05:21:46.183Zrunning 0 tests
11492023-12-13T05:21:46.183Z
11502023-12-13T05:21:46.183Ztest result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
11512023-12-13T05:21:46.183Z
11522023-12-13T05:21:46.184Z+ for test_bin in tests/*
11532023-12-13T05:21:46.184Z+ ./tests/end_to_end_tests-c0f56f9124e0823c
11542023-12-13T05:21:46.215Z
11552023-12-13T05:21:46.215Zrunning 2 tests
11562023-12-13T05:21:46.411Ztest no_spoof::no_spoof_here ... ok
11572023-12-13T05:22:22.067Ztest instance_launch::instance_launch ... ok
11582023-12-13T05:22:22.067Z
11592023-12-13T05:22:22.067Ztest result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 35.88s
11602023-12-13T05:22:22.069Z
11612023-12-13T05:22:22.093Z+ _exit_trap
11622023-12-13T05:22:22.093Z+ local status=0
11632023-12-13T05:22:22.094Z+ [[ 0 -eq 0 ]]
11642023-12-13T05:22:22.094Z+ exit 0
11652023-12-13T05:22:22.094Zprocess exited: duration 627144 ms, exit code 0
 
11662023-12-13T05:22:22.725Zfound 93 output files
11672023-12-13T05:22:22.725Zuploading: /var/svc/log/oxide-sled-agent:default.log (698307 bytes)
11682023-12-13T05:22:24.898Zuploaded: /var/svc/log/oxide-sled-agent:default.log
11692023-12-13T05:22:24.898Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_60a7bc03-7bcd-431b-b092-e5c0d502d872/root/var/svc/log/oxide-cockroachdb:default.log (0 bytes)
11702023-12-13T05:22:25.968Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_60a7bc03-7bcd-431b-b092-e5c0d502d872/root/var/svc/log/oxide-cockroachdb:default.log
11712023-12-13T05:22:25.993Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_60a7bc03-7bcd-431b-b092-e5c0d502d872/root/var/svc/log/oxide-cockroachdb:default.log.0 (5413 bytes)
11722023-12-13T05:22:27.065Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_60a7bc03-7bcd-431b-b092-e5c0d502d872/root/var/svc/log/oxide-cockroachdb:default.log.0
11732023-12-13T05:22:27.065Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_160c3ac4-6b7d-494d-a36e-209ce3657189/root/var/svc/log/oxide-crucible-agent:default.log (21390 bytes)
11742023-12-13T05:22:28.638Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_160c3ac4-6b7d-494d-a36e-209ce3657189/root/var/svc/log/oxide-crucible-agent:default.log
11752023-12-13T05:22:28.639Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_160c3ac4-6b7d-494d-a36e-209ce3657189/root/var/svc/log/oxide-crucible-agent:default.log.0 (3436 bytes)
11762023-12-13T05:22:29.737Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_160c3ac4-6b7d-494d-a36e-209ce3657189/root/var/svc/log/oxide-crucible-agent:default.log.0
11772023-12-13T05:22:29.737Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_160c3ac4-6b7d-494d-a36e-209ce3657189/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
11782023-12-13T05:22:30.815Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_160c3ac4-6b7d-494d-a36e-209ce3657189/root/var/svc/log/oxide-crucible-downstairs:default.log
11792023-12-13T05:22:30.815Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_160c3ac4-6b7d-494d-a36e-209ce3657189/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
11802023-12-13T05:22:32.327Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_160c3ac4-6b7d-494d-a36e-209ce3657189/root/var/svc/log/oxide-crucible-downstairs:default.log.0
11812023-12-13T05:22:32.327Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_160c3ac4-6b7d-494d-a36e-209ce3657189/root/var/svc/log/oxide-crucible-downstairs:downstairs-176c0386-48db-48dc-b7e3-e0091c2eaa8c.log (5515 bytes)
11822023-12-13T05:22:33.446Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_160c3ac4-6b7d-494d-a36e-209ce3657189/root/var/svc/log/oxide-crucible-downstairs:downstairs-176c0386-48db-48dc-b7e3-e0091c2eaa8c.log
11832023-12-13T05:22:33.446Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_160c3ac4-6b7d-494d-a36e-209ce3657189/root/var/svc/log/oxide-crucible-downstairs:downstairs-25430967-e481-4332-9520-984fdffbd590.log (3038 bytes)
11842023-12-13T05:22:34.547Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_160c3ac4-6b7d-494d-a36e-209ce3657189/root/var/svc/log/oxide-crucible-downstairs:downstairs-25430967-e481-4332-9520-984fdffbd590.log
11852023-12-13T05:22:34.547Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_10ae3a76-668f-4e88-b9f8-2597600c8207/root/var/svc/log/oxide-cockroachdb:default.log (0 bytes)
11862023-12-13T05:22:35.625Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_10ae3a76-668f-4e88-b9f8-2597600c8207/root/var/svc/log/oxide-cockroachdb:default.log
11872023-12-13T05:22:35.626Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_10ae3a76-668f-4e88-b9f8-2597600c8207/root/var/svc/log/oxide-cockroachdb:default.log.0 (5386 bytes)
11882023-12-13T05:22:36.718Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_10ae3a76-668f-4e88-b9f8-2597600c8207/root/var/svc/log/oxide-cockroachdb:default.log.0
11892023-12-13T05:22:36.720Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_cockroachdb_b084062d-0cec-4924-9778-1d4993e7e774/root/var/svc/log/oxide-cockroachdb:default.log (0 bytes)
11902023-12-13T05:22:37.789Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_cockroachdb_b084062d-0cec-4924-9778-1d4993e7e774/root/var/svc/log/oxide-cockroachdb:default.log
11912023-12-13T05:22:37.796Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_cockroachdb_b084062d-0cec-4924-9778-1d4993e7e774/root/var/svc/log/oxide-cockroachdb:default.log.0 (5412 bytes)
11922023-12-13T05:22:38.884Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_cockroachdb_b084062d-0cec-4924-9778-1d4993e7e774/root/var/svc/log/oxide-cockroachdb:default.log.0
11932023-12-13T05:22:38.884Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_7157960a-b632-46cb-9291-e492c7092911/root/var/svc/log/oxide-crucible-agent:default.log (6358 bytes)
11942023-12-13T05:22:40.007Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_7157960a-b632-46cb-9291-e492c7092911/root/var/svc/log/oxide-crucible-agent:default.log
11952023-12-13T05:22:40.007Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_7157960a-b632-46cb-9291-e492c7092911/root/var/svc/log/oxide-crucible-agent:default.log.0 (3436 bytes)
11962023-12-13T05:22:41.108Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_7157960a-b632-46cb-9291-e492c7092911/root/var/svc/log/oxide-crucible-agent:default.log.0
11972023-12-13T05:22:41.108Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_7157960a-b632-46cb-9291-e492c7092911/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
11982023-12-13T05:22:42.179Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_7157960a-b632-46cb-9291-e492c7092911/root/var/svc/log/oxide-crucible-downstairs:default.log
11992023-12-13T05:22:42.180Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_7157960a-b632-46cb-9291-e492c7092911/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
12002023-12-13T05:22:43.272Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_7157960a-b632-46cb-9291-e492c7092911/root/var/svc/log/oxide-crucible-downstairs:default.log.0
12012023-12-13T05:22:43.272Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_7157960a-b632-46cb-9291-e492c7092911/root/var/svc/log/oxide-crucible-downstairs:downstairs-274dcaf4-70d4-438f-839e-d3b932665eed.log (2933 bytes)
12022023-12-13T05:22:44.774Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_7157960a-b632-46cb-9291-e492c7092911/root/var/svc/log/oxide-crucible-downstairs:downstairs-274dcaf4-70d4-438f-839e-d3b932665eed.log
12032023-12-13T05:22:44.774Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_e07445e9-2d22-48ee-9bc3-3ebfe83e423d/root/var/svc/log/oxide-crucible-agent:default.log (16607 bytes)
12042023-12-13T05:22:46.381Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_e07445e9-2d22-48ee-9bc3-3ebfe83e423d/root/var/svc/log/oxide-crucible-agent:default.log
12052023-12-13T05:22:46.381Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_e07445e9-2d22-48ee-9bc3-3ebfe83e423d/root/var/svc/log/oxide-crucible-agent:default.log.0 (3436 bytes)
12062023-12-13T05:22:47.483Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_e07445e9-2d22-48ee-9bc3-3ebfe83e423d/root/var/svc/log/oxide-crucible-agent:default.log.0
12072023-12-13T05:22:47.484Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_e07445e9-2d22-48ee-9bc3-3ebfe83e423d/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
12082023-12-13T05:22:48.556Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_e07445e9-2d22-48ee-9bc3-3ebfe83e423d/root/var/svc/log/oxide-crucible-downstairs:default.log
12092023-12-13T05:22:48.580Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_e07445e9-2d22-48ee-9bc3-3ebfe83e423d/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
12102023-12-13T05:22:49.707Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_e07445e9-2d22-48ee-9bc3-3ebfe83e423d/root/var/svc/log/oxide-crucible-downstairs:default.log.0
12112023-12-13T05:22:49.745Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_e07445e9-2d22-48ee-9bc3-3ebfe83e423d/root/var/svc/log/oxide-crucible-downstairs:downstairs-76898057-416e-4efe-9d0a-c1bc6d4bb799.log (5515 bytes)
12122023-12-13T05:22:50.847Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_e07445e9-2d22-48ee-9bc3-3ebfe83e423d/root/var/svc/log/oxide-crucible-downstairs:downstairs-76898057-416e-4efe-9d0a-c1bc6d4bb799.log
12132023-12-13T05:22:50.848Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_e07445e9-2d22-48ee-9bc3-3ebfe83e423d/root/var/svc/log/oxide-crucible-downstairs:downstairs-f0aa0671-4fa1-4375-aa74-1402ccc7b285.log (2932 bytes)
12142023-12-13T05:22:51.953Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_e07445e9-2d22-48ee-9bc3-3ebfe83e423d/root/var/svc/log/oxide-crucible-downstairs:downstairs-f0aa0671-4fa1-4375-aa74-1402ccc7b285.log
12152023-12-13T05:22:51.954Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_pantry_2725327c-6f52-48ab-b165-6a8f9f4945bd/root/var/svc/log/oxide-crucible-pantry:default.log (1222737 bytes)
12162023-12-13T05:22:54.754Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_pantry_2725327c-6f52-48ab-b165-6a8f9f4945bd/root/var/svc/log/oxide-crucible-pantry:default.log
12172023-12-13T05:22:54.754Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_pantry_2725327c-6f52-48ab-b165-6a8f9f4945bd/root/var/svc/log/oxide-crucible-pantry:default.log.0 (2333 bytes)
12182023-12-13T05:22:58.463Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_pantry_2725327c-6f52-48ab-b165-6a8f9f4945bd/root/var/svc/log/oxide-crucible-pantry:default.log.0
12192023-12-13T05:22:58.463Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_nexus_9df1ae5d-663b-47e8-a5d9-32b405dbe239/root/var/svc/log/oxide-nexus:default.log (33960780 bytes)
12202023-12-13T05:23:10.533Zupload warning: file "/pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_nexus_9df1ae5d-663b-47e8-a5d9-32b405dbe239/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 33960780 -> 33968866
12212023-12-13T05:23:11.615Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_nexus_9df1ae5d-663b-47e8-a5d9-32b405dbe239/root/var/svc/log/oxide-nexus:default.log
12222023-12-13T05:23:11.615Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_nexus_9df1ae5d-663b-47e8-a5d9-32b405dbe239/root/var/svc/log/oxide-nexus:default.log.0 (120 bytes)
12232023-12-13T05:23:12.711Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_nexus_9df1ae5d-663b-47e8-a5d9-32b405dbe239/root/var/svc/log/oxide-nexus:default.log.0
12242023-12-13T05:23:12.711Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_nexus_dc8b0170-d0cb-407a-a361-7dd0dffdc2ee/root/var/svc/log/oxide-nexus:default.log (805445 bytes)
12252023-12-13T05:23:13.379Zupload warning: file "/pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_nexus_dc8b0170-d0cb-407a-a361-7dd0dffdc2ee/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 805445 -> 813529
12262023-12-13T05:23:14.468Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_nexus_dc8b0170-d0cb-407a-a361-7dd0dffdc2ee/root/var/svc/log/oxide-nexus:default.log
12272023-12-13T05:23:14.468Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_nexus_dc8b0170-d0cb-407a-a361-7dd0dffdc2ee/root/var/svc/log/oxide-nexus:default.log.0 (75 bytes)
12282023-12-13T05:23:15.562Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_nexus_dc8b0170-d0cb-407a-a361-7dd0dffdc2ee/root/var/svc/log/oxide-nexus:default.log.0
12292023-12-13T05:23:15.562Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_cockroachdb_6e958392-bbd9-4833-88f1-9698aeb350be/root/var/svc/log/oxide-cockroachdb:default.log (0 bytes)
12302023-12-13T05:23:16.632Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_cockroachdb_6e958392-bbd9-4833-88f1-9698aeb350be/root/var/svc/log/oxide-cockroachdb:default.log
12312023-12-13T05:23:16.632Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_cockroachdb_6e958392-bbd9-4833-88f1-9698aeb350be/root/var/svc/log/oxide-cockroachdb:default.log.0 (5412 bytes)
12322023-12-13T05:23:17.730Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_cockroachdb_6e958392-bbd9-4833-88f1-9698aeb350be/root/var/svc/log/oxide-cockroachdb:default.log.0
12332023-12-13T05:23:17.731Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_pantry_32e75302-03fb-4e17-8356-44b2cd27d525/root/var/svc/log/oxide-crucible-pantry:default.log (0 bytes)
12342023-12-13T05:23:18.800Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_pantry_32e75302-03fb-4e17-8356-44b2cd27d525/root/var/svc/log/oxide-crucible-pantry:default.log
12352023-12-13T05:23:18.801Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_pantry_32e75302-03fb-4e17-8356-44b2cd27d525/root/var/svc/log/oxide-crucible-pantry:default.log.0 (2338 bytes)
12362023-12-13T05:23:19.891Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_pantry_32e75302-03fb-4e17-8356-44b2cd27d525/root/var/svc/log/oxide-crucible-pantry:default.log.0
12372023-12-13T05:23:19.891Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_cockroachdb_424b4534-e78a-4876-8feb-74b9789416d8/root/var/svc/log/oxide-cockroachdb:default.log (0 bytes)
12382023-12-13T05:23:20.962Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_cockroachdb_424b4534-e78a-4876-8feb-74b9789416d8/root/var/svc/log/oxide-cockroachdb:default.log
12392023-12-13T05:23:20.963Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_cockroachdb_424b4534-e78a-4876-8feb-74b9789416d8/root/var/svc/log/oxide-cockroachdb:default.log.0 (5413 bytes)
12402023-12-13T05:23:22.058Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_cockroachdb_424b4534-e78a-4876-8feb-74b9789416d8/root/var/svc/log/oxide-cockroachdb:default.log.0
12412023-12-13T05:23:22.058Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_1bccd775-a5cd-49c3-bd84-00d0dab2f7ef/root/var/svc/log/oxide-crucible-agent:default.log (10719 bytes)
12422023-12-13T05:23:23.196Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_1bccd775-a5cd-49c3-bd84-00d0dab2f7ef/root/var/svc/log/oxide-crucible-agent:default.log
12432023-12-13T05:23:23.196Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_1bccd775-a5cd-49c3-bd84-00d0dab2f7ef/root/var/svc/log/oxide-crucible-agent:default.log.0 (3436 bytes)
12442023-12-13T05:23:24.716Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_1bccd775-a5cd-49c3-bd84-00d0dab2f7ef/root/var/svc/log/oxide-crucible-agent:default.log.0
12452023-12-13T05:23:24.716Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_1bccd775-a5cd-49c3-bd84-00d0dab2f7ef/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
12462023-12-13T05:23:26.194Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_1bccd775-a5cd-49c3-bd84-00d0dab2f7ef/root/var/svc/log/oxide-crucible-downstairs:default.log
12472023-12-13T05:23:26.194Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_1bccd775-a5cd-49c3-bd84-00d0dab2f7ef/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
12482023-12-13T05:23:27.290Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_1bccd775-a5cd-49c3-bd84-00d0dab2f7ef/root/var/svc/log/oxide-crucible-downstairs:default.log.0
12492023-12-13T05:23:27.316Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_1bccd775-a5cd-49c3-bd84-00d0dab2f7ef/root/var/svc/log/oxide-crucible-downstairs:downstairs-01b41390-a0d3-43d1-8278-334f1e02aceb.log (3038 bytes)
12502023-12-13T05:23:29.443Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_1bccd775-a5cd-49c3-bd84-00d0dab2f7ef/root/var/svc/log/oxide-crucible-downstairs:downstairs-01b41390-a0d3-43d1-8278-334f1e02aceb.log
12512023-12-13T05:23:29.443Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_cfe1623b-deb5-4bb2-ad1e-333e71bc87fb/root/var/svc/log/oxide-crucible-agent:default.log (6358 bytes)
12522023-12-13T05:23:30.547Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_cfe1623b-deb5-4bb2-ad1e-333e71bc87fb/root/var/svc/log/oxide-crucible-agent:default.log
12532023-12-13T05:23:30.573Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_cfe1623b-deb5-4bb2-ad1e-333e71bc87fb/root/var/svc/log/oxide-crucible-agent:default.log.0 (3434 bytes)
12542023-12-13T05:23:31.665Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_cfe1623b-deb5-4bb2-ad1e-333e71bc87fb/root/var/svc/log/oxide-crucible-agent:default.log.0
12552023-12-13T05:23:31.665Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_cfe1623b-deb5-4bb2-ad1e-333e71bc87fb/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
12562023-12-13T05:23:32.736Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_cfe1623b-deb5-4bb2-ad1e-333e71bc87fb/root/var/svc/log/oxide-crucible-downstairs:default.log
12572023-12-13T05:23:32.736Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_cfe1623b-deb5-4bb2-ad1e-333e71bc87fb/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
12582023-12-13T05:23:33.828Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_cfe1623b-deb5-4bb2-ad1e-333e71bc87fb/root/var/svc/log/oxide-crucible-downstairs:default.log.0
12592023-12-13T05:23:33.828Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_cfe1623b-deb5-4bb2-ad1e-333e71bc87fb/root/var/svc/log/oxide-crucible-downstairs:downstairs-ed833262-fbcc-424b-bd31-ce63911ef6c8.log (2929 bytes)
12602023-12-13T05:23:34.924Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_cfe1623b-deb5-4bb2-ad1e-333e71bc87fb/root/var/svc/log/oxide-crucible-downstairs:downstairs-ed833262-fbcc-424b-bd31-ce63911ef6c8.log
12612023-12-13T05:23:34.925Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_internal_dns_d5fa1d13-03d4-4377-bd02-a397af6d145f/root/var/svc/log/oxide-internal_dns:default.log (27543 bytes)
12622023-12-13T05:23:35.038Zupload warning: file "/pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_internal_dns_d5fa1d13-03d4-4377-bd02-a397af6d145f/root/var/svc/log/oxide-internal_dns:default.log" changed size mid upload: 27543 -> 31295
12632023-12-13T05:23:36.111Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_internal_dns_d5fa1d13-03d4-4377-bd02-a397af6d145f/root/var/svc/log/oxide-internal_dns:default.log
12642023-12-13T05:23:36.112Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_internal_dns_d5fa1d13-03d4-4377-bd02-a397af6d145f/root/var/svc/log/oxide-internal_dns:default.log.0 (100445 bytes)
12652023-12-13T05:23:37.318Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_internal_dns_d5fa1d13-03d4-4377-bd02-a397af6d145f/root/var/svc/log/oxide-internal_dns:default.log.0
12662023-12-13T05:23:37.318Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_clickhouse_f87ee328-6a8a-4b2d-9e67-97495dd78de0/root/var/svc/log/oxide-clickhouse:default.log (651008 bytes)
12672023-12-13T05:23:37.596Zupload warning: file "/pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_clickhouse_f87ee328-6a8a-4b2d-9e67-97495dd78de0/root/var/svc/log/oxide-clickhouse:default.log" changed size mid upload: 651008 -> 765018
12682023-12-13T05:23:38.673Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_clickhouse_f87ee328-6a8a-4b2d-9e67-97495dd78de0/root/var/svc/log/oxide-clickhouse:default.log
12692023-12-13T05:23:38.674Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_clickhouse_f87ee328-6a8a-4b2d-9e67-97495dd78de0/root/var/svc/log/oxide-clickhouse:default.log.0 (129730 bytes)
12702023-12-13T05:23:39.966Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_clickhouse_f87ee328-6a8a-4b2d-9e67-97495dd78de0/root/var/svc/log/oxide-clickhouse:default.log.0
12712023-12-13T05:23:39.966Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_30567c02-bab2-4f33-aae7-10e756deafff/root/var/svc/log/oxide-crucible-agent:default.log (14114 bytes)
12722023-12-13T05:23:41.083Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_30567c02-bab2-4f33-aae7-10e756deafff/root/var/svc/log/oxide-crucible-agent:default.log
12732023-12-13T05:23:41.084Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_30567c02-bab2-4f33-aae7-10e756deafff/root/var/svc/log/oxide-crucible-agent:default.log.0 (3436 bytes)
12742023-12-13T05:23:42.644Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_30567c02-bab2-4f33-aae7-10e756deafff/root/var/svc/log/oxide-crucible-agent:default.log.0
12752023-12-13T05:23:42.644Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_30567c02-bab2-4f33-aae7-10e756deafff/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
12762023-12-13T05:23:43.719Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_30567c02-bab2-4f33-aae7-10e756deafff/root/var/svc/log/oxide-crucible-downstairs:default.log
12772023-12-13T05:23:43.719Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_30567c02-bab2-4f33-aae7-10e756deafff/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
12782023-12-13T05:23:44.812Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_30567c02-bab2-4f33-aae7-10e756deafff/root/var/svc/log/oxide-crucible-downstairs:default.log.0
12792023-12-13T05:23:44.814Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_30567c02-bab2-4f33-aae7-10e756deafff/root/var/svc/log/oxide-crucible-downstairs:downstairs-c712842d-07e1-4f09-8808-120b6145df2e.log (6408 bytes)
12802023-12-13T05:23:45.928Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_30567c02-bab2-4f33-aae7-10e756deafff/root/var/svc/log/oxide-crucible-downstairs:downstairs-c712842d-07e1-4f09-8808-120b6145df2e.log
12812023-12-13T05:23:45.928Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_30567c02-bab2-4f33-aae7-10e756deafff/root/var/svc/log/oxide-crucible-downstairs:snapshot-c712842d-07e1-4f09-8808-120b6145df2e-8b254072-f4b4-466e-b1a6-8c3feab28c95.log (5156 bytes)
12822023-12-13T05:23:47.047Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_30567c02-bab2-4f33-aae7-10e756deafff/root/var/svc/log/oxide-crucible-downstairs:snapshot-c712842d-07e1-4f09-8808-120b6145df2e-8b254072-f4b4-466e-b1a6-8c3feab28c95.log
12832023-12-13T05:23:47.047Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_external_dns_bc7f3216-ef4b-4f2d-9f42-d8cde8af038b/root/var/svc/log/oxide-external_dns:default.log (28152 bytes)
12842023-12-13T05:23:47.224Zupload warning: file "/pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_external_dns_bc7f3216-ef4b-4f2d-9f42-d8cde8af038b/root/var/svc/log/oxide-external_dns:default.log" changed size mid upload: 28152 -> 31925
12852023-12-13T05:23:48.293Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_external_dns_bc7f3216-ef4b-4f2d-9f42-d8cde8af038b/root/var/svc/log/oxide-external_dns:default.log
12862023-12-13T05:23:48.293Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_external_dns_bc7f3216-ef4b-4f2d-9f42-d8cde8af038b/root/var/svc/log/oxide-external_dns:default.log.0 (2554 bytes)
12872023-12-13T05:23:49.391Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_external_dns_bc7f3216-ef4b-4f2d-9f42-d8cde8af038b/root/var/svc/log/oxide-external_dns:default.log.0
12882023-12-13T05:23:49.416Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_ntp_12d9fff4-405b-4a0c-a4f4-b3d2413b256e/root/var/svc/log/oxide-ntp:default.log (66 bytes)
12892023-12-13T05:23:50.487Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_ntp_12d9fff4-405b-4a0c-a4f4-b3d2413b256e/root/var/svc/log/oxide-ntp:default.log
12902023-12-13T05:23:50.487Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_ntp_12d9fff4-405b-4a0c-a4f4-b3d2413b256e/root/var/svc/log/oxide-ntp:default.log.0 (982 bytes)
12912023-12-13T05:23:51.590Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_ntp_12d9fff4-405b-4a0c-a4f4-b3d2413b256e/root/var/svc/log/oxide-ntp:default.log.0
12922023-12-13T05:23:51.591Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_31fa899e-2c91-47bf-8b75-f3e8976d6cf2/root/var/svc/log/oxide-crucible-agent:default.log (0 bytes)
12932023-12-13T05:23:52.663Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_31fa899e-2c91-47bf-8b75-f3e8976d6cf2/root/var/svc/log/oxide-crucible-agent:default.log
12942023-12-13T05:23:52.663Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_31fa899e-2c91-47bf-8b75-f3e8976d6cf2/root/var/svc/log/oxide-crucible-agent:default.log.0 (3434 bytes)
12952023-12-13T05:23:53.757Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_31fa899e-2c91-47bf-8b75-f3e8976d6cf2/root/var/svc/log/oxide-crucible-agent:default.log.0
12962023-12-13T05:23:53.757Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_31fa899e-2c91-47bf-8b75-f3e8976d6cf2/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
12972023-12-13T05:23:54.830Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_31fa899e-2c91-47bf-8b75-f3e8976d6cf2/root/var/svc/log/oxide-crucible-downstairs:default.log
12982023-12-13T05:23:54.830Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_31fa899e-2c91-47bf-8b75-f3e8976d6cf2/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
12992023-12-13T05:23:55.927Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_31fa899e-2c91-47bf-8b75-f3e8976d6cf2/root/var/svc/log/oxide-crucible-downstairs:default.log.0
13002023-12-13T05:23:55.928Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_d04b9c38-9134-4b45-9f7c-ded66ffdc85f/root/var/svc/log/oxide-crucible-agent:default.log (34184 bytes)
13012023-12-13T05:23:57.176Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_d04b9c38-9134-4b45-9f7c-ded66ffdc85f/root/var/svc/log/oxide-crucible-agent:default.log
13022023-12-13T05:23:57.176Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_d04b9c38-9134-4b45-9f7c-ded66ffdc85f/root/var/svc/log/oxide-crucible-agent:default.log.0 (3436 bytes)
13032023-12-13T05:23:58.289Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_d04b9c38-9134-4b45-9f7c-ded66ffdc85f/root/var/svc/log/oxide-crucible-agent:default.log.0
13042023-12-13T05:23:58.289Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_d04b9c38-9134-4b45-9f7c-ded66ffdc85f/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
13052023-12-13T05:23:59.366Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_d04b9c38-9134-4b45-9f7c-ded66ffdc85f/root/var/svc/log/oxide-crucible-downstairs:default.log
13062023-12-13T05:23:59.366Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_d04b9c38-9134-4b45-9f7c-ded66ffdc85f/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
13072023-12-13T05:24:00.496Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_d04b9c38-9134-4b45-9f7c-ded66ffdc85f/root/var/svc/log/oxide-crucible-downstairs:default.log.0
13082023-12-13T05:24:00.496Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_d04b9c38-9134-4b45-9f7c-ded66ffdc85f/root/var/svc/log/oxide-crucible-downstairs:downstairs-9ebf14d5-9a00-4122-914a-ab79e2e1a77e.log (3038 bytes)
13092023-12-13T05:24:01.601Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_d04b9c38-9134-4b45-9f7c-ded66ffdc85f/root/var/svc/log/oxide-crucible-downstairs:downstairs-9ebf14d5-9a00-4122-914a-ab79e2e1a77e.log
13102023-12-13T05:24:01.601Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_d04b9c38-9134-4b45-9f7c-ded66ffdc85f/root/var/svc/log/oxide-crucible-downstairs:downstairs-d82c740f-3a42-48e0-bea2-d18319115b5d.log (6406 bytes)
13112023-12-13T05:24:03.183Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_d04b9c38-9134-4b45-9f7c-ded66ffdc85f/root/var/svc/log/oxide-crucible-downstairs:downstairs-d82c740f-3a42-48e0-bea2-d18319115b5d.log
13122023-12-13T05:24:03.183Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_d04b9c38-9134-4b45-9f7c-ded66ffdc85f/root/var/svc/log/oxide-crucible-downstairs:downstairs-f3a51bde-6262-4118-8c4c-89edab205a97.log (5516 bytes)
13132023-12-13T05:24:04.280Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_d04b9c38-9134-4b45-9f7c-ded66ffdc85f/root/var/svc/log/oxide-crucible-downstairs:downstairs-f3a51bde-6262-4118-8c4c-89edab205a97.log
13142023-12-13T05:24:04.280Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_d04b9c38-9134-4b45-9f7c-ded66ffdc85f/root/var/svc/log/oxide-crucible-downstairs:snapshot-d82c740f-3a42-48e0-bea2-d18319115b5d-8b254072-f4b4-466e-b1a6-8c3feab28c95.log (5156 bytes)
13152023-12-13T05:24:05.394Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_d04b9c38-9134-4b45-9f7c-ded66ffdc85f/root/var/svc/log/oxide-crucible-downstairs:snapshot-d82c740f-3a42-48e0-bea2-d18319115b5d-8b254072-f4b4-466e-b1a6-8c3feab28c95.log
13162023-12-13T05:24:05.419Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_3d1dbcf8-31c2-4907-9c33-76d8b31d3794/root/var/svc/log/oxide-external_dns:default.log (30548 bytes)
13172023-12-13T05:24:05.553Zupload warning: file "/pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_3d1dbcf8-31c2-4907-9c33-76d8b31d3794/root/var/svc/log/oxide-external_dns:default.log" changed size mid upload: 30548 -> 34310
13182023-12-13T05:24:06.666Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_3d1dbcf8-31c2-4907-9c33-76d8b31d3794/root/var/svc/log/oxide-external_dns:default.log
13192023-12-13T05:24:06.666Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_3d1dbcf8-31c2-4907-9c33-76d8b31d3794/root/var/svc/log/oxide-external_dns:default.log.0 (75 bytes)
13202023-12-13T05:24:07.783Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_3d1dbcf8-31c2-4907-9c33-76d8b31d3794/root/var/svc/log/oxide-external_dns:default.log.0
13212023-12-13T05:24:07.783Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_5643f65f-b1e6-4f36-9f00-ac36119962e3/root/var/svc/log/oxide-internal_dns:default.log (27542 bytes)
13222023-12-13T05:24:08.340Zupload warning: file "/pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_5643f65f-b1e6-4f36-9f00-ac36119962e3/root/var/svc/log/oxide-internal_dns:default.log" changed size mid upload: 27542 -> 31297
13232023-12-13T05:24:09.415Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_5643f65f-b1e6-4f36-9f00-ac36119962e3/root/var/svc/log/oxide-internal_dns:default.log
13242023-12-13T05:24:09.415Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_5643f65f-b1e6-4f36-9f00-ac36119962e3/root/var/svc/log/oxide-internal_dns:default.log.0 (104926 bytes)
13252023-12-13T05:24:10.810Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_5643f65f-b1e6-4f36-9f00-ac36119962e3/root/var/svc/log/oxide-internal_dns:default.log.0
13262023-12-13T05:24:10.811Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_52fbf5fc-4b8d-45f3-95af-87bd85d3514c/root/var/svc/log/oxide-nexus:default.log (2666549 bytes)
13272023-12-13T05:24:13.580Zupload warning: file "/pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_52fbf5fc-4b8d-45f3-95af-87bd85d3514c/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 2666549 -> 2751815
13282023-12-13T05:24:14.654Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_52fbf5fc-4b8d-45f3-95af-87bd85d3514c/root/var/svc/log/oxide-nexus:default.log
13292023-12-13T05:24:14.654Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_52fbf5fc-4b8d-45f3-95af-87bd85d3514c/root/var/svc/log/oxide-nexus:default.log.0 (120 bytes)
13302023-12-13T05:24:15.752Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_52fbf5fc-4b8d-45f3-95af-87bd85d3514c/root/var/svc/log/oxide-nexus:default.log.0
13312023-12-13T05:24:15.752Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_oximeter_916c7ed2-6f94-4b70-b159-8507ba9f8d1b/root/var/svc/log/oxide-oximeter:default.log (239221 bytes)
13322023-12-13T05:24:16.341Zupload warning: file "/pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_oximeter_916c7ed2-6f94-4b70-b159-8507ba9f8d1b/root/var/svc/log/oxide-oximeter:default.log" changed size mid upload: 239221 -> 297566
13332023-12-13T05:24:17.416Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_oximeter_916c7ed2-6f94-4b70-b159-8507ba9f8d1b/root/var/svc/log/oxide-oximeter:default.log
13342023-12-13T05:24:17.416Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_oximeter_916c7ed2-6f94-4b70-b159-8507ba9f8d1b/root/var/svc/log/oxide-oximeter:default.log.0 (11980 bytes)
13352023-12-13T05:24:18.566Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_oximeter_916c7ed2-6f94-4b70-b159-8507ba9f8d1b/root/var/svc/log/oxide-oximeter:default.log.0
13362023-12-13T05:24:18.566Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_0cb04175-0aee-40d5-ad7b-d155caa34d88/root/var/svc/log/oxide-crucible-agent:default.log (14116 bytes)
13372023-12-13T05:24:19.703Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_0cb04175-0aee-40d5-ad7b-d155caa34d88/root/var/svc/log/oxide-crucible-agent:default.log
13382023-12-13T05:24:19.727Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_0cb04175-0aee-40d5-ad7b-d155caa34d88/root/var/svc/log/oxide-crucible-agent:default.log.0 (3436 bytes)
13392023-12-13T05:24:20.808Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_0cb04175-0aee-40d5-ad7b-d155caa34d88/root/var/svc/log/oxide-crucible-agent:default.log.0
13402023-12-13T05:24:20.808Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_0cb04175-0aee-40d5-ad7b-d155caa34d88/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
13412023-12-13T05:24:21.878Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_0cb04175-0aee-40d5-ad7b-d155caa34d88/root/var/svc/log/oxide-crucible-downstairs:default.log
13422023-12-13T05:24:21.878Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_0cb04175-0aee-40d5-ad7b-d155caa34d88/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
13432023-12-13T05:24:22.971Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_0cb04175-0aee-40d5-ad7b-d155caa34d88/root/var/svc/log/oxide-crucible-downstairs:default.log.0
13442023-12-13T05:24:22.971Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_0cb04175-0aee-40d5-ad7b-d155caa34d88/root/var/svc/log/oxide-crucible-downstairs:downstairs-3d3d907f-2e14-451d-bfa0-b028649e6ee4.log (6407 bytes)
13452023-12-13T05:24:24.088Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_0cb04175-0aee-40d5-ad7b-d155caa34d88/root/var/svc/log/oxide-crucible-downstairs:downstairs-3d3d907f-2e14-451d-bfa0-b028649e6ee4.log
13462023-12-13T05:24:24.088Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_0cb04175-0aee-40d5-ad7b-d155caa34d88/root/var/svc/log/oxide-crucible-downstairs:snapshot-3d3d907f-2e14-451d-bfa0-b028649e6ee4-8b254072-f4b4-466e-b1a6-8c3feab28c95.log (5157 bytes)
13472023-12-13T05:24:25.195Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_0cb04175-0aee-40d5-ad7b-d155caa34d88/root/var/svc/log/oxide-crucible-downstairs:snapshot-3d3d907f-2e14-451d-bfa0-b028649e6ee4-8b254072-f4b4-466e-b1a6-8c3feab28c95.log
13482023-12-13T05:24:25.195Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_b1a429fd-f5ac-4d28-9600-524ac191644c/root/var/svc/log/oxide-crucible-pantry:default.log (0 bytes)
13492023-12-13T05:24:26.323Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_b1a429fd-f5ac-4d28-9600-524ac191644c/root/var/svc/log/oxide-crucible-pantry:default.log
13502023-12-13T05:24:26.323Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_b1a429fd-f5ac-4d28-9600-524ac191644c/root/var/svc/log/oxide-crucible-pantry:default.log.0 (2338 bytes)
13512023-12-13T05:24:27.466Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_b1a429fd-f5ac-4d28-9600-524ac191644c/root/var/svc/log/oxide-crucible-pantry:default.log.0
13522023-12-13T05:24:27.511Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_93218afa-3969-4308-aec1-fcf5b39775ff/root/var/svc/log/oxide-internal_dns:default.log (27536 bytes)
13532023-12-13T05:24:28.257Zupload warning: file "/pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_93218afa-3969-4308-aec1-fcf5b39775ff/root/var/svc/log/oxide-internal_dns:default.log" changed size mid upload: 27536 -> 35047
13542023-12-13T05:24:29.899Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_93218afa-3969-4308-aec1-fcf5b39775ff/root/var/svc/log/oxide-internal_dns:default.log
13552023-12-13T05:24:29.899Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_93218afa-3969-4308-aec1-fcf5b39775ff/root/var/svc/log/oxide-internal_dns:default.log.0 (102477 bytes)
13562023-12-13T05:24:31.669Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_93218afa-3969-4308-aec1-fcf5b39775ff/root/var/svc/log/oxide-internal_dns:default.log.0
13572023-12-13T05:24:31.669Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_ntp_12d9fff4-405b-4a0c-a4f4-b3d2413b256e/root/var/log/chrony/measurements.log (16714 bytes)
13582023-12-13T05:24:31.959Zupload warning: file "/pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_ntp_12d9fff4-405b-4a0c-a4f4-b3d2413b256e/root/var/log/chrony/measurements.log" changed size mid upload: 16714 -> 20687
13592023-12-13T05:24:33.550Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_ntp_12d9fff4-405b-4a0c-a4f4-b3d2413b256e/root/var/log/chrony/measurements.log
13602023-12-13T05:24:33.550Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_ntp_12d9fff4-405b-4a0c-a4f4-b3d2413b256e/root/var/log/chrony/statistics.log (9360 bytes)
13612023-12-13T05:24:33.821Zupload warning: file "/pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_ntp_12d9fff4-405b-4a0c-a4f4-b3d2413b256e/root/var/log/chrony/statistics.log" changed size mid upload: 9360 -> 11817
13622023-12-13T05:24:38.377Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_ntp_12d9fff4-405b-4a0c-a4f4-b3d2413b256e/root/var/log/chrony/statistics.log
13632023-12-13T05:24:38.377Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_ntp_12d9fff4-405b-4a0c-a4f4-b3d2413b256e/root/var/log/chrony/tracking.log (1984 bytes)
13642023-12-13T05:24:58.866Zupload warning: file "/pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_ntp_12d9fff4-405b-4a0c-a4f4-b3d2413b256e/root/var/log/chrony/tracking.log" changed size mid upload: 1984 -> 2248
13652023-12-13T05:24:59.767Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_ntp_12d9fff4-405b-4a0c-a4f4-b3d2413b256e/root/var/log/chrony/tracking.log