01HDJR63ZWVD7Q7JC4WDXNNH0D: helios / deploy

BasicConfig {
    output_rules: [
        "%/var/svc/log/oxide-sled-agent:default.log*",
        "%/pool/ext/*/crypt/zone/oxz_*/root/var/svc/log/oxide-*.log*",
        "%/pool/ext/*/crypt/zone/oxz_*/root/var/svc/log/system-illumos-*.log*",
        "%/pool/ext/*/crypt/zone/oxz_ntp_*/root/var/log/chrony/*.log*",
        "!/pool/ext/*/crypt/zone/oxz_propolis-server_*/root/var/svc/log/*.log*",
        "%/pool/ext/*/crypt/debug/global/oxide-sled-agent:default.log.*",
        "%/pool/ext/*/crypt/debug/oxz_*/oxide-*.log.*",
        "%/pool/ext/*/crypt/debug/oxz_*/system-illumos-*.log.*",
        "!/pool/ext/*/crypt/debug/oxz_propolis-server_*/*.log.*",
    ],
    rust_toolchain: None,
    target: Some(
        "lab-2.0-opte-0.25",
    ),
    access_repos: [],
    publish: [],
    skip_clone: true,
}

Buildomat Job: 01HDJR8RREGGX1Z6X2ZSS6EKED

Tags:

Artefacts:

Output:

SEQ GLOBAL TIME DETAILS
12023-10-25T07:20:12.785Zjob dependencies complete; ready to run (waiting for 52 m 35 s)
22023-10-25T07:21:01.077Zjob assigned to worker 01HDJV99JV0JPV6KBNK4TXN7P5 (queued for 48 s)
32023-10-25T07:21:01.256Zhttp://172.20.0.5:9969/ipxe/sock... ok
42023-10-25T07:21:01.307Zhttp://172.20.0.5:9969/os/sock/platform/i86pc/kernel/amd64/unix... ok
52023-10-25T07:21:01.340Zhttp://172.20.0.5:9969/os/sock/platform/i86pc/amd64/boot_archive... ok
62023-10-25T07:21:01.381Zhttp://172.20.0.5:9969/os/sock/platform/i86pc/amd64/boot_archive.hash... ok
72023-10-25T07:21:01.437Zhttp://172.20.0.5:9969/postboot/sock... ok
82023-10-25T07:21:01.474ZwcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U
92023-10-25T07:21:01.507Zcpu0: microcode has been updated from version 0xa001133 to 0xa0011d1
102023-10-25T07:21:01.544ZOxide Helios Version helios-2.0.22241 64-bit
112023-10-25T07:21:01.583ZConfiguring devices.
122023-10-25T07:21:01.619Z##################################################
132023-10-25T07:21:01.660Z#### ############# ##
142023-10-25T07:21:01.718Z### ### ############ ##
152023-10-25T07:21:01.774Z## ### # ## ### ## Oxide Computer Company ##
162023-10-25T07:21:01.856Z## ## ## ### # ### ##
172023-10-25T07:21:01.919Z## # ### #### #### This Station Under ##
182023-10-25T07:21:01.965Z### ### #### # ### Computer Control ##
192023-10-25T07:21:02.044Z#### #### ### ## ##
202023-10-25T07:21:02.112Z##################################################
212023-10-25T07:21:02.190ZHostname: sock
222023-10-25T07:21:02.293ZLoading smf(7) serice descriptions:
232023-10-25T07:21:02.731Zdownloading input: /input/ci-tools/work/end-to-end-tests/bootstrap-7505a19fa56a5ab1.gz
242023-10-25T07:21:04.492Zdownloaded input: /input/ci-tools/work/end-to-end-tests/bootstrap-7505a19fa56a5ab1.gz
252023-10-25T07:21:04.526Zdownloading input: /input/ci-tools/work/end-to-end-tests/bootstrap.gz
262023-10-25T07:21:29.543Zdownloaded input: /input/ci-tools/work/end-to-end-tests/bootstrap.gz
272023-10-25T07:21:29.568Zdownloading input: /input/ci-tools/work/end-to-end-tests/end_to_end_tests-a486f79ff30d68c1.gz
282023-10-25T07:21:56.092Zdownloaded input: /input/ci-tools/work/end-to-end-tests/end_to_end_tests-a486f79ff30d68c1.gz
292023-10-25T07:21:56.143Zdownloading input: /input/ci-tools/work/caboose-util.gz
302023-10-25T07:21:57.342Zdownloaded input: /input/ci-tools/work/caboose-util.gz
312023-10-25T07:21:57.376Zdownloading input: /input/ci-tools/work/tufaceous.gz
322023-10-25T07:22:00.314Zdownloaded input: /input/ci-tools/work/tufaceous.gz
332023-10-25T07:22:00.338Zdownloading input: /input/package/work/version.txt
342023-10-25T07:22:00.363Zdownloaded input: /input/package/work/version.txt
352023-10-25T07:22:00.388Zdownloading input: /input/package/work/package.tar.gz
362023-10-25T07:22:17.696Zdownloaded input: /input/package/work/package.tar.gz
372023-10-25T07:22:17.723Zdownloading input: /input/package/work/global-zone-packages.tar.gz
382023-10-25T07:22:44.131Zdownloaded input: /input/package/work/global-zone-packages.tar.gz
392023-10-25T07:22:44.161Zdownloading input: /input/package/work/trampoline-global-zone-packages.tar.gz
402023-10-25T07:23:04.574Zdownloaded input: /input/package/work/trampoline-global-zone-packages.tar.gz
412023-10-25T07:23:04.615Zdownloading input: /input/package/work/zones/clickhouse.tar.gz
422023-10-25T07:23:55.543Zdownloaded input: /input/package/work/zones/clickhouse.tar.gz
432023-10-25T07:23:55.600Zdownloading input: /input/package/work/zones/clickhouse_keeper.tar.gz
442023-10-25T07:24:50.423Zdownloaded input: /input/package/work/zones/clickhouse_keeper.tar.gz
452023-10-25T07:24:51.387Zdownloading input: /input/package/work/zones/cockroachdb.tar.gz
462023-10-25T07:25:24.400Zdownloaded input: /input/package/work/zones/cockroachdb.tar.gz
472023-10-25T07:25:24.431Zdownloading input: /input/package/work/zones/crucible-pantry.tar.gz
482023-10-25T07:25:32.320Zdownloaded input: /input/package/work/zones/crucible-pantry.tar.gz
492023-10-25T07:25:32.363Zdownloading input: /input/package/work/zones/crucible.tar.gz
502023-10-25T07:25:45.232Zdownloaded input: /input/package/work/zones/crucible.tar.gz
512023-10-25T07:25:45.275Zdownloading input: /input/package/work/zones/external-dns.tar.gz
522023-10-25T07:25:52.809Zdownloaded input: /input/package/work/zones/external-dns.tar.gz
532023-10-25T07:25:53.243Zdownloading input: /input/package/work/zones/internal-dns.tar.gz
542023-10-25T07:25:57.807Zdownloaded input: /input/package/work/zones/internal-dns.tar.gz
552023-10-25T07:25:57.847Zdownloading input: /input/package/work/zones/ntp.tar.gz
562023-10-25T07:25:57.879Zdownloaded input: /input/package/work/zones/ntp.tar.gz
572023-10-25T07:25:57.917Zdownloading input: /input/package/work/zones/omicron-gateway-asic.tar.gz
582023-10-25T07:26:02.190Zdownloaded input: /input/package/work/zones/omicron-gateway-asic.tar.gz
592023-10-25T07:26:02.215Zdownloading input: /input/package/work/zones/omicron-gateway-softnpu.tar.gz
602023-10-25T07:26:06.112Zdownloaded input: /input/package/work/zones/omicron-gateway-softnpu.tar.gz
612023-10-25T07:26:06.149Zdownloading input: /input/package/work/zones/omicron-nexus-single-sled.tar.gz
622023-10-25T07:26:28.284Zdownloaded input: /input/package/work/zones/omicron-nexus-single-sled.tar.gz
632023-10-25T07:26:28.315Zdownloading input: /input/package/work/zones/omicron-nexus.tar.gz
642023-10-25T07:26:48.939Zdownloaded input: /input/package/work/zones/omicron-nexus.tar.gz
652023-10-25T07:26:48.963Zdownloading input: /input/package/work/zones/overlay.tar.gz
662023-10-25T07:26:49.000Zdownloaded input: /input/package/work/zones/overlay.tar.gz
672023-10-25T07:26:49.039Zdownloading input: /input/package/work/zones/oximeter-collector.tar.gz
682023-10-25T07:26:55.664Zdownloaded input: /input/package/work/zones/oximeter-collector.tar.gz
692023-10-25T07:26:55.696Zdownloading input: /input/package/work/zones/propolis-server.tar.gz
702023-10-25T07:27:16.181Zdownloaded input: /input/package/work/zones/propolis-server.tar.gz
712023-10-25T07:27:16.239Zdownloading input: /input/package/work/zones/switch-asic.tar.gz
722023-10-25T07:28:04.010Zdownloaded input: /input/package/work/zones/switch-asic.tar.gz
732023-10-25T07:28:04.036Zdownloading input: /input/package/work/zones/switch-softnpu.tar.gz
742023-10-25T07:28:34.735Zdownloaded input: /input/package/work/zones/switch-softnpu.tar.gz
 
752023-10-25T07:28:34.765Zstarting task 0: "setup"
762023-10-25T07:28:34.805Z++ uname -s
772023-10-25T07:28:34.832Z+ kern=SunOS
782023-10-25T07:28:34.858Z+ case "$kern" in
792023-10-25T07:28:34.887Z+ groupadd -g 12345 build
802023-10-25T07:28:34.920Z+ useradd -u 12345 -g build -d /home/build -s /bin/bash -c build -P 'Primary Administrator' build
812023-10-25T07:28:36.811Z+ zfs create -o mountpoint=/work rpool/work
822023-10-25T07:28:36.865Z++ awk '$2 == "/home" { print $3 }' /etc/mnttab
832023-10-25T07:28:36.895Z+ home_fs=zfs
842023-10-25T07:28:36.929Z+ [[ zfs == autofs ]]
852023-10-25T07:28:36.964Z+ mkdir -p /home/build
862023-10-25T07:28:36.991Z+ chown build:build /home/build /work
872023-10-25T07:28:37.860Z+ chmod 0700 /home/build /work
882023-10-25T07:28:37.913Zprocess exited: duration 3069 ms, exit code 0
 
892023-10-25T07:28:38.374Zstarting task 1: "authentication"
902023-10-25T07:28:38.442Zprocess exited: duration 43 ms, exit code 0
 
912023-10-25T07:28:38.493Zstarting task 2: "build"
922023-10-25T07:28:38.526Z+ trap _exit_trap EXIT
932023-10-25T07:28:38.551Z+ pfexec mdb -kwe 'secpolicy_fs_linkdir/v 55 48 89 e5 b8 01 00 00 00 5d c3'
942023-10-25T07:28:38.722Zsecpolicy_fs_linkdir: 0x55 = 0x55
952023-10-25T07:28:38.750Zsecpolicy_fs_linkdir+1: 0xb8 = 0x48
962023-10-25T07:28:38.778Zsecpolicy_fs_linkdir+2: 0x1 = 0x89
972023-10-25T07:28:38.809Zsecpolicy_fs_linkdir+3: 0 = 0xe5
982023-10-25T07:28:38.842Zsecpolicy_fs_linkdir+4: 0 = 0xb8
992023-10-25T07:28:38.870Zsecpolicy_fs_linkdir+5: 0 = 0x1
1002023-10-25T07:28:38.897Zsecpolicy_fs_linkdir+6: 0x48 = 0x0
1012023-10-25T07:28:38.941Zsecpolicy_fs_linkdir+7: 0x89 = 0x0
1022023-10-25T07:28:38.967Zsecpolicy_fs_linkdir+8: 0xe5 = 0x0
1032023-10-25T07:28:38.990Zsecpolicy_fs_linkdir+9: 0x48 = 0x5d
1042023-10-25T07:28:39.014Zsecpolicy_fs_linkdir+0xa: 0x83 = 0xc3
1052023-10-25T07:28:39.038Z+ [[ -d /opt/oxide ]]
1062023-10-25T07:28:39.062Z+ pfexec mv /opt/oxide /opt/oxide-underneath
1072023-10-25T07:28:39.085Z+ pfexec mkdir /opt/oxide
1082023-10-25T07:28:39.109Z+ pfexec mount -F tmpfs -O swap /opt/oxide
1092023-10-25T07:28:39.132Z+ [[ -d /opt/oxide-underneath ]]
1102023-10-25T07:28:39.156Z+ cd /opt/oxide-underneath
1112023-10-25T07:28:39.180Z+ pfexec tar ceEp@/f - .
1122023-10-25T07:28:39.204Z+ cd /opt/oxide
1132023-10-25T07:28:39.228Z+ pfexec tar xveEp@/f -
1142023-10-25T07:28:39.658Zx ., 0 bytes, 0 tape blocks
1152023-10-25T07:28:39.685Zx ./opte, 0 bytes, 0 tape blocks
1162023-10-25T07:28:39.715Zx ./opte/bin, 0 bytes, 0 tape blocks
1172023-10-25T07:28:39.743Zx ./opte/bin/opteadm, 50714832 bytes, 99053 tape blocks
1182023-10-25T07:28:39.769Z+ pfexec rm -rf /opt/oxide-underneath
1192023-10-25T07:28:39.793Z+ pfexec /sbin/zfs create -o mountpoint=/zone rpool/zone
1202023-10-25T07:28:39.819Z+ pfexec mkdir /opt/oxide/work
1212023-10-25T07:28:39.843Z+ pfexec chown build:build /opt/oxide/work
1222023-10-25T07:28:39.868Z+ cd /opt/oxide/work
1232023-10-25T07:28:39.898Z+ ptime -m tar xvzf /input/package/work/package.tar.gz
1242023-10-25T07:28:40.448ZDecompressing '/input/package/work/package.tar.gz' with '/usr/bin/gzcat'...
1252023-10-25T07:28:40.489Zx out/mg-ddm-gz.tar, 63638528 bytes, 124294 tape blocks
1262023-10-25T07:28:40.517Zx out/omicron-sled-agent.tar, 115152384 bytes, 224907 tape blocks
1272023-10-25T07:28:40.546Zx out/target/test, 75 bytes, 1 tape blocks
1282023-10-25T07:28:40.579Zx out/npuzone/npuzone, 37381232 bytes, 73011 tape blocks
1292023-10-25T07:28:40.605Zx package-manifest.toml, 22037 bytes, 44 tape blocks
1302023-10-25T07:28:40.636Zx smf/sled-agent/non-gimlet/config.toml, 3097 bytes, 7 tape blocks
1312023-10-25T07:28:40.667Zx target/release/omicron-package, 38200896 bytes, 74612 tape blocks
1322023-10-25T07:28:40.697Zx tools/create_virtual_hardware.sh, 2986 bytes, 6 tape blocks
1332023-10-25T07:28:40.727Zx tools/virtual_hardware.sh, 3408 bytes, 7 tape blocks
1342023-10-25T07:28:40.756Zx tools/scrimlet/create-softnpu-zone.sh, 619 bytes, 2 tape blocks
1352023-10-25T07:28:40.784Zx tools/scrimlet/destroy-softnpu-zone.sh, 195 bytes, 1 tape blocks
1362023-10-25T07:28:40.813Zx tools/scrimlet/softnpu-init.sh, 2148 bytes, 5 tape blocks
1372023-10-25T07:28:40.845Zx tools/scrimlet/softnpu-zone.txt, 330 bytes, 1 tape blocks
1382023-10-25T07:28:40.877Zx tools/scrimlet/softnpu.toml, 171 bytes, 1 tape blocks
1392023-10-25T07:28:40.902Z
1402023-10-25T07:28:40.942Zreal 1.551970554
1412023-10-25T07:28:40.993Zuser 1.394815305
1422023-10-25T07:28:41.024Zsys 0.277467897
1432023-10-25T07:28:41.048Ztrap 0.000064550
1442023-10-25T07:28:41.074Ztflt 0.000000000
1452023-10-25T07:28:41.108Zdflt 0.000000000
1462023-10-25T07:28:41.138Zkflt 0.000000000
1472023-10-25T07:28:41.163Zlock 0.000000000
1482023-10-25T07:28:41.196Zslp 1.392438779
1492023-10-25T07:28:41.238Zlat 0.036928715
1502023-10-25T07:28:41.678Zstop 0.000121811
1512023-10-25T07:28:41.702Z+ cp /input/package/work/zones/clickhouse_keeper.tar.gz /input/package/work/zones/clickhouse.tar.gz /input/package/work/zones/cockroachdb.tar.gz /input/package/work/zones/crucible-pantry.tar.gz /input/package/work/zones/crucible.tar.gz /input/package/work/zones/external-dns.tar.gz /input/package/work/zones/internal-dns.tar.gz /input/package/work/zones/ntp.tar.gz /input/package/work/zones/omicron-gateway-asic.tar.gz /input/package/work/zones/omicron-gateway-softnpu.tar.gz /input/package/work/zones/omicron-nexus-single-sled.tar.gz /input/package/work/zones/omicron-nexus.tar.gz /input/package/work/zones/overlay.tar.gz /input/package/work/zones/oximeter-collector.tar.gz /input/package/work/zones/propolis-server.tar.gz /input/package/work/zones/switch-asic.tar.gz /input/package/work/zones/switch-softnpu.tar.gz out/
1522023-10-25T07:28:41.725Z+ mv out/omicron-nexus-single-sled.tar.gz out/omicron-nexus.tar.gz
1532023-10-25T07:28:41.749Z+ mkdir tests
1542023-10-25T07:28:41.772Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1552023-10-25T07:28:41.796Z+ ptime -m gunzip
1562023-10-25T07:28:41.819Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap-7505a19fa56a5ab1
1572023-10-25T07:28:41.843Z
1582023-10-25T07:28:41.869Zreal 0.144232115
1592023-10-25T07:28:41.904Zuser 0.129594791
1602023-10-25T07:28:41.935Zsys 0.014407819
1612023-10-25T07:28:41.964Ztrap 0.000010820
1622023-10-25T07:28:42.032Ztflt 0.000000000
1632023-10-25T07:28:42.061Zdflt 0.000000000
1642023-10-25T07:28:42.092Zkflt 0.000000000
1652023-10-25T07:28:42.117Zlock 0.000000000
1662023-10-25T07:28:42.140Zslp 0.000018010
1672023-10-25T07:28:42.164Zlat 0.000020569
1682023-10-25T07:28:42.187Zstop 0.000117592
1692023-10-25T07:28:42.211Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap-7505a19fa56a5ab1
1702023-10-25T07:28:42.234Z+ chmod a+x tests/bootstrap-7505a19fa56a5ab1
1712023-10-25T07:28:42.258Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1722023-10-25T07:28:42.282Z+ ptime -m gunzip
1732023-10-25T07:28:42.306Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap
1742023-10-25T07:28:43.913Z
1752023-10-25T07:28:43.942Zreal 2.230401123
1762023-10-25T07:28:43.969Zuser 2.102724752
1772023-10-25T07:28:43.996Zsys 0.127276533
1782023-10-25T07:28:44.022Ztrap 0.000171473
1792023-10-25T07:28:44.045Ztflt 0.000000000
1802023-10-25T07:28:44.069Zdflt 0.000000000
1812023-10-25T07:28:44.104Zkflt 0.000000000
1822023-10-25T07:28:44.141Zlock 0.000000000
1832023-10-25T07:28:44.181Zslp 0.000018950
1842023-10-25T07:28:44.221Zlat 0.000022140
1852023-10-25T07:28:44.249Zstop 0.000125382
1862023-10-25T07:28:44.275Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap
1872023-10-25T07:28:44.299Z+ chmod a+x tests/bootstrap
1882023-10-25T07:28:44.323Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1892023-10-25T07:28:44.347Z+ ptime -m gunzip
1902023-10-25T07:28:44.371Z++ basename /input/ci-tools/work/end-to-end-tests/end_to_end_tests-a486f79ff30d68c1
1912023-10-25T07:28:46.485Z
1922023-10-25T07:28:46.509Zreal 2.559185587
1932023-10-25T07:28:46.533Zuser 2.393963431
1942023-10-25T07:28:46.556Zsys 0.164806977
1952023-10-25T07:28:46.580Ztrap 0.000192323
1962023-10-25T07:28:46.604Ztflt 0.000000000
1972023-10-25T07:28:46.627Zdflt 0.000000000
1982023-10-25T07:28:46.651Zkflt 0.000000000
1992023-10-25T07:28:46.675Zlock 0.000000000
2002023-10-25T07:28:46.699Zslp 0.000017340
2012023-10-25T07:28:46.723Zlat 0.000022800
2022023-10-25T07:28:46.747Zstop 0.000122961
2032023-10-25T07:28:46.770Z++ basename /input/ci-tools/work/end-to-end-tests/end_to_end_tests-a486f79ff30d68c1
2042023-10-25T07:28:46.795Z+ chmod a+x tests/end_to_end_tests-a486f79ff30d68c1
2052023-10-25T07:28:46.818Z+ bmat address ls
2062023-10-25T07:28:46.842ZNAME CIDR FIRST LAST COUNT
2072023-10-25T07:28:46.871Zextra 10.151.2.0/24 10.151.2.164 10.151.2.227 64
2082023-10-25T07:28:46.903Z++ bmat address ls -f extra -Ho first,last,count
2092023-10-25T07:28:46.934Z+ set -- 10.151.2.164 10.151.2.227 64
2102023-10-25T07:28:46.958Z+ EXTRA_IP_START=10.151.2.164
2112023-10-25T07:28:46.982Z+ EXTRA_IP_END=10.151.2.227
2122023-10-25T07:28:47.005Z+ EXTRA_IP_COUNT=64
2132023-10-25T07:28:47.029Z+ (( EXTRA_IP_COUNT >= 32 ))
2142023-10-25T07:28:47.054Z+ EXTRA_IP_BASE=10.151.2
2152023-10-25T07:28:47.078Z+ EXTRA_IP_FOCTET=164
2162023-10-25T07:28:47.101Z+ EXTRA_IP_LOCTET=227
2172023-10-25T07:28:47.130Z+ SERVICE_IP_POOL_START=10.151.2.164
2182023-10-25T07:28:47.156Z+ SERVICE_IP_POOL_END=10.151.2.173
2192023-10-25T07:28:47.197Z+ DNS_IP1=10.151.2.164
2202023-10-25T07:28:47.221Z+ DNS_IP2=10.151.2.165
2212023-10-25T07:28:47.245Z+ UPLINK_IP=10.151.2.174
2222023-10-25T07:28:47.268Z+ IPPOOL_START=10.151.2.175
2232023-10-25T07:28:47.405Z+ IPPOOL_END=10.151.2.227
2242023-10-25T07:28:47.434Z++ ipadm show-addr -po type,addr
2252023-10-25T07:28:47.458Z++ awk '-F[:/]' '$1 == "dhcp" {print $2}'
2262023-10-25T07:28:47.482Z+ GATEWAY_IP=10.151.2.100
2272023-10-25T07:28:47.506Z+ [[ -n 10.151.2.100 ]]
2282023-10-25T07:28:47.529Z+ ping -s 10.151.2.100 56 1
2292023-10-25T07:28:47.559ZPING 10.151.2.100: 56 data bytes
2302023-10-25T07:28:47.585Z64 bytes from 10.151.2.100: icmp_seq=0. time=0.010 ms
2312023-10-25T07:28:47.612Z
2322023-10-25T07:28:47.639Z----10.151.2.100 PING Statistics----
2332023-10-25T07:28:47.670Z1 packets transmitted, 1 packets received, 0% packet loss
2342023-10-25T07:28:47.696Zround-trip (ms) min/avg/max/stddev = 0.010/0.010/0.010/-nan
2352023-10-25T07:28:47.746Z++ arp -an
2362023-10-25T07:28:47.771Z++ awk -vgw=10.151.2.100 '$2 == gw {print $NF}'
2372023-10-25T07:28:47.795Z+ GATEWAY_MAC=18:c0:4d:0d:a0:2a
2382023-10-25T07:28:47.819Z+ routeadm -e ipv4-forwarding -u
2392023-10-25T07:28:47.844Z+ PXA_START=10.151.2.164
2402023-10-25T07:28:47.877Z+ PXA_END=10.151.2.227
2412023-10-25T07:28:47.904Z+ export GATEWAY_IP GATEWAY_MAC PXA_START PXA_END
2422023-10-25T07:28:47.927Z+ pfexec zpool create -f scratch c1t1d0 c2t1d0
2432023-10-25T07:28:47.951Z+ ZPOOL_VDEV_DIR=/scratch
2442023-10-25T07:28:47.975Z+ ptime -m pfexec ./tools/create_virtual_hardware.sh
2452023-10-25T07:28:47.999Z+++ dirname ./tools/create_virtual_hardware.sh
2462023-10-25T07:28:48.023Z++ cd ./tools
2472023-10-25T07:28:48.046Z++ pwd
2482023-10-25T07:28:48.070Z+ SOURCE_DIR=/opt/oxide/work/tools
2492023-10-25T07:28:48.099Z+ OMICRON_TOP=/opt/oxide/work/tools/..
2502023-10-25T07:28:48.126Z+ . /opt/oxide/work/tools/virtual_hardware.sh
2512023-10-25T07:28:48.150Z++ MARKER=/etc/opt/oxide/NO_INSTALL
2522023-10-25T07:28:48.173Z++ [[ -f /etc/opt/oxide/NO_INSTALL ]]
2532023-10-25T07:28:48.221Z++ dladm show-phys -p -o LINK
2542023-10-25T07:28:48.245Z++ head -1
2552023-10-25T07:28:48.268Z+ PHYSICAL_LINK=igb0
2562023-10-25T07:28:48.293Z+ echo 'Using igb0 as physical link'
2572023-10-25T07:28:48.317ZUsing igb0 as physical link
2582023-10-25T07:28:48.342Z+ ensure_run_as_root
2592023-10-25T07:28:48.366Z++ id -u
2602023-10-25T07:28:48.389Z+ [[ 0 -ne 0 ]]
2612023-10-25T07:28:48.413Z+ ensure_zpools
2622023-10-25T07:28:48.436Z+ ZPOOL_TYPES=('oxp_' 'oxi_')
2632023-10-25T07:28:48.460Z+ for ZPOOL_TYPE in "${ZPOOL_TYPES[@]}"
2642023-10-25T07:28:48.484Z+ readarray -t ZPOOLS
2652023-10-25T07:28:48.507Z++ grep '"oxp_' /opt/oxide/work/tools/../smf/sled-agent/non-gimlet/config.toml
2662023-10-25T07:28:48.531Z++ sed 's/[ ",]//g'
2672023-10-25T07:28:48.554Z+ for ZPOOL in "${ZPOOLS[@]}"
2682023-10-25T07:28:48.578Z+ echo 'Zpool: [oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b]'
2692023-10-25T07:28:48.603ZZpool: [oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b]
2702023-10-25T07:28:48.630Z+ VDEV_PATH=/scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
2712023-10-25T07:28:48.658Z+ [[ -f /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
2722023-10-25T07:28:48.684Z+ dd if=/dev/zero of=/scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
2732023-10-25T07:28:48.717Z0+0 records in
2742023-10-25T07:28:48.748Z0+0 records out
2752023-10-25T07:28:49.188ZZFS vdev /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
2762023-10-25T07:28:49.218Z0 bytes (0 iB) transferred in 0.000005 secs (0 iB/sec)
2772023-10-25T07:28:49.275ZZFS zpool oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b exists
2782023-10-25T07:28:49.305Z+ success 'ZFS vdev /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
2792023-10-25T07:28:49.334ZZpool: [oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03]
2802023-10-25T07:28:49.361Z+ set +x
2812023-10-25T07:28:49.391ZZFS vdev /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
2822023-10-25T07:28:49.416Z++ zpool list -o name
2832023-10-25T07:28:49.448ZZFS zpool oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
2842023-10-25T07:28:49.486Z++ grep oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b
2852023-10-25T07:28:49.514ZZpool: [oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03]
2862023-10-25T07:28:49.540Z+ [[ -z '' ]]
2872023-10-25T07:28:49.578ZZFS vdev /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
2882023-10-25T07:28:49.603Z+ zpool create -f oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
2892023-10-25T07:28:49.627ZZFS zpool oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
2902023-10-25T07:28:49.655Z+ success 'ZFS zpool oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
2912023-10-25T07:28:49.681ZZpool: [oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03]
2922023-10-25T07:28:49.706Z+ set +x
2932023-10-25T07:28:49.729ZZFS vdev /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
2942023-10-25T07:28:49.753Z+ for ZPOOL in "${ZPOOLS[@]}"
2952023-10-25T07:28:49.782ZZFS zpool oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
2962023-10-25T07:28:49.812Z+ echo 'Zpool: [oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03]'
2972023-10-25T07:28:49.837ZZpool: [oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03]
2982023-10-25T07:28:49.860Z+ VDEV_PATH=/scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
2992023-10-25T07:28:49.886ZZFS vdev /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
3002023-10-25T07:28:49.913Z+ [[ -f /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3012023-10-25T07:28:49.937ZZFS zpool oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
3022023-10-25T07:28:49.960Z+ dd if=/dev/zero of=/scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3032023-10-25T07:28:49.983ZZpool: [oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930]
3042023-10-25T07:28:50.007Z0+0 records in
3052023-10-25T07:28:50.030ZZFS vdev /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev exists
3062023-10-25T07:28:50.054Z0+0 records out
3072023-10-25T07:28:50.083ZZFS zpool oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 exists
3082023-10-25T07:28:50.118Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
3092023-10-25T07:28:50.147ZZpool: [oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60]
3102023-10-25T07:28:50.172Z+ success 'ZFS vdev /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3112023-10-25T07:28:50.209ZZFS vdev /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev exists
3122023-10-25T07:28:50.236Z+ set +x
3132023-10-25T07:28:50.259ZZFS zpool oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 exists
3142023-10-25T07:28:50.293Z++ zpool list -o name
3152023-10-25T07:28:50.319ZZpool: [oxp_31bd71cd-4736-4a12-a387-9b74b050396f]
3162023-10-25T07:28:50.344Z++ grep oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03
3172023-10-25T07:28:50.371ZZFS vdev /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev exists
3182023-10-25T07:28:50.397Z+ [[ -z '' ]]
3192023-10-25T07:28:50.421ZZFS zpool oxp_31bd71cd-4736-4a12-a387-9b74b050396f exists
3202023-10-25T07:28:50.444Z+ zpool create -f oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3212023-10-25T07:28:50.472ZZpool: [oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7]
3222023-10-25T07:28:50.499Z+ success 'ZFS zpool oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3232023-10-25T07:28:50.938ZZFS vdev /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev exists
3242023-10-25T07:28:51.378Z+ set +x
3252023-10-25T07:28:51.403ZZFS zpool oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 exists
3262023-10-25T07:28:51.434Z+ for ZPOOL in "${ZPOOLS[@]}"
3272023-10-25T07:28:51.468ZZpool: [oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b]
3282023-10-25T07:28:51.495Z+ echo 'Zpool: [oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3292023-10-25T07:28:51.520ZZFS vdev /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
3302023-10-25T07:28:51.543Z+ VDEV_PATH=/scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3312023-10-25T07:28:51.569ZZFS zpool oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b exists
3322023-10-25T07:28:51.592Z+ [[ -f /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3332023-10-25T07:28:51.616ZZpool: [oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b]
3342023-10-25T07:28:51.640Z+ dd if=/dev/zero of=/scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3352023-10-25T07:28:51.667ZZFS vdev /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
3362023-10-25T07:28:51.693Z0+0 records in
3372023-10-25T07:28:51.720ZZFS zpool oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b exists
3382023-10-25T07:28:51.744Z0+0 records out
3392023-10-25T07:28:51.768ZSimnet net0/sc0_0 exists
3402023-10-25T07:28:51.794Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3412023-10-25T07:28:51.818ZSimnet net1/sc1_0 exists
3422023-10-25T07:28:52.269Z+ success 'ZFS vdev /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3432023-10-25T07:28:52.734ZVnic sc0_1 exists
3442023-10-25T07:28:52.870Z+ set +x
3452023-10-25T07:28:52.893Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/softnpu/image/c1c42398c82b0220c8b5fa3bfba9c7a3bcaa0943/softnpu.sha256.txt
3462023-10-25T07:28:52.918Z++ zpool list -o name
3472023-10-25T07:28:52.949Z++ grep oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03
3482023-10-25T07:28:52.972Z+ [[ -z '' ]]
3492023-10-25T07:28:52.999Z+ zpool create -f oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3502023-10-25T07:28:53.029Z+ success 'ZFS zpool oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3512023-10-25T07:28:53.053Z+ set +x
3522023-10-25T07:28:53.076Z+ for ZPOOL in "${ZPOOLS[@]}"
3532023-10-25T07:28:53.100Z+ echo 'Zpool: [oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3542023-10-25T07:28:53.123Z+ VDEV_PATH=/scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3552023-10-25T07:28:53.148Z+ [[ -f /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3562023-10-25T07:28:53.172Z+ dd if=/dev/zero of=/scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3572023-10-25T07:28:53.196Z0+0 records in
3582023-10-25T07:28:53.219Z0+0 records out
3592023-10-25T07:28:53.243Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3602023-10-25T07:28:53.267Z+ success 'ZFS vdev /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3612023-10-25T07:28:53.290Z+ set +x
3622023-10-25T07:28:53.314Z++ zpool list -o name
3632023-10-25T07:28:53.338Z++ grep oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03
3642023-10-25T07:28:53.370Z+ [[ -z '' ]]
3652023-10-25T07:28:53.394Z+ zpool create -f oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3662023-10-25T07:28:53.417Z+ success 'ZFS zpool oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3672023-10-25T07:28:53.442Z+ set +x
3682023-10-25T07:28:53.467Z+ for ZPOOL in "${ZPOOLS[@]}"
3692023-10-25T07:28:53.493Z+ echo 'Zpool: [oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3702023-10-25T07:28:53.525Z+ VDEV_PATH=/scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3712023-10-25T07:28:53.550Z+ [[ -f /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3722023-10-25T07:28:53.577Z+ dd if=/dev/zero of=/scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3732023-10-25T07:28:53.600Z0+0 records in
3742023-10-25T07:28:53.624Z0+0 records out
3752023-10-25T07:28:53.648Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3762023-10-25T07:28:53.671Z+ success 'ZFS vdev /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3772023-10-25T07:28:53.695Z+ set +x
3782023-10-25T07:28:53.719Z++ zpool list -o name
3792023-10-25T07:28:53.742Z++ grep oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03
3802023-10-25T07:28:53.766Z+ [[ -z '' ]]
3812023-10-25T07:28:53.790Z+ zpool create -f oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3822023-10-25T07:28:53.813Z+ success 'ZFS zpool oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3832023-10-25T07:28:53.837Z+ set +x
3842023-10-25T07:28:53.860Z+ for ZPOOL in "${ZPOOLS[@]}"
3852023-10-25T07:28:53.883Z+ echo 'Zpool: [oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930]'
3862023-10-25T07:28:53.910Z+ VDEV_PATH=/scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev
3872023-10-25T07:28:53.937Z+ [[ -f /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev ]]
3882023-10-25T07:28:53.975Z+ dd if=/dev/zero of=/scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev bs=1 count=0 seek=20G
3892023-10-25T07:28:53.999Z0+0 records in
3902023-10-25T07:28:54.022Z0+0 records out
3912023-10-25T07:28:54.046Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3922023-10-25T07:28:54.074Z+ success 'ZFS vdev /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev exists'
3932023-10-25T07:28:54.101Z+ set +x
3942023-10-25T07:28:54.127Z++ zpool list -o name
3952023-10-25T07:28:54.153Z++ grep oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930
3962023-10-25T07:28:54.179Z+ [[ -z '' ]]
3972023-10-25T07:28:54.207Z+ zpool create -f oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev
3982023-10-25T07:28:54.233Z+ success 'ZFS zpool oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 exists'
3992023-10-25T07:28:54.259Z+ set +x
4002023-10-25T07:28:54.283Z+ for ZPOOL in "${ZPOOLS[@]}"
4012023-10-25T07:28:54.308Z+ echo 'Zpool: [oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60]'
4022023-10-25T07:28:54.343Z+ VDEV_PATH=/scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev
4032023-10-25T07:28:54.440Z+ [[ -f /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev ]]
4042023-10-25T07:28:54.470Z+ dd if=/dev/zero of=/scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev bs=1 count=0 seek=20G
4052023-10-25T07:28:54.497Z0+0 records in
4062023-10-25T07:28:54.521Z0+0 records out
4072023-10-25T07:28:54.546Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4082023-10-25T07:28:54.573Z+ success 'ZFS vdev /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev exists'
4092023-10-25T07:28:54.599Z+ set +x
4102023-10-25T07:28:54.627Z++ zpool list -o name
4112023-10-25T07:28:54.656Z++ grep oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60
4122023-10-25T07:28:54.684Z+ [[ -z '' ]]
4132023-10-25T07:28:54.711Z+ zpool create -f oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev
4142023-10-25T07:28:54.740Z+ success 'ZFS zpool oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 exists'
4152023-10-25T07:28:54.774Z+ set +x
4162023-10-25T07:28:54.804Z+ for ZPOOL in "${ZPOOLS[@]}"
4172023-10-25T07:28:54.836Z+ echo 'Zpool: [oxp_31bd71cd-4736-4a12-a387-9b74b050396f]'
4182023-10-25T07:28:54.867Z+ VDEV_PATH=/scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev
4192023-10-25T07:28:54.902Z+ [[ -f /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev ]]
4202023-10-25T07:28:54.935Z+ dd if=/dev/zero of=/scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev bs=1 count=0 seek=20G
4212023-10-25T07:28:54.962Z0+0 records in
4222023-10-25T07:28:54.986Z0+0 records out
4232023-10-25T07:28:55.010Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4242023-10-25T07:28:55.036Z+ success 'ZFS vdev /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev exists'
4252023-10-25T07:28:55.059Z+ set +x
4262023-10-25T07:28:55.085Z++ zpool list -o name
4272023-10-25T07:28:55.108Z++ grep oxp_31bd71cd-4736-4a12-a387-9b74b050396f
4282023-10-25T07:28:55.132Z+ [[ -z '' ]]
4292023-10-25T07:28:55.155Z+ zpool create -f oxp_31bd71cd-4736-4a12-a387-9b74b050396f /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev
4302023-10-25T07:28:55.179Z+ success 'ZFS zpool oxp_31bd71cd-4736-4a12-a387-9b74b050396f exists'
4312023-10-25T07:28:55.202Z+ set +x
4322023-10-25T07:28:55.229Z+ for ZPOOL in "${ZPOOLS[@]}"
4332023-10-25T07:28:55.255Z+ echo 'Zpool: [oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7]'
4342023-10-25T07:28:55.279Z+ VDEV_PATH=/scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev
4352023-10-25T07:28:55.312Z+ [[ -f /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev ]]
4362023-10-25T07:28:55.343Z+ dd if=/dev/zero of=/scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev bs=1 count=0 seek=20G
4372023-10-25T07:28:55.375Z0+0 records in
4382023-10-25T07:28:55.400Z0+0 records out
4392023-10-25T07:28:55.424Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4402023-10-25T07:28:55.456Z+ success 'ZFS vdev /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev exists'
4412023-10-25T07:28:55.480Z+ set +x
4422023-10-25T07:28:55.503Z++ zpool list -o name
4432023-10-25T07:28:55.527Zdownloading softnpu ...done
4442023-10-25T07:28:55.552Z++ grep oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7
4452023-10-25T07:28:55.575Z+ [[ -z '' ]]
4462023-10-25T07:28:55.599Z+ zpool create -f oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev
4472023-10-25T07:28:55.622Z+ success 'ZFS zpool oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 exists'
4482023-10-25T07:28:55.646Z+ set +x
4492023-10-25T07:28:55.669Z+ for ZPOOL_TYPE in "${ZPOOL_TYPES[@]}"
4502023-10-25T07:28:55.694Z+ readarray -t ZPOOLS
4512023-10-25T07:28:55.718Z++ grep '"oxi_' /opt/oxide/work/tools/../smf/sled-agent/non-gimlet/config.toml
4522023-10-25T07:28:55.748Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/sidecar-lite/release/f397c5da66b2b54b0a288d89f0afaee00eb9a576/libsidecar_lite.so.sha256.txt
4532023-10-25T07:28:55.786Z++ sed 's/[ ",]//g'
4542023-10-25T07:28:55.825Z+ for ZPOOL in "${ZPOOLS[@]}"
4552023-10-25T07:28:55.862Z+ echo 'Zpool: [oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b]'
4562023-10-25T07:28:55.895Z+ VDEV_PATH=/scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4572023-10-25T07:28:55.927Z+ [[ -f /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
4582023-10-25T07:28:55.953Z+ dd if=/dev/zero of=/scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
4592023-10-25T07:28:55.980Z0+0 records in
4602023-10-25T07:28:56.005Z0+0 records out
4612023-10-25T07:28:56.031Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
4622023-10-25T07:28:56.058Z+ success 'ZFS vdev /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
4632023-10-25T07:28:56.083Z+ set +x
4642023-10-25T07:28:56.109Z++ zpool list -o name
4652023-10-25T07:28:56.138Z++ grep oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b
4662023-10-25T07:28:56.169Z+ [[ -z '' ]]
4672023-10-25T07:28:56.195Z+ zpool create -f oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4682023-10-25T07:28:56.222Z+ success 'ZFS zpool oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
4692023-10-25T07:28:56.249Z+ set +x
4702023-10-25T07:28:56.282Z+ for ZPOOL in "${ZPOOLS[@]}"
4712023-10-25T07:28:56.320Z+ echo 'Zpool: [oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b]'
4722023-10-25T07:28:56.361Z+ VDEV_PATH=/scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4732023-10-25T07:28:56.402Z+ [[ -f /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
4742023-10-25T07:28:56.430Z+ dd if=/dev/zero of=/scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
4752023-10-25T07:28:56.503Z0+0 records in
4762023-10-25T07:28:56.526Z0+0 records out
4772023-10-25T07:28:56.550Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
4782023-10-25T07:28:56.575Z+ success 'ZFS vdev /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
4792023-10-25T07:28:57.041Z+ set +x
4802023-10-25T07:28:57.065Z++ zpool list -o name
4812023-10-25T07:28:57.089Z++ grep oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b
4822023-10-25T07:28:57.112Z+ [[ -z '' ]]
4832023-10-25T07:28:57.136Z+ zpool create -f oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4842023-10-25T07:28:57.166Z+ success 'ZFS zpool oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
4852023-10-25T07:28:57.191Z+ set +x
4862023-10-25T07:28:57.214Z+ ensure_simulated_links igb0
4872023-10-25T07:28:57.239Z+ local PHYSICAL_LINK=igb0
4882023-10-25T07:28:57.263Z+ INDICES=("0" "1")
4892023-10-25T07:28:57.291Z+ for I in "${INDICES[@]}"
4902023-10-25T07:28:57.318Z++ get_simnet_name_if_exists net0
4912023-10-25T07:28:57.346Z++ dladm show-simnet -p -o LINK net0
4922023-10-25T07:28:57.375Z++ echo ''
4932023-10-25T07:28:57.404Z+ [[ -z '' ]]
4942023-10-25T07:28:57.437Z+ dladm create-simnet -t net0
4952023-10-25T07:28:57.466Z+ dladm create-simnet -t sc0_0
4962023-10-25T07:28:57.499Z+ dladm modify-simnet -t -p net0 sc0_0
4972023-10-25T07:28:57.555Z+ dladm set-linkprop -p mtu=9000 sc0_0
4982023-10-25T07:28:57.585Z+ success 'Simnet net0/sc0_0 exists'
4992023-10-25T07:28:57.616Z+ set +x
5002023-10-25T07:28:57.646Z+ for I in "${INDICES[@]}"
5012023-10-25T07:28:57.684Z++ get_simnet_name_if_exists net1
5022023-10-25T07:28:57.717Z++ dladm show-simnet -p -o LINK net1
5032023-10-25T07:28:57.743Z++ echo ''
5042023-10-25T07:28:57.774Z+ [[ -z '' ]]
5052023-10-25T07:28:57.805Z+ dladm create-simnet -t net1
5062023-10-25T07:28:57.833Z+ dladm create-simnet -t sc1_0
5072023-10-25T07:28:57.861Z+ dladm modify-simnet -t -p net1 sc1_0
5082023-10-25T07:28:57.897Z+ dladm set-linkprop -p mtu=9000 sc1_0
5092023-10-25T07:28:57.926Z+ success 'Simnet net1/sc1_0 exists'
5102023-10-25T07:28:57.967Z+ set +x
5112023-10-25T07:28:58.054Z++ get_vnic_name_if_exists sc0_1
5122023-10-25T07:28:58.084Z++ dladm show-vnic -p -o LINK sc0_1
5132023-10-25T07:28:58.115Z++ echo ''
5142023-10-25T07:28:58.145Z+ [[ -z '' ]]
5152023-10-25T07:28:58.174Z+ dladm create-vnic -t sc0_1 -l igb0 -m a8:e1:de:01:70:1d
5162023-10-25T07:28:58.203Z+ [[ -v PROMISC_FILT_OFF ]]
5172023-10-25T07:28:58.239Z+ success 'Vnic sc0_1 exists'
5182023-10-25T07:28:58.265Z+ set +x
5192023-10-25T07:28:58.293Z+ warn_if_no_proxy_arp
5202023-10-25T07:28:58.325Z+ [[ -v PXA_START ]]
5212023-10-25T07:28:58.354Z+ [[ -v PXA_END ]]
5222023-10-25T07:28:58.384Z+ ensure_softnpu_zone
5232023-10-25T07:28:58.410Z+ zoneadm list
5242023-10-25T07:28:58.438Z+ grep -q sidecar_softnpu
5252023-10-25T07:28:58.463Z+ [[ -f out/npuzone/npuzone ]]
5262023-10-25T07:28:58.490Z+ out/npuzone/npuzone create sidecar --omicron-zone --ports sc0_0,tfportrear0_0 --ports sc0_1,tfportqsfp0_0 --sidecar-lite-branch omicron-tracking
5272023-10-25T07:28:59.987Zdownloading asic_program.so ...done
5282023-10-25T07:29:00.033Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/sidecar-lite/release/f397c5da66b2b54b0a288d89f0afaee00eb9a576/scadm.sha256.txt
5292023-10-25T07:29:09.070Zdownloading scadm ...done
5302023-10-25T07:29:09.099Zcreating port tfportrear0_0 for sc0_0
5312023-10-25T07:29:09.124Zcreating port tfportqsfp0_0 for sc0_1
5322023-10-25T07:29:09.149Zsoftnpu zone setup
5332023-10-25T07:29:09.173Zconfigure zone
5342023-10-25T07:29:09.196Zinstall zone
5352023-10-25T07:29:10.140Zboot zone
5362023-10-25T07:29:13.248Z[sidecar_softnpu] svcs milestone/network
5372023-10-25T07:29:13.275ZSTATE STIME FMRI
5382023-10-25T07:29:13.299Zonline 7:29:11 svc:/milestone/network:default
5392023-10-25T07:29:13.334Z[sidecar_softnpu] /softnpu/npu start
5402023-10-25T07:29:13.363Z
5412023-10-25T07:29:13.391Z+ /opt/oxide/work/tools/scrimlet/softnpu-init.sh
5422023-10-25T07:29:13.417Z+ GATEWAY_IP=10.151.2.100
5432023-10-25T07:29:13.452Z+ echo 'Using 10.151.2.100 as gateway ip'
5442023-10-25T07:29:13.477ZUsing 10.151.2.100 as gateway ip
5452023-10-25T07:29:13.500Z+ [[ ! -v GATEWAY_MAC ]]
5462023-10-25T07:29:13.935Z+ GATEWAY_MAC=18:c0:4d:0d:a0:2a
5472023-10-25T07:29:13.964Z++ grep -c -E '^([0-9a-fA-F]{2}:){5}[0-9a-fA-F]{2}$' /dev/fd/63
5482023-10-25T07:29:13.996Z+++ echo 18:c0:4d:0d:a0:2a
5492023-10-25T07:29:14.021Z+ COUNT=1
5502023-10-25T07:29:14.044Z+ [[ 1 -ne 1 ]]
5512023-10-25T07:29:14.068Z+ echo 'Using 18:c0:4d:0d:a0:2a as gateway mac'
5522023-10-25T07:29:14.135ZUsing 18:c0:4d:0d:a0:2a as gateway mac
5532023-10-25T07:29:14.168Z+ z_scadm add-arp-entry 10.151.2.100 18:c0:4d:0d:a0:2a
5542023-10-25T07:29:14.203Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone add-arp-entry 10.151.2.100 18:c0:4d:0d:a0:2a
5552023-10-25T07:29:14.234Z+ PXA_MAC=a8:e1:de:01:70:1d
5562023-10-25T07:29:14.265Z+ [[ -v PXA_START ]]
5572023-10-25T07:29:14.293Z+ z_scadm add-proxy-arp 10.151.2.164 10.151.2.227 a8:e1:de:01:70:1d
5582023-10-25T07:29:14.321Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone add-proxy-arp 10.151.2.164 10.151.2.227 a8:e1:de:01:70:1d
5592023-10-25T07:29:14.355Z+ z_scadm dump-state
5602023-10-25T07:29:14.401Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone dump-state
5612023-10-25T07:29:14.434Zlocal v6:
5622023-10-25T07:29:14.876Zlocal v4:
5632023-10-25T07:29:14.904Zrouter v6:
5642023-10-25T07:29:14.929Zrouter v4:
5652023-10-25T07:29:14.952Zresolver v4:
5662023-10-25T07:29:14.976Z10.151.2.100 -> 18:c0:4d:0d:a0:2a
5672023-10-25T07:29:15.006Zresolver v6:
5682023-10-25T07:29:15.029Znat v4:
5692023-10-25T07:29:15.053Znat_v6:
5702023-10-25T07:29:15.077Zport_mac:
5712023-10-25T07:29:15.101Zproxy arp:
5722023-10-25T07:29:15.125Z10.151.2.164/10.151.2.227: a8:e1:de:01:70:1d
5732023-10-25T07:29:15.160Z+ success 'softnpu zone exists'
5742023-10-25T07:29:15.184Z+ set +x
5752023-10-25T07:29:15.208Zsoftnpu zone exists
5762023-10-25T07:29:15.231Z
5772023-10-25T07:29:15.255Zreal 26.569802034
5782023-10-25T07:29:15.286Zuser 0.726416715
5792023-10-25T07:29:15.319Zsys 1.609806801
5802023-10-25T07:29:15.347Ztrap 0.000079246
5812023-10-25T07:29:15.375Ztflt 0.000171303
5822023-10-25T07:29:15.400Zdflt 0.000852247
5832023-10-25T07:29:15.426Zkflt 0.000000000
5842023-10-25T07:29:15.457Zlock 20:46.613283121
5852023-10-25T07:29:15.484Zslp 1:18.889930820
5862023-10-25T07:29:15.512Zlat 0.079162599
5872023-10-25T07:29:15.545Zstop 0.360996655
5882023-10-25T07:29:15.577Z+ tar xf out/omicron-sled-agent.tar pkg/config-rss.toml
5892023-10-25T07:29:15.609Z++ sed -n 's/silo_name = "\(.*\)"/\1/p' pkg/config-rss.toml
5902023-10-25T07:29:16.040Z+ SILO_NAME=recovery
5912023-10-25T07:29:16.067Z++ sed -n 's/external_dns_zone_name = "\(.*\)"/\1/p' pkg/config-rss.toml
5922023-10-25T07:29:16.095Z+ EXTERNAL_DNS_DOMAIN=oxide.test
5932023-10-25T07:29:16.123Z+ sed -i~ '
5942023-10-25T07:29:16.192Z /^external_dns_ips/c\
5952023-10-25T07:29:16.217Zexternal_dns_ips = [ "10.151.2.164", "10.151.2.165" ]
5962023-10-25T07:29:16.240Z /^\[\[internal_services_ip_pool_ranges/,/^$/ {
5972023-10-25T07:29:16.264Z /^first/c\
5982023-10-25T07:29:16.298Zfirst = "10.151.2.164"
5992023-10-25T07:29:16.328Z /^last/c\
6002023-10-25T07:29:16.351Zlast = "10.151.2.173"
6012023-10-25T07:29:16.375Z }
6022023-10-25T07:29:16.398Z /^\[rack_network_config/,/^$/ {
6032023-10-25T07:29:16.422Z /^infra_ip_first/c\
6042023-10-25T07:29:16.445Zinfra_ip_first = "10.151.2.174"
6052023-10-25T07:29:16.469Z /^infra_ip_last/c\
6062023-10-25T07:29:16.495Zinfra_ip_last = "10.151.2.174"
6072023-10-25T07:29:16.523Z }
6082023-10-25T07:29:16.555Z /^\[\[rack_network_config.ports/,/^$/ {
6092023-10-25T07:29:16.581Z--- pkg/config-rss.toml~ Mon Jul 24 01:21:28 2006
6102023-10-25T07:29:16.605Z /^routes/c\
6112023-10-25T07:29:16.629Z+++ pkg/config-rss.toml Wed Oct 25 07:29:13 2023
6122023-10-25T07:29:16.662Zroutes = \[{nexthop = "10.151.2.100", destination = "0.0.0.0/0"}\]
6132023-10-25T07:29:16.687Z@@ -41,7 +41,7 @@
6142023-10-25T07:29:16.710Z /^addresses/c\
6152023-10-25T07:29:16.734Z # the DNS domain delegated to the rack by the customer. Each of these addresses
6162023-10-25T07:29:16.757Zaddresses = \["10.151.2.174/32"\]
6172023-10-25T07:29:16.781Z # must be contained in one of the "internal services" IP Pool ranges listed
6182023-10-25T07:29:16.805Z }
6192023-10-25T07:29:16.828Z # below.
6202023-10-25T07:29:16.851Z' pkg/config-rss.toml
6212023-10-25T07:29:16.875Z-external_dns_ips = [ "192.168.1.20", "192.168.1.21" ]
6222023-10-25T07:29:16.899Z+ diff -u pkg/config-rss.toml~ pkg/config-rss.toml
6232023-10-25T07:29:16.922Z+external_dns_ips = [ "10.151.2.164", "10.151.2.165" ]
6242023-10-25T07:29:16.946Z+ true
6252023-10-25T07:29:16.976Z
6262023-10-25T07:29:17.003Z+ tar rvf out/omicron-sled-agent.tar pkg/config-rss.toml
6272023-10-25T07:29:17.038Z # Initial TLS certificates for the external API
6282023-10-25T07:29:17.063Z+ rm -f pkg/config-rss.toml pkg/config-rss.toml~
6292023-10-25T07:29:17.087Z #
6302023-10-25T07:29:17.117Z+ TLS_NAME=recovery.sys.oxide.test
6312023-10-25T07:29:17.144Z@@ -77,8 +77,8 @@
6322023-10-25T07:29:17.171Z+ openssl req -newkey rsa:4096 -x509 -sha256 -days 3 -nodes -out pkg/initial-tls-cert.pem -keyout pkg/initial-tls-key.pem -subj /CN=recovery.sys.oxide.test -addext subjectAltName=DNS:recovery.sys.oxide.test -addext basicConstraints=critical,CA:FALSE -config /dev/stdin
6332023-10-25T07:29:17.217Z #
6342023-10-25T07:29:17.241Z..+...........+...+.......+...+..+...+.+......+.........+......+.....+.+..+................+.........+........+...+.+.....+.+............+......+..............+.......+...+......+..+..........+...+.....+.......+..+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*....+..+.............+..+...+....+...+...+.....+....+...........+...+......+.+.........+.....+....+.....+....+...........+....+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*........+.........+....+.....+.......+........+.+............+..+.+......+.....+......+...+.+.....+....+.....+......+....+..+.........+.....................+...+.+...+...+.........+.....+.............+..+.+..+...+.............+...............+..+....+.........+......+......+...+.....+....+..+...+.......+......+...+.....+............+....+.........+.........+...+...+.................+...+.+...+...........+......+...+.......+.....+.........+.........+.............+..............+..........+........+.+.....+................+......+...+..............+.+..+....+...+..+.............+...+..............+...+...............+.+.....+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
6352023-10-25T07:29:17.265Z # For more on this and what to put here, see docs/how-to-run.adoc.
6362023-10-25T07:29:17.288Z [[internal_services_ip_pool_ranges]]
6372023-10-25T07:29:17.312Z-first = "192.168.1.20"
6382023-10-25T07:29:17.336Z-last = "192.168.1.29"
6392023-10-25T07:29:17.360Z+first = "10.151.2.164"
6402023-10-25T07:29:17.386Z+last = "10.151.2.173"
6412023-10-25T07:29:17.419Z
6422023-10-25T07:29:17.451Z # TODO - this configuration is subject to change going forward. Ultimately these
6432023-10-25T07:29:17.476Z # parameters should be provided to the control plane via wicket, but we need to
6442023-10-25T07:29:17.500Z@@ -92,8 +92,8 @@
6452023-10-25T07:29:17.523Z.............+......+...+...+.......+...+........+.........+.+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.........+......+......+............+..+.+..............+.........+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*......+...........+.......................................+........................+......+.+.........+.....+.........+.+...+...........+....+..+.+............+...+..+..........+.....+.+............+..+..........+.....................+............+.........+.....+....+............+...+...........+....+.....................+...+...+........+......+...+..........+...+............+..+......+...+......+.+.....+.........+....+.....+......................+...+.....+.........+...........................+......+.......+.....+.+.........+........+....+..............+.+........+.+......+.........+..+............+..........+..+...............+..........+..................+.....+...+..................+.......+...+.....+......+...+.......+......+........+.......+.........+...+......+.....+.+..............+..........+.....+......+....+.....+......+......+.........+....+..............+.................................+..........+...+.....+..........+..+...................+...+..+.+.....+......................+........+....+..................+.....+.+..............+.+.....+.......+.........+...........+..........+............+.....+.+......+......+........+........................+.+............+.....+....+.....+....+.................+............+.+.....+.+......+............+.....+.....................+....+.....+..........+...+.....+.......+...+..............+..............................+.+.......................+....+..+......+.......+......+.........+...+..+..........+...........+..........+..+...+......................+..+...+.......+.....+......+......................+..+..........+..+..................+............+.+...+.....+.+......+.....+.......+........+.......+..+.+.....+..........+....................+...+....+...+..............+...+......+............+....+.................................+.....+.+......+.....+.........+.......+..+.+......+...+.....................+.................+...+...............+......+.+...+.....................+...+........+..................+.+........+.+..+.........+...+.......+...........+.......+...+..+.+..+.........+.........+.+...+...........................+......+..+...............+..........+......+.....+......+...+.+..............+..................+............+............+...+.......+...+..+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
6462023-10-25T07:29:17.548Z # A range of IP addresses used by Boundary Services on the external network. In
6472023-10-25T07:29:17.580Z-----
6482023-10-25T07:29:17.609Z # a real system, these would be addresses of the uplink ports on the Sidecar.
6492023-10-25T07:29:17.639Z+ tar rvf out/omicron-sled-agent.tar pkg/initial-tls-cert.pem pkg/initial-tls-key.pem
6502023-10-25T07:29:17.667Z # With softnpu, only one address is used.
6512023-10-25T07:29:17.693Z+ rm -f pkg/initial-tls-cert.pem pkg/initial-tls-key.pem
6522023-10-25T07:29:17.716Z-infra_ip_first = "192.168.1.30"
6532023-10-25T07:29:17.740Z+ rmdir pkg
6542023-10-25T07:29:17.765Z-infra_ip_last = "192.168.1.30"
6552023-10-25T07:29:17.789Z+ E2E_TLS_CERT=/opt/oxide/sled-agent/pkg/initial-tls-cert.pem
6562023-10-25T07:29:17.839Z+infra_ip_first = "10.151.2.174"
6572023-10-25T07:29:17.871Z+ pfexec mkdir -p /usr/oxide
6582023-10-25T07:29:17.897Z+infra_ip_last = "10.151.2.174"
6592023-10-25T07:29:17.922Z+ pfexec rm -f /usr/oxide/tcpproxy
6602023-10-25T07:29:17.945Z
6612023-10-25T07:29:17.969Z+ pfexec curl -sSfL -o /usr/oxide/tcpproxy http://catacomb.eng.oxide.computer:12346/tcpproxy
6622023-10-25T07:29:17.992Z # Configurations for BGP routers to run on the scrimlets.
6632023-10-25T07:29:18.016Z+ pfexec chmod +x /usr/oxide/tcpproxy
6642023-10-25T07:29:18.039Z bgp = []
6652023-10-25T07:29:18.063Z+ pfexec rm -f /var/svc/manifest/site/tcpproxy.xml
6662023-10-25T07:29:18.086Z@@ -101,9 +101,9 @@
6672023-10-25T07:29:18.110Z+ pfexec curl -sSfL -o /var/svc/manifest/site/tcpproxy.xml http://catacomb.eng.oxide.computer:12346/tcpproxy.xml
6682023-10-25T07:29:18.134Z # You can configure multiple uplinks by repeating the following stanza
6692023-10-25T07:29:18.157Z+ pfexec svccfg import /var/svc/manifest/site/tcpproxy.xml
6702023-10-25T07:29:18.184Z [[rack_network_config.ports]]
6712023-10-25T07:29:18.208Z+ OMICRON_NO_UNINSTALL=1
6722023-10-25T07:29:18.250Z # Routes associated with this port.
6732023-10-25T07:29:18.277Z+ ptime -m pfexec ./target/release/omicron-package -t test install
6742023-10-25T07:29:18.307Z-routes = [{nexthop = "192.168.1.199", destination = "0.0.0.0/0"}]
6752023-10-25T07:29:18.335ZOct 25 07:29:15.921 DEBG target[test]: Target({"image": "standard", "machine": "non-gimlet", "rack-topology": "single-sled", "switch": "softnpu"})
6762023-10-25T07:29:18.358Z+routes = [{nexthop = "10.151.2.100", destination = "0.0.0.0/0"}]
6772023-10-25T07:29:18.385ZOct 25 07:29:15.923 INFO Installing service, dst: /opt/oxide/clickhouse.tar.gz, src: out/clickhouse.tar.gz
6782023-10-25T07:29:18.419Z # Addresses associated with this port.
6792023-10-25T07:29:18.446ZOct 25 07:29:15.923 INFO Installing service, dst: /opt/oxide/ntp.tar.gz, src: out/ntp.tar.gz
6802023-10-25T07:29:18.473Z-addresses = ["192.168.1.30/32"]
6812023-10-25T07:29:18.497ZOct 25 07:29:15.923 INFO Installing service, dst: /opt/oxide/crucible_pantry.tar.gz, src: out/crucible-pantry.tar.gz
6822023-10-25T07:29:18.520Z+addresses = ["10.151.2.174/32"]
6832023-10-25T07:29:18.544ZOct 25 07:29:15.923 INFO Installing service, dst: /opt/oxide/mg-ddm.tar, src: out/mg-ddm-gz.tar
6842023-10-25T07:29:18.568Z # Name of the uplink port. This should always be "qsfp0" when using softnpu.
6852023-10-25T07:29:18.593ZOct 25 07:29:15.923 INFO Installing service, dst: /opt/oxide/external_dns.tar.gz, src: out/external-dns.tar.gz
6862023-10-25T07:29:18.623Z port = "qsfp0"
6872023-10-25T07:29:18.650ZOct 25 07:29:15.923 INFO Installing service, dst: /opt/oxide/internal_dns.tar.gz, src: out/internal-dns.tar.gz
6882023-10-25T07:29:18.673Z # The speed of this port.
6892023-10-25T07:29:18.697ZOct 25 07:29:15.923 INFO Installing service, dst: /opt/oxide/switch.tar.gz, src: out/switch-softnpu.tar.gz
6902023-10-25T07:29:18.726Za pkg/config-rss.toml 7K
6912023-10-25T07:29:18.757ZOct 25 07:29:15.923 INFO Installing service, dst: /opt/oxide/clickhouse_keeper.tar.gz, src: out/clickhouse_keeper.tar.gz
6922023-10-25T07:29:18.788Za pkg/initial-tls-cert.pem 2K
6932023-10-25T07:29:18.826ZOct 25 07:29:15.923 INFO Installing service, dst: /opt/oxide/overlay.tar.gz, src: out/overlay.tar.gz
6942023-10-25T07:29:18.860Za pkg/initial-tls-key.pem 4K
6952023-10-25T07:29:18.895ZOct 25 07:29:15.923 INFO Installing service, dst: /opt/oxide/mgs.tar.gz, src: out/omicron-gateway-softnpu.tar.gz
6962023-10-25T07:29:18.924ZOct 25 07:29:15.923 INFO Installing service, dst: /opt/oxide/cockroachdb.tar.gz, src: out/cockroachdb.tar.gz
6972023-10-25T07:29:18.951ZOct 25 07:29:15.923 INFO Installing service, dst: /opt/oxide/oximeter.tar.gz, src: out/oximeter-collector.tar.gz
6982023-10-25T07:29:18.985ZOct 25 07:29:15.923 INFO Installing service, dst: /opt/oxide/crucible.tar.gz, src: out/crucible.tar.gz
6992023-10-25T07:29:19.013ZOct 25 07:29:15.923 INFO Installing service, dst: /opt/oxide/propolis-server.tar.gz, src: out/propolis-server.tar.gz
7002023-10-25T07:29:19.048ZOct 25 07:29:15.923 INFO Installing service, dst: /opt/oxide/nexus.tar.gz, src: out/omicron-nexus.tar.gz
7012023-10-25T07:29:19.494ZOct 25 07:29:15.923 INFO Installing service, dst: /opt/oxide/sled-agent.tar, src: out/omicron-sled-agent.tar
7022023-10-25T07:29:19.518ZOct 25 07:29:16.142 INFO Unpacking service tarball, service_path: /opt/oxide/mg-ddm, tar_path: /opt/oxide/mg-ddm.tar
7032023-10-25T07:29:19.544ZOct 25 07:29:16.178 INFO Unpacking service tarball, service_path: /opt/oxide/sled-agent, tar_path: /opt/oxide/sled-agent.tar
7042023-10-25T07:29:19.576ZOct 25 07:29:16.238 INFO Installing bootstrap service from /opt/oxide/sled-agent/pkg/manifest.xml
7052023-10-25T07:29:19.605Z
7062023-10-25T07:29:19.631Zreal 0.458031389
7072023-10-25T07:29:19.655Zuser 0.096489520
7082023-10-25T07:29:19.679Zsys 1.752696731
7092023-10-25T07:29:19.703Ztrap 0.000008440
7102023-10-25T07:29:19.727Ztflt 0.000013019
7112023-10-25T07:29:20.198Zdflt 0.000354265
7122023-10-25T07:29:20.224Zkflt 0.000002640
7132023-10-25T07:29:20.249Zlock 41.172937356
7142023-10-25T07:29:20.273Zslp 0.673090521
7152023-10-25T07:29:20.297Zlat 0.002936480
7162023-10-25T07:29:20.323Zstop 0.225458969
7172023-10-25T07:29:20.347Z+ retry=0
7182023-10-25T07:29:20.371Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7192023-10-25T07:29:20.804Z+ [[ 0 -gt 30 ]]
7202023-10-25T07:29:20.832Z+ sleep 1
7212023-10-25T07:29:20.857Z+ retry=1
7222023-10-25T07:29:21.730Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7232023-10-25T07:29:21.758Z+ [[ 1 -gt 30 ]]
7242023-10-25T07:29:21.786Z+ sleep 1
7252023-10-25T07:29:21.815Z+ retry=2
7262023-10-25T07:29:21.845Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7272023-10-25T07:29:43.415Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone dump-state
7282023-10-25T07:29:43.441Zlocal v6:
7292023-10-25T07:29:43.465Zfe80::288a:fff:fe5a:3ee9
7302023-10-25T07:29:43.489Zfd00:99::1
7312023-10-25T07:29:43.512Zlocal v4:
7322023-10-25T07:29:43.536Z10.151.2.174
7332023-10-25T07:29:43.559Zrouter v6:
7342023-10-25T07:29:43.583Zfdb0:18c0:4d0d:a02a::/64 -> fe80::3c62:7bff:fee4:f46d (1) vid=0
7352023-10-25T07:29:43.606Zfd00:1122:3344:101::/64 -> fe80::3c62:7bff:fee4:f46d (1) vid=0
7362023-10-25T07:29:43.630Zrouter v4:
7372023-10-25T07:29:43.653Z0.0.0.0/0 -> 10.151.2.100 (2) vid=0
7382023-10-25T07:29:43.677Zresolver v4:
7392023-10-25T07:29:43.702Z10.151.2.174 -> ba:b4:b3:60:f2:59
7402023-10-25T07:29:43.731Z10.151.2.100 -> 18:c0:4d:0d:a0:2a
7412023-10-25T07:29:43.757Zresolver v6:
7422023-10-25T07:29:43.781Zfe80::3c62:7bff:fee4:f46d -> 3e:62:7b:e4:f4:6d
7432023-10-25T07:29:43.805Zfe80::288a:fff:fe5a:3ee9 -> 2a:8a:0f:5a:3e:e9
7442023-10-25T07:29:43.864Znat v4:
7452023-10-25T07:29:43.888Znat_v6:
7462023-10-25T07:29:43.912Zport_mac:
7472023-10-25T07:29:43.936Z1000: a8:40:25:31:05:31
7482023-10-25T07:29:43.959Z1: a8:40:25:31:05:32
7492023-10-25T07:29:43.987Zproxy arp:
7502023-10-25T07:29:44.011Z10.151.2.164/10.151.2.227: a8:e1:de:01:70:1d
7512023-10-25T07:29:44.036Z+ retry=0
7522023-10-25T07:29:44.060Z+++ zoneadm list -n
7532023-10-25T07:29:44.084Z+++ grep oxz_ntp
7542023-10-25T07:29:44.107Z++ pfexec svcs -z -Hostate oxide/ntp
7552023-10-25T07:29:44.137Zsvcs: zone '-Hostate': entity not found
7562023-10-25T07:29:44.164Z++ true
7572023-10-25T07:29:44.192Z+ [[ '' != online ]]
7582023-10-25T07:29:44.218Z+ [[ 0 -gt 60 ]]
7592023-10-25T07:29:44.242Z+ sleep 1
7602023-10-25T07:29:44.465Z+ retry=1
7612023-10-25T07:29:44.489Z+++ zoneadm list -n
7622023-10-25T07:29:44.513Z+++ grep oxz_ntp
7632023-10-25T07:29:44.537Z++ pfexec svcs -z -Hostate oxide/ntp
7642023-10-25T07:29:44.561Zsvcs: zone '-Hostate': entity not found
7652023-10-25T07:29:44.585Z++ true
7662023-10-25T07:29:45.026Z+ [[ '' != online ]]
7672023-10-25T07:29:45.053Z+ [[ 1 -gt 60 ]]
7682023-10-25T07:29:45.082Z+ sleep 1
7692023-10-25T07:29:45.490Z+ retry=2
7702023-10-25T07:29:45.515Z+++ zoneadm list -n
7712023-10-25T07:29:45.543Z+++ grep oxz_ntp
7722023-10-25T07:29:45.570Z++ pfexec svcs -z -Hostate oxide/ntp
7732023-10-25T07:29:45.594Zsvcs: zone '-Hostate': entity not found
7742023-10-25T07:29:45.618Z++ true
7752023-10-25T07:29:45.642Z+ [[ '' != online ]]
7762023-10-25T07:29:45.665Z+ [[ 2 -gt 60 ]]
7772023-10-25T07:29:45.692Z+ sleep 1
7782023-10-25T07:29:46.508Z+ retry=3
7792023-10-25T07:29:46.535Z+++ zoneadm list -n
7802023-10-25T07:29:46.560Z+++ grep oxz_ntp
7812023-10-25T07:29:46.583Z++ pfexec svcs -z -Hostate oxide/ntp
7822023-10-25T07:29:46.607Zsvcs: zone '-Hostate': entity not found
7832023-10-25T07:29:46.630Z++ true
7842023-10-25T07:29:46.654Z+ [[ '' != online ]]
7852023-10-25T07:29:46.677Z+ [[ 3 -gt 60 ]]
7862023-10-25T07:29:46.700Z+ sleep 1
7872023-10-25T07:29:47.530Z+ retry=4
7882023-10-25T07:29:47.554Z+++ zoneadm list -n
7892023-10-25T07:29:47.579Z+++ grep oxz_ntp
7902023-10-25T07:29:47.604Z++ pfexec svcs -z -Hostate oxide/ntp
7912023-10-25T07:29:47.630Zsvcs: zone '-Hostate': entity not found
7922023-10-25T07:29:47.654Z++ true
7932023-10-25T07:29:47.678Z+ [[ '' != online ]]
7942023-10-25T07:29:47.701Z+ [[ 4 -gt 60 ]]
7952023-10-25T07:29:47.725Z+ sleep 1
7962023-10-25T07:29:48.552Z+ retry=5
7972023-10-25T07:29:48.576Z+++ zoneadm list -n
7982023-10-25T07:29:48.600Z+++ grep oxz_ntp
7992023-10-25T07:29:48.623Z++ pfexec svcs -z -Hostate oxide/ntp
8002023-10-25T07:29:48.648Zsvcs: zone '-Hostate': entity not found
8012023-10-25T07:29:48.676Z++ true
8022023-10-25T07:29:48.703Z+ [[ '' != online ]]
8032023-10-25T07:29:48.730Z+ [[ 5 -gt 60 ]]
8042023-10-25T07:29:48.756Z+ sleep 1
8052023-10-25T07:29:49.575Z+ retry=6
8062023-10-25T07:29:50.049Z+++ zoneadm list -n
8072023-10-25T07:29:50.082Z+++ grep oxz_ntp
8082023-10-25T07:29:50.115Z++ pfexec svcs -z -Hostate oxide/ntp
8092023-10-25T07:29:50.145Zsvcs: zone '-Hostate': entity not found
8102023-10-25T07:29:50.207Z++ true
8112023-10-25T07:29:50.247Z+ [[ '' != online ]]
8122023-10-25T07:29:50.274Z+ [[ 6 -gt 60 ]]
8132023-10-25T07:29:50.298Z+ sleep 1
8142023-10-25T07:29:50.594Z+ retry=7
8152023-10-25T07:29:50.619Z+++ zoneadm list -n
8162023-10-25T07:29:50.646Z+++ grep oxz_ntp
8172023-10-25T07:29:50.682Z++ pfexec svcs -z -Hostate oxide/ntp
8182023-10-25T07:29:50.717Zsvcs: zone '-Hostate': entity not found
8192023-10-25T07:29:50.741Z++ true
8202023-10-25T07:29:50.764Z+ [[ '' != online ]]
8212023-10-25T07:29:50.788Z+ [[ 7 -gt 60 ]]
8222023-10-25T07:29:50.811Z+ sleep 1
8232023-10-25T07:29:51.616Z+ retry=8
8242023-10-25T07:29:51.643Z+++ zoneadm list -n
8252023-10-25T07:29:51.666Z+++ grep oxz_ntp
8262023-10-25T07:29:51.690Z++ pfexec svcs -z -Hostate oxide/ntp
8272023-10-25T07:29:51.713Zsvcs: zone '-Hostate': entity not found
8282023-10-25T07:29:51.737Z++ true
8292023-10-25T07:29:51.761Z+ [[ '' != online ]]
8302023-10-25T07:29:51.787Z+ [[ 8 -gt 60 ]]
8312023-10-25T07:29:51.811Z+ sleep 1
8322023-10-25T07:29:52.639Z+ retry=9
8332023-10-25T07:29:52.663Z+++ zoneadm list -n
8342023-10-25T07:29:52.690Z+++ grep oxz_ntp
8352023-10-25T07:29:52.714Z++ pfexec svcs -z -Hostate oxide/ntp
8362023-10-25T07:29:52.737Zsvcs: zone '-Hostate': entity not found
8372023-10-25T07:29:52.826Z++ true
8382023-10-25T07:29:52.857Z+ [[ '' != online ]]
8392023-10-25T07:29:52.886Z+ [[ 9 -gt 60 ]]
8402023-10-25T07:29:52.915Z+ sleep 1
8412023-10-25T07:29:53.662Z+ retry=10
8422023-10-25T07:29:53.687Z+++ zoneadm list -n
8432023-10-25T07:29:53.711Z+++ grep oxz_ntp
8442023-10-25T07:29:53.739Z++ pfexec svcs -z oxz_ntp_f155e354-4667-41a4-86df-7b7c9baf046a -Hostate oxide/ntp
8452023-10-25T07:29:53.769Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
8462023-10-25T07:29:53.797Z++ true
8472023-10-25T07:29:53.821Z+ [[ '' != online ]]
8482023-10-25T07:29:53.848Z+ [[ 10 -gt 60 ]]
8492023-10-25T07:29:53.875Z+ sleep 1
8502023-10-25T07:29:54.700Z+ retry=11
8512023-10-25T07:29:54.734Z+++ zoneadm list -n
8522023-10-25T07:29:54.765Z+++ grep oxz_ntp
8532023-10-25T07:29:54.809Z++ pfexec svcs -z oxz_ntp_f155e354-4667-41a4-86df-7b7c9baf046a -Hostate oxide/ntp
8542023-10-25T07:29:54.838Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
8552023-10-25T07:29:54.865Z++ true
8562023-10-25T07:29:54.907Z+ [[ '' != online ]]
8572023-10-25T07:29:54.943Z+ [[ 11 -gt 60 ]]
8582023-10-25T07:29:54.971Z+ sleep 1
8592023-10-25T07:29:55.725Z+ retry=12
8602023-10-25T07:29:55.749Z+++ zoneadm list -n
8612023-10-25T07:29:55.772Z+++ grep oxz_ntp
8622023-10-25T07:29:55.796Z++ pfexec svcs -z oxz_ntp_f155e354-4667-41a4-86df-7b7c9baf046a -Hostate oxide/ntp
8632023-10-25T07:29:55.820Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
8642023-10-25T07:29:55.843Z++ true
8652023-10-25T07:29:55.867Z+ [[ '' != online ]]
8662023-10-25T07:29:55.890Z+ [[ 12 -gt 60 ]]
8672023-10-25T07:29:55.914Z+ sleep 1
8682023-10-25T07:29:56.757Z+ retry=13
8692023-10-25T07:29:56.784Z+++ zoneadm list -n
8702023-10-25T07:29:56.808Z+++ grep oxz_ntp
8712023-10-25T07:29:56.832Z++ pfexec svcs -z oxz_ntp_f155e354-4667-41a4-86df-7b7c9baf046a -Hostate oxide/ntp
8722023-10-25T07:29:56.858Z+ [[ disabled != online ]]
8732023-10-25T07:29:56.883Z+ [[ 13 -gt 60 ]]
8742023-10-25T07:29:56.908Z+ sleep 1
8752023-10-25T07:29:57.778Z+ retry=14
8762023-10-25T07:29:57.811Z+++ zoneadm list -n
8772023-10-25T07:29:57.838Z+++ grep oxz_ntp
8782023-10-25T07:29:57.867Z++ pfexec svcs -z oxz_ntp_f155e354-4667-41a4-86df-7b7c9baf046a -Hostate oxide/ntp
8792023-10-25T07:29:57.894Z+ [[ disabled != online ]]
8802023-10-25T07:29:57.921Z+ [[ 14 -gt 60 ]]
8812023-10-25T07:29:57.944Z+ sleep 1
8822023-10-25T07:29:58.803Z+ retry=15
8832023-10-25T07:29:58.832Z+++ zoneadm list -n
8842023-10-25T07:29:58.938Z+++ grep oxz_ntp
8852023-10-25T07:29:58.965Z++ pfexec svcs -z oxz_ntp_f155e354-4667-41a4-86df-7b7c9baf046a -Hostate oxide/ntp
8862023-10-25T07:29:58.997Z+ [[ disabled != online ]]
8872023-10-25T07:29:59.027Z+ [[ 15 -gt 60 ]]
8882023-10-25T07:29:59.053Z+ sleep 1
8892023-10-25T07:29:59.821Z+ retry=16
8902023-10-25T07:29:59.845Z+++ zoneadm list -n
8912023-10-25T07:29:59.874Z+++ grep oxz_ntp
8922023-10-25T07:29:59.902Z++ pfexec svcs -z oxz_ntp_f155e354-4667-41a4-86df-7b7c9baf046a -Hostate oxide/ntp
8932023-10-25T07:29:59.928Z+ [[ disabled != online ]]
8942023-10-25T07:29:59.954Z+ [[ 16 -gt 60 ]]
8952023-10-25T07:29:59.991Z+ sleep 1
8962023-10-25T07:30:00.845Z+ retry=17
8972023-10-25T07:30:00.869Z+++ zoneadm list -n
8982023-10-25T07:30:00.892Z+++ grep oxz_ntp
8992023-10-25T07:30:00.916Z++ pfexec svcs -z oxz_ntp_f155e354-4667-41a4-86df-7b7c9baf046a -Hostate oxide/ntp
9002023-10-25T07:30:00.940Z+ [[ disabled != online ]]
9012023-10-25T07:30:00.963Z+ [[ 17 -gt 60 ]]
9022023-10-25T07:30:00.989Z+ sleep 1
9032023-10-25T07:30:01.866Z+ retry=18
9042023-10-25T07:30:01.892Z+++ zoneadm list -n
9052023-10-25T07:30:01.917Z+++ grep oxz_ntp
9062023-10-25T07:30:01.941Z++ pfexec svcs -z oxz_ntp_f155e354-4667-41a4-86df-7b7c9baf046a -Hostate oxide/ntp
9072023-10-25T07:30:01.965Z+ [[ disabled != online ]]
9082023-10-25T07:30:01.989Z+ [[ 18 -gt 60 ]]
9092023-10-25T07:30:02.013Z+ sleep 1
9102023-10-25T07:30:02.888Z+ retry=19
9112023-10-25T07:30:02.913Z+++ zoneadm list -n
9122023-10-25T07:30:02.937Z+++ grep oxz_ntp
9132023-10-25T07:30:02.961Z++ pfexec svcs -z oxz_ntp_f155e354-4667-41a4-86df-7b7c9baf046a -Hostate oxide/ntp
9142023-10-25T07:30:02.985Z+ [[ disabled != online ]]
9152023-10-25T07:30:03.016Z+ [[ 19 -gt 60 ]]
9162023-10-25T07:30:03.046Z+ sleep 1
 
9172023-10-25T07:30:03.351Zsock console login: Oct 25 07:30:03 sock ip: ip: joining multicasts failed (4) on vopte0 - will use link layer broadcasts for multicast
 
9182023-10-25T07:30:03.911Z+ retry=20
9192023-10-25T07:30:03.938Z+++ zoneadm list -n
9202023-10-25T07:30:03.963Z+++ grep oxz_ntp
9212023-10-25T07:30:03.992Z++ pfexec svcs -z oxz_ntp_f155e354-4667-41a4-86df-7b7c9baf046a -Hostate oxide/ntp
9222023-10-25T07:30:04.461Z+ [[ online != online ]]
9232023-10-25T07:30:04.488Z+ echo 'Waited for chrony: 20s'
9242023-10-25T07:30:04.513Z+ export RUST_BACKTRACE=1
9252023-10-25T07:30:04.537ZWaited for chrony: 20s
9262023-10-25T07:30:04.563Z+ RUST_BACKTRACE=1
9272023-10-25T07:30:04.587Z+ export E2E_TLS_CERT IPPOOL_START IPPOOL_END
9282023-10-25T07:30:04.619Z+ ./tests/bootstrap
9292023-10-25T07:30:04.643ZUsing external DNS server at 10.151.2.164:53
9302023-10-25T07:30:04.668Z2023-10-25 07:30:04.116683478 UTC: attempting to log into API
9312023-10-25T07:30:18.997Z2023-10-25 07:30:19.280084121 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9322023-10-25T07:30:20.025Z2023-10-25 07:30:20.281710763 UTC: attempting to log into API
9332023-10-25T07:30:35.032Z2023-10-25 07:30:35.314826666 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9342023-10-25T07:30:36.033Z2023-10-25 07:30:36.315044357 UTC: attempting to log into API
9352023-10-25T07:30:51.070Z2023-10-25 07:30:51.311005429 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9362023-10-25T07:30:52.073Z2023-10-25 07:30:52.309332541 UTC: attempting to log into API
9372023-10-25T07:31:07.105Z2023-10-25 07:31:07.301929354 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9382023-10-25T07:31:08.106Z2023-10-25 07:31:08.300304886 UTC: attempting to log into API
 
9392023-10-25T07:31:14.225ZOct 25 07:31:14 sok ip: ip: joining multicasts failed (4) on vopte1 - will use link layer broadcasts for multicast
9402023-10-25T07:31:20.745ZOct 25 07:31:20 sock ip: ip: joining multicasts failed (4) on vopte2 - will use link layer broadcasts for multicast
 
9412023-10-25T07:31:23.165Z2023-10-25 07:31:23.317466520 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9422023-10-25T07:31:24.168Z2023-10-25 07:31:24.316296381 UTC: attempting to log into API
 
9432023-10-25T07:31:26.484ZOct 25 07:31:26 sock ip: ip: joining multicasts failed (4) on vopte3 - will use link layer broadcasts for multicast
9442023-10-25T07:31:33.003ZOct 25 07:31:32 sok ip: ip: joining multicasts failed (4) on vopte4 - will use link layer broadcasts for multicast
9452023-10-25T07:31:45.477ZOct 25 07:31:39 sock ip: ip: joining multicasts failed (4) on vopte5 - will use link layer broadcasts for multicast
 
9462023-10-25T07:31:45.506Z2023-10-25 07:31:39.308579498 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9472023-10-25T07:31:45.589Z2023-10-25 07:31:40.306902349 UTC: attempting to log into API
9482023-10-25T07:31:55.233Z2023-10-25 07:31:55.299030731 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9492023-10-25T07:31:56.235Z2023-10-25 07:31:56.298068864 UTC: attempting to log into API
9502023-10-25T07:32:11.268Z2023-10-25 07:32:11.290041992 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9512023-10-25T07:32:12.278Z2023-10-25 07:32:12.288179879 UTC: attempting to log into API
9522023-10-25T07:32:23.564Z2023-10-25 07:32:23.042350250 UTC: login succeeded
9532023-10-25T07:32:23.919Zwaiting for nexus to come up...
9542023-10-25T07:32:24.082Zcreating IP pool... 10.151.2.175 - 10.151.2.227
9552023-10-25T07:32:24.201Zensuring datasets are ready...
9562023-10-25T07:32:26.022Zlet's roll.
9572023-10-25T07:32:55.215Z+ rm ./tests/bootstrap
9582023-10-25T07:32:56.619Z+ for test_bin in tests/*
9592023-10-25T07:32:57.185Z+ ./tests/bootstrap-7505a19fa56a5ab1
9602023-10-25T07:32:57.323Z
9612023-10-25T07:32:57.585Zrunning 0 tests
9622023-10-25T07:32:57.709Z
9632023-10-25T07:32:57.800Ztest result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
9642023-10-25T07:32:57.856Z
9652023-10-25T07:32:58.166Z+ for test_bin in tests/*
9662023-10-25T07:32:58.262Z+ ./tests/end_to_end_tests-a486f79ff30d68c1
9672023-10-25T07:32:58.341Z
9682023-10-25T07:32:58.378Zrunning 2 tests
9692023-10-25T07:32:58.406Ztest no_spoof::no_spoof_here ... ok
9702023-10-25T07:33:25.703Ztest instance_launch::instance_launch has been running for over 60 seconds
9712023-10-25T07:35:18.897Ztest instance_launch::instance_launch ... ok
9722023-10-25T07:35:18.926Z
9732023-10-25T07:35:18.952Ztest result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 173.19s
9742023-10-25T07:35:18.978Z
9752023-10-25T07:35:19.005Z+ _exit_trap
9762023-10-25T07:35:19.029Z+ local status=0
9772023-10-25T07:35:19.058Z+ [[ 0 -eq 0 ]]
9782023-10-25T07:35:19.085Z+ exit 0
9792023-10-25T07:35:19.112Zprocess exited: duration 400386 ms, exit code 0
 
9802023-10-25T07:35:19.219Zfound 51 output files
9812023-10-25T07:35:19.390Zuploading: /var/svc/log/oxide-sled-agent:default.log (454148 bytes)
9822023-10-25T07:35:20.469Zuploaded: /var/svc/log/oxide-sled-agent:default.log
9832023-10-25T07:35:20.500Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_965df092-0b6e-44f2-946f-004e3b779ff6/root/var/svc/log/oxide-internal_dns:default.log (93576 bytes)
9842023-10-25T07:35:21.715Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_965df092-0b6e-44f2-946f-004e3b779ff6/root/var/svc/log/oxide-internal_dns:default.log
9852023-10-25T07:35:21.740Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_965df092-0b6e-44f2-946f-004e3b779ff6/root/var/svc/log/oxide-internal_dns:default.log.0 (5907 bytes)
9862023-10-25T07:35:22.811Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_965df092-0b6e-44f2-946f-004e3b779ff6/root/var/svc/log/oxide-internal_dns:default.log.0
9872023-10-25T07:35:22.836Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_oximeter_fddbc2f8-b0a9-4a43-a608-e2bd2260b4f8/root/var/svc/log/oxide-oximeter:default.log (174491 bytes)
9882023-10-25T07:35:23.016Zupload warning: file "/pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_oximeter_fddbc2f8-b0a9-4a43-a608-e2bd2260b4f8/root/var/svc/log/oxide-oximeter:default.log" changed size mid upload: 174491 -> 176598
9892023-10-25T07:35:24.095Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_oximeter_fddbc2f8-b0a9-4a43-a608-e2bd2260b4f8/root/var/svc/log/oxide-oximeter:default.log
9902023-10-25T07:35:24.120Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_clickhouse_4b831784-be3a-416f-9a4c-97df1e394941/root/var/svc/log/oxide-clickhouse:default.log (645376 bytes)
9912023-10-25T07:35:25.415Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_clickhouse_4b831784-be3a-416f-9a4c-97df1e394941/root/var/svc/log/oxide-clickhouse:default.log
9922023-10-25T07:35:25.855Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_206c4b76-edfd-4576-85fb-24874541e443/root/var/svc/log/oxide-crucible-agent:default.log (13684 bytes)
9932023-10-25T07:35:26.538Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_206c4b76-edfd-4576-85fb-24874541e443/root/var/svc/log/oxide-crucible-agent:default.log
9942023-10-25T07:35:26.563Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_206c4b76-edfd-4576-85fb-24874541e443/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
9952023-10-25T07:35:27.634Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_206c4b76-edfd-4576-85fb-24874541e443/root/var/svc/log/oxide-crucible-downstairs:default.log
9962023-10-25T07:35:27.659Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_206c4b76-edfd-4576-85fb-24874541e443/root/var/svc/log/oxide-crucible-downstairs:downstairs-a90c7733-33d6-444a-bab8-94114100f5fc.log (3050 bytes)
9972023-10-25T07:35:29.158Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_206c4b76-edfd-4576-85fb-24874541e443/root/var/svc/log/oxide-crucible-downstairs:downstairs-a90c7733-33d6-444a-bab8-94114100f5fc.log
9982023-10-25T07:35:29.254Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_f155e354-4667-41a4-86df-7b7c9baf046a/root/var/svc/log/oxide-ntp:default.log (1062 bytes)
9992023-10-25T07:35:30.331Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_f155e354-4667-41a4-86df-7b7c9baf046a/root/var/svc/log/oxide-ntp:default.log
10002023-10-25T07:35:30.356Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_f155e354-4667-41a4-86df-7b7c9baf046a/root/var/svc/log/oxide-ntp:default.log.0 (120 bytes)
10012023-10-25T07:35:31.428Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_f155e354-4667-41a4-86df-7b7c9baf046a/root/var/svc/log/oxide-ntp:default.log.0
10022023-10-25T07:35:31.455Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_364b5bc8-206e-419c-8274-3dcf5f310857/root/var/svc/log/oxide-crucible-agent:default.log (14579 bytes)
10032023-10-25T07:35:32.567Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_364b5bc8-206e-419c-8274-3dcf5f310857/root/var/svc/log/oxide-crucible-agent:default.log
10042023-10-25T07:35:32.593Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_364b5bc8-206e-419c-8274-3dcf5f310857/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10052023-10-25T07:35:33.665Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_364b5bc8-206e-419c-8274-3dcf5f310857/root/var/svc/log/oxide-crucible-downstairs:default.log
10062023-10-25T07:35:33.692Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_364b5bc8-206e-419c-8274-3dcf5f310857/root/var/svc/log/oxide-crucible-downstairs:downstairs-0e353560-d242-4729-aead-6d8577a32558.log (5536 bytes)
10072023-10-25T07:35:34.762Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_364b5bc8-206e-419c-8274-3dcf5f310857/root/var/svc/log/oxide-crucible-downstairs:downstairs-0e353560-d242-4729-aead-6d8577a32558.log
10082023-10-25T07:35:34.793Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_cockroachdb_ab6141a3-7962-414b-be50-515c9df514ff/root/var/svc/log/oxide-cockroachdb:default.log (5369 bytes)
10092023-10-25T07:35:35.925Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_cockroachdb_ab6141a3-7962-414b-be50-515c9df514ff/root/var/svc/log/oxide-cockroachdb:default.log
10102023-10-25T07:35:35.972Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_b20dca9b-d1f3-4966-908f-951821ffdc70/root/var/svc/log/oxide-crucible-agent:default.log (3436 bytes)
10112023-10-25T07:35:37.044Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_b20dca9b-d1f3-4966-908f-951821ffdc70/root/var/svc/log/oxide-crucible-agent:default.log
10122023-10-25T07:35:37.072Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_b20dca9b-d1f3-4966-908f-951821ffdc70/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10132023-10-25T07:35:38.151Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_b20dca9b-d1f3-4966-908f-951821ffdc70/root/var/svc/log/oxide-crucible-downstairs:default.log
10142023-10-25T07:35:38.181Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_external_dns_ac17cb9b-2923-45b5-923c-18026cf3059d/root/var/svc/log/oxide-external_dns:default.log (15611 bytes)
10152023-10-25T07:35:39.301Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_external_dns_ac17cb9b-2923-45b5-923c-18026cf3059d/root/var/svc/log/oxide-external_dns:default.log
10162023-10-25T07:35:39.343Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_cockroachdb_5c97c543-4ea0-4ee5-b78f-ce3d5c998700/root/var/svc/log/oxide-cockroachdb:default.log (5368 bytes)
10172023-10-25T07:35:40.398Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_cockroachdb_5c97c543-4ea0-4ee5-b78f-ce3d5c998700/root/var/svc/log/oxide-cockroachdb:default.log
10182023-10-25T07:35:40.423Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_cockroachdb_dcf19c04-9532-4f94-a98f-da3c8046b92e/root/var/svc/log/oxide-cockroachdb:default.log (5369 bytes)
10192023-10-25T07:35:41.497Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_cockroachdb_dcf19c04-9532-4f94-a98f-da3c8046b92e/root/var/svc/log/oxide-cockroachdb:default.log
10202023-10-25T07:35:41.522Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_7f3f8904-e99e-49c5-bfc5-fa3c04981115/root/var/svc/log/oxide-crucible-agent:default.log (24392 bytes)
10212023-10-25T07:35:42.642Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_7f3f8904-e99e-49c5-bfc5-fa3c04981115/root/var/svc/log/oxide-crucible-agent:default.log
10222023-10-25T07:35:42.667Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_7f3f8904-e99e-49c5-bfc5-fa3c04981115/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10232023-10-25T07:35:43.741Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_7f3f8904-e99e-49c5-bfc5-fa3c04981115/root/var/svc/log/oxide-crucible-downstairs:default.log
10242023-10-25T07:35:43.767Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_7f3f8904-e99e-49c5-bfc5-fa3c04981115/root/var/svc/log/oxide-crucible-downstairs:downstairs-68328638-8300-45ca-a610-de041311045f.log (3050 bytes)
10252023-10-25T07:35:44.907Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_7f3f8904-e99e-49c5-bfc5-fa3c04981115/root/var/svc/log/oxide-crucible-downstairs:downstairs-68328638-8300-45ca-a610-de041311045f.log
10262023-10-25T07:35:44.939Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_7f3f8904-e99e-49c5-bfc5-fa3c04981115/root/var/svc/log/oxide-crucible-downstairs:downstairs-6a15842b-15e0-4490-b399-77e2ca289249.log (5537 bytes)
10272023-10-25T07:35:46.032Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_7f3f8904-e99e-49c5-bfc5-fa3c04981115/root/var/svc/log/oxide-crucible-downstairs:downstairs-6a15842b-15e0-4490-b399-77e2ca289249.log
10282023-10-25T07:35:46.057Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_ab12860f-6dda-41c9-85bc-0f5e176f3c92/root/var/svc/log/oxide-crucible-agent:default.log (3436 bytes)
10292023-10-25T07:35:47.132Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_ab12860f-6dda-41c9-85bc-0f5e176f3c92/root/var/svc/log/oxide-crucible-agent:default.log
10302023-10-25T07:35:47.157Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_ab12860f-6dda-41c9-85bc-0f5e176f3c92/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10312023-10-25T07:35:48.270Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_ab12860f-6dda-41c9-85bc-0f5e176f3c92/root/var/svc/log/oxide-crucible-downstairs:default.log
10322023-10-25T07:35:48.296Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_pantry_68469549-8998-46da-89bb-928e981c6460/root/var/svc/log/oxide-crucible-pantry:default.log (2337 bytes)
10332023-10-25T07:35:49.368Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_pantry_68469549-8998-46da-89bb-928e981c6460/root/var/svc/log/oxide-crucible-pantry:default.log
10342023-10-25T07:35:49.394Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_nexus_c18262cf-7222-4f51-aee5-37dcfa593d70/root/var/svc/log/oxide-nexus:default.log (521150 bytes)
10352023-10-25T07:35:49.667Zupload warning: file "/pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_nexus_c18262cf-7222-4f51-aee5-37dcfa593d70/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 521150 -> 581153
10362023-10-25T07:35:50.752Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_nexus_c18262cf-7222-4f51-aee5-37dcfa593d70/root/var/svc/log/oxide-nexus:default.log
10372023-10-25T07:35:50.777Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_pantry_3e1680e7-ad99-4972-95e8-92651aabb0e4/root/var/svc/log/oxide-crucible-pantry:default.log (2333 bytes)
10382023-10-25T07:35:51.850Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_pantry_3e1680e7-ad99-4972-95e8-92651aabb0e4/root/var/svc/log/oxide-crucible-pantry:default.log
10392023-10-25T07:35:51.881Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_cockroachdb_549f52ba-97ce-4a02-917e-5fe5c22ac55f/root/var/svc/log/oxide-cockroachdb:default.log (5343 bytes)
10402023-10-25T07:35:52.968Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_cockroachdb_549f52ba-97ce-4a02-917e-5fe5c22ac55f/root/var/svc/log/oxide-cockroachdb:default.log
10412023-10-25T07:35:52.994Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_cockroachdb_9ab0d7a2-255c-4672-a26d-1620b04904dd/root/var/svc/log/oxide-cockroachdb:default.log (5370 bytes)
10422023-10-25T07:35:54.067Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_cockroachdb_9ab0d7a2-255c-4672-a26d-1620b04904dd/root/var/svc/log/oxide-cockroachdb:default.log
10432023-10-25T07:35:54.093Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_0348d0ca-3e7c-4d78-8d4c-cb0269d633ac/root/var/svc/log/oxide-crucible-agent:default.log (3436 bytes)
10442023-10-25T07:35:55.193Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_0348d0ca-3e7c-4d78-8d4c-cb0269d633ac/root/var/svc/log/oxide-crucible-agent:default.log
10452023-10-25T07:35:55.241Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_0348d0ca-3e7c-4d78-8d4c-cb0269d633ac/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10462023-10-25T07:35:56.314Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_0348d0ca-3e7c-4d78-8d4c-cb0269d633ac/root/var/svc/log/oxide-crucible-downstairs:default.log
10472023-10-25T07:35:56.339Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_external_dns_3c79b749-b606-44f5-a137-99534ffd6b4b/root/var/svc/log/oxide-external_dns:default.log (15611 bytes)
10482023-10-25T07:35:56.409Zupload warning: file "/pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_external_dns_3c79b749-b606-44f5-a137-99534ffd6b4b/root/var/svc/log/oxide-external_dns:default.log" changed size mid upload: 15611 -> 19388
10492023-10-25T07:35:57.482Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_external_dns_3c79b749-b606-44f5-a137-99534ffd6b4b/root/var/svc/log/oxide-external_dns:default.log
10502023-10-25T07:35:57.507Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_internal_dns_1f52ac37-3a0a-4874-9873-8979a4155077/root/var/svc/log/oxide-internal_dns:default.log (93565 bytes)
10512023-10-25T07:35:57.646Zupload warning: file "/pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_internal_dns_1f52ac37-3a0a-4874-9873-8979a4155077/root/var/svc/log/oxide-internal_dns:default.log" changed size mid upload: 93565 -> 97321
10522023-10-25T07:35:58.725Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_internal_dns_1f52ac37-3a0a-4874-9873-8979a4155077/root/var/svc/log/oxide-internal_dns:default.log
10532023-10-25T07:35:58.753Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_internal_dns_1f52ac37-3a0a-4874-9873-8979a4155077/root/var/svc/log/oxide-internal_dns:default.log.0 (11596 bytes)
10542023-10-25T07:35:59.852Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_internal_dns_1f52ac37-3a0a-4874-9873-8979a4155077/root/var/svc/log/oxide-internal_dns:default.log.0
10552023-10-25T07:35:59.891Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_85ff9678-7351-4224-965f-6cd43fc99da2/root/var/svc/log/oxide-nexus:default.log (4557276 bytes)
10562023-10-25T07:36:01.659Zupload warning: file "/pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_85ff9678-7351-4224-965f-6cd43fc99da2/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 4557276 -> 4618499
10572023-10-25T07:36:02.757Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_85ff9678-7351-4224-965f-6cd43fc99da2/root/var/svc/log/oxide-nexus:default.log
10582023-10-25T07:36:02.782Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_3f8146e4-a9b3-4851-badc-967edc34936e/root/var/svc/log/oxide-crucible-agent:default.log (13684 bytes)
10592023-10-25T07:36:03.906Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_3f8146e4-a9b3-4851-badc-967edc34936e/root/var/svc/log/oxide-crucible-agent:default.log
10602023-10-25T07:36:03.938Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_3f8146e4-a9b3-4851-badc-967edc34936e/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10612023-10-25T07:36:05.420Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_3f8146e4-a9b3-4851-badc-967edc34936e/root/var/svc/log/oxide-crucible-downstairs:default.log
10622023-10-25T07:36:05.448Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_3f8146e4-a9b3-4851-badc-967edc34936e/root/var/svc/log/oxide-crucible-downstairs:downstairs-0f16a3a7-9149-406f-b21a-c7c1ba207374.log (5535 bytes)
10632023-10-25T07:36:06.525Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_3f8146e4-a9b3-4851-badc-967edc34936e/root/var/svc/log/oxide-crucible-downstairs:downstairs-0f16a3a7-9149-406f-b21a-c7c1ba207374.log
10642023-10-25T07:36:06.550Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_5281082d-454d-4578-8a40-9685f2bc250c/root/var/svc/log/oxide-crucible-agent:default.log (14113 bytes)
10652023-10-25T07:36:07.645Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_5281082d-454d-4578-8a40-9685f2bc250c/root/var/svc/log/oxide-crucible-agent:default.log
10662023-10-25T07:36:07.695Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_5281082d-454d-4578-8a40-9685f2bc250c/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10672023-10-25T07:36:08.777Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_5281082d-454d-4578-8a40-9685f2bc250c/root/var/svc/log/oxide-crucible-downstairs:default.log
10682023-10-25T07:36:08.804Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_5281082d-454d-4578-8a40-9685f2bc250c/root/var/svc/log/oxide-crucible-downstairs:downstairs-d125b63d-ffa5-4d22-9b0e-56c8d480be99.log (3050 bytes)
10692023-10-25T07:36:10.308Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_5281082d-454d-4578-8a40-9685f2bc250c/root/var/svc/log/oxide-crucible-downstairs:downstairs-d125b63d-ffa5-4d22-9b0e-56c8d480be99.log
10702023-10-25T07:36:10.333Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_c4d36bd4-1c3a-4e75-b807-9bb0e0278a53/root/var/svc/log/oxide-crucible-agent:default.log (3436 bytes)
10712023-10-25T07:36:11.415Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_c4d36bd4-1c3a-4e75-b807-9bb0e0278a53/root/var/svc/log/oxide-crucible-agent:default.log
10722023-10-25T07:36:11.441Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_c4d36bd4-1c3a-4e75-b807-9bb0e0278a53/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10732023-10-25T07:36:12.515Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_c4d36bd4-1c3a-4e75-b807-9bb0e0278a53/root/var/svc/log/oxide-crucible-downstairs:default.log
10742023-10-25T07:36:12.540Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_a860316d-8e07-475b-a42c-f06826a62671/root/var/svc/log/oxide-crucible-pantry:default.log (2338 bytes)
10752023-10-25T07:36:13.623Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_a860316d-8e07-475b-a42c-f06826a62671/root/var/svc/log/oxide-crucible-pantry:default.log
10762023-10-25T07:36:14.060Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_5f85facf-ada3-4e08-b53d-566f078677c7/root/var/svc/log/oxide-internal_dns:default.log (93569 bytes)
10772023-10-25T07:36:14.089Zupload warning: file "/pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_5f85facf-ada3-4e08-b53d-566f078677c7/root/var/svc/log/oxide-internal_dns:default.log" changed size mid upload: 93569 -> 97325
10782023-10-25T07:36:14.862Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_5f85facf-ada3-4e08-b53d-566f078677c7/root/var/svc/log/oxide-internal_dns:default.log
10792023-10-25T07:36:14.886Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_5f85facf-ada3-4e08-b53d-566f078677c7/root/var/svc/log/oxide-internal_dns:default.log.0 (9158 bytes)
10802023-10-25T07:36:15.981Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_5f85facf-ada3-4e08-b53d-566f078677c7/root/var/svc/log/oxide-internal_dns:default.log.0
10812023-10-25T07:36:16.054Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_9f3d1ae4-3c38-4796-b430-c081b52a3826/root/var/svc/log/oxide-nexus:default.log (597200 bytes)
10822023-10-25T07:36:16.334Zupload warning: file "/pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_9f3d1ae4-3c38-4796-b430-c081b52a3826/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 597200 -> 659023
10832023-10-25T07:36:17.429Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_9f3d1ae4-3c38-4796-b430-c081b52a3826/root/var/svc/log/oxide-nexus:default.log
10842023-10-25T07:36:17.453Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_f155e354-4667-41a4-86df-7b7c9baf046a/root/var/log/chrony/measurements.log (16988 bytes)
10852023-10-25T07:36:17.521Zupload warning: file "/pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_f155e354-4667-41a4-86df-7b7c9baf046a/root/var/log/chrony/measurements.log" changed size mid upload: 16988 -> 18632
10862023-10-25T07:36:18.594Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_f155e354-4667-41a4-86df-7b7c9baf046a/root/var/log/chrony/measurements.log
10872023-10-25T07:36:18.626Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_f155e354-4667-41a4-86df-7b7c9baf046a/root/var/log/chrony/statistics.log (9243 bytes)
10882023-10-25T07:36:18.676Zupload warning: file "/pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_f155e354-4667-41a4-86df-7b7c9baf046a/root/var/log/chrony/statistics.log" changed size mid upload: 9243 -> 10530
10892023-10-25T07:36:19.757Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_f155e354-4667-41a4-86df-7b7c9baf046a/root/var/log/chrony/statistics.log
10902023-10-25T07:36:19.782Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_f155e354-4667-41a4-86df-7b7c9baf046a/root/var/log/chrony/tracking.log (1849 bytes)
10912023-10-25T07:36:19.807Zupload warning: file "/pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_f155e354-4667-41a4-86df-7b7c9baf046a/root/var/log/chrony/tracking.log" changed size mid upload: 1849 -> 1981
10922023-10-25T07:36:20.885Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_f155e354-4667-41a4-86df-7b7c9baf046a/root/var/log/chrony/tracking.log