01HD54G5DKA0CYV28V09NR7KHM: helios / deploy

BasicConfig {
    output_rules: [
        "%/var/svc/log/oxide-sled-agent:default.log*",
        "%/pool/ext/*/crypt/zone/oxz_*/root/var/svc/log/oxide-*.log*",
        "%/pool/ext/*/crypt/zone/oxz_*/root/var/svc/log/system-illumos-*.log*",
        "%/pool/ext/*/crypt/zone/oxz_ntp_*/root/var/log/chrony/*.log*",
        "!/pool/ext/*/crypt/zone/oxz_propolis-server_*/root/var/svc/log/*.log*",
        "%/pool/ext/*/crypt/debug/global/oxide-sled-agent:default.log.*",
        "%/pool/ext/*/crypt/debug/oxz_*/oxide-*.log.*",
        "%/pool/ext/*/crypt/debug/oxz_*/system-illumos-*.log.*",
        "!/pool/ext/*/crypt/debug/oxz_propolis-server_*/*.log.*",
    ],
    rust_toolchain: None,
    target: Some(
        "lab-2.0-opte-0.25",
    ),
    access_repos: [],
    publish: [],
    skip_clone: true,
}

Buildomat Job: 01HD54H56VACDA2ZWYPPDSX6K6

Tags:

Artefacts:

Output:

SEQ GLOBAL TIME DETAILS
12023-10-20T00:21:55.196Zjob dependencies complete; ready to run (waiting for 49 m 22 s)
22023-10-20T00:28:42.069Zjob assigned to worker 01HD57PWH6QKKFWF47KJR0P6RB (queued for 6 m 46 s)
32023-10-20T00:28:42.181Zhttp://172.20.0.5:9969/ipxe/buskin... ok
42023-10-20T00:28:42.224Zhttp://172.20.0.5:9969/os/buskin/platform/i86pc/kernel/amd64/unix... ok
52023-10-20T00:28:42.258Zhttp://172.20.0.5:9969/os/buskin/platform/i86pc/amd64/boot_archive... ok
62023-10-20T00:28:42.291Zhttp://172.20.0.5:9969/os/buskin/platform/i86pc/amd64/boot_archive.hash... ok
72023-10-20T00:28:42.325Zhttp://172.20.0.5:9969/postboot/buskin... ok
82023-10-20T00:28:42.358ZwcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U
92023-10-20T00:28:42.391Zcpu0: microcode has been updated from version 0xa001133 to 0xa0011d1
102023-10-20T00:28:42.425ZOxide Helios Version helios-2.0.22241 64-bit
112023-10-20T00:28:42.458ZConfiguring devices.
122023-10-20T00:28:42.492Z##################################################
132023-10-20T00:28:42.525Z#### ############# ##
142023-10-20T00:28:42.558Z### ### ############ ##
152023-10-20T00:28:42.592Z## ### # ## ### ## Oxide Computer Company ##
162023-10-20T00:28:42.625Z## ## ## ### # ### ##
172023-10-20T00:28:42.659Z## # ### #### #### This Station Under ##
182023-10-20T00:28:42.701Z### ### #### # ### Computer Control ##
192023-10-20T00:28:42.737Z#### #### ### ## ##
202023-10-20T00:28:42.770Z##################################################
212023-10-20T00:28:42.804ZHostname: buskin
222023-10-20T00:28:42.837ZLoading smf(7) serice descriptions:
232023-10-20T00:28:43.971Zdownloading input: /input/ci-tools/work/end-to-end-tests/bootstrap-8a64e1a08208e018.gz
242023-10-20T00:28:45.307Zdownloaded input: /input/ci-tools/work/end-to-end-tests/bootstrap-8a64e1a08208e018.gz
252023-10-20T00:28:45.331Zdownloading input: /input/ci-tools/work/end-to-end-tests/bootstrap.gz
262023-10-20T00:29:02.490Zdownloaded input: /input/ci-tools/work/end-to-end-tests/bootstrap.gz
272023-10-20T00:29:02.517Zdownloading input: /input/ci-tools/work/end-to-end-tests/end_to_end_tests-ac1cb3326a4ba018.gz
282023-10-20T00:29:21.519Zdownloaded input: /input/ci-tools/work/end-to-end-tests/end_to_end_tests-ac1cb3326a4ba018.gz
292023-10-20T00:29:21.543Zdownloading input: /input/ci-tools/work/caboose-util.gz
302023-10-20T00:29:22.445Zdownloaded input: /input/ci-tools/work/caboose-util.gz
312023-10-20T00:29:22.470Zdownloading input: /input/ci-tools/work/tufaceous.gz
322023-10-20T00:29:25.181Zdownloaded input: /input/ci-tools/work/tufaceous.gz
332023-10-20T00:29:25.205Zdownloading input: /input/package/work/version.txt
342023-10-20T00:29:25.232Zdownloaded input: /input/package/work/version.txt
352023-10-20T00:29:25.258Zdownloading input: /input/package/work/package.tar.gz
362023-10-20T00:29:40.657Zdownloaded input: /input/package/work/package.tar.gz
372023-10-20T00:29:40.682Zdownloading input: /input/package/work/global-zone-packages.tar.gz
382023-10-20T00:30:03.830Zdownloaded input: /input/package/work/global-zone-packages.tar.gz
392023-10-20T00:30:03.855Zdownloading input: /input/package/work/trampoline-global-zone-packages.tar.gz
402023-10-20T00:30:10.555Zdownloaded input: /input/package/work/trampoline-global-zone-packages.tar.gz
412023-10-20T00:30:10.579Zdownloading input: /input/package/work/zones/clickhouse.tar.gz
422023-10-20T00:30:47.154Zdownloaded input: /input/package/work/zones/clickhouse.tar.gz
432023-10-20T00:30:47.179Zdownloading input: /input/package/work/zones/clickhouse_keeper.tar.gz
442023-10-20T00:31:17.952Zdownloaded input: /input/package/work/zones/clickhouse_keeper.tar.gz
452023-10-20T00:31:17.976Zdownloading input: /input/package/work/zones/cockroachdb.tar.gz
462023-10-20T00:31:41.704Zdownloaded input: /input/package/work/zones/cockroachdb.tar.gz
472023-10-20T00:31:41.732Zdownloading input: /input/package/work/zones/crucible-pantry.tar.gz
482023-10-20T00:31:45.869Zdownloaded input: /input/package/work/zones/crucible-pantry.tar.gz
492023-10-20T00:31:45.894Zdownloading input: /input/package/work/zones/crucible.tar.gz
502023-10-20T00:31:53.374Zdownloaded input: /input/package/work/zones/crucible.tar.gz
512023-10-20T00:31:53.400Zdownloading input: /input/package/work/zones/external-dns.tar.gz
522023-10-20T00:31:57.690Zdownloaded input: /input/package/work/zones/external-dns.tar.gz
532023-10-20T00:31:57.715Zdownloading input: /input/package/work/zones/internal-dns.tar.gz
542023-10-20T00:32:01.643Zdownloaded input: /input/package/work/zones/internal-dns.tar.gz
552023-10-20T00:32:01.681Zdownloading input: /input/package/work/zones/ntp.tar.gz
562023-10-20T00:32:01.721Zdownloaded input: /input/package/work/zones/ntp.tar.gz
572023-10-20T00:32:01.745Zdownloading input: /input/package/work/zones/omicron-gateway-asic.tar.gz
582023-10-20T00:32:05.490Zdownloaded input: /input/package/work/zones/omicron-gateway-asic.tar.gz
592023-10-20T00:32:05.515Zdownloading input: /input/package/work/zones/omicron-gateway-softnpu.tar.gz
602023-10-20T00:32:08.501Zdownloaded input: /input/package/work/zones/omicron-gateway-softnpu.tar.gz
612023-10-20T00:32:08.525Zdownloading input: /input/package/work/zones/omicron-nexus-single-sled.tar.gz
622023-10-20T00:32:21.811Zdownloaded input: /input/package/work/zones/omicron-nexus-single-sled.tar.gz
632023-10-20T00:32:21.835Zdownloading input: /input/package/work/zones/omicron-nexus.tar.gz
642023-10-20T00:32:35.017Zdownloaded input: /input/package/work/zones/omicron-nexus.tar.gz
652023-10-20T00:32:35.046Zdownloading input: /input/package/work/zones/overlay.tar.gz
662023-10-20T00:32:35.111Zdownloaded input: /input/package/work/zones/overlay.tar.gz
672023-10-20T00:32:35.135Zdownloading input: /input/package/work/zones/oximeter-collector.tar.gz
682023-10-20T00:32:39.213Zdownloaded input: /input/package/work/zones/oximeter-collector.tar.gz
692023-10-20T00:32:39.252Zdownloading input: /input/package/work/zones/propolis-server.tar.gz
702023-10-20T00:32:52.774Zdownloaded input: /input/package/work/zones/propolis-server.tar.gz
712023-10-20T00:32:52.799Zdownloading input: /input/package/work/zones/switch-asic.tar.gz
722023-10-20T00:33:39.398Zdownloaded input: /input/package/work/zones/switch-asic.tar.gz
732023-10-20T00:33:39.423Zdownloading input: /input/package/work/zones/switch-softnpu.tar.gz
742023-10-20T00:34:07.603Zdownloaded input: /input/package/work/zones/switch-softnpu.tar.gz
 
752023-10-20T00:34:07.627Zstarting task 0: "setup"
762023-10-20T00:34:07.660Z++ uname -s
772023-10-20T00:34:07.684Z+ kern=SunOS
782023-10-20T00:34:07.708Z+ case "$kern" in
792023-10-20T00:34:07.731Z+ groupadd -g 12345 build
802023-10-20T00:34:07.755Z+ useradd -u 12345 -g build -d /home/build -s /bin/bash -c build -P 'Primary Administrator' build
812023-10-20T00:34:09.670Z+ zfs create -o mountpoint=/work rpool/work
822023-10-20T00:34:09.697Z++ awk '$2 == "/home" { print $3 }' /etc/mnttab
832023-10-20T00:34:09.721Z+ home_fs=zfs
842023-10-20T00:34:09.745Z+ [[ zfs == autofs ]]
852023-10-20T00:34:09.769Z+ mkdir -p /home/build
862023-10-20T00:34:09.793Z+ chown build:build /home/build /work
872023-10-20T00:34:11.706Z+ chmod 0700 /home/build /work
882023-10-20T00:34:11.729Zprocess exited: duration 4058 ms, exit code 0
 
892023-10-20T00:34:11.777Zstarting task 1: "authentication"
902023-10-20T00:34:11.852Zprocess exited: duration 44 ms, exit code 0
 
912023-10-20T00:34:11.906Zstarting task 2: "build"
922023-10-20T00:34:11.938Z+ trap _exit_trap EXIT
932023-10-20T00:34:11.961Z+ pfexec mdb -kwe 'secpolicy_fs_linkdir/v 55 48 89 e5 b8 01 00 00 00 5d c3'
942023-10-20T00:34:12.132Zsecpolicy_fs_linkdir: 0x55 = 0x55
952023-10-20T00:34:12.156Zsecpolicy_fs_linkdir+1: 0xb8 = 0x48
962023-10-20T00:34:12.180Zsecpolicy_fs_linkdir+2: 0x1 = 0x89
972023-10-20T00:34:12.204Zsecpolicy_fs_linkdir+3: 0 = 0xe5
982023-10-20T00:34:12.228Zsecpolicy_fs_linkdir+4: 0 = 0xb8
992023-10-20T00:34:12.252Zsecpolicy_fs_linkdir+5: 0 = 0x1
1002023-10-20T00:34:12.276Zsecpolicy_fs_linkdir+6: 0x48 = 0x0
1012023-10-20T00:34:12.299Zsecpolicy_fs_linkdir+7: 0x89 = 0x0
1022023-10-20T00:34:12.732Zsecpolicy_fs_linkdir+8: 0xe5 = 0x0
1032023-10-20T00:34:12.755Zsecpolicy_fs_linkdir+9: 0x48 = 0x5d
1042023-10-20T00:34:12.779Zsecpolicy_fs_linkdir+0xa: 0x83 = 0xc3
1052023-10-20T00:34:12.804Z+ [[ -d /opt/oxide ]]
1062023-10-20T00:34:12.828Z+ pfexec mv /opt/oxide /opt/oxide-underneath
1072023-10-20T00:34:12.852Z+ pfexec mkdir /opt/oxide
1082023-10-20T00:34:12.876Z+ pfexec mount -F tmpfs -O swap /opt/oxide
1092023-10-20T00:34:12.900Z+ [[ -d /opt/oxide-underneath ]]
1102023-10-20T00:34:12.923Z+ cd /opt/oxide-underneath
1112023-10-20T00:34:12.947Z+ pfexec tar ceEp@/f - .
1122023-10-20T00:34:12.971Z+ cd /opt/oxide
1132023-10-20T00:34:12.997Z+ pfexec tar xveEp@/f -
1142023-10-20T00:34:13.021Zx ., 0 bytes, 0 tape blocks
1152023-10-20T00:34:13.045Zx ./opte, 0 bytes, 0 tape blocks
1162023-10-20T00:34:13.069Zx ./opte/bin, 0 bytes, 0 tape blocks
1172023-10-20T00:34:13.092Zx ./opte/bin/opteadm, 50714832 bytes, 99053 tape blocks
1182023-10-20T00:34:13.119Z+ pfexec rm -rf /opt/oxide-underneath
1192023-10-20T00:34:13.144Z+ pfexec /sbin/zfs create -o mountpoint=/zone rpool/zone
1202023-10-20T00:34:13.167Z+ pfexec mkdir /opt/oxide/work
1212023-10-20T00:34:13.191Z+ pfexec chown build:build /opt/oxide/work
1222023-10-20T00:34:13.214Z+ cd /opt/oxide/work
1232023-10-20T00:34:13.238Z+ ptime -m tar xvzf /input/package/work/package.tar.gz
1242023-10-20T00:34:13.901ZDecompressing '/input/package/work/package.tar.gz' with '/usr/bin/gzcat'...
1252023-10-20T00:34:13.925Zx out/maghemite.tar, 74957312 bytes, 146401 tape blocks
1262023-10-20T00:34:13.974Zx out/omicron-sled-agent.tar, 114068992 bytes, 222791 tape blocks
1272023-10-20T00:34:13.998Zx out/target/test, 75 bytes, 1 tape blocks
1282023-10-20T00:34:14.024Zx out/npuzone/npuzone, 37379944 bytes, 73008 tape blocks
1292023-10-20T00:34:14.057Zx package-manifest.toml, 21303 bytes, 42 tape blocks
1302023-10-20T00:34:14.082Zx smf/sled-agent/non-gimlet/config.toml, 3097 bytes, 7 tape blocks
1312023-10-20T00:34:14.108Zx target/release/omicron-package, 38081152 bytes, 74378 tape blocks
1322023-10-20T00:34:14.132Zx tools/create_virtual_hardware.sh, 2818 bytes, 6 tape blocks
1332023-10-20T00:34:14.157Zx tools/virtual_hardware.sh, 3408 bytes, 7 tape blocks
1342023-10-20T00:34:14.200Zx tools/scrimlet/create-softnpu-zone.sh, 619 bytes, 2 tape blocks
1352023-10-20T00:34:14.226Zx tools/scrimlet/destroy-softnpu-zone.sh, 195 bytes, 1 tape blocks
1362023-10-20T00:34:14.250Zx tools/scrimlet/softnpu-init.sh, 2148 bytes, 5 tape blocks
1372023-10-20T00:34:14.273Zx tools/scrimlet/softnpu-zone.txt, 330 bytes, 1 tape blocks
1382023-10-20T00:34:14.298Zx tools/scrimlet/softnpu.toml, 171 bytes, 1 tape blocks
1392023-10-20T00:34:14.323Z
1402023-10-20T00:34:14.346Zreal 1.609114877
1412023-10-20T00:34:14.372Zuser 1.448353122
1422023-10-20T00:34:14.396Zsys 0.284973687
1432023-10-20T00:34:14.420Ztrap 0.000061389
1442023-10-20T00:34:14.443Ztflt 0.000000000
1452023-10-20T00:34:14.466Zdflt 0.000000000
1462023-10-20T00:34:14.490Zkflt 0.000000000
1472023-10-20T00:34:14.515Zlock 0.000000000
1482023-10-20T00:34:14.540Zslp 1.446039079
1492023-10-20T00:34:14.564Zlat 0.036604878
1502023-10-20T00:34:14.588Zstop 0.000126713
1512023-10-20T00:34:15.022Z+ cp /input/package/work/zones/clickhouse_keeper.tar.gz /input/package/work/zones/clickhouse.tar.gz /input/package/work/zones/cockroachdb.tar.gz /input/package/work/zones/crucible-pantry.tar.gz /input/package/work/zones/crucible.tar.gz /input/package/work/zones/external-dns.tar.gz /input/package/work/zones/internal-dns.tar.gz /input/package/work/zones/ntp.tar.gz /input/package/work/zones/omicron-gateway-asic.tar.gz /input/package/work/zones/omicron-gateway-softnpu.tar.gz /input/package/work/zones/omicron-nexus-single-sled.tar.gz /input/package/work/zones/omicron-nexus.tar.gz /input/package/work/zones/overlay.tar.gz /input/package/work/zones/oximeter-collector.tar.gz /input/package/work/zones/propolis-server.tar.gz /input/package/work/zones/switch-asic.tar.gz /input/package/work/zones/switch-softnpu.tar.gz out/
1522023-10-20T00:34:15.048Z+ mv out/omicron-nexus-single-sled.tar.gz out/omicron-nexus.tar.gz
1532023-10-20T00:34:15.071Z+ mkdir tests
1542023-10-20T00:34:15.095Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1552023-10-20T00:34:15.118Z+ ptime -m gunzip
1562023-10-20T00:34:15.142Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap-8a64e1a08208e018
1572023-10-20T00:34:15.165Z
1582023-10-20T00:34:15.189Zreal 0.141284060
1592023-10-20T00:34:15.212Zuser 0.128451697
1602023-10-20T00:34:15.236Zsys 0.012612829
1612023-10-20T00:34:15.260Ztrap 0.000009430
1622023-10-20T00:34:15.283Ztflt 0.000000000
1632023-10-20T00:34:15.307Zdflt 0.000000000
1642023-10-20T00:34:15.330Zkflt 0.000000000
1652023-10-20T00:34:15.354Zlock 0.000000000
1662023-10-20T00:34:15.377Zslp 0.000020340
1672023-10-20T00:34:15.402Zlat 0.000016970
1682023-10-20T00:34:15.425Zstop 0.000116482
1692023-10-20T00:34:15.449Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap-8a64e1a08208e018
1702023-10-20T00:34:15.472Z+ chmod a+x tests/bootstrap-8a64e1a08208e018
1712023-10-20T00:34:15.497Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1722023-10-20T00:34:15.520Z+ ptime -m gunzip
1732023-10-20T00:34:15.543Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap
1742023-10-20T00:34:18.554Z
1752023-10-20T00:34:18.580Zreal 2.242540225
1762023-10-20T00:34:18.604Zuser 2.102684610
1772023-10-20T00:34:18.631Zsys 0.139459438
1782023-10-20T00:34:18.663Ztrap 0.000159152
1792023-10-20T00:34:18.692Ztflt 0.000000000
1802023-10-20T00:34:18.715Zdflt 0.000000000
1812023-10-20T00:34:18.740Zkflt 0.000000000
1822023-10-20T00:34:18.764Zlock 0.000000000
1832023-10-20T00:34:18.788Zslp 0.000015310
1842023-10-20T00:34:18.812Zlat 0.000024321
1852023-10-20T00:34:18.837Zstop 0.000128152
1862023-10-20T00:34:18.861Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap
1872023-10-20T00:34:18.886Z+ chmod a+x tests/bootstrap
1882023-10-20T00:34:18.911Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1892023-10-20T00:34:18.965Z+ ptime -m gunzip
1902023-10-20T00:34:18.992Z++ basename /input/ci-tools/work/end-to-end-tests/end_to_end_tests-ac1cb3326a4ba018
1912023-10-20T00:34:19.899Z
1922023-10-20T00:34:19.924Zreal 2.537966895
1932023-10-20T00:34:19.951Zuser 2.359528590
1942023-10-20T00:34:19.976Zsys 0.178050387
1952023-10-20T00:34:20.001Ztrap 0.000161413
1962023-10-20T00:34:20.026Ztflt 0.000000000
1972023-10-20T00:34:20.050Zdflt 0.000000000
1982023-10-20T00:34:20.076Zkflt 0.000000000
1992023-10-20T00:34:20.102Zlock 0.000000000
2002023-10-20T00:34:20.127Zslp 0.000021160
2012023-10-20T00:34:20.152Zlat 0.000021570
2022023-10-20T00:34:20.178Zstop 0.000125782
2032023-10-20T00:34:20.205Z++ basename /input/ci-tools/work/end-to-end-tests/end_to_end_tests-ac1cb3326a4ba018
2042023-10-20T00:34:20.232Z+ chmod a+x tests/end_to_end_tests-ac1cb3326a4ba018
2052023-10-20T00:34:20.262Z+ bmat address ls
2062023-10-20T00:34:20.299ZNAME CIDR FIRST LAST COUNT
2072023-10-20T00:34:20.349Zextra 10.151.1.0/24 10.151.1.164 10.151.1.227 64
2082023-10-20T00:34:20.393Z++ bmat address ls -f extra -Ho first,last,count
2092023-10-20T00:34:20.450Z+ set -- 10.151.1.164 10.151.1.227 64
2102023-10-20T00:34:20.509Z+ EXTRA_IP_START=10.151.1.164
2112023-10-20T00:34:20.563Z+ EXTRA_IP_END=10.151.1.227
2122023-10-20T00:34:20.617Z+ EXTRA_IP_COUNT=64
2132023-10-20T00:34:20.645Z+ (( EXTRA_IP_COUNT >= 32 ))
2142023-10-20T00:34:20.672Z+ EXTRA_IP_BASE=10.151.1
2152023-10-20T00:34:20.697Z+ EXTRA_IP_FOCTET=164
2162023-10-20T00:34:20.735Z+ EXTRA_IP_LOCTET=227
2172023-10-20T00:34:20.761Z+ SERVICE_IP_POOL_START=10.151.1.164
2182023-10-20T00:34:20.786Z+ SERVICE_IP_POOL_END=10.151.1.173
2192023-10-20T00:34:20.815Z+ DNS_IP1=10.151.1.164
2202023-10-20T00:34:20.841Z+ DNS_IP2=10.151.1.165
2212023-10-20T00:34:20.871Z+ UPLINK_IP=10.151.1.174
2222023-10-20T00:34:20.907Z+ IPPOOL_START=10.151.1.175
2232023-10-20T00:34:20.934Z+ IPPOOL_END=10.151.1.227
2242023-10-20T00:34:20.958Z++ ipadm show-addr -po type,addr
2252023-10-20T00:34:20.984Z++ awk '-F[:/]' '$1 == "dhcp" {print $2}'
2262023-10-20T00:34:21.034Z+ GATEWAY_IP=10.151.1.100
2272023-10-20T00:34:21.471Z+ [[ -n 10.151.1.100 ]]
2282023-10-20T00:34:21.505Z+ ping -s 10.151.1.100 56 1
2292023-10-20T00:34:21.538ZPING 10.151.1.100: 56 data bytes
2302023-10-20T00:34:21.566Z64 bytes from 10.151.1.100: icmp_seq=0. time=0.012 ms
2312023-10-20T00:34:21.594Z
2322023-10-20T00:34:21.632Z----10.151.1.100 PING Statistics----
2332023-10-20T00:34:21.656Z1 packets transmitted, 1 packets received, 0% packet loss
2342023-10-20T00:34:21.681Zround-trip (ms) min/avg/max/stddev = 0.012/0.012/0.012/-nan
2352023-10-20T00:34:21.705Z++ arp -an
2362023-10-20T00:34:21.730Z++ awk -vgw=10.151.1.100 '$2 == gw {print $NF}'
2372023-10-20T00:34:21.754Z+ GATEWAY_MAC=18:c0:4d:0d:9f:b2
2382023-10-20T00:34:21.779Z+ routeadm -e ipv4-forwarding -u
2392023-10-20T00:34:21.804Z+ PXA_START=10.151.1.164
2402023-10-20T00:34:21.829Z+ PXA_END=10.151.1.227
2412023-10-20T00:34:22.261Z+ export GATEWAY_IP GATEWAY_MAC PXA_START PXA_END
2422023-10-20T00:34:22.290Z+ pfexec zpool create -f scratch c1t1d0 c2t1d0
2432023-10-20T00:34:22.321Z+ ZPOOL_VDEV_DIR=/scratch
2442023-10-20T00:34:22.753Z+ ptime -m pfexec ./tools/create_virtual_hardware.sh
2452023-10-20T00:34:22.781Z+++ dirname ./tools/create_virtual_hardware.sh
2462023-10-20T00:34:22.809Z++ cd ./tools
2472023-10-20T00:34:22.836Z++ pwd
2482023-10-20T00:34:22.862Z+ SOURCE_DIR=/opt/oxide/work/tools
2492023-10-20T00:34:22.898Z+ OMICRON_TOP=/opt/oxide/work/tools/..
2502023-10-20T00:34:22.923Z+ . /opt/oxide/work/tools/virtual_hardware.sh
2512023-10-20T00:34:22.948Z++ MARKER=/etc/opt/oxide/NO_INSTALL
2522023-10-20T00:34:22.972Z++ [[ -f /etc/opt/oxide/NO_INSTALL ]]
2532023-10-20T00:34:22.997Z++ dladm show-phys -p -o LINK
2542023-10-20T00:34:23.021Z++ head -1
2552023-10-20T00:34:23.046Z+ PHYSICAL_LINK=igb0
2562023-10-20T00:34:23.071Z+ echo 'Using igb0 as physical link'
2572023-10-20T00:34:23.096ZUsing igb0 as physical link
2582023-10-20T00:34:23.136Z+ ensure_run_as_root
2592023-10-20T00:34:23.164Z++ id -u
2602023-10-20T00:34:23.193Z+ [[ 0 -ne 0 ]]
2612023-10-20T00:34:23.219Z+ ensure_zpools
2622023-10-20T00:34:23.252Z+ ZPOOL_TYPES=('oxp_' 'oxi_')
2632023-10-20T00:34:23.278Z+ for ZPOOL_TYPE in "${ZPOOL_TYPES[@]}"
2642023-10-20T00:34:23.302Z+ readarray -t ZPOOLS
2652023-10-20T00:34:23.327Z++ grep '"oxp_' /opt/oxide/work/tools/../smf/sled-agent/non-gimlet/config.toml
2662023-10-20T00:34:23.352Z++ sed 's/[ ",]//g'
2672023-10-20T00:34:23.376Z+ for ZPOOL in "${ZPOOLS[@]}"
2682023-10-20T00:34:23.409Z+ echo 'Zpool: [oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b]'
2692023-10-20T00:34:23.434ZZpool: [oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b]
2702023-10-20T00:34:23.459Z+ VDEV_PATH=/scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
2712023-10-20T00:34:23.483Z+ [[ -f /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
2722023-10-20T00:34:23.508Z+ dd if=/dev/zero of=/scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
2732023-10-20T00:34:23.532Z0+0 records in
2742023-10-20T00:34:23.566ZZFS vdev /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
2752023-10-20T00:34:23.605Z0+0 records out
2762023-10-20T00:34:23.629ZZFS zpool oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b exists
2772023-10-20T00:34:23.653Z0 bytes (0 iB) transferred in 0.000005 secs (0 iB/sec)
2782023-10-20T00:34:23.715ZZpool: [oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03]
2792023-10-20T00:34:23.745Z+ success 'ZFS vdev /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
2802023-10-20T00:34:23.769ZZFS vdev /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
2812023-10-20T00:34:23.794Z+ set +x
2822023-10-20T00:34:23.819ZZFS zpool oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
2832023-10-20T00:34:23.843Z++ zpool list -o name
2842023-10-20T00:34:23.868ZZpool: [oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03]
2852023-10-20T00:34:23.893Z++ grep oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b
2862023-10-20T00:34:23.917ZZFS vdev /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
2872023-10-20T00:34:23.943Z+ [[ -z '' ]]
2882023-10-20T00:34:23.996ZZFS zpool oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
2892023-10-20T00:34:24.024Z+ zpool create -f oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
2902023-10-20T00:34:24.048ZZpool: [oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03]
2912023-10-20T00:34:24.078Z+ success 'ZFS zpool oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
2922023-10-20T00:34:24.103ZZFS vdev /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
2932023-10-20T00:34:24.151Z+ set +x
2942023-10-20T00:34:24.177ZZFS zpool oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
2952023-10-20T00:34:24.202Z+ for ZPOOL in "${ZPOOLS[@]}"
2962023-10-20T00:34:24.226ZZpool: [oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03]
2972023-10-20T00:34:24.251Z+ echo 'Zpool: [oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03]'
2982023-10-20T00:34:24.275ZZFS vdev /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
2992023-10-20T00:34:24.300Z+ VDEV_PATH=/scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3002023-10-20T00:34:24.325ZZFS zpool oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
3012023-10-20T00:34:24.349Z+ [[ -f /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3022023-10-20T00:34:24.374ZZpool: [oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930]
3032023-10-20T00:34:24.401Z+ dd if=/dev/zero of=/scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3042023-10-20T00:34:24.429ZZFS vdev /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev exists
3052023-10-20T00:34:24.455Z0+0 records in
3062023-10-20T00:34:24.481ZZFS zpool oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 exists
3072023-10-20T00:34:24.506Z0+0 records out
3082023-10-20T00:34:24.532ZZpool: [oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60]
3092023-10-20T00:34:24.557Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3102023-10-20T00:34:24.582ZZFS vdev /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev exists
3112023-10-20T00:34:24.613Z+ success 'ZFS vdev /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3122023-10-20T00:34:24.640ZZFS zpool oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 exists
3132023-10-20T00:34:24.664Z+ set +x
3142023-10-20T00:34:24.689ZZpool: [oxp_31bd71cd-4736-4a12-a387-9b74b050396f]
3152023-10-20T00:34:24.713Z++ zpool list -o name
3162023-10-20T00:34:24.739ZZFS vdev /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev exists
3172023-10-20T00:34:24.764Z++ grep oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03
3182023-10-20T00:34:24.789ZZFS zpool oxp_31bd71cd-4736-4a12-a387-9b74b050396f exists
3192023-10-20T00:34:24.814Z+ [[ -z '' ]]
3202023-10-20T00:34:24.839ZZpool: [oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7]
3212023-10-20T00:34:24.864Z+ zpool create -f oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3222023-10-20T00:34:24.894ZZFS vdev /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev exists
3232023-10-20T00:34:24.921Z+ success 'ZFS zpool oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3242023-10-20T00:34:24.947ZZFS zpool oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 exists
3252023-10-20T00:34:24.972Z+ set +x
3262023-10-20T00:34:24.997ZZpool: [oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b]
3272023-10-20T00:34:25.430Z+ for ZPOOL in "${ZPOOLS[@]}"
3282023-10-20T00:34:25.860ZZFS vdev /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
3292023-10-20T00:34:25.885Z+ echo 'Zpool: [oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3302023-10-20T00:34:25.910ZZFS zpool oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b exists
3312023-10-20T00:34:25.935Z+ VDEV_PATH=/scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3322023-10-20T00:34:25.960ZZpool: [oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b]
3332023-10-20T00:34:25.985Z+ [[ -f /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3342023-10-20T00:34:26.010ZZFS vdev /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
3352023-10-20T00:34:26.034Z+ dd if=/dev/zero of=/scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3362023-10-20T00:34:26.059ZZFS zpool oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b exists
3372023-10-20T00:34:26.084Z0+0 records in
3382023-10-20T00:34:26.108ZSimnet net0/sc0_0 exists
3392023-10-20T00:34:26.133Z0+0 records out
3402023-10-20T00:34:26.159ZSimnet net1/sc1_0 exists
3412023-10-20T00:34:26.185Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
3422023-10-20T00:34:26.210ZVnic sc0_1 exists
3432023-10-20T00:34:26.235Z+ success 'ZFS vdev /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3442023-10-20T00:34:26.260Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/softnpu/image/c1c42398c82b0220c8b5fa3bfba9c7a3bcaa0943/softnpu.sha256.txt
3452023-10-20T00:34:26.284Z+ set +x
3462023-10-20T00:34:26.311Z++ zpool list -o name
3472023-10-20T00:34:26.338Z++ grep oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03
3482023-10-20T00:34:26.365Z+ [[ -z '' ]]
3492023-10-20T00:34:26.390Z+ zpool create -f oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3502023-10-20T00:34:26.414Z+ success 'ZFS zpool oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3512023-10-20T00:34:26.439Z+ set +x
3522023-10-20T00:34:26.463Z+ for ZPOOL in "${ZPOOLS[@]}"
3532023-10-20T00:34:26.488Z+ echo 'Zpool: [oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3542023-10-20T00:34:26.513Z+ VDEV_PATH=/scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3552023-10-20T00:34:26.538Z+ [[ -f /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3562023-10-20T00:34:26.562Z+ dd if=/dev/zero of=/scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3572023-10-20T00:34:26.587Z0+0 records in
3582023-10-20T00:34:26.611Z0+0 records out
3592023-10-20T00:34:26.635Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3602023-10-20T00:34:26.660Z+ success 'ZFS vdev /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3612023-10-20T00:34:26.685Z+ set +x
3622023-10-20T00:34:26.710Z++ zpool list -o name
3632023-10-20T00:34:26.734Z++ grep oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03
3642023-10-20T00:34:26.758Z+ [[ -z '' ]]
3652023-10-20T00:34:26.789Z+ zpool create -f oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3662023-10-20T00:34:26.813Z+ success 'ZFS zpool oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3672023-10-20T00:34:26.838Z+ set +x
3682023-10-20T00:34:26.863Z+ for ZPOOL in "${ZPOOLS[@]}"
3692023-10-20T00:34:26.887Z+ echo 'Zpool: [oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3702023-10-20T00:34:26.925Z+ VDEV_PATH=/scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3712023-10-20T00:34:26.953Z+ [[ -f /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3722023-10-20T00:34:26.981Z+ dd if=/dev/zero of=/scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3732023-10-20T00:34:27.005Z0+0 records in
3742023-10-20T00:34:27.034Z0+0 records out
3752023-10-20T00:34:27.058Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3762023-10-20T00:34:27.083Z+ success 'ZFS vdev /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3772023-10-20T00:34:27.107Z+ set +x
3782023-10-20T00:34:27.132Z++ zpool list -o name
3792023-10-20T00:34:27.157Z++ grep oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03
3802023-10-20T00:34:27.181Z+ [[ -z '' ]]
3812023-10-20T00:34:27.206Z+ zpool create -f oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3822023-10-20T00:34:27.231Z+ success 'ZFS zpool oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3832023-10-20T00:34:27.255Z+ set +x
3842023-10-20T00:34:27.279Z+ for ZPOOL in "${ZPOOLS[@]}"
3852023-10-20T00:34:27.304Z+ echo 'Zpool: [oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930]'
3862023-10-20T00:34:27.328Z+ VDEV_PATH=/scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev
3872023-10-20T00:34:27.352Z+ [[ -f /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev ]]
3882023-10-20T00:34:27.377Z+ dd if=/dev/zero of=/scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev bs=1 count=0 seek=20G
3892023-10-20T00:34:27.405Z0+0 records in
3902023-10-20T00:34:27.429Z0+0 records out
3912023-10-20T00:34:27.457Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3922023-10-20T00:34:27.487Z+ success 'ZFS vdev /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev exists'
3932023-10-20T00:34:27.514Z+ set +x
3942023-10-20T00:34:27.552Z++ zpool list -o name
3952023-10-20T00:34:27.581Z++ grep oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930
3962023-10-20T00:34:27.606Z+ [[ -z '' ]]
3972023-10-20T00:34:27.631Z+ zpool create -f oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev
3982023-10-20T00:34:27.656Z+ success 'ZFS zpool oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 exists'
3992023-10-20T00:34:27.681Z+ set +x
4002023-10-20T00:34:27.714Z+ for ZPOOL in "${ZPOOLS[@]}"
4012023-10-20T00:34:27.741Z+ echo 'Zpool: [oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60]'
4022023-10-20T00:34:27.766Z+ VDEV_PATH=/scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev
4032023-10-20T00:34:27.828Z+ [[ -f /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev ]]
4042023-10-20T00:34:27.863Z+ dd if=/dev/zero of=/scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev bs=1 count=0 seek=20G
4052023-10-20T00:34:27.889Z0+0 records in
4062023-10-20T00:34:27.916Z0+0 records out
4072023-10-20T00:34:27.940Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4082023-10-20T00:34:27.965Z+ success 'ZFS vdev /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev exists'
4092023-10-20T00:34:27.989Z+ set +x
4102023-10-20T00:34:28.014Z++ zpool list -o name
4112023-10-20T00:34:28.039Z++ grep oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60
4122023-10-20T00:34:28.065Z+ [[ -z '' ]]
4132023-10-20T00:34:28.089Z+ zpool create -f oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev
4142023-10-20T00:34:28.115Z+ success 'ZFS zpool oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 exists'
4152023-10-20T00:34:28.144Z+ set +x
4162023-10-20T00:34:28.174Z+ for ZPOOL in "${ZPOOLS[@]}"
4172023-10-20T00:34:28.199Z+ echo 'Zpool: [oxp_31bd71cd-4736-4a12-a387-9b74b050396f]'
4182023-10-20T00:34:28.224Z+ VDEV_PATH=/scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev
4192023-10-20T00:34:28.249Z+ [[ -f /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev ]]
4202023-10-20T00:34:28.274Z+ dd if=/dev/zero of=/scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev bs=1 count=0 seek=20G
4212023-10-20T00:34:28.304Z0+0 records in
4222023-10-20T00:34:28.331Z0+0 records out
4232023-10-20T00:34:28.355Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
4242023-10-20T00:34:28.380Z+ success 'ZFS vdev /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev exists'
4252023-10-20T00:34:28.404Z+ set +x
4262023-10-20T00:34:28.429Z++ zpool list -o name
4272023-10-20T00:34:28.453Z++ grep oxp_31bd71cd-4736-4a12-a387-9b74b050396f
4282023-10-20T00:34:28.478Z+ [[ -z '' ]]
4292023-10-20T00:34:28.503Z+ zpool create -f oxp_31bd71cd-4736-4a12-a387-9b74b050396f /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev
4302023-10-20T00:34:28.528Z+ success 'ZFS zpool oxp_31bd71cd-4736-4a12-a387-9b74b050396f exists'
4312023-10-20T00:34:28.554Z+ set +x
4322023-10-20T00:34:28.584Z+ for ZPOOL in "${ZPOOLS[@]}"
4332023-10-20T00:34:28.612Z+ echo 'Zpool: [oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7]'
4342023-10-20T00:34:28.638Z+ VDEV_PATH=/scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev
4352023-10-20T00:34:28.664Z+ [[ -f /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev ]]
4362023-10-20T00:34:28.689Z+ dd if=/dev/zero of=/scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev bs=1 count=0 seek=20G
4372023-10-20T00:34:28.714Z0+0 records in
4382023-10-20T00:34:28.755Z0+0 records out
4392023-10-20T00:34:28.782Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
4402023-10-20T00:34:28.807Z+ success 'ZFS vdev /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev exists'
4412023-10-20T00:34:28.864Z+ set +x
4422023-10-20T00:34:28.888Z++ zpool list -o name
4432023-10-20T00:34:28.939Z++ grep oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7
4442023-10-20T00:34:28.963Z+ [[ -z '' ]]
4452023-10-20T00:34:28.993Z+ zpool create -f oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev
4462023-10-20T00:34:29.022Z+ success 'ZFS zpool oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 exists'
4472023-10-20T00:34:29.047Z+ set +x
4482023-10-20T00:34:29.072Zdownloading softnpu ...done
4492023-10-20T00:34:29.098Z+ for ZPOOL_TYPE in "${ZPOOL_TYPES[@]}"
4502023-10-20T00:34:29.132Z+ readarray -t ZPOOLS
4512023-10-20T00:34:29.160Z++ grep '"oxi_' /opt/oxide/work/tools/../smf/sled-agent/non-gimlet/config.toml
4522023-10-20T00:34:29.190Z++ sed 's/[ ",]//g'
4532023-10-20T00:34:29.221Z+ for ZPOOL in "${ZPOOLS[@]}"
4542023-10-20T00:34:29.246Z+ echo 'Zpool: [oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b]'
4552023-10-20T00:34:29.272Z+ VDEV_PATH=/scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4562023-10-20T00:34:29.296Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/sidecar-lite/release/4ba9c18928edece80f83153b9d78b76cf093da19/libsidecar_lite.so.sha256.txt
4572023-10-20T00:34:29.321Z+ [[ -f /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
4582023-10-20T00:34:29.345Z+ dd if=/dev/zero of=/scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
4592023-10-20T00:34:29.370Z0+0 records in
4602023-10-20T00:34:29.394Z0+0 records out
4612023-10-20T00:34:29.419Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4622023-10-20T00:34:29.443Z+ success 'ZFS vdev /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
4632023-10-20T00:34:29.468Z+ set +x
4642023-10-20T00:34:29.495Z++ zpool list -o name
4652023-10-20T00:34:29.526Z++ grep oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b
4662023-10-20T00:34:29.554Z+ [[ -z '' ]]
4672023-10-20T00:34:29.579Z+ zpool create -f oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4682023-10-20T00:34:29.603Z+ success 'ZFS zpool oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
4692023-10-20T00:34:29.643Z+ set +x
4702023-10-20T00:34:29.675Z+ for ZPOOL in "${ZPOOLS[@]}"
4712023-10-20T00:34:29.699Z+ echo 'Zpool: [oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b]'
4722023-10-20T00:34:29.724Z+ VDEV_PATH=/scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4732023-10-20T00:34:29.749Z+ [[ -f /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
4742023-10-20T00:34:29.774Z+ dd if=/dev/zero of=/scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
4752023-10-20T00:34:29.800Z0+0 records in
4762023-10-20T00:34:29.824Z0+0 records out
4772023-10-20T00:34:29.849Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4782023-10-20T00:34:29.892Z+ success 'ZFS vdev /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
4792023-10-20T00:34:29.919Z+ set +x
4802023-10-20T00:34:29.943Z++ zpool list -o name
4812023-10-20T00:34:29.984Z++ grep oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b
4822023-10-20T00:34:30.013Z+ [[ -z '' ]]
4832023-10-20T00:34:30.048Z+ zpool create -f oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4842023-10-20T00:34:30.082Z+ success 'ZFS zpool oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
4852023-10-20T00:34:30.107Z+ set +x
4862023-10-20T00:34:30.132Z+ ensure_simulated_links igb0
4872023-10-20T00:34:30.164Z+ local PHYSICAL_LINK=igb0
4882023-10-20T00:34:30.192Z+ INDICES=("0" "1")
4892023-10-20T00:34:30.217Z+ for I in "${INDICES[@]}"
4902023-10-20T00:34:30.241Z++ get_simnet_name_if_exists net0
4912023-10-20T00:34:30.266Z++ dladm show-simnet -p -o LINK net0
4922023-10-20T00:34:30.291Z++ echo ''
4932023-10-20T00:34:30.315Z+ [[ -z '' ]]
4942023-10-20T00:34:30.342Z+ dladm create-simnet -t net0
4952023-10-20T00:34:30.368Z+ dladm create-simnet -t sc0_0
4962023-10-20T00:34:30.393Z+ dladm modify-simnet -t -p net0 sc0_0
4972023-10-20T00:34:30.418Z+ dladm set-linkprop -p mtu=9000 sc0_0
4982023-10-20T00:34:30.443Z+ success 'Simnet net0/sc0_0 exists'
4992023-10-20T00:34:30.468Z+ set +x
5002023-10-20T00:34:30.493Z+ for I in "${INDICES[@]}"
5012023-10-20T00:34:30.521Z++ get_simnet_name_if_exists net1
5022023-10-20T00:34:30.545Z++ dladm show-simnet -p -o LINK net1
5032023-10-20T00:34:30.573Z++ echo ''
5042023-10-20T00:34:30.598Z+ [[ -z '' ]]
5052023-10-20T00:34:30.623Z+ dladm create-simnet -t net1
5062023-10-20T00:34:30.648Z+ dladm create-simnet -t sc1_0
5072023-10-20T00:34:30.672Z+ dladm modify-simnet -t -p net1 sc1_0
5082023-10-20T00:34:30.697Z+ dladm set-linkprop -p mtu=9000 sc1_0
5092023-10-20T00:34:30.721Z+ success 'Simnet net1/sc1_0 exists'
5102023-10-20T00:34:30.750Z+ set +x
5112023-10-20T00:34:30.776Z++ get_vnic_name_if_exists sc0_1
5122023-10-20T00:34:30.805Z++ dladm show-vnic -p -o LINK sc0_1
5132023-10-20T00:34:30.831Z++ echo ''
5142023-10-20T00:34:30.856Z+ [[ -z '' ]]
5152023-10-20T00:34:30.885Z+ dladm create-vnic -t sc0_1 -l igb0 -m a8:e1:de:01:70:1d
5162023-10-20T00:34:30.991Z+ success 'Vnic sc0_1 exists'
5172023-10-20T00:34:31.016Z+ set +x
5182023-10-20T00:34:31.041Z+ warn_if_no_proxy_arp
5192023-10-20T00:34:31.066Z+ [[ -v PXA_START ]]
5202023-10-20T00:34:31.091Z+ [[ -v PXA_END ]]
5212023-10-20T00:34:31.115Z+ ensure_softnpu_zone
5222023-10-20T00:34:31.140Z+ zoneadm list
5232023-10-20T00:34:31.165Z+ grep -q sidecar_softnpu
5242023-10-20T00:34:31.190Z+ [[ -f out/npuzone/npuzone ]]
5252023-10-20T00:34:31.215Z+ out/npuzone/npuzone create sidecar --omicron-zone --ports sc0_0,tfportrear0_0 --ports sc0_1,tfportqsfp0_0
5262023-10-20T00:34:33.436Zdownloading asic_program.so ...done
5272023-10-20T00:34:33.484Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/sidecar-lite/release/4ba9c18928edece80f83153b9d78b76cf093da19/scadm.sha256.txt
5282023-10-20T00:34:38.298Zdownloading scadm ...done
5292023-10-20T00:34:38.325Zcreating port tfportrear0_0 for sc0_0
5302023-10-20T00:34:38.353Zcreating port tfportqsfp0_0 for sc0_1
5312023-10-20T00:34:38.382Zsoftnpu zone setup
5322023-10-20T00:34:38.411Zconfigure zone
5332023-10-20T00:34:38.444Zinstall zone
5342023-10-20T00:34:39.382Zboot zone
5352023-10-20T00:34:42.496Z[sidecar_softnpu] svcs milestone/network
5362023-10-20T00:34:42.524ZSTATE STIME FMRI
5372023-10-20T00:34:42.548Zonline 0:34:40 svc:/milestone/network:default
5382023-10-20T00:34:42.575Z[sidecar_softnpu] /softnpu/npu start
5392023-10-20T00:34:42.599Z
5402023-10-20T00:34:43.038Z+ /opt/oxide/work/tools/scrimlet/softnpu-init.sh
5412023-10-20T00:34:43.064Z+ GATEWAY_IP=10.151.1.100
5422023-10-20T00:34:43.089Z+ echo 'Using 10.151.1.100 as gateway ip'
5432023-10-20T00:34:43.114ZUsing 10.151.1.100 as gateway ip
5442023-10-20T00:34:43.138Z+ [[ ! -v GATEWAY_MAC ]]
5452023-10-20T00:34:43.163Z+ GATEWAY_MAC=18:c0:4d:0d:9f:b2
5462023-10-20T00:34:43.187Z++ grep -c -E '^([0-9a-fA-F]{2}:){5}[0-9a-fA-F]{2}$' /dev/fd/63
5472023-10-20T00:34:43.211Z+++ echo 18:c0:4d:0d:9f:b2
5482023-10-20T00:34:43.249Z+ COUNT=1
5492023-10-20T00:34:43.275Z+ [[ 1 -ne 1 ]]
5502023-10-20T00:34:43.301Z+ echo 'Using 18:c0:4d:0d:9f:b2 as gateway mac'
5512023-10-20T00:34:43.341ZUsing 18:c0:4d:0d:9f:b2 as gateway mac
5522023-10-20T00:34:43.371Z+ z_scadm add-arp-entry 10.151.1.100 18:c0:4d:0d:9f:b2
5532023-10-20T00:34:43.395Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone add-arp-entry 10.151.1.100 18:c0:4d:0d:9f:b2
5542023-10-20T00:34:43.420Z+ PXA_MAC=a8:e1:de:01:70:1d
5552023-10-20T00:34:43.445Z+ [[ -v PXA_START ]]
5562023-10-20T00:34:43.470Z+ z_scadm add-proxy-arp 10.151.1.164 10.151.1.227 a8:e1:de:01:70:1d
5572023-10-20T00:34:43.494Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone add-proxy-arp 10.151.1.164 10.151.1.227 a8:e1:de:01:70:1d
5582023-10-20T00:34:43.519Z+ z_scadm dump-state
5592023-10-20T00:34:43.544Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone dump-state
5602023-10-20T00:34:43.569Zlocal v6:
5612023-10-20T00:34:43.593Zlocal v4:
5622023-10-20T00:34:43.618Zrouter v6:
5632023-10-20T00:34:43.646Zrouter v4:
5642023-10-20T00:34:43.671Zresolver v4:
5652023-10-20T00:34:43.696Z10.151.1.100 -> 18:c0:4d:0d:9f:b2
5662023-10-20T00:34:43.721Zresolver v6:
5672023-10-20T00:34:43.746Znat v4:
5682023-10-20T00:34:43.773Znat_v6:
5692023-10-20T00:34:43.799Zport_mac:
5702023-10-20T00:34:43.824Zproxy arp:
5712023-10-20T00:34:43.849Z10.151.1.164/10.151.1.227: a8:e1:de:01:70:1d
5722023-10-20T00:34:43.874Z+ success 'softnpu zone exists'
5732023-10-20T00:34:43.901Z+ set +x
5742023-10-20T00:34:43.958Zsoftnpu zone exists
5752023-10-20T00:34:43.984Z
5762023-10-20T00:34:44.010Zreal 22.387637339
5772023-10-20T00:34:44.036Zuser 0.717510725
5782023-10-20T00:34:44.061Zsys 1.630067400
5792023-10-20T00:34:44.086Ztrap 0.000078938
5802023-10-20T00:34:44.111Ztflt 0.000266427
5812023-10-20T00:34:44.139Zdflt 0.001064502
5822023-10-20T00:34:44.164Zkflt 0.000000000
5832023-10-20T00:34:44.189Zlock 17:22.677677267
5842023-10-20T00:34:44.214Zslp 1:06.343352375
5852023-10-20T00:34:44.239Zlat 0.079948520
5862023-10-20T00:34:44.264Zstop 0.358025345
5872023-10-20T00:34:44.288Z+ tar xf out/omicron-sled-agent.tar pkg/config-rss.toml
5882023-10-20T00:34:44.313Z++ sed -n 's/silo_name = "\(.*\)"/\1/p' pkg/config-rss.toml
5892023-10-20T00:34:44.337Z+ SILO_NAME=recovery
5902023-10-20T00:34:44.365Z++ sed -n 's/external_dns_zone_name = "\(.*\)"/\1/p' pkg/config-rss.toml
5912023-10-20T00:34:44.395Z+ EXTERNAL_DNS_DOMAIN=oxide.test
5922023-10-20T00:34:44.433Z+ sed -i~ '
5932023-10-20T00:34:44.457Z /^external_dns_ips/c\
5942023-10-20T00:34:44.482Zexternal_dns_ips = [ "10.151.1.164", "10.151.1.165" ]
5952023-10-20T00:34:44.506Z /^\[\[internal_services_ip_pool_ranges/,/^$/ {
5962023-10-20T00:34:44.531Z /^first/c\
5972023-10-20T00:34:44.556Zfirst = "10.151.1.164"
5982023-10-20T00:34:44.584Z /^last/c\
5992023-10-20T00:34:45.042Zlast = "10.151.1.173"
6002023-10-20T00:34:45.069Z }
6012023-10-20T00:34:45.094Z /^\[rack_network_config/,/^$/ {
6022023-10-20T00:34:45.123Z /^infra_ip_first/c\
6032023-10-20T00:34:45.148Zinfra_ip_first = "10.151.1.174"
6042023-10-20T00:34:45.173Z /^infra_ip_last/c\
6052023-10-20T00:34:45.198Zinfra_ip_last = "10.151.1.174"
6062023-10-20T00:34:45.222Z--- pkg/config-rss.toml~ Mon Jul 24 01:21:28 2006
6072023-10-20T00:34:45.247Z }
6082023-10-20T00:34:45.272Z+++ pkg/config-rss.toml Fri Oct 20 00:34:42 2023
6092023-10-20T00:34:45.296Z /^\[\[rack_network_config.uplinks/,/^$/ {
6102023-10-20T00:34:45.321Z@@ -41,7 +41,7 @@
6112023-10-20T00:34:45.345Z /^gateway_ip/c\
6122023-10-20T00:34:45.402Z # the DNS domain delegated to the rack by the customer. Each of these addresses
6132023-10-20T00:34:45.426Zgateway_ip = "10.151.1.100"
6142023-10-20T00:34:45.451Z # must be contained in one of the "internal services" IP Pool ranges listed
6152023-10-20T00:34:45.476Z /^uplink_cidr/c\
6162023-10-20T00:34:45.501Z # below.
6172023-10-20T00:34:45.525Zuplink_cidr = "10.151.1.174/32"
6182023-10-20T00:34:45.556Z-external_dns_ips = [ "192.168.1.20", "192.168.1.21" ]
6192023-10-20T00:34:45.582Z }
6202023-10-20T00:34:45.606Z+external_dns_ips = [ "10.151.1.164", "10.151.1.165" ]
6212023-10-20T00:34:45.631Z' pkg/config-rss.toml
6222023-10-20T00:34:45.656Z
6232023-10-20T00:34:45.680Z+ diff -u pkg/config-rss.toml~ pkg/config-rss.toml
6242023-10-20T00:34:45.706Z # Initial TLS certificates for the external API
6252023-10-20T00:34:45.730Z+ true
6262023-10-20T00:34:45.755Z #
6272023-10-20T00:34:45.780Z+ tar rvf out/omicron-sled-agent.tar pkg/config-rss.toml
6282023-10-20T00:34:45.804Z@@ -77,8 +77,8 @@
6292023-10-20T00:34:45.829Z+ rm -f pkg/config-rss.toml pkg/config-rss.toml~
6302023-10-20T00:34:45.853Z #
6312023-10-20T00:34:45.878Z+ TLS_NAME=recovery.sys.oxide.test
6322023-10-20T00:34:45.902Z # For more on this and what to put here, see docs/how-to-run.adoc.
6332023-10-20T00:34:45.927Z+ openssl req -newkey rsa:4096 -x509 -sha256 -days 3 -nodes -out pkg/initial-tls-cert.pem -keyout pkg/initial-tls-key.pem -subj /CN=recovery.sys.oxide.test -addext subjectAltName=DNS:recovery.sys.oxide.test -addext basicConstraints=critical,CA:FALSE -config /dev/stdin
6342023-10-20T00:34:45.953Z [[internal_services_ip_pool_ranges]]
6352023-10-20T00:34:45.979Z.....+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.........+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*...+....................+...+...+....+...+...........+............+...+.+...+...+...+......+...........+..........+..................+..+............+.+......+...........+...+....+...............+.....+...............+...+...+.+......+............+........+...+.......+............+........+...+.......+..+.......+.........+..........................+......+....+...+......+..+...+.........+..........+............+...+.....+......+.......+..............+.+...+..+...+....+.....+............+.........+.......+........................+...+......+......+........+......+......+.+..+.............+........+.......+............+..+......+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
6362023-10-20T00:34:46.003Z-first = "192.168.1.20"
6372023-10-20T00:34:46.028Z..............+.+...+..+......+.......+.....+.+..+.......+.....+......+.+......+..+.......+.....+.+..+.+...........+.........+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*..+......+.+............+......+..+...+............+.........+...+..........+.....+.+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*...+..................+...+...............+.....+......+...+.+.........+........+....+..............+....+...+...+...........+..........+......+.....+...+.+.....+...............+.+........+.+.....+...+..............................+.+..+............+...+....+......+...+...........+.+..+......................+.....+.........+...................+..+...+...+.......+..+....+.............................+.......+........+......+....+........+...+............+.......+...+..+.........................+.....+.............+...............+.....+.........+.........+............+...+......+.............+......+.....+....+............+......+..+.......+.....+..................+......+....+...+..............+.+.....+.......+.....+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
6382023-10-20T00:34:46.053Z-last = "192.168.1.29"
6392023-10-20T00:34:46.078Z-----
6402023-10-20T00:34:46.103Z+first = "10.151.1.164"
6412023-10-20T00:34:46.127Z+ tar rvf out/omicron-sled-agent.tar pkg/initial-tls-cert.pem pkg/initial-tls-key.pem
6422023-10-20T00:34:46.152Z+last = "10.151.1.173"
6432023-10-20T00:34:46.579Z+ rm -f pkg/initial-tls-cert.pem pkg/initial-tls-key.pem
6442023-10-20T00:34:46.604Z
6452023-10-20T00:34:46.629Z+ rmdir pkg
6462023-10-20T00:34:46.653Z # TODO - this configuration is subject to change going forward. Ultimately these
6472023-10-20T00:34:46.717Z+ E2E_TLS_CERT=/opt/oxide/sled-agent/pkg/initial-tls-cert.pem
6482023-10-20T00:34:46.742Z # parameters should be provided to the control plane via wicket, but we need to
6492023-10-20T00:34:46.767Z+ pfexec mkdir -p /usr/oxide
6502023-10-20T00:34:46.792Z@@ -91,13 +91,13 @@
6512023-10-20T00:34:46.825Z+ pfexec rm -f /usr/oxide/tcpproxy
6522023-10-20T00:34:46.850Z # A range of IP addresses used by Boundary Services on the external network. In
6532023-10-20T00:34:46.874Z+ pfexec curl -sSfL -o /usr/oxide/tcpproxy http://catacomb.eng.oxide.computer:12346/tcpproxy
6542023-10-20T00:34:46.907Z # a real system, these would be addresses of the uplink ports on the Sidecar.
6552023-10-20T00:34:46.933Z+ pfexec chmod +x /usr/oxide/tcpproxy
6562023-10-20T00:34:46.957Z # With softnpu, only one address is used.
6572023-10-20T00:34:46.982Z+ pfexec rm -f /var/svc/manifest/site/tcpproxy.xml
6582023-10-20T00:34:47.007Z-infra_ip_first = "192.168.1.30"
6592023-10-20T00:34:47.031Z+ pfexec curl -sSfL -o /var/svc/manifest/site/tcpproxy.xml http://catacomb.eng.oxide.computer:12346/tcpproxy.xml
6602023-10-20T00:34:47.056Z-infra_ip_last = "192.168.1.30"
6612023-10-20T00:34:47.080Z+ pfexec svccfg import /var/svc/manifest/site/tcpproxy.xml
6622023-10-20T00:34:47.105Z+infra_ip_first = "10.151.1.174"
6632023-10-20T00:34:47.129Z+ OMICRON_NO_UNINSTALL=1
6642023-10-20T00:34:47.154Z+infra_ip_last = "10.151.1.174"
6652023-10-20T00:34:47.179Z+ ptime -m pfexec ./target/release/omicron-package -t test install
6662023-10-20T00:34:47.203Z
6672023-10-20T00:34:47.638ZOct 20 00:34:44.293 DEBG target[test]: Target({"image": "standard", "machine": "non-gimlet", "rack-topology": "single-sled", "switch": "softnpu"})
6682023-10-20T00:34:47.664Z # You can configure multiple uplinks by repeating the following stanza
6692023-10-20T00:34:47.688ZOct 20 00:34:44.295 INFO Installing service, dst: /opt/oxide/clickhouse.tar.gz, src: out/clickhouse.tar.gz
6702023-10-20T00:34:47.713Z [[rack_network_config.uplinks]]
6712023-10-20T00:34:47.744ZOct 20 00:34:44.295 INFO Installing service, dst: /opt/oxide/ntp.tar.gz, src: out/ntp.tar.gz
6722023-10-20T00:34:47.772Z # The gateway IP for the rack's external network
6732023-10-20T00:34:47.799ZOct 20 00:34:44.295 INFO Installing service, dst: /opt/oxide/oximeter.tar.gz, src: out/oximeter-collector.tar.gz
6742023-10-20T00:34:47.824Z-gateway_ip = "192.168.1.199"
6752023-10-20T00:34:47.849ZOct 20 00:34:44.295 INFO Installing service, dst: /opt/oxide/nexus.tar.gz, src: out/omicron-nexus.tar.gz
6762023-10-20T00:34:47.874Z+gateway_ip = "10.151.1.100"
6772023-10-20T00:34:47.899ZOct 20 00:34:44.295 INFO Installing service, dst: /opt/oxide/internal_dns.tar.gz, src: out/internal-dns.tar.gz
6782023-10-20T00:34:47.924Z # Name of the uplink port. This should always be "qsfp0" when using softnpu.
6792023-10-20T00:34:47.953ZOct 20 00:34:44.295 INFO Installing service, dst: /opt/oxide/cockroachdb.tar.gz, src: out/cockroachdb.tar.gz
6802023-10-20T00:34:47.979Z uplink_port = "qsfp0"
6812023-10-20T00:34:48.003ZOct 20 00:34:44.295 INFO Installing service, dst: /opt/oxide/clickhouse_keeper.tar.gz, src: out/clickhouse_keeper.tar.gz
6822023-10-20T00:34:48.028Z uplink_port_speed = "40G"
6832023-10-20T00:34:48.052ZOct 20 00:34:44.296 INFO Installing service, dst: /opt/oxide/overlay.tar.gz, src: out/overlay.tar.gz
6842023-10-20T00:34:48.078Z@@ -104,7 +104,7 @@
6852023-10-20T00:34:48.105ZOct 20 00:34:44.296 INFO Installing service, dst: /opt/oxide/crucible_pantry.tar.gz, src: out/crucible-pantry.tar.gz
6862023-10-20T00:34:48.131Z uplink_port_fec="none"
6872023-10-20T00:34:48.156ZOct 20 00:34:44.296 INFO Installing service, dst: /opt/oxide/propolis-server.tar.gz, src: out/propolis-server.tar.gz
6882023-10-20T00:34:48.182Z # For softnpu, an address within the "infra" block above that will be used for
6892023-10-20T00:34:48.208ZOct 20 00:34:44.296 INFO Installing service, dst: /opt/oxide/mgs.tar.gz, src: out/omicron-gateway-softnpu.tar.gz
6902023-10-20T00:34:48.233Z # the softnpu uplink port. You can just pick the first address in that pool.
6912023-10-20T00:34:48.258ZOct 20 00:34:44.296 INFO Installing service, dst: /opt/oxide/sled-agent.tar, src: out/omicron-sled-agent.tar
6922023-10-20T00:34:48.283Z-uplink_cidr = "192.168.1.30/32"
6932023-10-20T00:34:48.307ZOct 20 00:34:44.296 INFO Installing service, dst: /opt/oxide/mg-ddm.tar, src: out/maghemite.tar
6942023-10-20T00:34:48.332Z+uplink_cidr = "10.151.1.174/32"
6952023-10-20T00:34:48.357ZOct 20 00:34:44.296 INFO Installing service, dst: /opt/oxide/switch.tar.gz, src: out/switch-softnpu.tar.gz
6962023-10-20T00:34:48.394Z # Switch to use for the uplink. For single-rack deployments this can be
6972023-10-20T00:34:48.421ZOct 20 00:34:44.296 INFO Installing service, dst: /opt/oxide/external_dns.tar.gz, src: out/external-dns.tar.gz
6982023-10-20T00:34:48.446Z # "switch0" (upper slot) or "switch1" (lower slot). For single-node softnpu
6992023-10-20T00:34:48.476ZOct 20 00:34:44.296 INFO Installing service, dst: /opt/oxide/crucible.tar.gz, src: out/crucible.tar.gz
7002023-10-20T00:34:48.500Z # and dendrite stub environments, use "switch0"
7012023-10-20T00:34:48.525ZOct 20 00:34:44.544 INFO Unpacking service tarball, service_path: /opt/oxide/mg-ddm, tar_path: /opt/oxide/mg-ddm.tar
7022023-10-20T00:34:48.549Za pkg/config-rss.toml 7K
7032023-10-20T00:34:48.574ZOct 20 00:34:44.586 INFO Unpacking service tarball, service_path: /opt/oxide/sled-agent, tar_path: /opt/oxide/sled-agent.tar
7042023-10-20T00:34:48.605Za pkg/initial-tls-cert.pem 2K
7052023-10-20T00:34:48.631ZOct 20 00:34:44.651 INFO Installing bootstrap service from /opt/oxide/sled-agent/pkg/manifest.xml
7062023-10-20T00:34:48.655Za pkg/initial-tls-key.pem 4K
7072023-10-20T00:34:48.698Z
7082023-10-20T00:34:48.723Zreal 0.498578287
7092023-10-20T00:34:48.748Zuser 0.096591323
7102023-10-20T00:34:48.772Zsys 1.989059725
7112023-10-20T00:34:48.797Ztrap 0.000004230
7122023-10-20T00:34:48.821Ztflt 0.000014669
7132023-10-20T00:34:48.846Zdflt 0.000241375
7142023-10-20T00:34:48.871Zkflt 0.000002420
7152023-10-20T00:34:48.895Zlock 44.966326775
7162023-10-20T00:34:48.946Zslp 0.713744104
7172023-10-20T00:34:48.971Zlat 0.002696756
7182023-10-20T00:34:48.996Zstop 0.206265737
7192023-10-20T00:34:49.029Z+ retry=0
7202023-10-20T00:34:49.054Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7212023-10-20T00:34:49.079Z+ [[ 0 -gt 30 ]]
7222023-10-20T00:34:49.103Z+ sleep 1
7232023-10-20T00:34:49.128Z+ retry=1
7242023-10-20T00:34:49.152Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7252023-10-20T00:34:49.177Z+ [[ 1 -gt 30 ]]
7262023-10-20T00:34:49.205Z+ sleep 1
7272023-10-20T00:34:49.231Z+ retry=2
7282023-10-20T00:34:49.260Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7292023-10-20T00:35:06.250Z+ [[ 2 -gt 30 ]]
7302023-10-20T00:35:06.275Z+ sleep 1
7312023-10-20T00:35:07.253Z+ retry=3
7322023-10-20T00:35:07.278Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7332023-10-20T00:35:07.303Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone dump-state
7342023-10-20T00:35:07.332Zlocal v6:
7352023-10-20T00:35:07.358Zfe80::80ef:d7ff:fe64:b18b
7362023-10-20T00:35:07.383Zlocal v4:
7372023-10-20T00:35:07.408Zrouter v6:
7382023-10-20T00:35:07.433Zrouter v4:
7392023-10-20T00:35:07.458Zresolver v4:
7402023-10-20T00:35:07.482Z10.151.1.100 -> 18:c0:4d:0d:9f:b2
7412023-10-20T00:35:07.507Zresolver v6:
7422023-10-20T00:35:07.532Zfe80::78a2:5bff:fe48:d5d3 -> 7a:a2:5b:48:d5:d3
7432023-10-20T00:35:07.557Zfe80::80ef:d7ff:fe64:b18b -> 82:ef:d7:64:b1:8b
7442023-10-20T00:35:07.582Znat v4:
7452023-10-20T00:35:07.607Znat_v6:
7462023-10-20T00:35:07.631Zport_mac:
7472023-10-20T00:35:07.656Z1: a8:40:25:62:42:fd
7482023-10-20T00:35:07.680Z1000: a8:40:25:62:42:fc
7492023-10-20T00:35:07.705Zproxy arp:
7502023-10-20T00:35:07.730Z10.151.1.164/10.151.1.227: a8:e1:de:01:70:1d
7512023-10-20T00:35:07.764Z+ retry=0
7522023-10-20T00:35:07.789Z+++ zoneadm list -n
7532023-10-20T00:35:07.813Z+++ grep oxz_ntp
7542023-10-20T00:35:07.842Z++ pfexec svcs -z -Hostate oxide/ntp
7552023-10-20T00:35:07.870Zsvcs: zone '-Hostate': entity not found
7562023-10-20T00:35:07.903Z++ true
7572023-10-20T00:35:07.931Z+ [[ '' != online ]]
7582023-10-20T00:35:07.977Z+ [[ 0 -gt 60 ]]
7592023-10-20T00:35:08.002Z+ sleep 1
7602023-10-20T00:35:08.321Z+ retry=1
7612023-10-20T00:35:08.349Z+++ zoneadm list -n
7622023-10-20T00:35:08.374Z+++ grep oxz_ntp
7632023-10-20T00:35:08.399Z++ pfexec svcs -z -Hostate oxide/ntp
7642023-10-20T00:35:08.424Zsvcs: zone '-Hostate': entity not found
7652023-10-20T00:35:08.449Z++ true
7662023-10-20T00:35:08.474Z+ [[ '' != online ]]
7672023-10-20T00:35:08.499Z+ [[ 1 -gt 60 ]]
7682023-10-20T00:35:08.524Z+ sleep 1
7692023-10-20T00:35:09.337Z+ retry=2
7702023-10-20T00:35:09.363Z+++ zoneadm list -n
7712023-10-20T00:35:09.388Z+++ grep oxz_ntp
7722023-10-20T00:35:09.412Z++ pfexec svcs -z -Hostate oxide/ntp
7732023-10-20T00:35:09.439Zsvcs: zone '-Hostate': entity not found
7742023-10-20T00:35:09.464Z++ true
7752023-10-20T00:35:09.489Z+ [[ '' != online ]]
7762023-10-20T00:35:09.514Z+ [[ 2 -gt 60 ]]
7772023-10-20T00:35:09.539Z+ sleep 1
7782023-10-20T00:35:10.359Z+ retry=3
7792023-10-20T00:35:10.384Z+++ zoneadm list -n
7802023-10-20T00:35:10.409Z+++ grep oxz_ntp
7812023-10-20T00:35:10.433Z++ pfexec svcs -z -Hostate oxide/ntp
7822023-10-20T00:35:10.458Zsvcs: zone '-Hostate': entity not found
7832023-10-20T00:35:10.483Z++ true
7842023-10-20T00:35:10.509Z+ [[ '' != online ]]
7852023-10-20T00:35:10.542Z+ [[ 3 -gt 60 ]]
7862023-10-20T00:35:10.570Z+ sleep 1
7872023-10-20T00:35:11.381Z+ retry=4
7882023-10-20T00:35:11.406Z+++ zoneadm list -n
7892023-10-20T00:35:11.431Z+++ grep oxz_ntp
7902023-10-20T00:35:11.456Z++ pfexec svcs -z -Hostate oxide/ntp
7912023-10-20T00:35:11.480Zsvcs: zone '-Hostate': entity not found
7922023-10-20T00:35:11.505Z++ true
7932023-10-20T00:35:11.530Z+ [[ '' != online ]]
7942023-10-20T00:35:11.556Z+ [[ 4 -gt 60 ]]
7952023-10-20T00:35:11.580Z+ sleep 1
7962023-10-20T00:35:12.402Z+ retry=5
7972023-10-20T00:35:12.428Z+++ zoneadm list -n
7982023-10-20T00:35:12.456Z+++ grep oxz_ntp
7992023-10-20T00:35:12.481Z++ pfexec svcs -z -Hostate oxide/ntp
8002023-10-20T00:35:12.506Zsvcs: zone '-Hostate': entity not found
8012023-10-20T00:35:12.531Z++ true
8022023-10-20T00:35:12.557Z+ [[ '' != online ]]
8032023-10-20T00:35:12.582Z+ [[ 5 -gt 60 ]]
8042023-10-20T00:35:12.608Z+ sleep 1
8052023-10-20T00:35:13.422Z+ retry=6
8062023-10-20T00:35:13.447Z+++ zoneadm list -n
8072023-10-20T00:35:13.472Z+++ grep oxz_ntp
8082023-10-20T00:35:13.507Z++ pfexec svcs -z -Hostate oxide/ntp
8092023-10-20T00:35:13.538Zsvcs: zone '-Hostate': entity not found
8102023-10-20T00:35:13.567Z++ true
8112023-10-20T00:35:13.592Z+ [[ '' != online ]]
8122023-10-20T00:35:14.026Z+ [[ 6 -gt 60 ]]
8132023-10-20T00:35:14.051Z+ sleep 1
8142023-10-20T00:35:14.443Z+ retry=7
8152023-10-20T00:35:14.468Z+++ zoneadm list -n
8162023-10-20T00:35:14.493Z+++ grep oxz_ntp
8172023-10-20T00:35:14.518Z++ pfexec svcs -z -Hostate oxide/ntp
8182023-10-20T00:35:14.543Zsvcs: zone '-Hostate': entity not found
8192023-10-20T00:35:14.568Z++ true
8202023-10-20T00:35:14.593Z+ [[ '' != online ]]
8212023-10-20T00:35:14.618Z+ [[ 7 -gt 60 ]]
8222023-10-20T00:35:14.644Z+ sleep 1
8232023-10-20T00:35:15.467Z+ retry=8
8242023-10-20T00:35:15.493Z+++ zoneadm list -n
8252023-10-20T00:35:15.518Z+++ grep oxz_ntp
8262023-10-20T00:35:15.542Z++ pfexec svcs -z -Hostate oxide/ntp
8272023-10-20T00:35:15.567Zsvcs: zone '-Hostate': entity not found
8282023-10-20T00:35:15.591Z++ true
8292023-10-20T00:35:15.616Z+ [[ '' != online ]]
8302023-10-20T00:35:15.640Z+ [[ 8 -gt 60 ]]
8312023-10-20T00:35:15.665Z+ sleep 1
8322023-10-20T00:35:16.490Z+ retry=9
8332023-10-20T00:35:16.519Z+++ zoneadm list -n
8342023-10-20T00:35:16.550Z+++ grep oxz_ntp
8352023-10-20T00:35:16.579Z++ pfexec svcs -z -Hostate oxide/ntp
8362023-10-20T00:35:16.604Zsvcs: zone '-Hostate': entity not found
8372023-10-20T00:35:16.631Z++ true
8382023-10-20T00:35:16.656Z+ [[ '' != online ]]
8392023-10-20T00:35:16.681Z+ [[ 9 -gt 60 ]]
8402023-10-20T00:35:16.705Z+ sleep 1
8412023-10-20T00:35:17.509Z+ retry=10
8422023-10-20T00:35:17.541Z+++ zoneadm list -n
8432023-10-20T00:35:17.572Z+++ grep oxz_ntp
8442023-10-20T00:35:17.604Z++ pfexec svcs -z -Hostate oxide/ntp
8452023-10-20T00:35:17.632Zsvcs: zone '-Hostate': entity not found
8462023-10-20T00:35:17.657Z++ true
8472023-10-20T00:35:17.682Z+ [[ '' != online ]]
8482023-10-20T00:35:17.707Z+ [[ 10 -gt 60 ]]
8492023-10-20T00:35:17.733Z+ sleep 1
8502023-10-20T00:35:18.584Z+ retry=11
8512023-10-20T00:35:32.022Z+++ zoneadm list -n
8522023-10-20T00:35:33.354Z+++ grep oxz_ntp
8532023-10-20T00:35:33.538Z++ pfexec svcs -z -Hostate oxide/ntp
 
8542023-10-20T00:35:33.640Zbuskin console login: Oct 20 00:35:32 buskin ip: ip: joining multicasts failed (4) on vopte0 - will use link layer broadcasts for multicast
 
8552023-10-20T00:35:33.794Zsvcs: zone '-Hostate': entity not found
8562023-10-20T00:35:33.931Z++ true
8572023-10-20T00:35:33.967Z+ [[ '' != online ]]
8582023-10-20T00:35:34.131Z+ [[ 11 -gt 60 ]]
8592023-10-20T00:35:34.162Z+ sleep 1
8602023-10-20T00:35:34.240Z+ retry=12
8612023-10-20T00:35:34.289Z+++ zoneadm list -n
8622023-10-20T00:35:34.404Z+++ grep oxz_ntp
8632023-10-20T00:35:34.471Z++ pfexec svcs -z -Hostate oxide/ntp
8642023-10-20T00:35:34.605Zsvcs: zone '-Hostate': entity not found
8652023-10-20T00:35:34.683Z++ true
8662023-10-20T00:35:34.719Z+ [[ '' != online ]]
8672023-10-20T00:35:34.747Z+ [[ 12 -gt 60 ]]
8682023-10-20T00:35:34.821Z+ sleep 1
8692023-10-20T00:35:34.846Z+ retry=13
8702023-10-20T00:35:34.879Z+++ zoneadm list -n
8712023-10-20T00:35:34.912Z+++ grep oxz_ntp
8722023-10-20T00:35:34.945Z++ pfexec svcs -z -Hostate oxide/ntp
8732023-10-20T00:35:34.975Zsvcs: zone '-Hostate': entity not found
8742023-10-20T00:35:35.006Z++ true
8752023-10-20T00:35:35.032Z+ [[ '' != online ]]
8762023-10-20T00:35:35.068Z+ [[ 13 -gt 60 ]]
8772023-10-20T00:35:35.100Z+ sleep 1
8782023-10-20T00:35:35.125Z+ retry=14
8792023-10-20T00:35:35.159Z+++ zoneadm list -n
8802023-10-20T00:35:35.184Z+++ grep oxz_ntp
8812023-10-20T00:35:35.222Z++ pfexec svcs -z oxz_ntp_5585bc36-20c5-4420-b2c7-196898249e8f -Hostate oxide/ntp
8822023-10-20T00:35:35.248Zsvcs: Could not bind to repository server for zone oxz_ntp_5585bc36-20c5-4420-b2c7-196898249e8f: repository server unavailable
8832023-10-20T00:35:35.275Z++ true
8842023-10-20T00:35:35.314Z+ [[ '' != online ]]
8852023-10-20T00:35:35.352Z+ [[ 14 -gt 60 ]]
8862023-10-20T00:35:35.383Z+ sleep 1
8872023-10-20T00:35:35.414Z+ retry=15
8882023-10-20T00:35:35.442Z+++ zoneadm list -n
8892023-10-20T00:35:35.493Z+++ grep oxz_ntp
8902023-10-20T00:35:35.536Z++ pfexec svcs -z oxz_ntp_5585bc36-20c5-4420-b2c7-196898249e8f -Hostate oxide/ntp
8912023-10-20T00:35:35.628Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
8922023-10-20T00:35:35.666Z++ true
8932023-10-20T00:35:35.691Z+ [[ '' != online ]]
8942023-10-20T00:35:35.717Z+ [[ 15 -gt 60 ]]
8952023-10-20T00:35:35.749Z+ sleep 1
8962023-10-20T00:35:35.789Z+ retry=16
8972023-10-20T00:35:35.817Z+++ zoneadm list -n
8982023-10-20T00:35:35.852Z+++ grep oxz_ntp
8992023-10-20T00:35:35.887Z++ pfexec svcs -z oxz_ntp_5585bc36-20c5-4420-b2c7-196898249e8f -Hostate oxide/ntp
9002023-10-20T00:35:35.927Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
9012023-10-20T00:35:35.957Z++ true
9022023-10-20T00:35:36.007Z+ [[ '' != online ]]
9032023-10-20T00:35:36.049Z+ [[ 16 -gt 60 ]]
9042023-10-20T00:35:36.077Z+ sleep 1
9052023-10-20T00:35:36.107Z+ retry=17
9062023-10-20T00:35:36.136Z+++ zoneadm list -n
9072023-10-20T00:35:36.161Z+++ grep oxz_ntp
9082023-10-20T00:35:36.187Z++ pfexec svcs -z oxz_ntp_5585bc36-20c5-4420-b2c7-196898249e8f -Hostate oxide/ntp
9092023-10-20T00:35:36.224Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
9102023-10-20T00:35:36.251Z++ true
9112023-10-20T00:35:36.286Z+ [[ '' != online ]]
9122023-10-20T00:35:36.312Z+ [[ 17 -gt 60 ]]
9132023-10-20T00:35:36.349Z+ sleep 1
9142023-10-20T00:35:36.388Z+ retry=18
9152023-10-20T00:35:36.435Z+++ zoneadm list -n
9162023-10-20T00:35:36.465Z+++ grep oxz_ntp
9172023-10-20T00:35:36.492Z++ pfexec svcs -z oxz_ntp_5585bc36-20c5-4420-b2c7-196898249e8f -Hostate oxide/ntp
9182023-10-20T00:35:36.525Z+ [[ disabled != online ]]
9192023-10-20T00:35:36.553Z+ [[ 18 -gt 60 ]]
9202023-10-20T00:35:36.584Z+ sleep 1
9212023-10-20T00:35:36.662Z+ retry=19
9222023-10-20T00:35:36.701ZWaited for chrony: 26s
9232023-10-20T00:35:36.730Z+++ zoneadm list -n
9242023-10-20T00:35:36.755Z+++ grep oxz_ntp
9252023-10-20T00:35:36.784Z++ pfexec svcs -z oxz_ntp_5585bc36-20c5-4420-b2c7-196898249e8f -Hostate oxide/ntp
9262023-10-20T00:35:36.809Z+ [[ disabled != online ]]
9272023-10-20T00:35:36.860Z+ [[ 19 -gt 60 ]]
9282023-10-20T00:35:36.907Z+ sleep 1
9292023-10-20T00:35:36.941Z+ retry=20
9302023-10-20T00:35:36.967Z+++ zoneadm list -n
9312023-10-20T00:35:37.018Z+++ grep oxz_ntp
9322023-10-20T00:35:37.045Z++ pfexec svcs -z oxz_ntp_5585bc36-20c5-4420-b2c7-196898249e8f -Hostate oxide/ntp
9332023-10-20T00:35:37.079Z+ [[ disabled != online ]]
9342023-10-20T00:35:37.114Z+ [[ 20 -gt 60 ]]
9352023-10-20T00:35:37.140Z+ sleep 1
9362023-10-20T00:35:37.173Z+ retry=21
9372023-10-20T00:35:37.200Z+++ zoneadm list -n
9382023-10-20T00:35:37.226Z+++ grep oxz_ntp
9392023-10-20T00:35:37.252Z++ pfexec svcs -z oxz_ntp_5585bc36-20c5-4420-b2c7-196898249e8f -Hostate oxide/ntp
9402023-10-20T00:35:37.287Z+ [[ disabled != online ]]
9412023-10-20T00:35:37.329Z+ [[ 21 -gt 60 ]]
9422023-10-20T00:35:37.358Z+ sleep 1
9432023-10-20T00:35:37.389Z+ retry=22
9442023-10-20T00:35:37.426Z+++ zoneadm list -n
9452023-10-20T00:35:37.466Z+++ grep oxz_ntp
9462023-10-20T00:35:37.498Z++ pfexec svcs -z oxz_ntp_5585bc36-20c5-4420-b2c7-196898249e8f -Hostate oxide/ntp
9472023-10-20T00:35:37.527Z+ [[ disabled != online ]]
9482023-10-20T00:35:37.566Z+ [[ 22 -gt 60 ]]
9492023-10-20T00:35:37.607Z+ sleep 1
9502023-10-20T00:35:37.646Z+ retry=23
9512023-10-20T00:35:37.717Z+++ zoneadm list -n
9522023-10-20T00:35:37.744Z+++ grep oxz_ntp
9532023-10-20T00:35:37.774Z++ pfexec svcs -z oxz_ntp_5585bc36-20c5-4420-b2c7-196898249e8f -Hostate oxide/ntp
9542023-10-20T00:35:37.805Z+ [[ disabled != online ]]
9552023-10-20T00:35:37.836Z+ [[ 23 -gt 60 ]]
9562023-10-20T00:35:37.864Z+ sleep 1
9572023-10-20T00:35:37.889Z+ retry=24
9582023-10-20T00:35:37.917Z+++ zoneadm list -n
9592023-10-20T00:35:37.945Z+++ grep oxz_ntp
9602023-10-20T00:35:37.979Z++ pfexec svcs -z oxz_ntp_5585bc36-20c5-4420-b2c7-196898249e8f -Hostate oxide/ntp
9612023-10-20T00:35:38.004Z+ [[ disabled != online ]]
9622023-10-20T00:35:38.037Z+ [[ 24 -gt 60 ]]
9632023-10-20T00:35:38.080Z+ sleep 1
9642023-10-20T00:35:38.108Z+ retry=25
9652023-10-20T00:35:38.138Z+++ zoneadm list -n
9662023-10-20T00:35:38.165Z+++ grep oxz_ntp
9672023-10-20T00:35:38.209Z++ pfexec svcs -z oxz_ntp_5585bc36-20c5-4420-b2c7-196898249e8f -Hostate oxide/ntp
9682023-10-20T00:35:38.238Z+ [[ disabled != online ]]
9692023-10-20T00:35:38.264Z+ [[ 25 -gt 60 ]]
9702023-10-20T00:35:38.324Z+ sleep 1
9712023-10-20T00:35:38.373Z+ retry=26
9722023-10-20T00:35:38.401Z+++ zoneadm list -n
9732023-10-20T00:35:38.426Z+++ grep oxz_ntp
9742023-10-20T00:35:38.457Z++ pfexec svcs -z oxz_ntp_5585bc36-20c5-4420-b2c7-196898249e8f -Hostate oxide/ntp
9752023-10-20T00:35:38.483Z+ [[ online != online ]]
9762023-10-20T00:35:38.509Z+ echo 'Waited for chrony: 26s'
9772023-10-20T00:35:38.540Z+ export RUST_BACKTRACE=1
9782023-10-20T00:35:38.574Z+ RUST_BACKTRACE=1
9792023-10-20T00:35:38.601Z+ export E2E_TLS_CERT IPPOOL_START IPPOOL_END
9802023-10-20T00:35:38.637Z+ ./tests/bootstrap
9812023-10-20T00:35:38.663ZUsing external DNS server at 10.151.1.164:53
9822023-10-20T00:35:38.696Z2023-10-20 00:35:34.031300052 UTC: attempting to log into API
9832023-10-20T00:35:48.971Z2023-10-20 00:35:49.302033860 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9842023-10-20T00:35:49.978Z2023-10-20 00:35:50.303153158 UTC: attempting to log into API
9852023-10-20T00:36:05.005Z2023-10-20 00:36:05.335944326 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9862023-10-20T00:36:06.005Z2023-10-20 00:36:06.337071214 UTC: attempting to log into API
9872023-10-20T00:36:21.060Z2023-10-20 00:36:21.370926425 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9882023-10-20T00:36:22.040Z2023-10-20 00:36:22.371774468 UTC: attempting to log into API
9892023-10-20T00:36:37.091Z2023-10-20 00:36:37.422645336 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9902023-10-20T00:36:38.092Z2023-10-20 00:36:38.423616311 UTC: attempting to log into API
 
9912023-10-20T00:36:39.936ZOct 20 00:36:39 buskin ip: ip: joining multicasts failed (4) on vopte4 - will use link layer broadcasts for multicast
9922023-10-20T00:36:46.206ZOct 20 00:36:45 buskin ip: ip: joining multicasts failed (4) on vopte2 - will use link layer broadcasts for multicast
9932023-10-20T00:36:52.196ZOct 20 00:36:52 bukin ip: ip: joining multicasts failed (4) on vopte1 - will use link layer broadcasts for multicast
 
9942023-10-20T00:36:53.128Z2023-10-20 00:36:53.437238321 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9952023-10-20T00:36:54.126Z2023-10-20 00:36:54.435311172 UTC: attempting to log into API
 
9962023-10-20T00:36:58.412ZOct 20 00:36:58 buskin ip: ip: joining multicasts failed (4) on vopte3 - will use link layer broadcasts for multicast
9972023-10-20T00:37:06.094ZOct 20 00:37:05 buskin ip: ip: joining multicasts failed (4) on vopte5 - will use link layer broadcasts for multicast
 
9982023-10-20T00:37:09.160Z2023-10-20 00:37:09.426455820 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9992023-10-20T00:37:10.160Z2023-10-20 00:37:10.424818558 UTC: attempting to log into API
10002023-10-20T00:37:25.196Z2023-10-20 00:37:25.417165261 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
10012023-10-20T00:37:26.194Z2023-10-20 00:37:26.415516578 UTC: attempting to log into API
10022023-10-20T00:37:26.975Z2023-10-20 00:37:27.193841241 UTC: login succeeded
10032023-10-20T00:37:27.008Zwaiting for nexus to come up...
10042023-10-20T00:37:27.073Zcreating IP pool... 10.151.1.175 - 10.151.1.227
10052023-10-20T00:37:27.143Zensuring datasets are ready...
10062023-10-20T00:37:29.575Zlet's roll.
10072023-10-20T00:37:29.627Z+ rm ./tests/bootstrap
10082023-10-20T00:37:29.672Z+ for test_bin in tests/*
10092023-10-20T00:37:29.722Z+ ./tests/bootstrap-8a64e1a08208e018
10102023-10-20T00:37:29.755Z
10112023-10-20T00:37:29.793Zrunning 0 tests
10122023-10-20T00:37:29.860Z
10132023-10-20T00:37:29.898Ztest result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
10142023-10-20T00:37:29.926Z
10152023-10-20T00:37:29.966Z+ for test_bin in tests/*
10162023-10-20T00:37:30.001Z+ ./tests/end_to_end_tests-ac1cb3326a4ba018
10172023-10-20T00:37:30.031Z
10182023-10-20T00:37:30.073Zrunning 2 tests
10192023-10-20T00:37:30.116Ztest no_spoof::no_spoof_here ... ok
10202023-10-20T00:38:29.862Ztest instance_launch::instance_launch has been running for over 60 seconds
10212023-10-20T00:40:28.022Ztest instance_launch::instance_launch ... ok
10222023-10-20T00:40:28.049Z
10232023-10-20T00:40:28.075Ztest result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 178.32s
10242023-10-20T00:40:28.100Z
10252023-10-20T00:40:28.124Z+ _exit_trap
10262023-10-20T00:40:28.149Z+ local status=0
10272023-10-20T00:40:28.174Z+ [[ 0 -eq 0 ]]
10282023-10-20T00:40:28.198Z+ exit 0
10292023-10-20T00:40:28.223Zprocess exited: duration 376101 ms, exit code 0
 
10302023-10-20T00:40:28.308Zfound 47 output files
10312023-10-20T00:40:28.360Zuploading: /var/svc/log/oxide-sled-agent:default.log (454693 bytes)
10322023-10-20T00:40:30.096Zuploaded: /var/svc/log/oxide-sled-agent:default.log
10332023-10-20T00:40:30.125Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_clickhouse_a4969012-6ade-4077-8c95-90fc0dd6fd14/root/var/svc/log/oxide-clickhouse:default.log (694860 bytes)
10342023-10-20T00:40:32.144Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_clickhouse_a4969012-6ade-4077-8c95-90fc0dd6fd14/root/var/svc/log/oxide-clickhouse:default.log
10352023-10-20T00:40:32.168Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_6def8f57-949f-4e72-b23e-aee54d1af2ee/root/var/svc/log/oxide-cockroachdb:default.log (5368 bytes)
10362023-10-20T00:40:33.239Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_6def8f57-949f-4e72-b23e-aee54d1af2ee/root/var/svc/log/oxide-cockroachdb:default.log
10372023-10-20T00:40:33.264Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_8b27fbc7-253e-4e88-8140-95e9f4923a41/root/var/svc/log/oxide-cockroachdb:default.log (5367 bytes)
10382023-10-20T00:40:34.336Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_8b27fbc7-253e-4e88-8140-95e9f4923a41/root/var/svc/log/oxide-cockroachdb:default.log
10392023-10-20T00:40:34.360Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_921208e9-a8c0-41b5-83d9-47f4576c4b18/root/var/svc/log/oxide-internal_dns:default.log (102320 bytes)
10402023-10-20T00:40:35.548Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_921208e9-a8c0-41b5-83d9-47f4576c4b18/root/var/svc/log/oxide-internal_dns:default.log
10412023-10-20T00:40:35.642Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_26c67608-87a8-4913-aaf4-b27c5801b5b6/root/var/svc/log/oxide-crucible-agent:default.log (3436 bytes)
10422023-10-20T00:40:36.716Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_26c67608-87a8-4913-aaf4-b27c5801b5b6/root/var/svc/log/oxide-crucible-agent:default.log
10432023-10-20T00:40:36.740Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_26c67608-87a8-4913-aaf4-b27c5801b5b6/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10442023-10-20T00:40:37.811Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_26c67608-87a8-4913-aaf4-b27c5801b5b6/root/var/svc/log/oxide-crucible-downstairs:default.log
10452023-10-20T00:40:37.835Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_8bd21879-b6df-42ea-a3c8-c71e34221481/root/var/svc/log/oxide-crucible-agent:default.log (14571 bytes)
10462023-10-20T00:40:38.931Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_8bd21879-b6df-42ea-a3c8-c71e34221481/root/var/svc/log/oxide-crucible-agent:default.log
10472023-10-20T00:40:38.955Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_8bd21879-b6df-42ea-a3c8-c71e34221481/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10482023-10-20T00:40:40.028Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_8bd21879-b6df-42ea-a3c8-c71e34221481/root/var/svc/log/oxide-crucible-downstairs:default.log
10492023-10-20T00:40:40.052Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_8bd21879-b6df-42ea-a3c8-c71e34221481/root/var/svc/log/oxide-crucible-downstairs:downstairs-5bba19de-376d-4809-8b52-3a5c971cd1ef.log (5535 bytes)
10502023-10-20T00:40:41.129Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_8bd21879-b6df-42ea-a3c8-c71e34221481/root/var/svc/log/oxide-crucible-downstairs:downstairs-5bba19de-376d-4809-8b52-3a5c971cd1ef.log
10512023-10-20T00:40:41.157Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_pantry_03c2c0f7-e5d8-45e2-911d-facb98913263/root/var/svc/log/oxide-crucible-pantry:default.log (2338 bytes)
10522023-10-20T00:40:42.232Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_pantry_03c2c0f7-e5d8-45e2-911d-facb98913263/root/var/svc/log/oxide-crucible-pantry:default.log
10532023-10-20T00:40:42.257Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_pantry_db1d803e-1537-4ad9-b312-de99fb3d82d5/root/var/svc/log/oxide-crucible-pantry:default.log (2333 bytes)
10542023-10-20T00:40:43.328Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_pantry_db1d803e-1537-4ad9-b312-de99fb3d82d5/root/var/svc/log/oxide-crucible-pantry:default.log
10552023-10-20T00:40:43.353Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_internal_dns_d0f2f1af-1b8c-4376-88e8-07c98e1faf2e/root/var/svc/log/oxide-internal_dns:default.log (100292 bytes)
10562023-10-20T00:40:44.544Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_internal_dns_d0f2f1af-1b8c-4376-88e8-07c98e1faf2e/root/var/svc/log/oxide-internal_dns:default.log
10572023-10-20T00:40:44.569Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_cockroachdb_1bd3732e-c325-4403-be9b-cd7e0d9bf2c7/root/var/svc/log/oxide-cockroachdb:default.log (5341 bytes)
10582023-10-20T00:40:45.642Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_cockroachdb_1bd3732e-c325-4403-be9b-cd7e0d9bf2c7/root/var/svc/log/oxide-cockroachdb:default.log
10592023-10-20T00:40:45.667Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_external_dns_dac113b7-cb8c-4992-bf38-7702522c9f80/root/var/svc/log/oxide-external_dns:default.log (15608 bytes)
10602023-10-20T00:40:46.761Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_external_dns_dac113b7-cb8c-4992-bf38-7702522c9f80/root/var/svc/log/oxide-external_dns:default.log
10612023-10-20T00:40:46.785Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_00741d78-c6dd-4a5d-a187-6099af189740/root/var/svc/log/oxide-crucible-agent:default.log (14149 bytes)
10622023-10-20T00:40:47.882Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_00741d78-c6dd-4a5d-a187-6099af189740/root/var/svc/log/oxide-crucible-agent:default.log
10632023-10-20T00:40:47.908Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_00741d78-c6dd-4a5d-a187-6099af189740/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10642023-10-20T00:40:48.992Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_00741d78-c6dd-4a5d-a187-6099af189740/root/var/svc/log/oxide-crucible-downstairs:default.log
10652023-10-20T00:40:49.018Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_00741d78-c6dd-4a5d-a187-6099af189740/root/var/svc/log/oxide-crucible-downstairs:downstairs-75bbc187-b179-494e-81e7-9663bfd3ecc7.log (3049 bytes)
10662023-10-20T00:40:50.088Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_00741d78-c6dd-4a5d-a187-6099af189740/root/var/svc/log/oxide-crucible-downstairs:downstairs-75bbc187-b179-494e-81e7-9663bfd3ecc7.log
10672023-10-20T00:40:50.113Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_pantry_188bda95-3db3-48e8-93a5-f0ee0792c373/root/var/svc/log/oxide-crucible-pantry:default.log (2338 bytes)
10682023-10-20T00:40:51.184Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_pantry_188bda95-3db3-48e8-93a5-f0ee0792c373/root/var/svc/log/oxide-crucible-pantry:default.log
10692023-10-20T00:40:51.210Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_oximeter_7aedbbab-5c65-4eaf-a4d7-306e5c92197f/root/var/svc/log/oxide-oximeter:default.log (182499 bytes)
10702023-10-20T00:40:51.369Zupload warning: file "/pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_oximeter_7aedbbab-5c65-4eaf-a4d7-306e5c92197f/root/var/svc/log/oxide-oximeter:default.log" changed size mid upload: 182499 -> 200354
10712023-10-20T00:40:52.441Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_oximeter_7aedbbab-5c65-4eaf-a4d7-306e5c92197f/root/var/svc/log/oxide-oximeter:default.log
10722023-10-20T00:40:52.465Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_cockroachdb_af20b2f1-3c2d-4dd8-ade8-ca3ce116f864/root/var/svc/log/oxide-cockroachdb:default.log (5368 bytes)
10732023-10-20T00:40:58.174Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_cockroachdb_af20b2f1-3c2d-4dd8-ade8-ca3ce116f864/root/var/svc/log/oxide-cockroachdb:default.log
10742023-10-20T00:40:58.208Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_2ca2645e-e8d7-41a1-af9f-ce5c687f8ed0/root/var/svc/log/oxide-crucible-agent:default.log (24823 bytes)
10752023-10-20T00:40:59.307Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_2ca2645e-e8d7-41a1-af9f-ce5c687f8ed0/root/var/svc/log/oxide-crucible-agent:default.log
10762023-10-20T00:40:59.342Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_2ca2645e-e8d7-41a1-af9f-ce5c687f8ed0/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10772023-10-20T00:41:00.405Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_2ca2645e-e8d7-41a1-af9f-ce5c687f8ed0/root/var/svc/log/oxide-crucible-downstairs:default.log
10782023-10-20T00:41:00.430Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_2ca2645e-e8d7-41a1-af9f-ce5c687f8ed0/root/var/svc/log/oxide-crucible-downstairs:downstairs-2e4fbe26-3ff8-410f-9b19-ca23ba771d1d.log (5531 bytes)
10792023-10-20T00:41:01.505Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_2ca2645e-e8d7-41a1-af9f-ce5c687f8ed0/root/var/svc/log/oxide-crucible-downstairs:downstairs-2e4fbe26-3ff8-410f-9b19-ca23ba771d1d.log
10802023-10-20T00:41:01.530Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_2ca2645e-e8d7-41a1-af9f-ce5c687f8ed0/root/var/svc/log/oxide-crucible-downstairs:downstairs-3129811b-6ae7-45e1-9877-f87a68e1cd25.log (3049 bytes)
10812023-10-20T00:41:02.603Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_2ca2645e-e8d7-41a1-af9f-ce5c687f8ed0/root/var/svc/log/oxide-crucible-downstairs:downstairs-3129811b-6ae7-45e1-9877-f87a68e1cd25.log
10822023-10-20T00:41:02.628Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_502eea51-1b83-4276-ba38-7706f311dc18/root/var/svc/log/oxide-crucible-agent:default.log (3436 bytes)
10832023-10-20T00:41:03.701Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_502eea51-1b83-4276-ba38-7706f311dc18/root/var/svc/log/oxide-crucible-agent:default.log
10842023-10-20T00:41:03.726Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_502eea51-1b83-4276-ba38-7706f311dc18/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10852023-10-20T00:41:04.799Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_502eea51-1b83-4276-ba38-7706f311dc18/root/var/svc/log/oxide-crucible-downstairs:default.log
10862023-10-20T00:41:04.824Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_bcb54cf9-f8ee-4360-ad39-7041ab4b7a11/root/var/svc/log/oxide-crucible-agent:default.log (14112 bytes)
10872023-10-20T00:41:05.920Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_bcb54cf9-f8ee-4360-ad39-7041ab4b7a11/root/var/svc/log/oxide-crucible-agent:default.log
10882023-10-20T00:41:05.944Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_bcb54cf9-f8ee-4360-ad39-7041ab4b7a11/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10892023-10-20T00:41:07.015Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_bcb54cf9-f8ee-4360-ad39-7041ab4b7a11/root/var/svc/log/oxide-crucible-downstairs:default.log
10902023-10-20T00:41:07.040Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_bcb54cf9-f8ee-4360-ad39-7041ab4b7a11/root/var/svc/log/oxide-crucible-downstairs:downstairs-2a0af7d6-1cb7-4dea-a1ae-70312449f718.log (3048 bytes)
10912023-10-20T00:41:08.112Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_bcb54cf9-f8ee-4360-ad39-7041ab4b7a11/root/var/svc/log/oxide-crucible-downstairs:downstairs-2a0af7d6-1cb7-4dea-a1ae-70312449f718.log
10922023-10-20T00:41:08.142Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_nexus_68b2b45e-e0d0-4171-835d-4303071ba40c/root/var/svc/log/oxide-nexus:default.log (545954 bytes)
10932023-10-20T00:41:08.379Zupload warning: file "/pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_nexus_68b2b45e-e0d0-4171-835d-4303071ba40c/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 545954 -> 606567
10942023-10-20T00:41:09.451Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_nexus_68b2b45e-e0d0-4171-835d-4303071ba40c/root/var/svc/log/oxide-nexus:default.log
10952023-10-20T00:41:09.476Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_2f948b51-7bbd-4a8d-bcef-3594deca80cb/root/var/svc/log/oxide-cockroachdb:default.log (5367 bytes)
10962023-10-20T00:41:10.547Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_2f948b51-7bbd-4a8d-bcef-3594deca80cb/root/var/svc/log/oxide-cockroachdb:default.log
10972023-10-20T00:41:10.574Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_6babdd03-2ab9-42e1-9aac-61a1542d77c5/root/var/svc/log/oxide-crucible-agent:default.log (14148 bytes)
10982023-10-20T00:41:11.666Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_6babdd03-2ab9-42e1-9aac-61a1542d77c5/root/var/svc/log/oxide-crucible-agent:default.log
10992023-10-20T00:41:11.691Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_6babdd03-2ab9-42e1-9aac-61a1542d77c5/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
11002023-10-20T00:41:12.761Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_6babdd03-2ab9-42e1-9aac-61a1542d77c5/root/var/svc/log/oxide-crucible-downstairs:default.log
11012023-10-20T00:41:12.786Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_6babdd03-2ab9-42e1-9aac-61a1542d77c5/root/var/svc/log/oxide-crucible-downstairs:downstairs-2e75c46e-c658-494b-afb0-d5b6a5982240.log (5535 bytes)
11022023-10-20T00:41:13.857Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_6babdd03-2ab9-42e1-9aac-61a1542d77c5/root/var/svc/log/oxide-crucible-downstairs:downstairs-2e75c46e-c658-494b-afb0-d5b6a5982240.log
11032023-10-20T00:41:13.882Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_eb349ef0-24fd-4667-9c88-8803b7449d37/root/var/svc/log/oxide-crucible-agent:default.log (3435 bytes)
11042023-10-20T00:41:14.953Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_eb349ef0-24fd-4667-9c88-8803b7449d37/root/var/svc/log/oxide-crucible-agent:default.log
11052023-10-20T00:41:14.977Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_eb349ef0-24fd-4667-9c88-8803b7449d37/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
11062023-10-20T00:41:16.049Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_eb349ef0-24fd-4667-9c88-8803b7449d37/root/var/svc/log/oxide-crucible-downstairs:default.log
11072023-10-20T00:41:16.074Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_a3d98407-39a3-4d5a-94ce-b95640f7157c/root/var/svc/log/oxide-external_dns:default.log (15608 bytes)
11082023-10-20T00:41:16.145Zupload warning: file "/pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_a3d98407-39a3-4d5a-94ce-b95640f7157c/root/var/svc/log/oxide-external_dns:default.log" changed size mid upload: 15608 -> 19385
11092023-10-20T00:41:17.217Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_a3d98407-39a3-4d5a-94ce-b95640f7157c/root/var/svc/log/oxide-external_dns:default.log
11102023-10-20T00:41:17.242Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_b9a16093-99bb-4223-a8ce-f447307e76ec/root/var/svc/log/oxide-internal_dns:default.log (100005 bytes)
11112023-10-20T00:41:17.379Zupload warning: file "/pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_b9a16093-99bb-4223-a8ce-f447307e76ec/root/var/svc/log/oxide-internal_dns:default.log" changed size mid upload: 100005 -> 103759
11122023-10-20T00:41:18.450Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_b9a16093-99bb-4223-a8ce-f447307e76ec/root/var/svc/log/oxide-internal_dns:default.log
11132023-10-20T00:41:18.476Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_ceb4cd2e-86c7-45a2-8d9f-69fc17ed2528/root/var/svc/log/oxide-nexus:default.log (474956 bytes)
11142023-10-20T00:41:18.711Zupload warning: file "/pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_ceb4cd2e-86c7-45a2-8d9f-69fc17ed2528/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 474956 -> 536178
11152023-10-20T00:41:19.790Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_ceb4cd2e-86c7-45a2-8d9f-69fc17ed2528/root/var/svc/log/oxide-nexus:default.log
11162023-10-20T00:41:19.814Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_211be812-6b6a-4645-98f7-2fb2c3107472/root/var/svc/log/oxide-crucible-agent:default.log (3435 bytes)
11172023-10-20T00:41:20.890Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_211be812-6b6a-4645-98f7-2fb2c3107472/root/var/svc/log/oxide-crucible-agent:default.log
11182023-10-20T00:41:20.914Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_211be812-6b6a-4645-98f7-2fb2c3107472/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
11192023-10-20T00:41:21.987Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_211be812-6b6a-4645-98f7-2fb2c3107472/root/var/svc/log/oxide-crucible-downstairs:default.log
11202023-10-20T00:41:22.012Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_5ad05592-56d3-4007-86e5-6e477b244e1e/root/var/svc/log/oxide-nexus:default.log (4531127 bytes)
11212023-10-20T00:41:23.299Zupload warning: file "/pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_5ad05592-56d3-4007-86e5-6e477b244e1e/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 4531127 -> 4592967
11222023-10-20T00:41:24.372Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_5ad05592-56d3-4007-86e5-6e477b244e1e/root/var/svc/log/oxide-nexus:default.log
11232023-10-20T00:41:24.397Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_5585bc36-20c5-4420-b2c7-196898249e8f/root/var/svc/log/oxide-ntp:default.log (1253 bytes)
11242023-10-20T00:41:24.421Zupload warning: file "/pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_5585bc36-20c5-4420-b2c7-196898249e8f/root/var/svc/log/oxide-ntp:default.log" changed size mid upload: 1253 -> 1321
11252023-10-20T00:41:25.490Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_5585bc36-20c5-4420-b2c7-196898249e8f/root/var/svc/log/oxide-ntp:default.log
11262023-10-20T00:41:25.515Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_5585bc36-20c5-4420-b2c7-196898249e8f/root/var/log/chrony/measurements.log (10549 bytes)
11272023-10-20T00:41:25.561Zupload warning: file "/pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_5585bc36-20c5-4420-b2c7-196898249e8f/root/var/log/chrony/measurements.log" changed size mid upload: 10549 -> 12056
11282023-10-20T00:41:26.641Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_5585bc36-20c5-4420-b2c7-196898249e8f/root/var/log/chrony/measurements.log
11292023-10-20T00:41:26.666Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_5585bc36-20c5-4420-b2c7-196898249e8f/root/var/log/chrony/statistics.log (6084 bytes)
11302023-10-20T00:41:27.117Zupload warning: file "/pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_5585bc36-20c5-4420-b2c7-196898249e8f/root/var/log/chrony/statistics.log" changed size mid upload: 6084 -> 7371
11312023-10-20T00:41:28.198Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_5585bc36-20c5-4420-b2c7-196898249e8f/root/var/log/chrony/statistics.log
11322023-10-20T00:41:28.222Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_5585bc36-20c5-4420-b2c7-196898249e8f/root/var/log/chrony/tracking.log (1722 bytes)
11332023-10-20T00:41:28.246Zupload warning: file "/pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_5585bc36-20c5-4420-b2c7-196898249e8f/root/var/log/chrony/tracking.log" changed size mid upload: 1722 -> 1986
11342023-10-20T00:41:29.317Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_5585bc36-20c5-4420-b2c7-196898249e8f/root/var/log/chrony/tracking.log