01HEDY0V9Q0EEF61PCM8QN7ES2: helios / deploy

BasicConfig {
    output_rules: [
        "%/var/svc/log/oxide-sled-agent:default.log*",
        "%/pool/ext/*/crypt/zone/oxz_*/root/var/svc/log/oxide-*.log*",
        "%/pool/ext/*/crypt/zone/oxz_*/root/var/svc/log/system-illumos-*.log*",
        "%/pool/ext/*/crypt/zone/oxz_ntp_*/root/var/log/chrony/*.log*",
        "!/pool/ext/*/crypt/zone/oxz_propolis-server_*/root/var/svc/log/*.log*",
        "%/pool/ext/*/crypt/debug/global/oxide-sled-agent:default.log.*",
        "%/pool/ext/*/crypt/debug/oxz_*/oxide-*.log.*",
        "%/pool/ext/*/crypt/debug/oxz_*/system-illumos-*.log.*",
        "!/pool/ext/*/crypt/debug/oxz_propolis-server_*/*.log.*",
    ],
    rust_toolchain: None,
    target: Some(
        "lab-2.0-opte-0.25",
    ),
    access_repos: [],
    publish: [],
    skip_clone: true,
}

Buildomat Job: 01HEDY1BA384K5FVDKR4H5FSQG

Tags:

Artefacts:

Output:

SEQ GLOBAL TIME DETAILS
12023-11-04T20:38:38.071Zjob dependencies complete; ready to run (waiting for 50 m 43 s)
22023-11-04T20:39:21.194Zjob assigned to worker 01HEE0YD78KQVWFJT4PKYRAHWB (queued for 43 s)
32023-11-04T20:39:21.206Zhttp://172.20.0.5:9969/ipxe/sock... ok
42023-11-04T20:39:21.263Zhttp://172.20.0.5:9969/os/sock/platform/i86pc/kernel/amd64/unix... ok
52023-11-04T20:39:21.300Zhttp://172.20.0.5:9969/os/sock/platform/i86pc/amd64/boot_archive... ok
62023-11-04T20:39:21.333Zhttp://172.20.0.5:9969/os/sock/platform/i86pc/amd64/boot_archive.hash... ok
72023-11-04T20:39:21.368Zhttp://172.20.0.5:9969/postboot/sock... ok
82023-11-04T20:39:21.402ZwcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U
92023-11-04T20:39:21.435Zcpu0: microcode has been updated from version 0xa001133 to 0xa0011d1
102023-11-04T20:39:21.469ZOxide Helios Version helios-2.0.22241 64-bit
112023-11-04T20:39:21.503ZConfiguring device.
122023-11-04T20:39:21.536Z##################################################
132023-11-04T20:39:21.597Z#### ############# ##
142023-11-04T20:39:21.626Z### ### ############ ##
152023-11-04T20:39:21.660Z## ### # ## ### ## Oxide Computer Company ##
162023-11-04T20:39:21.694Z## ## ## ### # ### ##
172023-11-04T20:39:21.728Z## # ### #### #### This Station Under ##
182023-11-04T20:39:21.761Z### ### #### # ### Computer Control ##
192023-11-04T20:39:21.795Z#### #### ### ## ##
202023-11-04T20:39:21.828Z##################################################
212023-11-04T20:39:21.861ZHostname: sock
222023-11-04T20:39:21.895ZLoading smf(7) serice descriptions:
232023-11-04T20:39:22.401Zdownloading input: /input/ci-tools/work/end-to-end-tests/bootstrap-346a6c0cadffef35.gz
242023-11-04T20:39:24.048Zdownloaded input: /input/ci-tools/work/end-to-end-tests/bootstrap-346a6c0cadffef35.gz
252023-11-04T20:39:24.072Zdownloading input: /input/ci-tools/work/end-to-end-tests/bootstrap.gz
262023-11-04T20:39:41.653Zdownloaded input: /input/ci-tools/work/end-to-end-tests/bootstrap.gz
272023-11-04T20:39:41.677Zdownloading input: /input/ci-tools/work/end-to-end-tests/end_to_end_tests-ad8d578e5459ec9d.gz
282023-11-04T20:40:01.505Zdownloaded input: /input/ci-tools/work/end-to-end-tests/end_to_end_tests-ad8d578e5459ec9d.gz
292023-11-04T20:40:01.529Zdownloading input: /input/ci-tools/work/caboose-util.gz
302023-11-04T20:40:02.471Zdownloaded input: /input/ci-tools/work/caboose-util.gz
312023-11-04T20:40:02.496Zdownloading input: /input/ci-tools/work/tufaceous.gz
322023-11-04T20:40:05.358Zdownloaded input: /input/ci-tools/work/tufaceous.gz
332023-11-04T20:40:05.383Zdownloading input: /input/package/work/version.txt
342023-11-04T20:40:05.408Zdownloaded input: /input/package/work/version.txt
352023-11-04T20:40:05.433Zdownloading input: /input/package/work/package.tar.gz
362023-11-04T20:40:21.466Zdownloaded input: /input/package/work/package.tar.gz
372023-11-04T20:40:21.492Zdownloading input: /input/package/work/global-zone-packages.tar.gz
382023-11-04T20:40:45.236Zdownloaded input: /input/package/work/global-zone-packages.tar.gz
392023-11-04T20:40:45.261Zdownloading input: /input/package/work/trampoline-global-zone-packages.tar.gz
402023-11-04T20:40:52.440Zdownloaded input: /input/package/work/trampoline-global-zone-packages.tar.gz
412023-11-04T20:40:52.466Zdownloading input: /input/package/work/zones/clickhouse.tar.gz
422023-11-04T20:41:28.314Zdownloaded input: /input/package/work/zones/clickhouse.tar.gz
432023-11-04T20:41:28.338Zdownloading input: /input/package/work/zones/clickhouse_keeper.tar.gz
442023-11-04T20:42:10.177Zdownloaded input: /input/package/work/zones/clickhouse_keeper.tar.gz
452023-11-04T20:42:10.205Zdownloading input: /input/package/work/zones/cockroachdb.tar.gz
462023-11-04T20:42:30.640Zdownloaded input: /input/package/work/zones/cockroachdb.tar.gz
472023-11-04T20:42:30.664Zdownloading input: /input/package/work/zones/crucible-pantry.tar.gz
482023-11-04T20:42:34.676Zdownloaded input: /input/package/work/zones/crucible-pantry.tar.gz
492023-11-04T20:42:34.700Zdownloading input: /input/package/work/zones/crucible.tar.gz
502023-11-04T20:42:42.821Zdownloaded input: /input/package/work/zones/crucible.tar.gz
512023-11-04T20:42:42.846Zdownloading input: /input/package/work/zones/external-dns.tar.gz
522023-11-04T20:42:47.362Zdownloaded input: /input/package/work/zones/external-dns.tar.gz
532023-11-04T20:42:47.387Zdownloading input: /input/package/work/zones/internal-dns.tar.gz
542023-11-04T20:42:51.631Zdownloaded input: /input/package/work/zones/internal-dns.tar.gz
552023-11-04T20:42:51.655Zdownloading input: /input/package/work/zones/ntp.tar.gz
562023-11-04T20:42:51.679Zdownloaded input: /input/package/work/zones/ntp.tar.gz
572023-11-04T20:42:51.704Zdownloading input: /input/package/work/zones/omicron-gateway-asic.tar.gz
582023-11-04T20:42:54.970Zdownloaded input: /input/package/work/zones/omicron-gateway-asic.tar.gz
592023-11-04T20:42:54.995Zdownloading input: /input/package/work/zones/omicron-gateway-softnpu.tar.gz
602023-11-04T20:42:58.450Zdownloaded input: /input/package/work/zones/omicron-gateway-softnpu.tar.gz
612023-11-04T20:42:58.474Zdownloading input: /input/package/work/zones/omicron-nexus-single-sled.tar.gz
622023-11-04T20:43:11.209Zdownloaded input: /input/package/work/zones/omicron-nexus-single-sled.tar.gz
632023-11-04T20:43:11.233Zdownloading input: /input/package/work/zones/omicron-nexus.tar.gz
642023-11-04T20:43:23.693Zdownloaded input: /input/package/work/zones/omicron-nexus.tar.gz
652023-11-04T20:43:23.717Zdownloading input: /input/package/work/zones/overlay.tar.gz
662023-11-04T20:43:23.741Zdownloaded input: /input/package/work/zones/overlay.tar.gz
672023-11-04T20:43:23.765Zdownloading input: /input/package/work/zones/oximeter-collector.tar.gz
682023-11-04T20:43:27.200Zdownloaded input: /input/package/work/zones/oximeter-collector.tar.gz
692023-11-04T20:43:27.225Zdownloading input: /input/package/work/zones/propolis-server.tar.gz
702023-11-04T20:43:38.601Zdownloaded input: /input/package/work/zones/propolis-server.tar.gz
712023-11-04T20:43:38.625Zdownloading input: /input/package/work/zones/switch-asic.tar.gz
722023-11-04T20:44:20.801Zdownloaded input: /input/package/work/zones/switch-asic.tar.gz
732023-11-04T20:44:20.825Zdownloading input: /input/package/work/zones/switch-softnpu.tar.gz
742023-11-04T20:44:49.289Zdownloaded input: /input/package/work/zones/switch-softnpu.tar.gz
 
752023-11-04T20:44:49.314Zstarting task 0: "setup"
762023-11-04T20:44:49.347Z++ uname -s
772023-11-04T20:44:49.371Z+ kern=SunOS
782023-11-04T20:44:49.395Z+ case "$kern" in
792023-11-04T20:44:49.418Z+ groupadd -g 12345 build
802023-11-04T20:44:49.442Z+ useradd -u 12345 -g build -d /home/build -s /bin/bash -c build -P 'Primary Administrator' build
812023-11-04T20:44:51.356Z+ zfs create -o mountpoint=/work rpool/work
822023-11-04T20:44:51.383Z++ awk '$2 == "/home" { print $3 }' /etc/mnttab
832023-11-04T20:44:51.408Z+ home_fs=zfs
842023-11-04T20:44:51.432Z+ [[ zfs == autofs ]]
852023-11-04T20:44:51.456Z+ mkdir -p /home/build
862023-11-04T20:44:51.480Z+ chown build:build /home/build /work
872023-11-04T20:44:53.387Z+ chmod 0700 /home/build /work
882023-11-04T20:44:53.411Zprocess exited: duration 4052 ms, exit code 0
 
892023-11-04T20:44:53.459Zstarting task 1: "authentication"
902023-11-04T20:44:53.528Zprocess exited: duration 44 ms, exit code 0
 
912023-11-04T20:44:53.576Zstarting task 2: "build"
922023-11-04T20:44:53.607Z+ trap _exit_trap EXIT
932023-11-04T20:44:53.631Z+ pfexec mdb -kwe 'secpolicy_fs_linkdir/v 55 48 89 e5 b8 01 00 00 00 5d c3'
942023-11-04T20:44:53.805Zsecpolicy_fs_linkdir: 0x55 = 0x55
952023-11-04T20:44:53.830Zsecpolicy_fs_linkdir+1: 0xb8 = 0x48
962023-11-04T20:44:53.854Zsecpolicy_fs_linkdir+2: 0x1 = 0x89
972023-11-04T20:44:53.878Zsecpolicy_fs_linkdir+3: 0 = 0xe5
982023-11-04T20:44:53.902Zsecpolicy_fs_linkdir+4: 0 = 0xb8
992023-11-04T20:44:53.927Zsecpolicy_fs_linkdir+5: 0 = 0x1
1002023-11-04T20:44:53.951Zsecpolicy_fs_linkdir+6: 0x48 = 0x0
1012023-11-04T20:44:53.976Zsecpolicy_fs_linkdir+7: 0x89 = 0x0
1022023-11-04T20:44:54.000Zsecpolicy_fs_linkdir+8: 0xe5 = 0x0
1032023-11-04T20:44:54.025Zsecpolicy_fs_linkdir+9: 0x48 = 0x5d
1042023-11-04T20:44:54.049Zsecpolicy_fs_linkdir+0xa: 0x83 = 0xc3
1052023-11-04T20:44:54.073Z+ [[ -d /opt/oxide ]]
1062023-11-04T20:44:54.098Z+ pfexec mv /opt/oxide /opt/oxide-underneath
1072023-11-04T20:44:54.122Z+ pfexec mkdir /opt/oxide
1082023-11-04T20:44:54.146Z+ pfexec mount -F tmpfs -O swap /opt/oxide
1092023-11-04T20:44:54.170Z+ [[ -d /opt/oxide-underneath ]]
1102023-11-04T20:44:54.194Z+ cd /opt/oxide-underneath
1112023-11-04T20:44:54.218Z+ pfexec tar ceEp@/f - .
1122023-11-04T20:44:54.242Z+ cd /opt/oxide
1132023-11-04T20:44:54.266Z+ pfexec tar xveEp@/f -
1142023-11-04T20:44:54.290Zx ., 0 bytes, 0 tape blocks
1152023-11-04T20:44:54.314Zx ./opte, 0 bytes, 0 tape blocks
1162023-11-04T20:44:54.338Zx ./opte/bin, 0 bytes, 0 tape blocks
1172023-11-04T20:44:54.363Zx ./opte/bin/opteadm, 50714832 bytes, 99053 tape blocks
1182023-11-04T20:44:54.390Z+ pfexec rm -rf /opt/oxide-underneath
1192023-11-04T20:44:54.413Z+ pfexec /sbin/zfs create -o mountpoint=/zone rpool/zone
1202023-11-04T20:44:54.437Z+ pfexec mkdir /opt/oxide/work
1212023-11-04T20:44:54.461Z+ pfexec chown build:build /opt/oxide/work
1222023-11-04T20:44:54.485Z+ cd /opt/oxide/work
1232023-11-04T20:44:54.509Z+ ptime -m tar xvzf /input/package/work/package.tar.gz
1242023-11-04T20:44:55.583ZDecompressing '/input/package/work/package.tar.gz' with '/usr/bin/gzcat'...
1252023-11-04T20:44:55.608Zx out/mg-ddm-gz.tar, 62974464 bytes, 122997 tape blocks
1262023-11-04T20:44:55.632Zx out/omicron-sled-agent.tar, 123253248 bytes, 240729 tape blocks
1272023-11-04T20:44:55.657Zx out/target/test, 75 bytes, 1 tape blocks
1282023-11-04T20:44:55.683Zx out/npuzone/npuzone, 37381232 bytes, 73011 tape blocks
1292023-11-04T20:44:55.707Zx package-manifest.toml, 22037 bytes, 44 tape blocks
1302023-11-04T20:44:55.731Zx smf/sled-agent/non-gimlet/config.toml, 3102 bytes, 7 tape blocks
1312023-11-04T20:44:55.756Zx target/release/omicron-package, 42682512 bytes, 83365 tape blocks
1322023-11-04T20:44:55.779Zx tools/create_virtual_hardware.sh, 3067 bytes, 6 tape blocks
1332023-11-04T20:44:55.803Zx tools/virtual_hardware.sh, 3408 bytes, 7 tape blocks
1342023-11-04T20:44:55.828Zx tools/scrimlet/create-softnpu-zone.sh, 619 bytes, 2 tape blocks
1352023-11-04T20:44:55.852Zx tools/scrimlet/destroy-softnpu-zone.sh, 195 bytes, 1 tape blocks
1362023-11-04T20:44:55.876Zx tools/scrimlet/softnpu-init.sh, 2148 bytes, 5 tape blocks
1372023-11-04T20:44:55.900Zx tools/scrimlet/softnpu-zone.txt, 330 bytes, 1 tape blocks
1382023-11-04T20:44:55.924Zx tools/scrimlet/softnpu.toml, 171 bytes, 1 tape blocks
1392023-11-04T20:44:55.948Z
1402023-11-04T20:44:55.972Zreal 1.615389982
1412023-11-04T20:44:55.996Zuser 1.451412795
1422023-11-04T20:44:56.021Zsys 0.288717010
1432023-11-04T20:44:56.044Ztrap 0.000060351
1442023-11-04T20:44:56.069Ztflt 0.000000000
1452023-11-04T20:44:56.093Zdflt 0.000000000
1462023-11-04T20:44:56.117Zkflt 0.000000000
1472023-11-04T20:44:56.142Zlock 0.000000000
1482023-11-04T20:44:56.166Zslp 1.452700798
1492023-11-04T20:44:56.190Zlat 0.035654639
1502023-11-04T20:44:56.213Zstop 0.000126913
1512023-11-04T20:44:56.238Z+ cp /input/package/work/zones/clickhouse_keeper.tar.gz /input/package/work/zones/clickhouse.tar.gz /input/package/work/zones/cockroachdb.tar.gz /input/package/work/zones/crucible-pantry.tar.gz /input/package/work/zones/crucible.tar.gz /input/package/work/zones/external-dns.tar.gz /input/package/work/zones/internal-dns.tar.gz /input/package/work/zones/ntp.tar.gz /input/package/work/zones/omicron-gateway-asic.tar.gz /input/package/work/zones/omicron-gateway-softnpu.tar.gz /input/package/work/zones/omicron-nexus-single-sled.tar.gz /input/package/work/zones/omicron-nexus.tar.gz /input/package/work/zones/overlay.tar.gz /input/package/work/zones/oximeter-collector.tar.gz /input/package/work/zones/propolis-server.tar.gz /input/package/work/zones/switch-asic.tar.gz /input/package/work/zones/switch-softnpu.tar.gz out/
1522023-11-04T20:44:56.642Z+ mv out/omicron-nexus-single-sled.tar.gz out/omicron-nexus.tar.gz
1532023-11-04T20:44:56.666Z+ mkdir tests
1542023-11-04T20:44:56.690Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1552023-11-04T20:44:56.714Z+ ptime -m gunzip
1562023-11-04T20:44:56.738Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap-346a6c0cadffef35
1572023-11-04T20:44:56.802Z
1582023-11-04T20:44:56.826Zreal 0.141964011
1592023-11-04T20:44:56.850Zuser 0.129523310
1602023-11-04T20:44:56.874Zsys 0.012211135
1612023-11-04T20:44:56.898Ztrap 0.000012460
1622023-11-04T20:44:56.922Ztflt 0.000000000
1632023-11-04T20:44:56.947Zdflt 0.000000000
1642023-11-04T20:44:56.971Zkflt 0.000000000
1652023-11-04T20:44:56.995Zlock 0.000000000
1662023-11-04T20:44:57.019Zslp 0.000022930
1672023-11-04T20:44:57.043Zlat 0.000018070
1682023-11-04T20:44:57.067Zstop 0.000118453
1692023-11-04T20:44:57.091Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap-346a6c0cadffef35
1702023-11-04T20:44:57.114Z+ chmod a+x tests/bootstrap-346a6c0cadffef35
1712023-11-04T20:44:57.138Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1722023-11-04T20:44:57.162Z+ ptime -m gunzip
1732023-11-04T20:44:57.186Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap
1742023-11-04T20:44:59.131Z
1752023-11-04T20:44:59.155Zreal 2.317685131
1762023-11-04T20:44:59.178Zuser 2.179645590
1772023-11-04T20:44:59.202Zsys 0.137637913
1782023-11-04T20:44:59.226Ztrap 0.000181663
1792023-11-04T20:44:59.250Ztflt 0.000000000
1802023-11-04T20:44:59.274Zdflt 0.000000000
1812023-11-04T20:44:59.298Zkflt 0.000000000
1822023-11-04T20:44:59.322Zlock 0.000000000
1832023-11-04T20:44:59.346Zslp 0.000021270
1842023-11-04T20:44:59.370Zlat 0.000020350
1852023-11-04T20:44:59.394Zstop 0.000124642
1862023-11-04T20:44:59.418Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap
1872023-11-04T20:44:59.441Z+ chmod a+x tests/bootstrap
1882023-11-04T20:44:59.465Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1892023-11-04T20:44:59.489Z+ ptime -m gunzip
1902023-11-04T20:44:59.518Z++ basename /input/ci-tools/work/end-to-end-tests/end_to_end_tests-ad8d578e5459ec9d
1912023-11-04T20:45:01.755Z
1922023-11-04T20:45:01.780Zreal 2.612488157
1932023-11-04T20:45:01.805Zuser 2.450981108
1942023-11-04T20:45:01.830Zsys 0.161068074
1952023-11-04T20:45:01.855Ztrap 0.000217938
1962023-11-04T20:45:01.880Ztflt 0.000000000
1972023-11-04T20:45:01.905Zdflt 0.000000000
1982023-11-04T20:45:01.929Zkflt 0.000000000
1992023-11-04T20:45:01.954Zlock 0.000000000
2002023-11-04T20:45:01.980Zslp 0.000020760
2012023-11-04T20:45:02.005Zlat 0.000022470
2022023-11-04T20:45:02.031Zstop 0.000123163
2032023-11-04T20:45:02.055Z++ basename /input/ci-tools/work/end-to-end-tests/end_to_end_tests-ad8d578e5459ec9d
2042023-11-04T20:45:02.080Z+ chmod a+x tests/end_to_end_tests-ad8d578e5459ec9d
2052023-11-04T20:45:02.105Z+ bmat address ls
2062023-11-04T20:45:02.130ZNAME CIDR FIRST LAST COUNT
2072023-11-04T20:45:02.154Zextra 10.151.2.0/24 10.151.2.164 10.151.2.227 64
2082023-11-04T20:45:02.179Z++ bmat address ls -f extra -Ho first,last,count
2092023-11-04T20:45:02.203Z+ set -- 10.151.2.164 10.151.2.227 64
2102023-11-04T20:45:02.228Z+ EXTRA_IP_START=10.151.2.164
2112023-11-04T20:45:02.252Z+ EXTRA_IP_END=10.151.2.227
2122023-11-04T20:45:02.277Z+ EXTRA_IP_COUNT=64
2132023-11-04T20:45:02.302Z+ (( EXTRA_IP_COUNT >= 32 ))
2142023-11-04T20:45:02.326Z+ EXTRA_IP_BASE=10.151.2
2152023-11-04T20:45:02.351Z+ EXTRA_IP_FOCTET=164
2162023-11-04T20:45:02.375Z+ EXTRA_IP_LOCTET=227
2172023-11-04T20:45:02.400Z+ SERVICE_IP_POOL_START=10.151.2.164
2182023-11-04T20:45:02.451Z+ SERVICE_IP_POOL_END=10.151.2.173
2192023-11-04T20:45:02.476Z+ DNS_IP1=10.151.2.164
2202023-11-04T20:45:02.501Z+ DNS_IP2=10.151.2.165
2212023-11-04T20:45:02.527Z+ UPLINK_IP=10.151.2.174
2222023-11-04T20:45:02.552Z+ IPPOOL_START=10.151.2.175
2232023-11-04T20:45:02.596Z+ IPPOOL_END=10.151.2.227
2242023-11-04T20:45:02.621Z++ ipadm show-addr -po type,addr
2252023-11-04T20:45:02.647Z++ awk '-F[:/]' '$1 == "dhcp" {print $2}'
2262023-11-04T20:45:02.671Z+ GATEWAY_IP=10.151.2.100
2272023-11-04T20:45:02.696Z+ [[ -n 10.151.2.100 ]]
2282023-11-04T20:45:02.721Z+ ping -s 10.151.2.100 56 1
2292023-11-04T20:45:02.745ZPING 10.151.2.100: 56 data bytes
2302023-11-04T20:45:02.770Z64 bytes from 10.151.2.100: icmp_seq=0. time=0.011 ms
2312023-11-04T20:45:02.795Z
2322023-11-04T20:45:02.820Z----10.151.2.100 PING Statistics----
2332023-11-04T20:45:02.845Z1 packets transmitted, 1 packets received, 0% packet loss
2342023-11-04T20:45:02.870Zround-trip (ms) min/avg/max/stddev = 0.011/0.011/0.011/-nan
2352023-11-04T20:45:02.897Z++ arp -an
2362023-11-04T20:45:02.922Z++ awk -vgw=10.151.2.100 '$2 == gw {print $NF}'
2372023-11-04T20:45:02.946Z+ GATEWAY_MAC=18:c0:4d:0d:a0:2a
2382023-11-04T20:45:02.971Z+ routeadm -e ipv4-forwarding -u
2392023-11-04T20:45:02.996Z+ PXA_START=10.151.2.164
2402023-11-04T20:45:03.021Z+ PXA_END=10.151.2.227
2412023-11-04T20:45:03.045Z+ export GATEWAY_IP GATEWAY_MAC PXA_START PXA_END
2422023-11-04T20:45:03.070Z+ pfexec zpool create -f scratch c1t1d0 c2t1d0
2432023-11-04T20:45:03.096Z+ ZPOOL_VDEV_DIR=/scratch
2442023-11-04T20:45:03.122Z+ ptime -m pfexec ./tools/create_virtual_hardware.sh
2452023-11-04T20:45:03.148Z+++ dirname ./tools/create_virtual_hardware.sh
2462023-11-04T20:45:03.174Z++ cd ./tools
2472023-11-04T20:45:03.201Z++ pwd
2482023-11-04T20:45:03.225Z+ SOURCE_DIR=/opt/oxide/work/tools
2492023-11-04T20:45:03.250Z+ OMICRON_TOP=/opt/oxide/work/tools/..
2502023-11-04T20:45:03.275Z+ SOFTNPU_MODE=zone
2512023-11-04T20:45:03.300Z+ . /opt/oxide/work/tools/virtual_hardware.sh
2522023-11-04T20:45:03.326Z++ MARKER=/etc/opt/oxide/NO_INSTALL
2532023-11-04T20:45:03.351Z++ [[ -f /etc/opt/oxide/NO_INSTALL ]]
2542023-11-04T20:45:03.378Z++ dladm show-phys -p -o LINK
2552023-11-04T20:45:03.403Z++ head -1
2562023-11-04T20:45:03.427Z+ PHYSICAL_LINK=igb0
2572023-11-04T20:45:03.453Z+ echo 'Using igb0 as physical link'
2582023-11-04T20:45:03.477ZUsing igb0 as physical link
2592023-11-04T20:45:03.504Z+ ensure_run_as_root
2602023-11-04T20:45:03.529Z++ id -u
2612023-11-04T20:45:03.554Z+ [[ 0 -ne 0 ]]
2622023-11-04T20:45:03.579Z+ ensure_zpools
2632023-11-04T20:45:03.624Z+ ZPOOL_TYPES=('oxp_' 'oxi_')
2642023-11-04T20:45:03.650Z+ for ZPOOL_TYPE in "${ZPOOL_TYPES[@]}"
2652023-11-04T20:45:03.675Z+ readarray -t ZPOOLS
2662023-11-04T20:45:03.700Z++ grep '"oxp_' /opt/oxide/work/tools/../smf/sled-agent/non-gimlet/config.toml
2672023-11-04T20:45:03.725Z++ sed 's/[ ",]//g'
2682023-11-04T20:45:03.749Z+ for ZPOOL in "${ZPOOLS[@]}"
2692023-11-04T20:45:03.774Z+ echo 'Zpool: [oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b]'
2702023-11-04T20:45:03.799ZZpool: [oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b]
2712023-11-04T20:45:03.825Z+ VDEV_PATH=/scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
2722023-11-04T20:45:03.853Z+ [[ -f /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
2732023-11-04T20:45:03.883Z+ dd if=/dev/zero of=/scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
2742023-11-04T20:45:03.919Z0+0 records in
2752023-11-04T20:45:03.947Z0+0 records out
2762023-11-04T20:45:03.977Z0 bytes (0 iB) transferred in 0.000007 secs (0 iB/sec)
2772023-11-04T20:45:04.007Z+ success 'ZFS vdev /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
2782023-11-04T20:45:04.040Z+ set +x
2792023-11-04T20:45:04.069ZZFS vdev /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
2802023-11-04T20:45:04.099Z++ zpool list -o name
2812023-11-04T20:45:04.129Z++ grep oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b
2822023-11-04T20:45:04.158ZZFS zpool oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b exists
2832023-11-04T20:45:04.187Z+ [[ -z '' ]]
2842023-11-04T20:45:04.218ZZpool: [oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03]
2852023-11-04T20:45:04.249Z+ zpool create -f oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
2862023-11-04T20:45:04.281ZZFS vdev /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
2872023-11-04T20:45:04.311Z+ success 'ZFS zpool oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
2882023-11-04T20:45:04.342ZZFS zpool oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
2892023-11-04T20:45:04.375Z+ set +x
2902023-11-04T20:45:04.406ZZpool: [oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03]
2912023-11-04T20:45:04.436Z+ for ZPOOL in "${ZPOOLS[@]}"
2922023-11-04T20:45:04.466ZZFS vdev /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
2932023-11-04T20:45:04.498Z+ echo 'Zpool: [oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03]'
2942023-11-04T20:45:04.528ZZFS zpool oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
2952023-11-04T20:45:04.561Z+ VDEV_PATH=/scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
2962023-11-04T20:45:04.591ZZpool: [oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03]
2972023-11-04T20:45:04.623Z+ [[ -f /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
2982023-11-04T20:45:04.676ZZFS vdev /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
2992023-11-04T20:45:05.111Z+ dd if=/dev/zero of=/scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3002023-11-04T20:45:05.152ZZFS zpool oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
3012023-11-04T20:45:05.178Z0+0 records in
3022023-11-04T20:45:05.203ZZpool: [oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03]
3032023-11-04T20:45:05.229Z0+0 records out
3042023-11-04T20:45:05.253ZZFS vdev /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
3052023-11-04T20:45:05.278Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3062023-11-04T20:45:05.302ZZFS zpool oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
3072023-11-04T20:45:05.327Z+ success 'ZFS vdev /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3082023-11-04T20:45:05.353ZZpool: [oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930]
3092023-11-04T20:45:05.378Z+ set +x
3102023-11-04T20:45:05.402ZZFS vdev /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev exists
3112023-11-04T20:45:05.426Z++ zpool list -o name
3122023-11-04T20:45:05.452ZZFS zpool oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 exists
3132023-11-04T20:45:05.476Z++ grep oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03
3142023-11-04T20:45:05.501ZZpool: [oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60]
3152023-11-04T20:45:05.526Z+ [[ -z '' ]]
3162023-11-04T20:45:05.550ZZFS vdev /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev exists
3172023-11-04T20:45:05.576Z+ zpool create -f oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3182023-11-04T20:45:05.601ZZFS zpool oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 exists
3192023-11-04T20:45:05.626Z+ success 'ZFS zpool oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3202023-11-04T20:45:05.650ZZpool: [oxp_31bd71cd-4736-4a12-a387-9b74b050396f]
3212023-11-04T20:45:05.675Z+ set +x
3222023-11-04T20:45:05.703ZZFS vdev /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev exists
3232023-11-04T20:45:05.728Z+ for ZPOOL in "${ZPOOLS[@]}"
3242023-11-04T20:45:05.752ZZFS zpool oxp_31bd71cd-4736-4a12-a387-9b74b050396f exists
3252023-11-04T20:45:05.777Z+ echo 'Zpool: [oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3262023-11-04T20:45:05.802ZZpool: [oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7]
3272023-11-04T20:45:05.827Z+ VDEV_PATH=/scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3282023-11-04T20:45:05.852ZZFS vdev /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev exists
3292023-11-04T20:45:05.876Z+ [[ -f /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3302023-11-04T20:45:05.901ZZFS zpool oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 exists
3312023-11-04T20:45:05.925Z+ dd if=/dev/zero of=/scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3322023-11-04T20:45:05.950ZZpool: [oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b]
3332023-11-04T20:45:05.976Z0+0 records in
3342023-11-04T20:45:06.002ZZFS vdev /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
3352023-11-04T20:45:06.027Z0+0 records out
3362023-11-04T20:45:06.052ZZFS zpool oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b exists
3372023-11-04T20:45:06.077Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3382023-11-04T20:45:06.101ZZpool: [oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b]
3392023-11-04T20:45:06.126Z+ success 'ZFS vdev /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3402023-11-04T20:45:06.151ZZFS vdev /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
3412023-11-04T20:45:06.175Z+ set +x
3422023-11-04T20:45:06.200ZZFS zpool oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b exists
3432023-11-04T20:45:06.225Z++ zpool list -o name
3442023-11-04T20:45:06.249ZSimnet net0/sc0_0 exists
3452023-11-04T20:45:06.273Z++ grep oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03
3462023-11-04T20:45:06.298ZSimnet net1/sc1_0 exists
3472023-11-04T20:45:06.323Z+ [[ -z '' ]]
3482023-11-04T20:45:06.347ZVnic sc0_1 exists
3492023-11-04T20:45:06.371Z+ zpool create -f oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3502023-11-04T20:45:06.396Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/softnpu/image/c1c42398c82b0220c8b5fa3bfba9c7a3bcaa0943/softnpu.sha256.txt
3512023-11-04T20:45:06.422Z+ success 'ZFS zpool oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3522023-11-04T20:45:06.447Z+ set +x
3532023-11-04T20:45:06.472Z+ for ZPOOL in "${ZPOOLS[@]}"
3542023-11-04T20:45:06.496Z+ echo 'Zpool: [oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3552023-11-04T20:45:06.521Z+ VDEV_PATH=/scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3562023-11-04T20:45:06.546Z+ [[ -f /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3572023-11-04T20:45:06.570Z+ dd if=/dev/zero of=/scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3582023-11-04T20:45:06.594Z0+0 records in
3592023-11-04T20:45:06.619Z0+0 records out
3602023-11-04T20:45:06.643Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3612023-11-04T20:45:06.668Z+ success 'ZFS vdev /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3622023-11-04T20:45:06.692Z+ set +x
3632023-11-04T20:45:06.729Z++ zpool list -o name
3642023-11-04T20:45:06.754Z++ grep oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03
3652023-11-04T20:45:06.778Z+ [[ -z '' ]]
3662023-11-04T20:45:06.802Z+ zpool create -f oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3672023-11-04T20:45:06.827Z+ success 'ZFS zpool oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3682023-11-04T20:45:06.851Z+ set +x
3692023-11-04T20:45:06.875Z+ for ZPOOL in "${ZPOOLS[@]}"
3702023-11-04T20:45:06.900Z+ echo 'Zpool: [oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3712023-11-04T20:45:06.925Z+ VDEV_PATH=/scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3722023-11-04T20:45:06.950Z+ [[ -f /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3732023-11-04T20:45:06.974Z+ dd if=/dev/zero of=/scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3742023-11-04T20:45:07.000Z0+0 records in
3752023-11-04T20:45:07.024Z0+0 records out
3762023-11-04T20:45:07.049Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
3772023-11-04T20:45:07.073Z+ success 'ZFS vdev /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3782023-11-04T20:45:07.103Z+ set +x
3792023-11-04T20:45:07.128Z++ zpool list -o name
3802023-11-04T20:45:07.153Z++ grep oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03
3812023-11-04T20:45:07.177Z+ [[ -z '' ]]
3822023-11-04T20:45:07.202Z+ zpool create -f oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3832023-11-04T20:45:07.226Z+ success 'ZFS zpool oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3842023-11-04T20:45:07.251Z+ set +x
3852023-11-04T20:45:07.275Z+ for ZPOOL in "${ZPOOLS[@]}"
3862023-11-04T20:45:07.300Z+ echo 'Zpool: [oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930]'
3872023-11-04T20:45:07.324Z+ VDEV_PATH=/scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev
3882023-11-04T20:45:07.348Z+ [[ -f /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev ]]
3892023-11-04T20:45:07.373Z+ dd if=/dev/zero of=/scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev bs=1 count=0 seek=20G
3902023-11-04T20:45:07.398Z0+0 records in
3912023-11-04T20:45:07.422Z0+0 records out
3922023-11-04T20:45:07.447Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3932023-11-04T20:45:07.471Z+ success 'ZFS vdev /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev exists'
3942023-11-04T20:45:07.496Z+ set +x
3952023-11-04T20:45:07.520Z++ zpool list -o name
3962023-11-04T20:45:07.547Z++ grep oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930
3972023-11-04T20:45:07.597Z+ [[ -z '' ]]
3982023-11-04T20:45:07.622Z+ zpool create -f oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev
3992023-11-04T20:45:07.646Z+ success 'ZFS zpool oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 exists'
4002023-11-04T20:45:07.673Z+ set +x
4012023-11-04T20:45:07.698Z+ for ZPOOL in "${ZPOOLS[@]}"
4022023-11-04T20:45:07.722Z+ echo 'Zpool: [oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60]'
4032023-11-04T20:45:07.755Z+ VDEV_PATH=/scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev
4042023-11-04T20:45:07.779Z+ [[ -f /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev ]]
4052023-11-04T20:45:07.803Z+ dd if=/dev/zero of=/scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev bs=1 count=0 seek=20G
4062023-11-04T20:45:07.828Z0+0 records in
4072023-11-04T20:45:07.852Z0+0 records out
4082023-11-04T20:45:07.876Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4092023-11-04T20:45:07.901Z+ success 'ZFS vdev /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev exists'
4102023-11-04T20:45:07.925Z+ set +x
4112023-11-04T20:45:07.950Z++ zpool list -o name
4122023-11-04T20:45:07.974Z++ grep oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60
4132023-11-04T20:45:07.999Z+ [[ -z '' ]]
4142023-11-04T20:45:08.023Z+ zpool create -f oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev
4152023-11-04T20:45:08.048Z+ success 'ZFS zpool oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 exists'
4162023-11-04T20:45:08.073Z+ set +x
4172023-11-04T20:45:08.098Z+ for ZPOOL in "${ZPOOLS[@]}"
4182023-11-04T20:45:08.123Z+ echo 'Zpool: [oxp_31bd71cd-4736-4a12-a387-9b74b050396f]'
4192023-11-04T20:45:08.147Z+ VDEV_PATH=/scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev
4202023-11-04T20:45:08.172Z+ [[ -f /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev ]]
4212023-11-04T20:45:08.196Z+ dd if=/dev/zero of=/scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev bs=1 count=0 seek=20G
4222023-11-04T20:45:08.220Z0+0 records in
4232023-11-04T20:45:08.245Z0+0 records out
4242023-11-04T20:45:08.269Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4252023-11-04T20:45:08.294Z+ success 'ZFS vdev /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev exists'
4262023-11-04T20:45:08.318Z+ set +x
4272023-11-04T20:45:08.343Z++ zpool list -o name
4282023-11-04T20:45:08.367Z++ grep oxp_31bd71cd-4736-4a12-a387-9b74b050396f
4292023-11-04T20:45:08.391Z+ [[ -z '' ]]
4302023-11-04T20:45:08.416Z+ zpool create -f oxp_31bd71cd-4736-4a12-a387-9b74b050396f /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev
4312023-11-04T20:45:08.441Z+ success 'ZFS zpool oxp_31bd71cd-4736-4a12-a387-9b74b050396f exists'
4322023-11-04T20:45:08.466Z+ set +x
4332023-11-04T20:45:08.493Z+ for ZPOOL in "${ZPOOLS[@]}"
4342023-11-04T20:45:08.517Z+ echo 'Zpool: [oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7]'
4352023-11-04T20:45:08.546Z+ VDEV_PATH=/scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev
4362023-11-04T20:45:08.570Z+ [[ -f /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev ]]
4372023-11-04T20:45:08.594Z+ dd if=/dev/zero of=/scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev bs=1 count=0 seek=20G
4382023-11-04T20:45:08.620Z0+0 records in
4392023-11-04T20:45:08.658Z0+0 records out
4402023-11-04T20:45:08.691Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4412023-11-04T20:45:08.716Z+ success 'ZFS vdev /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev exists'
4422023-11-04T20:45:08.740Z+ set +x
4432023-11-04T20:45:08.788Z++ zpool list -o name
4442023-11-04T20:45:08.812Z++ grep oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7
4452023-11-04T20:45:08.837Z+ [[ -z '' ]]
4462023-11-04T20:45:08.861Z+ zpool create -f oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev
4472023-11-04T20:45:08.886Z+ success 'ZFS zpool oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 exists'
4482023-11-04T20:45:08.910Z+ set +x
4492023-11-04T20:45:08.937Z+ for ZPOOL_TYPE in "${ZPOOL_TYPES[@]}"
4502023-11-04T20:45:08.964Z+ readarray -t ZPOOLS
4512023-11-04T20:45:08.988Z++ grep '"oxi_' /opt/oxide/work/tools/../smf/sled-agent/non-gimlet/config.toml
4522023-11-04T20:45:09.013Z++ sed 's/[ ",]//g'
4532023-11-04T20:45:09.037Z+ for ZPOOL in "${ZPOOLS[@]}"
4542023-11-04T20:45:09.062Z+ echo 'Zpool: [oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b]'
4552023-11-04T20:45:09.086Z+ VDEV_PATH=/scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4562023-11-04T20:45:09.514Z+ [[ -f /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
4572023-11-04T20:45:09.543Z+ dd if=/dev/zero of=/scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
4582023-11-04T20:45:09.568Z0+0 records in
4592023-11-04T20:45:09.592Z0+0 records out
4602023-11-04T20:45:09.617Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
4612023-11-04T20:45:09.643Z+ success 'ZFS vdev /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
4622023-11-04T20:45:09.667Z+ set +x
4632023-11-04T20:45:09.691Z++ zpool list -o name
4642023-11-04T20:45:09.716Z++ grep oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b
4652023-11-04T20:45:09.740Z+ [[ -z '' ]]
4662023-11-04T20:45:09.765Z+ zpool create -f oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4672023-11-04T20:45:09.814Z+ success 'ZFS zpool oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
4682023-11-04T20:45:09.838Z+ set +x
4692023-11-04T20:45:09.862Z+ for ZPOOL in "${ZPOOLS[@]}"
4702023-11-04T20:45:09.886Z+ echo 'Zpool: [oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b]'
4712023-11-04T20:45:09.910Z+ VDEV_PATH=/scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4722023-11-04T20:45:09.934Z+ [[ -f /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
4732023-11-04T20:45:09.960Z+ dd if=/dev/zero of=/scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
4742023-11-04T20:45:09.984Z0+0 records in
4752023-11-04T20:45:10.009Z0+0 records out
4762023-11-04T20:45:10.033Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4772023-11-04T20:45:10.057Z+ success 'ZFS vdev /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
4782023-11-04T20:45:10.082Z+ set +x
4792023-11-04T20:45:10.106Z++ zpool list -o name
4802023-11-04T20:45:10.131Z++ grep oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b
4812023-11-04T20:45:10.155Z+ [[ -z '' ]]
4822023-11-04T20:45:10.179Zdownloading softnpu ...done
4832023-11-04T20:45:10.203Z+ zpool create -f oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4842023-11-04T20:45:10.227Z+ success 'ZFS zpool oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
4852023-11-04T20:45:10.251Z+ set +x
4862023-11-04T20:45:10.275Z+ [[ zone == \z\o\n\e ]]
4872023-11-04T20:45:10.344Z+ ensure_simulated_links igb0
4882023-11-04T20:45:10.368Z+ local PHYSICAL_LINK=igb0
4892023-11-04T20:45:10.392Z+ INDICES=("0" "1")
4902023-11-04T20:45:10.417Z+ for I in "${INDICES[@]}"
4912023-11-04T20:45:10.442Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/sidecar-lite/release/f0585a29fb0285f7a1220c1118856b0e5c1f75c5/libsidecar_lite.so.sha256.txt
4922023-11-04T20:45:10.467Z++ get_simnet_name_if_exists net0
4932023-11-04T20:45:10.492Z++ dladm show-simnet -p -o LINK net0
4942023-11-04T20:45:10.518Z++ echo ''
4952023-11-04T20:45:10.543Z+ [[ -z '' ]]
4962023-11-04T20:45:10.569Z+ dladm create-simnet -t net0
4972023-11-04T20:45:10.601Z+ dladm create-simnet -t sc0_0
4982023-11-04T20:45:10.627Z+ dladm modify-simnet -t -p net0 sc0_0
4992023-11-04T20:45:10.654Z+ dladm set-linkprop -p mtu=9000 sc0_0
5002023-11-04T20:45:10.680Z+ success 'Simnet net0/sc0_0 exists'
5012023-11-04T20:45:10.707Z+ set +x
5022023-11-04T20:45:10.737Z+ for I in "${INDICES[@]}"
5032023-11-04T20:45:10.764Z++ get_simnet_name_if_exists net1
5042023-11-04T20:45:10.791Z++ dladm show-simnet -p -o LINK net1
5052023-11-04T20:45:10.853Z++ echo ''
5062023-11-04T20:45:10.884Z+ [[ -z '' ]]
5072023-11-04T20:45:10.910Z+ dladm create-simnet -t net1
5082023-11-04T20:45:10.934Z+ dladm create-simnet -t sc1_0
5092023-11-04T20:45:10.958Z+ dladm modify-simnet -t -p net1 sc1_0
5102023-11-04T20:45:10.983Z+ dladm set-linkprop -p mtu=9000 sc1_0
5112023-11-04T20:45:11.007Z+ success 'Simnet net1/sc1_0 exists'
5122023-11-04T20:45:11.031Z+ set +x
5132023-11-04T20:45:11.055Z++ get_vnic_name_if_exists sc0_1
5142023-11-04T20:45:11.079Z++ dladm show-vnic -p -o LINK sc0_1
5152023-11-04T20:45:11.103Z++ echo ''
5162023-11-04T20:45:11.128Z+ [[ -z '' ]]
5172023-11-04T20:45:11.152Z+ dladm create-vnic -t sc0_1 -l igb0 -m a8:e1:de:01:70:1d
5182023-11-04T20:45:11.177Z+ [[ -v PROMISC_FILT_OFF ]]
5192023-11-04T20:45:11.200Z+ success 'Vnic sc0_1 exists'
5202023-11-04T20:45:11.225Z+ set +x
5212023-11-04T20:45:11.249Z+ warn_if_no_proxy_arp
5222023-11-04T20:45:11.273Z+ [[ -v PXA_START ]]
5232023-11-04T20:45:11.321Z+ [[ -v PXA_END ]]
5242023-11-04T20:45:11.355Z+ ensure_softnpu_zone
5252023-11-04T20:45:11.784Z+ zoneadm list
5262023-11-04T20:45:11.820Z+ grep -q sidecar_softnpu
5272023-11-04T20:45:12.275Z+ [[ -f out/npuzone/npuzone ]]
5282023-11-04T20:45:12.300Z+ out/npuzone/npuzone create sidecar --omicron-zone --ports sc0_0,tfportrear0_0 --ports sc0_1,tfportqsfp0_0 --sidecar-lite-branch main
5292023-11-04T20:45:14.579Zdownloading asic_program.so ...done
5302023-11-04T20:45:14.625Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/sidecar-lite/release/f0585a29fb0285f7a1220c1118856b0e5c1f75c5/scadm.sha256.txt
5312023-11-04T20:45:20.805Zdownloading scadm ...done
5322023-11-04T20:45:20.829Zcreating port tfportrear0_0 for sc0_0
5332023-11-04T20:45:20.852Zcreating port tfportqsfp0_0 for sc0_1
5342023-11-04T20:45:20.876Zsoftnpu zone setup
5352023-11-04T20:45:20.900Zconfigure zone
5362023-11-04T20:45:20.924Zinstall zone
5372023-11-04T20:45:21.919Zboot zone
5382023-11-04T20:45:25.030Z[sidecar_softnpu] svcs milestone/network
5392023-11-04T20:45:25.056ZSTATE STIME FMRI
5402023-11-04T20:45:25.080Zonline 20:45:23 svc:/milestone/network:default
5412023-11-04T20:45:25.104Z[sidecar_softnpu] /softnpu/npu start
5422023-11-04T20:45:25.128Z
5432023-11-04T20:45:25.153Z+ /opt/oxide/work/tools/scrimlet/softnpu-init.sh
5442023-11-04T20:45:25.176Z+ GATEWAY_IP=10.151.2.100
5452023-11-04T20:45:25.200Z+ echo 'Using 10.151.2.100 as gateway ip'
5462023-11-04T20:45:25.225ZUsing 10.151.2.100 as gateway ip
5472023-11-04T20:45:25.255Z+ [[ ! -v GATEWAY_MAC ]]
5482023-11-04T20:45:25.279Z+ GATEWAY_MAC=18:c0:4d:0d:a0:2a
5492023-11-04T20:45:25.303Z++ grep -c -E '^([0-9a-fA-F]{2}:){5}[0-9a-fA-F]{2}$' /dev/fd/63
5502023-11-04T20:45:25.327Z+++ echo 18:c0:4d:0d:a0:2a
5512023-11-04T20:45:25.394Z+ COUNT=1
5522023-11-04T20:45:25.418Z+ [[ 1 -ne 1 ]]
5532023-11-04T20:45:25.442Z+ echo 'Using 18:c0:4d:0d:a0:2a as gateway mac'
5542023-11-04T20:45:25.466ZUsing 18:c0:4d:0d:a0:2a as gateway mac
5552023-11-04T20:45:25.490Z+ z_scadm add-arp-entry 10.151.2.100 18:c0:4d:0d:a0:2a
5562023-11-04T20:45:25.513Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone add-arp-entry 10.151.2.100 18:c0:4d:0d:a0:2a
5572023-11-04T20:45:25.537Z+ PXA_MAC=a8:e1:de:01:70:1d
5582023-11-04T20:45:25.561Z+ [[ -v PXA_START ]]
5592023-11-04T20:45:25.585Z+ z_scadm add-proxy-arp 10.151.2.164 10.151.2.227 a8:e1:de:01:70:1d
5602023-11-04T20:45:25.609Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone add-proxy-arp 10.151.2.164 10.151.2.227 a8:e1:de:01:70:1d
5612023-11-04T20:45:25.633Z+ z_scadm dump-state
5622023-11-04T20:45:25.657Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone dump-state
5632023-11-04T20:45:25.681Zlocal v6:
5642023-11-04T20:45:25.705Zlocal v4:
5652023-11-04T20:45:25.729Zrouter v6:
5662023-11-04T20:45:25.753Zrouter v4_idx:
5672023-11-04T20:45:25.777Zrouter v4_routes:
5682023-11-04T20:45:25.801Zresolver v4:
5692023-11-04T20:45:25.825Z10.151.2.100 -> 18:c0:4d:0d:a0:2a
5702023-11-04T20:45:25.849Zresolver v6:
5712023-11-04T20:45:25.873Znat v4:
5722023-11-04T20:45:25.897Znat_v6:
5732023-11-04T20:45:25.921Zport_mac:
5742023-11-04T20:45:25.945Zproxy arp:
5752023-11-04T20:45:25.968Z10.151.2.164/10.151.2.227: a8:e1:de:01:70:1d
5762023-11-04T20:45:25.992Z+ success 'softnpu zone exists'
5772023-11-04T20:45:26.016Z+ set +x
5782023-11-04T20:45:26.040Zsoftnpu zone exists
5792023-11-04T20:45:26.064Z
5802023-11-04T20:45:26.088Zreal 22.996166193
5812023-11-04T20:45:26.112Zuser 0.739679034
5822023-11-04T20:45:26.136Zsys 1.655373966
5832023-11-04T20:45:26.160Ztrap 0.000080485
5842023-11-04T20:45:26.184Ztflt 0.000211787
5852023-11-04T20:45:26.208Zdflt 0.001159577
5862023-11-04T20:45:26.232Zkflt 0.000000000
5872023-11-04T20:45:26.280Zlock 17:49.832638256
5882023-11-04T20:45:26.304Zslp 1:08.126939749
5892023-11-04T20:45:26.328Zlat 0.084249725
5902023-11-04T20:45:26.352Zstop 0.359311021
5912023-11-04T20:45:26.376Z+ tar xf out/omicron-sled-agent.tar pkg/config-rss.toml
5922023-11-04T20:45:26.407Z++ sed -n 's/silo_name = "\(.*\)"/\1/p' pkg/config-rss.toml
5932023-11-04T20:45:26.431Z+ SILO_NAME=recovery
5942023-11-04T20:45:26.455Z++ sed -n 's/external_dns_zone_name = "\(.*\)"/\1/p' pkg/config-rss.toml
5952023-11-04T20:45:26.479Z+ EXTERNAL_DNS_DOMAIN=oxide.test
5962023-11-04T20:45:26.504Z+ sed -i~ '
5972023-11-04T20:45:26.528Z /^external_dns_ips/c\
5982023-11-04T20:45:26.552Zexternal_dns_ips = [ "10.151.2.164", "10.151.2.165" ]
5992023-11-04T20:45:26.576Z /^\[\[internal_services_ip_pool_ranges/,/^$/ {
6002023-11-04T20:45:26.600Z /^first/c\
6012023-11-04T20:45:26.624Zfirst = "10.151.2.164"
6022023-11-04T20:45:26.648Z /^last/c\
6032023-11-04T20:45:26.671Zlast = "10.151.2.173"
6042023-11-04T20:45:26.695Z }
6052023-11-04T20:45:26.719Z /^\[rack_network_config/,/^$/ {
6062023-11-04T20:45:26.743Z /^infra_ip_first/c\
6072023-11-04T20:45:26.768Zinfra_ip_first = "10.151.2.174"
6082023-11-04T20:45:26.793Z /^infra_ip_last/c\
6092023-11-04T20:45:26.818Zinfra_ip_last = "10.151.2.174"
6102023-11-04T20:45:26.843Z }
6112023-11-04T20:45:26.867Z /^\[\[rack_network_config.ports/,/^$/ {
6122023-11-04T20:45:26.890Z--- pkg/config-rss.toml~ Mon Jul 24 01:21:28 2006
6132023-11-04T20:45:26.914Z /^routes/c\
6142023-11-04T20:45:26.938Z+++ pkg/config-rss.toml Sat Nov 4 20:45:25 2023
6152023-11-04T20:45:26.962Zroutes = \[{nexthop = "10.151.2.100", destination = "0.0.0.0/0"}\]
6162023-11-04T20:45:26.986Z@@ -41,7 +41,7 @@
6172023-11-04T20:45:27.010Z /^addresses/c\
6182023-11-04T20:45:27.034Z # the DNS domain delegated to the rack by the customer. Each of these addresses
6192023-11-04T20:45:27.060Zaddresses = \["10.151.2.174/32"\]
6202023-11-04T20:45:27.084Z # must be contained in one of the "internal services" IP Pool ranges listed
6212023-11-04T20:45:27.108Z }
6222023-11-04T20:45:27.132Z # below.
6232023-11-04T20:45:27.156Z' pkg/config-rss.toml
6242023-11-04T20:45:27.180Z-external_dns_ips = [ "192.168.1.20", "192.168.1.21" ]
6252023-11-04T20:45:27.204Z+ diff -u pkg/config-rss.toml~ pkg/config-rss.toml
6262023-11-04T20:45:27.228Z+external_dns_ips = [ "10.151.2.164", "10.151.2.165" ]
6272023-11-04T20:45:27.252Z+ true
6282023-11-04T20:45:27.276Z
6292023-11-04T20:45:27.305Z+ tar rvf out/omicron-sled-agent.tar pkg/config-rss.toml
6302023-11-04T20:45:27.329Z # Initial TLS certificates for the external API
6312023-11-04T20:45:27.353Z+ rm -f pkg/config-rss.toml pkg/config-rss.toml~
6322023-11-04T20:45:27.376Z #
6332023-11-04T20:45:27.426Z+ TLS_NAME=recovery.sys.oxide.test
6342023-11-04T20:45:27.450Z@@ -77,8 +77,8 @@
6352023-11-04T20:45:27.474Z+ openssl req -newkey rsa:4096 -x509 -sha256 -days 3 -nodes -out pkg/initial-tls-cert.pem -keyout pkg/initial-tls-key.pem -subj /CN=recovery.sys.oxide.test -addext subjectAltName=DNS:recovery.sys.oxide.test -addext basicConstraints=critical,CA:FALSE -config /dev/stdin
6362023-11-04T20:45:27.498Z #
6372023-11-04T20:45:27.522Z.+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*..........+..+.+...+..+....+.........+..+....+...+......+..+...+.+.........+............+...+..+....+.....+................+...+..+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*..............+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
6382023-11-04T20:45:27.546Z # For more on this and what to put here, see docs/how-to-run.adoc.
6392023-11-04T20:45:27.570Z [[internal_services_ip_pool_ranges]]
6402023-11-04T20:45:27.594Z-first = "192.168.1.20"
6412023-11-04T20:45:27.618Z-last = "192.168.1.29"
6422023-11-04T20:45:27.642Z+first = "10.151.2.164"
6432023-11-04T20:45:27.666Z+last = "10.151.2.173"
6442023-11-04T20:45:27.691Z
6452023-11-04T20:45:27.715Z # TODO - this configuration is subject to change going forward. Ultimately these
6462023-11-04T20:45:27.739Z # parameters should be provided to the control plane via wicket, but we need to
6472023-11-04T20:45:27.764Z@@ -92,8 +92,8 @@
6482023-11-04T20:45:27.788Z # A range of IP addresses used by Boundary Services on the external network. In
6492023-11-04T20:45:27.812Z # a real system, these would be addresses of the uplink ports on the Sidecar.
6502023-11-04T20:45:27.836Z # With softnpu, only one address is used.
6512023-11-04T20:45:27.859Z-infra_ip_first = "192.168.1.30"
6522023-11-04T20:45:27.883Z-infra_ip_last = "192.168.1.30"
6532023-11-04T20:45:27.907Z+infra_ip_first = "10.151.2.174"
6542023-11-04T20:45:27.931Z+infra_ip_last = "10.151.2.174"
6552023-11-04T20:45:27.955Z
6562023-11-04T20:45:27.979Z # Configurations for BGP routers to run on the scrimlets.
6572023-11-04T20:45:28.003Z bgp = []
6582023-11-04T20:45:28.027Z@@ -101,9 +101,9 @@
6592023-11-04T20:45:28.051Z # You can configure multiple uplinks by repeating the following stanza
6602023-11-04T20:45:28.075Z [[rack_network_config.ports]]
6612023-11-04T20:45:28.099Z # Routes associated with this port.
6622023-11-04T20:45:28.123Z-routes = [{nexthop = "192.168.1.199", destination = "0.0.0.0/0"}]
6632023-11-04T20:45:28.147Z+routes = [{nexthop = "10.151.2.100", destination = "0.0.0.0/0"}]
6642023-11-04T20:45:28.171Z # Addresses associated with this port.
6652023-11-04T20:45:28.195Z........+...+..+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.......+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.......+......+.....+.+...+.....+...+.........+..................+......+...................+....................+...............+......+.......+..+.+.....+....+.................+.+...+.........+.....+.......+.........+..+............+...+....+...+..+....+..+...+....+......+...........+...+.+............+........+................+...+...+........+....+...+....................+.......+...+..+..........+...........+.+...............+..............+..........+..............+.+..+.........+.............+...+..+............+.+........+.+..............+....+........+...+...........................+............+.+............+.....+.........+...+.+.....+.+.....+.+........+............+...+....+...+............+...+.....+.......+...........+..............................+.......+..+...+............+.......+.....+.+..............+...+.+.........+..+....+............+..+.......+.........+.....+..........+..+....+...+..+.+.....+....+...+...+..............+.+..+.+.........+............+...............+.......................+............+.+..+............+.....................+.........+....+.....+...............+....+...........+...............+.........+...+...+......+.+...+.........+..+.+...+..............+......+.......+..+...+...+.+........+.............+..+.+...........+.+..+...+....+.....+.+............+..+.+...+...............+.........+......+.....+......+.+.........+.........+..+...............+.........+.........+......+.............+.....+.+.....+....+........+....+.....+..................+.+..+............+......+......+......+...............+....+........+...+....+...+...........+......+.+........+......+.+.....+....+...+.....+............+......+.......+............+...+.....+.+...............+.........+......+......+......+..+...+.......+..............................+..+.............+..+..................+...............+.+...........+.............+...+..............+....+.........+.....+.+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
6662023-11-04T20:45:28.220Z-addresses = ["192.168.1.30/32"]
6672023-11-04T20:45:28.243Z-----
6682023-11-04T20:45:28.267Z+addresses = ["10.151.2.174/32"]
6692023-11-04T20:45:28.291Z+ tar rvf out/omicron-sled-agent.tar pkg/initial-tls-cert.pem pkg/initial-tls-key.pem
6702023-11-04T20:45:28.331Z # Name of the uplink port. This should always be "qsfp0" when using softnpu.
6712023-11-04T20:45:28.355Z+ rm -f pkg/initial-tls-cert.pem pkg/initial-tls-key.pem
6722023-11-04T20:45:28.378Z port = "qsfp0"
6732023-11-04T20:45:28.402Z+ rmdir pkg
6742023-11-04T20:45:28.426Z # The speed of this port.
6752023-11-04T20:45:28.450Z+ E2E_TLS_CERT=/opt/oxide/sled-agent/pkg/initial-tls-cert.pem
6762023-11-04T20:45:28.474Za pkg/config-rss.toml 7K
6772023-11-04T20:45:28.498Z+ pfexec mkdir -p /usr/oxide
6782023-11-04T20:45:28.523Za pkg/initial-tls-cert.pem 2K
6792023-11-04T20:45:28.546Z+ pfexec rm -f /usr/oxide/tcpproxy
6802023-11-04T20:45:28.570Za pkg/initial-tls-key.pem 4K
6812023-11-04T20:45:28.595Z+ pfexec curl -sSfL -o /usr/oxide/tcpproxy http://catacomb.eng.oxide.computer:12346/tcpproxy
6822023-11-04T20:45:28.619Z+ pfexec chmod +x /usr/oxide/tcpproxy
6832023-11-04T20:45:28.643Z+ pfexec rm -f /var/svc/manifest/site/tcpproxy.xml
6842023-11-04T20:45:28.667Z+ pfexec curl -sSfL -o /var/svc/manifest/site/tcpproxy.xml http://catacomb.eng.oxide.computer:12346/tcpproxy.xml
6852023-11-04T20:45:28.691Z+ pfexec svccfg import /var/svc/manifest/site/tcpproxy.xml
6862023-11-04T20:45:28.715Z+ OMICRON_NO_UNINSTALL=1
6872023-11-04T20:45:28.739Z+ ptime -m pfexec ./target/release/omicron-package -t test install
6882023-11-04T20:45:28.763ZNov 04 20:45:27.016 DEBG target[test]: Target({"image": "standard", "machine": "non-gimlet", "rack-topology": "single-sled", "switch": "softnpu"})
6892023-11-04T20:45:28.787ZNov 04 20:45:27.018 INFO Installing service, dst: /opt/oxide/clickhouse.tar.gz, src: out/clickhouse.tar.gz
6902023-11-04T20:45:28.811ZNov 04 20:45:27.018 INFO Installing service, dst: /opt/oxide/crucible_pantry.tar.gz, src: out/crucible-pantry.tar.gz
6912023-11-04T20:45:28.835ZNov 04 20:45:27.018 INFO Installing service, dst: /opt/oxide/clickhouse_keeper.tar.gz, src: out/clickhouse_keeper.tar.gz
6922023-11-04T20:45:28.859ZNov 04 20:45:27.018 INFO Installing service, dst: /opt/oxide/cockroachdb.tar.gz, src: out/cockroachdb.tar.gz
6932023-11-04T20:45:28.883ZNov 04 20:45:27.018 INFO Installing service, dst: /opt/oxide/sled-agent.tar, src: out/omicron-sled-agent.tar
6942023-11-04T20:45:28.907ZNov 04 20:45:27.018 INFO Installing service, dst: /opt/oxide/ntp.tar.gz, src: out/ntp.tar.gz
6952023-11-04T20:45:28.931ZNov 04 20:45:27.018 INFO Installing service, dst: /opt/oxide/nexus.tar.gz, src: out/omicron-nexus.tar.gz
6962023-11-04T20:45:28.955ZNov 04 20:45:27.018 INFO Installing service, dst: /opt/oxide/overlay.tar.gz, src: out/overlay.tar.gz
6972023-11-04T20:45:28.991ZNov 04 20:45:27.018 INFO Installing service, dst: /opt/oxide/switch.tar.gz, src: out/switch-softnpu.tar.gz
6982023-11-04T20:45:29.015ZNov 04 20:45:27.018 INFO Installing service, dst: /opt/oxide/internal_dns.tar.gz, src: out/internal-dns.tar.gz
6992023-11-04T20:45:29.038ZNov 04 20:45:27.018 INFO Installing service, dst: /opt/oxide/mg-ddm.tar, src: out/mg-ddm-gz.tar
7002023-11-04T20:45:29.062ZNov 04 20:45:27.018 INFO Installing service, dst: /opt/oxide/oximeter.tar.gz, src: out/oximeter-collector.tar.gz
7012023-11-04T20:45:29.086ZNov 04 20:45:27.019 INFO Installing service, dst: /opt/oxide/propolis-server.tar.gz, src: out/propolis-server.tar.gz
7022023-11-04T20:45:29.110ZNov 04 20:45:27.019 INFO Installing service, dst: /opt/oxide/mgs.tar.gz, src: out/omicron-gateway-softnpu.tar.gz
7032023-11-04T20:45:29.134ZNov 04 20:45:27.019 INFO Installing service, dst: /opt/oxide/external_dns.tar.gz, src: out/external-dns.tar.gz
7042023-11-04T20:45:29.158ZNov 04 20:45:27.019 INFO Installing service, dst: /opt/oxide/crucible.tar.gz, src: out/crucible.tar.gz
7052023-11-04T20:45:29.182ZNov 04 20:45:27.273 INFO Unpacking service tarball, service_path: /opt/oxide/mg-ddm, tar_path: /opt/oxide/mg-ddm.tar
7062023-11-04T20:45:29.206ZNov 04 20:45:27.307 INFO Unpacking service tarball, service_path: /opt/oxide/sled-agent, tar_path: /opt/oxide/sled-agent.tar
7072023-11-04T20:45:29.230ZNov 04 20:45:27.373 INFO Installing bootstrap service from /opt/oxide/sled-agent/pkg/manifest.xml
7082023-11-04T20:45:29.253Z
7092023-11-04T20:45:29.278Zreal 0.497900516
7102023-11-04T20:45:29.301Zuser 0.102689096
7112023-11-04T20:45:29.325Zsys 2.004928400
7122023-11-04T20:45:29.356Ztrap 0.000000410
7132023-11-04T20:45:29.381Ztflt 0.000121752
7142023-11-04T20:45:29.405Zdflt 0.000577395
7152023-11-04T20:45:29.428Zkflt 0.000000000
7162023-11-04T20:45:29.452Zlock 44.507629548
7172023-11-04T20:45:29.476Zslp 0.710974135
7182023-11-04T20:45:29.500Zlat 0.004127916
7192023-11-04T20:45:29.524Zstop 0.204120700
7202023-11-04T20:45:29.548Z+ retry=0
7212023-11-04T20:45:29.572Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7222023-11-04T20:45:29.596Z+ [[ 0 -gt 30 ]]
7232023-11-04T20:45:29.620Z+ sleep 1
7242023-11-04T20:45:29.644Z+ retry=1
7252023-11-04T20:45:29.668Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7262023-11-04T20:45:29.692Z+ [[ 1 -gt 30 ]]
7272023-11-04T20:45:29.717Z+ sleep 1
7282023-11-04T20:45:29.740Z+ retry=2
7292023-11-04T20:45:29.764Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7302023-11-04T20:45:29.788Z+ [[ 2 -gt 30 ]]
7312023-11-04T20:45:29.812Z+ sleep 1
7322023-11-04T20:45:30.531Z+ retry=3
7332023-11-04T20:45:30.556Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7342023-11-04T20:45:49.736Z+ [[ 3 -gt 30 ]]
7352023-11-04T20:45:49.760Z+ sleep 1
7362023-11-04T20:45:50.739Z+ retry=4
7372023-11-04T20:45:50.764Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7382023-11-04T20:45:50.788Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone dump-state
7392023-11-04T20:45:50.812Zlocal v6:
7402023-11-04T20:45:50.836Zfd00:99::1
7412023-11-04T20:45:50.859Zfe80::6844:b1ff:fef1:b7d8
7422023-11-04T20:45:50.889Zlocal v4:
7432023-11-04T20:45:50.913Z10.151.2.174
7442023-11-04T20:45:50.937Zrouter v6:
7452023-11-04T20:45:50.961Zrouter v4_idx:
7462023-11-04T20:45:50.985Z0.0.0.0/0 -> 0
7472023-11-04T20:45:51.009Zrouter v4_routes:
7482023-11-04T20:45:51.033Z0 -> 10.151.2.100 (2)
7492023-11-04T20:45:51.057Zresolver v4:
7502023-11-04T20:45:51.081Z10.151.2.100 -> 18:c0:4d:0d:a0:2a
7512023-11-04T20:45:51.106Zresolver v6:
7522023-11-04T20:45:51.130Zfe80::6844:b1ff:fef1:b7d8 -> 6a:44:b1:f1:b7:d8
7532023-11-04T20:45:51.154Zfe80::10d4:dfff:feb0:cfe0 -> 12:d4:df:b0:cf:e0
7542023-11-04T20:45:51.179Znat v4:
7552023-11-04T20:45:51.203Znat_v6:
7562023-11-04T20:45:51.227Zport_mac:
7572023-11-04T20:45:51.251Z1: a8:40:25:58:78:17
7582023-11-04T20:45:51.275Z1000: a8:40:25:58:78:16
7592023-11-04T20:45:51.301Zproxy arp:
7602023-11-04T20:45:51.325Z10.151.2.164/10.151.2.227: a8:e1:de:01:70:1d
7612023-11-04T20:45:51.349Z+ retry=0
7622023-11-04T20:45:51.372Z+++ zoneadm list -n
7632023-11-04T20:45:51.396Z+++ grep oxz_ntp
7642023-11-04T20:45:51.420Z++ pfexec svcs -z -Hostate oxide/ntp
7652023-11-04T20:45:51.444Zsvcs: zone '-Hostate': entity not found
7662023-11-04T20:45:51.468Z++ true
7672023-11-04T20:45:51.492Z+ [[ '' != online ]]
7682023-11-04T20:45:51.515Z+ [[ 0 -gt 60 ]]
7692023-11-04T20:45:51.540Z+ sleep 1
7702023-11-04T20:45:51.802Z+ retry=1
7712023-11-04T20:45:51.826Z+++ zoneadm list -n
7722023-11-04T20:45:51.850Z+++ grep oxz_ntp
7732023-11-04T20:45:51.874Z++ pfexec svcs -z -Hostate oxide/ntp
7742023-11-04T20:45:51.915Zsvcs: zone '-Hostate': entity not found
7752023-11-04T20:45:51.939Z++ true
7762023-11-04T20:45:51.963Z+ [[ '' != online ]]
7772023-11-04T20:45:51.987Z+ [[ 1 -gt 60 ]]
7782023-11-04T20:45:52.013Z+ sleep 1
7792023-11-04T20:45:52.832Z+ retry=2
7802023-11-04T20:45:52.856Z+++ grep oxz_ntp
7812023-11-04T20:45:52.882Z+++ zoneadm list -n
7822023-11-04T20:45:52.907Z++ pfexec svcs -z -Hostate oxide/ntp
7832023-11-04T20:45:52.941Zsvcs: zone '-Hostate': entity not found
7842023-11-04T20:45:52.965Z++ true
7852023-11-04T20:45:52.990Z+ [[ '' != online ]]
7862023-11-04T20:45:53.016Z+ [[ 2 -gt 60 ]]
7872023-11-04T20:45:53.040Z+ sleep 1
7882023-11-04T20:45:53.878Z+ retry=3
7892023-11-04T20:45:53.903Z+++ zoneadm list -n
7902023-11-04T20:45:53.927Z+++ grep oxz_ntp
7912023-11-04T20:45:53.967Z++ pfexec svcs -z -Hostate oxide/ntp
7922023-11-04T20:45:53.991Zsvcs: zone '-Hostate': entity not found
7932023-11-04T20:45:54.015Z++ true
7942023-11-04T20:45:54.040Z+ [[ '' != online ]]
7952023-11-04T20:45:54.064Z+ [[ 3 -gt 60 ]]
7962023-11-04T20:45:54.088Z+ sleep 1
7972023-11-04T20:45:54.899Z+ retry=4
7982023-11-04T20:45:54.924Z+++ zoneadm list -n
7992023-11-04T20:45:54.949Z+++ grep oxz_ntp
8002023-11-04T20:45:54.991Z++ pfexec svcs -z -Hostate oxide/ntp
8012023-11-04T20:45:55.016Zsvcs: zone '-Hostate': entity not found
8022023-11-04T20:45:55.040Z++ true
8032023-11-04T20:45:55.064Z+ [[ '' != online ]]
8042023-11-04T20:45:55.089Z+ [[ 4 -gt 60 ]]
8052023-11-04T20:45:55.113Z+ sleep 1
8062023-11-04T20:45:55.922Z+ retry=5
8072023-11-04T20:45:55.946Z+++ zoneadm list -n
8082023-11-04T20:45:55.971Z+++ grep oxz_ntp
8092023-11-04T20:45:56.020Z++ pfexec svcs -z -Hostate oxide/ntp
8102023-11-04T20:45:56.044Zsvcs: zone '-Hostate': entity not found
8112023-11-04T20:45:56.070Z++ true
8122023-11-04T20:45:56.093Z+ [[ '' != online ]]
8132023-11-04T20:45:56.117Z+ [[ 5 -gt 60 ]]
8142023-11-04T20:45:56.141Z+ sleep 1
8152023-11-04T20:45:56.942Z+ retry=6
8162023-11-04T20:45:56.967Z+++ zoneadm list -n
8172023-11-04T20:45:56.991Z+++ grep oxz_ntp
8182023-11-04T20:45:57.015Z++ pfexec svcs -z -Hostate oxide/ntp
8192023-11-04T20:45:57.043Zsvcs: zone '-Hostate': entity not found
8202023-11-04T20:45:57.067Z++ true
8212023-11-04T20:45:57.091Z+ [[ '' != online ]]
8222023-11-04T20:45:57.114Z+ [[ 6 -gt 60 ]]
8232023-11-04T20:45:57.138Z+ sleep 1
8242023-11-04T20:45:57.964Z+ retry=7
8252023-11-04T20:45:57.988Z+++ zoneadm list -n
8262023-11-04T20:45:58.013Z+++ grep oxz_ntp
8272023-11-04T20:45:58.037Z++ pfexec svcs -z -Hostate oxide/ntp
8282023-11-04T20:45:58.068Zsvcs: zone '-Hostate': entity not found
8292023-11-04T20:45:58.092Z++ true
8302023-11-04T20:45:58.116Z+ [[ '' != online ]]
8312023-11-04T20:45:58.140Z+ [[ 7 -gt 60 ]]
8322023-11-04T20:45:58.164Z+ sleep 1
8332023-11-04T20:45:58.985Z+ retry=8
8342023-11-04T20:45:59.010Z+++ zoneadm list -n
8352023-11-04T20:45:59.033Z+++ grep oxz_ntp
8362023-11-04T20:45:59.057Z++ pfexec svcs -z -Hostate oxide/ntp
8372023-11-04T20:45:59.099Zsvcs: zone '-Hostate': entity not found
8382023-11-04T20:45:59.123Z++ true
8392023-11-04T20:45:59.147Z+ [[ '' != online ]]
8402023-11-04T20:45:59.171Z+ [[ 8 -gt 60 ]]
8412023-11-04T20:45:59.195Z+ sleep 1
8422023-11-04T20:46:00.007Z+ retry=9
8432023-11-04T20:46:00.031Z+++ zoneadm list -n
8442023-11-04T20:46:00.055Z+++ grep oxz_ntp
8452023-11-04T20:46:00.081Z++ pfexec svcs -z -Hostate oxide/ntp
8462023-11-04T20:46:00.124Zsvcs: zone '-Hostate': entity not found
8472023-11-04T20:46:00.148Z++ true
8482023-11-04T20:46:00.172Z+ [[ '' != online ]]
8492023-11-04T20:46:00.196Z+ [[ 9 -gt 60 ]]
8502023-11-04T20:46:00.221Z+ sleep 1
8512023-11-04T20:46:01.029Z+ retry=10
8522023-11-04T20:46:01.054Z+++ zoneadm list -n
8532023-11-04T20:46:01.079Z+++ grep oxz_ntp
8542023-11-04T20:46:01.104Z++ pfexec svcs -z -Hostate oxide/ntp
8552023-11-04T20:46:01.150Zsvcs: zone '-Hostate': entity not found
8562023-11-04T20:46:01.175Z++ true
8572023-11-04T20:46:01.200Z+ [[ '' != online ]]
8582023-11-04T20:46:01.224Z+ [[ 10 -gt 60 ]]
8592023-11-04T20:46:01.249Z+ sleep 1
8602023-11-04T20:46:02.051Z+ retry=11
8612023-11-04T20:46:02.076Z+++ zoneadm list -n
8622023-11-04T20:46:02.100Z+++ grep oxz_ntp
8632023-11-04T20:46:02.125Z++ pfexec svcs -z -Hostate oxide/ntp
8642023-11-04T20:46:02.176Zsvcs: zone '-Hostate': entity not found
8652023-11-04T20:46:02.201Z++ true
8662023-11-04T20:46:02.226Z+ [[ '' != online ]]
8672023-11-04T20:46:02.250Z+ [[ 11 -gt 60 ]]
8682023-11-04T20:46:02.275Z+ sleep 1
8692023-11-04T20:46:03.073Z+ retry=12
8702023-11-04T20:46:03.099Z+++ zoneadm list -n
8712023-11-04T20:46:03.123Z+++ grep oxz_ntp
8722023-11-04T20:46:03.149Z++ pfexec svcs -z -Hostate oxide/ntp
8732023-11-04T20:46:03.173Zsvcs: zone '-Hostate': entity not found
8742023-11-04T20:46:03.201Z++ true
8752023-11-04T20:46:03.225Z+ [[ '' != online ]]
8762023-11-04T20:46:03.250Z+ [[ 12 -gt 60 ]]
8772023-11-04T20:46:03.275Z+ sleep 1
8782023-11-04T20:46:04.095Z+ retry=13
8792023-11-04T20:46:04.120Z+++ zoneadm list -n
8802023-11-04T20:46:04.145Z+++ grep oxz_ntp
8812023-11-04T20:46:04.169Z++ pfexec svcs -z -Hostate oxide/ntp
8822023-11-04T20:46:04.194Zsvcs: zone '-Hostate': entity not found
8832023-11-04T20:46:04.228Z++ true
8842023-11-04T20:46:04.254Z+ [[ '' != online ]]
8852023-11-04T20:46:04.278Z+ [[ 13 -gt 60 ]]
8862023-11-04T20:46:04.303Z+ sleep 1
8872023-11-04T20:46:05.116Z+ retry=14
8882023-11-04T20:46:05.141Z+++ zoneadm list -n
8892023-11-04T20:46:05.165Z+++ grep oxz_ntp
8902023-11-04T20:46:05.189Z++ pfexec svcs -z oxz_ntp_26ba4142-c85c-4eb8-a460-b28328a85e57 -Hostate oxide/ntp
8912023-11-04T20:46:05.213Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
8922023-11-04T20:46:05.253Z++ true
8932023-11-04T20:46:05.277Z+ [[ '' != online ]]
8942023-11-04T20:46:05.301Z+ [[ 14 -gt 60 ]]
8952023-11-04T20:46:05.325Z+ sleep 1
8962023-11-04T20:46:06.149Z+ retry=15
8972023-11-04T20:46:06.174Z+++ zoneadm list -n
8982023-11-04T20:46:06.197Z+++ grep oxz_ntp
8992023-11-04T20:46:06.222Z++ pfexec svcs -z oxz_ntp_26ba4142-c85c-4eb8-a460-b28328a85e57 -Hostate oxide/ntp
9002023-11-04T20:46:06.246Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
9012023-11-04T20:46:06.279Z++ true
9022023-11-04T20:46:06.305Z+ [[ '' != online ]]
9032023-11-04T20:46:06.329Z+ [[ 15 -gt 60 ]]
9042023-11-04T20:46:06.354Z+ sleep 1
9052023-11-04T20:46:07.180Z+ retry=16
9062023-11-04T20:46:07.205Z+++ zoneadm list -n
9072023-11-04T20:46:07.229Z+++ grep oxz_ntp
9082023-11-04T20:46:07.253Z++ pfexec svcs -z oxz_ntp_26ba4142-c85c-4eb8-a460-b28328a85e57 -Hostate oxide/ntp
9092023-11-04T20:46:07.277Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
9102023-11-04T20:46:07.305Z++ true
9112023-11-04T20:46:07.329Z+ [[ '' != online ]]
9122023-11-04T20:46:07.353Z+ [[ 16 -gt 60 ]]
9132023-11-04T20:46:07.377Z+ sleep 1
9142023-11-04T20:46:08.211Z+ retry=17
9152023-11-04T20:46:08.235Z+++ zoneadm list -n
9162023-11-04T20:46:08.259Z+++ grep oxz_ntp
9172023-11-04T20:46:08.283Z++ pfexec svcs -z oxz_ntp_26ba4142-c85c-4eb8-a460-b28328a85e57 -Hostate oxide/ntp
9182023-11-04T20:46:08.331Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
9192023-11-04T20:46:08.355Z++ true
9202023-11-04T20:46:08.379Z+ [[ '' != online ]]
9212023-11-04T20:46:08.403Z+ [[ 17 -gt 60 ]]
9222023-11-04T20:46:08.427Z+ sleep 1
9232023-11-04T20:46:09.241Z+ retry=18
9242023-11-04T20:46:09.267Z+++ zoneadm list -n
9252023-11-04T20:46:09.291Z+++ grep oxz_ntp
9262023-11-04T20:46:09.315Z++ pfexec svcs -z oxz_ntp_26ba4142-c85c-4eb8-a460-b28328a85e57 -Hostate oxide/ntp
9272023-11-04T20:46:09.357Z+ [[ disabled != online ]]
9282023-11-04T20:46:09.381Z+ [[ 18 -gt 60 ]]
9292023-11-04T20:46:09.404Z+ sleep 1
9302023-11-04T20:46:10.264Z+ retry=19
9312023-11-04T20:46:10.288Z+++ zoneadm list -n
9322023-11-04T20:46:10.312Z+++ grep oxz_ntp
9332023-11-04T20:46:10.336Z++ pfexec svcs -z oxz_ntp_26ba4142-c85c-4eb8-a460-b28328a85e57 -Hostate oxide/ntp
9342023-11-04T20:46:10.382Z+ [[ disabled != online ]]
9352023-11-04T20:46:10.406Z+ [[ 19 -gt 60 ]]
9362023-11-04T20:46:10.430Z+ sleep 1
9372023-11-04T20:46:11.286Z+ retry=20
9382023-11-04T20:46:11.310Z+++ zoneadm list -n
9392023-11-04T20:46:11.334Z+++ grep oxz_ntp
9402023-11-04T20:46:11.358Z++ pfexec svcs -z oxz_ntp_26ba4142-c85c-4eb8-a460-b28328a85e57 -Hostate oxide/ntp
9412023-11-04T20:46:11.382Z+ [[ disabled != online ]]
9422023-11-04T20:46:11.409Z+ [[ 20 -gt 60 ]]
9432023-11-04T20:46:11.433Z+ sleep 1
9442023-11-04T20:46:12.308Z+ retry=21
9452023-11-04T20:46:12.334Z+++ zoneadm list -n
9462023-11-04T20:46:12.358Z+++ grep oxz_ntp
9472023-11-04T20:46:12.382Z++ pfexec svcs -z oxz_ntp_26ba4142-c85c-4eb8-a460-b28328a85e57 -Hostate oxide/ntp
9482023-11-04T20:46:12.434Z+ [[ disabled != online ]]
9492023-11-04T20:46:12.458Z+ [[ 21 -gt 60 ]]
9502023-11-04T20:46:12.482Z+ sleep 1
9512023-11-04T20:46:13.330Z+ retry=22
9522023-11-04T20:46:13.354Z+++ zoneadm list -n
9532023-11-04T20:46:13.378Z+++ grep oxz_ntp
9542023-11-04T20:46:13.403Z++ pfexec svcs -z oxz_ntp_26ba4142-c85c-4eb8-a460-b28328a85e57 -Hostate oxide/ntp
9552023-11-04T20:46:13.427Z+ [[ disabled != online ]]
9562023-11-04T20:46:13.459Z+ [[ 22 -gt 60 ]]
9572023-11-04T20:46:13.483Z+ sleep 1
9582023-11-04T20:46:14.353Z+ retry=23
9592023-11-04T20:46:14.377Z+++ zoneadm list -n
9602023-11-04T20:46:14.401Z+++ grep oxz_ntp
9612023-11-04T20:46:14.425Z++ pfexec svcs -z oxz_ntp_26ba4142-c85c-4eb8-a460-b28328a85e57 -Hostate oxide/ntp
9622023-11-04T20:46:14.449Z+ [[ disabled != online ]]
9632023-11-04T20:46:14.484Z+ [[ 23 -gt 60 ]]
9642023-11-04T20:46:14.508Z+ sleep 1
9652023-11-04T20:46:15.374Z+ retry=24
9662023-11-04T20:46:15.398Z+++ zoneadm list -n
9672023-11-04T20:46:15.422Z+++ grep oxz_ntp
9682023-11-04T20:46:15.446Z++ pfexec svcs -z oxz_ntp_26ba4142-c85c-4eb8-a460-b28328a85e57 -Hostate oxide/ntp
9692023-11-04T20:46:15.470Z+ [[ disabled != online ]]
 
9702023-11-04T20:46:15.510Zsock console login: Nov 4 20:46:15 sok ip: ip: joining multicasts failed (4) on vopte0 - will use link layer broadcasts for multicast
 
9712023-11-04T20:46:15.516Z+ [[ 24 -gt 60 ]]
9722023-11-04T20:46:15.540Z+ sleep 1
9732023-11-04T20:46:16.396Z+ retry=25
9742023-11-04T20:46:16.421Z+++ zoneadm list -n
9752023-11-04T20:46:16.445Z+++ grep oxz_ntp
9762023-11-04T20:46:16.469Z++ pfexec svcs -z oxz_ntp_26ba4142-c85c-4eb8-a460-b28328a85e57 -Hostate oxide/ntp
9772023-11-04T20:46:16.493Z+ [[ online != online ]]
9782023-11-04T20:46:16.537Z+ echo 'Waited for chrony: 25s'
9792023-11-04T20:46:16.561ZWaited for chrony: 25s
9802023-11-04T20:46:16.586Z+ export RUST_BACKTRACE=1
9812023-11-04T20:46:16.610Z+ RUST_BACKTRACE=1
9822023-11-04T20:46:16.635Z+ export E2E_TLS_CERT IPPOOL_START IPPOOL_END
9832023-11-04T20:46:16.659Z+ ./tests/bootstrap
9842023-11-04T20:46:16.683ZUsing external DNS server at 10.151.2.164:53
9852023-11-04T20:46:16.708Z2023-11-04 20:46:16.466823156 UTC: attempting to log into API
9862023-11-04T20:46:31.487Z2023-11-04 20:46:31.479514222 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9872023-11-04T20:46:32.488Z2023-11-04 20:46:32.480692130 UTC: attempting to log into API
9882023-11-04T20:46:47.526Z2023-11-04 20:46:47.518752499 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9892023-11-04T20:46:48.527Z2023-11-04 20:46:48.519525564 UTC: attempting to log into API
9902023-11-04T20:47:03.571Z2023-11-04 20:47:03.561506686 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9912023-11-04T20:47:04.572Z2023-11-04 20:47:04.562644335 UTC: attempting to log into API
9922023-11-04T20:47:19.608Z2023-11-04 20:47:19.598976261 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9932023-11-04T20:47:20.609Z2023-11-04 20:47:20.599574616 UTC: attempting to log into API
 
9942023-11-04T20:47:29.829ZNov 4 20:47:29 sock ip: ip: joining multicasts failed (4) on vopte5 - will use link layer broadcasts for multicast
 
9952023-11-04T20:47:35.661Z2023-11-04 20:47:35.651228279 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
 
9962023-11-04T20:47:36.398ZNov 4 20:47:36 sock ip: ip: joining multicasts failed (4) on vopte1 - will use link layer broadcasts for multicast
 
9972023-11-04T20:47:36.662Z2023-11-04 20:47:36.652327077 UTC: attempting to log into API
 
9982023-11-04T20:47:42.181ZNov 4 20:47:41 sok ip: ip: joining multicasts failed (4) on vopte2 - will use link layer broadcasts for multicast
9992023-11-04T20:47:48.733ZNov 4 20:47:48 sok ip: ip: joining multicasts failed (4) on vopte3 - will use link layer broadcasts for multicast
 
10002023-11-04T20:47:51.695Z2023-11-04 20:47:51.685598744 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
10012023-11-04T20:47:52.696Z2023-11-04 20:47:52.686625269 UTC: attempting to log into API
 
10022023-11-04T20:47:55.813ZNov 4 20:47:55 sok ip: ip: joining multicasts failed (4) on vopte4 - will use link layer broadcasts for multicast
 
10032023-11-04T20:48:07.729Z2023-11-04 20:48:07.719627719 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
10042023-11-04T20:48:08.730Z2023-11-04 20:48:08.720746317 UTC: attempting to log into API
10052023-11-04T20:48:23.763Z2023-11-04 20:48:23.753483016 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
10062023-11-04T20:48:24.763Z2023-11-04 20:48:24.754254385 UTC: attempting to log into API
10072023-11-04T20:48:35.544Z2023-11-04 20:48:35.534972174 UTC: login succeeded
10082023-11-04T20:48:35.577Zwaiting for nexus to come up...
10092023-11-04T20:48:35.643Zcreating IP pool... 10.151.2.175 - 10.151.2.227
10102023-11-04T20:48:35.701Zensuring datasets are ready...
10112023-11-04T20:48:38.323Zlet's roll.
10122023-11-04T20:48:38.348Z+ rm ./tests/bootstrap
10132023-11-04T20:48:38.408Z+ for test_bin in tests/*
10142023-11-04T20:48:38.433Z+ ./tests/bootstrap-346a6c0cadffef35
10152023-11-04T20:48:38.458Z
10162023-11-04T20:48:38.482Zrunning 0 tests
10172023-11-04T20:48:38.506Z
10182023-11-04T20:48:38.531Ztest result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
10192023-11-04T20:48:38.556Z
10202023-11-04T20:48:38.580Z+ for test_bin in tests/*
10212023-11-04T20:48:38.605Z+ ./tests/end_to_end_tests-ad8d578e5459ec9d
10222023-11-04T20:48:38.630Z
10232023-11-04T20:48:38.654Zrunning 2 tests
10242023-11-04T20:48:38.679Ztest no_spoof::no_spoof_here ... ok
10252023-11-04T20:49:38.433Ztest instance_launch::instance_launch has been running for over 60 seconds
10262023-11-04T20:51:34.561Ztest instance_launch::instance_launch ... ok
10272023-11-04T20:51:34.586Z
10282023-11-04T20:51:34.610Ztest result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 176.13s
10292023-11-04T20:51:34.635Z
10302023-11-04T20:51:34.659Z+ _exit_trap
10312023-11-04T20:51:34.684Z+ local status=0
10322023-11-04T20:51:34.708Z+ [[ 0 -eq 0 ]]
10332023-11-04T20:51:34.732Z+ exit 0
10342023-11-04T20:51:34.756Zprocess exited: duration 400971 ms, exit code 0
 
10352023-11-04T20:51:34.837Zfound 47 output files
10362023-11-04T20:51:34.889Zuploading: /var/svc/log/oxide-sled-agent:default.log (645911 bytes)
10372023-11-04T20:51:36.129Zuploaded: /var/svc/log/oxide-sled-agent:default.log
10382023-11-04T20:51:36.155Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_fdaf0194-9fec-46da-ae9d-68085c62a4b7/root/var/svc/log/oxide-crucible-agent:default.log (3436 bytes)
10392023-11-04T20:51:37.227Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_fdaf0194-9fec-46da-ae9d-68085c62a4b7/root/var/svc/log/oxide-crucible-agent:default.log
10402023-11-04T20:51:37.251Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_fdaf0194-9fec-46da-ae9d-68085c62a4b7/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10412023-11-04T20:51:38.322Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_fdaf0194-9fec-46da-ae9d-68085c62a4b7/root/var/svc/log/oxide-crucible-downstairs:default.log
10422023-11-04T20:51:38.348Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_72925d21-abd5-4d8c-bd35-7547b64d34f8/root/var/svc/log/oxide-nexus:default.log (622058 bytes)
10432023-11-04T20:51:38.577Zupload warning: file "/pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_72925d21-abd5-4d8c-bd35-7547b64d34f8/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 622058 -> 622666
10442023-11-04T20:51:39.647Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_72925d21-abd5-4d8c-bd35-7547b64d34f8/root/var/svc/log/oxide-nexus:default.log
10452023-11-04T20:51:39.676Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_26ba4142-c85c-4eb8-a460-b28328a85e57/root/var/svc/log/oxide-ntp:default.log (852 bytes)
10462023-11-04T20:51:40.747Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_26ba4142-c85c-4eb8-a460-b28328a85e57/root/var/svc/log/oxide-ntp:default.log
10472023-11-04T20:51:40.771Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_oximeter_281c02ce-6517-4665-bea7-729449033991/root/var/svc/log/oxide-oximeter:default.log (178109 bytes)
10482023-11-04T20:51:40.932Zupload warning: file "/pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_oximeter_281c02ce-6517-4665-bea7-729449033991/root/var/svc/log/oxide-oximeter:default.log" changed size mid upload: 178109 -> 183655
10492023-11-04T20:51:42.005Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_oximeter_281c02ce-6517-4665-bea7-729449033991/root/var/svc/log/oxide-oximeter:default.log
10502023-11-04T20:51:42.029Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_9a582bb3-5aee-4563-b85a-8d9fe2cb7dc8/root/var/svc/log/oxide-cockroachdb:default.log (5367 bytes)
10512023-11-04T20:51:43.100Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_9a582bb3-5aee-4563-b85a-8d9fe2cb7dc8/root/var/svc/log/oxide-cockroachdb:default.log
10522023-11-04T20:51:43.124Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_a955f775-c493-46fd-ba03-8d53e243e867/root/var/svc/log/oxide-crucible-agent:default.log (3434 bytes)
10532023-11-04T20:51:44.196Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_a955f775-c493-46fd-ba03-8d53e243e867/root/var/svc/log/oxide-crucible-agent:default.log
10542023-11-04T20:51:44.220Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_a955f775-c493-46fd-ba03-8d53e243e867/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10552023-11-04T20:51:45.292Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_a955f775-c493-46fd-ba03-8d53e243e867/root/var/svc/log/oxide-crucible-downstairs:default.log
10562023-11-04T20:51:45.316Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_1554354b-4d89-430f-818a-4a7639de9c10/root/var/svc/log/oxide-external_dns:default.log (15610 bytes)
10572023-11-04T20:51:46.410Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_external_dns_1554354b-4d89-430f-818a-4a7639de9c10/root/var/svc/log/oxide-external_dns:default.log
10582023-11-04T20:51:46.434Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_00499140-b2aa-46b2-b449-30a753b5fd16/root/var/svc/log/oxide-internal_dns:default.log (101105 bytes)
10592023-11-04T20:51:47.615Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_00499140-b2aa-46b2-b449-30a753b5fd16/root/var/svc/log/oxide-internal_dns:default.log
10602023-11-04T20:51:47.640Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_cockroachdb_7010992a-5846-4ed8-ae3f-e29ae18cef62/root/var/svc/log/oxide-cockroachdb:default.log (5368 bytes)
10612023-11-04T20:51:48.712Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_cockroachdb_7010992a-5846-4ed8-ae3f-e29ae18cef62/root/var/svc/log/oxide-cockroachdb:default.log
10622023-11-04T20:51:48.737Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_cockroachdb_91bcbd61-900f-49d9-8066-c9e4465f1c35/root/var/svc/log/oxide-cockroachdb:default.log (5368 bytes)
10632023-11-04T20:51:49.809Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_cockroachdb_91bcbd61-900f-49d9-8066-c9e4465f1c35/root/var/svc/log/oxide-cockroachdb:default.log
10642023-11-04T20:51:49.833Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_7dd9aa94-4cb6-4e37-a4a4-7a5ae741b03a/root/var/svc/log/oxide-crucible-agent:default.log (24359 bytes)
10652023-11-04T20:51:50.960Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_7dd9aa94-4cb6-4e37-a4a4-7a5ae741b03a/root/var/svc/log/oxide-crucible-agent:default.log
10662023-11-04T20:51:50.984Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_7dd9aa94-4cb6-4e37-a4a4-7a5ae741b03a/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10672023-11-04T20:51:52.055Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_7dd9aa94-4cb6-4e37-a4a4-7a5ae741b03a/root/var/svc/log/oxide-crucible-downstairs:default.log
10682023-11-04T20:51:52.080Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_7dd9aa94-4cb6-4e37-a4a4-7a5ae741b03a/root/var/svc/log/oxide-crucible-downstairs:downstairs-30baa1c3-d7aa-4ece-8c3f-976df97293b6.log (5536 bytes)
10692023-11-04T20:51:53.152Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_7dd9aa94-4cb6-4e37-a4a4-7a5ae741b03a/root/var/svc/log/oxide-crucible-downstairs:downstairs-30baa1c3-d7aa-4ece-8c3f-976df97293b6.log
10702023-11-04T20:51:53.176Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_7dd9aa94-4cb6-4e37-a4a4-7a5ae741b03a/root/var/svc/log/oxide-crucible-downstairs:downstairs-cb852d86-684f-4767-82eb-62f664a8031c.log (3048 bytes)
10712023-11-04T20:51:54.247Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_7dd9aa94-4cb6-4e37-a4a4-7a5ae741b03a/root/var/svc/log/oxide-crucible-downstairs:downstairs-cb852d86-684f-4767-82eb-62f664a8031c.log
10722023-11-04T20:51:54.273Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_nexus_e938fc50-abed-4298-aec3-1b406e49b001/root/var/svc/log/oxide-nexus:default.log (521751 bytes)
10732023-11-04T20:51:54.499Zupload warning: file "/pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_nexus_e938fc50-abed-4298-aec3-1b406e49b001/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 521751 -> 522968
10742023-11-04T20:51:55.571Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_nexus_e938fc50-abed-4298-aec3-1b406e49b001/root/var/svc/log/oxide-nexus:default.log
10752023-11-04T20:51:55.596Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_cockroachdb_04c177d2-f57c-4306-b3d3-39179d06d6aa/root/var/svc/log/oxide-cockroachdb:default.log (5342 bytes)
10762023-11-04T20:51:56.703Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_cockroachdb_04c177d2-f57c-4306-b3d3-39179d06d6aa/root/var/svc/log/oxide-cockroachdb:default.log
10772023-11-04T20:51:56.727Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_cockroachdb_5a685f81-c217-4c02-850a-a2dc8ed31347/root/var/svc/log/oxide-cockroachdb:default.log (5369 bytes)
10782023-11-04T20:51:57.798Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_cockroachdb_5a685f81-c217-4c02-850a-a2dc8ed31347/root/var/svc/log/oxide-cockroachdb:default.log
10792023-11-04T20:51:57.823Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_internal_dns_a4ef4360-0f25-4ad2-9cc4-76b0380af201/root/var/svc/log/oxide-internal_dns:default.log (99073 bytes)
10802023-11-04T20:51:57.936Zupload warning: file "/pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_internal_dns_a4ef4360-0f25-4ad2-9cc4-76b0380af201/root/var/svc/log/oxide-internal_dns:default.log" changed size mid upload: 99073 -> 102829
10812023-11-04T20:51:59.007Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_internal_dns_a4ef4360-0f25-4ad2-9cc4-76b0380af201/root/var/svc/log/oxide-internal_dns:default.log
10822023-11-04T20:51:59.046Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_nexus_5299ba63-2d0f-4b50-804d-1d5b5894a2c9/root/var/svc/log/oxide-nexus:default.log (4579817 bytes)
10832023-11-04T20:52:00.264Zupload warning: file "/pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_nexus_5299ba63-2d0f-4b50-804d-1d5b5894a2c9/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 4579817 -> 4639213
10842023-11-04T20:52:01.337Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_nexus_5299ba63-2d0f-4b50-804d-1d5b5894a2c9/root/var/svc/log/oxide-nexus:default.log
10852023-11-04T20:52:01.361Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_180e2fd4-4f98-4404-936b-4330c71862c0/root/var/svc/log/oxide-crucible-agent:default.log (3434 bytes)
10862023-11-04T20:52:02.485Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_180e2fd4-4f98-4404-936b-4330c71862c0/root/var/svc/log/oxide-crucible-agent:default.log
10872023-11-04T20:52:02.511Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_180e2fd4-4f98-4404-936b-4330c71862c0/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10882023-11-04T20:52:03.536Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_180e2fd4-4f98-4404-936b-4330c71862c0/root/var/svc/log/oxide-crucible-downstairs:default.log
10892023-11-04T20:52:03.561Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_46ae965c-ccfd-44bd-8eb8-91e843bc18b1/root/var/svc/log/oxide-crucible-agent:default.log (3435 bytes)
10902023-11-04T20:52:04.633Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_46ae965c-ccfd-44bd-8eb8-91e843bc18b1/root/var/svc/log/oxide-crucible-agent:default.log
10912023-11-04T20:52:04.657Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_46ae965c-ccfd-44bd-8eb8-91e843bc18b1/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10922023-11-04T20:52:05.725Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_46ae965c-ccfd-44bd-8eb8-91e843bc18b1/root/var/svc/log/oxide-crucible-downstairs:default.log
10932023-11-04T20:52:05.750Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_604e6705-869b-4286-9157-dd8cbf184def/root/var/svc/log/oxide-crucible-agent:default.log (3436 bytes)
10942023-11-04T20:52:06.822Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_604e6705-869b-4286-9157-dd8cbf184def/root/var/svc/log/oxide-crucible-agent:default.log
10952023-11-04T20:52:06.846Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_604e6705-869b-4286-9157-dd8cbf184def/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10962023-11-04T20:52:07.918Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_604e6705-869b-4286-9157-dd8cbf184def/root/var/svc/log/oxide-crucible-downstairs:default.log
10972023-11-04T20:52:07.943Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_8beb839e-5a50-4e20-b304-af4399b27eb3/root/var/svc/log/oxide-crucible-agent:default.log (14114 bytes)
10982023-11-04T20:52:09.036Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_8beb839e-5a50-4e20-b304-af4399b27eb3/root/var/svc/log/oxide-crucible-agent:default.log
10992023-11-04T20:52:09.060Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_8beb839e-5a50-4e20-b304-af4399b27eb3/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
11002023-11-04T20:52:10.130Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_8beb839e-5a50-4e20-b304-af4399b27eb3/root/var/svc/log/oxide-crucible-downstairs:default.log
11012023-11-04T20:52:10.154Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_8beb839e-5a50-4e20-b304-af4399b27eb3/root/var/svc/log/oxide-crucible-downstairs:downstairs-1783b77e-1f64-47c9-9cd0-d61fa5a53faf.log (5535 bytes)
11022023-11-04T20:52:11.228Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_8beb839e-5a50-4e20-b304-af4399b27eb3/root/var/svc/log/oxide-crucible-downstairs:downstairs-1783b77e-1f64-47c9-9cd0-d61fa5a53faf.log
11032023-11-04T20:52:11.252Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_9247a449-3ebe-4f58-b563-7d7345564505/root/var/svc/log/oxide-crucible-agent:default.log (24400 bytes)
11042023-11-04T20:52:12.367Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_9247a449-3ebe-4f58-b563-7d7345564505/root/var/svc/log/oxide-crucible-agent:default.log
11052023-11-04T20:52:12.393Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_9247a449-3ebe-4f58-b563-7d7345564505/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
11062023-11-04T20:52:13.467Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_9247a449-3ebe-4f58-b563-7d7345564505/root/var/svc/log/oxide-crucible-downstairs:default.log
11072023-11-04T20:52:13.491Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_9247a449-3ebe-4f58-b563-7d7345564505/root/var/svc/log/oxide-crucible-downstairs:downstairs-822dba4f-e9cd-4f87-9160-6c54cf22362b.log (3051 bytes)
11082023-11-04T20:52:14.571Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_9247a449-3ebe-4f58-b563-7d7345564505/root/var/svc/log/oxide-crucible-downstairs:downstairs-822dba4f-e9cd-4f87-9160-6c54cf22362b.log
11092023-11-04T20:52:14.595Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_9247a449-3ebe-4f58-b563-7d7345564505/root/var/svc/log/oxide-crucible-downstairs:downstairs-bb563ce9-3bfc-4240-880b-37a6a5d04e0e.log (5536 bytes)
11102023-11-04T20:52:15.665Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_9247a449-3ebe-4f58-b563-7d7345564505/root/var/svc/log/oxide-crucible-downstairs:downstairs-bb563ce9-3bfc-4240-880b-37a6a5d04e0e.log
11112023-11-04T20:52:15.689Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_pantry_320e77f1-e0d5-4f4a-9e85-04f0a0a02c80/root/var/svc/log/oxide-crucible-pantry:default.log (2338 bytes)
11122023-11-04T20:52:16.760Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_pantry_320e77f1-e0d5-4f4a-9e85-04f0a0a02c80/root/var/svc/log/oxide-crucible-pantry:default.log
11132023-11-04T20:52:16.784Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_pantry_4006100a-9ad2-4bdc-91c6-9e3dfb530324/root/var/svc/log/oxide-crucible-pantry:default.log (2333 bytes)
11142023-11-04T20:52:17.854Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_pantry_4006100a-9ad2-4bdc-91c6-9e3dfb530324/root/var/svc/log/oxide-crucible-pantry:default.log
11152023-11-04T20:52:17.878Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_external_dns_59bf56a2-fadd-4acf-83d6-7afef272596c/root/var/svc/log/oxide-external_dns:default.log (15610 bytes)
11162023-11-04T20:52:17.923Zupload warning: file "/pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_external_dns_59bf56a2-fadd-4acf-83d6-7afef272596c/root/var/svc/log/oxide-external_dns:default.log" changed size mid upload: 15610 -> 19388
11172023-11-04T20:52:18.994Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_external_dns_59bf56a2-fadd-4acf-83d6-7afef272596c/root/var/svc/log/oxide-external_dns:default.log
11182023-11-04T20:52:19.019Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_clickhouse_7dd9db68-9a29-4824-9c89-e6d0a7b8b129/root/var/svc/log/oxide-clickhouse:default.log (620636 bytes)
11192023-11-04T20:52:19.798Zupload warning: file "/pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_clickhouse_7dd9db68-9a29-4824-9c89-e6d0a7b8b129/root/var/svc/log/oxide-clickhouse:default.log" changed size mid upload: 620636 -> 723157
11202023-11-04T20:52:20.868Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_clickhouse_7dd9db68-9a29-4824-9c89-e6d0a7b8b129/root/var/svc/log/oxide-clickhouse:default.log
11212023-11-04T20:52:20.892Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_pantry_988694c8-0563-4dbd-a3d7-52f4a9a0c109/root/var/svc/log/oxide-crucible-pantry:default.log (2338 bytes)
11222023-11-04T20:52:21.961Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_pantry_988694c8-0563-4dbd-a3d7-52f4a9a0c109/root/var/svc/log/oxide-crucible-pantry:default.log
11232023-11-04T20:52:21.985Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_0de098aa-666a-43da-a98f-f8979c755d44/root/var/svc/log/oxide-crucible-agent:default.log (14116 bytes)
11242023-11-04T20:52:23.077Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_0de098aa-666a-43da-a98f-f8979c755d44/root/var/svc/log/oxide-crucible-agent:default.log
11252023-11-04T20:52:23.101Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_0de098aa-666a-43da-a98f-f8979c755d44/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
11262023-11-04T20:52:24.171Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_0de098aa-666a-43da-a98f-f8979c755d44/root/var/svc/log/oxide-crucible-downstairs:default.log
11272023-11-04T20:52:24.195Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_0de098aa-666a-43da-a98f-f8979c755d44/root/var/svc/log/oxide-crucible-downstairs:downstairs-2861a6bb-0a91-4964-af66-3014c75f45e5.log (3049 bytes)
11282023-11-04T20:52:25.264Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_0de098aa-666a-43da-a98f-f8979c755d44/root/var/svc/log/oxide-crucible-downstairs:downstairs-2861a6bb-0a91-4964-af66-3014c75f45e5.log
11292023-11-04T20:52:25.289Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_c3813caa-13b6-414d-a043-7c688bb48492/root/var/svc/log/oxide-internal_dns:default.log (103544 bytes)
11302023-11-04T20:52:25.422Zupload warning: file "/pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_c3813caa-13b6-414d-a043-7c688bb48492/root/var/svc/log/oxide-internal_dns:default.log" changed size mid upload: 103544 -> 107300
11312023-11-04T20:52:26.492Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_c3813caa-13b6-414d-a043-7c688bb48492/root/var/svc/log/oxide-internal_dns:default.log
11322023-11-04T20:52:26.516Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_26ba4142-c85c-4eb8-a460-b28328a85e57/root/var/log/chrony/measurements.log (13289 bytes)
11332023-11-04T20:52:26.562Zupload warning: file "/pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_26ba4142-c85c-4eb8-a460-b28328a85e57/root/var/log/chrony/measurements.log" changed size mid upload: 13289 -> 15344
11342023-11-04T20:52:27.631Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_26ba4142-c85c-4eb8-a460-b28328a85e57/root/var/log/chrony/measurements.log
11352023-11-04T20:52:27.655Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_26ba4142-c85c-4eb8-a460-b28328a85e57/root/var/log/chrony/statistics.log (6903 bytes)
11362023-11-04T20:52:27.679Zupload warning: file "/pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_26ba4142-c85c-4eb8-a460-b28328a85e57/root/var/log/chrony/statistics.log" changed size mid upload: 6903 -> 8190
11372023-11-04T20:52:28.748Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_26ba4142-c85c-4eb8-a460-b28328a85e57/root/var/log/chrony/statistics.log
11382023-11-04T20:52:28.772Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_26ba4142-c85c-4eb8-a460-b28328a85e57/root/var/log/chrony/tracking.log (1453 bytes)
11392023-11-04T20:52:28.796Zupload warning: file "/pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_26ba4142-c85c-4eb8-a460-b28328a85e57/root/var/log/chrony/tracking.log" changed size mid upload: 1453 -> 1585
11402023-11-04T20:52:29.864Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_26ba4142-c85c-4eb8-a460-b28328a85e57/root/var/log/chrony/tracking.log