01HFPWJG5WFRGND4H7C02GZBH7: helios / deploy

BasicConfig {
    output_rules: [
        "%/var/svc/log/oxide-sled-agent:default.log*",
        "%/pool/ext/*/crypt/zone/oxz_*/root/var/svc/log/oxide-*.log*",
        "%/pool/ext/*/crypt/zone/oxz_*/root/var/svc/log/system-illumos-*.log*",
        "%/pool/ext/*/crypt/zone/oxz_ntp_*/root/var/log/chrony/*.log*",
        "!/pool/ext/*/crypt/zone/oxz_propolis-server_*/root/var/svc/log/*.log*",
        "%/pool/ext/*/crypt/debug/global/oxide-sled-agent:default.log.*",
        "%/pool/ext/*/crypt/debug/oxz_*/oxide-*.log.*",
        "%/pool/ext/*/crypt/debug/oxz_*/system-illumos-*.log.*",
        "!/pool/ext/*/crypt/debug/oxz_propolis-server_*/*.log.*",
    ],
    rust_toolchain: None,
    target: Some(
        "lab-2.0-opte-0.25",
    ),
    access_repos: [],
    publish: [],
    skip_clone: true,
}

Buildomat Job: 01HFPWK29SXJ7033MRHD9CKHDB

Tags:

Artefacts:

Output:

SEQ GLOBAL TIME DETAILS
12023-11-20T18:21:41.251Zjob dependencies complete; ready to run (waiting for 50 m 11 s)
22023-11-20T18:23:56.112Zjob assigned to worker 01HFPZF1JHRVM3FRA2YSK77MK7 (queued for 2 m 14 s)
32023-11-20T18:23:56.236Zhttp://172.20.0.5:9969/ipxe/sock... ok
42023-11-20T18:23:56.285Zhttp://172.20.0.5:9969/os/sock/platform/i86pc/kernel/amd64/unix... ok
52023-11-20T18:23:56.320Zhttp://172.20.0.5:9969/os/sock/platform/i86pc/amd64/boot_archive... ok
62023-11-20T18:23:56.354Zhttp://172.20.0.5:9969/os/sock/platform/i86pc/amd64/boot_archive.hash... ok
72023-11-20T18:23:56.388Zhttp://172.20.0.5:9969/postboot/sock... ok
82023-11-20T18:23:56.422ZwcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U
92023-11-20T18:23:56.455Zcpu0: microcode has been updated from version 0xa001133 to 0xa0011d1
102023-11-20T18:23:56.489ZOxide Helios Version helios-2.0.22241 64-bit
112023-11-20T18:23:56.522ZConfiguring devices.
122023-11-20T18:23:56.567Z##################################################
132023-11-20T18:23:56.619Z#### ############# ##
142023-11-20T18:23:56.659Z### ### ############ ##
152023-11-20T18:23:56.703Z## ### # ## ### ## Oxide Computer Company ##
162023-11-20T18:23:56.739Z## ## ## ### # ### ##
172023-11-20T18:23:56.778Z## # ### #### #### This Station Under ##
182023-11-20T18:23:56.812Z### ### #### # ### Computer Control ##
192023-11-20T18:23:56.845Z#### #### ### ## ##
202023-11-20T18:23:56.883Z##################################################
212023-11-20T18:23:56.931ZHostname: sock
222023-11-20T18:23:56.973ZLoading smf(7) service descriptions:
232023-11-20T18:23:58.221Zdownloading input: /input/ci-tools/work/end-to-end-tests/bootstrap-7746be776f1918b3.gz
242023-11-20T18:24:01.252Zdownloaded input: /input/ci-tools/work/end-to-end-tests/bootstrap-7746be776f1918b3.gz
252023-11-20T18:24:01.325Zdownloading input: /input/ci-tools/work/end-to-end-tests/bootstrap.gz
262023-11-20T18:24:23.366Zdownloaded input: /input/ci-tools/work/end-to-end-tests/bootstrap.gz
272023-11-20T18:24:23.390Zdownloading input: /input/ci-tools/work/end-to-end-tests/end_to_end_tests-cb4cc51c87b08c71.gz
282023-11-20T18:24:51.416Zdownloaded input: /input/ci-tools/work/end-to-end-tests/end_to_end_tests-cb4cc51c87b08c71.gz
292023-11-20T18:24:51.440Zdownloading input: /input/ci-tools/work/caboose-util.gz
302023-11-20T18:24:54.361Zdownloaded input: /input/ci-tools/work/caboose-util.gz
312023-11-20T18:24:54.388Zdownloading input: /input/ci-tools/work/tufaceous.gz
322023-11-20T18:24:57.938Zdownloaded input: /input/ci-tools/work/tufaceous.gz
332023-11-20T18:24:57.961Zdownloading input: /input/package/work/version.txt
342023-11-20T18:24:57.985Zdownloaded input: /input/package/work/version.txt
352023-11-20T18:24:58.009Zdownloading input: /input/package/work/package.tar.gz
362023-11-20T18:25:24.447Zdownloaded input: /input/package/work/package.tar.gz
372023-11-20T18:25:24.472Zdownloading input: /input/package/work/global-zone-packages.tar.gz
382023-11-20T18:26:02.768Zdownloaded input: /input/package/work/global-zone-packages.tar.gz
392023-11-20T18:26:02.795Zdownloading input: /input/package/work/trampoline-global-zone-packages.tar.gz
402023-11-20T18:26:12.897Zdownloaded input: /input/package/work/trampoline-global-zone-packages.tar.gz
412023-11-20T18:26:12.927Zdownloading input: /input/package/work/zones/clickhouse.tar.gz
422023-11-20T18:27:05.636Zdownloaded input: /input/package/work/zones/clickhouse.tar.gz
432023-11-20T18:27:05.663Zdownloading input: /input/package/work/zones/clickhouse_keeper.tar.gz
442023-11-20T18:27:52.510Zdownloaded input: /input/package/work/zones/clickhouse_keeper.tar.gz
452023-11-20T18:27:52.534Zdownloading input: /input/package/work/zones/cockroachdb.tar.gz
462023-11-20T18:28:16.274Zdownloaded input: /input/package/work/zones/cockroachdb.tar.gz
472023-11-20T18:28:16.300Zdownloading input: /input/package/work/zones/crucible-pantry.tar.gz
482023-11-20T18:28:20.863Zdownloaded input: /input/package/work/zones/crucible-pantry.tar.gz
492023-11-20T18:28:20.894Zdownloading input: /input/package/work/zones/crucible.tar.gz
502023-11-20T18:28:28.367Zdownloaded input: /input/package/work/zones/crucible.tar.gz
512023-11-20T18:28:28.391Zdownloading input: /input/package/work/zones/external-dns.tar.gz
522023-11-20T18:28:33.455Zdownloaded input: /input/package/work/zones/external-dns.tar.gz
532023-11-20T18:28:33.480Zdownloading input: /input/package/work/zones/internal-dns.tar.gz
542023-11-20T18:28:37.674Zdownloaded input: /input/package/work/zones/internal-dns.tar.gz
552023-11-20T18:28:37.699Zdownloading input: /input/package/work/zones/ntp.tar.gz
562023-11-20T18:28:37.727Zdownloaded input: /input/package/work/zones/ntp.tar.gz
572023-11-20T18:28:37.761Zdownloading input: /input/package/work/zones/omicron-gateway-asic.tar.gz
582023-11-20T18:28:40.875Zdownloaded input: /input/package/work/zones/omicron-gateway-asic.tar.gz
592023-11-20T18:28:40.899Zdownloading input: /input/package/work/zones/omicron-gateway-softnpu.tar.gz
602023-11-20T18:28:44.133Zdownloaded input: /input/package/work/zones/omicron-gateway-softnpu.tar.gz
612023-11-20T18:28:44.158Zdownloading input: /input/package/work/zones/omicron-nexus-single-sled.tar.gz
622023-11-20T18:28:56.771Zdownloaded input: /input/package/work/zones/omicron-nexus-single-sled.tar.gz
632023-11-20T18:28:56.797Zdownloading input: /input/package/work/zones/omicron-nexus.tar.gz
642023-11-20T18:29:09.601Zdownloaded input: /input/package/work/zones/omicron-nexus.tar.gz
652023-11-20T18:29:09.625Zdownloading input: /input/package/work/zones/overlay.tar.gz
662023-11-20T18:29:09.670Zdownloaded input: /input/package/work/zones/overlay.tar.gz
672023-11-20T18:29:09.695Zdownloading input: /input/package/work/zones/oximeter-collector.tar.gz
682023-11-20T18:29:15.539Zdownloaded input: /input/package/work/zones/oximeter-collector.tar.gz
692023-11-20T18:29:15.562Zdownloading input: /input/package/work/zones/propolis-server.tar.gz
702023-11-20T18:29:27.102Zdownloaded input: /input/package/work/zones/propolis-server.tar.gz
712023-11-20T18:29:27.138Zdownloading input: /input/package/work/zones/switch-asic.tar.gz
722023-11-20T18:30:09.783Zdownloaded input: /input/package/work/zones/switch-asic.tar.gz
732023-11-20T18:30:09.807Zdownloading input: /input/package/work/zones/switch-softnpu.tar.gz
742023-11-20T18:30:41.865Zdownloaded input: /input/package/work/zones/switch-softnpu.tar.gz
 
752023-11-20T18:30:41.894Zstarting task 0: "setup"
762023-11-20T18:30:41.933Z++ uname -s
772023-11-20T18:30:41.971Z+ kern=SunOS
782023-11-20T18:30:42.012Z+ case "$kern" in
792023-11-20T18:30:42.040Z+ groupadd -g 12345 build
802023-11-20T18:30:42.068Z+ useradd -u 12345 -g build -d /home/build -s /bin/bash -c build -P 'Primary Administrator' build
812023-11-20T18:30:43.937Z+ zfs create -o mountpoint=/work rpool/work
822023-11-20T18:30:43.967Z++ awk '$2 == "/home" { print $3 }' /etc/mnttab
832023-11-20T18:30:43.991Z+ home_fs=zfs
842023-11-20T18:30:44.015Z+ [[ zfs == autofs ]]
852023-11-20T18:30:44.040Z+ mkdir -p /home/build
862023-11-20T18:30:44.064Z+ chown build:build /home/build /work
872023-11-20T18:30:45.975Z+ chmod 0700 /home/build /work
882023-11-20T18:30:45.999Zprocess exited: duration 4059 ms, exit code 0
 
892023-11-20T18:30:46.052Zstarting task 1: "authentication"
902023-11-20T18:30:46.121Zprocess exited: duration 44 ms, exit code 0
 
912023-11-20T18:30:46.170Zstarting task 2: "build"
922023-11-20T18:30:46.204Z+ trap _exit_trap EXIT
932023-11-20T18:30:46.230Z+ pfexec mdb -kwe 'secpolicy_fs_linkdir/v 55 48 89 e5 b8 01 00 00 00 5d c3'
942023-11-20T18:30:46.402Zsecpolicy_fs_linkdir: 0x55 = 0x55
952023-11-20T18:30:46.428Zsecpolicy_fs_linkdir+1: 0xb8 = 0x48
962023-11-20T18:30:46.452Zsecpolicy_fs_linkdir+2: 0x1 = 0x89
972023-11-20T18:30:46.478Zsecpolicy_fs_linkdir+3: 0 = 0xe5
982023-11-20T18:30:46.502Zsecpolicy_fs_linkdir+4: 0 = 0xb8
992023-11-20T18:30:46.527Zsecpolicy_fs_linkdir+5: 0 = 0x1
1002023-11-20T18:30:46.551Zsecpolicy_fs_linkdir+6: 0x48 = 0x0
1012023-11-20T18:30:46.579Zsecpolicy_fs_linkdir+7: 0x89 = 0x0
1022023-11-20T18:30:46.603Zsecpolicy_fs_linkdir+8: 0xe5 = 0x0
1032023-11-20T18:30:46.628Zsecpolicy_fs_linkdir+9: 0x48 = 0x5d
1042023-11-20T18:30:46.711Zsecpolicy_fs_linkdir+0xa: 0x83 = 0xc3
1052023-11-20T18:30:46.736Z+ [[ -d /opt/oxide ]]
1062023-11-20T18:30:46.762Z+ pfexec mv /opt/oxide /opt/oxide-underneath
1072023-11-20T18:30:46.788Z+ pfexec mkdir /opt/oxide
1082023-11-20T18:30:47.232Z+ pfexec mount -F tmpfs -O swap /opt/oxide
1092023-11-20T18:30:47.261Z+ [[ -d /opt/oxide-underneath ]]
1102023-11-20T18:30:47.292Z+ cd /opt/oxide-underneath
1112023-11-20T18:30:47.316Z+ pfexec tar ceEp@/f - .
1122023-11-20T18:30:47.340Z+ cd /opt/oxide
1132023-11-20T18:30:47.365Z+ pfexec tar xveEp@/f -
1142023-11-20T18:30:47.389Zx ., 0 bytes, 0 tape blocks
1152023-11-20T18:30:47.414Zx ./opte, 0 bytes, 0 tape blocks
1162023-11-20T18:30:47.439Zx ./opte/bin, 0 bytes, 0 tape blocks
1172023-11-20T18:30:47.463Zx ./opte/bin/opteadm, 50714832 bytes, 99053 tape blocks
1182023-11-20T18:30:47.488Z+ pfexec rm -rf /opt/oxide-underneath
1192023-11-20T18:30:47.512Z+ pfexec /sbin/zfs create -o mountpoint=/zone rpool/zone
1202023-11-20T18:30:47.537Z+ pfexec mkdir /opt/oxide/work
1212023-11-20T18:30:47.562Z+ pfexec chown build:build /opt/oxide/work
1222023-11-20T18:30:47.589Z+ cd /opt/oxide/work
1232023-11-20T18:30:47.613Z+ ptime -m tar xvzf /input/package/work/package.tar.gz
1242023-11-20T18:30:48.111ZDecompressing '/input/package/work/package.tar.gz' with '/usr/bin/gzcat'...
1252023-11-20T18:30:48.137Zx out/mg-ddm-gz.tar, 62723072 bytes, 122506 tape blocks
1262023-11-20T18:30:48.161Zx out/omicron-sled-agent.tar, 111626752 bytes, 218021 tape blocks
1272023-11-20T18:30:48.186Zx out/target/test, 75 bytes, 1 tape blocks
1282023-11-20T18:30:48.211Zx out/npuzone/npuzone, 37380856 bytes, 73010 tape blocks
1292023-11-20T18:30:48.235Zx package-manifest.toml, 22143 bytes, 44 tape blocks
1302023-11-20T18:30:48.289Zx smf/sled-agent/non-gimlet/config.toml, 3298 bytes, 7 tape blocks
1312023-11-20T18:30:48.317Zx target/release/omicron-package, 42522496 bytes, 83052 tape blocks
1322023-11-20T18:30:48.352Zx tools/create_virtual_hardware.sh, 3176 bytes, 7 tape blocks
1332023-11-20T18:30:48.792Zx tools/virtual_hardware.sh, 3408 bytes, 7 tape blocks
1342023-11-20T18:30:48.816Zx tools/scrimlet/create-softnpu-zone.sh, 619 bytes, 2 tape blocks
1352023-11-20T18:30:48.841Zx tools/scrimlet/destroy-softnpu-zone.sh, 195 bytes, 1 tape blocks
1362023-11-20T18:30:48.865Zx tools/scrimlet/softnpu-init.sh, 2148 bytes, 5 tape blocks
1372023-11-20T18:30:48.894Zx tools/scrimlet/softnpu-zone.txt, 330 bytes, 1 tape blocks
1382023-11-20T18:30:48.934Zx tools/scrimlet/softnpu.toml, 171 bytes, 1 tape blocks
1392023-11-20T18:30:48.966Z
1402023-11-20T18:30:48.993Zreal 1.537867018
1412023-11-20T18:30:49.017Zuser 1.387132561
1422023-11-20T18:30:49.041Zsys 0.265512075
1432023-11-20T18:30:49.065Ztrap 0.000062812
1442023-11-20T18:30:49.089Ztflt 0.000000000
1452023-11-20T18:30:49.117Zdflt 0.000000000
1462023-11-20T18:30:49.141Zkflt 0.000000000
1472023-11-20T18:30:49.166Zlock 0.000000000
1482023-11-20T18:30:49.190Zslp 1.386517239
1492023-11-20T18:30:49.214Zlat 0.034160012
1502023-11-20T18:30:49.239Zstop 0.000122624
1512023-11-20T18:30:49.263Z+ cp /input/package/work/zones/clickhouse_keeper.tar.gz /input/package/work/zones/clickhouse.tar.gz /input/package/work/zones/cockroachdb.tar.gz /input/package/work/zones/crucible-pantry.tar.gz /input/package/work/zones/crucible.tar.gz /input/package/work/zones/external-dns.tar.gz /input/package/work/zones/internal-dns.tar.gz /input/package/work/zones/ntp.tar.gz /input/package/work/zones/omicron-gateway-asic.tar.gz /input/package/work/zones/omicron-gateway-softnpu.tar.gz /input/package/work/zones/omicron-nexus-single-sled.tar.gz /input/package/work/zones/omicron-nexus.tar.gz /input/package/work/zones/overlay.tar.gz /input/package/work/zones/oximeter-collector.tar.gz /input/package/work/zones/propolis-server.tar.gz /input/package/work/zones/switch-asic.tar.gz /input/package/work/zones/switch-softnpu.tar.gz out/
1522023-11-20T18:30:49.287Z+ mv out/omicron-nexus-single-sled.tar.gz out/omicron-nexus.tar.gz
1532023-11-20T18:30:49.312Z+ mkdir tests
1542023-11-20T18:30:49.336Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1552023-11-20T18:30:49.361Z+ ptime -m gunzip
1562023-11-20T18:30:49.385Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap-7746be776f1918b3
1572023-11-20T18:30:49.410Z
1582023-11-20T18:30:49.434Zreal 0.138609320
1592023-11-20T18:30:49.459Zuser 0.127337320
1602023-11-20T18:30:49.483Zsys 0.011018731
1612023-11-20T18:30:49.507Ztrap 0.000016290
1622023-11-20T18:30:49.532Ztflt 0.000000000
1632023-11-20T18:30:49.556Zdflt 0.000000000
1642023-11-20T18:30:49.581Zkflt 0.000000000
1652023-11-20T18:30:49.605Zlock 0.000000000
1662023-11-20T18:30:49.630Zslp 0.000018880
1672023-11-20T18:30:49.654Zlat 0.000022850
1682023-11-20T18:30:49.679Zstop 0.000134123
1692023-11-20T18:30:49.703Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap-7746be776f1918b3
1702023-11-20T18:30:49.727Z+ chmod a+x tests/bootstrap-7746be776f1918b3
1712023-11-20T18:30:49.758Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1722023-11-20T18:30:49.829Z+ ptime -m gunzip
1732023-11-20T18:30:49.854Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap
1742023-11-20T18:30:51.354Z
1752023-11-20T18:30:51.379Zreal 1.975874387
1762023-11-20T18:30:51.404Zuser 1.858927247
1772023-11-20T18:30:51.429Zsys 0.116553317
1782023-11-20T18:30:51.455Ztrap 0.000152986
1792023-11-20T18:30:51.479Ztflt 0.000000000
1802023-11-20T18:30:51.563Zdflt 0.000000000
1812023-11-20T18:30:51.588Zkflt 0.000000000
1822023-11-20T18:30:51.612Zlock 0.000000000
1832023-11-20T18:30:51.639Zslp 0.000019810
1842023-11-20T18:30:51.676Zlat 0.000031510
1852023-11-20T18:30:51.704Zstop 0.000121743
1862023-11-20T18:30:51.732Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap
1872023-11-20T18:30:51.759Z+ chmod a+x tests/bootstrap
1882023-11-20T18:30:51.787Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1892023-11-20T18:30:51.818Z+ ptime -m gunzip
1902023-11-20T18:30:51.845Z++ basename /input/ci-tools/work/end-to-end-tests/end_to_end_tests-cb4cc51c87b08c71
1912023-11-20T18:30:53.641Z
1922023-11-20T18:30:53.667Zreal 2.273553980
1932023-11-20T18:30:53.692Zuser 2.128433890
1942023-11-20T18:30:53.717Zsys 0.144749329
1952023-11-20T18:30:53.744Ztrap 0.000143224
1962023-11-20T18:30:53.769Ztflt 0.000000000
1972023-11-20T18:30:53.794Zdflt 0.000000000
1982023-11-20T18:30:53.818Zkflt 0.000000000
1992023-11-20T18:30:53.843Zlock 0.000000000
2002023-11-20T18:30:53.867Zslp 0.000023290
2012023-11-20T18:30:53.891Zlat 0.000022211
2022023-11-20T18:30:53.915Zstop 0.000117472
2032023-11-20T18:30:53.940Z++ basename /input/ci-tools/work/end-to-end-tests/end_to_end_tests-cb4cc51c87b08c71
2042023-11-20T18:30:53.964Z+ chmod a+x tests/end_to_end_tests-cb4cc51c87b08c71
2052023-11-20T18:30:53.990Z+ bmat address ls
2062023-11-20T18:30:54.019ZNAME CIDR FIRST LAST COUNT
2072023-11-20T18:30:54.047Zextra 10.151.2.0/24 10.151.2.164 10.151.2.227 64
2082023-11-20T18:30:54.081Z++ bmat address ls -f extra -Ho first,last,count
2092023-11-20T18:30:54.113Z+ set -- 10.151.2.164 10.151.2.227 64
2102023-11-20T18:30:54.159Z+ EXTRA_IP_START=10.151.2.164
2112023-11-20T18:30:54.184Z+ EXTRA_IP_END=10.151.2.227
2122023-11-20T18:30:54.208Z+ EXTRA_IP_COUNT=64
2132023-11-20T18:30:54.234Z+ (( EXTRA_IP_COUNT >= 32 ))
2142023-11-20T18:30:54.259Z+ EXTRA_IP_BASE=10.151.2
2152023-11-20T18:30:54.283Z+ EXTRA_IP_FOCTET=164
2162023-11-20T18:30:54.309Z+ EXTRA_IP_LOCTET=227
2172023-11-20T18:30:54.334Z+ SERVICE_IP_POOL_START=10.151.2.164
2182023-11-20T18:30:54.359Z+ SERVICE_IP_POOL_END=10.151.2.173
2192023-11-20T18:30:54.383Z+ DNS_IP1=10.151.2.164
2202023-11-20T18:30:54.407Z+ DNS_IP2=10.151.2.165
2212023-11-20T18:30:54.441Z+ UPLINK_IP=10.151.2.174
2222023-11-20T18:30:54.468Z+ IPPOOL_START=10.151.2.175
2232023-11-20T18:30:54.492Z+ IPPOOL_END=10.151.2.227
2242023-11-20T18:30:54.526Z++ ipadm show-addr -po type,addr
2252023-11-20T18:30:54.554Z++ awk '-F[:/]' '$1 == "dhcp" {print $2}'
2262023-11-20T18:30:54.579Z+ GATEWAY_IP=10.151.2.100
2272023-11-20T18:30:54.604Z+ [[ -n 10.151.2.100 ]]
2282023-11-20T18:30:54.628Z+ ping -s 10.151.2.100 56 1
2292023-11-20T18:30:54.653ZPING 10.151.2.100: 56 data bytes
2302023-11-20T18:30:54.677Z64 bytes from 10.151.2.100: icmp_seq=0. time=0.011 ms
2312023-11-20T18:30:54.702Z
2322023-11-20T18:30:54.727Z----10.151.2.100 PING Statistics----
2332023-11-20T18:30:54.752Z1 packets transmitted, 1 packets received, 0% packet loss
2342023-11-20T18:30:54.776Zround-trip (ms) min/avg/max/stddev = 0.011/0.011/0.011/-nan
2352023-11-20T18:30:54.801Z++ arp -an
2362023-11-20T18:30:54.825Z++ awk -vgw=10.151.2.100 '$2 == gw {print $NF}'
2372023-11-20T18:30:54.854Z+ GATEWAY_MAC=18:c0:4d:0d:a0:2a
2382023-11-20T18:30:54.886Z+ routeadm -e ipv4-forwarding -u
2392023-11-20T18:30:54.910Z+ PXA_START=10.151.2.164
2402023-11-20T18:30:54.935Z+ PXA_END=10.151.2.227
2412023-11-20T18:30:54.968Z+ export GATEWAY_IP GATEWAY_MAC PXA_START PXA_END
2422023-11-20T18:30:55.010Z+ pfexec zpool create -f scratch c1t1d0 c2t1d0
2432023-11-20T18:30:55.045Z+ ZPOOL_VDEV_DIR=/scratch
2442023-11-20T18:30:55.075Z+ ptime -m pfexec ./tools/create_virtual_hardware.sh
2452023-11-20T18:30:55.103Z+++ dirname ./tools/create_virtual_hardware.sh
2462023-11-20T18:30:55.131Z++ cd ./tools
2472023-11-20T18:30:55.157Z++ pwd
2482023-11-20T18:30:55.212Z+ SOURCE_DIR=/opt/oxide/work/tools
2492023-11-20T18:30:55.241Z+ OMICRON_TOP=/opt/oxide/work/tools/..
2502023-11-20T18:30:55.265Z+ SOFTNPU_MODE=zone
2512023-11-20T18:30:55.296Z+ . /opt/oxide/work/tools/virtual_hardware.sh
2522023-11-20T18:30:55.329Z++ MARKER=/etc/opt/oxide/NO_INSTALL
2532023-11-20T18:30:55.355Z++ [[ -f /etc/opt/oxide/NO_INSTALL ]]
2542023-11-20T18:30:55.382Z++ dladm show-phys -p -o LINK
2552023-11-20T18:30:55.420Z++ head -1
2562023-11-20T18:30:55.446Z+ PHYSICAL_LINK=igb0
2572023-11-20T18:30:55.473Z+ echo 'Using igb0 as physical link'
2582023-11-20T18:30:55.509ZUsing igb0 as physical link
2592023-11-20T18:30:55.535Z+ ensure_run_as_root
2602023-11-20T18:30:55.560Z++ id -u
2612023-11-20T18:30:55.585Z+ [[ 0 -ne 0 ]]
2622023-11-20T18:30:55.609Z+ ensure_zpools
2632023-11-20T18:30:55.634Z+ ZPOOL_TYPES=('oxp_' 'oxi_')
2642023-11-20T18:30:55.658Z+ for ZPOOL_TYPE in "${ZPOOL_TYPES[@]}"
2652023-11-20T18:30:55.682Z+ readarray -t ZPOOLS
2662023-11-20T18:30:55.707Z++ grep '"oxp_' /opt/oxide/work/tools/../smf/sled-agent/non-gimlet/config.toml
2672023-11-20T18:30:55.732Z++ sed 's/[ ",]//g'
2682023-11-20T18:30:55.760Z+ for ZPOOL in "${ZPOOLS[@]}"
2692023-11-20T18:30:55.794Z+ echo 'Zpool: [oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b]'
2702023-11-20T18:30:55.823ZZpool: [oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b]
2712023-11-20T18:30:55.855Z+ VDEV_PATH=/scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
2722023-11-20T18:30:55.893Z+ [[ -f /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
2732023-11-20T18:30:55.923Z+ dd if=/dev/zero of=/scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
2742023-11-20T18:30:55.951ZZFS vdev /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
2752023-11-20T18:30:56.391Z0+0 records in
2762023-11-20T18:30:56.418ZZFS zpool oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b exists
2772023-11-20T18:30:56.448Z0+0 records out
2782023-11-20T18:30:56.476ZZpool: [oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03]
2792023-11-20T18:30:56.503Z0 bytes (0 iB) transferred in 0.000006 secs (0 iB/sec)
2802023-11-20T18:30:56.531ZZFS vdev /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
2812023-11-20T18:30:56.561Z+ success 'ZFS vdev /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
2822023-11-20T18:30:56.587ZZFS zpool oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
2832023-11-20T18:30:56.615Z+ set +x
2842023-11-20T18:30:56.641ZZpool: [oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03]
2852023-11-20T18:30:56.670Z++ zpool list -o name
2862023-11-20T18:30:56.695ZZFS vdev /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
2872023-11-20T18:30:56.719Z++ grep oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b
2882023-11-20T18:30:56.743ZZFS zpool oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
2892023-11-20T18:30:56.768Z+ [[ -z '' ]]
2902023-11-20T18:30:56.792ZZpool: [oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03]
2912023-11-20T18:30:56.817Z+ zpool create -f oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
2922023-11-20T18:30:56.841ZZFS vdev /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
2932023-11-20T18:30:56.865Z+ success 'ZFS zpool oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
2942023-11-20T18:30:56.891ZZFS zpool oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
2952023-11-20T18:30:56.916Z+ set +x
2962023-11-20T18:30:56.941ZZpool: [oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03]
2972023-11-20T18:30:56.966Z+ for ZPOOL in "${ZPOOLS[@]}"
2982023-11-20T18:30:56.991ZZFS vdev /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
2992023-11-20T18:30:57.019Z+ echo 'Zpool: [oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3002023-11-20T18:30:57.047ZZFS zpool oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
3012023-11-20T18:30:57.079Z+ VDEV_PATH=/scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3022023-11-20T18:30:57.111ZZpool: [oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930]
3032023-11-20T18:30:57.144Z+ [[ -f /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3042023-11-20T18:30:57.181ZZFS vdev /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev exists
3052023-11-20T18:30:57.208Z+ dd if=/dev/zero of=/scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3062023-11-20T18:30:57.236ZZFS zpool oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 exists
3072023-11-20T18:30:57.310Z0+0 records in
3082023-11-20T18:30:57.345ZZpool: [oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60]
3092023-11-20T18:30:57.787Z0+0 records out
3102023-11-20T18:30:57.818ZZFS vdev /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev exists
3112023-11-20T18:30:57.855Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
3122023-11-20T18:30:57.889ZZFS zpool oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 exists
3132023-11-20T18:30:57.925Z+ success 'ZFS vdev /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3142023-11-20T18:30:57.965ZZpool: [oxp_31bd71cd-4736-4a12-a387-9b74b050396f]
3152023-11-20T18:30:58.402Z+ set +x
3162023-11-20T18:30:58.433ZZFS vdev /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev exists
3172023-11-20T18:30:58.459Z++ zpool list -o name
3182023-11-20T18:30:58.513ZZFS zpool oxp_31bd71cd-4736-4a12-a387-9b74b050396f exists
3192023-11-20T18:30:58.538Z++ grep oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03
3202023-11-20T18:30:58.564ZZpool: [oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7]
3212023-11-20T18:30:58.589Z+ [[ -z '' ]]
3222023-11-20T18:30:58.624ZZFS vdev /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev exists
3232023-11-20T18:30:59.071Z+ zpool create -f oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3242023-11-20T18:30:59.119ZZFS zpool oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 exists
3252023-11-20T18:30:59.162Z+ success 'ZFS zpool oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3262023-11-20T18:30:59.610ZZpool: [oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b]
3272023-11-20T18:30:59.646Z+ set +x
3282023-11-20T18:30:59.681ZZFS vdev /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
3292023-11-20T18:30:59.716Z+ for ZPOOL in "${ZPOOLS[@]}"
3302023-11-20T18:30:59.744ZZFS zpool oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b exists
3312023-11-20T18:30:59.777Z+ echo 'Zpool: [oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3322023-11-20T18:30:59.805ZZpool: [oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b]
3332023-11-20T18:30:59.831Z+ VDEV_PATH=/scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3342023-11-20T18:30:59.855ZZFS vdev /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
3352023-11-20T18:30:59.880Z+ [[ -f /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3362023-11-20T18:30:59.904ZZFS zpool oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b exists
3372023-11-20T18:30:59.929Z+ dd if=/dev/zero of=/scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3382023-11-20T18:30:59.953ZSimnet net0/sc0_0 exists
3392023-11-20T18:30:59.981Z0+0 records in
3402023-11-20T18:31:00.018ZSimnet net1/sc1_0 exists
3412023-11-20T18:31:00.481Z0+0 records out
3422023-11-20T18:31:00.922ZVnic sc0_1 exists
3432023-11-20T18:31:00.950Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
3442023-11-20T18:31:00.987Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/softnpu/image/dec63e67156fe6e958991bbfa090629868115ab5/softnpu.sha256.txt
3452023-11-20T18:31:01.019Z+ success 'ZFS vdev /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3462023-11-20T18:31:01.056Z+ set +x
3472023-11-20T18:31:01.086Z++ zpool list -o name
3482023-11-20T18:31:01.116Z++ grep oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03
3492023-11-20T18:31:01.141Z+ [[ -z '' ]]
3502023-11-20T18:31:01.166Z+ zpool create -f oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3512023-11-20T18:31:01.192Z+ success 'ZFS zpool oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3522023-11-20T18:31:01.217Z+ set +x
3532023-11-20T18:31:01.241Z+ for ZPOOL in "${ZPOOLS[@]}"
3542023-11-20T18:31:01.266Z+ echo 'Zpool: [oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3552023-11-20T18:31:01.291Z+ VDEV_PATH=/scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3562023-11-20T18:31:01.316Z+ [[ -f /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3572023-11-20T18:31:01.346Z+ dd if=/dev/zero of=/scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3582023-11-20T18:31:01.371Z0+0 records in
3592023-11-20T18:31:01.398Z0+0 records out
3602023-11-20T18:31:01.433Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3612023-11-20T18:31:01.873Z+ success 'ZFS vdev /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3622023-11-20T18:31:01.904Z+ set +x
3632023-11-20T18:31:01.933Z++ zpool list -o name
3642023-11-20T18:31:01.972Z++ grep oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03
3652023-11-20T18:31:02.841Z+ [[ -z '' ]]
3662023-11-20T18:31:02.876Z+ zpool create -f oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3672023-11-20T18:31:02.904Z+ success 'ZFS zpool oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3682023-11-20T18:31:02.938Z+ set +x
3692023-11-20T18:31:02.976Z+ for ZPOOL in "${ZPOOLS[@]}"
3702023-11-20T18:31:03.023Z+ echo 'Zpool: [oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3712023-11-20T18:31:03.060Z+ VDEV_PATH=/scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3722023-11-20T18:31:03.092Z+ [[ -f /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3732023-11-20T18:31:03.119Z+ dd if=/dev/zero of=/scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3742023-11-20T18:31:03.144Z0+0 records in
3752023-11-20T18:31:03.168Z0+0 records out
3762023-11-20T18:31:03.193Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3772023-11-20T18:31:03.314Z+ success 'ZFS vdev /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3782023-11-20T18:31:03.338Z+ set +x
3792023-11-20T18:31:03.364Z++ zpool list -o name
3802023-11-20T18:31:03.388Z++ grep oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03
3812023-11-20T18:31:03.421Z+ [[ -z '' ]]
3822023-11-20T18:31:03.896Z+ zpool create -f oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3832023-11-20T18:31:04.353Z+ success 'ZFS zpool oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3842023-11-20T18:31:04.391Z+ set +x
3852023-11-20T18:31:04.427Z+ for ZPOOL in "${ZPOOLS[@]}"
3862023-11-20T18:31:04.455Z+ echo 'Zpool: [oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930]'
3872023-11-20T18:31:04.481Z+ VDEV_PATH=/scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev
3882023-11-20T18:31:04.508Z+ [[ -f /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev ]]
3892023-11-20T18:31:04.534Z+ dd if=/dev/zero of=/scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev bs=1 count=0 seek=20G
3902023-11-20T18:31:04.561Z0+0 records in
3912023-11-20T18:31:04.589Z0+0 records out
3922023-11-20T18:31:04.617Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3932023-11-20T18:31:04.642Z+ success 'ZFS vdev /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev exists'
3942023-11-20T18:31:04.699Z+ set +x
3952023-11-20T18:31:04.734Z++ zpool list -o name
3962023-11-20T18:31:04.779Z++ grep oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930
3972023-11-20T18:31:04.816Z+ [[ -z '' ]]
3982023-11-20T18:31:05.262Z+ zpool create -f oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev
3992023-11-20T18:31:05.288Z+ success 'ZFS zpool oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 exists'
4002023-11-20T18:31:05.314Z+ set +x
4012023-11-20T18:31:05.339Z+ for ZPOOL in "${ZPOOLS[@]}"
4022023-11-20T18:31:05.374Z+ echo 'Zpool: [oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60]'
4032023-11-20T18:31:05.401Z+ VDEV_PATH=/scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev
4042023-11-20T18:31:05.425Z+ [[ -f /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev ]]
4052023-11-20T18:31:05.450Z+ dd if=/dev/zero of=/scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev bs=1 count=0 seek=20G
4062023-11-20T18:31:05.475Z0+0 records in
4072023-11-20T18:31:05.500Z0+0 records out
4082023-11-20T18:31:05.526Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
4092023-11-20T18:31:05.554Z+ success 'ZFS vdev /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev exists'
4102023-11-20T18:31:05.582Z+ set +x
4112023-11-20T18:31:05.614Z++ zpool list -o name
4122023-11-20T18:31:05.659Z++ grep oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60
4132023-11-20T18:31:05.693Z+ [[ -z '' ]]
4142023-11-20T18:31:05.742Z+ zpool create -f oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev
4152023-11-20T18:31:05.780Z+ success 'ZFS zpool oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 exists'
4162023-11-20T18:31:05.810Z+ set +x
4172023-11-20T18:31:05.851Z+ for ZPOOL in "${ZPOOLS[@]}"
4182023-11-20T18:31:06.326Z+ echo 'Zpool: [oxp_31bd71cd-4736-4a12-a387-9b74b050396f]'
4192023-11-20T18:31:06.358Z+ VDEV_PATH=/scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev
4202023-11-20T18:31:06.398Z+ [[ -f /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev ]]
4212023-11-20T18:31:06.440Z+ dd if=/dev/zero of=/scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev bs=1 count=0 seek=20G
4222023-11-20T18:31:06.472Z0+0 records in
4232023-11-20T18:31:06.502Z0+0 records out
4242023-11-20T18:31:06.531Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4252023-11-20T18:31:06.560Z+ success 'ZFS vdev /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev exists'
4262023-11-20T18:31:06.679Z+ set +x
4272023-11-20T18:31:06.714Z++ zpool list -o name
4282023-11-20T18:31:06.791Z++ grep oxp_31bd71cd-4736-4a12-a387-9b74b050396f
4292023-11-20T18:31:06.818Z+ [[ -z '' ]]
4302023-11-20T18:31:06.861Z+ zpool create -f oxp_31bd71cd-4736-4a12-a387-9b74b050396f /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev
4312023-11-20T18:31:06.896Z+ success 'ZFS zpool oxp_31bd71cd-4736-4a12-a387-9b74b050396f exists'
4322023-11-20T18:31:06.925Z+ set +x
4332023-11-20T18:31:06.957Z+ for ZPOOL in "${ZPOOLS[@]}"
4342023-11-20T18:31:06.984Z+ echo 'Zpool: [oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7]'
4352023-11-20T18:31:07.013Z+ VDEV_PATH=/scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev
4362023-11-20T18:31:07.040Z+ [[ -f /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev ]]
4372023-11-20T18:31:07.069Z+ dd if=/dev/zero of=/scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev bs=1 count=0 seek=20G
4382023-11-20T18:31:07.097Z0+0 records in
4392023-11-20T18:31:07.133Z0+0 records out
4402023-11-20T18:31:07.166Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4412023-11-20T18:31:07.601Z+ success 'ZFS vdev /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev exists'
4422023-11-20T18:31:07.632Z+ set +x
4432023-11-20T18:31:07.669Z++ zpool list -o name
4442023-11-20T18:31:08.107Z++ grep oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7
4452023-11-20T18:31:08.143Z+ [[ -z '' ]]
4462023-11-20T18:31:08.170Z+ zpool create -f oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev
4472023-11-20T18:31:08.196Z+ success 'ZFS zpool oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 exists'
4482023-11-20T18:31:08.234Z+ set +x
4492023-11-20T18:31:08.268Z+ for ZPOOL_TYPE in "${ZPOOL_TYPES[@]}"
4502023-11-20T18:31:08.300Z+ readarray -t ZPOOLS
4512023-11-20T18:31:08.341Z++ grep '"oxi_' /opt/oxide/work/tools/../smf/sled-agent/non-gimlet/config.toml
4522023-11-20T18:31:08.903Z++ sed 's/[ ",]//g'
4532023-11-20T18:31:08.943Z+ for ZPOOL in "${ZPOOLS[@]}"
4542023-11-20T18:31:08.980Z+ echo 'Zpool: [oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b]'
4552023-11-20T18:31:09.014Z+ VDEV_PATH=/scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4562023-11-20T18:31:09.051Z+ [[ -f /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
4572023-11-20T18:31:09.090Z+ dd if=/dev/zero of=/scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
4582023-11-20T18:31:09.119Z0+0 records in
4592023-11-20T18:31:09.148Z0+0 records out
4602023-11-20T18:31:09.182Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4612023-11-20T18:31:09.213Z+ success 'ZFS vdev /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
4622023-11-20T18:31:09.243Z+ set +x
4632023-11-20T18:31:09.274Z++ zpool list -o name
4642023-11-20T18:31:09.307Z++ grep oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b
4652023-11-20T18:31:09.338Z+ [[ -z '' ]]
4662023-11-20T18:31:09.770Z+ zpool create -f oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4672023-11-20T18:31:09.807Z+ success 'ZFS zpool oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
4682023-11-20T18:31:09.847Z+ set +x
4692023-11-20T18:31:09.927Z+ for ZPOOL in "${ZPOOLS[@]}"
4702023-11-20T18:31:10.371Z+ echo 'Zpool: [oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b]'
4712023-11-20T18:31:10.402Z+ VDEV_PATH=/scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4722023-11-20T18:31:10.426Z+ [[ -f /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
4732023-11-20T18:31:10.453Z+ dd if=/dev/zero of=/scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
4742023-11-20T18:31:10.892Z0+0 records in
4752023-11-20T18:31:10.976Z0+0 records out
4762023-11-20T18:31:11.008Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4772023-11-20T18:31:11.033Z+ success 'ZFS vdev /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
4782023-11-20T18:31:11.058Z+ set +x
4792023-11-20T18:31:11.083Z++ zpool list -o name
4802023-11-20T18:31:11.109Z++ grep oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b
4812023-11-20T18:31:11.139Z+ [[ -z '' ]]
4822023-11-20T18:31:11.165Z+ zpool create -f oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4832023-11-20T18:31:11.191Z+ success 'ZFS zpool oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
4842023-11-20T18:31:11.215Z+ set +x
4852023-11-20T18:31:11.240Z+ [[ zone == \z\o\n\e ]]
4862023-11-20T18:31:11.265Z+ ensure_simulated_links igb0
4872023-11-20T18:31:11.289Z+ local PHYSICAL_LINK=igb0
4882023-11-20T18:31:11.313Z+ INDICES=("0" "1")
4892023-11-20T18:31:11.345Z+ for I in "${INDICES[@]}"
4902023-11-20T18:31:11.382Z++ get_simnet_name_if_exists net0
4912023-11-20T18:31:11.408Z++ dladm show-simnet -p -o LINK net0
4922023-11-20T18:31:11.439Z++ echo ''
4932023-11-20T18:31:11.469Z+ [[ -z '' ]]
4942023-11-20T18:31:11.494Z+ dladm create-simnet -t net0
4952023-11-20T18:31:11.518Z+ dladm create-simnet -t sc0_0
4962023-11-20T18:31:11.542Z+ dladm modify-simnet -t -p net0 sc0_0
4972023-11-20T18:31:11.574Z+ dladm set-linkprop -p mtu=9000 sc0_0
4982023-11-20T18:31:11.603Z+ success 'Simnet net0/sc0_0 exists'
4992023-11-20T18:31:11.628Z+ set +x
5002023-11-20T18:31:11.652Zdownloading softnpu ...done
5012023-11-20T18:31:11.677Z+ for I in "${INDICES[@]}"
5022023-11-20T18:31:11.703Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/sidecar-lite/release/f0585a29fb0285f7a1220c1118856b0e5c1f75c5/libsidecar_lite.so.sha256.txt
5032023-11-20T18:31:11.728Z++ get_simnet_name_if_exists net1
5042023-11-20T18:31:11.752Z++ dladm show-simnet -p -o LINK net1
5052023-11-20T18:31:11.780Z++ echo ''
5062023-11-20T18:31:11.816Z+ [[ -z '' ]]
5072023-11-20T18:31:11.848Z+ dladm create-simnet -t net1
5082023-11-20T18:31:11.878Z+ dladm create-simnet -t sc1_0
5092023-11-20T18:31:11.916Z+ dladm modify-simnet -t -p net1 sc1_0
5102023-11-20T18:31:11.957Z+ dladm set-linkprop -p mtu=9000 sc1_0
5112023-11-20T18:31:12.027Z+ success 'Simnet net1/sc1_0 exists'
5122023-11-20T18:31:12.057Z+ set +x
5132023-11-20T18:31:12.084Z++ get_vnic_name_if_exists sc0_1
5142023-11-20T18:31:12.109Z++ dladm show-vnic -p -o LINK sc0_1
5152023-11-20T18:31:12.134Z++ echo ''
5162023-11-20T18:31:12.159Z+ [[ -z '' ]]
5172023-11-20T18:31:12.183Z+ dladm create-vnic -t sc0_1 -l igb0 -m a8:e1:de:01:70:1d
5182023-11-20T18:31:12.208Z+ [[ -v PROMISC_FILT_OFF ]]
5192023-11-20T18:31:12.232Z+ success 'Vnic sc0_1 exists'
5202023-11-20T18:31:12.256Z+ set +x
5212023-11-20T18:31:12.281Z+ warn_if_no_proxy_arp
5222023-11-20T18:31:12.306Z+ [[ -v PXA_START ]]
5232023-11-20T18:31:12.343Z+ [[ -v PXA_END ]]
5242023-11-20T18:31:12.379Z+ ensure_softnpu_zone
5252023-11-20T18:31:12.417Z+ zoneadm list
5262023-11-20T18:31:12.444Z+ grep -q sidecar_softnpu
5272023-11-20T18:31:12.469Z+ [[ -f out/npuzone/npuzone ]]
5282023-11-20T18:31:12.494Z+ out/npuzone/npuzone create sidecar --omicron-zone --ports sc0_0,tfportrear0_0 --ports sc0_1,tfportqsfp0_0 --sidecar-lite-commit f0585a29fb0285f7a1220c1118856b0e5c1f75c5 --softnpu-commit dec63e67156fe6e958991bbfa090629868115ab5
5292023-11-20T18:31:14.599Zdownloading asic_program.so ...done
5302023-11-20T18:31:14.628Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/sidecar-lite/release/f0585a29fb0285f7a1220c1118856b0e5c1f75c5/scadm.sha256.txt
5312023-11-20T18:31:24.343Zdownloading scadm ...done
5322023-11-20T18:31:24.368Zcreating port tfportrear0_0 for sc0_0
5332023-11-20T18:31:24.392Zcreating port tfportqsfp0_0 for sc0_1
5342023-11-20T18:31:24.417Zsoftnpu zone setup
5352023-11-20T18:31:24.452Zconfigure zone
5362023-11-20T18:31:24.477Zinstall zone
5372023-11-20T18:31:25.481Zboot zone
5382023-11-20T18:31:28.988Z[sidecar_softnpu] svcs milestone/network
5392023-11-20T18:31:29.013ZSTATE STIME FMRI
5402023-11-20T18:31:29.037Zonline 18:31:26 svc:/milestone/network:default
5412023-11-20T18:31:29.062Z[sidecar_softnpu] /softnpu/npu start
5422023-11-20T18:31:29.086Z
5432023-11-20T18:31:29.111Z+ /opt/oxide/work/tools/scrimlet/softnpu-init.sh
5442023-11-20T18:31:29.135Z+ GATEWAY_IP=10.151.2.100
5452023-11-20T18:31:29.160Z+ echo 'Using 10.151.2.100 as gateway ip'
5462023-11-20T18:31:29.187ZUsing 10.151.2.100 as gateway ip
5472023-11-20T18:31:29.219Z+ [[ ! -v GATEWAY_MAC ]]
5482023-11-20T18:31:29.254Z+ GATEWAY_MAC=18:c0:4d:0d:a0:2a
5492023-11-20T18:31:29.279Z++ grep -c -E '^([0-9a-fA-F]{2}:){5}[0-9a-fA-F]{2}$' /dev/fd/63
5502023-11-20T18:31:29.310Z+++ echo 18:c0:4d:0d:a0:2a
5512023-11-20T18:31:29.782Z+ COUNT=1
5522023-11-20T18:31:29.808Z+ [[ 1 -ne 1 ]]
5532023-11-20T18:31:29.835Z+ echo 'Using 18:c0:4d:0d:a0:2a as gateway mac'
5542023-11-20T18:31:29.862ZUsing 18:c0:4d:0d:a0:2a as gateway mac
5552023-11-20T18:31:29.887Z+ z_scadm add-arp-entry 10.151.2.100 18:c0:4d:0d:a0:2a
5562023-11-20T18:31:29.913Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone add-arp-entry 10.151.2.100 18:c0:4d:0d:a0:2a
5572023-11-20T18:31:29.940Z+ PXA_MAC=a8:e1:de:01:70:1d
5582023-11-20T18:31:29.965Z+ [[ -v PXA_START ]]
5592023-11-20T18:31:29.989Z+ z_scadm add-proxy-arp 10.151.2.164 10.151.2.227 a8:e1:de:01:70:1d
5602023-11-20T18:31:30.016Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone add-proxy-arp 10.151.2.164 10.151.2.227 a8:e1:de:01:70:1d
5612023-11-20T18:31:30.044Z+ z_scadm dump-state
5622023-11-20T18:31:30.072Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone dump-state
5632023-11-20T18:31:30.104Zlocal v6:
5642023-11-20T18:31:30.136Zlocal v4:
5652023-11-20T18:31:30.171Zrouter v6:
5662023-11-20T18:31:30.207Zrouter v4_idx:
5672023-11-20T18:31:30.256Zrouter v4_routes:
5682023-11-20T18:31:30.287Zresolver v4:
5692023-11-20T18:31:30.319Z10.151.2.100 -> 18:c0:4d:0d:a0:2a
5702023-11-20T18:31:30.344Zresolver v6:
5712023-11-20T18:31:30.369Znat v4:
5722023-11-20T18:31:30.394Znat_v6:
5732023-11-20T18:31:30.418Zport_mac:
5742023-11-20T18:31:30.443Zproxy arp:
5752023-11-20T18:31:30.467Z10.151.2.164/10.151.2.227: a8:e1:de:01:70:1d
5762023-11-20T18:31:30.492Z+ success 'softnpu zone exists'
5772023-11-20T18:31:30.516Z+ set +x
5782023-11-20T18:31:30.541Zsoftnpu zone exists
5792023-11-20T18:31:30.568Z
5802023-11-20T18:31:30.593Zreal 34.764329624
5812023-11-20T18:31:30.620Zuser 0.709222049
5822023-11-20T18:31:30.661Zsys 1.644260363
5832023-11-20T18:31:30.686Ztrap 0.000066487
5842023-11-20T18:31:30.710Ztflt 0.000115476
5852023-11-20T18:31:30.735Zdflt 0.001477388
5862023-11-20T18:31:30.760Zkflt 0.000000000
5872023-11-20T18:31:30.791Zlock 27:09.411246954
5882023-11-20T18:31:30.819Zslp 1:44.299664013
5892023-11-20T18:31:30.843Zlat 0.069442721
5902023-11-20T18:31:30.868Zstop 0.352988948
5912023-11-20T18:31:30.893Z+ tar xf out/omicron-sled-agent.tar pkg/config-rss.toml
5922023-11-20T18:31:30.917Z++ sed -n 's/silo_name = "\(.*\)"/\1/p' pkg/config-rss.toml
5932023-11-20T18:31:30.944Z+ SILO_NAME=recovery
5942023-11-20T18:31:30.968Z++ sed -n 's/external_dns_zone_name = "\(.*\)"/\1/p' pkg/config-rss.toml
5952023-11-20T18:31:30.993Z+ EXTERNAL_DNS_DOMAIN=oxide.test
5962023-11-20T18:31:31.017Z+ sed -i~ '
5972023-11-20T18:31:31.042Z /^external_dns_ips/c\
5982023-11-20T18:31:31.070Zexternal_dns_ips = [ "10.151.2.164", "10.151.2.165" ]
5992023-11-20T18:31:31.110Z /^\[\[internal_services_ip_pool_ranges/,/^$/ {
6002023-11-20T18:31:31.145Z /^first/c\
6012023-11-20T18:31:31.170Zfirst = "10.151.2.164"
6022023-11-20T18:31:31.195Z /^last/c\
6032023-11-20T18:31:31.219Zlast = "10.151.2.173"
6042023-11-20T18:31:31.245Z--- pkg/config-rss.toml~ Mon Jul 24 01:21:28 2006
6052023-11-20T18:31:31.270Z }
6062023-11-20T18:31:31.295Z+++ pkg/config-rss.toml Mon Nov 20 18:31:28 2023
6072023-11-20T18:31:31.319Z /^\[rack_network_config/,/^$/ {
6082023-11-20T18:31:31.344Z@@ -41,7 +41,7 @@
6092023-11-20T18:31:31.368Z /^infra_ip_first/c\
6102023-11-20T18:31:31.393Z # the DNS domain delegated to the rack by the customer. Each of these addresses
6112023-11-20T18:31:31.418Zinfra_ip_first = "10.151.2.174"
6122023-11-20T18:31:31.442Z # must be contained in one of the "internal services" IP Pool ranges listed
6132023-11-20T18:31:31.466Z /^infra_ip_last/c\
6142023-11-20T18:31:31.491Z # below.
6152023-11-20T18:31:31.520Zinfra_ip_last = "10.151.2.174"
6162023-11-20T18:31:31.553Z-external_dns_ips = [ "192.168.1.20", "192.168.1.21" ]
6172023-11-20T18:31:31.578Z }
6182023-11-20T18:31:31.612Z+external_dns_ips = [ "10.151.2.164", "10.151.2.165" ]
6192023-11-20T18:31:32.081Z /^\[\[rack_network_config.ports/,/^$/ {
6202023-11-20T18:31:32.108Z
6212023-11-20T18:31:32.134Z /^routes/c\
6222023-11-20T18:31:32.570Z # Initial TLS certificates for the external API
6232023-11-20T18:31:32.600Zroutes = \[{nexthop = "10.151.2.100", destination = "0.0.0.0/0"}\]
6242023-11-20T18:31:32.625Z #
6252023-11-20T18:31:32.654Z /^addresses/c\
6262023-11-20T18:31:32.679Z@@ -77,8 +77,8 @@
6272023-11-20T18:31:32.723Zaddresses = \["10.151.2.174/32"\]
6282023-11-20T18:31:32.749Z #
6292023-11-20T18:31:32.776Z }
6302023-11-20T18:31:32.805Z # For more on this and what to put here, see docs/how-to-run.adoc.
6312023-11-20T18:31:32.834Z' pkg/config-rss.toml
6322023-11-20T18:31:32.860Z [[internal_services_ip_pool_ranges]]
6332023-11-20T18:31:32.885Z+ diff -u pkg/config-rss.toml~ pkg/config-rss.toml
6342023-11-20T18:31:32.909Z-first = "192.168.1.20"
6352023-11-20T18:31:32.935Z+ true
6362023-11-20T18:31:32.960Z-last = "192.168.1.29"
6372023-11-20T18:31:32.985Z+ tar rvf out/omicron-sled-agent.tar pkg/config-rss.toml
6382023-11-20T18:31:33.011Z+first = "10.151.2.164"
6392023-11-20T18:31:33.043Z+ rm -f pkg/config-rss.toml pkg/config-rss.toml~
6402023-11-20T18:31:33.068Z+last = "10.151.2.173"
6412023-11-20T18:31:33.099Z+ TLS_NAME=recovery.sys.oxide.test
6422023-11-20T18:31:33.131Z
6432023-11-20T18:31:33.160Z+ openssl req -newkey rsa:4096 -x509 -sha256 -days 3 -nodes -out pkg/initial-tls-cert.pem -keyout pkg/initial-tls-key.pem -subj /CN=recovery.sys.oxide.test -addext subjectAltName=DNS:recovery.sys.oxide.test -addext basicConstraints=critical,CA:FALSE -config /dev/stdin
6442023-11-20T18:31:33.186Z # TODO - this configuration is subject to change going forward. Ultimately these
6452023-11-20T18:31:33.211Z...+......+...........+...+......+.+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*....+.....+.+..+.......+...+.....+...+....+..+.........+.+........+.............+..............+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.....+.......+...........+.........+.......+...........+...+.......+.....+...+...................+..+....+....................+.+...............+...........+............+...+....+.....+.............+..+.+..+.......+........+.+......+.........+..+....+...+......+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
6462023-11-20T18:31:33.236Z # parameters should be provided to the control plane via wicket, but we need to
6472023-11-20T18:31:33.261Z.......+...+.+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.+...+.............+..+...+...+..........+...........+...+.+..+....+...+..+..........+......+..+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*....+......+..................+...+...............+.....+.........................+...+.........+..+...+...............+......+............+...................+.....+.+.....+.........+.+.....+.........+.+............+..+....+......+...+..+...+......+.+......+...+..+..................+.+........+.+......+.....+......+.........+.........+..........+..+.......+........+......................+.....+............+....+.....+......+.........+..........+..............+...+............+.+..+....+......+..+.....................+.........+...+....+......+..+......+.+...+...+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
6482023-11-20T18:31:33.286Z@@ -92,8 +92,8 @@
6492023-11-20T18:31:34.094Z-----
6502023-11-20T18:31:34.131Z # A range of IP addresses used by Boundary Services on the external network. In
6512023-11-20T18:31:34.171Z+ tar rvf out/omicron-sled-agent.tar pkg/initial-tls-cert.pem pkg/initial-tls-key.pem
6522023-11-20T18:31:34.201Z # a real system, these would be addresses of the uplink ports on the Sidecar.
6532023-11-20T18:31:34.237Z+ rm -f pkg/initial-tls-cert.pem pkg/initial-tls-key.pem
6542023-11-20T18:31:34.268Z # With softnpu, only one address is used.
6552023-11-20T18:31:34.295Z+ rmdir pkg
6562023-11-20T18:31:34.320Z-infra_ip_first = "192.168.1.30"
6572023-11-20T18:31:34.344Z+ E2E_TLS_CERT=/opt/oxide/sled-agent/pkg/initial-tls-cert.pem
6582023-11-20T18:31:34.369Z-infra_ip_last = "192.168.1.30"
6592023-11-20T18:31:34.394Z+ pfexec mkdir -p /usr/oxide
6602023-11-20T18:31:34.419Z+infra_ip_first = "10.151.2.174"
6612023-11-20T18:31:34.443Z+ pfexec rm -f /usr/oxide/tcpproxy
6622023-11-20T18:31:34.468Z+infra_ip_last = "10.151.2.174"
6632023-11-20T18:31:34.493Z+ pfexec curl -sSfL -o /usr/oxide/tcpproxy http://catacomb.eng.oxide.computer:12346/tcpproxy
6642023-11-20T18:31:34.520Z
6652023-11-20T18:31:34.549Z+ pfexec chmod +x /usr/oxide/tcpproxy
6662023-11-20T18:31:34.582Z # Configurations for BGP routers to run on the scrimlets.
6672023-11-20T18:31:34.622Z+ pfexec rm -f /var/svc/manifest/site/tcpproxy.xml
6682023-11-20T18:31:34.655Z bgp = []
6692023-11-20T18:31:34.683Z+ pfexec curl -sSfL -o /var/svc/manifest/site/tcpproxy.xml http://catacomb.eng.oxide.computer:12346/tcpproxy.xml
6702023-11-20T18:31:34.711Z@@ -101,9 +101,9 @@
6712023-11-20T18:31:34.736Z+ pfexec svccfg import /var/svc/manifest/site/tcpproxy.xml
6722023-11-20T18:31:34.760Z # You can configure multiple uplinks by repeating the following stanza
6732023-11-20T18:31:34.798Z+ OMICRON_NO_UNINSTALL=1
6742023-11-20T18:31:34.822Z [[rack_network_config.ports]]
6752023-11-20T18:31:34.850Z+ ptime -m pfexec ./target/release/omicron-package -t test install
6762023-11-20T18:31:34.875Z # Routes associated with this port.
6772023-11-20T18:31:34.901ZNov 20 18:31:29.340 DEBG target[test]: Target({"image": "standard", "machine": "non-gimlet", "rack-topology": "single-sled", "switch": "softnpu"})
6782023-11-20T18:31:34.926Z-routes = [{nexthop = "192.168.1.199", destination = "0.0.0.0/0"}]
6792023-11-20T18:31:34.950ZNov 20 18:31:29.342 INFO Installing service, dst: /opt/oxide/clickhouse.tar.gz, src: out/clickhouse.tar.gz
6802023-11-20T18:31:34.974Z+routes = [{nexthop = "10.151.2.100", destination = "0.0.0.0/0"}]
6812023-11-20T18:31:34.999ZNov 20 18:31:29.342 INFO Installing service, dst: /opt/oxide/clickhouse_keeper.tar.gz, src: out/clickhouse_keeper.tar.gz
6822023-11-20T18:31:35.030Z # Addresses associated with this port.
6832023-11-20T18:31:35.060ZNov 20 18:31:29.342 INFO Installing service, dst: /opt/oxide/cockroachdb.tar.gz, src: out/cockroachdb.tar.gz
6842023-11-20T18:31:35.088Z-addresses = ["192.168.1.30/32"]
6852023-11-20T18:31:35.117ZNov 20 18:31:29.342 INFO Installing service, dst: /opt/oxide/crucible_pantry.tar.gz, src: out/crucible-pantry.tar.gz
6862023-11-20T18:31:35.146Z+addresses = ["10.151.2.174/32"]
6872023-11-20T18:31:35.177ZNov 20 18:31:29.342 INFO Installing service, dst: /opt/oxide/overlay.tar.gz, src: out/overlay.tar.gz
6882023-11-20T18:31:35.207Z # Name of the uplink port. This should always be "qsfp0" when using softnpu.
6892023-11-20T18:31:35.234ZNov 20 18:31:29.342 INFO Installing service, dst: /opt/oxide/oximeter.tar.gz, src: out/oximeter-collector.tar.gz
6902023-11-20T18:31:35.261Z port = "qsfp0"
6912023-11-20T18:31:35.291ZNov 20 18:31:29.342 INFO Installing service, dst: /opt/oxide/sled-agent.tar, src: out/omicron-sled-agent.tar
6922023-11-20T18:31:35.320Z # The speed of this port.
6932023-11-20T18:31:35.353ZNov 20 18:31:29.342 INFO Installing service, dst: /opt/oxide/switch.tar.gz, src: out/switch-softnpu.tar.gz
6942023-11-20T18:31:35.381Za pkg/config-rss.toml 7K
6952023-11-20T18:31:35.419ZNov 20 18:31:29.342 INFO Installing service, dst: /opt/oxide/ntp.tar.gz, src: out/ntp.tar.gz
6962023-11-20T18:31:35.456Za pkg/initial-tls-cert.pem 2K
6972023-11-20T18:31:35.498ZNov 20 18:31:29.342 INFO Installing service, dst: /opt/oxide/propolis-server.tar.gz, src: out/propolis-server.tar.gz
6982023-11-20T18:31:35.531Za pkg/initial-tls-key.pem 4K
6992023-11-20T18:31:35.567ZNov 20 18:31:29.342 INFO Installing service, dst: /opt/oxide/nexus.tar.gz, src: out/omicron-nexus.tar.gz
7002023-11-20T18:31:35.604ZNov 20 18:31:29.342 INFO Installing service, dst: /opt/oxide/mg-ddm.tar, src: out/mg-ddm-gz.tar
7012023-11-20T18:31:35.643ZNov 20 18:31:29.342 INFO Installing service, dst: /opt/oxide/internal_dns.tar.gz, src: out/internal-dns.tar.gz
7022023-11-20T18:31:35.678ZNov 20 18:31:29.342 INFO Installing service, dst: /opt/oxide/external_dns.tar.gz, src: out/external-dns.tar.gz
7032023-11-20T18:31:35.713ZNov 20 18:31:29.342 INFO Installing service, dst: /opt/oxide/mgs.tar.gz, src: out/omicron-gateway-softnpu.tar.gz
7042023-11-20T18:31:36.179ZNov 20 18:31:29.342 INFO Installing service, dst: /opt/oxide/crucible.tar.gz, src: out/crucible.tar.gz
7052023-11-20T18:31:36.207ZNov 20 18:31:29.596 INFO Unpacking service tarball, service_path: /opt/oxide/mg-ddm, tar_path: /opt/oxide/mg-ddm.tar
7062023-11-20T18:31:36.234ZNov 20 18:31:29.632 INFO Unpacking service tarball, service_path: /opt/oxide/sled-agent, tar_path: /opt/oxide/sled-agent.tar
7072023-11-20T18:31:36.258ZNov 20 18:31:29.690 INFO Installing bootstrap service from /opt/oxide/sled-agent/pkg/manifest.xml
7082023-11-20T18:31:36.283Z
7092023-11-20T18:31:36.316Zreal 0.491143127
7102023-11-20T18:31:36.864Zuser 0.097892908
7112023-11-20T18:31:36.901Zsys 1.993883008
7122023-11-20T18:31:36.932Ztrap 0.000000190
7132023-11-20T18:31:36.968Ztflt 0.000168753
7142023-11-20T18:31:37.000Zdflt 0.001200453
7152023-11-20T18:31:37.047Zkflt 0.000000000
7162023-11-20T18:31:37.478Zlock 44.126234988
7172023-11-20T18:31:37.507Zslp 0.716802566
7182023-11-20T18:31:37.537Zlat 0.003539987
7192023-11-20T18:31:37.982Zstop 0.223412695
7202023-11-20T18:31:38.014Z+ retry=0
7212023-11-20T18:31:38.048Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7222023-11-20T18:31:38.080Z+ [[ 0 -gt 30 ]]
7232023-11-20T18:31:38.115Z+ sleep 1
7242023-11-20T18:31:38.143Z+ retry=1
7252023-11-20T18:31:38.170Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7262023-11-20T18:31:38.612Z+ [[ 1 -gt 30 ]]
7272023-11-20T18:31:38.647Z+ sleep 1
7282023-11-20T18:31:38.686Z+ retry=2
7292023-11-20T18:31:38.816Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7302023-11-20T18:31:38.840Z+ [[ 2 -gt 30 ]]
7312023-11-20T18:31:38.864Z+ sleep 1
7322023-11-20T18:31:38.890Z+ retry=3
7332023-11-20T18:31:38.919Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7342023-11-20T18:31:38.977Z+ [[ 3 -gt 30 ]]
7352023-11-20T18:31:39.002Z+ sleep 1
7362023-11-20T18:31:39.029Z+ retry=4
7372023-11-20T18:31:39.055Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7382023-11-20T18:31:59.689Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone dump-state
7392023-11-20T18:31:59.723Zlocal v6:
7402023-11-20T18:31:59.748Zfe80::80cd:e3ff:fe26:ea67
7412023-11-20T18:31:59.773Zfd00:99::1
7422023-11-20T18:31:59.797Zlocal v4:
7432023-11-20T18:31:59.825Z10.151.2.174
7442023-11-20T18:31:59.850Zrouter v6:
7452023-11-20T18:31:59.885Zfdb0:18c0:4d0d:a02a::/64 -> fe80::fcd4:30ff:fe58:92a (1)
7462023-11-20T18:31:59.913Zfd00:1122:3344:2::/64 -> fe80::fcd4:30ff:fe58:92a (1)
7472023-11-20T18:31:59.938Zfd00:1122:3344:101::/64 -> fe80::fcd4:30ff:fe58:92a (1)
7482023-11-20T18:31:59.962Zfd00:1122:3344:1::/64 -> fe80::fcd4:30ff:fe58:92a (1)
7492023-11-20T18:31:59.987Zfd00:1122:3344:3::/64 -> fe80::fcd4:30ff:fe58:92a (1)
7502023-11-20T18:32:00.012Zrouter v4_idx:
7512023-11-20T18:32:00.039Z0.0.0.0/0 -> 0
7522023-11-20T18:32:00.064Zrouter v4_routes:
7532023-11-20T18:32:00.089Z0 -> 10.151.2.100 (2)
7542023-11-20T18:32:00.113Zresolver v4:
7552023-11-20T18:32:00.138Z10.151.2.100 -> 18:c0:4d:0d:a0:2a
7562023-11-20T18:32:00.168Z10.151.2.174 -> ca:ad:50:0a:85:89
7572023-11-20T18:32:00.201Zresolver v6:
7582023-11-20T18:32:00.230Zfe80::fcd4:30ff:fe58:92a -> fe:d4:30:58:09:2a
7592023-11-20T18:32:00.679Zfe80::80cd:e3ff:fe26:ea67 -> 82:cd:e3:26:ea:67
7602023-11-20T18:32:00.717Znat v4:
7612023-11-20T18:32:00.761Z10.151.2.169 0/16383 -> fd00:1122:3344:101::1 100/a8:40:25:ff:d0:eb
7622023-11-20T18:32:00.787Znat_v6:
7632023-11-20T18:32:00.815Zport_mac:
7642023-11-20T18:32:00.841Z1000: a8:40:25:07:4f:dd
7652023-11-20T18:32:00.866Z1: a8:40:25:07:4f:de
7662023-11-20T18:32:00.890Zproxy arp:
7672023-11-20T18:32:00.914Z10.151.2.164/10.151.2.227: a8:e1:de:01:70:1d
7682023-11-20T18:32:00.942Z+ retry=0
7692023-11-20T18:32:00.972Z+++ zoneadm list -n
7702023-11-20T18:32:00.996Z+++ grep oxz_ntp
7712023-11-20T18:32:01.020Z++ pfexec svcs -z oxz_ntp_3a2950ba-6446-481f-8a0c-fc8b3ecad888 -Hostate oxide/ntp
7722023-11-20T18:32:01.049Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
7732023-11-20T18:32:01.076Z++ true
7742023-11-20T18:32:01.113Z+ [[ '' != online ]]
7752023-11-20T18:32:01.150Z+ [[ 0 -gt 60 ]]
7762023-11-20T18:32:01.182Z+ sleep 1
7772023-11-20T18:32:01.218Z+ retry=1
7782023-11-20T18:32:01.245Z+++ zoneadm list -n
7792023-11-20T18:32:01.271Z+++ grep oxz_ntp
7802023-11-20T18:32:01.297Z++ pfexec svcs -z oxz_ntp_3a2950ba-6446-481f-8a0c-fc8b3ecad888 -Hostate oxide/ntp
7812023-11-20T18:32:01.326Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
7822023-11-20T18:32:01.357Z++ true
7832023-11-20T18:32:01.385Z+ [[ '' != online ]]
7842023-11-20T18:32:01.412Z+ [[ 1 -gt 60 ]]
7852023-11-20T18:32:01.436Z+ sleep 1
7862023-11-20T18:32:01.791Z+ retry=2
7872023-11-20T18:32:01.816Z+++ zoneadm list -n
7882023-11-20T18:32:01.841Z+++ grep oxz_ntp
7892023-11-20T18:32:01.866Z++ pfexec svcs -z oxz_ntp_3a2950ba-6446-481f-8a0c-fc8b3ecad888 -Hostate oxide/ntp
7902023-11-20T18:32:01.890Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
7912023-11-20T18:32:01.915Z++ true
7922023-11-20T18:32:01.942Z+ [[ '' != online ]]
7932023-11-20T18:32:01.968Z+ [[ 2 -gt 60 ]]
7942023-11-20T18:32:01.994Z+ sleep 1
7952023-11-20T18:32:02.821Z+ retry=3
7962023-11-20T18:32:02.846Z+++ zoneadm list -n
7972023-11-20T18:32:02.873Z+++ grep oxz_ntp
7982023-11-20T18:32:02.900Z++ pfexec svcs -z oxz_ntp_3a2950ba-6446-481f-8a0c-fc8b3ecad888 -Hostate oxide/ntp
7992023-11-20T18:32:02.925Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
8002023-11-20T18:32:02.951Z++ true
8012023-11-20T18:32:02.977Z+ [[ '' != online ]]
8022023-11-20T18:32:03.003Z+ [[ 3 -gt 60 ]]
8032023-11-20T18:32:03.033Z+ sleep 1
8042023-11-20T18:32:03.853Z+ retry=4
8052023-11-20T18:32:03.880Z+++ zoneadm list -n
8062023-11-20T18:32:03.905Z+++ grep oxz_ntp
8072023-11-20T18:32:03.930Z++ pfexec svcs -z oxz_ntp_3a2950ba-6446-481f-8a0c-fc8b3ecad888 -Hostate oxide/ntp
8082023-11-20T18:32:03.958Z+ [[ disabled != online ]]
8092023-11-20T18:32:03.983Z+ [[ 4 -gt 60 ]]
8102023-11-20T18:32:04.007Z+ sleep 1
8112023-11-20T18:32:04.888Z+ retry=5
8122023-11-20T18:32:04.913Z+++ zoneadm list -n
8132023-11-20T18:32:04.938Z+++ grep oxz_ntp
8142023-11-20T18:32:04.963Z++ pfexec svcs -z oxz_ntp_3a2950ba-6446-481f-8a0c-fc8b3ecad888 -Hostate oxide/ntp
8152023-11-20T18:32:04.988Z+ [[ disabled != online ]]
8162023-11-20T18:32:05.014Z+ [[ 5 -gt 60 ]]
8172023-11-20T18:32:05.040Z+ sleep 1
8182023-11-20T18:32:05.884Z+ retry=6
8192023-11-20T18:32:05.917Z+++ zoneadm list -n
8202023-11-20T18:32:05.942Z+++ grep oxz_ntp
8212023-11-20T18:32:05.966Z++ pfexec svcs -z oxz_ntp_3a2950ba-6446-481f-8a0c-fc8b3ecad888 -Hostate oxide/ntp
8222023-11-20T18:32:05.991Z+ [[ disabled != online ]]
8232023-11-20T18:32:06.015Z+ [[ 6 -gt 60 ]]
8242023-11-20T18:32:06.041Z+ sleep 1
8252023-11-20T18:32:07.012Z+ retry=7
8262023-11-20T18:32:07.039Z+++ zoneadm list -n
8272023-11-20T18:32:07.066Z+++ grep oxz_ntp
8282023-11-20T18:32:07.092Z++ pfexec svcs -z oxz_ntp_3a2950ba-6446-481f-8a0c-fc8b3ecad888 -Hostate oxide/ntp
8292023-11-20T18:32:07.117Z+ [[ disabled != online ]]
8302023-11-20T18:32:07.141Z+ [[ 7 -gt 60 ]]
8312023-11-20T18:32:07.166Z+ sleep 1
8322023-11-20T18:32:07.930Z+ retry=8
8332023-11-20T18:32:07.954Z+++ zoneadm list -n
8342023-11-20T18:32:07.979Z+++ grep oxz_ntp
8352023-11-20T18:32:08.004Z++ pfexec svcs -z oxz_ntp_3a2950ba-6446-481f-8a0c-fc8b3ecad888 -Hostate oxide/ntp
8362023-11-20T18:32:08.043Z+ [[ disabled != online ]]
8372023-11-20T18:32:08.067Z+ [[ 8 -gt 60 ]]
8382023-11-20T18:32:08.091Z+ sleep 1
8392023-11-20T18:32:08.952Z+ retry=9
8402023-11-20T18:32:08.977Z+++ zoneadm list -n
8412023-11-20T18:32:09.001Z+++ grep oxz_ntp
8422023-11-20T18:32:09.026Z++ pfexec svcs -z oxz_ntp_3a2950ba-6446-481f-8a0c-fc8b3ecad888 -Hostate oxide/ntp
8432023-11-20T18:32:09.070Z+ [[ disabled != online ]]
8442023-11-20T18:32:09.095Z+ [[ 9 -gt 60 ]]
8452023-11-20T18:32:09.119Z+ sleep 1
8462023-11-20T18:32:09.975Z+ retry=10
8472023-11-20T18:32:10.001Z+++ zoneadm list -n
8482023-11-20T18:32:10.025Z+++ grep oxz_ntp
8492023-11-20T18:32:10.049Z++ pfexec svcs -z oxz_ntp_3a2950ba-6446-481f-8a0c-fc8b3ecad888 -Hostate oxide/ntp
8502023-11-20T18:32:10.108Z+ [[ disabled != online ]]
8512023-11-20T18:32:10.139Z+ [[ 10 -gt 60 ]]
8522023-11-20T18:32:10.168Z+ sleep 1
 
8532023-11-20T18:32:10.195Zsock console login: Nov 20 18:32:08 sock ip: ip: joining multicasts failed (4) on vopte0 - will use link layer broadcasts for multicast
 
8542023-11-20T18:32:10.995Z+ retry=11
8552023-11-20T18:32:11.021Z+++ zoneadm list -n
8562023-11-20T18:32:11.045Z+++ grep oxz_ntp
8572023-11-20T18:32:11.071Z++ pfexec svcs -z oxz_ntp_3a2950ba-6446-481f-8a0c-fc8b3ecad888 -Hostate oxide/ntp
8582023-11-20T18:32:11.098Z+ [[ online != online ]]
8592023-11-20T18:32:11.135Z+ echo 'Waited for chrony: 11s'
8602023-11-20T18:32:11.160Z+ export RUST_BACKTRACE=1
8612023-11-20T18:32:11.185Z+ RUST_BACKTRACE=1
8622023-11-20T18:32:11.208ZWaited for chrony: 11s
8632023-11-20T18:32:11.233Z+ export E2E_TLS_CERT IPPOOL_START IPPOOL_END
8642023-11-20T18:32:11.258Z+ ./tests/bootstrap
8652023-11-20T18:32:11.283ZUsing external DNS server at 10.151.2.164:53
8662023-11-20T18:32:11.307Z2023-11-20 18:32:10.409558524 UTC: attempting to log into API
8672023-11-20T18:32:26.083Z2023-11-20 18:32:26.315620599 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8682023-11-20T18:32:27.083Z2023-11-20 18:32:27.316291479 UTC: attempting to log into API
8692023-11-20T18:32:42.117Z2023-11-20 18:32:42.349022951 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8702023-11-20T18:32:43.117Z2023-11-20 18:32:43.349804265 UTC: attempting to log into API
8712023-11-20T18:32:58.150Z2023-11-20 18:32:58.382738882 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8722023-11-20T18:32:59.151Z2023-11-20 18:32:59.383544237 UTC: attempting to log into API
8732023-11-20T18:33:14.183Z2023-11-20 18:33:14.416671921 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8742023-11-20T18:33:15.184Z2023-11-20 18:33:15.417566338 UTC: attempting to log into API
8752023-11-20T18:33:30.216Z2023-11-20 18:33:30.024583782 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8762023-11-20T18:33:31.217Z2023-11-20 18:33:31.028394662 UTC: attempting to log into API
8772023-11-20T18:33:46.250Z2023-11-20 18:33:46.102697580 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8782023-11-20T18:33:47.251Z2023-11-20 18:33:47.106095037 UTC: attempting to log into API
8792023-11-20T18:34:02.286Z2023-11-20 18:34:02.180353179 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8802023-11-20T18:34:03.285Z2023-11-20 18:34:03.183726517 UTC: attempting to log into API
8812023-11-20T18:34:18.344Z2023-11-20 18:34:18.281877348 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8822023-11-20T18:34:19.343Z2023-11-20 18:34:19.285120526 UTC: attempting to log into API
 
8832023-11-20T18:34:20.077ZNov 20 18:34:19 sok ip: ip: joining multicasts failed (4) on vopte4 - will use link layer broadcasts for multicast
8842023-11-20T18:34:27.856ZNov 20 18:34:27 sock ip: ip: joining multicasts failed (4) on vopte2 - will use link layer broadcasts for multicast
 
8852023-11-20T18:34:34.388Z2023-11-20 18:34:34.372218506 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8862023-11-20T18:34:35.396Z2023-11-20 18:34:35.375094089 UTC: attempting to log into API
 
8872023-11-20T18:34:35.884ZNov 20 18:34:35 sok ip: ip: joining multicasts failed (4) on vopte3 - will use link layer broadcasts for multicast
8882023-11-20T18:34:42.667ZNov 20 18:34:42 sock ip: ip: joining multicasts failed (4) on vopte5 - will use link layer broadcasts for multicast
8892023-11-20T18:34:49.882ZNov 20 18:34:48 sock ip: ip: joining multicasts failed (4) on vopte1 - will use link layer broadcasts for multicast
 
8902023-11-20T18:34:50.427Z2023-11-20 18:34:50.409276236 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8912023-11-20T18:34:51.423Z2023-11-20 18:34:51.410551142 UTC: attempting to log into API
8922023-11-20T18:35:06.457Z2023-11-20 18:35:06.445251905 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8932023-11-20T18:35:07.457Z2023-11-20 18:35:07.446495701 UTC: attempting to log into API
8942023-11-20T18:35:22.491Z2023-11-20 18:35:22.481692030 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8952023-11-20T18:35:23.491Z2023-11-20 18:35:23.482778220 UTC: attempting to log into API
8962023-11-20T18:35:29.282Z2023-11-20 18:35:29.272739430 UTC: login succeeded
8972023-11-20T18:35:29.313Zwaiting for nexus to come up...
8982023-11-20T18:35:29.385Zcreating IP pool... 10.151.2.175 - 10.151.2.227
8992023-11-20T18:35:29.436Zensuring datasets are ready...
9002023-11-20T18:35:31.907Zlet's roll.
9012023-11-20T18:35:31.933Z+ rm ./tests/bootstrap
9022023-11-20T18:35:31.981Z+ for test_bin in tests/*
9032023-11-20T18:35:32.006Z+ ./tests/bootstrap-7746be776f1918b3
9042023-11-20T18:35:32.030Z
9052023-11-20T18:35:32.055Zrunning 0 tests
9062023-11-20T18:35:32.079Z
9072023-11-20T18:35:32.122Ztest result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
9082023-11-20T18:35:32.147Z
9092023-11-20T18:35:32.176Z+ for test_bin in tests/*
9102023-11-20T18:35:32.202Z+ ./tests/end_to_end_tests-cb4cc51c87b08c71
9112023-11-20T18:35:32.227Z
9122023-11-20T18:35:32.252Zrunning 2 tests
9132023-11-20T18:35:32.276Ztest no_spoof::no_spoof_here ... ok
9142023-11-20T18:36:32.010Ztest instance_launch::instance_launch has been running for over 60 seconds
9152023-11-20T18:38:30.174Ztest instance_launch::instance_launch ... ok
9162023-11-20T18:38:30.198Z
9172023-11-20T18:38:30.223Ztest result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 178.17s
9182023-11-20T18:38:30.247Z
9192023-11-20T18:38:30.271Z+ _exit_trap
9202023-11-20T18:38:30.296Z+ local status=0
9212023-11-20T18:38:30.320Z+ [[ 0 -eq 0 ]]
9222023-11-20T18:38:30.345Z+ exit 0
9232023-11-20T18:38:30.369Zprocess exited: duration 463990 ms, exit code 0
 
9242023-11-20T18:38:30.450Zfound 47 output files
9252023-11-20T18:38:30.520Zuploading: /var/svc/log/oxide-sled-agent:default.log (682263 bytes)
9262023-11-20T18:38:32.273Zuploaded: /var/svc/log/oxide-sled-agent:default.log
9272023-11-20T18:38:32.298Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_5cc21d68-9aae-4ae1-921d-cc0b737f5e76/root/var/svc/log/oxide-cockroachdb:default.log (5368 bytes)
9282023-11-20T18:38:33.369Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_5cc21d68-9aae-4ae1-921d-cc0b737f5e76/root/var/svc/log/oxide-cockroachdb:default.log
9292023-11-20T18:38:33.395Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_7fbc3a71-a6c3-403a-8df2-5102c0b01425/root/var/svc/log/oxide-crucible-agent:default.log (14116 bytes)
9302023-11-20T18:38:34.490Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_7fbc3a71-a6c3-403a-8df2-5102c0b01425/root/var/svc/log/oxide-crucible-agent:default.log
9312023-11-20T18:38:34.514Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_7fbc3a71-a6c3-403a-8df2-5102c0b01425/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
9322023-11-20T18:38:35.592Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_7fbc3a71-a6c3-403a-8df2-5102c0b01425/root/var/svc/log/oxide-crucible-downstairs:default.log
9332023-11-20T18:38:35.622Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_7fbc3a71-a6c3-403a-8df2-5102c0b01425/root/var/svc/log/oxide-crucible-downstairs:downstairs-4b463056-7c55-4f32-89f0-0c9f9a7a609a.log (5536 bytes)
9342023-11-20T18:38:36.695Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_7fbc3a71-a6c3-403a-8df2-5102c0b01425/root/var/svc/log/oxide-crucible-downstairs:downstairs-4b463056-7c55-4f32-89f0-0c9f9a7a609a.log
9352023-11-20T18:38:36.719Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_2b97c09b-2b28-4b96-b201-2eb1418f2750/root/var/svc/log/oxide-crucible-agent:default.log (3435 bytes)
9362023-11-20T18:38:37.791Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_2b97c09b-2b28-4b96-b201-2eb1418f2750/root/var/svc/log/oxide-crucible-agent:default.log
9372023-11-20T18:38:37.816Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_2b97c09b-2b28-4b96-b201-2eb1418f2750/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
9382023-11-20T18:38:38.887Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_2b97c09b-2b28-4b96-b201-2eb1418f2750/root/var/svc/log/oxide-crucible-downstairs:default.log
9392023-11-20T18:38:38.911Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_98405858-f953-4ba4-8736-ad87e933650c/root/var/svc/log/oxide-crucible-agent:default.log (3435 bytes)
9402023-11-20T18:38:39.983Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_98405858-f953-4ba4-8736-ad87e933650c/root/var/svc/log/oxide-crucible-agent:default.log
9412023-11-20T18:38:40.007Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_98405858-f953-4ba4-8736-ad87e933650c/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
9422023-11-20T18:38:41.090Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_98405858-f953-4ba4-8736-ad87e933650c/root/var/svc/log/oxide-crucible-downstairs:default.log
9432023-11-20T18:38:41.117Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_f8bca242-9a19-4eac-bf1c-8cee00b11a6e/root/var/svc/log/oxide-crucible-agent:default.log (14115 bytes)
9442023-11-20T18:38:42.206Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_f8bca242-9a19-4eac-bf1c-8cee00b11a6e/root/var/svc/log/oxide-crucible-agent:default.log
9452023-11-20T18:38:42.231Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_f8bca242-9a19-4eac-bf1c-8cee00b11a6e/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
9462023-11-20T18:38:43.302Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_f8bca242-9a19-4eac-bf1c-8cee00b11a6e/root/var/svc/log/oxide-crucible-downstairs:default.log
9472023-11-20T18:38:43.327Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_f8bca242-9a19-4eac-bf1c-8cee00b11a6e/root/var/svc/log/oxide-crucible-downstairs:downstairs-5a43f9b4-c9af-4bad-91d1-4a6bb1ad7c37.log (5536 bytes)
9482023-11-20T18:38:44.412Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_f8bca242-9a19-4eac-bf1c-8cee00b11a6e/root/var/svc/log/oxide-crucible-downstairs:downstairs-5a43f9b4-c9af-4bad-91d1-4a6bb1ad7c37.log
9492023-11-20T18:38:44.439Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_oximeter_ef87913b-854f-402f-aa14-306af3093cc8/root/var/svc/log/oxide-oximeter:default.log (193404 bytes)
9502023-11-20T18:38:44.628Zupload warning: file "/pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_oximeter_ef87913b-854f-402f-aa14-306af3093cc8/root/var/svc/log/oxide-oximeter:default.log" changed size mid upload: 193404 -> 207664
9512023-11-20T18:38:45.705Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_oximeter_ef87913b-854f-402f-aa14-306af3093cc8/root/var/svc/log/oxide-oximeter:default.log
9522023-11-20T18:38:45.733Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_clickhouse_3512f599-0566-45f3-8d51-20115641fb4a/root/var/svc/log/oxide-clickhouse:default.log (708613 bytes)
9532023-11-20T18:38:46.087Zupload warning: file "/pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_clickhouse_3512f599-0566-45f3-8d51-20115641fb4a/root/var/svc/log/oxide-clickhouse:default.log" changed size mid upload: 708613 -> 774650
9542023-11-20T18:38:47.571Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_clickhouse_3512f599-0566-45f3-8d51-20115641fb4a/root/var/svc/log/oxide-clickhouse:default.log
9552023-11-20T18:38:47.599Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_cockroachdb_24165f27-dc65-47f6-9172-4e1840df0506/root/var/svc/log/oxide-cockroachdb:default.log (5343 bytes)
9562023-11-20T18:38:48.260Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_cockroachdb_24165f27-dc65-47f6-9172-4e1840df0506/root/var/svc/log/oxide-cockroachdb:default.log
9572023-11-20T18:38:48.285Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_cockroachdb_5c85f226-e6df-414a-9adf-e0991901e6d2/root/var/svc/log/oxide-cockroachdb:default.log (5369 bytes)
9582023-11-20T18:38:49.357Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_cockroachdb_5c85f226-e6df-414a-9adf-e0991901e6d2/root/var/svc/log/oxide-cockroachdb:default.log
9592023-11-20T18:38:49.385Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_pantry_94f92b33-8220-4de0-a5d6-1088da160494/root/var/svc/log/oxide-crucible-pantry:default.log (2337 bytes)
9602023-11-20T18:38:50.466Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_pantry_94f92b33-8220-4de0-a5d6-1088da160494/root/var/svc/log/oxide-crucible-pantry:default.log
9612023-11-20T18:38:50.494Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_nexus_dd2c99d2-4b96-4f7f-a65c-53cbf77e6131/root/var/svc/log/oxide-nexus:default.log (699530 bytes)
9622023-11-20T18:38:51.166Zupload warning: file "/pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_nexus_dd2c99d2-4b96-4f7f-a65c-53cbf77e6131/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 699530 -> 763085
9632023-11-20T18:38:52.241Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_nexus_dd2c99d2-4b96-4f7f-a65c-53cbf77e6131/root/var/svc/log/oxide-nexus:default.log
9642023-11-20T18:38:52.265Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_6a172a2f-c9e6-40cb-a03e-05b522149e7b/root/var/svc/log/oxide-crucible-agent:default.log (14113 bytes)
9652023-11-20T18:38:53.359Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_6a172a2f-c9e6-40cb-a03e-05b522149e7b/root/var/svc/log/oxide-crucible-agent:default.log
9662023-11-20T18:38:53.384Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_6a172a2f-c9e6-40cb-a03e-05b522149e7b/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
9672023-11-20T18:38:54.840Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_6a172a2f-c9e6-40cb-a03e-05b522149e7b/root/var/svc/log/oxide-crucible-downstairs:default.log
9682023-11-20T18:38:54.871Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_6a172a2f-c9e6-40cb-a03e-05b522149e7b/root/var/svc/log/oxide-crucible-downstairs:downstairs-b38a20ab-1b4f-4dbb-afe4-9331585587d3.log (3048 bytes)
9692023-11-20T18:38:55.937Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_6a172a2f-c9e6-40cb-a03e-05b522149e7b/root/var/svc/log/oxide-crucible-downstairs:downstairs-b38a20ab-1b4f-4dbb-afe4-9331585587d3.log
9702023-11-20T18:38:55.978Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_internal_dns_026f940f-a6ed-4067-ab96-c82d10c8912f/root/var/svc/log/oxide-internal_dns:default.log (103543 bytes)
9712023-11-20T18:38:56.135Zupload warning: file "/pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_internal_dns_026f940f-a6ed-4067-ab96-c82d10c8912f/root/var/svc/log/oxide-internal_dns:default.log" changed size mid upload: 103543 -> 107300
9722023-11-20T18:38:57.207Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_internal_dns_026f940f-a6ed-4067-ab96-c82d10c8912f/root/var/svc/log/oxide-internal_dns:default.log
9732023-11-20T18:38:57.231Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_internal_dns_aa99d44c-f04b-4cbe-a6c3-8f4b622eef0b/root/var/svc/log/oxide-internal_dns:default.log (99068 bytes)
9742023-11-20T18:38:57.371Zupload warning: file "/pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_internal_dns_aa99d44c-f04b-4cbe-a6c3-8f4b622eef0b/root/var/svc/log/oxide-internal_dns:default.log" changed size mid upload: 99068 -> 102824
9752023-11-20T18:38:58.445Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_internal_dns_aa99d44c-f04b-4cbe-a6c3-8f4b622eef0b/root/var/svc/log/oxide-internal_dns:default.log
9762023-11-20T18:38:58.469Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_internal_dns_d8d0a761-04f3-434c-abd3-76198e07b6e7/root/var/svc/log/oxide-internal_dns:default.log (101115 bytes)
9772023-11-20T18:38:58.610Zupload warning: file "/pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_internal_dns_d8d0a761-04f3-434c-abd3-76198e07b6e7/root/var/svc/log/oxide-internal_dns:default.log" changed size mid upload: 101115 -> 104870
9782023-11-20T18:38:59.681Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_internal_dns_d8d0a761-04f3-434c-abd3-76198e07b6e7/root/var/svc/log/oxide-internal_dns:default.log
9792023-11-20T18:38:59.707Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_cockroachdb_9206f566-4359-483d-89c8-b54369adb707/root/var/svc/log/oxide-cockroachdb:default.log (5366 bytes)
9802023-11-20T18:39:00.778Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_cockroachdb_9206f566-4359-483d-89c8-b54369adb707/root/var/svc/log/oxide-cockroachdb:default.log
9812023-11-20T18:39:00.802Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_external_dns_337eda60-7725-445c-a86d-8d8fe97b32c5/root/var/svc/log/oxide-external_dns:default.log (15613 bytes)
9822023-11-20T18:39:00.872Zupload warning: file "/pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_external_dns_337eda60-7725-445c-a86d-8d8fe97b32c5/root/var/svc/log/oxide-external_dns:default.log" changed size mid upload: 15613 -> 19391
9832023-11-20T18:39:01.956Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_external_dns_337eda60-7725-445c-a86d-8d8fe97b32c5/root/var/svc/log/oxide-external_dns:default.log
9842023-11-20T18:39:01.984Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_external_dns_f333d179-3898-43ed-9387-221edf85d328/root/var/svc/log/oxide-external_dns:default.log (15612 bytes)
9852023-11-20T18:39:02.053Zupload warning: file "/pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_external_dns_f333d179-3898-43ed-9387-221edf85d328/root/var/svc/log/oxide-external_dns:default.log" changed size mid upload: 15612 -> 19391
9862023-11-20T18:39:03.125Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_external_dns_f333d179-3898-43ed-9387-221edf85d328/root/var/svc/log/oxide-external_dns:default.log
9872023-11-20T18:39:03.150Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_cockroachdb_71362719-ff94-4ef0-9bf6-47056c8c4dee/root/var/svc/log/oxide-cockroachdb:default.log (5368 bytes)
9882023-11-20T18:39:04.234Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_cockroachdb_71362719-ff94-4ef0-9bf6-47056c8c4dee/root/var/svc/log/oxide-cockroachdb:default.log
9892023-11-20T18:39:04.264Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_3e671fa3-d6e8-474e-894d-08b116608a9e/root/var/svc/log/oxide-crucible-agent:default.log (14117 bytes)
9902023-11-20T18:39:05.389Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_3e671fa3-d6e8-474e-894d-08b116608a9e/root/var/svc/log/oxide-crucible-agent:default.log
9912023-11-20T18:39:05.414Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_3e671fa3-d6e8-474e-894d-08b116608a9e/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
9922023-11-20T18:39:06.490Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_3e671fa3-d6e8-474e-894d-08b116608a9e/root/var/svc/log/oxide-crucible-downstairs:default.log
9932023-11-20T18:39:06.515Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_3e671fa3-d6e8-474e-894d-08b116608a9e/root/var/svc/log/oxide-crucible-downstairs:downstairs-3757af4f-0934-49e5-9e48-d3b7e67e3d19.log (5535 bytes)
9942023-11-20T18:39:07.586Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_3e671fa3-d6e8-474e-894d-08b116608a9e/root/var/svc/log/oxide-crucible-downstairs:downstairs-3757af4f-0934-49e5-9e48-d3b7e67e3d19.log
9952023-11-20T18:39:07.610Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_08919c66-60fb-446d-8dcb-42a558825836/root/var/svc/log/oxide-crucible-agent:default.log (14113 bytes)
9962023-11-20T18:39:08.707Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_08919c66-60fb-446d-8dcb-42a558825836/root/var/svc/log/oxide-crucible-agent:default.log
9972023-11-20T18:39:08.731Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_08919c66-60fb-446d-8dcb-42a558825836/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
9982023-11-20T18:39:09.802Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_08919c66-60fb-446d-8dcb-42a558825836/root/var/svc/log/oxide-crucible-downstairs:default.log
9992023-11-20T18:39:09.826Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_08919c66-60fb-446d-8dcb-42a558825836/root/var/svc/log/oxide-crucible-downstairs:downstairs-6285b911-1480-4de9-89cb-79dce5a2a409.log (3048 bytes)
10002023-11-20T18:39:10.896Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_08919c66-60fb-446d-8dcb-42a558825836/root/var/svc/log/oxide-crucible-downstairs:downstairs-6285b911-1480-4de9-89cb-79dce5a2a409.log
10012023-11-20T18:39:10.921Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_pantry_af1b6949-eeee-4f7a-8935-fff427e97d8a/root/var/svc/log/oxide-crucible-pantry:default.log (2338 bytes)
10022023-11-20T18:39:11.992Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_pantry_af1b6949-eeee-4f7a-8935-fff427e97d8a/root/var/svc/log/oxide-crucible-pantry:default.log
10032023-11-20T18:39:12.032Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_nexus_be34c8bc-6bc0-479a-a128-8f11bef026aa/root/var/svc/log/oxide-nexus:default.log (4434034 bytes)
10042023-11-20T18:39:13.273Zupload warning: file "/pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_nexus_be34c8bc-6bc0-479a-a128-8f11bef026aa/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 4434034 -> 4498198
10052023-11-20T18:39:14.344Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_nexus_be34c8bc-6bc0-479a-a128-8f11bef026aa/root/var/svc/log/oxide-nexus:default.log
10062023-11-20T18:39:14.379Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_nexus_bfc4c023-fabc-4307-b4c4-7b3cf96df3f6/root/var/svc/log/oxide-nexus:default.log (1037822 bytes)
10072023-11-20T18:39:15.065Zupload warning: file "/pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_nexus_bfc4c023-fabc-4307-b4c4-7b3cf96df3f6/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 1037822 -> 1107332
10082023-11-20T18:39:16.139Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_nexus_bfc4c023-fabc-4307-b4c4-7b3cf96df3f6/root/var/svc/log/oxide-nexus:default.log
10092023-11-20T18:39:16.164Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_f52131f4-5838-4276-aa0a-0f716cfaa097/root/var/svc/log/oxide-crucible-agent:default.log (3436 bytes)
10102023-11-20T18:39:17.251Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_f52131f4-5838-4276-aa0a-0f716cfaa097/root/var/svc/log/oxide-crucible-agent:default.log
10112023-11-20T18:39:17.276Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_f52131f4-5838-4276-aa0a-0f716cfaa097/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10122023-11-20T18:39:18.352Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_f52131f4-5838-4276-aa0a-0f716cfaa097/root/var/svc/log/oxide-crucible-downstairs:default.log
10132023-11-20T18:39:18.379Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_f2335ef7-d3af-47e0-92b1-5c8e59295e51/root/var/svc/log/oxide-crucible-pantry:default.log (2333 bytes)
10142023-11-20T18:39:19.451Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_f2335ef7-d3af-47e0-92b1-5c8e59295e51/root/var/svc/log/oxide-crucible-pantry:default.log
10152023-11-20T18:39:19.475Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_3a2950ba-6446-481f-8a0c-fc8b3ecad888/root/var/svc/log/oxide-ntp:default.log (1176 bytes)
10162023-11-20T18:39:20.546Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_3a2950ba-6446-481f-8a0c-fc8b3ecad888/root/var/svc/log/oxide-ntp:default.log
10172023-11-20T18:39:20.570Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_73bcf5c2-0985-4958-8f6f-dbbbaabe6a83/root/var/svc/log/oxide-crucible-agent:default.log (14117 bytes)
10182023-11-20T18:39:21.669Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_73bcf5c2-0985-4958-8f6f-dbbbaabe6a83/root/var/svc/log/oxide-crucible-agent:default.log
10192023-11-20T18:39:21.695Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_73bcf5c2-0985-4958-8f6f-dbbbaabe6a83/root/var/svc/log/oxide-crucible-downstairs:default.log (75 bytes)
10202023-11-20T18:39:22.771Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_73bcf5c2-0985-4958-8f6f-dbbbaabe6a83/root/var/svc/log/oxide-crucible-downstairs:default.log
10212023-11-20T18:39:22.797Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_73bcf5c2-0985-4958-8f6f-dbbbaabe6a83/root/var/svc/log/oxide-crucible-downstairs:downstairs-94057796-fc49-4861-8686-584325da7b18.log (3051 bytes)
10222023-11-20T18:39:23.869Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_73bcf5c2-0985-4958-8f6f-dbbbaabe6a83/root/var/svc/log/oxide-crucible-downstairs:downstairs-94057796-fc49-4861-8686-584325da7b18.log
10232023-11-20T18:39:23.893Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_3a2950ba-6446-481f-8a0c-fc8b3ecad888/root/var/log/chrony/measurements.log (13563 bytes)
10242023-11-20T18:39:23.940Zupload warning: file "/pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_3a2950ba-6446-481f-8a0c-fc8b3ecad888/root/var/log/chrony/measurements.log" changed size mid upload: 13563 -> 15070
10252023-11-20T18:39:25.048Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_3a2950ba-6446-481f-8a0c-fc8b3ecad888/root/var/log/chrony/measurements.log
10262023-11-20T18:39:25.073Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_3a2950ba-6446-481f-8a0c-fc8b3ecad888/root/var/log/chrony/statistics.log (7488 bytes)
10272023-11-20T18:39:25.124Zupload warning: file "/pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_3a2950ba-6446-481f-8a0c-fc8b3ecad888/root/var/log/chrony/statistics.log" changed size mid upload: 7488 -> 8658
10282023-11-20T18:39:26.599Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_3a2950ba-6446-481f-8a0c-fc8b3ecad888/root/var/log/chrony/statistics.log
10292023-11-20T18:39:26.623Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_3a2950ba-6446-481f-8a0c-fc8b3ecad888/root/var/log/chrony/tracking.log (1721 bytes)
10302023-11-20T18:39:26.648Zupload warning: file "/pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_3a2950ba-6446-481f-8a0c-fc8b3ecad888/root/var/log/chrony/tracking.log" changed size mid upload: 1721 -> 1853
10312023-11-20T18:39:27.718Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_ntp_3a2950ba-6446-481f-8a0c-fc8b3ecad888/root/var/log/chrony/tracking.log