01HE753E1R2C5MKQTGVGWPAN2X: helios / deploy

BasicConfig {
    output_rules: [
        "%/var/svc/log/oxide-sled-agent:default.log*",
        "%/pool/ext/*/crypt/zone/oxz_*/root/var/svc/log/oxide-*.log*",
        "%/pool/ext/*/crypt/zone/oxz_*/root/var/svc/log/system-illumos-*.log*",
        "%/pool/ext/*/crypt/zone/oxz_ntp_*/root/var/log/chrony/*.log*",
        "!/pool/ext/*/crypt/zone/oxz_propolis-server_*/root/var/svc/log/*.log*",
        "%/pool/ext/*/crypt/debug/global/oxide-sled-agent:default.log.*",
        "%/pool/ext/*/crypt/debug/oxz_*/oxide-*.log.*",
        "%/pool/ext/*/crypt/debug/oxz_*/system-illumos-*.log.*",
        "!/pool/ext/*/crypt/debug/oxz_propolis-server_*/*.log.*",
    ],
    rust_toolchain: None,
    target: Some(
        "lab-2.0-opte-0.25",
    ),
    access_repos: [],
    publish: [],
    skip_clone: true,
}

Buildomat Job: 01HE75451H8K369KEX4A48VJQY

Tags:

Artefacts:

Output:

SEQ GLOBAL TIME DETAILS
12023-11-02T05:33:32.020Zjob dependencies complete; ready to run (waiting for 56 m 26 s)
22023-11-02T05:34:17.413Zjob assigned to worker 01HE78BPDB7V91V0PSHR5APF0P (queued for 45 s)
32023-11-02T05:34:17.548Zhttp://172.20.0.5:9969/ipxe/sock... ok
42023-11-02T05:34:17.599Zhttp://172.20.0.5:9969/os/sock/platform/i86pc/kernel/amd64/unix... ok
52023-11-02T05:34:17.637Zhttp://172.20.0.5:9969/os/sock/platform/i86pc/amd64/boot_archive... ok
62023-11-02T05:34:17.671Zhttp://172.20.0.5:9969/os/sock/platform/i86pc/amd64/boot_archive.hash... ok
72023-11-02T05:34:17.704Zhttp://172.20.0.5:9969/postboot/sock... ok
82023-11-02T05:34:17.737ZwcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U
92023-11-02T05:34:17.771Zcpu0: microcode has been updated from version 0xa001133 to 0xa0011d1
102023-11-02T05:34:17.804ZOxide Helios Version helios-2.0.22241 64-bit
112023-11-02T05:34:17.838ZConfiguring device.
122023-11-02T05:34:17.871Z#################################################
132023-11-02T05:34:17.912Z#### ############# ##
142023-11-02T05:34:17.949Z### ### ############ ##
152023-11-02T05:34:17.983Z## ### # ## ### ## Oxide Computer Company ##
162023-11-02T05:34:18.017Z## ## ## ### # ### ##
172023-11-02T05:34:18.049Z## # ### #### #### This Station Under ##
182023-11-02T05:34:18.083Z### ### #### # ### Computer Control ##
192023-11-02T05:34:18.117Z#### #### ### ## ##
202023-11-02T05:34:18.172Z##################################################
212023-11-02T05:34:18.206ZHostname: sock
222023-11-02T05:34:18.242ZLoading smf(7) service descriptions:
232023-11-02T05:34:19.137Zdownloading input: /input/ci-tools/work/end-to-end-tests/bootstrap-c43fba0fc8ca36e8.gz
242023-11-02T05:34:20.671Zdownloaded input: /input/ci-tools/work/end-to-end-tests/bootstrap-c43fba0fc8ca36e8.gz
252023-11-02T05:34:20.696Zdownloading input: /input/ci-tools/work/end-to-end-tests/bootstrap.gz
262023-11-02T05:34:39.265Zdownloaded input: /input/ci-tools/work/end-to-end-tests/bootstrap.gz
272023-11-02T05:34:39.289Zdownloading input: /input/ci-tools/work/end-to-end-tests/end_to_end_tests-db7e95353d866fe2.gz
282023-11-02T05:34:59.803Zdownloaded input: /input/ci-tools/work/end-to-end-tests/end_to_end_tests-db7e95353d866fe2.gz
292023-11-02T05:34:59.828Zdownloading input: /input/ci-tools/work/caboose-util.gz
302023-11-02T05:35:01.064Zdownloaded input: /input/ci-tools/work/caboose-util.gz
312023-11-02T05:35:01.108Zdownloading input: /input/ci-tools/work/tufaceous.gz
322023-11-02T05:35:04.260Zdownloaded input: /input/ci-tools/work/tufaceous.gz
332023-11-02T05:35:04.285Zdownloading input: /input/package/work/version.txt
342023-11-02T05:35:04.314Zdownloaded input: /input/package/work/version.txt
352023-11-02T05:35:04.344Zdownloading input: /input/package/work/package.tar.gz
362023-11-02T05:35:21.702Zdownloaded input: /input/package/work/package.tar.gz
372023-11-02T05:35:21.734Zdownloading input: /input/package/work/global-zone-packages.tar.gz
382023-11-02T05:35:46.690Zdownloaded input: /input/package/work/global-zone-packages.tar.gz
392023-11-02T05:35:46.728Zdownloading input: /input/package/work/trampoline-global-zone-packages.tar.gz
402023-11-02T05:35:54.928Zdownloaded input: /input/package/work/trampoline-global-zone-packages.tar.gz
412023-11-02T05:35:54.971Zdownloading input: /input/package/work/zones/clickhouse.tar.gz
422023-11-02T05:36:49.574Zdownloaded input: /input/package/work/zones/clickhouse.tar.gz
432023-11-02T05:36:49.607Zdownloading input: /input/package/work/zones/clickhouse_keeper.tar.gz
442023-11-02T05:37:30.653Zdownloaded input: /input/package/work/zones/clickhouse_keeper.tar.gz
452023-11-02T05:37:30.691Zdownloading input: /input/package/work/zones/cockroachdb.tar.gz
462023-11-02T05:37:56.673Zdownloaded input: /input/package/work/zones/cockroachdb.tar.gz
472023-11-02T05:37:56.714Zdownloading input: /input/package/work/zones/crucible-pantry.tar.gz
482023-11-02T05:38:02.351Zdownloaded input: /input/package/work/zones/crucible-pantry.tar.gz
492023-11-02T05:38:02.379Zdownloading input: /input/package/work/zones/crucible.tar.gz
502023-11-02T05:38:12.526Zdownloaded input: /input/package/work/zones/crucible.tar.gz
512023-11-02T05:38:12.562Zdownloading input: /input/package/work/zones/external-dns.tar.gz
522023-11-02T05:38:18.161Zdownloaded input: /input/package/work/zones/external-dns.tar.gz
532023-11-02T05:38:18.186Zdownloading input: /input/package/work/zones/internal-dns.tar.gz
542023-11-02T05:38:23.374Zdownloaded input: /input/package/work/zones/internal-dns.tar.gz
552023-11-02T05:38:23.401Zdownloading input: /input/package/work/zones/ntp.tar.gz
562023-11-02T05:38:23.428Zdownloaded input: /input/package/work/zones/ntp.tar.gz
572023-11-02T05:38:23.458Zdownloading input: /input/package/work/zones/omicron-gateway-asic.tar.gz
582023-11-02T05:38:27.445Zdownloaded input: /input/package/work/zones/omicron-gateway-asic.tar.gz
592023-11-02T05:38:27.478Zdownloading input: /input/package/work/zones/omicron-gateway-softnpu.tar.gz
602023-11-02T05:38:32.662Zdownloaded input: /input/package/work/zones/omicron-gateway-softnpu.tar.gz
612023-11-02T05:38:33.093Zdownloading input: /input/package/work/zones/omicron-nexus-single-sled.tar.gz
622023-11-02T05:38:46.154Zdownloaded input: /input/package/work/zones/omicron-nexus-single-sled.tar.gz
632023-11-02T05:38:46.183Zdownloading input: /input/package/work/zones/omicron-nexus.tar.gz
642023-11-02T05:38:59.363Zdownloaded input: /input/package/work/zones/omicron-nexus.tar.gz
652023-11-02T05:38:59.416Zdownloading input: /input/package/work/zones/overlay.tar.gz
662023-11-02T05:38:59.451Zdownloaded input: /input/package/work/zones/overlay.tar.gz
672023-11-02T05:38:59.475Zdownloading input: /input/package/work/zones/oximeter-collector.tar.gz
682023-11-02T05:39:03.421Zdownloaded input: /input/package/work/zones/oximeter-collector.tar.gz
692023-11-02T05:39:03.455Zdownloading input: /input/package/work/zones/propolis-server.tar.gz
702023-11-02T05:39:15.746Zdownloaded input: /input/package/work/zones/propolis-server.tar.gz
712023-11-02T05:39:15.783Zdownloading input: /input/package/work/zones/switch-asic.tar.gz
722023-11-02T05:40:02.992Zdownloaded input: /input/package/work/zones/switch-asic.tar.gz
732023-11-02T05:40:03.024Zdownloading input: /input/package/work/zones/switch-softnpu.tar.gz
742023-11-02T05:40:33.783Zdownloaded input: /input/package/work/zones/switch-softnpu.tar.gz
 
752023-11-02T05:40:33.808Zstarting task 0: "setup"
762023-11-02T05:40:33.843Z++ uname -s
772023-11-02T05:40:33.870Z+ kern=SunOS
782023-11-02T05:40:33.896Z+ case "$kern" in
792023-11-02T05:40:33.924Z+ groupadd -g 12345 build
802023-11-02T05:40:33.949Z+ useradd -u 12345 -g build -d /home/build -s /bin/bash -c build -P 'Primary Administrator' build
812023-11-02T05:40:35.852Z+ zfs create -o mountpoint=/work rpool/work
822023-11-02T05:40:35.888Z++ awk '$2 == "/home" { print $3 }' /etc/mnttab
832023-11-02T05:40:35.914Z+ home_fs=zfs
842023-11-02T05:40:35.939Z+ [[ zfs == autofs ]]
852023-11-02T05:40:35.964Z+ mkdir -p /home/build
862023-11-02T05:40:35.989Z+ chown build:build /home/build /work
872023-11-02T05:40:37.895Z+ chmod 0700 /home/build /work
882023-11-02T05:40:37.922Zprocess exited: duration 4064 ms, exit code 0
 
892023-11-02T05:40:37.974Zstarting task 1: "authentication"
902023-11-02T05:40:38.047Zprocess exited: duration 46 ms, exit code 0
 
912023-11-02T05:40:38.098Zstarting task 2: "build"
922023-11-02T05:40:38.130Z+ trap _exit_trap EXIT
932023-11-02T05:40:38.155Z+ pfexec mdb -kwe 'secpolicy_fs_linkdir/v 55 48 89 e5 b8 01 00 00 00 5d c3'
942023-11-02T05:40:38.325Zsecpolicy_fs_linkdir: 0x55 = 0x55
952023-11-02T05:40:38.350Zsecpolicy_fs_linkdir+1: 0xb8 = 0x48
962023-11-02T05:40:38.376Zsecpolicy_fs_linkdir+2: 0x1 = 0x89
972023-11-02T05:40:38.406Zsecpolicy_fs_linkdir+3: 0 = 0xe5
982023-11-02T05:40:38.438Zsecpolicy_fs_linkdir+4: 0 = 0xb8
992023-11-02T05:40:38.463Zsecpolicy_fs_linkdir+5: 0 = 0x1
1002023-11-02T05:40:38.489Zsecpolicy_fs_linkdir+6: 0x48 = 0x0
1012023-11-02T05:40:38.517Zsecpolicy_fs_linkdir+7: 0x89 = 0x0
1022023-11-02T05:40:38.542Zsecpolicy_fs_linkdir+8: 0xe5 = 0x0
1032023-11-02T05:40:38.567Zsecpolicy_fs_linkdir+9: 0x48 = 0x5d
1042023-11-02T05:40:38.592Zsecpolicy_fs_linkdir+0xa: 0x83 = 0xc3
1052023-11-02T05:40:38.617Z+ [[ -d /opt/oxide ]]
1062023-11-02T05:40:38.653Z+ pfexec mv /opt/oxide /opt/oxide-underneath
1072023-11-02T05:40:38.678Z+ pfexec mkdir /opt/oxide
1082023-11-02T05:40:38.703Z+ pfexec mount -F tmpfs -O swap /opt/oxide
1092023-11-02T05:40:38.730Z+ [[ -d /opt/oxide-underneath ]]
1102023-11-02T05:40:38.755Z+ cd /opt/oxide-underneath
1112023-11-02T05:40:38.780Z+ pfexec tar ceEp@/f - .
1122023-11-02T05:40:38.804Z+ cd /opt/oxide
1132023-11-02T05:40:38.829Z+ pfexec tar xveEp@/f -
1142023-11-02T05:40:38.855Zx ., 0 bytes, 0 tape blocks
1152023-11-02T05:40:38.881Zx ./opte, 0 bytes, 0 tape blocks
1162023-11-02T05:40:38.906Zx ./opte/bin, 0 bytes, 0 tape blocks
1172023-11-02T05:40:38.931Zx ./opte/bin/opteadm, 50714832 bytes, 99053 tape blocks
1182023-11-02T05:40:38.956Z+ pfexec rm -rf /opt/oxide-underneath
1192023-11-02T05:40:38.981Z+ pfexec /sbin/zfs create -o mountpoint=/zone rpool/zone
1202023-11-02T05:40:39.006Z+ pfexec mkdir /opt/oxide/work
1212023-11-02T05:40:39.032Z+ pfexec chown build:build /opt/oxide/work
1222023-11-02T05:40:39.057Z+ cd /opt/oxide/work
1232023-11-02T05:40:39.081Z+ ptime -m tar xvzf /input/package/work/package.tar.gz
1242023-11-02T05:40:40.100ZDecompressing '/input/package/work/package.tar.gz' with '/usr/bin/gzcat'...
1252023-11-02T05:40:40.127Zx out/mg-ddm-gz.tar, 62974464 bytes, 122997 tape blocks
1262023-11-02T05:40:40.153Zx out/omicron-sled-agent.tar, 123320832 bytes, 240861 tape blocks
1272023-11-02T05:40:40.178Zx out/target/test, 75 bytes, 1 tape blocks
1282023-11-02T05:40:40.204Zx out/npuzone/npuzone, 37381232 bytes, 73011 tape blocks
1292023-11-02T05:40:40.230Zx package-manifest.toml, 22037 bytes, 44 tape blocks
1302023-11-02T05:40:40.255Zx smf/sled-agent/non-gimlet/config.toml, 3097 bytes, 7 tape blocks
1312023-11-02T05:40:40.280Zx target/release/omicron-package, 42650048 bytes, 83301 tape blocks
1322023-11-02T05:40:40.327Zx tools/create_virtual_hardware.sh, 2974 bytes, 6 tape blocks
1332023-11-02T05:40:40.354Zx tools/virtual_hardware.sh, 3408 bytes, 7 tape blocks
1342023-11-02T05:40:40.379Zx tools/scrimlet/create-softnpu-zone.sh, 619 bytes, 2 tape blocks
1352023-11-02T05:40:40.404Zx tools/scrimlet/destroy-softnpu-zone.sh, 195 bytes, 1 tape blocks
1362023-11-02T05:40:40.431Zx tools/scrimlet/softnpu-init.sh, 2148 bytes, 5 tape blocks
1372023-11-02T05:40:40.455Zx tools/scrimlet/softnpu-zone.txt, 330 bytes, 1 tape blocks
1382023-11-02T05:40:40.480Zx tools/scrimlet/softnpu.toml, 171 bytes, 1 tape blocks
1392023-11-02T05:40:40.504Z
1402023-11-02T05:40:40.529Zreal 1.611225736
1412023-11-02T05:40:40.553Zuser 1.453694853
1422023-11-02T05:40:40.578Zsys 0.279964495
1432023-11-02T05:40:40.603Ztrap 0.000071051
1442023-11-02T05:40:40.627Ztflt 0.000000000
1452023-11-02T05:40:40.652Zdflt 0.000000000
1462023-11-02T05:40:40.676Zkflt 0.000000000
1472023-11-02T05:40:40.701Zlock 0.000000000
1482023-11-02T05:40:40.726Zslp 1.449841477
1492023-11-02T05:40:40.750Zlat 0.036613472
1502023-11-02T05:40:40.775Zstop 0.000130803
1512023-11-02T05:40:40.799Z+ cp /input/package/work/zones/clickhouse_keeper.tar.gz /input/package/work/zones/clickhouse.tar.gz /input/package/work/zones/cockroachdb.tar.gz /input/package/work/zones/crucible-pantry.tar.gz /input/package/work/zones/crucible.tar.gz /input/package/work/zones/external-dns.tar.gz /input/package/work/zones/internal-dns.tar.gz /input/package/work/zones/ntp.tar.gz /input/package/work/zones/omicron-gateway-asic.tar.gz /input/package/work/zones/omicron-gateway-softnpu.tar.gz /input/package/work/zones/omicron-nexus-single-sled.tar.gz /input/package/work/zones/omicron-nexus.tar.gz /input/package/work/zones/overlay.tar.gz /input/package/work/zones/oximeter-collector.tar.gz /input/package/work/zones/propolis-server.tar.gz /input/package/work/zones/switch-asic.tar.gz /input/package/work/zones/switch-softnpu.tar.gz out/
1522023-11-02T05:40:41.159Z+ mv out/omicron-nexus-single-sled.tar.gz out/omicron-nexus.tar.gz
1532023-11-02T05:40:41.184Z+ mkdir tests
1542023-11-02T05:40:41.209Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1552023-11-02T05:40:41.234Z+ ptime -m gunzip
1562023-11-02T05:40:41.259Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap-c43fba0fc8ca36e8
1572023-11-02T05:40:41.319Z
1582023-11-02T05:40:41.364Zreal 0.141217826
1592023-11-02T05:40:41.392Zuser 0.129488566
1602023-11-02T05:40:41.417Zsys 0.011480173
1612023-11-02T05:40:41.441Ztrap 0.000012100
1622023-11-02T05:40:41.466Ztflt 0.000000000
1632023-11-02T05:40:41.906Zdflt 0.000000000
1642023-11-02T05:40:41.931Zkflt 0.000000000
1652023-11-02T05:40:41.956Zlock 0.000000000
1662023-11-02T05:40:41.981Zslp 0.000018130
1672023-11-02T05:40:42.006Zlat 0.000023641
1682023-11-02T05:40:42.031Zstop 0.000125392
1692023-11-02T05:40:42.056Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap-c43fba0fc8ca36e8
1702023-11-02T05:40:42.081Z+ chmod a+x tests/bootstrap-c43fba0fc8ca36e8
1712023-11-02T05:40:42.106Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1722023-11-02T05:40:42.131Z+ ptime -m gunzip
1732023-11-02T05:40:42.157Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap
1742023-11-02T05:40:43.621Z
1752023-11-02T05:40:43.647Zreal 2.291023521
1762023-11-02T05:40:43.673Zuser 2.151873641
1772023-11-02T05:40:43.698Zsys 0.138793509
1782023-11-02T05:40:43.724Ztrap 0.000136864
1792023-11-02T05:40:43.749Ztflt 0.000000000
1802023-11-02T05:40:43.775Zdflt 0.000000000
1812023-11-02T05:40:43.799Zkflt 0.000000000
1822023-11-02T05:40:43.824Zlock 0.000000000
1832023-11-02T05:40:43.849Zslp 0.000018750
1842023-11-02T05:40:43.874Zlat 0.000018231
1852023-11-02T05:40:43.947Zstop 0.000118023
1862023-11-02T05:40:44.017Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap
1872023-11-02T05:40:44.086Z+ chmod a+x tests/bootstrap
1882023-11-02T05:40:44.111Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1892023-11-02T05:40:44.136Z+ ptime -m gunzip
1902023-11-02T05:40:44.210Z++ basename /input/ci-tools/work/end-to-end-tests/end_to_end_tests-db7e95353d866fe2
1912023-11-02T05:40:46.255Z
1922023-11-02T05:40:46.280Zreal 2.622353211
1932023-11-02T05:40:46.307Zuser 2.450743930
1942023-11-02T05:40:46.332Zsys 0.171221091
1952023-11-02T05:40:46.357Ztrap 0.000165793
1962023-11-02T05:40:46.384Ztflt 0.000000000
1972023-11-02T05:40:46.409Zdflt 0.000000000
1982023-11-02T05:40:46.434Zkflt 0.000000000
1992023-11-02T05:40:46.459Zlock 0.000000000
2002023-11-02T05:40:46.484Zslp 0.000020800
2012023-11-02T05:40:46.510Zlat 0.000016880
2022023-11-02T05:40:46.534Zstop 0.000123933
2032023-11-02T05:40:46.558Z++ basename /input/ci-tools/work/end-to-end-tests/end_to_end_tests-db7e95353d866fe2
2042023-11-02T05:40:46.627Z+ chmod a+x tests/end_to_end_tests-db7e95353d866fe2
2052023-11-02T05:40:46.651Z+ bmat address ls
2062023-11-02T05:40:46.677ZNAME CIDR FIRST LAST COUNT
2072023-11-02T05:40:46.702Zextra 10.151.2.0/24 10.151.2.164 10.151.2.227 64
2082023-11-02T05:40:46.727Z++ bmat address ls -f extra -Ho first,last,count
2092023-11-02T05:40:46.752Z+ set -- 10.151.2.164 10.151.2.227 64
2102023-11-02T05:40:46.778Z+ EXTRA_IP_START=10.151.2.164
2112023-11-02T05:40:46.802Z+ EXTRA_IP_END=10.151.2.227
2122023-11-02T05:40:46.846Z+ EXTRA_IP_COUNT=64
2132023-11-02T05:40:46.873Z+ (( EXTRA_IP_COUNT >= 32 ))
2142023-11-02T05:40:46.903Z+ EXTRA_IP_BASE=10.151.2
2152023-11-02T05:40:46.928Z+ EXTRA_IP_FOCTET=164
2162023-11-02T05:40:46.953Z+ EXTRA_IP_LOCTET=227
2172023-11-02T05:40:46.978Z+ SERVICE_IP_POOL_START=10.151.2.164
2182023-11-02T05:40:47.004Z+ SERVICE_IP_POOL_END=10.151.2.173
2192023-11-02T05:40:47.029Z+ DNS_IP1=10.151.2.164
2202023-11-02T05:40:47.054Z+ DNS_IP2=10.151.2.165
2212023-11-02T05:40:47.078Z+ UPLINK_IP=10.151.2.174
2222023-11-02T05:40:47.104Z+ IPPOOL_START=10.151.2.175
2232023-11-02T05:40:47.129Z+ IPPOOL_END=10.151.2.227
2242023-11-02T05:40:47.154Z++ ipadm show-addr -po type,addr
2252023-11-02T05:40:47.179Z++ awk '-F[:/]' '$1 == "dhcp" {print $2}'
2262023-11-02T05:40:47.205Z+ GATEWAY_IP=10.151.2.100
2272023-11-02T05:40:47.230Z+ [[ -n 10.151.2.100 ]]
2282023-11-02T05:40:47.254Z+ ping -s 10.151.2.100 56 1
2292023-11-02T05:40:47.280ZPING 10.151.2.100: 56 data bytes
2302023-11-02T05:40:47.306Z64 bytes from 10.151.2.100: icmp_seq=0. time=0.010 ms
2312023-11-02T05:40:47.331Z
2322023-11-02T05:40:47.356Z----10.151.2.100 PING Statistics----
2332023-11-02T05:40:47.382Z1 packets transmitted, 1 packets received, 0% packet loss
2342023-11-02T05:40:47.407Zround-trip (ms) min/avg/max/stddev = 0.010/0.010/0.010/-nan
2352023-11-02T05:40:47.433Z++ arp -an
2362023-11-02T05:40:47.458Z++ awk -vgw=10.151.2.100 '$2 == gw {print $NF}'
2372023-11-02T05:40:47.487Z+ GATEWAY_MAC=18:c0:4d:0d:a0:2a
2382023-11-02T05:40:47.512Z+ routeadm -e ipv4-forwarding -u
2392023-11-02T05:40:47.536Z+ PXA_START=10.151.2.164
2402023-11-02T05:40:47.562Z+ PXA_END=10.151.2.227
2412023-11-02T05:40:47.587Z+ export GATEWAY_IP GATEWAY_MAC PXA_START PXA_END
2422023-11-02T05:40:47.613Z+ pfexec zpool create -f scratch c1t1d0 c2t1d0
2432023-11-02T05:40:47.638Z+ ZPOOL_VDEV_DIR=/scratch
2442023-11-02T05:40:47.664Z+ ptime -m pfexec ./tools/create_virtual_hardware.sh
2452023-11-02T05:40:47.689Z+++ dirname ./tools/create_virtual_hardware.sh
2462023-11-02T05:40:47.713Z++ cd ./tools
2472023-11-02T05:40:47.738Z++ pwd
2482023-11-02T05:40:47.763Z+ SOURCE_DIR=/opt/oxide/work/tools
2492023-11-02T05:40:47.788Z+ OMICRON_TOP=/opt/oxide/work/tools/..
2502023-11-02T05:40:47.835Z+ . /opt/oxide/work/tools/virtual_hardware.sh
2512023-11-02T05:40:47.893Z++ MARKER=/etc/opt/oxide/NO_INSTALL
2522023-11-02T05:40:47.918Z++ [[ -f /etc/opt/oxide/NO_INSTALL ]]
2532023-11-02T05:40:47.943Z++ dladm show-phys -p -o LINK
2542023-11-02T05:40:47.970Z++ head -1
2552023-11-02T05:40:47.995Z+ PHYSICAL_LINK=igb0
2562023-11-02T05:40:48.032Z+ echo 'Using igb0 as physical link'
2572023-11-02T05:40:48.057ZUsing igb0 as physical link
2582023-11-02T05:40:48.086Z+ ensure_run_as_root
2592023-11-02T05:40:48.112Z++ id -u
2602023-11-02T05:40:48.140Z+ [[ 0 -ne 0 ]]
2612023-11-02T05:40:48.169Z+ ensure_zpools
2622023-11-02T05:40:48.201Z+ ZPOOL_TYPES=('oxp_' 'oxi_')
2632023-11-02T05:40:48.228Z+ for ZPOOL_TYPE in "${ZPOOL_TYPES[@]}"
2642023-11-02T05:40:48.253Z+ readarray -t ZPOOLS
2652023-11-02T05:40:48.277Z++ grep '"oxp_' /opt/oxide/work/tools/../smf/sled-agent/non-gimlet/config.toml
2662023-11-02T05:40:48.302Z++ sed 's/[ ",]//g'
2672023-11-02T05:40:48.326Z+ for ZPOOL in "${ZPOOLS[@]}"
2682023-11-02T05:40:48.351Z+ echo 'Zpool: [oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b]'
2692023-11-02T05:40:48.375Z+ VDEV_PATH=/scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
2702023-11-02T05:40:48.400ZZpool: [oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b]
2712023-11-02T05:40:48.424Z+ [[ -f /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
2722023-11-02T05:40:48.448Z+ dd if=/dev/zero of=/scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
2732023-11-02T05:40:48.473Z0+0 records in
2742023-11-02T05:40:48.497Z0+0 records out
2752023-11-02T05:40:48.521ZZFS vdev /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
2762023-11-02T05:40:48.546Z0 bytes (0 iB) transferred in 0.000006 secs (0 iB/sec)
2772023-11-02T05:40:48.575ZZFS zpool oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b exists
2782023-11-02T05:40:48.607Z+ success 'ZFS vdev /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
2792023-11-02T05:40:48.635ZZpool: [oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03]
2802023-11-02T05:40:48.661Z+ set +x
2812023-11-02T05:40:48.686ZZFS vdev /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
2822023-11-02T05:40:48.711Z++ zpool list -o name
2832023-11-02T05:40:48.736ZZFS zpool oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
2842023-11-02T05:40:48.761Z++ grep oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b
2852023-11-02T05:40:48.786ZZpool: [oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03]
2862023-11-02T05:40:48.812Z+ [[ -z '' ]]
2872023-11-02T05:40:48.837ZZFS vdev /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
2882023-11-02T05:40:48.862Z+ zpool create -f oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
2892023-11-02T05:40:48.887ZZFS zpool oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
2902023-11-02T05:40:48.932Z+ success 'ZFS zpool oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
2912023-11-02T05:40:48.959ZZpool: [oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03]
2922023-11-02T05:40:48.984Z+ set +x
2932023-11-02T05:40:49.010ZZFS vdev /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
2942023-11-02T05:40:49.042Z+ for ZPOOL in "${ZPOOLS[@]}"
2952023-11-02T05:40:49.075ZZFS zpool oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
2962023-11-02T05:40:49.100Z+ echo 'Zpool: [oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03]'
2972023-11-02T05:40:49.125ZZpool: [oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03]
2982023-11-02T05:40:49.151Z+ VDEV_PATH=/scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
2992023-11-02T05:40:49.231ZZFS vdev /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
3002023-11-02T05:40:49.262Z+ [[ -f /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3012023-11-02T05:40:49.294ZZFS zpool oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
3022023-11-02T05:40:49.324Z+ dd if=/dev/zero of=/scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3032023-11-02T05:40:49.356ZZpool: [oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930]
3042023-11-02T05:40:49.385Z0+0 records in
3052023-11-02T05:40:49.420ZZFS vdev /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev exists
3062023-11-02T05:40:49.450Z0+0 records out
3072023-11-02T05:40:49.480ZZFS zpool oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 exists
3082023-11-02T05:40:49.510Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3092023-11-02T05:40:49.539ZZpool: [oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60]
3102023-11-02T05:40:49.570Z+ success 'ZFS vdev /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3112023-11-02T05:40:49.599ZZFS vdev /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev exists
3122023-11-02T05:40:49.629Z+ set +x
3132023-11-02T05:40:49.661ZZFS zpool oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 exists
3142023-11-02T05:40:49.690Z++ zpool list -o name
3152023-11-02T05:40:49.723ZZpool: [oxp_31bd71cd-4736-4a12-a387-9b74b050396f]
3162023-11-02T05:40:49.749Z++ grep oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03
3172023-11-02T05:40:49.773ZZFS vdev /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev exists
3182023-11-02T05:40:49.798Z+ [[ -z '' ]]
3192023-11-02T05:40:49.823ZZFS zpool oxp_31bd71cd-4736-4a12-a387-9b74b050396f exists
3202023-11-02T05:40:49.849Z+ zpool create -f oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3212023-11-02T05:40:49.874ZZpool: [oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7]
3222023-11-02T05:40:49.903Z+ success 'ZFS zpool oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3232023-11-02T05:40:49.928ZZFS vdev /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev exists
3242023-11-02T05:40:49.986Z+ set +x
3252023-11-02T05:40:50.013ZZFS zpool oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 exists
3262023-11-02T05:40:50.038Z+ for ZPOOL in "${ZPOOLS[@]}"
3272023-11-02T05:40:50.062ZZpool: [oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b]
3282023-11-02T05:40:50.089Z+ echo 'Zpool: [oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3292023-11-02T05:40:50.114ZZFS vdev /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
3302023-11-02T05:40:50.146Z+ VDEV_PATH=/scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3312023-11-02T05:40:50.175ZZFS zpool oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b exists
3322023-11-02T05:40:50.200Z+ [[ -f /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3332023-11-02T05:40:50.226ZZpool: [oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b]
3342023-11-02T05:40:50.251Z+ dd if=/dev/zero of=/scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3352023-11-02T05:40:50.277ZZFS vdev /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
3362023-11-02T05:40:50.312Z0+0 records in
3372023-11-02T05:40:50.338ZZFS zpool oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b exists
3382023-11-02T05:40:50.363Z0+0 records out
3392023-11-02T05:40:50.388ZSimnet net0/sc0_0 exists
3402023-11-02T05:40:50.414Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3412023-11-02T05:40:50.440ZSimnet net1/sc1_0 exists
3422023-11-02T05:40:50.464Z+ success 'ZFS vdev /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3432023-11-02T05:40:50.489ZVnic sc0_1 exists
3442023-11-02T05:40:50.517Z+ set +x
3452023-11-02T05:40:50.542Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/softnpu/image/c1c42398c82b0220c8b5fa3bfba9c7a3bcaa0943/softnpu.sha256.txt
3462023-11-02T05:40:50.571Z++ zpool list -o name
3472023-11-02T05:40:50.608Z++ grep oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03
3482023-11-02T05:40:50.643Z+ [[ -z '' ]]
3492023-11-02T05:40:50.669Z+ zpool create -f oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3502023-11-02T05:40:50.694Z+ success 'ZFS zpool oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3512023-11-02T05:40:50.719Z+ set +x
3522023-11-02T05:40:50.744Z+ for ZPOOL in "${ZPOOLS[@]}"
3532023-11-02T05:40:50.770Z+ echo 'Zpool: [oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3542023-11-02T05:40:50.795Z+ VDEV_PATH=/scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3552023-11-02T05:40:50.821Z+ [[ -f /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3562023-11-02T05:40:50.845Z+ dd if=/dev/zero of=/scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3572023-11-02T05:40:50.871Z0+0 records in
3582023-11-02T05:40:50.898Z0+0 records out
3592023-11-02T05:40:50.924Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3602023-11-02T05:40:50.949Z+ success 'ZFS vdev /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3612023-11-02T05:40:50.974Z+ set +x
3622023-11-02T05:40:51.023Z++ zpool list -o name
3632023-11-02T05:40:51.061Z++ grep oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03
3642023-11-02T05:40:51.533Z+ [[ -z '' ]]
3652023-11-02T05:40:51.559Z+ zpool create -f oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3662023-11-02T05:40:51.583Z+ success 'ZFS zpool oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3672023-11-02T05:40:51.608Z+ set +x
3682023-11-02T05:40:51.632Z+ for ZPOOL in "${ZPOOLS[@]}"
3692023-11-02T05:40:51.656Z+ echo 'Zpool: [oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3702023-11-02T05:40:51.682Z+ VDEV_PATH=/scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3712023-11-02T05:40:51.707Z+ [[ -f /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3722023-11-02T05:40:51.731Z+ dd if=/dev/zero of=/scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3732023-11-02T05:40:51.755Z0+0 records in
3742023-11-02T05:40:51.781Z0+0 records out
3752023-11-02T05:40:51.806Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
3762023-11-02T05:40:51.831Z+ success 'ZFS vdev /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3772023-11-02T05:40:51.855Z+ set +x
3782023-11-02T05:40:51.879Z++ zpool list -o name
3792023-11-02T05:40:51.903Z++ grep oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03
3802023-11-02T05:40:51.928Z+ [[ -z '' ]]
3812023-11-02T05:40:51.959Z+ zpool create -f oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3822023-11-02T05:40:52.429Z+ success 'ZFS zpool oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3832023-11-02T05:40:52.459Z+ set +x
3842023-11-02T05:40:52.483Z+ for ZPOOL in "${ZPOOLS[@]}"
3852023-11-02T05:40:52.508Z+ echo 'Zpool: [oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930]'
3862023-11-02T05:40:52.532Z+ VDEV_PATH=/scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev
3872023-11-02T05:40:52.556Z+ [[ -f /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev ]]
3882023-11-02T05:40:52.581Z+ dd if=/dev/zero of=/scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev bs=1 count=0 seek=20G
3892023-11-02T05:40:52.605Z0+0 records in
3902023-11-02T05:40:52.630Z0+0 records out
3912023-11-02T05:40:52.654Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3922023-11-02T05:40:52.678Z+ success 'ZFS vdev /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev exists'
3932023-11-02T05:40:52.703Z+ set +x
3942023-11-02T05:40:52.727Z++ zpool list -o name
3952023-11-02T05:40:52.751Z++ grep oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930
3962023-11-02T05:40:52.776Z+ [[ -z '' ]]
3972023-11-02T05:40:52.800Z+ zpool create -f oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev
3982023-11-02T05:40:52.827Z+ success 'ZFS zpool oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 exists'
3992023-11-02T05:40:53.266Z+ set +x
4002023-11-02T05:40:53.290Z+ for ZPOOL in "${ZPOOLS[@]}"
4012023-11-02T05:40:53.315Z+ echo 'Zpool: [oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60]'
4022023-11-02T05:40:53.341Z+ VDEV_PATH=/scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev
4032023-11-02T05:40:53.367Z+ [[ -f /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev ]]
4042023-11-02T05:40:53.392Z+ dd if=/dev/zero of=/scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev bs=1 count=0 seek=20G
4052023-11-02T05:40:53.416Z0+0 records in
4062023-11-02T05:40:53.442Z0+0 records out
4072023-11-02T05:40:53.467Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4082023-11-02T05:40:53.491Z+ success 'ZFS vdev /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev exists'
4092023-11-02T05:40:53.517Z+ set +x
4102023-11-02T05:40:53.541Z++ zpool list -o name
4112023-11-02T05:40:53.567Z++ grep oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60
4122023-11-02T05:40:53.592Z+ [[ -z '' ]]
4132023-11-02T05:40:53.617Z+ zpool create -f oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev
4142023-11-02T05:40:53.642Z+ success 'ZFS zpool oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 exists'
4152023-11-02T05:40:53.724Z+ set +x
4162023-11-02T05:40:53.756Z+ for ZPOOL in "${ZPOOLS[@]}"
4172023-11-02T05:40:53.784Z+ echo 'Zpool: [oxp_31bd71cd-4736-4a12-a387-9b74b050396f]'
4182023-11-02T05:40:53.810Z+ VDEV_PATH=/scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev
4192023-11-02T05:40:53.847Z+ [[ -f /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev ]]
4202023-11-02T05:40:53.878Z+ dd if=/dev/zero of=/scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev bs=1 count=0 seek=20G
4212023-11-02T05:40:53.904Z0+0 records in
4222023-11-02T05:40:53.931Z0+0 records out
4232023-11-02T05:40:53.956Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4242023-11-02T05:40:53.981Z+ success 'ZFS vdev /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev exists'
4252023-11-02T05:40:54.006Z+ set +x
4262023-11-02T05:40:54.031Z++ zpool list -o name
4272023-11-02T05:40:54.056Z++ grep oxp_31bd71cd-4736-4a12-a387-9b74b050396f
4282023-11-02T05:40:54.081Z+ [[ -z '' ]]
4292023-11-02T05:40:54.151Z+ zpool create -f oxp_31bd71cd-4736-4a12-a387-9b74b050396f /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev
4302023-11-02T05:40:54.203Z+ success 'ZFS zpool oxp_31bd71cd-4736-4a12-a387-9b74b050396f exists'
4312023-11-02T05:40:54.228Z+ set +x
4322023-11-02T05:40:54.252Z+ for ZPOOL in "${ZPOOLS[@]}"
4332023-11-02T05:40:54.277Z+ echo 'Zpool: [oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7]'
4342023-11-02T05:40:54.302Z+ VDEV_PATH=/scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev
4352023-11-02T05:40:54.327Z+ [[ -f /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev ]]
4362023-11-02T05:40:54.353Z+ dd if=/dev/zero of=/scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev bs=1 count=0 seek=20G
4372023-11-02T05:40:54.377Z0+0 records in
4382023-11-02T05:40:54.403Z0+0 records out
4392023-11-02T05:40:54.429Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4402023-11-02T05:40:54.453Z+ success 'ZFS vdev /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev exists'
4412023-11-02T05:40:54.480Z+ set +x
4422023-11-02T05:40:54.512Z++ zpool list -o name
4432023-11-02T05:40:54.537Z++ grep oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7
4442023-11-02T05:40:54.562Z+ [[ -z '' ]]
4452023-11-02T05:40:54.587Z+ zpool create -f oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev
4462023-11-02T05:40:54.611Z+ success 'ZFS zpool oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 exists'
4472023-11-02T05:40:54.636Z+ set +x
4482023-11-02T05:40:54.662Z+ for ZPOOL_TYPE in "${ZPOOL_TYPES[@]}"
4492023-11-02T05:40:54.687Z+ readarray -t ZPOOLS
4502023-11-02T05:40:54.712Z++ grep '"oxi_' /opt/oxide/work/tools/../smf/sled-agent/non-gimlet/config.toml
4512023-11-02T05:40:54.737Z++ sed 's/[ ",]//g'
4522023-11-02T05:40:54.761Z+ for ZPOOL in "${ZPOOLS[@]}"
4532023-11-02T05:40:54.787Z+ echo 'Zpool: [oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b]'
4542023-11-02T05:40:54.812Z+ VDEV_PATH=/scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4552023-11-02T05:40:54.837Z+ [[ -f /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
4562023-11-02T05:40:54.866Z+ dd if=/dev/zero of=/scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
4572023-11-02T05:40:55.311Z0+0 records in
4582023-11-02T05:40:55.340Z0+0 records out
4592023-11-02T05:40:55.364Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4602023-11-02T05:40:55.389Z+ success 'ZFS vdev /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
4612023-11-02T05:40:55.413Z+ set +x
4622023-11-02T05:40:55.438Z++ zpool list -o name
4632023-11-02T05:40:55.462Z++ grep oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b
4642023-11-02T05:40:55.487Z+ [[ -z '' ]]
4652023-11-02T05:40:55.512Z+ zpool create -f oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4662023-11-02T05:40:55.536Z+ success 'ZFS zpool oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
4672023-11-02T05:40:55.560Z+ set +x
4682023-11-02T05:40:55.585Z+ for ZPOOL in "${ZPOOLS[@]}"
4692023-11-02T05:40:55.609Z+ echo 'Zpool: [oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b]'
4702023-11-02T05:40:55.633Z+ VDEV_PATH=/scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4712023-11-02T05:40:55.658Z+ [[ -f /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
4722023-11-02T05:40:55.682Z+ dd if=/dev/zero of=/scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
4732023-11-02T05:40:55.712Z0+0 records in
4742023-11-02T05:40:56.147Z0+0 records out
4752023-11-02T05:40:56.174Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4762023-11-02T05:40:56.238Z+ success 'ZFS vdev /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
4772023-11-02T05:40:56.265Z+ set +x
4782023-11-02T05:40:56.301Z++ zpool list -o name
4792023-11-02T05:40:56.331Z++ grep oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b
4802023-11-02T05:40:56.358Z+ [[ -z '' ]]
4812023-11-02T05:40:56.384Z+ zpool create -f oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4822023-11-02T05:40:56.409Z+ success 'ZFS zpool oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
4832023-11-02T05:40:56.434Z+ set +x
4842023-11-02T05:40:56.459Z+ ensure_simulated_links igb0
4852023-11-02T05:40:56.484Z+ local PHYSICAL_LINK=igb0
4862023-11-02T05:40:56.511Z+ INDICES=("0" "1")
4872023-11-02T05:40:56.536Z+ for I in "${INDICES[@]}"
4882023-11-02T05:40:56.563Z++ get_simnet_name_if_exists net0
4892023-11-02T05:40:56.597Z++ dladm show-simnet -p -o LINK net0
4902023-11-02T05:40:57.034Z++ echo ''
4912023-11-02T05:40:57.065Z+ [[ -z '' ]]
4922023-11-02T05:40:57.095Z+ dladm create-simnet -t net0
4932023-11-02T05:40:57.121Z+ dladm create-simnet -t sc0_0
4942023-11-02T05:40:57.146Z+ dladm modify-simnet -t -p net0 sc0_0
4952023-11-02T05:40:57.170Z+ dladm set-linkprop -p mtu=9000 sc0_0
4962023-11-02T05:40:57.194Z+ success 'Simnet net0/sc0_0 exists'
4972023-11-02T05:40:57.218Z+ set +x
4982023-11-02T05:40:57.267Z+ for I in "${INDICES[@]}"
4992023-11-02T05:40:57.292Z++ get_simnet_name_if_exists net1
5002023-11-02T05:40:57.317Z++ dladm show-simnet -p -o LINK net1
5012023-11-02T05:40:57.341Zdownloading softnpu ...done
5022023-11-02T05:40:57.366Z++ echo ''
5032023-11-02T05:40:57.390Z+ [[ -z '' ]]
5042023-11-02T05:40:57.415Z+ dladm create-simnet -t net1
5052023-11-02T05:40:57.439Z+ dladm create-simnet -t sc1_0
5062023-11-02T05:40:57.463Z+ dladm modify-simnet -t -p net1 sc1_0
5072023-11-02T05:40:57.487Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/sidecar-lite/release/f0585a29fb0285f7a1220c1118856b0e5c1f75c5/libsidecar_lite.so.sha256.txt
5082023-11-02T05:40:57.520Z+ dladm set-linkprop -p mtu=9000 sc1_0
5092023-11-02T05:40:57.553Z+ success 'Simnet net1/sc1_0 exists'
5102023-11-02T05:40:57.580Z+ set +x
5112023-11-02T05:40:57.604Z++ get_vnic_name_if_exists sc0_1
5122023-11-02T05:40:57.629Z++ dladm show-vnic -p -o LINK sc0_1
5132023-11-02T05:40:57.654Z++ echo ''
5142023-11-02T05:40:57.678Z+ [[ -z '' ]]
5152023-11-02T05:40:57.704Z+ dladm create-vnic -t sc0_1 -l igb0 -m a8:e1:de:01:70:1d
5162023-11-02T05:40:57.729Z+ [[ -v PROMISC_FILT_OFF ]]
5172023-11-02T05:40:57.754Z+ success 'Vnic sc0_1 exists'
5182023-11-02T05:40:57.779Z+ set +x
5192023-11-02T05:40:57.804Z+ warn_if_no_proxy_arp
5202023-11-02T05:40:57.829Z+ [[ -v PXA_START ]]
5212023-11-02T05:40:57.854Z+ [[ -v PXA_END ]]
5222023-11-02T05:40:57.879Z+ ensure_softnpu_zone
5232023-11-02T05:40:57.904Z+ zoneadm list
5242023-11-02T05:40:57.934Z+ grep -q sidecar_softnpu
5252023-11-02T05:40:57.967Z+ [[ -f out/npuzone/npuzone ]]
5262023-11-02T05:40:57.994Z+ out/npuzone/npuzone create sidecar --omicron-zone --ports sc0_0,tfportrear0_0 --ports sc0_1,tfportqsfp0_0 --sidecar-lite-branch main
5272023-11-02T05:40:59.867Zdownloading asic_program.so ...done
5282023-11-02T05:40:59.915Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/sidecar-lite/release/f0585a29fb0285f7a1220c1118856b0e5c1f75c5/scadm.sha256.txt
5292023-11-02T05:41:05.514Zdownloading scadm ...done
5302023-11-02T05:41:05.563Zcreating port tfportrear0_0 for sc0_0
5312023-11-02T05:41:05.591Zcreating port tfportqsfp0_0 for sc0_1
5322023-11-02T05:41:05.616Zsoftnpu zone setup
5332023-11-02T05:41:05.642Zconfigure zone
5342023-11-02T05:41:05.667Zinstall zone
5352023-11-02T05:41:06.655Zboot zone
5362023-11-02T05:41:09.765Z[sidecar_softnpu] svcs milestone/network
5372023-11-02T05:41:09.793ZSTATE STIME FMRI
5382023-11-02T05:41:09.818Zonline 5:41:07 svc:/milestone/network:default
5392023-11-02T05:41:09.843Z[sidecar_softnpu] /softnpu/npu start
5402023-11-02T05:41:09.868Z
5412023-11-02T05:41:09.893Z+ /opt/oxide/work/tools/scrimlet/softnpu-init.sh
5422023-11-02T05:41:09.918Z+ GATEWAY_IP=10.151.2.100
5432023-11-02T05:41:09.944Z+ echo 'Using 10.151.2.100 as gateway ip'
5442023-11-02T05:41:09.969ZUsing 10.151.2.100 as gateway ip
5452023-11-02T05:41:09.995Z+ [[ ! -v GATEWAY_MAC ]]
5462023-11-02T05:41:10.020Z+ GATEWAY_MAC=18:c0:4d:0d:a0:2a
5472023-11-02T05:41:10.046Z++ grep -c -E '^([0-9a-fA-F]{2}:){5}[0-9a-fA-F]{2}$' /dev/fd/63
5482023-11-02T05:41:10.071Z+++ echo 18:c0:4d:0d:a0:2a
5492023-11-02T05:41:10.096Z+ COUNT=1
5502023-11-02T05:41:10.121Z+ [[ 1 -ne 1 ]]
5512023-11-02T05:41:10.148Z+ echo 'Using 18:c0:4d:0d:a0:2a as gateway mac'
5522023-11-02T05:41:10.174ZUsing 18:c0:4d:0d:a0:2a as gateway mac
5532023-11-02T05:41:10.199Z+ z_scadm add-arp-entry 10.151.2.100 18:c0:4d:0d:a0:2a
5542023-11-02T05:41:10.225Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone add-arp-entry 10.151.2.100 18:c0:4d:0d:a0:2a
5552023-11-02T05:41:10.250Z+ PXA_MAC=a8:e1:de:01:70:1d
5562023-11-02T05:41:10.275Z+ [[ -v PXA_START ]]
5572023-11-02T05:41:10.300Z+ z_scadm add-proxy-arp 10.151.2.164 10.151.2.227 a8:e1:de:01:70:1d
5582023-11-02T05:41:10.326Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone add-proxy-arp 10.151.2.164 10.151.2.227 a8:e1:de:01:70:1d
5592023-11-02T05:41:10.352Z+ z_scadm dump-state
5602023-11-02T05:41:10.377Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone dump-state
5612023-11-02T05:41:10.402Zlocal v6:
5622023-11-02T05:41:10.427Zlocal v4:
5632023-11-02T05:41:10.452Zrouter v6:
5642023-11-02T05:41:10.477Zrouter v4_idx:
5652023-11-02T05:41:10.507Zrouter v4_routes:
5662023-11-02T05:41:10.535Zresolver v4:
5672023-11-02T05:41:10.561Z10.151.2.100 -> 18:c0:4d:0d:a0:2a
5682023-11-02T05:41:10.586Zresolver v6:
5692023-11-02T05:41:10.612Znat v4:
5702023-11-02T05:41:10.639Znat_v6:
5712023-11-02T05:41:10.664Zport_mac:
5722023-11-02T05:41:10.691Zproxy arp:
5732023-11-02T05:41:10.717Z10.151.2.164/10.151.2.227: a8:e1:de:01:70:1d
5742023-11-02T05:41:10.742Z+ success 'softnpu zone exists'
5752023-11-02T05:41:10.767Z+ set +x
5762023-11-02T05:41:10.808Zsoftnpu zone exists
5772023-11-02T05:41:10.833Z
5782023-11-02T05:41:10.859Zreal 23.259586647
5792023-11-02T05:41:10.885Zuser 0.729561937
5802023-11-02T05:41:10.911Zsys 1.648600976
5812023-11-02T05:41:10.943Ztrap 0.000090938
5822023-11-02T05:41:10.968Ztflt 0.000128357
5832023-11-02T05:41:10.994Zdflt 0.001040092
5842023-11-02T05:41:11.020Zkflt 0.000000000
5852023-11-02T05:41:11.046Zlock 18:05.636979296
5862023-11-02T05:41:11.073Zslp 1:09.149598715
5872023-11-02T05:41:11.098Zlat 0.085511173
5882023-11-02T05:41:11.123Zstop 0.416694682
5892023-11-02T05:41:11.148Z+ tar xf out/omicron-sled-agent.tar pkg/config-rss.toml
5902023-11-02T05:41:11.174Z++ sed -n 's/silo_name = "\(.*\)"/\1/p' pkg/config-rss.toml
5912023-11-02T05:41:11.199Z+ SILO_NAME=recovery
5922023-11-02T05:41:11.225Z++ sed -n 's/external_dns_zone_name = "\(.*\)"/\1/p' pkg/config-rss.toml
5932023-11-02T05:41:11.250Z+ EXTERNAL_DNS_DOMAIN=oxide.test
5942023-11-02T05:41:11.275Z+ sed -i~ '
5952023-11-02T05:41:11.300Z /^external_dns_ips/c\
5962023-11-02T05:41:11.325Zexternal_dns_ips = [ "10.151.2.164", "10.151.2.165" ]
5972023-11-02T05:41:11.351Z /^\[\[internal_services_ip_pool_ranges/,/^$/ {
5982023-11-02T05:41:11.376Z /^first/c\
5992023-11-02T05:41:11.401Zfirst = "10.151.2.164"
6002023-11-02T05:41:11.427Z /^last/c\
6012023-11-02T05:41:11.453Zlast = "10.151.2.173"
6022023-11-02T05:41:11.480Z }
6032023-11-02T05:41:11.506Z /^\[rack_network_config/,/^$/ {
6042023-11-02T05:41:11.531Z /^infra_ip_first/c\
6052023-11-02T05:41:11.557Zinfra_ip_first = "10.151.2.174"
6062023-11-02T05:41:11.592Z /^infra_ip_last/c\
6072023-11-02T05:41:11.617Zinfra_ip_last = "10.151.2.174"
6082023-11-02T05:41:11.643Z--- pkg/config-rss.toml~ Mon Jul 24 01:21:28 2006
6092023-11-02T05:41:11.668Z }
6102023-11-02T05:41:11.694Z+++ pkg/config-rss.toml Thu Nov 2 05:41:09 2023
6112023-11-02T05:41:11.719Z /^\[\[rack_network_config.ports/,/^$/ {
6122023-11-02T05:41:11.744Z@@ -41,7 +41,7 @@
6132023-11-02T05:41:11.769Z /^routes/c\
6142023-11-02T05:41:11.794Z # the DNS domain delegated to the rack by the customer. Each of these addresses
6152023-11-02T05:41:11.843Zroutes = \[{nexthop = "10.151.2.100", destination = "0.0.0.0/0"}\]
6162023-11-02T05:41:11.871Z # must be contained in one of the "internal services" IP Pool ranges listed
6172023-11-02T05:41:11.896Z /^addresses/c\
6182023-11-02T05:41:11.921Z # below.
6192023-11-02T05:41:11.949Zaddresses = \["10.151.2.174/32"\]
6202023-11-02T05:41:11.974Z-external_dns_ips = [ "192.168.1.20", "192.168.1.21" ]
6212023-11-02T05:41:12.003Z }
6222023-11-02T05:41:12.042Z+external_dns_ips = [ "10.151.2.164", "10.151.2.165" ]
6232023-11-02T05:41:12.075Z' pkg/config-rss.toml
6242023-11-02T05:41:12.100Z
6252023-11-02T05:41:12.125Z+ diff -u pkg/config-rss.toml~ pkg/config-rss.toml
6262023-11-02T05:41:12.151Z # Initial TLS certificates for the external API
6272023-11-02T05:41:12.176Z+ true
6282023-11-02T05:41:12.201Z #
6292023-11-02T05:41:12.226Z+ tar rvf out/omicron-sled-agent.tar pkg/config-rss.toml
6302023-11-02T05:41:12.251Z@@ -77,8 +77,8 @@
6312023-11-02T05:41:12.276Z+ rm -f pkg/config-rss.toml pkg/config-rss.toml~
6322023-11-02T05:41:12.302Z #
6332023-11-02T05:41:12.327Z+ TLS_NAME=recovery.sys.oxide.test
6342023-11-02T05:41:12.352Z # For more on this and what to put here, see docs/how-to-run.adoc.
6352023-11-02T05:41:12.379Z+ openssl req -newkey rsa:4096 -x509 -sha256 -days 3 -nodes -out pkg/initial-tls-cert.pem -keyout pkg/initial-tls-key.pem -subj /CN=recovery.sys.oxide.test -addext subjectAltName=DNS:recovery.sys.oxide.test -addext basicConstraints=critical,CA:FALSE -config /dev/stdin
6362023-11-02T05:41:12.404Z [[internal_services_ip_pool_ranges]]
6372023-11-02T05:41:12.430Z-first = "192.168.1.20"
6382023-11-02T05:41:12.454Z-last = "192.168.1.29"
6392023-11-02T05:41:12.480Z+first = "10.151.2.164"
6402023-11-02T05:41:12.505Z+last = "10.151.2.173"
6412023-11-02T05:41:12.530Z
6422023-11-02T05:41:12.555Z # TODO - this configuration is subject to change going forward. Ultimately these
6432023-11-02T05:41:12.995Z # parameters should be provided to the control plane via wicket, but we need to
6442023-11-02T05:41:13.019Z@@ -92,8 +92,8 @@
6452023-11-02T05:41:13.045Z # A range of IP addresses used by Boundary Services on the external network. In
6462023-11-02T05:41:13.078Z # a real system, these would be addresses of the uplink ports on the Sidecar.
6472023-11-02T05:41:13.104Z # With softnpu, only one address is used.
6482023-11-02T05:41:13.131Z-infra_ip_first = "192.168.1.30"
6492023-11-02T05:41:13.179Z-infra_ip_last = "192.168.1.30"
6502023-11-02T05:41:13.205Z+infra_ip_first = "10.151.2.174"
6512023-11-02T05:41:13.230Z+infra_ip_last = "10.151.2.174"
6522023-11-02T05:41:13.255Z
6532023-11-02T05:41:13.281Z # Configurations for BGP routers to run on the scrimlets.
6542023-11-02T05:41:13.307Z bgp = []
6552023-11-02T05:41:13.332Z@@ -101,9 +101,9 @@
6562023-11-02T05:41:13.358Z # You can configure multiple uplinks by repeating the following stanza
6572023-11-02T05:41:13.384Z [[rack_network_config.ports]]
6582023-11-02T05:41:13.410Z # Routes associated with this port.
6592023-11-02T05:41:13.435Z-routes = [{nexthop = "192.168.1.199", destination = "0.0.0.0/0"}]
6602023-11-02T05:41:13.460Z+routes = [{nexthop = "10.151.2.100", destination = "0.0.0.0/0"}]
6612023-11-02T05:41:13.486Z # Addresses associated with this port.
6622023-11-02T05:41:13.511Z-addresses = ["192.168.1.30/32"]
6632023-11-02T05:41:13.537Z+addresses = ["10.151.2.174/32"]
6642023-11-02T05:41:13.562Z # Name of the uplink port. This should always be "qsfp0" when using softnpu.
6652023-11-02T05:41:13.592Z port = "qsfp0"
6662023-11-02T05:41:13.620Z # The speed of this port.
6672023-11-02T05:41:13.644Za pkg/config-rss.toml 7K
6682023-11-02T05:41:13.669Z..+......+......+........+...................+......+...........+....+..+............+...+.......+...........+.+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.+.........+.....+.......+..+......+.......+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*.........+.+...+..+.+.....+.........+.+............+..........................+.+...........+.......+.....+...............+...............+...+.........+.+............+......+..+...............+...+.........+...+.........+...+......+.+........+.+..+.......+........+....+.....+......+.........+.+..+...+............+.+.........+......+......+.....................+..+......+.+........+...+........................+.+..+...............+...+....+......+..+.......+.........+......+.....+......+...+......+...+......+.+.....................+.....+.............+......+.........+.....+......................+..+...+....+......+..............+....+..+.......+..+.+...+...........+...................+.....+.+...+...+..............+...............+.+............+..+................+...+.........+.....+......+.+...+.................+...+.+.....+.+...........+...+......+.+.....+...+...................+.....+.+..............................+..+............+.....................+.........................+.........+..+.+.........+...+...........+.+...+...............+.....+.+...............+..+..........+..+.........................+........+.........+......+....+...........+...+.+......+........+.+.........+..+...............+..........+..............+...+...............+......+..........+..............+............+...+..........+...+..+...+...+......+......+...+.......+...+......+...........+...+.+...+......+.....+.+.....................+...+......+........+.......+.....+.+............+..+.+.....+............................+..+...+......+.+..+.................................+.........+.........+....+......+.........+.....+.......+......+..+.+..+..........+...............+..+.....................+..........+..+...+.......+.....+....+........+..................+.........+.+.........+......+.....+.......+.....................+...............+..+.............+.....+............+.........+.+...+.....+......+.+.....+.......+...........+.....................+.+.....+.......+..+.............+......+......+...+...........+.......+...+......+..............+....+..+................+..+...+.......+...........+.........+.......+........................+......+........+..............................+...+....+........+.+..............................+...........+...+......+....+.....+...+.......+........+...+....+........+..................+.........+..........+...........+.......+..+......+.......+..+...+....+...+....................+..........+.....+..........+...........+..........+.....+..........+..+...+....+...+.....+.............+...........+....+..............................+......+...+...+..+..................+...+.+........+...+.........+.+........+.......+...+......+.....+......+...+.+...+......+............+..+.+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
6692023-11-02T05:41:13.695Z..+........+...+.......+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*......+.........+...........+.+...+............+..+...+.........+......+.............+......+...+......+..+...+....+...+...+..+...+.+.....+.+..+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*....+...........+.....................+.+..............+....+..+.......+...+..............+...+......+.+......+..+...+..................+....+...+........+......+.+...+..+.........+..............................+.+...+............+...+.....+....+...................................+...+............+......+.+.........+..+....+...............+........+.+.........+..+...+...........................+......+............+.+.........+......+..+...+.........+.+......+......+........+...............+.+.........+..+..........+..................+........................+...........+......+...+.+...........+........................+....+.....+.+......+.....+..................+...............+......+.+..............+....+...........+....+...+..............+...+...............+.......+......+..+...+...+.+.....+...............+.+........+.........+..........+...+..................+.........+..+.........+....+..........................+.......+..+.+..+....+...........+.............+..+.............+......+..............+.........+......+....+...+..............+....+.....+....+..+......+...............+.........+.+.........+...+............+...........+.......+.....+.....................+.+......+..+......+.+...+..+.+....................+.............+.....+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
6702023-11-02T05:41:13.721Z-----
6712023-11-02T05:41:13.747Z+ tar rvf out/omicron-sled-agent.tar pkg/initial-tls-cert.pem pkg/initial-tls-key.pem
6722023-11-02T05:41:13.772Za pkg/initial-tls-cert.pem 2K
6732023-11-02T05:41:13.797Za pkg/initial-tls-key.pem 4K
6742023-11-02T05:41:13.821Z+ rm -f pkg/initial-tls-cert.pem pkg/initial-tls-key.pem
6752023-11-02T05:41:13.845Z+ rmdir pkg
6762023-11-02T05:41:13.870Z+ E2E_TLS_CERT=/opt/oxide/sled-agent/pkg/initial-tls-cert.pem
6772023-11-02T05:41:13.908Z+ pfexec mkdir -p /usr/oxide
6782023-11-02T05:41:13.933Z+ pfexec rm -f /usr/oxide/tcpproxy
6792023-11-02T05:41:13.957Z+ pfexec curl -sSfL -o /usr/oxide/tcpproxy http://catacomb.eng.oxide.computer:12346/tcpproxy
6802023-11-02T05:41:13.982Z+ pfexec chmod +x /usr/oxide/tcpproxy
6812023-11-02T05:41:14.011Z+ pfexec rm -f /var/svc/manifest/site/tcpproxy.xml
6822023-11-02T05:41:14.049Z+ pfexec curl -sSfL -o /var/svc/manifest/site/tcpproxy.xml http://catacomb.eng.oxide.computer:12346/tcpproxy.xml
6832023-11-02T05:41:14.082Z+ pfexec svccfg import /var/svc/manifest/site/tcpproxy.xml
6842023-11-02T05:41:14.107Z+ OMICRON_NO_UNINSTALL=1
6852023-11-02T05:41:15.362Z+ ptime -m pfexec ./target/release/omicron-package -t test install
6862023-11-02T05:41:15.396ZNov 02 05:41:12.532 DEBG target[test]: Target({"image": "standard", "machine": "non-gimlet", "rack-topology": "single-sled", "switch": "softnpu"})
6872023-11-02T05:41:15.455ZNov 02 05:41:12.534 INFO Installing service, dst: /opt/oxide/clickhouse.tar.gz, src: out/clickhouse.tar.gz
6882023-11-02T05:41:15.480ZNov 02 05:41:12.534 INFO Installing service, dst: /opt/oxide/clickhouse_keeper.tar.gz, src: out/clickhouse_keeper.tar.gz
6892023-11-02T05:41:15.507ZNov 02 05:41:12.534 INFO Installing service, dst: /opt/oxide/nexus.tar.gz, src: out/omicron-nexus.tar.gz
6902023-11-02T05:41:15.533ZNov 02 05:41:12.534 INFO Installing service, dst: /opt/oxide/overlay.tar.gz, src: out/overlay.tar.gz
6912023-11-02T05:41:15.557ZNov 02 05:41:12.534 INFO Installing service, dst: /opt/oxide/switch.tar.gz, src: out/switch-softnpu.tar.gz
6922023-11-02T05:41:15.583ZNov 02 05:41:12.534 INFO Installing service, dst: /opt/oxide/crucible_pantry.tar.gz, src: out/crucible-pantry.tar.gz
6932023-11-02T05:41:15.608ZNov 02 05:41:12.534 INFO Installing service, dst: /opt/oxide/mg-ddm.tar, src: out/mg-ddm-gz.tar
6942023-11-02T05:41:15.634ZNov 02 05:41:12.535 INFO Installing service, dst: /opt/oxide/cockroachdb.tar.gz, src: out/cockroachdb.tar.gz
6952023-11-02T05:41:15.659ZNov 02 05:41:12.535 INFO Installing service, dst: /opt/oxide/crucible.tar.gz, src: out/crucible.tar.gz
6962023-11-02T05:41:15.684ZNov 02 05:41:12.535 INFO Installing service, dst: /opt/oxide/oximeter.tar.gz, src: out/oximeter-collector.tar.gz
6972023-11-02T05:41:15.711ZNov 02 05:41:12.535 INFO Installing service, dst: /opt/oxide/propolis-server.tar.gz, src: out/propolis-server.tar.gz
6982023-11-02T05:41:15.750ZNov 02 05:41:12.535 INFO Installing service, dst: /opt/oxide/sled-agent.tar, src: out/omicron-sled-agent.tar
6992023-11-02T05:41:15.784ZNov 02 05:41:12.535 INFO Installing service, dst: /opt/oxide/internal_dns.tar.gz, src: out/internal-dns.tar.gz
7002023-11-02T05:41:15.814ZNov 02 05:41:12.535 INFO Installing service, dst: /opt/oxide/mgs.tar.gz, src: out/omicron-gateway-softnpu.tar.gz
7012023-11-02T05:41:15.848ZNov 02 05:41:12.535 INFO Installing service, dst: /opt/oxide/external_dns.tar.gz, src: out/external-dns.tar.gz
7022023-11-02T05:41:15.874ZNov 02 05:41:12.535 INFO Installing service, dst: /opt/oxide/ntp.tar.gz, src: out/ntp.tar.gz
7032023-11-02T05:41:15.899ZNov 02 05:41:12.761 INFO Unpacking service tarball, service_path: /opt/oxide/mg-ddm, tar_path: /opt/oxide/mg-ddm.tar
7042023-11-02T05:41:15.925ZNov 02 05:41:12.796 INFO Unpacking service tarball, service_path: /opt/oxide/sled-agent, tar_path: /opt/oxide/sled-agent.tar
7052023-11-02T05:41:15.951ZNov 02 05:41:12.860 INFO Installing bootstrap service from /opt/oxide/sled-agent/pkg/manifest.xml
7062023-11-02T05:41:16.036Z
7072023-11-02T05:41:16.065Zreal 0.466744423
7082023-11-02T05:41:16.090Zuser 0.099115293
7092023-11-02T05:41:16.115Zsys 1.899383267
7102023-11-02T05:41:16.141Ztrap 0.000004780
7112023-11-02T05:41:16.167Ztflt 0.000028649
7122023-11-02T05:41:16.199Zdflt 0.001096657
7132023-11-02T05:41:16.224Zkflt 0.000000000
7142023-11-02T05:41:16.249Zlock 41.855203088
7152023-11-02T05:41:16.273Zslp 0.679631366
7162023-11-02T05:41:16.300Zlat 0.003023037
7172023-11-02T05:41:16.326Zstop 0.198822586
7182023-11-02T05:41:16.351Z+ retry=0
7192023-11-02T05:41:16.376Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7202023-11-02T05:41:16.401Z+ [[ 0 -gt 30 ]]
7212023-11-02T05:41:16.426Z+ sleep 1
7222023-11-02T05:41:16.451Z+ retry=1
7232023-11-02T05:41:16.476Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7242023-11-02T05:41:16.503Z+ [[ 1 -gt 30 ]]
7252023-11-02T05:41:16.529Z+ sleep 1
7262023-11-02T05:41:16.554Z+ retry=2
7272023-11-02T05:41:16.579Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7282023-11-02T05:41:40.485Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone dump-state
7292023-11-02T05:41:40.515Zlocal v6:
7302023-11-02T05:41:40.540Zfe80::2cc6:23ff:fec2:8608
7312023-11-02T05:41:40.565Zfd00:99::1
7322023-11-02T05:41:40.590Zlocal v4:
7332023-11-02T05:41:40.616Z10.151.2.174
7342023-11-02T05:41:40.641Zrouter v6:
7352023-11-02T05:41:40.666Zfdb0:18c0:4d0d:a02a::/64 -> fe80::3491:77ff:fe97:d1c4 (1)
7362023-11-02T05:41:40.691Zfd00:1122:3344:101::/64 -> fe80::3491:77ff:fe97:d1c4 (1)
7372023-11-02T05:41:40.725Zrouter v4_idx:
7382023-11-02T05:41:40.752Z0.0.0.0/0 -> 0
7392023-11-02T05:41:40.777Zrouter v4_routes:
7402023-11-02T05:41:40.802Z0 -> 10.151.2.100 (2)
7412023-11-02T05:41:40.826Zresolver v4:
7422023-11-02T05:41:40.851Z10.151.2.174 -> 52:3f:77:9b:0a:a4
7432023-11-02T05:41:40.877Z10.151.2.100 -> 18:c0:4d:0d:a0:2a
7442023-11-02T05:41:40.902Zresolver v6:
7452023-11-02T05:41:40.927Zfe80::3491:77ff:fe97:d1c4 -> 36:91:77:97:d1:c4
7462023-11-02T05:41:40.953Zfe80::2cc6:23ff:fec2:8608 -> 2e:c6:23:c2:86:08
7472023-11-02T05:41:40.978Znat v4:
7482023-11-02T05:41:41.004Znat_v6:
7492023-11-02T05:41:41.029Zport_mac:
7502023-11-02T05:41:41.054Z1: a8:40:25:1e:0f:da
7512023-11-02T05:41:41.079Z1000: a8:40:25:1e:0f:d9
7522023-11-02T05:41:41.104Zproxy arp:
7532023-11-02T05:41:41.129Z10.151.2.164/10.151.2.227: a8:e1:de:01:70:1d
7542023-11-02T05:41:41.154Z+ retry=0
7552023-11-02T05:41:41.182Z+++ zoneadm list -n
7562023-11-02T05:41:41.207Z+++ grep oxz_ntp
7572023-11-02T05:41:41.244Z++ pfexec svcs -z -Hostate oxide/ntp
7582023-11-02T05:41:41.269Zsvcs: zone '-Hostate': entity not found
7592023-11-02T05:41:41.294Z++ true
7602023-11-02T05:41:41.321Z+ [[ '' != online ]]
7612023-11-02T05:41:41.348Z+ [[ 0 -gt 60 ]]
7622023-11-02T05:41:41.375Z+ sleep 1
7632023-11-02T05:41:41.540Z+ retry=1
7642023-11-02T05:41:41.565Z+++ zoneadm list -n
7652023-11-02T05:41:41.590Z+++ grep oxz_ntp
7662023-11-02T05:41:41.617Z++ pfexec svcs -z -Hostate oxide/ntp
7672023-11-02T05:41:41.643Zsvcs: zone '-Hostate': entity not found
7682023-11-02T05:41:41.669Z++ true
7692023-11-02T05:41:41.698Z+ [[ '' != online ]]
7702023-11-02T05:41:41.725Z+ [[ 1 -gt 60 ]]
7712023-11-02T05:41:41.749Z+ sleep 1
7722023-11-02T05:41:42.563Z+ retry=2
7732023-11-02T05:41:42.589Z+++ zoneadm list -n
7742023-11-02T05:41:42.625Z+++ grep oxz_ntp
7752023-11-02T05:41:42.650Z++ pfexec svcs -z -Hostate oxide/ntp
7762023-11-02T05:41:42.682Zsvcs: zone '-Hostate': entity not found
7772023-11-02T05:41:42.710Z++ true
7782023-11-02T05:41:42.737Z+ [[ '' != online ]]
7792023-11-02T05:41:42.768Z+ [[ 2 -gt 60 ]]
7802023-11-02T05:41:42.800Z+ sleep 1
7812023-11-02T05:41:44.064Z+ retry=3
7822023-11-02T05:41:44.092Z+++ zoneadm list -n
7832023-11-02T05:41:44.118Z+++ grep oxz_ntp
7842023-11-02T05:41:44.172Z++ pfexec svcs -z -Hostate oxide/ntp
7852023-11-02T05:41:44.237Zsvcs: zone '-Hostate': entity not found
7862023-11-02T05:41:44.408Z++ true
7872023-11-02T05:41:44.447Z+ [[ '' != online ]]
7882023-11-02T05:41:44.478Z+ [[ 3 -gt 60 ]]
7892023-11-02T05:41:44.512Z+ sleep 1
7902023-11-02T05:41:44.846Z+ retry=4
7912023-11-02T05:41:44.909Z+++ zoneadm list -n
7922023-11-02T05:41:44.940Z+++ grep oxz_ntp
7932023-11-02T05:41:45.376Z++ pfexec svcs -z -Hostate oxide/ntp
7942023-11-02T05:41:45.406Zsvcs: zone '-Hostate': entity not found
7952023-11-02T05:41:45.448Z++ true
7962023-11-02T05:41:45.478Z+ [[ '' != online ]]
7972023-11-02T05:41:45.511Z+ [[ 4 -gt 60 ]]
7982023-11-02T05:41:45.540Z+ sleep 1
7992023-11-02T05:41:45.641Z+ retry=5
8002023-11-02T05:41:45.677Z+++ zoneadm list -n
8012023-11-02T05:41:45.706Z+++ grep oxz_ntp
8022023-11-02T05:41:45.744Z++ pfexec svcs -z -Hostate oxide/ntp
8032023-11-02T05:41:45.779Zsvcs: zone '-Hostate': entity not found
8042023-11-02T05:41:45.809Z++ true
8052023-11-02T05:41:45.835Z+ [[ '' != online ]]
8062023-11-02T05:41:46.278Z+ [[ 5 -gt 60 ]]
8072023-11-02T05:41:46.304Z+ sleep 1
8082023-11-02T05:41:46.656Z+ retry=6
8092023-11-02T05:41:46.717Z+++ zoneadm list -n
8102023-11-02T05:41:46.743Z+++ grep oxz_ntp
8112023-11-02T05:41:46.768Z++ pfexec svcs -z -Hostate oxide/ntp
8122023-11-02T05:41:46.794Zsvcs: zone '-Hostate': entity not found
8132023-11-02T05:41:46.821Z++ true
8142023-11-02T05:41:46.846Z+ [[ '' != online ]]
8152023-11-02T05:41:46.871Z+ [[ 6 -gt 60 ]]
8162023-11-02T05:41:46.896Z+ sleep 1
8172023-11-02T05:41:47.678Z+ retry=7
8182023-11-02T05:41:47.703Z+++ zoneadm list -n
8192023-11-02T05:41:47.755Z+++ grep oxz_ntp
8202023-11-02T05:41:47.788Z++ pfexec svcs -z -Hostate oxide/ntp
8212023-11-02T05:41:47.819Zsvcs: zone '-Hostate': entity not found
8222023-11-02T05:41:47.847Z++ true
8232023-11-02T05:41:47.893Z+ [[ '' != online ]]
8242023-11-02T05:41:47.921Z+ [[ 7 -gt 60 ]]
8252023-11-02T05:41:47.946Z+ sleep 1
8262023-11-02T05:41:48.700Z+ retry=8
8272023-11-02T05:41:48.725Z+++ zoneadm list -n
8282023-11-02T05:41:48.750Z+++ grep oxz_ntp
8292023-11-02T05:41:48.809Z++ pfexec svcs -z -Hostate oxide/ntp
8302023-11-02T05:41:48.839Zsvcs: zone '-Hostate': entity not found
8312023-11-02T05:41:48.869Z++ true
8322023-11-02T05:41:49.312Z+ [[ '' != online ]]
8332023-11-02T05:41:49.366Z+ [[ 8 -gt 60 ]]
8342023-11-02T05:41:49.391Z+ sleep 1
8352023-11-02T05:41:49.721Z+ retry=9
8362023-11-02T05:41:49.746Z+++ zoneadm list -n
8372023-11-02T05:41:49.770Z+++ grep oxz_ntp
8382023-11-02T05:41:49.796Z++ pfexec svcs -z oxz_ntp_7b014f06-174e-4581-90a8-5c25f48af080 -Hostate oxide/ntp
8392023-11-02T05:41:49.821Zsvcs: Could not bind to repository server for zone oxz_ntp_7b014f06-174e-4581-90a8-5c25f48af080: repository server unavailable
8402023-11-02T05:41:49.846Z++ true
8412023-11-02T05:41:49.875Z+ [[ '' != online ]]
8422023-11-02T05:41:49.902Z+ [[ 9 -gt 60 ]]
8432023-11-02T05:41:49.927Z+ sleep 1
8442023-11-02T05:41:50.762Z+ retry=10
8452023-11-02T05:41:50.810Z+++ zoneadm list -n
8462023-11-02T05:41:50.843Z+++ grep oxz_ntp
8472023-11-02T05:41:50.869Z++ pfexec svcs -z oxz_ntp_7b014f06-174e-4581-90a8-5c25f48af080 -Hostate oxide/ntp
8482023-11-02T05:41:50.900Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
8492023-11-02T05:41:50.926Z++ true
8502023-11-02T05:41:50.954Z+ [[ '' != online ]]
8512023-11-02T05:41:50.983Z+ [[ 10 -gt 60 ]]
8522023-11-02T05:41:51.008Z+ sleep 1
8532023-11-02T05:41:51.777Z+ retry=11
8542023-11-02T05:41:51.802Z+++ zoneadm list -n
8552023-11-02T05:41:51.828Z+++ grep oxz_ntp
8562023-11-02T05:41:51.853Z++ pfexec svcs -z oxz_ntp_7b014f06-174e-4581-90a8-5c25f48af080 -Hostate oxide/ntp
8572023-11-02T05:41:51.878Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
8582023-11-02T05:41:51.908Z++ true
8592023-11-02T05:41:51.934Z+ [[ '' != online ]]
8602023-11-02T05:41:51.960Z+ [[ 11 -gt 60 ]]
8612023-11-02T05:41:51.985Z+ sleep 1
8622023-11-02T05:41:52.807Z+ retry=12
8632023-11-02T05:41:52.833Z+++ zoneadm list -n
8642023-11-02T05:41:52.858Z+++ grep oxz_ntp
8652023-11-02T05:41:52.883Z++ pfexec svcs -z oxz_ntp_7b014f06-174e-4581-90a8-5c25f48af080 -Hostate oxide/ntp
8662023-11-02T05:41:52.908Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
8672023-11-02T05:41:52.934Z++ true
8682023-11-02T05:41:52.959Z+ [[ '' != online ]]
8692023-11-02T05:41:52.986Z+ [[ 12 -gt 60 ]]
8702023-11-02T05:41:53.011Z+ sleep 1
8712023-11-02T05:41:53.837Z+ retry=13
8722023-11-02T05:41:53.863Z+++ zoneadm list -n
8732023-11-02T05:41:53.888Z+++ grep oxz_ntp
8742023-11-02T05:41:53.914Z++ pfexec svcs -z oxz_ntp_7b014f06-174e-4581-90a8-5c25f48af080 -Hostate oxide/ntp
8752023-11-02T05:41:53.940Z+ [[ disabled != online ]]
8762023-11-02T05:41:53.965Z+ [[ 13 -gt 60 ]]
8772023-11-02T05:41:53.990Z+ sleep 1
8782023-11-02T05:41:54.861Z+ retry=14
8792023-11-02T05:41:54.888Z+++ zoneadm list -n
8802023-11-02T05:41:55.327Z+++ grep oxz_ntp
8812023-11-02T05:41:55.353Z++ pfexec svcs -z oxz_ntp_7b014f06-174e-4581-90a8-5c25f48af080 -Hostate oxide/ntp
8822023-11-02T05:41:55.378Z+ [[ disabled != online ]]
8832023-11-02T05:41:55.417Z+ [[ 14 -gt 60 ]]
8842023-11-02T05:41:55.448Z+ sleep 1
8852023-11-02T05:41:55.884Z+ retry=15
8862023-11-02T05:41:55.910Z+++ zoneadm list -n
8872023-11-02T05:41:55.936Z+++ grep oxz_ntp
8882023-11-02T05:41:55.961Z++ pfexec svcs -z oxz_ntp_7b014f06-174e-4581-90a8-5c25f48af080 -Hostate oxide/ntp
8892023-11-02T05:41:55.986Z+ [[ disabled != online ]]
8902023-11-02T05:41:56.011Z+ [[ 15 -gt 60 ]]
8912023-11-02T05:41:56.036Z+ sleep 1
8922023-11-02T05:41:56.907Z+ retry=16
8932023-11-02T05:41:56.932Z+++ zoneadm list -n
8942023-11-02T05:41:56.960Z+++ grep oxz_ntp
8952023-11-02T05:41:56.985Z++ pfexec svcs -z oxz_ntp_7b014f06-174e-4581-90a8-5c25f48af080 -Hostate oxide/ntp
8962023-11-02T05:41:57.012Z+ [[ disabled != online ]]
8972023-11-02T05:41:57.037Z+ [[ 16 -gt 60 ]]
8982023-11-02T05:41:57.062Z+ sleep 1
8992023-11-02T05:41:57.929Z+ retry=17
9002023-11-02T05:41:57.954Z+++ zoneadm list -n
9012023-11-02T05:41:57.979Z+++ grep oxz_ntp
9022023-11-02T05:41:58.005Z++ pfexec svcs -z oxz_ntp_7b014f06-174e-4581-90a8-5c25f48af080 -Hostate oxide/ntp
9032023-11-02T05:41:58.030Z+ [[ disabled != online ]]
9042023-11-02T05:41:58.055Z+ [[ 17 -gt 60 ]]
9052023-11-02T05:41:58.079Z+ sleep 1
9062023-11-02T05:41:58.950Z+ retry=18
9072023-11-02T05:41:58.975Z+++ zoneadm list -n
9082023-11-02T05:41:58.999Z+++ grep oxz_ntp
9092023-11-02T05:41:59.024Z++ pfexec svcs -z oxz_ntp_7b014f06-174e-4581-90a8-5c25f48af080 -Hostate oxide/ntp
9102023-11-02T05:41:59.049Z+ [[ disabled != online ]]
9112023-11-02T05:41:59.073Z+ [[ 18 -gt 60 ]]
9122023-11-02T05:41:59.098Z+ sleep 1
 
9132023-11-02T05:41:59.903Zsock console login: Nov 2 05:41:59 sock ip: ip: joining multicasts failed (4) on vopte0 - will use link layer broadcasts for multicast
 
9142023-11-02T05:41:59.973Z+ retry=19
9152023-11-02T05:41:59.998Z+++ zoneadm list -n
9162023-11-02T05:42:00.023Z+++ grep oxz_ntp
9172023-11-02T05:42:00.048Z++ pfexec svcs -z oxz_ntp_7b014f06-174e-4581-90a8-5c25f48af080 -Hostate oxide/ntp
9182023-11-02T05:42:00.074Z+ [[ online != online ]]
9192023-11-02T05:42:00.100Z+ echo 'Waited for chrony: 19s'
9202023-11-02T05:42:00.126ZWaited for chrony: 19s
9212023-11-02T05:42:00.153Z+ export RUST_BACKTRACE=1
9222023-11-02T05:42:00.182Z+ RUST_BACKTRACE=1
9232023-11-02T05:42:00.208Z+ export E2E_TLS_CERT IPPOOL_START IPPOOL_END
9242023-11-02T05:42:00.233Z+ ./tests/bootstrap
9252023-11-02T05:42:00.257ZUsing external DNS server at 10.151.2.164:53
9262023-11-02T05:42:00.282Z2023-11-02 05:41:59.725216600 UTC: attempting to log into API
9272023-11-02T05:42:15.058Z2023-11-02 05:42:15.124800241 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9282023-11-02T05:42:16.059Z2023-11-02 05:42:16.125807318 UTC: attempting to log into API
9292023-11-02T05:42:31.092Z2023-11-02 05:42:31.158651599 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9302023-11-02T05:42:32.094Z2023-11-02 05:42:32.160515938 UTC: attempting to log into API
9312023-11-02T05:42:47.143Z2023-11-02 05:42:47.209265662 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9322023-11-02T05:42:48.146Z2023-11-02 05:42:48.209878508 UTC: attempting to log into API
9332023-11-02T05:43:03.192Z2023-11-02 05:43:03.250277990 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9342023-11-02T05:43:04.184Z2023-11-02 05:43:04.251249065 UTC: attempting to log into API
 
9352023-11-02T05:43:06.771ZNov 2 05:43:06 sok ip: ip: joining multicasts failed (4) on vopte2 - will use link layer broadcasts for multicast
9362023-11-02T05:43:13.315ZNov 2 05:43:13 sock ip: ip: joining multicasts failed (4) on vopte4 - will use link layer broadcasts for multicast
 
9372023-11-02T05:43:19.218Z2023-11-02 05:43:19.262448378 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9382023-11-02T05:43:20.219Z2023-11-02 05:43:20.261374111 UTC: attempting to log into API
 
9392023-11-02T05:43:21.094ZNov 2 05:43:20 sock ip: ip: joining multicasts failed (4) on vopte5 - will use link layer broadcasts for multicast
9402023-11-02T05:43:26.823ZNov 2 05:43:26 sock ip: ip: joining multicasts failed (4) on vopte1 - will use link layer broadcasts for multicast
9412023-11-02T05:43:32.832ZNov 2 05:43:32 sock ip: ip: joining multicasts failed (4) on vopte3 - will use link layer broadcasts for multicast
 
9422023-11-02T05:43:35.252Z2023-11-02 05:43:35.253536674 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9432023-11-02T05:43:36.670Z2023-11-02 05:43:36.251914522 UTC: attempting to log into API
9442023-11-02T05:43:51.286Z2023-11-02 05:43:51.274352657 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
9452023-11-02T05:43:52.287Z2023-11-02 05:43:52.275204830 UTC: attempting to log into API
9462023-11-02T05:44:03.082Z2023-11-02 05:44:03.070010377 UTC: login succeeded
9472023-11-02T05:44:03.114Zwaiting for nexus to come up...
9482023-11-02T05:44:03.182Zcreating IP pool... 10.151.2.175 - 10.151.2.227
9492023-11-02T05:44:03.494Zensuring datasets are ready...
9502023-11-02T05:44:05.626Zlet's roll.
9512023-11-02T05:44:05.651Z+ rm ./tests/bootstrap
9522023-11-02T05:44:05.711Z+ for test_bin in tests/*
9532023-11-02T05:44:05.736Z+ ./tests/bootstrap-c43fba0fc8ca36e8
9542023-11-02T05:44:05.762Z
9552023-11-02T05:44:05.786Zrunning 0 tests
9562023-11-02T05:44:05.811Z
9572023-11-02T05:44:05.839Ztest result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
9582023-11-02T05:44:05.867Z
9592023-11-02T05:44:05.899Z+ for test_bin in tests/*
9602023-11-02T05:44:05.924Z+ ./tests/end_to_end_tests-db7e95353d866fe2
9612023-11-02T05:44:05.953Z
9622023-11-02T05:44:05.978Zrunning 2 tests
9632023-11-02T05:44:06.002Ztest no_spoof::no_spoof_here ... ok
9642023-11-02T05:45:05.735Ztest instance_launch::instance_launch has been running for over 60 seconds
9652023-11-02T05:47:03.495Ztest instance_launch::instance_launch ... ok
9662023-11-02T05:47:03.520Z
9672023-11-02T05:47:03.546Ztest result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 177.76s
9682023-11-02T05:47:03.571Z
9692023-11-02T05:47:03.597Z+ _exit_trap
9702023-11-02T05:47:03.623Z+ local status=0
9712023-11-02T05:47:03.648Z+ [[ 0 -eq 0 ]]
9722023-11-02T05:47:03.674Z+ exit 0
9732023-11-02T05:47:03.699Zprocess exited: duration 385382 ms, exit code 0
 
9742023-11-02T05:47:03.774Zfound 90 output files
9752023-11-02T05:47:03.907Zuploading: /var/svc/log/oxide-sled-agent:default.log (428068 bytes)
9762023-11-02T05:47:05.127Zuploaded: /var/svc/log/oxide-sled-agent:default.log
9772023-11-02T05:47:05.154Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_9b506cd1-cdef-4154-9cdc-2a1d0174b2f5/root/var/svc/log/oxide-crucible-agent:default.log (0 bytes)
9782023-11-02T05:47:06.199Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_9b506cd1-cdef-4154-9cdc-2a1d0174b2f5/root/var/svc/log/oxide-crucible-agent:default.log
9792023-11-02T05:47:06.224Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_9b506cd1-cdef-4154-9cdc-2a1d0174b2f5/root/var/svc/log/oxide-crucible-agent:default.log.0 (3436 bytes)
9802023-11-02T05:47:07.295Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_9b506cd1-cdef-4154-9cdc-2a1d0174b2f5/root/var/svc/log/oxide-crucible-agent:default.log.0
9812023-11-02T05:47:07.320Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_9b506cd1-cdef-4154-9cdc-2a1d0174b2f5/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
9822023-11-02T05:47:08.367Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_9b506cd1-cdef-4154-9cdc-2a1d0174b2f5/root/var/svc/log/oxide-crucible-downstairs:default.log
9832023-11-02T05:47:08.392Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_9b506cd1-cdef-4154-9cdc-2a1d0174b2f5/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
9842023-11-02T05:47:09.463Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_9b506cd1-cdef-4154-9cdc-2a1d0174b2f5/root/var/svc/log/oxide-crucible-downstairs:default.log.0
9852023-11-02T05:47:09.488Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_94c0c207-1235-4f87-ae36-89043bc40a88/root/var/svc/log/oxide-nexus:default.log (124267 bytes)
9862023-11-02T05:47:10.702Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_94c0c207-1235-4f87-ae36-89043bc40a88/root/var/svc/log/oxide-nexus:default.log
9872023-11-02T05:47:10.738Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_94c0c207-1235-4f87-ae36-89043bc40a88/root/var/svc/log/oxide-nexus:default.log.0 (400240 bytes)
9882023-11-02T05:47:11.998Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_94c0c207-1235-4f87-ae36-89043bc40a88/root/var/svc/log/oxide-nexus:default.log.0
9892023-11-02T05:47:12.025Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_95af46d7-200e-4c43-b4ee-50364a4f7df5/root/var/svc/log/oxide-cockroachdb:default.log (0 bytes)
9902023-11-02T05:47:13.071Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_95af46d7-200e-4c43-b4ee-50364a4f7df5/root/var/svc/log/oxide-cockroachdb:default.log
9912023-11-02T05:47:13.097Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_95af46d7-200e-4c43-b4ee-50364a4f7df5/root/var/svc/log/oxide-cockroachdb:default.log.0 (5369 bytes)
9922023-11-02T05:47:14.181Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_95af46d7-200e-4c43-b4ee-50364a4f7df5/root/var/svc/log/oxide-cockroachdb:default.log.0
9932023-11-02T05:47:14.206Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_c876aed1-f03d-41b4-9ac8-16ba122426e7/root/var/svc/log/oxide-crucible-pantry:default.log (0 bytes)
9942023-11-02T05:47:15.240Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_c876aed1-f03d-41b4-9ac8-16ba122426e7/root/var/svc/log/oxide-crucible-pantry:default.log
9952023-11-02T05:47:15.265Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_c876aed1-f03d-41b4-9ac8-16ba122426e7/root/var/svc/log/oxide-crucible-pantry:default.log.0 (2338 bytes)
9962023-11-02T05:47:16.751Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_c876aed1-f03d-41b4-9ac8-16ba122426e7/root/var/svc/log/oxide-crucible-pantry:default.log.0
9972023-11-02T05:47:16.775Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_oximeter_c57a3af8-27c8-4d02-a526-6d4179390511/root/var/svc/log/oxide-oximeter:default.log (107629 bytes)
9982023-11-02T05:47:16.914Zupload warning: file "/pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_oximeter_c57a3af8-27c8-4d02-a526-6d4179390511/root/var/svc/log/oxide-oximeter:default.log" changed size mid upload: 107629 -> 120998
9992023-11-02T05:47:17.987Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_oximeter_c57a3af8-27c8-4d02-a526-6d4179390511/root/var/svc/log/oxide-oximeter:default.log
10002023-11-02T05:47:18.012Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_oximeter_c57a3af8-27c8-4d02-a526-6d4179390511/root/var/svc/log/oxide-oximeter:default.log.0 (99024 bytes)
10012023-11-02T05:47:19.203Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_oximeter_c57a3af8-27c8-4d02-a526-6d4179390511/root/var/svc/log/oxide-oximeter:default.log.0
10022023-11-02T05:47:19.227Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_pantry_2c936ded-cf9f-46a8-ad9a-9415d1c98fbd/root/var/svc/log/oxide-crucible-pantry:default.log (0 bytes)
10032023-11-02T05:47:20.278Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_pantry_2c936ded-cf9f-46a8-ad9a-9415d1c98fbd/root/var/svc/log/oxide-crucible-pantry:default.log
10042023-11-02T05:47:20.303Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_pantry_2c936ded-cf9f-46a8-ad9a-9415d1c98fbd/root/var/svc/log/oxide-crucible-pantry:default.log.0 (2338 bytes)
10052023-11-02T05:47:21.377Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_pantry_2c936ded-cf9f-46a8-ad9a-9415d1c98fbd/root/var/svc/log/oxide-crucible-pantry:default.log.0
10062023-11-02T05:47:21.401Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_internal_dns_ea4293c7-0df5-477b-9f2e-d597dc993f3d/root/var/svc/log/oxide-internal_dns:default.log (7509 bytes)
10072023-11-02T05:47:22.539Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_internal_dns_ea4293c7-0df5-477b-9f2e-d597dc993f3d/root/var/svc/log/oxide-internal_dns:default.log
10082023-11-02T05:47:22.566Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_internal_dns_ea4293c7-0df5-477b-9f2e-d597dc993f3d/root/var/svc/log/oxide-internal_dns:default.log.0 (107891 bytes)
10092023-11-02T05:47:23.756Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_internal_dns_ea4293c7-0df5-477b-9f2e-d597dc993f3d/root/var/svc/log/oxide-internal_dns:default.log.0
10102023-11-02T05:47:23.781Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_cockroachdb_c0547c5a-b1dd-4615-ba6b-049836b360b7/root/var/svc/log/oxide-cockroachdb:default.log (0 bytes)
10112023-11-02T05:47:24.831Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_cockroachdb_c0547c5a-b1dd-4615-ba6b-049836b360b7/root/var/svc/log/oxide-cockroachdb:default.log
10122023-11-02T05:47:24.856Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_cockroachdb_c0547c5a-b1dd-4615-ba6b-049836b360b7/root/var/svc/log/oxide-cockroachdb:default.log.0 (5368 bytes)
10132023-11-02T05:47:25.933Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_cockroachdb_c0547c5a-b1dd-4615-ba6b-049836b360b7/root/var/svc/log/oxide-cockroachdb:default.log.0
10142023-11-02T05:47:25.958Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_4ac6eff3-d96d-427e-aaa9-6a8d15cbd851/root/var/svc/log/oxide-crucible-agent:default.log (0 bytes)
10152023-11-02T05:47:27.416Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_4ac6eff3-d96d-427e-aaa9-6a8d15cbd851/root/var/svc/log/oxide-crucible-agent:default.log
10162023-11-02T05:47:27.441Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_4ac6eff3-d96d-427e-aaa9-6a8d15cbd851/root/var/svc/log/oxide-crucible-agent:default.log.0 (3436 bytes)
10172023-11-02T05:47:28.513Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_4ac6eff3-d96d-427e-aaa9-6a8d15cbd851/root/var/svc/log/oxide-crucible-agent:default.log.0
10182023-11-02T05:47:28.540Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_4ac6eff3-d96d-427e-aaa9-6a8d15cbd851/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
10192023-11-02T05:47:29.589Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_4ac6eff3-d96d-427e-aaa9-6a8d15cbd851/root/var/svc/log/oxide-crucible-downstairs:default.log
10202023-11-02T05:47:29.614Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_4ac6eff3-d96d-427e-aaa9-6a8d15cbd851/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
10212023-11-02T05:47:30.686Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_4ac6eff3-d96d-427e-aaa9-6a8d15cbd851/root/var/svc/log/oxide-crucible-downstairs:default.log.0
10222023-11-02T05:47:30.710Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_fd352a48-baa9-49c1-8d6c-d986794381ba/root/var/svc/log/oxide-crucible-agent:default.log (4322 bytes)
10232023-11-02T05:47:31.782Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_fd352a48-baa9-49c1-8d6c-d986794381ba/root/var/svc/log/oxide-crucible-agent:default.log
10242023-11-02T05:47:31.806Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_fd352a48-baa9-49c1-8d6c-d986794381ba/root/var/svc/log/oxide-crucible-agent:default.log.0 (9796 bytes)
10252023-11-02T05:47:32.899Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_fd352a48-baa9-49c1-8d6c-d986794381ba/root/var/svc/log/oxide-crucible-agent:default.log.0
10262023-11-02T05:47:32.933Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_fd352a48-baa9-49c1-8d6c-d986794381ba/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
10272023-11-02T05:47:33.972Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_fd352a48-baa9-49c1-8d6c-d986794381ba/root/var/svc/log/oxide-crucible-downstairs:default.log
10282023-11-02T05:47:33.996Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_fd352a48-baa9-49c1-8d6c-d986794381ba/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
10292023-11-02T05:47:35.068Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_fd352a48-baa9-49c1-8d6c-d986794381ba/root/var/svc/log/oxide-crucible-downstairs:default.log.0
10302023-11-02T05:47:35.092Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_fd352a48-baa9-49c1-8d6c-d986794381ba/root/var/svc/log/oxide-crucible-downstairs:downstairs-e95135e8-21c2-403b-b41e-fc941e09476e.log (1006 bytes)
10312023-11-02T05:47:36.165Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_fd352a48-baa9-49c1-8d6c-d986794381ba/root/var/svc/log/oxide-crucible-downstairs:downstairs-e95135e8-21c2-403b-b41e-fc941e09476e.log
10322023-11-02T05:47:36.189Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_fd352a48-baa9-49c1-8d6c-d986794381ba/root/var/svc/log/oxide-crucible-downstairs:downstairs-e95135e8-21c2-403b-b41e-fc941e09476e.log.0 (4531 bytes)
10332023-11-02T05:47:37.267Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_crucible_fd352a48-baa9-49c1-8d6c-d986794381ba/root/var/svc/log/oxide-crucible-downstairs:downstairs-e95135e8-21c2-403b-b41e-fc941e09476e.log.0
10342023-11-02T05:47:37.291Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_internal_dns_c2f9069b-ad09-41be-aead-318a16fbb0f1/root/var/svc/log/oxide-internal_dns:default.log (7511 bytes)
10352023-11-02T05:47:37.337Zupload warning: file "/pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_internal_dns_c2f9069b-ad09-41be-aead-318a16fbb0f1/root/var/svc/log/oxide-internal_dns:default.log" changed size mid upload: 7511 -> 11267
10362023-11-02T05:47:38.410Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_internal_dns_c2f9069b-ad09-41be-aead-318a16fbb0f1/root/var/svc/log/oxide-internal_dns:default.log
10372023-11-02T05:47:38.434Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_internal_dns_c2f9069b-ad09-41be-aead-318a16fbb0f1/root/var/svc/log/oxide-internal_dns:default.log.0 (112949 bytes)
10382023-11-02T05:47:39.622Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_internal_dns_c2f9069b-ad09-41be-aead-318a16fbb0f1/root/var/svc/log/oxide-internal_dns:default.log.0
10392023-11-02T05:47:39.646Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_clickhouse_8a147cbc-2f50-436d-a4c6-5ec4b58e890d/root/var/svc/log/oxide-clickhouse:default.log (397042 bytes)
10402023-11-02T05:47:39.879Zupload warning: file "/pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_clickhouse_8a147cbc-2f50-436d-a4c6-5ec4b58e890d/root/var/svc/log/oxide-clickhouse:default.log" changed size mid upload: 397042 -> 513574
10412023-11-02T05:47:40.954Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_clickhouse_8a147cbc-2f50-436d-a4c6-5ec4b58e890d/root/var/svc/log/oxide-clickhouse:default.log
10422023-11-02T05:47:40.980Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_clickhouse_8a147cbc-2f50-436d-a4c6-5ec4b58e890d/root/var/svc/log/oxide-clickhouse:default.log.0 (396811 bytes)
10432023-11-02T05:47:42.240Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_clickhouse_8a147cbc-2f50-436d-a4c6-5ec4b58e890d/root/var/svc/log/oxide-clickhouse:default.log.0
10442023-11-02T05:47:42.264Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_cockroachdb_2644a4c4-6dfa-4057-8a46-fa8f559a2b4b/root/var/svc/log/oxide-cockroachdb:default.log (0 bytes)
10452023-11-02T05:47:43.317Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_cockroachdb_2644a4c4-6dfa-4057-8a46-fa8f559a2b4b/root/var/svc/log/oxide-cockroachdb:default.log
10462023-11-02T05:47:43.342Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_cockroachdb_2644a4c4-6dfa-4057-8a46-fa8f559a2b4b/root/var/svc/log/oxide-cockroachdb:default.log.0 (5343 bytes)
10472023-11-02T05:47:44.425Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_cockroachdb_2644a4c4-6dfa-4057-8a46-fa8f559a2b4b/root/var/svc/log/oxide-cockroachdb:default.log.0
10482023-11-02T05:47:44.451Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_cockroachdb_59cd2f05-2752-4ddd-b8a9-219fd9f53ed1/root/var/svc/log/oxide-cockroachdb:default.log (0 bytes)
10492023-11-02T05:47:45.500Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_cockroachdb_59cd2f05-2752-4ddd-b8a9-219fd9f53ed1/root/var/svc/log/oxide-cockroachdb:default.log
10502023-11-02T05:47:45.526Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_cockroachdb_59cd2f05-2752-4ddd-b8a9-219fd9f53ed1/root/var/svc/log/oxide-cockroachdb:default.log.0 (5368 bytes)
10512023-11-02T05:47:46.597Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_cockroachdb_59cd2f05-2752-4ddd-b8a9-219fd9f53ed1/root/var/svc/log/oxide-cockroachdb:default.log.0
10522023-11-02T05:47:46.626Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_external_dns_0cd35555-4c09-44c6-ad6a-222e76578ffc/root/var/svc/log/oxide-external_dns:default.log (7552 bytes)
10532023-11-02T05:47:46.673Zupload warning: file "/pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_external_dns_0cd35555-4c09-44c6-ad6a-222e76578ffc/root/var/svc/log/oxide-external_dns:default.log" changed size mid upload: 7552 -> 11330
10542023-11-02T05:47:47.744Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_external_dns_0cd35555-4c09-44c6-ad6a-222e76578ffc/root/var/svc/log/oxide-external_dns:default.log
10552023-11-02T05:47:47.769Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_external_dns_0cd35555-4c09-44c6-ad6a-222e76578ffc/root/var/svc/log/oxide-external_dns:default.log.0 (8059 bytes)
10562023-11-02T05:47:48.864Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_external_dns_0cd35555-4c09-44c6-ad6a-222e76578ffc/root/var/svc/log/oxide-external_dns:default.log.0
10572023-11-02T05:47:48.889Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_external_dns_e8dd1883-fc3c-4c38-ab63-de18c275a7ca/root/var/svc/log/oxide-external_dns:default.log (7552 bytes)
10582023-11-02T05:47:48.935Zupload warning: file "/pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_external_dns_e8dd1883-fc3c-4c38-ab63-de18c275a7ca/root/var/svc/log/oxide-external_dns:default.log" changed size mid upload: 7552 -> 11325
10592023-11-02T05:47:50.010Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_external_dns_e8dd1883-fc3c-4c38-ab63-de18c275a7ca/root/var/svc/log/oxide-external_dns:default.log
10602023-11-02T05:47:50.036Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_external_dns_e8dd1883-fc3c-4c38-ab63-de18c275a7ca/root/var/svc/log/oxide-external_dns:default.log.0 (8060 bytes)
10612023-11-02T05:47:51.132Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_external_dns_e8dd1883-fc3c-4c38-ab63-de18c275a7ca/root/var/svc/log/oxide-external_dns:default.log.0
10622023-11-02T05:47:51.157Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_cockroachdb_a242c005-82c5-412b-8a36-ebacf1694e89/root/var/svc/log/oxide-cockroachdb:default.log (0 bytes)
10632023-11-02T05:47:52.208Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_cockroachdb_a242c005-82c5-412b-8a36-ebacf1694e89/root/var/svc/log/oxide-cockroachdb:default.log
10642023-11-02T05:47:52.261Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_cockroachdb_a242c005-82c5-412b-8a36-ebacf1694e89/root/var/svc/log/oxide-cockroachdb:default.log.0 (5369 bytes)
10652023-11-02T05:47:53.335Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_cockroachdb_a242c005-82c5-412b-8a36-ebacf1694e89/root/var/svc/log/oxide-cockroachdb:default.log.0
10662023-11-02T05:47:53.362Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_a58ba6e3-61ba-4003-a5dd-d613e703be99/root/var/svc/log/oxide-crucible-agent:default.log (0 bytes)
10672023-11-02T05:47:54.412Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_a58ba6e3-61ba-4003-a5dd-d613e703be99/root/var/svc/log/oxide-crucible-agent:default.log
10682023-11-02T05:47:54.436Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_a58ba6e3-61ba-4003-a5dd-d613e703be99/root/var/svc/log/oxide-crucible-agent:default.log.0 (14113 bytes)
10692023-11-02T05:47:55.531Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_a58ba6e3-61ba-4003-a5dd-d613e703be99/root/var/svc/log/oxide-crucible-agent:default.log.0
10702023-11-02T05:47:55.558Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_a58ba6e3-61ba-4003-a5dd-d613e703be99/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
10712023-11-02T05:47:56.602Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_a58ba6e3-61ba-4003-a5dd-d613e703be99/root/var/svc/log/oxide-crucible-downstairs:default.log
10722023-11-02T05:47:56.626Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_a58ba6e3-61ba-4003-a5dd-d613e703be99/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
10732023-11-02T05:47:57.699Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_a58ba6e3-61ba-4003-a5dd-d613e703be99/root/var/svc/log/oxide-crucible-downstairs:default.log.0
10742023-11-02T05:47:57.723Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_a58ba6e3-61ba-4003-a5dd-d613e703be99/root/var/svc/log/oxide-crucible-downstairs:downstairs-e6998842-b421-45b3-96f2-8a9b765245b1.log (0 bytes)
10752023-11-02T05:47:58.785Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_a58ba6e3-61ba-4003-a5dd-d613e703be99/root/var/svc/log/oxide-crucible-downstairs:downstairs-e6998842-b421-45b3-96f2-8a9b765245b1.log
10762023-11-02T05:47:58.811Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_a58ba6e3-61ba-4003-a5dd-d613e703be99/root/var/svc/log/oxide-crucible-downstairs:downstairs-e6998842-b421-45b3-96f2-8a9b765245b1.log.0 (3049 bytes)
10772023-11-02T05:47:59.883Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_a58ba6e3-61ba-4003-a5dd-d613e703be99/root/var/svc/log/oxide-crucible-downstairs:downstairs-e6998842-b421-45b3-96f2-8a9b765245b1.log.0
10782023-11-02T05:47:59.911Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_ccc300c3-2655-4230-b589-0b70686d4a2b/root/var/svc/log/oxide-crucible-agent:default.log (0 bytes)
10792023-11-02T05:48:00.955Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_ccc300c3-2655-4230-b589-0b70686d4a2b/root/var/svc/log/oxide-crucible-agent:default.log
10802023-11-02T05:48:00.980Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_ccc300c3-2655-4230-b589-0b70686d4a2b/root/var/svc/log/oxide-crucible-agent:default.log.0 (14580 bytes)
10812023-11-02T05:48:02.080Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_ccc300c3-2655-4230-b589-0b70686d4a2b/root/var/svc/log/oxide-crucible-agent:default.log.0
10822023-11-02T05:48:02.110Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_ccc300c3-2655-4230-b589-0b70686d4a2b/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
10832023-11-02T05:48:03.153Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_ccc300c3-2655-4230-b589-0b70686d4a2b/root/var/svc/log/oxide-crucible-downstairs:default.log
10842023-11-02T05:48:03.178Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_ccc300c3-2655-4230-b589-0b70686d4a2b/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
10852023-11-02T05:48:04.269Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_ccc300c3-2655-4230-b589-0b70686d4a2b/root/var/svc/log/oxide-crucible-downstairs:default.log.0
10862023-11-02T05:48:04.296Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_ccc300c3-2655-4230-b589-0b70686d4a2b/root/var/svc/log/oxide-crucible-downstairs:downstairs-c6c0b4bd-2265-4269-954c-8cac64428714.log (0 bytes)
10872023-11-02T05:48:05.343Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_ccc300c3-2655-4230-b589-0b70686d4a2b/root/var/svc/log/oxide-crucible-downstairs:downstairs-c6c0b4bd-2265-4269-954c-8cac64428714.log
10882023-11-02T05:48:05.367Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_ccc300c3-2655-4230-b589-0b70686d4a2b/root/var/svc/log/oxide-crucible-downstairs:downstairs-c6c0b4bd-2265-4269-954c-8cac64428714.log.0 (3051 bytes)
10892023-11-02T05:48:06.439Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_ccc300c3-2655-4230-b589-0b70686d4a2b/root/var/svc/log/oxide-crucible-downstairs:downstairs-c6c0b4bd-2265-4269-954c-8cac64428714.log.0
10902023-11-02T05:48:06.463Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_f798b0b5-279c-4f7e-bf5e-2219ba53135a/root/var/svc/log/oxide-crucible-agent:default.log (4784 bytes)
10912023-11-02T05:48:07.538Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_f798b0b5-279c-4f7e-bf5e-2219ba53135a/root/var/svc/log/oxide-crucible-agent:default.log
10922023-11-02T05:48:07.563Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_f798b0b5-279c-4f7e-bf5e-2219ba53135a/root/var/svc/log/oxide-crucible-agent:default.log.0 (20043 bytes)
10932023-11-02T05:48:08.681Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_f798b0b5-279c-4f7e-bf5e-2219ba53135a/root/var/svc/log/oxide-crucible-agent:default.log.0
10942023-11-02T05:48:08.706Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_f798b0b5-279c-4f7e-bf5e-2219ba53135a/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
10952023-11-02T05:48:09.753Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_f798b0b5-279c-4f7e-bf5e-2219ba53135a/root/var/svc/log/oxide-crucible-downstairs:default.log
10962023-11-02T05:48:09.778Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_f798b0b5-279c-4f7e-bf5e-2219ba53135a/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
10972023-11-02T05:48:10.854Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_f798b0b5-279c-4f7e-bf5e-2219ba53135a/root/var/svc/log/oxide-crucible-downstairs:default.log.0
10982023-11-02T05:48:10.879Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_f798b0b5-279c-4f7e-bf5e-2219ba53135a/root/var/svc/log/oxide-crucible-downstairs:downstairs-63f0eed1-2131-433b-8b49-579fab78a16b.log (1006 bytes)
10992023-11-02T05:48:11.950Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_f798b0b5-279c-4f7e-bf5e-2219ba53135a/root/var/svc/log/oxide-crucible-downstairs:downstairs-63f0eed1-2131-433b-8b49-579fab78a16b.log
11002023-11-02T05:48:11.974Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_f798b0b5-279c-4f7e-bf5e-2219ba53135a/root/var/svc/log/oxide-crucible-downstairs:downstairs-63f0eed1-2131-433b-8b49-579fab78a16b.log.0 (4530 bytes)
11012023-11-02T05:48:13.045Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_f798b0b5-279c-4f7e-bf5e-2219ba53135a/root/var/svc/log/oxide-crucible-downstairs:downstairs-63f0eed1-2131-433b-8b49-579fab78a16b.log.0
11022023-11-02T05:48:13.072Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_f798b0b5-279c-4f7e-bf5e-2219ba53135a/root/var/svc/log/oxide-crucible-downstairs:downstairs-bf14f32f-bf87-4536-9785-2772d2968ea8.log (0 bytes)
11032023-11-02T05:48:14.116Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_f798b0b5-279c-4f7e-bf5e-2219ba53135a/root/var/svc/log/oxide-crucible-downstairs:downstairs-bf14f32f-bf87-4536-9785-2772d2968ea8.log
11042023-11-02T05:48:14.141Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_f798b0b5-279c-4f7e-bf5e-2219ba53135a/root/var/svc/log/oxide-crucible-downstairs:downstairs-bf14f32f-bf87-4536-9785-2772d2968ea8.log.0 (3048 bytes)
11052023-11-02T05:48:15.262Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_f798b0b5-279c-4f7e-bf5e-2219ba53135a/root/var/svc/log/oxide-crucible-downstairs:downstairs-bf14f32f-bf87-4536-9785-2772d2968ea8.log.0
11062023-11-02T05:48:15.288Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_nexus_55b029ac-878e-49b2-9b82-9d48e005ad6f/root/var/svc/log/oxide-nexus:default.log (124249 bytes)
11072023-11-02T05:48:15.470Zupload warning: file "/pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_nexus_55b029ac-878e-49b2-9b82-9d48e005ad6f/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 124249 -> 186689
11082023-11-02T05:48:16.541Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_nexus_55b029ac-878e-49b2-9b82-9d48e005ad6f/root/var/svc/log/oxide-nexus:default.log
11092023-11-02T05:48:16.567Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_nexus_55b029ac-878e-49b2-9b82-9d48e005ad6f/root/var/svc/log/oxide-nexus:default.log.0 (481264 bytes)
11102023-11-02T05:48:17.844Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_nexus_55b029ac-878e-49b2-9b82-9d48e005ad6f/root/var/svc/log/oxide-nexus:default.log.0
11112023-11-02T05:48:17.871Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_ntp_7b014f06-174e-4581-90a8-5c25f48af080/root/var/svc/log/oxide-ntp:default.log (0 bytes)
11122023-11-02T05:48:18.915Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_ntp_7b014f06-174e-4581-90a8-5c25f48af080/root/var/svc/log/oxide-ntp:default.log
11132023-11-02T05:48:18.940Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_ntp_7b014f06-174e-4581-90a8-5c25f48af080/root/var/svc/log/oxide-ntp:default.log.0 (1119 bytes)
11142023-11-02T05:48:20.012Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_ntp_7b014f06-174e-4581-90a8-5c25f48af080/root/var/svc/log/oxide-ntp:default.log.0
11152023-11-02T05:48:20.036Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_498d5ed7-987b-4af8-8c32-38b0752b2d93/root/var/svc/log/oxide-internal_dns:default.log (17589 bytes)
11162023-11-02T05:48:20.107Zupload warning: file "/pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_498d5ed7-987b-4af8-8c32-38b0752b2d93/root/var/svc/log/oxide-internal_dns:default.log" changed size mid upload: 17589 -> 21343
11172023-11-02T05:48:21.178Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_498d5ed7-987b-4af8-8c32-38b0752b2d93/root/var/svc/log/oxide-internal_dns:default.log
11182023-11-02T05:48:21.203Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_498d5ed7-987b-4af8-8c32-38b0752b2d93/root/var/svc/log/oxide-internal_dns:default.log.0 (97096 bytes)
11192023-11-02T05:48:22.418Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_498d5ed7-987b-4af8-8c32-38b0752b2d93/root/var/svc/log/oxide-internal_dns:default.log.0
11202023-11-02T05:48:22.444Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_263f7591-d1a2-4aca-92cb-a0be7abd5abe/root/var/svc/log/oxide-nexus:default.log (1413909 bytes)
11212023-11-02T05:48:23.167Zupload warning: file "/pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_263f7591-d1a2-4aca-92cb-a0be7abd5abe/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 1413909 -> 1476954
11222023-11-02T05:48:24.240Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_263f7591-d1a2-4aca-92cb-a0be7abd5abe/root/var/svc/log/oxide-nexus:default.log
11232023-11-02T05:48:24.272Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_263f7591-d1a2-4aca-92cb-a0be7abd5abe/root/var/svc/log/oxide-nexus:default.log.0 (3357747 bytes)
11242023-11-02T05:48:26.896Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_263f7591-d1a2-4aca-92cb-a0be7abd5abe/root/var/svc/log/oxide-nexus:default.log.0
11252023-11-02T05:48:26.921Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_1bb26d81-f618-4bcc-ae03-ee25f44096d0/root/var/svc/log/oxide-crucible-agent:default.log (4788 bytes)
11262023-11-02T05:48:27.993Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_1bb26d81-f618-4bcc-ae03-ee25f44096d0/root/var/svc/log/oxide-crucible-agent:default.log
11272023-11-02T05:48:28.018Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_1bb26d81-f618-4bcc-ae03-ee25f44096d0/root/var/svc/log/oxide-crucible-agent:default.log.0 (9794 bytes)
11282023-11-02T05:48:29.114Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_1bb26d81-f618-4bcc-ae03-ee25f44096d0/root/var/svc/log/oxide-crucible-agent:default.log.0
11292023-11-02T05:48:29.138Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_1bb26d81-f618-4bcc-ae03-ee25f44096d0/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
11302023-11-02T05:48:30.187Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_1bb26d81-f618-4bcc-ae03-ee25f44096d0/root/var/svc/log/oxide-crucible-downstairs:default.log
11312023-11-02T05:48:30.213Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_1bb26d81-f618-4bcc-ae03-ee25f44096d0/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
11322023-11-02T05:48:31.284Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_1bb26d81-f618-4bcc-ae03-ee25f44096d0/root/var/svc/log/oxide-crucible-downstairs:default.log.0
11332023-11-02T05:48:31.308Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_1bb26d81-f618-4bcc-ae03-ee25f44096d0/root/var/svc/log/oxide-crucible-downstairs:downstairs-a1034974-04a4-447a-8c3b-f005bd43cc33.log (1006 bytes)
11342023-11-02T05:48:32.379Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_1bb26d81-f618-4bcc-ae03-ee25f44096d0/root/var/svc/log/oxide-crucible-downstairs:downstairs-a1034974-04a4-447a-8c3b-f005bd43cc33.log
11352023-11-02T05:48:32.404Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_1bb26d81-f618-4bcc-ae03-ee25f44096d0/root/var/svc/log/oxide-crucible-downstairs:downstairs-a1034974-04a4-447a-8c3b-f005bd43cc33.log.0 (4529 bytes)
11362023-11-02T05:48:33.476Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_1bb26d81-f618-4bcc-ae03-ee25f44096d0/root/var/svc/log/oxide-crucible-downstairs:downstairs-a1034974-04a4-447a-8c3b-f005bd43cc33.log.0
11372023-11-02T05:48:33.502Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_a1048330-7a29-4ebd-bdb4-34bafadb1822/root/var/svc/log/oxide-crucible-agent:default.log (0 bytes)
11382023-11-02T05:48:34.552Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_a1048330-7a29-4ebd-bdb4-34bafadb1822/root/var/svc/log/oxide-crucible-agent:default.log
11392023-11-02T05:48:34.577Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_a1048330-7a29-4ebd-bdb4-34bafadb1822/root/var/svc/log/oxide-crucible-agent:default.log.0 (3433 bytes)
11402023-11-02T05:48:35.647Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_a1048330-7a29-4ebd-bdb4-34bafadb1822/root/var/svc/log/oxide-crucible-agent:default.log.0
11412023-11-02T05:48:35.672Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_a1048330-7a29-4ebd-bdb4-34bafadb1822/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
11422023-11-02T05:48:36.718Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_a1048330-7a29-4ebd-bdb4-34bafadb1822/root/var/svc/log/oxide-crucible-downstairs:default.log
11432023-11-02T05:48:36.751Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_a1048330-7a29-4ebd-bdb4-34bafadb1822/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
11442023-11-02T05:48:37.823Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_a1048330-7a29-4ebd-bdb4-34bafadb1822/root/var/svc/log/oxide-crucible-downstairs:default.log.0
11452023-11-02T05:48:37.849Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_cf753767-589f-401a-b808-6d8dfdb84b9a/root/var/svc/log/oxide-crucible-agent:default.log (0 bytes)
11462023-11-02T05:48:38.893Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_cf753767-589f-401a-b808-6d8dfdb84b9a/root/var/svc/log/oxide-crucible-agent:default.log
11472023-11-02T05:48:38.918Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_cf753767-589f-401a-b808-6d8dfdb84b9a/root/var/svc/log/oxide-crucible-agent:default.log.0 (3434 bytes)
11482023-11-02T05:48:39.991Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_cf753767-589f-401a-b808-6d8dfdb84b9a/root/var/svc/log/oxide-crucible-agent:default.log.0
11492023-11-02T05:48:40.017Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_cf753767-589f-401a-b808-6d8dfdb84b9a/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
11502023-11-02T05:48:41.062Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_cf753767-589f-401a-b808-6d8dfdb84b9a/root/var/svc/log/oxide-crucible-downstairs:default.log
11512023-11-02T05:48:41.087Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_cf753767-589f-401a-b808-6d8dfdb84b9a/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
11522023-11-02T05:48:42.158Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_cf753767-589f-401a-b808-6d8dfdb84b9a/root/var/svc/log/oxide-crucible-downstairs:default.log.0
11532023-11-02T05:48:42.183Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_1875a38d-7185-4ee6-ba04-4ff3d67568e7/root/var/svc/log/oxide-crucible-pantry:default.log (0 bytes)
11542023-11-02T05:48:43.232Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_1875a38d-7185-4ee6-ba04-4ff3d67568e7/root/var/svc/log/oxide-crucible-pantry:default.log
11552023-11-02T05:48:43.257Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_1875a38d-7185-4ee6-ba04-4ff3d67568e7/root/var/svc/log/oxide-crucible-pantry:default.log.0 (2333 bytes)
11562023-11-02T05:48:44.330Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_1875a38d-7185-4ee6-ba04-4ff3d67568e7/root/var/svc/log/oxide-crucible-pantry:default.log.0
11572023-11-02T05:48:44.354Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_ntp_7b014f06-174e-4581-90a8-5c25f48af080/root/var/log/chrony/measurements.log (8494 bytes)
11582023-11-02T05:48:44.402Zupload warning: file "/pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_ntp_7b014f06-174e-4581-90a8-5c25f48af080/root/var/log/chrony/measurements.log" changed size mid upload: 8494 -> 10686
11592023-11-02T05:48:45.475Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_ntp_7b014f06-174e-4581-90a8-5c25f48af080/root/var/log/chrony/measurements.log
11602023-11-02T05:48:45.501Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_ntp_7b014f06-174e-4581-90a8-5c25f48af080/root/var/log/chrony/statistics.log (3861 bytes)
11612023-11-02T05:48:45.525Zupload warning: file "/pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_ntp_7b014f06-174e-4581-90a8-5c25f48af080/root/var/log/chrony/statistics.log" changed size mid upload: 3861 -> 5733
11622023-11-02T05:48:46.598Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_ntp_7b014f06-174e-4581-90a8-5c25f48af080/root/var/log/chrony/statistics.log
11632023-11-02T05:48:46.622Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_ntp_7b014f06-174e-4581-90a8-5c25f48af080/root/var/log/chrony/tracking.log (1457 bytes)
11642023-11-02T05:48:46.647Zupload warning: file "/pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_ntp_7b014f06-174e-4581-90a8-5c25f48af080/root/var/log/chrony/tracking.log" changed size mid upload: 1457 -> 1721
11652023-11-02T05:48:47.720Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_ntp_7b014f06-174e-4581-90a8-5c25f48af080/root/var/log/chrony/tracking.log