01HHGTSZ6R0C05ZQMKW1C5GD10: helios / deploy

BasicConfig {
    output_rules: [
        "%/var/svc/log/oxide-sled-agent:default.log*",
        "%/pool/ext/*/crypt/zone/oxz_*/root/var/svc/log/oxide-*.log*",
        "%/pool/ext/*/crypt/zone/oxz_*/root/var/svc/log/system-illumos-*.log*",
        "%/pool/ext/*/crypt/zone/oxz_ntp_*/root/var/log/chrony/*.log*",
        "!/pool/ext/*/crypt/zone/oxz_propolis-server_*/root/var/svc/log/*.log*",
        "%/pool/ext/*/crypt/debug/global/oxide-sled-agent:default.log.*",
        "%/pool/ext/*/crypt/debug/oxz_*/oxide-*.log.*",
        "%/pool/ext/*/crypt/debug/oxz_*/system-illumos-*.log.*",
        "!/pool/ext/*/crypt/debug/oxz_propolis-server_*/*.log.*",
    ],
    rust_toolchain: None,
    target: Some(
        "lab-2.0-opte-0.27",
    ),
    access_repos: [],
    publish: [],
    skip_clone: true,
}

Buildomat Job: 01HHGTTY0DRQ7Q96VECNWBG8T5

Tags:

Artefacts:

Output:

SEQ GLOBAL TIME DETAILS
12023-12-13T06:24:58.437Zjob dependencies complete; ready to run (waiting for 48 m 10 s)
22023-12-13T06:28:42.050Zjob assigned to worker 01HHGXKB726C6Z2F15FN7VN03M (queued for 3 m 43 s)
32023-12-13T06:28:42.100Zhttp://172.20.0.5:9969/ipxe/sock... ok
42023-12-13T06:28:42.350Zxcvsh0tfvd xcvsh0tfvd xcvsh0tfvd xcvsh0tfvd xcvsh0tfvd xcvsh0tfvd xcvsh0tfvd xcvsh0tfvd
52023-12-13T06:28:42.462Zhttp://172.20.0.5:9969/ipxe/sock... ok
62023-12-13T06:28:42.517Zhttp://172.20.0.5:9969/os/sock/platform/i86pc/kernel/amd64/unix... ok
72023-12-13T06:28:42.970Zhttp://172.20.0.5:9969/os/sock/platform/i86pc/amd64/boot_archive... ok
82023-12-13T06:28:43.018Zhttp://172.20.0.5:9969/os/sock/platform/i86pc/amd64/boot_archive.hash... ok
92023-12-13T06:28:43.084Zhttp://172.20.0.5:9969/postboot/sock... ok
102023-12-13T06:28:43.220ZwcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U
112023-12-13T06:28:43.264Zcpu0: microcode has been updated from version 0xa001133 to 0xa0011d1
122023-12-13T06:28:43.315ZOxide Helios Version helios-2.0.22333 64-bit
132023-12-13T06:28:43.347ZConfiguring devices.
142023-12-13T06:28:43.383Z##################################################
152023-12-13T06:28:43.424Z#### ############# ##
162023-12-13T06:28:43.460Z### ### ############ ##
172023-12-13T06:28:43.507Z## ### # ## ### ## Oxide Computer Company ##
182023-12-13T06:28:43.551Z## ## ## ### # ### ##
192023-12-13T06:28:43.594Z## # ### #### #### This Station Under ##
202023-12-13T06:28:43.640Z### ### #### # ### Computer Control ##
212023-12-13T06:28:43.691Z#### #### ### ## ##
222023-12-13T06:28:43.736Z##################################################
232023-12-13T06:28:43.778ZHostname: sock
242023-12-13T06:28:43.811ZLoading smf(7) service descriptions:
252023-12-13T06:28:44.376Zdownloading input: /input/ci-tools/work/end-to-end-tests/bootstrap-b98d9b3a65a4fd47.gz
262023-12-13T06:28:46.311Zdownloaded input: /input/ci-tools/work/end-to-end-tests/bootstrap-b98d9b3a65a4fd47.gz
272023-12-13T06:28:46.312Zdownloading input: /input/ci-tools/work/end-to-end-tests/bootstrap.gz
282023-12-13T06:29:05.999Zdownloaded input: /input/ci-tools/work/end-to-end-tests/bootstrap.gz
292023-12-13T06:29:06.025Zdownloading input: /input/ci-tools/work/end-to-end-tests/end_to_end_tests-e373b32ce9650dc1.gz
302023-12-13T06:29:31.986Zdownloaded input: /input/ci-tools/work/end-to-end-tests/end_to_end_tests-e373b32ce9650dc1.gz
312023-12-13T06:29:32.015Zdownloading input: /input/ci-tools/work/caboose-util.gz
322023-12-13T06:29:33.403Zdownloaded input: /input/ci-tools/work/caboose-util.gz
332023-12-13T06:29:33.404Zdownloading input: /input/ci-tools/work/tufaceous.gz
342023-12-13T06:29:36.487Zdownloaded input: /input/ci-tools/work/tufaceous.gz
352023-12-13T06:29:36.512Zdownloading input: /input/package/work/version.txt
362023-12-13T06:29:36.537Zdownloaded input: /input/package/work/version.txt
372023-12-13T06:29:36.538Zdownloading input: /input/package/work/package.tar.gz
382023-12-13T06:29:51.174Zdownloaded input: /input/package/work/package.tar.gz
392023-12-13T06:29:51.174Zdownloading input: /input/package/work/global-zone-packages.tar.gz
402023-12-13T06:30:14.109Zdownloaded input: /input/package/work/global-zone-packages.tar.gz
412023-12-13T06:30:14.109Zdownloading input: /input/package/work/trampoline-global-zone-packages.tar.gz
422023-12-13T06:30:20.845Zdownloaded input: /input/package/work/trampoline-global-zone-packages.tar.gz
432023-12-13T06:30:20.869Zdownloading input: /input/package/work/zones/clickhouse.tar.gz
442023-12-13T06:30:55.028Zdownloaded input: /input/package/work/zones/clickhouse.tar.gz
452023-12-13T06:30:55.053Zdownloading input: /input/package/work/zones/clickhouse_keeper.tar.gz
462023-12-13T06:31:29.987Zdownloaded input: /input/package/work/zones/clickhouse_keeper.tar.gz
472023-12-13T06:31:30.012Zdownloading input: /input/package/work/zones/cockroachdb.tar.gz
482023-12-13T06:31:51.985Zdownloaded input: /input/package/work/zones/cockroachdb.tar.gz
492023-12-13T06:31:52.010Zdownloading input: /input/package/work/zones/crucible-pantry.tar.gz
502023-12-13T06:31:56.222Zdownloaded input: /input/package/work/zones/crucible-pantry.tar.gz
512023-12-13T06:31:56.222Zdownloading input: /input/package/work/zones/crucible.tar.gz
522023-12-13T06:32:03.581Zdownloaded input: /input/package/work/zones/crucible.tar.gz
532023-12-13T06:32:03.581Zdownloading input: /input/package/work/zones/external-dns.tar.gz
542023-12-13T06:32:09.272Zdownloaded input: /input/package/work/zones/external-dns.tar.gz
552023-12-13T06:32:09.298Zdownloading input: /input/package/work/zones/internal-dns.tar.gz
562023-12-13T06:32:13.527Zdownloaded input: /input/package/work/zones/internal-dns.tar.gz
572023-12-13T06:32:13.551Zdownloading input: /input/package/work/zones/ntp.tar.gz
582023-12-13T06:32:13.575Zdownloaded input: /input/package/work/zones/ntp.tar.gz
592023-12-13T06:32:13.575Zdownloading input: /input/package/work/zones/omicron-gateway-asic.tar.gz
602023-12-13T06:32:17.115Zdownloaded input: /input/package/work/zones/omicron-gateway-asic.tar.gz
612023-12-13T06:32:17.115Zdownloading input: /input/package/work/zones/omicron-gateway-softnpu.tar.gz
622023-12-13T06:32:20.799Zdownloaded input: /input/package/work/zones/omicron-gateway-softnpu.tar.gz
632023-12-13T06:32:20.799Zdownloading input: /input/package/work/zones/omicron-nexus-single-sled.tar.gz
642023-12-13T06:32:34.991Zdownloaded input: /input/package/work/zones/omicron-nexus-single-sled.tar.gz
652023-12-13T06:32:35.015Zdownloading input: /input/package/work/zones/omicron-nexus.tar.gz
662023-12-13T06:32:48.597Zdownloaded input: /input/package/work/zones/omicron-nexus.tar.gz
672023-12-13T06:32:48.598Zdownloading input: /input/package/work/zones/overlay.tar.gz
682023-12-13T06:32:48.639Zdownloaded input: /input/package/work/zones/overlay.tar.gz
692023-12-13T06:32:48.640Zdownloading input: /input/package/work/zones/oximeter-collector.tar.gz
702023-12-13T06:32:55.187Zdownloaded input: /input/package/work/zones/oximeter-collector.tar.gz
712023-12-13T06:32:55.188Zdownloading input: /input/package/work/zones/propolis-server.tar.gz
722023-12-13T06:33:08.657Zdownloaded input: /input/package/work/zones/propolis-server.tar.gz
732023-12-13T06:33:08.657Zdownloading input: /input/package/work/zones/switch-asic.tar.gz
742023-12-13T06:33:53.608Zdownloaded input: /input/package/work/zones/switch-asic.tar.gz
752023-12-13T06:33:53.608Zdownloading input: /input/package/work/zones/switch-softnpu.tar.gz
762023-12-13T06:34:48.666Zdownloaded input: /input/package/work/zones/switch-softnpu.tar.gz
 
772023-12-13T06:34:48.666Zstarting task 0: "setup"
782023-12-13T06:34:48.694Z++ uname -s
792023-12-13T06:34:48.694Z+ kern=SunOS
802023-12-13T06:34:48.694Z+ case "$kern" in
812023-12-13T06:34:48.695Z+ groupadd -g 12345 build
822023-12-13T06:34:48.695Z+ useradd -u 12345 -g build -d /home/build -s /bin/bash -c build -P 'Primary Administrator' build
832023-12-13T06:34:50.685Z+ zfs create -o mountpoint=/work rpool/work
842023-12-13T06:34:50.730Z++ awk '$2 == "/home" { print $3 }' /etc/mnttab
852023-12-13T06:34:50.758Z+ home_fs=zfs
862023-12-13T06:34:50.758Z+ [[ zfs == autofs ]]
872023-12-13T06:34:50.758Z+ mkdir -p /home/build
882023-12-13T06:34:50.758Z+ chown build:build /home/build /work
892023-12-13T06:34:52.739Z+ chmod 0700 /home/build /work
902023-12-13T06:34:52.765Zprocess exited: duration 4076 ms, exit code 0
 
912023-12-13T06:34:52.815Zstarting task 1: "authentication"
922023-12-13T06:34:52.859Zprocess exited: duration 45 ms, exit code 0
 
932023-12-13T06:34:52.908Zstarting task 2: "build"
942023-12-13T06:34:52.932Z+ trap _exit_trap EXIT
952023-12-13T06:34:52.932Z+ pfexec mdb -kwe 'secpolicy_fs_linkdir/v 55 48 89 e5 b8 01 00 00 00 5d c3'
962023-12-13T06:34:53.115Zsecpolicy_fs_linkdir: 0x55 = 0x55
972023-12-13T06:34:53.116Zsecpolicy_fs_linkdir+1: 0xb8 = 0x48
982023-12-13T06:34:53.116Zsecpolicy_fs_linkdir+2: 0x1 = 0x89
992023-12-13T06:34:53.116Zsecpolicy_fs_linkdir+3: 0 = 0xe5
1002023-12-13T06:34:53.116Zsecpolicy_fs_linkdir+4: 0 = 0xb8
1012023-12-13T06:34:53.117Zsecpolicy_fs_linkdir+5: 0 = 0x1
1022023-12-13T06:34:53.141Zsecpolicy_fs_linkdir+6: 0x48 = 0x0
1032023-12-13T06:34:53.142Zsecpolicy_fs_linkdir+7: 0x89 = 0x0
1042023-12-13T06:34:53.142Zsecpolicy_fs_linkdir+8: 0xe5 = 0x0
1052023-12-13T06:34:53.142Zsecpolicy_fs_linkdir+9: 0x48 = 0x5d
1062023-12-13T06:34:53.142Zsecpolicy_fs_linkdir+0xa: 0x83 = 0xc3
1072023-12-13T06:34:53.142Z+ [[ -d /opt/oxide ]]
1082023-12-13T06:34:53.142Z+ pfexec mv /opt/oxide /opt/oxide-underneath
1092023-12-13T06:34:53.142Z+ pfexec mkdir /opt/oxide
1102023-12-13T06:34:53.142Z+ pfexec mount -F tmpfs -O swap /opt/oxide
1112023-12-13T06:34:53.142Z+ [[ -d /opt/oxide-underneath ]]
1122023-12-13T06:34:53.142Z+ cd /opt/oxide-underneath
1132023-12-13T06:34:53.142Z+ pfexec tar ceEp@/f - .
1142023-12-13T06:34:53.142Z+ cd /opt/oxide
1152023-12-13T06:34:53.142Z+ pfexec tar xveEp@/f -
1162023-12-13T06:34:53.246Zx ., 0 bytes, 0 tape blocks
1172023-12-13T06:34:53.246Zx ./opte, 0 bytes, 0 tape blocks
1182023-12-13T06:34:53.246Zx ./opte/bin, 0 bytes, 0 tape blocks
1192023-12-13T06:34:53.246Zx ./opte/bin/opteadm, 52567720 bytes, 102672 tape blocks
1202023-12-13T06:34:53.271Z+ pfexec rm -rf /opt/oxide-underneath
1212023-12-13T06:34:53.271Z+ pfexec /sbin/zfs create -o mountpoint=/zone rpool/zone
1222023-12-13T06:34:53.308Z+ pfexec mkdir /opt/oxide/work
1232023-12-13T06:34:53.308Z+ pfexec chown build:build /opt/oxide/work
1242023-12-13T06:34:53.309Z+ cd /opt/oxide/work
1252023-12-13T06:34:53.309Z+ ptime -m tar xvzf /input/package/work/package.tar.gz
1262023-12-13T06:34:54.825ZDecompressing '/input/package/work/package.tar.gz' with '/usr/bin/gzcat'...
1272023-12-13T06:34:54.825Zx out/mg-ddm-gz.tar, 62723072 bytes, 122506 tape blocks
1282023-12-13T06:34:54.825Zx out/omicron-sled-agent.tar, 111883264 bytes, 218522 tape blocks
1292023-12-13T06:34:54.825Zx out/target/test, 75 bytes, 1 tape blocks
1302023-12-13T06:34:54.825Zx out/npuzone/npuzone, 37184104 bytes, 72626 tape blocks
1312023-12-13T06:34:54.825Zx package-manifest.toml, 22143 bytes, 44 tape blocks
1322023-12-13T06:34:54.825Zx smf/sled-agent/non-gimlet/config.toml, 3448 bytes, 7 tape blocks
1332023-12-13T06:34:54.825Zx target/release/omicron-package, 42517232 bytes, 83042 tape blocks
1342023-12-13T06:34:54.825Zx tools/create_virtual_hardware.sh, 3176 bytes, 7 tape blocks
1352023-12-13T06:34:54.825Zx tools/virtual_hardware.sh, 3408 bytes, 7 tape blocks
1362023-12-13T06:34:54.825Zx tools/scrimlet/create-softnpu-zone.sh, 619 bytes, 2 tape blocks
1372023-12-13T06:34:54.825Zx tools/scrimlet/destroy-softnpu-zone.sh, 195 bytes, 1 tape blocks
1382023-12-13T06:34:54.825Zx tools/scrimlet/softnpu-init.sh, 2148 bytes, 5 tape blocks
1392023-12-13T06:34:54.825Zx tools/scrimlet/softnpu-zone.txt, 330 bytes, 1 tape blocks
1402023-12-13T06:34:54.825Zx tools/scrimlet/softnpu.toml, 171 bytes, 1 tape blocks
1412023-12-13T06:34:55.276Z
1422023-12-13T06:34:55.276Zreal 1.540944971
1432023-12-13T06:34:55.276Zuser 1.385450048
1442023-12-13T06:34:55.276Zsys 0.270526278
1452023-12-13T06:34:55.276Ztrap 0.000058760
1462023-12-13T06:34:55.276Ztflt 0.000000000
1472023-12-13T06:34:55.276Zdflt 0.000000000
1482023-12-13T06:34:55.276Zkflt 0.000000000
1492023-12-13T06:34:55.276Zlock 0.000000000
1502023-12-13T06:34:55.277Zslp 1.387684744
1512023-12-13T06:34:55.277Zlat 0.036148135
1522023-12-13T06:34:55.277Zstop 0.000124104
1532023-12-13T06:34:55.277Z+ cp /input/package/work/zones/clickhouse_keeper.tar.gz /input/package/work/zones/clickhouse.tar.gz /input/package/work/zones/cockroachdb.tar.gz /input/package/work/zones/crucible-pantry.tar.gz /input/package/work/zones/crucible.tar.gz /input/package/work/zones/external-dns.tar.gz /input/package/work/zones/internal-dns.tar.gz /input/package/work/zones/ntp.tar.gz /input/package/work/zones/omicron-gateway-asic.tar.gz /input/package/work/zones/omicron-gateway-softnpu.tar.gz /input/package/work/zones/omicron-nexus-single-sled.tar.gz /input/package/work/zones/omicron-nexus.tar.gz /input/package/work/zones/overlay.tar.gz /input/package/work/zones/oximeter-collector.tar.gz /input/package/work/zones/propolis-server.tar.gz /input/package/work/zones/switch-asic.tar.gz /input/package/work/zones/switch-softnpu.tar.gz out/
1542023-12-13T06:34:55.928Z+ mv out/omicron-nexus-single-sled.tar.gz out/omicron-nexus.tar.gz
1552023-12-13T06:34:55.952Z+ mkdir tests
1562023-12-13T06:34:55.952Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1572023-12-13T06:34:55.952Z+ ptime -m gunzip
1582023-12-13T06:34:55.952Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap-b98d9b3a65a4fd47
1592023-12-13T06:34:56.101Z
1602023-12-13T06:34:56.127Zreal 0.140123388
1612023-12-13T06:34:56.127Zuser 0.129657640
1622023-12-13T06:34:56.127Zsys 0.010234529
1632023-12-13T06:34:56.127Ztrap 0.000010990
1642023-12-13T06:34:56.127Ztflt 0.000000000
1652023-12-13T06:34:56.127Zdflt 0.000000000
1662023-12-13T06:34:56.127Zkflt 0.000000000
1672023-12-13T06:34:56.127Zlock 0.000000000
1682023-12-13T06:34:56.127Zslp 0.000019780
1692023-12-13T06:34:56.127Zlat 0.000020000
1702023-12-13T06:34:56.127Zstop 0.000114774
1712023-12-13T06:34:56.127Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap-b98d9b3a65a4fd47
1722023-12-13T06:34:56.127Z+ chmod a+x tests/bootstrap-b98d9b3a65a4fd47
1732023-12-13T06:34:56.127Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1742023-12-13T06:34:56.127Z+ ptime -m gunzip
1752023-12-13T06:34:56.127Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap
1762023-12-13T06:34:58.352Z
1772023-12-13T06:34:58.399Zreal 2.247799442
1782023-12-13T06:34:58.399Zuser 2.117984635
1792023-12-13T06:34:58.399Zsys 0.129397165
1802023-12-13T06:34:58.400Ztrap 0.000191596
1812023-12-13T06:34:58.400Ztflt 0.000000000
1822023-12-13T06:34:58.400Zdflt 0.000000000
1832023-12-13T06:34:58.400Zkflt 0.000000000
1842023-12-13T06:34:58.401Zlock 0.000000000
1852023-12-13T06:34:58.401Zslp 0.000012060
1862023-12-13T06:34:58.401Zlat 0.000031310
1872023-12-13T06:34:58.401Zstop 0.000127504
1882023-12-13T06:34:58.401Z++ basename /input/ci-tools/work/end-to-end-tests/bootstrap
1892023-12-13T06:34:58.401Z+ chmod a+x tests/bootstrap
1902023-12-13T06:34:58.401Z+ for p in /input/ci-tools/work/end-to-end-tests/*.gz
1912023-12-13T06:34:58.401Z+ ptime -m gunzip
1922023-12-13T06:34:58.401Z++ basename /input/ci-tools/work/end-to-end-tests/end_to_end_tests-e373b32ce9650dc1
1932023-12-13T06:35:00.664Z
1942023-12-13T06:35:00.690Zreal 2.299898727
1952023-12-13T06:35:00.690Zuser 2.157214394
1962023-12-13T06:35:00.690Zsys 0.142281822
1972023-12-13T06:35:00.690Ztrap 0.000180544
1982023-12-13T06:35:00.690Ztflt 0.000000000
1992023-12-13T06:35:00.690Zdflt 0.000000000
2002023-12-13T06:35:00.690Zkflt 0.000000000
2012023-12-13T06:35:00.690Zlock 0.000000000
2022023-12-13T06:35:00.690Zslp 0.000021290
2032023-12-13T06:35:00.691Zlat 0.000020421
2042023-12-13T06:35:00.691Zstop 0.000118223
2052023-12-13T06:35:00.691Z++ basename /input/ci-tools/work/end-to-end-tests/end_to_end_tests-e373b32ce9650dc1
2062023-12-13T06:35:00.691Z+ chmod a+x tests/end_to_end_tests-e373b32ce9650dc1
2072023-12-13T06:35:00.691Z+ bmat address ls
2082023-12-13T06:35:00.691ZNAME CIDR FIRST LAST COUNT
2092023-12-13T06:35:00.691Zextra 10.151.2.0/24 10.151.2.164 10.151.2.227 64
2102023-12-13T06:35:00.691Z++ bmat address ls -f extra -Ho first,last,count
2112023-12-13T06:35:00.717Z+ set -- 10.151.2.164 10.151.2.227 64
2122023-12-13T06:35:00.717Z+ EXTRA_IP_START=10.151.2.164
2132023-12-13T06:35:00.717Z+ EXTRA_IP_END=10.151.2.227
2142023-12-13T06:35:00.717Z+ EXTRA_IP_COUNT=64
2152023-12-13T06:35:00.717Z+ (( EXTRA_IP_COUNT >= 32 ))
2162023-12-13T06:35:00.717Z+ EXTRA_IP_BASE=10.151.2
2172023-12-13T06:35:00.717Z+ EXTRA_IP_FOCTET=164
2182023-12-13T06:35:00.718Z+ EXTRA_IP_LOCTET=227
2192023-12-13T06:35:00.718Z+ SERVICE_IP_POOL_START=10.151.2.164
2202023-12-13T06:35:00.718Z+ SERVICE_IP_POOL_END=10.151.2.173
2212023-12-13T06:35:00.718Z+ DNS_IP1=10.151.2.164
2222023-12-13T06:35:00.718Z+ DNS_IP2=10.151.2.165
2232023-12-13T06:35:00.718Z+ UPLINK_IP=10.151.2.174
2242023-12-13T06:35:00.718Z+ IPPOOL_START=10.151.2.175
2252023-12-13T06:35:00.718Z+ IPPOOL_END=10.151.2.227
2262023-12-13T06:35:00.718Z++ ipadm show-addr -po type,addr
2272023-12-13T06:35:00.718Z++ awk '-F[:/]' '$1 == "dhcp" {print $2}'
2282023-12-13T06:35:00.718Z+ GATEWAY_IP=10.151.2.100
2292023-12-13T06:35:00.718Z+ [[ -n 10.151.2.100 ]]
2302023-12-13T06:35:00.718Z+ ping -s 10.151.2.100 56 1
2312023-12-13T06:35:00.718ZPING 10.151.2.100: 56 data bytes
2322023-12-13T06:35:00.718Z64 bytes from 10.151.2.100: icmp_seq=0. time=0.011 ms
2332023-12-13T06:35:00.718Z
2342023-12-13T06:35:00.718Z----10.151.2.100 PING Statistics----
2352023-12-13T06:35:00.718Z1 packets transmitted, 1 packets received, 0% packet loss
2362023-12-13T06:35:00.718Zround-trip (ms) min/avg/max/stddev = 0.011/0.011/0.011/-nan
2372023-12-13T06:35:00.718Z++ arp -an
2382023-12-13T06:35:00.718Z++ awk -vgw=10.151.2.100 '$2 == gw {print $NF}'
2392023-12-13T06:35:00.743Z+ GATEWAY_MAC=18:c0:4d:0d:a0:2a
2402023-12-13T06:35:00.743Z+ routeadm -e ipv4-forwarding -u
2412023-12-13T06:35:00.828Z+ PXA_START=10.151.2.164
2422023-12-13T06:35:00.828Z+ PXA_END=10.151.2.227
2432023-12-13T06:35:00.853Z+ export GATEWAY_IP GATEWAY_MAC PXA_START PXA_END
2442023-12-13T06:35:00.853Z+ pfexec zpool create -f scratch c1t1d0 c2t1d0
2452023-12-13T06:35:00.976Z+ ZPOOL_VDEV_DIR=/scratch
2462023-12-13T06:35:00.976Z+ ptime -m pfexec ./tools/create_virtual_hardware.sh
2472023-12-13T06:35:01.000Z+++ dirname ./tools/create_virtual_hardware.sh
2482023-12-13T06:35:01.000Z++ cd ./tools
2492023-12-13T06:35:01.000Z++ pwd
2502023-12-13T06:35:01.000Z+ SOURCE_DIR=/opt/oxide/work/tools
2512023-12-13T06:35:01.000Z+ OMICRON_TOP=/opt/oxide/work/tools/..
2522023-12-13T06:35:01.000Z+ SOFTNPU_MODE=zone
2532023-12-13T06:35:01.001Z+ . /opt/oxide/work/tools/virtual_hardware.sh
2542023-12-13T06:35:01.001Z++ MARKER=/etc/opt/oxide/NO_INSTALL
2552023-12-13T06:35:01.001Z++ [[ -f /etc/opt/oxide/NO_INSTALL ]]
2562023-12-13T06:35:01.001Z++ dladm show-phys -p -o LINK
2572023-12-13T06:35:01.001Z++ head -1
2582023-12-13T06:35:01.058Z+ PHYSICAL_LINK=igb0
2592023-12-13T06:35:01.058Z+ echo 'Using igb0 as physical link'
2602023-12-13T06:35:01.083ZUsing igb0 as physical link
2612023-12-13T06:35:01.083Z+ ensure_run_as_root
2622023-12-13T06:35:01.083Z++ id -u
2632023-12-13T06:35:01.083Z+ [[ 0 -ne 0 ]]
2642023-12-13T06:35:01.083Z+ ensure_zpools
2652023-12-13T06:35:01.083Z+ ZPOOL_TYPES=('oxp_' 'oxi_')
2662023-12-13T06:35:01.083Z+ for ZPOOL_TYPE in "${ZPOOL_TYPES[@]}"
2672023-12-13T06:35:01.083Z+ readarray -t ZPOOLS
2682023-12-13T06:35:01.083Z++ grep '"oxp_' /opt/oxide/work/tools/../smf/sled-agent/non-gimlet/config.toml
2692023-12-13T06:35:01.083Z++ sed 's/[ ",]//g'
2702023-12-13T06:35:01.083Z+ for ZPOOL in "${ZPOOLS[@]}"
2712023-12-13T06:35:01.083Z+ echo 'Zpool: [oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b]'
2722023-12-13T06:35:01.083ZZpool: [oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b]
2732023-12-13T06:35:01.083Z+ VDEV_PATH=/scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
2742023-12-13T06:35:01.083Z+ [[ -f /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
2752023-12-13T06:35:01.083Z+ dd if=/dev/zero of=/scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
2762023-12-13T06:35:01.108Z0+0 records in
2772023-12-13T06:35:01.108Z0+0 records out
2782023-12-13T06:35:01.109Z0 bytes (0 iB) transferred in 0.000006 secs (0 iB/sec)
2792023-12-13T06:35:01.109Z+ success 'ZFS vdev /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
2802023-12-13T06:35:01.109Z+ set +x
2812023-12-13T06:35:01.109ZZFS vdev /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
2822023-12-13T06:35:01.109Z++ zpool list -o name
2832023-12-13T06:35:01.109Z++ grep oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b
2842023-12-13T06:35:01.109Z+ [[ -z '' ]]
2852023-12-13T06:35:01.109Z+ zpool create -f oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
2862023-12-13T06:35:01.151Z+ success 'ZFS zpool oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
2872023-12-13T06:35:01.175Z+ set +x
2882023-12-13T06:35:01.175ZZFS zpool oxp_d462a7f7-b628-40fe-80ff-4e4189e2d62b exists
2892023-12-13T06:35:01.175Z+ for ZPOOL in "${ZPOOLS[@]}"
2902023-12-13T06:35:01.175Z+ echo 'Zpool: [oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03]'
2912023-12-13T06:35:01.175ZZpool: [oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03]
2922023-12-13T06:35:01.175Z+ VDEV_PATH=/scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
2932023-12-13T06:35:01.176Z+ [[ -f /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
2942023-12-13T06:35:01.176Z+ dd if=/dev/zero of=/scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
2952023-12-13T06:35:01.176Z0+0 records in
2962023-12-13T06:35:01.176Z0+0 records out
2972023-12-13T06:35:01.176Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
2982023-12-13T06:35:01.176Z+ success 'ZFS vdev /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
2992023-12-13T06:35:01.176Z+ set +x
3002023-12-13T06:35:01.176ZZFS vdev /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
3012023-12-13T06:35:01.176Z++ zpool list -o name
3022023-12-13T06:35:01.176Z++ grep oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03
3032023-12-13T06:35:01.176Z+ [[ -z '' ]]
3042023-12-13T06:35:01.176Z+ zpool create -f oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3052023-12-13T06:35:01.200Z+ success 'ZFS zpool oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3062023-12-13T06:35:01.200Z+ set +x
3072023-12-13T06:35:01.200ZZFS zpool oxp_e4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
3082023-12-13T06:35:01.200Z+ for ZPOOL in "${ZPOOLS[@]}"
3092023-12-13T06:35:01.200Z+ echo 'Zpool: [oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3102023-12-13T06:35:01.200ZZpool: [oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03]
3112023-12-13T06:35:01.200Z+ VDEV_PATH=/scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3122023-12-13T06:35:01.200Z+ [[ -f /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3132023-12-13T06:35:01.201Z+ dd if=/dev/zero of=/scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3142023-12-13T06:35:01.201Z0+0 records in
3152023-12-13T06:35:01.201Z0+0 records out
3162023-12-13T06:35:01.201Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3172023-12-13T06:35:01.201Z+ success 'ZFS vdev /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3182023-12-13T06:35:01.201Z+ set +x
3192023-12-13T06:35:01.201ZZFS vdev /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
3202023-12-13T06:35:01.225Z++ zpool list -o name
3212023-12-13T06:35:01.226Z++ grep oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03
3222023-12-13T06:35:01.226Z+ [[ -z '' ]]
3232023-12-13T06:35:01.226Z+ zpool create -f oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3242023-12-13T06:35:01.250Z+ success 'ZFS zpool oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3252023-12-13T06:35:01.250Z+ set +x
3262023-12-13T06:35:01.250ZZFS zpool oxp_f4b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
3272023-12-13T06:35:01.250Z+ for ZPOOL in "${ZPOOLS[@]}"
3282023-12-13T06:35:01.251Z+ echo 'Zpool: [oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3292023-12-13T06:35:01.251ZZpool: [oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03]
3302023-12-13T06:35:01.251Z+ VDEV_PATH=/scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3312023-12-13T06:35:01.251Z+ [[ -f /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3322023-12-13T06:35:01.251Z+ dd if=/dev/zero of=/scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3332023-12-13T06:35:01.251Z0+0 records in
3342023-12-13T06:35:01.251Z0+0 records out
3352023-12-13T06:35:01.251Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3362023-12-13T06:35:01.251Z+ success 'ZFS vdev /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3372023-12-13T06:35:01.251Z+ set +x
3382023-12-13T06:35:01.251ZZFS vdev /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
3392023-12-13T06:35:01.275Z++ zpool list -o name
3402023-12-13T06:35:01.278Z++ grep oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03
3412023-12-13T06:35:01.278Z+ [[ -z '' ]]
3422023-12-13T06:35:01.278Z+ zpool create -f oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3432023-12-13T06:35:01.303Z+ success 'ZFS zpool oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3442023-12-13T06:35:01.303Z+ set +x
3452023-12-13T06:35:01.303ZZFS zpool oxp_14b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
3462023-12-13T06:35:01.303Z+ for ZPOOL in "${ZPOOLS[@]}"
3472023-12-13T06:35:01.303Z+ echo 'Zpool: [oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03]'
3482023-12-13T06:35:01.303ZZpool: [oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03]
3492023-12-13T06:35:01.303Z+ VDEV_PATH=/scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3502023-12-13T06:35:01.303Z+ [[ -f /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev ]]
3512023-12-13T06:35:01.303Z+ dd if=/dev/zero of=/scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev bs=1 count=0 seek=20G
3522023-12-13T06:35:01.303Z0+0 records in
3532023-12-13T06:35:01.303Z0+0 records out
3542023-12-13T06:35:01.303Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
3552023-12-13T06:35:01.303Z+ success 'ZFS vdev /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists'
3562023-12-13T06:35:01.303Z+ set +x
3572023-12-13T06:35:01.303ZZFS vdev /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev exists
3582023-12-13T06:35:01.303Z++ zpool list -o name
3592023-12-13T06:35:01.303Z++ grep oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03
3602023-12-13T06:35:01.328Z+ [[ -z '' ]]
3612023-12-13T06:35:01.328Z+ zpool create -f oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 /scratch/oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03.vdev
3622023-12-13T06:35:01.352Z+ success 'ZFS zpool oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 exists'
3632023-12-13T06:35:01.353Z+ set +x
3642023-12-13T06:35:01.353ZZFS zpool oxp_24b4dc87-ab46-49fb-a4b4-d361ae214c03 exists
3652023-12-13T06:35:01.353Z+ for ZPOOL in "${ZPOOLS[@]}"
3662023-12-13T06:35:01.353Z+ echo 'Zpool: [oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930]'
3672023-12-13T06:35:01.353ZZpool: [oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930]
3682023-12-13T06:35:01.353Z+ VDEV_PATH=/scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev
3692023-12-13T06:35:01.354Z+ [[ -f /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev ]]
3702023-12-13T06:35:01.354Z+ dd if=/dev/zero of=/scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev bs=1 count=0 seek=20G
3712023-12-13T06:35:01.354Z0+0 records in
3722023-12-13T06:35:01.354Z0+0 records out
3732023-12-13T06:35:01.354Z0 bytes (0 iB) transferred in 0.000005 secs (0 iB/sec)
3742023-12-13T06:35:01.354Z+ success 'ZFS vdev /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev exists'
3752023-12-13T06:35:01.354Z+ set +x
3762023-12-13T06:35:01.354ZZFS vdev /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev exists
3772023-12-13T06:35:01.354Z++ zpool list -o name
3782023-12-13T06:35:01.354Z++ grep oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930
3792023-12-13T06:35:01.380Z+ [[ -z '' ]]
3802023-12-13T06:35:01.380Z+ zpool create -f oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 /scratch/oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930.vdev
3812023-12-13T06:35:01.404Z+ success 'ZFS zpool oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 exists'
3822023-12-13T06:35:01.405Z+ set +x
3832023-12-13T06:35:01.405ZZFS zpool oxp_cd70d7f6-2354-4bf2-8012-55bf9eaf7930 exists
3842023-12-13T06:35:01.405Z+ for ZPOOL in "${ZPOOLS[@]}"
3852023-12-13T06:35:01.405Z+ echo 'Zpool: [oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60]'
3862023-12-13T06:35:01.405ZZpool: [oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60]
3872023-12-13T06:35:01.405Z+ VDEV_PATH=/scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev
3882023-12-13T06:35:01.405Z+ [[ -f /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev ]]
3892023-12-13T06:35:01.406Z+ dd if=/dev/zero of=/scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev bs=1 count=0 seek=20G
3902023-12-13T06:35:01.406Z0+0 records in
3912023-12-13T06:35:01.406Z0+0 records out
3922023-12-13T06:35:01.406Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
3932023-12-13T06:35:01.406Z+ success 'ZFS vdev /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev exists'
3942023-12-13T06:35:01.406Z+ set +x
3952023-12-13T06:35:01.406ZZFS vdev /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev exists
3962023-12-13T06:35:01.406Z++ zpool list -o name
3972023-12-13T06:35:01.406Z++ grep oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60
3982023-12-13T06:35:01.436Z+ [[ -z '' ]]
3992023-12-13T06:35:01.436Z+ zpool create -f oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 /scratch/oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60.vdev
4002023-12-13T06:35:01.492Z+ success 'ZFS zpool oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 exists'
4012023-12-13T06:35:01.493Z+ set +x
4022023-12-13T06:35:01.493ZZFS zpool oxp_ceb4461c-cf56-4719-ad3c-14430bfdfb60 exists
4032023-12-13T06:35:01.493Z+ for ZPOOL in "${ZPOOLS[@]}"
4042023-12-13T06:35:01.493Z+ echo 'Zpool: [oxp_31bd71cd-4736-4a12-a387-9b74b050396f]'
4052023-12-13T06:35:01.493ZZpool: [oxp_31bd71cd-4736-4a12-a387-9b74b050396f]
4062023-12-13T06:35:01.493Z+ VDEV_PATH=/scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev
4072023-12-13T06:35:01.493Z+ [[ -f /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev ]]
4082023-12-13T06:35:01.493Z+ dd if=/dev/zero of=/scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev bs=1 count=0 seek=20G
4092023-12-13T06:35:01.493Z0+0 records in
4102023-12-13T06:35:01.493Z0+0 records out
4112023-12-13T06:35:01.493Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
4122023-12-13T06:35:01.493Z+ success 'ZFS vdev /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev exists'
4132023-12-13T06:35:01.493Z+ set +x
4142023-12-13T06:35:01.493ZZFS vdev /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev exists
4152023-12-13T06:35:01.493Z++ zpool list -o name
4162023-12-13T06:35:01.493Z++ grep oxp_31bd71cd-4736-4a12-a387-9b74b050396f
4172023-12-13T06:35:01.518Z+ [[ -z '' ]]
4182023-12-13T06:35:01.518Z+ zpool create -f oxp_31bd71cd-4736-4a12-a387-9b74b050396f /scratch/oxp_31bd71cd-4736-4a12-a387-9b74b050396f.vdev
4192023-12-13T06:35:01.518Z+ success 'ZFS zpool oxp_31bd71cd-4736-4a12-a387-9b74b050396f exists'
4202023-12-13T06:35:01.518Z+ set +x
4212023-12-13T06:35:01.518ZZFS zpool oxp_31bd71cd-4736-4a12-a387-9b74b050396f exists
4222023-12-13T06:35:01.519Z+ for ZPOOL in "${ZPOOLS[@]}"
4232023-12-13T06:35:01.519Z+ echo 'Zpool: [oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7]'
4242023-12-13T06:35:01.519ZZpool: [oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7]
4252023-12-13T06:35:01.519Z+ VDEV_PATH=/scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev
4262023-12-13T06:35:01.519Z+ [[ -f /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev ]]
4272023-12-13T06:35:01.519Z+ dd if=/dev/zero of=/scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev bs=1 count=0 seek=20G
4282023-12-13T06:35:01.519Z0+0 records in
4292023-12-13T06:35:01.519Z0+0 records out
4302023-12-13T06:35:01.519Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4312023-12-13T06:35:01.519Z+ success 'ZFS vdev /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev exists'
4322023-12-13T06:35:01.519Z+ set +x
4332023-12-13T06:35:01.519ZZFS vdev /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev exists
4342023-12-13T06:35:01.519Z++ zpool list -o name
4352023-12-13T06:35:01.519Z++ grep oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7
4362023-12-13T06:35:01.519Z+ [[ -z '' ]]
4372023-12-13T06:35:01.519Z+ zpool create -f oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 /scratch/oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7.vdev
4382023-12-13T06:35:01.549Z+ success 'ZFS zpool oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 exists'
4392023-12-13T06:35:01.549Z+ set +x
4402023-12-13T06:35:01.575ZZFS zpool oxp_616b26df-e62a-4c68-b506-f4a923d8aaf7 exists
4412023-12-13T06:35:01.575Z+ for ZPOOL_TYPE in "${ZPOOL_TYPES[@]}"
4422023-12-13T06:35:01.575Z+ readarray -t ZPOOLS
4432023-12-13T06:35:01.575Z++ grep '"oxi_' /opt/oxide/work/tools/../smf/sled-agent/non-gimlet/config.toml
4442023-12-13T06:35:01.575Z++ sed 's/[ ",]//g'
4452023-12-13T06:35:01.575Z+ for ZPOOL in "${ZPOOLS[@]}"
4462023-12-13T06:35:01.576Z+ echo 'Zpool: [oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b]'
4472023-12-13T06:35:01.576ZZpool: [oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b]
4482023-12-13T06:35:01.576Z+ VDEV_PATH=/scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4492023-12-13T06:35:01.576Z+ [[ -f /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
4502023-12-13T06:35:01.576Z+ dd if=/dev/zero of=/scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
4512023-12-13T06:35:01.576Z0+0 records in
4522023-12-13T06:35:01.576Z0+0 records out
4532023-12-13T06:35:01.576Z0 bytes (0 iB) transferred in 0.000003 secs (0 iB/sec)
4542023-12-13T06:35:01.576Z+ success 'ZFS vdev /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
4552023-12-13T06:35:01.576Z+ set +x
4562023-12-13T06:35:01.576ZZFS vdev /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
4572023-12-13T06:35:01.576Z++ zpool list -o name
4582023-12-13T06:35:01.576Z++ grep oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b
4592023-12-13T06:35:01.576Z+ [[ -z '' ]]
4602023-12-13T06:35:01.576Z+ zpool create -f oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4612023-12-13T06:35:01.609Z+ success 'ZFS zpool oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
4622023-12-13T06:35:01.635Z+ set +x
4632023-12-13T06:35:01.635ZZFS zpool oxi_a462a7f7-b628-40fe-80ff-4e4189e2d62b exists
4642023-12-13T06:35:01.635Z+ for ZPOOL in "${ZPOOLS[@]}"
4652023-12-13T06:35:01.635ZZpool: [oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b]
4662023-12-13T06:35:01.635Z+ echo 'Zpool: [oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b]'
4672023-12-13T06:35:01.635Z+ VDEV_PATH=/scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4682023-12-13T06:35:01.635Z+ [[ -f /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev ]]
4692023-12-13T06:35:01.635Z+ dd if=/dev/zero of=/scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev bs=1 count=0 seek=20G
4702023-12-13T06:35:01.635Z0+0 records in
4712023-12-13T06:35:01.635Z0+0 records out
4722023-12-13T06:35:01.636Z0 bytes (0 iB) transferred in 0.000004 secs (0 iB/sec)
4732023-12-13T06:35:01.636Z+ success 'ZFS vdev /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists'
4742023-12-13T06:35:01.636Z+ set +x
4752023-12-13T06:35:01.636ZZFS vdev /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev exists
4762023-12-13T06:35:01.636Z++ zpool list -o name
4772023-12-13T06:35:01.637Z++ grep oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b
4782023-12-13T06:35:01.637Z+ [[ -z '' ]]
4792023-12-13T06:35:01.637Z+ zpool create -f oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b /scratch/oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b.vdev
4802023-12-13T06:35:01.662Z+ success 'ZFS zpool oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b exists'
4812023-12-13T06:35:01.662Z+ set +x
4822023-12-13T06:35:01.687ZZFS zpool oxi_b462a7f7-b628-40fe-80ff-4e4189e2d62b exists
4832023-12-13T06:35:01.687Z+ [[ zone == \z\o\n\e ]]
4842023-12-13T06:35:01.687Z+ ensure_simulated_links igb0
4852023-12-13T06:35:01.687Z+ local PHYSICAL_LINK=igb0
4862023-12-13T06:35:01.687Z+ INDICES=("0" "1")
4872023-12-13T06:35:01.687Z+ for I in "${INDICES[@]}"
4882023-12-13T06:35:01.687Z++ get_simnet_name_if_exists net0
4892023-12-13T06:35:01.687Z++ dladm show-simnet -p -o LINK net0
4902023-12-13T06:35:01.687Z++ echo ''
4912023-12-13T06:35:01.687Z+ [[ -z '' ]]
4922023-12-13T06:35:01.687Z+ dladm create-simnet -t net0
4932023-12-13T06:35:01.687Z+ dladm create-simnet -t sc0_0
4942023-12-13T06:35:01.687Z+ dladm modify-simnet -t -p net0 sc0_0
4952023-12-13T06:35:01.687Z+ dladm set-linkprop -p mtu=9000 sc0_0
4962023-12-13T06:35:01.687Z+ success 'Simnet net0/sc0_0 exists'
4972023-12-13T06:35:01.687Z+ set +x
4982023-12-13T06:35:01.687ZSimnet net0/sc0_0 exists
4992023-12-13T06:35:01.687Z+ for I in "${INDICES[@]}"
5002023-12-13T06:35:01.712Z++ get_simnet_name_if_exists net1
5012023-12-13T06:35:01.712Z++ dladm show-simnet -p -o LINK net1
5022023-12-13T06:35:01.712Z++ echo ''
5032023-12-13T06:35:01.712Z+ [[ -z '' ]]
5042023-12-13T06:35:01.712Z+ dladm create-simnet -t net1
5052023-12-13T06:35:01.712Z+ dladm create-simnet -t sc1_0
5062023-12-13T06:35:01.712Z+ dladm modify-simnet -t -p net1 sc1_0
5072023-12-13T06:35:01.712Z+ dladm set-linkprop -p mtu=9000 sc1_0
5082023-12-13T06:35:01.712Z+ success 'Simnet net1/sc1_0 exists'
5092023-12-13T06:35:01.713Z+ set +x
5102023-12-13T06:35:01.713ZSimnet net1/sc1_0 exists
5112023-12-13T06:35:01.713Z++ get_vnic_name_if_exists sc0_1
5122023-12-13T06:35:01.713Z++ dladm show-vnic -p -o LINK sc0_1
5132023-12-13T06:35:01.738Z++ echo ''
5142023-12-13T06:35:01.738Z+ [[ -z '' ]]
5152023-12-13T06:35:01.738Z+ dladm create-vnic -t sc0_1 -l igb0 -m a8:e1:de:01:70:1d
5162023-12-13T06:35:01.738Z+ [[ -v PROMISC_FILT_OFF ]]
5172023-12-13T06:35:01.738Z+ success 'Vnic sc0_1 exists'
5182023-12-13T06:35:01.738Z+ set +x
5192023-12-13T06:35:01.738ZVnic sc0_1 exists
5202023-12-13T06:35:01.738Z+ warn_if_no_proxy_arp
5212023-12-13T06:35:01.738Z+ [[ -v PXA_START ]]
5222023-12-13T06:35:01.738Z+ [[ -v PXA_END ]]
5232023-12-13T06:35:01.738Z+ ensure_softnpu_zone
5242023-12-13T06:35:01.738Z+ zoneadm list
5252023-12-13T06:35:01.738Z+ grep -q sidecar_softnpu
5262023-12-13T06:35:01.738Z+ [[ -f out/npuzone/npuzone ]]
5272023-12-13T06:35:01.738Z+ out/npuzone/npuzone create sidecar --omicron-zone --ports sc0_0,tfportrear0_0 --ports sc0_1,tfportqsfp0_0 --sidecar-lite-commit 45ed98fea5824feb4d42f45bbf218e597dc9fc58 --softnpu-commit dbab082dfa89da5db5ca2325c257089d2f130092
5282023-12-13T06:35:01.763Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/softnpu/image/dbab082dfa89da5db5ca2325c257089d2f130092/softnpu.sha256.txt
5292023-12-13T06:35:07.158Zdownloading softnpu ...done
5302023-12-13T06:35:07.191Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/sidecar-lite/release/45ed98fea5824feb4d42f45bbf218e597dc9fc58/libsidecar_lite.so.sha256.txt
5312023-12-13T06:35:12.152Zdownloading asic_program.so ...done
5322023-12-13T06:35:12.178Zremote url: https://buildomat.eng.oxide.computer/public/file/oxidecomputer/sidecar-lite/release/45ed98fea5824feb4d42f45bbf218e597dc9fc58/scadm.sha256.txt
5332023-12-13T06:35:17.109Zdownloading scadm ...done
5342023-12-13T06:35:17.133Zcreating port tfportrear0_0 for sc0_0
5352023-12-13T06:35:17.134Zcreating port tfportqsfp0_0 for sc0_1
5362023-12-13T06:35:17.134Zsoftnpu zone setup
5372023-12-13T06:35:17.134Zconfigure zone
5382023-12-13T06:35:17.158Zinstall zone
5392023-12-13T06:35:18.225Zboot zone
5402023-12-13T06:35:21.330Z[sidecar_softnpu] svcs milestone/network
5412023-12-13T06:35:21.357ZSTATE STIME FMRI
5422023-12-13T06:35:21.357Zonline 6:35:19 svc:/milestone/network:default
5432023-12-13T06:35:21.357Z[sidecar_softnpu] /softnpu/npu start
5442023-12-13T06:35:21.382Z
5452023-12-13T06:35:21.406Z+ /opt/oxide/work/tools/scrimlet/softnpu-init.sh
5462023-12-13T06:35:21.407Z+ GATEWAY_IP=10.151.2.100
5472023-12-13T06:35:21.407Z+ echo 'Using 10.151.2.100 as gateway ip'
5482023-12-13T06:35:21.407ZUsing 10.151.2.100 as gateway ip
5492023-12-13T06:35:21.407Z+ [[ ! -v GATEWAY_MAC ]]
5502023-12-13T06:35:21.407Z+ GATEWAY_MAC=18:c0:4d:0d:a0:2a
5512023-12-13T06:35:21.407Z++ grep -c -E '^([0-9a-fA-F]{2}:){5}[0-9a-fA-F]{2}$' /dev/fd/63
5522023-12-13T06:35:21.407Z+++ echo 18:c0:4d:0d:a0:2a
5532023-12-13T06:35:21.407Z+ COUNT=1
5542023-12-13T06:35:21.407Z+ [[ 1 -ne 1 ]]
5552023-12-13T06:35:21.407Z+ echo 'Using 18:c0:4d:0d:a0:2a as gateway mac'
5562023-12-13T06:35:21.407ZUsing 18:c0:4d:0d:a0:2a as gateway mac
5572023-12-13T06:35:21.407Z+ z_scadm add-arp-entry 10.151.2.100 18:c0:4d:0d:a0:2a
5582023-12-13T06:35:21.407Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone add-arp-entry 10.151.2.100 18:c0:4d:0d:a0:2a
5592023-12-13T06:35:21.431Z+ PXA_MAC=a8:e1:de:01:70:1d
5602023-12-13T06:35:21.431Z+ [[ -v PXA_START ]]
5612023-12-13T06:35:21.431Z+ z_scadm add-proxy-arp 10.151.2.164 10.151.2.227 a8:e1:de:01:70:1d
5622023-12-13T06:35:21.431Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone add-proxy-arp 10.151.2.164 10.151.2.227 a8:e1:de:01:70:1d
5632023-12-13T06:35:21.463Z+ z_scadm dump-state
5642023-12-13T06:35:21.463Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone dump-state
5652023-12-13T06:35:21.487Zlocal v6:
5662023-12-13T06:35:21.487Zlocal v4:
5672023-12-13T06:35:21.487Zrouter v6:
5682023-12-13T06:35:21.487Zrouter v4_idx:
5692023-12-13T06:35:21.487Zrouter v4_routes:
5702023-12-13T06:35:21.487Zresolver v4:
5712023-12-13T06:35:21.487Z10.151.2.100 -> 18:c0:4d:0d:a0:2a
5722023-12-13T06:35:21.487Zresolver v6:
5732023-12-13T06:35:21.487Znat v4:
5742023-12-13T06:35:21.487Znat_v6:
5752023-12-13T06:35:21.487Zport_mac:
5762023-12-13T06:35:21.487Zproxy arp:
5772023-12-13T06:35:21.487Z10.151.2.164/10.151.2.227: a8:e1:de:01:70:1d
5782023-12-13T06:35:21.487Z+ success 'softnpu zone exists'
5792023-12-13T06:35:21.487Z+ set +x
5802023-12-13T06:35:21.487Zsoftnpu zone exists
5812023-12-13T06:35:21.488Z
5822023-12-13T06:35:21.488Zreal 20.509584279
5832023-12-13T06:35:21.488Zuser 0.714686016
5842023-12-13T06:35:21.488Zsys 1.612287697
5852023-12-13T06:35:21.488Ztrap 0.000073029
5862023-12-13T06:35:21.488Ztflt 0.000351872
5872023-12-13T06:35:21.488Zdflt 0.001655050
5882023-12-13T06:35:21.488Zkflt 0.000002020
5892023-12-13T06:35:21.488Zlock 15:41.161186147
5902023-12-13T06:35:21.488Zslp 1:01.390493267
5912023-12-13T06:35:21.488Zlat 0.054080217
5922023-12-13T06:35:21.488Zstop 0.357535437
5932023-12-13T06:35:21.488Z+ tar xf out/omicron-sled-agent.tar pkg/config-rss.toml
5942023-12-13T06:35:21.522Z++ sed -n 's/silo_name = "\(.*\)"/\1/p' pkg/config-rss.toml
5952023-12-13T06:35:21.574Z+ SILO_NAME=recovery
5962023-12-13T06:35:21.574Z++ sed -n 's/external_dns_zone_name = "\(.*\)"/\1/p' pkg/config-rss.toml
5972023-12-13T06:35:21.575Z+ EXTERNAL_DNS_DOMAIN=oxide.test
5982023-12-13T06:35:21.575Z+ sed -i~ '
5992023-12-13T06:35:21.575Z /^external_dns_ips/c\
6002023-12-13T06:35:21.575Zexternal_dns_ips = [ "10.151.2.164", "10.151.2.165" ]
6012023-12-13T06:35:21.575Z /^\[\[internal_services_ip_pool_ranges/,/^$/ {
6022023-12-13T06:35:21.575Z /^first/c\
6032023-12-13T06:35:21.575Zfirst = "10.151.2.164"
6042023-12-13T06:35:21.575Z /^last/c\
6052023-12-13T06:35:21.575Zlast = "10.151.2.173"
6062023-12-13T06:35:21.575Z }
6072023-12-13T06:35:21.575Z /^\[rack_network_config/,/^$/ {
6082023-12-13T06:35:21.575Z /^infra_ip_first/c\
6092023-12-13T06:35:21.575Zinfra_ip_first = "10.151.2.174"
6102023-12-13T06:35:21.575Z /^infra_ip_last/c\
6112023-12-13T06:35:21.575Zinfra_ip_last = "10.151.2.174"
6122023-12-13T06:35:21.575Z }
6132023-12-13T06:35:21.575Z /^\[\[rack_network_config.ports/,/^$/ {
6142023-12-13T06:35:21.575Z /^routes/c\
6152023-12-13T06:35:21.575Zroutes = \[{nexthop = "10.151.2.100", destination = "0.0.0.0/0"}\]
6162023-12-13T06:35:21.575Z /^addresses/c\
6172023-12-13T06:35:21.575Zaddresses = \["10.151.2.174/32"\]
6182023-12-13T06:35:21.575Z }
6192023-12-13T06:35:21.575Z' pkg/config-rss.toml
6202023-12-13T06:35:21.575Z+ diff -u pkg/config-rss.toml~ pkg/config-rss.toml
6212023-12-13T06:35:21.575Z--- pkg/config-rss.toml~ Mon Jul 24 01:21:28 2006
6222023-12-13T06:35:21.576Z+++ pkg/config-rss.toml Wed Dec 13 06:35:21 2023
6232023-12-13T06:35:21.576Z@@ -41,7 +41,7 @@
6242023-12-13T06:35:21.576Z # the DNS domain delegated to the rack by the customer. Each of these addresses
6252023-12-13T06:35:21.576Z # must be contained in one of the "internal services" IP Pool ranges listed
6262023-12-13T06:35:21.576Z # below.
6272023-12-13T06:35:21.576Z-external_dns_ips = [ "192.168.1.20", "192.168.1.21" ]
6282023-12-13T06:35:21.576Z+external_dns_ips = [ "10.151.2.164", "10.151.2.165" ]
6292023-12-13T06:35:21.576Z
6302023-12-13T06:35:21.576Z # Initial TLS certificates for the external API
6312023-12-13T06:35:21.576Z #
6322023-12-13T06:35:21.576Z@@ -77,8 +77,8 @@
6332023-12-13T06:35:21.576Z #
6342023-12-13T06:35:21.576Z # For more on this and what to put here, see docs/how-to-run.adoc.
6352023-12-13T06:35:21.576Z [[internal_services_ip_pool_ranges]]
6362023-12-13T06:35:21.576Z-first = "192.168.1.20"
6372023-12-13T06:35:21.576Z-last = "192.168.1.29"
6382023-12-13T06:35:21.576Z+first = "10.151.2.164"
6392023-12-13T06:35:21.576Z+last = "10.151.2.173"
6402023-12-13T06:35:21.576Z
6412023-12-13T06:35:21.576Z # TODO - this configuration is subject to change going forward. Ultimately these
6422023-12-13T06:35:21.576Z # parameters should be provided to the control plane via wicket, but we need to
6432023-12-13T06:35:21.577Z@@ -92,8 +92,8 @@
6442023-12-13T06:35:21.577Z # A range of IP addresses used by Boundary Services on the external network. In
6452023-12-13T06:35:21.577Z # a real system, these would be addresses of the uplink ports on the Sidecar.
6462023-12-13T06:35:21.577Z # With softnpu, only one address is used.
6472023-12-13T06:35:21.577Z-infra_ip_first = "192.168.1.30"
6482023-12-13T06:35:21.577Z-infra_ip_last = "192.168.1.30"
6492023-12-13T06:35:21.577Z+infra_ip_first = "10.151.2.174"
6502023-12-13T06:35:21.577Z+infra_ip_last = "10.151.2.174"
6512023-12-13T06:35:21.577Z
6522023-12-13T06:35:21.577Z # Configurations for BGP routers to run on the scrimlets.
6532023-12-13T06:35:21.577Z bgp = []
6542023-12-13T06:35:21.578Z@@ -101,9 +101,9 @@
6552023-12-13T06:35:21.578Z # You can configure multiple uplinks by repeating the following stanza
6562023-12-13T06:35:21.578Z [[rack_network_config.ports]]
6572023-12-13T06:35:21.578Z # Routes associated with this port.
6582023-12-13T06:35:21.578Z-routes = [{nexthop = "192.168.1.199", destination = "0.0.0.0/0"}]
6592023-12-13T06:35:21.578Z+routes = [{nexthop = "10.151.2.100", destination = "0.0.0.0/0"}]
6602023-12-13T06:35:21.578Z # Addresses associated with this port.
6612023-12-13T06:35:21.578Z-addresses = ["192.168.1.30/32"]
6622023-12-13T06:35:21.578Z+addresses = ["10.151.2.174/32"]
6632023-12-13T06:35:21.578Z # Name of the uplink port. This should always be "qsfp0" when using softnpu.
6642023-12-13T06:35:21.578Z port = "qsfp0"
6652023-12-13T06:35:21.578Z # The speed of this port.
6662023-12-13T06:35:21.578Z+ true
6672023-12-13T06:35:21.578Z+ tar rvf out/omicron-sled-agent.tar pkg/config-rss.toml
6682023-12-13T06:35:21.578Za pkg/config-rss.toml 7K
6692023-12-13T06:35:21.579Z+ rm -f pkg/config-rss.toml pkg/config-rss.toml~
6702023-12-13T06:35:21.579Z+ TLS_NAME=recovery.sys.oxide.test
6712023-12-13T06:35:21.579Z+ openssl req -newkey rsa:4096 -x509 -sha256 -days 3 -nodes -out pkg/initial-tls-cert.pem -keyout pkg/initial-tls-key.pem -subj /CN=recovery.sys.oxide.test -addext subjectAltName=DNS:recovery.sys.oxide.test -addext basicConstraints=critical,CA:FALSE -config /dev/stdin
6722023-12-13T06:35:22.517Z.+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*......+.....+.+...+..+...+......+......+...+..........+.................+....+...+......+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*........+.+.........+..+.........+............+.........+.+..+..........+..................+.....+.............+.........+......+..+...+.+...................................+...+...+...+......+.........+...................+...........+.+........+......+...............+.+...........+.......+..................+...+..+...+........................+.......+..............+...........................+..........+.....+...+.......+.....+.......+......+.........+...........+....+..................+..+...............+............+.+..+......+.+.........+...............+..+....+...............+.........+..+............+.+......+.........+..+...+....+.....+......+..........+..............+............+...+...+............+..........+..................+...+..+....+...+..................+............+...+..+.+.........+........+.......+......+.....+.......+...+.....+...............+......+..........+.....+.+..............+.............+..+......+...+.........+...+..........+...+.....+.+.....+....+......+...+..+.+.....+.......+..+.+............+...............+......+............+...+............+...+...........+...+.......+...+...............+..+.+........+....+...+..+...+............................+......+...+.................+.+..+............+......+...............+.........+.............+.....+...............+.+...........+.+........+..........+.....+......+.+.........+...........+.......+...+..................+......+...+...+...+..+.................................+....+.................+......+...+......+...+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
6732023-12-13T06:35:22.685Z...+......+...+..+.+........+.+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*...+.....+.+..+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++*...+..............+....+.....+....+............+...+..................+........+.......+...+.....+......+.+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
6742023-12-13T06:35:22.709Z-----
6752023-12-13T06:35:22.733Z+ tar rvf out/omicron-sled-agent.tar pkg/initial-tls-cert.pem pkg/initial-tls-key.pem
6762023-12-13T06:35:22.733Za pkg/initial-tls-cert.pem 2K
6772023-12-13T06:35:22.733Za pkg/initial-tls-key.pem 4K
6782023-12-13T06:35:22.733Z+ rm -f pkg/initial-tls-cert.pem pkg/initial-tls-key.pem
6792023-12-13T06:35:22.733Z+ rmdir pkg
6802023-12-13T06:35:22.733Z+ E2E_TLS_CERT=/opt/oxide/sled-agent/pkg/initial-tls-cert.pem
6812023-12-13T06:35:22.733Z+ pfexec mkdir -p /usr/oxide
6822023-12-13T06:35:22.733Z+ pfexec curl -sSfL -o /usr/oxide/oxide http://catacomb.eng.oxide.computer:12346/oxide-v0.1.0
6832023-12-13T06:35:23.462Z+ pfexec chmod +x /usr/oxide/oxide
6842023-12-13T06:35:23.490Z+ curl -sSfL -o debian-11-genericcloud-amd64.raw http://catacomb.eng.oxide.computer:12346/debian-11-genericcloud-amd64.raw
6852023-12-13T06:35:41.765Z+ OMICRON_NO_UNINSTALL=1
6862023-12-13T06:35:41.765Z+ ptime -m pfexec ./target/release/omicron-package -t test install
6872023-12-13T06:35:41.791ZDec 13 06:35:41.669 DEBG target[test]: Target({"image": "standard", "machine": "non-gimlet", "rack-topology": "single-sled", "switch": "softnpu"})
6882023-12-13T06:35:41.791ZDec 13 06:35:41.671 INFO Installing service, dst: /opt/oxide/clickhouse.tar.gz, src: out/clickhouse.tar.gz
6892023-12-13T06:35:41.791ZDec 13 06:35:41.671 INFO Installing service, dst: /opt/oxide/crucible_pantry.tar.gz, src: out/crucible-pantry.tar.gz
6902023-12-13T06:35:41.791ZDec 13 06:35:41.671 INFO Installing service, dst: /opt/oxide/internal_dns.tar.gz, src: out/internal-dns.tar.gz
6912023-12-13T06:35:41.791ZDec 13 06:35:41.671 INFO Installing service, dst: /opt/oxide/crucible.tar.gz, src: out/crucible.tar.gz
6922023-12-13T06:35:41.791ZDec 13 06:35:41.671 INFO Installing service, dst: /opt/oxide/external_dns.tar.gz, src: out/external-dns.tar.gz
6932023-12-13T06:35:41.791ZDec 13 06:35:41.671 INFO Installing service, dst: /opt/oxide/ntp.tar.gz, src: out/ntp.tar.gz
6942023-12-13T06:35:41.791ZDec 13 06:35:41.671 INFO Installing service, dst: /opt/oxide/nexus.tar.gz, src: out/omicron-nexus.tar.gz
6952023-12-13T06:35:41.791ZDec 13 06:35:41.671 INFO Installing service, dst: /opt/oxide/cockroachdb.tar.gz, src: out/cockroachdb.tar.gz
6962023-12-13T06:35:41.791ZDec 13 06:35:41.671 INFO Installing service, dst: /opt/oxide/oximeter.tar.gz, src: out/oximeter-collector.tar.gz
6972023-12-13T06:35:41.791ZDec 13 06:35:41.671 INFO Installing service, dst: /opt/oxide/mg-ddm.tar, src: out/mg-ddm-gz.tar
6982023-12-13T06:35:41.791ZDec 13 06:35:41.671 INFO Installing service, dst: /opt/oxide/propolis-server.tar.gz, src: out/propolis-server.tar.gz
6992023-12-13T06:35:41.791ZDec 13 06:35:41.671 INFO Installing service, dst: /opt/oxide/clickhouse_keeper.tar.gz, src: out/clickhouse_keeper.tar.gz
7002023-12-13T06:35:41.791ZDec 13 06:35:41.671 INFO Installing service, dst: /opt/oxide/overlay.tar.gz, src: out/overlay.tar.gz
7012023-12-13T06:35:41.791ZDec 13 06:35:41.671 INFO Installing service, dst: /opt/oxide/mgs.tar.gz, src: out/omicron-gateway-softnpu.tar.gz
7022023-12-13T06:35:41.791ZDec 13 06:35:41.671 INFO Installing service, dst: /opt/oxide/sled-agent.tar, src: out/omicron-sled-agent.tar
7032023-12-13T06:35:41.791ZDec 13 06:35:41.671 INFO Installing service, dst: /opt/oxide/switch.tar.gz, src: out/switch-softnpu.tar.gz
7042023-12-13T06:35:42.026ZDec 13 06:35:41.926 INFO Unpacking service tarball, service_path: /opt/oxide/mg-ddm, tar_path: /opt/oxide/mg-ddm.tar
7052023-12-13T06:35:42.062ZDec 13 06:35:41.962 INFO Unpacking service tarball, service_path: /opt/oxide/sled-agent, tar_path: /opt/oxide/sled-agent.tar
7062023-12-13T06:35:42.168ZDec 13 06:35:42.031 INFO Installing bootstrap service from /opt/oxide/sled-agent/pkg/manifest.xml
7072023-12-13T06:35:42.274Z
7082023-12-13T06:35:42.298Zreal 0.514782615
7092023-12-13T06:35:42.298Zuser 0.100029110
7102023-12-13T06:35:42.298Zsys 2.105441443
7112023-12-13T06:35:42.298Ztrap 0.000000609
7122023-12-13T06:35:42.298Ztflt 0.000027290
7132023-12-13T06:35:42.298Zdflt 0.000846461
7142023-12-13T06:35:42.298Zkflt 0.000005220
7152023-12-13T06:35:42.299Zlock 46.202119622
7162023-12-13T06:35:42.299Zslp 0.718207048
7172023-12-13T06:35:42.299Zlat 0.003236643
7182023-12-13T06:35:42.299Zstop 0.314958244
7192023-12-13T06:35:42.299Z+ retry=0
7202023-12-13T06:35:42.299Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7212023-12-13T06:35:42.299Z+ [[ 0 -gt 30 ]]
7222023-12-13T06:35:42.299Z+ sleep 1
7232023-12-13T06:35:43.290Z+ retry=1
7242023-12-13T06:35:43.290Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7252023-12-13T06:35:43.315Z+ [[ 1 -gt 30 ]]
7262023-12-13T06:35:43.315Z+ sleep 1
7272023-12-13T06:35:44.329Z+ retry=2
7282023-12-13T06:35:44.329Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7292023-12-13T06:35:44.329Z+ [[ 2 -gt 30 ]]
7302023-12-13T06:35:44.329Z+ sleep 1
7312023-12-13T06:35:45.319Z+ retry=3
7322023-12-13T06:35:45.319Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7332023-12-13T06:35:45.344Z+ [[ 3 -gt 30 ]]
7342023-12-13T06:35:45.344Z+ sleep 1
7352023-12-13T06:35:46.332Z+ retry=4
7362023-12-13T06:35:46.332Z+ curl --head --silent -o /dev/null 'http://[fd00:1122:3344:101::2]:12224/'
7372023-12-13T06:36:20.207Z+ pfexec zlogin sidecar_softnpu /softnpu/scadm --server /softnpu/server --client /softnpu/client standalone dump-state
7382023-12-13T06:36:20.340Zlocal v6:
7392023-12-13T06:36:20.340Zfd00:99::1
7402023-12-13T06:36:20.340Zfe80::c8cc:28ff:fe9c:277f
7412023-12-13T06:36:20.340Zlocal v4:
7422023-12-13T06:36:20.340Z10.151.2.174
7432023-12-13T06:36:20.340Zrouter v6:
7442023-12-13T06:36:20.340Zfd00:1122:3344:3::/64 -> fe80::886a:38ff:feda:f99 (1)
7452023-12-13T06:36:20.340Zfdb0:18c0:4d0d:a02a::/64 -> fe80::886a:38ff:feda:f99 (1)
7462023-12-13T06:36:20.340Zfd00:1122:3344:2::/64 -> fe80::886a:38ff:feda:f99 (1)
7472023-12-13T06:36:20.341Zfd00:1122:3344:101::/64 -> fe80::886a:38ff:feda:f99 (1)
7482023-12-13T06:36:20.341Zfd00:1122:3344:1::/64 -> fe80::886a:38ff:feda:f99 (1)
7492023-12-13T06:36:20.341Zrouter v4_idx:
7502023-12-13T06:36:20.341Z0.0.0.0/0 -> 0
7512023-12-13T06:36:20.341Zrouter v4_routes:
7522023-12-13T06:36:20.341Z0 -> 10.151.2.100 (2)
7532023-12-13T06:36:20.341Zresolver v4:
7542023-12-13T06:36:20.341Z10.151.2.100 -> 18:c0:4d:0d:a0:2a
7552023-12-13T06:36:20.341Z10.151.2.174 -> b6:8f:f0:55:d3:8e
7562023-12-13T06:36:20.342Zresolver v6:
7572023-12-13T06:36:20.342Zfe80::c8cc:28ff:fe9c:277f -> ca:cc:28:9c:27:7f
7582023-12-13T06:36:20.342Zfe80::886a:38ff:feda:f99 -> 8a:6a:38:da:0f:99
7592023-12-13T06:36:20.343Znat v4:
7602023-12-13T06:36:20.343Z10.151.2.169 0/16383 -> fd00:1122:3344:101::1 100/a8:40:25:ff:c4:c9
7612023-12-13T06:36:20.343Znat_v6:
7622023-12-13T06:36:20.343Zport_mac:
7632023-12-13T06:36:20.343Z1: a8:40:25:3d:3a:bc
7642023-12-13T06:36:20.343Z1000: a8:40:25:3d:3a:bb
7652023-12-13T06:36:20.343Zproxy arp:
7662023-12-13T06:36:20.343Z10.151.2.164/10.151.2.227: a8:e1:de:01:70:1d
7672023-12-13T06:36:20.343Z+ retry=0
7682023-12-13T06:36:20.344Z+++ zoneadm list -n
7692023-12-13T06:36:20.344Z+++ grep oxz_ntp
7702023-12-13T06:36:20.344Z++ pfexec svcs -z oxz_ntp_6c95d6e6-e4e5-43a3-ba3b-3d3584cfe251 -Hostate oxide/ntp
7712023-12-13T06:36:20.344Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
7722023-12-13T06:36:20.344Z++ true
7732023-12-13T06:36:20.344Z+ [[ '' != online ]]
7742023-12-13T06:36:20.344Z+ [[ 0 -gt 60 ]]
7752023-12-13T06:36:20.344Z+ sleep 1
7762023-12-13T06:36:20.344Z+ retry=1
7772023-12-13T06:36:20.344Z+++ zoneadm list -n
7782023-12-13T06:36:20.344Z+++ grep oxz_ntp
7792023-12-13T06:36:20.344Z++ pfexec svcs -z oxz_ntp_6c95d6e6-e4e5-43a3-ba3b-3d3584cfe251 -Hostate oxide/ntp
7802023-12-13T06:36:20.344Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
7812023-12-13T06:36:20.344Z++ true
7822023-12-13T06:36:20.344Z+ [[ '' != online ]]
7832023-12-13T06:36:20.344Z+ [[ 1 -gt 60 ]]
7842023-12-13T06:36:20.344Z+ sleep 1
7852023-12-13T06:36:20.344Z+ retry=2
7862023-12-13T06:36:20.344Z+++ zoneadm list -n
7872023-12-13T06:36:20.344Z+++ grep oxz_ntp
7882023-12-13T06:36:20.344Z++ pfexec svcs -z oxz_ntp_6c95d6e6-e4e5-43a3-ba3b-3d3584cfe251 -Hostate oxide/ntp
7892023-12-13T06:36:20.344Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
7902023-12-13T06:36:20.345Z++ true
7912023-12-13T06:36:20.345Z+ [[ '' != online ]]
7922023-12-13T06:36:20.345Z+ [[ 2 -gt 60 ]]
7932023-12-13T06:36:20.345Z+ sleep 1
7942023-12-13T06:36:20.345Z+ retry=3
7952023-12-13T06:36:20.345Z+++ zoneadm list -n
7962023-12-13T06:36:20.345Z+++ grep oxz_ntp
7972023-12-13T06:36:20.373Z++ pfexec svcs -z oxz_ntp_6c95d6e6-e4e5-43a3-ba3b-3d3584cfe251 -Hostate oxide/ntp
7982023-12-13T06:36:20.373Zsvcs: Pattern 'oxide/ntp' doesn't match any instances
7992023-12-13T06:36:20.373Z++ true
8002023-12-13T06:36:20.373Z+ [[ '' != online ]]
8012023-12-13T06:36:20.373Z+ [[ 3 -gt 60 ]]
8022023-12-13T06:36:20.373Z+ sleep 1
8032023-12-13T06:36:20.373Z+ retry=4
8042023-12-13T06:36:20.373Z+++ zoneadm list -n
8052023-12-13T06:36:20.373Z+++ grep oxz_ntp
8062023-12-13T06:36:20.373Z++ pfexec svcs -z oxz_ntp_6c95d6e6-e4e5-43a3-ba3b-3d3584cfe251 -Hostate oxide/ntp
8072023-12-13T06:36:20.374Z+ [[ disabled != online ]]
8082023-12-13T06:36:20.374Z+ [[ 4 -gt 60 ]]
8092023-12-13T06:36:20.374Z+ sleep 1
8102023-12-13T06:36:20.374Z+ retry=5
8112023-12-13T06:36:20.374Z+++ zoneadm list -n
8122023-12-13T06:36:20.374Z+++ grep oxz_ntp
8132023-12-13T06:36:20.375Z++ pfexec svcs -z oxz_ntp_6c95d6e6-e4e5-43a3-ba3b-3d3584cfe251 -Hostate oxide/ntp
8142023-12-13T06:36:20.375Z+ [[ disabled != online ]]
8152023-12-13T06:36:20.375Z+ [[ 5 -gt 60 ]]
8162023-12-13T06:36:20.375Z+ sleep 1
8172023-12-13T06:36:20.375Z+ retry=6
8182023-12-13T06:36:20.375Z+++ zoneadm list -n
8192023-12-13T06:36:20.375Z+++ grep oxz_ntp
8202023-12-13T06:36:20.375Z++ pfexec svcs -z oxz_ntp_6c95d6e6-e4e5-43a3-ba3b-3d3584cfe251 -Hostate oxide/ntp
8212023-12-13T06:36:20.375Z+ [[ disabled != online ]]
8222023-12-13T06:36:20.376Z+ [[ 6 -gt 60 ]]
8232023-12-13T06:36:20.376Z+ sleep 1
8242023-12-13T06:36:20.376Z+ retry=7
8252023-12-13T06:36:20.376Z+++ zoneadm list -n
8262023-12-13T06:36:20.376Z+++ grep oxz_ntp
8272023-12-13T06:36:20.376Z++ pfexec svcs -z oxz_ntp_6c95d6e6-e4e5-43a3-ba3b-3d3584cfe251 -Hostate oxide/ntp
8282023-12-13T06:36:20.376Z+ [[ disabled != online ]]
8292023-12-13T06:36:20.376Z+ [[ 7 -gt 60 ]]
8302023-12-13T06:36:20.376Z+ sleep 1
8312023-12-13T06:36:20.376Z+ retry=8
8322023-12-13T06:36:20.376Z+++ zoneadm list -n
8332023-12-13T06:36:20.376Z+++ grep oxz_ntp
8342023-12-13T06:36:20.376Z++ pfexec svcs -z oxz_ntp_6c95d6e6-e4e5-43a3-ba3b-3d3584cfe251 -Hostate oxide/ntp
8352023-12-13T06:36:20.377Z+ [[ disabled != online ]]
8362023-12-13T06:36:20.377Z+ [[ 8 -gt 60 ]]
8372023-12-13T06:36:20.377Z+ sleep 1
8382023-12-13T06:36:20.817Z+ retry=9
8392023-12-13T06:36:20.850Z+++ zoneadm list -n
8402023-12-13T06:36:20.850Z+++ grep oxz_ntp
8412023-12-13T06:36:20.850Z++ pfexec svcs -z oxz_ntp_6c95d6e6-e4e5-43a3-ba3b-3d3584cfe251 -Hostate oxide/ntp
8422023-12-13T06:36:20.850Z+ [[ disabled != online ]]
8432023-12-13T06:36:20.851Z+ [[ 9 -gt 60 ]]
8442023-12-13T06:36:20.851Z+ sleep 1
8452023-12-13T06:36:21.837Z+ retry=10
8462023-12-13T06:36:21.865Z+++ zoneadm list -n
8472023-12-13T06:36:21.865Z+++ grep oxz_ntp
8482023-12-13T06:36:21.865Z++ pfexec svcs -z oxz_ntp_6c95d6e6-e4e5-43a3-ba3b-3d3584cfe251 -Hostate oxide/ntp
8492023-12-13T06:36:21.865Z+ [[ disabled != online ]]
8502023-12-13T06:36:21.865Z+ [[ 10 -gt 60 ]]
8512023-12-13T06:36:21.865Z+ sleep 1
 
8522023-12-13T06:36:22.246Zsock console login: Dec 13 06:36:21 sock ip: ip: joining multicasts failed (4) on vopte0 - will use link layer broadcasts for multicast
 
8532023-12-13T06:36:22.859Z+ retry=11
8542023-12-13T06:36:22.884Z+++ zoneadm list -n
8552023-12-13T06:36:22.884Z+++ grep oxz_ntp
8562023-12-13T06:36:22.884Z++ pfexec svcs -z oxz_ntp_6c95d6e6-e4e5-43a3-ba3b-3d3584cfe251 -Hostate oxide/ntp
8572023-12-13T06:36:22.884Z+ [[ online != online ]]
8582023-12-13T06:36:22.884Z+ echo 'Waited for chrony: 11s'
8592023-12-13T06:36:22.885ZWaited for chrony: 11s
8602023-12-13T06:36:22.885Z+ export RUST_BACKTRACE=1
8612023-12-13T06:36:22.885Z+ RUST_BACKTRACE=1
8622023-12-13T06:36:22.885Z+ export E2E_TLS_CERT IPPOOL_START IPPOOL_END
8632023-12-13T06:36:22.885Z++ ./tests/bootstrap
8642023-12-13T06:36:22.911ZUsing external DNS server at 10.151.2.164:53
8652023-12-13T06:36:22.911Z2023-12-13 06:36:22.810435054 UTC: attempting to log into API
8662023-12-13T06:36:37.947Z2023-12-13 06:36:37.871086675 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8672023-12-13T06:36:38.947Z2023-12-13 06:36:38.874529239 UTC: attempting to log into API
8682023-12-13T06:36:53.981Z2023-12-13 06:36:53.948800191 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8692023-12-13T06:36:54.982Z2023-12-13 06:36:54.953261845 UTC: attempting to log into API
8702023-12-13T06:37:10.015Z2023-12-13 06:37:10.005279218 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8712023-12-13T06:37:11.016Z2023-12-13 06:37:11.006168496 UTC: attempting to log into API
8722023-12-13T06:37:26.048Z2023-12-13 06:37:26.038068844 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8732023-12-13T06:37:27.049Z2023-12-13 06:37:27.038980082 UTC: attempting to log into API
8742023-12-13T06:37:42.082Z2023-12-13 06:37:42.069656273 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8752023-12-13T06:37:43.084Z2023-12-13 06:37:43.071396748 UTC: attempting to log into API
8762023-12-13T06:37:58.117Z2023-12-13 06:37:58.103839465 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8772023-12-13T06:37:59.129Z2023-12-13 06:37:59.104839237 UTC: attempting to log into API
8782023-12-13T06:38:14.158Z2023-12-13 06:38:14.144361933 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8792023-12-13T06:38:15.160Z2023-12-13 06:38:15.145914012 UTC: attempting to log into API
8802023-12-13T06:38:30.197Z2023-12-13 06:38:30.183313253 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8812023-12-13T06:38:31.198Z2023-12-13 06:38:31.184468760 UTC: attempting to log into API
8822023-12-13T06:38:46.250Z2023-12-13 06:38:46.237331215 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8832023-12-13T06:38:47.252Z2023-12-13 06:38:47.238489589 UTC: attempting to log into API
 
8842023-12-13T06:39:01.841ZDec 13 06:39:01 sock ip: ip: joining multicasts failed (4) on vopte3 - will use link layer broadcasts for multicast
 
8852023-12-13T06:39:02.285Z2023-12-13 06:39:02.271430775 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8862023-12-13T06:39:03.285Z2023-12-13 06:39:03.272513779 UTC: attempting to log into API
 
8872023-12-13T06:39:09.040ZDec 13 06:39:08 sock ip: ip: joining multicasts failed (4) on vopte1 - will use link layer broadcasts for multicast
8882023-12-13T06:39:15.704ZDec 13 06:39:15 sock ip: ip: joining multicasts failed (4) on vopte2 - will use link layer broadcasts for multicast
 
8892023-12-13T06:39:18.318Z2023-12-13 06:39:18.305373564 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8902023-12-13T06:39:19.325Z2023-12-13 06:39:19.306976395 UTC: attempting to log into API
 
8912023-12-13T06:39:22.897ZDec 13 06:39:22 sock ip: ip: joining multicasts failed (4) on vopte4 - will use link layer broadcasts for multicast
8922023-12-13T06:39:30.620ZDec 13 06:39:29 sock ip: ip: joining multicasts failed (4) on vopte5 - will use link layer broadcasts for multicast
 
8932023-12-13T06:39:34.359Z2023-12-13 06:39:34.339497639 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8942023-12-13T06:39:35.354Z2023-12-13 06:39:35.340582893 UTC: attempting to log into API
8952023-12-13T06:39:50.426Z2023-12-13 06:39:50.372117771 UTC: login failed: logging in: error sending request for url (https://recovery.sys.oxide.test/v1/login/recovery/local): error trying to connect: operation timed out
8962023-12-13T06:39:51.418Z2023-12-13 06:39:51.373055240 UTC: attempting to log into API
8972023-12-13T06:39:52.232Z2023-12-13 06:39:52.161780205 UTC: login succeeded
8982023-12-13T06:39:52.323Zwaiting for nexus to come up...
8992023-12-13T06:39:52.418Zcreating IP pool... 10.151.2.175 - 10.151.2.227
9002023-12-13T06:39:52.418Zensuring datasets are ready...
9012023-12-13T06:39:55.174Zlet's roll.
9022023-12-13T06:39:55.199Z+ eval 'OXIDE_HOST=https://recovery.sys.oxide.test
9032023-12-13T06:39:55.199ZOXIDE_RESOLVE=recovery.sys.oxide.test:443:10.151.2.166
9042023-12-13T06:39:55.199ZOXIDE_TOKEN=oxide-token-459d9357f56d4fa43408e5365cebea4cc8f99a7d'
9052023-12-13T06:39:55.199Z++ OXIDE_HOST=https://recovery.sys.oxide.test
9062023-12-13T06:39:55.200Z++ OXIDE_RESOLVE=recovery.sys.oxide.test:443:10.151.2.166
9072023-12-13T06:39:55.200Z++ OXIDE_TOKEN=oxide-token-459d9357f56d4fa43408e5365cebea4cc8f99a7d
9082023-12-13T06:39:55.200Z+ export OXIDE_HOST OXIDE_TOKEN
9092023-12-13T06:39:55.200Z+ retry=0
9102023-12-13T06:39:55.200Z+ curl -sSf https://recovery.sys.oxide.test/v1/ping --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem
9112023-12-13T06:39:55.200Z+ /usr/oxide/oxide --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem project create --name images --description 'some images'
9122023-12-13T06:39:55.645Z{"status":"ok"}success
9132023-12-13T06:39:55.646ZProject {
9142023-12-13T06:39:55.646Z description: "some images",
9152023-12-13T06:39:55.646Z id: 49f0d6a6-dd6c-4a7a-86ba-90c3aeff930c,
9162023-12-13T06:39:55.646Z name: Name(
9172023-12-13T06:39:55.646Z "images",
9182023-12-13T06:39:55.646Z ),
9192023-12-13T06:39:55.670Z time_created: 2023-12-13T06:39:55.257730Z,
9202023-12-13T06:39:55.670Z time_modified: 2023-12-13T06:39:55.257730Z,
9212023-12-13T06:39:55.670Z}
9222023-12-13T06:39:55.670Z+ /usr/oxide/oxide --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem disk import --path debian-11-genericcloud-amd64.raw --disk debian11-boot --project images --description 'debian 11 cloud image from distros' --snapshot debian11-snapshot --image debian11 --image-description 'debian 11 original base image' --image-os debian --image-version 11
9232023-12-13T06:45:30.284Zdone!
9242023-12-13T06:45:30.379Z+ /usr/oxide/oxide --resolve recovery.sys.oxide.test:443:10.151.2.166 --cacert /opt/oxide/sled-agent/pkg/initial-tls-cert.pem image promote --project images --image debian11
9252023-12-13T06:45:30.483Zsuccess
9262023-12-13T06:45:30.483ZImage {
9272023-12-13T06:45:30.514Z block_size: ByteCount(
9282023-12-13T06:45:30.514Z 512,
9292023-12-13T06:45:30.514Z ),
9302023-12-13T06:45:30.514Z description: "debian 11 original base image",
9312023-12-13T06:45:30.514Z digest: None,
9322023-12-13T06:45:30.514Z id: 0bd43762-0e42-4e4b-9b62-379f995be302,
9332023-12-13T06:45:30.514Z name: Name(
9342023-12-13T06:45:30.514Z "debian11",
9352023-12-13T06:45:30.514Z ),
9362023-12-13T06:45:30.514Z os: "debian",
9372023-12-13T06:45:30.514Z project_id: None,
9382023-12-13T06:45:30.514Z size: ByteCount(
9392023-12-13T06:45:30.514Z 2147483648,
9402023-12-13T06:45:30.515Z ),
9412023-12-13T06:45:30.515Z time_created: 2023-12-13T06:45:30.245780Z,
9422023-12-13T06:45:30.515Z time_modified: 2023-12-13T06:45:30.461317Z,
9432023-12-13T06:45:30.516Z url: None,
9442023-12-13T06:45:30.516Z version: "11",
9452023-12-13T06:45:30.516Z}
9462023-12-13T06:45:30.516Z+ rm ./tests/bootstrap
9472023-12-13T06:45:30.569Z+ for test_bin in tests/*
9482023-12-13T06:45:30.569Z+ ./tests/bootstrap-b98d9b3a65a4fd47
9492023-12-13T06:45:30.593Z
9502023-12-13T06:45:30.593Zrunning 0 tests
9512023-12-13T06:45:30.593Z
9522023-12-13T06:45:30.593Ztest result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
9532023-12-13T06:45:30.593Z
9542023-12-13T06:45:30.593Z+ for test_bin in tests/*
9552023-12-13T06:45:30.593Z+ ./tests/end_to_end_tests-e373b32ce9650dc1
9562023-12-13T06:45:30.617Z
9572023-12-13T06:45:30.618Zrunning 2 tests
9582023-12-13T06:45:30.832Ztest no_spoof::no_spoof_here ... ok
9592023-12-13T06:46:04.657Ztest instance_launch::instance_launch ... ok
9602023-12-13T06:46:04.657Z
9612023-12-13T06:46:04.657Ztest result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 34.05s
9622023-12-13T06:46:04.657Z
9632023-12-13T06:46:04.682Z+ _exit_trap
9642023-12-13T06:46:04.682Z+ local status=0
9652023-12-13T06:46:04.682Z+ [[ 0 -eq 0 ]]
9662023-12-13T06:46:04.682Z+ exit 0
9672023-12-13T06:46:04.682Zprocess exited: duration 671764 ms, exit code 0
 
9682023-12-13T06:46:04.786Zfound 99 output files
9692023-12-13T06:46:04.786Zuploading: /var/svc/log/oxide-sled-agent:default.log (829416 bytes)
9702023-12-13T06:46:06.849Zuploaded: /var/svc/log/oxide-sled-agent:default.log
9712023-12-13T06:46:06.849Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_4202958f-7c82-4bf1-9b24-143b5e8cf456/root/var/svc/log/oxide-cockroachdb:default.log (0 bytes)
9722023-12-13T06:46:07.931Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_4202958f-7c82-4bf1-9b24-143b5e8cf456/root/var/svc/log/oxide-cockroachdb:default.log
9732023-12-13T06:46:07.931Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_4202958f-7c82-4bf1-9b24-143b5e8cf456/root/var/svc/log/oxide-cockroachdb:default.log.0 (5385 bytes)
9742023-12-13T06:46:09.027Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_4202958f-7c82-4bf1-9b24-143b5e8cf456/root/var/svc/log/oxide-cockroachdb:default.log.0
9752023-12-13T06:46:09.027Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_e3f37bcd-79ce-497a-853c-539deb111314/root/var/svc/log/oxide-cockroachdb:default.log (0 bytes)
9762023-12-13T06:46:10.098Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_e3f37bcd-79ce-497a-853c-539deb111314/root/var/svc/log/oxide-cockroachdb:default.log
9772023-12-13T06:46:10.098Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_e3f37bcd-79ce-497a-853c-539deb111314/root/var/svc/log/oxide-cockroachdb:default.log.0 (5413 bytes)
9782023-12-13T06:46:11.195Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_e3f37bcd-79ce-497a-853c-539deb111314/root/var/svc/log/oxide-cockroachdb:default.log.0
9792023-12-13T06:46:11.195Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_1292a764-f313-480b-8193-f706cc5b6c7b/root/var/svc/log/oxide-crucible-agent:default.log (7758 bytes)
9802023-12-13T06:46:12.310Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_1292a764-f313-480b-8193-f706cc5b6c7b/root/var/svc/log/oxide-crucible-agent:default.log
9812023-12-13T06:46:12.335Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_1292a764-f313-480b-8193-f706cc5b6c7b/root/var/svc/log/oxide-crucible-agent:default.log.0 (19614 bytes)
9822023-12-13T06:46:13.448Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_1292a764-f313-480b-8193-f706cc5b6c7b/root/var/svc/log/oxide-crucible-agent:default.log.0
9832023-12-13T06:46:13.449Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_1292a764-f313-480b-8193-f706cc5b6c7b/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
9842023-12-13T06:46:14.519Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_1292a764-f313-480b-8193-f706cc5b6c7b/root/var/svc/log/oxide-crucible-downstairs:default.log
9852023-12-13T06:46:14.519Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_1292a764-f313-480b-8193-f706cc5b6c7b/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
9862023-12-13T06:46:15.616Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_1292a764-f313-480b-8193-f706cc5b6c7b/root/var/svc/log/oxide-crucible-downstairs:default.log.0
9872023-12-13T06:46:15.617Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_1292a764-f313-480b-8193-f706cc5b6c7b/root/var/svc/log/oxide-crucible-downstairs:downstairs-23d776bd-e11e-41d3-80f1-af58d02b054d.log (0 bytes)
9882023-12-13T06:46:16.692Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_1292a764-f313-480b-8193-f706cc5b6c7b/root/var/svc/log/oxide-crucible-downstairs:downstairs-23d776bd-e11e-41d3-80f1-af58d02b054d.log
9892023-12-13T06:46:16.692Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_1292a764-f313-480b-8193-f706cc5b6c7b/root/var/svc/log/oxide-crucible-downstairs:downstairs-23d776bd-e11e-41d3-80f1-af58d02b054d.log.0 (3050 bytes)
9902023-12-13T06:46:17.811Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_1292a764-f313-480b-8193-f706cc5b6c7b/root/var/svc/log/oxide-crucible-downstairs:downstairs-23d776bd-e11e-41d3-80f1-af58d02b054d.log.0
9912023-12-13T06:46:17.811Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_1292a764-f313-480b-8193-f706cc5b6c7b/root/var/svc/log/oxide-crucible-downstairs:downstairs-4e9f612d-17dd-447a-a755-594dd4ce1dc8.log (1904 bytes)
9922023-12-13T06:46:18.909Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_1292a764-f313-480b-8193-f706cc5b6c7b/root/var/svc/log/oxide-crucible-downstairs:downstairs-4e9f612d-17dd-447a-a755-594dd4ce1dc8.log
9932023-12-13T06:46:18.909Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_1292a764-f313-480b-8193-f706cc5b6c7b/root/var/svc/log/oxide-crucible-downstairs:downstairs-4e9f612d-17dd-447a-a755-594dd4ce1dc8.log.0 (4529 bytes)
9942023-12-13T06:46:20.004Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_1292a764-f313-480b-8193-f706cc5b6c7b/root/var/svc/log/oxide-crucible-downstairs:downstairs-4e9f612d-17dd-447a-a755-594dd4ce1dc8.log.0
9952023-12-13T06:46:20.004Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_1292a764-f313-480b-8193-f706cc5b6c7b/root/var/svc/log/oxide-crucible-downstairs:snapshot-4e9f612d-17dd-447a-a755-594dd4ce1dc8-1a70593c-ed4b-48da-aba9-a3e4e480a772.log (5154 bytes)
9962023-12-13T06:46:21.098Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_1292a764-f313-480b-8193-f706cc5b6c7b/root/var/svc/log/oxide-crucible-downstairs:snapshot-4e9f612d-17dd-447a-a755-594dd4ce1dc8-1a70593c-ed4b-48da-aba9-a3e4e480a772.log
9972023-12-13T06:46:21.098Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_b9dbca9d-f5a2-4c57-993d-0cf14cd8f8fc/root/var/svc/log/oxide-internal_dns:default.log (3756 bytes)
9982023-12-13T06:46:22.212Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_b9dbca9d-f5a2-4c57-993d-0cf14cd8f8fc/root/var/svc/log/oxide-internal_dns:default.log
9992023-12-13T06:46:22.212Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_b9dbca9d-f5a2-4c57-993d-0cf14cd8f8fc/root/var/svc/log/oxide-internal_dns:default.log.0 (127760 bytes)
10002023-12-13T06:46:23.534Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_b9dbca9d-f5a2-4c57-993d-0cf14cd8f8fc/root/var/svc/log/oxide-internal_dns:default.log.0
10012023-12-13T06:46:23.534Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_4d2ad6b5-6521-419b-a8f6-c0b5667485eb/root/var/svc/log/oxide-nexus:default.log (86941 bytes)
10022023-12-13T06:46:23.737Zupload warning: file "/pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_4d2ad6b5-6521-419b-a8f6-c0b5667485eb/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 86941 -> 88157
10032023-12-13T06:46:24.808Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_4d2ad6b5-6521-419b-a8f6-c0b5667485eb/root/var/svc/log/oxide-nexus:default.log
10042023-12-13T06:46:24.809Zuploading: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_4d2ad6b5-6521-419b-a8f6-c0b5667485eb/root/var/svc/log/oxide-nexus:default.log.0 (921463 bytes)
10052023-12-13T06:46:26.795Zuploaded: /pool/ext/14b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_nexus_4d2ad6b5-6521-419b-a8f6-c0b5667485eb/root/var/svc/log/oxide-nexus:default.log.0
10062023-12-13T06:46:26.795Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_cf7bb334-ee34-4de8-a593-130cd4e169d8/root/var/svc/log/oxide-cockroachdb:default.log (0 bytes)
10072023-12-13T06:46:27.906Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_cf7bb334-ee34-4de8-a593-130cd4e169d8/root/var/svc/log/oxide-cockroachdb:default.log
10082023-12-13T06:46:27.906Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_cf7bb334-ee34-4de8-a593-130cd4e169d8/root/var/svc/log/oxide-cockroachdb:default.log.0 (5867 bytes)
10092023-12-13T06:46:29.398Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_cockroachdb_cf7bb334-ee34-4de8-a593-130cd4e169d8/root/var/svc/log/oxide-cockroachdb:default.log.0
10102023-12-13T06:46:29.398Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_0e439546-60f7-43be-9ded-bd45248b3d46/root/var/svc/log/oxide-crucible-agent:default.log (0 bytes)
10112023-12-13T06:46:30.495Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_0e439546-60f7-43be-9ded-bd45248b3d46/root/var/svc/log/oxide-crucible-agent:default.log
10122023-12-13T06:46:30.495Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_0e439546-60f7-43be-9ded-bd45248b3d46/root/var/svc/log/oxide-crucible-agent:default.log.0 (3435 bytes)
10132023-12-13T06:46:31.589Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_0e439546-60f7-43be-9ded-bd45248b3d46/root/var/svc/log/oxide-crucible-agent:default.log.0
10142023-12-13T06:46:31.589Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_0e439546-60f7-43be-9ded-bd45248b3d46/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
10152023-12-13T06:46:32.659Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_0e439546-60f7-43be-9ded-bd45248b3d46/root/var/svc/log/oxide-crucible-downstairs:default.log
10162023-12-13T06:46:32.659Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_0e439546-60f7-43be-9ded-bd45248b3d46/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
10172023-12-13T06:46:33.754Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_0e439546-60f7-43be-9ded-bd45248b3d46/root/var/svc/log/oxide-crucible-downstairs:default.log.0
10182023-12-13T06:46:33.754Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_63d19713-226b-44b9-be21-0fec38d6aabb/root/var/svc/log/oxide-crucible-agent:default.log (5928 bytes)
10192023-12-13T06:46:34.849Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_63d19713-226b-44b9-be21-0fec38d6aabb/root/var/svc/log/oxide-crucible-agent:default.log
10202023-12-13T06:46:34.849Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_63d19713-226b-44b9-be21-0fec38d6aabb/root/var/svc/log/oxide-crucible-agent:default.log.0 (3436 bytes)
10212023-12-13T06:46:35.945Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_63d19713-226b-44b9-be21-0fec38d6aabb/root/var/svc/log/oxide-crucible-agent:default.log.0
10222023-12-13T06:46:35.945Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_63d19713-226b-44b9-be21-0fec38d6aabb/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
10232023-12-13T06:46:37.014Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_63d19713-226b-44b9-be21-0fec38d6aabb/root/var/svc/log/oxide-crucible-downstairs:default.log
10242023-12-13T06:46:37.015Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_63d19713-226b-44b9-be21-0fec38d6aabb/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
10252023-12-13T06:46:38.109Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_63d19713-226b-44b9-be21-0fec38d6aabb/root/var/svc/log/oxide-crucible-downstairs:default.log.0
10262023-12-13T06:46:38.109Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_63d19713-226b-44b9-be21-0fec38d6aabb/root/var/svc/log/oxide-crucible-downstairs:downstairs-563f4eb1-594e-4aef-b472-847a33e572ec.log (2932 bytes)
10272023-12-13T06:46:39.206Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_63d19713-226b-44b9-be21-0fec38d6aabb/root/var/svc/log/oxide-crucible-downstairs:downstairs-563f4eb1-594e-4aef-b472-847a33e572ec.log
10282023-12-13T06:46:39.206Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_834be57c-4dff-4da4-8da0-bed590823081/root/var/svc/log/oxide-crucible-agent:default.log (5930 bytes)
10292023-12-13T06:46:40.300Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_834be57c-4dff-4da4-8da0-bed590823081/root/var/svc/log/oxide-crucible-agent:default.log
10302023-12-13T06:46:40.300Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_834be57c-4dff-4da4-8da0-bed590823081/root/var/svc/log/oxide-crucible-agent:default.log.0 (3433 bytes)
10312023-12-13T06:46:41.394Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_834be57c-4dff-4da4-8da0-bed590823081/root/var/svc/log/oxide-crucible-agent:default.log.0
10322023-12-13T06:46:41.395Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_834be57c-4dff-4da4-8da0-bed590823081/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
10332023-12-13T06:46:42.465Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_834be57c-4dff-4da4-8da0-bed590823081/root/var/svc/log/oxide-crucible-downstairs:default.log
10342023-12-13T06:46:42.465Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_834be57c-4dff-4da4-8da0-bed590823081/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
10352023-12-13T06:46:43.559Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_834be57c-4dff-4da4-8da0-bed590823081/root/var/svc/log/oxide-crucible-downstairs:default.log.0
10362023-12-13T06:46:43.559Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_834be57c-4dff-4da4-8da0-bed590823081/root/var/svc/log/oxide-crucible-downstairs:downstairs-6a89e933-67a9-4f24-b9af-efe1e9cbba88.log (2930 bytes)
10372023-12-13T06:46:44.688Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_834be57c-4dff-4da4-8da0-bed590823081/root/var/svc/log/oxide-crucible-downstairs:downstairs-6a89e933-67a9-4f24-b9af-efe1e9cbba88.log
10382023-12-13T06:46:44.688Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_1a716dfe-13c4-425b-82ad-98c5b9a8e7a8/root/var/svc/log/oxide-crucible-pantry:default.log (78055 bytes)
10392023-12-13T06:46:45.901Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_1a716dfe-13c4-425b-82ad-98c5b9a8e7a8/root/var/svc/log/oxide-crucible-pantry:default.log
10402023-12-13T06:46:45.902Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_1a716dfe-13c4-425b-82ad-98c5b9a8e7a8/root/var/svc/log/oxide-crucible-pantry:default.log.0 (1147135 bytes)
10412023-12-13T06:46:48.051Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_1a716dfe-13c4-425b-82ad-98c5b9a8e7a8/root/var/svc/log/oxide-crucible-pantry:default.log.0
10422023-12-13T06:46:48.051Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_71e4c81c-01ec-4989-9eec-dadde20e0c82/root/var/svc/log/oxide-crucible-pantry:default.log (0 bytes)
10432023-12-13T06:46:49.121Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_71e4c81c-01ec-4989-9eec-dadde20e0c82/root/var/svc/log/oxide-crucible-pantry:default.log
10442023-12-13T06:46:49.122Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_71e4c81c-01ec-4989-9eec-dadde20e0c82/root/var/svc/log/oxide-crucible-pantry:default.log.0 (2337 bytes)
10452023-12-13T06:46:50.215Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_pantry_71e4c81c-01ec-4989-9eec-dadde20e0c82/root/var/svc/log/oxide-crucible-pantry:default.log.0
10462023-12-13T06:46:50.215Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_78790969-f959-4dbc-8b9a-9c6fbc413bf9/root/var/svc/log/oxide-internal_dns:default.log (3754 bytes)
10472023-12-13T06:46:50.283Zupload warning: file "/pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_78790969-f959-4dbc-8b9a-9c6fbc413bf9/root/var/svc/log/oxide-internal_dns:default.log" changed size mid upload: 3754 -> 7507
10482023-12-13T06:46:51.353Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_78790969-f959-4dbc-8b9a-9c6fbc413bf9/root/var/svc/log/oxide-internal_dns:default.log
10492023-12-13T06:46:51.354Zuploading: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_78790969-f959-4dbc-8b9a-9c6fbc413bf9/root/var/svc/log/oxide-internal_dns:default.log.0 (123319 bytes)
10502023-12-13T06:46:52.646Zuploaded: /pool/ext/24b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_internal_dns_78790969-f959-4dbc-8b9a-9c6fbc413bf9/root/var/svc/log/oxide-internal_dns:default.log.0
10512023-12-13T06:46:52.646Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_clickhouse_0e17a6f7-bb1c-4708-a92b-8989c36cd8d3/root/var/svc/log/oxide-clickhouse:default.log (87002 bytes)
10522023-12-13T06:46:52.955Zupload warning: file "/pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_clickhouse_0e17a6f7-bb1c-4708-a92b-8989c36cd8d3/root/var/svc/log/oxide-clickhouse:default.log" changed size mid upload: 87002 -> 167063
10532023-12-13T06:47:06.239Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_clickhouse_0e17a6f7-bb1c-4708-a92b-8989c36cd8d3/root/var/svc/log/oxide-clickhouse:default.log
10542023-12-13T06:47:06.239Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_clickhouse_0e17a6f7-bb1c-4708-a92b-8989c36cd8d3/root/var/svc/log/oxide-clickhouse:default.log.0 (637662 bytes)
10552023-12-13T06:47:07.843Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_clickhouse_0e17a6f7-bb1c-4708-a92b-8989c36cd8d3/root/var/svc/log/oxide-clickhouse:default.log.0
10562023-12-13T06:47:07.843Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_b623a645-71e4-47d7-9a9f-5690818b0e68/root/var/svc/log/oxide-crucible-agent:default.log (0 bytes)
10572023-12-13T06:47:09.367Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_b623a645-71e4-47d7-9a9f-5690818b0e68/root/var/svc/log/oxide-crucible-agent:default.log
10582023-12-13T06:47:09.367Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_b623a645-71e4-47d7-9a9f-5690818b0e68/root/var/svc/log/oxide-crucible-agent:default.log.0 (3436 bytes)
10592023-12-13T06:47:10.483Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_b623a645-71e4-47d7-9a9f-5690818b0e68/root/var/svc/log/oxide-crucible-agent:default.log.0
10602023-12-13T06:47:10.483Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_b623a645-71e4-47d7-9a9f-5690818b0e68/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
10612023-12-13T06:47:11.555Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_b623a645-71e4-47d7-9a9f-5690818b0e68/root/var/svc/log/oxide-crucible-downstairs:default.log
10622023-12-13T06:47:11.555Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_b623a645-71e4-47d7-9a9f-5690818b0e68/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
10632023-12-13T06:47:12.650Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_crucible_b623a645-71e4-47d7-9a9f-5690818b0e68/root/var/svc/log/oxide-crucible-downstairs:default.log.0
10642023-12-13T06:47:12.650Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_external_dns_dbd562a6-352e-4fd4-8124-fec6bb412043/root/var/svc/log/oxide-external_dns:default.log (3777 bytes)
10652023-12-13T06:47:12.734Zupload warning: file "/pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_external_dns_dbd562a6-352e-4fd4-8124-fec6bb412043/root/var/svc/log/oxide-external_dns:default.log" changed size mid upload: 3777 -> 7555
10662023-12-13T06:47:13.810Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_external_dns_dbd562a6-352e-4fd4-8124-fec6bb412043/root/var/svc/log/oxide-external_dns:default.log
10672023-12-13T06:47:13.811Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_external_dns_dbd562a6-352e-4fd4-8124-fec6bb412043/root/var/svc/log/oxide-external_dns:default.log.0 (23163 bytes)
10682023-12-13T06:47:14.951Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_external_dns_dbd562a6-352e-4fd4-8124-fec6bb412043/root/var/svc/log/oxide-external_dns:default.log.0
10692023-12-13T06:47:14.951Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_nexus_c2b46da6-4b55-43b0-ac4b-8282863aa18a/root/var/svc/log/oxide-nexus:default.log (80581 bytes)
10702023-12-13T06:47:15.268Zupload warning: file "/pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_nexus_c2b46da6-4b55-43b0-ac4b-8282863aa18a/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 80581 -> 153136
10712023-12-13T06:47:16.339Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_nexus_c2b46da6-4b55-43b0-ac4b-8282863aa18a/root/var/svc/log/oxide-nexus:default.log
10722023-12-13T06:47:16.339Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_nexus_c2b46da6-4b55-43b0-ac4b-8282863aa18a/root/var/svc/log/oxide-nexus:default.log.0 (704010 bytes)
10732023-12-13T06:47:18.292Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_nexus_c2b46da6-4b55-43b0-ac4b-8282863aa18a/root/var/svc/log/oxide-nexus:default.log.0
10742023-12-13T06:47:18.292Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_ntp_6c95d6e6-e4e5-43a3-ba3b-3d3584cfe251/root/var/svc/log/oxide-ntp:default.log (0 bytes)
10752023-12-13T06:47:19.413Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_ntp_6c95d6e6-e4e5-43a3-ba3b-3d3584cfe251/root/var/svc/log/oxide-ntp:default.log
10762023-12-13T06:47:19.413Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_ntp_6c95d6e6-e4e5-43a3-ba3b-3d3584cfe251/root/var/svc/log/oxide-ntp:default.log.0 (778 bytes)
10772023-12-13T06:47:20.508Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_ntp_6c95d6e6-e4e5-43a3-ba3b-3d3584cfe251/root/var/svc/log/oxide-ntp:default.log.0
10782023-12-13T06:47:20.508Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_oximeter_1ec26749-1b83-4c43-9e43-9dab8f3f00da/root/var/svc/log/oxide-oximeter:default.log (37605 bytes)
10792023-12-13T06:47:20.708Zupload warning: file "/pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_oximeter_1ec26749-1b83-4c43-9e43-9dab8f3f00da/root/var/svc/log/oxide-oximeter:default.log" changed size mid upload: 37605 -> 78632
10802023-12-13T06:47:21.778Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_oximeter_1ec26749-1b83-4c43-9e43-9dab8f3f00da/root/var/svc/log/oxide-oximeter:default.log
10812023-12-13T06:47:21.779Zuploading: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_oximeter_1ec26749-1b83-4c43-9e43-9dab8f3f00da/root/var/svc/log/oxide-oximeter:default.log.0 (185313 bytes)
10822023-12-13T06:47:23.158Zuploaded: /pool/ext/616b26df-e62a-4c68-b506-f4a923d8aaf7/crypt/zone/oxz_oximeter_1ec26749-1b83-4c43-9e43-9dab8f3f00da/root/var/svc/log/oxide-oximeter:default.log.0
10832023-12-13T06:47:23.158Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_8246e2e5-d44c-4c57-81a5-aeaaf523d56c/root/var/svc/log/oxide-crucible-agent:default.log (0 bytes)
10842023-12-13T06:47:24.229Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_8246e2e5-d44c-4c57-81a5-aeaaf523d56c/root/var/svc/log/oxide-crucible-agent:default.log
10852023-12-13T06:47:24.230Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_8246e2e5-d44c-4c57-81a5-aeaaf523d56c/root/var/svc/log/oxide-crucible-agent:default.log.0 (14151 bytes)
10862023-12-13T06:47:25.346Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_8246e2e5-d44c-4c57-81a5-aeaaf523d56c/root/var/svc/log/oxide-crucible-agent:default.log.0
10872023-12-13T06:47:25.348Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_8246e2e5-d44c-4c57-81a5-aeaaf523d56c/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
10882023-12-13T06:47:26.422Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_8246e2e5-d44c-4c57-81a5-aeaaf523d56c/root/var/svc/log/oxide-crucible-downstairs:default.log
10892023-12-13T06:47:26.422Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_8246e2e5-d44c-4c57-81a5-aeaaf523d56c/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
10902023-12-13T06:47:27.519Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_8246e2e5-d44c-4c57-81a5-aeaaf523d56c/root/var/svc/log/oxide-crucible-downstairs:default.log.0
10912023-12-13T06:47:27.519Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_8246e2e5-d44c-4c57-81a5-aeaaf523d56c/root/var/svc/log/oxide-crucible-downstairs:downstairs-c5205a14-376a-4d50-a482-3be47a2617d2.log (0 bytes)
10922023-12-13T06:47:28.589Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_8246e2e5-d44c-4c57-81a5-aeaaf523d56c/root/var/svc/log/oxide-crucible-downstairs:downstairs-c5205a14-376a-4d50-a482-3be47a2617d2.log
10932023-12-13T06:47:28.590Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_8246e2e5-d44c-4c57-81a5-aeaaf523d56c/root/var/svc/log/oxide-crucible-downstairs:downstairs-c5205a14-376a-4d50-a482-3be47a2617d2.log.0 (3050 bytes)
10942023-12-13T06:47:29.695Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_crucible_8246e2e5-d44c-4c57-81a5-aeaaf523d56c/root/var/svc/log/oxide-crucible-downstairs:downstairs-c5205a14-376a-4d50-a482-3be47a2617d2.log.0
10952023-12-13T06:47:29.695Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_external_dns_a3b0f71c-0188-4968-bb40-4fcbe4032ecc/root/var/svc/log/oxide-external_dns:default.log (3777 bytes)
10962023-12-13T06:47:29.772Zupload warning: file "/pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_external_dns_a3b0f71c-0188-4968-bb40-4fcbe4032ecc/root/var/svc/log/oxide-external_dns:default.log" changed size mid upload: 3777 -> 7552
10972023-12-13T06:47:30.852Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_external_dns_a3b0f71c-0188-4968-bb40-4fcbe4032ecc/root/var/svc/log/oxide-external_dns:default.log
10982023-12-13T06:47:30.853Zuploading: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_external_dns_a3b0f71c-0188-4968-bb40-4fcbe4032ecc/root/var/svc/log/oxide-external_dns:default.log.0 (23167 bytes)
10992023-12-13T06:47:31.998Zuploaded: /pool/ext/cd70d7f6-2354-4bf2-8012-55bf9eaf7930/crypt/zone/oxz_external_dns_a3b0f71c-0188-4968-bb40-4fcbe4032ecc/root/var/svc/log/oxide-external_dns:default.log.0
11002023-12-13T06:47:31.998Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_a2168a07-3770-43d5-a975-fe511a071cf9/root/var/svc/log/oxide-crucible-agent:default.log (23933 bytes)
11012023-12-13T06:47:33.150Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_a2168a07-3770-43d5-a975-fe511a071cf9/root/var/svc/log/oxide-crucible-agent:default.log
11022023-12-13T06:47:33.150Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_a2168a07-3770-43d5-a975-fe511a071cf9/root/var/svc/log/oxide-crucible-agent:default.log.0 (20506 bytes)
11032023-12-13T06:47:34.323Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_a2168a07-3770-43d5-a975-fe511a071cf9/root/var/svc/log/oxide-crucible-agent:default.log.0
11042023-12-13T06:47:34.323Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_a2168a07-3770-43d5-a975-fe511a071cf9/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
11052023-12-13T06:47:35.365Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_a2168a07-3770-43d5-a975-fe511a071cf9/root/var/svc/log/oxide-crucible-downstairs:default.log
11062023-12-13T06:47:35.365Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_a2168a07-3770-43d5-a975-fe511a071cf9/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
11072023-12-13T06:47:36.469Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_a2168a07-3770-43d5-a975-fe511a071cf9/root/var/svc/log/oxide-crucible-downstairs:default.log.0
11082023-12-13T06:47:36.469Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_a2168a07-3770-43d5-a975-fe511a071cf9/root/var/svc/log/oxide-crucible-downstairs:downstairs-264a1d26-b8b2-4c2d-9b83-7a711905ffbc.log (5514 bytes)
11092023-12-13T06:47:37.569Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_a2168a07-3770-43d5-a975-fe511a071cf9/root/var/svc/log/oxide-crucible-downstairs:downstairs-264a1d26-b8b2-4c2d-9b83-7a711905ffbc.log
11102023-12-13T06:47:37.571Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_a2168a07-3770-43d5-a975-fe511a071cf9/root/var/svc/log/oxide-crucible-downstairs:downstairs-a1682200-44e7-4aad-8a98-7560b9ce516a.log (0 bytes)
11112023-12-13T06:47:38.644Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_a2168a07-3770-43d5-a975-fe511a071cf9/root/var/svc/log/oxide-crucible-downstairs:downstairs-a1682200-44e7-4aad-8a98-7560b9ce516a.log
11122023-12-13T06:47:38.644Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_a2168a07-3770-43d5-a975-fe511a071cf9/root/var/svc/log/oxide-crucible-downstairs:downstairs-a1682200-44e7-4aad-8a98-7560b9ce516a.log.0 (3050 bytes)
11132023-12-13T06:47:41.175Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_a2168a07-3770-43d5-a975-fe511a071cf9/root/var/svc/log/oxide-crucible-downstairs:downstairs-a1682200-44e7-4aad-8a98-7560b9ce516a.log.0
11142023-12-13T06:47:41.175Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_a2168a07-3770-43d5-a975-fe511a071cf9/root/var/svc/log/oxide-crucible-downstairs:downstairs-c6c4f32a-397a-47a5-bdce-a014141fa861.log (2932 bytes)
11152023-12-13T06:47:42.253Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_a2168a07-3770-43d5-a975-fe511a071cf9/root/var/svc/log/oxide-crucible-downstairs:downstairs-c6c4f32a-397a-47a5-bdce-a014141fa861.log
11162023-12-13T06:47:42.253Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_a2168a07-3770-43d5-a975-fe511a071cf9/root/var/svc/log/oxide-crucible-downstairs:downstairs-ff5c3954-706e-463f-8a6a-9de765cb6694.log (1902 bytes)
11172023-12-13T06:47:43.352Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_a2168a07-3770-43d5-a975-fe511a071cf9/root/var/svc/log/oxide-crucible-downstairs:downstairs-ff5c3954-706e-463f-8a6a-9de765cb6694.log
11182023-12-13T06:47:43.352Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_a2168a07-3770-43d5-a975-fe511a071cf9/root/var/svc/log/oxide-crucible-downstairs:downstairs-ff5c3954-706e-463f-8a6a-9de765cb6694.log.0 (4529 bytes)
11192023-12-13T06:47:44.444Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_a2168a07-3770-43d5-a975-fe511a071cf9/root/var/svc/log/oxide-crucible-downstairs:downstairs-ff5c3954-706e-463f-8a6a-9de765cb6694.log.0
11202023-12-13T06:47:44.444Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_a2168a07-3770-43d5-a975-fe511a071cf9/root/var/svc/log/oxide-crucible-downstairs:snapshot-ff5c3954-706e-463f-8a6a-9de765cb6694-1a70593c-ed4b-48da-aba9-a3e4e480a772.log (5154 bytes)
11212023-12-13T06:47:45.538Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_crucible_a2168a07-3770-43d5-a975-fe511a071cf9/root/var/svc/log/oxide-crucible-downstairs:snapshot-ff5c3954-706e-463f-8a6a-9de765cb6694-1a70593c-ed4b-48da-aba9-a3e4e480a772.log
11222023-12-13T06:47:45.538Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_internal_dns_b4d118f0-3b3a-4885-b4cc-29ec593913d7/root/var/svc/log/oxide-internal_dns:default.log (3755 bytes)
11232023-12-13T06:47:45.607Zupload warning: file "/pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_internal_dns_b4d118f0-3b3a-4885-b4cc-29ec593913d7/root/var/svc/log/oxide-internal_dns:default.log" changed size mid upload: 3755 -> 11265
11242023-12-13T06:47:46.677Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_internal_dns_b4d118f0-3b3a-4885-b4cc-29ec593913d7/root/var/svc/log/oxide-internal_dns:default.log
11252023-12-13T06:47:46.677Zuploading: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_internal_dns_b4d118f0-3b3a-4885-b4cc-29ec593913d7/root/var/svc/log/oxide-internal_dns:default.log.0 (124532 bytes)
11262023-12-13T06:47:47.991Zuploaded: /pool/ext/ceb4461c-cf56-4719-ad3c-14430bfdfb60/crypt/zone/oxz_internal_dns_b4d118f0-3b3a-4885-b4cc-29ec593913d7/root/var/svc/log/oxide-internal_dns:default.log.0
11272023-12-13T06:47:47.993Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_cockroachdb_42923cc8-26a8-4278-a81f-a122ff9b3c20/root/var/svc/log/oxide-cockroachdb:default.log (0 bytes)
11282023-12-13T06:47:49.062Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_cockroachdb_42923cc8-26a8-4278-a81f-a122ff9b3c20/root/var/svc/log/oxide-cockroachdb:default.log
11292023-12-13T06:47:49.062Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_cockroachdb_42923cc8-26a8-4278-a81f-a122ff9b3c20/root/var/svc/log/oxide-cockroachdb:default.log.0 (5414 bytes)
11302023-12-13T06:47:50.155Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_cockroachdb_42923cc8-26a8-4278-a81f-a122ff9b3c20/root/var/svc/log/oxide-cockroachdb:default.log.0
11312023-12-13T06:47:50.155Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_cockroachdb_f199bf85-7c0c-4b9c-ae48-4bb04a9f5ac8/root/var/svc/log/oxide-cockroachdb:default.log (0 bytes)
11322023-12-13T06:47:51.231Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_cockroachdb_f199bf85-7c0c-4b9c-ae48-4bb04a9f5ac8/root/var/svc/log/oxide-cockroachdb:default.log
11332023-12-13T06:47:51.231Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_cockroachdb_f199bf85-7c0c-4b9c-ae48-4bb04a9f5ac8/root/var/svc/log/oxide-cockroachdb:default.log.0 (5413 bytes)
11342023-12-13T06:47:52.327Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_cockroachdb_f199bf85-7c0c-4b9c-ae48-4bb04a9f5ac8/root/var/svc/log/oxide-cockroachdb:default.log.0
11352023-12-13T06:47:52.327Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_pantry_b303730f-2a33-4405-a33f-861d2cc654ab/root/var/svc/log/oxide-crucible-pantry:default.log (0 bytes)
11362023-12-13T06:47:53.399Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_pantry_b303730f-2a33-4405-a33f-861d2cc654ab/root/var/svc/log/oxide-crucible-pantry:default.log
11372023-12-13T06:47:53.400Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_pantry_b303730f-2a33-4405-a33f-861d2cc654ab/root/var/svc/log/oxide-crucible-pantry:default.log.0 (2338 bytes)
11382023-12-13T06:47:54.494Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_crucible_pantry_b303730f-2a33-4405-a33f-861d2cc654ab/root/var/svc/log/oxide-crucible-pantry:default.log.0
11392023-12-13T06:47:54.494Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_nexus_14f7a0d4-f78e-4fbe-b3ce-b9baa26d8784/root/var/svc/log/oxide-nexus:default.log (3797351 bytes)
11402023-12-13T06:47:57.084Zupload warning: file "/pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_nexus_14f7a0d4-f78e-4fbe-b3ce-b9baa26d8784/root/var/svc/log/oxide-nexus:default.log" changed size mid upload: 3797351 -> 3936215
11412023-12-13T06:47:58.169Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_nexus_14f7a0d4-f78e-4fbe-b3ce-b9baa26d8784/root/var/svc/log/oxide-nexus:default.log
11422023-12-13T06:47:58.169Zuploading: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_nexus_14f7a0d4-f78e-4fbe-b3ce-b9baa26d8784/root/var/svc/log/oxide-nexus:default.log.0 (32025855 bytes)
11432023-12-13T06:48:15.692Zuploaded: /pool/ext/d462a7f7-b628-40fe-80ff-4e4189e2d62b/crypt/zone/oxz_nexus_14f7a0d4-f78e-4fbe-b3ce-b9baa26d8784/root/var/svc/log/oxide-nexus:default.log.0
11442023-12-13T06:48:15.692Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_c33950df-a1cd-4f4f-9e9a-09ba89e2ac6d/root/var/svc/log/oxide-crucible-agent:default.log (10681 bytes)
11452023-12-13T06:48:16.808Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_c33950df-a1cd-4f4f-9e9a-09ba89e2ac6d/root/var/svc/log/oxide-crucible-agent:default.log
11462023-12-13T06:48:16.808Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_c33950df-a1cd-4f4f-9e9a-09ba89e2ac6d/root/var/svc/log/oxide-crucible-agent:default.log.0 (3436 bytes)
11472023-12-13T06:48:17.902Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_c33950df-a1cd-4f4f-9e9a-09ba89e2ac6d/root/var/svc/log/oxide-crucible-agent:default.log.0
11482023-12-13T06:48:17.902Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_c33950df-a1cd-4f4f-9e9a-09ba89e2ac6d/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
11492023-12-13T06:48:18.972Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_c33950df-a1cd-4f4f-9e9a-09ba89e2ac6d/root/var/svc/log/oxide-crucible-downstairs:default.log
11502023-12-13T06:48:18.972Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_c33950df-a1cd-4f4f-9e9a-09ba89e2ac6d/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
11512023-12-13T06:48:20.069Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_c33950df-a1cd-4f4f-9e9a-09ba89e2ac6d/root/var/svc/log/oxide-crucible-downstairs:default.log.0
11522023-12-13T06:48:20.069Zuploading: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_c33950df-a1cd-4f4f-9e9a-09ba89e2ac6d/root/var/svc/log/oxide-crucible-downstairs:downstairs-cf48804a-e588-46e7-bae7-4b5345652396.log (5514 bytes)
11532023-12-13T06:48:21.164Zuploaded: /pool/ext/e4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_c33950df-a1cd-4f4f-9e9a-09ba89e2ac6d/root/var/svc/log/oxide-crucible-downstairs:downstairs-cf48804a-e588-46e7-bae7-4b5345652396.log
11542023-12-13T06:48:21.164Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_3a154fe9-359c-456f-be6f-51bb8496e1b0/root/var/svc/log/oxide-crucible-agent:default.log (18006 bytes)
11552023-12-13T06:48:22.301Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_3a154fe9-359c-456f-be6f-51bb8496e1b0/root/var/svc/log/oxide-crucible-agent:default.log
11562023-12-13T06:48:22.301Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_3a154fe9-359c-456f-be6f-51bb8496e1b0/root/var/svc/log/oxide-crucible-agent:default.log.0 (9368 bytes)
11572023-12-13T06:48:23.418Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_3a154fe9-359c-456f-be6f-51bb8496e1b0/root/var/svc/log/oxide-crucible-agent:default.log.0
11582023-12-13T06:48:23.418Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_3a154fe9-359c-456f-be6f-51bb8496e1b0/root/var/svc/log/oxide-crucible-downstairs:default.log (0 bytes)
11592023-12-13T06:48:24.497Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_3a154fe9-359c-456f-be6f-51bb8496e1b0/root/var/svc/log/oxide-crucible-downstairs:default.log
11602023-12-13T06:48:24.497Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_3a154fe9-359c-456f-be6f-51bb8496e1b0/root/var/svc/log/oxide-crucible-downstairs:default.log.0 (75 bytes)
11612023-12-13T06:48:25.594Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_3a154fe9-359c-456f-be6f-51bb8496e1b0/root/var/svc/log/oxide-crucible-downstairs:default.log.0
11622023-12-13T06:48:25.594Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_3a154fe9-359c-456f-be6f-51bb8496e1b0/root/var/svc/log/oxide-crucible-downstairs:downstairs-6c3851a4-a29c-49f2-9b16-9c19a56184a8.log (5515 bytes)
11632023-12-13T06:48:26.690Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_3a154fe9-359c-456f-be6f-51bb8496e1b0/root/var/svc/log/oxide-crucible-downstairs:downstairs-6c3851a4-a29c-49f2-9b16-9c19a56184a8.log
11642023-12-13T06:48:26.690Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_3a154fe9-359c-456f-be6f-51bb8496e1b0/root/var/svc/log/oxide-crucible-downstairs:downstairs-e68f4f8d-14f0-4c16-8c13-41638c3e78ab.log (1904 bytes)
11652023-12-13T06:48:27.782Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_3a154fe9-359c-456f-be6f-51bb8496e1b0/root/var/svc/log/oxide-crucible-downstairs:downstairs-e68f4f8d-14f0-4c16-8c13-41638c3e78ab.log
11662023-12-13T06:48:27.782Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_3a154fe9-359c-456f-be6f-51bb8496e1b0/root/var/svc/log/oxide-crucible-downstairs:downstairs-e68f4f8d-14f0-4c16-8c13-41638c3e78ab.log.0 (4529 bytes)
11672023-12-13T06:48:28.877Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_3a154fe9-359c-456f-be6f-51bb8496e1b0/root/var/svc/log/oxide-crucible-downstairs:downstairs-e68f4f8d-14f0-4c16-8c13-41638c3e78ab.log.0
11682023-12-13T06:48:28.879Zuploading: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_3a154fe9-359c-456f-be6f-51bb8496e1b0/root/var/svc/log/oxide-crucible-downstairs:snapshot-e68f4f8d-14f0-4c16-8c13-41638c3e78ab-1a70593c-ed4b-48da-aba9-a3e4e480a772.log (5154 bytes)
11692023-12-13T06:48:29.988Zuploaded: /pool/ext/f4b4dc87-ab46-49fb-a4b4-d361ae214c03/crypt/zone/oxz_crucible_3a154fe9-359c-456f-be6f-51bb8496e1b0/root/var/svc/log/oxide-crucible-downstairs:snapshot-e68f4f8d-14f0-4c16-8c13-41638c3e78ab-1a70593c-ed4b-48da-aba9-a3e4e480a772.log
11702023-12-13T06:48:29.988Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_ntp_6c95d6e6-e4e5-43a3-ba3b-3d3584cfe251/root/var/log/chrony/measurements.log (21509 bytes)
11712023-12-13T06:48:30.079Zupload warning: file "/pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_ntp_6c95d6e6-e4e5-43a3-ba3b-3d3584cfe251/root/var/log/chrony/measurements.log" changed size mid upload: 21509 -> 26989
11722023-12-13T06:48:31.149Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_ntp_6c95d6e6-e4e5-43a3-ba3b-3d3584cfe251/root/var/log/chrony/measurements.log
11732023-12-13T06:48:31.150Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_ntp_6c95d6e6-e4e5-43a3-ba3b-3d3584cfe251/root/var/log/chrony/statistics.log (14391 bytes)
11742023-12-13T06:48:31.240Zupload warning: file "/pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_ntp_6c95d6e6-e4e5-43a3-ba3b-3d3584cfe251/root/var/log/chrony/statistics.log" changed size mid upload: 14391 -> 19071
11752023-12-13T06:48:32.311Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_ntp_6c95d6e6-e4e5-43a3-ba3b-3d3584cfe251/root/var/log/chrony/statistics.log
11762023-12-13T06:48:32.311Zuploading: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_ntp_6c95d6e6-e4e5-43a3-ba3b-3d3584cfe251/root/var/log/chrony/tracking.log (1719 bytes)
11772023-12-13T06:48:32.357Zupload warning: file "/pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_ntp_6c95d6e6-e4e5-43a3-ba3b-3d3584cfe251/root/var/log/chrony/tracking.log" changed size mid upload: 1719 -> 2115
11782023-12-13T06:48:33.426Zuploaded: /pool/ext/31bd71cd-4736-4a12-a387-9b74b050396f/crypt/zone/oxz_ntp_6c95d6e6-e4e5-43a3-ba3b-3d3584cfe251/root/var/log/chrony/tracking.log