01JP0TT60NSEBJ3P7GWCSWZW1V: phd-run-migrate-from-base

BasicConfig {
    output_rules: [
        "/tmp/phd-runner.log",
        "/tmp/phd-tmp-files.tar.gz",
    ],
    rust_toolchain: None,
    target: Some(
        "lab-2.0-opte",
    ),
    access_repos: [],
    publish: [],
    skip_clone: true,
}

Buildomat Job: 01JP0TTM6PWCN4F2SCS1KKP61E

Tags:

Artefacts:

Output:

SEQ GLOBAL TIME DETAILS
12025-03-10T20:17:06.210Zjob dependencies complete; ready to run (waiting for 7 m 32 s)
22025-03-10T21:06:43.515Zjob assigned to worker 01JP0Y1N69XZ1ZBWWDW0XEK2AQ [factory lab, buskin/13918] (queued for 49 m 37 s)
32025-03-10T21:06:43.787Zhttp://172.20.0.5:9969/ipxe/buskin... ok
42025-03-10T21:06:43.888Zhttp://172.20.0.5:9969/os/buskin/platform/i86pc/kernel/amd64/unix... ok
52025-03-10T21:06:43.922Zhttp://172.20.0.5:9969/os/buskin/platform/i86pc/amd64/boot_archive... ok
62025-03-10T21:06:43.967Zhttp://172.20.0.5:9969/os/buskin/platform/i86pc/amd64/boot_archive.hash... ok
72025-03-10T21:06:44.012Zhttp://172.20.0.5:9969/postboot/buskin... ok
82025-03-10T21:06:44.046ZwcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U wcnCroia6U
92025-03-10T21:06:44.080Zcpu0: microcode has been updated from version 0xa001133 to 0xa0011d5
102025-03-10T21:06:44.114ZOxide Helios Version helios-2.0.23032 64-bit
112025-03-10T21:06:44.158ZConfiguring devices.
122025-03-10T21:06:44.192Z##################################################
132025-03-10T21:06:44.226Z#### ############# ##
142025-03-10T21:06:44.261Z### ### ############ ##
152025-03-10T21:06:44.294Z## ### # ## ### ## Oxide Computer Company ##
162025-03-10T21:06:44.328Z## ## ## ### # ### ##
172025-03-10T21:06:44.362Z## # ### #### #### This Station Under ##
182025-03-10T21:06:44.395Z### ### #### # ### Computer Control ##
192025-03-10T21:06:44.429Z#### #### ### ## ##
202025-03-10T21:06:44.462Z##################################################
212025-03-10T21:06:44.496ZHostname: buskin
222025-03-10T21:06:44.540ZLoading smf(7) service descriptions:
232025-03-10T21:06:45.470Zdownloading input: /input/phd-build/out/phd-run-with-args.sh
242025-03-10T21:06:45.494Zdownloaded input: /input/phd-build/out/phd-run-with-args.sh
252025-03-10T21:06:45.494Zdownloading input: /input/phd-build/out/phd-runner.sha256.txt
262025-03-10T21:06:45.520Zdownloaded input: /input/phd-build/out/phd-runner.sha256.txt
272025-03-10T21:06:45.521Zdownloading input: /input/phd-build/out/phd-runner.tar.gz
282025-03-10T21:06:58.115Zdownloaded input: /input/phd-build/out/phd-runner.tar.gz
292025-03-10T21:06:58.115Zdownloading input: /input/phd-build/out/propolis-server-debug.sha256.txt
302025-03-10T21:06:58.167Zdownloaded input: /input/phd-build/out/propolis-server-debug.sha256.txt
312025-03-10T21:06:58.167Zdownloading input: /input/phd-build/out/propolis-server-debug.tar.gz
322025-03-10T21:07:22.259Zdownloaded input: /input/phd-build/out/propolis-server-debug.tar.gz
 
332025-03-10T21:07:22.284Zstarting task 0: "setup"
342025-03-10T21:07:22.284Z++ uname -s
352025-03-10T21:07:22.284Z+ kern=SunOS
362025-03-10T21:07:22.284Z+ build_user=build
372025-03-10T21:07:22.284Z+ build_uid=12345
382025-03-10T21:07:22.284Z+ work_dir=/work
392025-03-10T21:07:22.284Z+ input_dir=/input
402025-03-10T21:07:22.284Z+ [[ 0 == 12345 ]]
412025-03-10T21:07:22.285Z+ case "$kern" in
422025-03-10T21:07:22.285Z+ groupadd -g 12345 build
432025-03-10T21:07:22.285Z+ useradd -u 12345 -g build -d /home/build -s /bin/bash -c build -P 'Primary Administrator' build
442025-03-10T21:07:24.277Z+ zfs create -o mountpoint=/work rpool/work
452025-03-10T21:07:24.301Z++ awk '$2 == "/home" { print $3 }' /etc/mnttab
462025-03-10T21:07:24.301Z+ home_fs=zfs
472025-03-10T21:07:24.301Z+ [[ zfs == autofs ]]
482025-03-10T21:07:24.301Z+ mkdir -p /home/build
492025-03-10T21:07:24.301Z+ chown build:build /home/build /work
502025-03-10T21:07:26.300Z+ chmod 0700 /home/build /work
512025-03-10T21:07:26.324Zprocess exited: duration 4043 ms, exit code 0
 
522025-03-10T21:07:26.373Zstarting task 1: "authentication"
532025-03-10T21:07:26.418Zprocess exited: duration 44 ms, exit code 0
 
542025-03-10T21:07:26.468Zstarting task 2: "build"
552025-03-10T21:07:26.493Z+ indir=/input
562025-03-10T21:07:26.493Z+ indir_suffix='phd-build/out/*.tar.gz'
572025-03-10T21:07:26.493Z+ phddir=/work/phd-test
582025-03-10T21:07:26.493Z+ pfexec zpool create -f phd-artifacts c1t1d0 c2t1d0
592025-03-10T21:07:26.602Z+ artifactdir=/phd-artifacts
602025-03-10T21:07:26.602Z+ banner Inputs
612025-03-10T21:07:26.627Z ###
622025-03-10T21:07:26.628Z # # # ##### # # ##### ####
632025-03-10T21:07:26.628Z # ## # # # # # # #
642025-03-10T21:07:26.628Z # # # # # # # # # ####
652025-03-10T21:07:26.628Z # # # # ##### # # # #
662025-03-10T21:07:26.628Z # # ## # # # # # #
672025-03-10T21:07:26.628Z ### # # # #### # ####
682025-03-10T21:07:26.628Z
692025-03-10T21:07:26.628Z+ find /input -ls
702025-03-10T21:07:26.628Z 34 1 drwxr-xr-x 3 root root 3 Mar 10 21:06 /input
712025-03-10T21:07:26.628Z 2 1 drwxr-xr-x 3 root root 3 Mar 10 21:06 /input/phd-build
722025-03-10T21:07:26.628Z 3 2 drwxr-xr-x 2 root root 7 Mar 10 21:06 /input/phd-build/out
732025-03-10T21:07:26.628Z 129 1 -rw-r--r-- 1 root root 65 Mar 10 21:06 /input/phd-build/out/phd-runner.sha256.txt
742025-03-10T21:07:26.628Z 256 1 -rw-r--r-- 1 root root 65 Mar 10 21:06 /input/phd-build/out/propolis-server-debug.sha256.txt
752025-03-10T21:07:26.628Z 128 2 -rw-r--r-- 1 root root 2099 Mar 10 21:06 /input/phd-build/out/phd-run-with-args.sh
762025-03-10T21:07:26.628Z 257 136499 -rw-r--r-- 1 root root 139683757 Mar 10 21:07 /input/phd-build/out/propolis-server-debug.tar.gz
772025-03-10T21:07:26.628Z 130 70231 -rw-r--r-- 1 root root 71869834 Mar 10 21:06 /input/phd-build/out/phd-runner.tar.gz
782025-03-10T21:07:26.628Z+ rm -rf /work/phd-test
792025-03-10T21:07:26.628Z+ mkdir /work/phd-test
802025-03-10T21:07:26.628Z+ for p in $indir/$indir_suffix
812025-03-10T21:07:26.628Z+ tar xzvf /input/phd-build/out/phd-runner.tar.gz -C /work/phd-test
822025-03-10T21:07:28.188ZDecompressing '/input/phd-build/out/phd-runner.tar.gz' with '/usr/bin/gzcat'...
832025-03-10T21:07:28.188Zx phd-runner, 254737928 bytes, 497536 tape blocks
842025-03-10T21:07:28.188Zx artifacts.toml, 548 bytes, 2 tape blocks
852025-03-10T21:07:28.213Z++ tar tf /input/phd-build/out/phd-runner.tar.gz
862025-03-10T21:07:29.702Z+ for f in $(tar tf "$p")
872025-03-10T21:07:29.702Z+ chmod +x /work/phd-test/phd-runner
882025-03-10T21:07:29.727Z+ for f in $(tar tf "$p")
892025-03-10T21:07:29.727Z+ chmod +x /work/phd-test/artifacts.toml
902025-03-10T21:07:29.727Z+ for p in $indir/$indir_suffix
912025-03-10T21:07:29.727Z+ tar xzvf /input/phd-build/out/propolis-server-debug.tar.gz -C /work/phd-test
922025-03-10T21:07:32.940ZDecompressing '/input/phd-build/out/propolis-server-debug.tar.gz' with '/usr/bin/gzcat'...
932025-03-10T21:07:32.941Zx propolis-server, 526748336 bytes, 1028806 tape blocks
942025-03-10T21:07:32.966Z++ tar tf /input/phd-build/out/propolis-server-debug.tar.gz
952025-03-10T21:07:35.868Z+ for f in $(tar tf "$p")
962025-03-10T21:07:35.900Z+ chmod +x /work/phd-test/propolis-server
972025-03-10T21:07:35.900Z+ ls /work/phd-test
982025-03-10T21:07:35.900Zartifacts.toml
992025-03-10T21:07:35.900Zphd-runner
1002025-03-10T21:07:35.900Zpropolis-server
1012025-03-10T21:07:35.901Z+ banner Setup
1022025-03-10T21:07:35.901Z #####
1032025-03-10T21:07:35.901Z# # ###### ##### # # #####
1042025-03-10T21:07:35.901Z# # # # # # #
1052025-03-10T21:07:35.901Z ##### ##### # # # # #
1062025-03-10T21:07:35.901Z # # # # # #####
1072025-03-10T21:07:35.901Z# # # # # # #
1082025-03-10T21:07:35.901Z ##### ###### # #### #
1092025-03-10T21:07:35.901Z
1102025-03-10T21:07:35.901Z+ tmpdir=/tmp/propolis-phd
1112025-03-10T21:07:35.901Z+ '[' '!' -d /tmp/propolis-phd ']'
1122025-03-10T21:07:35.901Z+ mkdir /tmp/propolis-phd
1132025-03-10T21:07:35.901Z+ banner Tests
1142025-03-10T21:07:35.901Z#######
1152025-03-10T21:07:35.901Z # ###### #### ##### ####
1162025-03-10T21:07:35.901Z # # # # #
1172025-03-10T21:07:35.901Z # ##### #### # ####
1182025-03-10T21:07:35.901Z # # # # #
1192025-03-10T21:07:35.901Z # # # # # # #
1202025-03-10T21:07:35.901Z # ###### #### # ####
1212025-03-10T21:07:35.901Z
1222025-03-10T21:07:35.901Z+ runner=/work/phd-test/phd-runner
1232025-03-10T21:07:35.901Z+ artifacts=/work/phd-test/artifacts.toml
1242025-03-10T21:07:35.901Z+ propolis=/work/phd-test/propolis-server
1252025-03-10T21:07:35.901Z+ ls /work/phd-test/phd-runner
1262025-03-10T21:07:35.901Z/work/phd-test/phd-runner
1272025-03-10T21:07:35.901Z+ ls /work/phd-test/artifacts.toml
1282025-03-10T21:07:35.901Z/work/phd-test/artifacts.toml
1292025-03-10T21:07:35.901Z+ ls /work/phd-test/propolis-server
1302025-03-10T21:07:35.901Z/work/phd-test/propolis-server
1312025-03-10T21:07:35.901Z+ args=($runner '--emit-bunyan' 'run' '--propolis-server-cmd' $propolis '--crucible-downstairs-commit' 'auto' '--artifact-toml-path' $artifacts '--tmp-directory' $tmpdir '--artifact-directory' $artifactdir '--parallelism' 2 $@)
1322025-03-10T21:07:35.901Z+ set +e
1332025-03-10T21:07:35.901Z+ RUST_BACKTRACE=1
1342025-03-10T21:07:35.901Z+ RUST_LOG=info,phd=debug
1352025-03-10T21:07:35.901Z+ ptime -m pfexec /work/phd-test/phd-runner --emit-bunyan run --propolis-server-cmd /work/phd-test/propolis-server --crucible-downstairs-commit auto --artifact-toml-path /work/phd-test/artifacts.toml --tmp-directory /tmp/propolis-phd --artifact-directory /phd-artifacts --parallelism 2 --include-filter phd_tests::migrate::from_base --base-propolis-branch master
1362025-03-10T21:07:35.902Z+ tee /tmp/phd-runner.log
1372025-03-10T21:07:35.927Z{"v":0,"name":"phd-runner","msg":"phd_runner","level":30,"hostname":"buskin","pid":862,"time":"2025-03-10T21:07:36.152943998Z","target":"phd_runner","line":36,"file":"phd-tests/runner/src/main.rs","runner_args":"ProcessArgs { command: Run(RunOptions { propolis_server_cmd: \"/work/phd-test/propolis-server\", base_propolis_branch: Some(\"master\"), base_propolis_commit: None, base_propolis_cmd: None, crucible_downstairs_cmd: None, crucible_downstairs_commit: Some(Auto), tmp_directory: \"/tmp/propolis-phd\", artifact_directory: Some(\"/phd-artifacts\"), server_logging_mode: TmpFile, parallelism: Some(2), default_guest_cpus: 2, default_guest_memory_mib: 512, artifact_toml_path: \"/work/phd-test/artifacts.toml\", default_guest_artifact: \"alpine\", default_bootrom_artifact: \"ovmf\", include_filter: [\"phd_tests::migrate::from_base\"], exclude_filter: [], max_buildomat_wait_secs: 1200 }), disable_ansi: false, emit_bunyan: true }"}
1382025-03-10T21:07:35.927Z{"v":0,"name":"phd-runner","msg":"running tests with max parallelism of 2","level":30,"hostname":"buskin","pid":862,"time":"2025-03-10T21:07:36.153336034Z","target":"phd_runner","line":167,"file":"phd-tests/runner/src/main.rs"}
1392025-03-10T21:07:35.927Z{"v":0,"name":"phd-runner","msg":"Created new artifact store from manifest","level":20,"hostname":"buskin","pid":862,"time":"2025-03-10T21:07:36.155648339Z","target":"phd_framework::artifacts::store","line":221,"file":"phd-tests/framework/src/artifacts/store.rs","store":"Store { local_dir: \"/phd-artifacts/runner-0\", artifacts: {\"alpine\": Mutex { data: StoredArtifact { description: Artifact { filename: \"alpine.iso\", kind: GuestOs(Alpine), source: RemoteServer { sha256: \"ba8007f74f9b54fbae3b2520da577831b4834778a498d732f091260c61aa7ca1\" }, untar: None }, cached_path: None } }, \"ovmf\": Mutex { data: StoredArtifact { description: Artifact { filename: \"OVMF_CODE.fd\", kind: Bootrom, source: Buildomat(BuildomatArtifact { repo: Repo(\"oxidecomputer/edk2\"), series: Series(\"image_debug\"), commit: Commit(\"6d92acf0a22718dd4175d7c64dbcf7aaec3740bd\"), sha256: \"29813374b58e3b77fb665f2d95cb3bab37d44fdd2c4fce2a70de9d76a3512a4f\" }), untar: None }, cached_path: None } }}, downloader: DownloadConfig { timeout: 600s, buildomat_backoff: ExponentialBackoff { current_interval: 1s, initial_interval: 1s, randomization_factor: 0.5, multiplier: 1.5, max_interval: 60s, start_time: Instant { tv_sec: 205, tv_nsec: 922383082 }, max_elapsed_time: Some(1200s), clock: SystemClock }, remote_server_uris: [\"https://oxide-omicron-build.s3.amazonaws.com\"] } }"}
1402025-03-10T21:07:35.927Z{"v":0,"name":"phd-runner","msg":"Adding Propolis server from local command","level":30,"hostname":"buskin","pid":862,"time":"2025-03-10T21:07:36.15569969Z","target":"phd_framework::artifacts::store","line":229,"file":"phd-tests/framework/src/artifacts/store.rs","propolis_server_cmd":"/work/phd-test/propolis-server"}
1412025-03-10T21:07:35.927Z{"v":0,"name":"phd-runner","msg":"Adding crucible-downstairs from Buildomat Git revision","level":30,"hostname":"buskin","pid":862,"time":"2025-03-10T21:07:36.155827262Z","target":"phd_framework::artifacts::store","line":313,"file":"phd-tests/framework/src/artifacts/store.rs","commit":"d2d8f8ad449df7e2befb7ee2723a442dd74b9b72"}
1422025-03-10T21:07:35.927Z{"v":0,"name":"phd-runner","msg":"Downloading file from Buildomat...","level":20,"hostname":"buskin","pid":862,"time":"2025-03-10T21:07:36.155964634Z","target":"phd_framework::artifacts::buildomat","line":215,"file":"phd-tests/framework/src/artifacts/buildomat.rs","timeout":"600s","uri":"https://buildomat.eng.oxide.computer/public/file/oxidecomputer/crucible/nightly-image/d2d8f8ad449df7e2befb7ee2723a442dd74b9b72/crucible-nightly.sha256.txt"}
1432025-03-10T21:07:36.026Z{"v":0,"name":"phd-runner","msg":"Adding 'current' Propolis server from Buildomat Git branch 'master'","level":30,"hostname":"buskin","pid":862,"time":"2025-03-10T21:07:36.263302348Z","target":"phd_framework::artifacts::store","line":250,"file":"phd-tests/framework/src/artifacts/store.rs"}
1442025-03-10T21:07:36.515Z{"v":0,"name":"phd-runner","msg":"Downloading file from Buildomat...","level":20,"hostname":"buskin","pid":862,"time":"2025-03-10T21:07:36.752877207Z","target":"phd_framework::artifacts::buildomat","line":215,"file":"phd-tests/framework/src/artifacts/buildomat.rs","timeout":"600s","uri":"https://buildomat.eng.oxide.computer/public/file/oxidecomputer/propolis/phd_build/9ee2e1f5f8d7cc3543b6985af36c0df7f81498c5/propolis-server.sha256.txt"}
1452025-03-10T21:07:36.780Z{"v":0,"name":"phd-runner","msg":"Created new artifact store from manifest","level":20,"hostname":"buskin","pid":862,"time":"2025-03-10T21:07:37.017692602Z","target":"phd_framework::artifacts::store","line":221,"file":"phd-tests/framework/src/artifacts/store.rs","store":"Store { local_dir: \"/phd-artifacts/runner-1\", artifacts: {\"alpine\": Mutex { data: StoredArtifact { description: Artifact { filename: \"alpine.iso\", kind: GuestOs(Alpine), source: RemoteServer { sha256: \"ba8007f74f9b54fbae3b2520da577831b4834778a498d732f091260c61aa7ca1\" }, untar: None }, cached_path: None } }, \"ovmf\": Mutex { data: StoredArtifact { description: Artifact { filename: \"OVMF_CODE.fd\", kind: Bootrom, source: Buildomat(BuildomatArtifact { repo: Repo(\"oxidecomputer/edk2\"), series: Series(\"image_debug\"), commit: Commit(\"6d92acf0a22718dd4175d7c64dbcf7aaec3740bd\"), sha256: \"29813374b58e3b77fb665f2d95cb3bab37d44fdd2c4fce2a70de9d76a3512a4f\" }), untar: None }, cached_path: None } }}, downloader: DownloadConfig { timeout: 600s, buildomat_backoff: ExponentialBackoff { current_interval: 1s, initial_interval: 1s, randomization_factor: 0.5, multiplier: 1.5, max_interval: 60s, start_time: Instant { tv_sec: 206, tv_nsec: 784618598 }, max_elapsed_time: Some(1200s), clock: SystemClock }, remote_server_uris: [\"https://oxide-omicron-build.s3.amazonaws.com\"] } }"}
1462025-03-10T21:07:36.780Z{"v":0,"name":"phd-runner","msg":"Adding Propolis server from local command","level":30,"hostname":"buskin","pid":862,"time":"2025-03-10T21:07:37.017786853Z","target":"phd_framework::artifacts::store","line":229,"file":"phd-tests/framework/src/artifacts/store.rs","propolis_server_cmd":"/work/phd-test/propolis-server"}
1472025-03-10T21:07:36.780Z{"v":0,"name":"phd-runner","msg":"Adding crucible-downstairs from Buildomat Git revision","level":30,"hostname":"buskin","pid":862,"time":"2025-03-10T21:07:37.017829044Z","target":"phd_framework::artifacts::store","line":313,"file":"phd-tests/framework/src/artifacts/store.rs","commit":"d2d8f8ad449df7e2befb7ee2723a442dd74b9b72"}
1482025-03-10T21:07:36.805Z{"v":0,"name":"phd-runner","msg":"Downloading file from Buildomat...","level":20,"hostname":"buskin","pid":862,"time":"2025-03-10T21:07:37.017859945Z","target":"phd_framework::artifacts::buildomat","line":215,"file":"phd-tests/framework/src/artifacts/buildomat.rs","timeout":"600s","uri":"https://buildomat.eng.oxide.computer/public/file/oxidecomputer/crucible/nightly-image/d2d8f8ad449df7e2befb7ee2723a442dd74b9b72/crucible-nightly.sha256.txt"}
1492025-03-10T21:07:36.885Z{"v":0,"name":"phd-runner","msg":"Adding 'current' Propolis server from Buildomat Git branch 'master'","level":30,"hostname":"buskin","pid":862,"time":"2025-03-10T21:07:37.12268031Z","target":"phd_framework::artifacts::store","line":250,"file":"phd-tests/framework/src/artifacts/store.rs"}
1502025-03-10T21:07:37.306Z{"v":0,"name":"phd-runner","msg":"Downloading file from Buildomat...","level":20,"hostname":"buskin","pid":862,"time":"2025-03-10T21:07:37.5431487Z","target":"phd_framework::artifacts::buildomat","line":215,"file":"phd-tests/framework/src/artifacts/buildomat.rs","timeout":"600s","uri":"https://buildomat.eng.oxide.computer/public/file/oxidecomputer/propolis/phd_build/9ee2e1f5f8d7cc3543b6985af36c0df7f81498c5/propolis-server.sha256.txt"}
1512025-03-10T21:07:37.409Z{"v":0,"name":"phd-runner","msg":"Running 3 test(s)","level":30,"hostname":"buskin","pid":862,"time":"2025-03-10T21:07:37.64691212Z","target":"phd_runner::execute","line":161,"file":"phd-tests/runner/src/execute.rs"}
1522025-03-10T21:07:37.438Z{"v":0,"name":"phd-runner","msg":"[EXECUTION_SETUP - START]","level":30,"hostname":"buskin","pid":862,"time":"2025-03-10T21:07:37.647301815Z","target":"phd_runner::fixtures","line":25,"file":"phd-tests/runner/src/fixtures.rs"}
1532025-03-10T21:07:37.438Z{"v":0,"name":"phd-runner","msg":"[EXECUTION_SETUP - START]","level":30,"hostname":"buskin","pid":862,"time":"2025-03-10T21:07:37.647322236Z","target":"phd_runner::fixtures","line":25,"file":"phd-tests/runner/src/fixtures.rs"}
1542025-03-10T21:07:37.438Z{"v":0,"name":"phd-runner","msg":"[EXECUTION_SETUP - END]","level":30,"hostname":"buskin","pid":862,"time":"2025-03-10T21:07:37.647461978Z","target":"phd_runner::fixtures","line":25,"file":"phd-tests/runner/src/fixtures.rs","elapsed_milliseconds":0}
1552025-03-10T21:07:37.438Z{"v":0,"name":"phd-runner","msg":"[EXECUTION_SETUP - END]","level":30,"hostname":"buskin","pid":862,"time":"2025-03-10T21:07:37.647477238Z","target":"phd_runner::fixtures","line":25,"file":"phd-tests/runner/src/fixtures.rs","elapsed_milliseconds":0}
1562025-03-10T21:07:37.438Z{"v":0,"name":"phd-runner","msg":"Starting test phd_tests::migrate::from_base::migration_from_base_and_back","level":30,"hostname":"buskin","pid":862,"time":"2025-03-10T21:07:37.64761763Z","target":"phd_runner::execute","line":101,"file":"phd-tests/runner/src/execute.rs"}
1572025-03-10T21:07:37.438Z{"v":0,"name":"phd-runner","msg":"Starting test phd_tests::migrate::from_base::serial_history","level":30,"hostname":"buskin","pid":862,"time":"2025-03-10T21:07:37.64762618Z","target":"phd_runner::execute","line":101,"file":"phd-tests/runner/src/execute.rs"}
1582025-03-10T21:07:37.439Z{"v":0,"name":"phd-runner","msg":"[TEST_SETUP - START]","level":30,"hostname":"buskin","pid":862,"time":"2025-03-10T21:07:37.647686561Z","target":"phd_runner::fixtures","line":41,"file":"phd-tests/runner/src/fixtures.rs"}
1592025-03-10T21:07:37.439Z{"v":0,"name":"phd-runner","msg":"[TEST_SETUP - START]","level":30,"hostname":"buskin","pid":862,"time":"2025-03-10T21:07:37.647688831Z","target":"phd_runner::fixtures","line":41,"file":"phd-tests/runner/src/fixtures.rs"}
1602025-03-10T21:07:37.439Z{"v":0,"name":"phd-runner","msg":"[TEST_SETUP - END]","level":30,"hostname":"buskin","pid":862,"time":"2025-03-10T21:07:37.647757222Z","target":"phd_runner::fixtures","line":41,"file":"phd-tests/runner/src/fixtures.rs","elapsed_milliseconds":0}
1612025-03-10T21:07:37.439Z{"v":0,"name":"phd-runner","msg":"[TEST_SETUP - END]","level":30,"hostname":"buskin","pid":862,"time":"2025-03-10T21:07:37.647758942Z","target":"phd_runner::fixtures","line":41,"file":"phd-tests/runner/src/fixtures.rs","elapsed_milliseconds":0}
1622025-03-10T21:07:37.439Z{"v":0,"name":"phd-runner","msg":"[TEST - START]","level":30,"hostname":"buskin","pid":862,"time":"2025-03-10T21:07:37.647955816Z","target":"phd_tests::migrate::from_base","line":52,"file":"phd-tests/tests/src/migrate.rs","path":"phd_tests::migrate::from_base::migration_from_base_and_back"}
1632025-03-10T21:07:37.439Z{"v":0,"name":"phd-runner","msg":"[TEST - START]","level":30,"hostname":"buskin","pid":862,"time":"2025-03-10T21:07:37.647960596Z","target":"phd_tests::migrate::from_base","line":40,"file":"phd-tests/tests/src/migrate.rs","path":"phd_tests::migrate::from_base::serial_history"}
1642025-03-10T21:07:37.439Z{"v":0,"name":"phd-runner","msg":"[TEST - EVENT] checking for existing copy of artifact","level":20,"hostname":"buskin","pid":862,"time":"2025-03-10T21:07:37.648343561Z","target":"phd_framework::artifacts::store","line":89,"file":"phd-tests/framework/src/artifacts/store.rs","maybe_path":"/phd-artifacts/runner-0/29813374b58e3b77fb665f2d95cb3bab37d44fdd2c4fce2a70de9d76a3512a4f/OVMF_CODE.fd","path":"phd_tests::migrate::from_base::serial_history"}
1652025-03-10T21:07:37.439Z{"v":0,"name":"phd-runner","msg":"[TEST - EVENT] checking for existing copy of artifact","level":20,"hostname":"buskin","pid":862,"time":"2025-03-10T21:07:37.648344011Z","target":"phd_framework::artifacts::store","line":89,"file":"phd-tests/framework/src/artifacts/store.rs","maybe_path":"/phd-artifacts/runner-1/29813374b58e3b77fb665f2d95cb3bab37d44fdd2c4fce2a70de9d76a3512a4f/OVMF_CODE.fd","path":"phd_tests::migrate::from_base::migration_from_base_and_back"}
1662025-03-10T21:07:37.439Z{"v":0,"name":"phd-runner","msg":"[TEST - EVENT] Downloading file from Buildomat...","level":20,"hostname":"buskin","pid":862,"time":"2025-03-10T21:07:37.648520484Z","target":"phd_framework::artifacts::buildomat","line":215,"file":"phd-tests/framework/src/artifacts/buildomat.rs","uri":"https://buildomat.eng.oxide.computer/public/file/oxidecomputer/edk2/image_debug/6d92acf0a22718dd4175d7c64dbcf7aaec3740bd/OVMF_CODE.fd","timeout":"600s","path":"phd_tests::migrate::from_base::migration_from_base_and_back"}
1672025-03-10T21:07:37.439Z{"v":0,"name":"phd-runner","msg":"[TEST - EVENT] Downloading file from Buildomat...","level":20,"hostname":"buskin","pid":862,"time":"2025-03-10T21:07:37.648531674Z","target":"phd_framework::artifacts::buildomat","line":215,"file":"phd-tests/framework/src/artifacts/buildomat.rs","timeout":"600s","uri":"https://buildomat.eng.oxide.computer/public/file/oxidecomputer/edk2/image_debug/6d92acf0a22718dd4175d7c64dbcf7aaec3740bd/OVMF_CODE.fd","path":"phd_tests::migrate::from_base::serial_history"}
1682025-03-10T21:07:38.008Z{"v":0,"name":"phd-runner","msg":"[TEST - EVENT] checking for existing copy of artifact","level":20,"hostname":"buskin","pid":862,"time":"2025-03-10T21:07:38.245241764Z","target":"phd_framework::artifacts::store","line":89,"file":"phd-tests/framework/src/artifacts/store.rs","maybe_path":"/phd-artifacts/runner-1/ba8007f74f9b54fbae3b2520da577831b4834778a498d732f091260c61aa7ca1/alpine.iso","path":"phd_tests::migrate::from_base::migration_from_base_and_back"}
1692025-03-10T21:07:38.033Z{"v":0,"name":"phd-runner","msg":"[TEST - EVENT] Downloading alpine.iso from https://oxide-omicron-build.s3.amazonaws.com/alpine.iso","level":20,"hostname":"buskin","pid":862,"time":"2025-03-10T21:07:38.245387506Z","target":"phd_framework::artifacts::store","line":621,"file":"phd-tests/framework/src/artifacts/store.rs","timeout":"600s","path":"phd_tests::migrate::from_base::migration_from_base_and_back"}
1702025-03-10T21:07:38.059Z{"v":0,"name":"phd-runner","msg":"[TEST - EVENT] checking for existing copy of artifact","level":20,"hostname":"buskin","pid":862,"time":"2025-03-10T21:07:38.274322169Z","target":"phd_framework::artifacts::store","line":89,"file":"phd-tests/framework/src/artifacts/store.rs","maybe_path":"/phd-artifacts/runner-0/ba8007f74f9b54fbae3b2520da577831b4834778a498d732f091260c61aa7ca1/alpine.iso","path":"phd_tests::migrate::from_base::serial_history"}
1712025-03-10T21:07:38.059Z{"v":0,"name":"phd-runner","msg":"[TEST - EVENT] Downloading alpine.iso from https://oxide-omicron-build.s3.amazonaws.com/alpine.iso","level":20,"hostname":"buskin","pid":862,"time":"2025-03-10T21:07:38.274460431Z","target":"phd_framework::artifacts::store","line":621,"file":"phd-tests/framework/src/artifacts/store.rs","timeout":"600s","path":"phd_tests::migrate::from_base::serial_history"}
1722025-03-10T21:08:12.126Z{"v":0,"name":"phd-runner","msg":"[TEST - EVENT] Verified artifact already exists","level":20,"hostname":"buskin","pid":862,"time":"2025-03-10T21:08:12.362485863Z","target":"phd_framework::artifacts::store","line":45,"file":"phd-tests/framework/src/artifacts/store.rs","path":"/phd-artifacts/runner-0/ba8007f74f9b54fbae3b2520da577831b4834778a498d732f091260c61aa7ca1/alpine.iso","path":"phd_tests::migrate::from_base::serial_history"}
1732025-03-10T21:08:12.151Z{"v":0,"name":"phd-runner","msg":"[TEST - EVENT] executing ZFS command","level":20,"hostname":"buskin","pid":862,"time":"2025-03-10T21:08:12.362632026Z","target":"phd_framework::zfs","line":304,"file":"phd-tests/framework/src/zfs.rs","verb":"list","args":"[\"/phd-artifacts/runner-0/ba8007f74f9b54fbae3b2520da577831b4834778a498d732f091260c61aa7ca1/alpine.iso\"]","path":"phd_tests::migrate::from_base::serial_history"}
1742025-03-10T21:08:12.176Z{"v":0,"name":"phd-runner","msg":"[TEST - EVENT] parsing zfs list output","level":20,"hostname":"buskin","pid":862,"time":"2025-03-10T21:08:12.405553323Z","target":"phd_framework::zfs","line":238,"file":"phd-tests/framework/src/zfs.rs","stdout":"NAME USED AVAIL REFER MOUNTPOINT\nphd-artifacts 3.90M 1.80T 3.80M /phd-artifacts\n","path":"phd_tests::migrate::from_base::serial_history"}
1752025-03-10T21:08:12.176Z{"v":0,"name":"phd-runner","msg":"[TEST - EVENT] executing ZFS command","level":20,"hostname":"buskin","pid":862,"time":"2025-03-10T21:08:12.40597801Z","target":"phd_framework::zfs","line":304,"file":"phd-tests/framework/src/zfs.rs","args":"[\"phd-artifacts@phd-f8ef2ae3-f355-46e0-85f8-83bde2919e44\"]","verb":"snapshot","path":"phd_tests::migrate::from_base::serial_history"}
1762025-03-10T21:08:12.226Z{"v":0,"name":"phd-runner","msg":"[TEST - EVENT] executing ZFS command","level":20,"hostname":"buskin","pid":862,"time":"2025-03-10T21:08:12.462673568Z","target":"phd_framework::zfs","line":304,"file":"phd-tests/framework/src/zfs.rs","args":"[\"phd-artifacts@phd-f8ef2ae3-f355-46e0-85f8-83bde2919e44\", \"phd-artifacts/phd-clone-3bb4d8f0-493e-49cf-9703-8c855f29d4fe\"]","verb":"clone","path":"phd_tests::migrate::from_base::serial_history"}
1772025-03-10T21:08:12.254Z{"v":0,"name":"phd-runner","msg":"[TEST - EVENT] executing ZFS command","level":20,"hostname":"buskin","pid":862,"time":"2025-03-10T21:08:12.490968091Z","target":"phd_framework::zfs","line":304,"file":"phd-tests/framework/src/zfs.rs","args":"[\"phd-artifacts/phd-clone-3bb4d8f0-493e-49cf-9703-8c855f29d4fe\"]","verb":"list","path":"phd_tests::migrate::from_base::serial_history"}
1782025-03-10T21:08:12.281Z{"v":0,"name":"phd-runner","msg":"[TEST - EVENT] parsing zfs list output","level":20,"hostname":"buskin","pid":862,"time":"2025-03-10T21:08:12.517906724Z","target":"phd_framework::zfs","line":238,"file":"phd-tests/framework/src/zfs.rs","stdout":"NAME USED AVAIL REFER MOUNTPOINT\nphd-artifacts/phd-clone-3bb4d8f0-493e-49cf-9703-8c855f29d4fe 0B 1.80T 50.8M /phd-artifacts/phd-clone-3bb4d8f0-493e-49cf-9703-8c855f29d4fe\n","path":"phd_tests::migrate::from_base::serial_history"}
1792025-03-10T21:08:12.306Z{"v":0,"name":"phd-runner","msg":"[NEW - START]","level":30,"hostname":"buskin","pid":862,"time":"2025-03-10T21:08:12.518596254Z","target":"phd_framework::test_vm","line":193,"file":"phd-tests/framework/src/test_vm/mod.rs","path":"phd_tests::migrate::from_base::serial_history"}
1802025-03-10T21:08:12.307Z{"v":0,"name":"phd-runner","msg":"[NEW - EVENT] phd_framework::test_vm","level":30,"hostname":"buskin","pid":862,"time":"2025-03-10T21:08:12.518657615Z","target":"phd_framework::test_vm","line":211,"file":"phd-tests/framework/src/test_vm/mod.rs","vm_name":"migration_serial_history_base","guest_os_kind":"Alpine","environment":"EnvironmentSpec { location: Local, propolis_artifact: \"__BASE_PROPOLIS\" }","path":"phd_tests::migrate::from_base::serial_history"}
1812025-03-10T21:08:12.307Z{"v":0,"name":"phd-runner","msg":"[NEW - EVENT] checking for existing copy of artifact","level":20,"hostname":"buskin","pid":862,"time":"2025-03-10T21:08:12.518715756Z","target":"phd_framework::artifacts::store","line":89,"file":"phd-tests/framework/src/artifacts/store.rs","maybe_path":"/phd-artifacts/runner-0/699c2d2c5e97e63733ab85fce044388baefe882761373e5c02190990dd286770/propolis-server.tar.gz","path":"phd_tests::migrate::from_base::serial_history"}
1822025-03-10T21:08:12.307Z{"v":0,"name":"phd-runner","msg":"[NEW - EVENT] Downloading file from Buildomat...","level":20,"hostname":"buskin","pid":862,"time":"2025-03-10T21:08:12.518779767Z","target":"phd_framework::artifacts::buildomat","line":215,"file":"phd-tests/framework/src/artifacts/buildomat.rs","timeout":"600s","uri":"https://buildomat.eng.oxide.computer/public/file/oxidecomputer/propolis/phd_build/9ee2e1f5f8d7cc3543b6985af36c0df7f81498c5/propolis-server.tar.gz","path":"phd_tests::migrate::from_base::serial_history"}
1832025-03-10T21:08:12.332Z{"v":0,"name":"phd-runner","msg":"[TEST - EVENT] Verified artifact already exists","level":20,"hostname":"buskin","pid":862,"time":"2025-03-10T21:08:12.552852759Z","target":"phd_framework::artifacts::store","line":45,"file":"phd-tests/framework/src/artifacts/store.rs","path":"/phd-artifacts/runner-1/ba8007f74f9b54fbae3b2520da577831b4834778a498d732f091260c61aa7ca1/alpine.iso","path":"phd_tests::migrate::from_base::migration_from_base_and_back"}
1842025-03-10T21:08:12.332Z{"v":0,"name":"phd-runner","msg":"[TEST - EVENT] executing ZFS command","level":20,"hostname":"buskin","pid":862,"time":"2025-03-10T21:08:12.55293189Z","target":"phd_framework::zfs","line":304,"file":"phd-tests/framework/src/zfs.rs","args":"[\"/phd-artifacts/runner-1/ba8007f74f9b54fbae3b2520da577831b4834778a498d732f091260c61aa7ca1/alpine.iso\"]","verb":"list","path":"phd_tests::migrate::from_base::migration_from_base_and_back"}
1852025-03-10T21:08:12.357Z{"v":0,"name":"phd-runner","msg":"[TEST - EVENT] parsing zfs list output","level":20,"hostname":"buskin","pid":862,"time":"2025-03-10T21:08:12.574786935Z","target":"phd_framework::zfs","line":238,"file":"phd-tests/framework/src/zfs.rs","stdout":"NAME USED AVAIL REFER MOUNTPOINT\nphd-artifacts 50.9M 1.80T 50.8M /phd-artifacts\n","path":"phd_tests::migrate::from_base::migration_from_base_and_back"}
1862025-03-10T21:08:12.357Z{"v":0,"name":"phd-runner","msg":"[TEST - EVENT] executing ZFS command","level":20,"hostname":"buskin","pid":862,"time":"2025-03-10T21:08:12.574910097Z","target":"phd_framework::zfs","line":304,"file":"phd-tests/framework/src/zfs.rs","args":"[\"phd-artifacts@phd-68aa1763-4002-4b53-ae10-15f8d1625940\"]","verb":"snapshot","path":"phd_tests::migrate::from_base::migration_from_base_and_back"}
1872025-03-10T21:08:12.396Z{"v":0,"name":"phd-runner","msg":"[TEST - EVENT] executing ZFS command","level":20,"hostname":"buskin","pid":862,"time":"2025-03-10T21:08:12.633739128Z","target":"phd_framework::zfs","line":304,"file":"phd-tests/framework/src/zfs.rs","verb":"clone","args":"[\"phd-artifacts@phd-68aa1763-4002-4b53-ae10-15f8d1625940\", \"phd-artifacts/phd-clone-9d36a2f8-ffed-490c-b405-c06b63013ecc\"]","path":"phd_tests::migrate::from_base::migration_from_base_and_back"}
1882025-03-10T21:08:12.421Z{"v":0,"name":"phd-runner","msg":"[TEST - EVENT] executing ZFS command","level":20,"hostname":"buskin","pid":862,"time":"2025-03-10T21:08:12.657066455Z","target":"phd_framework::zfs","line":304,"file":"phd-tests/framework/src/zfs.rs","args":"[\"phd-artifacts/phd-clone-9d36a2f8-ffed-490c-b405-c06b63013ecc\"]","verb":"list","path":"phd_tests::migrate::from_base::migration_from_base_and_back"}
1892025-03-10T21:08:12.447Z{"v":0,"name":"phd-runner","msg":"[TEST - EVENT] parsing zfs list output","level":20,"hostname":"buskin","pid":862,"time":"2025-03-10T21:08:12.671283263Z","target":"phd_framework::zfs","line":238,"file":"phd-tests/framework/src/zfs.rs","stdout":"NAME USED AVAIL REFER MOUNTPOINT\nphd-artifacts/phd-clone-9d36a2f8-ffed-490c-b405-c06b63013ecc 0B 1.80T 97.9M /phd-artifacts/phd-clone-9d36a2f8-ffed-490c-b405-c06b63013ecc\n","path":"phd_tests::migrate::from_base::migration_from_base_and_back"}
1902025-03-10T21:08:12.447Z{"v":0,"name":"phd-runner","msg":"[NEW - START]","level":30,"hostname":"buskin","pid":862,"time":"2025-03-10T21:08:12.671793151Z","target":"phd_framework::test_vm","line":193,"file":"phd-tests/framework/src/test_vm/mod.rs","path":"phd_tests::migrate::from_base::migration_from_base_and_back"}
1912025-03-10T21:08:12.447Z{"v":0,"name":"phd-runner","msg":"[NEW - EVENT] phd_framework::test_vm","level":30,"hostname":"buskin","pid":862,"time":"2025-03-10T21:08:12.671853792Z","target":"phd_framework::test_vm","line":211,"file":"phd-tests/framework/src/test_vm/mod.rs","vm_name":"migration_from_base_and_back","environment":"EnvironmentSpec { location: Local, propolis_artifact: \"__BASE_PROPOLIS\" }","guest_os_kind":"Alpine","path":"phd_tests::migrate::from_base::migration_from_base_and_back"}
1922025-03-10T21:08:12.447Z{"v":0,"name":"phd-runner","msg":"[NEW - EVENT] checking for existing copy of artifact","level":20,"hostname":"buskin","pid":862,"time":"2025-03-10T21:08:12.671917853Z","target":"phd_framework::artifacts::store","line":89,"file":"phd-tests/framework/src/artifacts/store.rs","maybe_path":"/phd-artifacts/runner-1/699c2d2c5e97e63733ab85fce044388baefe882761373e5c02190990dd286770/propolis-server.tar.gz","path":"phd_tests::migrate::from_base::migration_from_base_and_back"}
1932025-03-10T21:08:12.447Z{"v":0,"name":"phd-runner","msg":"[NEW - EVENT] Downloading file from Buildomat...","level":20,"hostname":"buskin","pid":862,"time":"2025-03-10T21:08:12.671980924Z","target":"phd_framework::artifacts::buildomat","line":215,"file":"phd-tests/framework/src/artifacts/buildomat.rs","timeout":"600s","uri":"https://buildomat.eng.oxide.computer/public/file/oxidecomputer/propolis/phd_build/9ee2e1f5f8d7cc3543b6985af36c0df7f81498c5/propolis-server.tar.gz","path":"phd_tests::migrate::from_base::migration_from_base_and_back"}
1942025-03-10T21:08:36.808Z{"v":0,"name":"phd-runner","msg":"[NEW - EVENT] Extracting artifact from tarball","level":20,"hostname":"buskin","pid":862,"time":"2025-03-10T21:08:37.044425432Z","target":"phd_framework::artifacts::store","line":162,"file":"phd-tests/framework/src/artifacts/store.rs","untar_path":"propolis-server","extracted_path":"/phd-artifacts/runner-0/699c2d2c5e97e63733ab85fce044388baefe882761373e5c02190990dd286770/propolis-server","path":"phd_tests::migrate::from_base::serial_history"}
1952025-03-10T21:08:36.832Z{"v":0,"name":"phd-runner","msg":"[NEW - EVENT] Extracting gzipped tarball...","level":20,"hostname":"buskin","pid":862,"time":"2025-03-10T21:08:37.044538973Z","target":"phd_framework::artifacts::store","line":521,"file":"phd-tests/framework/src/artifacts/store.rs","path":"phd_tests::migrate::from_base::serial_history"}
1962025-03-10T21:08:36.978Z{"v":0,"name":"phd-runner","msg":"[NEW - EVENT] Extracting artifact from tarball","level":20,"hostname":"buskin","pid":862,"time":"2025-03-10T21:08:37.215307869Z","target":"phd_framework::artifacts::store","line":162,"file":"phd-tests/framework/src/artifacts/store.rs","untar_path":"propolis-server","extracted_path":"/phd-artifacts/runner-1/699c2d2c5e97e63733ab85fce044388baefe882761373e5c02190990dd286770/propolis-server","path":"phd_tests::migrate::from_base::migration_from_base_and_back"}
1972025-03-10T21:08:36.978Z{"v":0,"name":"phd-runner","msg":"[NEW - EVENT] Extracting gzipped tarball...","level":20,"hostname":"buskin","pid":862,"time":"2025-03-10T21:08:37.21537375Z","target":"phd_framework::artifacts::store","line":521,"file":"phd-tests/framework/src/artifacts/store.rs","path":"phd_tests::migrate::from_base::migration_from_base_and_back"}
1982025-03-10T21:08:50.225Z{"v":0,"name":"phd-runner","msg":"[NEW - EVENT] Launching Propolis server","level":30,"hostname":"buskin","pid":862,"time":"2025-03-10T21:08:50.461262574Z","target":"phd_framework::test_vm::server","line":55,"file":"phd-tests/framework/src/test_vm/server.rs","bootrom_path":"\"/phd-artifacts/runner-0/29813374b58e3b77fb665f2d95cb3bab37d44fdd2c4fce2a70de9d76a3512a4f/OVMF_CODE.fd\"","server_path":"\"/phd-artifacts/runner-0/699c2d2c5e97e63733ab85fce044388baefe882761373e5c02190990dd286770/propolis-server\"","server_addr":"127.0.0.1:9000","path":"phd_tests::migrate::from_base::serial_history"}
1992025-03-10T21:08:50.225Z{"v":0,"name":"phd-runner","msg":"[NEW - EVENT] Opening server log files","level":30,"hostname":"buskin","pid":862,"time":"2025-03-10T21:08:50.461359406Z","target":"phd_framework::server_log_mode","line":64,"file":"phd-tests/framework/src/server_log_mode.rs","stderr_path":"\"/tmp/propolis-phd/runner-0/migration_serial_history_base.stderr.log\"","stdout_path":"\"/tmp/propolis-phd/runner-0/migration_serial_history_base.stdout.log\"","path":"phd_tests::migrate::from_base::serial_history"}
2002025-03-10T21:08:50.339Z{"v":0,"name":"phd-runner","msg":"[NEW - EVENT] Launched server with pid 871","level":30,"hostname":"buskin","pid":862,"time":"2025-03-10T21:08:50.576038002Z","target":"phd_framework::test_vm::server","line":107,"file":"phd-tests/framework/src/test_vm/server.rs","path":"phd_tests::migrate::from_base::serial_history"}
2012025-03-10T21:08:50.364Z{"v":0,"name":"phd-runner","msg":"[NEW - END]","level":30,"hostname":"buskin","pid":862,"time":"2025-03-10T21:08:50.576435068Z","target":"phd_framework::test_vm","line":193,"file":"phd-tests/framework/src/test_vm/mod.rs","elapsed_milliseconds":38057,"path":"phd_tests::migrate::from_base::serial_history"}
2022025-03-10T21:08:50.364Z{"v":0,"name":"phd-runner","msg":"[INSTANCE_ENSURE_INTERNAL - START]","level":30,"hostname":"buskin","pid":862,"time":"2025-03-10T21:08:50.576608721Z","target":"phd_framework::test_vm","line":276,"file":"phd-tests/framework/src/test_vm/mod.rs","vm_id":"69048244-dda4-48f2-81d9-16e8db1760ae","path":"phd_tests::migrate::from_base::serial_history","vm":"migration_serial_history_base"}
2032025-03-10T21:08:50.364Z{"v":0,"name":"phd-runner","msg":"[INSTANCE_ENSURE_INTERNAL - EVENT] retriable error from instance_spec_ensure","level":30,"hostname":"buskin","pid":862,"time":"2025-03-10T21:08:50.577559065Z","target":"phd_framework::test_vm","line":320,"file":"phd-tests/framework/src/test_vm/mod.rs","e":"Communication Error: error sending request for url (http://127.0.0.1:9000/instance)","vm_id":"69048244-dda4-48f2-81d9-16e8db1760ae","path":"phd_tests::migrate::from_base::serial_history","vm":"migration_serial_history_base"}
2042025-03-10T21:08:50.364Z{"v":0,"name":"phd-runner","msg":"[NEW - EVENT] Launching Propolis server","level":30,"hostname":"buskin","pid":862,"time":"2025-03-10T21:08:50.597401489Z","target":"phd_framework::test_vm::server","line":55,"file":"phd-tests/framework/src/test_vm/server.rs","server_path":"\"/phd-artifacts/runner-1/699c2d2c5e97e63733ab85fce044388baefe882761373e5c02190990dd286770/propolis-server\"","bootrom_path":"\"/phd-artifacts/runner-1/29813374b58e3b77fb665f2d95cb3bab37d44fdd2c4fce2a70de9d76a3512a4f/OVMF_CODE.fd\"","server_addr":"127.0.0.1:9125","path":"phd_tests::migrate::from_base::migration_from_base_and_back"}
2052025-03-10T21:08:50.365Z{"v":0,"name":"phd-runner","msg":"[NEW - EVENT] Opening server log files","level":30,"hostname":"buskin","pid":862,"time":"2025-03-10T21:08:50.597486331Z","target":"phd_framework::server_log_mode","line":64,"file":"phd-tests/framework/src/server_log_mode.rs","stdout_path":"\"/tmp/propolis-phd/runner-1/migration_from_base_and_back.stdout.log\"","stderr_path":"\"/tmp/propolis-phd/runner-1/migration_from_base_and_back.stderr.log\"","path":"phd_tests::migrate::from_base::migration_from_base_and_back"}
2062025-03-10T21:08:50.402Z{"v":0,"name":"phd-runner","msg":"[NEW - EVENT] Launched server with pid 872","level":30,"hostname":"buskin","pid":862,"time":"2025-03-10T21:08:50.639291301Z","target":"phd_framework::test_vm::server","line":107,"file":"phd-tests/framework/src/test_vm/server.rs","path":"phd_tests::migrate::from_base::migration_from_base_and_back"}
2072025-03-10T21:08:50.427Z{"v":0,"name":"phd-runner","msg":"[NEW - END]","level":30,"hostname":"buskin","pid":862,"time":"2025-03-10T21:08:50.639521084Z","target":"phd_framework::test_vm","line":193,"file":"phd-tests/framework/src/test_vm/mod.rs","path":"phd_tests::migrate::from_base::migration_from_base_and_back","elapsed_milliseconds":37967}
2082025-03-10T21:08:50.427Z{"v":0,"name":"phd-runner","msg":"[INSTANCE_ENSURE_INTERNAL - START]","level":30,"hostname":"buskin","pid":862,"time":"2025-03-10T21:08:50.639632526Z","target":"phd_framework::test_vm","line":276,"file":"phd-tests/framework/src/test_vm/mod.rs","path":"phd_tests::migrate::from_base::migration_from_base_and_back","vm":"migration_from_base_and_back","vm_id":"b1850a39-20a7-473a-9111-106afa27f677"}
2092025-03-10T21:08:50.427Z{"v":0,"name":"phd-runner","msg":"[INSTANCE_ENSURE_INTERNAL - EVENT] retriable error from instance_spec_ensure","level":30,"hostname":"buskin","pid":862,"time":"2025-03-10T21:08:50.640216635Z","target":"phd_framework::test_vm","line":320,"file":"phd-tests/framework/src/test_vm/mod.rs","e":"Communication Error: error sending request for url (http://127.0.0.1:9125/instance)","path":"phd_tests::migrate::from_base::migration_from_base_and_back","vm":"migration_from_base_and_back","vm_id":"b1850a39-20a7-473a-9111-106afa27f677"}
2102025-03-10T21:08:50.865Z{"v":0,"name":"phd-runner","msg":"[INSTANCE_ENSURE_INTERNAL - EVENT] permanent error from instance_spec_ensure","level":50,"hostname":"buskin","pid":862,"time":"2025-03-10T21:08:51.101395488Z","target":"phd_framework::test_vm","line":324,"file":"phd-tests/framework/src/test_vm/mod.rs","e":"Error Response: status: 500 Internal Server Error; headers: {\"content-type\": \"application/json\", \"x-request-id\": \"6261907a-2dca-45bb-ba5d-6602517a27ec\", \"content-length\": \"124\", \"date\": \"Mon, 10 Mar 2025 21:08:51 GMT\"}; value: Error { error_code: Some(\"Internal\"), message: \"Internal Server Error\", request_id: \"6261907a-2dca-45bb-ba5d-6602517a27ec\" }","vm_id":"69048244-dda4-48f2-81d9-16e8db1760ae","path":"phd_tests::migrate::from_base::serial_history","vm":"migration_serial_history_base"}
2112025-03-10T21:08:50.889Z{"v":0,"name":"phd-runner","msg":"[INSTANCE_ENSURE_INTERNAL - END]","level":30,"hostname":"buskin","pid":862,"time":"2025-03-10T21:08:51.102346213Z","target":"phd_framework::test_vm","line":276,"file":"phd-tests/framework/src/test_vm/mod.rs","vm":"migration_serial_history_base","elapsed_milliseconds":525,"path":"phd_tests::migrate::from_base::serial_history","vm_id":"69048244-dda4-48f2-81d9-16e8db1760ae"}
2122025-03-10T21:08:50.890Z{"v":0,"name":"phd-runner","msg":"[TEST - EVENT] Killing Propolis server process","level":20,"hostname":"buskin","pid":862,"time":"2025-03-10T21:08:51.102458325Z","target":"phd_framework::test_vm::server","line":125,"file":"phd-tests/framework/src/test_vm/server.rs","self.address":"127.0.0.1:9000","path":"phd_tests::migrate::from_base::serial_history"}
2132025-03-10T21:08:50.921Z{"v":0,"name":"phd-runner","msg":"[TEST - EVENT] zfs clone dropped","level":20,"hostname":"buskin","pid":862,"time":"2025-03-10T21:08:51.157219443Z","target":"phd_framework::zfs","line":83,"file":"phd-tests/framework/src/zfs.rs","path":"phd_tests::migrate::from_base::serial_history"}
2142025-03-10T21:08:50.946Z{"v":0,"name":"phd-runner","msg":"[TEST - EVENT] executing ZFS command","level":20,"hostname":"buskin","pid":862,"time":"2025-03-10T21:08:51.157290184Z","target":"phd_framework::zfs","line":304,"file":"phd-tests/framework/src/zfs.rs","args":"[\"phd-artifacts/phd-clone-3bb4d8f0-493e-49cf-9703-8c855f29d4fe\"]","verb":"destroy","path":"phd_tests::migrate::from_base::serial_history"}
2152025-03-10T21:08:51.104Z{"v":0,"name":"phd-runner","msg":"[TEST - EVENT] zfs snapshot dropped","level":20,"hostname":"buskin","pid":862,"time":"2025-03-10T21:08:51.340974348Z","target":"phd_framework::zfs","line":57,"file":"phd-tests/framework/src/zfs.rs","path":"phd_tests::migrate::from_base::serial_history"}
2162025-03-10T21:08:51.104Z{"v":0,"name":"phd-runner","msg":"[TEST - EVENT] executing ZFS command","level":20,"hostname":"buskin","pid":862,"time":"2025-03-10T21:08:51.341054719Z","target":"phd_framework::zfs","line":304,"file":"phd-tests/framework/src/zfs.rs","verb":"destroy","args":"[\"phd-artifacts@phd-f8ef2ae3-f355-46e0-85f8-83bde2919e44\"]","path":"phd_tests::migrate::from_base::serial_history"}
2172025-03-10T21:08:51.154Z{"v":0,"name":"phd-runner","msg":"[TEST - END]","level":30,"hostname":"buskin","pid":862,"time":"2025-03-10T21:08:51.391163856Z","target":"phd_tests::migrate::from_base","line":40,"file":"phd-tests/tests/src/migrate.rs","elapsed_milliseconds":73743,"path":"phd_tests::migrate::from_base::serial_history"}
2182025-03-10T21:08:51.257Z{"v":0,"name":"phd-runner","msg":"test phd_tests::migrate::from_base::serial_history ... FAILED: Error Response: status: 500 Internal Server Error; headers: {\"content-type\": \"application/json\", \"x-request-id\": \"6261907a-2dca-45bb-ba5d-6602517a27ec\", \"content-length\": \"124\", \"date\": \"Mon, 10 Mar 2025 21:08:51 GMT\"}; value: Error { error_code: Some(\"Internal\"), message: \"Internal Server Error\", request_id: \"6261907a-2dca-45bb-ba5d-6602517a27ec\" }\n\nStack backtrace:\n 0: anyhow::error::<impl core::convert::From<E> for anyhow::Error>::from\n at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.89/src/backtrace.rs:27:14\n 1: <core::result::Result<T,F> as core::ops::try_trait::FromResidual<core::result::Result<core::convert::Infallible,E>>>::from_residual\n at /rustc/4d91de4e48198da2e33413efdcd9cd2cc0c46688/library/core/src/result.rs:2014:27\n 2: phd_framework::test_vm::TestVm::instance_ensure_internal::{{closure}}::{{closure}}\n at ./oxidecomputer/propolis/phd-tests/framework/src/test_vm/mod.rs:336:9\n 3: <tracing::instrument::Instrumented<T> as core::future::future::Future>::poll\n at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tracing-0.1.40/src/instrument.rs:321:9\n 4: phd_framework::test_vm::TestVm::instance_ensure_internal::{{closure}}\n at ./oxidecomputer/propolis/phd-tests/framework/src/test_vm/mod.rs:276:5\n 5: phd_framework::test_vm::TestVm::instance_ensure::{{closure}}\n at ./oxidecomputer/propolis/phd-tests/framework/src/test_vm/mod.rs:414:22\n 6: phd_framework::test_vm::TestVm::launch::{{closure}}\n at ./oxidecomputer/propolis/phd-tests/framework/src/test_vm/mod.rs:402:32\n 7: phd_tests::migrate::run_serial_history_test::{{closure}}\n at ./oxidecomputer/propolis/phd-tests/tests/src/migrate.rs:345:21\n 8: phd_tests::migrate::from_base::serial_history::{{closure}}::{{closure}}\n at ./oxidecomputer/propolis/phd-tests/tests/src/migrate.rs:46:10\n 9: <tracing::instrument::Instrumented<T> as core::future::future::Future>::poll\n at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tracing-0.1.40/src/instrument.rs:321:9\n 10: phd_tests::migrate::from_base::serial_history::{{closure}}\n at ./oxidecomputer/propolis/phd-tests/tests/src/migrate.rs:40:5\n 11: <core::pin::Pin<P> as core::future::future::Future>::poll\n at /rustc/4d91de4e48198da2e33413efdcd9cd2cc0c46688/library/core/src/future/future.rs:124:9\n 12: phd_testcase::TestCase::run::{{closure}}\n at ./oxidecomputer/propolis/phd-tests/testcase/src/lib.rs:79:32\n 13: phd_runner::execute::run_tests_with_ctx::{{closure}}::run_tests::{{closure}}\n at ./oxidecomputer/propolis/phd-tests/runner/src/execute.rs:117:58\n 14: tokio::runtime::task::core::Core<T,S>::poll::{{closure}}\n at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.40.0/src/runtime/task/core.rs:331:17\n 15: tokio::loom::std::unsafe_cell::UnsafeCell<T>::with_mut\n at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.40.0/src/loom/std/unsafe_cell.rs:16:9\n 16: tokio::runtime::task::core::Core<T,S>::poll\n at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.40.0/src/runtime/task/core.rs:320:13\n 17: tokio::runtime::task::harness::poll_future::{{closure}}\n at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.40.0/src/runtime/task/harness.rs:500:19\n 18: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once\n at /rustc/4d91de4e48198da2e33413efdcd9cd2cc0c46688/library/core/src/panic/unwind_safe.rs:272:9\n 19: std::panicking::try::do_call\n at /rustc/4d91de4e48198da2e33413efdcd9cd2cc0c46688/library/std/src/panicking.rs:584:40\n 20: __rust_try\n 21: std::panicking::try\n at /rustc/4d91de4e48198da2e33413efdcd9cd2cc0c46688/library/std/src/panicking.rs:547:19\n 22: std::panic::catch_unwind\n at /rustc/4d91de4e48198da2e33413efdcd9cd2cc0c46688/library/std/src/panic.rs:358:14\n 23: tokio::runtime::task::harness::poll_future\n at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.40.0/src/runtime/task/harness.rs:488:18\n 24: tokio::runtime::task::harness::Harness<T,S>::poll_inner\n at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.40.0/src/runtime/task/harness.rs:209:27\n 25: tokio::runtime::task::harness::Harness<T,S>::poll\n at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.40.0/src/runtime/task/harness.rs:154:15\n 26: tokio::runtime::task::raw::poll\n at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.40.0/src/runtime/task/raw.rs:271:5\n 27: tokio::runtime::task::raw::RawTask::poll\n at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.40.0/src/runtime/task/raw.rs:201:18\n 28: tokio::runtime::task::LocalNotified<S>::run\n at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.40.0/src/runtime/task/mod.rs:436:9\n 29: tokio::runtime::scheduler::multi_thread::worker::Context::run_task::{{closure}}\n at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.40.0/src/runtime/scheduler/multi_thread/worker.rs:661:17\n 30: tokio::runtime::coop::with_budget\n at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.40.0/src/runtime/coop.rs:107:5\n 31: tokio::runtime::coop::budget\n at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.40.0/src/runtime/coop.rs:73:5\n 32: tokio::runtime::scheduler::multi_thread::worker::Context::run_task\n at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.40.0/src/runtime/scheduler/multi_thread/worker.rs:597:9\n 33: tokio::runtime::scheduler::multi_thread::worker::Context::run\n at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.40.0/src/runtime/scheduler/multi_thread/worker.rs:548:24\n 34: tokio::runtime::scheduler::multi_thread::worker::run::{{closure}}::{{closure}}\n at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.40.0/src/runtime/scheduler/multi_thread/worker.rs:513:21\n 35: tokio::runtime::context::scoped::Scoped<T>::set\n at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.40.0/src/runtime/context/scoped.rs:40:9\n 36: tokio::runtime::context::set_scheduler::{{closure}}\n at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.40.0/src/runtime/context.rs:180:26\n 37: std::thread::local::LocalKey<T>::try_with\n at /rustc/4d91de4e48198da2e33413efdcd9cd2cc0c46688/library/std/src/thread/local.rs:308:12\n 38: std::thread::local::LocalKey<T>::with\n at /rustc/4d91de4e48198da2e33413efdcd9cd2cc0c46688/library/std/src/thread/local.rs:272:9\n 39: tokio::runtime::context::set_scheduler\n at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.40.0/src/runtime/context.rs:180:9\n 40: tokio::runtime::scheduler::multi_thread::worker::run::{{closure}}\n at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.40.0/src/runtime/scheduler/multi_thread/worker.rs:508:9\n 41: tokio::runtime::context::runtime::enter_runtime\n at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.40.0/src/runtime/context/runtime.rs:65:16\n 42: tokio::runtime::scheduler::multi_thread::worker::run\n at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.40.0/src/runtime/scheduler/multi_thread/worker.rs:500:5\n 43: tokio::runtime::scheduler::multi_thread::worker::Launch::launch::{{closure}}\n at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.40.0/src/runtime/scheduler/multi_thread/worker.rs:466:45\n 44: <tokio::runtime::blocking::task::BlockingTask<T> as core::future::future::Future>::poll\n at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.40.0/src/runtime/blocking/task.rs:42:21\n 45: tokio::runtime::task::core::Core<T,S>::poll::{{closure}}\n at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.40.0/src/runtime/task/core.rs:331:17\n 46: tokio::loom::std::unsafe_cell::UnsafeCell<T>::with_mut\n at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.40.0/src/loom/std/unsafe_cell.rs:16:9\n 47: tokio::runtime::task::core::Core<T,S>::poll\n at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.40.0/src/runtime/task/core.rs:320:13\n 48: tokio::runtime::task::harness::poll_future::{{closure}}\n at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.40.0/src/runtime/task/harness.rs:500:19\n 49: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once\n at /rustc/4d91de4e48198da2e33413efdcd9cd2cc0c46688/library/core/src/panic/unwind_safe.rs:272:9\n 50: std::panicking::try::do_call\n at /rustc/4d91de4e48198da2e33413efdcd9cd2cc0c46688/library/std/src/panicking.rs:584:40\n 51: __rust_try\n 52: std::panicking::try\n at /rustc/4d91de4e48198da2e33413efdcd9cd2cc0c46688/library/std/src/panicking.rs:547:19\n 53: std::panic::catch_unwind\n at /rustc/4d91de4e48198da2e33413efdcd9cd2cc0c46688/library/std/src/panic.rs:358:14\n 54: tokio::runtime::task::harness::poll_future\n at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.40.0/src/runtime/task/harness.rs:488:18\n 55: tokio::runtime::task::harness::Harness<T,S>::poll_inner\n at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.40.0/src/runtime/task/harness.rs:209:27\n 56: tokio::runtime::task::harness::Harness<T,S>::poll\n at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.40.0/src/runtime/task/harness.rs:154:15\n 57: tokio::runtime::task::raw::poll\n at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.40.0/src/runtime/task/raw.rs:271:5\n 58: tokio::runtime::task::raw::RawTask::poll\n at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.40.0/src/runtime/task/raw.rs:201:18\n 59: tokio::runtime::task::UnownedTask<S>::run\n at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.40.0/src/runtime/task/mod.rs:473:9\n 60: tokio::runtime::blocking::pool::Task::run\n at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.40.0/src/runtime/blocking/pool.rs:160:9\n 61: tokio::runtime::blocking::pool::Inner::run\n at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.40.0/src/runtime/blocking/pool.rs:518:17\n 62: tokio::runtime::blocking::pool::Spawner::spawn_thread::{{closure}}\n at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.40.0/src/runtime/blocking/pool.rs:476:13\n 63: std::sys::backtrace::__rust_begin_short_backtrace\n at /rustc/4d91de4e48198da2e33413efdcd9cd2cc0c46688/library/std/src/sys/backtrace.rs:152:18\n 64: std::thread::Builder::spawn_unchecked_::{{closure}}::{{closure}}\n at /rustc/4d91de4e48198da2e33413efdcd9cd2cc0c46688/library/std/src/thread/mod.rs:564:17\n 65: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once\n at /rustc/4d91de4e48198da2e33413efdcd9cd2cc0c46688/library/core/src/panic/unwind_safe.rs:272:9\n 66: std::panicking::try::do_call\n at /rustc/4d91de4e48198da2e33413efdcd9cd2cc0c46688/library/std/src/panicking.rs:584:40\n 67: __rust_try\n 68: std::panicking::try\n at /rustc/4d91de4e48198da2e33413efdcd9cd2cc0c46688/library/std/src/panicking.rs:547:19\n 69: std::panic::catch_unwind\n at /rustc/4d91de4e48198da2e33413efdcd9cd2cc0c46688/library/std/src/panic.rs:358:14\n 70: std::thread::Builder::spawn_unchecked_::{{closure}}\n at /rustc/4d91de4e48198da2e33413efdcd9cd2cc0c46688/library/std/src/thread/mod.rs:562:30\n 71: core::ops::function::FnOnce::call_once{{vtable.shim}}\n at /rustc/4d91de4e48198da2e33413efdcd9cd2cc0c46688/library/core/src/ops/function.rs:250:5\n 72: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once\n at /rustc/4d91de4e48198da2e33413efdcd9cd2cc0c46688/library/alloc/src/boxed.rs:1993:9\n 73: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once\n at /rustc/4d91de4e48198da2e33413efdcd9cd2cc0c46688/library/alloc/src/boxed.rs:1993:9\n 74: std::sys::pal::unix::thread::Thread::new::thread_start\n at /rustc/4d91de4e48198da2e33413efdcd9cd2cc0c46688/library/std/src/sys/pal/unix/thread.rs:106:17\n 75: _thrp_setup\n 76: <unknown>","level":30,"hostname":"buskin","pid":862,"time":"2025-03-10T21:08:51.492783933Z","target":"phd_runner::execute","line":119,"file":"phd-tests/runner/src/execute.rs"}
2192025-03-10T21:08:51.258Z{"v":0,"name":"phd-runner","msg":"[TEST_CLEANUP - START]","level":30,"hostname":"buskin","pid":862,"time":"2025-03-10T21:08:51.492866484Z","target":"phd_runner::fixtures","line":51,"file":"phd-tests/runner/src/fixtures.rs"}
2202025-03-10T21:08:51.258Z{"v":0,"name":"phd-runner","msg":"[TEST_CLEANUP - END]","level":30,"hostname":"buskin","pid":862,"time":"2025-03-10T21:08:51.492964946Z","target":"phd_runner::fixtures","line":51,"file":"phd-tests/runner/src/fixtures.rs","elapsed_milliseconds":0}
2212025-03-10T21:08:51.283Z{"v":0,"name":"phd-runner","msg":"Starting test phd_tests::migrate::from_base::can_migrate_from_base","level":30,"hostname":"buskin","pid":862,"time":"2025-03-10T21:08:51.493009636Z","target":"phd_runner::execute","line":101,"file":"phd-tests/runner/src/execute.rs"}
2222025-03-10T21:08:51.284Z{"v":0,"name":"phd-runner","msg":"[TEST_SETUP - START]","level":30,"hostname":"buskin","pid":862,"time":"2025-03-10T21:08:51.493035977Z","target":"phd_runner::fixtures","line":41,"file":"phd-tests/runner/src/fixtures.rs"}
2232025-03-10T21:08:51.284Z{"v":0,"name":"phd-runner","msg":"[TEST_SETUP - END]","level":30,"hostname":"buskin","pid":862,"time":"2025-03-10T21:08:51.493063947Z","target":"phd_runner::fixtures","line":41,"file":"phd-tests/runner/src/fixtures.rs","elapsed_milliseconds":0}
2242025-03-10T21:08:51.284Z{"v":0,"name":"phd-runner","msg":"[TEST - START]","level":30,"hostname":"buskin","pid":862,"time":"2025-03-10T21:08:51.493104788Z","target":"phd_tests::migrate::from_base","line":34,"file":"phd-tests/tests/src/migrate.rs","path":"phd_tests::migrate::from_base::can_migrate_from_base"}
2252025-03-10T21:08:51.284Z{"v":0,"name":"phd-runner","msg":"[TEST - EVENT] Verified artifact already exists","level":20,"hostname":"buskin","pid":862,"time":"2025-03-10T21:08:51.493186329Z","target":"phd_framework::artifacts::store","line":45,"file":"phd-tests/framework/src/artifacts/store.rs","path":"/phd-artifacts/runner-0/29813374b58e3b77fb665f2d95cb3bab37d44fdd2c4fce2a70de9d76a3512a4f/OVMF_CODE.fd","path":"phd_tests::migrate::from_base::can_migrate_from_base"}
2262025-03-10T21:08:51.284Z{"v":0,"name":"phd-runner","msg":"[TEST - EVENT] Verified artifact already exists","level":20,"hostname":"buskin","pid":862,"time":"2025-03-10T21:08:51.49322899Z","target":"phd_framework::artifacts::store","line":45,"file":"phd-tests/framework/src/artifacts/store.rs","path":"/phd-artifacts/runner-0/ba8007f74f9b54fbae3b2520da577831b4834778a498d732f091260c61aa7ca1/alpine.iso","path":"phd_tests::migrate::from_base::can_migrate_from_base"}
2272025-03-10T21:08:51.284Z{"v":0,"name":"phd-runner","msg":"[TEST - EVENT] Verified artifact already exists","level":20,"hostname":"buskin","pid":862,"time":"2025-03-10T21:08:51.49327354Z","target":"phd_framework::artifacts::store","line":45,"file":"phd-tests/framework/src/artifacts/store.rs","path":"/phd-artifacts/runner-0/ba8007f74f9b54fbae3b2520da577831b4834778a498d732f091260c61aa7ca1/alpine.iso","path":"phd_tests::migrate::from_base::can_migrate_from_base"}
2282025-03-10T21:08:51.284Z{"v":0,"name":"phd-runner","msg":"[TEST - EVENT] executing ZFS command","level":20,"hostname":"buskin","pid":862,"time":"2025-03-10T21:08:51.493322661Z","target":"phd_framework::zfs","line":304,"file":"phd-tests/framework/src/zfs.rs","verb":"list","args":"[\"/phd-artifacts/runner-0/ba8007f74f9b54fbae3b2520da577831b4834778a498d732f091260c61aa7ca1/alpine.iso\"]","path":"phd_tests::migrate::from_base::can_migrate_from_base"}
2292025-03-10T21:08:51.310Z{"v":0,"name":"phd-runner","msg":"[TEST - EVENT] parsing zfs list output","level":20,"hostname":"buskin","pid":862,"time":"2025-03-10T21:08:51.543307807Z","target":"phd_framework::zfs","line":238,"file":"phd-tests/framework/src/zfs.rs","stdout":"NAME USED AVAIL REFER MOUNTPOINT\nphd-artifacts 1.34G 1.80T 1.34G /phd-artifacts\n","path":"phd_tests::migrate::from_base::can_migrate_from_base"}
2302025-03-10T21:08:51.310Z{"v":0,"name":"phd-runner","msg":"[TEST - EVENT] executing ZFS command","level":20,"hostname":"buskin","pid":862,"time":"2025-03-10T21:08:51.543397038Z","target":"phd_framework::zfs","line":304,"file":"phd-tests/framework/src/zfs.rs","args":"[\"phd-artifacts@phd-1c3f9a47-443c-4e05-8e70-ef38dfc3fa6e\"]","verb":"snapshot","path":"phd_tests::migrate::from_base::can_migrate_from_base"}
2312025-03-10T21:08:51.347Z{"v":0,"name":"phd-runner","msg":"[TEST - EVENT] executing ZFS command","level":20,"hostname":"buskin","pid":862,"time":"2025-03-10T21:08:51.584220953Z","target":"phd_framework::zfs","line":304,"file":"phd-tests/framework/src/zfs.rs","args":"[\"phd-artifacts@phd-1c3f9a47-443c-4e05-8e70-ef38dfc3fa6e\", \"phd-artifacts/phd-clone-e2dc39fc-3c98-41a4-b6ba-a24d44613e82\"]","verb":"clone","path":"phd_tests::migrate::from_base::can_migrate_from_base"}
2322025-03-10T21:08:51.387Z{"v":0,"name":"phd-runner","msg":"[TEST - EVENT] executing ZFS command","level":20,"hostname":"buskin","pid":862,"time":"2025-03-10T21:08:51.623597766Z","target":"phd_framework::zfs","line":304,"file":"phd-tests/framework/src/zfs.rs","verb":"list","args":"[\"phd-artifacts/phd-clone-e2dc39fc-3c98-41a4-b6ba-a24d44613e82\"]","path":"phd_tests::migrate::from_base::can_migrate_from_base"}
2332025-03-10T21:08:51.427Z{"v":0,"name":"phd-runner","msg":"[TEST - EVENT] parsing zfs list output","level":20,"hostname":"buskin","pid":862,"time":"2025-03-10T21:08:51.663306024Z","target":"phd_framework::zfs","line":238,"file":"phd-tests/framework/src/zfs.rs","stdout":"NAME USED AVAIL REFER MOUNTPOINT\nphd-artifacts/phd-clone-e2dc39fc-3c98-41a4-b6ba-a24d44613e82 0B 1.80T 1.34G /phd-artifacts/phd-clone-e2dc39fc-3c98-41a4-b6ba-a24d44613e82\n","path":"phd_tests::migrate::from_base::can_migrate_from_base"}
2342025-03-10T21:08:51.451Z{"v":0,"name":"phd-runner","msg":"[NEW - START]","level":30,"hostname":"buskin","pid":862,"time":"2025-03-10T21:08:51.663760011Z","target":"phd_framework::test_vm","line":193,"file":"phd-tests/framework/src/test_vm/mod.rs","path":"phd_tests::migrate::from_base::can_migrate_from_base"}
2352025-03-10T21:08:51.452Z{"v":0,"name":"phd-runner","msg":"[NEW - EVENT] phd_framework::test_vm","level":30,"hostname":"buskin","pid":862,"time":"2025-03-10T21:08:51.663821403Z","target":"phd_framework::test_vm","line":211,"file":"phd-tests/framework/src/test_vm/mod.rs","guest_os_kind":"Alpine","environment":"EnvironmentSpec { location: Local, propolis_artifact: \"__BASE_PROPOLIS\" }","vm_name":"migration_from_base","path":"phd_tests::migrate::from_base::can_migrate_from_base"}
2362025-03-10T21:08:51.452Z{"v":0,"name":"phd-runner","msg":"[NEW - EVENT] Verified artifact already exists","level":20,"hostname":"buskin","pid":862,"time":"2025-03-10T21:08:51.663880593Z","target":"phd_framework::artifacts::store","line":45,"file":"phd-tests/framework/src/artifacts/store.rs","path":"/phd-artifacts/runner-0/699c2d2c5e97e63733ab85fce044388baefe882761373e5c02190990dd286770/propolis-server","path":"phd_tests::migrate::from_base::can_migrate_from_base"}
2372025-03-10T21:08:51.452Z{"v":0,"name":"phd-runner","msg":"[NEW - EVENT] Launching Propolis server","level":30,"hostname":"buskin","pid":862,"time":"2025-03-10T21:08:51.663936044Z","target":"phd_framework::test_vm::server","line":55,"file":"phd-tests/framework/src/test_vm/server.rs","bootrom_path":"\"/phd-artifacts/runner-0/29813374b58e3b77fb665f2d95cb3bab37d44fdd2c4fce2a70de9d76a3512a4f/OVMF_CODE.fd\"","server_path":"\"/phd-artifacts/runner-0/699c2d2c5e97e63733ab85fce044388baefe882761373e5c02190990dd286770/propolis-server\"","server_addr":"127.0.0.1:9000","path":"phd_tests::migrate::from_base::can_migrate_from_base"}
2382025-03-10T21:08:51.452Z{"v":0,"name":"phd-runner","msg":"[NEW - EVENT] Opening server log files","level":30,"hostname":"buskin","pid":862,"time":"2025-03-10T21:08:51.663977945Z","target":"phd_framework::server_log_mode","line":64,"file":"phd-tests/framework/src/server_log_mode.rs","stderr_path":"\"/tmp/propolis-phd/runner-0/migration_from_base.stderr.log\"","stdout_path":"\"/tmp/propolis-phd/runner-0/migration_from_base.stdout.log\"","path":"phd_tests::migrate::from_base::can_migrate_from_base"}
2392025-03-10T21:08:51.476Z{"v":0,"name":"phd-runner","msg":"[NEW - EVENT] Launched server with pid 880","level":30,"hostname":"buskin","pid":862,"time":"2025-03-10T21:08:51.696821488Z","target":"phd_framework::test_vm::server","line":107,"file":"phd-tests/framework/src/test_vm/server.rs","path":"phd_tests::migrate::from_base::can_migrate_from_base"}
2402025-03-10T21:08:51.477Z{"v":0,"name":"phd-runner","msg":"[NEW - END]","level":30,"hostname":"buskin","pid":862,"time":"2025-03-10T21:08:51.697144573Z","target":"phd_framework::test_vm","line":193,"file":"phd-tests/framework/src/test_vm/mod.rs","path":"phd_tests::migrate::from_base::can_migrate_from_base","elapsed_milliseconds":33}
2412025-03-10T21:08:51.477Z{"v":0,"name":"phd-runner","msg":"[INSTANCE_ENSURE_INTERNAL - START]","level":30,"hostname":"buskin","pid":862,"time":"2025-03-10T21:08:51.697291565Z","target":"phd_framework::test_vm","line":276,"file":"phd-tests/framework/src/test_vm/mod.rs","path":"phd_tests::migrate::from_base::can_migrate_from_base","vm":"migration_from_base","vm_id":"cadd903e-2784-4c2d-b4d0-e1f60ff9ed27"}
2422025-03-10T21:08:51.477Z{"v":0,"name":"phd-runner","msg":"[INSTANCE_ENSURE_INTERNAL - EVENT] retriable error from instance_spec_ensure","level":30,"hostname":"buskin","pid":862,"time":"2025-03-10T21:08:51.697807203Z","target":"phd_framework::test_vm","line":320,"file":"phd-tests/framework/src/test_vm/mod.rs","e":"Communication Error: error sending request for url (http://127.0.0.1:9000/instance)","path":"phd_tests::migrate::from_base::can_migrate_from_base","vm":"migration_from_base","vm_id":"cadd903e-2784-4c2d-b4d0-e1f60ff9ed27"}
2432025-03-10T21:08:51.501Z{"v":0,"name":"phd-runner","msg":"[INSTANCE_ENSURE_INTERNAL - EVENT] permanent error from instance_spec_ensure","level":50,"hostname":"buskin","pid":862,"time":"2025-03-10T21:08:51.718231086Z","target":"phd_framework::test_vm","line":324,"file":"phd-tests/framework/src/test_vm/mod.rs","e":"Error Response: status: 500 Internal Server Error; headers: {\"content-type\": \"application/json\", \"x-request-id\": \"dfff96e5-efaf-4183-95a5-953338c065d9\", \"content-length\": \"124\", \"date\": \"Mon, 10 Mar 2025 21:08:51 GMT\"}; value: Error { error_code: Some(\"Internal\"), message: \"Internal Server Error\", request_id: \"dfff96e5-efaf-4183-95a5-953338c065d9\" }","path":"phd_tests::migrate::from_base::migration_from_base_and_back","vm":"migration_from_base_and_back","vm_id":"b1850a39-20a7-473a-9111-106afa27f677"}
2442025-03-10T21:08:51.502Z{"v":0,"name":"phd-runner","msg":"[INSTANCE_ENSURE_INTERNAL - END]","level":30,"hostname":"buskin","pid":862,"time":"2025-03-10T21:08:51.718420219Z","target":"phd_framework::test_vm","line":276,"file":"phd-tests/framework/src/test_vm/mod.rs","vm_id":"b1850a39-20a7-473a-9111-106afa27f677","elapsed_milliseconds":1078,"path":"phd_tests::migrate::from_base::migration_from_base_and_back","vm":"migration_from_base_and_back"}
2452025-03-10T21:08:51.502Z{"v":0,"name":"phd-runner","msg":"[TEST - EVENT] Killing Propolis server process","level":20,"hostname":"buskin","pid":862,"time":"2025-03-10T21:08:51.71850577Z","target":"phd_framework::test_vm::server","line":125,"file":"phd-tests/framework/src/test_vm/server.rs","self.address":"127.0.0.1:9125","path":"phd_tests::migrate::from_base::migration_from_base_and_back"}
2462025-03-10T21:08:51.526Z{"v":0,"name":"phd-runner","msg":"[TEST - EVENT] zfs clone dropped","level":20,"hostname":"buskin","pid":862,"time":"2025-03-10T21:08:51.76092196Z","target":"phd_framework::zfs","line":83,"file":"phd-tests/framework/src/zfs.rs","path":"phd_tests::migrate::from_base::migration_from_base_and_back"}
2472025-03-10T21:08:51.526Z{"v":0,"name":"phd-runner","msg":"[TEST - EVENT] executing ZFS command","level":20,"hostname":"buskin","pid":862,"time":"2025-03-10T21:08:51.761010221Z","target":"phd_framework::zfs","line":304,"file":"phd-tests/framework/src/zfs.rs","verb":"destroy","args":"[\"phd-artifacts/phd-clone-9d36a2f8-ffed-490c-b405-c06b63013ecc\"]","path":"phd_tests::migrate::from_base::migration_from_base_and_back"}
2482025-03-10T21:08:51.575Z{"v":0,"name":"phd-runner","msg":"[TEST - EVENT] zfs snapshot dropped","level":20,"hostname":"buskin","pid":862,"time":"2025-03-10T21:08:51.811817609Z","target":"phd_framework::zfs","line":57,"file":"phd-tests/framework/src/zfs.rs","path":"phd_tests::migrate::from_base::migration_from_base_and_back"}
2492025-03-10T21:08:51.600Z{"v":0,"name":"phd-runner","msg":"[TEST - EVENT] executing ZFS command","level":20,"hostname":"buskin","pid":862,"time":"2025-03-10T21:08:51.81187743Z","target":"phd_framework::zfs","line":304,"file":"phd-tests/framework/src/zfs.rs","verb":"destroy","args":"[\"phd-artifacts@phd-68aa1763-4002-4b53-ae10-15f8d1625940\"]","path":"phd_tests::migrate::from_base::migration_from_base_and_back"}
2502025-03-10T21:08:51.625Z{"v":0,"name":"phd-runner","msg":"[TEST - END]","level":30,"hostname":"buskin","pid":862,"time":"2025-03-10T21:08:51.854391551Z","target":"phd_tests::migrate::from_base","line":52,"file":"phd-tests/tests/src/migrate.rs","path":"phd_tests::migrate::from_base::migration_from_base_and_back","elapsed_milliseconds":74206}
2512025-03-10T21:08:51.625Z{"v":0,"name":"phd-runner","msg":"test phd_tests::migrate::from_base::migration_from_base_and_back ... FAILED: Error Response: status: 500 Internal Server Error; headers: {\"content-type\": \"application/json\", \"x-request-id\": \"dfff96e5-efaf-4183-95a5-953338c065d9\", \"content-length\": \"124\", \"date\": \"Mon, 10 Mar 2025 21:08:51 GMT\"}; value: Error { error_code: Some(\"Internal\"), message: \"Internal Server Error\", request_id: \"dfff96e5-efaf-4183-95a5-953338c065d9\" }\n\nStack backtrace:\n 0: anyhow::error::<impl core::convert::From<E> for anyhow::Error>::from\n at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.89/src/backtrace.rs:27:14\n 1: <core::result::Result<T,F> as core::ops::try_trait::FromResidual<core::result::Result<core::convert::Infallible,E>>>::from_residual\n at /rustc/4d91de4e48198da2e33413efdcd9cd2cc0c46688/library/core/src/result.rs:2014:27\n 2: phd_framework::test_vm::TestVm::instance_ensure_internal::{{closure}}::{{closure}}\n at ./oxidecomputer/propolis/phd-tests/framework/src/test_vm/mod.rs:336:9\n 3: <tracing::instrument::Instrumented<T> as core::future::future::Future>::poll\n at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tracing-0.1.40/src/instrument.rs:321:9\n 4: phd_framework::test_vm::TestVm::instance_ensure_internal::{{closure}}\n at ./oxidecomputer/propolis/phd-tests/framework/src/test_vm/mod.rs:276:5\n 5: phd_framework::test_vm::TestVm::instance_ensure::{{closure}}\n at ./oxidecomputer/propolis/phd-tests/framework/src/test_vm/mod.rs:414:22\n 6: phd_framework::test_vm::TestVm::launch::{{closure}}\n at ./oxidecomputer/propolis/phd-tests/framework/src/test_vm/mod.rs:402:32\n 7: phd_tests::migrate::from_base::migration_from_base_and_back::{{closure}}::{{closure}}\n at ./oxidecomputer/propolis/phd-tests/tests/src/migrate.rs:56:25\n 8: <tracing::instrument::Instrumented<T> as core::future::future::Future>::poll\n at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tracing-0.1.40/src/instrument.rs:321:9\n 9: phd_tests::migrate::from_base::migration_from_base_and_back::{{closure}}\n at ./oxidecomputer/propolis/phd-tests/tests/src/migrate.rs:52:5\n 10: <core::pin::Pin<P> as core::future::future::Future>::poll\n at /rustc/4d91de4e48198da2e33413efdcd9cd2cc0c46688/library/core/src/future/future.rs:124:9\n 11: phd_testcase::TestCase::run::{{closure}}\n at ./oxidecomputer/propolis/phd-tests/testcase/src/lib.rs:79:32\n 12: phd_runner::execute::run_tests_with_ctx::{{closure}}::run_tests::{{closure}}\n at ./oxidecomputer/propolis/phd-tests/runner/src/execute.rs:117:58\n 13: tokio::runtime::task::core::Core<T,S>::poll::{{closure}}\n at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.40.0/src/runtime/task/core.rs:331:17\n 14: tokio::loom::std::unsafe_cell::UnsafeCell<T>::with_mut\n at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.40.0/src/loom/std/unsafe_cell.rs:16:9\n 15: tokio::runtime::task::core::Core<T,S>::poll\n at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.40.0/src/runtime/task/core.rs:320:13\n 16: tokio::runtime::task::harness::poll_future::{{closure}}\n at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.40.0/src/runtime/task/harness.rs:500:19\n 17: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once\n at /rustc/4d91de4e48198da2e33413efdcd9cd2cc0c46688/library/core/src/panic/unwind_safe.rs:272:9\n 18: std::panicking::try::do_call\n at /rustc/4d91de4e48198da2e33413efdcd9cd2cc0c46688/library/std/src/panicking.rs:584:40\n 19: __rust_try\n 20: std::panicking::try\n at /rustc/4d91de4e48198da2e33413efdcd9cd2cc0c46688/library/std/src/panicking.rs:547:19\n 21: std::panic::catch_unwind\n at /rustc/4d91de4e48198da2e33413efdcd9cd2cc0c46688/library/std/src/panic.rs:358:14\n 22: tokio::runtime::task::harness::poll_future\n at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.40.0/src/runtime/task/harness.rs:488:18\n 23: tokio::runtime::task::harness::Harness<T,S>::poll_inner\n at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.40.0/src/runtime/task/harness.rs:209:27\n 24: tokio::runtime::task::harness::Harness<T,S>::poll\n at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.40.0/src/runtime/task/harness.rs:154:15\n 25: tokio::runtime::task::raw::poll\n at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.40.0/src/runtime/task/raw.rs:271:5\n 26: tokio::runtime::task::raw::RawTask::poll\n at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.40.0/src/runtime/task/raw.rs:201:18\n 27: tokio::runtime::task::LocalNotified<S>::run\n at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.40.0/src/runtime/task/mod.rs:436:9\n 28: tokio::runtime::scheduler::multi_thread::worker::Context::run_task::{{closure}}\n at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.40.0/src/runtime/scheduler/multi_thread/worker.rs:661:17\n 29: tokio::runtime::coop::with_budget\n at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.40.0/src/runtime/coop.rs:107:5\n 30: tokio::runtime::coop::budget\n at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.40.0/src/runtime/coop.rs:73:5\n 31: tokio::runtime::scheduler::multi_thread::worker::Context::run_task\n at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.40.0/src/runtime/scheduler/multi_thread/worker.rs:597:9\n 32: tokio::runtime::scheduler::multi_thread::worker::Context::run\n at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.40.0/src/runtime/scheduler/multi_thread/worker.rs:548:24\n 33: tokio::runtime::scheduler::multi_thread::worker::run::{{closure}}::{{closure}}\n at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.40.0/src/runtime/scheduler/multi_thread/worker.rs:513:21\n 34: tokio::runtime::context::scoped::Scoped<T>::set\n at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.40.0/src/runtime/context/scoped.rs:40:9\n 35: tokio::runtime::context::set_scheduler::{{closure}}\n at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.40.0/src/runtime/context.rs:180:26\n 36: std::thread::local::LocalKey<T>::try_with\n at /rustc/4d91de4e48198da2e33413efdcd9cd2cc0c46688/library/std/src/thread/local.rs:308:12\n 37: std::thread::local::LocalKey<T>::with\n at /rustc/4d91de4e48198da2e33413efdcd9cd2cc0c46688/library/std/src/thread/local.rs:272:9\n 38: tokio::runtime::context::set_scheduler\n at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.40.0/src/runtime/context.rs:180:9\n 39: tokio::runtime::scheduler::multi_thread::worker::run::{{closure}}\n at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.40.0/src/runtime/scheduler/multi_thread/worker.rs:508:9\n 40: tokio::runtime::context::runtime::enter_runtime\n at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.40.0/src/runtime/context/runtime.rs:65:16\n 41: tokio::runtime::scheduler::multi_thread::worker::run\n at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.40.0/src/runtime/scheduler/multi_thread/worker.rs:500:5\n 42: tokio::runtime::scheduler::multi_thread::worker::Launch::launch::{{closure}}\n at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.40.0/src/runtime/scheduler/multi_thread/worker.rs:466:45\n 43: <tokio::runtime::blocking::task::BlockingTask<T> as core::future::future::Future>::poll\n at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.40.0/src/runtime/blocking/task.rs:42:21\n 44: tokio::runtime::task::core::Core<T,S>::poll::{{closure}}\n at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.40.0/src/runtime/task/core.rs:331:17\n 45: tokio::loom::std::unsafe_cell::UnsafeCell<T>::with_mut\n at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.40.0/src/loom/std/unsafe_cell.rs:16:9\n 46: tokio::runtime::task::core::Core<T,S>::poll\n at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.40.0/src/runtime/task/core.rs:320:13\n 47: tokio::runtime::task::harness::poll_future::{{closure}}\n at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.40.0/src/runtime/task/harness.rs:500:19\n 48: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once\n at /rustc/4d91de4e48198da2e33413efdcd9cd2cc0c46688/library/core/src/panic/unwind_safe.rs:272:9\n 49: std::panicking::try::do_call\n at /rustc/4d91de4e48198da2e33413efdcd9cd2cc0c46688/library/std/src/panicking.rs:584:40\n 50: __rust_try\n 51: std::panicking::try\n at /rustc/4d91de4e48198da2e33413efdcd9cd2cc0c46688/library/std/src/panicking.rs:547:19\n 52: std::panic::catch_unwind\n at /rustc/4d91de4e48198da2e33413efdcd9cd2cc0c46688/library/std/src/panic.rs:358:14\n 53: tokio::runtime::task::harness::poll_future\n at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.40.0/src/runtime/task/harness.rs:488:18\n 54: tokio::runtime::task::harness::Harness<T,S>::poll_inner\n at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.40.0/src/runtime/task/harness.rs:209:27\n 55: tokio::runtime::task::harness::Harness<T,S>::poll\n at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.40.0/src/runtime/task/harness.rs:154:15\n 56: tokio::runtime::task::raw::poll\n at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.40.0/src/runtime/task/raw.rs:271:5\n 57: tokio::runtime::task::raw::RawTask::poll\n at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.40.0/src/runtime/task/raw.rs:201:18\n 58: tokio::runtime::task::UnownedTask<S>::run\n at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.40.0/src/runtime/task/mod.rs:473:9\n 59: tokio::runtime::blocking::pool::Task::run\n at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.40.0/src/runtime/blocking/pool.rs:160:9\n 60: tokio::runtime::blocking::pool::Inner::run\n at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.40.0/src/runtime/blocking/pool.rs:518:17\n 61: tokio::runtime::blocking::pool::Spawner::spawn_thread::{{closure}}\n at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.40.0/src/runtime/blocking/pool.rs:476:13\n 62: std::sys::backtrace::__rust_begin_short_backtrace\n at /rustc/4d91de4e48198da2e33413efdcd9cd2cc0c46688/library/std/src/sys/backtrace.rs:152:18\n 63: std::thread::Builder::spawn_unchecked_::{{closure}}::{{closure}}\n at /rustc/4d91de4e48198da2e33413efdcd9cd2cc0c46688/library/std/src/thread/mod.rs:564:17\n 64: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once\n at /rustc/4d91de4e48198da2e33413efdcd9cd2cc0c46688/library/core/src/panic/unwind_safe.rs:272:9\n 65: std::panicking::try::do_call\n at /rustc/4d91de4e48198da2e33413efdcd9cd2cc0c46688/library/std/src/panicking.rs:584:40\n 66: __rust_try\n 67: std::panicking::try\n at /rustc/4d91de4e48198da2e33413efdcd9cd2cc0c46688/library/std/src/panicking.rs:547:19\n 68: std::panic::catch_unwind\n at /rustc/4d91de4e48198da2e33413efdcd9cd2cc0c46688/library/std/src/panic.rs:358:14\n 69: std::thread::Builder::spawn_unchecked_::{{closure}}\n at /rustc/4d91de4e48198da2e33413efdcd9cd2cc0c46688/library/std/src/thread/mod.rs:562:30\n 70: core::ops::function::FnOnce::call_once{{vtable.shim}}\n at /rustc/4d91de4e48198da2e33413efdcd9cd2cc0c46688/library/core/src/ops/function.rs:250:5\n 71: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once\n at /rustc/4d91de4e48198da2e33413efdcd9cd2cc0c46688/library/alloc/src/boxed.rs:1993:9\n 72: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once\n at /rustc/4d91de4e48198da2e33413efdcd9cd2cc0c46688/library/alloc/src/boxed.rs:1993:9\n 73: std::sys::pal::unix::thread::Thread::new::thread_start\n at /rustc/4d91de4e48198da2e33413efdcd9cd2cc0c46688/library/std/src/sys/pal/unix/thread.rs:106:17\n 74: _thrp_setup\n 75: <unknown>","level":30,"hostname":"buskin","pid":862,"time":"2025-03-10T21:08:51.855310105Z","target":"phd_runner::execute","line":119,"file":"phd-tests/runner/src/execute.rs"}
2522025-03-10T21:08:51.626Z{"v":0,"name":"phd-runner","msg":"[TEST_CLEANUP - START]","level":30,"hostname":"buskin","pid":862,"time":"2025-03-10T21:08:51.855360486Z","target":"phd_runner::fixtures","line":51,"file":"phd-tests/runner/src/fixtures.rs"}
2532025-03-10T21:08:51.626Z{"v":0,"name":"phd-runner","msg":"[TEST_CLEANUP - END]","level":30,"hostname":"buskin","pid":862,"time":"2025-03-10T21:08:51.855421987Z","target":"phd_runner::fixtures","line":51,"file":"phd-tests/runner/src/fixtures.rs","elapsed_milliseconds":0}
2542025-03-10T21:08:51.627Z{"v":0,"name":"phd-runner","msg":"[EXECUTION_CLEANUP - START]","level":30,"hostname":"buskin","pid":862,"time":"2025-03-10T21:08:51.855467468Z","target":"phd_runner::fixtures","line":35,"file":"phd-tests/runner/src/fixtures.rs"}
2552025-03-10T21:08:51.627Z{"v":0,"name":"phd-runner","msg":"[EXECUTION_CLEANUP - END]","level":30,"hostname":"buskin","pid":862,"time":"2025-03-10T21:08:51.855495418Z","target":"phd_runner::fixtures","line":35,"file":"phd-tests/runner/src/fixtures.rs","elapsed_milliseconds":0}
2562025-03-10T21:08:51.852Z{"v":0,"name":"phd-runner","msg":"[INSTANCE_ENSURE_INTERNAL - EVENT] permanent error from instance_spec_ensure","level":50,"hostname":"buskin","pid":862,"time":"2025-03-10T21:08:52.088590588Z","target":"phd_framework::test_vm","line":324,"file":"phd-tests/framework/src/test_vm/mod.rs","e":"Error Response: status: 500 Internal Server Error; headers: {\"content-type\": \"application/json\", \"x-request-id\": \"571e4e3d-1a71-4701-8991-5958d651cb01\", \"content-length\": \"124\", \"date\": \"Mon, 10 Mar 2025 21:08:52 GMT\"}; value: Error { error_code: Some(\"Internal\"), message: \"Internal Server Error\", request_id: \"571e4e3d-1a71-4701-8991-5958d651cb01\" }","path":"phd_tests::migrate::from_base::can_migrate_from_base","vm":"migration_from_base","vm_id":"cadd903e-2784-4c2d-b4d0-e1f60ff9ed27"}
2572025-03-10T21:08:51.878Z{"v":0,"name":"phd-runner","msg":"[INSTANCE_ENSURE_INTERNAL - END]","level":30,"hostname":"buskin","pid":862,"time":"2025-03-10T21:08:52.088774541Z","target":"phd_framework::test_vm","line":276,"file":"phd-tests/framework/src/test_vm/mod.rs","path":"phd_tests::migrate::from_base::can_migrate_from_base","vm_id":"cadd903e-2784-4c2d-b4d0-e1f60ff9ed27","elapsed_milliseconds":391,"vm":"migration_from_base"}
2582025-03-10T21:08:51.878Z{"v":0,"name":"phd-runner","msg":"[TEST - EVENT] Killing Propolis server process","level":20,"hostname":"buskin","pid":862,"time":"2025-03-10T21:08:52.088869732Z","target":"phd_framework::test_vm::server","line":125,"file":"phd-tests/framework/src/test_vm/server.rs","self.address":"127.0.0.1:9000","path":"phd_tests::migrate::from_base::can_migrate_from_base"}
2592025-03-10T21:08:51.903Z{"v":0,"name":"phd-runner","msg":"[TEST - EVENT] zfs clone dropped","level":20,"hostname":"buskin","pid":862,"time":"2025-03-10T21:08:52.137844712Z","target":"phd_framework::zfs","line":83,"file":"phd-tests/framework/src/zfs.rs","path":"phd_tests::migrate::from_base::can_migrate_from_base"}
2602025-03-10T21:08:51.903Z{"v":0,"name":"phd-runner","msg":"[TEST - EVENT] executing ZFS command","level":20,"hostname":"buskin","pid":862,"time":"2025-03-10T21:08:52.137943594Z","target":"phd_framework::zfs","line":304,"file":"phd-tests/framework/src/zfs.rs","args":"[\"phd-artifacts/phd-clone-e2dc39fc-3c98-41a4-b6ba-a24d44613e82\"]","verb":"destroy","path":"phd_tests::migrate::from_base::can_migrate_from_base"}
2612025-03-10T21:08:51.956Z{"v":0,"name":"phd-runner","msg":"[TEST - EVENT] zfs snapshot dropped","level":20,"hostname":"buskin","pid":862,"time":"2025-03-10T21:08:52.192393708Z","target":"phd_framework::zfs","line":57,"file":"phd-tests/framework/src/zfs.rs","path":"phd_tests::migrate::from_base::can_migrate_from_base"}
2622025-03-10T21:08:51.980Z{"v":0,"name":"phd-runner","msg":"[TEST - EVENT] executing ZFS command","level":20,"hostname":"buskin","pid":862,"time":"2025-03-10T21:08:52.192448579Z","target":"phd_framework::zfs","line":304,"file":"phd-tests/framework/src/zfs.rs","args":"[\"phd-artifacts@phd-1c3f9a47-443c-4e05-8e70-ef38dfc3fa6e\"]","verb":"destroy","path":"phd_tests::migrate::from_base::can_migrate_from_base"}
2632025-03-10T21:08:52.005Z{"v":0,"name":"phd-runner","msg":"[TEST - END]","level":30,"hostname":"buskin","pid":862,"time":"2025-03-10T21:08:52.24085775Z","target":"phd_tests::migrate::from_base","line":34,"file":"phd-tests/tests/src/migrate.rs","path":"phd_tests::migrate::from_base::can_migrate_from_base","elapsed_milliseconds":747}
2642025-03-10T21:08:52.005Z{"v":0,"name":"phd-runner","msg":"test phd_tests::migrate::from_base::can_migrate_from_base ... FAILED: Error Response: status: 500 Internal Server Error; headers: {\"content-type\": \"application/json\", \"x-request-id\": \"571e4e3d-1a71-4701-8991-5958d651cb01\", \"content-length\": \"124\", \"date\": \"Mon, 10 Mar 2025 21:08:52 GMT\"}; value: Error { error_code: Some(\"Internal\"), message: \"Internal Server Error\", request_id: \"571e4e3d-1a71-4701-8991-5958d651cb01\" }\n\nStack backtrace:\n 0: anyhow::error::<impl core::convert::From<E> for anyhow::Error>::from\n at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/anyhow-1.0.89/src/backtrace.rs:27:14\n 1: <core::result::Result<T,F> as core::ops::try_trait::FromResidual<core::result::Result<core::convert::Infallible,E>>>::from_residual\n at /rustc/4d91de4e48198da2e33413efdcd9cd2cc0c46688/library/core/src/result.rs:2014:27\n 2: phd_framework::test_vm::TestVm::instance_ensure_internal::{{closure}}::{{closure}}\n at ./oxidecomputer/propolis/phd-tests/framework/src/test_vm/mod.rs:336:9\n 3: <tracing::instrument::Instrumented<T> as core::future::future::Future>::poll\n at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tracing-0.1.40/src/instrument.rs:321:9\n 4: phd_framework::test_vm::TestVm::instance_ensure_internal::{{closure}}\n at ./oxidecomputer/propolis/phd-tests/framework/src/test_vm/mod.rs:276:5\n 5: phd_framework::test_vm::TestVm::instance_ensure::{{closure}}\n at ./oxidecomputer/propolis/phd-tests/framework/src/test_vm/mod.rs:414:22\n 6: phd_framework::test_vm::TestVm::launch::{{closure}}\n at ./oxidecomputer/propolis/phd-tests/framework/src/test_vm/mod.rs:402:32\n 7: phd_tests::migrate::run_smoke_test::{{closure}}\n at ./oxidecomputer/propolis/phd-tests/tests/src/migrate.rs:315:21\n 8: phd_tests::migrate::from_base::can_migrate_from_base::{{closure}}::{{closure}}\n at ./oxidecomputer/propolis/phd-tests/tests/src/migrate.rs:37:14\n 9: <tracing::instrument::Instrumented<T> as core::future::future::Future>::poll\n at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tracing-0.1.40/src/instrument.rs:321:9\n 10: phd_tests::migrate::from_base::can_migrate_from_base::{{closure}}\n at ./oxidecomputer/propolis/phd-tests/tests/src/migrate.rs:34:5\n 11: <core::pin::Pin<P> as core::future::future::Future>::poll\n at /rustc/4d91de4e48198da2e33413efdcd9cd2cc0c46688/library/core/src/future/future.rs:124:9\n 12: phd_testcase::TestCase::run::{{closure}}\n at ./oxidecomputer/propolis/phd-tests/testcase/src/lib.rs:79:32\n 13: phd_runner::execute::run_tests_with_ctx::{{closure}}::run_tests::{{closure}}\n at ./oxidecomputer/propolis/phd-tests/runner/src/execute.rs:117:58\n 14: tokio::runtime::task::core::Core<T,S>::poll::{{closure}}\n at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.40.0/src/runtime/task/core.rs:331:17\n 15: tokio::loom::std::unsafe_cell::UnsafeCell<T>::with_mut\n at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.40.0/src/loom/std/unsafe_cell.rs:16:9\n 16: tokio::runtime::task::core::Core<T,S>::poll\n at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.40.0/src/runtime/task/core.rs:320:13\n 17: tokio::runtime::task::harness::poll_future::{{closure}}\n at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.40.0/src/runtime/task/harness.rs:500:19\n 18: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once\n at /rustc/4d91de4e48198da2e33413efdcd9cd2cc0c46688/library/core/src/panic/unwind_safe.rs:272:9\n 19: std::panicking::try::do_call\n at /rustc/4d91de4e48198da2e33413efdcd9cd2cc0c46688/library/std/src/panicking.rs:584:40\n 20: __rust_try\n 21: std::panicking::try\n at /rustc/4d91de4e48198da2e33413efdcd9cd2cc0c46688/library/std/src/panicking.rs:547:19\n 22: std::panic::catch_unwind\n at /rustc/4d91de4e48198da2e33413efdcd9cd2cc0c46688/library/std/src/panic.rs:358:14\n 23: tokio::runtime::task::harness::poll_future\n at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.40.0/src/runtime/task/harness.rs:488:18\n 24: tokio::runtime::task::harness::Harness<T,S>::poll_inner\n at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.40.0/src/runtime/task/harness.rs:209:27\n 25: tokio::runtime::task::harness::Harness<T,S>::poll\n at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.40.0/src/runtime/task/harness.rs:154:15\n 26: tokio::runtime::task::raw::poll\n at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.40.0/src/runtime/task/raw.rs:271:5\n 27: tokio::runtime::task::raw::RawTask::poll\n at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.40.0/src/runtime/task/raw.rs:201:18\n 28: tokio::runtime::task::LocalNotified<S>::run\n at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.40.0/src/runtime/task/mod.rs:436:9\n 29: tokio::runtime::scheduler::multi_thread::worker::Context::run_task::{{closure}}\n at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.40.0/src/runtime/scheduler/multi_thread/worker.rs:661:17\n 30: tokio::runtime::coop::with_budget\n at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.40.0/src/runtime/coop.rs:107:5\n 31: tokio::runtime::coop::budget\n at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.40.0/src/runtime/coop.rs:73:5\n 32: tokio::runtime::scheduler::multi_thread::worker::Context::run_task\n at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.40.0/src/runtime/scheduler/multi_thread/worker.rs:597:9\n 33: tokio::runtime::scheduler::multi_thread::worker::Context::run\n at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.40.0/src/runtime/scheduler/multi_thread/worker.rs:548:24\n 34: tokio::runtime::scheduler::multi_thread::worker::run::{{closure}}::{{closure}}\n at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.40.0/src/runtime/scheduler/multi_thread/worker.rs:513:21\n 35: tokio::runtime::context::scoped::Scoped<T>::set\n at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.40.0/src/runtime/context/scoped.rs:40:9\n 36: tokio::runtime::context::set_scheduler::{{closure}}\n at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.40.0/src/runtime/context.rs:180:26\n 37: std::thread::local::LocalKey<T>::try_with\n at /rustc/4d91de4e48198da2e33413efdcd9cd2cc0c46688/library/std/src/thread/local.rs:308:12\n 38: std::thread::local::LocalKey<T>::with\n at /rustc/4d91de4e48198da2e33413efdcd9cd2cc0c46688/library/std/src/thread/local.rs:272:9\n 39: tokio::runtime::context::set_scheduler\n at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.40.0/src/runtime/context.rs:180:9\n 40: tokio::runtime::scheduler::multi_thread::worker::run::{{closure}}\n at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.40.0/src/runtime/scheduler/multi_thread/worker.rs:508:9\n 41: tokio::runtime::context::runtime::enter_runtime\n at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.40.0/src/runtime/context/runtime.rs:65:16\n 42: tokio::runtime::scheduler::multi_thread::worker::run\n at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.40.0/src/runtime/scheduler/multi_thread/worker.rs:500:5\n 43: tokio::runtime::scheduler::multi_thread::worker::Launch::launch::{{closure}}\n at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.40.0/src/runtime/scheduler/multi_thread/worker.rs:466:45\n 44: <tokio::runtime::blocking::task::BlockingTask<T> as core::future::future::Future>::poll\n at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.40.0/src/runtime/blocking/task.rs:42:21\n 45: tokio::runtime::task::core::Core<T,S>::poll::{{closure}}\n at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.40.0/src/runtime/task/core.rs:331:17\n 46: tokio::loom::std::unsafe_cell::UnsafeCell<T>::with_mut\n at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.40.0/src/loom/std/unsafe_cell.rs:16:9\n 47: tokio::runtime::task::core::Core<T,S>::poll\n at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.40.0/src/runtime/task/core.rs:320:13\n 48: tokio::runtime::task::harness::poll_future::{{closure}}\n at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.40.0/src/runtime/task/harness.rs:500:19\n 49: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once\n at /rustc/4d91de4e48198da2e33413efdcd9cd2cc0c46688/library/core/src/panic/unwind_safe.rs:272:9\n 50: std::panicking::try::do_call\n at /rustc/4d91de4e48198da2e33413efdcd9cd2cc0c46688/library/std/src/panicking.rs:584:40\n 51: __rust_try\n 52: std::panicking::try\n at /rustc/4d91de4e48198da2e33413efdcd9cd2cc0c46688/library/std/src/panicking.rs:547:19\n 53: std::panic::catch_unwind\n at /rustc/4d91de4e48198da2e33413efdcd9cd2cc0c46688/library/std/src/panic.rs:358:14\n 54: tokio::runtime::task::harness::poll_future\n at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.40.0/src/runtime/task/harness.rs:488:18\n 55: tokio::runtime::task::harness::Harness<T,S>::poll_inner\n at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.40.0/src/runtime/task/harness.rs:209:27\n 56: tokio::runtime::task::harness::Harness<T,S>::poll\n at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.40.0/src/runtime/task/harness.rs:154:15\n 57: tokio::runtime::task::raw::poll\n at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.40.0/src/runtime/task/raw.rs:271:5\n 58: tokio::runtime::task::raw::RawTask::poll\n at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.40.0/src/runtime/task/raw.rs:201:18\n 59: tokio::runtime::task::UnownedTask<S>::run\n at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.40.0/src/runtime/task/mod.rs:473:9\n 60: tokio::runtime::blocking::pool::Task::run\n at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.40.0/src/runtime/blocking/pool.rs:160:9\n 61: tokio::runtime::blocking::pool::Inner::run\n at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.40.0/src/runtime/blocking/pool.rs:518:17\n 62: tokio::runtime::blocking::pool::Spawner::spawn_thread::{{closure}}\n at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.40.0/src/runtime/blocking/pool.rs:476:13\n 63: std::sys::backtrace::__rust_begin_short_backtrace\n at /rustc/4d91de4e48198da2e33413efdcd9cd2cc0c46688/library/std/src/sys/backtrace.rs:152:18\n 64: std::thread::Builder::spawn_unchecked_::{{closure}}::{{closure}}\n at /rustc/4d91de4e48198da2e33413efdcd9cd2cc0c46688/library/std/src/thread/mod.rs:564:17\n 65: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once\n at /rustc/4d91de4e48198da2e33413efdcd9cd2cc0c46688/library/core/src/panic/unwind_safe.rs:272:9\n 66: std::panicking::try::do_call\n at /rustc/4d91de4e48198da2e33413efdcd9cd2cc0c46688/library/std/src/panicking.rs:584:40\n 67: __rust_try\n 68: std::panicking::try\n at /rustc/4d91de4e48198da2e33413efdcd9cd2cc0c46688/library/std/src/panicking.rs:547:19\n 69: std::panic::catch_unwind\n at /rustc/4d91de4e48198da2e33413efdcd9cd2cc0c46688/library/std/src/panic.rs:358:14\n 70: std::thread::Builder::spawn_unchecked_::{{closure}}\n at /rustc/4d91de4e48198da2e33413efdcd9cd2cc0c46688/library/std/src/thread/mod.rs:562:30\n 71: core::ops::function::FnOnce::call_once{{vtable.shim}}\n at /rustc/4d91de4e48198da2e33413efdcd9cd2cc0c46688/library/core/src/ops/function.rs:250:5\n 72: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once\n at /rustc/4d91de4e48198da2e33413efdcd9cd2cc0c46688/library/alloc/src/boxed.rs:1993:9\n 73: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once\n at /rustc/4d91de4e48198da2e33413efdcd9cd2cc0c46688/library/alloc/src/boxed.rs:1993:9\n 74: std::sys::pal::unix::thread::Thread::new::thread_start\n at /rustc/4d91de4e48198da2e33413efdcd9cd2cc0c46688/library/std/src/sys/pal/unix/thread.rs:106:17\n 75: _thrp_setup\n 76: <unknown>","level":30,"hostname":"buskin","pid":862,"time":"2025-03-10T21:08:52.241832265Z","target":"phd_runner::execute","line":119,"file":"phd-tests/runner/src/execute.rs"}
2652025-03-10T21:08:52.006Z{"v":0,"name":"phd-runner","msg":"[TEST_CLEANUP - START]","level":30,"hostname":"buskin","pid":862,"time":"2025-03-10T21:08:52.241874096Z","target":"phd_runner::fixtures","line":51,"file":"phd-tests/runner/src/fixtures.rs"}
2662025-03-10T21:08:52.032Z{"v":0,"name":"phd-runner","msg":"[TEST_CLEANUP - END]","level":30,"hostname":"buskin","pid":862,"time":"2025-03-10T21:08:52.241935627Z","target":"phd_runner::fixtures","line":51,"file":"phd-tests/runner/src/fixtures.rs","elapsed_milliseconds":0}
2672025-03-10T21:08:52.032Z{"v":0,"name":"phd-runner","msg":"[EXECUTION_CLEANUP - START]","level":30,"hostname":"buskin","pid":862,"time":"2025-03-10T21:08:52.241975207Z","target":"phd_runner::fixtures","line":35,"file":"phd-tests/runner/src/fixtures.rs"}
2682025-03-10T21:08:52.032Z{"v":0,"name":"phd-runner","msg":"[EXECUTION_CLEANUP - END]","level":30,"hostname":"buskin","pid":862,"time":"2025-03-10T21:08:52.242002098Z","target":"phd_runner::fixtures","line":35,"file":"phd-tests/runner/src/fixtures.rs","elapsed_milliseconds":0}
2692025-03-10T21:08:52.032Z
2702025-03-10T21:08:52.032Zfailures:
2712025-03-10T21:08:52.032Z phd_tests::migrate::from_base::serial_history
2722025-03-10T21:08:52.032Z phd_tests::migrate::from_base::migration_from_base_and_back
2732025-03-10T21:08:52.032Z phd_tests::migrate::from_base::can_migrate_from_base
2742025-03-10T21:08:52.032Z
2752025-03-10T21:08:52.032Ztest result: FAILED. 0 passed; 3 failed; 0 skipped; 0 not run; finished in 74.60s
2762025-03-10T21:08:52.032Z
2772025-03-10T21:08:52.032Z{"v":0,"name":"phd-runner","msg":"phd_runner","level":20,"hostname":"buskin","pid":862,"time":"2025-03-10T21:08:52.242309112Z","target":"phd_runner","line":41,"file":"phd-tests/runner/src/main.rs","exit_code":3}
2782025-03-10T21:08:52.163Z
2792025-03-10T21:08:52.188Zreal 1:16.115175100
2802025-03-10T21:08:52.188Zuser 38.049869825
2812025-03-10T21:08:52.188Zsys 3.810575121
2822025-03-10T21:08:52.188Ztrap 0.005033254
2832025-03-10T21:08:52.188Ztflt 0.000648649
2842025-03-10T21:08:52.188Zdflt 0.000648343
2852025-03-10T21:08:52.188Zkflt 0.000000000
2862025-03-10T21:08:52.188Zlock 1:01:40.021689139
2872025-03-10T21:08:52.188Zslp 1:05.644067633
2882025-03-10T21:08:52.188Zlat 0.103209801
2892025-03-10T21:08:52.188Zstop 31.774033034
2902025-03-10T21:08:52.188Z+ failcount=3
2912025-03-10T21:08:52.188Z+ set -e
2922025-03-10T21:08:52.188Z+ set +e
2932025-03-10T21:08:52.188Z+ tar -czvf /tmp/phd-tmp-files.tar.gz -C /tmp/propolis-phd '/tmp/propolis-phd/*.log'
2942025-03-10T21:08:52.188Ztar: /tmp/propolis-phd/*.log: No such file or directory
2952025-03-10T21:08:52.188ZCompressing '/tmp/phd-tmp-files.tar.gz' with '/usr/bin/gzip'...
2962025-03-10T21:08:52.188Z+ for runnerdir in $tmpdir/runner-*
2972025-03-10T21:08:52.189Z+ '[' -d /tmp/propolis-phd/runner-0 ']'
2982025-03-10T21:08:52.189Z+ tar -rzvf /tmp/phd-tmp-files.tar.gz -C /tmp/propolis-phd/runner-0 /tmp/propolis-phd/runner-0/migration_from_base.stderr.log /tmp/propolis-phd/runner-0/migration_from_base.stdout.log /tmp/propolis-phd/runner-0/migration_serial_history_base.stderr.log /tmp/propolis-phd/runner-0/migration_serial_history_base.stdout.log
2992025-03-10T21:08:52.189Za /tmp/propolis-phd/runner-0/migration_from_base.stderr.log 0K
3002025-03-10T21:08:52.189Za /tmp/propolis-phd/runner-0/migration_from_base.stdout.log 20K
3012025-03-10T21:08:52.189Za /tmp/propolis-phd/runner-0/migration_serial_history_base.stderr.log 0K
3022025-03-10T21:08:52.189Za /tmp/propolis-phd/runner-0/migration_serial_history_base.stdout.log 20K
3032025-03-10T21:08:52.189ZCompressing '/tmp/phd-tmp-files.tar' with '/usr/bin/gzip'...
3042025-03-10T21:08:52.189Z+ for runnerdir in $tmpdir/runner-*
3052025-03-10T21:08:52.189Z+ '[' -d /tmp/propolis-phd/runner-1 ']'
3062025-03-10T21:08:52.189Z+ tar -rzvf /tmp/phd-tmp-files.tar.gz -C /tmp/propolis-phd/runner-1 /tmp/propolis-phd/runner-1/migration_from_base_and_back.stderr.log /tmp/propolis-phd/runner-1/migration_from_base_and_back.stdout.log
3072025-03-10T21:08:52.189Za /tmp/propolis-phd/runner-1/migration_from_base_and_back.stderr.log 0K
3082025-03-10T21:08:52.189Za /tmp/propolis-phd/runner-1/migration_from_base_and_back.stdout.log 20K
3092025-03-10T21:08:52.190ZCompressing '/tmp/phd-tmp-files.tar' with '/usr/bin/gzip'...
3102025-03-10T21:08:52.190Z+ set -e
3112025-03-10T21:08:52.190Z+ exitcode=0
3122025-03-10T21:08:52.190Z+ '[' 3 -eq 0 ']'
3132025-03-10T21:08:52.190Z+ echo
3142025-03-10T21:08:52.190Z
3152025-03-10T21:08:52.190Z+ echo 'SOME TESTS FAILED'
3162025-03-10T21:08:52.190ZSOME TESTS FAILED
3172025-03-10T21:08:52.190Z+ echo
3182025-03-10T21:08:52.190Z
3192025-03-10T21:08:52.190Z+ exitcode=1
3202025-03-10T21:08:52.190Z+ find /dev/vmm -mindepth 1 -maxdepth 1
3212025-03-10T21:08:52.190Z+ read
3222025-03-10T21:08:52.190Z+ exit 1
3232025-03-10T21:08:52.190Zprocess exited: duration 85718 ms, exit code 1
 
3242025-03-10T21:08:52.240Zfound 2 output files
3252025-03-10T21:08:52.241Zuploading: /tmp/phd-runner.log (92653 bytes)
3262025-03-10T21:08:53.369Zuploaded: /tmp/phd-runner.log
3272025-03-10T21:08:53.369Zuploading: /tmp/phd-tmp-files.tar.gz (3736 bytes)
3282025-03-10T21:08:54.466Zuploaded: /tmp/phd-tmp-files.tar.gz