(stdout not captured) (stderr not captured) (stdout not captured) (stderr not captured) thread 'example::tests::builder_zone_counts' panicked at nexus/reconfigurator/planning/src/example.rs:591:9: assertion failed: string doesn't match the contents of file: "tests/output/example_builder_zone_counts_blueprint.txt" see diffset above set EXPECTORATE=overwrite if these changes are intentional stack backtrace: 0: rust_begin_unwind at /rustc/4d91de4e48198da2e33413efdcd9cd2cc0c46688/library/std/src/panicking.rs:692:5 1: core::panicking::panic_fmt at /rustc/4d91de4e48198da2e33413efdcd9cd2cc0c46688/library/core/src/panicking.rs:75:14 2: expectorate::assert_contents at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/expectorate-1.1.0/src/lib.rs:64:9 3: builder_zone_counts at ./src/example.rs:591:9 4: nexus_reconfigurator_planning::example::tests::builder_zone_counts::{{closure}} at ./src/example.rs:572:29 5: core::ops::function::FnOnce::call_once at /rustc/4d91de4e48198da2e33413efdcd9cd2cc0c46688/library/core/src/ops/function.rs:250:5 6: core::ops::function::FnOnce::call_once at /rustc/4d91de4e48198da2e33413efdcd9cd2cc0c46688/library/core/src/ops/function.rs:250:5 note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace. running 1 test @@ -485,5 +485,6 @@ created at::::::::::: 1970-01-01T00:00:00.000Z comment:::::::::::::: (none) internal DNS version: 1 external DNS version: 1 + PENDING MGS-MANAGED UPDATES: 0 test example::tests::builder_zone_counts ... FAILED failures: failures: example::tests::builder_zone_counts test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 48 filtered out; finished in 0.90s log file: /var/tmp/omicron_tmp/nexus_reconfigurator_planning-801ad5dea45473c0-example_builder_zone_counts.17221.0.log note: configured to log to "/var/tmp/omicron_tmp/nexus_reconfigurator_planning-801ad5dea45473c0-example_builder_zone_counts.17221.0.log" thread 'example::tests::builder_zone_counts' panicked at nexus/reconfigurator/planning/src/example.rs:591:9: assertion failed: string doesn't match the contents of file: "tests/output/example_builder_zone_counts_blueprint.txt" see diffset above set EXPECTORATE=overwrite if these changes are intentional stack backtrace: 0: rust_begin_unwind at /rustc/4d91de4e48198da2e33413efdcd9cd2cc0c46688/library/std/src/panicking.rs:692:5 1: core::panicking::panic_fmt at /rustc/4d91de4e48198da2e33413efdcd9cd2cc0c46688/library/core/src/panicking.rs:75:14 2: expectorate::assert_contents at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/expectorate-1.1.0/src/lib.rs:64:9 3: builder_zone_counts at ./src/example.rs:591:9 4: nexus_reconfigurator_planning::example::tests::builder_zone_counts::{{closure}} at ./src/example.rs:572:29 5: core::ops::function::FnOnce::call_once at /rustc/4d91de4e48198da2e33413efdcd9cd2cc0c46688/library/core/src/ops/function.rs:250:5 6: core::ops::function::FnOnce::call_once at /rustc/4d91de4e48198da2e33413efdcd9cd2cc0c46688/library/core/src/ops/function.rs:250:5 note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace. thread 'planner::test::planner_decommissions_sleds' panicked at nexus/reconfigurator/planning/src/planner.rs:2890:9: assertion failed: string doesn't match the contents of file: "tests/output/planner_decommissions_sleds_bp2.txt" see diffset above set EXPECTORATE=overwrite if these changes are intentional stack backtrace: 0: rust_begin_unwind at /rustc/4d91de4e48198da2e33413efdcd9cd2cc0c46688/library/std/src/panicking.rs:692:5 1: core::panicking::panic_fmt at /rustc/4d91de4e48198da2e33413efdcd9cd2cc0c46688/library/core/src/panicking.rs:75:14 2: expectorate::assert_contents at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/expectorate-1.1.0/src/lib.rs:64:9 3: nexus_reconfigurator_planning::planner::test::planner_decommissions_sleds at ./src/planner.rs:2890:9 4: nexus_reconfigurator_planning::planner::test::planner_decommissions_sleds::{{closure}} at ./src/planner.rs:2854:37 5: core::ops::function::FnOnce::call_once at /rustc/4d91de4e48198da2e33413efdcd9cd2cc0c46688/library/core/src/ops/function.rs:250:5 6: core::ops::function::FnOnce::call_once at /rustc/4d91de4e48198da2e33413efdcd9cd2cc0c46688/library/core/src/ops/function.rs:250:5 note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace. running 1 test @@ -292,5 +292,6 @@ created at::::::::::: 1970-01-01T00:00:00.000Z comment:::::::::::::: sled a1b477db-b629-48eb-911d-1ccdafca75b9 expunged (expunged 10 disks, 47 datasets, 15 zones) internal DNS version: 1 external DNS version: 1 + PENDING MGS-MANAGED UPDATES: 0 test planner::test::planner_decommissions_sleds ... FAILED failures: failures: planner::test::planner_decommissions_sleds test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 48 filtered out; finished in 0.67s log file: /var/tmp/omicron_tmp/nexus_reconfigurator_planning-801ad5dea45473c0-planner_decommissions_sleds.17224.0.log note: configured to log to "/var/tmp/omicron_tmp/nexus_reconfigurator_planning-801ad5dea45473c0-planner_decommissions_sleds.17224.0.log" thread 'planner::test::planner_decommissions_sleds' panicked at nexus/reconfigurator/planning/src/planner.rs:2890:9: assertion failed: string doesn't match the contents of file: "tests/output/planner_decommissions_sleds_bp2.txt" see diffset above set EXPECTORATE=overwrite if these changes are intentional stack backtrace: 0: rust_begin_unwind at /rustc/4d91de4e48198da2e33413efdcd9cd2cc0c46688/library/std/src/panicking.rs:692:5 1: core::panicking::panic_fmt at /rustc/4d91de4e48198da2e33413efdcd9cd2cc0c46688/library/core/src/panicking.rs:75:14 2: expectorate::assert_contents at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/expectorate-1.1.0/src/lib.rs:64:9 3: nexus_reconfigurator_planning::planner::test::planner_decommissions_sleds at ./src/planner.rs:2890:9 4: nexus_reconfigurator_planning::planner::test::planner_decommissions_sleds::{{closure}} at ./src/planner.rs:2854:37 5: core::ops::function::FnOnce::call_once at /rustc/4d91de4e48198da2e33413efdcd9cd2cc0c46688/library/core/src/ops/function.rs:250:5 6: core::ops::function::FnOnce::call_once at /rustc/4d91de4e48198da2e33413efdcd9cd2cc0c46688/library/core/src/ops/function.rs:250:5 note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace. thread 'planner::test::test_expunge_clickhouse_clusters' panicked at nexus/reconfigurator/planning/src/planner.rs:3755:9: assertion failed: string doesn't match the contents of file: "tests/output/planner_expunge_clickhouse_clusters_3_4.txt" see diffset above set EXPECTORATE=overwrite if these changes are intentional stack backtrace: 0: rust_begin_unwind at /rustc/4d91de4e48198da2e33413efdcd9cd2cc0c46688/library/std/src/panicking.rs:692:5 1: core::panicking::panic_fmt at /rustc/4d91de4e48198da2e33413efdcd9cd2cc0c46688/library/core/src/panicking.rs:75:14 2: expectorate::assert_contents at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/expectorate-1.1.0/src/lib.rs:64:9 3: test_expunge_clickhouse_clusters at ./src/planner.rs:3755:9 4: nexus_reconfigurator_planning::planner::test::test_expunge_clickhouse_clusters::{{closure}} at ./src/planner.rs:3626:42 5: core::ops::function::FnOnce::call_once at /rustc/4d91de4e48198da2e33413efdcd9cd2cc0c46688/library/core/src/ops/function.rs:250:5 6: core::ops::function::FnOnce::call_once at /rustc/4d91de4e48198da2e33413efdcd9cd2cc0c46688/library/core/src/ops/function.rs:250:5 note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace. running 1 test @@ -390,19 +390,19 @@ max used keeper id::::::::::::::::::::::::::::: 3 (unchanged) cluster name::::::::::::::::::::::::::::::::::: oximeter_cluster (unchanged) cluster secret::::::::::::::::::::::::::::::::: 61731868-2fb2-48cc-b135-1eebcfbff058 (unchanged) highest seen keeper leader committed log index: 1 (unchanged) - clickhouse keepers generation 2 -> 3: + clickhouse keepers generation 2 -> 3: ------------------------------------------------ zone id keeper id ------------------------------------------------ 2b368dcd-bb4b-4415-9daf-0d36fd769fed 1 3f78fa04-0b7e-4959-9718-c232c76500e1 2 - 1998de68-a305-44f7-8ebc-cc5613c9e6fb 0 - clickhouse servers generation 2 -> 3: + clickhouse servers generation 2 -> 3: ------------------------------------------------ zone id server id ------------------------------------------------ 097ab295-aec4-427f-9124-bb4e8aecfeca 1 f0341fc6-6a73-4318-8787-dd0c6b26c385 2 test planner::test::test_expunge_clickhouse_clusters ... FAILED failures: failures: planner::test::test_expunge_clickhouse_clusters test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 48 filtered out; finished in 0.31s log file: /var/tmp/omicron_tmp/nexus_reconfigurator_planning-801ad5dea45473c0-planner_expunge_clickhouse_clusters.17234.0.log note: configured to log to "/var/tmp/omicron_tmp/nexus_reconfigurator_planning-801ad5dea45473c0-planner_expunge_clickhouse_clusters.17234.0.log" thread 'planner::test::test_expunge_clickhouse_clusters' panicked at nexus/reconfigurator/planning/src/planner.rs:3755:9: assertion failed: string doesn't match the contents of file: "tests/output/planner_expunge_clickhouse_clusters_3_4.txt" see diffset above set EXPECTORATE=overwrite if these changes are intentional stack backtrace: 0: rust_begin_unwind at /rustc/4d91de4e48198da2e33413efdcd9cd2cc0c46688/library/std/src/panicking.rs:692:5 1: core::panicking::panic_fmt at /rustc/4d91de4e48198da2e33413efdcd9cd2cc0c46688/library/core/src/panicking.rs:75:14 2: expectorate::assert_contents at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/expectorate-1.1.0/src/lib.rs:64:9 3: test_expunge_clickhouse_clusters at ./src/planner.rs:3755:9 4: nexus_reconfigurator_planning::planner::test::test_expunge_clickhouse_clusters::{{closure}} at ./src/planner.rs:3626:42 5: core::ops::function::FnOnce::call_once at /rustc/4d91de4e48198da2e33413efdcd9cd2cc0c46688/library/core/src/ops/function.rs:250:5 6: core::ops::function::FnOnce::call_once at /rustc/4d91de4e48198da2e33413efdcd9cd2cc0c46688/library/core/src/ops/function.rs:250:5 note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace. thread 'planner::test::test_expunge_clickhouse_zones_after_policy_is_changed' panicked at nexus/reconfigurator/planning/src/planner.rs:3954:9: assertion failed: string doesn't match the contents of file: "tests/output/planner_expunge_clickhouse_zones_after_policy_is_changed_3_4.txt" see diffset above set EXPECTORATE=overwrite if these changes are intentional stack backtrace: 0: rust_begin_unwind at /rustc/4d91de4e48198da2e33413efdcd9cd2cc0c46688/library/std/src/panicking.rs:692:5 1: core::panicking::panic_fmt at /rustc/4d91de4e48198da2e33413efdcd9cd2cc0c46688/library/core/src/panicking.rs:75:14 2: expectorate::assert_contents at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/expectorate-1.1.0/src/lib.rs:64:9 3: test_expunge_clickhouse_zones_after_policy_is_changed at ./src/planner.rs:3954:9 4: nexus_reconfigurator_planning::planner::test::test_expunge_clickhouse_zones_after_policy_is_changed::{{closure}} at ./src/planner.rs:3844:63 5: core::ops::function::FnOnce::call_once at /rustc/4d91de4e48198da2e33413efdcd9cd2cc0c46688/library/core/src/ops/function.rs:250:5 6: core::ops::function::FnOnce::call_once at /rustc/4d91de4e48198da2e33413efdcd9cd2cc0c46688/library/core/src/ops/function.rs:250:5 note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace. running 1 test @@ -320,19 +320,19 @@ - max used keeper id::::::::::::::::::::::::::::: 3 - cluster name::::::::::::::::::::::::::::::::::: oximeter_cluster - cluster secret::::::::::::::::::::::::::::::::: 7894a059-89b5-407b-ad17-1d8d885e2f67 - highest seen keeper leader committed log index: 0 - clickhouse keepers at generation 2: + clickhouse keepers at generation 2: ------------------------------------------------ zone id keeper id ------------------------------------------------ - 0326e8e8-1d52-495e-9773-cd3fa559edbe 1 - 10db7ee4-04bb-430a-836d-c4308716e995 2 - 14b84b08-598b-4756-99ec-eea98f8fa46e 3 - clickhouse servers at generation 2: + clickhouse servers at generation 2: ------------------------------------------------ zone id server id ------------------------------------------------ - 1b6b784b-d157-4544-b7ff-6c71f3622a0f 1 - fe8b84ef-ee63-4dd9-9110-4a8dd696b5c4 2 test planner::test::test_expunge_clickhouse_zones_after_policy_is_changed ... FAILED failures: failures: planner::test::test_expunge_clickhouse_zones_after_policy_is_changed test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 48 filtered out; finished in 0.31s log file: /var/tmp/omicron_tmp/nexus_reconfigurator_planning-801ad5dea45473c0-planner_expunge_clickhouse_zones_after_policy_is_changed.17235.0.log note: configured to log to "/var/tmp/omicron_tmp/nexus_reconfigurator_planning-801ad5dea45473c0-planner_expunge_clickhouse_zones_after_policy_is_changed.17235.0.log" thread 'planner::test::test_expunge_clickhouse_zones_after_policy_is_changed' panicked at nexus/reconfigurator/planning/src/planner.rs:3954:9: assertion failed: string doesn't match the contents of file: "tests/output/planner_expunge_clickhouse_zones_after_policy_is_changed_3_4.txt" see diffset above set EXPECTORATE=overwrite if these changes are intentional stack backtrace: 0: rust_begin_unwind at /rustc/4d91de4e48198da2e33413efdcd9cd2cc0c46688/library/std/src/panicking.rs:692:5 1: core::panicking::panic_fmt at /rustc/4d91de4e48198da2e33413efdcd9cd2cc0c46688/library/core/src/panicking.rs:75:14 2: expectorate::assert_contents at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/expectorate-1.1.0/src/lib.rs:64:9 3: test_expunge_clickhouse_zones_after_policy_is_changed at ./src/planner.rs:3954:9 4: nexus_reconfigurator_planning::planner::test::test_expunge_clickhouse_zones_after_policy_is_changed::{{closure}} at ./src/planner.rs:3844:63 5: core::ops::function::FnOnce::call_once at /rustc/4d91de4e48198da2e33413efdcd9cd2cc0c46688/library/core/src/ops/function.rs:250:5 6: core::ops::function::FnOnce::call_once at /rustc/4d91de4e48198da2e33413efdcd9cd2cc0c46688/library/core/src/ops/function.rs:250:5 note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace. thread 'planner::test::test_nexus_allocation_skips_nonprovisionable_sleds' panicked at nexus/reconfigurator/planning/src/planner.rs:2642:9: assertion failed: string doesn't match the contents of file: "tests/output/planner_nonprovisionable_bp2.txt" see diffset above set EXPECTORATE=overwrite if these changes are intentional stack backtrace: 0: rust_begin_unwind at /rustc/4d91de4e48198da2e33413efdcd9cd2cc0c46688/library/std/src/panicking.rs:692:5 1: core::panicking::panic_fmt at /rustc/4d91de4e48198da2e33413efdcd9cd2cc0c46688/library/core/src/panicking.rs:75:14 2: expectorate::assert_contents at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/expectorate-1.1.0/src/lib.rs:64:9 3: test_nexus_allocation_skips_nonprovisionable_sleds at ./src/planner.rs:2642:9 4: nexus_reconfigurator_planning::planner::test::test_nexus_allocation_skips_nonprovisionable_sleds::{{closure}} at ./src/planner.rs:2520:60 5: core::ops::function::FnOnce::call_once at /rustc/4d91de4e48198da2e33413efdcd9cd2cc0c46688/library/core/src/ops/function.rs:250:5 6: core::ops::function::FnOnce::call_once at /rustc/4d91de4e48198da2e33413efdcd9cd2cc0c46688/library/core/src/ops/function.rs:250:5 note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace. running 1 test 1 -> 2: marked non-provisionable 2d1cb4f2-cf44-40fc-b118-85036eb732a9 1 -> 2: expunged 48d95fef-bc9f-4f50-9a53-1e075836291d 1 -> 2: decommissioned 68d24ac5-f341-49ea-a92a-0381b52ab387 @@ -464,5 +464,6 @@ created at::::::::::: 1970-01-01T00:00:00.000Z comment:::::::::::::: sled 48d95fef-bc9f-4f50-9a53-1e075836291d expunged (expunged 10 disks, 45 datasets, 14 zones) internal DNS version: 1 external DNS version: 1 + PENDING MGS-MANAGED UPDATES: 0 test planner::test::test_nexus_allocation_skips_nonprovisionable_sleds ... FAILED failures: failures: planner::test::test_nexus_allocation_skips_nonprovisionable_sleds test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 48 filtered out; finished in 0.22s log file: /var/tmp/omicron_tmp/nexus_reconfigurator_planning-801ad5dea45473c0-planner_nexus_allocation_skips_nonprovisionable_sleds.17237.0.log note: configured to log to "/var/tmp/omicron_tmp/nexus_reconfigurator_planning-801ad5dea45473c0-planner_nexus_allocation_skips_nonprovisionable_sleds.17237.0.log" thread 'planner::test::test_nexus_allocation_skips_nonprovisionable_sleds' panicked at nexus/reconfigurator/planning/src/planner.rs:2642:9: assertion failed: string doesn't match the contents of file: "tests/output/planner_nonprovisionable_bp2.txt" see diffset above set EXPECTORATE=overwrite if these changes are intentional stack backtrace: 0: rust_begin_unwind at /rustc/4d91de4e48198da2e33413efdcd9cd2cc0c46688/library/std/src/panicking.rs:692:5 1: core::panicking::panic_fmt at /rustc/4d91de4e48198da2e33413efdcd9cd2cc0c46688/library/core/src/panicking.rs:75:14 2: expectorate::assert_contents at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/expectorate-1.1.0/src/lib.rs:64:9 3: test_nexus_allocation_skips_nonprovisionable_sleds at ./src/planner.rs:2642:9 4: nexus_reconfigurator_planning::planner::test::test_nexus_allocation_skips_nonprovisionable_sleds::{{closure}} at ./src/planner.rs:2520:60 5: core::ops::function::FnOnce::call_once at /rustc/4d91de4e48198da2e33413efdcd9cd2cc0c46688/library/core/src/ops/function.rs:250:5 6: core::ops::function::FnOnce::call_once at /rustc/4d91de4e48198da2e33413efdcd9cd2cc0c46688/library/core/src/ops/function.rs:250:5 note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace. thread 'planner::test::test_plan_deploy_all_clickhouse_cluster_nodes' panicked at nexus/reconfigurator/planning/src/planner.rs:3317:9: assertion failed: string doesn't match the contents of file: "tests/output/planner_deploy_all_keeper_nodes_1_2.txt" see diffset above set EXPECTORATE=overwrite if these changes are intentional stack backtrace: 0: rust_begin_unwind at /rustc/4d91de4e48198da2e33413efdcd9cd2cc0c46688/library/std/src/panicking.rs:692:5 1: core::panicking::panic_fmt at /rustc/4d91de4e48198da2e33413efdcd9cd2cc0c46688/library/core/src/panicking.rs:75:14 2: expectorate::assert_contents at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/expectorate-1.1.0/src/lib.rs:64:9 3: test_plan_deploy_all_clickhouse_cluster_nodes at ./src/planner.rs:3317:9 4: nexus_reconfigurator_planning::planner::test::test_plan_deploy_all_clickhouse_cluster_nodes::{{closure}} at ./src/planner.rs:3279:55 5: core::ops::function::FnOnce::call_once at /rustc/4d91de4e48198da2e33413efdcd9cd2cc0c46688/library/core/src/ops/function.rs:250:5 6: core::ops::function::FnOnce::call_once at /rustc/4d91de4e48198da2e33413efdcd9cd2cc0c46688/library/core/src/ops/function.rs:250:5 note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace. running 1 test @@ -302,19 +302,19 @@ + max used keeper id::::::::::::::::::::::::::::: 3 + cluster name::::::::::::::::::::::::::::::::::: oximeter_cluster + cluster secret::::::::::::::::::::::::::::::::: 6a984e2e-20be-4ed9-b572-d7ef063c67f7 + highest seen keeper leader committed log index: 0 - clickhouse keepers at generation 2: + clickhouse keepers at generation 2: ------------------------------------------------ zone id keeper id ------------------------------------------------ + 31a66e47-6252-4f38-aa92-f48637da152c 1 + 7aa82b80-0c37-4f20-9398-bfce4707f327 2 + dd5ab8b1-e6e7-40e0-9ea6-b6a16f485b52 3 - clickhouse servers at generation 2: + clickhouse servers at generation 2: ------------------------------------------------ zone id server id ------------------------------------------------ + 6fedfc94-2ae6-4cc7-a968-5185207284f4 1 + 812be70a-f237-47d5-ab49-f9450fe948db 2 test planner::test::test_plan_deploy_all_clickhouse_cluster_nodes ... FAILED failures: failures: planner::test::test_plan_deploy_all_clickhouse_cluster_nodes test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 48 filtered out; finished in 0.28s log file: /var/tmp/omicron_tmp/nexus_reconfigurator_planning-801ad5dea45473c0-planner_deploy_all_keeper_nodes.17238.0.log note: configured to log to "/var/tmp/omicron_tmp/nexus_reconfigurator_planning-801ad5dea45473c0-planner_deploy_all_keeper_nodes.17238.0.log" thread 'planner::test::test_plan_deploy_all_clickhouse_cluster_nodes' panicked at nexus/reconfigurator/planning/src/planner.rs:3317:9: assertion failed: string doesn't match the contents of file: "tests/output/planner_deploy_all_keeper_nodes_1_2.txt" see diffset above set EXPECTORATE=overwrite if these changes are intentional stack backtrace: 0: rust_begin_unwind at /rustc/4d91de4e48198da2e33413efdcd9cd2cc0c46688/library/std/src/panicking.rs:692:5 1: core::panicking::panic_fmt at /rustc/4d91de4e48198da2e33413efdcd9cd2cc0c46688/library/core/src/panicking.rs:75:14 2: expectorate::assert_contents at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/expectorate-1.1.0/src/lib.rs:64:9 3: test_plan_deploy_all_clickhouse_cluster_nodes at ./src/planner.rs:3317:9 4: nexus_reconfigurator_planning::planner::test::test_plan_deploy_all_clickhouse_cluster_nodes::{{closure}} at ./src/planner.rs:3279:55 5: core::ops::function::FnOnce::call_once at /rustc/4d91de4e48198da2e33413efdcd9cd2cc0c46688/library/core/src/ops/function.rs:250:5 6: core::ops::function::FnOnce::call_once at /rustc/4d91de4e48198da2e33413efdcd9cd2cc0c46688/library/core/src/ops/function.rs:250:5 note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace. thread 'test_omdb_success_cases' panicked at dev-tools/omdb/tests/test_all_output.rs:258:5: assertion failed: string doesn't match the contents of file: "tests/successes.out" see diffset above set EXPECTORATE=overwrite if these changes are intentional stack backtrace: 0: rust_begin_unwind at /rustc/4d91de4e48198da2e33413efdcd9cd2cc0c46688/library/std/src/panicking.rs:692:5 1: core::panicking::panic_fmt at /rustc/4d91de4e48198da2e33413efdcd9cd2cc0c46688/library/core/src/panicking.rs:75:14 2: expectorate::assert_contents at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/expectorate-1.1.0/src/lib.rs:64:9 3: {async_fn#0} at ./tests/test_all_output.rs:258:5 4: {async_block#0} at ./tests/test_all_output.rs:121:1 5: poll<&mut dyn core::future::future::Future<Output=()>> at /rustc/4d91de4e48198da2e33413efdcd9cd2cc0c46688/library/core/src/future/future.rs:124:9 6: poll<&mut core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>> at /rustc/4d91de4e48198da2e33413efdcd9cd2cc0c46688/library/core/src/future/future.rs:124:9 7: {closure#0}<core::pin::Pin<&mut core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>> at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.43.0/src/runtime/scheduler/current_thread/mod.rs:729:57 8: with_budget<core::task::poll::Poll<()>, tokio::runtime::scheduler::current_thread::{impl#8}::block_on::{closure#0}::{closure#0}::{closure_env#0}<core::pin::Pin<&mut core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>>> at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.43.0/src/runtime/coop.rs:107:5 9: budget<core::task::poll::Poll<()>, tokio::runtime::scheduler::current_thread::{impl#8}::block_on::{closure#0}::{closure#0}::{closure_env#0}<core::pin::Pin<&mut core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>>> at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.43.0/src/runtime/coop.rs:73:5 10: {closure#0}<core::pin::Pin<&mut core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>> at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.43.0/src/runtime/scheduler/current_thread/mod.rs:729:25 11: tokio::runtime::scheduler::current_thread::Context::enter at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.43.0/src/runtime/scheduler/current_thread/mod.rs:428:19 12: {closure#0}<core::pin::Pin<&mut core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>> at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.43.0/src/runtime/scheduler/current_thread/mod.rs:728:36 13: tokio::runtime::scheduler::current_thread::CoreGuard::enter::{{closure}} at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.43.0/src/runtime/scheduler/current_thread/mod.rs:807:68 14: tokio::runtime::context::scoped::Scoped<T>::set at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.43.0/src/runtime/context/scoped.rs:40:9 15: tokio::runtime::context::set_scheduler::{{closure}} at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.43.0/src/runtime/context.rs:180:26 16: try_with<tokio::runtime::context::Context, tokio::runtime::context::set_scheduler::{closure_env#0}<(alloc::boxed::Box<tokio::runtime::scheduler::current_thread::Core, alloc::alloc::Global>, core::option::Option<()>), tokio::runtime::scheduler::current_thread::{impl#8}::enter::{closure_env#0}<tokio::runtime::scheduler::current_thread::{impl#8}::block_on::{closure_env#0}<core::pin::Pin<&mut core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>>, core::option::Option<()>>>, (alloc::boxed::Box<tokio::runtime::scheduler::current_thread::Core, alloc::alloc::Global>, core::option::Option<()>)> at /rustc/4d91de4e48198da2e33413efdcd9cd2cc0c46688/library/std/src/thread/local.rs:308:12 17: std::thread::local::LocalKey<T>::with at /rustc/4d91de4e48198da2e33413efdcd9cd2cc0c46688/library/std/src/thread/local.rs:272:9 18: tokio::runtime::context::set_scheduler at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.43.0/src/runtime/context.rs:180:9 19: tokio::runtime::scheduler::current_thread::CoreGuard::enter at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.43.0/src/runtime/scheduler/current_thread/mod.rs:807:27 20: tokio::runtime::scheduler::current_thread::CoreGuard::block_on at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.43.0/src/runtime/scheduler/current_thread/mod.rs:716:19 21: {closure#0}<core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>> at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.43.0/src/runtime/scheduler/current_thread/mod.rs:196:28 22: tokio::runtime::context::runtime::enter_runtime at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.43.0/src/runtime/context/runtime.rs:65:16 23: block_on<core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>> at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.43.0/src/runtime/scheduler/current_thread/mod.rs:184:9 24: tokio::runtime::runtime::Runtime::block_on_inner at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.43.0/src/runtime/runtime.rs:368:47 25: block_on<core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>> at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.43.0/src/runtime/runtime.rs:342:13 26: test_omdb_success_cases at ./tests/test_all_output.rs:121:1 27: test_all_output::test_omdb_success_cases::{{closure}} at ./tests/test_all_output.rs:122:70 28: core::ops::function::FnOnce::call_once at /rustc/4d91de4e48198da2e33413efdcd9cd2cc0c46688/library/core/src/ops/function.rs:250:5 29: core::ops::function::FnOnce::call_once at /rustc/4d91de4e48198da2e33413efdcd9cd2cc0c46688/library/core/src/ops/function.rs:250:5 note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace. WARN: dropped CockroachInstance without cleaning it up first (there may still be a child process running and a temporary directory leaked) WARN: temporary directory leaked: "/var/tmp/omicron_tmp/.tmpgOpnZV" If you would like to access the database for debugging, run the following: # Run the database cargo xtask db-dev run --no-populate --store-dir "/var/tmp/omicron_tmp/.tmpgOpnZV/data" # Access the database. Note the port may change if you run multiple databases. cockroach sql --host=localhost:32221 --insecure WARN: dropped ClickHouse process without cleaning it up first (there may still be a child process running (PID 28282) and a temporary directory leaked, /var/tmp/omicron_tmp/test_all_output-7c515e0b96a9562f-test_omdb_success_cases.28275.1-clickhouse-m9rJSm) failed to clean up ClickHouse data dir: - /var/tmp/omicron_tmp/test_all_output-7c515e0b96a9562f-test_omdb_success_cases.28275.1-clickhouse-m9rJSm: File exists (os error 17) WARN: dropped DendriteInstance without cleaning it up first (there may still be a child process running and a temporary directory leaked) WARN: dendrite temporary directory leaked: /var/tmp/omicron_tmp/.tmpw3Rnkp WARN: dropped DendriteInstance without cleaning it up first (there may still be a child process running and a temporary directory leaked) WARN: dendrite temporary directory leaked: /var/tmp/omicron_tmp/.tmptkK9kr WARN: dropped MgdInstance without cleaning it up first (there may still be a child process running and a temporary directory leaked) WARN: mgd temporary directory leaked: /var/tmp/omicron_tmp/.tmpZbgYxZ WARN: dropped MgdInstance without cleaning it up first (there may still be a child process running and a temporary directory leaked) WARN: mgd temporary directory leaked: /var/tmp/omicron_tmp/.tmpcssYHe running 1 test running commands with args: ["db", "disks", "list"] running commands with args: ["db", "dns", "show"] running commands with args: ["db", "dns", "diff", "external", "2"] running commands with args: ["db", "dns", "names", "external", "2"] running commands with args: ["db", "instances"] running commands with args: ["db", "sleds"] running commands with args: ["db", "sleds", "-F", "discretionary"] running commands with args: ["mgs", "inventory"] running commands with args: ["nexus", "background-tasks", "doc"] running commands with args: ["nexus", "background-tasks", "show"] running commands with args: ["nexus", "background-tasks", "show", "saga_recovery"] running commands with args: ["nexus", "background-tasks", "show", "blueprint_loader", "blueprint_executor"] running commands with args: ["nexus", "background-tasks", "show", "dns_internal"] running commands with args: ["nexus", "background-tasks", "show", "dns_external"] running commands with args: ["nexus", "background-tasks", "show", "all"] running commands with args: ["nexus", "sagas", "list"] running commands with args: ["--destructive", "nexus", "sagas", "demo-create"] running commands with args: ["nexus", "sagas", "list"] running commands with args: ["--destructive", "nexus", "background-tasks", "activate", "inventory_collection"] running commands with args: ["nexus", "blueprints", "list"] running commands with args: ["nexus", "blueprints", "show", "e77eef41-3144-4035-89a1-c97dd61b8e4a"] running commands with args: ["nexus", "blueprints", "show", "current-target"] running commands with args: ["nexus", "blueprints", "diff", "e77eef41-3144-4035-89a1-c97dd61b8e4a", "current-target"] running commands with args: ["reconfigurator", "export", "/var/tmp/omicron_tmp/.tmpBUfHxE/reconfigurator-export.out"] @@ -1421,10 +1421,11 @@ created at::::::::::: <REDACTED_TIMESTAMP> comment:::::::::::::: initial test blueprint internal DNS version: 1 external DNS version: 2 + PENDING MGS-MANAGED UPDATES: 0 --------------------------------------------- stderr: note: using Nexus URL http://127.0.0.1:REDACTED_PORT/ ============================================= @@ -1517,10 +1518,11 @@ created at::::::::::: <REDACTED_TIMESTAMP> comment:::::::::::::: initial test blueprint internal DNS version: 1 external DNS version: 2 + PENDING MGS-MANAGED UPDATES: 0 --------------------------------------------- stderr: note: using Nexus URL http://127.0.0.1:REDACTED_PORT/ ============================================= test test_omdb_success_cases ... FAILED failures: failures: test_omdb_success_cases test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 2 filtered out; finished in 35.22s log file: /var/tmp/omicron_tmp/test_all_output-7c515e0b96a9562f-test_omdb_success_cases.28275.0.log note: configured to log to "/var/tmp/omicron_tmp/test_all_output-7c515e0b96a9562f-test_omdb_success_cases.28275.0.log" DB URL: postgresql://root@[::1]:64496/omicron?sslmode=disable DB address: [::1]:64496 log file: /var/tmp/omicron_tmp/test_all_output-7c515e0b96a9562f-test_omdb_success_cases.28275.2.log note: configured to log to "/var/tmp/omicron_tmp/test_all_output-7c515e0b96a9562f-test_omdb_success_cases.28275.2.log" log file: /var/tmp/omicron_tmp/test_all_output-7c515e0b96a9562f-test_omdb_success_cases.28275.3.log note: configured to log to "/var/tmp/omicron_tmp/test_all_output-7c515e0b96a9562f-test_omdb_success_cases.28275.3.log" log file: /var/tmp/omicron_tmp/test_all_output-7c515e0b96a9562f-test_omdb_success_case.28275.4.log note: configured to log to "/var/tmp/omicron_tmp/test_all_output-7c515e0b96a9562f-test_omdb_success_case.28275.4.log" thread 'test_omdb_success_cases' panicked at dev-tools/omdb/tests/test_all_output.rs:258:5: assertion failed: string doesn't match the contents of file: "tests/successes.out" see diffset above set EXPECTORATE=overwrite if these changes are intentional stack backtrace: 0: rust_begin_unwind at /rustc/4d91de4e48198da2e33413efdcd9cd2cc0c46688/library/std/src/panicking.rs:692:5 1: core::panicking::panic_fmt at /rustc/4d91de4e48198da2e33413efdcd9cd2cc0c46688/library/core/src/panicking.rs:75:14 2: expectorate::assert_contents at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/expectorate-1.1.0/src/lib.rs:64:9 3: {async_fn#0} at ./tests/test_all_output.rs:258:5 4: {async_block#0} at ./tests/test_all_output.rs:121:1 5: poll<&mut dyn core::future::future::Future<Output=()>> at /rustc/4d91de4e48198da2e33413efdcd9cd2cc0c46688/library/core/src/future/future.rs:124:9 6: poll<&mut core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>> at /rustc/4d91de4e48198da2e33413efdcd9cd2cc0c46688/library/core/src/future/future.rs:124:9 7: {closure#0}<core::pin::Pin<&mut core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>> at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.43.0/src/runtime/scheduler/current_thread/mod.rs:729:57 8: with_budget<core::task::poll::Poll<()>, tokio::runtime::scheduler::current_thread::{impl#8}::block_on::{closure#0}::{closure#0}::{closure_env#0}<core::pin::Pin<&mut core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>>> at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.43.0/src/runtime/coop.rs:107:5 9: budget<core::task::poll::Poll<()>, tokio::runtime::scheduler::current_thread::{impl#8}::block_on::{closure#0}::{closure#0}::{closure_env#0}<core::pin::Pin<&mut core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>>> at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.43.0/src/runtime/coop.rs:73:5 10: {closure#0}<core::pin::Pin<&mut core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>> at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.43.0/src/runtime/scheduler/current_thread/mod.rs:729:25 11: tokio::runtime::scheduler::current_thread::Context::enter at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.43.0/src/runtime/scheduler/current_thread/mod.rs:428:19 12: {closure#0}<core::pin::Pin<&mut core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>> at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.43.0/src/runtime/scheduler/current_thread/mod.rs:728:36 13: tokio::runtime::scheduler::current_thread::CoreGuard::enter::{{closure}} at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.43.0/src/runtime/scheduler/current_thread/mod.rs:807:68 14: tokio::runtime::context::scoped::Scoped<T>::set at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.43.0/src/runtime/context/scoped.rs:40:9 15: tokio::runtime::context::set_scheduler::{{closure}} at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.43.0/src/runtime/context.rs:180:26 16: try_with<tokio::runtime::context::Context, tokio::runtime::context::set_scheduler::{closure_env#0}<(alloc::boxed::Box<tokio::runtime::scheduler::current_thread::Core, alloc::alloc::Global>, core::option::Option<()>), tokio::runtime::scheduler::current_thread::{impl#8}::enter::{closure_env#0}<tokio::runtime::scheduler::current_thread::{impl#8}::block_on::{closure_env#0}<core::pin::Pin<&mut core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>>>, core::option::Option<()>>>, (alloc::boxed::Box<tokio::runtime::scheduler::current_thread::Core, alloc::alloc::Global>, core::option::Option<()>)> at /rustc/4d91de4e48198da2e33413efdcd9cd2cc0c46688/library/std/src/thread/local.rs:308:12 17: std::thread::local::LocalKey<T>::with at /rustc/4d91de4e48198da2e33413efdcd9cd2cc0c46688/library/std/src/thread/local.rs:272:9 18: tokio::runtime::context::set_scheduler at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.43.0/src/runtime/context.rs:180:9 19: tokio::runtime::scheduler::current_thread::CoreGuard::enter at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.43.0/src/runtime/scheduler/current_thread/mod.rs:807:27 20: tokio::runtime::scheduler::current_thread::CoreGuard::block_on at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.43.0/src/runtime/scheduler/current_thread/mod.rs:716:19 21: {closure#0}<core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>> at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.43.0/src/runtime/scheduler/current_thread/mod.rs:196:28 22: tokio::runtime::context::runtime::enter_runtime at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.43.0/src/runtime/context/runtime.rs:65:16 23: block_on<core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>> at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.43.0/src/runtime/scheduler/current_thread/mod.rs:184:9 24: tokio::runtime::runtime::Runtime::block_on_inner at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.43.0/src/runtime/runtime.rs:368:47 25: block_on<core::pin::Pin<&mut dyn core::future::future::Future<Output=()>>> at /home/build/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.43.0/src/runtime/runtime.rs:342:13 26: test_omdb_success_cases at ./tests/test_all_output.rs:121:1 27: test_all_output::test_omdb_success_cases::{{closure}} at ./tests/test_all_output.rs:122:70 28: core::ops::function::FnOnce::call_once at /rustc/4d91de4e48198da2e33413efdcd9cd2cc0c46688/library/core/src/ops/function.rs:250:5 29: core::ops::function::FnOnce::call_once at /rustc/4d91de4e48198da2e33413efdcd9cd2cc0c46688/library/core/src/ops/function.rs:250:5 note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace. WARN: dropped CockroachInstance without cleaning it up first (there may still be a child process running and a temporary directory leaked) WARN: temporary directory leaked: "/var/tmp/omicron_tmp/.tmpgOpnZV" If you would like to access the database for debugging, run the following: # Run the database cargo xtask db-dev run --no-populate --store-dir "/var/tmp/omicron_tmp/.tmpgOpnZV/data" # Access the database. Note the port may change if you run multiple databases. cockroach sql --host=localhost:32221 --insecure WARN: dropped ClickHouse process without cleaning it up first (there may still be a child process running (PID 28282) and a temporary directory leaked, /var/tmp/omicron_tmp/test_all_output-7c515e0b96a9562f-test_omdb_success_cases.28275.1-clickhouse-m9rJSm) failed to clean up ClickHouse data dir: - /var/tmp/omicron_tmp/test_all_output-7c515e0b96a9562f-test_omdb_success_cases.28275.1-clickhouse-m9rJSm: File exists (os error 17) WARN: dropped DendriteInstance without cleaning it up first (there may still be a child process running and a temporary directory leaked) WARN: dendrite temporary directory leaked: /var/tmp/omicron_tmp/.tmpw3Rnkp WARN: dropped DendriteInstance without cleaning it up first (there may still be a child process running and a temporary directory leaked) WARN: dendrite temporary directory leaked: /var/tmp/omicron_tmp/.tmptkK9kr WARN: dropped MgdInstance without cleaning it up first (there may still be a child process running and a temporary directory leaked) WARN: mgd temporary directory leaked: /var/tmp/omicron_tmp/.tmpZbgYxZ WARN: dropped MgdInstance without cleaning it up first (there may still be a child process running and a temporary directory leaked) WARN: mgd temporary directory leaked: /var/tmp/omicron_tmp/.tmpcssYHe