[E2E] Validate e2e comparison configuration before the simulation runs - #3273
matthew7838 wants to merge 44 commits into
Conversation
Adds E2ETestResultsHandler.validate_comparison_configuration(), called at the start of the END_TO_END_TESTING task, so a bad expected results path, an invalid must-change or expected results file, a bad conversion table, or a flagged name missing from every domain's expected results fails the task in seconds instead of after the full simulation. The comparison-time checks stay in place as a backstop.
matthew7838
changed the title
|
Current Coverage: 99% Mypy errors on e2e-prevalidation branch: 1164 |
1 similar comment
|
Current Coverage: 99% Mypy errors on e2e-prevalidation branch: 1164 |
…fore the simulation Extends the pre-simulation validation to every path the comparison and the expected results update use afterwards: each expected results file must be valid and sit in the task's filters directory, and each actual_results_path must be a prefix of the results file name the run writes. The UPDATE_E2E_TEST_RESULTS task now validates its configuration before the simulation as well.
|
Current Coverage: 99% Mypy errors on e2e-prevalidation branch: 1164 |
|
Current Coverage: 99% Mypy errors on e2e-prevalidation branch: 1164 |
The shared loader only requires an expected_results entry, as the comparison did on dev. validate_update_configuration checks for the keys that update_expected_test_results writes back.
|
Current Coverage: 99% Mypy errors on e2e-prevalidation branch: 1164 |
|
Current Coverage: 99% Mypy errors on e2e-prevalidation branch: 1164 |
|
Current Coverage: 99% Mypy errors on e2e-prevalidation branch: 1164 |
|
Current Coverage: 99% Mypy errors on e2e-prevalidation branch: 1164 |
|
Current Coverage: 99% Mypy errors on e2e-prevalidation branch: 1164 |
|
Current Coverage: 99% Mypy errors on e2e-prevalidation branch: 1164 |
| """ | ||
| om = OutputManager() | ||
| info_map: dict[str, Any] = { | ||
| "class": E2ETestResultsHandler.__class__.__name__, |
There was a problem hiding this comment.
| "class": E2ETestResultsHandler.__class__.__name__, | |
| "class": E2ETestResultsHandler.__name__, |
E2ETestResultsHandler.__class__.__name__ gives "type"
There was a problem hiding this comment.
Do a search and make sure to correct all occurrences
| """ | ||
| om = OutputManager() | ||
| info_map: dict[str, Any] = { | ||
| "class": E2ETestResultsHandler.__class__.__name__, |
There was a problem hiding this comment.
| "class": E2ETestResultsHandler.__class__.__name__, | |
| "class": E2ETestResultsHandler.__name__, |
|
Current Coverage: 99% Mypy errors on e2e-prevalidation branch: 1164 |
2 participants
Validates the end-to-end testing configuration before the simulation runs, so a configuration mistake fails the task in seconds instead of after the full run.
Context
Issue(s) closed by this pull request: closes #3222
What
E2ETestResultsHandler.validate_comparison_configuration()andvalidate_update_configuration(), called at the start of theEND_TO_END_TESTINGandUPDATE_E2E_TEST_RESULTStasks before the simulation._load_expected_results_file()(used by the validation and the comparison) and_validate_result_path_set().Why
The paths the comparison and the update use afterwards (must-change files, expected-results files, conversion CSV, actual results prefixes) were only read once the simulation had finished, so a bad path or a mistyped flagged name surfaced at the end of the task.
How
Before the simulation, every result path set is checked: the expected-results file must exist, be valid JSON with an
expected_resultsentry, and sit in the task's filters directory, andactual_results_pathmust be a prefix of the{output_prefix}_saved_variables_{filter name}_file name the run writes. For the comparison task the must-change files and the conversion table must also load, and every flagged name must be a key of some domain's expected results. For the update task the file must also contain the keys the update writes back. Failures are logged and raised, so the task terminates before the simulation.Test plan
actual_results_path) and forTaskManager(validation runs before the simulation for both tasks; a failure skips the simulation and the comparison or update).expected_results_path, oractual_results_pathfails the comparison task in ~4 s before the simulation, and a mistypedexpected_results_pathfails the update task the same way.Input Changes
Output Changes