You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Updates the options for E2E testing by establishing the ability to run multiple simulations of each type tested by E2E testing (freestall, open_lot, animals_only, field_and_feed) across different, user-specified random-seeds and averaging the results.
Key change to Inputs:
task attribute random_seed: int -> random_seeds: list[int] (random seed is now an array of ints)
New E2E testing process:
Expand e2e task args as multi-run args (a simulation run for each random seed specified in the e2e task grouped by simulation type).
Run all simulations and save the results individually as usual (the results from different runs of the same simulation type are enumerated to be able to keep them grouped but separate in the outputs dir).
Group e2e results by output_prefixONLY AFTER all simulations have completed.
Average the results by group (i.e. by output_prefix aka freestall, open_lot, etc.) and by module over each run.
Save the averaged results.
Compare the averaged results to the expected results as before.
Why
Allows for a wider array of simulation data to be used in e2e testing to determine how affected the model is by changes to the code or inputs.
How
Added new functions to TaskManager:
_group_end_to_end_testing_runs()
_run_end_to_end_testing_comparisons()
_expand_end_to_end_testing_args()
_process_end_to_end_testing_group()
Updated random_seed in task properties from int to array of ints.
Updated all affected task files.
Added new functions to E2ETestResultsHandler:
average_test_results()
_average_results()
_validate_values()
_validate_results()
_load_results()
_extract_results_paths()
Updated OutputManager reporting by listing the random seeds across which the results were averaged.
Test plan
Run e2e testing as usual - it should pass (as of 9/3/2026)
Update the e2e task by adding some additional random seeds to the
🚨 Please update the changelog. This PR cannot be merged until changelog_WIP.md is updated.
🚨 Unauthorized changes detected in protected files. Please remove these changes if they are not intended.
🚨 Some tests have failed.
🚨 Please update the changelog. This PR cannot be merged until changelog_WIP.md is updated.
🚨 Unauthorized changes detected in protected files. Please remove these changes if they are not intended.
🚨 Unauthorized changes detected in protected files. Please remove these changes if they are not intended.
🚨 Flake8 linting errors were found. Please fix the linting issues.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Context
Issue(s) closed by this pull request: closes #3155
What
Updates the options for E2E testing by establishing the ability to run multiple simulations of each type tested by E2E testing (
freestall,open_lot,animals_only,field_and_feed) across different, user-specified random-seeds and averaging the results.Key change to Inputs:
taskattributerandom_seed: int ->random_seeds: list[int] (random seed is now an array of ints)New E2E testing process:
output_prefixONLY AFTER all simulations have completed.output_prefixakafreestall,open_lot, etc.) and by module over each run.Why
Allows for a wider array of simulation data to be used in e2e testing to determine how affected the model is by changes to the code or inputs.
How
TaskManager:_group_end_to_end_testing_runs()_run_end_to_end_testing_comparisons()_expand_end_to_end_testing_args()_process_end_to_end_testing_group()random_seedin task properties frominttoarrayofints.E2ETestResultsHandler:average_test_results()_average_results()_validate_values()_validate_results()_load_results()_extract_results_paths()OutputManagerreporting by listing the random seeds across which the results were averaged.Test plan
Input Changes
Output Changes
Filter