Skip to content

[E2E][Testing] Set up E2E test averaging - #3209

Draft
ew3361zh wants to merge 23 commits into
devfrom
run-multi-e2e
Draft

ew3361zh wants to merge 23 commits into
devfrom
run-multi-e2e

Conversation

@ew3361zh

@ew3361zh ew3361zh commented Aug 14, 2026

Copy link
Copy Markdown
Collaborator

Context

Issue(s) closed by this pull request: closes #3155

What

Updates the options for E2E testing by establishing the ability to run multiple simulations of each type tested by E2E testing (freestall, open_lot, animals_only, field_and_feed) across different, user-specified random-seeds and averaging the results.

Key change to Inputs:

  • task attribute random_seed: int -> random_seeds: list[int] (random seed is now an array of ints)

New E2E testing process:

  1. Expand e2e task args as multi-run args (a simulation run for each random seed specified in the e2e task grouped by simulation type).
  2. Run all simulations and save the results individually as usual (the results from different runs of the same simulation type are enumerated to be able to keep them grouped but separate in the outputs dir).
  3. Group e2e results by output_prefix ONLY AFTER all simulations have completed.
  4. Average the results by group (i.e. by output_prefix aka freestall, open_lot, etc.) and by module over each run.
  5. Save the averaged results.
  6. Compare the averaged results to the expected results as before.

Why

Allows for a wider array of simulation data to be used in e2e testing to determine how affected the model is by changes to the code or inputs.

How

  1. Added new functions to TaskManager:
    • _group_end_to_end_testing_runs()
    • _run_end_to_end_testing_comparisons()
    • _expand_end_to_end_testing_args()
    • _process_end_to_end_testing_group()
  2. Updated random_seed in task properties from int to array of ints.
  3. Updated all affected task files.
  4. Added new functions to E2ETestResultsHandler:
    • average_test_results()
    • _average_results()
    • _validate_values()
    • _validate_results()
    • _load_results()
    • _extract_results_paths()
  5. Updated OutputManager reporting by listing the random seeds across which the results were averaged.

Test plan

  1. Run e2e testing as usual - it should pass (as of 9/3/2026)
  2. Update the e2e task by adding some additional random seeds to the

Input Changes

Output Changes

  • N/A

Filter

@github-actions

github-actions Bot commented Sep 2, 2026

Copy link
Copy Markdown
Contributor

Current Coverage: %

Mypy errors on run-multi-e2e branch: 1134
Mypy errors on dev branch: 1133
1 more errors on run-multi-e2e branch

@github-actions

github-actions Bot commented Sep 2, 2026

Copy link
Copy Markdown
Contributor

🚨 Please update the changelog. This PR cannot be merged until changelog_WIP.md is updated.
🚨 Unauthorized changes detected in protected files. Please remove these changes if they are not intended.
🚨 Some tests have failed.

@ew3361zh
ew3361zh changed the base branch from dev to ppo-initial-setup September 2, 2026 21:14
@ew3361zh
ew3361zh changed the base branch from ppo-initial-setup to dev September 2, 2026 21:14
@github-actions

github-actions Bot commented Sep 3, 2026

Copy link
Copy Markdown
Contributor

Current Coverage: 99%

Mypy errors on run-multi-e2e branch: 1165
Mypy errors on dev branch: 1164
1 more errors on run-multi-e2e branch

@github-actions

github-actions Bot commented Sep 3, 2026

Copy link
Copy Markdown
Contributor

🚨 Please update the changelog. This PR cannot be merged until changelog_WIP.md is updated.
🚨 Unauthorized changes detected in protected files. Please remove these changes if they are not intended.

@github-actions

github-actions Bot commented Sep 8, 2026

Copy link
Copy Markdown
Contributor

Current Coverage: 99%

Mypy errors on run-multi-e2e branch: 1165
Mypy errors on dev branch: 1164
1 more errors on run-multi-e2e branch

@github-actions

github-actions Bot commented Sep 8, 2026

Copy link
Copy Markdown
Contributor

🚨 Unauthorized changes detected in protected files. Please remove these changes if they are not intended.
🚨 Flake8 linting errors were found. Please fix the linting issues.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[E2E] Support averaging results

1 participant