Skip to content

Run the seed prover in CI, and keep its results with each release #80

Open

Description

@CaffeinatedCoder

CI checks only that every seed still applies (OracleSeedTests.EverySeed_AppliesToTheCurrentTree). Whether each seed is actually caught by the oracles it names is checked only by dotnet run tools/oracles.cs seeds, and running it is a local release obligation. Nothing records that it was run, or what it found, for a given release.

Change:

  • run the seed prover in CI: as a step of release.yml before publishing, a scheduled job, or both
  • fail the release if any seed is not caught
  • attach the machine-readable result (each seed, the oracles it named, caught or known) to the GitHub release

#55 should land first, since the prover can currently count a seed as a known candidate when a different row in the same test method failed.

Found in an external review of the repository.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area/releaseVersioning, changelogs, release wiringtestsAdds or changes tests only (oracles, harnesses); no library code

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions