Skip to content

feat(evals): add lightweight smoke tests and grading safeguards - #31

Merged
aroff merged 1 commit into
mainfrom
feat/eval-suite-smoke-and-grading
Oct 2, 2026
Merged

aroff merged 1 commit into
mainfrom
feat/eval-suite-smoke-and-grading

Conversation

@aroff

@aroff aroff commented Oct 2, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Keep FastSkill evaluation lightweight while making the results easier to interpret:

  • Add a six-case, one-trial smoke command without native-judge prerequisites. Preserve the full 42-case benchmark as an opt-in run.
  • Rename trace-fragment matching so it is not presented as semantic answer accuracy. Remove the incorrect prohibition that rejected correct refusals mentioning a nonexistent publish command.
  • Add labelled judge-calibration contrasts without claiming they have been calibrated against a live judge.
  • Restrict hosted CI to its configured Claude authentication route, forward the target model explicitly, and require negative fixtures to fail for their intended check.
  • Preserve original metadata and per-trial paths in the consultation negative control, asserting the specific consultation flip and unchanged tool-budget verdict.
  • Add create/repair/extend authoring exercises with actual validation and file/row preservation checks.

@aroff
aroff marked this pull request as ready for review October 2, 2026 15:13
@aroff
aroff merged commit de32f9f into main Oct 2, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant