Skip to content

feat(evals): define and verify skill-driven eval authoring - #333

Merged
aroff merged 4 commits into
mainfrom
feat/eval-authoring-skill
Sep 14, 2026
Merged

aroff merged 4 commits into
mainfrom
feat/eval-authoring-skill

Conversation

@aroff

@aroff aroff commented Sep 14, 2026 •

Copy link
Copy Markdown
Contributor

Summary

  • Define the skill-first eval-authoring workflow, independent outcome/adherence criteria, and acceptance requirements.
  • Keep six executable engine contract tests and repaired CLI documentation in the CLI repository.
  • Distribute authoring through the existing fastskill/ package in companion draft feat(skill): integrate eval authoring into FastSkill package skill#30, not through a duplicate skills/eval-authoring/ tree here. Assets, fixtures, and package tests now live in their owning skills repository.
  • Retain the compiler-recommended recursion-limit adjustment needed by strict nightly Clippy.
  • Update the verification report with the current 11-file inventory and preserve historical evidence.

Verification

  • Original Linux regression: 1,872 Nextest tests passed; later shared-cache failure passed with isolated FASTSKILL_CACHE_DIR.
  • Strict workspace Clippy, formatting, source-size, documentation parity, and diff checks passed. Changed CLI entrypoint: 91.22% lines / 96.00% branches.
  • Claude and Codex authored suites; Claude continued from Codex files. The deliberate 15.00 defect failed the independent 17.00 criterion; corrected target and rescore passed.
  • Owner-approved native judge tools-advanced rejected 15.00 and accepted 17.00 with zero judge errors. Credentials were not committed.
  • After relocation: six engine contract tests and 13 documentation/related tests passed; formatting/source-size/diff checks passed again.
  • Companion package: 12 deterministic tests, actual ZIP installation, installed-entrypoint authoring exercise, preservation checks, and harness line coverage recorded in its report.

Limitations and coordination

Both PRs must remain drafts. Never merge or enable auto-merge. The new main-branch distribution links depend on the companion PR landing.

Historical measurements are not a fresh full model baseline. Shared-cache isolation and cross-machine absolute saved-run paths remain documented limitations. Companion Bash branch instrumentation is unavailable; line coverage and behavioral error/preservation tests are recorded. GitHub CI is separate from local verification.

@aroff aroff changed the title feat(evals): add skill-driven eval authoring workflow feat(evals): define and verify skill-driven eval authoring Sep 14, 2026
@aroff
aroff marked this pull request as ready for review September 14, 2026 23:34
@aroff
aroff enabled auto-merge (squash) September 14, 2026 23:34
@aroff
aroff merged commit 041afa0 into main Sep 14, 2026
14 checks passed
@aroff
aroff deleted the feat/eval-authoring-skill branch September 14, 2026 23:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant