Skip to content

feat(skill): integrate eval authoring into FastSkill package - #30

Merged
aroff merged 1 commit into
mainfrom
feat/eval-authoring-in-fastskill
Sep 14, 2026
Merged

aroff merged 1 commit into
mainfrom
feat/eval-authoring-in-fastskill

Conversation

@aroff

@aroff aroff commented Sep 14, 2026

Copy link
Copy Markdown
Contributor

Summary

Integrate eval authoring into the existing fastskill/ skill, rather than distributing a second skill from the CLI repository. The entrypoint routes to the authoring workflow, supporting references, and a copyable invoice example. Version the expanded package as 2.1.0.

Retain the useful repository-only evals/ harness, generator, fixtures, and historical measurements. Move authoring scenarios into evals/authoring/fixtures/, outside the released ZIP. Document the assessment and exact changed-file coverage inventory in specs/002-eval-authoring-consolidation.md.

Repair staging that could erase an existing destination, propagate runtime failures from subset/full sweeps, and correct stale documentation and CI comments. Add deterministic package, preservation, invalid-input, and runner tests to CI.

Companion CLI cleanup: gofastskill/fastskill#333. These are coordinated draft changes; the linked main-branch documentation becomes available after the companion changes land. Do not merge or enable auto-merge.

Verification

  • server02, pinned FastSkill 0.9.230: 12 Python tests passed; namespace check, generator parity, vacuity guard, and diff check passed.
  • Real ZIP installation as 2.1.0; copied example and all three preserved v2 suites validated.
  • Installed-package Claude Sonnet 4.5 suite extension completed; independent validation and byte/row preservation checks passed. This is file authoring, not a new target-runtime sweep.
  • Existing positive/negative v1 fixtures exercised; failure detected for its intended reason.
  • Bash line coverage: run.sh 93.1%, stage.sh 100%; exact embedded Python 100%. Bash branch instrumentation is unavailable and not claimed.
  • Companion CLI: six engine contract and 13 documentation/related tests passed after relocation; prior full Linux, two-tool authoring, and calibrated native-judge evidence retained.

Limitations

No new full live model baseline is claimed. Existing baseline numbers are historical. The existing negative-control script remains useful but its assumptions and missing budget-invariance assertion are documented. Native-judge placeholders require owner configuration. GitHub CI status is reported separately from local evidence.

@aroff
aroff marked this pull request as ready for review September 14, 2026 23:33
@aroff
aroff merged commit cff0415 into main Sep 14, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant