Skip to content

Say what each grader requires, as a checklist - #31

Open
johnl-amd wants to merge 1 commit into
mainfrom
requirements-checklist
Open

johnl-amd wants to merge 1 commit into
mainfrom
requirements-checklist

Conversation

@johnl-amd

Copy link
Copy Markdown
Collaborator

Adds docs/checklist.md: every bar the three graders hold a skill to, in one list.

  • A section per grader, in the order they run, each opening with what that grader actually does before listing what it wants.
  • The rules are taken from the code rather than restated from the prose: the SKILL.md limits from structure.py, the dataset shape and Tier 0 coverage from datasets.py, the four kinds of behavioral check from engine/scorers.py, the pass mark and the two infrastructure checks from routing_gate.
  • The CI section keeps reusable.yml and skill-evals.yml apart and gives the former's full input list. routing_room and infra_paths belong only to the latter, which the current docs make easy to get wrong.
  • Notes two things that fail quietly: routing ignores a case's workspace and sends the prompt verbatim, and an error or degraded case counts towards neither side of the bar.
  • Linked from the README and from the top of both existing docs.

Closes #9

Verified with skillscope structural --docs '*.md,docs/**/*.md': the 37 internal references across the four markdown files resolve.

The bars a skill is held to were spread across usage.md,
authoring-evals.md and the error strings the code raises, so there was
nowhere to look before asking for a run, or after a red one. Collect
them in one list, grouped by grader and written for someone meeting the
harness for the first time, and link it from the README and both docs.

Signed-off-by: John Lybeck <john.lybeck@amd.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Create checklist showing requirements to pass each test

1 participant