Skip to content

skill-testing.md

Latest commit

 

History

History
125 lines (91 loc) · 5 KB

skill-testing.md

File metadata and controls

125 lines (91 loc) · 5 KB
Raw

Skill Testing Guide

This document provides a comprehensive methodology for testing, evaluating, and optimizing skills in this repository. It covers eval creation, fixture management, iterative improvement, trigger testing, and description optimization.

1. Recommended workflow

  1. Create eval cases in skills/<skill-name>/evals/evals.json
  2. Add fixture files under skills/<skill-name>/evals/files/ when prompts depend on local files
  3. Start a Copilot CLI session from the repository root
  4. Ask Copilot to use the skill-creator skill to test the target skill
  5. Review outputs and diffs
  6. Improve the skill description or body
  7. Re-run targeted evals, then the full suite
  8. Repeat until regressions are stable and output quality is consistently better

2. Eval Cases

Store evals in skills/<skill-name>/evals/evals.json:

{
  "skill_name": "my-skill",
  "evals": [
    {
      "id": "happy-path-1",
      "prompt": "A realistic prompt a real user would type, with concrete detail",
      "expected_output": "Short description of what a successful result should do",
      "files": []
    }
  ]
}

Write prompts that look like real user requests, not abstract test descriptions. Include a mix of happy-path, regression, output-format, edge, negative, and paraphrased cases.

3. Fixtures

If the skill operates on files, place sample inputs in skills/<skill-name>/evals/files/ and reference those files from the eval entries. Use small, representative fixtures.

4. With/Without Skill Comparisons

Ask for a side-by-side comparison in the Copilot CLI session:

Use the skill-creator skill to compare outputs for skills/my-skill with and without the skill enabled.
Compare on: correctness, structure adherence, completeness, and output verbosity.
Only fix the smallest part of the skill that explains the largest failure cluster. Repeat until all evals pass.

Compare correctness, completeness, structure, latency, verbosity, and formatting stability.

5. Reviewing Outputs and Diffs

Review changed outputs and fixture diffs. Classify each change as improvement, acceptable variation, regression, or unclear. Avoid making fixtures so strict that harmless wording improvements fail the test.

6. Iterative Improvement

When an eval fails, update the smallest possible part of the skill and re-run the affected cases first. Avoid large rewrites unless the skill is fundamentally mis-scoped.

7. Regression-First Loop

  1. Run the full eval suite and save the baseline
  2. Pick the largest failure cluster
  3. Make one small edit to the skill
  4. Re-run only affected evals and regressions
  5. Review output diffs
  6. Run the full suite again
  7. Keep or revert the change
  8. Repeat until stable

8. Body vs. Trigger Testing

  • Body testing: Use evals/evals.json to verify that once the skill is active, it behaves correctly.
  • Trigger testing: Create skills/<skill-name>/evals/trigger-eval.json to test whether the skill activates for the right queries.

9. Description Optimization

Ask Copilot to optimize the description against your trigger eval set:

Use the skill-creator skill to optimize the description for skills/my-skill
using skills/my-skill/evals/trigger-eval.json.
Keep minimalist (≤ 1024 chars); structured domain nouns/verbs preferred over explicit keyword lists.
Include a NOT for: boundary clause. Report precision and recall per candidate. Repeat until all evals pass.

A good description uses structured domain nouns and a NOT for: boundary. An explicit keyword list is not required.

10. What “good enough” looks like

  • All smoke tests and known regressions passing
  • No critical format failures
  • Positive or neutral delta versus baseline
  • Stable behavior across paraphrases
  • Strong trigger accuracy on both train and validation queries

11. Practical Tips

  • Use realistic prompts with concrete nouns, filenames, and intent
  • Prefer several focused evals over one large vague eval
  • Keep fixtures small and scenario-specific
  • Update fixtures only after confirming the new behavior is actually better
  • Do not optimize solely for exact string matches
  • If a change improves one eval but harms unrelated ones, revert it

12. Minimal CLI Loop

VS Code Copilot Chat (or gh copilot):
→ "Use the skill-creator skill to test my skill at skills/my-skill.
   Run all evals and report pass rate and baseline delta."
→ inspect results and diffs — classify each change as improvement, regression, or neutral
→ edit SKILL.md or fixtures (smallest change that fixes the largest failure cluster)
→ "Use the skill-creator skill to rerun the evals for skills/my-skill."
→ "Use the skill-creator skill to optimize the description for skills/my-skill
   using skills/my-skill/evals/trigger-eval.json.
   Keep ≤ 1024 chars; include a NOT for: boundary clause. Repeat until all evals pass."
→ repeat until stable

For more on evals, see Demystifying Evals for AI Agents and the skill-creator skill documentation.