From 11e673c24c1dc5233f4d5162788f18886d2ca168 Mon Sep 17 00:00:00 2001 From: unknown Date: Sun, 13 Sep 2026 13:55:44 +0400 Subject: [PATCH 1/4] feat(evals): checkpoint skill-driven authoring implementation --- .gitignore | 8 + CONTEXT.md | 17 ++ .../tests/documented_check_contract.rs | 68 +++++- docs/adr/0011-skill-first-eval-authoring.md | 22 ++ .../0012-separate-outcomes-from-adherence.md | 7 + docs/adr/0013-skill-driven-eval-authoring.md | 15 ++ docs/qa/eval-authoring-verification.md | 79 +++++++ docs/requirements/eval-authoring-proposal.md | 91 ++++++++ docs/requirements/eval-authoring-skill-prd.md | 219 ++++++++++++++++++ .../eval-authoring-support-prd.md | 95 ++++++++ skills/eval-authoring/SKILL.md | 85 +++++++ .../examples/invoice-extraction/SKILL.md | 10 + .../invoice-extraction/evals/checks.toml | 22 ++ .../invoice-extraction/evals/judge-prompt.md | 10 + .../invoice-extraction/evals/prompts.csv | 4 + .../invoice-extraction/evals/review-notes.md | 11 + .../invoice-extraction/fixtures/basic.txt | 4 + .../fixtures/missing-number.txt | 3 + .../invoice-extraction/skill-project.toml | 10 + skills/eval-authoring/references/examples.md | 36 +++ skills/eval-authoring/references/handoff.md | 31 +++ skills/eval-authoring/references/semantics.md | 31 +++ tests/cli/eval_authoring_assets_test.rs | 197 ++++++++++++++++ tests/cli/mod.rs | 1 + .../eval-authoring/defective-total/SKILL.md | 8 + .../defective-total/ground-truth.md | 3 + .../eval-authoring/environment-variants.md | 9 + .../eval-authoring/existing-suite/SKILL.md | 8 + .../existing-suite/evals/checks.toml | 4 + .../existing-suite/evals/prompts.csv | 2 + .../existing-suite/evals/user-note.md | 3 + .../existing-suite/fixtures/release.md | 3 + .../existing-suite/skill-project.toml | 3 + .../eval-authoring/invoice-no-evals/SKILL.md | 8 + .../fixtures/ambiguous-date.txt | 4 + .../invoice-no-evals/fixtures/basic.txt | 4 + .../fixtures/missing-number.txt | 3 + .../fixtures/multiple-currencies.txt | 5 + .../eval-authoring/scripted-workflow.md | 12 + .../eval-authoring/unnecessary-step/SKILL.md | 8 + .../unnecessary-step/scripted-answers.md | 5 + .../eval-authoring/unsupported-order/SKILL.md | 8 + webdocs/evals-quality/setup.mdx | 21 +- 43 files changed, 1188 insertions(+), 9 deletions(-) create mode 100644 docs/adr/0011-skill-first-eval-authoring.md create mode 100644 docs/adr/0012-separate-outcomes-from-adherence.md create mode 100644 docs/adr/0013-skill-driven-eval-authoring.md create mode 100644 docs/qa/eval-authoring-verification.md create mode 100644 docs/requirements/eval-authoring-proposal.md create mode 100644 docs/requirements/eval-authoring-skill-prd.md create mode 100644 docs/requirements/eval-authoring-support-prd.md create mode 100644 skills/eval-authoring/SKILL.md create mode 100644 skills/eval-authoring/examples/invoice-extraction/SKILL.md create mode 100644 skills/eval-authoring/examples/invoice-extraction/evals/checks.toml create mode 100644 skills/eval-authoring/examples/invoice-extraction/evals/judge-prompt.md create mode 100644 skills/eval-authoring/examples/invoice-extraction/evals/prompts.csv create mode 100644 skills/eval-authoring/examples/invoice-extraction/evals/review-notes.md create mode 100644 skills/eval-authoring/examples/invoice-extraction/fixtures/basic.txt create mode 100644 skills/eval-authoring/examples/invoice-extraction/fixtures/missing-number.txt create mode 100644 skills/eval-authoring/examples/invoice-extraction/skill-project.toml create mode 100644 skills/eval-authoring/references/examples.md create mode 100644 skills/eval-authoring/references/handoff.md create mode 100644 skills/eval-authoring/references/semantics.md create mode 100644 tests/cli/eval_authoring_assets_test.rs create mode 100644 tests/fixtures/eval-authoring/defective-total/SKILL.md create mode 100644 tests/fixtures/eval-authoring/defective-total/ground-truth.md create mode 100644 tests/fixtures/eval-authoring/environment-variants.md create mode 100644 tests/fixtures/eval-authoring/existing-suite/SKILL.md create mode 100644 tests/fixtures/eval-authoring/existing-suite/evals/checks.toml create mode 100644 tests/fixtures/eval-authoring/existing-suite/evals/prompts.csv create mode 100644 tests/fixtures/eval-authoring/existing-suite/evals/user-note.md create mode 100644 tests/fixtures/eval-authoring/existing-suite/fixtures/release.md create mode 100644 tests/fixtures/eval-authoring/existing-suite/skill-project.toml create mode 100644 tests/fixtures/eval-authoring/invoice-no-evals/SKILL.md create mode 100644 tests/fixtures/eval-authoring/invoice-no-evals/fixtures/ambiguous-date.txt create mode 100644 tests/fixtures/eval-authoring/invoice-no-evals/fixtures/basic.txt create mode 100644 tests/fixtures/eval-authoring/invoice-no-evals/fixtures/missing-number.txt create mode 100644 tests/fixtures/eval-authoring/invoice-no-evals/fixtures/multiple-currencies.txt create mode 100644 tests/fixtures/eval-authoring/scripted-workflow.md create mode 100644 tests/fixtures/eval-authoring/unnecessary-step/SKILL.md create mode 100644 tests/fixtures/eval-authoring/unnecessary-step/scripted-answers.md create mode 100644 tests/fixtures/eval-authoring/unsupported-order/SKILL.md diff --git a/.gitignore b/.gitignore index 08b4df44..387004f5 100644 --- a/.gitignore +++ b/.gitignore @@ -218,6 +218,12 @@ tmp/ # Specific to this project skills/ +# This repository ships the maintained eval-authoring skill; keep other local +# skill workspaces ignored. +!skills/ +skills/* +!skills/eval-authoring/ +!skills/eval-authoring/** *.skill *.zip temp_skills/ @@ -338,6 +344,8 @@ repomix-output.xml coverage-after coverage-before skill-project.toml +!skills/eval-authoring/examples/invoice-extraction/skill-project.toml +!tests/fixtures/eval-authoring/existing-suite/skill-project.toml # Local opencode runtime state .opencode/opencode.json /webdocs/lib/generated/ diff --git a/CONTEXT.md b/CONTEXT.md index 3783eb48..82af3647 100644 --- a/CONTEXT.md +++ b/CONTEXT.md @@ -206,6 +206,23 @@ _Avoid_: Sweep, report (a rendering), summary (one run directory's, upstream), r A rendering of measurements already made — one run directory's summary, or one or more scorecards — as a table, JSON, or one self-contained HTML file with every asset embedded. Producing a report never runs an agent and never re-scores; it reads scorecard files, not run directories, so it can be made anywhere the files are. _Avoid_: Dashboard, page, export, scorecard (the measurement it renders) +### Eval authoring + +**Eval-authoring skill**: +A skill that guides an authoring agent through designing evaluation suites according to FastSkill's layouts and standards. It is distinct from the target skill being evaluated. + +**Authoring agent**: +The user-chosen agentic tool that helps design and edit an evaluation suite. It need not be the agent evaluated by that suite. + +**Outcome correctness**: +Whether the result of a skill-assisted task satisfies its intended acceptance criteria, independently of the prescribed method. + +**Skill adherence**: +Whether the agent follows the skill's prescribed method. It is distinct from outcome correctness and is mandatory only where the author identifies the method as a requirement. + +**Coverage outline**: +A proposed selection of behaviors and boundaries to evaluate for a skill, before detailed cases and graders are authored. It is not a Benchmark, which identifies an executable measurement definition. + ## Resolved decisions - **Historical `sync` is removed.** Modern targets read skills directly. Propagation has exactly two diff --git a/crates/fastskill-evals/tests/documented_check_contract.rs b/crates/fastskill-evals/tests/documented_check_contract.rs index 17a35ca1..b10f0bad 100644 --- a/crates/fastskill-evals/tests/documented_check_contract.rs +++ b/crates/fastskill-evals/tests/documented_check_contract.rs @@ -7,7 +7,9 @@ #![allow(clippy::expect_used, clippy::unwrap_used)] -use fastskill_evals::{run_checks, suite_passes, ChecksToml}; +use fastskill_evals::{ + effective_checks, run_checks, suite_passes, validate_case_checks, CheckDefinition, ChecksToml, +}; use std::path::Path; /// `max_command_count` is a legacy alias; results report the canonical @@ -175,3 +177,67 @@ required = true Path::new("/tmp") ))); } + +/// The setup guide's central authoring contract: `should_trigger` creates a +/// required per-case expectation even when checks.toml contributes nothing. +#[test] +fn should_trigger_generates_positive_and_negative_required_checks() { + for expected in [true, false] { + let checks = effective_checks(&[], "case-1", expected); + assert_eq!(checks.len(), 1); + assert!(matches!(checks[0], CheckDefinition::SkillInvoked { .. })); + if let CheckDefinition::SkillInvoked { + expected: actual, + required, + cases, + .. + } = &checks[0] + { + assert_eq!(*actual, expected); + assert!(*required); + assert_eq!(cases.as_deref(), Some(["case-1".to_string()].as_slice())); + } + } +} + +/// Exact case selectors control applicability, and an explicit trigger check +/// replaces the implicit one only on the selected case. A contradictory +/// replacement is rejected rather than silently changing CSV intent. +#[test] +fn case_scoping_and_explicit_trigger_precedence_match_the_docs() { + let parsed: ChecksToml = toml::from_str( + r#" +[[check]] +name = "skill_invoked" +expected = true +cases = ["positive"] + +[[check]] +name = "command_contains" +pattern = "invoice_number" +cases = ["positive"] +"#, + ) + .unwrap(); + + let positive = effective_checks(&parsed.checks, "positive", true); + assert_eq!( + positive.len(), + 2, + "explicit trigger plus scoped outcome check" + ); + let negative = effective_checks(&parsed.checks, "negative", false); + assert_eq!(negative.len(), 1, "other-case checks must be excluded"); + assert!(matches!( + negative[0], + CheckDefinition::SkillInvoked { + expected: false, + .. + } + )); + + assert!(validate_case_checks(&parsed.checks, "positive", true).is_ok()); + let error = validate_case_checks(&parsed.checks, "positive", false).unwrap_err(); + assert!(error.to_string().contains("EVAL_CHECKS_INVALID")); + assert!(error.to_string().contains("positive")); +} diff --git a/docs/adr/0011-skill-first-eval-authoring.md b/docs/adr/0011-skill-first-eval-authoring.md new file mode 100644 index 00000000..44d4842a --- /dev/null +++ b/docs/adr/0011-skill-first-eval-authoring.md @@ -0,0 +1,22 @@ +# Start assisted eval authoring from the skill + +Status: accepted. Date: 2026-09-12. + +## Context + +Designing meaningful eval cases requires human attention to intended outcomes, coverage, and grading criteria. The initial authoring experience could start from a skill that needs tests or from an observed session that should become a regression case. These solve different entry-point problems. + +## Decision + +The first assisted-authoring experience starts from an existing skill and helps its author design the skill's evaluation tests. It does not require session history or a known failure. + +Inspecting sessions to discover regression cases or improve skills is deferred for a separate discussion. It is not a prerequisite or committed deliverable of the initial change. + +This decision settles scope only. It does not select the authoring model, interaction style, suite representation, calibration method, acceptance workflow, or implementation ownership. + +## Consequences + +- Initial design and acceptance criteria must cover a skill with no prior usage sessions. +- Session import adapters and transcript reconstruction are outside the initial scope. +- The next decisions must establish how intended behavior is clarified and how generated cases and graders are validated. +- Future session-derived cases can be considered independently without delaying skill-first authoring. diff --git a/docs/adr/0012-separate-outcomes-from-adherence.md b/docs/adr/0012-separate-outcomes-from-adherence.md new file mode 100644 index 00000000..05d28bac --- /dev/null +++ b/docs/adr/0012-separate-outcomes-from-adherence.md @@ -0,0 +1,7 @@ +# Separate outcome correctness from skill adherence + +Status: accepted. Date: 2026-09-12. + +Generated evals must distinguish a useful, correct outcome from compliance with a skill's prescribed method. Measure these separately and make adherence mandatory only when the author identifies it as a requirement. Automatically treating every instruction as correctness would encode existing mistakes and could reward unnecessary tool use; treating outcomes alone as sufficient would miss genuinely required procedures. + +This decision does not choose the storage schema or change the current engine's implicit trigger checks. Implementation must reconcile those existing checks with the authoring distinction before claiming support. Pilot success establishes evidence about cases and graders, not that the skill passes the suite. diff --git a/docs/adr/0013-skill-driven-eval-authoring.md b/docs/adr/0013-skill-driven-eval-authoring.md new file mode 100644 index 00000000..e5e8b10f --- /dev/null +++ b/docs/adr/0013-skill-driven-eval-authoring.md @@ -0,0 +1,15 @@ +# Drive eval authoring through a skill and the CLI + +Status: accepted. Date: 2026-09-12. + +Users design evals in their chosen agentic tool with FastSkill CLI and an eval-authoring skill that explains its layouts and standards. The authoring skill guides inspection, coverage discussion, case construction, and iteration; FastSkill and its eval engine define and validate artifacts, execute evaluations, and preserve measurement evidence. This keeps conversational judgment in the user's agent and enforceable mechanics in code, avoiding a separate model integration and interview orchestrator inside FastSkill. + +Drafts use ordinary files and version control. A dedicated interview engine and draft-state service are deferred. Requirements that CI relies on must be represented in artifacts and checked by code; instructions in the authoring skill alone cannot establish verified status. + +Authoring-tool choice is independent of evaluation-runtime support. An agent able to follow the skill, edit files, and invoke the CLI can author a suite without having a FastSkill evaluation backend. Executing that agent as the subject of an eval still requires a supported backend and observable evidence. + +The authoring agent owns conversational authorization: it uses the user's existing authorization or asks about pilot scope before invoking the CLI. FastSkill validates explicit parameters and returns results or errors; it does not infer consent, remember conversational approvals, or conduct an interview. A spending ceiling is not a prerequisite of this change and must not be promised unless implemented. + +The skill checks CLI compatibility and follows authoritative format references and validation output. Exact supported versions and distribution location remain implementation details to investigate. Review covers the outline and then the suite, with individual questions for ambiguities. Calibration labels follow author-confirmed criteria or independently verifiable facts. + +The first delivery is the authoring skill with supporting documentation and validation corrections for existing contracts. Unsupported test intentions are reported explicitly; new grader types, schemas, or execution capabilities require separate proposals. The PRDs are drafts for external review, not a claim of implementation readiness or completed verification. diff --git a/docs/qa/eval-authoring-verification.md b/docs/qa/eval-authoring-verification.md new file mode 100644 index 00000000..d47fa6cf --- /dev/null +++ b/docs/qa/eval-authoring-verification.md @@ -0,0 +1,79 @@ +# Eval-authoring verification record + +Status: implementation evidence in progress. Updated: 2026-09-13. + +## Current evidence (supersedes historical status tables below) + +- Pinned FastSkill 0.9.230 built successfully. Six documented contract tests, four CLI asset tests, six documentation parity tests, and formatting passed before the latest example metadata assertion; that assertion still requires execution. +- Both authoring tools ran: Codex CLI 0.153.4 with `gpt-5.6-luna`, and Claude Code 2.1.269 with `claude-sonnet-4-5`. Generated suites validated against the pinned binary. Initial Claude attempts required trigger-scope, judge-path, schema, and template repairs. +- Claude continued the Codex-authored suite using files, repaired extra CSV columns and answer leakage, validated four cases/four checks, and recorded why changed prompts require new target evidence. Handoff cost reported by Claude: USD 0.2934362. Raw local record: `work/claude-handoff.json` beside this checkout. +- A real Claude Sonnet 4.5 defect pilot consulted the target, returned `15.00`, and failed the independent `17.00` expectation with exit code zero and no execution error. Changing only the target to include the final item produced a passing rerun with the same criterion. Both runs and the failed workspace are retained locally under `work/pilot-defect-claude-runs/2026-09-12T20-23-38Z/claude` and `work/pilot-corrected-claude-runs/2026-09-12T20-24-46Z/claude`. +- The conversion pilot passed the numeric outcome check, failed mandatory consultation, and failed the scratch-file check without gating because it was advisory. This demonstrates distinct outcome/adherence results, not a fully passing conversion suite. Evidence: `work/pilot-conversion-claude-runs/2026-09-12T20-23-45Z/claude`. +- Runtime prerequisite checks exposed `EVAL_ISOLATION_NO_SKILL` for missing metadata and `RUNTIME_UNKNOWN_ID` for codex in the current registry (available: aikit, claude). Standalone authoring-tool availability does not establish FastSkill target-runtime availability. The shipped example now includes metadata required for isolation. +- Strict clippy failed on a recursion-depth diagnostic in unchanged `crates/fastskill-cli/src/main.rs`. A new test panic lint was repaired. Full workspace regression was restarted with two compilation jobs after the unconstrained build impaired responsiveness; completion remains pending. +- No production Rust files changed. Numeric production line/branch coverage is not claimed; changed tests require execution, and skill/fixture behavior requires the inventory below. + +Final local checks on 2026-09-13: the latest four asset tests passed, including the metadata assertion; formatting passed; source-size checking passed after removing CR characters from the script input stream (the checkout uses CRLF). The workspace command stopped at the CLI test binary with 457 passed and seven failed. Two isolation tests failed with missing project configuration during parallel execution and both passed when rerun with `--test-threads=1`. Two other failures report Windows privilege error 1314 during symlink operations; remaining add/update failures concern project state and require separate diagnosis. These are failures in unchanged files, not established regressions from this change, but the full gate is not green. + +Rescoring the corrected pilot's saved artifacts passed without another model call. No relevant native-judge API key was present in this process, and the example endpoint/model remain placeholders; live calibration is unverified. + +`cargo clippy -p fastskill-evals --tests --locked -- -D warnings` passed after the test lint repair. `git diff --check` passed. These focused results do not replace the failing workspace-wide gate. + +Remaining release gaps: full workspace regression, strict clippy resolution/baseline evidence, remaining fixture and prerequisite branches, judge calibration/credential availability, and final per-file evidence inventory. No PR has been created. Historical environment blockers below are retained as attempt history, not current claims. + +This record keeps deterministic checks, live authoring exercises, and release gaps distinct. It must not be read as a release attestation while any mandatory row is incomplete. + +## Compatibility and confirmed validation behavior + +- Intended supported baseline: FastSkill 0.9.230 at `dd983e89e5fee977325b77ae386b879e8310ed26`, with `aikit-evals` pinned to `eb50f2c5509d4345cab65af0f7faed012ab0e557`. +- Local installed binary: FastSkill 0.9.208. It parsed the shipped invoice example with `fastskill eval validate --json`, reporting three cases and one deterministic check, but it predates the declared baseline and therefore does not establish 0.9.230 behavioral compatibility. +- Confirmed source contracts at the pin: per-case implicit trigger checks from `should_trigger`; exact `cases` selectors; explicit applicable `skill_invoked` precedence; contradiction rejection; required/advisory behavior; path-reference consultation proxy. +- No new validation defect requiring production-source changes was identified. The existing engine and CLI already enforce the material contracts; this change repairs maintained guidance and adds contract tests. + +## Requirement-to-test mapping + +| Requirements | Evidence | Status | +| --- | --- | --- | +| WF1, A1, AT1, ST4, ST6 | `invoice-no-evals`; skill compatibility/help instructions; package validator | deterministic assets complete; live workflow incomplete | +| WF2, A2, AT3 | scripted workflow and unnecessary-step author decisions | fixture assertions complete; live workflow incomplete | +| WF3-WF4, A3-A5, AT4-AT5, ST1-ST5 | shipped example; semantics/examples references; documented contract tests | authored; Rust execution blocked by environment | +| WF5, A6-A7, AT6 | judge contrast examples, real `eval validate`, invalid contradiction test | parse evidence present; executed judge calibration unavailable | +| WF6, A8, AT9-AT10, ST8 | explicit pilot-scope instructions and scripted authorized/unauthorized branches | instructions/fixtures present; live exercise incomplete | +| WF7, A9, AT7-AT8, ST7 | defective target, environment variants, evidence categories | fixtures present; live target run incomplete | +| WF8, A10, AT11 | existing suite/user note, rescore/rejudge/rerun guidance, handoff template | fixture assertion authored; live agent switch incomplete | +| ST2-ST3 | shipped suite validation and `documented_check_contract.rs` trigger/selector/contradiction tests | local 0.9.208 parse passed; pinned Rust test not yet executed | + +## Changed-file coverage classes + +- `skills/eval-authoring/SKILL.md`: every normative workflow section maps to WF1-WF8 and AT1-AT11 above; structural validation passed with `quick_validate.py`. +- `skills/eval-authoring/references/*.md`: linked from the entrypoint and exercised by the asset integration test; command and semantic claims map to the maintained docs and pinned contract tests. +- `skills/eval-authoring/examples/invoice-extraction/**`: every config/prompt/fixture/note is parsed or asserted by `eval_authoring_assets_test`; the suite also passed file validation under local FastSkill 0.9.208, with the compatibility limitation above. +- `tests/fixtures/eval-authoring/**`: the asset integration test checks discovery and independent ground truth; the existing suite itself is parseable; live behavioral use remains mandatory. +- `crates/fastskill-evals/tests/documented_check_contract.rs`: focused assertions cover positive/negative implicit checks, exact selectors, explicit precedence, contradiction, required/advisory behavior, and trace-only matching. +- `tests/cli/eval_authoring_assets_test.rs` and `tests/cli/mod.rs`: Cargo discovery is wired, meaningful assertions and a controlled contradiction failure are present; execution is currently blocked before compilation. +- `webdocs/evals-quality/setup.mdx`: repaired claims map to the documented contract test and shipped example; link/parity regressions remain to run. +- `.gitignore`: controlled check is that only `skills/eval-authoring/**` is unignored while another `skills/*` path remains ignored. +- `CONTEXT.md`, ADR-0011 through ADR-0013, and the three authoring requirement documents: accepted design handoff transferred unchanged; their requirements map through this record. + +No production executable Rust file is modified, so the 90% line and 80% branch thresholds do not apply to production source in this change. Test-source execution and meaningful assertion evidence are still mandatory. + +## Commands and observed results + +| Command or exercise | Result | +| --- | --- | +| `quick_validate.py skills/eval-authoring` | passed (`Skill is valid!`) | +| `cargo fmt --all -- --check` | initially reported formatting-only diffs; `cargo fmt --all` applied them | +| local FastSkill 0.9.208 `eval validate --json` in shipped example | passed; three cases, one explicit check; version is outside declared compatibility | +| pinned Cargo test invocation | blocked before compilation: sandboxed Windows Cargo cannot read the installed Git system config; local path patches bypassed Git dependencies, then crates.io TLS failed with `SEC_E_NO_CREDENTIALS`; offline cache lacks `ammonia` | +| Codex CLI 0.153.4, `gpt-5.6-luna`, isolated authoring exercise | attempted twice; first rejected incompatible CLI flags, second stopped before model invocation because the sandbox account has no resolvable Codex home | +| second authoring tool | unavailable: `pi` and `claude` not installed; Cursor 3.17.8 exposes the editor launcher, not a headless authoring-agent/model interface | + +## Release gaps + +- Execute the focused tests, full relevant regression suite, command/document parity checks, source-size check, clippy, and coverage instrumentation in a Rust environment with usable registry access. +- Complete WF1-WF8, AT1-AT11, and ST1-ST8 behavioral exercises with at least two distinct capable authoring tools, including preservation and handoff. +- Verify actual selected model IDs and authentication for both tools; record all attempts and sanitized artifacts. +- Exercise a real target pilot and, where supported, judge contrasts. Missing standalone calibration remains an explicit product limitation. +- Produce final changed-file coverage results. Branch coverage is unavailable until instrumentation runs. + +Until these gaps close or the owner explicitly waives a gate, no pull request may be created. diff --git a/docs/requirements/eval-authoring-proposal.md b/docs/requirements/eval-authoring-proposal.md new file mode 100644 index 00000000..cc25abb0 --- /dev/null +++ b/docs/requirements/eval-authoring-proposal.md @@ -0,0 +1,91 @@ +# Assisted eval design for skills + +Status: working discussion record, superseded for delivery scope by the [eval-authoring skill PRD](eval-authoring-skill-prd.md) and [support PRD](eval-authoring-support-prd.md). Product boundaries are agreed; PRDs are drafts for external review. See ADR-0011 through ADR-0013. + +## Agreed authoring principles + +- Immediately propose a short coverage outline after inspecting the skill. Ask only questions whose answers materially change the tests; avoid a fixed questionnaire. +- Measure outcome correctness separately from skill adherence. Adherence becomes mandatory only when the author identifies it as a requirement; do not silently promote every skill instruction to a correctness condition. +- Aim to deliver a piloted suite. If execution prerequisites are missing, preserve an explicitly unverified draft rather than claiming completion of verification. +- Exercise graders against acceptable and unacceptable examples. The skill passing its own tests is not a condition of successful authoring. + +The authoring conversation runs in the user's chosen agentic tool, guided by an eval-authoring skill. FastSkill CLI and the evaluation engine own supported formats, validation, execution, and evidence. Drafts are ordinary files under version control; a dedicated interview engine and draft-state service are deferred. + +Additional agreed decisions: + +- Review the coverage outline first, then the suite as a whole. Request individual clarification for ambiguous requirements or questionable grading examples rather than requiring approval of every case. +- The authoring agent may propose graders and calibration examples. Labels must derive from author-confirmed criteria or independently verifiable facts; synthetic examples are evidence, not proof of correctness. +- The authoring skill checks CLI version compatibility and consults authoritative format references and validation results. Unsupported versions receive an explicit compatibility message; exact supported versions are an implementation deliverable. +- The user's authoring agent handles all conversational authorization. It uses existing authorization or asks once about the proposed pilot when necessary, then invokes explicit CLI parameters. FastSkill neither interprets conversational consent nor stores conversational approvals. +- A monetary ceiling is not required for this authoring change. An agreed set of cases, runtimes, and trial counts bounds the pilot's scope, not its actual monetary cost. Only controls implemented by the CLI may be described as enforced. + +V1 builds the authoring skill first using existing suite/CLI capabilities, with documentation and existing-contract validation fixes. Unsupported intentions are reported rather than weakened. Engine expansion requires separate proposals. Distribution location, exact compatibility versions, and test mechanics require implementation investigation, not a new conversational subsystem. + +## Problem + +Creating meaningful cases consumes more human attention than writing CSV or TOML. Authors must reconstruct the task, decide what a correct outcome means, remove incidental context, supply fixtures, and establish that the grader distinguishes success from failure. Existing execution and scorecard infrastructure does not solve this authoring problem. + +## Proposed outcome + +An author provides a target skill to their chosen agentic tool and uses FastSkill's eval-authoring skill to design its evaluation tests. That agent inspects the target, proposes coverage and cases, surfaces uncertain assumptions, and invokes CLI validation and pilots as appropriate. Accepted cases enter an ordinary version-controlled suite with provenance. Human attention is spent on intent and ambiguous criteria rather than file syntax. + +The initial scope is skill-to-suite authoring without requiring prior sessions or known failures. Session inspection for regression discovery and skill improvement is a separate future discussion. The skill's current instructions are evidence of intended behavior, not sufficient ground truth: a generated suite must not merely encode an existing mistake as the expected outcome. + +## Proposed workflow + +1. Select the skill to evaluate and inspect its instructions and relevant supporting files. +2. Propose a coverage plan spanning intended outcomes, triggering, non-triggering, boundaries, and failure handling where applicable. Identify missing intent before generating a large suite. +3. Propose expected outcomes and deterministic assertions where possible. Label inferred expectations and request clarification where correctness depends on unstated intent. +4. Show the proposed prompt, fixtures, assertions, evidence provenance, and unresolved assumptions together for review. +5. Pilot graders against acceptable and unacceptable example outcomes or deliberately constructed contrasts, labeled from author-confirmed criteria or independently verifiable facts. Do not treat a successful pilot as proof of validity. +6. Pilot execution on selected supported runtimes, reporting unavailable evidence separately from failure. Additional trials and ablation are a later expansion unless needed for the first use case. +7. Accept the candidate explicitly into the suite. Later executions and scorecards use the existing evaluation workflow. + +All steps describe proposed behavior. They are not existing commands or implementation commitments. + +## Initial scope recommendation + +Start with skill inspection, intent clarification, coverage planning, candidate generation, suite-level review with targeted clarifications, grader calibration where supported, and conversion to the existing suite format. Report expectations that current capabilities cannot faithfully express; preserve unverified work explicitly. + +Session import, session-derived regression discovery, and session-based skill improvement are deferred by the agreed scope. Also propose deferring automatic monitoring, unrestricted filesystem reconstruction, full MCP simulation, automatic skill rewriting, and autonomous promotion of generated cases until the initial authoring workflow is validated. + +## Candidate ADR topics + +Create numbered ADRs after the grilling session resolves the choices. Use the repository's context/decision/consequences structure and keep unresolved ADRs proposed. + +| Topic | Recommended starting position | Decision to resolve | +| --- | --- | --- | +| Authoring entry point | Agreed: start from a skill and design its eval tests | Recorded in ADR-0011; session inspection deferred. | +| Authoring boundary | Agreed: user's agent plus authoring skill; CLI owns enforceable mechanics | ADR-0013; exact distribution and compatibility versions require investigation. | +| Candidate lifecycle | Ordinary files and version control; outline then suite-level review | Verification artifact details depend on the remaining engine-scope decision. | +| Case representation | Compile into existing CSV/check/judge formats, retain authoring metadata separately | Are richer fixtures or multi-turn cases essential enough to require schema changes now? | +| Ground truth and calibration | Agreed: author-confirmed criteria or independently verifiable facts establish labels | Preserve unverified work where prerequisites are missing; calibration implementation depends on engine scope. | +| Authoring inputs and data handling | User-selected agent owns the conversation and its provider setup | No new FastSkill authoring provider or conversational approval store. | +| Execution and ownership | Reusable calibration primitives upstream where appropriate; FastSkill owns CLI/project integration | Which parts belong in aikit-evals, and what isolation guarantees must pilots enforce? | + +## Success measures + +Measure human active minutes and clarification turns per accepted case, including subsequent repairs. Compare with manual FastSkill authoring and assisted plugin-eval authoring on comparable tasks. Do not optimize for the number of generated cases. + +Track whether accepted cases distinguish known failures from acceptable outcomes, how often reviewers reject or rewrite suggested criteria, and how often graders change verdict on unchanged evidence. Record supported-runtime coverage and infrastructure errors separately. Numerical acceptance targets require a baseline and are not invented here. + +## Questions to challenge in the grilling session + +- Where should review checkpoints fall after the coverage outline is proposed? +- Can the case be reproduced without the original private repository or external service state? +- Who decides what a correct answer is when the skill's instructions are ambiguous or wrong? +- Is a cross-agent case still testing the same outcome after tool-specific details are removed? +- How do we prevent the generator from making a grader that simply rewards its own proposed answer? +- Can the workflow reduce total human work after fixture repair, review, and grader maintenance are counted? +- What happens when a proposed case cannot be expressed faithfully in the existing engine? +- How are sensitive source details removed without destroying the behavior being tested? + +## Related material + +- [Skill-first eval authoring ADR](../adr/0011-skill-first-eval-authoring.md). +- [Outcome correctness and adherence ADR](../adr/0012-separate-outcomes-from-adherence.md). +- [Skill-driven authoring ADR](../adr/0013-skill-driven-eval-authoring.md). +- [Command taxonomy ADR](../adr/0010-command-taxonomy.md): any new authoring actions must use the existing explicit namespace model. +- [Eval measurement integrity requirements](eval-measurement-integrity.md). +- [Eval judging requirements](eval-judge.md). +- [Eval scorecard reporting requirements](eval-scorecard-report.md). diff --git a/docs/requirements/eval-authoring-skill-prd.md b/docs/requirements/eval-authoring-skill-prd.md new file mode 100644 index 00000000..9f5bdbf2 --- /dev/null +++ b/docs/requirements/eval-authoring-skill-prd.md @@ -0,0 +1,219 @@ +# Skill-driven eval authoring + +Status: draft for external review. Version: 0.2. Date: 2026-09-12. + +Product and architecture direction is agreed in ADR-0011 through ADR-0013. This document specifies proposed delivery requirements; it does not claim implementation or validation. It supersedes the working authoring proposal for this delivery. + +## Problem and intended outcome + +FastSkill can execute, judge, rescore, and report evaluation suites across supported agent runtimes. Creating those suites still requires substantial human attention: selecting meaningful scenarios, defining correctness, learning file conventions, designing graders, and interpreting validation and pilot failures. + +Provide an eval-authoring skill that works inside the user's chosen agentic tool. It should guide the agent from an existing target skill to a reviewable suite in FastSkill's supported formats, with pilot evidence where execution is possible. FastSkill remains a CLI and evaluation toolchain; it does not become a conversational agent. + +The primary outcome is reduced human active time per useful case, including review and repair. The number of generated cases is not a success metric by itself. + +## Users and initial scope + +The primary user maintains a skill and needs to design its first evaluation suite. No session history or known failure is required. The user's authoring agent must be able to read the authoring skill, inspect relevant files, edit suite files, and invoke FastSkill CLI. Authoring support does not imply that the same agent is supported as an evaluation target. + +V1 delivers the authoring skill, focused references and examples, and supporting documentation/validation corrections described in the companion PRD. It uses existing suite and CLI capabilities. If an intended check cannot be represented faithfully, the workflow records the limitation rather than silently weakening the expectation. + +Out of scope: a FastSkill interview engine, an authoring model API integration, a draft-state service, new case schemas or grader types, automated session import, skill optimization, native ablation, MCP simulation, new sandbox guarantees, and monetary spending caps. Necessary engine extensions discovered during development require separate proposals. + +## Before and target state + +| Area | Before | Target | +| --- | --- | --- | +| Starting point | Author learns formats and constructs cases | Agent inspects a target skill and proposes coverage | +| Human contribution | Scenario construction, syntax, commands, debugging | Intended behavior, consequential ambiguities, review | +| Authoring environment | No dedicated supported workflow identified | User's chosen agent plus FastSkill authoring skill | +| Artifacts | Existing CSV/TOML, prompts, fixtures, run evidence | Same execution formats, with concise human-readable review notes | +| Validation and execution | CLI capabilities available separately | Agent invokes those capabilities and explains evidence | +| Unsupported expectations | Easy to approximate accidentally | Explicit limitation tied to the affected behavior | + +Claude plugin evals already provide assisted suite generation and pilots. This delivery aims to narrow that authoring gap while preserving choice of authoring agent and access to FastSkill's existing cross-runtime measurements and historical scorecards. It does not claim parity in grader expressiveness, plugin integration, mocks, ablation, or execution isolation. + +## Responsibility boundary + +| Responsibility | Owner | +| --- | --- | +| Conversation, clarification, user review, execution authorization | User's authoring agent, guided by the authoring skill | +| Layout guidance, workflow instructions, examples | Eval-authoring skill and maintained references | +| Supported formats, parsing, deterministic validation | FastSkill CLI and relevant existing libraries | +| Trial execution, configured judging, scoring, evidence | Existing eval toolchain | +| Definition of intended outcome where judgment is needed | Author | + +The authoring skill cannot establish machine-enforced guarantees merely by instructing the agent to follow a procedure. No prose note is a CI attestation. Existing agent authentication is not proof that separately configured eval judges are available. + +## Reference user workflow for implementation and testing + +This workflow is a normative acceptance reference for the authoring skill and CLI integration. Implementers must create purpose-built target skills and fixtures to exercise it; testing only the authoring instructions' text or isolated CLI commands is insufficient. The example conversations illustrate intent, not exact wording that tests must match. + +The user stays in their chosen agentic tool. All conversation below is conducted by that agent. FastSkill only receives explicit CLI invocations and returns results. + +| Step | User and authoring-agent interaction | Implementation and test evidence | +| --- | --- | --- | +| WF1: Start | User: "Help me design eval tests for my invoice-extraction skill." Agent loads the eval-authoring skill, inspects the target and resources, and checks CLI compatibility. | Loaded skill/version, relevant inspected inputs, actual version/help output; no target suite initially exists. | +| WF2: Outline | Agent proposes ordinary invoices, missing fields, ambiguous dates, multiple currencies, and unrelated requests. It asks: "When the invoice number is missing, should extraction return null or fail?" User confirms null and reviews coverage. | Coverage outline precedes detailed suite generation; consequential answer is retained and no fixed questionnaire is required. | +| WF3: Generate | Agent writes supported prompts, checks, judge rubrics where needed, and fixtures. It distinguishes extraction correctness from triggering or required procedure. | Real generated files parse; missing-number criterion reflects null; trigger settings match engine semantics; unsupported expectations are explicit. | +| WF4: Review | Agent presents the suite together: each case's intent, expected result, and check method. It asks separately only about remaining ambiguities. | Suite-level review and targeted questions; no mandatory approval loop for every row. | +| WF5: Validate and challenge | Agent invokes CLI validation, repairs malformed files, and challenges graders with acceptable and unacceptable examples. | Actual validation output; labels tied to confirmed criteria; executed calibration distinguished from qualitative review or unavailable tooling. | +| WF6: Pilot authorization | If not already authorized, agent proposes "six cases, one trial each, on the selected runtime" and explains that this consumes model usage. The user can change scope. | Agent, not CLI, asks when needed. Prior authorization is respected. Invoked cases/runtimes/trials match the agreed scope; no unsupported cost guarantee. | +| WF7: Execute and explain | Agent invokes FastSkill. It explains which cases passed, which failed, and which could not be measured, linking the evidence. | Real run artifacts and results. The seeded ambiguous-date failure is identified without treating missing evidence as passing. | +| WF8: Refine and hand over | User: "Make the date criterion clearer and rerun the affected case." Agent updates the criterion and performs an appropriate supported rescore or rerun. It hands over files, evidence, and unresolved limits. | Unrelated user content is preserved; a new target run occurs only when needed; no criterion is weakened merely to obtain a pass; another authoring agent can continue from files. | + +Expected end state: editable suite and fixtures; review notes; actual validation/pilot artifacts; explicit target failures and unsupported expectations; and clear identification of anything still unverified. Target failure is compatible with successful test authoring. + +Failure branches must include missing CLI/runtime/judge prerequisites, incompatible CLI version, malformed generated configuration, unsupported checks, missing calibration capability, and a user changing the pilot scope. Preserve useful work in each branch and report the precise limitation. + +## Functional requirements + +### A1 — Inspect before asking + +The authoring agent identifies the selected target skill, inspects its instructions and relevant supporting files, and checks the installed CLI version. It consults compatible references and actual help/validation output rather than assuming a command or field exists. The workflow does not require a specific authoring vendor, subscription, or model. + +### A2 — Propose coverage, then clarify + +Present a concise coverage outline before generating a large suite. Consider useful outcomes, positive and negative triggering, boundaries, and failure handling where relevant. Ask only questions whose answers materially change tests. Review the outline before proceeding to the detailed suite; avoid a fixed questionnaire and arbitrary case-count quotas. + +### A3 — Separate outcomes and adherence + +Each proposed expectation identifies whether it measures outcome correctness or skill adherence. A prescribed method is mandatory only when the author identifies it as a requirement. The agent must surface a conflict between a skill instruction and the intended outcome instead of silently resolving it. + +The reviewed engine generates trigger checks from `should_trigger`. V1 must explain and respect that behavior, including explicit-check overrides and contradiction validation. It must not label a case outcome-only if the emitted configuration still has a mandatory adherence gate. If an intended scoring policy cannot be represented, identify it as unsupported; do not alter engine semantics under this PRD. + +### A4 — Write supported, reviewable files + +Generate ordinary suite files, judge prompts, and fixtures using documented supported formats. Preserve existing user files and show material changes when editing an existing suite. Retain concise review notes identifying coverage, requirement origins, assumptions, and unsupported expectations. These notes are ordinary documentation, not a new suite schema or managed lifecycle. + +### A5 — Review the suite together + +Show the proposed cases and criteria as a coherent suite. Request individual clarification for ambiguities or questionable examples, rather than approval of every row. Review acceptance does not imply that the target skill passed or that all graders were validated. + +### A6 — Challenge graders with contrasting outcomes + +Propose acceptable and unacceptable example outcomes, with labels grounded in author-confirmed criteria or independently verifiable facts. Record the reason for each label. Synthetic examples and a second model's agreement are not independent ground truth. + +Exercise actual graders against contrasting evidence where existing supported interfaces permit it. Where they do not, distinguish qualitative review from executed calibration and preserve the missing verification explicitly. Never fabricate an official run artifact to pretend a grader was exercised. Adding a standalone calibration command is outside v1 and may be proposed separately. + +### A7 — Validate and repair + +Invoke existing config validation and runtime observability checks appropriate to the selected targets. Fix authored formatting/reference mistakes and repeat the relevant check. Separate invalid configuration, missing prerequisites, unobservable measurements, and actual target failures. Validation success alone is not evidence of behavioral correctness. + +### A8 — Let the authoring agent authorize pilots + +The agent uses existing user authorization or asks about a concrete pilot scope when authorization is unclear. Scope identifies cases, evaluation runtimes, and trial counts. FastSkill receives explicit command parameters; it does not interpret the conversation, ask consent questions, or maintain an approval store. + +Pilot scope bounds work, not monetary cost. Do not promise a budget cap unsupported by the CLI. Missing evaluation-target or judge credentials must be reported without treating the authoring agent's credentials as interchangeable. + +### A9 — Preserve honest completion evidence + +Aim for a piloted suite. Report independently: author review, configuration validation, execution attempted/completed, grading/calibration evidence, target verdicts, and unresolved limitations. A target failing a useful test is a valid authoring result. Missing prerequisites leave an unverified draft; they must not be represented as passing or fully verified work. + +Link actual run artifacts where available. Human-readable completion notes are not consumed as trusted verification by CI. Existing machine-readable artifacts retain their current meanings. + +### A10 — Support continuation through files + +Another capable authoring agent can continue from the suite and review notes without reconstructing a proprietary conversation state. The workflow must not require a hosted authoring service, session database, or FastSkill-owned interview state. + +## Deliverables + +- An installable eval-authoring skill with concise workflow instructions and focused supporting references. +- Examples covering positive/negative triggering, outcome judging, and an unsupported expectation handled explicitly. +- Instructions for validation, authorized pilot execution, result review, and handoff through ordinary files. +- A declared CLI compatibility range backed by checks from the companion PRD. +- Evidence from end-to-end authoring exercises and a record of unresolved limitations. +- Purpose-built test skills, fixture data, scripted user answers, and workflow tests covering WF1–WF8 and the acceptance scenarios below. +- A changed-file coverage inventory and reproducible validation evidence meeting the testing requirements below. + +## Acceptance scenarios + +| ID | Scenario | Required evidence | +| --- | --- | --- | +| AT1 | Target skill has no evals | Coverage outline, relevant clarification, generated suite, validation result | +| AT2 | User's agent is not an eval backend | Authoring succeeds; execution targets only supported runtimes | +| AT3 | Skill instruction conflicts with intended outcome | Conflict surfaced; resulting criteria reflect author's decision | +| AT4 | Positive and negative triggering cases | Expectations match actual current scoring semantics | +| AT5 | Required assertion is not supported | Limitation reported; no misleading weaker substitute | +| AT6 | Known acceptable and unacceptable examples | Labels explained; actual calibration distinguished from qualitative review | +| AT7 | CLI, target runtime, or judge prerequisite missing | Work retained with precise unmet prerequisite and unverified status | +| AT8 | Target skill fails a well-formed case | Failure retained as evidence; criteria not weakened merely to obtain a pass | +| AT9 | Pilot scope already authorized | No redundant permission interview; explicit scoped command invoked | +| AT10 | Pilot scope not authorized | Authoring agent asks; CLI remains non-conversational | +| AT11 | Author switches agent | Suite and notes are sufficient to resume without a proprietary state service | + +## Validation and success measures + +Run documented authoring exercises with at least two different capable authoring tools as a release check. Record versions, prerequisite setup, generated artifacts, validation results, human interventions, and paid runs. This is evidence of portability, not a universal compatibility claim. The same workflow and behavioral expectations must apply across tools; exact generated prose and case counts need not match. + +### Test skills and fixtures to create + +Create isolated, synthetic test assets in the chosen repository's test/fixture layout. They must not depend on private production data or change the developer's global skill installations. + +| Test asset | Purpose and required exercises | +| --- | --- | +| Invoice-extraction skill with no evals | Main WF1–WF8 flow; text invoice fixtures with known values, missing number, ambiguous date, multiple currencies, and an unrelated request. | +| Target skill with a deliberate behavioral defect | Ensure the generated test detects an incorrect outcome and the authoring agent retains the failure instead of changing expectations. The defect and expected outcome must be established independently of the generated grader. | +| Target skill with an unnecessary prescribed step | Exercise the distinction between useful outcome and adherence; scripted author answers settle whether the step is mandatory. | +| Target requiring an unsupported assertion | Verify honest limitation handling without invented commands, formats, or misleading substitute checks. | +| Existing suite with unrelated user edits | Verify preservation, targeted edits, rescoring/rerun selection, and handoff between agents. | +| Environment/configuration variants | Missing credentials or runtime, incompatible version, invalid references, and unavailable observations; use controlled stubs for deterministic failure tests where appropriate. | + +Keep expected labels, known defects, and scripted user answers outside the target's visible execution inputs where feasible. Separate test-owned ground truth from agent-generated outputs. Real workflows must use the actual authoring skill and CLI; stubs are appropriate for controlled failure branches, not as replacements for all end-to-end evidence. + +### Authorized live-test tools and models + +The project owner permits implementation testing with the following available-tool choices. These are permitted options, not a requirement to run every combination or a claim that each alias is installed or supported as an eval backend. + +| Tool | Permitted model/provider choice | Setup requirement | +| --- | --- | --- | +| pi | Models available through the team's LiteLLM endpoint | Discover configured endpoint/model aliases and use an existing approved token; request secure provisioning if absent. | +| Cursor | Available Grok models | Verify the installed tool's actual supported model identifier and authentication. | +| Codex | GPT Luna, as requested by the owner | Resolve the actual available identifier; do not guess an alias or silently substitute another model. | +| Claude | Sonnet 4, as requested by the owner | Verify the actual available identifier and credentials; report unavailability. | + +These tools can be authoring agents. Use them as evaluation targets only where the FastSkill backend supports them and can observe the selected checks. FastSkill's native judge endpoint/model configuration is separate from authoring-agent authentication and must satisfy the existing judge contract. + +Use small, recorded pilot batches within this authorized testing scope; no new blanket approval is needed merely to exercise an allowed combination. Request missing credentials or environment permissions when required, never print tokens, and never place tokens in fixtures, prompts, logs, PRDs, or committed artifacts. Record endpoint identity/model IDs without secret values. If fewer than two authoring tools can run, retain a blocked portability release check rather than claiming completion from one tool. + +### High coverage for every touched file + +Every added or modified file in the implementation, including skill instructions, supporting references, fixtures, scripts, configuration, and code, must have explicit test or verification coverage appropriate to its contents. Submit a changed-file inventory mapping each file to test IDs, results, and remaining gaps. A high repository-wide average is not a substitute for coverage of an individual changed file. + +For this PRD, high executable-code coverage means at least **90% line coverage per added/modified executable source file**, and **80% branch coverage where supported by the toolchain**, with stricter repository gates taking precedence. Cover all changed contract behaviors and critical failure paths explicitly; numeric coverage alone is insufficient. Test files themselves need discovery/execution evidence and meaningful assertions, not recursive tests of tests. Document instrumentation limitations; do not represent unmeasurable branch coverage as passing. Uncovered behavior or unmet thresholds remains a release gap unless explicitly waived by the owner. + +For non-executable files, do not claim a source-line percentage. Require: + +- Skill instructions: every normative behavior mapped to an executed workflow/acceptance scenario, including negative and recovery branches. +- References and Markdown: all command/config examples validated against the supported CLI, local links resolved, and material behavioral claims mapped to tests or verified code contracts. +- Fixtures/templates/configuration: every supplied asset parsed or exercised; invalid examples fail for the intended reason; expected labels and outcomes checked independently. +- Changed tests/build/CI configuration: demonstrate that intended tests run and that a controlled failure is detected by the relevant gate where applicable. + +Avoid tests that only search for wording in SKILL.md or mirror implementation logic. Observe emitted suite files, CLI invocations, exit/results semantics, preserved edits, and real failure detection. Separate deterministic CI checks from live-model exercises; record every live attempt and flaky result instead of selectively reporting successful retries. + +### Developer handoff and definition of done + +The implementation handoff must contain the installable authoring skill and references; test targets and fixtures; requirement-to-test and changed-file coverage mappings; reproducible commands; CLI/agent/model versions; sanitized live-run artifacts; and a clear list of limitations. Run relevant existing regression and command/document parity checks in addition to new coverage. + +Release is ready only when WF1–WF8, AT1–AT11, and the companion support criteria have evidence; the two-tool portability check and changed-file coverage gates pass; and known limitations are explicit. A deliberately failing target is expected evidence and does not fail authoring acceptance when correctly detected. This PRD update itself does not claim those implementation tests have been run. + +Measure active human minutes per accepted useful case, clarification count, correction/rejection rate, layout errors caught by validation, and time spent repairing graders. Compare comparable tasks with manual FastSkill authoring and the reviewed Claude-assisted workflow when available. A quantitative improvement target must follow a measured baseline; no percentage advantage is claimed now. + +## External review questions and implementation investigations + +- Does the acceptance matrix distinguish enough evidence to prevent an unverified suite appearing release-ready? +- Is the usefulness of v1 acceptable when full contrast calibration is unavailable for a grader? +- Determine skill distribution repository/path and supported versions using the maintained distribution conventions. No new installation mechanism is presumed. +- Confirm the smallest safe representation of review notes and examples, without creating a new engine schema. +- Identify current-interface limitations with reproducible cases; route required new capabilities to separate proposals. + +These are review and feasibility items. The user/agent/CLI responsibility boundary is settled. + +## Related decisions and requirements + +- [ADR-0011: skill-first authoring](../adr/0011-skill-first-eval-authoring.md) +- [ADR-0012: outcomes and adherence](../adr/0012-separate-outcomes-from-adherence.md) +- [ADR-0013: skill and CLI boundary](../adr/0013-skill-driven-eval-authoring.md) +- [Authoring support PRD](eval-authoring-support-prd.md) +- [Existing judge requirements](eval-judge.md) +- [Existing measurement-integrity requirements](eval-measurement-integrity.md) diff --git a/docs/requirements/eval-authoring-support-prd.md b/docs/requirements/eval-authoring-support-prd.md new file mode 100644 index 00000000..91ad527b --- /dev/null +++ b/docs/requirements/eval-authoring-support-prd.md @@ -0,0 +1,95 @@ +# Documentation and validation support for eval authoring + +Status: draft for external review. Version: 0.2. Date: 2026-09-12. + +This companion to the eval-authoring skill PRD covers the narrow tooling and documentation work needed to make its instructions reliable. It does not introduce an authoring orchestrator or expand evaluation expressiveness. + +## Problem + +A portable authoring skill must teach the behavior of the installed FastSkill release. Documentation drift can cause it to generate valid-looking suites that measure the wrong thing. The reviewed 0.9.230 checkout already contains material contradictions: its setup guide calls `should_trigger` inert, says deterministic checks necessarily apply globally, and describes no-check scoring as process-exit-only. The pinned engine generates per-case trigger checks and supports case selectors. + +The source baseline is FastSkill `dd983e89e5fee977325b77ae386b879e8310ed26` and its aikit pin `eb50f2c5509d4345cab65af0f7faed012ab0e557`. Revalidate against the implementation checkout before making fixes; these findings are not assertions about every version. + +## Objective and boundary + +Make supported authoring formats, commands, and semantics discoverable, correct, and testable. FastSkill remains responsible for validating explicit files and command parameters, producing evidence, and explaining errors. The user's authoring agent remains responsible for conversation, intent, and authorization. + +Use existing help, command specification, documentation, and validation surfaces. Add or repair validation only for existing format contracts. New graders, case schemas, calibration commands, budget controls, and runtime isolation changes need separate scope and are not implied here. + +## Requirements + +### S1 — Correct the maintained reference + +Audit and correct eval setup and command examples against pinned source behavior. Cover implicit trigger checks, explicit-check precedence, case selectors, contradiction handling, consultation evidence by backend, required/advisory checks, and scoring thresholds. Explain that path-reference evidence is a proxy for consultation, not proof that instructions were followed. + +Document the distinction between configuration validation, actual execution, judgment, and rescoring. Do not claim that a supported command validates evidence it does not inspect. + +### S2 — Keep ownership of rules clear + +The parser and engine define enforceable formats and scoring. Maintained references explain them; the authoring skill links to those references instead of copying large independent schemas. Examples must identify their supported release range. The skill must not assume features from a later CLI merely because its own instructions are newer. + +Determine whether current version/help/spec surfaces provide enough compatibility information before proposing additions. Do not introduce a new discovery API by default. + +### S3 — Provide actionable existing-contract validation + +Inventory existing validation behavior before implementing anything. Where a currently supported field has inadequate validation, propose a narrow correction with a reproducer. Errors should identify the affected file, case/check/judge where available, and the violated rule. Preserve established CLI and machine-readable output contracts unless a compatibility change is separately reviewed. + +Do not convert missing runtime evidence into a passing check, or missing measurements into zero scores. Do not add conversational prompting or approval persistence. + +### S4 — Ship executable examples + +Maintain small fixtures exercising positive/negative triggering, case-scoped checks, optional adherence where representable, a configured outcome judge, and an unsupported expectation described as such. Keep secrets out of examples and use explicit placeholders for separately supplied endpoint credentials. + +Validate syntax and deterministic behavior without paid calls. Any live agent or judge exercise is a separate, explicitly scoped validation activity. Do not substitute a model-generated success narrative for command output. + +### S5 — Connect authoring guidance to real evidence + +Examples and the skill must show how to interpret config errors, backend observability exclusions, execution failures, and target-quality failures separately. Document which current interfaces can exercise graders on saved evidence and which cannot. If a standalone calibration path does not exist, report that gap rather than inventing a command or forging artifacts. + +CI decisions continue to rely on existing structured results and exit semantics. Authoring notes do not introduce a trusted `verified` flag or new gate. + +### S6 — Preserve execution boundaries + +Correct any guidance that confuses scratch-workspace skill isolation with OS-level confinement. Describe actual permissions and backend limitations of the supported version. The authoring agent obtains any needed user authorization and invokes explicit CLI arguments. This PRD neither adds a sandbox nor promises a monetary cap. + +## Acceptance criteria + +| ID | Criterion | +| --- | --- | +| ST1 | Maintained authoring references no longer contradict the supported release on trigger expectations, selectors, or no-check behavior | +| ST2 | All shipped suite examples parse under the declared supported versions; expected invalid examples fail for the intended reason | +| ST3 | Deterministic tests demonstrate positive/negative expectations and explicit-check contradiction behavior | +| ST4 | Authoring instructions use actual help/spec/validation interfaces and no invented CLI commands | +| ST5 | Documentation and examples distinguish outcome checks, adherence checks, and aggregate verdicts accurately | +| ST6 | Compatibility failures are explicit; silently assuming newer fields is not taught | +| ST7 | Missing prerequisites, absent observations, and quality failures are explained separately | +| ST8 | No conversational interview, approval state store, or new engine capability is introduced under this support work | + +## Delivery and ownership + +### Workflow and coverage obligations + +Use the primary PRD's [reference workflow](eval-authoring-skill-prd.md#reference-user-workflow-for-implementation-and-testing), test skills, live-tool choices, and [changed-file coverage requirements](eval-authoring-skill-prd.md#high-coverage-for-every-touched-file) as mandatory implementation and testing references. They apply equally to files changed under this support PRD and to any associated upstream fixes. + +Every changed file must appear in the handoff coverage inventory. Executable source files require at least 90% per-file line coverage and 80% branch coverage where instrumentation supports it, or stricter existing repository gates. Skill and documentation files require behavioral/example/link verification rather than artificial line percentages. Every supplied fixture must be exercised, and test/CI changes need execution evidence. Unmet thresholds and unavailable checks are visible release gaps, not silently skipped success. + +In addition to ST1–ST8, demonstrate that repaired guidance works during real skill-driven authoring: generate and validate suites from purpose-built target skills, preserve deliberately detected failures, and distinguish unsupported expectations from valid passing checks. Use controlled deterministic tests for parser/validator defects and scoped live exercises for agent behavior. The owner's permitted tools and models are recorded in the primary PRD; credentials must be provisioned securely and model identifiers verified before use. + +Implement reference/example corrections in FastSkill and the chosen skill distribution location. Changes to shared parsers or validation belong in the owning library when a confirmed existing-contract bug requires them; FastSkill should not duplicate upstream evaluation semantics. Any upstream change must be pinned and verified in the consuming checkout. + +Before development, identify the release compatibility matrix and enumerate confirmed validation defects. A new validation feature is not justified merely because it would be convenient for an agent. + +## Risks and review items + +- A prose reference can drift again: cover material behavioral claims with focused tests or existing parity checks where appropriate. +- Implicit trigger checks can conflict with author expectations of outcome-only scoring: demonstrate the actual configuration or mark the requested policy unsupported. +- Pilots may be expensive or unavailable: deterministic documentation checks must remain possible without live credentials. +- Broader evaluation gaps may emerge: document them separately rather than expanding the authoring release silently. + +## Related material + +- [Eval-authoring skill PRD](eval-authoring-skill-prd.md) +- [ADR-0013](../adr/0013-skill-driven-eval-authoring.md) +- [Eval setup reference](../../webdocs/evals-quality/setup.mdx) +- [Eval command reference](../../webdocs/cli-reference/eval-command.mdx) +- [Observability preflight](../../crates/fastskill-cli/src/commands/eval/observability.rs) diff --git a/skills/eval-authoring/SKILL.md b/skills/eval-authoring/SKILL.md new file mode 100644 index 00000000..9f80c42b --- /dev/null +++ b/skills/eval-authoring/SKILL.md @@ -0,0 +1,85 @@ +--- +name: eval-authoring +description: Design, validate, pilot, and hand off FastSkill evaluation suites for an existing skill. Use when creating or repairing prompts.csv, checks.toml, judge rubrics, fixtures, or review notes; do not use to add unsupported evaluator features. +--- + +# Author FastSkill evals + +Turn an existing target skill into a reviewable suite in the formats supported by the installed FastSkill CLI. The user chooses the authoring agent. FastSkill validates explicit files, runs selected evaluation targets, and returns evidence; it does not conduct the conversation or remember approval. + +## Start from evidence + +1. Locate the target `SKILL.md`, its project root, relevant references, and any existing eval files. Preserve unrelated content and local edits. +2. Run `fastskill --version`, `fastskill eval --help`, and `fastskill eval validate --help`. Compare the installed version with the compatibility note below; treat live help and validation as authoritative for command availability. +3. Read the maintained [Eval Setup](../../webdocs/evals-quality/setup.mdx) and [eval command](../../webdocs/cli-reference/eval-command.mdx) references from the matching checkout. Read [references/semantics.md](references/semantics.md) before choosing checks or judges. + +This skill is maintained against FastSkill `0.9.230` and the pinned `aikit-evals` revision in that release. On another version, inspect its matching references and validate every field; report the suite as compatibility-unverified if the relevant contracts cannot be confirmed. Never assume that newer skill guidance adds a CLI field. + +## Design before generating + +After inspection, propose a short coverage outline spanning useful outcomes, positive and negative triggering, boundaries, and failure handling where relevant. Ask only questions whose answers materially change a case or criterion. Retain the answer and review the outline before creating a large suite. + +For every expectation, label it in review notes as: + +- **outcome**: the result must satisfy the user's acceptance criterion; or +- **adherence**: the agent must consult or follow a prescribed method. + +Make adherence mandatory only when the author says the method is required. Surface conflicts between target instructions and intended outcomes. Do not silently choose one or weaken an unsupported expectation. + +## Author and review + +Use ordinary `skill-project.toml`, CSV, TOML, Markdown prompt, and fixture files. Prefer deterministic checks for evidence they actually observe and a judge only for outcome criteria that need semantic review. Keep expected labels and known defects outside inputs visible to the evaluated target where practical. + +Check CSV row widths against the header and inspect parsed expected values. Ask the target to solve the task without supplying the expected answer in its prompt; a requested output format should use placeholders rather than the ground-truth result. File validation alone does not establish these properties. + +For a mixed positive/negative suite, normally omit explicit `skill_invoked` checks and let each row's `should_trigger` value generate the correctly polarized check. Never add one global positive check: it applies to negative rows too and the supported validator rejects the contradiction. If explicit trigger checks are necessary, scope them to exact case IDs and keep their polarity equal to those rows. + +Resolve judge prompt paths relative to `checks.toml`, not the project root. A judge can grade only evidence rendered into its prompt. Put independently established expected values in extra CSV columns such as `expected`, render them with `{{case.expected}}`, and do not claim a judge can inspect an unchanged fixture merely because it exists in the workspace. + +In the supported schema, repeat singular `[[judge.criterion]]` tables and render the evaluated answer with `{{trial.final_answer}}`. Validate these names instead of guessing plural tables or generic output fields. + +Write concise `evals/review-notes.md` recording: + +- coverage and the origin of each expectation; +- outcome versus adherence classification; +- author-confirmed decisions and remaining assumptions; +- unsupported expectations and missing prerequisites; +- validation, calibration, pilot, and target-verdict evidence as separate statuses. + +Present the cases and criteria as one coherent suite. Ask targeted follow-ups only for unresolved ambiguities; do not require approval row by row. For concrete layouts and limitation wording, read [references/examples.md](references/examples.md). + +## Validate, challenge, and pilot + +Run `fastskill eval validate` first, then validate selected runtime observability with `--agent ` or `--all`. Repair authored syntax and references, rerunning the failing check. + +Before the default isolated pilot, ensure `skill-project.toml` declares `[metadata].id` for the target skill. File validation can succeed without this pilot prerequisite. + +Keep these categories distinct: + +- invalid configuration; +- missing CLI, runtime, endpoint, or credential prerequisite; +- measurement not observable on a backend; +- execution error; +- well-formed quality failure. + +Challenge each judge with at least one author-confirmed or independently verifiable acceptable outcome and one unacceptable outcome. Record the label reason. If the installed interfaces cannot execute a standalone contrast, call it qualitative review and explicitly leave calibration unverified; never forge run artifacts. `eval judge` can judge or rejudge saved run evidence, but it is not a general calibration command. + +Before a paid or model-backed pilot, use existing user authorization or propose an explicit scope naming cases, evaluation runtimes, and trial counts. If unclear, ask once. Do not promise a monetary cap. Authoring-agent credentials do not prove that evaluation runtimes or native judges are configured. + +Run only the authorized scope with explicit parameters, for example: + +```bash +fastskill eval run --agent codex --case invoice-basic --trials 1 --output-dir ./eval-runs +``` + +Explain actual artifacts and distinguish pass, fail, error, excluded/unobservable evidence, and missing judgment. A useful test that exposes a target defect is successful authoring evidence; do not change the criterion merely to obtain a pass. + +## Refine and hand off + +Use `eval score` when only deterministic checks changed and saved evidence remains suitable, `eval judge --rejudge` when only judge configuration changed and saved evidence remains suitable, and rerun the target when its prompt, fixture, target skill, runtime/model, or required evidence changed. Preserve unrelated edits. + +Hand over editable suite files, fixtures, review notes, exact commands, run artifact paths, observed versions/model IDs, and unresolved limitations. Another capable authoring agent must be able to continue from files without conversation state. Use [references/handoff.md](references/handoff.md) for the completion record. + +## Boundaries + +Do not invent checks, schemas, graders, calibration commands, interview services, approval stores, native ablation, mocks, sandbox guarantees, or cost ceilings. Skill isolation controls discovered skills, not OS-level filesystem or network confinement. When the engine cannot faithfully represent an intended assertion, keep it as an explicit unsupported expectation and propose any engine extension separately. diff --git a/skills/eval-authoring/examples/invoice-extraction/SKILL.md b/skills/eval-authoring/examples/invoice-extraction/SKILL.md new file mode 100644 index 00000000..0a4ad91b --- /dev/null +++ b/skills/eval-authoring/examples/invoice-extraction/SKILL.md @@ -0,0 +1,10 @@ +--- +name: invoice-extraction +description: Extract invoice fields from plain-text invoices into JSON when the user asks for invoice data. +metadata: + id: invoice-extraction +--- + +# Extract invoices + +Return JSON with `invoice_number`, `date`, `currency`, and `total`. Use JSON null for a missing invoice number. Do not activate for unrelated questions. diff --git a/skills/eval-authoring/examples/invoice-extraction/evals/checks.toml b/skills/eval-authoring/examples/invoice-extraction/evals/checks.toml new file mode 100644 index 00000000..3387ef46 --- /dev/null +++ b/skills/eval-authoring/examples/invoice-extraction/evals/checks.toml @@ -0,0 +1,22 @@ +[[check]] +name = "command_contains" +pattern = "normalize_currency" +required = false +cases = ["invoice-basic"] + +[[judge]] +name = "extraction-correctness" +cases = ["invoice-basic", "invoice-missing-number"] +prompt_file = "judge-prompt.md" +model = "replace-with-approved-judge-model" + +[[judge.criterion]] +name = "correct_fields" +kind = "bool" +description = "The JSON values agree with the invoice fixture, and a missing invoice number is JSON null." + +[judge_defaults] +base_url = "https://replace-with-approved-endpoint.example/v1" +api_key_env = "FASTSKILL_EXAMPLE_JUDGE_KEY" +temperature = 0.0 +max_tokens = 1024 diff --git a/skills/eval-authoring/examples/invoice-extraction/evals/judge-prompt.md b/skills/eval-authoring/examples/invoice-extraction/evals/judge-prompt.md new file mode 100644 index 00000000..0baa444a --- /dev/null +++ b/skills/eval-authoring/examples/invoice-extraction/evals/judge-prompt.md @@ -0,0 +1,10 @@ +Review the extracted invoice JSON against the task and recorded workspace evidence. + +Task: {{case.prompt}} +Answer: {{trial.final_answer}} +Expected facts: {{case.expected}} + +{{rubric}} + +Reply using this contract: +{{output_contract}} diff --git a/skills/eval-authoring/examples/invoice-extraction/evals/prompts.csv b/skills/eval-authoring/examples/invoice-extraction/evals/prompts.csv new file mode 100644 index 00000000..ad250a53 --- /dev/null +++ b/skills/eval-authoring/examples/invoice-extraction/evals/prompts.csv @@ -0,0 +1,4 @@ +id,prompt,should_trigger,tags,workspace_subdir,expected +invoice-basic,"Extract the invoice in fixtures/basic.txt as JSON.",true,"outcome,positive",,"{""invoice_number"":""INV-1042"",""date"":""2026-09-01"",""currency"":""USD"",""total"":123.45}" +invoice-missing-number,"Extract the invoice in fixtures/missing-number.txt as JSON. Use null for missing fields.",true,"outcome,boundary",,"{""invoice_number"":null,""date"":""2026-09-02"",""currency"":""EUR"",""total"":78.90}" +invoice-unrelated,"Explain the difference between a receipt and an invoice.",false,"trigger,negative",, diff --git a/skills/eval-authoring/examples/invoice-extraction/evals/review-notes.md b/skills/eval-authoring/examples/invoice-extraction/evals/review-notes.md new file mode 100644 index 00000000..16a0abe4 --- /dev/null +++ b/skills/eval-authoring/examples/invoice-extraction/evals/review-notes.md @@ -0,0 +1,11 @@ +# Invoice extraction eval review + +Compatibility: authored for FastSkill 0.9.230; validate against another installed release before use. + +Coverage: ordinary extraction, missing invoice number, and unrelated-request non-triggering. The author confirmed that a missing number must be JSON null. Test-owned expected facts are supplied through the `expected` CSV column and rendered into the judge prompt. Field correctness is an outcome expectation. Skill consultation/non-consultation is an adherence/selection expectation imposed by `should_trigger` semantics. Currency-normalization command text is advisory only. + +Calibration contrasts: JSON null is acceptable for the missing number; the string `"UNKNOWN"` is unacceptable. These labels are based on the confirmed criterion. They have not been executed against a judge in this example. + +Unsupported: ordered OCR-then-normalization calls cannot be asserted by the current deterministic check types. No weaker substitute is claimed. + +Status: example files are syntax-validated without contacting the placeholder judge endpoint. No runtime pilot or judge calibration is claimed. diff --git a/skills/eval-authoring/examples/invoice-extraction/fixtures/basic.txt b/skills/eval-authoring/examples/invoice-extraction/fixtures/basic.txt new file mode 100644 index 00000000..2f093a08 --- /dev/null +++ b/skills/eval-authoring/examples/invoice-extraction/fixtures/basic.txt @@ -0,0 +1,4 @@ +Invoice number: INV-1042 +Date: 2026-09-01 +Currency: USD +Total: 123.45 diff --git a/skills/eval-authoring/examples/invoice-extraction/fixtures/missing-number.txt b/skills/eval-authoring/examples/invoice-extraction/fixtures/missing-number.txt new file mode 100644 index 00000000..cb81df95 --- /dev/null +++ b/skills/eval-authoring/examples/invoice-extraction/fixtures/missing-number.txt @@ -0,0 +1,3 @@ +Date: 2026-09-02 +Currency: EUR +Total: 78.90 diff --git a/skills/eval-authoring/examples/invoice-extraction/skill-project.toml b/skills/eval-authoring/examples/invoice-extraction/skill-project.toml new file mode 100644 index 00000000..0e27dbdc --- /dev/null +++ b/skills/eval-authoring/examples/invoice-extraction/skill-project.toml @@ -0,0 +1,10 @@ +[metadata] +id = "invoice-extraction" +version = "1.0.0" + +[tool.fastskill.eval] +prompts = "evals/prompts.csv" +checks = "evals/checks.toml" +trials_per_case = 1 +pass_threshold = 1.0 +fail_on_missing_agent = true diff --git a/skills/eval-authoring/references/examples.md b/skills/eval-authoring/references/examples.md new file mode 100644 index 00000000..965478b5 --- /dev/null +++ b/skills/eval-authoring/references/examples.md @@ -0,0 +1,36 @@ +# Authoring examples + +The executable example is in [`../examples/invoice-extraction`](../examples/invoice-extraction). It is compatible with FastSkill 0.9.230. Run `fastskill eval validate` from that directory without credentials; select a runtime only when intentionally checking its availability and observability. + +## Review table + +| Case | Intent | Expected result | Measurement | +| --- | --- | --- | --- | +| `invoice-basic` | Ordinary invoice | Correct number, date, currency, and total | outcome judge; trigger required by `should_trigger=true` | +| `invoice-missing-number` | Missing field | `invoice_number` is null | outcome judge; trigger required | +| `invoice-unrelated` | Unrelated request | Skill is not consulted | adherence/selection through `should_trigger=false` | + +An optional procedure can be recorded without gating: + +```toml +[[check]] +name = "command_contains" +pattern = "normalize_currency" +required = false +cases = ["invoice-basic"] +``` + +Do not describe the case as outcome-only while `should_trigger=true`: the implicit consultation check remains required. If outcome-only scoring is intended and the current format cannot disable the implicit check without contradicting the CSV, record that policy as unsupported rather than changing `should_trigger` deceptively. + +## Contrasting judge outcomes + +Criterion: “The response represents a missing invoice number as JSON null.” + +- Acceptable: `{"invoice_number":null,"date":"2026-09-01"}`. Label reason: the confirmed missing-number policy is null. +- Unacceptable: `{"invoice_number":"UNKNOWN","date":"2026-09-01"}`. Label reason: the sentinel string violates the confirmed policy. + +These labels support review. They are executed calibration evidence only if passed through a supported real judge interface and the resulting artifact is retained. + +## Unsupported expectation + +“Verify that the agent called OCR before parsing and then called currency normalization, in that order.” The current checks can inspect substring presence and aggregate tool-call count, but do not provide a structured ordered-call assertion. Do not substitute two `command_contains` checks and claim order was verified. Keep the intended assertion in review notes as unsupported and propose the engine feature separately. diff --git a/skills/eval-authoring/references/handoff.md b/skills/eval-authoring/references/handoff.md new file mode 100644 index 00000000..4f054971 --- /dev/null +++ b/skills/eval-authoring/references/handoff.md @@ -0,0 +1,31 @@ +# Eval suite handoff + +Record each category independently. + +```markdown +## Coverage and decisions +- Target skill and version: +- Coverage outline reviewed: +- Author-confirmed decisions: +- Outcome expectations: +- Adherence expectations: +- Unsupported expectations: + +## Verification +- FastSkill version and compatibility status: +- Configuration validation command/result: +- Runtime observability command/result: +- Judge contrast labels and provenance: +- Executed calibration artifacts, or unverified reason: +- Authorized pilot scope: +- Pilot commands and run directories: +- Target passes/failures/errors/unmeasured evidence: + +## Continuation +- Files changed: +- Preserved unrelated edits: +- Rescore, rejudge, or rerun decision: +- Missing prerequisites and next action: +``` + +Do not collapse these fields into “verified.” A reviewed and parse-valid suite may still be unexecuted; an executed trial may lack a judgment; a target failure may demonstrate a useful test. diff --git a/skills/eval-authoring/references/semantics.md b/skills/eval-authoring/references/semantics.md new file mode 100644 index 00000000..55cf89b4 --- /dev/null +++ b/skills/eval-authoring/references/semantics.md @@ -0,0 +1,31 @@ +# FastSkill 0.9.230 eval semantics + +Use this reference while mapping expectations to the maintained Eval Setup page. It highlights decisions that are easy to get wrong; it is not a duplicate schema. + +## Triggering and case scope + +`should_trigger` is executable scoring input in this release. For each case, the engine creates a required `skill_invoked` expectation with the same polarity unless an explicit `skill_invoked` check applies to that case. An explicit applicable check replaces the implicit one. Validation rejects an explicit polarity that contradicts the case's `should_trigger` value. + +Every deterministic check accepts an optional `cases = ["exact-id", ...]` selector. It applies only to those exact case IDs. Scope mixed positive and negative behavior deliberately. + +In a mixed-polarity suite, omit explicit `skill_invoked` checks unless they add necessary specificity. A global `expected = true` check applies to the negative rows and contradicts them; comments do not create per-case polarity. When explicit checks are used, select exact cases and match each selected row's `should_trigger` value. + +`skill_invoked` recognizes a structured `Skill` invocation or, on supported decoders, a tool input that refers to the staged skill-document path. The path form is evidence of consultation, not proof that the agent read or followed the instructions. Measure outcome correctness separately. + +With no checks file—or an empty check list—the implicit `should_trigger` expectation still applies. Process exit alone is not the verdict. + +## Verdicts and evidence + +Required deterministic checks are all-or-nothing for a trial. `required = false` keeps a check observable without gating. Across trials, `pass_threshold` controls the aggregated case result. Judge criteria have their own normalized scores; a judge gates only when it declares `min_score`. + +Validation proves that configured files parse and that file-only contracts are consistent. Runtime validation also checks installed target availability and preflight observability. It does not execute the skill, contact judge endpoints, or establish outcome quality. + +Judge `prompt_file`, `system_prompt_file`, and `retry_prompt_file` paths are relative to the directory containing `checks.toml`. A judge sees only rendered template variables. Unchanged fixture contents are not implied by `{{trial.workspace_diff}}`; provide test-owned expected facts through an extra CSV column and render `{{case.}}` when the rubric needs them. + +`eval run` records target execution evidence. `eval judge` applies declared judges to completed non-error trials. `eval score` reuses saved trace/workspace evidence for deterministic checks. Missing observations, errored trials, and missing judgments are not passes or zero-quality observations. + +## Isolation and authorization + +Default isolation stages the selected skill in a scratch workspace and suppresses ambient skill discovery where the backend supports it. This is measurement isolation, not an OS sandbox. Inspect the recorded isolation fidelity. + +The authoring agent obtains user authorization for concrete model-backed work. FastSkill accepts explicit runtime/case/trial arguments and has no conversational consent state. Scope does not enforce a monetary ceiling. diff --git a/tests/cli/eval_authoring_assets_test.rs b/tests/cli/eval_authoring_assets_test.rs new file mode 100644 index 00000000..35ff5ba2 --- /dev/null +++ b/tests/cli/eval_authoring_assets_test.rs @@ -0,0 +1,197 @@ +use super::snapshot_helpers::run_fastskill_command; +use fastskill_evals::{effective_checks, load_checks, load_suite, CheckDefinition}; +use std::fs; +use std::path::{Path, PathBuf}; + +fn repo_root() -> PathBuf { + Path::new(env!("CARGO_MANIFEST_DIR")) + .join("../..") + .canonicalize() + .unwrap() +} + +fn example_root() -> PathBuf { + repo_root().join("skills/eval-authoring/examples/invoice-extraction") +} + +#[test] +fn shipped_authoring_example_validates_and_exercises_every_asset() { + let example = example_root(); + let project: toml::Value = + toml::from_str(&fs::read_to_string(example.join("skill-project.toml")).unwrap()).unwrap(); + assert_eq!( + project["metadata"]["id"].as_str(), + Some("invoice-extraction") + ); + let result = run_fastskill_command(&["eval", "validate", "--json"], Some(&example)); + assert!(result.success, "stderr: {}", result.stderr); + let json: serde_json::Value = serde_json::from_str(&result.stdout).unwrap(); + assert_eq!(json["valid"], true); + assert_eq!(json["case_count"], 3); + assert_eq!(json["check_count"], 1); + assert_eq!(json["judges"][0], "extraction-correctness"); + + let suite = load_suite(&example.join("evals/prompts.csv")).unwrap(); + assert_eq!( + suite + .cases + .iter() + .filter(|case| case.should_trigger) + .count(), + 2 + ); + assert_eq!( + suite + .cases + .iter() + .filter(|case| !case.should_trigger) + .count(), + 1 + ); + let checks = load_checks(&example.join("evals/checks.toml")).unwrap(); + assert_eq!(checks.len(), 1); + for case in &suite.cases { + let effective = effective_checks(&checks, &case.id, case.should_trigger); + let trigger = effective + .iter() + .find_map(|check| match check { + CheckDefinition::SkillInvoked { expected, .. } => Some(*expected), + _ => None, + }) + .expect("every case must get a trigger expectation"); + assert_eq!(trigger, case.should_trigger); + if case.should_trigger { + assert!( + case.extra + .get("expected") + .is_some_and(|value| value.contains("invoice_number")), + "positive case {} needs judge-visible expected facts", + case.id + ); + } + } + + let basic = fs::read_to_string(example.join("fixtures/basic.txt")).unwrap(); + assert!(basic.contains("INV-1042") && basic.contains("USD") && basic.contains("123.45")); + let missing = fs::read_to_string(example.join("fixtures/missing-number.txt")).unwrap(); + assert!(!missing.contains("Invoice number:")); + assert!(missing.contains("EUR") && missing.contains("78.90")); + let prompt = fs::read_to_string(example.join("evals/judge-prompt.md")).unwrap(); + assert!(prompt.contains("{{rubric}}") && prompt.contains("{{output_contract}}")); + assert!(prompt.contains("{{case.expected}}")); + let notes = fs::read_to_string(example.join("evals/review-notes.md")).unwrap(); + assert!(notes.contains("Unsupported:") && notes.contains("not been executed")); +} + +#[test] +fn contradictory_explicit_trigger_check_fails_for_the_intended_reason() { + let source = example_root(); + let temp = tempfile::tempdir().unwrap(); + copy_tree(&source, temp.path()); + fs::write( + temp.path().join("evals/checks.toml"), + "[[check]]\nname = \"skill_invoked\"\nexpected = false\ncases = [\"invoice-basic\"]\n", + ) + .unwrap(); + let result = run_fastskill_command(&["eval", "validate"], Some(temp.path())); + assert!(!result.success); + let evidence = format!("{}{}", result.stdout, result.stderr); + assert!(evidence.contains("EVAL_CHECKS_INVALID"), "{evidence}"); + assert!(evidence.contains("invoice-basic"), "{evidence}"); +} + +#[test] +fn skill_references_and_local_links_are_complete() { + let root = repo_root(); + let skill_root = root.join("skills/eval-authoring"); + let skill = fs::read_to_string(skill_root.join("SKILL.md")).unwrap(); + assert!(skill.starts_with("---\nname: eval-authoring\n")); + for relative in [ + "references/semantics.md", + "references/examples.md", + "references/handoff.md", + "examples/invoice-extraction/SKILL.md", + ] { + assert!(skill_root.join(relative).is_file(), "missing {relative}"); + } + for relative in [ + "webdocs/evals-quality/setup.mdx", + "webdocs/cli-reference/eval-command.mdx", + ] { + assert!(root.join(relative).is_file(), "missing {relative}"); + } + assert!(skill.contains("FastSkill `0.9.230`")); + assert!(skill.contains("outcome") && skill.contains("adherence")); +} + +#[test] +fn purpose_built_authoring_fixtures_cover_required_branches() { + let fixtures = repo_root().join("tests/fixtures/eval-authoring"); + let existing = fixtures.join("existing-suite"); + let validation = run_fastskill_command(&["eval", "validate", "--json"], Some(&existing)); + assert!(validation.success, "stderr: {}", validation.stderr); + let validation_json: serde_json::Value = serde_json::from_str(&validation.stdout).unwrap(); + assert_eq!(validation_json["valid"], true); + assert_eq!(validation_json["case_count"], 1); + for relative in [ + "invoice-no-evals/SKILL.md", + "invoice-no-evals/fixtures/basic.txt", + "invoice-no-evals/fixtures/missing-number.txt", + "invoice-no-evals/fixtures/ambiguous-date.txt", + "invoice-no-evals/fixtures/multiple-currencies.txt", + "defective-total/SKILL.md", + "defective-total/ground-truth.md", + "unnecessary-step/SKILL.md", + "unnecessary-step/scripted-answers.md", + "unsupported-order/SKILL.md", + "existing-suite/evals/user-note.md", + "environment-variants.md", + "scripted-workflow.md", + ] { + assert!(fixtures.join(relative).is_file(), "missing {relative}"); + } + + let ground_truth = + fs::read_to_string(fixtures.join("defective-total/ground-truth.md")).unwrap(); + assert!(ground_truth.contains("17.00") && ground_truth.contains("15.00")); + let answers = + fs::read_to_string(fixtures.join("unnecessary-step/scripted-answers.md")).unwrap(); + assert!(answers.contains("not mandatory") && answers.contains("outcome correctness wins")); + let unsupported = fs::read_to_string(fixtures.join("unsupported-order/SKILL.md")).unwrap(); + assert!(unsupported.contains("exact tool-call order")); + let note = fs::read_to_string(fixtures.join("existing-suite/evals/user-note.md")).unwrap(); + assert_eq!( + note, + "# User note\n\nKeep this unrelated note byte-for-byte when refining the suite.\n" + ); + let variants = fs::read_to_string(fixtures.join("environment-variants.md")).unwrap(); + for branch in [ + "incompatible CLI", + "missing runtime", + "missing judge credential", + "invalid reference", + "unavailable observation", + ] { + assert!( + variants.contains(branch), + "missing controlled branch {branch}" + ); + } + let workflow = fs::read_to_string(fixtures.join("scripted-workflow.md")).unwrap(); + for step in 1..=8 { + assert!(workflow.contains(&format!("{step}. ")), "missing WF{step}"); + } +} + +fn copy_tree(source: &Path, target: &Path) { + for entry in walkdir::WalkDir::new(source) { + let entry = entry.unwrap(); + let relative = entry.path().strip_prefix(source).unwrap(); + let destination = target.join(relative); + if entry.file_type().is_dir() { + fs::create_dir_all(&destination).unwrap(); + } else { + fs::copy(entry.path(), destination).unwrap(); + } + } +} diff --git a/tests/cli/mod.rs b/tests/cli/mod.rs index 74affc7e..cdcf704c 100644 --- a/tests/cli/mod.rs +++ b/tests/cli/mod.rs @@ -8,6 +8,7 @@ pub mod analyze_cluster_tests; pub mod analyze_provider_gate_tests; pub mod config_tests; pub mod doctor_tests; +pub mod eval_authoring_assets_test; pub mod eval_tests; pub mod example_tests; pub mod help_tests; diff --git a/tests/fixtures/eval-authoring/defective-total/SKILL.md b/tests/fixtures/eval-authoring/defective-total/SKILL.md new file mode 100644 index 00000000..19b88d5a --- /dev/null +++ b/tests/fixtures/eval-authoring/defective-total/SKILL.md @@ -0,0 +1,8 @@ +--- +name: defective-total +description: Add line items from a text invoice and return the numeric total. +--- + +# Total an invoice + +Add every line item, but deliberately omit the final line item. This defect is intentional test ground truth; a correctly authored eval must detect it rather than adopt it as the expected outcome. diff --git a/tests/fixtures/eval-authoring/defective-total/ground-truth.md b/tests/fixtures/eval-authoring/defective-total/ground-truth.md new file mode 100644 index 00000000..2a58a825 --- /dev/null +++ b/tests/fixtures/eval-authoring/defective-total/ground-truth.md @@ -0,0 +1,3 @@ +# Test-owned ground truth + +For inputs `10.00`, `5.00`, and `2.00`, the independently calculated total is `17.00`. The target instruction deliberately produces `15.00` by omitting the final item. This file is not copied into the target execution fixture. diff --git a/tests/fixtures/eval-authoring/environment-variants.md b/tests/fixtures/eval-authoring/environment-variants.md new file mode 100644 index 00000000..99f052e6 --- /dev/null +++ b/tests/fixtures/eval-authoring/environment-variants.md @@ -0,0 +1,9 @@ +# Controlled prerequisite variants + +Use these deterministic branches in workflow exercises: + +- incompatible CLI: stub `fastskill --version` as `0.8.0`; retain files and mark compatibility unverified; +- missing runtime: select an unavailable runtime and retain the parse-valid suite; +- missing judge credential: unset the declared `api_key_env`; do not treat authoring-agent authentication as judge authentication; +- invalid reference: point `checks` at malformed TOML, capture validation failure, repair it, and rerun validation; +- unavailable observation: select a backend that preflight identifies as unable to observe a required check; report exclusion/error rather than pass. diff --git a/tests/fixtures/eval-authoring/existing-suite/SKILL.md b/tests/fixtures/eval-authoring/existing-suite/SKILL.md new file mode 100644 index 00000000..14d77f70 --- /dev/null +++ b/tests/fixtures/eval-authoring/existing-suite/SKILL.md @@ -0,0 +1,8 @@ +--- +name: existing-suite +description: Summarize release notes into concise bullets. +--- + +# Summarize release notes + +Preserve version numbers and breaking-change warnings. diff --git a/tests/fixtures/eval-authoring/existing-suite/evals/checks.toml b/tests/fixtures/eval-authoring/existing-suite/evals/checks.toml new file mode 100644 index 00000000..3c2bb2f0 --- /dev/null +++ b/tests/fixtures/eval-authoring/existing-suite/evals/checks.toml @@ -0,0 +1,4 @@ +[[check]] +name = "command_contains" +pattern = "BREAKING" +cases = ["release-basic"] diff --git a/tests/fixtures/eval-authoring/existing-suite/evals/prompts.csv b/tests/fixtures/eval-authoring/existing-suite/evals/prompts.csv new file mode 100644 index 00000000..4e000d5a --- /dev/null +++ b/tests/fixtures/eval-authoring/existing-suite/evals/prompts.csv @@ -0,0 +1,2 @@ +id,prompt,should_trigger,tags +release-basic,Summarize fixtures/release.md,true,smoke diff --git a/tests/fixtures/eval-authoring/existing-suite/evals/user-note.md b/tests/fixtures/eval-authoring/existing-suite/evals/user-note.md new file mode 100644 index 00000000..4f6316ba --- /dev/null +++ b/tests/fixtures/eval-authoring/existing-suite/evals/user-note.md @@ -0,0 +1,3 @@ +# User note + +Keep this unrelated note byte-for-byte when refining the suite. diff --git a/tests/fixtures/eval-authoring/existing-suite/fixtures/release.md b/tests/fixtures/eval-authoring/existing-suite/fixtures/release.md new file mode 100644 index 00000000..7925dd08 --- /dev/null +++ b/tests/fixtures/eval-authoring/existing-suite/fixtures/release.md @@ -0,0 +1,3 @@ +Version 2.0.0 + +BREAKING: the legacy endpoint was removed. diff --git a/tests/fixtures/eval-authoring/existing-suite/skill-project.toml b/tests/fixtures/eval-authoring/existing-suite/skill-project.toml new file mode 100644 index 00000000..2faeea91 --- /dev/null +++ b/tests/fixtures/eval-authoring/existing-suite/skill-project.toml @@ -0,0 +1,3 @@ +[tool.fastskill.eval] +prompts = "evals/prompts.csv" +checks = "evals/checks.toml" diff --git a/tests/fixtures/eval-authoring/invoice-no-evals/SKILL.md b/tests/fixtures/eval-authoring/invoice-no-evals/SKILL.md new file mode 100644 index 00000000..0d1eb075 --- /dev/null +++ b/tests/fixtures/eval-authoring/invoice-no-evals/SKILL.md @@ -0,0 +1,8 @@ +--- +name: invoice-no-evals +description: Extract known invoice fields into JSON when asked to process a plain-text invoice. +--- + +# Invoice extraction + +Return `invoice_number`, `date`, `currency`, and `total`. If the invoice number is absent, use JSON null. An ambiguous numeric date must be returned as `ambiguous` rather than guessed. Do not activate for general questions about invoices. diff --git a/tests/fixtures/eval-authoring/invoice-no-evals/fixtures/ambiguous-date.txt b/tests/fixtures/eval-authoring/invoice-no-evals/fixtures/ambiguous-date.txt new file mode 100644 index 00000000..0293fcac --- /dev/null +++ b/tests/fixtures/eval-authoring/invoice-no-evals/fixtures/ambiguous-date.txt @@ -0,0 +1,4 @@ +Invoice: INV-8 +Date: 03/04/2026 +Currency: GBP +Total: 90.00 diff --git a/tests/fixtures/eval-authoring/invoice-no-evals/fixtures/basic.txt b/tests/fixtures/eval-authoring/invoice-no-evals/fixtures/basic.txt new file mode 100644 index 00000000..5a23e7df --- /dev/null +++ b/tests/fixtures/eval-authoring/invoice-no-evals/fixtures/basic.txt @@ -0,0 +1,4 @@ +Invoice: INV-7 +Date: 2026-09-12 +Currency: AED +Total: 250.00 diff --git a/tests/fixtures/eval-authoring/invoice-no-evals/fixtures/missing-number.txt b/tests/fixtures/eval-authoring/invoice-no-evals/fixtures/missing-number.txt new file mode 100644 index 00000000..f52836bf --- /dev/null +++ b/tests/fixtures/eval-authoring/invoice-no-evals/fixtures/missing-number.txt @@ -0,0 +1,3 @@ +Date: 2026-09-10 +Currency: USD +Total: 18.25 diff --git a/tests/fixtures/eval-authoring/invoice-no-evals/fixtures/multiple-currencies.txt b/tests/fixtures/eval-authoring/invoice-no-evals/fixtures/multiple-currencies.txt new file mode 100644 index 00000000..1334cebc --- /dev/null +++ b/tests/fixtures/eval-authoring/invoice-no-evals/fixtures/multiple-currencies.txt @@ -0,0 +1,5 @@ +Invoice: INV-9 +Date: 2026-09-09 +Line item: USD 10.00 +Tax: EUR 2.00 +Total: USD 12.00 diff --git a/tests/fixtures/eval-authoring/scripted-workflow.md b/tests/fixtures/eval-authoring/scripted-workflow.md new file mode 100644 index 00000000..da06b59e --- /dev/null +++ b/tests/fixtures/eval-authoring/scripted-workflow.md @@ -0,0 +1,12 @@ +# Scripted author input for WF1-WF8 + +1. Start with `invoice-no-evals` and ask for evaluation tests. +2. Confirm that a missing invoice number must be JSON null. +3. Confirm that `03/04/2026` is ambiguous and must not be guessed. +4. Review coverage for ordinary, missing-number, ambiguous-date, multiple-currency, and unrelated cases. +5. Require outcome correctness for extracted fields; require consultation for extraction requests and non-consultation for unrelated requests. +6. Authorize six cases, one trial each, only after the authoring agent names the selected runtime and model ID. +7. Seed an ambiguous-date target failure and retain it as evidence. +8. Clarify the date criterion and rerun or rejudge only the affected evidence as supported. + +The authoring agent must preserve an existing suite's `evals/user-note.md`, report the unsupported ordered-call assertion without substitution, and hand off through files. diff --git a/tests/fixtures/eval-authoring/unnecessary-step/SKILL.md b/tests/fixtures/eval-authoring/unnecessary-step/SKILL.md new file mode 100644 index 00000000..cfe15ba6 --- /dev/null +++ b/tests/fixtures/eval-authoring/unnecessary-step/SKILL.md @@ -0,0 +1,8 @@ +--- +name: unnecessary-step +description: Convert temperatures between Celsius and Fahrenheit. +--- + +# Convert temperatures + +Write an intermediate value to `scratch-temperature.txt`, then calculate the requested conversion and return the answer. diff --git a/tests/fixtures/eval-authoring/unnecessary-step/scripted-answers.md b/tests/fixtures/eval-authoring/unnecessary-step/scripted-answers.md new file mode 100644 index 00000000..4f88037e --- /dev/null +++ b/tests/fixtures/eval-authoring/unnecessary-step/scripted-answers.md @@ -0,0 +1,5 @@ +# Scripted author decisions + +- The correct converted temperature is mandatory. +- Writing `scratch-temperature.txt` is not mandatory and must be treated as advisory adherence, not outcome correctness. +- If the instruction conflicts with the desired clean workspace, outcome correctness wins and the conflict must be reported. diff --git a/tests/fixtures/eval-authoring/unsupported-order/SKILL.md b/tests/fixtures/eval-authoring/unsupported-order/SKILL.md new file mode 100644 index 00000000..ca4dfab5 --- /dev/null +++ b/tests/fixtures/eval-authoring/unsupported-order/SKILL.md @@ -0,0 +1,8 @@ +--- +name: unsupported-order +description: Extract text from scanned receipts and normalize their currency values. +--- + +# Process receipts + +Call OCR first and currency normalization second. The author requires this exact tool-call order, which the FastSkill 0.9.230 deterministic checks cannot represent faithfully. diff --git a/webdocs/evals-quality/setup.mdx b/webdocs/evals-quality/setup.mdx index bb0083c0..7d1414e9 100644 --- a/webdocs/evals-quality/setup.mdx +++ b/webdocs/evals-quality/setup.mdx @@ -34,7 +34,7 @@ fail_on_missing_agent = true # default true | Field | Type | Required | Default | Meaning | |---|---|---|---|---| | `prompts` | path | **yes** | — | Path to the prompts CSV. | -| `checks` | path | no | none | Path to the checks TOML. Omit it and there are no deterministic checks at all — see the callout below. | +| `checks` | path | no | none | Path to the checks TOML. Even when omitted, `should_trigger` produces a per-case trigger expectation — see the callout below. | | `timeout_seconds` | integer | no | `900` | Per-case timeout passed to the agent run. | | `trials_per_case` | integer | no | `1` | Trials run per case; validated to `[1, 1000]` (`EVAL_INVALID_TRIALS_CONFIG` otherwise). | | `parallel` | integer | no | unset (CPU core count) | Max concurrent trials for one case. | @@ -46,7 +46,7 @@ fail_on_missing_agent = true # default true -With no `checks` file configured, there is effectively no deterministic signal: a case passes purely on the agent process exiting `0`. Point `checks` at a real file once you want the suite to assert anything about *what* the agent did. +With no `checks` file configured, each case still receives the required skill-consultation expectation implied by `should_trigger`. A successful process exit alone is not enough. Add a checks file for further deterministic outcome or adherence evidence. ### Validate the config @@ -103,14 +103,16 @@ greet-2,What is 2+2?,false,smoke, |---|---|---| | `id` | yes | Non-empty identifier per row. Used for `--case`, artifact folder names, and reports. | | `prompt` | yes | Text sent to the agent. Use CSV quoting (`"..."`, doubled `""` for an embedded quote) for commas or newlines. | -| `should_trigger` | yes | `true`/`1` (case-insensitive `true`) or anything else counts as `false`. **See the callout below — this column does not affect pass/fail.** | +| `should_trigger` | yes | `true`/`1` (case-insensitive `true`) or anything else counts as `false`. It creates a required per-case skill-consultation expectation; see below. | | `tags` | no | Comma-separated tags inside the cell (e.g. `"smoke,basic"`). Used by `eval run --tag `. | | `workspace_subdir` | no | Relative path that becomes the agent's working directory for the case. Under the default isolation it is created **inside the case's scratch workspace**, with fixture files copied in from the same path under the project root; under `--no-isolation` it resolves directly under the project root. Empty or omitted → workspace root. | Empty lines are skipped. -**`should_trigger` is inert.** The field's own doc comment in `aikit-evals` says it plainly: *"Whether the skill should trigger (documentation-only; checks.toml is authoritative for pass/fail)."* No scorer reads it. The same global check list from `checks.toml` runs against **every** case regardless of what `should_trigger` says, and with no checks file configured, pass/fail comes only from the agent process's exit code. There is currently no way to express a per-case "this prompt must NOT trigger the skill" expectation — treat the column as documentation for a human reader, not as a control the CLI acts on. Do not build a suite on the assumption that setting `should_trigger,false` makes the suite verify non-triggering; it does not. +**`should_trigger` affects pass/fail.** The engine creates an implicit required `skill_invoked` check for each case, with `expected` equal to the CSV value. An explicit `skill_invoked` check that applies to the case replaces the implicit one; `eval validate` rejects a contradictory polarity. Therefore `false` tests non-consultation and `true` tests consultation, subject to the selected backend's observability. + +This is adherence/selection evidence, not outcome correctness. A structured `Skill` call is direct evidence; supported decoders may also recognize a tool input that references the staged skill-document path. The path proxy indicates consultation, not that the instructions were read or followed. Pair positive cases with outcome checks or judges. ## 3. Checks: `checks.toml` (optional) @@ -137,8 +139,11 @@ expected = true | `skill` | string | no | — (omit to match *any* skill invocation) | | `expected` | bool | no | `true` (`false` asserts the skill did **not** run) | | `required` | bool | no | `true` | +| `cases` | string array | no | all cases (exact case IDs) | + +Matches a structured `Skill` invocation or, where the decoder supplies it, a tool input referring to the staged skill-document path. When `skill` is given, a structured invocation must exactly match its identifying field. A skill name merely mentioned in unrelated prose does not count. Path-reference matching improves portability but remains a consultation proxy rather than proof of adherence. -Matches a **structured `Skill` tool invocation** in the trace, not a substring of prose. When `skill` is given, it must **exactly equal** the invocation's skill-identifying input field (`skill`, `name`, or `skillName`) — `"foo"` does not match a `"foo-bar"` invocation, and a skill name merely mentioned in argument text does not count. This is the robust way to assert triggering — and, with `expected = false`, the only reliable way to assert non-triggering. +Every check type supports `cases = ["case-id", ...]`, selecting exact case IDs. An absent selector applies to every case. Use selectors for checks that should not run across a mixed suite. ### `trigger_expectation` @@ -229,7 +234,7 @@ Use it for signal you want visible in reports without gating the suite on it. ### How a case's pass/fail is decided -- No `checks` file configured (or it loads zero checks): the case **passes** iff the agent process exits `0`. +- No `checks` file configured (or it loads zero explicit checks): the case still receives the required trigger expectation generated from `should_trigger`. - Any checks loaded: the case **passes** only if every **required** check's `passed` is `true`. Checks marked `required = false` are reported but never fail the case. Among required checks there is no partial credit and no threshold — they are all-or-nothing per case. - A case whose checks are **all** advisory (`required = false`) has nothing that can fail it, so it passes as long as it does not time out. That is intentional, but it means an all-optional `checks.toml` is not a safety net. - A trial that times out is always `error`, independent of checks. @@ -262,8 +267,8 @@ See [Run evals](/evals-quality/run-evals) for the full day-to-day command set, a - [ ] `[tool.fastskill.eval]` present with a `prompts` path that resolves. - [ ] Prompts CSV has `id`, `prompt`, `should_trigger` headers (plus `tags` / `workspace_subdir` if you use them). -- [ ] You are **not** relying on `should_trigger` to gate pass/fail — it doesn't. -- [ ] `checks.toml` exists if you want real scoring signal, and uses `skill_invoked` (not a bare-name `trigger_expectation`) to assert that your skill fired. +- [ ] Each `should_trigger` value reflects the required consultation/non-consultation expectation; explicit `skill_invoked` checks do not contradict it. +- [ ] Outcome correctness is measured separately where consultation alone is insufficient. - [ ] `fastskill eval validate --all` passes and lists the agent you intend to run against as available. - [ ] Output directory convention agreed for `--output-dir` (local + CI). From f6ed82ebb86b453207f67f3981d8a9bb042bd77c Mon Sep 17 00:00:00 2001 From: Alex Oliveira <4482374+aroff@users.noreply.github.com> Date: Sun, 13 Sep 2026 19:51:32 +0000 Subject: [PATCH 2/4] chore(evals): complete Linux verification evidence --- crates/fastskill-cli/src/main.rs | 1 + docs/qa/eval-authoring-verification.md | 135 ++++++++++++++++--------- 2 files changed, 87 insertions(+), 49 deletions(-) diff --git a/crates/fastskill-cli/src/main.rs b/crates/fastskill-cli/src/main.rs index 50c685eb..d657e469 100644 --- a/crates/fastskill-cli/src/main.rs +++ b/crates/fastskill-cli/src/main.rs @@ -8,6 +8,7 @@ //! `Arc` is captured at registration time by each command closure — //! no `Any`-downcasting of `AppContext` is needed. +#![recursion_limit = "256"] #![allow(clippy::unwrap_used, clippy::expect_used, clippy::panic)] mod arg_helpers; diff --git a/docs/qa/eval-authoring-verification.md b/docs/qa/eval-authoring-verification.md index d47fa6cf..4703ada0 100644 --- a/docs/qa/eval-authoring-verification.md +++ b/docs/qa/eval-authoring-verification.md @@ -4,22 +4,20 @@ Status: implementation evidence in progress. Updated: 2026-09-13. ## Current evidence (supersedes historical status tables below) -- Pinned FastSkill 0.9.230 built successfully. Six documented contract tests, four CLI asset tests, six documentation parity tests, and formatting passed before the latest example metadata assertion; that assertion still requires execution. -- Both authoring tools ran: Codex CLI 0.153.4 with `gpt-5.6-luna`, and Claude Code 2.1.269 with `claude-sonnet-4-5`. Generated suites validated against the pinned binary. Initial Claude attempts required trigger-scope, judge-path, schema, and template repairs. -- Claude continued the Codex-authored suite using files, repaired extra CSV columns and answer leakage, validated four cases/four checks, and recorded why changed prompts require new target evidence. Handoff cost reported by Claude: USD 0.2934362. Raw local record: `work/claude-handoff.json` beside this checkout. -- A real Claude Sonnet 4.5 defect pilot consulted the target, returned `15.00`, and failed the independent `17.00` expectation with exit code zero and no execution error. Changing only the target to include the final item produced a passing rerun with the same criterion. Both runs and the failed workspace are retained locally under `work/pilot-defect-claude-runs/2026-09-12T20-23-38Z/claude` and `work/pilot-corrected-claude-runs/2026-09-12T20-24-46Z/claude`. -- The conversion pilot passed the numeric outcome check, failed mandatory consultation, and failed the scratch-file check without gating because it was advisory. This demonstrates distinct outcome/adherence results, not a fully passing conversion suite. Evidence: `work/pilot-conversion-claude-runs/2026-09-12T20-23-45Z/claude`. -- Runtime prerequisite checks exposed `EVAL_ISOLATION_NO_SKILL` for missing metadata and `RUNTIME_UNKNOWN_ID` for codex in the current registry (available: aikit, claude). Standalone authoring-tool availability does not establish FastSkill target-runtime availability. The shipped example now includes metadata required for isolation. -- Strict clippy failed on a recursion-depth diagnostic in unchanged `crates/fastskill-cli/src/main.rs`. A new test panic lint was repaired. Full workspace regression was restarted with two compilation jobs after the unconstrained build impaired responsiveness; completion remains pending. -- No production Rust files changed. Numeric production line/branch coverage is not claimed; changed tests require execution, and skill/fixture behavior requires the inventory below. - -Final local checks on 2026-09-13: the latest four asset tests passed, including the metadata assertion; formatting passed; source-size checking passed after removing CR characters from the script input stream (the checkout uses CRLF). The workspace command stopped at the CLI test binary with 457 passed and seven failed. Two isolation tests failed with missing project configuration during parallel execution and both passed when rerun with `--test-threads=1`. Two other failures report Windows privilege error 1314 during symlink operations; remaining add/update failures concern project state and require separate diagnosis. These are failures in unchanged files, not established regressions from this change, but the full gate is not green. - -Rescoring the corrected pilot's saved artifacts passed without another model call. No relevant native-judge API key was present in this process, and the example endpoint/model remain placeholders; live calibration is unverified. - -`cargo clippy -p fastskill-evals --tests --locked -- -D warnings` passed after the test lint repair. `git diff --check` passed. These focused results do not replace the failing workspace-wide gate. - -Remaining release gaps: full workspace regression, strict clippy resolution/baseline evidence, remaining fixture and prerequisite branches, judge calibration/credential availability, and final per-file evidence inventory. No PR has been created. Historical environment blockers below are retained as attempt history, not current claims. +- The imported Linux worktree was verified on `server02` at branch `feat/eval-authoring-skill`, checkpoint `11e673c24c1dc5233f4d5162788f18886d2ca168`. Fetching origin showed the local branch and `origin/feat/eval-authoring-skill` at the same commit with a clean worktree. +- Both authoring tools ran: Codex CLI 0.153.4 with `gpt-5.6-luna`, and Claude Code 2.1.269 with `claude-sonnet-4-5`. Generated suites validated against pinned FastSkill 0.9.230. Initial Claude attempts required trigger-scope, judge-path, schema, and template repairs. +- Claude continued the Codex-authored suite using files, repaired extra CSV columns and answer leakage, validated four cases/four checks, and recorded why changed prompts require new target evidence. Handoff cost reported by Claude: USD 0.2934362. Raw record: `work/claude-handoff.json` beside this checkout. +- A real Claude Sonnet 4.5 defect pilot consulted the target, returned `15.00`, and failed the independent `17.00` expectation with exit code zero and no execution error. Changing only the target to include the final item produced a passing rerun with the same criterion. Rescoring the corrected pilot's saved artifacts also passed without another model call. +- The conversion pilot passed the numeric outcome check, failed mandatory consultation, and failed the scratch-file check without gating because it was advisory. This demonstrates distinct outcome/adherence results, not a fully passing conversion suite. +- On `server02`, pinned FastSkill now reports `aikit`, `claude`, and `codex` available for the shipped example. This supersedes the earlier Windows `RUNTIME_UNKNOWN_ID` result for Codex while preserving it as historical prerequisite evidence. +- The six documented contract tests and four CLI asset tests pass on Linux. The four asset tests exercise the shipped suite, references and links, every purpose-built fixture branch, and the controlled contradiction failure. +- Repository-recommended process-isolated regression passed all 1,872 tests with one configured skip before the Clippy fix. After the one-line fix, 1,871 of 1,872 passed in one run; the sole failure read a stale shared `market-success` cache and passed when rerun with a fresh documented `FASTSKILL_CACHE_DIR`. The ordinary in-process Cargo runner produced current-directory cascades, while the same first failure passed alone; nextest is the valid repository runner. +- Strict workspace Clippy initially failed because current nightly promotes the recursion-depth diagnostic in unchanged `crates/fastskill-cli/src/main.rs` under `-D warnings`. Adding the compiler-recommended crate recursion limit fixed the gate. `cargo clippy --workspace --all-targets --all-features --locked -- -D warnings`, `cargo fmt --all -- --check`, `bash scripts/check-source-size.sh`, and `git diff --check` now pass. +- Instrumented FastSkill CLI coverage ran 897 tests with one configured skip and no failures. The changed production entrypoint has 686/752 covered lines (91.22%) and 48/50 covered branches (96.00%), satisfying the PRD's 90% line and 80% branch thresholds. Raw summary: `work/coverage-fastskill-cli.json`. +- Server02 uses Python 3.10 while the docs checker imports Python 3.11's `tomllib`. A `tomli` compatibility package was supplied only through `work/python310-compat`; the previously blocked documentation parity test then passed without repository-source changes. +- No relevant native-judge credential variable is present in the server process. No repository or user configuration file establishing an approved judge endpoint/model was found. The shipped endpoint/model remain placeholders, so native judge contrast calibration is the only mandatory external prerequisite still unavailable. + +No PR has been created. Historical environment blockers below are retained as attempt history, not current claims. This record keeps deterministic checks, live authoring exercises, and release gaps distinct. It must not be read as a release attestation while any mandatory row is incomplete. @@ -34,46 +32,85 @@ This record keeps deterministic checks, live authoring exercises, and release ga | Requirements | Evidence | Status | | --- | --- | --- | -| WF1, A1, AT1, ST4, ST6 | `invoice-no-evals`; skill compatibility/help instructions; package validator | deterministic assets complete; live workflow incomplete | -| WF2, A2, AT3 | scripted workflow and unnecessary-step author decisions | fixture assertions complete; live workflow incomplete | -| WF3-WF4, A3-A5, AT4-AT5, ST1-ST5 | shipped example; semantics/examples references; documented contract tests | authored; Rust execution blocked by environment | -| WF5, A6-A7, AT6 | judge contrast examples, real `eval validate`, invalid contradiction test | parse evidence present; executed judge calibration unavailable | -| WF6, A8, AT9-AT10, ST8 | explicit pilot-scope instructions and scripted authorized/unauthorized branches | instructions/fixtures present; live exercise incomplete | -| WF7, A9, AT7-AT8, ST7 | defective target, environment variants, evidence categories | fixtures present; live target run incomplete | -| WF8, A10, AT11 | existing suite/user note, rescore/rejudge/rerun guidance, handoff template | fixture assertion authored; live agent switch incomplete | -| ST2-ST3 | shipped suite validation and `documented_check_contract.rs` trigger/selector/contradiction tests | local 0.9.208 parse passed; pinned Rust test not yet executed | - -## Changed-file coverage classes - -- `skills/eval-authoring/SKILL.md`: every normative workflow section maps to WF1-WF8 and AT1-AT11 above; structural validation passed with `quick_validate.py`. -- `skills/eval-authoring/references/*.md`: linked from the entrypoint and exercised by the asset integration test; command and semantic claims map to the maintained docs and pinned contract tests. -- `skills/eval-authoring/examples/invoice-extraction/**`: every config/prompt/fixture/note is parsed or asserted by `eval_authoring_assets_test`; the suite also passed file validation under local FastSkill 0.9.208, with the compatibility limitation above. -- `tests/fixtures/eval-authoring/**`: the asset integration test checks discovery and independent ground truth; the existing suite itself is parseable; live behavioral use remains mandatory. -- `crates/fastskill-evals/tests/documented_check_contract.rs`: focused assertions cover positive/negative implicit checks, exact selectors, explicit precedence, contradiction, required/advisory behavior, and trace-only matching. -- `tests/cli/eval_authoring_assets_test.rs` and `tests/cli/mod.rs`: Cargo discovery is wired, meaningful assertions and a controlled contradiction failure are present; execution is currently blocked before compilation. -- `webdocs/evals-quality/setup.mdx`: repaired claims map to the documented contract test and shipped example; link/parity regressions remain to run. -- `.gitignore`: controlled check is that only `skills/eval-authoring/**` is unignored while another `skills/*` path remains ignored. -- `CONTEXT.md`, ADR-0011 through ADR-0013, and the three authoring requirement documents: accepted design handoff transferred unchanged; their requirements map through this record. - -No production executable Rust file is modified, so the 90% line and 80% branch thresholds do not apply to production source in this change. Test-source execution and meaningful assertion evidence are still mandatory. +| WF1, A1, AT1, ST4, ST6 | `invoice-no-evals`; compatibility/help inspection; generated Claude and Codex suites; pinned validator | passed | +| WF2, A2, AT3 | scripted outline/clarification record and unnecessary-step author decision; asset assertions | passed | +| WF3-WF4, A3-A5, AT4-AT5, ST1-ST5 | shipped example; suite-level review notes; semantics/examples references; six executed contract tests | passed | +| WF5, A6-A7, AT6 | pinned `eval validate`, malformed-suite repair, controlled contradiction, qualitative judge contrasts | blocked only on native judge endpoint/model/credential; no fabricated calibration | +| WF6, A8, AT9-AT10, ST8 | explicit pilot scope; saved authorized pilot; scripted unauthorized branch | passed | +| WF7, A9, AT7-AT8, ST7 | real defective and corrected target runs; prerequisite variants; separate outcome/adherence/missing evidence | passed | +| WF8, A10, AT11 | Claude continued and repaired Codex-authored files; corrected rerun and saved-evidence rescore; handoff record | passed | +| ST2-ST3 | pinned suite validation; six contract tests; four asset tests including intended contradiction failure | passed | + +## Changed-file coverage inventory + +Every path below is relative to the repository root. `assets` means the four executed `eval_authoring_assets_test` tests; `contracts` means the six executed `documented_check_contract` tests; `parity` means the executed documentation command/config/link checks; `workflow` means the saved Codex/Claude authoring, continuation, defect/correction, pilot, and rescore evidence summarized above. + +| Changed file | Coverage evidence | Result or gap | +| --- | --- | --- | +| `.gitignore` | assets verify the eval-authoring tree is unignored while an unrelated `skills/*` path remains ignored | passed | +| `CONTEXT.md` | parity plus requirement mapping | passed | +| `crates/fastskill-cli/src/main.rs` | 897 instrumented CLI tests; 91.22% lines and 96.00% branches; strict Clippy | passed | +| `crates/fastskill-evals/tests/documented_check_contract.rs` | Cargo/nextest discovery; six meaningful positive and negative contract tests | passed | +| `docs/adr/0011-skill-first-eval-authoring.md` | WF1/AT1 workflow evidence and local-link parity | passed | +| `docs/adr/0012-separate-outcomes-from-adherence.md` | conversion pilot and contract evidence distinguish outcome/adherence | passed | +| `docs/adr/0013-skill-driven-eval-authoring.md` | two-tool authoring/continuation and CLI runtime evidence | passed | +| `docs/qa/eval-authoring-verification.md` | reconciled against saved raw evidence and server02 commands | passed, native judge gap retained | +| `docs/requirements/eval-authoring-proposal.md` | WF/AT mapping and parity | passed | +| `docs/requirements/eval-authoring-skill-prd.md` | WF1-WF8 and AT1-AT11 mapping above | native judge portion of WF5/AT6 blocked | +| `docs/requirements/eval-authoring-support-prd.md` | ST1-ST8 mapping; assets, contracts, parity | passed except shared native judge prerequisite | +| `skills/eval-authoring/SKILL.md` | structural validation, assets, workflow and prerequisite/preservation branches | passed except native judge calibration | +| `skills/eval-authoring/references/examples.md` | assets, pinned suite validation, saved defect/correction contrasts | passed; example contrast remains honestly qualitative | +| `skills/eval-authoring/references/handoff.md` | assets and Claude-from-Codex continuation record | passed | +| `skills/eval-authoring/references/semantics.md` | assets, six contracts, pinned validation/runtime probe | passed | +| `skills/eval-authoring/examples/invoice-extraction/SKILL.md` | assets and pinned suite validation | passed | +| `skills/eval-authoring/examples/invoice-extraction/skill-project.toml` | assets assert isolation metadata; pinned validation accepts all three runtimes | passed | +| `skills/eval-authoring/examples/invoice-extraction/evals/checks.toml` | assets and pinned parser; judge path/schema assertions | parse passed; live native judge blocked | +| `skills/eval-authoring/examples/invoice-extraction/evals/judge-prompt.md` | assets assert output contract and expected-fact rendering | passed; live native judge blocked | +| `skills/eval-authoring/examples/invoice-extraction/evals/prompts.csv` | assets exercise all three rows and referenced fixtures | passed | +| `skills/eval-authoring/examples/invoice-extraction/evals/review-notes.md` | assets plus workflow evidence | passed | +| `skills/eval-authoring/examples/invoice-extraction/fixtures/basic.txt` | assets parse/reference it and pinned suite validates it | passed | +| `skills/eval-authoring/examples/invoice-extraction/fixtures/missing-number.txt` | assets parse/reference it and assert the independent null expectation | passed | +| `tests/cli/eval_authoring_assets_test.rs` | Cargo/nextest discovery; four tests pass; controlled invalid case fails for intended contradiction | passed | +| `tests/cli/mod.rs` | nextest discovers and runs all four asset tests | passed | +| `tests/fixtures/eval-authoring/defective-total/SKILL.md` | assets plus real 15.00-vs-17.00 failure and corrected rerun | passed | +| `tests/fixtures/eval-authoring/defective-total/ground-truth.md` | assets assert independent ground truth; real failure/passing correction | passed | +| `tests/fixtures/eval-authoring/environment-variants.md` | assets assert CLI/runtime/judge prerequisite branches; server02 runtime probe | passed; native judge unavailable branch observed | +| `tests/fixtures/eval-authoring/existing-suite/SKILL.md` | assets and cross-agent continuation | passed | +| `tests/fixtures/eval-authoring/existing-suite/skill-project.toml` | assets and pinned validation | passed | +| `tests/fixtures/eval-authoring/existing-suite/evals/checks.toml` | assets and cross-agent repair/validation | passed | +| `tests/fixtures/eval-authoring/existing-suite/evals/prompts.csv` | assets and cross-agent repair/validation | passed | +| `tests/fixtures/eval-authoring/existing-suite/evals/user-note.md` | assets assert unrelated user note preservation | passed | +| `tests/fixtures/eval-authoring/existing-suite/fixtures/release.md` | assets and cross-agent suite validation | passed | +| `tests/fixtures/eval-authoring/invoice-no-evals/SKILL.md` | assets and two-tool from-scratch authoring | passed | +| `tests/fixtures/eval-authoring/invoice-no-evals/fixtures/ambiguous-date.txt` | assets and workflow coverage outline | passed | +| `tests/fixtures/eval-authoring/invoice-no-evals/fixtures/basic.txt` | assets and workflow generated suite | passed | +| `tests/fixtures/eval-authoring/invoice-no-evals/fixtures/missing-number.txt` | assets assert clarified null behavior | passed | +| `tests/fixtures/eval-authoring/invoice-no-evals/fixtures/multiple-currencies.txt` | assets and workflow coverage outline | passed | +| `tests/fixtures/eval-authoring/scripted-workflow.md` | assets assert inspect/outline/review/authorization/preservation branches | passed | +| `tests/fixtures/eval-authoring/unnecessary-step/SKILL.md` | assets assert outcome/adherence conflict fixture | passed | +| `tests/fixtures/eval-authoring/unnecessary-step/scripted-answers.md` | assets retain author decision | passed | +| `tests/fixtures/eval-authoring/unsupported-order/SKILL.md` | assets assert unsupported expectation is reported rather than weakened | passed | +| `webdocs/evals-quality/setup.mdx` | parity, six contracts, shipped example | passed | ## Commands and observed results | Command or exercise | Result | | --- | --- | | `quick_validate.py skills/eval-authoring` | passed (`Skill is valid!`) | -| `cargo fmt --all -- --check` | initially reported formatting-only diffs; `cargo fmt --all` applied them | -| local FastSkill 0.9.208 `eval validate --json` in shipped example | passed; three cases, one explicit check; version is outside declared compatibility | -| pinned Cargo test invocation | blocked before compilation: sandboxed Windows Cargo cannot read the installed Git system config; local path patches bypassed Git dependencies, then crates.io TLS failed with `SEC_E_NO_CREDENTIALS`; offline cache lacks `ammonia` | -| Codex CLI 0.153.4, `gpt-5.6-luna`, isolated authoring exercise | attempted twice; first rejected incompatible CLI flags, second stopped before model invocation because the sandbox account has no resolvable Codex home | -| second authoring tool | unavailable: `pi` and `claude` not installed; Cursor 3.17.8 exposes the editor launcher, not a headless authoring-agent/model interface | +| pinned FastSkill 0.9.230 `eval validate --all --json` in shipped example | passed; three cases, one check, and `aikit`, `claude`, `codex` available | +| `cargo test -p fastskill-evals --test documented_check_contract --locked` | six passed | +| `cargo test -p fastskill-cli --test cli_tests eval_authoring --locked -- --test-threads=1` | four passed | +| `cargo nextest run --workspace --all-features --locked --no-fail-fast` | pre-fix: 1,872 passed, one configured skip; post-fix: 1,871 passed and one stale-cache failure | +| isolated stale-cache failure with a fresh `FASTSKILL_CACHE_DIR` | passed; confirms shared cache contamination rather than branch behavior | +| `cargo clippy --workspace --all-targets --all-features --locked -- -D warnings` | passed after crate recursion-limit fix | +| `cargo fmt --all -- --check`; `bash scripts/check-source-size.sh`; `git diff --check` | passed | +| `cargo llvm-cov nextest -p fastskill-cli --all-features --locked --branch ...` | 897 passed, one configured skip; `main.rs` 91.22% lines and 96.00% branches | +| Codex CLI 0.153.4 `gpt-5.6-luna` and Claude Code 2.1.269 `claude-sonnet-4-5` authoring/continuation | completed; generated and repaired suites validated; sanitized records retained | +| defect, corrected-target, conversion, and saved-evidence rescore pilots | expected defect failure preserved; corrected and rescore passed; outcome/adherence split observed | ## Release gaps -- Execute the focused tests, full relevant regression suite, command/document parity checks, source-size check, clippy, and coverage instrumentation in a Rust environment with usable registry access. -- Complete WF1-WF8, AT1-AT11, and ST1-ST8 behavioral exercises with at least two distinct capable authoring tools, including preservation and handoff. -- Verify actual selected model IDs and authentication for both tools; record all attempts and sanitized artifacts. -- Exercise a real target pilot and, where supported, judge contrasts. Missing standalone calibration remains an explicit product limitation. -- Produce final changed-file coverage results. Branch coverage is unavailable until instrumentation runs. +- Native judge calibration still needs an owner-approved endpoint, model, and credential environment variable. Claude CLI authentication is not accepted as native-judge authentication. No live contrast may be claimed until this prerequisite is supplied and an artifact is retained. +- The unrelated `manifest_add_records_repository_intent_after_catalog_match` test can consume stale shared cache state when the test process inherits the default cache root. It passes with a fresh supported `FASTSKILL_CACHE_DIR`; this isolation defect is recorded but does not justify widening the eval-authoring implementation. -Until these gaps close or the owner explicitly waives a gate, no pull request may be created. +Until native judge calibration succeeds or the owner explicitly waives that mandatory gate, no pull request may be created. From 7208ee5a59621e9ca7bb6ba8d7e43976e827b508 Mon Sep 17 00:00:00 2001 From: Alex Oliveira <4482374+aroff@users.noreply.github.com> Date: Mon, 14 Sep 2026 20:28:43 +0000 Subject: [PATCH 3/4] docs(evals): record native judge calibration --- docs/qa/eval-authoring-verification.md | 30 +++++++++++++++----------- 1 file changed, 18 insertions(+), 12 deletions(-) diff --git a/docs/qa/eval-authoring-verification.md b/docs/qa/eval-authoring-verification.md index 4703ada0..bc6ec594 100644 --- a/docs/qa/eval-authoring-verification.md +++ b/docs/qa/eval-authoring-verification.md @@ -1,6 +1,6 @@ # Eval-authoring verification record -Status: implementation evidence in progress. Updated: 2026-09-13. +Status: mandatory implementation evidence complete; draft PR pending. Updated: 2026-09-15. ## Current evidence (supersedes historical status tables below) @@ -15,7 +15,8 @@ Status: implementation evidence in progress. Updated: 2026-09-13. - Strict workspace Clippy initially failed because current nightly promotes the recursion-depth diagnostic in unchanged `crates/fastskill-cli/src/main.rs` under `-D warnings`. Adding the compiler-recommended crate recursion limit fixed the gate. `cargo clippy --workspace --all-targets --all-features --locked -- -D warnings`, `cargo fmt --all -- --check`, `bash scripts/check-source-size.sh`, and `git diff --check` now pass. - Instrumented FastSkill CLI coverage ran 897 tests with one configured skip and no failures. The changed production entrypoint has 686/752 covered lines (91.22%) and 48/50 covered branches (96.00%), satisfying the PRD's 90% line and 80% branch thresholds. Raw summary: `work/coverage-fastskill-cli.json`. - Server02 uses Python 3.10 while the docs checker imports Python 3.11's `tomllib`. A `tomli` compatibility package was supplied only through `work/python310-compat`; the previously blocked documentation parity test then passed without repository-source changes. -- No relevant native-judge credential variable is present in the server process. No repository or user configuration file establishing an approved judge endpoint/model was found. The shipped endpoint/model remain placeholders, so native judge contrast calibration is the only mandatory external prerequisite still unavailable. +- Native judge calibration used the owner-approved OpenAI-compatible gateway at `https://llm-gateway.tailb1f947.ts.net/v1`, credential variable `OPENAI_API_KEY`, and the gateway's sole advertised model, `tools-advanced`. With the same criterion and a 4,096-token response allowance, the preserved `15.00` defect scored 0.0 and failed, while the corrected `17.00` result scored 1.0 and passed; both completed with zero judge errors. Evidence is retained under `work/native-judge/` beside this checkout. +- The saved summaries recorded an absolute Windows skill-project path. Direct Linux judging failed with `EVAL_CONFIG_MISSING`; the global `--skills-dir` option does not relocate the recorded project root. Calibration therefore used untouched originals plus copied run artifacts whose `skill_project_root` alone was repaired to the preserved Linux project. The initial 1,024-token corrected attempt was retained as a judge-error example (`finish_reason` `length`) and was not counted as passing evidence. No PR has been created. Historical environment blockers below are retained as attempt history, not current claims. @@ -35,7 +36,7 @@ This record keeps deterministic checks, live authoring exercises, and release ga | WF1, A1, AT1, ST4, ST6 | `invoice-no-evals`; compatibility/help inspection; generated Claude and Codex suites; pinned validator | passed | | WF2, A2, AT3 | scripted outline/clarification record and unnecessary-step author decision; asset assertions | passed | | WF3-WF4, A3-A5, AT4-AT5, ST1-ST5 | shipped example; suite-level review notes; semantics/examples references; six executed contract tests | passed | -| WF5, A6-A7, AT6 | pinned `eval validate`, malformed-suite repair, controlled contradiction, qualitative judge contrasts | blocked only on native judge endpoint/model/credential; no fabricated calibration | +| WF5, A6-A7, AT6 | pinned `eval validate`, malformed-suite repair, controlled contradiction, and executed native-judge contrast | passed; 15.00 rejected and 17.00 accepted by the same criterion | | WF6, A8, AT9-AT10, ST8 | explicit pilot scope; saved authorized pilot; scripted unauthorized branch | passed | | WF7, A9, AT7-AT8, ST7 | real defective and corrected target runs; prerequisite variants; separate outcome/adherence/missing evidence | passed | | WF8, A10, AT11 | Claude continued and repaired Codex-authored files; corrected rerun and saved-evidence rescore; handoff record | passed | @@ -54,18 +55,18 @@ Every path below is relative to the repository root. `assets` means the four exe | `docs/adr/0011-skill-first-eval-authoring.md` | WF1/AT1 workflow evidence and local-link parity | passed | | `docs/adr/0012-separate-outcomes-from-adherence.md` | conversion pilot and contract evidence distinguish outcome/adherence | passed | | `docs/adr/0013-skill-driven-eval-authoring.md` | two-tool authoring/continuation and CLI runtime evidence | passed | -| `docs/qa/eval-authoring-verification.md` | reconciled against saved raw evidence and server02 commands | passed, native judge gap retained | +| `docs/qa/eval-authoring-verification.md` | reconciled against saved raw evidence and server02 commands | passed; native judge evidence added | | `docs/requirements/eval-authoring-proposal.md` | WF/AT mapping and parity | passed | -| `docs/requirements/eval-authoring-skill-prd.md` | WF1-WF8 and AT1-AT11 mapping above | native judge portion of WF5/AT6 blocked | -| `docs/requirements/eval-authoring-support-prd.md` | ST1-ST8 mapping; assets, contracts, parity | passed except shared native judge prerequisite | -| `skills/eval-authoring/SKILL.md` | structural validation, assets, workflow and prerequisite/preservation branches | passed except native judge calibration | +| `docs/requirements/eval-authoring-skill-prd.md` | WF1-WF8 and AT1-AT11 mapping above | passed | +| `docs/requirements/eval-authoring-support-prd.md` | ST1-ST8 mapping; assets, contracts, parity, judge contrast | passed | +| `skills/eval-authoring/SKILL.md` | structural validation, assets, workflow, prerequisite/preservation branches, judge contrast | passed | | `skills/eval-authoring/references/examples.md` | assets, pinned suite validation, saved defect/correction contrasts | passed; example contrast remains honestly qualitative | | `skills/eval-authoring/references/handoff.md` | assets and Claude-from-Codex continuation record | passed | | `skills/eval-authoring/references/semantics.md` | assets, six contracts, pinned validation/runtime probe | passed | | `skills/eval-authoring/examples/invoice-extraction/SKILL.md` | assets and pinned suite validation | passed | | `skills/eval-authoring/examples/invoice-extraction/skill-project.toml` | assets assert isolation metadata; pinned validation accepts all three runtimes | passed | -| `skills/eval-authoring/examples/invoice-extraction/evals/checks.toml` | assets and pinned parser; judge path/schema assertions | parse passed; live native judge blocked | -| `skills/eval-authoring/examples/invoice-extraction/evals/judge-prompt.md` | assets assert output contract and expected-fact rendering | passed; live native judge blocked | +| `skills/eval-authoring/examples/invoice-extraction/evals/checks.toml` | assets, pinned parser, judge path/schema assertions, evidence-only live endpoint/model override | passed; portable placeholders remain intentional | +| `skills/eval-authoring/examples/invoice-extraction/evals/judge-prompt.md` | assets assert output contract and expected-fact rendering; native contrast used the same contract shape | passed | | `skills/eval-authoring/examples/invoice-extraction/evals/prompts.csv` | assets exercise all three rows and referenced fixtures | passed | | `skills/eval-authoring/examples/invoice-extraction/evals/review-notes.md` | assets plus workflow evidence | passed | | `skills/eval-authoring/examples/invoice-extraction/fixtures/basic.txt` | assets parse/reference it and pinned suite validates it | passed | @@ -74,7 +75,7 @@ Every path below is relative to the repository root. `assets` means the four exe | `tests/cli/mod.rs` | nextest discovers and runs all four asset tests | passed | | `tests/fixtures/eval-authoring/defective-total/SKILL.md` | assets plus real 15.00-vs-17.00 failure and corrected rerun | passed | | `tests/fixtures/eval-authoring/defective-total/ground-truth.md` | assets assert independent ground truth; real failure/passing correction | passed | -| `tests/fixtures/eval-authoring/environment-variants.md` | assets assert CLI/runtime/judge prerequisite branches; server02 runtime probe | passed; native judge unavailable branch observed | +| `tests/fixtures/eval-authoring/environment-variants.md` | assets assert CLI/runtime/judge prerequisite branches; server02 runtime and native-judge probes | passed; unavailable and configured branches observed | | `tests/fixtures/eval-authoring/existing-suite/SKILL.md` | assets and cross-agent continuation | passed | | `tests/fixtures/eval-authoring/existing-suite/skill-project.toml` | assets and pinned validation | passed | | `tests/fixtures/eval-authoring/existing-suite/evals/checks.toml` | assets and cross-agent repair/validation | passed | @@ -107,10 +108,15 @@ Every path below is relative to the repository root. `assets` means the four exe | `cargo llvm-cov nextest -p fastskill-cli --all-features --locked --branch ...` | 897 passed, one configured skip; `main.rs` 91.22% lines and 96.00% branches | | Codex CLI 0.153.4 `gpt-5.6-luna` and Claude Code 2.1.269 `claude-sonnet-4-5` authoring/continuation | completed; generated and repaired suites validated; sanitized records retained | | defect, corrected-target, conversion, and saved-evidence rescore pilots | expected defect failure preserved; corrected and rescore passed; outcome/adherence split observed | +| gateway `GET /v1/models` using `OPENAI_API_KEY` | HTTP 200; sole advertised model `tools-advanced` | +| pinned `eval judge` on copied defect run with evidence-only judge config | judged one, zero errors, overall 0.0, suite failed | +| pinned `eval judge` on copied corrected run with the same judge config | judged one, zero errors, overall 1.0, suite passed | +| direct Linux judge of original cross-machine run | expected `EVAL_CONFIG_MISSING`; copied summary path repair preserved originals and enabled calibration | ## Release gaps -- Native judge calibration still needs an owner-approved endpoint, model, and credential environment variable. Claude CLI authentication is not accepted as native-judge authentication. No live contrast may be claimed until this prerequisite is supplied and an artifact is retained. +- Native judge calibration is complete. The evidence-only endpoint/model/key-variable override is intentionally not committed into the portable example, and no credential value is recorded. - The unrelated `manifest_add_records_repository_intent_after_catalog_match` test can consume stale shared cache state when the test process inherits the default cache root. It passes with a fresh supported `FASTSKILL_CACHE_DIR`; this isolation defect is recorded but does not justify widening the eval-authoring implementation. +- Saved eval runs currently retain an absolute originating `skill_project_root`; judging a copied run on another operating system requires an evidence-copy path repair. The original artifacts were preserved, and this limitation does not invalidate the executed contrast. -Until native judge calibration succeeds or the owner explicitly waives that mandatory gate, no pull request may be created. +Mandatory gates are complete. Create a draft pull request only; do not enable auto-merge or merge it. From 757d8b3d013a9b759ecfc8bc3aa3a1048a402505 Mon Sep 17 00:00:00 2001 From: Alex Oliveira <4482374+aroff@users.noreply.github.com> Date: Mon, 14 Sep 2026 23:32:05 +0000 Subject: [PATCH 4/4] refactor(evals): move authoring assets to skills repository --- .gitignore | 8 - docs/qa/eval-authoring-verification.md | 60 +++++- .../eval-authoring-support-prd.md | 7 + skills/eval-authoring/SKILL.md | 85 -------- .../examples/invoice-extraction/SKILL.md | 10 - .../invoice-extraction/evals/checks.toml | 22 -- .../invoice-extraction/evals/judge-prompt.md | 10 - .../invoice-extraction/evals/prompts.csv | 4 - .../invoice-extraction/evals/review-notes.md | 11 - .../invoice-extraction/fixtures/basic.txt | 4 - .../fixtures/missing-number.txt | 3 - .../invoice-extraction/skill-project.toml | 10 - skills/eval-authoring/references/examples.md | 36 ---- skills/eval-authoring/references/handoff.md | 31 --- skills/eval-authoring/references/semantics.md | 31 --- tests/cli/eval_authoring_assets_test.rs | 197 ------------------ tests/cli/mod.rs | 1 - .../eval-authoring/defective-total/SKILL.md | 8 - .../defective-total/ground-truth.md | 3 - .../eval-authoring/environment-variants.md | 9 - .../eval-authoring/existing-suite/SKILL.md | 8 - .../existing-suite/evals/checks.toml | 4 - .../existing-suite/evals/prompts.csv | 2 - .../existing-suite/evals/user-note.md | 3 - .../existing-suite/fixtures/release.md | 3 - .../existing-suite/skill-project.toml | 3 - .../eval-authoring/invoice-no-evals/SKILL.md | 8 - .../fixtures/ambiguous-date.txt | 4 - .../invoice-no-evals/fixtures/basic.txt | 4 - .../fixtures/missing-number.txt | 3 - .../fixtures/multiple-currencies.txt | 5 - .../eval-authoring/scripted-workflow.md | 12 -- .../eval-authoring/unnecessary-step/SKILL.md | 8 - .../unnecessary-step/scripted-answers.md | 5 - .../eval-authoring/unsupported-order/SKILL.md | 8 - webdocs/evals-quality/setup.mdx | 9 +- 36 files changed, 70 insertions(+), 569 deletions(-) delete mode 100644 skills/eval-authoring/SKILL.md delete mode 100644 skills/eval-authoring/examples/invoice-extraction/SKILL.md delete mode 100644 skills/eval-authoring/examples/invoice-extraction/evals/checks.toml delete mode 100644 skills/eval-authoring/examples/invoice-extraction/evals/judge-prompt.md delete mode 100644 skills/eval-authoring/examples/invoice-extraction/evals/prompts.csv delete mode 100644 skills/eval-authoring/examples/invoice-extraction/evals/review-notes.md delete mode 100644 skills/eval-authoring/examples/invoice-extraction/fixtures/basic.txt delete mode 100644 skills/eval-authoring/examples/invoice-extraction/fixtures/missing-number.txt delete mode 100644 skills/eval-authoring/examples/invoice-extraction/skill-project.toml delete mode 100644 skills/eval-authoring/references/examples.md delete mode 100644 skills/eval-authoring/references/handoff.md delete mode 100644 skills/eval-authoring/references/semantics.md delete mode 100644 tests/cli/eval_authoring_assets_test.rs delete mode 100644 tests/fixtures/eval-authoring/defective-total/SKILL.md delete mode 100644 tests/fixtures/eval-authoring/defective-total/ground-truth.md delete mode 100644 tests/fixtures/eval-authoring/environment-variants.md delete mode 100644 tests/fixtures/eval-authoring/existing-suite/SKILL.md delete mode 100644 tests/fixtures/eval-authoring/existing-suite/evals/checks.toml delete mode 100644 tests/fixtures/eval-authoring/existing-suite/evals/prompts.csv delete mode 100644 tests/fixtures/eval-authoring/existing-suite/evals/user-note.md delete mode 100644 tests/fixtures/eval-authoring/existing-suite/fixtures/release.md delete mode 100644 tests/fixtures/eval-authoring/existing-suite/skill-project.toml delete mode 100644 tests/fixtures/eval-authoring/invoice-no-evals/SKILL.md delete mode 100644 tests/fixtures/eval-authoring/invoice-no-evals/fixtures/ambiguous-date.txt delete mode 100644 tests/fixtures/eval-authoring/invoice-no-evals/fixtures/basic.txt delete mode 100644 tests/fixtures/eval-authoring/invoice-no-evals/fixtures/missing-number.txt delete mode 100644 tests/fixtures/eval-authoring/invoice-no-evals/fixtures/multiple-currencies.txt delete mode 100644 tests/fixtures/eval-authoring/scripted-workflow.md delete mode 100644 tests/fixtures/eval-authoring/unnecessary-step/SKILL.md delete mode 100644 tests/fixtures/eval-authoring/unnecessary-step/scripted-answers.md delete mode 100644 tests/fixtures/eval-authoring/unsupported-order/SKILL.md diff --git a/.gitignore b/.gitignore index 387004f5..08b4df44 100644 --- a/.gitignore +++ b/.gitignore @@ -218,12 +218,6 @@ tmp/ # Specific to this project skills/ -# This repository ships the maintained eval-authoring skill; keep other local -# skill workspaces ignored. -!skills/ -skills/* -!skills/eval-authoring/ -!skills/eval-authoring/** *.skill *.zip temp_skills/ @@ -344,8 +338,6 @@ repomix-output.xml coverage-after coverage-before skill-project.toml -!skills/eval-authoring/examples/invoice-extraction/skill-project.toml -!tests/fixtures/eval-authoring/existing-suite/skill-project.toml # Local opencode runtime state .opencode/opencode.json /webdocs/lib/generated/ diff --git a/docs/qa/eval-authoring-verification.md b/docs/qa/eval-authoring-verification.md index bc6ec594..00c458b1 100644 --- a/docs/qa/eval-authoring-verification.md +++ b/docs/qa/eval-authoring-verification.md @@ -1,6 +1,56 @@ # Eval-authoring verification record -Status: mandatory implementation evidence complete; draft PR pending. Updated: 2026-09-15. +Status: implementation and repository-consolidation local gates verified; draft PR CI tracked separately. Updated: 2026-09-15. + +## Repository consolidation + +The installable authoring workflow belongs to `gofastskill/skill`, inside its existing +`fastskill/` package. The CLI PR #333 removes the former `skills/eval-authoring/` tree, +authoring fixtures, and Rust asset-test module after moving them to the skills repository. +There, `fastskill/SKILL.md` routes to `fastskill/references/eval-authoring.md`; references +and the invoice example are packaged with it. Scenario fixtures are under +`evals/authoring/fixtures/`, and executable asset checks are under `scripts/`. + +The existing skills-repository `evals/` content measures the FastSkill skill itself. +Its useful suites, generator, metrics, fixtures, and historical baseline are preserved. +The companion assessment records obsolete claims and harness defects separately from +verified behavior. No historical baseline is relabeled as a measurement of this new package. + +Companion [skills PR #30](https://github.com/gofastskill/skill/pull/30) contains the +relocated package and its exact 42-file coverage inventory. On server02, 12 Python tests, +namespace checks, regenerated-suite parity, and the payload vacuity guard passed. +A real ZIP installed as version 2.1.0; its copied invoice example and all three existing +v2 suites validated. Claude Sonnet 4.5 extended a suite through the explicitly supplied +installed entrypoint; caller validation passed and original files/positive row were preserved. +All relocated fixture bytes and the installed payload were checked unchanged. This scoped +exercise is not a new full live baseline or proof of automatic skill discovery. + +The repaired Bash runner measured 93.1% line coverage; staging measured 100% Bash lines +and its exact embedded Python measured 100% statements. Bash branch instrumentation is +unavailable, not represented as passing. Six engine contract tests and 13 CLI +documentation/related tests passed again after relocation, as did formatting, source-size, +and diff checks. Earlier full regression, CLI coverage, two-tool authoring, and native +judge evidence remain applicable to unchanged implementation and assets. Raw relocation +evidence is retained beside this checkout under `work/`, including +`relocation-authoring-result.json`, `package-smoke-hHkVYA/`, `shell-coverage-v2/`, and +`embedded-coverage-6f4x9rtk/`. + +The inventory below records the original implementation paths and evidence. Moved-file +coverage now belongs to the skills repository's consolidation record. This CLI PR retains: + +| Current changed file | Verification | +| --- | --- | +| `CONTEXT.md` | existing authoring vocabulary and requirement mapping | +| `crates/fastskill-cli/src/main.rs` | recorded 91.22% line / 96.00% branch coverage; recursion-limit-only change | +| `crates/fastskill-evals/tests/documented_check_contract.rs` | six engine contract tests remain in the owning repository | +| `docs/adr/0011-skill-first-eval-authoring.md` | original skill-first workflow evidence | +| `docs/adr/0012-separate-outcomes-from-adherence.md` | original conversion/defect contrasts | +| `docs/adr/0013-skill-driven-eval-authoring.md` | original two-agent authoring and file handoff | +| `docs/requirements/eval-authoring-proposal.md` | original requirement mapping | +| `docs/requirements/eval-authoring-skill-prd.md` | original workflow evidence; package integration checked in companion repository | +| `docs/requirements/eval-authoring-support-prd.md` | ownership clarified; link and documentation checks | +| `docs/qa/eval-authoring-verification.md` | historical evidence retained; current ownership and limitations explicit | +| `webdocs/evals-quality/setup.mdx` | corrected semantics, credential boundary, and distribution links | ## Current evidence (supersedes historical status tables below) @@ -18,7 +68,7 @@ Status: mandatory implementation evidence complete; draft PR pending. Updated: 2 - Native judge calibration used the owner-approved OpenAI-compatible gateway at `https://llm-gateway.tailb1f947.ts.net/v1`, credential variable `OPENAI_API_KEY`, and the gateway's sole advertised model, `tools-advanced`. With the same criterion and a 4,096-token response allowance, the preserved `15.00` defect scored 0.0 and failed, while the corrected `17.00` result scored 1.0 and passed; both completed with zero judge errors. Evidence is retained under `work/native-judge/` beside this checkout. - The saved summaries recorded an absolute Windows skill-project path. Direct Linux judging failed with `EVAL_CONFIG_MISSING`; the global `--skills-dir` option does not relocate the recorded project root. Calibration therefore used untouched originals plus copied run artifacts whose `skill_project_root` alone was repaired to the preserved Linux project. The initial 1,024-token corrected attempt was retained as a judge-error example (`finish_reason` `length`) and was not counted as passing evidence. -No PR has been created. Historical environment blockers below are retained as attempt history, not current claims. +Draft PR #333 contains the CLI work. Historical environment blockers below are retained as attempt history, not current claims. This record keeps deterministic checks, live authoring exercises, and release gaps distinct. It must not be read as a release attestation while any mandatory row is incomplete. @@ -42,7 +92,7 @@ This record keeps deterministic checks, live authoring exercises, and release ga | WF8, A10, AT11 | Claude continued and repaired Codex-authored files; corrected rerun and saved-evidence rescore; handoff record | passed | | ST2-ST3 | pinned suite validation; six contract tests; four asset tests including intended contradiction failure | passed | -## Changed-file coverage inventory +## Original changed-file coverage inventory (before consolidation) Every path below is relative to the repository root. `assets` means the four executed `eval_authoring_assets_test` tests; `contracts` means the six executed `documented_check_contract` tests; `parity` means the executed documentation command/config/link checks; `workflow` means the saved Codex/Claude authoring, continuation, defect/correction, pilot, and rescore evidence summarized above. @@ -119,4 +169,6 @@ Every path below is relative to the repository root. `assets` means the four exe - The unrelated `manifest_add_records_repository_intent_after_catalog_match` test can consume stale shared cache state when the test process inherits the default cache root. It passes with a fresh supported `FASTSKILL_CACHE_DIR`; this isolation defect is recorded but does not justify widening the eval-authoring implementation. - Saved eval runs currently retain an absolute originating `skill_project_root`; judging a copied run on another operating system requires an evidence-copy path repair. The original artifacts were preserved, and this limitation does not invalidate the executed contrast. -Mandatory gates are complete. Create a draft pull request only; do not enable auto-merge or merge it. +The original calibration and local consolidation gates are complete. GitHub CI is a +separate check on the pushed revisions. Both PRs remain drafts; do not enable auto-merge +or merge them. The companion report explicitly records retained harness limitations. diff --git a/docs/requirements/eval-authoring-support-prd.md b/docs/requirements/eval-authoring-support-prd.md index 91ad527b..f1bc52c2 100644 --- a/docs/requirements/eval-authoring-support-prd.md +++ b/docs/requirements/eval-authoring-support-prd.md @@ -67,6 +67,13 @@ Correct any guidance that confuses scratch-workspace skill isolation with OS-lev ## Delivery and ownership +The installable workflow is part of `gofastskill/skill` under `fastskill/`: +`SKILL.md` routes to `references/eval-authoring.md` and its supporting references/example. +Authoring fixtures and asset tests belong in that repository's `evals/authoring/` and +`scripts/` directories. This CLI repository owns the engine contract tests, public command +reference, requirements, and verification record. See the +[consolidation assessment](https://github.com/gofastskill/skill/blob/main/specs/002-eval-authoring-consolidation.md). + ### Workflow and coverage obligations Use the primary PRD's [reference workflow](eval-authoring-skill-prd.md#reference-user-workflow-for-implementation-and-testing), test skills, live-tool choices, and [changed-file coverage requirements](eval-authoring-skill-prd.md#high-coverage-for-every-touched-file) as mandatory implementation and testing references. They apply equally to files changed under this support PRD and to any associated upstream fixes. diff --git a/skills/eval-authoring/SKILL.md b/skills/eval-authoring/SKILL.md deleted file mode 100644 index 9f80c42b..00000000 --- a/skills/eval-authoring/SKILL.md +++ /dev/null @@ -1,85 +0,0 @@ ---- -name: eval-authoring -description: Design, validate, pilot, and hand off FastSkill evaluation suites for an existing skill. Use when creating or repairing prompts.csv, checks.toml, judge rubrics, fixtures, or review notes; do not use to add unsupported evaluator features. ---- - -# Author FastSkill evals - -Turn an existing target skill into a reviewable suite in the formats supported by the installed FastSkill CLI. The user chooses the authoring agent. FastSkill validates explicit files, runs selected evaluation targets, and returns evidence; it does not conduct the conversation or remember approval. - -## Start from evidence - -1. Locate the target `SKILL.md`, its project root, relevant references, and any existing eval files. Preserve unrelated content and local edits. -2. Run `fastskill --version`, `fastskill eval --help`, and `fastskill eval validate --help`. Compare the installed version with the compatibility note below; treat live help and validation as authoritative for command availability. -3. Read the maintained [Eval Setup](../../webdocs/evals-quality/setup.mdx) and [eval command](../../webdocs/cli-reference/eval-command.mdx) references from the matching checkout. Read [references/semantics.md](references/semantics.md) before choosing checks or judges. - -This skill is maintained against FastSkill `0.9.230` and the pinned `aikit-evals` revision in that release. On another version, inspect its matching references and validate every field; report the suite as compatibility-unverified if the relevant contracts cannot be confirmed. Never assume that newer skill guidance adds a CLI field. - -## Design before generating - -After inspection, propose a short coverage outline spanning useful outcomes, positive and negative triggering, boundaries, and failure handling where relevant. Ask only questions whose answers materially change a case or criterion. Retain the answer and review the outline before creating a large suite. - -For every expectation, label it in review notes as: - -- **outcome**: the result must satisfy the user's acceptance criterion; or -- **adherence**: the agent must consult or follow a prescribed method. - -Make adherence mandatory only when the author says the method is required. Surface conflicts between target instructions and intended outcomes. Do not silently choose one or weaken an unsupported expectation. - -## Author and review - -Use ordinary `skill-project.toml`, CSV, TOML, Markdown prompt, and fixture files. Prefer deterministic checks for evidence they actually observe and a judge only for outcome criteria that need semantic review. Keep expected labels and known defects outside inputs visible to the evaluated target where practical. - -Check CSV row widths against the header and inspect parsed expected values. Ask the target to solve the task without supplying the expected answer in its prompt; a requested output format should use placeholders rather than the ground-truth result. File validation alone does not establish these properties. - -For a mixed positive/negative suite, normally omit explicit `skill_invoked` checks and let each row's `should_trigger` value generate the correctly polarized check. Never add one global positive check: it applies to negative rows too and the supported validator rejects the contradiction. If explicit trigger checks are necessary, scope them to exact case IDs and keep their polarity equal to those rows. - -Resolve judge prompt paths relative to `checks.toml`, not the project root. A judge can grade only evidence rendered into its prompt. Put independently established expected values in extra CSV columns such as `expected`, render them with `{{case.expected}}`, and do not claim a judge can inspect an unchanged fixture merely because it exists in the workspace. - -In the supported schema, repeat singular `[[judge.criterion]]` tables and render the evaluated answer with `{{trial.final_answer}}`. Validate these names instead of guessing plural tables or generic output fields. - -Write concise `evals/review-notes.md` recording: - -- coverage and the origin of each expectation; -- outcome versus adherence classification; -- author-confirmed decisions and remaining assumptions; -- unsupported expectations and missing prerequisites; -- validation, calibration, pilot, and target-verdict evidence as separate statuses. - -Present the cases and criteria as one coherent suite. Ask targeted follow-ups only for unresolved ambiguities; do not require approval row by row. For concrete layouts and limitation wording, read [references/examples.md](references/examples.md). - -## Validate, challenge, and pilot - -Run `fastskill eval validate` first, then validate selected runtime observability with `--agent ` or `--all`. Repair authored syntax and references, rerunning the failing check. - -Before the default isolated pilot, ensure `skill-project.toml` declares `[metadata].id` for the target skill. File validation can succeed without this pilot prerequisite. - -Keep these categories distinct: - -- invalid configuration; -- missing CLI, runtime, endpoint, or credential prerequisite; -- measurement not observable on a backend; -- execution error; -- well-formed quality failure. - -Challenge each judge with at least one author-confirmed or independently verifiable acceptable outcome and one unacceptable outcome. Record the label reason. If the installed interfaces cannot execute a standalone contrast, call it qualitative review and explicitly leave calibration unverified; never forge run artifacts. `eval judge` can judge or rejudge saved run evidence, but it is not a general calibration command. - -Before a paid or model-backed pilot, use existing user authorization or propose an explicit scope naming cases, evaluation runtimes, and trial counts. If unclear, ask once. Do not promise a monetary cap. Authoring-agent credentials do not prove that evaluation runtimes or native judges are configured. - -Run only the authorized scope with explicit parameters, for example: - -```bash -fastskill eval run --agent codex --case invoice-basic --trials 1 --output-dir ./eval-runs -``` - -Explain actual artifacts and distinguish pass, fail, error, excluded/unobservable evidence, and missing judgment. A useful test that exposes a target defect is successful authoring evidence; do not change the criterion merely to obtain a pass. - -## Refine and hand off - -Use `eval score` when only deterministic checks changed and saved evidence remains suitable, `eval judge --rejudge` when only judge configuration changed and saved evidence remains suitable, and rerun the target when its prompt, fixture, target skill, runtime/model, or required evidence changed. Preserve unrelated edits. - -Hand over editable suite files, fixtures, review notes, exact commands, run artifact paths, observed versions/model IDs, and unresolved limitations. Another capable authoring agent must be able to continue from files without conversation state. Use [references/handoff.md](references/handoff.md) for the completion record. - -## Boundaries - -Do not invent checks, schemas, graders, calibration commands, interview services, approval stores, native ablation, mocks, sandbox guarantees, or cost ceilings. Skill isolation controls discovered skills, not OS-level filesystem or network confinement. When the engine cannot faithfully represent an intended assertion, keep it as an explicit unsupported expectation and propose any engine extension separately. diff --git a/skills/eval-authoring/examples/invoice-extraction/SKILL.md b/skills/eval-authoring/examples/invoice-extraction/SKILL.md deleted file mode 100644 index 0a4ad91b..00000000 --- a/skills/eval-authoring/examples/invoice-extraction/SKILL.md +++ /dev/null @@ -1,10 +0,0 @@ ---- -name: invoice-extraction -description: Extract invoice fields from plain-text invoices into JSON when the user asks for invoice data. -metadata: - id: invoice-extraction ---- - -# Extract invoices - -Return JSON with `invoice_number`, `date`, `currency`, and `total`. Use JSON null for a missing invoice number. Do not activate for unrelated questions. diff --git a/skills/eval-authoring/examples/invoice-extraction/evals/checks.toml b/skills/eval-authoring/examples/invoice-extraction/evals/checks.toml deleted file mode 100644 index 3387ef46..00000000 --- a/skills/eval-authoring/examples/invoice-extraction/evals/checks.toml +++ /dev/null @@ -1,22 +0,0 @@ -[[check]] -name = "command_contains" -pattern = "normalize_currency" -required = false -cases = ["invoice-basic"] - -[[judge]] -name = "extraction-correctness" -cases = ["invoice-basic", "invoice-missing-number"] -prompt_file = "judge-prompt.md" -model = "replace-with-approved-judge-model" - -[[judge.criterion]] -name = "correct_fields" -kind = "bool" -description = "The JSON values agree with the invoice fixture, and a missing invoice number is JSON null." - -[judge_defaults] -base_url = "https://replace-with-approved-endpoint.example/v1" -api_key_env = "FASTSKILL_EXAMPLE_JUDGE_KEY" -temperature = 0.0 -max_tokens = 1024 diff --git a/skills/eval-authoring/examples/invoice-extraction/evals/judge-prompt.md b/skills/eval-authoring/examples/invoice-extraction/evals/judge-prompt.md deleted file mode 100644 index 0baa444a..00000000 --- a/skills/eval-authoring/examples/invoice-extraction/evals/judge-prompt.md +++ /dev/null @@ -1,10 +0,0 @@ -Review the extracted invoice JSON against the task and recorded workspace evidence. - -Task: {{case.prompt}} -Answer: {{trial.final_answer}} -Expected facts: {{case.expected}} - -{{rubric}} - -Reply using this contract: -{{output_contract}} diff --git a/skills/eval-authoring/examples/invoice-extraction/evals/prompts.csv b/skills/eval-authoring/examples/invoice-extraction/evals/prompts.csv deleted file mode 100644 index ad250a53..00000000 --- a/skills/eval-authoring/examples/invoice-extraction/evals/prompts.csv +++ /dev/null @@ -1,4 +0,0 @@ -id,prompt,should_trigger,tags,workspace_subdir,expected -invoice-basic,"Extract the invoice in fixtures/basic.txt as JSON.",true,"outcome,positive",,"{""invoice_number"":""INV-1042"",""date"":""2026-09-01"",""currency"":""USD"",""total"":123.45}" -invoice-missing-number,"Extract the invoice in fixtures/missing-number.txt as JSON. Use null for missing fields.",true,"outcome,boundary",,"{""invoice_number"":null,""date"":""2026-09-02"",""currency"":""EUR"",""total"":78.90}" -invoice-unrelated,"Explain the difference between a receipt and an invoice.",false,"trigger,negative",, diff --git a/skills/eval-authoring/examples/invoice-extraction/evals/review-notes.md b/skills/eval-authoring/examples/invoice-extraction/evals/review-notes.md deleted file mode 100644 index 16a0abe4..00000000 --- a/skills/eval-authoring/examples/invoice-extraction/evals/review-notes.md +++ /dev/null @@ -1,11 +0,0 @@ -# Invoice extraction eval review - -Compatibility: authored for FastSkill 0.9.230; validate against another installed release before use. - -Coverage: ordinary extraction, missing invoice number, and unrelated-request non-triggering. The author confirmed that a missing number must be JSON null. Test-owned expected facts are supplied through the `expected` CSV column and rendered into the judge prompt. Field correctness is an outcome expectation. Skill consultation/non-consultation is an adherence/selection expectation imposed by `should_trigger` semantics. Currency-normalization command text is advisory only. - -Calibration contrasts: JSON null is acceptable for the missing number; the string `"UNKNOWN"` is unacceptable. These labels are based on the confirmed criterion. They have not been executed against a judge in this example. - -Unsupported: ordered OCR-then-normalization calls cannot be asserted by the current deterministic check types. No weaker substitute is claimed. - -Status: example files are syntax-validated without contacting the placeholder judge endpoint. No runtime pilot or judge calibration is claimed. diff --git a/skills/eval-authoring/examples/invoice-extraction/fixtures/basic.txt b/skills/eval-authoring/examples/invoice-extraction/fixtures/basic.txt deleted file mode 100644 index 2f093a08..00000000 --- a/skills/eval-authoring/examples/invoice-extraction/fixtures/basic.txt +++ /dev/null @@ -1,4 +0,0 @@ -Invoice number: INV-1042 -Date: 2026-09-01 -Currency: USD -Total: 123.45 diff --git a/skills/eval-authoring/examples/invoice-extraction/fixtures/missing-number.txt b/skills/eval-authoring/examples/invoice-extraction/fixtures/missing-number.txt deleted file mode 100644 index cb81df95..00000000 --- a/skills/eval-authoring/examples/invoice-extraction/fixtures/missing-number.txt +++ /dev/null @@ -1,3 +0,0 @@ -Date: 2026-09-02 -Currency: EUR -Total: 78.90 diff --git a/skills/eval-authoring/examples/invoice-extraction/skill-project.toml b/skills/eval-authoring/examples/invoice-extraction/skill-project.toml deleted file mode 100644 index 0e27dbdc..00000000 --- a/skills/eval-authoring/examples/invoice-extraction/skill-project.toml +++ /dev/null @@ -1,10 +0,0 @@ -[metadata] -id = "invoice-extraction" -version = "1.0.0" - -[tool.fastskill.eval] -prompts = "evals/prompts.csv" -checks = "evals/checks.toml" -trials_per_case = 1 -pass_threshold = 1.0 -fail_on_missing_agent = true diff --git a/skills/eval-authoring/references/examples.md b/skills/eval-authoring/references/examples.md deleted file mode 100644 index 965478b5..00000000 --- a/skills/eval-authoring/references/examples.md +++ /dev/null @@ -1,36 +0,0 @@ -# Authoring examples - -The executable example is in [`../examples/invoice-extraction`](../examples/invoice-extraction). It is compatible with FastSkill 0.9.230. Run `fastskill eval validate` from that directory without credentials; select a runtime only when intentionally checking its availability and observability. - -## Review table - -| Case | Intent | Expected result | Measurement | -| --- | --- | --- | --- | -| `invoice-basic` | Ordinary invoice | Correct number, date, currency, and total | outcome judge; trigger required by `should_trigger=true` | -| `invoice-missing-number` | Missing field | `invoice_number` is null | outcome judge; trigger required | -| `invoice-unrelated` | Unrelated request | Skill is not consulted | adherence/selection through `should_trigger=false` | - -An optional procedure can be recorded without gating: - -```toml -[[check]] -name = "command_contains" -pattern = "normalize_currency" -required = false -cases = ["invoice-basic"] -``` - -Do not describe the case as outcome-only while `should_trigger=true`: the implicit consultation check remains required. If outcome-only scoring is intended and the current format cannot disable the implicit check without contradicting the CSV, record that policy as unsupported rather than changing `should_trigger` deceptively. - -## Contrasting judge outcomes - -Criterion: “The response represents a missing invoice number as JSON null.” - -- Acceptable: `{"invoice_number":null,"date":"2026-09-01"}`. Label reason: the confirmed missing-number policy is null. -- Unacceptable: `{"invoice_number":"UNKNOWN","date":"2026-09-01"}`. Label reason: the sentinel string violates the confirmed policy. - -These labels support review. They are executed calibration evidence only if passed through a supported real judge interface and the resulting artifact is retained. - -## Unsupported expectation - -“Verify that the agent called OCR before parsing and then called currency normalization, in that order.” The current checks can inspect substring presence and aggregate tool-call count, but do not provide a structured ordered-call assertion. Do not substitute two `command_contains` checks and claim order was verified. Keep the intended assertion in review notes as unsupported and propose the engine feature separately. diff --git a/skills/eval-authoring/references/handoff.md b/skills/eval-authoring/references/handoff.md deleted file mode 100644 index 4f054971..00000000 --- a/skills/eval-authoring/references/handoff.md +++ /dev/null @@ -1,31 +0,0 @@ -# Eval suite handoff - -Record each category independently. - -```markdown -## Coverage and decisions -- Target skill and version: -- Coverage outline reviewed: -- Author-confirmed decisions: -- Outcome expectations: -- Adherence expectations: -- Unsupported expectations: - -## Verification -- FastSkill version and compatibility status: -- Configuration validation command/result: -- Runtime observability command/result: -- Judge contrast labels and provenance: -- Executed calibration artifacts, or unverified reason: -- Authorized pilot scope: -- Pilot commands and run directories: -- Target passes/failures/errors/unmeasured evidence: - -## Continuation -- Files changed: -- Preserved unrelated edits: -- Rescore, rejudge, or rerun decision: -- Missing prerequisites and next action: -``` - -Do not collapse these fields into “verified.” A reviewed and parse-valid suite may still be unexecuted; an executed trial may lack a judgment; a target failure may demonstrate a useful test. diff --git a/skills/eval-authoring/references/semantics.md b/skills/eval-authoring/references/semantics.md deleted file mode 100644 index 55cf89b4..00000000 --- a/skills/eval-authoring/references/semantics.md +++ /dev/null @@ -1,31 +0,0 @@ -# FastSkill 0.9.230 eval semantics - -Use this reference while mapping expectations to the maintained Eval Setup page. It highlights decisions that are easy to get wrong; it is not a duplicate schema. - -## Triggering and case scope - -`should_trigger` is executable scoring input in this release. For each case, the engine creates a required `skill_invoked` expectation with the same polarity unless an explicit `skill_invoked` check applies to that case. An explicit applicable check replaces the implicit one. Validation rejects an explicit polarity that contradicts the case's `should_trigger` value. - -Every deterministic check accepts an optional `cases = ["exact-id", ...]` selector. It applies only to those exact case IDs. Scope mixed positive and negative behavior deliberately. - -In a mixed-polarity suite, omit explicit `skill_invoked` checks unless they add necessary specificity. A global `expected = true` check applies to the negative rows and contradicts them; comments do not create per-case polarity. When explicit checks are used, select exact cases and match each selected row's `should_trigger` value. - -`skill_invoked` recognizes a structured `Skill` invocation or, on supported decoders, a tool input that refers to the staged skill-document path. The path form is evidence of consultation, not proof that the agent read or followed the instructions. Measure outcome correctness separately. - -With no checks file—or an empty check list—the implicit `should_trigger` expectation still applies. Process exit alone is not the verdict. - -## Verdicts and evidence - -Required deterministic checks are all-or-nothing for a trial. `required = false` keeps a check observable without gating. Across trials, `pass_threshold` controls the aggregated case result. Judge criteria have their own normalized scores; a judge gates only when it declares `min_score`. - -Validation proves that configured files parse and that file-only contracts are consistent. Runtime validation also checks installed target availability and preflight observability. It does not execute the skill, contact judge endpoints, or establish outcome quality. - -Judge `prompt_file`, `system_prompt_file`, and `retry_prompt_file` paths are relative to the directory containing `checks.toml`. A judge sees only rendered template variables. Unchanged fixture contents are not implied by `{{trial.workspace_diff}}`; provide test-owned expected facts through an extra CSV column and render `{{case.}}` when the rubric needs them. - -`eval run` records target execution evidence. `eval judge` applies declared judges to completed non-error trials. `eval score` reuses saved trace/workspace evidence for deterministic checks. Missing observations, errored trials, and missing judgments are not passes or zero-quality observations. - -## Isolation and authorization - -Default isolation stages the selected skill in a scratch workspace and suppresses ambient skill discovery where the backend supports it. This is measurement isolation, not an OS sandbox. Inspect the recorded isolation fidelity. - -The authoring agent obtains user authorization for concrete model-backed work. FastSkill accepts explicit runtime/case/trial arguments and has no conversational consent state. Scope does not enforce a monetary ceiling. diff --git a/tests/cli/eval_authoring_assets_test.rs b/tests/cli/eval_authoring_assets_test.rs deleted file mode 100644 index 35ff5ba2..00000000 --- a/tests/cli/eval_authoring_assets_test.rs +++ /dev/null @@ -1,197 +0,0 @@ -use super::snapshot_helpers::run_fastskill_command; -use fastskill_evals::{effective_checks, load_checks, load_suite, CheckDefinition}; -use std::fs; -use std::path::{Path, PathBuf}; - -fn repo_root() -> PathBuf { - Path::new(env!("CARGO_MANIFEST_DIR")) - .join("../..") - .canonicalize() - .unwrap() -} - -fn example_root() -> PathBuf { - repo_root().join("skills/eval-authoring/examples/invoice-extraction") -} - -#[test] -fn shipped_authoring_example_validates_and_exercises_every_asset() { - let example = example_root(); - let project: toml::Value = - toml::from_str(&fs::read_to_string(example.join("skill-project.toml")).unwrap()).unwrap(); - assert_eq!( - project["metadata"]["id"].as_str(), - Some("invoice-extraction") - ); - let result = run_fastskill_command(&["eval", "validate", "--json"], Some(&example)); - assert!(result.success, "stderr: {}", result.stderr); - let json: serde_json::Value = serde_json::from_str(&result.stdout).unwrap(); - assert_eq!(json["valid"], true); - assert_eq!(json["case_count"], 3); - assert_eq!(json["check_count"], 1); - assert_eq!(json["judges"][0], "extraction-correctness"); - - let suite = load_suite(&example.join("evals/prompts.csv")).unwrap(); - assert_eq!( - suite - .cases - .iter() - .filter(|case| case.should_trigger) - .count(), - 2 - ); - assert_eq!( - suite - .cases - .iter() - .filter(|case| !case.should_trigger) - .count(), - 1 - ); - let checks = load_checks(&example.join("evals/checks.toml")).unwrap(); - assert_eq!(checks.len(), 1); - for case in &suite.cases { - let effective = effective_checks(&checks, &case.id, case.should_trigger); - let trigger = effective - .iter() - .find_map(|check| match check { - CheckDefinition::SkillInvoked { expected, .. } => Some(*expected), - _ => None, - }) - .expect("every case must get a trigger expectation"); - assert_eq!(trigger, case.should_trigger); - if case.should_trigger { - assert!( - case.extra - .get("expected") - .is_some_and(|value| value.contains("invoice_number")), - "positive case {} needs judge-visible expected facts", - case.id - ); - } - } - - let basic = fs::read_to_string(example.join("fixtures/basic.txt")).unwrap(); - assert!(basic.contains("INV-1042") && basic.contains("USD") && basic.contains("123.45")); - let missing = fs::read_to_string(example.join("fixtures/missing-number.txt")).unwrap(); - assert!(!missing.contains("Invoice number:")); - assert!(missing.contains("EUR") && missing.contains("78.90")); - let prompt = fs::read_to_string(example.join("evals/judge-prompt.md")).unwrap(); - assert!(prompt.contains("{{rubric}}") && prompt.contains("{{output_contract}}")); - assert!(prompt.contains("{{case.expected}}")); - let notes = fs::read_to_string(example.join("evals/review-notes.md")).unwrap(); - assert!(notes.contains("Unsupported:") && notes.contains("not been executed")); -} - -#[test] -fn contradictory_explicit_trigger_check_fails_for_the_intended_reason() { - let source = example_root(); - let temp = tempfile::tempdir().unwrap(); - copy_tree(&source, temp.path()); - fs::write( - temp.path().join("evals/checks.toml"), - "[[check]]\nname = \"skill_invoked\"\nexpected = false\ncases = [\"invoice-basic\"]\n", - ) - .unwrap(); - let result = run_fastskill_command(&["eval", "validate"], Some(temp.path())); - assert!(!result.success); - let evidence = format!("{}{}", result.stdout, result.stderr); - assert!(evidence.contains("EVAL_CHECKS_INVALID"), "{evidence}"); - assert!(evidence.contains("invoice-basic"), "{evidence}"); -} - -#[test] -fn skill_references_and_local_links_are_complete() { - let root = repo_root(); - let skill_root = root.join("skills/eval-authoring"); - let skill = fs::read_to_string(skill_root.join("SKILL.md")).unwrap(); - assert!(skill.starts_with("---\nname: eval-authoring\n")); - for relative in [ - "references/semantics.md", - "references/examples.md", - "references/handoff.md", - "examples/invoice-extraction/SKILL.md", - ] { - assert!(skill_root.join(relative).is_file(), "missing {relative}"); - } - for relative in [ - "webdocs/evals-quality/setup.mdx", - "webdocs/cli-reference/eval-command.mdx", - ] { - assert!(root.join(relative).is_file(), "missing {relative}"); - } - assert!(skill.contains("FastSkill `0.9.230`")); - assert!(skill.contains("outcome") && skill.contains("adherence")); -} - -#[test] -fn purpose_built_authoring_fixtures_cover_required_branches() { - let fixtures = repo_root().join("tests/fixtures/eval-authoring"); - let existing = fixtures.join("existing-suite"); - let validation = run_fastskill_command(&["eval", "validate", "--json"], Some(&existing)); - assert!(validation.success, "stderr: {}", validation.stderr); - let validation_json: serde_json::Value = serde_json::from_str(&validation.stdout).unwrap(); - assert_eq!(validation_json["valid"], true); - assert_eq!(validation_json["case_count"], 1); - for relative in [ - "invoice-no-evals/SKILL.md", - "invoice-no-evals/fixtures/basic.txt", - "invoice-no-evals/fixtures/missing-number.txt", - "invoice-no-evals/fixtures/ambiguous-date.txt", - "invoice-no-evals/fixtures/multiple-currencies.txt", - "defective-total/SKILL.md", - "defective-total/ground-truth.md", - "unnecessary-step/SKILL.md", - "unnecessary-step/scripted-answers.md", - "unsupported-order/SKILL.md", - "existing-suite/evals/user-note.md", - "environment-variants.md", - "scripted-workflow.md", - ] { - assert!(fixtures.join(relative).is_file(), "missing {relative}"); - } - - let ground_truth = - fs::read_to_string(fixtures.join("defective-total/ground-truth.md")).unwrap(); - assert!(ground_truth.contains("17.00") && ground_truth.contains("15.00")); - let answers = - fs::read_to_string(fixtures.join("unnecessary-step/scripted-answers.md")).unwrap(); - assert!(answers.contains("not mandatory") && answers.contains("outcome correctness wins")); - let unsupported = fs::read_to_string(fixtures.join("unsupported-order/SKILL.md")).unwrap(); - assert!(unsupported.contains("exact tool-call order")); - let note = fs::read_to_string(fixtures.join("existing-suite/evals/user-note.md")).unwrap(); - assert_eq!( - note, - "# User note\n\nKeep this unrelated note byte-for-byte when refining the suite.\n" - ); - let variants = fs::read_to_string(fixtures.join("environment-variants.md")).unwrap(); - for branch in [ - "incompatible CLI", - "missing runtime", - "missing judge credential", - "invalid reference", - "unavailable observation", - ] { - assert!( - variants.contains(branch), - "missing controlled branch {branch}" - ); - } - let workflow = fs::read_to_string(fixtures.join("scripted-workflow.md")).unwrap(); - for step in 1..=8 { - assert!(workflow.contains(&format!("{step}. ")), "missing WF{step}"); - } -} - -fn copy_tree(source: &Path, target: &Path) { - for entry in walkdir::WalkDir::new(source) { - let entry = entry.unwrap(); - let relative = entry.path().strip_prefix(source).unwrap(); - let destination = target.join(relative); - if entry.file_type().is_dir() { - fs::create_dir_all(&destination).unwrap(); - } else { - fs::copy(entry.path(), destination).unwrap(); - } - } -} diff --git a/tests/cli/mod.rs b/tests/cli/mod.rs index cdcf704c..74affc7e 100644 --- a/tests/cli/mod.rs +++ b/tests/cli/mod.rs @@ -8,7 +8,6 @@ pub mod analyze_cluster_tests; pub mod analyze_provider_gate_tests; pub mod config_tests; pub mod doctor_tests; -pub mod eval_authoring_assets_test; pub mod eval_tests; pub mod example_tests; pub mod help_tests; diff --git a/tests/fixtures/eval-authoring/defective-total/SKILL.md b/tests/fixtures/eval-authoring/defective-total/SKILL.md deleted file mode 100644 index 19b88d5a..00000000 --- a/tests/fixtures/eval-authoring/defective-total/SKILL.md +++ /dev/null @@ -1,8 +0,0 @@ ---- -name: defective-total -description: Add line items from a text invoice and return the numeric total. ---- - -# Total an invoice - -Add every line item, but deliberately omit the final line item. This defect is intentional test ground truth; a correctly authored eval must detect it rather than adopt it as the expected outcome. diff --git a/tests/fixtures/eval-authoring/defective-total/ground-truth.md b/tests/fixtures/eval-authoring/defective-total/ground-truth.md deleted file mode 100644 index 2a58a825..00000000 --- a/tests/fixtures/eval-authoring/defective-total/ground-truth.md +++ /dev/null @@ -1,3 +0,0 @@ -# Test-owned ground truth - -For inputs `10.00`, `5.00`, and `2.00`, the independently calculated total is `17.00`. The target instruction deliberately produces `15.00` by omitting the final item. This file is not copied into the target execution fixture. diff --git a/tests/fixtures/eval-authoring/environment-variants.md b/tests/fixtures/eval-authoring/environment-variants.md deleted file mode 100644 index 99f052e6..00000000 --- a/tests/fixtures/eval-authoring/environment-variants.md +++ /dev/null @@ -1,9 +0,0 @@ -# Controlled prerequisite variants - -Use these deterministic branches in workflow exercises: - -- incompatible CLI: stub `fastskill --version` as `0.8.0`; retain files and mark compatibility unverified; -- missing runtime: select an unavailable runtime and retain the parse-valid suite; -- missing judge credential: unset the declared `api_key_env`; do not treat authoring-agent authentication as judge authentication; -- invalid reference: point `checks` at malformed TOML, capture validation failure, repair it, and rerun validation; -- unavailable observation: select a backend that preflight identifies as unable to observe a required check; report exclusion/error rather than pass. diff --git a/tests/fixtures/eval-authoring/existing-suite/SKILL.md b/tests/fixtures/eval-authoring/existing-suite/SKILL.md deleted file mode 100644 index 14d77f70..00000000 --- a/tests/fixtures/eval-authoring/existing-suite/SKILL.md +++ /dev/null @@ -1,8 +0,0 @@ ---- -name: existing-suite -description: Summarize release notes into concise bullets. ---- - -# Summarize release notes - -Preserve version numbers and breaking-change warnings. diff --git a/tests/fixtures/eval-authoring/existing-suite/evals/checks.toml b/tests/fixtures/eval-authoring/existing-suite/evals/checks.toml deleted file mode 100644 index 3c2bb2f0..00000000 --- a/tests/fixtures/eval-authoring/existing-suite/evals/checks.toml +++ /dev/null @@ -1,4 +0,0 @@ -[[check]] -name = "command_contains" -pattern = "BREAKING" -cases = ["release-basic"] diff --git a/tests/fixtures/eval-authoring/existing-suite/evals/prompts.csv b/tests/fixtures/eval-authoring/existing-suite/evals/prompts.csv deleted file mode 100644 index 4e000d5a..00000000 --- a/tests/fixtures/eval-authoring/existing-suite/evals/prompts.csv +++ /dev/null @@ -1,2 +0,0 @@ -id,prompt,should_trigger,tags -release-basic,Summarize fixtures/release.md,true,smoke diff --git a/tests/fixtures/eval-authoring/existing-suite/evals/user-note.md b/tests/fixtures/eval-authoring/existing-suite/evals/user-note.md deleted file mode 100644 index 4f6316ba..00000000 --- a/tests/fixtures/eval-authoring/existing-suite/evals/user-note.md +++ /dev/null @@ -1,3 +0,0 @@ -# User note - -Keep this unrelated note byte-for-byte when refining the suite. diff --git a/tests/fixtures/eval-authoring/existing-suite/fixtures/release.md b/tests/fixtures/eval-authoring/existing-suite/fixtures/release.md deleted file mode 100644 index 7925dd08..00000000 --- a/tests/fixtures/eval-authoring/existing-suite/fixtures/release.md +++ /dev/null @@ -1,3 +0,0 @@ -Version 2.0.0 - -BREAKING: the legacy endpoint was removed. diff --git a/tests/fixtures/eval-authoring/existing-suite/skill-project.toml b/tests/fixtures/eval-authoring/existing-suite/skill-project.toml deleted file mode 100644 index 2faeea91..00000000 --- a/tests/fixtures/eval-authoring/existing-suite/skill-project.toml +++ /dev/null @@ -1,3 +0,0 @@ -[tool.fastskill.eval] -prompts = "evals/prompts.csv" -checks = "evals/checks.toml" diff --git a/tests/fixtures/eval-authoring/invoice-no-evals/SKILL.md b/tests/fixtures/eval-authoring/invoice-no-evals/SKILL.md deleted file mode 100644 index 0d1eb075..00000000 --- a/tests/fixtures/eval-authoring/invoice-no-evals/SKILL.md +++ /dev/null @@ -1,8 +0,0 @@ ---- -name: invoice-no-evals -description: Extract known invoice fields into JSON when asked to process a plain-text invoice. ---- - -# Invoice extraction - -Return `invoice_number`, `date`, `currency`, and `total`. If the invoice number is absent, use JSON null. An ambiguous numeric date must be returned as `ambiguous` rather than guessed. Do not activate for general questions about invoices. diff --git a/tests/fixtures/eval-authoring/invoice-no-evals/fixtures/ambiguous-date.txt b/tests/fixtures/eval-authoring/invoice-no-evals/fixtures/ambiguous-date.txt deleted file mode 100644 index 0293fcac..00000000 --- a/tests/fixtures/eval-authoring/invoice-no-evals/fixtures/ambiguous-date.txt +++ /dev/null @@ -1,4 +0,0 @@ -Invoice: INV-8 -Date: 03/04/2026 -Currency: GBP -Total: 90.00 diff --git a/tests/fixtures/eval-authoring/invoice-no-evals/fixtures/basic.txt b/tests/fixtures/eval-authoring/invoice-no-evals/fixtures/basic.txt deleted file mode 100644 index 5a23e7df..00000000 --- a/tests/fixtures/eval-authoring/invoice-no-evals/fixtures/basic.txt +++ /dev/null @@ -1,4 +0,0 @@ -Invoice: INV-7 -Date: 2026-09-12 -Currency: AED -Total: 250.00 diff --git a/tests/fixtures/eval-authoring/invoice-no-evals/fixtures/missing-number.txt b/tests/fixtures/eval-authoring/invoice-no-evals/fixtures/missing-number.txt deleted file mode 100644 index f52836bf..00000000 --- a/tests/fixtures/eval-authoring/invoice-no-evals/fixtures/missing-number.txt +++ /dev/null @@ -1,3 +0,0 @@ -Date: 2026-09-10 -Currency: USD -Total: 18.25 diff --git a/tests/fixtures/eval-authoring/invoice-no-evals/fixtures/multiple-currencies.txt b/tests/fixtures/eval-authoring/invoice-no-evals/fixtures/multiple-currencies.txt deleted file mode 100644 index 1334cebc..00000000 --- a/tests/fixtures/eval-authoring/invoice-no-evals/fixtures/multiple-currencies.txt +++ /dev/null @@ -1,5 +0,0 @@ -Invoice: INV-9 -Date: 2026-09-09 -Line item: USD 10.00 -Tax: EUR 2.00 -Total: USD 12.00 diff --git a/tests/fixtures/eval-authoring/scripted-workflow.md b/tests/fixtures/eval-authoring/scripted-workflow.md deleted file mode 100644 index da06b59e..00000000 --- a/tests/fixtures/eval-authoring/scripted-workflow.md +++ /dev/null @@ -1,12 +0,0 @@ -# Scripted author input for WF1-WF8 - -1. Start with `invoice-no-evals` and ask for evaluation tests. -2. Confirm that a missing invoice number must be JSON null. -3. Confirm that `03/04/2026` is ambiguous and must not be guessed. -4. Review coverage for ordinary, missing-number, ambiguous-date, multiple-currency, and unrelated cases. -5. Require outcome correctness for extracted fields; require consultation for extraction requests and non-consultation for unrelated requests. -6. Authorize six cases, one trial each, only after the authoring agent names the selected runtime and model ID. -7. Seed an ambiguous-date target failure and retain it as evidence. -8. Clarify the date criterion and rerun or rejudge only the affected evidence as supported. - -The authoring agent must preserve an existing suite's `evals/user-note.md`, report the unsupported ordered-call assertion without substitution, and hand off through files. diff --git a/tests/fixtures/eval-authoring/unnecessary-step/SKILL.md b/tests/fixtures/eval-authoring/unnecessary-step/SKILL.md deleted file mode 100644 index cfe15ba6..00000000 --- a/tests/fixtures/eval-authoring/unnecessary-step/SKILL.md +++ /dev/null @@ -1,8 +0,0 @@ ---- -name: unnecessary-step -description: Convert temperatures between Celsius and Fahrenheit. ---- - -# Convert temperatures - -Write an intermediate value to `scratch-temperature.txt`, then calculate the requested conversion and return the answer. diff --git a/tests/fixtures/eval-authoring/unnecessary-step/scripted-answers.md b/tests/fixtures/eval-authoring/unnecessary-step/scripted-answers.md deleted file mode 100644 index 4f88037e..00000000 --- a/tests/fixtures/eval-authoring/unnecessary-step/scripted-answers.md +++ /dev/null @@ -1,5 +0,0 @@ -# Scripted author decisions - -- The correct converted temperature is mandatory. -- Writing `scratch-temperature.txt` is not mandatory and must be treated as advisory adherence, not outcome correctness. -- If the instruction conflicts with the desired clean workspace, outcome correctness wins and the conflict must be reported. diff --git a/tests/fixtures/eval-authoring/unsupported-order/SKILL.md b/tests/fixtures/eval-authoring/unsupported-order/SKILL.md deleted file mode 100644 index ca4dfab5..00000000 --- a/tests/fixtures/eval-authoring/unsupported-order/SKILL.md +++ /dev/null @@ -1,8 +0,0 @@ ---- -name: unsupported-order -description: Extract text from scanned receipts and normalize their currency values. ---- - -# Process receipts - -Call OCR first and currency normalization second. The author requires this exact tool-call order, which the FastSkill 0.9.230 deterministic checks cannot represent faithfully. diff --git a/webdocs/evals-quality/setup.mdx b/webdocs/evals-quality/setup.mdx index 7d1414e9..357ae164 100644 --- a/webdocs/evals-quality/setup.mdx +++ b/webdocs/evals-quality/setup.mdx @@ -7,14 +7,19 @@ description: "Author an eval suite: the [tool.fastskill.eval] manifest table, th This page is the reference for **authoring** an eval suite: the config table in `skill-project.toml`, the prompts CSV a suite is made of, and the optional checks TOML that actually decides pass/fail. Work through it top to bottom once, then use [Run evals](/evals-quality/run-evals) for the day-to-day commands. +For guided suite design, use the [FastSkill skill](https://github.com/gofastskill/skill/tree/main/fastskill). +Its [eval-authoring workflow](https://github.com/gofastskill/skill/blob/main/fastskill/references/eval-authoring.md) +and copyable invoice example are maintained and packaged in the skills repository. + ## Prerequisites - `skill-project.toml` exists (project root or skill directory — `fastskill` walks up the tree to find it). - `SKILL.md` exists alongside it if you're authoring a single skill's suite. - For `fastskill eval run`: a supported agent CLI on `PATH`. Agent keys come from aikit-sdk: `aikit`, `claude`, `codex`, `cursor`, `gemini`, and `pi`. A key is accepted only when its CLI is detected on the current machine; a typo and an uninstalled agent both error with `RUNTIME_UNKNOWN_ID`. `fastskill eval validate --all` reports which keys are detected in *your* environment. -Evals do not require embedding or `OPENAI_API_KEY`; those are used by `index rebuild` and -`skill search --local`. +Deterministic validation and scoring need no model credential. Target execution needs the +selected runtime's authentication; native judges separately need their configured endpoint, +model, and credential environment variable. Embedding configuration is unrelated. ## 1. The `[tool.fastskill.eval]` table