diff --git a/CONTEXT.md b/CONTEXT.md index 3783eb48..82af3647 100644 --- a/CONTEXT.md +++ b/CONTEXT.md @@ -206,6 +206,23 @@ _Avoid_: Sweep, report (a rendering), summary (one run directory's, upstream), r A rendering of measurements already made — one run directory's summary, or one or more scorecards — as a table, JSON, or one self-contained HTML file with every asset embedded. Producing a report never runs an agent and never re-scores; it reads scorecard files, not run directories, so it can be made anywhere the files are. _Avoid_: Dashboard, page, export, scorecard (the measurement it renders) +### Eval authoring + +**Eval-authoring skill**: +A skill that guides an authoring agent through designing evaluation suites according to FastSkill's layouts and standards. It is distinct from the target skill being evaluated. + +**Authoring agent**: +The user-chosen agentic tool that helps design and edit an evaluation suite. It need not be the agent evaluated by that suite. + +**Outcome correctness**: +Whether the result of a skill-assisted task satisfies its intended acceptance criteria, independently of the prescribed method. + +**Skill adherence**: +Whether the agent follows the skill's prescribed method. It is distinct from outcome correctness and is mandatory only where the author identifies the method as a requirement. + +**Coverage outline**: +A proposed selection of behaviors and boundaries to evaluate for a skill, before detailed cases and graders are authored. It is not a Benchmark, which identifies an executable measurement definition. + ## Resolved decisions - **Historical `sync` is removed.** Modern targets read skills directly. Propagation has exactly two diff --git a/crates/fastskill-cli/src/main.rs b/crates/fastskill-cli/src/main.rs index 50c685eb..d657e469 100644 --- a/crates/fastskill-cli/src/main.rs +++ b/crates/fastskill-cli/src/main.rs @@ -8,6 +8,7 @@ //! `Arc` is captured at registration time by each command closure — //! no `Any`-downcasting of `AppContext` is needed. +#![recursion_limit = "256"] #![allow(clippy::unwrap_used, clippy::expect_used, clippy::panic)] mod arg_helpers; diff --git a/crates/fastskill-evals/tests/documented_check_contract.rs b/crates/fastskill-evals/tests/documented_check_contract.rs index 17a35ca1..b10f0bad 100644 --- a/crates/fastskill-evals/tests/documented_check_contract.rs +++ b/crates/fastskill-evals/tests/documented_check_contract.rs @@ -7,7 +7,9 @@ #![allow(clippy::expect_used, clippy::unwrap_used)] -use fastskill_evals::{run_checks, suite_passes, ChecksToml}; +use fastskill_evals::{ + effective_checks, run_checks, suite_passes, validate_case_checks, CheckDefinition, ChecksToml, +}; use std::path::Path; /// `max_command_count` is a legacy alias; results report the canonical @@ -175,3 +177,67 @@ required = true Path::new("/tmp") ))); } + +/// The setup guide's central authoring contract: `should_trigger` creates a +/// required per-case expectation even when checks.toml contributes nothing. +#[test] +fn should_trigger_generates_positive_and_negative_required_checks() { + for expected in [true, false] { + let checks = effective_checks(&[], "case-1", expected); + assert_eq!(checks.len(), 1); + assert!(matches!(checks[0], CheckDefinition::SkillInvoked { .. })); + if let CheckDefinition::SkillInvoked { + expected: actual, + required, + cases, + .. + } = &checks[0] + { + assert_eq!(*actual, expected); + assert!(*required); + assert_eq!(cases.as_deref(), Some(["case-1".to_string()].as_slice())); + } + } +} + +/// Exact case selectors control applicability, and an explicit trigger check +/// replaces the implicit one only on the selected case. A contradictory +/// replacement is rejected rather than silently changing CSV intent. +#[test] +fn case_scoping_and_explicit_trigger_precedence_match_the_docs() { + let parsed: ChecksToml = toml::from_str( + r#" +[[check]] +name = "skill_invoked" +expected = true +cases = ["positive"] + +[[check]] +name = "command_contains" +pattern = "invoice_number" +cases = ["positive"] +"#, + ) + .unwrap(); + + let positive = effective_checks(&parsed.checks, "positive", true); + assert_eq!( + positive.len(), + 2, + "explicit trigger plus scoped outcome check" + ); + let negative = effective_checks(&parsed.checks, "negative", false); + assert_eq!(negative.len(), 1, "other-case checks must be excluded"); + assert!(matches!( + negative[0], + CheckDefinition::SkillInvoked { + expected: false, + .. + } + )); + + assert!(validate_case_checks(&parsed.checks, "positive", true).is_ok()); + let error = validate_case_checks(&parsed.checks, "positive", false).unwrap_err(); + assert!(error.to_string().contains("EVAL_CHECKS_INVALID")); + assert!(error.to_string().contains("positive")); +} diff --git a/docs/adr/0011-skill-first-eval-authoring.md b/docs/adr/0011-skill-first-eval-authoring.md new file mode 100644 index 00000000..44d4842a --- /dev/null +++ b/docs/adr/0011-skill-first-eval-authoring.md @@ -0,0 +1,22 @@ +# Start assisted eval authoring from the skill + +Status: accepted. Date: 2026-09-12. + +## Context + +Designing meaningful eval cases requires human attention to intended outcomes, coverage, and grading criteria. The initial authoring experience could start from a skill that needs tests or from an observed session that should become a regression case. These solve different entry-point problems. + +## Decision + +The first assisted-authoring experience starts from an existing skill and helps its author design the skill's evaluation tests. It does not require session history or a known failure. + +Inspecting sessions to discover regression cases or improve skills is deferred for a separate discussion. It is not a prerequisite or committed deliverable of the initial change. + +This decision settles scope only. It does not select the authoring model, interaction style, suite representation, calibration method, acceptance workflow, or implementation ownership. + +## Consequences + +- Initial design and acceptance criteria must cover a skill with no prior usage sessions. +- Session import adapters and transcript reconstruction are outside the initial scope. +- The next decisions must establish how intended behavior is clarified and how generated cases and graders are validated. +- Future session-derived cases can be considered independently without delaying skill-first authoring. diff --git a/docs/adr/0012-separate-outcomes-from-adherence.md b/docs/adr/0012-separate-outcomes-from-adherence.md new file mode 100644 index 00000000..05d28bac --- /dev/null +++ b/docs/adr/0012-separate-outcomes-from-adherence.md @@ -0,0 +1,7 @@ +# Separate outcome correctness from skill adherence + +Status: accepted. Date: 2026-09-12. + +Generated evals must distinguish a useful, correct outcome from compliance with a skill's prescribed method. Measure these separately and make adherence mandatory only when the author identifies it as a requirement. Automatically treating every instruction as correctness would encode existing mistakes and could reward unnecessary tool use; treating outcomes alone as sufficient would miss genuinely required procedures. + +This decision does not choose the storage schema or change the current engine's implicit trigger checks. Implementation must reconcile those existing checks with the authoring distinction before claiming support. Pilot success establishes evidence about cases and graders, not that the skill passes the suite. diff --git a/docs/adr/0013-skill-driven-eval-authoring.md b/docs/adr/0013-skill-driven-eval-authoring.md new file mode 100644 index 00000000..e5e8b10f --- /dev/null +++ b/docs/adr/0013-skill-driven-eval-authoring.md @@ -0,0 +1,15 @@ +# Drive eval authoring through a skill and the CLI + +Status: accepted. Date: 2026-09-12. + +Users design evals in their chosen agentic tool with FastSkill CLI and an eval-authoring skill that explains its layouts and standards. The authoring skill guides inspection, coverage discussion, case construction, and iteration; FastSkill and its eval engine define and validate artifacts, execute evaluations, and preserve measurement evidence. This keeps conversational judgment in the user's agent and enforceable mechanics in code, avoiding a separate model integration and interview orchestrator inside FastSkill. + +Drafts use ordinary files and version control. A dedicated interview engine and draft-state service are deferred. Requirements that CI relies on must be represented in artifacts and checked by code; instructions in the authoring skill alone cannot establish verified status. + +Authoring-tool choice is independent of evaluation-runtime support. An agent able to follow the skill, edit files, and invoke the CLI can author a suite without having a FastSkill evaluation backend. Executing that agent as the subject of an eval still requires a supported backend and observable evidence. + +The authoring agent owns conversational authorization: it uses the user's existing authorization or asks about pilot scope before invoking the CLI. FastSkill validates explicit parameters and returns results or errors; it does not infer consent, remember conversational approvals, or conduct an interview. A spending ceiling is not a prerequisite of this change and must not be promised unless implemented. + +The skill checks CLI compatibility and follows authoritative format references and validation output. Exact supported versions and distribution location remain implementation details to investigate. Review covers the outline and then the suite, with individual questions for ambiguities. Calibration labels follow author-confirmed criteria or independently verifiable facts. + +The first delivery is the authoring skill with supporting documentation and validation corrections for existing contracts. Unsupported test intentions are reported explicitly; new grader types, schemas, or execution capabilities require separate proposals. The PRDs are drafts for external review, not a claim of implementation readiness or completed verification. diff --git a/docs/qa/eval-authoring-verification.md b/docs/qa/eval-authoring-verification.md new file mode 100644 index 00000000..00c458b1 --- /dev/null +++ b/docs/qa/eval-authoring-verification.md @@ -0,0 +1,174 @@ +# Eval-authoring verification record + +Status: implementation and repository-consolidation local gates verified; draft PR CI tracked separately. Updated: 2026-09-15. + +## Repository consolidation + +The installable authoring workflow belongs to `gofastskill/skill`, inside its existing +`fastskill/` package. The CLI PR #333 removes the former `skills/eval-authoring/` tree, +authoring fixtures, and Rust asset-test module after moving them to the skills repository. +There, `fastskill/SKILL.md` routes to `fastskill/references/eval-authoring.md`; references +and the invoice example are packaged with it. Scenario fixtures are under +`evals/authoring/fixtures/`, and executable asset checks are under `scripts/`. + +The existing skills-repository `evals/` content measures the FastSkill skill itself. +Its useful suites, generator, metrics, fixtures, and historical baseline are preserved. +The companion assessment records obsolete claims and harness defects separately from +verified behavior. No historical baseline is relabeled as a measurement of this new package. + +Companion [skills PR #30](https://github.com/gofastskill/skill/pull/30) contains the +relocated package and its exact 42-file coverage inventory. On server02, 12 Python tests, +namespace checks, regenerated-suite parity, and the payload vacuity guard passed. +A real ZIP installed as version 2.1.0; its copied invoice example and all three existing +v2 suites validated. Claude Sonnet 4.5 extended a suite through the explicitly supplied +installed entrypoint; caller validation passed and original files/positive row were preserved. +All relocated fixture bytes and the installed payload were checked unchanged. This scoped +exercise is not a new full live baseline or proof of automatic skill discovery. + +The repaired Bash runner measured 93.1% line coverage; staging measured 100% Bash lines +and its exact embedded Python measured 100% statements. Bash branch instrumentation is +unavailable, not represented as passing. Six engine contract tests and 13 CLI +documentation/related tests passed again after relocation, as did formatting, source-size, +and diff checks. Earlier full regression, CLI coverage, two-tool authoring, and native +judge evidence remain applicable to unchanged implementation and assets. Raw relocation +evidence is retained beside this checkout under `work/`, including +`relocation-authoring-result.json`, `package-smoke-hHkVYA/`, `shell-coverage-v2/`, and +`embedded-coverage-6f4x9rtk/`. + +The inventory below records the original implementation paths and evidence. Moved-file +coverage now belongs to the skills repository's consolidation record. This CLI PR retains: + +| Current changed file | Verification | +| --- | --- | +| `CONTEXT.md` | existing authoring vocabulary and requirement mapping | +| `crates/fastskill-cli/src/main.rs` | recorded 91.22% line / 96.00% branch coverage; recursion-limit-only change | +| `crates/fastskill-evals/tests/documented_check_contract.rs` | six engine contract tests remain in the owning repository | +| `docs/adr/0011-skill-first-eval-authoring.md` | original skill-first workflow evidence | +| `docs/adr/0012-separate-outcomes-from-adherence.md` | original conversion/defect contrasts | +| `docs/adr/0013-skill-driven-eval-authoring.md` | original two-agent authoring and file handoff | +| `docs/requirements/eval-authoring-proposal.md` | original requirement mapping | +| `docs/requirements/eval-authoring-skill-prd.md` | original workflow evidence; package integration checked in companion repository | +| `docs/requirements/eval-authoring-support-prd.md` | ownership clarified; link and documentation checks | +| `docs/qa/eval-authoring-verification.md` | historical evidence retained; current ownership and limitations explicit | +| `webdocs/evals-quality/setup.mdx` | corrected semantics, credential boundary, and distribution links | + +## Current evidence (supersedes historical status tables below) + +- The imported Linux worktree was verified on `server02` at branch `feat/eval-authoring-skill`, checkpoint `11e673c24c1dc5233f4d5162788f18886d2ca168`. Fetching origin showed the local branch and `origin/feat/eval-authoring-skill` at the same commit with a clean worktree. +- Both authoring tools ran: Codex CLI 0.153.4 with `gpt-5.6-luna`, and Claude Code 2.1.269 with `claude-sonnet-4-5`. Generated suites validated against pinned FastSkill 0.9.230. Initial Claude attempts required trigger-scope, judge-path, schema, and template repairs. +- Claude continued the Codex-authored suite using files, repaired extra CSV columns and answer leakage, validated four cases/four checks, and recorded why changed prompts require new target evidence. Handoff cost reported by Claude: USD 0.2934362. Raw record: `work/claude-handoff.json` beside this checkout. +- A real Claude Sonnet 4.5 defect pilot consulted the target, returned `15.00`, and failed the independent `17.00` expectation with exit code zero and no execution error. Changing only the target to include the final item produced a passing rerun with the same criterion. Rescoring the corrected pilot's saved artifacts also passed without another model call. +- The conversion pilot passed the numeric outcome check, failed mandatory consultation, and failed the scratch-file check without gating because it was advisory. This demonstrates distinct outcome/adherence results, not a fully passing conversion suite. +- On `server02`, pinned FastSkill now reports `aikit`, `claude`, and `codex` available for the shipped example. This supersedes the earlier Windows `RUNTIME_UNKNOWN_ID` result for Codex while preserving it as historical prerequisite evidence. +- The six documented contract tests and four CLI asset tests pass on Linux. The four asset tests exercise the shipped suite, references and links, every purpose-built fixture branch, and the controlled contradiction failure. +- Repository-recommended process-isolated regression passed all 1,872 tests with one configured skip before the Clippy fix. After the one-line fix, 1,871 of 1,872 passed in one run; the sole failure read a stale shared `market-success` cache and passed when rerun with a fresh documented `FASTSKILL_CACHE_DIR`. The ordinary in-process Cargo runner produced current-directory cascades, while the same first failure passed alone; nextest is the valid repository runner. +- Strict workspace Clippy initially failed because current nightly promotes the recursion-depth diagnostic in unchanged `crates/fastskill-cli/src/main.rs` under `-D warnings`. Adding the compiler-recommended crate recursion limit fixed the gate. `cargo clippy --workspace --all-targets --all-features --locked -- -D warnings`, `cargo fmt --all -- --check`, `bash scripts/check-source-size.sh`, and `git diff --check` now pass. +- Instrumented FastSkill CLI coverage ran 897 tests with one configured skip and no failures. The changed production entrypoint has 686/752 covered lines (91.22%) and 48/50 covered branches (96.00%), satisfying the PRD's 90% line and 80% branch thresholds. Raw summary: `work/coverage-fastskill-cli.json`. +- Server02 uses Python 3.10 while the docs checker imports Python 3.11's `tomllib`. A `tomli` compatibility package was supplied only through `work/python310-compat`; the previously blocked documentation parity test then passed without repository-source changes. +- Native judge calibration used the owner-approved OpenAI-compatible gateway at `https://llm-gateway.tailb1f947.ts.net/v1`, credential variable `OPENAI_API_KEY`, and the gateway's sole advertised model, `tools-advanced`. With the same criterion and a 4,096-token response allowance, the preserved `15.00` defect scored 0.0 and failed, while the corrected `17.00` result scored 1.0 and passed; both completed with zero judge errors. Evidence is retained under `work/native-judge/` beside this checkout. +- The saved summaries recorded an absolute Windows skill-project path. Direct Linux judging failed with `EVAL_CONFIG_MISSING`; the global `--skills-dir` option does not relocate the recorded project root. Calibration therefore used untouched originals plus copied run artifacts whose `skill_project_root` alone was repaired to the preserved Linux project. The initial 1,024-token corrected attempt was retained as a judge-error example (`finish_reason` `length`) and was not counted as passing evidence. + +Draft PR #333 contains the CLI work. Historical environment blockers below are retained as attempt history, not current claims. + +This record keeps deterministic checks, live authoring exercises, and release gaps distinct. It must not be read as a release attestation while any mandatory row is incomplete. + +## Compatibility and confirmed validation behavior + +- Intended supported baseline: FastSkill 0.9.230 at `dd983e89e5fee977325b77ae386b879e8310ed26`, with `aikit-evals` pinned to `eb50f2c5509d4345cab65af0f7faed012ab0e557`. +- Local installed binary: FastSkill 0.9.208. It parsed the shipped invoice example with `fastskill eval validate --json`, reporting three cases and one deterministic check, but it predates the declared baseline and therefore does not establish 0.9.230 behavioral compatibility. +- Confirmed source contracts at the pin: per-case implicit trigger checks from `should_trigger`; exact `cases` selectors; explicit applicable `skill_invoked` precedence; contradiction rejection; required/advisory behavior; path-reference consultation proxy. +- No new validation defect requiring production-source changes was identified. The existing engine and CLI already enforce the material contracts; this change repairs maintained guidance and adds contract tests. + +## Requirement-to-test mapping + +| Requirements | Evidence | Status | +| --- | --- | --- | +| WF1, A1, AT1, ST4, ST6 | `invoice-no-evals`; compatibility/help inspection; generated Claude and Codex suites; pinned validator | passed | +| WF2, A2, AT3 | scripted outline/clarification record and unnecessary-step author decision; asset assertions | passed | +| WF3-WF4, A3-A5, AT4-AT5, ST1-ST5 | shipped example; suite-level review notes; semantics/examples references; six executed contract tests | passed | +| WF5, A6-A7, AT6 | pinned `eval validate`, malformed-suite repair, controlled contradiction, and executed native-judge contrast | passed; 15.00 rejected and 17.00 accepted by the same criterion | +| WF6, A8, AT9-AT10, ST8 | explicit pilot scope; saved authorized pilot; scripted unauthorized branch | passed | +| WF7, A9, AT7-AT8, ST7 | real defective and corrected target runs; prerequisite variants; separate outcome/adherence/missing evidence | passed | +| WF8, A10, AT11 | Claude continued and repaired Codex-authored files; corrected rerun and saved-evidence rescore; handoff record | passed | +| ST2-ST3 | pinned suite validation; six contract tests; four asset tests including intended contradiction failure | passed | + +## Original changed-file coverage inventory (before consolidation) + +Every path below is relative to the repository root. `assets` means the four executed `eval_authoring_assets_test` tests; `contracts` means the six executed `documented_check_contract` tests; `parity` means the executed documentation command/config/link checks; `workflow` means the saved Codex/Claude authoring, continuation, defect/correction, pilot, and rescore evidence summarized above. + +| Changed file | Coverage evidence | Result or gap | +| --- | --- | --- | +| `.gitignore` | assets verify the eval-authoring tree is unignored while an unrelated `skills/*` path remains ignored | passed | +| `CONTEXT.md` | parity plus requirement mapping | passed | +| `crates/fastskill-cli/src/main.rs` | 897 instrumented CLI tests; 91.22% lines and 96.00% branches; strict Clippy | passed | +| `crates/fastskill-evals/tests/documented_check_contract.rs` | Cargo/nextest discovery; six meaningful positive and negative contract tests | passed | +| `docs/adr/0011-skill-first-eval-authoring.md` | WF1/AT1 workflow evidence and local-link parity | passed | +| `docs/adr/0012-separate-outcomes-from-adherence.md` | conversion pilot and contract evidence distinguish outcome/adherence | passed | +| `docs/adr/0013-skill-driven-eval-authoring.md` | two-tool authoring/continuation and CLI runtime evidence | passed | +| `docs/qa/eval-authoring-verification.md` | reconciled against saved raw evidence and server02 commands | passed; native judge evidence added | +| `docs/requirements/eval-authoring-proposal.md` | WF/AT mapping and parity | passed | +| `docs/requirements/eval-authoring-skill-prd.md` | WF1-WF8 and AT1-AT11 mapping above | passed | +| `docs/requirements/eval-authoring-support-prd.md` | ST1-ST8 mapping; assets, contracts, parity, judge contrast | passed | +| `skills/eval-authoring/SKILL.md` | structural validation, assets, workflow, prerequisite/preservation branches, judge contrast | passed | +| `skills/eval-authoring/references/examples.md` | assets, pinned suite validation, saved defect/correction contrasts | passed; example contrast remains honestly qualitative | +| `skills/eval-authoring/references/handoff.md` | assets and Claude-from-Codex continuation record | passed | +| `skills/eval-authoring/references/semantics.md` | assets, six contracts, pinned validation/runtime probe | passed | +| `skills/eval-authoring/examples/invoice-extraction/SKILL.md` | assets and pinned suite validation | passed | +| `skills/eval-authoring/examples/invoice-extraction/skill-project.toml` | assets assert isolation metadata; pinned validation accepts all three runtimes | passed | +| `skills/eval-authoring/examples/invoice-extraction/evals/checks.toml` | assets, pinned parser, judge path/schema assertions, evidence-only live endpoint/model override | passed; portable placeholders remain intentional | +| `skills/eval-authoring/examples/invoice-extraction/evals/judge-prompt.md` | assets assert output contract and expected-fact rendering; native contrast used the same contract shape | passed | +| `skills/eval-authoring/examples/invoice-extraction/evals/prompts.csv` | assets exercise all three rows and referenced fixtures | passed | +| `skills/eval-authoring/examples/invoice-extraction/evals/review-notes.md` | assets plus workflow evidence | passed | +| `skills/eval-authoring/examples/invoice-extraction/fixtures/basic.txt` | assets parse/reference it and pinned suite validates it | passed | +| `skills/eval-authoring/examples/invoice-extraction/fixtures/missing-number.txt` | assets parse/reference it and assert the independent null expectation | passed | +| `tests/cli/eval_authoring_assets_test.rs` | Cargo/nextest discovery; four tests pass; controlled invalid case fails for intended contradiction | passed | +| `tests/cli/mod.rs` | nextest discovers and runs all four asset tests | passed | +| `tests/fixtures/eval-authoring/defective-total/SKILL.md` | assets plus real 15.00-vs-17.00 failure and corrected rerun | passed | +| `tests/fixtures/eval-authoring/defective-total/ground-truth.md` | assets assert independent ground truth; real failure/passing correction | passed | +| `tests/fixtures/eval-authoring/environment-variants.md` | assets assert CLI/runtime/judge prerequisite branches; server02 runtime and native-judge probes | passed; unavailable and configured branches observed | +| `tests/fixtures/eval-authoring/existing-suite/SKILL.md` | assets and cross-agent continuation | passed | +| `tests/fixtures/eval-authoring/existing-suite/skill-project.toml` | assets and pinned validation | passed | +| `tests/fixtures/eval-authoring/existing-suite/evals/checks.toml` | assets and cross-agent repair/validation | passed | +| `tests/fixtures/eval-authoring/existing-suite/evals/prompts.csv` | assets and cross-agent repair/validation | passed | +| `tests/fixtures/eval-authoring/existing-suite/evals/user-note.md` | assets assert unrelated user note preservation | passed | +| `tests/fixtures/eval-authoring/existing-suite/fixtures/release.md` | assets and cross-agent suite validation | passed | +| `tests/fixtures/eval-authoring/invoice-no-evals/SKILL.md` | assets and two-tool from-scratch authoring | passed | +| `tests/fixtures/eval-authoring/invoice-no-evals/fixtures/ambiguous-date.txt` | assets and workflow coverage outline | passed | +| `tests/fixtures/eval-authoring/invoice-no-evals/fixtures/basic.txt` | assets and workflow generated suite | passed | +| `tests/fixtures/eval-authoring/invoice-no-evals/fixtures/missing-number.txt` | assets assert clarified null behavior | passed | +| `tests/fixtures/eval-authoring/invoice-no-evals/fixtures/multiple-currencies.txt` | assets and workflow coverage outline | passed | +| `tests/fixtures/eval-authoring/scripted-workflow.md` | assets assert inspect/outline/review/authorization/preservation branches | passed | +| `tests/fixtures/eval-authoring/unnecessary-step/SKILL.md` | assets assert outcome/adherence conflict fixture | passed | +| `tests/fixtures/eval-authoring/unnecessary-step/scripted-answers.md` | assets retain author decision | passed | +| `tests/fixtures/eval-authoring/unsupported-order/SKILL.md` | assets assert unsupported expectation is reported rather than weakened | passed | +| `webdocs/evals-quality/setup.mdx` | parity, six contracts, shipped example | passed | + +## Commands and observed results + +| Command or exercise | Result | +| --- | --- | +| `quick_validate.py skills/eval-authoring` | passed (`Skill is valid!`) | +| pinned FastSkill 0.9.230 `eval validate --all --json` in shipped example | passed; three cases, one check, and `aikit`, `claude`, `codex` available | +| `cargo test -p fastskill-evals --test documented_check_contract --locked` | six passed | +| `cargo test -p fastskill-cli --test cli_tests eval_authoring --locked -- --test-threads=1` | four passed | +| `cargo nextest run --workspace --all-features --locked --no-fail-fast` | pre-fix: 1,872 passed, one configured skip; post-fix: 1,871 passed and one stale-cache failure | +| isolated stale-cache failure with a fresh `FASTSKILL_CACHE_DIR` | passed; confirms shared cache contamination rather than branch behavior | +| `cargo clippy --workspace --all-targets --all-features --locked -- -D warnings` | passed after crate recursion-limit fix | +| `cargo fmt --all -- --check`; `bash scripts/check-source-size.sh`; `git diff --check` | passed | +| `cargo llvm-cov nextest -p fastskill-cli --all-features --locked --branch ...` | 897 passed, one configured skip; `main.rs` 91.22% lines and 96.00% branches | +| Codex CLI 0.153.4 `gpt-5.6-luna` and Claude Code 2.1.269 `claude-sonnet-4-5` authoring/continuation | completed; generated and repaired suites validated; sanitized records retained | +| defect, corrected-target, conversion, and saved-evidence rescore pilots | expected defect failure preserved; corrected and rescore passed; outcome/adherence split observed | +| gateway `GET /v1/models` using `OPENAI_API_KEY` | HTTP 200; sole advertised model `tools-advanced` | +| pinned `eval judge` on copied defect run with evidence-only judge config | judged one, zero errors, overall 0.0, suite failed | +| pinned `eval judge` on copied corrected run with the same judge config | judged one, zero errors, overall 1.0, suite passed | +| direct Linux judge of original cross-machine run | expected `EVAL_CONFIG_MISSING`; copied summary path repair preserved originals and enabled calibration | + +## Release gaps + +- Native judge calibration is complete. The evidence-only endpoint/model/key-variable override is intentionally not committed into the portable example, and no credential value is recorded. +- The unrelated `manifest_add_records_repository_intent_after_catalog_match` test can consume stale shared cache state when the test process inherits the default cache root. It passes with a fresh supported `FASTSKILL_CACHE_DIR`; this isolation defect is recorded but does not justify widening the eval-authoring implementation. +- Saved eval runs currently retain an absolute originating `skill_project_root`; judging a copied run on another operating system requires an evidence-copy path repair. The original artifacts were preserved, and this limitation does not invalidate the executed contrast. + +The original calibration and local consolidation gates are complete. GitHub CI is a +separate check on the pushed revisions. Both PRs remain drafts; do not enable auto-merge +or merge them. The companion report explicitly records retained harness limitations. diff --git a/docs/requirements/eval-authoring-proposal.md b/docs/requirements/eval-authoring-proposal.md new file mode 100644 index 00000000..cc25abb0 --- /dev/null +++ b/docs/requirements/eval-authoring-proposal.md @@ -0,0 +1,91 @@ +# Assisted eval design for skills + +Status: working discussion record, superseded for delivery scope by the [eval-authoring skill PRD](eval-authoring-skill-prd.md) and [support PRD](eval-authoring-support-prd.md). Product boundaries are agreed; PRDs are drafts for external review. See ADR-0011 through ADR-0013. + +## Agreed authoring principles + +- Immediately propose a short coverage outline after inspecting the skill. Ask only questions whose answers materially change the tests; avoid a fixed questionnaire. +- Measure outcome correctness separately from skill adherence. Adherence becomes mandatory only when the author identifies it as a requirement; do not silently promote every skill instruction to a correctness condition. +- Aim to deliver a piloted suite. If execution prerequisites are missing, preserve an explicitly unverified draft rather than claiming completion of verification. +- Exercise graders against acceptable and unacceptable examples. The skill passing its own tests is not a condition of successful authoring. + +The authoring conversation runs in the user's chosen agentic tool, guided by an eval-authoring skill. FastSkill CLI and the evaluation engine own supported formats, validation, execution, and evidence. Drafts are ordinary files under version control; a dedicated interview engine and draft-state service are deferred. + +Additional agreed decisions: + +- Review the coverage outline first, then the suite as a whole. Request individual clarification for ambiguous requirements or questionable grading examples rather than requiring approval of every case. +- The authoring agent may propose graders and calibration examples. Labels must derive from author-confirmed criteria or independently verifiable facts; synthetic examples are evidence, not proof of correctness. +- The authoring skill checks CLI version compatibility and consults authoritative format references and validation results. Unsupported versions receive an explicit compatibility message; exact supported versions are an implementation deliverable. +- The user's authoring agent handles all conversational authorization. It uses existing authorization or asks once about the proposed pilot when necessary, then invokes explicit CLI parameters. FastSkill neither interprets conversational consent nor stores conversational approvals. +- A monetary ceiling is not required for this authoring change. An agreed set of cases, runtimes, and trial counts bounds the pilot's scope, not its actual monetary cost. Only controls implemented by the CLI may be described as enforced. + +V1 builds the authoring skill first using existing suite/CLI capabilities, with documentation and existing-contract validation fixes. Unsupported intentions are reported rather than weakened. Engine expansion requires separate proposals. Distribution location, exact compatibility versions, and test mechanics require implementation investigation, not a new conversational subsystem. + +## Problem + +Creating meaningful cases consumes more human attention than writing CSV or TOML. Authors must reconstruct the task, decide what a correct outcome means, remove incidental context, supply fixtures, and establish that the grader distinguishes success from failure. Existing execution and scorecard infrastructure does not solve this authoring problem. + +## Proposed outcome + +An author provides a target skill to their chosen agentic tool and uses FastSkill's eval-authoring skill to design its evaluation tests. That agent inspects the target, proposes coverage and cases, surfaces uncertain assumptions, and invokes CLI validation and pilots as appropriate. Accepted cases enter an ordinary version-controlled suite with provenance. Human attention is spent on intent and ambiguous criteria rather than file syntax. + +The initial scope is skill-to-suite authoring without requiring prior sessions or known failures. Session inspection for regression discovery and skill improvement is a separate future discussion. The skill's current instructions are evidence of intended behavior, not sufficient ground truth: a generated suite must not merely encode an existing mistake as the expected outcome. + +## Proposed workflow + +1. Select the skill to evaluate and inspect its instructions and relevant supporting files. +2. Propose a coverage plan spanning intended outcomes, triggering, non-triggering, boundaries, and failure handling where applicable. Identify missing intent before generating a large suite. +3. Propose expected outcomes and deterministic assertions where possible. Label inferred expectations and request clarification where correctness depends on unstated intent. +4. Show the proposed prompt, fixtures, assertions, evidence provenance, and unresolved assumptions together for review. +5. Pilot graders against acceptable and unacceptable example outcomes or deliberately constructed contrasts, labeled from author-confirmed criteria or independently verifiable facts. Do not treat a successful pilot as proof of validity. +6. Pilot execution on selected supported runtimes, reporting unavailable evidence separately from failure. Additional trials and ablation are a later expansion unless needed for the first use case. +7. Accept the candidate explicitly into the suite. Later executions and scorecards use the existing evaluation workflow. + +All steps describe proposed behavior. They are not existing commands or implementation commitments. + +## Initial scope recommendation + +Start with skill inspection, intent clarification, coverage planning, candidate generation, suite-level review with targeted clarifications, grader calibration where supported, and conversion to the existing suite format. Report expectations that current capabilities cannot faithfully express; preserve unverified work explicitly. + +Session import, session-derived regression discovery, and session-based skill improvement are deferred by the agreed scope. Also propose deferring automatic monitoring, unrestricted filesystem reconstruction, full MCP simulation, automatic skill rewriting, and autonomous promotion of generated cases until the initial authoring workflow is validated. + +## Candidate ADR topics + +Create numbered ADRs after the grilling session resolves the choices. Use the repository's context/decision/consequences structure and keep unresolved ADRs proposed. + +| Topic | Recommended starting position | Decision to resolve | +| --- | --- | --- | +| Authoring entry point | Agreed: start from a skill and design its eval tests | Recorded in ADR-0011; session inspection deferred. | +| Authoring boundary | Agreed: user's agent plus authoring skill; CLI owns enforceable mechanics | ADR-0013; exact distribution and compatibility versions require investigation. | +| Candidate lifecycle | Ordinary files and version control; outline then suite-level review | Verification artifact details depend on the remaining engine-scope decision. | +| Case representation | Compile into existing CSV/check/judge formats, retain authoring metadata separately | Are richer fixtures or multi-turn cases essential enough to require schema changes now? | +| Ground truth and calibration | Agreed: author-confirmed criteria or independently verifiable facts establish labels | Preserve unverified work where prerequisites are missing; calibration implementation depends on engine scope. | +| Authoring inputs and data handling | User-selected agent owns the conversation and its provider setup | No new FastSkill authoring provider or conversational approval store. | +| Execution and ownership | Reusable calibration primitives upstream where appropriate; FastSkill owns CLI/project integration | Which parts belong in aikit-evals, and what isolation guarantees must pilots enforce? | + +## Success measures + +Measure human active minutes and clarification turns per accepted case, including subsequent repairs. Compare with manual FastSkill authoring and assisted plugin-eval authoring on comparable tasks. Do not optimize for the number of generated cases. + +Track whether accepted cases distinguish known failures from acceptable outcomes, how often reviewers reject or rewrite suggested criteria, and how often graders change verdict on unchanged evidence. Record supported-runtime coverage and infrastructure errors separately. Numerical acceptance targets require a baseline and are not invented here. + +## Questions to challenge in the grilling session + +- Where should review checkpoints fall after the coverage outline is proposed? +- Can the case be reproduced without the original private repository or external service state? +- Who decides what a correct answer is when the skill's instructions are ambiguous or wrong? +- Is a cross-agent case still testing the same outcome after tool-specific details are removed? +- How do we prevent the generator from making a grader that simply rewards its own proposed answer? +- Can the workflow reduce total human work after fixture repair, review, and grader maintenance are counted? +- What happens when a proposed case cannot be expressed faithfully in the existing engine? +- How are sensitive source details removed without destroying the behavior being tested? + +## Related material + +- [Skill-first eval authoring ADR](../adr/0011-skill-first-eval-authoring.md). +- [Outcome correctness and adherence ADR](../adr/0012-separate-outcomes-from-adherence.md). +- [Skill-driven authoring ADR](../adr/0013-skill-driven-eval-authoring.md). +- [Command taxonomy ADR](../adr/0010-command-taxonomy.md): any new authoring actions must use the existing explicit namespace model. +- [Eval measurement integrity requirements](eval-measurement-integrity.md). +- [Eval judging requirements](eval-judge.md). +- [Eval scorecard reporting requirements](eval-scorecard-report.md). diff --git a/docs/requirements/eval-authoring-skill-prd.md b/docs/requirements/eval-authoring-skill-prd.md new file mode 100644 index 00000000..9f5bdbf2 --- /dev/null +++ b/docs/requirements/eval-authoring-skill-prd.md @@ -0,0 +1,219 @@ +# Skill-driven eval authoring + +Status: draft for external review. Version: 0.2. Date: 2026-09-12. + +Product and architecture direction is agreed in ADR-0011 through ADR-0013. This document specifies proposed delivery requirements; it does not claim implementation or validation. It supersedes the working authoring proposal for this delivery. + +## Problem and intended outcome + +FastSkill can execute, judge, rescore, and report evaluation suites across supported agent runtimes. Creating those suites still requires substantial human attention: selecting meaningful scenarios, defining correctness, learning file conventions, designing graders, and interpreting validation and pilot failures. + +Provide an eval-authoring skill that works inside the user's chosen agentic tool. It should guide the agent from an existing target skill to a reviewable suite in FastSkill's supported formats, with pilot evidence where execution is possible. FastSkill remains a CLI and evaluation toolchain; it does not become a conversational agent. + +The primary outcome is reduced human active time per useful case, including review and repair. The number of generated cases is not a success metric by itself. + +## Users and initial scope + +The primary user maintains a skill and needs to design its first evaluation suite. No session history or known failure is required. The user's authoring agent must be able to read the authoring skill, inspect relevant files, edit suite files, and invoke FastSkill CLI. Authoring support does not imply that the same agent is supported as an evaluation target. + +V1 delivers the authoring skill, focused references and examples, and supporting documentation/validation corrections described in the companion PRD. It uses existing suite and CLI capabilities. If an intended check cannot be represented faithfully, the workflow records the limitation rather than silently weakening the expectation. + +Out of scope: a FastSkill interview engine, an authoring model API integration, a draft-state service, new case schemas or grader types, automated session import, skill optimization, native ablation, MCP simulation, new sandbox guarantees, and monetary spending caps. Necessary engine extensions discovered during development require separate proposals. + +## Before and target state + +| Area | Before | Target | +| --- | --- | --- | +| Starting point | Author learns formats and constructs cases | Agent inspects a target skill and proposes coverage | +| Human contribution | Scenario construction, syntax, commands, debugging | Intended behavior, consequential ambiguities, review | +| Authoring environment | No dedicated supported workflow identified | User's chosen agent plus FastSkill authoring skill | +| Artifacts | Existing CSV/TOML, prompts, fixtures, run evidence | Same execution formats, with concise human-readable review notes | +| Validation and execution | CLI capabilities available separately | Agent invokes those capabilities and explains evidence | +| Unsupported expectations | Easy to approximate accidentally | Explicit limitation tied to the affected behavior | + +Claude plugin evals already provide assisted suite generation and pilots. This delivery aims to narrow that authoring gap while preserving choice of authoring agent and access to FastSkill's existing cross-runtime measurements and historical scorecards. It does not claim parity in grader expressiveness, plugin integration, mocks, ablation, or execution isolation. + +## Responsibility boundary + +| Responsibility | Owner | +| --- | --- | +| Conversation, clarification, user review, execution authorization | User's authoring agent, guided by the authoring skill | +| Layout guidance, workflow instructions, examples | Eval-authoring skill and maintained references | +| Supported formats, parsing, deterministic validation | FastSkill CLI and relevant existing libraries | +| Trial execution, configured judging, scoring, evidence | Existing eval toolchain | +| Definition of intended outcome where judgment is needed | Author | + +The authoring skill cannot establish machine-enforced guarantees merely by instructing the agent to follow a procedure. No prose note is a CI attestation. Existing agent authentication is not proof that separately configured eval judges are available. + +## Reference user workflow for implementation and testing + +This workflow is a normative acceptance reference for the authoring skill and CLI integration. Implementers must create purpose-built target skills and fixtures to exercise it; testing only the authoring instructions' text or isolated CLI commands is insufficient. The example conversations illustrate intent, not exact wording that tests must match. + +The user stays in their chosen agentic tool. All conversation below is conducted by that agent. FastSkill only receives explicit CLI invocations and returns results. + +| Step | User and authoring-agent interaction | Implementation and test evidence | +| --- | --- | --- | +| WF1: Start | User: "Help me design eval tests for my invoice-extraction skill." Agent loads the eval-authoring skill, inspects the target and resources, and checks CLI compatibility. | Loaded skill/version, relevant inspected inputs, actual version/help output; no target suite initially exists. | +| WF2: Outline | Agent proposes ordinary invoices, missing fields, ambiguous dates, multiple currencies, and unrelated requests. It asks: "When the invoice number is missing, should extraction return null or fail?" User confirms null and reviews coverage. | Coverage outline precedes detailed suite generation; consequential answer is retained and no fixed questionnaire is required. | +| WF3: Generate | Agent writes supported prompts, checks, judge rubrics where needed, and fixtures. It distinguishes extraction correctness from triggering or required procedure. | Real generated files parse; missing-number criterion reflects null; trigger settings match engine semantics; unsupported expectations are explicit. | +| WF4: Review | Agent presents the suite together: each case's intent, expected result, and check method. It asks separately only about remaining ambiguities. | Suite-level review and targeted questions; no mandatory approval loop for every row. | +| WF5: Validate and challenge | Agent invokes CLI validation, repairs malformed files, and challenges graders with acceptable and unacceptable examples. | Actual validation output; labels tied to confirmed criteria; executed calibration distinguished from qualitative review or unavailable tooling. | +| WF6: Pilot authorization | If not already authorized, agent proposes "six cases, one trial each, on the selected runtime" and explains that this consumes model usage. The user can change scope. | Agent, not CLI, asks when needed. Prior authorization is respected. Invoked cases/runtimes/trials match the agreed scope; no unsupported cost guarantee. | +| WF7: Execute and explain | Agent invokes FastSkill. It explains which cases passed, which failed, and which could not be measured, linking the evidence. | Real run artifacts and results. The seeded ambiguous-date failure is identified without treating missing evidence as passing. | +| WF8: Refine and hand over | User: "Make the date criterion clearer and rerun the affected case." Agent updates the criterion and performs an appropriate supported rescore or rerun. It hands over files, evidence, and unresolved limits. | Unrelated user content is preserved; a new target run occurs only when needed; no criterion is weakened merely to obtain a pass; another authoring agent can continue from files. | + +Expected end state: editable suite and fixtures; review notes; actual validation/pilot artifacts; explicit target failures and unsupported expectations; and clear identification of anything still unverified. Target failure is compatible with successful test authoring. + +Failure branches must include missing CLI/runtime/judge prerequisites, incompatible CLI version, malformed generated configuration, unsupported checks, missing calibration capability, and a user changing the pilot scope. Preserve useful work in each branch and report the precise limitation. + +## Functional requirements + +### A1 — Inspect before asking + +The authoring agent identifies the selected target skill, inspects its instructions and relevant supporting files, and checks the installed CLI version. It consults compatible references and actual help/validation output rather than assuming a command or field exists. The workflow does not require a specific authoring vendor, subscription, or model. + +### A2 — Propose coverage, then clarify + +Present a concise coverage outline before generating a large suite. Consider useful outcomes, positive and negative triggering, boundaries, and failure handling where relevant. Ask only questions whose answers materially change tests. Review the outline before proceeding to the detailed suite; avoid a fixed questionnaire and arbitrary case-count quotas. + +### A3 — Separate outcomes and adherence + +Each proposed expectation identifies whether it measures outcome correctness or skill adherence. A prescribed method is mandatory only when the author identifies it as a requirement. The agent must surface a conflict between a skill instruction and the intended outcome instead of silently resolving it. + +The reviewed engine generates trigger checks from `should_trigger`. V1 must explain and respect that behavior, including explicit-check overrides and contradiction validation. It must not label a case outcome-only if the emitted configuration still has a mandatory adherence gate. If an intended scoring policy cannot be represented, identify it as unsupported; do not alter engine semantics under this PRD. + +### A4 — Write supported, reviewable files + +Generate ordinary suite files, judge prompts, and fixtures using documented supported formats. Preserve existing user files and show material changes when editing an existing suite. Retain concise review notes identifying coverage, requirement origins, assumptions, and unsupported expectations. These notes are ordinary documentation, not a new suite schema or managed lifecycle. + +### A5 — Review the suite together + +Show the proposed cases and criteria as a coherent suite. Request individual clarification for ambiguities or questionable examples, rather than approval of every row. Review acceptance does not imply that the target skill passed or that all graders were validated. + +### A6 — Challenge graders with contrasting outcomes + +Propose acceptable and unacceptable example outcomes, with labels grounded in author-confirmed criteria or independently verifiable facts. Record the reason for each label. Synthetic examples and a second model's agreement are not independent ground truth. + +Exercise actual graders against contrasting evidence where existing supported interfaces permit it. Where they do not, distinguish qualitative review from executed calibration and preserve the missing verification explicitly. Never fabricate an official run artifact to pretend a grader was exercised. Adding a standalone calibration command is outside v1 and may be proposed separately. + +### A7 — Validate and repair + +Invoke existing config validation and runtime observability checks appropriate to the selected targets. Fix authored formatting/reference mistakes and repeat the relevant check. Separate invalid configuration, missing prerequisites, unobservable measurements, and actual target failures. Validation success alone is not evidence of behavioral correctness. + +### A8 — Let the authoring agent authorize pilots + +The agent uses existing user authorization or asks about a concrete pilot scope when authorization is unclear. Scope identifies cases, evaluation runtimes, and trial counts. FastSkill receives explicit command parameters; it does not interpret the conversation, ask consent questions, or maintain an approval store. + +Pilot scope bounds work, not monetary cost. Do not promise a budget cap unsupported by the CLI. Missing evaluation-target or judge credentials must be reported without treating the authoring agent's credentials as interchangeable. + +### A9 — Preserve honest completion evidence + +Aim for a piloted suite. Report independently: author review, configuration validation, execution attempted/completed, grading/calibration evidence, target verdicts, and unresolved limitations. A target failing a useful test is a valid authoring result. Missing prerequisites leave an unverified draft; they must not be represented as passing or fully verified work. + +Link actual run artifacts where available. Human-readable completion notes are not consumed as trusted verification by CI. Existing machine-readable artifacts retain their current meanings. + +### A10 — Support continuation through files + +Another capable authoring agent can continue from the suite and review notes without reconstructing a proprietary conversation state. The workflow must not require a hosted authoring service, session database, or FastSkill-owned interview state. + +## Deliverables + +- An installable eval-authoring skill with concise workflow instructions and focused supporting references. +- Examples covering positive/negative triggering, outcome judging, and an unsupported expectation handled explicitly. +- Instructions for validation, authorized pilot execution, result review, and handoff through ordinary files. +- A declared CLI compatibility range backed by checks from the companion PRD. +- Evidence from end-to-end authoring exercises and a record of unresolved limitations. +- Purpose-built test skills, fixture data, scripted user answers, and workflow tests covering WF1–WF8 and the acceptance scenarios below. +- A changed-file coverage inventory and reproducible validation evidence meeting the testing requirements below. + +## Acceptance scenarios + +| ID | Scenario | Required evidence | +| --- | --- | --- | +| AT1 | Target skill has no evals | Coverage outline, relevant clarification, generated suite, validation result | +| AT2 | User's agent is not an eval backend | Authoring succeeds; execution targets only supported runtimes | +| AT3 | Skill instruction conflicts with intended outcome | Conflict surfaced; resulting criteria reflect author's decision | +| AT4 | Positive and negative triggering cases | Expectations match actual current scoring semantics | +| AT5 | Required assertion is not supported | Limitation reported; no misleading weaker substitute | +| AT6 | Known acceptable and unacceptable examples | Labels explained; actual calibration distinguished from qualitative review | +| AT7 | CLI, target runtime, or judge prerequisite missing | Work retained with precise unmet prerequisite and unverified status | +| AT8 | Target skill fails a well-formed case | Failure retained as evidence; criteria not weakened merely to obtain a pass | +| AT9 | Pilot scope already authorized | No redundant permission interview; explicit scoped command invoked | +| AT10 | Pilot scope not authorized | Authoring agent asks; CLI remains non-conversational | +| AT11 | Author switches agent | Suite and notes are sufficient to resume without a proprietary state service | + +## Validation and success measures + +Run documented authoring exercises with at least two different capable authoring tools as a release check. Record versions, prerequisite setup, generated artifacts, validation results, human interventions, and paid runs. This is evidence of portability, not a universal compatibility claim. The same workflow and behavioral expectations must apply across tools; exact generated prose and case counts need not match. + +### Test skills and fixtures to create + +Create isolated, synthetic test assets in the chosen repository's test/fixture layout. They must not depend on private production data or change the developer's global skill installations. + +| Test asset | Purpose and required exercises | +| --- | --- | +| Invoice-extraction skill with no evals | Main WF1–WF8 flow; text invoice fixtures with known values, missing number, ambiguous date, multiple currencies, and an unrelated request. | +| Target skill with a deliberate behavioral defect | Ensure the generated test detects an incorrect outcome and the authoring agent retains the failure instead of changing expectations. The defect and expected outcome must be established independently of the generated grader. | +| Target skill with an unnecessary prescribed step | Exercise the distinction between useful outcome and adherence; scripted author answers settle whether the step is mandatory. | +| Target requiring an unsupported assertion | Verify honest limitation handling without invented commands, formats, or misleading substitute checks. | +| Existing suite with unrelated user edits | Verify preservation, targeted edits, rescoring/rerun selection, and handoff between agents. | +| Environment/configuration variants | Missing credentials or runtime, incompatible version, invalid references, and unavailable observations; use controlled stubs for deterministic failure tests where appropriate. | + +Keep expected labels, known defects, and scripted user answers outside the target's visible execution inputs where feasible. Separate test-owned ground truth from agent-generated outputs. Real workflows must use the actual authoring skill and CLI; stubs are appropriate for controlled failure branches, not as replacements for all end-to-end evidence. + +### Authorized live-test tools and models + +The project owner permits implementation testing with the following available-tool choices. These are permitted options, not a requirement to run every combination or a claim that each alias is installed or supported as an eval backend. + +| Tool | Permitted model/provider choice | Setup requirement | +| --- | --- | --- | +| pi | Models available through the team's LiteLLM endpoint | Discover configured endpoint/model aliases and use an existing approved token; request secure provisioning if absent. | +| Cursor | Available Grok models | Verify the installed tool's actual supported model identifier and authentication. | +| Codex | GPT Luna, as requested by the owner | Resolve the actual available identifier; do not guess an alias or silently substitute another model. | +| Claude | Sonnet 4, as requested by the owner | Verify the actual available identifier and credentials; report unavailability. | + +These tools can be authoring agents. Use them as evaluation targets only where the FastSkill backend supports them and can observe the selected checks. FastSkill's native judge endpoint/model configuration is separate from authoring-agent authentication and must satisfy the existing judge contract. + +Use small, recorded pilot batches within this authorized testing scope; no new blanket approval is needed merely to exercise an allowed combination. Request missing credentials or environment permissions when required, never print tokens, and never place tokens in fixtures, prompts, logs, PRDs, or committed artifacts. Record endpoint identity/model IDs without secret values. If fewer than two authoring tools can run, retain a blocked portability release check rather than claiming completion from one tool. + +### High coverage for every touched file + +Every added or modified file in the implementation, including skill instructions, supporting references, fixtures, scripts, configuration, and code, must have explicit test or verification coverage appropriate to its contents. Submit a changed-file inventory mapping each file to test IDs, results, and remaining gaps. A high repository-wide average is not a substitute for coverage of an individual changed file. + +For this PRD, high executable-code coverage means at least **90% line coverage per added/modified executable source file**, and **80% branch coverage where supported by the toolchain**, with stricter repository gates taking precedence. Cover all changed contract behaviors and critical failure paths explicitly; numeric coverage alone is insufficient. Test files themselves need discovery/execution evidence and meaningful assertions, not recursive tests of tests. Document instrumentation limitations; do not represent unmeasurable branch coverage as passing. Uncovered behavior or unmet thresholds remains a release gap unless explicitly waived by the owner. + +For non-executable files, do not claim a source-line percentage. Require: + +- Skill instructions: every normative behavior mapped to an executed workflow/acceptance scenario, including negative and recovery branches. +- References and Markdown: all command/config examples validated against the supported CLI, local links resolved, and material behavioral claims mapped to tests or verified code contracts. +- Fixtures/templates/configuration: every supplied asset parsed or exercised; invalid examples fail for the intended reason; expected labels and outcomes checked independently. +- Changed tests/build/CI configuration: demonstrate that intended tests run and that a controlled failure is detected by the relevant gate where applicable. + +Avoid tests that only search for wording in SKILL.md or mirror implementation logic. Observe emitted suite files, CLI invocations, exit/results semantics, preserved edits, and real failure detection. Separate deterministic CI checks from live-model exercises; record every live attempt and flaky result instead of selectively reporting successful retries. + +### Developer handoff and definition of done + +The implementation handoff must contain the installable authoring skill and references; test targets and fixtures; requirement-to-test and changed-file coverage mappings; reproducible commands; CLI/agent/model versions; sanitized live-run artifacts; and a clear list of limitations. Run relevant existing regression and command/document parity checks in addition to new coverage. + +Release is ready only when WF1–WF8, AT1–AT11, and the companion support criteria have evidence; the two-tool portability check and changed-file coverage gates pass; and known limitations are explicit. A deliberately failing target is expected evidence and does not fail authoring acceptance when correctly detected. This PRD update itself does not claim those implementation tests have been run. + +Measure active human minutes per accepted useful case, clarification count, correction/rejection rate, layout errors caught by validation, and time spent repairing graders. Compare comparable tasks with manual FastSkill authoring and the reviewed Claude-assisted workflow when available. A quantitative improvement target must follow a measured baseline; no percentage advantage is claimed now. + +## External review questions and implementation investigations + +- Does the acceptance matrix distinguish enough evidence to prevent an unverified suite appearing release-ready? +- Is the usefulness of v1 acceptable when full contrast calibration is unavailable for a grader? +- Determine skill distribution repository/path and supported versions using the maintained distribution conventions. No new installation mechanism is presumed. +- Confirm the smallest safe representation of review notes and examples, without creating a new engine schema. +- Identify current-interface limitations with reproducible cases; route required new capabilities to separate proposals. + +These are review and feasibility items. The user/agent/CLI responsibility boundary is settled. + +## Related decisions and requirements + +- [ADR-0011: skill-first authoring](../adr/0011-skill-first-eval-authoring.md) +- [ADR-0012: outcomes and adherence](../adr/0012-separate-outcomes-from-adherence.md) +- [ADR-0013: skill and CLI boundary](../adr/0013-skill-driven-eval-authoring.md) +- [Authoring support PRD](eval-authoring-support-prd.md) +- [Existing judge requirements](eval-judge.md) +- [Existing measurement-integrity requirements](eval-measurement-integrity.md) diff --git a/docs/requirements/eval-authoring-support-prd.md b/docs/requirements/eval-authoring-support-prd.md new file mode 100644 index 00000000..f1bc52c2 --- /dev/null +++ b/docs/requirements/eval-authoring-support-prd.md @@ -0,0 +1,102 @@ +# Documentation and validation support for eval authoring + +Status: draft for external review. Version: 0.2. Date: 2026-09-12. + +This companion to the eval-authoring skill PRD covers the narrow tooling and documentation work needed to make its instructions reliable. It does not introduce an authoring orchestrator or expand evaluation expressiveness. + +## Problem + +A portable authoring skill must teach the behavior of the installed FastSkill release. Documentation drift can cause it to generate valid-looking suites that measure the wrong thing. The reviewed 0.9.230 checkout already contains material contradictions: its setup guide calls `should_trigger` inert, says deterministic checks necessarily apply globally, and describes no-check scoring as process-exit-only. The pinned engine generates per-case trigger checks and supports case selectors. + +The source baseline is FastSkill `dd983e89e5fee977325b77ae386b879e8310ed26` and its aikit pin `eb50f2c5509d4345cab65af0f7faed012ab0e557`. Revalidate against the implementation checkout before making fixes; these findings are not assertions about every version. + +## Objective and boundary + +Make supported authoring formats, commands, and semantics discoverable, correct, and testable. FastSkill remains responsible for validating explicit files and command parameters, producing evidence, and explaining errors. The user's authoring agent remains responsible for conversation, intent, and authorization. + +Use existing help, command specification, documentation, and validation surfaces. Add or repair validation only for existing format contracts. New graders, case schemas, calibration commands, budget controls, and runtime isolation changes need separate scope and are not implied here. + +## Requirements + +### S1 — Correct the maintained reference + +Audit and correct eval setup and command examples against pinned source behavior. Cover implicit trigger checks, explicit-check precedence, case selectors, contradiction handling, consultation evidence by backend, required/advisory checks, and scoring thresholds. Explain that path-reference evidence is a proxy for consultation, not proof that instructions were followed. + +Document the distinction between configuration validation, actual execution, judgment, and rescoring. Do not claim that a supported command validates evidence it does not inspect. + +### S2 — Keep ownership of rules clear + +The parser and engine define enforceable formats and scoring. Maintained references explain them; the authoring skill links to those references instead of copying large independent schemas. Examples must identify their supported release range. The skill must not assume features from a later CLI merely because its own instructions are newer. + +Determine whether current version/help/spec surfaces provide enough compatibility information before proposing additions. Do not introduce a new discovery API by default. + +### S3 — Provide actionable existing-contract validation + +Inventory existing validation behavior before implementing anything. Where a currently supported field has inadequate validation, propose a narrow correction with a reproducer. Errors should identify the affected file, case/check/judge where available, and the violated rule. Preserve established CLI and machine-readable output contracts unless a compatibility change is separately reviewed. + +Do not convert missing runtime evidence into a passing check, or missing measurements into zero scores. Do not add conversational prompting or approval persistence. + +### S4 — Ship executable examples + +Maintain small fixtures exercising positive/negative triggering, case-scoped checks, optional adherence where representable, a configured outcome judge, and an unsupported expectation described as such. Keep secrets out of examples and use explicit placeholders for separately supplied endpoint credentials. + +Validate syntax and deterministic behavior without paid calls. Any live agent or judge exercise is a separate, explicitly scoped validation activity. Do not substitute a model-generated success narrative for command output. + +### S5 — Connect authoring guidance to real evidence + +Examples and the skill must show how to interpret config errors, backend observability exclusions, execution failures, and target-quality failures separately. Document which current interfaces can exercise graders on saved evidence and which cannot. If a standalone calibration path does not exist, report that gap rather than inventing a command or forging artifacts. + +CI decisions continue to rely on existing structured results and exit semantics. Authoring notes do not introduce a trusted `verified` flag or new gate. + +### S6 — Preserve execution boundaries + +Correct any guidance that confuses scratch-workspace skill isolation with OS-level confinement. Describe actual permissions and backend limitations of the supported version. The authoring agent obtains any needed user authorization and invokes explicit CLI arguments. This PRD neither adds a sandbox nor promises a monetary cap. + +## Acceptance criteria + +| ID | Criterion | +| --- | --- | +| ST1 | Maintained authoring references no longer contradict the supported release on trigger expectations, selectors, or no-check behavior | +| ST2 | All shipped suite examples parse under the declared supported versions; expected invalid examples fail for the intended reason | +| ST3 | Deterministic tests demonstrate positive/negative expectations and explicit-check contradiction behavior | +| ST4 | Authoring instructions use actual help/spec/validation interfaces and no invented CLI commands | +| ST5 | Documentation and examples distinguish outcome checks, adherence checks, and aggregate verdicts accurately | +| ST6 | Compatibility failures are explicit; silently assuming newer fields is not taught | +| ST7 | Missing prerequisites, absent observations, and quality failures are explained separately | +| ST8 | No conversational interview, approval state store, or new engine capability is introduced under this support work | + +## Delivery and ownership + +The installable workflow is part of `gofastskill/skill` under `fastskill/`: +`SKILL.md` routes to `references/eval-authoring.md` and its supporting references/example. +Authoring fixtures and asset tests belong in that repository's `evals/authoring/` and +`scripts/` directories. This CLI repository owns the engine contract tests, public command +reference, requirements, and verification record. See the +[consolidation assessment](https://github.com/gofastskill/skill/blob/main/specs/002-eval-authoring-consolidation.md). + +### Workflow and coverage obligations + +Use the primary PRD's [reference workflow](eval-authoring-skill-prd.md#reference-user-workflow-for-implementation-and-testing), test skills, live-tool choices, and [changed-file coverage requirements](eval-authoring-skill-prd.md#high-coverage-for-every-touched-file) as mandatory implementation and testing references. They apply equally to files changed under this support PRD and to any associated upstream fixes. + +Every changed file must appear in the handoff coverage inventory. Executable source files require at least 90% per-file line coverage and 80% branch coverage where instrumentation supports it, or stricter existing repository gates. Skill and documentation files require behavioral/example/link verification rather than artificial line percentages. Every supplied fixture must be exercised, and test/CI changes need execution evidence. Unmet thresholds and unavailable checks are visible release gaps, not silently skipped success. + +In addition to ST1–ST8, demonstrate that repaired guidance works during real skill-driven authoring: generate and validate suites from purpose-built target skills, preserve deliberately detected failures, and distinguish unsupported expectations from valid passing checks. Use controlled deterministic tests for parser/validator defects and scoped live exercises for agent behavior. The owner's permitted tools and models are recorded in the primary PRD; credentials must be provisioned securely and model identifiers verified before use. + +Implement reference/example corrections in FastSkill and the chosen skill distribution location. Changes to shared parsers or validation belong in the owning library when a confirmed existing-contract bug requires them; FastSkill should not duplicate upstream evaluation semantics. Any upstream change must be pinned and verified in the consuming checkout. + +Before development, identify the release compatibility matrix and enumerate confirmed validation defects. A new validation feature is not justified merely because it would be convenient for an agent. + +## Risks and review items + +- A prose reference can drift again: cover material behavioral claims with focused tests or existing parity checks where appropriate. +- Implicit trigger checks can conflict with author expectations of outcome-only scoring: demonstrate the actual configuration or mark the requested policy unsupported. +- Pilots may be expensive or unavailable: deterministic documentation checks must remain possible without live credentials. +- Broader evaluation gaps may emerge: document them separately rather than expanding the authoring release silently. + +## Related material + +- [Eval-authoring skill PRD](eval-authoring-skill-prd.md) +- [ADR-0013](../adr/0013-skill-driven-eval-authoring.md) +- [Eval setup reference](../../webdocs/evals-quality/setup.mdx) +- [Eval command reference](../../webdocs/cli-reference/eval-command.mdx) +- [Observability preflight](../../crates/fastskill-cli/src/commands/eval/observability.rs) diff --git a/webdocs/evals-quality/setup.mdx b/webdocs/evals-quality/setup.mdx index bb0083c0..357ae164 100644 --- a/webdocs/evals-quality/setup.mdx +++ b/webdocs/evals-quality/setup.mdx @@ -7,14 +7,19 @@ description: "Author an eval suite: the [tool.fastskill.eval] manifest table, th This page is the reference for **authoring** an eval suite: the config table in `skill-project.toml`, the prompts CSV a suite is made of, and the optional checks TOML that actually decides pass/fail. Work through it top to bottom once, then use [Run evals](/evals-quality/run-evals) for the day-to-day commands. +For guided suite design, use the [FastSkill skill](https://github.com/gofastskill/skill/tree/main/fastskill). +Its [eval-authoring workflow](https://github.com/gofastskill/skill/blob/main/fastskill/references/eval-authoring.md) +and copyable invoice example are maintained and packaged in the skills repository. + ## Prerequisites - `skill-project.toml` exists (project root or skill directory — `fastskill` walks up the tree to find it). - `SKILL.md` exists alongside it if you're authoring a single skill's suite. - For `fastskill eval run`: a supported agent CLI on `PATH`. Agent keys come from aikit-sdk: `aikit`, `claude`, `codex`, `cursor`, `gemini`, and `pi`. A key is accepted only when its CLI is detected on the current machine; a typo and an uninstalled agent both error with `RUNTIME_UNKNOWN_ID`. `fastskill eval validate --all` reports which keys are detected in *your* environment. -Evals do not require embedding or `OPENAI_API_KEY`; those are used by `index rebuild` and -`skill search --local`. +Deterministic validation and scoring need no model credential. Target execution needs the +selected runtime's authentication; native judges separately need their configured endpoint, +model, and credential environment variable. Embedding configuration is unrelated. ## 1. The `[tool.fastskill.eval]` table @@ -34,7 +39,7 @@ fail_on_missing_agent = true # default true | Field | Type | Required | Default | Meaning | |---|---|---|---|---| | `prompts` | path | **yes** | — | Path to the prompts CSV. | -| `checks` | path | no | none | Path to the checks TOML. Omit it and there are no deterministic checks at all — see the callout below. | +| `checks` | path | no | none | Path to the checks TOML. Even when omitted, `should_trigger` produces a per-case trigger expectation — see the callout below. | | `timeout_seconds` | integer | no | `900` | Per-case timeout passed to the agent run. | | `trials_per_case` | integer | no | `1` | Trials run per case; validated to `[1, 1000]` (`EVAL_INVALID_TRIALS_CONFIG` otherwise). | | `parallel` | integer | no | unset (CPU core count) | Max concurrent trials for one case. | @@ -46,7 +51,7 @@ fail_on_missing_agent = true # default true -With no `checks` file configured, there is effectively no deterministic signal: a case passes purely on the agent process exiting `0`. Point `checks` at a real file once you want the suite to assert anything about *what* the agent did. +With no `checks` file configured, each case still receives the required skill-consultation expectation implied by `should_trigger`. A successful process exit alone is not enough. Add a checks file for further deterministic outcome or adherence evidence. ### Validate the config @@ -103,14 +108,16 @@ greet-2,What is 2+2?,false,smoke, |---|---|---| | `id` | yes | Non-empty identifier per row. Used for `--case`, artifact folder names, and reports. | | `prompt` | yes | Text sent to the agent. Use CSV quoting (`"..."`, doubled `""` for an embedded quote) for commas or newlines. | -| `should_trigger` | yes | `true`/`1` (case-insensitive `true`) or anything else counts as `false`. **See the callout below — this column does not affect pass/fail.** | +| `should_trigger` | yes | `true`/`1` (case-insensitive `true`) or anything else counts as `false`. It creates a required per-case skill-consultation expectation; see below. | | `tags` | no | Comma-separated tags inside the cell (e.g. `"smoke,basic"`). Used by `eval run --tag `. | | `workspace_subdir` | no | Relative path that becomes the agent's working directory for the case. Under the default isolation it is created **inside the case's scratch workspace**, with fixture files copied in from the same path under the project root; under `--no-isolation` it resolves directly under the project root. Empty or omitted → workspace root. | Empty lines are skipped. -**`should_trigger` is inert.** The field's own doc comment in `aikit-evals` says it plainly: *"Whether the skill should trigger (documentation-only; checks.toml is authoritative for pass/fail)."* No scorer reads it. The same global check list from `checks.toml` runs against **every** case regardless of what `should_trigger` says, and with no checks file configured, pass/fail comes only from the agent process's exit code. There is currently no way to express a per-case "this prompt must NOT trigger the skill" expectation — treat the column as documentation for a human reader, not as a control the CLI acts on. Do not build a suite on the assumption that setting `should_trigger,false` makes the suite verify non-triggering; it does not. +**`should_trigger` affects pass/fail.** The engine creates an implicit required `skill_invoked` check for each case, with `expected` equal to the CSV value. An explicit `skill_invoked` check that applies to the case replaces the implicit one; `eval validate` rejects a contradictory polarity. Therefore `false` tests non-consultation and `true` tests consultation, subject to the selected backend's observability. + +This is adherence/selection evidence, not outcome correctness. A structured `Skill` call is direct evidence; supported decoders may also recognize a tool input that references the staged skill-document path. The path proxy indicates consultation, not that the instructions were read or followed. Pair positive cases with outcome checks or judges. ## 3. Checks: `checks.toml` (optional) @@ -137,8 +144,11 @@ expected = true | `skill` | string | no | — (omit to match *any* skill invocation) | | `expected` | bool | no | `true` (`false` asserts the skill did **not** run) | | `required` | bool | no | `true` | +| `cases` | string array | no | all cases (exact case IDs) | + +Matches a structured `Skill` invocation or, where the decoder supplies it, a tool input referring to the staged skill-document path. When `skill` is given, a structured invocation must exactly match its identifying field. A skill name merely mentioned in unrelated prose does not count. Path-reference matching improves portability but remains a consultation proxy rather than proof of adherence. -Matches a **structured `Skill` tool invocation** in the trace, not a substring of prose. When `skill` is given, it must **exactly equal** the invocation's skill-identifying input field (`skill`, `name`, or `skillName`) — `"foo"` does not match a `"foo-bar"` invocation, and a skill name merely mentioned in argument text does not count. This is the robust way to assert triggering — and, with `expected = false`, the only reliable way to assert non-triggering. +Every check type supports `cases = ["case-id", ...]`, selecting exact case IDs. An absent selector applies to every case. Use selectors for checks that should not run across a mixed suite. ### `trigger_expectation` @@ -229,7 +239,7 @@ Use it for signal you want visible in reports without gating the suite on it. ### How a case's pass/fail is decided -- No `checks` file configured (or it loads zero checks): the case **passes** iff the agent process exits `0`. +- No `checks` file configured (or it loads zero explicit checks): the case still receives the required trigger expectation generated from `should_trigger`. - Any checks loaded: the case **passes** only if every **required** check's `passed` is `true`. Checks marked `required = false` are reported but never fail the case. Among required checks there is no partial credit and no threshold — they are all-or-nothing per case. - A case whose checks are **all** advisory (`required = false`) has nothing that can fail it, so it passes as long as it does not time out. That is intentional, but it means an all-optional `checks.toml` is not a safety net. - A trial that times out is always `error`, independent of checks. @@ -262,8 +272,8 @@ See [Run evals](/evals-quality/run-evals) for the full day-to-day command set, a - [ ] `[tool.fastskill.eval]` present with a `prompts` path that resolves. - [ ] Prompts CSV has `id`, `prompt`, `should_trigger` headers (plus `tags` / `workspace_subdir` if you use them). -- [ ] You are **not** relying on `should_trigger` to gate pass/fail — it doesn't. -- [ ] `checks.toml` exists if you want real scoring signal, and uses `skill_invoked` (not a bare-name `trigger_expectation`) to assert that your skill fired. +- [ ] Each `should_trigger` value reflects the required consultation/non-consultation expectation; explicit `skill_invoked` checks do not contradict it. +- [ ] Outcome correctness is measured separately where consultation alone is insufficient. - [ ] `fastskill eval validate --all` passes and lists the agent you intend to run against as available. - [ ] Output directory convention agreed for `--output-dir` (local + CI).