feat(evals): define and verify skill-driven eval authoring

CONTEXT.md

@@ -206,6 +206,23 @@ _Avoid_: Sweep, report (a rendering), summary (one run directory's, upstream), r
 A rendering of measurements already made — one run directory's summary, or one or more scorecards — as a table, JSON, or one self-contained HTML file with every asset embedded. Producing a report never runs an agent and never re-scores; it reads scorecard files, not run directories, so it can be made anywhere the files are.
 _Avoid_: Dashboard, page, export, scorecard (the measurement it renders)
 
+### Eval authoring
+
+**Eval-authoring skill**:
+A skill that guides an authoring agent through designing evaluation suites according to FastSkill's layouts and standards. It is distinct from the target skill being evaluated.
+
+**Authoring agent**:
+The user-chosen agentic tool that helps design and edit an evaluation suite. It need not be the agent evaluated by that suite.
+
+**Outcome correctness**:
+Whether the result of a skill-assisted task satisfies its intended acceptance criteria, independently of the prescribed method.
+
+**Skill adherence**:
+Whether the agent follows the skill's prescribed method. It is distinct from outcome correctness and is mandatory only where the author identifies the method as a requirement.
+
+**Coverage outline**:
+A proposed selection of behaviors and boundaries to evaluate for a skill, before detailed cases and graders are authored. It is not a Benchmark, which identifies an executable measurement definition.
+
 ## Resolved decisions
 
 - **Historical `sync` is removed.** Modern targets read skills directly. Propagation has exactly two

crates/fastskill-cli/src/main.rs

@@ -8,6 +8,7 @@
 //! `Arc<FsState>` is captured at registration time by each command closure —
 //! no `Any`-downcasting of `AppContext` is needed.
 
+#![recursion_limit = "256"]
 #![allow(clippy::unwrap_used, clippy::expect_used, clippy::panic)]
 
 mod arg_helpers;

crates/fastskill-evals/tests/documented_check_contract.rs

@@ -7,7 +7,9 @@
 
 #![allow(clippy::expect_used, clippy::unwrap_used)]
 
-use fastskill_evals::{run_checks, suite_passes, ChecksToml};
+use fastskill_evals::{
+    effective_checks, run_checks, suite_passes, validate_case_checks, CheckDefinition, ChecksToml,
+};
 use std::path::Path;
 
 /// `max_command_count` is a legacy alias; results report the canonical
@@ -175,3 +177,67 @@ required = true
         Path::new("/tmp")
     )));
 }
+
+/// The setup guide's central authoring contract: `should_trigger` creates a
+/// required per-case expectation even when checks.toml contributes nothing.
+#[test]
+fn should_trigger_generates_positive_and_negative_required_checks() {
+    for expected in [true, false] {
+        let checks = effective_checks(&[], "case-1", expected);
+        assert_eq!(checks.len(), 1);
+        assert!(matches!(checks[0], CheckDefinition::SkillInvoked { .. }));
+        if let CheckDefinition::SkillInvoked {
+            expected: actual,
+            required,
+            cases,
+            ..
+        } = &checks[0]
+        {
+            assert_eq!(*actual, expected);
+            assert!(*required);
+            assert_eq!(cases.as_deref(), Some(["case-1".to_string()].as_slice()));
+        }
+    }
+}
+
+/// Exact case selectors control applicability, and an explicit trigger check
+/// replaces the implicit one only on the selected case. A contradictory
+/// replacement is rejected rather than silently changing CSV intent.
+#[test]
+fn case_scoping_and_explicit_trigger_precedence_match_the_docs() {
+    let parsed: ChecksToml = toml::from_str(
+        r#"
+[[check]]
+name = "skill_invoked"
+expected = true
+cases = ["positive"]
+
+[[check]]
+name = "command_contains"
+pattern = "invoice_number"
+cases = ["positive"]
+"#,
+    )
+    .unwrap();
+
+    let positive = effective_checks(&parsed.checks, "positive", true);
+    assert_eq!(
+        positive.len(),
+        2,
+        "explicit trigger plus scoped outcome check"
+    );
+    let negative = effective_checks(&parsed.checks, "negative", false);
+    assert_eq!(negative.len(), 1, "other-case checks must be excluded");
+    assert!(matches!(
+        negative[0],
+        CheckDefinition::SkillInvoked {
+            expected: false,
+            ..
+        }
+    ));
+
+    assert!(validate_case_checks(&parsed.checks, "positive", true).is_ok());
+    let error = validate_case_checks(&parsed.checks, "positive", false).unwrap_err();
+    assert!(error.to_string().contains("EVAL_CHECKS_INVALID"));
+    assert!(error.to_string().contains("positive"));
+}

docs/adr/0011-skill-first-eval-authoring.md

@@ -0,0 +1,22 @@
+# Start assisted eval authoring from the skill
+
+Status: accepted. Date: 2026-09-12.
+
+## Context
+
+Designing meaningful eval cases requires human attention to intended outcomes, coverage, and grading criteria. The initial authoring experience could start from a skill that needs tests or from an observed session that should become a regression case. These solve different entry-point problems.
+
+## Decision
+
+The first assisted-authoring experience starts from an existing skill and helps its author design the skill's evaluation tests. It does not require session history or a known failure.
+
+Inspecting sessions to discover regression cases or improve skills is deferred for a separate discussion. It is not a prerequisite or committed deliverable of the initial change.
+
+This decision settles scope only. It does not select the authoring model, interaction style, suite representation, calibration method, acceptance workflow, or implementation ownership.
+
+## Consequences
+
+- Initial design and acceptance criteria must cover a skill with no prior usage sessions.
+- Session import adapters and transcript reconstruction are outside the initial scope.
+- The next decisions must establish how intended behavior is clarified and how generated cases and graders are validated.
+- Future session-derived cases can be considered independently without delaying skill-first authoring.

docs/adr/0012-separate-outcomes-from-adherence.md

@@ -0,0 +1,7 @@
+# Separate outcome correctness from skill adherence
+
+Status: accepted. Date: 2026-09-12.
+
+Generated evals must distinguish a useful, correct outcome from compliance with a skill's prescribed method. Measure these separately and make adherence mandatory only when the author identifies it as a requirement. Automatically treating every instruction as correctness would encode existing mistakes and could reward unnecessary tool use; treating outcomes alone as sufficient would miss genuinely required procedures.
+
+This decision does not choose the storage schema or change the current engine's implicit trigger checks. Implementation must reconcile those existing checks with the authoring distinction before claiming support. Pilot success establishes evidence about cases and graders, not that the skill passes the suite.

docs/adr/0013-skill-driven-eval-authoring.md

@@ -0,0 +1,15 @@
+# Drive eval authoring through a skill and the CLI
+
+Status: accepted. Date: 2026-09-12.
+
+Users design evals in their chosen agentic tool with FastSkill CLI and an eval-authoring skill that explains its layouts and standards. The authoring skill guides inspection, coverage discussion, case construction, and iteration; FastSkill and its eval engine define and validate artifacts, execute evaluations, and preserve measurement evidence. This keeps conversational judgment in the user's agent and enforceable mechanics in code, avoiding a separate model integration and interview orchestrator inside FastSkill.
+
+Drafts use ordinary files and version control. A dedicated interview engine and draft-state service are deferred. Requirements that CI relies on must be represented in artifacts and checked by code; instructions in the authoring skill alone cannot establish verified status.
+
+Authoring-tool choice is independent of evaluation-runtime support. An agent able to follow the skill, edit files, and invoke the CLI can author a suite without having a FastSkill evaluation backend. Executing that agent as the subject of an eval still requires a supported backend and observable evidence.
+
+The authoring agent owns conversational authorization: it uses the user's existing authorization or asks about pilot scope before invoking the CLI. FastSkill validates explicit parameters and returns results or errors; it does not infer consent, remember conversational approvals, or conduct an interview. A spending ceiling is not a prerequisite of this change and must not be promised unless implemented.
+
+The skill checks CLI compatibility and follows authoritative format references and validation output. Exact supported versions and distribution location remain implementation details to investigate. Review covers the outline and then the suite, with individual questions for ambiguities. Calibration labels follow author-confirmed criteria or independently verifiable facts.
+
+The first delivery is the authoring skill with supporting documentation and validation corrections for existing contracts. Unsupported test intentions are reported explicitly; new grader types, schemas, or execution capabilities require separate proposals. The PRDs are drafts for external review, not a claim of implementation readiness or completed verification.