Grade a skill's cases without the skill, and report which ones still need it

README.md

@@ -15,13 +15,14 @@ SPDX-License-Identifier: MIT
 
 ![Skillscope](assets/banner.png)
 
-A test harness for AI agent skills. Install the harness, or point a workflow at your repo. Skillscope enables three main types of tests:
+A test harness for AI agent skills. Install the harness, or point a workflow at your repo. Skillscope enables four main types of tests:
 
 * **Structural**: Checks whether the skill is well structured (contains the expected files, a proper hierarchy, etc.).
 * **Routing**: Checks whether the skill triggers when it should and stays quiet when it shouldn't. You may also point to more than one skill here to explore how skills interfere with each other's routing.
 * **Behavioral**: Runs the agent end to end and uses an LLM judge to confirm the skill behaves correctly.
+* **Staleness**: Runs the behavioral cases a second time with the skill withheld, and compares. Tells you whether the cases still need the skill, or whether the model has caught up and the suite is passing on its own.
 
-All three read one dataset per skill, `<skill>/evals/evals.json`. How to write one is in [docs/authoring-evals.md](docs/authoring-evals.md).
+All four read one dataset per skill, `<skill>/evals/evals.json`. How to write one is in [docs/authoring-evals.md](docs/authoring-evals.md).
 
 > [!IMPORTANT]
 > **Early Days**: Flags, workflow inputs, and report formats may evolve quickly as this repo takes shape.
@@ -37,6 +38,7 @@ skillscope structural                        # no agent, no tokens
 skillscope structural --external             # the same, plus checking external URLs
 skillscope behavioral --skill my-skill       # needs an authenticated `claude` CLI
 skillscope routing --routing-room my-skill,its-neighbour  # also needs a key in the environment
+skillscope staleness --skill my-skill        # the same key: its baseline has to be an empty room
 ```
 
 `routing` needs `ANTHROPIC_API_KEY` or `ANTHROPIC_AUTH_TOKEN` set, and refuses
@@ -47,6 +49,12 @@ describe a room nobody asked for. `behavioral` installs one skill and grades
 what the agent produced, so a stray skill is at worst noise there, and a plain
 `claude login` is fine.
 
+`staleness` is in `routing`'s camp, not `behavioral`'s, and refuses on the
+same terms. Its baseline half has to be a room with **no** skills in it, so
+the runner's own are not noise there — they are the measurement. Either set
+one of those variables, or use `--engine claude-code`, whose guest has no
+user-level skills at all.
+
 ## Run it from your CI
 
 Simply add an `evals.json` file to your skill and a workflow that points to our reusable workflow. This will triger all graders to run directly from your repo.
@@ -72,6 +80,7 @@ You may also choose to customize or even disable some tests if you prefer it. Se
 | `skillscope structural` | Skill folders, datasets, and markdown references. |
 | `skillscope routing` | Which skill fires, with several installed together. |
 | `skillscope behavioral` | What a skill does once it has fired. |
+| `skillscope staleness` | Whether those cases still need the skill. |
 | `skillscope select` | The CI plan for a change, as JSON. |
 | `skillscope list-skills` | The skills that have a dataset, as JSON. |
 | `skillscope template` | The dataset template a new skill starts from. |

action.yml

@@ -16,7 +16,7 @@ description: >-
 
 inputs:
   command:
-    description: 'The skillscope subcommand: structural, routing, behavioral, select, list-skills, template.'
+    description: 'The skillscope subcommand: structural, routing, behavioral, staleness, select, list-skills, template.'
     required: true
   args:
     description: "Further arguments for the subcommand, shell-quoted."
@@ -47,8 +47,10 @@ inputs:
     default: "true"
   install-claude:
     description: >-
-      Install Node and the claude CLI. Needed by `routing` and `behavioral`,
-      pointless for `structural` and `select`.
+      Install Node and the claude CLI. Needed by `routing`, `behavioral` and
+      `staleness` -- staleness drives the behavioral leg for both of its
+      halves, so it needs the CLI exactly as much. Pointless for `structural`
+      and `select`.
     required: false
     default: "false"
   node-version:

docs/authoring-evals.md

@@ -224,8 +224,15 @@ skillscope structural                               # your folder, prose, and da
 skillscope behavioral --skill <your-skill>          # your skill, end to end
 skillscope routing --routing-room <your-skill>,<a-neighbour>
 skillscope routing --only <case-id> --keep-logs logs # one case, keeping the transcript
+skillscope staleness --skill <your-skill>           # do your cases still need the skill?
 ```
 
+`staleness` is the one aimed at this file's subject. It runs your behavioral
+cases twice, once with the skill installed and once without, and reports which
+of them passed either way. Those are cases that are not evidence about the
+skill — either it has stopped mattering, or the case never tested it. It costs
+two runs, so it belongs on a schedule rather than in the loop.
+
 `structural` is instant and needs no agent. Everything else needs the `claude`
 CLI authenticated, plus whatever your own cases need. Flags, what each check
 asserts, the routing bar, and how to read a report are in

docs/usage.md

@@ -247,6 +247,117 @@ Legs with a scoped environment run as a separate job, because a job's
 credentials are fixed before its matrix expands. A repo that declares no scoped
 environment gets one matrix, labels and all.
 
+## Is the skill still earning its place?
+
+A behavioral run says a skill's cases pass. It cannot say they pass *because
+of* the skill, and those are different claims: an expectation any competent
+agent meets is a measurement of the model with the skill's name on it.
+
+`skillscope staleness` runs the same cases twice, once with the skill
+installed and once with nothing installed, and reports which ones moved.
+
+```bash
+skillscope staleness --skill my-skill
+```
+
+On the default engine this needs `ANTHROPIC_API_KEY` or `ANTHROPIC_AUTH_TOKEN`
+in the environment and refuses to start without one, the same way `routing`
+does and for a stronger version of the same reason: the key is what lets the
+run redirect the CLI away from the runner's own config dir, and the baseline
+half has to be a room with no skills in it. Without it, every skill installed
+on the machine joins the baseline, and a case one of them solves there is
+reported as a case the skill under test was not needed for. A plain `claude
+login` is enough for `behavioral` and is not enough here. `--engine
+claude-code` needs no key for this, since its guest has no user-level skills
+at all.
+
+| Verdict | Means |
+| --- | --- |
+| `earns_it` | passed with the skill, failed without it — the case discriminates |
+| `harmful` | failed with the skill, passed without it — the skill steered the agent wrong |
+| `no_signal` | passed either way — says nothing about the skill |
+| `fails_either_way` | failed both times — a broken case, or a hard one |
+| `guards_only` | asserts nothing the skill could be credited for — see the guard rails below |
+| `ungraded` | nothing could be concluded: a run errored on one side or both, a case was missing from a side, its baseline was not empty, or whether the baseline was empty could not be established |
+
+A verdict says what changed, not why. So the graded side also records whether
+the agent *reached for* the skill, using the same detector the routing leg
+uses: installing a skill does not make an agent use it, and "installed and the
+outcome differed" is a weaker claim than "the skill fired and the outcome
+differed". An `earns_it` where the skill never fired is reported as the lead it
+is rather than as a result. Guard rails -- `unexpected_behavior` -- are kept
+out of the verdict entirely, since a good one holds whether or not the skill is
+installed; they are counted on their own, where a guard broken *with* the skill
+installed is the serious finding.
+
+`no_signal` is the verdict worth acting on, and it is two findings wearing one
+name: either the skill has stopped mattering, or the case was never evidence
+about it in the first place. The second is the more common, which makes this a
+check on the dataset as much as on the skill.
+
+Both halves write which side they were, whose skill they were about, where that
+skill was on disk, and which invocation wrote them into their inspect log
+metadata (`skillscope_baseline`, `skillscope_skill`, `skillscope_skill_dir`,
+`skillscope_staleness_run`). So a finished run can be looked at again without
+paying for it twice:
+
+```bash
+skillscope staleness --from-logs .skillscope/logs
+```
+
+That reaches no model and runs no preflight, having nothing to spend.
+
+Every key is load-bearing. `skillscope_skill_dir` is what lets a rebuild judge
+whether the baseline was really empty: the check recognises a tool call that
+reached the skill by comparing against the skill's absolute path, so a reader
+that worked that path out for itself would be working out its own rather than
+the run's, and would report a baseline the run had voided as a clean one. A log
+written without the key is read as an unknown room rather than an empty one.
+
+The run id is the key that makes this safe to point at `.skillscope/logs`. That directory is shared with `behavioral`
+and `routing`, is gitignored wholesale, and nothing prunes it — and every
+ordinary `behavioral` run writes the same `skillscope_baseline: false` a graded
+half does, with no run id. A plain `behavioral` log is therefore shaped exactly
+like a graded half, and always will be, since only `staleness` mints an id.
+Pairing on the skill alone would splice an unrelated run from last month in as
+this comparison's "with skill" side, under a table that reads exactly as
+confidently as a real one.
+
+So logs carrying no run id are passed over, and a directory of plain
+`behavioral` logs rebuilds as nothing rather than as a comparison. Logs written
+before the id existed are passed over for the same reason; `--unlabelled-logs`
+reads them:
+
+```bash
+skillscope staleness --from-logs .skillscope/logs --unlabelled-logs
+```
+
+That asserts the directory holds nothing else for the skills in it. Even then,
+a group of unlabelled logs needs a baseline half, which nothing but `staleness`
+ever writes, so ordinary `behavioral` logs stay out whatever flags you pass.
+Where one side holds several unlabelled logs there is a choice to make and
+nothing to make it with, so every case in that group is reported `ungraded`.
+
+The model and effort are not reported for a rebuild: nothing reads them off
+the logs, and the logs need not agree on them.
+
+Budget more wall clock than a `behavioral` run of the same cases, and more
+than double it. `--timeout` defaults to twice what the single-pass commands
+use for that reason. Both passes share one `--timeout`, and the baseline is
+reliably the slower half: the agent holding the skill goes straight to the
+work, while the one without it explores. On a two-case skill the graded pass
+took 2m23s and the baseline had not finished its second case half an hour
+later. A bound that expires partway reports the unreached cases as `ungraded`,
+after the graded half has already been paid for.
+
+It reports and never gates, for the reason the `claude-code` leg does not gate
+either: the agent is nondeterministic on both sides, so a single pair is a
+place to look rather than a verdict. [`tools/benchmark_engines.py`](../tools/benchmark_engines.py)
+is the shape of the honest version of this comparison — it measures agreement
+against how much one engine already disagrees with itself — and a staleness
+number deserves the same scepticism. It also costs twice what `behavioral`
+costs, which is why no workflow runs it for you.
+
 ## Which engine grades a run
 
 `--engine` chooses what actually runs the cases. The dataset, the CLI and the
@@ -279,7 +390,9 @@ The host leg cannot read the CLI's session-init event, so it has no way to
 notice such contamination or report it -- and therefore refuses to run a
 routing leg at all unless `ANTHROPIC_API_KEY` or `ANTHROPIC_AUTH_TOKEN` is
 set, which is what lets it redirect the CLI's config dir away from the
-runner's own. Refusing beats being
+runner's own. It refuses a `staleness` run on the same terms and for a
+stronger version of the same reason: there the empty room is not one input to
+the measurement, it is the measurement. Refusing beats being
 quietly wrong about every case.
 
 Both legs stop at the decision, by different routes. `claude-code`'s calls

pyproject.toml

@@ -19,7 +19,7 @@ authors = [{ name = "Advanced Micro Devices, Inc." }]
 #
 # It was an extra while `legacy` existed: that engine was standard library only,
 # so a repo that had not migrated installed nothing new. `legacy` is gone, and
-# keeping the extra would ship a tool whose two graded commands both fail on a
+# keeping the extra would ship a tool whose graded commands all fail on a
 # plain `pip install` -- which is what CI found, by running the suite without
 # extras and getting `ModuleNotFoundError: inspect_ai` from the default engine.
 #

skillscope/behavior.py

@@ -29,6 +29,18 @@ class BehaviorOutcome:
     elapsed_s: float
     checks: list[dict] = field(default_factory=list)
     error: str | None = None
+    # The skill the agent actually reached for, if any. A behavioral run
+    # installs a skill but cannot make the agent use it, and until this was
+    # recorded a case that passed was credited to a skill that may never have
+    # fired. `skillscope staleness` is what the distinction is for: "installed
+    # and the outcome differed" is a weaker claim than "the skill fired and the
+    # outcome differed", and only the second is evidence about the skill.
+    activated: str | None = None
+    # Why this run was not the room it claimed to be: another skill fired, or
+    # the agent read the skill's own source off disk. Harmless noise in a
+    # behavioral run, which grades what the agent produced. Fatal to a
+    # staleness baseline, whose whole claim is that nothing was available.
+    contaminated: str | None = None
     # Set when the failure was the model provider's rather than the skill's.
     # A behavioral case that failed on a gateway 504 is a skill nobody
     # measured, and reporting it beside one that genuinely failed its