Add GitHub Copilot CLI as a harness, with routing that needs no model credential

README.md

@@ -85,6 +85,8 @@ the repo under test — worth adding to `.gitignore`.
 * [docs/authoring-evals.md](docs/authoring-evals.md): writing a skill's tests.
 * [docs/usage.md](docs/usage.md): every flag, every workflow input, and what
   each check actually asserts.
+* [docs/harnesses.md](docs/harnesses.md): what an agent harness is, and how to
+  add one.
 
 ## Development
 

docs/harnesses.md

@@ -0,0 +1,148 @@
+<!--
+Copyright Advanced Micro Devices, Inc.
+
+SPDX-License-Identifier: MIT
+-->
+
+# Harnesses
+
+A **harness** is the agent that is handed a prompt and a workspace and does the
+work: Claude Code, GitHub Copilot CLI, Codex, and so on. It is a different
+question from where the agent runs. `--engine` says *where* (on the host, or in
+a container); the harness says *what*. The engines drive a harness as a
+subprocess, so the contract between them is small, and this page is that
+contract.
+
+**A harness defines the agent once, for both answers to *where*.** On the host the
+engine starts `command(spec)` as a subprocess: the executable resolved on the host,
+and the runner's environment underneath what `env(spec)` sets. A sandbox leg starts
+the same command inside the container, rather than delegating to a separate agent
+definition such as an `inspect_swe` one. Otherwise every agent would exist twice in
+this repo, with its own command line, environment, skill staging and event handling,
+free to drift apart.
+
+Two harnesses are registered, `claude-code` and `copilot`, and the contract exists
+so that another is an adapter and a set of tests rather than a change to the
+engines. `--harness` chooses one; see [usage.md](usage.md#which-engine-grades-a-run).
+
+## The harnesses
+
+| | `claude-code` | `copilot` |
+| --- | --- | --- |
+| Binary | `claude` | `copilot` |
+| Skills directory | `.claude/skills` | `.claude/skills` |
+| A skill activates by | a `Skill` tool call | a `skill` tool call |
+| Session lists its skills | yes (`system/init`) | no; `copilot skill list --json` does, without a model call |
+| Authenticates with | `ANTHROPIC_API_KEY` or `ANTHROPIC_AUTH_TOKEN` | `COPILOT_GITHUB_TOKEN`, `GH_TOKEN` or `GITHUB_TOKEN` |
+| Config isolated by | `CLAUDE_CONFIG_DIR` | `COPILOT_HOME` |
+| Prompt | stdin | stdin |
+| A run is billed in | dollars | premium requests |
+
+Things to know about `copilot` before reading its numbers:
+
+* **Its room always contains built-in skills.** Copilot ships a few of its own,
+  so they are present beside the skills a run stages and can win a case.
+* **It reports its answer separately from its closing event.** The harness
+  remembers the last assistant message and attaches it to the end of the run.
+* **It reports tokens once, at the end,** and no dollar cost, so a spend cap in
+  dollars cannot bind.
+* **It runs unsandboxed with every permission granted** (`--allow-all`), as the
+  host engine's Claude Code does, because a non-interactive run cannot answer a
+  prompt. Use a throwaway runner.
+* **It uses its own model names** (`claude-sonnet-5.5`, not `sonnet`), and
+  models differ greatly in how many premium requests one run costs.
+
+## What a harness provides
+
+A harness is a subclass of `skillscope.harness.Harness`, registered in
+`skillscope/harness/__init__.py`.
+
+| Member | What it is |
+| --- | --- |
+| `name` | The name reports and `--harness` use. |
+| `binary` | The executable's name. Whoever starts the process resolves it: on the host with `shutil.which`, in a container by the guest's `PATH`. |
+| `skills_dir` | Where, relative to the workspace, the harness discovers skills (`.claude/skills` for Claude Code). The engine stages each skill there. |
+| `has_skill_tool` | Whether activating a skill is a tool call in the stream. A harness that activates a skill by reading its `SKILL.md` sets this `False`, and routing reads activation from the file access instead. |
+| `prompt_on_stdin` | Whether the prompt is written to stdin. A harness that takes it as an argument builds it into `command` and sets this `False`; stdin is closed either way, because some CLIs wait on it. |
+| `require()` | Refuse to start **on the host** when the harness is not installed there. The default checks `binary` is on the runner's `PATH`. A leg that starts the agent in a container checks the image instead. |
+| `default_model`, `resolve_model(model)` | The model used when the caller names none, and how the harness names the one they did. A harness with its own names maps the generic aliases (`opus`, `sonnet`, `haiku`) onto them. |
+| `pin_model(model)` | The model a CI run is held to, so results stay comparable run to run. The default pins nothing. |
+| `credentials`, `can_isolate_room(environ)` | The environment variables a credential can come from. A routing run keeps the runner's own config and skills out of the session only when the harness authenticates from one of them. |
+| `supports_sandbox` | Whether a container leg exists (`--engine sandbox`). |
+| `cost_flags(max_budget_usd)` | The CLI's own spend controls for a case. Empty when it has none, which the report then says rather than implying a cap. |
+| `version()` | What `<binary> --version` prints, for the report. |
+| `preflight(model)` | One trivial prompt through the harness's own command, to prove before any case runs that the credential, the model name and the network work. The default is enough for most harnesses. |
+| `command(spec)` | The argv that runs one case, headless, streaming JSON events. The executable is named (`binary`), never an absolute path on this machine, so the same command can be started on the host or in a container. |
+| `env(spec)` | What the harness sets in the process's environment, and nothing else: its own settings, and whatever points it at `spec.config_dir` so the runner's own settings and skills stay out of the session. Not the runner's environment (`PATH`, `HOME`, every key on the machine), which a container must not inherit. Whoever starts the process puts this on top of the environment it has. |
+| `host_command(spec)`, `host_env(spec)` | What a run started on the host executes: `command(spec)` with its executable found by `shutil.which` (on Windows an npm-installed CLI is a `.cmd` shim that only that finds), and `os.environ` with `env(spec)` on top. A sandbox leg uses `command` and `env` directly. |
+| `normalize(event)` | Turns one event the harness printed into canonical events (below). Stateless. |
+| `normalizer()` | A translator for one run's events, which may keep state. The engine asks for a fresh one per case. Override it when an event cannot be translated on its own, for instance when the closing event does not repeat the answer and it has to be remembered from an earlier one. The default returns `normalize`. |
+
+`spec` is a `RunSpec`: the workspace, the prompt, the model and effort, an
+optional throwaway config directory, and any spend or session flags the caller
+built. The workspace and the config directory are strings naming a path wherever
+the agent runs: a host path on the host, a path inside the container in a sandbox.
+
+## The canonical event form
+
+The engines read one event shape, and a harness translates into it. It is
+Claude Code's stream-json, reduced to what is read:
+
+| Event | Carries |
+| --- | --- |
+| `{"type": "system", "subtype": "init", "skills": [...]}` | The skills the session registered. Optional: a harness that cannot say reports nothing, and the room check says it did not run rather than reporting an empty room. |
+| `{"type": "assistant", "message": {"content": [{"type": "tool_use", "name": ..., "input": {...}}]}}` | A tool call. Activating a skill is a `tool_use` named `Skill` with `input.skill` set. |
+| `{"type": "user", "message": {"content": [{"type": "tool_result", "content": ...}]}}` | A tool result. |
+| `{"type": "result", "result": "..."}` | The end of the run, and its closing answer. A stream without one is a run that did not finish. |
+
+A harness whose stream already has this shape keeps the default `normalize`,
+which passes events through. Any other maps its events onto these and returns an
+empty list for events that carry nothing the engines read. What a harness remembers about a stream belongs in the
+translator `normalizer()` returns, never on the harness itself: one harness
+instance serves every case in a run, and cases run concurrently. Every stop rule,
+budget, score and judge sees canonical events only, never the raw ones, so each
+is written once for every harness.
+
+## Adding a harness
+
+1. Subclass `Harness` in `skillscope/harness/<name>.py`, fill in the members
+   above, and register it in `skillscope/harness/__init__.py`.
+2. Record two real runs of the CLI, headless, on a trivial skill: one where it
+   activates the skill and finishes, one where it finishes without activating
+   one. Scrub tokens and paths. Put them in
+   `tests/fixtures/harness/<name>/trigger.jsonl` and `no_trigger.jsonl`, with an
+   `expected.json` saying what the engine should read out of each (see the
+   Claude Code one) and the CLI version they were recorded with.
+3. Run `python -m unittest tests.test_harness`. The contract test runs once per
+   registered harness and fails if there are no fixtures. It checks activation,
+   completion, the closing answer, that the command carries the model and effort
+   and no credential, that the config directory is isolated, that two runs translated at the same time
+   do not share memory, and that a skill is staged where the harness looks and
+   without its `evals/` directory.
+4. Say in the pull request how the harness differs from the others in ways a
+   reader of a report should know: how it activates skills, what it cannot
+   report, and what it needs to authenticate.
+
+A report from one harness is its own result. Harnesses send different system
+prompts, offer different tools, and activate skills in different ways, so a score
+from one is not comparable with a score from another.
+
+## What is not yet behind the contract
+
+The contract covers staging, the command, the environment, the events, the model,
+the credential check, the spend controls and the preflight. These still assume
+Claude Code, and a harness other than it cannot yet use them:
+
+* **the judge** that grades `behavioral` expectations goes through an Anthropic
+  model whatever agent produced the run, so `behavioral` is refused on any other
+  harness;
+* **the room check**: Claude Code reports at session start which skills it
+  loaded, and the engine compares that with what was staged. A harness with no
+  such event leaves the check not run;
+* **the sandbox leg**: `--engine claude-code` still runs Claude Code through
+  `inspect_swe`, with its own command line, environment, skill staging and event
+  handling. It should start the harness's `command(spec)` in the container, so that
+  each agent is defined once. Until it does, a harness other than Claude Code has no
+  sandboxed leg;
+* **`.claude/skills`** as the path routing treats as a skills inspection, rather
+  than a harness's own `skills_dir`.

docs/usage.md

@@ -249,18 +249,39 @@ environment gets one matrix, labels and all.
 
 ## Which engine grades a run
 
-`--engine` chooses what actually runs the cases. The dataset, the CLI and the
-reports are identical whichever you pick; only the thing driving the agent
-changes.
+Two choices decide what runs the cases, and they are separate: `--engine` is
+*where* the agent runs, and `--harness` is *which* agent. The dataset and the
+reports are identical whichever you pick.
 
-| `--engine` | What runs | Runs in | Needs |
-| --- | --- | --- | --- |
-| `claude-code-no-sandbox` (default) | The real CLI, under `inspect_ai` | the host | the CLI on `PATH` |
-| `claude-code` | The real CLI, via `inspect_swe` | a sandbox | `skillscope[verify]`, Linux only |
+| `--engine` | Where it runs | Needs |
+| --- | --- | --- |
+| `host` (default) | the host, under `inspect_ai` | the harness's CLI on `PATH` |
+| `sandbox` | a container, via `inspect_swe` | `skillscope[verify]`, Linux only, and a harness with a sandboxed leg |
 
-Both drive the agent a skill is written for, so what differs between them
-is **where the agent runs**, not what it is. That is the axis worth choosing
-along: the host measures the machine as it is, with whatever else is installed
+`claude-code-no-sandbox` and `claude-code` are the older spellings of `host` and
+`sandbox`. They name an agent as well as a place, so they mean Claude Code, and
+combining one with a different `--harness` is refused.
+
+| `--harness` | Agent | Sandboxed leg |
+| --- | --- | --- |
+| `claude-code` (default) | Claude Code | yes |
+| `copilot` | GitHub Copilot CLI | no |
+
+`--harness` and `--engine` can also come from `$SKILLSCOPE_HARNESS` and
+`$SKILLSCOPE_ENGINE`. Each harness names models its own way and has its own
+default; `--model opus`, `sonnet` and `haiku` mean the right thing for every
+harness, and CI holds each harness to one model. A report from one harness is its
+own result and is not comparable with another's. How a harness works, and how to
+add one, is in [harnesses.md](harnesses.md).
+
+**`copilot` today runs routing only.** It authenticates from
+`COPILOT_GITHUB_TOKEN`, `GH_TOKEN` or `GITHUB_TOKEN`, needs no model credential,
+and in Actions needs `permissions: copilot-requests: write` in the calling
+workflow. `behavioral` is refused on it until the judge that grades its
+expectations is available through the harness. Its preflight sends one real
+prompt, which is billed like any other.
+
+Where the agent runs is the axis worth choosing along: the host measures the machine as it is, with whatever else is installed
 on it, and the sandbox measures the skill alone. When the two disagree, the
 disagreement is usually a fact about one of those environments rather than
 about the skill -- which is a thing one engine on its own cannot tell you.

pyproject.toml

@@ -62,7 +62,7 @@ verify = ["skillscope[inspect]", "inspect-swe>=0.2.71"]
 skillscope = "skillscope.cli:main"
 
 [tool.setuptools]
-packages = ["skillscope", "skillscope.engine"]
+packages = ["skillscope", "skillscope.engine", "skillscope.harness"]
 
 [tool.setuptools.package-data]
 skillscope = ["data/*.json", "schema/*.json"]

skillscope/agent.py

@@ -77,17 +77,23 @@ def enforce_model_policy(model: str | None) -> str | None:
     return AUTOMATED_MODEL
 
 
-def claude_env() -> dict[str, str]:
-    """Environment for `claude` subprocesses.
+def claude_settings() -> dict[str, str]:
+    """What `claude` is told, beyond whatever environment it is started in.
 
     Disable the CLI's internal retry loop by default so a network/auth problem
     (e.g. not connected to the network that can reach the API) fails fast
     instead of being retried into a long, confusing hang. The caller can still
     override by exporting ``CLAUDE_CODE_MAX_RETRIES``.
+
+    Only the settings, not the runner's environment: where the CLI runs decides
+    what else it is given.
     """
-    env = dict(os.environ)
-    env.setdefault("CLAUDE_CODE_MAX_RETRIES", "0")
-    return env
+    return {"CLAUDE_CODE_MAX_RETRIES": os.environ.get("CLAUDE_CODE_MAX_RETRIES", "0")}
+
+
+def claude_env() -> dict[str, str]:
+    """Environment for `claude` subprocesses started on this machine."""
+    return {**os.environ, **claude_settings()}
 
 
 def check_api_reachable(model: str | None = DEFAULT_MODEL, timeout: float = 60) -> tuple[bool, str]: