diff --git a/README.md b/README.md index f3600f0..5db84d9 100644 --- a/README.md +++ b/README.md @@ -85,6 +85,8 @@ the repo under test — worth adding to `.gitignore`. * [docs/authoring-evals.md](docs/authoring-evals.md): writing a skill's tests. * [docs/usage.md](docs/usage.md): every flag, every workflow input, and what each check actually asserts. +* [docs/harnesses.md](docs/harnesses.md): what an agent harness is, and how to + add one. ## Development diff --git a/docs/harnesses.md b/docs/harnesses.md new file mode 100644 index 0000000..67818cb --- /dev/null +++ b/docs/harnesses.md @@ -0,0 +1,112 @@ + + +# Harnesses + +A **harness** is the agent that is handed a prompt and a workspace and does the +work: Claude Code, GitHub Copilot CLI, Codex, and so on. It is a different +question from where the agent runs. `--engine` says *where* (on the host, or in +a container); the harness says *what*. The engines drive a harness as a +subprocess, so the contract between them is small, and this page is that +contract. + +**A harness defines the agent once, for both answers to *where*.** On the host the +engine starts `command(spec)` as a subprocess: the executable resolved on the host, +and the runner's environment underneath what `env(spec)` sets. A sandbox leg starts +the same command inside the container, rather than delegating to a separate agent +definition such as an `inspect_swe` one. Otherwise every agent would exist twice in +this repo, with its own command line, environment, skill staging and event handling, +free to drift apart. + +Today one harness is registered, `claude-code`. The contract exists so that a +second one is an adapter and a set of tests rather than a change to the engines. + +## What a harness provides + +A harness is a subclass of `skillscope.harness.Harness`, registered in +`skillscope/harness/__init__.py`. + +| Member | What it is | +| --- | --- | +| `name` | The name reports and `--harness` use. | +| `binary` | The executable's name. Whoever starts the process resolves it: on the host with `shutil.which`, in a container by the guest's `PATH`. | +| `skills_dir` | Where, relative to the workspace, the harness discovers skills (`.claude/skills` for Claude Code). The engine stages each skill there. | +| `has_skill_tool` | Whether activating a skill is a tool call in the stream. A harness that activates a skill by reading its `SKILL.md` sets this `False`, and routing reads activation from the file access instead. | +| `prompt_on_stdin` | Whether the prompt is written to stdin. A harness that takes it as an argument builds it into `command` and sets this `False`; stdin is closed either way, because some CLIs wait on it. | +| `require()` | Refuse to start **on the host** when the harness is not installed there. The default checks `binary` is on the runner's `PATH`. A leg that starts the agent in a container checks the image instead. | +| `command(spec)` | The argv that runs one case, headless, streaming JSON events. The executable is named (`binary`), never an absolute path on this machine, so the same command can be started on the host or in a container. | +| `env(spec)` | What the harness sets in the process's environment, and nothing else: its own settings, and whatever points it at `spec.config_dir` so the runner's own settings and skills stay out of the session. Not the runner's environment (`PATH`, `HOME`, every key on the machine), which a container must not inherit. Whoever starts the process puts this on top of the environment it has. | +| `host_command(spec)`, `host_env(spec)` | What a run started on the host executes: `command(spec)` with its executable found by `shutil.which` (on Windows an npm-installed CLI is a `.cmd` shim that only that finds), and `os.environ` with `env(spec)` on top. A sandbox leg uses `command` and `env` directly. | +| `normalize(event)` | Turns one event the harness printed into canonical events (below). Stateless. | +| `normalizer()` | A translator for one run's events, which may keep state. The engine asks for a fresh one per case. Override it when an event cannot be translated on its own, for instance when the closing event does not repeat the answer and it has to be remembered from an earlier one. The default returns `normalize`. | + +`spec` is a `RunSpec`: the workspace, the prompt, the model and effort, an +optional throwaway config directory, and any spend or session flags the caller +built. The workspace and the config directory are strings naming a path wherever +the agent runs: a host path on the host, a path inside the container in a sandbox. + +## The canonical event form + +The engines read one event shape, and a harness translates into it. It is +Claude Code's stream-json, reduced to what is read: + +| Event | Carries | +| --- | --- | +| `{"type": "system", "subtype": "init", "skills": [...]}` | The skills the session registered. Optional: a harness that cannot say reports nothing, and the room check says it did not run rather than reporting an empty room. | +| `{"type": "assistant", "message": {"content": [{"type": "tool_use", "name": ..., "input": {...}}]}}` | A tool call. Activating a skill is a `tool_use` named `Skill` with `input.skill` set. | +| `{"type": "user", "message": {"content": [{"type": "tool_result", "content": ...}]}}` | A tool result. | +| `{"type": "result", "result": "..."}` | The end of the run, and its closing answer. A stream without one is a run that did not finish. | + +A harness whose stream already has this shape keeps the default `normalize`, +which passes events through. Any other maps its events onto these and returns an +empty list for events that carry nothing the engines read. What a harness remembers about a stream belongs in the +translator `normalizer()` returns, never on the harness itself: one harness +instance serves every case in a run, and cases run concurrently. Every stop rule, +budget, score and judge sees canonical events only, never the raw ones, so each +is written once for every harness. + +## Adding a harness + +1. Subclass `Harness` in `skillscope/harness/.py`, fill in the members + above, and register it in `skillscope/harness/__init__.py`. +2. Record two real runs of the CLI, headless, on a trivial skill: one where it + activates the skill and finishes, one where it finishes without activating + one. Scrub tokens and paths. Put them in + `tests/fixtures/harness//trigger.jsonl` and `no_trigger.jsonl`, with an + `expected.json` saying what the engine should read out of each (see the + Claude Code one) and the CLI version they were recorded with. +3. Run `python -m unittest tests.test_harness`. The contract test runs once per + registered harness and fails if there are no fixtures. It checks activation, + completion, the closing answer, that the command carries the model and effort + and no credential, that the config directory is isolated, that two runs translated at the same time + do not share memory, and that a skill is staged where the harness looks and + without its `evals/` directory. +4. Say in the pull request how the harness differs from the others in ways a + reader of a report should know: how it activates skills, what it cannot + report, and what it needs to authenticate. + +A report from one harness is its own result. Harnesses send different system +prompts, offer different tools, and activate skills in different ways, so a score +from one is not comparable with a score from another. + +## What is not yet behind the contract + +The contract covers staging, the command, the environment and the events. These +still assume Claude Code, and a second harness will need to move them behind it: + +* choosing a harness from the command line (there is no `--harness` flag yet); +* the reachability preflight and the credential check that decides whether the + runner's own config can be kept out of a routing run (`ANTHROPIC_*`); +* the spend-limit flags, which are probed from `claude --help`; +* the CI model pin, which forces `opus`; +* the judge, which uses an Anthropic model through `inspect_ai`; +* **the sandbox leg**: `--engine claude-code` still runs Claude Code through + `inspect_swe`, with its own command line, environment, skill staging and event + handling. It should start the harness's `command(spec)` in the container, so that + each agent is defined once. Until it does, a harness other than Claude Code has no + sandboxed leg; +* `.claude/skills` as the path routing treats as a skills inspection, rather than + a harness's own `skills_dir`. diff --git a/docs/usage.md b/docs/usage.md index 442d1b1..3ad6a34 100644 --- a/docs/usage.md +++ b/docs/usage.md @@ -259,7 +259,8 @@ changes. | `claude-code` | The real CLI, via `inspect_swe` | a sandbox | `skillscope[verify]`, Linux only | Both drive the agent a skill is written for, so what differs between them -is **where the agent runs**, not what it is. That is the axis worth choosing +is **where the agent runs**, not what it is. What it is -- the harness -- is a +separate question, answered in [harnesses.md](harnesses.md). That is the axis worth choosing along: the host measures the machine as it is, with whatever else is installed on it, and the sandbox measures the skill alone. When the two disagree, the disagreement is usually a fact about one of those environments rather than diff --git a/pyproject.toml b/pyproject.toml index 3ab8e4e..3b13eb0 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -62,7 +62,7 @@ verify = ["skillscope[inspect]", "inspect-swe>=0.2.71"] skillscope = "skillscope.cli:main" [tool.setuptools] -packages = ["skillscope", "skillscope.engine"] +packages = ["skillscope", "skillscope.engine", "skillscope.harness"] [tool.setuptools.package-data] skillscope = ["data/*.json", "schema/*.json"] diff --git a/skillscope/agent.py b/skillscope/agent.py index c37fc3c..8a2c551 100644 --- a/skillscope/agent.py +++ b/skillscope/agent.py @@ -77,17 +77,23 @@ def enforce_model_policy(model: str | None) -> str | None: return AUTOMATED_MODEL -def claude_env() -> dict[str, str]: - """Environment for `claude` subprocesses. +def claude_settings() -> dict[str, str]: + """What `claude` is told, beyond whatever environment it is started in. Disable the CLI's internal retry loop by default so a network/auth problem (e.g. not connected to the network that can reach the API) fails fast instead of being retried into a long, confusing hang. The caller can still override by exporting ``CLAUDE_CODE_MAX_RETRIES``. + + Only the settings, not the runner's environment: where the CLI runs decides + what else it is given. """ - env = dict(os.environ) - env.setdefault("CLAUDE_CODE_MAX_RETRIES", "0") - return env + return {"CLAUDE_CODE_MAX_RETRIES": os.environ.get("CLAUDE_CODE_MAX_RETRIES", "0")} + + +def claude_env() -> dict[str, str]: + """Environment for `claude` subprocesses started on this machine.""" + return {**os.environ, **claude_settings()} def check_api_reachable(model: str | None = DEFAULT_MODEL, timeout: float = 60) -> tuple[bool, str]: diff --git a/skillscope/engine/no_sandbox.py b/skillscope/engine/no_sandbox.py index 8d31bae..e614ede 100644 --- a/skillscope/engine/no_sandbox.py +++ b/skillscope/engine/no_sandbox.py @@ -33,6 +33,7 @@ from typing import Callable from .. import agent as legacy_agent +from .. import harness as harnesses from .. import usage from .. import routing as routing_core @@ -64,7 +65,7 @@ EVAL_FIXTURE_DIRNAME = "evals" -def require_local() -> None: +def require_local(harness: harnesses.Harness | None = None) -> None: """This driver runs on the host, so settle the sandbox on the host. Two cases, and they are not the same question. @@ -96,8 +97,7 @@ def require_local() -> None: "Unset it to run on the host, or use --engine claude-code for a " "sandboxed run of the real harness on Linux." ) - if not shutil.which("claude"): - raise SystemExit("error: 'claude' CLI not found on PATH") + (harness or harnesses.get()).require() async def _workspace() -> str: @@ -251,9 +251,15 @@ async def _stream_until( workspace: str, env: dict, stop_when: Callable[[dict], str | None], + normalize: Callable[[dict], list[dict]] | None = None, ) -> tuple[list[dict], str | None, int | None, str]: """Run the CLI, reading its stream, and stop the moment `stop_when` says to. + `normalize` turns each event the CLI prints into the canonical events the + rest of the engine reads (see `harness.Harness.normalize`). The stop rule + sees the canonical events, never the raw ones, so one rule serves every + harness. Left `None`, events pass through as they came. + The reason this exists rather than `inspect_ai.util.subprocess`: that call returns once the process has exited, so nothing it gives back can arrive in time to end the run early. For a behavioral case that is exactly right -- @@ -323,8 +329,11 @@ async def _drain_stderr() -> None: event = json.loads(line) except json.JSONDecodeError: continue - events.append(event) - stop_reason = stop_when(event) + for canonical in normalize(event) if normalize else [event]: + events.append(canonical) + stop_reason = stop_when(canonical) + if stop_reason is not None: + break if stop_reason is not None: break finally: @@ -338,15 +347,18 @@ async def _drain_stderr() -> None: return events, stop_reason, proc.returncode, "".join(errors) -def install_skill(skill_dir: Path, workspace: str) -> None: +def install_skill( + skill_dir: Path, workspace: str, skills_dir: str | None = None +) -> None: """Put the skill where the real harness looks for it. + `/` inside the workspace, which for Claude Code is `.claude/skills/` inside a directory the CLI is given with - `--add-dir`, which is what `inspect_swe` does via its own `skills=` - argument. Staging is the driver's job, and this driver is the one that has - to do it by hand: skip it and the agent runs with no skill at all, - answering from the prompt and scoring like it -- which looks like a bad - skill rather than a missing one. + `--add-dir`, and what `inspect_swe` does via its own `skills=` argument. + Another harness names its own directory. Staging is the driver's job, and + this driver is the one that has to do it by hand: skip it and the agent + runs with no skill at all, answering from the prompt and scoring like it -- + which looks like a bad skill rather than a missing one. Everything except the skill's own test suite. This used to copy the directory wholesale, which put `evals/evals.json` inside the room the agent @@ -364,7 +376,7 @@ def install_skill(skill_dir: Path, workspace: str) -> None: are read from the skill directory on disk and seeded into the *workspace*, which is a different path. """ - dest = Path(workspace) / ".claude" / "skills" / skill_dir.name + dest = Path(workspace) / (skills_dir or harnesses.get().skills_dir) / skill_dir.name dest.parent.mkdir(parents=True, exist_ok=True) shutil.copytree( skill_dir, @@ -396,7 +408,8 @@ def _record_room(state, event: dict, room_names: list[str] | None) -> None: state.store.set(EXTRA_SKILLS_KEY, routing_core.init_extra_skills(event, room_names)) -def claude_code_no_sandbox( +def harness_solver( + harness: harnesses.Harness, model: str | None, effort: str | None, skill_dir: Path, @@ -407,6 +420,13 @@ def claude_code_no_sandbox( ): """Solver: install the skill, run the real CLI once, record what it did. + `harness` is the agent that runs the case. It decides where the skill is + staged, the command line and environment of the process, and how its output + is translated into the canonical events everything below reads; this driver + owns the rest, and owns it once for every harness. `claude_code_no_sandbox` + is this with Claude Code, which is what every caller used before there was + a choice. + `extra_flags` are the CLI's own controls -- `--max-budget-usd` and `--no-session-persistence` -- built by `routing.host_cost_flags`, which probes `--help` first: an older build rejects an unknown flag and every @@ -444,34 +464,37 @@ def claude_code_no_sandbox( from inspect_ai.solver import solver @solver - def _claude_code_no_sandbox(): + def _harness_solver(): async def solve(state, generate): from inspect_ai.util import subprocess as sandbox_subprocess workspace = await _workspace() - install_skill(skill_dir, workspace) + install_skill(skill_dir, workspace, harness.skills_dir) prompt = state.input_text - cmd = [ - shutil.which("claude"), "-p", - "--output-format", "stream-json", "--verbose", - "--dangerously-skip-permissions", - "--add-dir", workspace, - ] - if model: - cmd += ["--model", model] - if effort: - cmd += ["--effort", effort] - cmd += list(extra_flags or []) - - env = legacy_agent.claude_env() - if config_dir is not None: - env["CLAUDE_CONFIG_DIR"] = str(config_dir) + spec = harnesses.RunSpec( + workspace=workspace, + prompt=prompt, + model=model, + effort=effort, + config_dir=str(config_dir) if config_dir is not None else None, + extra_flags=tuple(extra_flags or ()), + ) + # Started on the host, so the executable is looked up here and the + # runner's environment goes under what the harness sets. + cmd = harness.host_command(spec) + env = harness.host_env(spec) + # Closed either way: a CLI that is not given its prompt on stdin + # can still sit waiting on it. + stdin_text = prompt if harness.prompt_on_stdin else "" + # One per case, not one per run: it may remember what it has read, + # and the next case must not inherit that. + normalize = harness.normalizer() events: list[dict] = [] if stop_when_factory is None: result = await sandbox_subprocess( - cmd, input=prompt, cwd=workspace, env=env, + cmd, input=stdin_text, cwd=workspace, env=env, ) returncode, stderr = result.returncode, result.stderr or "" for line in (result.stdout or "").splitlines(): @@ -479,13 +502,14 @@ async def solve(state, generate): if not line: continue try: - events.append(json.loads(line)) + events.extend(normalize(json.loads(line))) except json.JSONDecodeError: continue else: async with _subprocess_concurrency(): events, stop_reason, returncode, stderr = await _stream_until( - cmd, prompt, workspace, env, stop_when_factory() + cmd, stdin_text, workspace, env, stop_when_factory(), + normalize, ) # Recorded whatever it was, including `None` for a stream that # ended on its own. A stop this driver performed leaves no @@ -495,7 +519,7 @@ async def solve(state, generate): if not events: raise RuntimeError( - "claude produced no parseable stream-json output " + f"{harness.binary} produced no parseable stream-json output " f"(exit {returncode}). stderr: {stderr[:300]}" ) @@ -506,10 +530,33 @@ async def solve(state, generate): messages, final = events_to_messages(events, prompt) state.messages.extend(messages) state.output = ModelOutput.from_content( - model=model or "claude", content=final or "(no final message)" + model=model or harness.binary, + content=final or "(no final message)", ) return state return solve - return _claude_code_no_sandbox() + return _harness_solver() + + +def claude_code_no_sandbox( + model: str | None, + effort: str | None, + skill_dir: Path, + config_dir: Path | None = None, + extra_flags: list[str] | None = None, + stop_when_factory: Callable[[], Callable[[dict], str | None]] | None = None, + room_names: list[str] | None = None, +): + """`harness_solver` with Claude Code, the harness every caller has used.""" + return harness_solver( + harnesses.get("claude-code"), + model, + effort, + skill_dir, + config_dir, + extra_flags, + stop_when_factory, + room_names, + ) diff --git a/skillscope/harness/__init__.py b/skillscope/harness/__init__.py new file mode 100644 index 0000000..c641d5d --- /dev/null +++ b/skillscope/harness/__init__.py @@ -0,0 +1,46 @@ +# Copyright Advanced Micro Devices, Inc. +# +# SPDX-License-Identifier: MIT + +"""The agent harnesses skillscope can drive. + +A harness is *what* runs a case; the engine is *where*. See `base.Harness` for +the contract and `docs/harnesses.md` for how to add one. +""" + +from __future__ import annotations + +from .base import Harness, RunSpec +from .claude_code import ClaudeCode + +DEFAULT = "claude-code" + +_REGISTRY: dict[str, Harness] = {} + + +def register(harness: Harness) -> Harness: + """Make `harness` selectable by its name. Registering a name twice is a bug.""" + if harness.name in _REGISTRY: + raise ValueError(f"harness {harness.name!r} is already registered") + _REGISTRY[harness.name] = harness + return harness + + +def names() -> list[str]: + return sorted(_REGISTRY) + + +def get(name: str | None = None) -> Harness: + """The harness called `name`, or the default one when none is named.""" + key = name or DEFAULT + try: + return _REGISTRY[key] + except KeyError: + raise SystemExit( + f"error: unknown harness {key!r}. Known harnesses: {', '.join(names())}." + ) from None + + +register(ClaudeCode()) + +__all__ = ["DEFAULT", "Harness", "RunSpec", "get", "names", "register"] diff --git a/skillscope/harness/base.py b/skillscope/harness/base.py new file mode 100644 index 0000000..b01d6da --- /dev/null +++ b/skillscope/harness/base.py @@ -0,0 +1,144 @@ +# Copyright Advanced Micro Devices, Inc. +# +# SPDX-License-Identifier: MIT + +"""What a harness is, as far as the engines are concerned. + +A *harness* is the agent that is handed a prompt and a workspace and does the +work: Claude Code, GitHub Copilot CLI, Codex, and so on. It is a different +question from where it runs (`--engine`: on the host or in a container), and +this module is the contract for answering it. + +The engines drive a harness as a subprocess, so the contract is small: how to +find it, where it looks for skills, how to start one case, what environment to +start it in, and how to read what it prints. Everything else -- staging, the +stop rule, usage, grading -- stays in the engine, once, for every harness. + +Adding a harness is one subclass of `Harness`, registered in +`skillscope/harness/__init__.py`, plus recorded output for the contract test in +`tests/test_harness.py`. `docs/harnesses.md` has the checklist. +""" + +from __future__ import annotations + +import os +import shutil +from abc import ABC, abstractmethod +from dataclasses import dataclass +from typing import Callable + + +@dataclass(frozen=True) +class RunSpec: + """Everything one case hands to a harness. + + A value object rather than keyword arguments so a harness that needs one + more input in future (a trust flag, a home directory) reads it from here + instead of every caller growing a parameter. + """ + + workspace: str + prompt: str + model: str | None = None + effort: str | None = None + # A throwaway config directory, so the runner's own settings and user-level + # skills stay out of the session. `None` when the caller does not isolate. + # A path wherever the agent runs, like `workspace`: on the host a host path, + # in a container a path inside it. + config_dir: str | None = None + # The harness's own spend or session controls, already built by the caller. + extra_flags: tuple[str, ...] = () + + +class Harness(ABC): + """One agent harness. Subclass, fill in the class attributes, register.""" + + # The name `--harness` and the reports use. + name: str + # The executable looked up on PATH. + binary: str + # Where, relative to the workspace, the harness discovers skills. + skills_dir: str + # Whether activating a skill is a tool call the stream names. A harness with + # no such call (it activates a skill by reading its SKILL.md) sets this + # `False`, and activation is read from the file access instead. + has_skill_tool: bool = True + # Whether the prompt is written to the process's stdin. A harness that takes + # it as an argument builds that into `command` and sets this `False`; its + # stdin is closed either way, because some CLIs wait on it. + prompt_on_stdin: bool = True + + def require(self) -> None: + """Refuse to start on the host when the harness is not installed there. + + This asks the runner's `PATH`, which is the right question for a run + started on the host. A leg that starts the agent in a container asks + about the image instead. + """ + if not shutil.which(self.binary): + raise SystemExit(f"error: '{self.binary}' CLI not found on PATH") + + @abstractmethod + def command(self, spec: RunSpec) -> list[str]: + """The argv that runs one case, headless, streaming JSON events. + + The executable is named, not located: `self.binary`, never an absolute + path on this machine. Whoever starts the process resolves it, so the same + command can be started on the host or inside a container. + """ + + @abstractmethod + def env(self, spec: RunSpec) -> dict[str, str]: + """What the harness sets in the process's environment, and nothing else. + + Its own settings, and whatever points it at `spec.config_dir` so the + runner's own settings and skills stay out of the session. Not the runner's + environment: `PATH`, `HOME` and every key on the machine are not the + harness's to carry, and a container must not inherit them. Whoever starts + the process puts this on top of the environment it has. + """ + + def host_command(self, spec: RunSpec) -> list[str]: + """`command(spec)` with its executable resolved on this machine. + + What a run started on the host executes. The lookup is `shutil.which` + rather than leaving it to the operating system, because on Windows a CLI + installed with npm is a `.cmd` shim that only the former finds. + """ + argv = self.command(spec) + return [shutil.which(argv[0]) or argv[0], *argv[1:]] + + def host_env(self, spec: RunSpec) -> dict[str, str]: + """The environment a run started on the host gets: this machine's, with + what the harness sets on top.""" + return {**os.environ, **self.env(spec)} + + def normalize(self, event: dict) -> list[dict]: + """Translate one event the harness printed into canonical events. + + The canonical form is the one the engines already read: Claude Code's + stream-json, reduced to `assistant` messages carrying `tool_use` + blocks, `user` messages carrying `tool_result` blocks, the + `system`/`init` event naming the skills the session registered, and a + final `result` event. A harness whose stream already has that shape + keeps this default; any other maps its events onto it, and returns an + empty list for the ones that carry nothing the engines read. + + Stateless, which is enough when each event can be translated on its + own. A harness that cannot -- one whose closing event does not repeat + the answer, so it has to be remembered from an earlier one -- overrides + `normalizer` instead. + """ + return [event] + + def normalizer(self) -> Callable[[dict], list[dict]]: + """A translator for the events of one run, free to keep state. + + The engine asks for a fresh one per case, so nothing a harness + remembers can leak from one case into the next. A harness instance is + shared by every case in a run, and cases run concurrently; the state + therefore lives in the returned callable, never on `self`. + + The default is the stateless `normalize`. + """ + return self.normalize diff --git a/skillscope/harness/claude_code.py b/skillscope/harness/claude_code.py new file mode 100644 index 0000000..9cd89c0 --- /dev/null +++ b/skillscope/harness/claude_code.py @@ -0,0 +1,42 @@ +# Copyright Advanced Micro Devices, Inc. +# +# SPDX-License-Identifier: MIT + +"""Claude Code as a harness. + +Not new behaviour: this is the command line and environment the host engine +has always used for the `claude` CLI, moved behind the `Harness` contract so +the engine no longer assumes it. Claude Code's stream-json is the canonical +event form, so `normalize` is the default identity. +""" + +from __future__ import annotations + +from .. import agent +from .base import Harness, RunSpec + + +class ClaudeCode(Harness): + name = "claude-code" + binary = "claude" + skills_dir = ".claude/skills" + + def command(self, spec: RunSpec) -> list[str]: + cmd = [ + self.binary, "-p", + "--output-format", "stream-json", "--verbose", + "--dangerously-skip-permissions", + "--add-dir", spec.workspace, + ] + if spec.model: + cmd += ["--model", spec.model] + if spec.effort: + cmd += ["--effort", spec.effort] + cmd += list(spec.extra_flags) + return cmd + + def env(self, spec: RunSpec) -> dict[str, str]: + env = agent.claude_settings() + if spec.config_dir is not None: + env["CLAUDE_CONFIG_DIR"] = spec.config_dir + return env diff --git a/tests/fixtures/harness/claude-code/expected.json b/tests/fixtures/harness/claude-code/expected.json new file mode 100644 index 0000000..44b0a3a --- /dev/null +++ b/tests/fixtures/harness/claude-code/expected.json @@ -0,0 +1,6 @@ +{ + "note": "Synthetic: hand-written to the stream-json shape, not a recording.", + "skills": ["alpha"], + "trigger": {"activation": "alpha", "completed": true, "final": "done"}, + "no_trigger": {"activation": null, "completed": true, "final": "That is not something a skill is needed for."} +} diff --git a/tests/fixtures/harness/claude-code/no_trigger.jsonl b/tests/fixtures/harness/claude-code/no_trigger.jsonl new file mode 100644 index 0000000..2f055f2 --- /dev/null +++ b/tests/fixtures/harness/claude-code/no_trigger.jsonl @@ -0,0 +1,3 @@ +{"type":"system","subtype":"init","skills":["alpha"]} +{"type":"assistant","message":{"content":[{"type":"text","text":"That is not something a skill is needed for."}]}} +{"type":"result","subtype":"success","result":"That is not something a skill is needed for."} diff --git a/tests/fixtures/harness/claude-code/trigger.jsonl b/tests/fixtures/harness/claude-code/trigger.jsonl new file mode 100644 index 0000000..6770bd2 --- /dev/null +++ b/tests/fixtures/harness/claude-code/trigger.jsonl @@ -0,0 +1,4 @@ +{"type":"system","subtype":"init","skills":["alpha","a-stray-skill"]} +{"type":"assistant","message":{"usage":{"input_tokens":11,"output_tokens":7},"content":[{"type":"tool_use","id":"t1","name":"Skill","input":{"skill":"alpha"}}]}} +{"type":"user","message":{"content":[{"type":"tool_result","tool_use_id":"t1","content":"Launching skill: alpha"}]}} +{"type":"result","subtype":"success","result":"done"} diff --git a/tests/test_harness.py b/tests/test_harness.py new file mode 100644 index 0000000..40c8043 --- /dev/null +++ b/tests/test_harness.py @@ -0,0 +1,603 @@ +# Copyright Advanced Micro Devices, Inc. +# +# SPDX-License-Identifier: MIT + +"""The harness contract: what every agent harness has to provide. + +Three kinds of test, in the order a new harness meets them: + +* the registry, which is how `--harness` will find one; +* the contract, run once per registered harness against the recorded output in + `tests/fixtures/harness//`, so a new harness is held to the same bar as + the first without anyone writing its tests from scratch; +* the seam, which drives the engine with a harness whose events look nothing + like Claude Code's, because a contract only the one real harness has ever + satisfied has not been shown to be a contract. +""" + +from __future__ import annotations + +import json +import os +import shutil +import sys +import tempfile +import unittest +from pathlib import Path +from unittest import mock + +from skillscope import harness, routing +from skillscope.engine import no_sandbox + +FIXTURES = Path(__file__).parent / "fixtures" / "harness" + + +class TestTheRegistry(unittest.TestCase): + def test_claude_code_is_the_default(self) -> None: + self.assertEqual(harness.DEFAULT, "claude-code") + self.assertEqual(harness.get().name, "claude-code") + self.assertIs(harness.get(), harness.get("claude-code")) + + def test_names_lists_what_is_registered(self) -> None: + self.assertIn("claude-code", harness.names()) + + def test_an_unknown_harness_is_refused_with_the_known_ones(self) -> None: + with self.assertRaises(SystemExit) as raised: + harness.get("no-such-agent") + message = str(raised.exception) + self.assertIn("no-such-agent", message) + self.assertIn("claude-code", message) + + def test_a_name_cannot_be_registered_twice(self) -> None: + with self.assertRaises(ValueError): + harness.register(harness.ClaudeCode()) + + +class TestClaudeCodeIsWhatTheEngineAlwaysRan(unittest.TestCase): + """The adapter replaced inline code, so pin the output it replaced. + + The command line and environment are the whole of what changed hands, and + nothing else in the suite compares them to the earlier values: the + end-to-end test runs a stub that ignores its flags. + """ + + def spec(self, **overrides) -> harness.RunSpec: + values = {"workspace": "/ws", "prompt": "do it", **overrides} + return harness.RunSpec(**values) + + def command(self, **overrides) -> list[str]: + return harness.get("claude-code").command(self.spec(**overrides)) + + def test_the_command_is_the_one_the_engine_built_inline(self) -> None: + # Naming the executable and not locating it is the one difference from + # what the engine built inline: `host_command` puts the located path back. + self.assertEqual( + self.command(model="opus", effort="high"), + [ + "claude", "-p", + "--output-format", "stream-json", "--verbose", + "--dangerously-skip-permissions", + "--add-dir", "/ws", + "--model", "opus", + "--effort", "high", + ], + ) + + def test_model_and_effort_are_left_out_when_not_given(self) -> None: + command = self.command() + self.assertNotIn("--model", command) + self.assertNotIn("--effort", command) + + def test_the_callers_flags_come_last_in_order(self) -> None: + command = self.command(extra_flags=("--max-budget-usd", "1")) + self.assertEqual(command[-2:], ["--max-budget-usd", "1"]) + + def test_run_on_the_host_it_is_the_executable_found_here(self) -> None: + claude = harness.get("claude-code") + with mock.patch("shutil.which", lambda _: "/usr/bin/claude"): + started = claude.host_command(self.spec(model="opus", effort="high")) + self.assertEqual(started[0], "/usr/bin/claude") + self.assertEqual(started[1:], self.command(model="opus", effort="high")[1:]) + + def test_an_executable_that_is_not_found_is_left_for_the_start_to_refuse(self) -> None: + with mock.patch("shutil.which", lambda _: None): + started = harness.get("claude-code").host_command(self.spec()) + self.assertEqual(started[0], "claude") + + def test_a_config_dir_is_only_exported_when_one_was_given(self) -> None: + claude = harness.get("claude-code") + self.assertEqual( + claude.env(self.spec(config_dir="/cfg"))["CLAUDE_CONFIG_DIR"], "/cfg" + ) + self.assertNotIn("CLAUDE_CONFIG_DIR", claude.env(self.spec())) + + def test_the_environment_is_only_what_the_harness_sets(self) -> None: + # Not the runner's: a container must not inherit its PATH, its HOME or + # every key on the machine. + claude = harness.get("claude-code") + ambient = {"PATH": "/runner/bin", "HOME": "/home/runner", "ANTHROPIC_API_KEY": "sk-x"} + with mock.patch.dict(os.environ, ambient): + env = claude.env(self.spec(config_dir="/cfg")) + self.assertEqual(set(env), {"CLAUDE_CODE_MAX_RETRIES", "CLAUDE_CONFIG_DIR"}) + + def test_on_the_host_the_environment_is_what_the_engine_always_built(self) -> None: + # The old inline code: the runner's environment, a retry default that the + # caller can override, and the isolation setting on top of everything. + claude = harness.get("claude-code") + for retries in (None, "3"): + with self.subTest(retries=retries), mock.patch.dict( + os.environ, {"CLAUDE_CONFIG_DIR": "/runners/own"}, clear=False + ): + os.environ.pop("CLAUDE_CODE_MAX_RETRIES", None) + if retries: + os.environ["CLAUDE_CODE_MAX_RETRIES"] = retries + expected = dict(os.environ) + expected.setdefault("CLAUDE_CODE_MAX_RETRIES", "0") + expected["CLAUDE_CONFIG_DIR"] = "/cfg" + self.assertEqual(claude.host_env(self.spec(config_dir="/cfg")), expected) + + def test_the_cli_is_told_not_to_retry_unless_the_caller_says_otherwise(self) -> None: + claude = harness.get("claude-code") + with mock.patch.dict(os.environ, clear=False): + os.environ.pop("CLAUDE_CODE_MAX_RETRIES", None) + self.assertEqual(claude.env(self.spec())["CLAUDE_CODE_MAX_RETRIES"], "0") + os.environ["CLAUDE_CODE_MAX_RETRIES"] = "3" + self.assertEqual(claude.env(self.spec())["CLAUDE_CODE_MAX_RETRIES"], "3") + + def test_a_missing_binary_is_refused_by_name(self) -> None: + with mock.patch("shutil.which", lambda _: None), self.assertRaises(SystemExit) as raised: + harness.get("claude-code").require() + self.assertEqual(str(raised.exception), "error: 'claude' CLI not found on PATH") + + +class TestEveryRegisteredHarnessMeetsTheContract(unittest.TestCase): + """Run for each harness that ships fixtures; a harness without them fails. + + A fixture directory holds `trigger.jsonl` (the harness activated the skill + and finished), `no_trigger.jsonl` (it finished without activating one) and + `expected.json` with what the engine should read out of each. Real + recordings, token-scrubbed, with the CLI version noted in `expected.json`; + a harness whose recordings change shape gets new fixtures, not a looser + test. + """ + + def fixtures(self, name: str) -> tuple[dict, dict[str, list[str]]]: + folder = FIXTURES / name + self.assertTrue( + folder.is_dir(), + f"harness {name!r} is registered but has no fixtures in {folder}", + ) + expected = json.loads((folder / "expected.json").read_text(encoding="utf-8")) + runs = { + kind: (folder / f"{kind}.jsonl").read_text(encoding="utf-8").splitlines() + for kind in ("trigger", "no_trigger") + } + return expected, runs + + def canonical(self, subject: harness.Harness, lines: list[str]) -> list[dict]: + events: list[dict] = [] + normalize = subject.normalizer() + for line in lines: + if line.strip(): + events.extend(normalize(json.loads(line))) + return events + + def test_what_the_engine_reads_out_of_a_recorded_run(self) -> None: + for name in harness.names(): + with self.subTest(harness=name): + subject = harness.get(name) + expected, runs = self.fixtures(name) + for kind, lines in runs.items(): + events = self.canonical(subject, lines) + want = expected[kind] + + activation = next( + ( + hit + for event in events + if (hit := routing.detect_activation( + event, expected["skills"], + allow_body_path=not subject.has_skill_tool, + )) + ), + None, + ) + self.assertEqual(activation, want["activation"], kind) + + completed, final = no_sandbox.run_completed(events) + self.assertEqual(completed, want["completed"], kind) + self.assertEqual(final, want["final"], kind) + + def test_the_room_is_read_off_the_init_event_when_the_harness_has_one(self) -> None: + for name in harness.names(): + with self.subTest(harness=name): + subject = harness.get(name) + expected, runs = self.fixtures(name) + events = self.canonical(subject, runs["trigger"]) + seen = [ + visible + for event in events + if (visible := routing.init_skills(event, expected["skills"])) is not None + ] + # A harness with no init event reports nothing, which the engine + # words as "the check did not run". It must never report a room + # it did not read. + for visible in seen: + self.assertEqual(visible, expected["skills"]) + + def test_two_runs_translated_at_once_do_not_share_memory(self) -> None: + # The engine runs cases concurrently against one harness instance, so + # whatever a harness remembers about a stream must live in the + # translator it hands out per run, not on the harness. + import itertools + + for name in harness.names(): + with self.subTest(harness=name): + subject = harness.get(name) + _, runs = self.fixtures(name) + alone = {kind: self.canonical(subject, lines) for kind, lines in runs.items()} + + first, second = subject.normalizer(), subject.normalizer() + mixed: dict[str, list[dict]] = {"trigger": [], "no_trigger": []} + for a, b in itertools.zip_longest(runs["trigger"], runs["no_trigger"]): + if a and a.strip(): + mixed["trigger"].extend(first(json.loads(a))) + if b and b.strip(): + mixed["no_trigger"].extend(second(json.loads(b))) + self.assertEqual(mixed, alone) + + def test_the_command_carries_the_model_the_effort_and_no_secret(self) -> None: + for name in harness.names(): + with self.subTest(harness=name): + subject = harness.get(name) + spec = harness.RunSpec( + workspace=str(Path(tempfile.gettempdir()) / "ws"), + prompt="the prompt", + model="a-model", + effort="high", + ) + with mock.patch.dict(os.environ, {"SOME_API_KEY": "sk-secret-value"}): + command = subject.command(spec) + self.assertIn("a-model", command) + self.assertIn("high", command) + self.assertFalse( + any("sk-secret-value" in part for part in command), + "a credential on the command line shows up in `ps` and in logs", + ) + + def test_the_config_is_isolated_from_the_runners_own(self) -> None: + for name in harness.names(): + with self.subTest(harness=name): + subject = harness.get(name) + spec = harness.RunSpec( + workspace="ws", prompt="p", config_dir="throwaway-config", + ) + env = subject.env(spec) + self.assertIn( + "throwaway-config", env.values(), + "no environment variable points the harness at the throwaway " + "config dir, so the runner's own skills would join the room", + ) + + def test_the_command_names_its_executable_and_does_not_locate_it(self) -> None: + # So the same command can be started on the host or in a container. An + # absolute path on the runner is wrong anywhere else. + for name in harness.names(): + with self.subTest(harness=name): + subject = harness.get(name) + command = subject.command(harness.RunSpec(workspace="ws", prompt="p")) + self.assertEqual(command[0], subject.binary) + + def test_the_environment_carries_only_what_the_harness_sets(self) -> None: + # Not the runner's PATH, its HOME or any key on it: a container would + # inherit all of it. + for name in harness.names(): + with self.subTest(harness=name): + subject = harness.get(name) + ambient = { + "PATH": "/runner/bin", "HOME": "/home/runner", + "SOME_API_KEY": "sk-secret-value", "SYSTEMROOT": "C:/Windows", + } + with mock.patch.dict(os.environ, ambient): + env = subject.env(harness.RunSpec( + workspace="ws", prompt="p", config_dir="throwaway-config", + )) + for variable in ambient: + self.assertNotIn(variable, env) + self.assertTrue(all(isinstance(v, str) for v in env.values())) + + def test_on_the_host_what_the_harness_sets_goes_on_top_of_the_runners(self) -> None: + for name in harness.names(): + with self.subTest(harness=name): + subject = harness.get(name) + spec = harness.RunSpec(workspace="ws", prompt="p", config_dir="cfg") + with mock.patch.dict(os.environ, {"AMBIENT_FOR_THE_TEST": "kept"}): + host = subject.host_env(spec) + self.assertEqual(host["AMBIENT_FOR_THE_TEST"], "kept") + for key, value in subject.env(spec).items(): + self.assertEqual(host[key], value) + + def test_a_skill_is_staged_where_the_harness_looks(self) -> None: + for name in harness.names(): + with self.subTest(harness=name): + subject = harness.get(name) + with tempfile.TemporaryDirectory() as tmp: + skill = Path(tmp) / "alpha" + (skill / "evals").mkdir(parents=True) + (skill / "SKILL.md").write_text("# alpha", encoding="utf-8") + (skill / "evals" / "evals.json").write_text("{}", encoding="utf-8") + workspace = Path(tmp) / "ws" + workspace.mkdir() + + no_sandbox.install_skill(skill, str(workspace), subject.skills_dir) + + staged = workspace / subject.skills_dir / "alpha" + self.assertTrue((staged / "SKILL.md").is_file()) + # The answer key stays out of the room for every harness. + self.assertFalse((staged / "evals").exists()) + + +# What the foreign harness below prints. Deliberately nothing like stream-json: +# no `type` keys, different names for everything. +_FOREIGN_AGENT = r""" +import json, os, sys + +def emit(**event): + print(json.dumps(event), flush=True) + +staged = os.path.isfile(os.path.join(".foreign", "skills", "alpha", "SKILL.md")) +# What the harness was handed: its prompt as an argument, and a stdin that was +# closed without ever being written to. +stdin_seen = sys.stdin.read() +emit(evt="started") +if "silent" not in sys.argv[1]: + emit(evt="said", text="all done") +emit(evt="skill", name="alpha") +emit( + evt="command", + command="echo foreign-marker staged=%s stdin=%r arg=%r" % (staged, stdin_seen, sys.argv[1]), +) +emit(evt="finished") +""" + + +class ForeignHarness(harness.Harness): + """A harness whose events share nothing with Claude Code's. + + Not registered: it exists to show the engine only needs what `Harness` + promises. It runs a script, takes its prompt as an argument, and keeps its + skills somewhere other than `.claude/skills`. + """ + + name = "foreign" + binary = Path(sys.executable).name + skills_dir = ".foreign/skills" + prompt_on_stdin = False + + def __init__(self, script: Path) -> None: + self.script = script + self.specs: list[harness.RunSpec] = [] + + def require(self) -> None: + return None + + def command(self, spec: harness.RunSpec) -> list[str]: + self.specs.append(spec) + return [sys.executable, "-u", str(self.script), spec.prompt] + + def env(self, spec: harness.RunSpec) -> dict[str, str]: + return {} + + def normalizer(self): + # Its closing event does not repeat the answer, so it is remembered from + # the one that said it -- which is why the translator is per run. + said = {"text": ""} + + def normalize(event: dict) -> list[dict]: + kind = event.get("evt") + if kind == "said": + said["text"] = event["text"] + return [] + if kind == "skill": + return [_tool_use("Skill", {"skill": event["name"]})] + if kind == "command": + return [_tool_use("Bash", {"command": event["command"]})] + if kind == "finished": + return [{"type": "result", "subtype": "success", "result": said["text"]}] + return [] + + return normalize + + +def _tool_use(name: str, arguments: dict) -> dict: + return { + "type": "assistant", + "message": {"content": [{"type": "tool_use", "name": name, "input": arguments}]}, + } + + +class TestTheSeamTranslatesAnotherHarnessesEvents(unittest.IsolatedAsyncioTestCase): + def setUp(self) -> None: + self.tmp = tempfile.mkdtemp(prefix="ss-foreign-") + self.addCleanup(shutil.rmtree, self.tmp, True) + self.script = Path(self.tmp) / "foreign_agent.py" + self.script.write_text(_FOREIGN_AGENT, encoding="utf-8") + self.foreign = ForeignHarness(self.script) + + async def stream(self, stop_when) -> tuple[list[dict], str | None]: + spec = harness.RunSpec(workspace=self.tmp, prompt="go") + events, reason, _, _ = await no_sandbox._stream_until( + self.foreign.host_command(spec), "", self.tmp, self.foreign.host_env(spec), + stop_when, self.foreign.normalizer(), + ) + return events, reason + + async def test_the_engine_reads_canonical_events_only(self) -> None: + events, reason = await self.stream(lambda event: None) + self.assertIsNone(reason) + # `started` carries nothing the engine reads, so it is dropped rather + # than passed through for something downstream to trip on. + calls = [call for event in events for call in routing._iter_tool_uses(event)] + self.assertEqual([name for name, _ in calls], ["Skill", "Bash"]) + self.assertEqual(calls[0][1], '{"skill": "alpha"}') + self.assertIn("foreign-marker", calls[1][1]) + completed, final = no_sandbox.run_completed(events) + self.assertEqual((completed, final), (True, "all done")) + + def test_each_translator_has_its_own_memory(self) -> None: + first, second = self.foreign.normalizer(), self.foreign.normalizer() + first({"evt": "said", "text": "first run"}) + [unspoken] = second({"evt": "finished"}) + self.assertEqual(unspoken["result"], "") + [spoken] = first({"evt": "finished"}) + self.assertEqual(spoken["result"], "first run") + + async def test_the_stop_rule_sees_canonical_events(self) -> None: + seen: list[dict] = [] + + def stop_when(event: dict) -> str | None: + seen.append(event) + return "activated" if routing.detect_activation(event, ["alpha"]) else None + + events, reason = await self.stream(stop_when) + self.assertEqual(reason, "activated") + self.assertEqual(len(events), 1) + self.assertTrue(all("evt" not in event for event in seen)) + + async def test_an_unreadable_line_is_skipped_not_fatal(self) -> None: + self.script.write_text( + "import json\nprint('not json', flush=True)\n" + "print(json.dumps({'evt': 'said', 'text': 'ok'}), flush=True)\n" + "print(json.dumps({'evt': 'finished'}), flush=True)\n", + encoding="utf-8", + ) + events, _ = await self.stream(lambda event: None) + self.assertEqual(no_sandbox.run_completed(events), (True, "ok")) + + +class TestTheEngineRunsACaseWithAnotherHarness(unittest.TestCase): + """The whole solver, under inspect, with a harness that is not Claude Code.""" + + def setUp(self) -> None: + try: + import inspect_ai # noqa: F401 + except ImportError: + self.skipTest("needs inspect_ai") + from skillscope.engine import sandbox as engine_sandbox + + env = mock.patch.dict(os.environ, {engine_sandbox.SANDBOX_ENV: "local"}) + env.start() + self.addCleanup(env.stop) + self.tmp = tempfile.mkdtemp(prefix="ss-foreign-e2e-") + self.addCleanup(shutil.rmtree, self.tmp, True) + self.script = Path(self.tmp) / "foreign_agent.py" + self.script.write_text(_FOREIGN_AGENT, encoding="utf-8") + self.foreign = ForeignHarness(self.script) + self.skill = Path(self.tmp) / "alpha" + self.skill.mkdir() + (self.skill / "SKILL.md").write_text( + "---\nname: alpha\ndescription: does alpha things\n---\n", encoding="utf-8" + ) + + def run_case(self, *, stop_when_factory=None, config_dir=None): + import contextlib + import io + + from inspect_ai import Task + from inspect_ai import eval as inspect_eval + from inspect_ai.dataset import Sample + + task = Task( + dataset=[Sample(id="c1", input="do the alpha thing")], + solver=no_sandbox.harness_solver( + self.foreign, "mockllm/model", None, self.skill, + config_dir=config_dir, stop_when_factory=stop_when_factory, + ), + sandbox="local", + ) + with contextlib.redirect_stdout(io.StringIO()): + [log] = inspect_eval( + task, model="mockllm/model", log_dir=str(Path(self.tmp) / "logs"), + display="plain", + ) + [sample] = log.samples + self.assertIsNone(sample.error, sample.error) + return sample + + def calls(self, sample) -> list[tuple[str, dict]]: + return [ + (call.function, call.arguments) + for message in sample.messages + for call in getattr(message, "tool_calls", None) or [] + ] + + def test_the_foreign_events_become_tool_calls_the_scorers_read(self) -> None: + sample = self.run_case() + functions = [function for function, _ in self.calls(sample)] + self.assertEqual(functions, ["Skill", "Bash"]) + self.assertIn("all done", sample.output.completion) + + def test_one_cases_answer_does_not_leak_into_the_next(self) -> None: + import contextlib + import io + + from inspect_ai import Task + from inspect_ai import eval as inspect_eval + from inspect_ai.dataset import Sample + + task = Task( + dataset=[ + Sample(id="talks", input="do the alpha thing"), + Sample(id="silent", input="be silent about the alpha thing"), + ], + solver=no_sandbox.harness_solver( + ForeignHarness(self.script), "mockllm/model", None, self.skill, + ), + sandbox="local", + ) + with contextlib.redirect_stdout(io.StringIO()): + [log] = inspect_eval( + task, model="mockllm/model", log_dir=str(Path(self.tmp) / "logs"), + display="plain", + # One at a time, so the second case really does follow the first. + max_samples=1, + ) + answers = {sample.id: sample.output.completion for sample in log.samples} + self.assertEqual(answers["talks"], "all done") + self.assertEqual(answers["silent"], "(no final message)") + + def test_the_harness_is_told_where_the_agent_runs_as_strings(self) -> None: + # A path wherever the agent runs, so a container leg can name one inside + # it: the engine hands over strings, not host Path objects. + self.run_case(config_dir=Path(self.tmp) / "cfg") + [spec] = self.foreign.specs + self.assertIsInstance(spec.config_dir, str) + self.assertEqual(spec.config_dir, str(Path(self.tmp) / "cfg")) + self.assertIsInstance(spec.workspace, str) + + def test_the_skill_is_staged_where_that_harness_looks(self) -> None: + # The agent script checks for the file itself, in the workspace it was + # started in, and reports what it found. + sample = self.run_case() + [(_, command)] = [call for call in self.calls(sample) if call[0] == "Bash"] + self.assertIn("staged=True", command["command"]) + + def test_a_prompt_taken_as_an_argument_is_not_also_written_to_stdin(self) -> None: + sample = self.run_case() + [(_, command)] = [call for call in self.calls(sample) if call[0] == "Bash"] + self.assertIn("arg='do the alpha thing'", command["command"]) + # Closed and empty, not left open: a CLI that waits on stdin would hang. + self.assertIn("stdin=''", command["command"]) + + def test_the_stop_rule_ends_the_run_on_a_foreign_activation(self) -> None: + def factory(): + return lambda event: ( + "activated" if routing.detect_activation(event, ["alpha"]) else None + ) + + sample = self.run_case(stop_when_factory=factory) + self.assertEqual([function for function, _ in self.calls(sample)], ["Skill"]) + + +if __name__ == "__main__": + unittest.main()