Repository navigation
Add a harness-neutral agent contract and adopt it for Copilot CLI - #1008
Merged
Sun Haoran (haoranpb) merged 7 commits intoNov 3, 2026
Merged
Sun Haoran (haoranpb) merged 7 commits into
Sun Haoran (haoranpb) merged 7 commits into
Conversation
Harness[T] reports an AgentIdentity (harness, version, configured model, provenance) before running and turns an AgentRequest[T] into an AgentResult (response, transcript, metrics, experiment configuration, observed model). It is generic over the task type so specialised harnesses (PR Review, BCal, hosted agents) take their own typed tasks, and AgentResult is a dataclass so harness-specific metrics subclasses pass through unchanged. The transcript uses the user/assistant turn shape LMChecklist already scores. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The Copilot, Claude, PR Review and BCal harness functions now return the core AgentResult instead of a (metrics, experiment) tuple, and pipelines read it through AgentRunner. Copilot and Claude also return their final message as the response, which was previously discarded. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
The new result type lacks regression coverage confirming that AgentResult preserves a supplied AgentMetrics subclass instance and its extra fields.
1 open issue
What changed in this PR
Adds a shared, harness-neutral agent contract to bcbench-core, allowing evaluation pipelines to work with different agent implementations through one typed interface.
Changes:
- Defines agent identity, typed requests, results, transcript turns, and the generic
Harness[T]protocol. - Makes environment input explicit and preserves metrics subclasses in a frozen result dataclass.
- Documents how applications construct and use harnesses.
| File | Description |
|---|---|
packages/bcbench-core/src/bcbench_core/agent/harness.py |
Introduces the shared agent models and harness protocol. |
packages/bcbench-core/README.md |
Documents the contract and its role in evaluation pipelines. |
Review effort: Balanced
Contributor
Code coverage overviewLanguages: Python Python / code-coverage/pytest
View coverage of the 10 most impacted files.
ℹ️ Calculations represent line coverage. Results may change as calculations are updated. Updated |
![github-code-quality[bot]](https://avatars.githubusercontent.com/u/9919?s=60&v=4){kind=small}
…anscript type CopilotHarness replaces run_copilot_agent: model, category, agent config, plugin root and runtime are construction-time settings, while the entry, workspace, output directory, environment and timeout arrive per run as an AgentRequest. identity() resolves the Copilot CLI version, replacing the CLI's direct get_copilot_version call. harness_runner adapts any Harness to the pipeline's AgentRunner, so other harnesses can follow the same path. The harness now scrubs BC credentials from the request's environment instead of reading the process environment. The transcript type is deferred until a harness produces one. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Sun Haoran (haoranpb)
changed the title
…contract EvaluationContext now holds the resolved AgentIdentity instead of separate agent_name, model and agent_version fields; results read harness, model and version from it. The environment is a harness concern, not per-run input, so AgentRequest no longer carries it and the Copilot harness reads it as before. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…its timeout a field AgentIdentity and Harness take an optional harness-name type, so BC-Bench uses AgentIdentity[AgentHarness] and reads the enum (and its metrics contract) without converting from str. CopilotHarness explicitly inherits Harness so ty checks it at the class definition, and its default timeout is a dataclass field instead of a module constant. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Drop AgentIdentity.provenance and AgentResult.observed_model until a harness needs them, simplify the module docstring, and remove the README section. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Wenjie Fan (gggdttt)
approved these changes
Nov 3, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #1007.
Adds a harness-neutral agent contract in
bcbench_core.agent.harness, designed against all known harnesses (Copilot, Claude, PR Review, BCal, LSA), and adopts it for GitHub Copilot CLI so the contract can be reviewed with a real implementation.Contract (core)
AgentIdentity[N]: harness name (typed by the application, e.g. BC-Bench'sAgentHarness), version (CLI semver, engine SHA, hosted agent version) and optional configured model. Resolved before running.AgentRequest[T]: typed task, work dir, output dir, optional timeout (None = harness default). Environment is a harness setting, not per-run input.AgentResult: response, metrics, experiment configuration. A frozen dataclass so metrics subclasses such asPRReviewMetricspass through unchanged.Harness[T]protocol:identity()andrun(request). Generic over the task type, so specialised harnesses (PR Review onCodeReviewEntry, BCal onNL2ALEntry) take their own typed tasks; ty rejects a mismatched harness, so no runtimeisinstancechecks.Deliberately out of scope: retries, scoring, dataset loading, workspace setup, config parsing. Transcript, provenance pins and observed model are deferred until a harness needs them.
Uptake (BC-Bench)
AgentResultinstead of a(metrics, experiment)tuple. Copilot and Claude now return their final message asresponse(previously discarded).CopilotHarnessreplacesrun_copilot_agent: model, category, agent config, plugin root and runtime are construction-time settings; entry, workspace, output dir, env and timeout arrive per run.identity()replaces the CLI's directget_copilot_versioncall.EvaluationContextcarries the resolvedAgentIdentity(context.agent) instead of separateagent_name,modelandagent_version.harness_runner(harness)is a temporary bridge fromHarnessto the pipelines'AgentRunner. Once Claude, PR Review and BCal becomeHarnessclasses, pipelines take aHarnessdirectly and both go away.Supersedes #1009.