Turn the coding-agent scenario into the flagship security + utility benchmark #253

Goal

Use the existing coding-agent scenario as a candidate flagship benchmark, not just a demo that proves Kernel can block things.

The benchmark must answer two questions at the same time:

1. Security: does the mediated boundary prevent actions outside the granted authority?
2. Utility: can the agent still complete the work it was legitimately asked to do without excessive false denials or policy ceremony?

A deny-all policy is not a successful result.

This remains a candidate beachhead. Gate 0 (#278) must validate that coding-agent security is a stronger external wedge than alternative application/workflow-agent use cases before broad promotion.

Canonical problem statement

Give a coding agent exactly the repository, shell and GitHub powers needed for the current step, without silently granting secrets, workflow rewrites, arbitrary network/package operations or merge authority—and leave evidence showing which mediated actions were allowed or denied.

Do not claim that Kernel makes a compromised model safe or provides a sandbox. The precise claim is that an unauthorized action submitted through the Kernel mediation path is rejected before the configured driver executes it.

Capability profile

Keep the public vocabulary small and concrete, for example:

• repo.read.files
• repo.write.files
• path-scoped write constraints
• shell.run.tests
• shell.run.package_manager
• shell.run.networked_command
• github.read.*
• github.create_pr
• github.merge_pr
• secrets.read — denied unless deliberately granted

The benchmark should prove constraints, not merely capability names. Examples:

• writes only under approved source/test paths;
• .github/workflows/** remains denied;
• approved test commands run, arbitrary shell/network commands do not;
• PR creation can require explicit escalation;
• merge remains unavailable after PR-create approval unless separately granted.

Reproducible scenario

Use a hermetic/local fixture so the core benchmark requires no LLM API key or external account.

review phase
  ALLOW repo.read + github.read
  DENY repo.write, shell/network, secrets

edit/test phase
  ALLOW bounded repo.write + shell.run.tests
  DENY workflows, secrets, arbitrary shell/network, publish

publish phase
  request github.create_pr
  → explicit approval / capability escalation
  → ALLOW create PR
  → still DENY merge

Visible evidence should include:

1. allowed repository read;
2. successful bounded edit;
3. successful test execution;
4. denied secret/environment access;
5. denied out-of-scope path write;
6. denied arbitrary/networked command;
7. approval-required PR creation;
8. approved PR creation succeeding;
9. merge still denied;
10. kernel.explain() / ActionTrace tying each mediated action to principal, capability/constraint, decision/reason and outcome.

Benchmark dimensions

Report both protected and useful behavior:

Where a native framework control or another authorization approach can run the same fixture, include it as a neutral comparison. The goal is to identify the incremental value of Kernel, not to construct a benchmark that competitors cannot participate in.

Framework coverage

If this scenario is later wired into OpenAI/LangChain/other agent frameworks, publish a surface-by-surface mediation table. Do not write “framework X is secured by Kernel” when only custom function tools cross the wrapper.

The offline fixture remains the canonical benchmark because it measures Kernel semantics independently of model quality or framework behavior.

Adoption artifacts

Only after the benchmark and Gate 0 evidence support the coding-agent wedge:

• <60-second terminal cast: useful read/edit/test → denied forbidden action → approved bounded escalation → trace;
• copy/paste released-package quickstart;
• three minimal policy examples: review-only, edit+test, create-PR;
• explicit “when not to use Kernel” and mediation-boundary box;
• machine-readable benchmark results committed with enough metadata to reproduce them.

Acceptance criteria

• Minimal coding-agent capability/constraint vocabulary is defined.
• Existing offline scenario covers useful ALLOW + security DENY + bounded escalation.
• Path-scoped writes and command-category restrictions are demonstrated.
• Approval does not implicitly grant adjacent authority (e.g. create PR ≠ merge).
• Benchmark reports security and utility metrics above.
• Every benchmark decision has inspectable ActionTrace / explanation evidence.
• Explicit non-claims match docs/security-contract.md.
• Any framework-backed variant publishes a mediation coverage matrix.
• Results are reproducible from the released weaver-kernel package or clearly identify unreleased commit SHA.
• Validate the execution-enforcement wedge against native guardrails and existing authz #278 validates coding agents as a worthwhile external wedge before broad launch/content promotion.

Dependencies / sequencing

• Security-contract and roadmap foundation: merged PR docs: define security contract and adoption-gated roadmap #276.
• Easy-path semantics: dx: framework middleware / drop-in mode (wrap an existing tool loop, don't rebuild the agent) #104.
• Product falsification / independent-user evidence: Validate the execution-enforcement wedge against native guardrails and existing authz #278.
• Known supported-path hardening remains higher priority than marketing artifacts.

Non-goals

• Proving prompt-injection resistance in general.
• Claiming a VM/container sandbox.
• Making the coding-agent vocabulary a universal standard before external use demonstrates portability.
• Growing a generic capability taxonomy before the concrete benchmark is excellent.