Build a reproducible acceptance harness for confidential workflows across components

Problem and scope

Component tests do not establish that a composed model, agent, tool and delegated peer preserve a selected confidentiality policy.

Provide a reusable acceptance harness with replaceable model, tool, transport, attestation and deployment adapters. Start with a deterministic synthetic workflow; label fresh hardware evidence separately.

Acceptance criteria

Define assets, principals, permitted recipients, trusted components and the exact claim evaluated by each run.
Join key provisioning, inference, a mediated tool call, a delegated peer call and response verification using a single bound transaction context.
Capture independent observations at release, receipt and execution boundaries; keep authorization, delivery, installation and execution outcomes separate.
Cover workload/key/configuration substitution, software downgrade, replay, missing evidence, bypass egress, forbidden disclosure, revocation and timeout with unknown outcome.
Pair expected refusals with valid controls and show that removing each tested gate causes its intended test to fail.
Produce machine-readable results with source revisions, dependency versions, policy/trust inputs, commands and explicit synthetic/replayed/live evidence classification.
Reproduce a live run across independently operated peers satisfying the selected policy. Sanitized public evidence must not expose protected data or device identifiers.

Dependencies and closure

Consume the response-binding, confinement, measured-identity, lifecycle, protected-execution and disclosure workstreams linked from the parent. agentrust-io/ca2a#47 remains the owner of cA2A hardware conformance and release-assurance decisions. A synthetic harness can complete its software milestone without closing live-hardware acceptance or promoting a component's release claim.

Published integration examples must use released packages, following this repository's contribution rules. Before required APIs are released, keep protocol development in the owning core repositories and mark the integrated milestone as waiting on those releases.

I would like to take the bounded synthetic/software milestone for #199.

I checked the integrations contribution rule before starting implementation. The current published releases are cMCP 0.5.0 and cA2A 0.2.0; the newer exact-output disclosure work and recent response-binding work landed after those releases. Because this repository requires published integrations to run against released packages, I do not want to build the public harness against main/unreleased APIs and then call it reproducible.

Proposed shape:

1. One run manifest, one explicit claim

Each run begins with a machine-readable manifest that states:

run ID / transaction context
protected assets and sensitivity
workload principal, represented user/delegator where present, delegated peer
permitted recipients / forbidden recipients
policy version and trust inputs
component package versions and source revisions
evidence classification per component: synthetic, replayed, hosted/live, hardware
exact claim under evaluation, e.g. “asset X may be processed by workload A, delegated to peer B within scope S, and released only to recipient R under policy P”

The result must never silently upgrade a synthetic component because another component used live evidence.

2. Replaceable adapters, fixed harness semantics

Keep the harness neutral and make components adapters:

key/provisioning adapter
model/inference adapter
mediated-tool adapter
delegated-peer / cA2A adapter
response-verification adapter
disclosure/release adapter
attestation/deployment adapter
independent observation adapters

Adapters return typed observations. They do not decide the overall verdict themselves.

3. Separate outcome dimensions

Do not collapse “authorized” into “happened.” Record at least:

authorization
delivery
receipt
installation/admission
execution
response verification
release/disclosure

Each can be established / contradicted / unavailable (or an equivalent closed vocabulary). A timeout after dispatch should remain unknown, not become success or failure by inference.

4. First deterministic synthetic workflow

Positive control:

key provisioning -> deterministic model/inference step -> mediated cMCP tool call -> scoped delegation to cA2A peer -> peer response -> response verification -> permitted release

The same transaction context must be bound across the artifacts used by the run; a post-hoc log correlation alone should not satisfy that criterion.

Negative cases:

workload substitution
key substitution
effective-config/policy substitution
software downgrade
replay
missing evidence
bypass egress
forbidden disclosure
revocation after authorization / before use
timeout after dispatch with unknown outcome

Each refusal gets a paired valid control.

5. Mutation proof for every claimed gate

For each negative case, deliberately remove or weaken the relevant gate and require the corresponding test to turn red. This prevents a “green” harness whose expected refusal is actually caused by some unrelated earlier check.

6. Evidence bundle

Emit a machine-readable result containing:

scenario ID
run/transaction ID
package versions and source revisions
commands
policy/trust input digests
adapter/evidence classifications
per-boundary observations
expected vs observed outcome
mutation-control result
explicit limitations

No protected payloads, device IDs, secrets or low-entropy confidential digests in the public evidence.

Release gate

I suggest we keep protocol/API development in the owning core repos until the required response-binding/disclosure surfaces are published, as CONTRIBUTING requires. Once compatible releases exist, the first integrations PR can pin those exact package versions and deliver the deterministic software milestone. The live independently-operated-peer/hardware milestone remains separate and should not be implied by the synthetic pass.

I can prepare the harness layout and test matrix against this contract now, but would avoid submitting code that imports unreleased APIs.

@altrudev

@altrudev please go ahead with the harness layout and test matrix for the synthetic/software milestone. That matches the scope here.

Keep protocol/API changes in the owning core repos, and pin released package versions when wiring up the integration. Mark any blocked cases as waiting on those releases. A synthetic pass completes only the software milestone; live peer and hardware acceptance remain open.

I reviewed the synthetic/software milestone again as an evidence-boundary problem before turning it into code. I think the harness should be split into a small fixed core plus replaceable adapters, with the test matrix proving each gate independently rather than treating one end-to-end green run as sufficient.

Harness layout

confidential-workflow-harness/
  README.md
  manifest.schema.json
  result.schema.json
  scenarios/
    positive-control.yaml
    workload-substitution.yaml
    key-substitution.yaml
    policy-substitution.yaml
    software-downgrade.yaml
    replay.yaml
    missing-evidence.yaml
    bypass-egress.yaml
    forbidden-disclosure.yaml
    revocation-before-use.yaml
    timeout-after-dispatch.yaml
  harness/
    model.py
    types.py
    runner.py
    verdict.py
    evidence.py
    adapters/
      provisioning.py
      inference.py
      mediated_tool.py
      delegated_peer.py
      response_verifier.py
      release.py
      attestation.py
      observation.py
  tests/
    test_positive_control.py
    test_negative_matrix.py
    test_mutation_proofs.py
    test_evidence_bundle.py
    test_unknown_outcomes.py

The harness core owns transaction binding, scenario sequencing, evidence classification and verdict semantics. Adapters emit observations only; they cannot upgrade the overall verdict.

Closed outcome vocabulary

For each boundary, record one of:

established
contradicted
unavailable
not_applicable

and keep these dimensions independent:

authorization
delivery
receipt
installation/admission
execution
response verification
release/disclosure

A timeout after dispatch therefore yields, for example, delivery=established and execution=unavailable; the harness must not infer execution from dispatch.

Binding rule

Each artifact used to establish the positive claim must carry or be cryptographically/structurally bound to the same transaction context. Timestamp proximity, log adjacency and request correlation are not sufficient substitutes.

A run manifest should minimally bind:

run/transaction ID
protected asset identifier (non-secret)
workload principal and represented/delegating principal
delegated peer
permitted and forbidden recipients
policy version/digest
trust-input digest
package versions/source revisions
evidence classification per adapter
exact claim under test

Test matrix

Scenario	Expected boundary	Expected result	Mutation proof
positive control	all required software boundaries	established	n/a
workload substitution	admission/execution	contradicted/refused	disable workload identity gate -> scenario must turn green
key substitution	authorization/verification	contradicted/refused	trust substituted key -> scenario must turn green
policy/config substitution	authorization/release	contradicted/refused	bypass policy binding -> scenario must turn green
software downgrade	admission	contradicted/refused	remove version floor -> scenario must turn green
replay	authorization/receipt	contradicted/refused	remove nonce/replay gate -> scenario must turn green
missing evidence	affected boundary only	unavailable	synthetic evidence insertion -> expected control becomes established
bypass egress	release	contradicted/refused	bypass release gate -> scenario must turn green
forbidden disclosure	release	contradicted/refused	authorize recipient -> paired control becomes established
revocation after authorization / before use	execution/release	contradicted/refused	suppress revocation check -> scenario must turn green
timeout after dispatch	execution	unavailable	deterministic completion control -> established

For every negative case, the paired valid control should differ only in the property under test.

Evidence bundle

Each run should emit one machine-readable result containing:

scenario ID and transaction ID
exact claim
package versions and source revisions
commands/entry point
policy and trust-input digests
adapter type + evidence classification (synthetic, replayed, hosted/live, hardware)
ordered observations with causal source/provenance
per-boundary outcomes
expected vs observed result
mutation-proof result
blocked/waiting-on-release markers
explicit limitations

No protected payloads, secrets, device identifiers or low-entropy digests of protected material should enter the public evidence bundle.

Evidence-integrity checks

Two additional checks seem important for this milestone:

branch preservation: retries/alternate paths remain separate lineages rather than being flattened into one successful timeline;
post-action contamination: later retry/recovery/reconciliation evidence cannot silently replace the evidence that existed at the earlier decision boundary.

These are harness-level properties and do not require protocol/API changes.

Release boundary

I will keep the initial implementation strictly on published package surfaces. Where a scenario depends on unreleased response-binding or disclosure APIs, the test should be present but explicitly marked waiting_on_release; it must not be simulated in a way that produces a false pass.

The software milestone can then be called complete only when:

the deterministic positive control passes on released packages,
every available negative case and its mutation proof behaves as expected,
blocked cases remain explicitly blocked rather than inferred,
the evidence bundle validates against schema, and
no result implies live-peer or hardware acceptance.

If this shape looks right, the next step is to implement the fixed core/schema + deterministic adapters, then wire released cMCP/cA2A surfaces as they are available.

@altrudev

@altrudev, go ahead with a focused PR for the core, schemas and deterministic software adapters.

Keep the valid comparison cases, but also weaken each safety check and show that its test fails. Making an input valid is a separate control.

If a required case is waiting on a release, keep the overall software milestone incomplete and report which parts passed. Pin released packages; live-peer and hardware acceptance remain separate.

Independent source/test review at integrations main 5f9b48e: python -m unittest discover -s tests -v passes all 23 harness tests. Timeout-after-dispatch correctly preserves delivery=established and main execution=unavailable; retry observations do not overwrite that summary.

A bounded retry-evidence gap is reproducible in harness/adapters/deterministic.py: the retry_lineage block emits retry execution=established unconditionally. With the unchanged synthetic inputs plus retry_lineage=True:

Input	Overall / main execution	Retry authorization / execution
Valid control	passed / established	established / established
workload=workload-X	refused / contradicted	contradicted / established
revoked_before_use=True	refused / contradicted	established / established
replay=True	refused / contradicted	established / established
software_version=0.9.0	refused / contradicted	established / established

Reproduction from integrations/agentrust-confidential-workflow-harness:

import sys
sys.path.insert(0, 'tests')
from test_harness import scenario
from harness import HarnessRunner
s = scenario(inputs={**scenario().inputs, 'workload': 'workload-X', 'retry_lineage': True})
r = HarnessRunner().run(s)
print(r.status, [(o.boundary.value, o.outcome.value) for o in r.observations if o.lineage == 'retry-1'])

The overall summary remains refused, so this is not a demonstrated overall acceptance bypass or live execution issue. It is an unsupported synthetic retry execution assertion in the machine-readable observations, most clearly visible when retry authorization itself is contradicted.

Suggested bounded follow-up: define whether the retry models actual guarded synthetic execution or a scripted observation. For guarded execution, evaluate authorization/admission/revocation/replay in that lineage and add valid/refused retry twins plus gate-removal controls; if scripted, represent its assumed execution distinctly so consumers cannot read it as an enforced result. Preserve main-lineage timeout uncertainty. This review proposes no protocol change or hardware claim and does not claim completion of the software milestone. Coordination with the existing harness owner requested before a corrective implementation.

Concrete supplemental draft for the retry-evidence finding: dinakarjs#3 at 9eba4dd, based on 5f9b48e main. Kept in my fork for coordination with the existing owner.

It shares admission/execution guard evaluation across main and retry lineages, adds retry receipt/admission observations and a causal admission parent, and covers workload/key/policy/downgrade/revocation/replay/missing evidence with seven positive/refused-or-unknown/gate-weakened comparisons. All 24 harness tests, targeted Ruff and diff checks pass on Python 3.12.14. Restoring unconditional retry execution=established in an isolated process kills all seven new refusal/unknown controls; no mutation committed.

Main-lineage timeout uncertainty remains intact. This is a deterministic software evidence correction, not live retry or hardware/protocol conformance; please review the proposed retry interpretation before upstream integration.

@dinakarjs

@dinakarjs reproduced at 5f9b48e: with retry_lineage=True, the workload, revocation and replay inputs refuse overall and still record retry execution as established. The retry should be guarded execution, evaluated by the same checks as the main lineage, which is what your #3 does and what the 09-21 instruction asked for. Please open it here, and @altrudev, as the harness owner, could you review it by Oct 20?

Opened the coordinated retry correction upstream as #316 at 69d3ef6, as requested. Carries the tested fork draft, its seven guarded retry controls and passing hosted checks; ready for harness-owner review by October 20. Main-lineage timeout uncertainty and synthetic-only scope remain explicit.

@altrudev

#316 merged on October 14 at 88528a7, so retry execution now goes through the same guards as the main lineage. On main at 52c2c96 the harness runs 27 tests, all passing.

The README still targets cmcp-runtime 0.5.0 and ca2a-runtime 0.2.0. PyPI now has cmcp-runtime 0.7.0 (September 30), which contains cmcp#663, and ca2a-runtime 0.4.0 (October 1). The release wait from September 19 no longer covers the confinement work; cmcp#660 is still open.

@altrudev, say here whether you are keeping the protocol adapters against those versions. If not, they open to anyone.

@altrudev

@altrudev following Imran's October 15 ownership question: are you retaining the cMCP/cA2A protocol adapters for this harness against the released cmcp-runtime 0.7.0 and ca2a-runtime 0.4.0 surfaces? I can contribute the bounded released-package adapter work if you are handing it over, or independent paired acceptance/refusal and mutation review if you are keeping it. I am inspecting the release/API compatibility now; no competing upstream implementation or ownership claim is being opened while your answer is pending. Main/retry timeout uncertainty and synthetic/replayed/live evidence classifications will remain separate.

@altrudev

Supplemental release-API readiness evidence is prepared in dinakarjs#4 at eca7cfb (fork draft, upstream baseline 84d1530).

Using installed cmcp-runtime 0.7.0 and ca2a-runtime 0.4.0, 14 component probes pass for response authentication/request binding/replay/expiry and exact-output disclosure/refusal/unknown delivery. Three process-local MAC/expiry/replay weakenings are detected with valid controls passing. Reproduced in a new Python 3.12.14 environment using the included 31-package hash lock; the existing deterministic harness separately passes 27 tests.

This establishes that these tested release APIs are available; it does not wire protocol adapters, complete the composed transaction/mutation matrix, or establish live peer/hardware evidence. Source imports do not substitute unreleased branches. @altrudev the adapter ownership question remains yours to answer; these test-only probes can support your implementation or a coordinated handoff. No competing upstream adapter implementation opened.

@imran-siddique

@imran-siddique @dinakarjs Thanks for the updates and the release-API verification work.

I'll retain responsibility for the cMCP/cA2A protocol adapters in the acceptance harness, targeting the released cmcp-runtime 0.7.0 and ca2a-runtime 0.4.0 interfaces.

I'd welcome independent paired acceptance/refusal testing and mutation review from @dinakarjs. Having the implementation and verification performed independently is useful here.

I'd like to keep the scope bounded to the released interfaces and the existing harness, with explicit evidence for authorization, request binding, replay, expiry, refusal, and observed outcomes.

We should continue distinguishing deterministic tests, synthetic/replayed evidence, and live execution. Passing component probes shouldn't be represented as completed end-to-end acceptance until the adapters are wired and the composed transaction matrix passes.

I'll work through the remaining implementation and EDRG acceptance gaps, with any unresolved limitations recorded explicitly.

Thanks for offering to collaborate.

@altrudev

@altrudev accepted: you retain protocol-adapter implementation; I will cover independent paired acceptance/refusal and mutation review against the released cmcp-runtime 0.7.0 / ca2a-runtime 0.4.0 interfaces and the existing harness.

The existing supplemental component probes remain at dinakarjs#4 (eca7cfb): 14 released-API controls with a 31-package hash lock and MAC/expiry/replay mutation checks. This is component readiness evidence, not adapter integration or composed acceptance.

For the adapter review, the bounded matrix will check: conformant delivery; wrong/missing authorization; altered request/output binding; replay; expiry boundaries; known refusal; and uncertain/unobserved delivery. Each refusal needs a conformant twin and an observed outcome, with deterministic, synthetic/replayed and live evidence distinguished. Implementation ownership remains yours. Please link the pinned adapter PR/head here when ready so I can run the composed matrix independently. End-to-end acceptance remains open until that implementation and matrix pass.

Navigation Menu

Repository navigation

Build a reproducible acceptance harness for confidential workflows across components #199

Problem and scope

Acceptance criteria

Dependencies and closure

1. One run manifest, one explicit claim

2. Replaceable adapters, fixed harness semantics

3. Separate outcome dimensions

4. First deterministic synthetic workflow

5. Mutation proof for every claimed gate

6. Evidence bundle

Release gate

Harness layout

Closed outcome vocabulary

Binding rule

Test matrix

Evidence bundle

Evidence-integrity checks

Release boundary

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Build a reproducible acceptance harness for confidential workflows across components #199

Description

Problem and scope

Acceptance criteria

Dependencies and closure

Activity

altrudev commented on Sep 19, 2026

1. One run manifest, one explicit claim

2. Replaceable adapters, fixed harness semantics

3. Separate outcome dimensions

4. First deterministic synthetic workflow

5. Mutation proof for every claimed gate

6. Evidence bundle

Release gate

imran-siddique commented on Sep 20, 2026

altrudev commented on Sep 21, 2026

Harness layout

Closed outcome vocabulary

Binding rule

Test matrix

Evidence bundle

Evidence-integrity checks

Release boundary

imran-siddique commented on Sep 21, 2026

dinakarjs commented on Oct 13, 2026

dinakarjs commented on Oct 13, 2026

imran-siddique commented on Oct 14, 2026

dinakarjs commented on Oct 14, 2026

imran-siddique commented on Oct 15, 2026

dinakarjs commented on Oct 15, 2026

dinakarjs commented on Oct 15, 2026

altrudev commented on Oct 15, 2026

dinakarjs commented on Oct 16, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions