Repository navigation
ci(harness-parity): make X-capture coverage opt-out for every ACP_BACKENDS_* set - #23532
Conversation
UX Review (Opus 5.5) — ⏭️ skippedRevision |
Design Review (Opus 5.5) — ✅ PASSDesign-level review of Design-Verdict: PASS No blocking design concerns. The change only tightens a CI gate and removes nothing that any reader depends on. Notes
[DESIGN-REVIEWED] b5152da Description read[DESCRIPTION-READ] 983f339ec397a7afa24b921815ac7578ca1f40dd8c2fca59b7285d68a2fe5b6f That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
First Principles Review (Opus 5.5) — ✅ PASSPremise-level review of First-Principles-Verdict: PASS Nothing blocks. One note: Notes
What this change shipsInventory (6 items) — 5 justifiedIntent: make sure no new
[FIRST-PRINCIPLES-REVIEWED] b5152da Description read[DESCRIPTION-READ] d4cf9c024beb849f0ad71a7f12afda606e76d063b7297feade65ad053977e430 That sha256 names the PR description this verdict read. Recompute it with |
Opus 5.5 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
GPT 6.1 Review — ✅ no blocking findingsGPT 6.1 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
d10d668 to
d8f4093
Compare
buluoray
left a comment
There was a problem hiding this comment.
Requesting changes on d8f4093e505efc2c555a784dd67844c13c9bede9: the design is right, but the branch goes red on current main.
What it does. The X-capture rule only checked sets someone remembered to list, so a set using a non-standard verb or _meta field could ship without a live capture just by being left out. Now every ACP_BACKENDS_* set must sit in exactly one of X_FEATURE_SETS, a shrink-only X_REGISTRATION_PENDING, or HARNESS_EXEMPT_SETS with a reason. This tightens the gate without touching X_CAPTURE_ALLOWLIST or its ceiling.
Blocking: two unclassified sets on main. Since this branch was cut, main gained ACP_BACKENDS_REPORT_USAGE (#23170) and ACP_BACKENDS_MCP_WIRE_REINJECT (#23622). Merging current main into this head and running HARNESS_BASE_REF=origin/main python3 scripts/check_harness_parity.py fails with both as "unclassified", and test/test_x_feature_captures.py fails test_tree_passes, test_an_unclassified_new_set_fails and test_dropping_an_x_set_is_refused. Please rebase and classify both. Main has no pending list yet, so one can still go into pending if it is an X set (raise the ceiling to match), or into exempt with its reason.
What I checked on this head. Gate green ("56 sets classified (11 pending)"), --test self-test passes, test_x_feature_captures.py + test_harness_parity.py 102 pass. Mutants, one at a time: unwiring check_coverage, allowing two lists, dropping the scope-member check, accepting a blank reason, letting pending grow, and letting a set leave pending with a narrowed scope or to anywhere each redden a named test.
Test gaps (please fix with the rebase).
- Removing the
_pending_ratchetcall fromenforce_x_capturesreddens nothing, in pytest or--test. The tests call the helper directly, so the shrink-only rule is not pinned through the gate itself. One test that runsenforce_x_capturesagainst a base with a larger pending list would close it. - Removing the "in a list but no longer defined" check also reddens nothing.
CI. On this head: 89 success, 11 skipped, 3 cancelled (Automated Rule Check, GPT 6.1 Review, Sensitive Change Review). GPT 6.1 has no completed verdict yet; the rebase will re-run it. Auto-merge is off.
d8f4093 to
55479d9
Compare
|
@buluoray thanks. Rebased onto Main had gained four unclassified sets by the time of the rebase: the two you named, plus
Your test gaps:
Local results: Could you take another look? |
|
Rebased onto |
052e466 to
5164e0a
Compare
|
Rebased onto |
CrysisDeu
left a comment
There was a problem hiding this comment.
Review at 5164e0ad: gate design is right, but it goes red on current main — please disarm auto-merge
@buluoray's six points are all resolved at this head (REPORT_USAGE pending with verb, MCP_WIRE_REINJECT as X scope None, both test gaps pinned via enforce_x_captures, GPT lane green). The three-list design (X / shrink-only pending with ceiling / exempt-with-reason) and the hard-fail on an unclassified set are what we want.
BLOCKER for landing as-is: main gained ACP_BACKENDS_AGENT_STARTED_TURNS = frozenset({ACP_BACKEND_CLAUDE}) (agent_sdk/backends_turns.py:633, #18285, after your merge base). It is in none of the three lists, so on the merged tree check_coverage reports it unclassified and test_tree_passes + fast-gate fail. git merge-tree is textually clean, which is why GitHub still says MERGEABLE — the merge queue should bounce it, but auto-merge is armed, so please disarm until fixed. Fix: rebase, add it to X_REGISTRATION_PENDING naming the _meta["_claude/origin"] end marker (no claude capture exists under test/fixtures/acp_frames/claude/), bump X_REGISTRATION_PENDING_CEILING 12 → 13, re-check for any other new sets, and update the body line "every set on main is classified".
Non-blocking (follow-up fine):
HARNESS_EXEMPT_SETShas no ratchet and only a blank reason is rejected (check_harness_parity.py:431,745). Once pending is shrink-only, exempt is the only escape for a new set — including one that gates permissions over a_metafield. Require the reason to start with one of the four categories (standard ACP:/routing-only:/off the wire:/Crew-side:) and assert it; consider CODEOWNERS on the gate file._base_pending(:759) only matches an annotated assignment; a plainX_REGISTRATION_PENDING = {...}later would silently turn the ratchet off. Acceptast.Assigntoo, or fail when the base once had the list and now parses nothing.
Posted by code-reviewer (Kiro Crew member) on behalf of @CrysisDeu. Full report on file.
buluoray
left a comment
There was a problem hiding this comment.
Re-reviewed at 5164e0a. This supersedes my request-changes review on d8f4093: every point from it is fixed. ACP_BACKENDS_REPORT_USAGE is in X_REGISTRATION_PENDING with its verbs, ACP_BACKENDS_MCP_WIRE_REINJECT is in X_FEATURE_SETS with scope None, the pending ceiling moved 11 -> 12 with it, and both new gate paths (shrink-only pending against the base ref, a listed set no longer defined) are pinned by tests that drive enforce_x_captures.
Blocking, and the same class as last round: main has gained another set since this branch's base. ACP_BACKENDS_AGENT_STARTED_TURNS (agent_sdk/backends_turns.py, #18285) is unclassified, so with this PR merged onto current main test/test_x_feature_captures.py fails three times with "ACP_BACKENDS_AGENT_STARTED_TURNS: unclassified ACP_BACKENDS_* set" (I ran it on a local merge of this head into origin/main). Auto-merge is on and main is strict:false, so the green checks on the old base would let this land and turn Fast Gate red on main. Please rebase onto current main and classify AGENT_STARTED_TURNS (X_FEATURE_SETS with a capture, X_REGISTRATION_PENDING with the ceiling bumped, or HARNESS_EXEMPT_SETS with a reason). CrysisDeu's review on this head reports the same set.
5164e0a to
96b9f2e
Compare
|
@buluoray @CrysisDeu thanks, both. Auto-merge is off. Rebased onto
Local checks: @buluoray could you re-review? |
buluoray
left a comment
There was a problem hiding this comment.
Re-reviewed at 96b9f2e. This supersedes my request-changes review on 5164e0a. That round's item is fixed: ACP_BACKENDS_AGENT_STARTED_TURNS is now in X_REGISTRATION_PENDING with its verb and the ceiling moved to 13, and the base-ref reader now also catches a plain (non-annotated) rewrite of X_REGISTRATION_PENDING, with a test.
Still blocking, same class: merged onto current origin/main (f265a52), test/test_x_feature_captures.py fails three times (lines 31, 53, 88). enforce_x_captures reports:
ACP_BACKENDS_PRUNE_UNSTARTABLE_MCP [copilot]: comment cites no acp_frames/copilot/*.jsonl
So a set main carries today, for the copilot backend, has no cited live capture under test/fixtures/acp_frames/copilot/. Please rebase and either cite a replayed capture above that set, move it to X_REGISTRATION_PENDING (bumping the ceiling), or exempt it with a reason. Since main keeps adding ACP_BACKENDS_* sets, it is worth running python3 -c 'import sys; sys.path.insert(0,"scripts"); import check_harness_parity as g; g.enforce_x_captures(None)' on a fresh merge with main right before each push.
…KENDS_* set Every ACP_BACKENDS_* set must now sit in exactly one of X_FEATURE_SETS, X_REGISTRATION_PENDING (X set, capture not yet recorded; shrink-only, a leaving set must join X_FEATURE_SETS) or HARNESS_EXEMPT_SETS (one-line reason each). An unclassified new set fails with a message naming the three choices. Registers the three sets that already pass the capture rule. X_CAPTURE_ALLOWLIST and its ceiling are unchanged.
96b9f2e to
b5152da
Compare
|
One more change before re-review. Head I rebased onto New head is |
|
@buluoray thanks. Review 5672975506 (on I ran your check just now on a fresh merge of this head with origin/main |
buluoray
left a comment
There was a problem hiding this comment.
Approving b5152da. This supersedes my request-changes review on 96b9f2e.
That round's item is fixed. ACP_BACKENDS_PRUNE_UNSTARTABLE_MCP is now scoped to {deepseek, goose} (scripts/check_harness_parity.py:407), and each of those cites its own replayed capture in the set's comment in agent_sdk/backends_mcp.py. Copilot's drop is measured in Copilot CLI's log with nothing on the wire, so leaving it out of the capture scope is the honest classification, and _narrowed keeps the scope from shrinking later.
Verified on a fresh local merge of this head into origin/main 21644f3: enforce_x_captures(None) passes with "62 sets classified (13 pending)", so main has added no unclassified set; test/test_x_feature_captures.py passes (30 tests); the whole-tree harness gate exits 0. Putting the scope back to None on that merge turns four tests red (test_tree_passes, test_an_unclassified_new_set_fails, test_the_gate_holds_pending_to_the_base_ref, test_dropping_an_x_set_is_refused).
CI on this head: 93 success, 11 skipped, 2 cancelled by concurrency, 0 failures. Design, First Principles, Opus and GPT lanes report no findings.
Non-blocking: the reason copilot is outside the scope is a source comment only. If more sets end up with a partial scope, a machine-read reason per excluded backend would keep these from drifting.
Problem / Motivation
Goal: Every new
ACP_BACKENDS_*set must be classified before it ships, so a non-standard capability can no longer skip the live-capture rule by not being listed.The X-capture rule in
scripts/check_harness_parity.pyonly checks sets listed inX_FEATURE_SETS. That list is opt-in. A new set that rides a non-standard verb or_metafield can ship without being listed, and then it never needs a live capture.ACP_BACKENDS_MATE_WORKER_KIT(#23010) landed that way; its capture has since been added (#23534). Main has 13 more X sets with no capture today.Why it matters
X sets decide behaviour from a proof recorded on one adapter version. When a set skips the rule, its proof can be prose or nothing, and a wrong member breaks users with no red check. Most ACP follow-up fixes came from exactly this class.
Not a goal
X_FEATURE_SETSwhen its capture lands.X_CAPTURE_ALLOWLISTor its ceiling of 4.What changed (motivation → approach → change)
Opt-in coverage lets a set escape by omission → make coverage opt-out → every
ACP_BACKENDS_*set must sit in exactly one of three lists:X_FEATURE_SETS: checked for a cited, replayed live capture, as before.X_REGISTRATION_PENDING(new): an X set with no capture yet, each entry naming its non-standard verb or field. Shrink-only against the base ref. A set that leaves it must joinX_FEATURE_SETSwith every member checked (scopeNone), never the exempt list. Ceiling 13 intest/test_x_feature_captures.py.HARNESS_EXEMPT_SETS(new): not an X feature, each entry with a one-line reason (standard ACP, routing-only, off the wire, Crew-side).A new set in none of them fails, and the message names all three choices. A set in two lists fails. Any
X_FEATURE_SETSscope must check at least one live member.This PR also registers the five sets that already pass the capture rule:
PERMISSION_KIND_FROM_TOOL_CALL,PRUNE_UNSTARTABLE_MCP,UNMARKED_COMPACTION_PAIR,MCP_WIRE_REINJECT(citeskas/mcp-live-apply-live.jsonlandkas/mcp-live-remove-live.jsonl) andMATE_WORKER_TOOL_LIMITS. The last one's comment inagent_sdk/backends_members.pynow citeskas/worker-kit-live.jsonl, where a crewmate'sexcludedTools [shell]withheldshellon KAS 0.66.58.Sets main gained since this branch was cut:
REPORT_USAGEgoes to pending (kiro-cli bills via_kiro.dev/metadatameteringUsage, KAS via_meta.kiroturn_completion; no replayed KAS capture keeps that frame),AGENT_STARTED_TURNS(#18285) goes to pending (claude marks the turn's end with_meta["_claude/origin"]; no claude capture of an agent-started turn exists yet), andMATE_WORKER_PROMPT_IN_CONTEXTis exempt (Crew-side: the worker prompt rides the context block).PRUNE_UNSTARTABLE_MCPis scoped to{deepseek, goose}, each with its own live capture. copilot (#22403) joined the set on main, but its drop is measured in Copilot CLI's log with nothing on the wire, so no ACP capture can show it._base_pendingreads the base list from an annotated or a plain assignment, so rewriting it as a plainX_REGISTRATION_PENDING = {...}cannot switch the ratchet off.docs/ci/harness-parity-gate.mddescribes the new contract.Alternative considered: put the 13 X sets in
X_CAPTURE_ALLOWLIST. That needs its ceiling raised from 4 to 17 and a carve-out in its ratchet, which loosens an existing gate. A separate pending list tightens without touching it.ACP_BACKENDS_MATE_WORKER_KITis already inX_FEATURE_SETSon main (#23534), so it is not in the pending list.Size: about 220 lines. 45 of them are the two new lists, one entry per line.
Backwards compatibility
Compatible: every set on main as of
ab57fba80is classified in this PR (62 sets), so the gate stays green on the current tree. Only a future unclassified set fails.Reader: .github/workflows/fast-gate.yml:check_harness_parity.py -- entry: release -- runs the same gate, green on this tree (
HARNESS_BASE_REF=origin/main python3 scripts/check_harness_parity.py)Reader: test/test_x_feature_captures.py:test_tree_passes -- entry: release -- passes on this tree
Tests
test_an_unclassified_new_set_fails: a new set in no list fails with the classification message. It passes once exempt with a reason, and fails again with a blank reason.test_a_set_in_two_lists_fails: a set in bothX_FEATURE_SETSand the exempt list fails.test_pending_only_shrinks_into_x_feature_sets: an added pending entry fails. A pending set moved to exempt fails. A pending set moved toX_FEATURE_SETSwith an empty scope fails. Moving it with scopeNonepasses.test_base_pending_is_read_back_from_the_gate_source: the base ref's pending list is parsed from the gate source, a plain-assignment list is read too, and a base without the list reads asNone.test_the_gate_holds_pending_to_the_base_ref: runsenforce_x_capturesagainst a base pending list; a set moved from pending to exempt fails, and an entry the base lacks fails.test_a_listed_set_no_longer_defined_fails: a pending or exempt entry whose set is gone fails through the gate.test_allowlists_stay_under_their_ceiling: now also caps the pending list at 13.pytest test/test_x_feature_captures.py test/test_harness_parity.py test/test_acp_frame_replay.py test/test_ratchet_scope.py test/test_main_ratchet_audit.py: 263 passed.python3 scripts/check_harness_parity.py --test: passed.Manual verification
HARNESS_BASE_REF=origin/main python3 scripts/check_harness_parity.py: green, "62 sets classified (13 pending)". Removing the_pending_ratchetcall, or the "no longer defined" check, each reddens a named test. Local GPT and Opus review on b5152da: no findings.Related Issues
Fixes #23531
Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)Contribution License Agreement
Contributions are accepted under the project's Apache-2.0 license.