Skip to content

ci(harness-parity): make X-capture coverage opt-out for every ACP_BACKENDS_* set - #23532

Merged
iamwhatever merged 1 commit into
mainfrom
fix/g2-registry-opt-out
Oct 28, 2026
Merged

iamwhatever merged 1 commit into
mainfrom
fix/g2-registry-opt-out

Conversation

@iamwhatever

@iamwhatever iamwhatever commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: Every new ACP_BACKENDS_* set must be classified before it ships, so a non-standard capability can no longer skip the live-capture rule by not being listed.

The X-capture rule in scripts/check_harness_parity.py only checks sets listed in X_FEATURE_SETS. That list is opt-in. A new set that rides a non-standard verb or _meta field can ship without being listed, and then it never needs a live capture. ACP_BACKENDS_MATE_WORKER_KIT (#23010) landed that way; its capture has since been added (#23534). Main has 13 more X sets with no capture today.

Why it matters

X sets decide behaviour from a proof recorded on one adapter version. When a set skips the rule, its proof can be prose or nothing, and a wrong member breaks users with no red check. Most ACP follow-up fixes came from exactly this class.

Not a goal

  • Recording the missing captures for the 13 pending sets. Each moves to X_FEATURE_SETS when its capture lands.
  • Changing X_CAPTURE_ALLOWLIST or its ceiling of 4.
  • Detecting a pending set that is renamed and then exempted. A rename shows in review, and the gate has no notion of set identity beyond the name.

What changed (motivation → approach → change)

Opt-in coverage lets a set escape by omission → make coverage opt-out → every ACP_BACKENDS_* set must sit in exactly one of three lists:

  • X_FEATURE_SETS: checked for a cited, replayed live capture, as before.
  • X_REGISTRATION_PENDING (new): an X set with no capture yet, each entry naming its non-standard verb or field. Shrink-only against the base ref. A set that leaves it must join X_FEATURE_SETS with every member checked (scope None), never the exempt list. Ceiling 13 in test/test_x_feature_captures.py.
  • HARNESS_EXEMPT_SETS (new): not an X feature, each entry with a one-line reason (standard ACP, routing-only, off the wire, Crew-side).

A new set in none of them fails, and the message names all three choices. A set in two lists fails. Any X_FEATURE_SETS scope must check at least one live member.

This PR also registers the five sets that already pass the capture rule: PERMISSION_KIND_FROM_TOOL_CALL, PRUNE_UNSTARTABLE_MCP, UNMARKED_COMPACTION_PAIR, MCP_WIRE_REINJECT (cites kas/mcp-live-apply-live.jsonl and kas/mcp-live-remove-live.jsonl) and MATE_WORKER_TOOL_LIMITS. The last one's comment in agent_sdk/backends_members.py now cites kas/worker-kit-live.jsonl, where a crewmate's excludedTools [shell] withheld shell on KAS 0.66.58.

Sets main gained since this branch was cut: REPORT_USAGE goes to pending (kiro-cli bills via _kiro.dev/metadata meteringUsage, KAS via _meta.kiro turn_completion; no replayed KAS capture keeps that frame), AGENT_STARTED_TURNS (#18285) goes to pending (claude marks the turn's end with _meta["_claude/origin"]; no claude capture of an agent-started turn exists yet), and MATE_WORKER_PROMPT_IN_CONTEXT is exempt (Crew-side: the worker prompt rides the context block).

PRUNE_UNSTARTABLE_MCP is scoped to {deepseek, goose}, each with its own live capture. copilot (#22403) joined the set on main, but its drop is measured in Copilot CLI's log with nothing on the wire, so no ACP capture can show it.

_base_pending reads the base list from an annotated or a plain assignment, so rewriting it as a plain X_REGISTRATION_PENDING = {...} cannot switch the ratchet off. docs/ci/harness-parity-gate.md describes the new contract.

Alternative considered: put the 13 X sets in X_CAPTURE_ALLOWLIST. That needs its ceiling raised from 4 to 17 and a carve-out in its ratchet, which loosens an existing gate. A separate pending list tightens without touching it.

ACP_BACKENDS_MATE_WORKER_KIT is already in X_FEATURE_SETS on main (#23534), so it is not in the pending list.

Size: about 220 lines. 45 of them are the two new lists, one entry per line.

Backwards compatibility

Compatible: every set on main as of ab57fba80 is classified in this PR (62 sets), so the gate stays green on the current tree. Only a future unclassified set fails.

Reader: .github/workflows/fast-gate.yml:check_harness_parity.py -- entry: release -- runs the same gate, green on this tree (HARNESS_BASE_REF=origin/main python3 scripts/check_harness_parity.py)
Reader: test/test_x_feature_captures.py:test_tree_passes -- entry: release -- passes on this tree

Tests

  • test_an_unclassified_new_set_fails: a new set in no list fails with the classification message. It passes once exempt with a reason, and fails again with a blank reason.
  • test_a_set_in_two_lists_fails: a set in both X_FEATURE_SETS and the exempt list fails.
  • test_pending_only_shrinks_into_x_feature_sets: an added pending entry fails. A pending set moved to exempt fails. A pending set moved to X_FEATURE_SETS with an empty scope fails. Moving it with scope None passes.
  • test_base_pending_is_read_back_from_the_gate_source: the base ref's pending list is parsed from the gate source, a plain-assignment list is read too, and a base without the list reads as None.
  • test_the_gate_holds_pending_to_the_base_ref: runs enforce_x_captures against a base pending list; a set moved from pending to exempt fails, and an entry the base lacks fails.
  • test_a_listed_set_no_longer_defined_fails: a pending or exempt entry whose set is gone fails through the gate.
  • test_allowlists_stay_under_their_ceiling: now also caps the pending list at 13.

pytest test/test_x_feature_captures.py test/test_harness_parity.py test/test_acp_frame_replay.py test/test_ratchet_scope.py test/test_main_ratchet_audit.py: 263 passed. python3 scripts/check_harness_parity.py --test: passed.

Manual verification

HARNESS_BASE_REF=origin/main python3 scripts/check_harness_parity.py: green, "62 sets classified (13 pending)". Removing the _pending_ratchet call, or the "no longer defined" check, each reddens a named test. Local GPT and Opus review on b5152da: no findings.

Related Issues

Fixes #23531

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

Contribution License Agreement

Contributions are accepted under the project's Apache-2.0 license.

@iamwhatever
iamwhatever requested review from a team and KiroCrewCaptain as code owners October 23, 2026 19:43
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Oct 23, 2026
@github-actions

github-actions Bot commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision b5152da4adb73f6b194503ca7ac0cfa5f2246d70 touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions

github-actions Bot commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of b5152da4adb73f6b194503ca7ac0cfa5f2246d70 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

No blocking design concerns. The change only tightens a CI gate and removes nothing that any reader depends on.

Notes

  • HARNESS_EXEMPT_SETS is now the only unratcheted route out of the capture rule. A new X set labelled "standard ACP" passes as long as nobody questions the label in review. That is better than escaping silently by omission, but it still depends on the reviewer catching it.

[DESIGN-REVIEWED] b5152da

Description read

[DESCRIPTION-READ] 983f339ec397a7afa24b921815ac7578ca1f40dd8c2fca59b7285d68a2fe5b6f

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ✅ PASS

Premise-level review of b5152da4adb73f6b194503ca7ac0cfa5f2246d70 — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

First-Principles-Verdict: PASS

Nothing blocks. One note: _base_pending is a second base-ref reader beside _base_allowlists.

Notes

  • Item 4 — duplicate of scripts/check_harness_parity.py:_base_allowlists: grepping git(["show", f"{base}:scripts/check_harness_parity.py"]) finds 2 readers, and both parse the same base source to pull top-level literals. Subtraction: fold the X_REGISTRATION_PENDING lookup into the existing reader's names scan (with a separate None for pending only), and delete _base_pending.

What this change ships

Inventory (6 items) — 5 justified

Intent: make sure no new ACP_BACKENDS_* set can skip the live-capture rule just because nobody listed it (#23531). FIX.

  1. The gate fails when a set is in none of the three lists, or in two of them — justified
  2. A new pending list of 13 X sets with no capture yet, which can only shrink and has a ceiling test — justified
  3. A new exempt list of 32 sets, each with a one-line reason — justified
  4. The base ref's pending list is read back to hold the ratchet — duplicate of _base_allowlists
  5. Five already-passing sets are added to X_FEATURE_SETS (PRUNE_UNSTARTABLE_MCP scoped to deepseek and goose) — justified
  6. The MATE_WORKER_TOOL_LIMITS comment now cites the KAS worker-kit capture — justified

[FIRST-PRINCIPLES-REVIEWED] b5152da

Description read

[DESCRIPTION-READ] d4cf9c024beb849f0ad71a7f12afda606e76d063b7297feade65ad053977e430

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed b5152da4adb73f6b194503ca7ac0cfa5f2246d70 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] b5152da

Verdict parsed from the review's SHA-scoped output markers for commit b5152da4adb73f6b194503ca7ac0cfa5f2246d70.

False positive or not applicable? A repository writer can comment:
/ai-review override fable b5152da4adb73f6b194503ca7ac0cfa5f2246d70 false-positive: <reason, 20 to 500 characters>

@github-actions

github-actions Bot commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of b5152da4adb73f6b194503ca7ac0cfa5f2246d70 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] b5152da

False positive or not applicable? A repository writer can comment:
/ai-review override gpt b5152da4adb73f6b194503ca7ac0cfa5f2246d70 false-positive: <reason, 20 to 500 characters>

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision merge conflict Branch has merge conflicts with its base — author must resolve before merge and removed readiness: checking Automated validation is still running labels Oct 23, 2026
@iamwhatever
iamwhatever force-pushed the fix/g2-registry-opt-out branch from d10d668 to d8f4093 Compare October 24, 2026 20:16
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: passed Eligible automated validation passed for the current revision and removed readiness: passed Eligible automated validation passed for the current revision merge conflict Branch has merge conflicts with its base — author must resolve before merge readiness: checking Automated validation is still running labels Oct 24, 2026

@buluoray buluoray left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Requesting changes on d8f4093e505efc2c555a784dd67844c13c9bede9: the design is right, but the branch goes red on current main.

What it does. The X-capture rule only checked sets someone remembered to list, so a set using a non-standard verb or _meta field could ship without a live capture just by being left out. Now every ACP_BACKENDS_* set must sit in exactly one of X_FEATURE_SETS, a shrink-only X_REGISTRATION_PENDING, or HARNESS_EXEMPT_SETS with a reason. This tightens the gate without touching X_CAPTURE_ALLOWLIST or its ceiling.

Blocking: two unclassified sets on main. Since this branch was cut, main gained ACP_BACKENDS_REPORT_USAGE (#23170) and ACP_BACKENDS_MCP_WIRE_REINJECT (#23622). Merging current main into this head and running HARNESS_BASE_REF=origin/main python3 scripts/check_harness_parity.py fails with both as "unclassified", and test/test_x_feature_captures.py fails test_tree_passes, test_an_unclassified_new_set_fails and test_dropping_an_x_set_is_refused. Please rebase and classify both. Main has no pending list yet, so one can still go into pending if it is an X set (raise the ceiling to match), or into exempt with its reason.

What I checked on this head. Gate green ("56 sets classified (11 pending)"), --test self-test passes, test_x_feature_captures.py + test_harness_parity.py 102 pass. Mutants, one at a time: unwiring check_coverage, allowing two lists, dropping the scope-member check, accepting a blank reason, letting pending grow, and letting a set leave pending with a narrowed scope or to anywhere each redden a named test.

Test gaps (please fix with the rebase).

  1. Removing the _pending_ratchet call from enforce_x_captures reddens nothing, in pytest or --test. The tests call the helper directly, so the shrink-only rule is not pinned through the gate itself. One test that runs enforce_x_captures against a base with a larger pending list would close it.
  2. Removing the "in a list but no longer defined" check also reddens nothing.

CI. On this head: 89 success, 11 skipped, 3 cancelled (Automated Rule Check, GPT 6.1 Review, Sensitive Change Review). GPT 6.1 has no completed verdict yet; the rebase will re-run it. Auto-merge is off.

@iamwhatever
iamwhatever enabled auto-merge (squash) October 25, 2026 06:35
@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: passed Eligible automated validation passed for the current revision labels Oct 25, 2026
@iamwhatever
iamwhatever force-pushed the fix/g2-registry-opt-out branch from d8f4093 to 55479d9 Compare October 26, 2026 02:32
@iamwhatever

Copy link
Copy Markdown
Contributor Author

@buluoray thanks. Rebased onto 6491d57d2 and amended into one commit, head 55479d94289710f9c35e2eba3d2eddc9d37597f7.

Main had gained four unclassified sets by the time of the rebase: the two you named, plus ACP_BACKENDS_MATE_WORKER_TOOL_LIMITS and ACP_BACKENDS_MATE_WORKER_PROMPT_IN_CONTEXT. Each one was classified from its code:

Set List Why
ACP_BACKENDS_MCP_WIRE_REINJECT (#23622) X_FEATURE_SETS (scope None) KAS _meta.kiro.customAgents re-sent by session/close + session/load. Its comment already cites kas/mcp-live-apply-live.jsonl and kas/mcp-live-remove-live.jsonl, both live and replayed.
ACP_BACKENDS_REPORT_USAGE (#23170) X_REGISTRATION_PENDING kiro-cli bills through _kiro.dev/metadata meteringUsage, KAS through _meta.kiro turn_completion. No replayed KAS capture keeps turn_completion (they all drop it as bookkeeping), so there is no capture to cite yet. Ceiling 11 → 12, which is exactly this one entry.
ACP_BACKENDS_MATE_WORKER_TOOL_LIMITS X_FEATURE_SETS (scope None) KAS projects excludedTools from _meta.kiro.customAgents. kas/worker-kit-live.jsonl shows a crewmate's excludedTools [shell] withholding shell. The set's comment in agent_sdk/backends_members.py now cites that capture.
ACP_BACKENDS_MATE_WORKER_PROMPT_IN_CONTEXT HARNESS_EXEMPT_SETS Crew-side: the worker prompt is carried in the [AGENT SYSTEM PROMPT] context block, so nothing on the wire is involved.

X_CAPTURE_ALLOWLIST is unchanged.

Your test gaps:

  1. _pending_ratchet not pinned through the gate. test_the_gate_holds_pending_to_the_base_ref runs enforce_x_captures("base") against a base pending list. A set moved from pending to exempt fails, and a pending entry the base lacks fails. With the call removed, this test goes red.
  2. "in a list but no longer defined". test_a_listed_set_no_longer_defined_fails plants a missing set in each of pending and exempt and runs the gate. With the check removed, this test goes red.

Local results: HARNESS_BASE_REF=origin/main python3 scripts/check_harness_parity.py is green ("60 sets classified (12 pending)"), and --test passes. The five test files named in the body pass (252 tests), plus 4,628 tests across every file that touches the backend sets. The one failure in that wider run, test_acp_backend_kas.py::TestConfigRoundTrip::test_wrong_typed_explicit_values_take_the_kiro_floor[None], also fails with this PR's working changes stashed, so it is local-environment. Local GPT and Opus review: no findings.

Could you take another look?

@iamwhatever
iamwhatever requested a review from buluoray October 26, 2026 02:32
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Oct 26, 2026
@iamwhatever

iamwhatever commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor Author

Rebased onto 33fd421f9 to pick up #24191, which fixes test_memory_files_substitution.py::TestTheInjectedImplementationAnswers::test_pruning_removes_from_the_seam_and_leaves_local_files_alone. That test failed Backend Tests shard 5 on Linux and Windows, and it also fails on the old base without this PR. New head is 052e46626817bef14bf69ac89eecd65b47c37fd0. The diff is unchanged, none of the new main commits touch this PR's files, and the gate still reads "60 sets classified (12 pending)".

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 26, 2026
@iamwhatever
iamwhatever force-pushed the fix/g2-registry-opt-out branch from 052e466 to 5164e0a Compare October 26, 2026 06:14
@iamwhatever

Copy link
Copy Markdown
Contributor Author

Rebased onto 908099fc2 to pick up #24292. On the previous head, Backend Tests shard 2 failed twice on test_pinned_staging_helper_contract.py::test_the_tree_walk_capability_probe_names_a_function_that_supports_dir_fd; #24292 is main's fix for that test. New head is 5164e0ad6321210dfbef839eac03e18fe1dfbee4. The diff is unchanged, none of the new main commits touch this PR's files, and the gate still reads "60 sets classified (12 pending)".

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Oct 26, 2026

@CrysisDeu CrysisDeu left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review at 5164e0ad: gate design is right, but it goes red on current main — please disarm auto-merge

@buluoray's six points are all resolved at this head (REPORT_USAGE pending with verb, MCP_WIRE_REINJECT as X scope None, both test gaps pinned via enforce_x_captures, GPT lane green). The three-list design (X / shrink-only pending with ceiling / exempt-with-reason) and the hard-fail on an unclassified set are what we want.

BLOCKER for landing as-is: main gained ACP_BACKENDS_AGENT_STARTED_TURNS = frozenset({ACP_BACKEND_CLAUDE}) (agent_sdk/backends_turns.py:633, #18285, after your merge base). It is in none of the three lists, so on the merged tree check_coverage reports it unclassified and test_tree_passes + fast-gate fail. git merge-tree is textually clean, which is why GitHub still says MERGEABLE — the merge queue should bounce it, but auto-merge is armed, so please disarm until fixed. Fix: rebase, add it to X_REGISTRATION_PENDING naming the _meta["_claude/origin"] end marker (no claude capture exists under test/fixtures/acp_frames/claude/), bump X_REGISTRATION_PENDING_CEILING 12 → 13, re-check for any other new sets, and update the body line "every set on main is classified".

Non-blocking (follow-up fine):

  • HARNESS_EXEMPT_SETS has no ratchet and only a blank reason is rejected (check_harness_parity.py:431,745). Once pending is shrink-only, exempt is the only escape for a new set — including one that gates permissions over a _meta field. Require the reason to start with one of the four categories (standard ACP: / routing-only: / off the wire: / Crew-side:) and assert it; consider CODEOWNERS on the gate file.
  • _base_pending (:759) only matches an annotated assignment; a plain X_REGISTRATION_PENDING = {...} later would silently turn the ratchet off. Accept ast.Assign too, or fail when the base once had the list and now parses nothing.

Posted by code-reviewer (Kiro Crew member) on behalf of @CrysisDeu. Full report on file.

@buluoray buluoray left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-reviewed at 5164e0a. This supersedes my request-changes review on d8f4093: every point from it is fixed. ACP_BACKENDS_REPORT_USAGE is in X_REGISTRATION_PENDING with its verbs, ACP_BACKENDS_MCP_WIRE_REINJECT is in X_FEATURE_SETS with scope None, the pending ceiling moved 11 -> 12 with it, and both new gate paths (shrink-only pending against the base ref, a listed set no longer defined) are pinned by tests that drive enforce_x_captures.

Blocking, and the same class as last round: main has gained another set since this branch's base. ACP_BACKENDS_AGENT_STARTED_TURNS (agent_sdk/backends_turns.py, #18285) is unclassified, so with this PR merged onto current main test/test_x_feature_captures.py fails three times with "ACP_BACKENDS_AGENT_STARTED_TURNS: unclassified ACP_BACKENDS_* set" (I ran it on a local merge of this head into origin/main). Auto-merge is on and main is strict:false, so the green checks on the old base would let this land and turn Fast Gate red on main. Please rebase onto current main and classify AGENT_STARTED_TURNS (X_FEATURE_SETS with a capture, X_REGISTRATION_PENDING with the ceiling bumped, or HARNESS_EXEMPT_SETS with a reason). CrysisDeu's review on this head reports the same set.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: passed Eligible automated validation passed for the current revision labels Oct 27, 2026
@iamwhatever
iamwhatever disabled auto-merge October 27, 2026 19:11
@iamwhatever
iamwhatever force-pushed the fix/g2-registry-opt-out branch from 5164e0a to 96b9f2e Compare October 27, 2026 19:17
@iamwhatever

Copy link
Copy Markdown
Contributor Author

@buluoray @CrysisDeu thanks, both. Auto-merge is off. Rebased onto 69507ed29 and amended into one commit. New head is 96b9f2eded434fc9df6e20e513eed3478cc9dfc8.

Point Fix
ACP_BACKENDS_AGENT_STARTED_TURNS (#18285) unclassified (both reviews) Added to X_REGISTRATION_PENDING as claude _meta["_claude/origin"] end-of-turn marker. No claude capture of an agent-started turn exists under test/fixtures/acp_frames/claude/. X_REGISTRATION_PENDING_CEILING 12 → 13, which is exactly this one entry.
Any other new set on main Re-scanned on 69507ed29: this was the only one unclassified. The gate reads "62 sets classified (13 pending)". 62 matches the number of unique ACP_BACKENDS_* definitions under src/.
Body line "every set on main is classified" Now names the base, 69507ed29, and the count, 62 sets. The other counts in the body are updated too.
_base_pending matched only ast.AnnAssign (CrysisDeu, non-blocking) It now also reads a plain ast.Assign, so rewriting the list as X_REGISTRATION_PENDING = {...} cannot switch the ratchet off. test_base_pending_is_read_back_from_the_gate_source covers the plain form.
HARNESS_EXEMPT_SETS reason prefix and CODEOWNERS (CrysisDeu, non-blocking) I'll do this as a follow-up, not in this PR. Requiring one of the four category prefixes is a new rule on all 32 current exempt entries, and CODEOWNERS is an owner call.

Local checks: HARNESS_BASE_REF=origin/main python3 scripts/check_harness_parity.py is green, and --test passes. The five test files named in the body pass (255 tests), as does the flake-pattern ratchet (62 tests). Local GPT and Opus review: no findings. Opus noted one non-blocking item: renaming a pending set and then exempting it. The body already lists that as a non-goal.

@buluoray could you re-review?

@iamwhatever
iamwhatever requested a review from buluoray October 27, 2026 19:18
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Oct 27, 2026

@buluoray buluoray left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-reviewed at 96b9f2e. This supersedes my request-changes review on 5164e0a. That round's item is fixed: ACP_BACKENDS_AGENT_STARTED_TURNS is now in X_REGISTRATION_PENDING with its verb and the ceiling moved to 13, and the base-ref reader now also catches a plain (non-annotated) rewrite of X_REGISTRATION_PENDING, with a test.

Still blocking, same class: merged onto current origin/main (f265a52), test/test_x_feature_captures.py fails three times (lines 31, 53, 88). enforce_x_captures reports:

ACP_BACKENDS_PRUNE_UNSTARTABLE_MCP [copilot]: comment cites no acp_frames/copilot/*.jsonl

So a set main carries today, for the copilot backend, has no cited live capture under test/fixtures/acp_frames/copilot/. Please rebase and either cite a replayed capture above that set, move it to X_REGISTRATION_PENDING (bumping the ceiling), or exempt it with a reason. Since main keeps adding ACP_BACKENDS_* sets, it is worth running python3 -c 'import sys; sys.path.insert(0,"scripts"); import check_harness_parity as g; g.enforce_x_captures(None)' on a fresh merge with main right before each push.

…KENDS_* set

Every ACP_BACKENDS_* set must now sit in exactly one of X_FEATURE_SETS,
X_REGISTRATION_PENDING (X set, capture not yet recorded; shrink-only,
a leaving set must join X_FEATURE_SETS) or HARNESS_EXEMPT_SETS (one-line
reason each). An unclassified new set fails with a message naming the
three choices. Registers the three sets that already pass the capture
rule. X_CAPTURE_ALLOWLIST and its ceiling are unchanged.
@iamwhatever
iamwhatever force-pushed the fix/g2-registry-opt-out branch from 96b9f2e to b5152da Compare October 27, 2026 20:24
@iamwhatever

Copy link
Copy Markdown
Contributor Author

One more change before re-review. Head 96b9f2ede went red on Harness Parity Gate and the Opus lane. #22403 had just added copilot to ACP_BACKENDS_PRUNE_UNSTARTABLE_MCP, and this PR registered that set with scope None, so copilot needed a capture.

I rebased onto ab57fba80 and scoped the entry to {deepseek, goose}. Each of those cites its own live capture. Copilot's drop is measured in Copilot CLI's log with nothing on the wire (see the set's comment in agent_sdk/backends_mcp.py), so no ACP capture can show it. _narrowed still refuses any later narrowing of this scope.

New head is b5152da4adb73f6b194503ca7ac0cfa5f2246d70. The gate reads "62 sets classified (13 pending)". Local GPT and Opus review: no findings.

@iamwhatever

Copy link
Copy Markdown
Contributor Author

@buluoray thanks. Review 5672975506 (on 96b9f2ede) is fixed at head b5152da4adb73f6b194503ca7ac0cfa5f2246d70. I scoped ACP_BACKENDS_PRUNE_UNSTARTABLE_MCP to {deepseek, goose}, and each of those cites its own replayed live capture. Copilot's drop is measured in Copilot CLI's log with nothing on the wire, so no ACP capture can show it. The details are in the comment above.

I ran your check just now on a fresh merge of this head with origin/main 21644f3f6. enforce_x_captures(None) passes with "62 sets classified (13 pending)", and test/test_x_feature_captures.py passes (30 tests). CI on this head is green apart from PR Readiness, which waits on your review. Could you take another look?

@iamwhatever
iamwhatever requested a review from buluoray October 27, 2026 21:26
@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Oct 27, 2026

@buluoray buluoray left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving b5152da. This supersedes my request-changes review on 96b9f2e.

That round's item is fixed. ACP_BACKENDS_PRUNE_UNSTARTABLE_MCP is now scoped to {deepseek, goose} (scripts/check_harness_parity.py:407), and each of those cites its own replayed capture in the set's comment in agent_sdk/backends_mcp.py. Copilot's drop is measured in Copilot CLI's log with nothing on the wire, so leaving it out of the capture scope is the honest classification, and _narrowed keeps the scope from shrinking later.

Verified on a fresh local merge of this head into origin/main 21644f3: enforce_x_captures(None) passes with "62 sets classified (13 pending)", so main has added no unclassified set; test/test_x_feature_captures.py passes (30 tests); the whole-tree harness gate exits 0. Putting the scope back to None on that merge turns four tests red (test_tree_passes, test_an_unclassified_new_set_fails, test_the_gate_holds_pending_to_the_base_ref, test_dropping_an_x_set_is_refused).

CI on this head: 93 success, 11 skipped, 2 cancelled by concurrency, 0 failures. Design, First Principles, Opus and GPT lanes report no findings.

Non-blocking: the reason copilot is outside the scope is a source comment only. If more sets end up with a partial scope, a machine-read reason per excluded backend would keep these from drifting.

@iamwhatever
iamwhatever merged commit b299d17 into main Oct 28, 2026
110 of 120 checks passed
@iamwhatever
iamwhatever deleted the fix/g2-registry-opt-out branch October 28, 2026 00:17
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Oct 28, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Harness parity X-capture rule is opt-in, so new non-standard ACP_BACKENDS_* sets skip it

3 participants