Skip to content

test: clear main's four reds left by #21753, #21598 and later backend edits - #22178

Merged
bolichen97 merged 1 commit into
mainfrom
refactor/linecap-fx12-harness-allowlist
Oct 19, 2026
Merged

bolichen97 merged 1 commit into
mainfrom
refactor/linecap-fx12-harness-allowlist

Conversation

@bolichen97

@bolichen97 bolichen97 commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: Turn main green again with one PR that clears all four of its current reds. Each red is a stale entry that a merged PR left behind. Two of them are hidden today: Await Fast Gate fails first, so main's backend shards never run them.

Red on main Cause Fix
Harness Parity Gate (Fast Gate), ACP_BACKENDS_MEMBER_CAPABILITIES [claude]: allowlisted but now proven; delete the entry. Main push run 39266672407 on 8e686e5, and again on later heads #21753 (c987572) committed a live claude saved-member-spec capture that a test replays, but scripts/check_harness_parity.py still lists ("ACP_BACKENDS_MEMBER_CAPABILITIES", "claude") in the x-capture allowlist delete that one row
test/test_name_grant_facade.py::test_the_patch_scan_reads_every_spelling_and_nothing_else[helper-binding] (assert set() == {'D'}) #21598 (79663b7) split llm_helpers.py. Its facade now serves name_grant from llm_helpers_parts/tool_approval.py through its module-level if TYPE_CHECKING: manifest. The test's spelling scan read only top-level imports, so it stopped seeing the llm_helpers.name_grant spelling. test/test_llm_helpers_deny_notice.py:240 really patches through it also read a module-level if TYPE_CHECKING: block in the scan
test/test_acp_frame_replay.py::test_a_live_header_names_the_agent_its_capture_answered_as (claude/compact-live.jsonl: _meta disagrees with its own initialize agentInfo, (None, '0.93.0')) #21753's three claude captures carry agent_version but no agent_name in their _meta header add "agent_name": "@agentclientprotocol/claude-agent-acp", the name each capture's own initialize answer gives, to the three headers
test/test_backend_cards_advertised.py::test_no_new_runtime_site_gates_a_field_backed_line_on_a_set, shrink half (lower SET_GATE_ALLOWLIST ...) after recent backend edits, three ACP_BACKENDS_KIRO_SLASH_COMMANDS rows overstate their files' membership-test counts: providers/acp_cli_overlay.py 1 (now 0), providers/acp_model_effort.py 3 (now 2), subagent_parts/model_choice.py 1 (now 0) delete the two zero rows and lower the third to 2; the grow half still passes

Why it matters

Fast Gate runs on every PR. While it is red, Await Fast Gate fails, the heavy matrix is skipped, and the Coverage Gate fails closed. So no PR rebased onto main can go green until both entries are fixed.

Not a goal

What changed

  • scripts/check_harness_parity.py: delete the ("ACP_BACKENDS_MEMBER_CAPABILITIES", "claude") x-capture allowlist row. This is a 1-line list edit, and it is exactly what the gate's own error message asks for.
  • test/fixtures/acp_frames/claude/{compact,mcp-stdio-mount,saved-spec-projection}-live.jsonl: add "agent_name": "@agentclientprotocol/claude-agent-acp" to the _meta header line, after agent_version, which is the key order every other backend's live capture uses. The frames are unchanged.
  • test/test_backend_cards_advertised.py: SET_GATE_ALLOWLIST loses the acp_cli_overlay.py and model_choice.py ACP_BACKENDS_KIRO_SLASH_COMMANDS rows (their counts are now 0), and acp_model_effort.py goes from 3 to 2. The real counts come from the test's own _set_gates() on main; note that its shrink message prints the old values, not the new counts.
  • test/test_name_grant_facade.py: _spellings() now also counts a module-level if TYPE_CHECKING: block as module level. That block is where a split module's facade lists the names it serves from an owner. The docstring says so. The llm_helpers.name_grant spelling comes back, so the corpus patch scan sees the real seam in test_llm_helpers_deny_notice.py again. The extra spellings can only add patched names, never hide one.

Backwards compatibility

Compatible. One CI list row, one test allowlist, three fixture header lines and one test helper change; no runtime code, path, key or field changes.

Tests

No new tests. The two reds are the proof:

Check main 6257f69 This head
python3 scripts/check_harness_parity.py rc 1 (allowlisted but now proven) rc 0 (every X-feature member cites a replayed live capture (17 + 3 allowlisted))
test/test_name_grant_facade.py 1 failed ([helper-binding]) passed
test/test_acp_frame_replay.py 1 failed (live-header check) passed
test/test_backend_cards_advertised.py 1 failed (set-gate shrink half) passed
test_backend_cards_advertised.py, test_acp_frame_replay.py, test_real_adapter_fixture_pins.py, test_x_feature_captures.py, test_harness_parity.py, test_name_grant_facade.py, test_llm_helpers_deny_notice.py 4 red 207 passed

flake8 and black are clean on the changed test files.

Manual verification

N/A: every change is checked by the gate or test that was red.

Related Issues

Part of #20192

Pattern harvest

The second red is a semantic merge race between two refactor PRs. #22069 pinned a spelling that #21598 had just moved behind a facade manifest. Facade-aware spelling discovery is now in this one test; other *_facade.py spelling scans that read only top-level imports could miss served names the same way. That is a follow-up, not part of this fix.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

Contribution License Agreement

Contributions are accepted under the project's Apache-2.0 license; see the Licensing section of CONTRIBUTING.md.

@bolichen97
bolichen97 requested review from a team and KiroCrewCaptain as code owners October 19, 2026 14:23
@bolichen97
bolichen97 requested a review from buluoray October 19, 2026 14:23
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Oct 19, 2026
@github-actions

github-actions Bot commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision 01f8ca6dbece082638d3cb7938ba9254eb2d7079 touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Advisory — does not block merge.

@github-actions

github-actions Bot commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of 01f8ca6dbece082638d3cb7938ba9254eb2d7079 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Check that llm_helpers.name_grant resolves at runtime through the facade's __getattr__, not only under TYPE_CHECKING; otherwise the restored spelling is static-only.

[DESIGN-REVIEWED] 01f8ca6

Description read

[DESCRIPTION-READ] 41b7e989fd7860b9314eb98c9fc80fee121ac97ab34d357510f56556f9115970

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ✅ PASS

Premise-level review of 01f8ca6dbece082638d3cb7938ba9254eb2d7079 — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: PASS

Check that the TYPE_CHECKING widening in _spellings() only adds spellings, and does not stop the scan from catching a patch it caught before.

What this change ships

Inventory (4 items) — 4 justified

Intent: get main's CI green again by clearing four stale pins left behind by merged PRs. This is a FIX.

  1. The Harness Parity Gate no longer allowlists claude's saved-member-spec capture, which a replayed live capture now proves. — justified
  2. The three claude live captures' _meta headers now name the agent their own initialize answer reports. — justified
  3. The set-gate allowlist drops two zero-count rows and lowers one count from 3 to 2. — justified
  4. The name-grant spelling scan now reads a module-level if TYPE_CHECKING: facade manifest, so the llm_helpers.name_grant spelling counts again. — justified

[FIRST-PRINCIPLES-REVIEWED] 01f8ca6

Description read

[DESCRIPTION-READ] a7407b257ff1522ad5154edf56c49afe3214532b29b1d9d812081456f76d0491

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed 01f8ca6dbece082638d3cb7938ba9254eb2d7079 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 01f8ca6

Verdict parsed from the review's SHA-scoped output markers for commit 01f8ca6dbece082638d3cb7938ba9254eb2d7079.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 01f8ca6dbece082638d3cb7938ba9254eb2d7079: <one-sentence reason>

@github-actions

github-actions Bot commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of 01f8ca6dbece082638d3cb7938ba9254eb2d7079 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 01f8ca6

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 01f8ca6dbece082638d3cb7938ba9254eb2d7079: <one-sentence reason>

@bolichen97
bolichen97 enabled auto-merge (squash) October 19, 2026 14:35
@bolichen97
bolichen97 force-pushed the refactor/linecap-fx12-harness-allowlist branch from 780b537 to 8a99388 Compare October 19, 2026 14:40
@bolichen97 bolichen97 changed the title ci(harness): drop the claude member-capabilities x-capture allowlist row #21753 proved test: clear main's two reds (stale claude x-capture allowlist row, name_grant patch-scan spelling) Oct 19, 2026
… edits

#21753 committed a live claude saved-member-spec capture that a test
replays, so the x-capture gate fails main with "allowlisted but now
proven; delete the entry". Delete that row from
scripts/check_harness_parity.py.

The same commit's three claude captures carry agent_version but no
agent_name in their _meta header, so test_acp_frame_replay's live-header
check fails against their own initialize agentInfo. Add
"agent_name": "@agentclientprotocol/claude-agent-acp" to each header.

test_backend_cards_advertised's SET_GATE_ALLOWLIST overstates three
counts after recent backend edits (its shrink half fails): drop the
acp_cli_overlay.py and model_choice.py KIRO_SLASH_COMMANDS rows (now 0)
and lower acp_model_effort.py to 2.

#21598 split llm_helpers.py, and its facade now serves name_grant from
llm_helpers_parts/tool_approval.py through its TYPE_CHECKING manifest.
test_name_grant_facade's spelling scan read only module-level imports, so
it lost the llm_helpers.name_grant spelling (which
test_llm_helpers_deny_notice.py patches) and its helper-binding case
fails. Count a module-level if TYPE_CHECKING: block as module level.

Part of #20192
buluoray
buluoray previously approved these changes Oct 19, 2026

@buluoray buluoray left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving at 8a99388. Applied onto current main and verified.

Harness Parity Gate: with HARNESS_BASE_REF=origin/main, scripts/check_harness_parity.py exits 1 on main (ACP_BACKENDS_MEMBER_CAPABILITIES [claude]: allowlisted but now proven) and exits 0 with this change; the --test self-test passes. The removed row is genuinely proven: backends_members.py cites test/fixtures/acp_frames/claude/saved-spec-projection-live.jsonl, the capture and its .expected.json are committed, and test/test_claude_live_captures.py replays it.

name_grant patch scan: test_the_patch_scan_reads_every_spelling_and_nothing_else[helper-binding] is red on main because the llm_helpers split moved its name_grant import under if TYPE_CHECKING:, while llm_helpers.name_grant still resolves to the facade at runtime. Folding top-level TYPE_CHECKING blocks into the module-level walk restores that spelling. test/test_harness_parity.py plus test/test_name_grant_facade.py: 94 passed with -n0 (main: 2 failed).

Not covered here, and still red on main: test_acp_frame_replay.py::test_a_live_header_names_the_agent_its_capture_answered_as. The three claude live captures from #21753 (compact-live, mcp-stdio-mount-live, saved-spec-projection-live) carry no _meta.agent_name, which #22064 now requires to match the initialize agentInfo name (@agentclientprotocol/claude-agent-acp). That is the Backend Tests shard 3 failure on this head, so this cannot go green until those headers are fixed; adding it here would let this PR clear all three reds at once. Small follow-up: the TYPE_CHECKING fold matches a bare TYPE_CHECKING name only, not typing.TYPE_CHECKING.

@bolichen97 bolichen97 changed the title test: clear main's two reds (stale claude x-capture allowlist row, name_grant patch-scan spelling) test: clear main's four reds left by #21753, #21598 and later backend edits Oct 19, 2026
@bolichen97
bolichen97 force-pushed the refactor/linecap-fx12-harness-allowlist branch from 8a99388 to 01f8ca6 Compare October 19, 2026 15:18

@buluoray buluoray left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving at 01f8ca6. This supersedes my approval at 8a99388.

Verified each of the four main reds on current origin/main (6257f69) and confirmed this head (origin/main plus one commit) clears all of them. With -n0 on test/test_harness_parity.py, test/test_name_grant_facade.py, test/test_acp_frame_replay.py, test/test_claude_live_captures.py and test/test_backend_cards_advertised.py, plus HARNESS_BASE_REF=origin/main python3 scripts/check_harness_parity.py: main has 4 failed / 151 passed and the gate reports ACP_BACKENDS_MEMBER_CAPABILITIES [claude] as allowlisted but now proven; this head has 155 passed and the gate passes (17 cited plus 3 allowlisted).

  1. Harness parity: deletes the (ACP_BACKENDS_MEMBER_CAPABILITIES, claude) row; the claude live captures prove it, so the allowlist shrinks.
  2. name_grant facade: the scan now reads the body of a module-level if TYPE_CHECKING: block (not its else). llm_helpers.py lists name_grant there, and at runtime llm_helpers.name_grant is kiro_crew.name_grant, so the spelling is real.
  3. Capture headers: in compact-live, mcp-stdio-mount-live and saved-spec-projection-live, _meta.agent_name @agentclientprotocol/claude-agent-acp and agent_version 0.93.0 equal the agentInfo name and version in each file own initialize result, so the header states what the capture answered as. A wrong agent_name reddens test_a_live_header_names_the_agent_its_capture_answered_as.
  4. SET_GATE_ALLOWLIST: new counts equal the runtime gates in source (acp_cli_overlay.py 0, model_choice.py 0, acp_model_effort.py 2 at lines 172 and 611; acp.py and acp_turns.py still 1). Every edit lowers or removes a row; raising acp_model_effort.py back to 3 reddens test_no_new_runtime_site_gates_a_field_backed_line_on_a_set.

CI at this sha when checked: 54 success, 10 skipped, 0 failed, 33 still running. Design and First Principles PASS; Opus and GPT no blocking findings. Auto-merge is on, so this lands once the required checks are green.

Small follow-up from before: the TYPE_CHECKING fold matches a bare TYPE_CHECKING name only, not typing.TYPE_CHECKING.

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Oct 19, 2026
@bolichen97
bolichen97 merged commit 3e39e7a into main Oct 19, 2026
152 of 154 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants