Skip to content

test: repair the set-gate ratchet and the claude capture headers - #22242

Closed
rnoack1 wants to merge 1 commit into
kirodotdev:mainfrom
rnoack1:fix/set-gate-ratchet-proven-slash-commands
Closed

rnoack1 wants to merge 1 commit into
kirodotdev:mainfrom
rnoack1:fix/set-gate-ratchet-proven-slash-commands

Conversation

@rnoack1

@rnoack1 rnoack1 commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: Main's CI stops failing on two test-side defects that no source change can
satisfy, so PRs can be merged again.

Two tests fail on every commit on main, for the same underlying reason: a gate landed and
the tree it measured moved out from under it.

  1. test_backend_cards_advertised.py::test_no_new_runtime_site_gates_a_field_backed_line_on_a_set
AssertionError: lower SET_GATE_ALLOWLIST to the new counts: {
  ('providers/acp_cli_overlay.py', 'ACP_BACKENDS_KIRO_SLASH_COMMANDS'): 1,
  ('providers/acp_model_effort.py', 'ACP_BACKENDS_KIRO_SLASH_COMMANDS'): 3,
  ('subagent_parts/model_choice.py', 'ACP_BACKENDS_KIRO_SLASH_COMMANDS'): 1}

#22056 added the ratchet at 79caee664f (01:09:41) with a baseline measured against the
tree as it then stood. #21376 landed at f161e2734a (01:10:56), 75 seconds later and with
the ratchet already in its ancestry, and removed three of the very gate sites the baseline
had just recorded. Its branch predated the ratchet, so nothing told it to re-measure.

  1. test_acp_frame_replay.py::test_a_live_header_names_the_agent_its_capture_answered_as
AssertionError: claude/compact-live.jsonl: _meta disagrees with its own initialize agentInfo
assert (None, '0.93.0') == ('@agentclientprotocol/claude-agent-acp', '0.93.0')

#21753 committed three live claude captures whose _meta omits agent_name. #22064 then
added the assertion that a live header must name the agent its own capture answered as. The
versions already agree at 0.93.0; only the name is absent.

Both baselines live as literals in test data, so neither failure can be cleared from the
source tree.

Why it matters

PR Readiness is the only required status check, it monitors ci.yml, and ci.yml runs
both tests. So every PR that reaches CI reds and cannot merge, whatever its own content.
Main has failed CI on each of 88914fd0d1, 2cf3d0d69f, 6257f698fe and 602ba3c1d4.

Not a goal

What changed (motivation → approach → change)

The ratchet. Symptom: shrunk fires with three entries. Root cause: the baseline
records gate sites #21376 deleted. Change: lower it to what the tree actually contains,
measured with the test's own _set_gates() at 88914fd0d1.

Entry (ACP_BACKENDS_KIRO_SLASH_COMMANDS) allowlisted found action
providers/acp_cli_overlay.py 1 0 entry removed
providers/acp_model_effort.py 3 2 lowered to 2
subagent_parts/model_choice.py 1 0 entry removed

Removed rather than set to 0, because an entry means "this site may gate on the set"; a site
that no longer does has nothing to allow, and a 0 would silently re-permit the next one.

The capture headers. Symptom: a live header's agent_name is None. Root cause: three
captures were committed without the field. Change: add it to each of the three, read off
that capture's OWN initialize answer, which is what the test's docstring prescribes
("read off the wire, not typed") and what claude/session.jsonl already does. Placed
immediately after agent_version to match that file's field order. All three answer as
@agentclientprotocol/claude-agent-acp. The edit is a textual insert of one field into the
header line, so no other byte of any capture moves, asserted per file against the parsed
object before and after.

Backwards compatibility

Compatible: test and fixture data only, and both changes tighten rather than relax. No
runtime code, config, wire field or public function is touched.

The diff removes two keys from SET_GATE_ALLOWLIST, so its readers are listed:

  • Reader: test/test_backend_cards_advertised.py:86 -- release (CI) -- the grown assertion reads the allowlist with .get(k, 0), so a removed key becomes a 0 ceiling and a re-introduced gate site at that path now fails. Proven below.
  • Reader: test/test_backend_cards_advertised.py:91 -- release (CI) -- the shrunk assertion, the one that was failing; it passes once the baseline matches the tree.
  • Reader: none elsewhere -- SET_GATE_ALLOWLIST appears only in that one file, 4 occurrences, all of them its definition and those two assertions.

The capture change adds a key and removes none. Its readers are
test_acp_frame_replay.py::test_a_live_header_names_the_agent_its_capture_answered_as and
the pin gate in test_real_adapter_fixture_pins.py, which the docstring notes keys on these
two fields; both pass (8 passed in the pin gate).

Tests

No new test. Both changes correct baselines inside existing tests, so those tests ARE the
coverage, and each was exercised in both directions.

Ratchet:

  • At 88914fd0d1 unmodified the file reds: 1 failed, 1 passed. With this change: 2 passed.
  • Tightening verified, not assumed: appending a synthetic
    backend in ACP_BACKENDS_KIRO_SLASH_COMMANDS site to providers/acp_cli_overlay.py now
    fails the grown assertion, naming {('providers/acp_cli_overlay.py', …): 1} -- the entry
    this PR removes. Before the change that same site was permitted. Probe reverted.

Capture headers:

  • At 88914fd0d1 unmodified the suite reds on the assertion quoted above. With this change:
    55 passed.
  • Staleness detection verified still live: bumping one header's agent_version to 0.99.0
    while its wire still answers 0.93.0 reds the same assertion. So the missing field is
    filled without blinding the check test(acp): red the real-adapter lane when a pin bump leaves live captures behind #22064 added. Probe reverted.
  • test_real_adapter_fixture_pins.py: 8 passed.

Neighbouring suites run together: test_ratchet_scope.py, test_main_ratchet_audit.py,
test_harness_parity.py, test_x_feature_captures.py, test_acp_frame_replay.py,
test_backend_cards_advertised.py, test_real_adapter_fixture_pins.py -- 211 passed, 3
failed, the 3 being the pre-existing x-capture failures named under "Not a goal". They fail
identically at 88914fd0d1 with this change reverted.

Manual verification

N/A -- both changes are test baselines, and both assertion directions are exercised above.

Screenshots / video

N/A -- no user-visible frontend surface is touched.

Related Issues

Corrects the baseline added by #22056 after #21376 removed the sites it recorded, and the
headers of the captures added by #21753 against the assertion added by #22064. Unblocks
#22186, which reds on these two tests rather than on its own change.

Pattern harvest

Rule candidate: review-prompt
Pattern: a gate whose baseline is a literal in test code or test data goes stale the moment a
concurrently-approved PR changes what it measures, and the staleness cannot be cleared from
the source tree. Both failures here are that shape, 75 seconds apart in one case. Two signals
a reviewer can act on: a ratchet or pin landing while PRs touching the measured sites are
already approved, and a "lower the baseline" style assertion with no instruction to
re-measure. Prefer deriving the baseline from the tree at a pinned ref, or land the gate
behind the PRs it measures.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

Contribution License Agreement

I confirm this contribution is made under the terms of the project's contribution licence.

@rnoack1
rnoack1 requested review from a team and KiroCrewCaptain as code owners October 19, 2026 17:23
@rnoack1
rnoack1 requested a review from mbriones98 October 19, 2026 17:24
@github-actions github-actions Bot added fork Pull request from a fork (external contributor) readiness: checking Automated validation is still running labels Oct 19, 2026
kirodotdev#22056's ratchet recorded sites kirodotdev#21376 removed 75s later. Three claude
captures from kirodotdev#21753 omit the agent_name kirodotdev#22064 then began requiring.
@rnoack1
rnoack1 force-pushed the fix/set-gate-ratchet-proven-slash-commands branch from 4dfd1f2 to 60d936f Compare October 19, 2026 17:30
@rnoack1 rnoack1 changed the title test: lower the set-gate ratchet to the counts #21376 left behind test: repair the set-gate ratchet and the claude capture headers Oct 19, 2026
@github-actions

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5, fork) — ⏭️ skipped

Revision 60d936f210b2185b5e0f17692576401b7004459b touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Advisory — does not block merge.

@github-actions

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5, fork) — ⏭️ skipped

Revision 60d936f210b2185b5e0f17692576401b7004459b ships no reviewable capability (docs, tests or generated files only), so there is nothing to inventory. Advisory — does not block merge.

@github-actions

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5, fork) — 🟡 CONCERNS

Design-level review of 60d936f210b2185b5e0f17692576401b7004459b via the fork AI-review pipeline — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: CONCERNS

This PR is already redundant: base 3e39e7a9 (#22178) already has the lowered SET_GATE_ALLOWLIST and all three agent_name headers, so it now adds nothing.

Watch

  • In the checked-out base, test/test_backend_cards_advertised.py:39-45 already holds the exact post-patch allowlist: acp_model_effort.py is at 2, and the acp_cli_overlay.py and model_choice.py KIRO_SLASH_COMMANDS rows are gone. All four claude/*.jsonl captures already carry agent_name. The diff's - lines describe a tree that main has moved past, so the stated Goal ("Main's CI stops failing…") is already met. Merging would rebase to an empty or conflicting diff, which is the same race between a literal baseline and concurrent PRs that the description warns about.
    Clears when: the branch is rebased onto current main and shows a non-empty diff that still clears a red. If not, close it as superseded by test: clear main's four reds left by #21753, #21598 and later backend edits #22178.

[DESIGN-REVIEWED] 60d936f

Description read

[DESCRIPTION-READ] b2181d0c207919105d04b15df7e7d6b77db16cd9033c7e2fd828311bf4ccf22c

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

Copy link
Copy Markdown
Contributor

GPT 6.1 Review (fork) — ✅ no blocking findings

Reviewed 60d936f210b2185b5e0f17692576401b7004459b via the fork AI-review pipeline; updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 60d936f

@github-actions

Copy link
Copy Markdown
Contributor

Opus 5.5 Review (fork) — ✅ no blocking findings

Reviewed 60d936f210b2185b5e0f17692576401b7004459b via the fork AI-review pipeline; updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 60d936f

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Oct 19, 2026

@mbriones98 mbriones98 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Checks green and readiness passed at 60d936f; no behavior change.

@iamwhatever

Copy link
Copy Markdown
Contributor

HOLD at 60d936f.

The Design Review's CONCERNS is still unanswered, and it is correct. #22178 (3e39e7a) already landed the same SET_GATE_ALLOWLIST lowering and the agent_name headers on the three claude captures. Merging this branch into current main (0104207) gives a tree identical to main, so the PR changes nothing.

To clear this: close it as superseded by #22178. Or, if a red still remains, rebase onto main, show the non-empty diff that clears it, and reply to the Design Review. The change itself is sound test data, and CI is green on this head.

— crew-pr-reviewer (r113)

@rnoack1

rnoack1 commented Oct 21, 2026

Copy link
Copy Markdown
Contributor Author

Closing as superseded by #22178, which merged on 2026-10-19 with the same changes:

  • test/test_backend_cards_advertised.py: main's SET_GATE_ALLOWLIST already has providers/acp_model_effort.py at 2, and the acp_cli_overlay.py and subagent_parts/model_choice.py ACP_BACKENDS_KIRO_SLASH_COMMANDS rows are already gone.
  • test/fixtures/acp_frames/claude/{compact-live,mcp-stdio-mount-live,saved-spec-projection-live}.jsonl: each header line on main already carries "agent_name": "@agentclientprotocol/claude-agent-acp".

Rebasing head 60d936f onto main would leave an empty diff, so there's nothing left to land.

@rnoack1 rnoack1 closed this Oct 21, 2026
@rnoack1
rnoack1 deleted the fix/set-gate-ratchet-proven-slash-commands branch October 21, 2026 15:02
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Oct 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

fork Pull request from a fork (external contributor)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants