Skip to content

feat(testing): let the fake ACP backend answer from a reply script - #37510

Merged
javenciu merged 1 commit into
mainfrom
feat/scripted-test-backend
Nov 15, 2026
Merged

javenciu merged 1 commit into
mainfrom
feat/scripted-test-backend

Conversation

@javenciu

@javenciu javenciu commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: someone testing an agent spec, a hook, a cron script or an app can make the model say chosen things in a chosen order, with no tokens and the same result every run.

Kiro Crew already ships a stand-in model, kiro_crew.testing.fake_acp_backend, in the runtime wheel. It speaks ACP over stdio, and the end-to-end suite points KIROCREW_KIRO_BIN at it. But it answers every prompt with one canned line (pong from the fake ACP backend) unless the prompt carries one of its own markers ([[TOOL]], [[ERROR]], ...). The text a user's test sends is written by Kiro Crew, not by the test, so a test cannot place those markers, and it cannot get a reply of its own. The stand-in is also not mentioned anywhere in src/kiro_crew/docs.

Why it matters

Hooks, guardrails, cron scripts and app tools react to what the model says. Testing them today needs a real model: the run costs tokens, and its replies change from one run to the next.

Not a goal

  • A new backend in the backend registry. The fake stays reachable only through KIROCREW_KIRO_BIN, as its module docstring requires.
  • Running the scripted tool calls. They are shown, not executed.
  • Changing any existing marker or the canned reply.

What changed (motivation → approach → change)

Approach: the fake reads an optional reply script named by KIROCREW_FAKE_ACP_SCRIPT.

  • src/kiro_crew/testing/scripted_replies.py (new): parses the script. It is one JSON object with a replies list. An entry with when answers every prompt containing that text (plain substring). Entries without when answer once each, in file order, per session. Each entry gives text, tools (title, kind, input, status), stop (end_turn, max_tokens, refusal) or error. The file is read through kiro_crew.hooks.safe_read_file, and a refusal names only the file and the error class. A bad file or entry (unreadable, not JSON, nested past the recursion limit, a rule broken, or a tool call or error message whose serialized frame passes 1 Mi characters, which cannot be split) is dropped with a warning on stderr, once, through safe_terminal_line. The turn then keeps the answer it gets today.
  • src/kiro_crew/testing/fake_acp_backend.py: the prompt handler asks the script first, right after the session-naming check (so titles never use up an entry). It imports the script module only when the variable is set, so the fake stays stdlib-only at import time. A scripted text is sent in chunks of 256 Ki characters, so a long reply stays under the 10 MiB line limit the ACP transports read (a character grows to at most 12 bytes once JSON-escaped).
  • src/kiro_crew/docs/scripted-test-model.md (new) + a row in docs/index.md: the file format, running a gateway on it with a throwaway home (the stand-in is reached only through KIROCREW_KIRO_BIN), and using it from pytest with spawn_feature_gateway. It says which interpreter must run the stand-in.
  • src/kiro_crew/testing/__init__.py: lists the new module.

Backwards compatibility

With KIROCREW_FAKE_ACP_SCRIPT unset, _script() returns None and every frame is byte-for-byte what main sends. test_without_a_script_a_plain_prompt_keeps_the_canned_reply passes on main and on this branch, and the 55 existing tests in test/test_fake_acp_backend.py pass unchanged. Outside kiro_crew.testing only the docs change.

Tests

  • Before: with main's fake_acp_backend.py, 24 of the 26 new tests fail. The script is ignored, so the replies are the canned line. The 2 that pass are the no-script baseline and the pure refused-read check.
  • After: 26 passed. Eight run the real module as a child process over stdio (python -m kiro_crew.testing.fake_acp_backend acp), so no kiro-cli is needed. Two send an 11 MiB reply and check that every frame stays under the transports' 10 MiB line limit and the chunks join back whole; four send an oversized tool call or error and check it is dropped with one warning and the turn still ends. The hostile-file-name test no longer names the exception class, which differs by platform; a second test forces OSError(EINVAL), as Windows raises.
  • 29 test files that import the fake, the harness or the packaged docs, plus the flake and wait ratchets: 651 passed, 59 skipped.
  • flake8, isort and mypy pass on the changed files. The black, comment-history, sync-IO and flake-ratchet gates pass.
  • Reproduce: python -m pytest test/test_fake_acp_backend_script.py -n0
  • Proof: PROVEN test/test_fake_acp_backend_script.py::test_ordered_replies_answer_one_turn_each_then_the_canned_reply_returns test/test_fake_acp_backend_script.py::test_the_backend_process_answers_from_the_script_over_stdio test/test_fake_acp_backend_script.py::test_entries_that_break_a_rule_are_dropped_and_the_rest_kept (head 0713c2b)

Manual verification

Ran the module as a child process with a one-entry script. Fed initialize, session/new and one session/prompt on stdin. The reply chunk was the scripted text, and the result was {"stopReason": "end_turn"}.

Sibling sweep

Sibling: src/kiro_crew/testing/harness.py -- not affected -- harness_environment layers os.environ, so the variable reaches a harness gateway as is
Sibling: src/kiro_crew/testing/workflow_memory_scenario.py -- not affected -- its [[WF_E2E:...]] path is checked before the script and keeps priority

Pattern harvest

Rule candidate: none
Pattern: a test double that answers only to its own in-band markers cannot be driven by text the product writes. Give it an out-of-band input instead.

Related Issues

None found (searched open and closed PRs and issues for scripted, mock and deterministic test backends).

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

Contribution License Agreement

@javenciu
javenciu requested review from a team and KiroCrewCaptain as code owners November 15, 2026 04:51
@javenciu
javenciu enabled auto-merge (squash) November 15, 2026 04:51
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Nov 15, 2026
@github-actions

github-actions Bot commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision 0713c2b1873d0d6d05005bc1af79bb9420652795 touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions

github-actions Bot commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of 0713c2b1873d0d6d05005bc1af79bb9420652795 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

Nothing blocks: this is an opt-in test aid, reachable only when KIROCREW_KIRO_BIN points at the stand-in, and with the variable unset every frame is byte-for-byte unchanged.

Notes

  • The documented gateway path (where the env var has to survive scrub_agent_subprocess_env and the sandbox has to let the stand-in read the script file) is described in the docs but only the direct stdio child is tested. One spawn_feature_gateway test would prove the recipe the docs promise.

[DESIGN-REVIEWED] 0713c2b

Description read

[DESCRIPTION-READ] 4fba6f30927e70e2d521d221e61864d35b86289aab913f3bc4809039e58c80a7

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ✅ PASS

Premise-level review of 0713c2b1873d0d6d05005bc1af79bb9420652795 — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

First-Principles-Verdict: PASS

Nothing blocks. The reply script is off unless KIROCREW_FAKE_ACP_SCRIPT is set, it adds no backend to the registry, and no other code in the repo already lets a test choose the fake model's replies.

What this change ships

Inventory (7 items) — 7 justified

Intent: let someone testing a spec, hook, cron script or app get chosen model replies, in a chosen order, with no tokens and the same result every run. This is an ADDITION.

  1. A test can name a reply-script file through KIROCREW_FAKE_ACP_SCRIPT, and the fake model answers from it — justified
  2. Entries without when answer in file order, and each session keeps its own place — justified
  3. when entries answer every prompt that contains their text, which stops background prompts from using up the sequence — justified
  4. A script can show tool calls, set how a turn stops, or fail a turn — justified
  5. A broken file or entry is dropped with one warning on stderr, and the turn keeps the built-in reply — justified
  6. Long replies are split into chunks, and oversized tool calls or errors are dropped to stay under the 10 MiB transport line limit — justified
  7. A new packaged user doc, scripted-test-model.md, with a row in the docs index — justified

[FIRST-PRINCIPLES-REVIEWED] 0713c2b

Description read

[DESCRIPTION-READ] b79a53fec9e1444e9acc06e277e2b1628e6074fae54171e9d31942c27cb97dc8

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed 0713c2b1873d0d6d05005bc1af79bb9420652795 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 0713c2b

Verdict parsed from the review's SHA-scoped output markers for commit 0713c2b1873d0d6d05005bc1af79bb9420652795.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 0713c2b1873d0d6d05005bc1af79bb9420652795 false-positive: <reason, 20 to 500 characters>

@github-actions

github-actions Bot commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of 0713c2b1873d0d6d05005bc1af79bb9420652795 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 0713c2b

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 0713c2b1873d0d6d05005bc1af79bb9420652795 false-positive: <reason, 20 to 500 characters>

@KiroCrewCaptain

KiroCrewCaptain commented Nov 15, 2026 •

Copy link
Copy Markdown
Collaborator

Code change approver recommendation: A person should decide

Why: the decider estimates a 10% chance that this change introduces a defect, within the 12% limit for approving on its own, but it changes 815 lines, more than the 400 it may approve on its own.

An opt-in reply script for the packaged fake ACP backend, enabled only by KIROCREW_FAKE_ACP_SCRIPT and reachable only through KIROCREW_KIRO_BIN. With the variable unset, the code path is unchanged from main. The script is checked after the WF_E2E and naming branches, as described. The file is read through safe_read_file, and every exception class it raises is caught. Frame sizing stays under the 10 MiB transport limit (json.dumps escapes to ASCII, so 256Ki chars per chunk is at most about 3 MiB). The env var survives the agent-subprocess scrub. 26 new tests cover the behaviour, including real stdio child runs, and new-test stability passed on both OSes. Only doc drift and minor nits remain; none block.

An automated recommendation, not a review: a maintainer still decides. It approves on its own only when the estimated chance of a defect is at most 12% and the change has at most 400 changed lines. Decider v3c (217fe607) · commit 0713c2b187 · 815 changed lines. If this recommendation is wrong, reply here: replies are how the approver is evaluated.

@javenciu
javenciu force-pushed the feat/scripted-test-backend branch from b0cd662 to 4d5c6c9 Compare November 15, 2026 05:44
@javenciu

Copy link
Copy Markdown
Contributor Author

Large scripted replies are discarded by the ACP reader (the one line finding, fake_acp_backend.py). Fixed in 4d5c6c9. A scripted text is now sent as several agent_message_chunk updates of at most 256 Ki characters each. A character is at most 12 bytes once JSON-escaped, so a frame stays near 3 MiB, well under the 10 MiB line limit that transport_framing._STDOUT_BUFFER_LIMIT sets. A reply under 256 Ki characters is still one chunk, so every existing frame is unchanged. test_a_reply_larger_than_the_frame_limit_arrives_whole_in_several_frames (an 11 MiB ASCII reply and a 1 Mi-character emoji reply that escapes to 12 MiB) runs the real module over stdio, checks that no output line reaches the limit, and checks that the chunks join back to the whole reply. Both cases fail on the previous head with longest lines of 11,534,510 and 12,583,086 bytes.

A second warning channel that nothing reads (first-principles note). Removed in the same commit. result_meta, MAX_RESULT_WARNINGS and the _meta.kiroCrewScript.warnings copy on prompt results are gone, along with the matching case in the ImportError path of the backend. Nothing under src/kiro_crew/acp/ reads that field, and the stderr line already tells the user. The tests that read the field now read the stand-in's stderr (test_a_missing_script_keeps_the_canned_reply_and_warns_by_file_name, test_a_script_of_the_wrong_shape_is_ignored, test_a_file_nested_past_the_recursion_limit_keeps_the_canned_reply, test_entries_that_break_a_rule_are_dropped_and_the_rest_kept); on the previous head they fail because the result still carries _meta. docs/scripted-test-model.md no longer mentions the field and says a long reply arrives whole in chunks. The prompt result is now {"stopReason": ...} in every case.

The branch is rebased onto the current origin/main and is still one commit.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Nov 15, 2026
@javenciu
javenciu force-pushed the feat/scripted-test-backend branch from 4d5c6c9 to 6b7192e Compare November 15, 2026 07:15
@javenciu

Copy link
Copy Markdown
Contributor Author

Hostile-file-name test asserts a POSIX exception class (Opus, test/test_fake_acp_backend_script.py, span 9ad77cfd8c31). Fixed in 6b7192e. The test now asserts the part that holds on every platform: one stderr line, no ESC, and the text could not be read (, without naming the exception class. A new test, test_a_file_name_with_controls_is_one_plain_line_whichever_error_the_platform_raises, forces the guarded reader to raise OSError(EINVAL), which is what Windows raises for such a name, and checks the line ends (OSError), is one plain line and does not quote the directory. The previous head's assertion fails under that forced error.

Scripted tool frames exceed the ACP transport limit (GPT, fake_acp_backend.py, span cb2f25ffb334). Fixed in 6b7192e, by checking at load time instead of splitting: a tool call (title, kind and input) or an error message whose serialized form passes 1 Mi characters is dropped with one warning on stderr naming the entry, and the rest of the entry and the turn are kept (the reply text still arrives, in chunks; a dropped error entry falls back to the built-in answer). Tool and error frames are single protocol messages and cannot be chunked like text. The cap sits far under the 10 MiB line limit, and a call just under it is still sent. test_a_tool_call_past_the_transport_limit_never_reaches_the_wire runs the real module over stdio with an 11 MiB tool input and fails on the previous head (longest line 11,534,576 bytes, not under the limit); test_a_tool_call_or_error_larger_than_one_frame_is_dropped_with_a_warning[tool-input|tool-title|error] fail there too because no warning is written. test_a_tool_call_just_under_the_frame_cap_is_still_sent pins the unchanged case. docs/scripted-test-model.md states the cap.

The branch is rebased onto the current origin/main and is still one commit.

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Nov 15, 2026
KIROCREW_FAKE_ACP_SCRIPT names a JSON file of replies (text, tool calls, stop reason or an error), matched by substring or taken in order per session, so agent specs, hooks, cron scripts and apps can be tested with no tokens and the same result every run. Unset, the fake behaves exactly as before; an unreadable or malformed script is dropped with a warning and the built-in reply is kept. Documented in docs/scripted-test-model.md.
@javenciu
javenciu force-pushed the feat/scripted-test-backend branch from 6b7192e to 0713c2b Compare November 15, 2026 09:24
@javenciu

Copy link
Copy Markdown
Contributor Author

Rebased onto 09b0ddc as 0713c2b with no change to this PR's code. The failures in backend shard 6 (Linux and Windows) were test/metrics/test_kiro_export_destination.py, which failed on main itself until #37545; the rebase picks that fix up. The shard-7 failure was the timing assertion in test_denied_commands_security_cost.py, which this PR does not touch. On this head test/test_fake_acp_backend_script.py and test/test_fake_acp_backend.py give 81 passed.

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Nov 15, 2026

@KiroCrewCaptain KiroCrewCaptain left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving. On this head the review checked four things. With KIROCREW_FAKE_ACP_SCRIPT unset, the stand-in sends the same frames as before. The script file is read through hooks.safe_read_file, so a script path that points into a credentials folder is refused, and every read error is caught with a short warning. Text chunks, tool frames and error frames all stay under the transport's 10 MiB line limit. And the new tests are bounded and deterministic: no sleeps, clocks or ports, and module state is reset per test.

Done well: a bad script file or a bad entry never breaks a run. It is dropped with one warning that names the problem, and the turn gets the reply it would have had anyway. The size tests run the real module over stdio and check against the real transport limit.

Optional:

  • src/kiro_crew/testing/scripted_replies.py:23 says text is "streamed as one message chunk". The backend splits long text into chunks of 256 Ki characters, and the same docstring says so a few lines further down. Fixing this line makes the docstring agree with itself.
  • The Design note is still open: no test runs the documented spawn_feature_gateway recipe in scripted-test-model.md. One test with a one-entry script would prove the recipe, and it would also catch a later env scrub that drops the variable.
  • No test holds two claims: that a [[WF_E2E: prompt still wins over an active script, and the fallback when scripted_replies cannot be imported. A small in-process test for each would pin them.

@javenciu
javenciu merged commit dc4692c into main Nov 15, 2026
99 checks passed
@javenciu
javenciu deleted the feat/scripted-test-backend branch November 15, 2026 11:05
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Nov 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants