Repository navigation
feat(testing): let the fake ACP backend answer from a reply script - #37510
Conversation
UX Review (Opus 5.5) — ⏭️ skippedRevision |
Design Review (Opus 5.5) — ✅ PASSDesign-level review of Design-Verdict: PASS Nothing blocks: this is an opt-in test aid, reachable only when Notes
[DESIGN-REVIEWED] 0713c2b Description read[DESCRIPTION-READ] 4fba6f30927e70e2d521d221e61864d35b86289aab913f3bc4809039e58c80a7 That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
First Principles Review (Opus 5.5) — ✅ PASSPremise-level review of First-Principles-Verdict: PASS Nothing blocks. The reply script is off unless What this change shipsInventory (7 items) — 7 justifiedIntent: let someone testing a spec, hook, cron script or app get chosen model replies, in a chosen order, with no tokens and the same result every run. This is an ADDITION.
[FIRST-PRINCIPLES-REVIEWED] 0713c2b Description read[DESCRIPTION-READ] b79a53fec9e1444e9acc06e277e2b1628e6074fae54171e9d31942c27cb97dc8 That sha256 names the PR description this verdict read. Recompute it with |
Opus 5.5 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
GPT 6.1 Review — ✅ no blocking findingsGPT 6.1 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
Code change approver recommendation: A person should decideWhy: the decider estimates a 10% chance that this change introduces a defect, within the 12% limit for approving on its own, but it changes 815 lines, more than the 400 it may approve on its own. An opt-in reply script for the packaged fake ACP backend, enabled only by KIROCREW_FAKE_ACP_SCRIPT and reachable only through KIROCREW_KIRO_BIN. With the variable unset, the code path is unchanged from main. The script is checked after the WF_E2E and naming branches, as described. The file is read through safe_read_file, and every exception class it raises is caught. Frame sizing stays under the 10 MiB transport limit (json.dumps escapes to ASCII, so 256Ki chars per chunk is at most about 3 MiB). The env var survives the agent-subprocess scrub. 26 new tests cover the behaviour, including real stdio child runs, and new-test stability passed on both OSes. Only doc drift and minor nits remain; none block. An automated recommendation, not a review: a maintainer still decides. It approves on its own only when the estimated chance of a defect is at most 12% and the change has at most 400 changed lines. Decider v3c (217fe607) · commit |
b0cd662 to
4d5c6c9
Compare
|
Large scripted replies are discarded by the ACP reader (the one line finding, A second warning channel that nothing reads (first-principles note). Removed in the same commit. The branch is rebased onto the current |
4d5c6c9 to
6b7192e
Compare
|
Hostile-file-name test asserts a POSIX exception class (Opus, Scripted tool frames exceed the ACP transport limit (GPT, The branch is rebased onto the current |
KIROCREW_FAKE_ACP_SCRIPT names a JSON file of replies (text, tool calls, stop reason or an error), matched by substring or taken in order per session, so agent specs, hooks, cron scripts and apps can be tested with no tokens and the same result every run. Unset, the fake behaves exactly as before; an unreadable or malformed script is dropped with a warning and the built-in reply is kept. Documented in docs/scripted-test-model.md.
6b7192e to
0713c2b
Compare
|
Rebased onto 09b0ddc as 0713c2b with no change to this PR's code. The failures in backend shard 6 (Linux and Windows) were |
KiroCrewCaptain
left a comment
There was a problem hiding this comment.
Approving. On this head the review checked four things. With KIROCREW_FAKE_ACP_SCRIPT unset, the stand-in sends the same frames as before. The script file is read through hooks.safe_read_file, so a script path that points into a credentials folder is refused, and every read error is caught with a short warning. Text chunks, tool frames and error frames all stay under the transport's 10 MiB line limit. And the new tests are bounded and deterministic: no sleeps, clocks or ports, and module state is reset per test.
Done well: a bad script file or a bad entry never breaks a run. It is dropped with one warning that names the problem, and the turn gets the reply it would have had anyway. The size tests run the real module over stdio and check against the real transport limit.
Optional:
src/kiro_crew/testing/scripted_replies.py:23saystextis "streamed as one message chunk". The backend splits long text into chunks of 256 Ki characters, and the same docstring says so a few lines further down. Fixing this line makes the docstring agree with itself.- The Design note is still open: no test runs the documented
spawn_feature_gatewayrecipe inscripted-test-model.md. One test with a one-entry script would prove the recipe, and it would also catch a later env scrub that drops the variable. - No test holds two claims: that a
[[WF_E2E:prompt still wins over an active script, and the fallback whenscripted_repliescannot be imported. A small in-process test for each would pin them.
Problem / Motivation
Goal: someone testing an agent spec, a hook, a cron script or an app can make the model say chosen things in a chosen order, with no tokens and the same result every run.
Kiro Crew already ships a stand-in model,
kiro_crew.testing.fake_acp_backend, in the runtime wheel. It speaks ACP over stdio, and the end-to-end suite pointsKIROCREW_KIRO_BINat it. But it answers every prompt with one canned line (pong from the fake ACP backend) unless the prompt carries one of its own markers ([[TOOL]],[[ERROR]], ...). The text a user's test sends is written by Kiro Crew, not by the test, so a test cannot place those markers, and it cannot get a reply of its own. The stand-in is also not mentioned anywhere insrc/kiro_crew/docs.Why it matters
Hooks, guardrails, cron scripts and app tools react to what the model says. Testing them today needs a real model: the run costs tokens, and its replies change from one run to the next.
Not a goal
KIROCREW_KIRO_BIN, as its module docstring requires.What changed (motivation → approach → change)
Approach: the fake reads an optional reply script named by
KIROCREW_FAKE_ACP_SCRIPT.src/kiro_crew/testing/scripted_replies.py(new): parses the script. It is one JSON object with areplieslist. An entry withwhenanswers every prompt containing that text (plain substring). Entries withoutwhenanswer once each, in file order, per session. Each entry givestext,tools(title, kind, input, status),stop(end_turn,max_tokens,refusal) orerror. The file is read throughkiro_crew.hooks.safe_read_file, and a refusal names only the file and the error class. A bad file or entry (unreadable, not JSON, nested past the recursion limit, a rule broken, or a tool call or error message whose serialized frame passes 1 Mi characters, which cannot be split) is dropped with a warning on stderr, once, throughsafe_terminal_line. The turn then keeps the answer it gets today.src/kiro_crew/testing/fake_acp_backend.py: the prompt handler asks the script first, right after the session-naming check (so titles never use up an entry). It imports the script module only when the variable is set, so the fake stays stdlib-only at import time. A scriptedtextis sent in chunks of 256 Ki characters, so a long reply stays under the 10 MiB line limit the ACP transports read (a character grows to at most 12 bytes once JSON-escaped).src/kiro_crew/docs/scripted-test-model.md(new) + a row indocs/index.md: the file format, running a gateway on it with a throwaway home (the stand-in is reached only throughKIROCREW_KIRO_BIN), and using it from pytest withspawn_feature_gateway. It says which interpreter must run the stand-in.src/kiro_crew/testing/__init__.py: lists the new module.Backwards compatibility
With
KIROCREW_FAKE_ACP_SCRIPTunset,_script()returnsNoneand every frame is byte-for-byte what main sends.test_without_a_script_a_plain_prompt_keeps_the_canned_replypasses on main and on this branch, and the 55 existing tests intest/test_fake_acp_backend.pypass unchanged. Outsidekiro_crew.testingonly the docs change.Tests
fake_acp_backend.py, 24 of the 26 new tests fail. The script is ignored, so the replies are the canned line. The 2 that pass are the no-script baseline and the pure refused-read check.python -m kiro_crew.testing.fake_acp_backend acp), so no kiro-cli is needed. Two send an 11 MiB reply and check that every frame stays under the transports' 10 MiB line limit and the chunks join back whole; four send an oversized tool call or error and check it is dropped with one warning and the turn still ends. The hostile-file-name test no longer names the exception class, which differs by platform; a second test forcesOSError(EINVAL), as Windows raises.flake8,isortandmypypass on the changed files. The black, comment-history, sync-IO and flake-ratchet gates pass.python -m pytest test/test_fake_acp_backend_script.py -n0Manual verification
Ran the module as a child process with a one-entry script. Fed
initialize,session/newand onesession/prompton stdin. The reply chunk was the scripted text, and the result was{"stopReason": "end_turn"}.Sibling sweep
Sibling:
src/kiro_crew/testing/harness.py-- not affected --harness_environmentlayersos.environ, so the variable reaches a harness gateway as isSibling:
src/kiro_crew/testing/workflow_memory_scenario.py-- not affected -- its[[WF_E2E:...]]path is checked before the script and keeps priorityPattern harvest
Rule candidate: none
Pattern: a test double that answers only to its own in-band markers cannot be driven by text the product writes. Give it an out-of-band input instead.
Related Issues
None found (searched open and closed PRs and issues for scripted, mock and deterministic test backends).
Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)Contribution License Agreement