Skip to content

fix(auto-improvement): re-check eval and quoted separators in the runner gate - #30602

Merged
bolichen97 merged 1 commit into
mainfrom
fix/29851-gate-eval-gh-repo-delete
Nov 13, 2026
Merged

bolichen97 merged 1 commit into
mainfrom
fix/29851-gate-eval-gh-repo-delete

Conversation

@iamwhatever

@iamwhatever iamwhatever commented Nov 5, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: The Auto Improvement runner's shell gate refuses a command it already refuses (git push, gh pr ready, ...) even when that command is run through eval, a nested shell script, or a wrapper, including when the script is spelled with bash's $'...' / $"..." quoting.

Today shell_command_refusal refuses git push --force origin main but allows eval git push --force origin main, sh -c 'git push; true' and script -c 'git push' /dev/null.

Why it matters

The runner is unattended and reads outsider-written PR text. A prompt injection that says "run eval git push ..." gets past the gate today. One word in front of a refused command should not turn it into an allowed one.

Not a goal

  • Item 2 of the issue: gating gh with an allowlist of read-only verbs. That is a design choice, left to a maintainer. This PR adds one denylist row only.
  • gh repo delete and the other repo-changing gh repo verbs (archive, rename, edit). Main has no rule for them, so refusing them is a new restriction: left for a follow-up with an off-by-default setting. An allowed row pins gh repo delete o/p --yes as allowed.
  • glab. The glab allowlist (feat(auto-improvement): support GitLab repositories alongside GitHub #22408, merged) edits the same tables; this PR reuses its _segments split and does not touch its rows.
  • Short script spellings (script -c'git push', --comm='git push'). Main allows these too; not fixed in this slice.
  • ANSI-C escapes inside $'...' (eval $'gh pr rea\x64y 1'). The walk reads $'...' as a quote and does not decode escapes; main allows these too (maintainer review 5887558754 asks for no new refusal mechanism here).
  • Quotes inside heredoc bodies. The walk does not track heredocs, so apostrophes in two bodies can still pair up across lines and hide a nested script between them (cat <<EOF / it's / EOF around sh -c "true;gh pr ready"). Main allows this too.

What changed (motivation → approach → change)

Command Before After
eval git push --force origin main allowed refused
eval 'GH_TOKEN=x gh pr merge 1', script -c 'git push' /dev/null allowed refused
gh repo delete o/p --yes allowed allowed
sh -c 'git push; true', eval 'true; git push' allowed refused
eval 'gh pr ready 1' # it's, eval $'gh pr ready 1' allowed refused
rg -F '$(curl' src, printf '%s\n' ';' curl, git commit -m "fix; gh pr ready" allowed allowed
echo hi # don't, heredoc with it's, echo hi > ./curl allowed allowed
watch 'git push', command -p eval 'curl …' refused refused

eval had no handling, so the gate checked eval itself and never the command behind it. It now has its own branch: the words after eval are joined and checked again as a script, with the existing depth limit, before any option or VAR=value stripping.

A refused command could also hide behind a ;, && or | inside a nested script, because the gate split on those characters before it read quotes. It now walks the characters with quote state, marks only unquoted separators and the substitution openers that bash runs ($( and backticks outside single quotes), then lets shlex (posix=True, punctuation_chars) split on those marks. Quoted text is data: printf '%s\n' ';' curl and rg -F '$(curl' src stay allowed, as on main. A ; inside an sh -c or eval script still starts a new command, and $(...) inside double quotes is still checked. No regex is involved.

Design: the gate checks the union of both splits. The old quote-blind split (main's) always runs, and the quote-aware split only adds more commands to check. shlex and bash read some quotes differently (comments, heredocs, ${x:-"'"}), so the quote-aware split alone can be fooled; with both, whatever main refuses stays refused, and a test over every table row pins that. When shlex cannot parse the text (an apostrophe in a heredoc, or an unclosed quote), only the quote-blind split runs, as on main; no command is refused for being unreadable (maintainer review 5887558754). The walk reads $'...' / $"..." as a quote opening at the $, as bash does, so the script inside is re-analyzed: eval $'gh pr ready 1' is refused because the command is found, and bash -c "printf $'a\n'" stays allowed. Redirections (<, >) do not split, so a redirect target such as ./curl is read as a file name, as on main.

A wrapper's command is checked both as plain joined words (what main checked, and what watch and script hand to a shell) and re-quoted (what an exec-style wrapper like sudo or env passes on); either refusing refuses. Each distinct command is checked once, and the second join only when it differs from the first, so nested wrappers cost about one check per level rather than four. script's -c / --command[=] value is checked as a script before its options are skipped, so script -q /dev/null and script -c 'ls' /dev/null stay allowed.

docs/system-specs/modules/auto-improvement.md describes the eval, script -c and quote-aware handling (D-143).

The gate's test rows move from test_ai_agent_runner_coverage.py to a new test_ai_agent_runner_shell_gate.py, which keeps the first file under the 2,000-line cap.

The denylist itself is unchanged: it gains no rows. Only ways around the existing rows are closed.

Backwards compatibility

Breaking: the gate now refuses eval <refused command>, a refused command after a separator inside a nested script, and script -c with a refused command. Main allowed all three; each only reaches a command main's denylist already names. No command is refused for being unreadable: text shlex cannot parse gets main's quote-blind split only, so eval 'echo hi' # it's, sh -c 'ls' # it's and bash -c "printf $'a\n'" stay allowed (maintainer review 5887558754). The walk treats an unquoted word-initial # as a comment to the end of the line, as bash does, so ls # it's fine and echo a#b stay allowed. Over-refusals kept on purpose, each needing eval, watch, script or a shell's -c: a heredoc body holding one of them around a refused command, a shell -c whose later positional argument holds a separator, and a wrapper in front of quoted $( or separator data (timeout 5 rg -F '$(curl' src), read through the plain wrapper join. Everything main refuses is still refused (the quote-blind split and the plain wrapper join both still run). Writer sweep on origin/main @ 48dee00: shell_command_refusal has one production caller, spine/agent_runner.py (the permission-request handler), and it only judges commands the model asks to run. No code or prompt under apps/builtins/auto_improvement sends eval or a nested script holding a refused command, so nothing the app itself runs is refused.

Removes nothing: the change adds refusals only; no key, path, record or function is removed.

Tests

test_ai_agent_runner_shell_gate.py (the moved denylist rows) gains _REFUSED rows for each row of the issue table; eval '...', sh -c '...' and bash -c '...' around a refused command; the quoted-= eval scripts; the separator bypasses eval 'gh pr ready 1; true', sh -c 'git push && true', sh -c 'git push; true', eval 'true; git push'; $(...) and backticks inside double quotes; a " nested in ${...} hiding git push; command -p eval 'curl …'; a git push hidden by a quote in a comment (true # '\ngit push\n# ') or by a " nested in $(; watch 'git push', watch -n 1 'gh pr ready 1'; $'...' hiding git push or gh pr merge; script --command 'git remote set-url ...', script -c 'git push' /dev/null and script --command='gh pr ready 1' /dev/null; a trailing # it's after eval '...', sh -c '...' and script --command '...', or before a newline and sh -c '...'; echo a#b; git push. Allowed rows add script -q /dev/null, script -c 'ls' /dev/null, echo hi # don't, a # don't comment after git status, a heredoc holding it's, printf x > test/bin/ssh, echo hi > ./curl, the quoted-data rows rg -F '$(curl' src, grep -F '`curl' README.md, printf '%s\n' ';' curl, printf '%s\n' '&&' curl, and printf $'a\n'. _REFUSED adds eval, bash -c, script -c, script --command= and watch around $'gh -R o/p pr ready 1', and eval $"gh -R o/p pr ready 1". New _ALLOWED rows pin the ordinary commands that must keep working: git commit -m "a; b", git commit -m "fix; gh pr ready", python3 -c "import os; print(1)", echo 'x && y', gh pr view 1 --jq ".body | length", grep -E "a|b" f, git log --format="%h; %s", eval echo hello, gh repo view o/p.

A property test runs over every refused and allowed row: whenever one quote-blind segment is refused on its own, the whole command must be refused. A cost test counts _refusal calls for "sudo "*6 + "x "*500 and caps them at 2 per wrapper level; removing either dedupe gives 127 calls and fails it.

Hand mutants against these rows: disabling the eval branch fails the eval rows; running the quote-blind split only when shlex fails fails 16 rows (8 refused rows plus the property test on them); dropping the plain wrapper join fails both watch rows and the command -p eval row; disabling the script -c check fails both script -c / --command= rows; dropping the walk's quote state fails the four quoted-data rows; dropping the re-quoted shlex.join wrapper join fails sudo script -c 'timeout 5 git push' /dev/null. Not reading $' / $" as a quote fails the eight $'...' / $"..." rows, including echo hi # it's / eval $'gh pr ready 1' / # it's. Dropping the walk's comment state fails the eight comment-apostrophe _REFUSED rows (the five # it's suffix rows, echo hi # it's / eval $'gh pr ready 1' / # it's, and the sh -c / eval "true;gh pr ready" pair).

Proof: not provable -- other: prove.py timed out on head b8cfda9 (baseline run over 900s); the hand mutants above stand in.

Manual verification

N/A — unit coverage sufficient: the gate is a pure function of the command string, and every shape is a test row.

Related Issues

Part of #29851

Sibling sweep

Swept: git grep -nE "_COMMAND_WRAPPERS|_SHELL_BINARIES|_FORBIDDEN_SUBCOMMANDS" -- src/
Sibling: src/kiro_crew/apps/builtins/auto_improvement/spine/agent_runner_gate.py -- changed here
Sibling: src/kiro_crew/hooks.py -- not affected -- the platform deny rules there are regex rules, not this wrapper-unwrapping tokenizer.

Pattern harvest

Rule candidate: review-prompt
Pattern: a shell denylist must read quotes before it splits on separators, and must unwrap every builtin that runs its arguments (eval, exec, command, builtin), not only PATH binaries.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@iamwhatever
iamwhatever requested review from a team and KiroCrewCaptain as code owners November 5, 2026 17:33
@iamwhatever
iamwhatever requested a review from buluoray November 5, 2026 17:33
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Nov 5, 2026
@github-actions

github-actions Bot commented Nov 5, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision b8cfda9065fdd88c502c136e939b8a1ce80656ff touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions

github-actions Bot commented Nov 5, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of b8cfda9065fdd88c502c136e939b8a1ce80656ff — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

Nothing blocks: the change only adds refusals for commands the denylist already names, and a revert undoes it cleanly.

Notes

  • This is still a denylist chase on a shell line. The description itself lists bypasses that remain open (ANSI-C escapes, heredoc quotes, short script spellings). The real fix is the issue's deferred item 2, an allowlist of read-only gh verbs, and it should be tracked so this walker doesn't keep growing one spelling at a time.
  • Keeping main's quote-blind split alongside the new one means nothing main refuses gets through. The cost is the over-refusals the PR accepts on purpose (timeout 5 rg -F '$(curl' src). Those are reversible, but they will show up as an unattended run that stalls.

[DESIGN-REVIEWED] b8cfda9

Description read

[DESCRIPTION-READ] 7550ac6b9471bb6fa99ed0a6cbe1e0866590d07aec8981f3e5c3f42de34a1cde

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Nov 5, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of b8cfda9065fdd88c502c136e939b8a1ce80656ff and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] b8cfda9

False positive or not applicable? A repository writer can comment:
/ai-review override gpt b8cfda9065fdd88c502c136e939b8a1ce80656ff false-positive: <reason, 20 to 500 characters>

@github-actions

github-actions Bot commented Nov 5, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed b8cfda9065fdd88c502c136e939b8a1ce80656ff — this comment is updated in place on each push.

Review details

No findings.

Notes:

  • The new walk misreads two inputs: a # right after an escaped space (a\ #), and backslash escapes inside $'...'. Either one lets sh -c "true;gh pr ready" slip past the quote-aware split. Main allows both commands too, so neither is a regression.

[OPUS-REVIEWED] b8cfda9

Verdict parsed from the review's SHA-scoped output markers for commit b8cfda9065fdd88c502c136e939b8a1ce80656ff.

False positive or not applicable? A repository writer can comment:
/ai-review override fable b8cfda9065fdd88c502c136e939b8a1ce80656ff false-positive: <reason, 20 to 500 characters>

@github-actions

github-actions Bot commented Nov 5, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ✅ PASS

Premise-level review of b8cfda9065fdd88c502c136e939b8a1ce80656ff — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

First-Principles-Verdict: PASS

Nothing blocks. Every item closes a way around a row the denylist already has, and the spec's existing "unwrap before it matches" decision covers that. The one note is that this is a text denylist, so bypass spellings will keep turning up.

Notes

  • Item 2 — symptom-level: the patch fixes spellings, not the cause. The cause is that the unattended runner can publish at all, and a text denylist has no limit on how a command can be spelled. ## Not a goal itself names 3 bypass spellings this PR leaves open: ANSI-C escapes in $'...', quotes pairing across heredoc bodies, and the short forms script -c'…' / --comm=. The cause-level fix is to withhold publish credentials from the runner, which is bigger than this PR and is deferred, not demanded. Subtraction: none in this PR. Remove the quote-aware walk once that credential fence exists.

What this change ships

Inventory (9 items) — 8 justified

Intent: stop a word in front of a refused command, such as eval, a nested script or a wrapper, from getting past the runner's shell gate. FIX.

  1. eval <refused command> is now refused — justified
  2. A separator inside a nested script now starts a new command that gets checked, using a quote-aware split added alongside the old quote-blind split — symptom-level: the bypass-spelling chase continues
  3. $'...' / $"..." scripts are now read as quoted text and their contents re-checked — justified
  4. An unquoted # at the start of a word is read as a comment, so an apostrophe in it no longer pairs with a quote elsewhere — justified
  5. The script given to script -c / --command is now checked — justified
  6. A wrapped command is checked both as plain joined words and re-quoted — justified
  7. Each distinct command is checked once, so nested wrappers no longer multiply the work — rides along
  8. The gate's test rows move to a new test file to stay under the line cap — rides along
  9. The spec describes the new behaviour as D-143 — justified

[FIRST-PRINCIPLES-REVIEWED] b8cfda9

Description read

[DESCRIPTION-READ] ae2c3cad26eba1704fed4f2d9c0e8b4912baa31e23a553acd9fac596c54eb5d7

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Nov 5, 2026
@iamwhatever
iamwhatever force-pushed the fix/29851-gate-eval-gh-repo-delete branch from 823d68b to 733e95e Compare November 5, 2026 17:41
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Nov 5, 2026
@iamwhatever

Copy link
Copy Markdown
Contributor Author

Disposition for the Opus 5.5 block at 823d68bca6ab (comment):

fixed in 733e95e: eval no longer goes through the wrapper path. It has its own branch, like the -c shell branch: the joined script (minus a leading --) is re-checked with _refusal(..., depth=depth + 1) before any option or VAR=value stripping, so a quoted script word that holds = is no longer dropped.
Tests: eval 'GH_TOKEN=x gh repo delete o/p --yes', eval 'git -c a=b push origin main' and eval -- git push are new _REFUSED rows. With the new branch disabled, 10 eval rows fail, these three among them.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Nov 5, 2026
@iamwhatever

iamwhatever commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor Author

Disposition for the Opus 5.5 block at 733e95e1a9de (comment):

fixed in 5c068a9: quote-aware split. _refusal now tokenizes with shlex (posix, punctuation_chars) and splits on separator tokens, so a ;/&&/| inside a nested eval/sh -c script starts a new command, while a quoted separator stays in its word. Unclosed quotes are refused instead of falling back to a plain split. Backticks and $( inside double quotes are still re-checked.
Tests: refused rows eval 'gh repo delete o/p --yes; true', sh -c 'git push && true', sh -c 'git push; true', eval 'true; git push' and unclosed sh -c 'git push; true; allowed rows git commit -m "a; b", git commit -m "fix; gh pr ready", python3 -c "import os; print(1)", echo 'x && y', gh pr view 1 --jq ".body | length", grep -E "a|b" f, git log --format="%h; %s". With main's replace chain put back, the 5 refused rows fail.

@iamwhatever
iamwhatever force-pushed the fix/29851-gate-eval-gh-repo-delete branch from 733e95e to 5c068a9 Compare November 6, 2026 02:11
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Nov 6, 2026
@iamwhatever

iamwhatever commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor Author

Disposition for the Opus 5.5 block at 5c068a9f5ffa (comment):

fixed in 49a41a9: (1) shlex misreads a # comment or a " nested in $(, so any text holding #, $( or a backtick also gets main's quote-blind separator split, and every command either split finds is checked; (2) watch re-checks " ".join(wrapped), since it hands its args to sh -c, while exec-style wrappers (sudo, env) keep shlex.join.
Tests: refused rows true # '\ngit push\n# ', echo "$(echo ")"; git push; echo "(")", watch 'git push', watch -n 1 'gh pr ready 1'; the 7 allowed rows stay allowed. Disabling the quote-blind pass fails the first two; reverting watch to shlex.join fails the last two.
Not done: treating # as a comment through shlex's commenters. shlex cuts a#b mid-word and eats the newline after a comment, which would merge two commands into one; the quote-blind pass already refuses both trigger shapes.

@iamwhatever
iamwhatever force-pushed the fix/29851-gate-eval-gh-repo-delete branch from 5c068a9 to 49a41a9 Compare November 6, 2026 02:39
@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Nov 6, 2026
@iamwhatever
iamwhatever force-pushed the fix/29851-gate-eval-gh-repo-delete branch from 49a41a9 to 5f83a61 Compare November 6, 2026 06:19
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Nov 6, 2026

@KiroCrewCaptain KiroCrewCaptain left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the quick fix. This head closes the # it's gap. The new fail-closed branch, though, now refuses ordinary wrapped commands that main allows.

Earlier items from my last review:

  • # it's turning off the refusals: fixed. The five rows and echo a#b; git push are refused, and echo hi # don't stays allowed.
  • Optional, < and > in seps: fixed. printf x > test/bin/ssh and echo hi > ./curl are pinned as allowed.
  • Optional, spec doc: fixed. It now covers eval, script -c and the quote-aware split.
  • Optional, Tests paragraph wording: fixed.
  1. agent_runner_gate.py:287-297. The fail-closed branch also runs on fragments from the quote-blind split. The trigger is a wrapper from _COMMAND_WRAPPERS in front of a shell -c whose script holds a separator. Examples:

    • timeout 300 bash -c 'cd /repo && python -m pytest -q'
    • nohup sh -c 'echo "start"; sleep 1'
    • env PYTHONPATH=src bash -c 'python -m pytest -q "test/a.py" | tail -5'
    • sudo bash -c 'apt update && apt install -y x'

    Each of these lexes cleanly at the top level. Line 311 adds the quote-blind split, which cuts inside the quoted script and yields a fragment like timeout 300 bash -c 'cd /repo. The wrapper branch (lines 395-400) then passes bash -c 'cd /repo to _refusal at depth+1, where shlex raises. The new branch sees bash and -c and returns "command holds a nested script the gate cannot parse". Main allows all of these, and the review reproduced each refusal on this head. Without the wrapper, bash -c 'cd /x && pytest "a.py"' is still allowed. The Backwards compatibility section only names text that shlex cannot parse as a whole, so it does not cover this. The result is that the unattended runner is refused its most common way to bound a test run.

    Smallest fix: apply the fail-closed refusal only to text that the gate lexes whole, meaning the top-level command or a token from the quote-aware split. For example, thread a blind: bool keyword into _refusal and set it when you recurse from a _segments fragment. The # it's rows stay refused, because the comment breaks the lex at depth 0. Add _ALLOWED rows such as timeout 300 bash -c 'cd /x && python -m pytest -q "t.py"', nohup sh -c 'echo "start"; sleep 1' and env A=1 bash -c 'ls; echo "x"'.

Done well: main's quote-blind split still always runs, and the property test pins that whatever main refuses stays refused. A wide random comparison against main found no command that main refuses and this head allows.

Optional:

  • PR Readiness is also red because a lane finding about eval $'git push' was upheld, and this head has no disposition for it. Main allows the same spellings, so it is not a regression. A disposition that rebuts or defers it as a pre-existing class fits better than adding an ANSI-C decoder.
  • A quoted lone separator is read as a split, so printf '%s\n' ';' curl is refused while main allows it. That contradicts "a quoted ; stays in its word". Either list it under Backwards compatibility or keep such a token as a word.
  • No row pins the re-quoted shlex.join wrapper join. A _REFUSED row such as sudo script -c 'timeout 5 git push' /dev/null would catch a later simplification back to a plain join.

@iamwhatever

iamwhatever commented Nov 9, 2026 •

Copy link
Copy Markdown
Contributor Author
  • span=18dc5b3b4a36 Bash-quoted eval scripts bypass the new command check

fixed in d5e2843: the quote-aware split now walks the characters with quote state; an unquoted $ directly before a quote sets a flag, and when the text also holds eval, watch, script or a shell -c the gate refuses (same scope as the lex-error fail-closed).
Rows added to _REFUSED: eval, bash -c, script -c, script --command= and watch around $'gh -R o/p pr ready 1', plus eval $"gh -R o/p pr ready 1"; a mutant without the flag fails all six.
A top-level $'gh' pr ready 1 with no nested script is listed under Not a goal: main allows it too.

@iamwhatever

iamwhatever commented Nov 9, 2026 •

Copy link
Copy Markdown
Contributor Author
  • span=18dc5b3b4a36 Literal search patterns are interpreted as executable substitutions

fixed in d5e2843: separators and substitution openers are marked by a character walk that tracks quote state, so only unquoted separators split and only $( / backticks outside single quotes are re-checked; quoted text is data. No regex.
rg -F '$(curl' src, grep -F 'curl' README.md, printf '%s\n' ';' curlandprintf '%s\n' '&&' curlare now_ALLOWED` rows, as on main; a mutant without the quote state fails all four.

@iamwhatever
iamwhatever force-pushed the fix/29851-gate-eval-gh-repo-delete branch from 5b792cd to d5e2843 Compare November 9, 2026 07:17
@iamwhatever

Copy link
Copy Markdown
Contributor Author

Head note: the F1/F2 fix named c0600f4ea8a6 in the two dispositions above landed as d5e2843, the same change rebased onto main b94f8cb.

@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Nov 9, 2026

@KiroCrewCaptain KiroCrewCaptain left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the fixes on this head. One item from my last review still reproduces on d5e2843, so I am requesting changes again.

Earlier items:

  • Item 1, wrapped bash -c scripts refused: still stands (see below).
  • Optional, the eval $'...' disposition: fixed. The disposition is posted, and eval $'gh -R o/p pr ready 1' is now refused.
  • Optional, a quoted lone separator read as a split: fixed. printf '%s\n' ';' curl is allowed again and pinned as an _ALLOWED row.
  • Optional, a row that pins the re-quoted shlex.join wrapper join: still open (see Optional below).
  1. src/kiro_crew/apps/builtins/auto_improvement/spine/agent_runner_gate.py:310-318. The fail-closed branch still runs on fragments from the quote-blind split. Trigger: a wrapper from _COMMAND_WRAPPERS in front of a shell -c script that holds a separator, for example timeout 300 bash -c 'cd /repo && python -m pytest -q', nohup sh -c 'echo "start"; sleep 1', env A=1 bash -c 'make && make test' or nice -n 5 bash -lc 'cd x; pytest'. Line 332 cuts inside the quoted script and gives timeout 300 bash -c 'cd /repo. The wrapper branch passes bash -c 'cd /repo to _refusal one level down, where shlex raises, and the branch returns "command holds a nested script the gate cannot parse". Main allows every one of these. On this head the bare bash -c 'cd /x && pytest' is still allowed. So the unattended runner cannot use its usual way to put a time limit on a test run, and the reason it is given is wrong. The Backwards compatibility section covers only text that shlex cannot parse as a whole, and these commands parse cleanly. The same check also fires inside a $(...) token: a heredoc commit message whose body has an apostrophe and the word script is refused here and allowed on main.
    Smallest fix: run the fail-closed check only on text the gate lexed whole, meaning the top-level command or a token from the quote-aware split. One way is a blind: bool keyword on _refusal, set when you recurse from a _segments fragment. The # it's rows stay refused, because the comment breaks the lex at depth 0, and timeout 300 bash -c 'cd x && git push' stays refused through the -c recursion. Add _ALLOWED rows such as timeout 300 bash -c 'cd /x && python -m pytest -q "t.py"', nohup sh -c 'echo "start"; sleep 1' and env A=1 bash -c 'ls; echo "x"'. If any over-refusal stays (for example the heredoc case), list it under Backwards compatibility.

Done well: main's quote-blind split still always runs, and the property test over every row pins that a command main refuses stays refused.

Optional:

  • Deleting the shlex.join(wrapped) half of the wrapper check leaves every test green and lets sudo script -c 'timeout 5 git push' /dev/null through again. A _REFUSED row for that command would pin it.
  • The last push widened the **Goal:** line and ## Not a goal with no request from a maintainer. The $'...' fix was already in scope as a defect in added code, so the original wording could stay.
  • The body still calls #22408 open (it is merged, and this PR builds on its _segments helper), and the Proof line names head 7878937 instead of the current head.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Nov 9, 2026
@iamwhatever

Copy link
Copy Markdown
Contributor Author
  • span=18dc5b3b4a36 Literal heredoc bodies are analyzed as executable scripts

Rebutted: an intended fail-closed trade-off, stated under Backwards compatibility. Each listed body is refused only because it holds eval, script -c / --command= or sh -c around a denied command; the gate does not track heredoc provenance and errs toward refusal.
Measured against main f068b8a: a heredoc body with no eval/-c/watch/script (gh pr ready 1; true in the body) is refused on main and here alike, and no such command is newly refused. The refusal is loud and the agent can rephrase; nothing runs.

@iamwhatever

Copy link
Copy Markdown
Contributor Author
  • span=18dc5b3b4a36 Quote-aware shell frames reinterpret positional arguments as commands

Rebutted: an intended fail-closed trade-off. The over-refusal needs a shell -c with a positional argument that holds a separator and a denied command (sh/bash/dash -c with -- "; gh pr ready"). Without -c the same argument is data and stays allowed, as on main (printf "%s\n" "; gh pr ready", echo "; gh pr ready" x).
Splitting the -c operand from later positional arguments needs frame provenance through both splits, which the adjudication judged disproportionate; the refusal is loud and nothing runs.

@KiroCrewCaptain KiroCrewCaptain left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the rebase. This head changes none of the touched files since d5e2843, so item 1 from my last review still reproduces. I am requesting changes again.

Earlier items:

  • Item 1, wrapped bash -c scripts refused: still stands (see below).
  • Optional, the eval $'...' disposition: fixed on the earlier head, and still fixed.
  • Optional, a quoted lone separator read as a split: fixed on the earlier head, and still fixed.
  • Optional, a row that pins the re-quoted shlex.join wrapper join: still open (see Optional below).
  1. auto_improvement/spine/agent_runner_gate.py:310-318. The fail-closed branch still runs on fragments from the quote-blind split. Trigger: a wrapper from _COMMAND_WRAPPERS (timeout, nohup, env, nice, time, sudo) in front of a shell -c script that holds &&, ; or |. Examples are timeout 300 bash -c 'cd /repo && python -m pytest -q', nohup sh -c 'echo "start"; sleep 1', env A=1 bash -c 'make && make test' and nice -n 5 bash -lc 'cd x; pytest'. Outcome: line 332 cuts inside the quoted script and gives timeout 300 bash -c 'cd /repo. The wrapper branch then passes bash -c 'cd /repo to _refusal one level down. There shlex raises, and the gate returns "command holds a nested script the gate cannot parse". The runner's permission handler then rejects the call as shell_denylist. Main allows every one of these commands, and bare bash -c 'cd /x && pytest' is still allowed on this head. So the unattended runner loses its usual way to put a time limit on a test run, and the reason it is given is wrong. These commands lex cleanly at the top level, so the Backwards compatibility section and spec bullet D-143 do not cover them. No _ALLOWED row has a wrapper, a shell -c and a separator together, so the test file stays green.
    Smallest fix: run the fail-closed check only on text the gate lexed whole, meaning the top-level command or a token from the quote-aware split. One way is a blind: bool keyword on _refusal, set when you recurse from a _segments fragment. timeout 300 bash -c 'cd x && git push' stays refused through the -c recursion. Add _ALLOWED rows such as timeout 300 bash -c 'cd /x && python -m pytest -q "t.py"', nohup sh -c 'echo "start"; sleep 1' and env A=1 bash -c 'ls; echo "x"'. If you keep an over-refusal on purpose (for example the heredoc case), list it under Backwards compatibility and in D-143.

Done well: main's quote-blind split still always runs, and the property test over every row checks that a command main refuses stays refused.

Optional:

  • If you delete the shlex.join(wrapped) half of the wrapper check, every test stays green and sudo script -c 'timeout 5 git push' /dev/null is allowed again. A _REFUSED row for that command would pin it.
  • The body still calls #22408 open. It is merged, and this PR builds on its _segments helper. The Proof line names head 7878937, not the current head. Please correct both outside the frozen sections.
  • The red checks on this head (backend shard 6, Coverage Gate and Frontend Tests 1) fail in files this diff does not touch.

@KiroCrewCaptain

KiroCrewCaptain commented Nov 13, 2026 •

Copy link
Copy Markdown
Collaborator

Code change approver recommendation: A person should decide

Why: the decider thinks the change is probably fine but estimates a 15% chance that this change introduces a defect, above the 12% limit for approving on its own.

The change only adds refusals: main's quote-blind split still runs on every command, and the new eval, script -c, $'...' and quote-aware handling each lead back to a command the denylist already names. I traced the key new refused and allowed rows by hand, plus the existing allowed commands in test_dogfood_learnings_tool_gate.py; all behave as expected. Only ValueError can come out of the tokenizer, and it is caught. There is one production caller and no state, config or limit changes. The blocking AI-lane findings are from an earlier commit and cover gaps that main shares and the PR lists as non-goals. Backend tests have not run in CI on this commit because of API rate limits and should be re-run before merging.

An automated recommendation, not a review: a maintainer still decides. It approves on its own only when the estimated chance of a defect is at most 12% and the change has at most 400 changed lines. Decider v3c (217fe607) · commit b8cfda9065 · 558 changed lines. If this recommendation is wrong, reply here: replies are how the approver is evaluated.

@iamwhatever

Copy link
Copy Markdown
Contributor Author
  • span=185f752d5ecd Comment apostrophes hide an eval $'...' line from the quote walk

fixed in e792c75bb500: $' and $" are now found by a plain substring search of the raw text, so a pair of comment apostrophes can no longer hide them from the walk. When such text also holds eval, watch, script or a shell -c, the gate refuses.
The exact command (echo hi # it's / eval $'gh pr ready 1' / # it's) is a _REFUSED row; a mutant without the search fails it. The same command was allowed on main and on the earlier head, so this head closes an existing gap.
Trade-off, in Backwards compatibility and D-143: a harmless $'...' inside such text is refused too.

@iamwhatever

Copy link
Copy Markdown
Contributor Author
  • span=185f752d5ecd A glued script -cCMD operand is not extracted

Rebutted: out of this slice. The glued short spelling (script -c'CMD') is listed under Not a goal, and main allows it too, so this PR opens no new path.
The PR closes the spellings script -c CMD, --command CMD and --command=CMD, each pinned by a _REFUSED row.

@iamwhatever

Copy link
Copy Markdown
Contributor Author
  • span=185f752d5ecd Comment apostrophes pair up and hide a nested script with a separator

fixed in 10b925281104: the quote walk now follows bash's comment rule. An unquoted # at the start of a word comments out the rest of the line, so an apostrophe in # it's never pairs with a quote on another line. A # inside a word is not a comment.
_REFUSED rows: true # it's / sh -c "true;gh pr ready" / # it's and its eval twin; a mutant without the comment state fails both. _ALLOWED rows: ls # it's fine, echo a#b.
Main allowed both rows too. Quotes inside heredoc bodies can still pair up across lines; that is listed under Not a goal (main allows it as well).

@iamwhatever

Copy link
Copy Markdown
Contributor Author
  • span=185f752d5ecd Glued script -cCMD and abbreviated --co= are not extracted

Rebutted: out of this slice. The glued short spelling (script -c'CMD') and abbreviated long options (--co=CMD) are listed under Not a goal, and main allows both, so this PR opens no new path.
The PR closes script -c CMD, --command CMD and --command=CMD, each pinned by a _REFUSED row.

@iamwhatever

Copy link
Copy Markdown
Contributor Author
  • span=18dc5b3b4a36 Exec-wrapper analysis reinterprets quoted data as commands

Rebutted: a known trade-off of the plain wrapper join, already stated under Backwards compatibility. watch and script hand their joined words to a shell, so the gate checks the plain join as well as the re-quoted shlex.join. A wrapper in front of quoted $( or separator data (timeout 5 rg -F '$(curl' src) is therefore read the way a shell would read it, and refused.
The refusal is loud and nothing runs; the agent can drop the wrapper. Keeping a separate join per wrapper kind is the provenance tracking the adjudication judged disproportionate.

@iamwhatever

Copy link
Copy Markdown
Contributor Author

@KiroCrewCaptain Review-ready on head fb99967: CI is green and readiness has passed. This head adds a comment state to the quote walk (an unquoted word-initial # comments out the line, as in bash), so comment apostrophes no longer pair up and hide a nested script. It also searches the raw text for $' and $", and skips the unparseable-text refusal on quote-blind fragments, so wrapped -c scripts with a separator are allowed as on main. Each AI finding has a disposition above. Could you re-review?

@bolichen97 bolichen97 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Requesting changes on one part. I read the whole diff and compared it with the head I last reviewed.

Fixed since my last review: the gh repo delete row is gone and the test now keeps it allowed. The eval, script -c and watch re-analysis, the quote-aware split and the wrapper handling apply main's existing git push and gh pr ready refusals to paths that skipped them, so they can stay on by default.

The new part is the fail-closed branch in _refusal in agent_runner_gate.py: if (not blind and tokens is None) or "$'" in text or '$"' in text: which returns "command holds a nested script the gate cannot parse". Main allows these commands, and this head refuses them:

  • bash -c "printf $'a\n'", which the spec in this PR calls out as refused although harmless.
  • eval 'echo hi' # it's, sh -c 'ls' # it's and any eval, watch, script or shell -c line with an unclosed quote.

That is a new condition that makes a harmless command fail where main passes. Please keep main's behavior for it by default. Two ways that need no new refusal:

  1. For text the quote walk cannot read, run only the quote-blind split, as the previous head of this PR did and as main does.
  2. For $'...' and $"...", read the $ as a plain quote opener in the character walk, so the script inside is re-analyzed. eval $'gh pr ready 1' is then refused because the command is found, and bash -c "printf $'a\n'" stays allowed.

If you want the strict refusal kept, put it behind a setting that is off by default, and add tests for both states. Keep the quote-blind split as the safety net in either case.

I will look again when the head moves.

@iamwhatever

Copy link
Copy Markdown
Contributor Author
  • span=185f752d5ecd Glued script -cCMD, -qcCMD and abbreviated --comm= are not extracted

Rebutted: out of this slice and allowed on main. Main 187b4da allows script -qc'gh pr ready 1' /dev/null and script --comm='gh pr ready 1' /dev/null, so this PR opens no new path. Both spellings are under Not a goal.
The maintainer asked this PR to add no new refusal beyond main's rows (#30602 (review)); the getopt-accurate option reader belongs to a follow-up slice.

@iamwhatever

Copy link
Copy Markdown
Contributor Author
  • span=18dc5b3b4a36 ANSI-C escapes bypass nested-script refusal

needs-a-decision: a real gap, but not one this PR opens. Main 187b4da allows eval $'gh -R owner/repo pr rea\x64y 1' and bash -c $'gh pr rea\x64y 1' too. This head reads $'...' as a quote at the $, per the maintainer review (#30602 (review)), and closes every unescaped spelling; it does not decode ANSI-C escapes.
Decoding them is a new parser layer; the maintainer asked for no new refusal mechanism here. Listed under Not a goal. This is security-class, so it is left to the owner: fix in a follow-up, or override on this head.

@iamwhatever

Copy link
Copy Markdown
Contributor Author

@bolichen97 @KiroCrewCaptain Review-ready on head b8cfda9: CI is green and readiness has passed. Per review 5887558754, this head adds no new refusal mechanism. Text the quote walk cannot read gets only main's quote-blind split, and the walk reads $'...' / $"..." as a quote opening at the $. So eval $'gh pr ready 1' is refused because the command is found, while bash -c "printf $'a\n'", eval 'echo hi' # it's and sh -c 'ls' # it's are allowed as on main. The bash comment rule stays in the walk. ANSI-C escapes and glued script -c'..' spellings, both allowed on main, are listed under Not a goal. Could you re-review?

@bolichen97 bolichen97 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving. I read the whole diff and compared it with the head I requested changes on (fb99967).

  • The fail-closed branch in _refusal is gone. Text the quote walk cannot read, such as an unclosed quote or an apostrophe in a heredoc body, now gets only the quote-blind split, as on main, and is never refused for being unreadable.
  • $'...' and $"..." are read as an ordinary quote opening at the $, so the script inside is re-analyzed. eval $'gh pr ready 1' is refused because the command is found, and bash -c "printf $'a\n'" stays allowed.
  • The blind flag is removed along with the branch it served, and the docs and the D-143 bullet match.
  • New _ALLOWED rows pin main's behavior: bash -c "printf $'a\n'", eval 'echo hi' # it's and sh -c 'ls' # it's. A new _REFUSED row covers eval $'gh pr ready 1'. The property test now calls the public gate, since no fragment is judged separately any more.
  • The eval, script -c and watch re-analysis, the quote-aware split and the wrapper handling still apply main's git push and gh pr ready refusals to paths that skipped them. No regex in product code.
  • The cancelled runs on this head (PR Hygiene, GPT 6.1 Review, PR Policy, Issue Gate, Sensitive Change Review) each have a newer passing run, so they are stale.

I did not run the tests locally.

@KiroCrewCaptain KiroCrewCaptain left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving head b8cfda9. This replaces my earlier change request.

Earlier items from my last review (on 6f3d6d7):

  • Item 1, wrapped bash -c scripts that hold a separator were refused: fixed. The fail-closed branch is gone. timeout 300 bash -c 'cd /repo && python -m pytest -q', nohup sh -c 'echo "start"; sleep 1', env A=1 bash -c 'make && make test' and nice -n 5 bash -lc 'cd x; pytest' are now allowed, as on main, and _ALLOWED rows pin this shape.
  • Optional, a row that pins the shlex.join(wrapped) join: fixed. sudo script -c 'timeout 5 git push' /dev/null is now a _REFUSED row.
  • Optional, body wording: fixed. The body now says #22408 is merged, and the Proof line names this head.

What I checked on this head: I ran the gate from this head and from main on the commands above. eval git push, eval $'gh pr ready 1' and the sudo script -c row are refused on this head and allowed on main. bash -c "printf $'a\n'", eval 'echo hi' # it's, sh -c 'ls' # it's, gh repo delete o/p --yes and a heredoc commit message with an apostrophe are allowed on both. The review also compared 60,000 random commands against main and found none that main refuses and this head allows. It confirmed that every moved test row is still present, and that the new test file fails 33 refused rows when run against main's gate. PR Readiness is green, and every review lane passes on this head.

Security: this changes the unattended runner's shell denylist, and the change only makes it stricter. Main's quote-blind split still runs on every command, so whatever main refuses is still refused. The new refusals are eval, nested-shell, script -c and $'...' spellings of commands the denylist already names. No denylist row is added, and no command is refused just because the gate cannot parse it.

Done well: the property test over every row and the call-count cost test each pin one of the two properties that matter, and neither depends on timing.

The design is signed off under the standing maintainer sign-off for changes by team members.

Optional:

  • No test row pins three new walk lines in agent_runner_gate.py, though each one blocks a real bypass: the substitution-opener mark, the \x00 re-walk of a quoted substitution, and the backslash-escape branch. All tests stay green if you delete any one of them. Rows such as echo "$(sh -c 'true; git push')" and echo \'; sh -c 'true; git push'; echo \' would pin them.
  • The quoted-substitution re-walk adds a few narrow over-refusals that the Backwards compatibility section does not name. One example is echo "$(date) curl", which this head refuses and main allows, although bash never runs curl there. You could list these, or re-walk only the text inside the substitution.
  • The fix can be proven: the new test file fails 33 rows against main's gate in a few seconds, so the Proof line can say PROVEN. Also, the if tokens is None mutant fails 14 tests (7 rows plus 7 property rows), not 16.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

readiness: passed Eligible automated validation passed for the current revision

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants