Skip to content

fix(shell-walk): read a heredoc body as data only when every command just reads it - #19986

Closed
mbriones98 wants to merge 1 commit into
mainfrom
fix/shell-walk-heredoc-reader-19588
Closed

mbriones98 wants to merge 1 commit into
mainfrom
fix/shell-walk-heredoc-reader-19588

Conversation

@mbriones98

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: Make the shared shell reader treat a heredoc body the way bash does and bound its descent on structure, so every argv floor reads the same, correct command shape.

security/shell_normalizer._shell_payload_walk is the one reader every argv floor goes through. It has two readings that do not match what the shell runs:

  1. A heredoc body is read as commands whatever consumes it. The walk folds a newline into ;, so it cannot tell bash <<EOF (runs the body) from cat <<EOF or git commit -F - <<EOF (reads it as data). A commit message, PR body or spec bullet that only names a flagged command is then read as that command.
  2. A nested payload that grows past its parent after $HOME expansion is not read. The walk descends a payload only while it is shorter than its parent's text, but $HOME is expanded per token before that comparison, so on a host with a long home path a payload exceeds the line and is skipped.

Why it matters

The first reading produces false refusals of ordinary data (a commit message, a file written through a heredoc). The second makes a correctness bound depend on a value outside the command, so the same command reads differently on two hosts. Both live in the one reader, so every floor inherits them.

Not a goal

  • No floor's deny matrix changes shape; this is the shared reader only.
  • No new command spelling is added to any catalog row.
  • Multi-heredoc and other non-simple shapes keep the fail-closed reading rather than gaining new attribution.

What changed (motivation → approach → change)

The reader (security/shell_normalizer.py) folds a heredoc body into the frame as ;-separated commands, which is why a data consumer's body is scanned as if it ran. The fix attributes a body to its consumer with a positive allow-list over the whole command: the body is set aside as data only when, with it removed, every command of every frame is a listed data consumer -- a bare cat/tee/head/grep-style program, or git/gh followed directly by a message subcommand (git commit/tag/notes, gh pr/issue/release). Any other program anywhere keeps every body: a shell or evaluator, xargs, a pipe into an unlisted program, a path such as ./f, a leading assignment, an operator glued inside a word, a process substitution, or git/gh with an option before the subcommand. The boundary is read from the raw line as the shell reads it: the one heredoc operator must be unquoted and unescaped (a << in quotes or behind a backslash is text), its delimiter ends at the first unquoted blank or metacharacter (<<eof>out is the tag eof and a redirect), and some later line must close the body with no backslash-newline before it (in an unquoted-delimiter body a backslash-newline joins two lines before the shell compares them to the delimiter). The opener line must also end its command at the newline and carry no comment, substitution, subshell or ANSI-C string. Any other shape keeps the body. The frame's source drops the body together with its tokens, so token readers and source readers see the same command.

The descent bound is measured with the home path discounted, so a payload that lives inside one token of its parent stays shorter than the parent whatever the home path's length. The bound stays structural and no longer varies with $HOME.

The module spec entry for the reader is updated to describe both, and states the heredoc attribution as the one reading the reader sets aside, with why it lives in the reader. The package line budget rises for the added helpers.

This unblocks the parked chmod floor work in #19563, which can drop its local heredoc workaround and rely on the reader once this lands. #14993 rewrites the same walk for performance and will need a rebase against whichever lands first.

Backwards compatibility

Compatible: no command that is refused today becomes allowed unless every command in it is a listed data consumer; the only behaviour that changes is that a provably-inert data body is no longer read as commands, which removes false refusals. The frames the walk returns for every runner shape are the same.

Removes nothing: no reader, record, key or public function is removed; the walk's return type and callers are unchanged.

Tests

test/test_shell_walk_heredoc_and_home_bound.py locks the walk's behaviour on neutral commands: a data-consumer body is set aside (and absent from the frame source); a body any unlisted program, path, assignment, glued operator, process substitution, unlisted git/gh subcommand or -c option could run is read; a backslash-continued opener line, multi-heredoc shapes, a quoted or escaped <<, a body no line closes, and a body line ending in a backslash keep their body; a delimiter with a glued redirect or parenthesis ends where the shell ends it; a substitution inside a data body is still its own frame; and a nested payload is descended for both a short and a long home path, with the walk terminating on deep nesting.

test/test_security_gate_liveness.py raises the package line budget to match the added helpers.

Manual verification

N/A — unit coverage sufficient; the change is in a pure text reader exercised directly by the new tests and the existing shell-normalizer / argv-floor / denied-command suites.

Related Issues

Fixes #19588
Supersedes #19951

Pattern harvest

Rule candidate: review-prompt
Pattern: a text reader that folds a data region into executable structure refuses data that merely names a flagged command; attribute the region to its consumer through a positive allow-list over the whole command, so an unrecognised program keeps the region as commands.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

…just reads it

The shared shell walk folded every heredoc body into the frame as commands,
so a message or file body that only names a command was read as that
command. Set the body aside, from both the frame's tokens and its source,
only when every command of every frame is a listed read-only data consumer
(or git/gh with a message subcommand directly after the program). The
boundary is read from the raw opener line as the shell reads it: one
unquoted, unescaped operator, a delimiter that ends at the first unquoted
blank or metacharacter, and a closing line with no backslash-newline before
it. Every other shape keeps the body.

Measure the walk's descent bound with the home path discounted, so a nested
payload is read whatever the length of $HOME.

Fixes #19588
@mbriones98
mbriones98 requested review from a team and KiroCrewCaptain as code owners October 13, 2026 22:39
@mbriones98
mbriones98 requested a review from bolichen97 October 13, 2026 22:39
@mbriones98

Copy link
Copy Markdown
Contributor Author

Sensitive change review

  • Head commit: 717bba8
  • What rule changed: the shared shell reader sets a heredoc body aside as data only when, with it removed, every command of every frame is a listed data consumer (bare cat/tee/head/grep-style program, or git/gh with a message subcommand right after the program) and the opener line ends at its newline. Its descent bound is measured with the home path discounted.
  • Before -> after (what was allowed/blocked, now): before, any heredoc body was read as commands, so a commit message or file body that only names a refused command was refused. After, that data body is allowed. Any other program, path, assignment, glued operator, process substitution, continued opener line or multi-heredoc shape keeps the body read as commands, as before. A nested payload under a long home path is now read where before it was skipped.
  • Worst case if wrong (breaks commands / leaks secret / locks user out): a body that does run is read as data, so a floor misses a command in it. The allow-list is limited to programs with no option that starts another program, and the decision is over the whole command, so one unlisted program anywhere restores the old reading.
  • Evidence checked (which sandbox result proves it): the reviewer shapes from both rounds are refused end to end through is_denied on this head: a continued opener line, | fish, ./f, git -c alias, a delimiter with a glued > or ), a quoted, double-quoted or escaped <<, a body no line closes, and a delimiter split by a backslash-newline. A 57-shape comparison against what bash actually runs found no line bash runs that the walk drops. A plain git commit -F - message is allowed.
  • No regression:
    • Tools still work (which tools/commands were run, result): git commit -F - heredoc, cat > file <<eof, tee, and gh pr create --body "$(cat <<eof ...)" read as data. Shells, xargs and evaluators still read the body.
    • Backward compatible (old config / old data / old callers still work, how checked): the walk's return type and callers are unchanged. 8,141 tests across the shell-normalizer, argv-floor, denied-rule, floor, heredoc and security suites pass locally.
    • Existing tests (which suites passed on this head): test_security*.py, test_shell*.py, test_*argv*.py, test_*denied*.py, test_*floor*.py, test_*perm_verb*.py, test_shell_walk_heredoc_and_home_bound.py. test_security_conductor_* fails identically locally without this change (environment), and CI runs it.
  • How to undo: revert the squash commit. The reader goes back to reading every heredoc body as commands.

@mbriones98

Copy link
Copy Markdown
Contributor Author

Review history carried forward from #19892 and #19951 (same change, recut as one commit). Each item was raised by the review lanes there and is addressed in this commit; plain notes, not disposition records.

  • Narrowing in the reader vs the spec rule: the spec entry states the heredoc attribution as the one reading the reader sets aside, and why; it is a positive allow-list over the whole command.
  • git/gh subcommands: admitted only through a message subcommand directly after the program; -c or any other option before it keeps the body.
  • Source kept the body: the walk returns the stripped source together with the stripped tokens.
  • Split the home-length bound into its own PR: kept together, as the issue asks for both and the parked follow-up pins both answers in one test file.
  • Runner list incomplete (| fish, ./f, git -c alias): inverted to a positive data-consumer list; any other program anywhere keeps every body.
  • Continued opener line: a trailing backslash or open quote on the opener line keeps the body.
  • Quoted or escaped << counted as an opener; delimiter glued to > or ): the operator is found by a quote- and escape-aware scan of the raw line, and the delimiter ends at the first unquoted blank or metacharacter; an unclosed body is kept.
  • Delimiter split by a backslash-newline: a body line ending in a backslash before the close keeps the body.
  • Heredoc list duplicates the shared data-consumer set: kept as a hand-picked stdin subset, because that set is chosen for argument reading only and several members run stdin-borne text; a test pins that the subset stays inside the shared set.
  • Unused parameter: removed.
  • Existing suites unchanged: the denied-command, argv-floor, floor and security suites pass unchanged (8,141 passed locally); no existing assertion was edited.

@github-actions

Copy link
Copy Markdown
Contributor

Sensitive Change Review

Sensitive files changed:

  • src/kiro_crew/security/shell_normalizer.py

  • @mbriones98 posted a complete reasoning template for 717bba8.

@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Oct 13, 2026
@github-actions

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — 🟡 CONCERNS

Premise-level review of 717bba8f6887fe2aaa72ec679e4ab879e7239455 — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: CONCERNS

_HEREDOC_DATA_PROGRAMS hand-copies 17 of the 53 members of the existing _DATA_CONSUMER_PROGRAMS in the same module, so there are now two data-consumer lists to keep in step.

Not justified as shipped

  • Item 1 — duplicate of shell_normalizer._DATA_CONSUMER_PROGRAMS: every member of the new _HEREDOC_DATA_PROGRAMS = frozenset({"cat", "tac", "tee", "head", … "sha256sum"}) is already in _DATA_CONSUMER_PROGRAMS. I counted 17 of 17 by extracting both literals. perm_verb_mention._PERM_VERB_MENTION_EXCLUDED_PROGRAMS already shows how to narrow that list for a stricter reader: subtract from it, so "a consumer added there in future is inherited here".
    Subtraction: delete the _HEREDOC_DATA_PROGRAMS literal. Use _DATA_CONSUMER_PROGRAMS minus a set of the members that can't take stdin as data (sed, echo, printf, the file movers, …).
    Clears when: the heredoc allow-list is derived from _DATA_CONSUMER_PROGRAMS instead of being a second hand-kept list.

What this change ships

Inventory (5 items) — 4 justified

Intent: stop the shared shell reader refusing commit messages and heredoc files that only name a flagged command, and stop its descent depending on $HOME. This is a FIX.

  1. A heredoc body read only by cat/tee/grep-style programs or git commit/gh pr is no longer scanned as commands — duplicate of _DATA_CONSUMER_PROGRAMS
  2. A nested payload is now read whatever the length of $HOME — justified
  3. The spec carves heredoc attribution out of the "narrowing never here" rule — justified
  4. The package line budget rises to 29,197 — justified
  5. A new test file pins both behaviours — justified

[FIRST-PRINCIPLES-REVIEWED] 717bba8

Description read

[DESCRIPTION-READ] ddae983566a5053c31c5ad4d1fb9eebc22bba28725ea849ee126185b845c80d3

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of 717bba8f6887fe2aaa72ec679e4ab879e7239455 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Check that the argv-floor callers at argv_floor.py:331/336/913 don't scan frame source for anything beyond a command shape that a data body should still feed.

[DESIGN-REVIEWED] 717bba8

Description read

[DESCRIPTION-READ] f5558ac4ff9fc68907f633202032f95a123e8d05ca8d9191855d5f6398aad0a7

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — 🔴 changes requested (blocking)

Reviewed 717bba8f6887fe2aaa72ec679e4ab879e7239455 — this comment is updated in place on each push.

A quoted string that spans lines can contain a fake heredoc opener, which strips real commands from the shell walk and lets git push origin main past the git-publish floor.

BLOCKING — src/kiro_crew/security/shell_normalizer.py:3661
opener_line = next(i for i, line in enumerate(lines) if _HEREDOC_OPEN_RE.search(line)) … openers = _unquoted_heredoc_openers(line) / if len(openers) != 1 or not _opener_line_is_closed(line):
Input cat "cat\ncat <<eof\n" ; git push origin main\neof → is_denied → _shell_payload_walk → _without_inert_heredoc_bodies → _without_data_heredoc_body. The quote state is read from the opener line alone, so the <<eof inside a " opened on an earlier line is taken as a real heredoc. The frame then rebuilds as ['cat','cat','cat','<<eof'], every command looks like a data consumer, and is_denied returns None. Bash still runs git push origin main, which the gate refuses when this stripping is disabled.
Fix: quote/escape-scan the whole source up to and including the opener line. Return None if a quote is open when the opener line begins, or if shlex.split("\n".join(lines[:opener_line + 1])) raises.

[BLOCK-MERGE] 717bba8
[OPUS-REVIEWED] 717bba8

Verdict parsed from the review's SHA-scoped output markers for commit 717bba8f6887fe2aaa72ec679e4ab879e7239455.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 717bba8f6887fe2aaa72ec679e4ab879e7239455: <one-sentence reason>

@github-actions

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ⛔ reviewer refused this diff

The provider declined to review 717bba8f6887fe2aaa72ec679e4ab879e7239455; the refusal is caused by the diff's own content, so a re-run is very unlikely to produce a verdict. A declined review is not an approval — a repository writer must adjudicate with /ai-review override gpt 717bba8f6887fe2aaa72ec679e4ab879e7239455: <one-sentence reason>.

This comment is updated in place on each push.

The refusal text is in the GPT 6.1 Review job logs. It is caused by the diff's content: re-runs have not been observed to clear this class of refusal.

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 717bba8f6887fe2aaa72ec679e4ab879e7239455: <one-sentence reason>

@mbriones98

Copy link
Copy Markdown
Contributor Author

Closing in favour of a design pass in #20006.

@mbriones98 mbriones98 closed this Oct 13, 2026
@github-actions github-actions Bot removed the readiness: checking Automated validation is still running label Oct 13, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Security Scope Review -- 🟡 CONCERNS

Which legitimate operations does 717bba8f6887fe2aaa72ec679e4ab879e7239455 newly refuse? Updated in place on each push. A script-confirmed regression blocks; the model proposes candidates and never decides a refusal.

Verdict basis: the reviewer's own judgement; no confirmed regression.

Security scope review -- adjudicated candidates

  • macos-latest (platform posix): 14 classified, 0 newly refused, 14 still allowed.
  • ubuntu-latest (platform posix): 14 classified, 0 newly refused, 14 still allowed.
  • windows-latest (platform windows): 3 classified, 0 newly refused, 3 still allowed.

No candidate flipped from allowed to refused.

Scope-Verdict: CONCERNS

TIGHTENING: the shared shell-payload walk's descent bound (_shell_payload_walk in src/kiro_crew/security/shell_normalizer.py) now measures length with the home path discounted -- - if len(payload) >= parent_len or payload in seen: / + if _structural_len(payload) >= parent_len or payload in seen:. Before, a nested payload whose $HOME/~ expanded past its parent's length was never walked. Now it is, so the inner command gets checked by every tier. The threat it refuses is a refused command hidden in a wrapper payload (bash -c "…$HOME…") padded with home references so the walk skips it. The heredoc half (_without_inert_heredoc_bodies) only sets text aside, so it can't add a refusal; I probed it anyway, for an accidental side effect.

ADJUDICATED: 17 candidates, 0 confirmed newly-refused (no confirmed rows have reached this review; the deterministic fold reports the per-leg results)

PLATFORM COVERAGE:

  • posix (14 rows): these probe the newly walked wrappers.
    • Read-only, test, log, config and scoped-push commands run inside bash -c / sh -c / bash -lc, with two or more ~/$HOME references each. These are the payloads that used to go unwalked and now reach the sensitive, exfil, catalog and git-publish checks.
    • Heredoc consumers agents use every day: git commit -F -, gh pr create --body "$(cat <<'EOF'…)", gh pr comment --body-file -, tee ~/…, cat | grep.
    • Python reading a heredoc, which is not on the data list and so must behave as on the base.
  • windows (3 rows): these probe the diff's one platform assumption. _structural_len does text.replace(os.path.expanduser("~"), "~"), a case-sensitive match against frame text that is otherwise lowercased, using a C:\Users\… home with backslashes.
    • A powershell -Command row using $HOME\….
    • A cmd /c row using %USERPROFILE%.
    • A Git Bash row using ~ with .venv\Scripts\python.exe and \ separators.
  • any (0 rows): every row carries a path or a program name.
  • Not covered: UNC home directories (\\server\users\name) and a $HOME with uppercase letters on posix. Both reach the same case-sensitive replace, and I wrote no row for either.

UNADJUDICATED:

  • The heredoc data-consumer list is a reader-level setting-aside, not a deny rule, so the classifier only sees its effect. The call path: argv_floor.py → _shell_payload_walk → _without_inert_heredoc_bodies → _runs_only_data_consumers. That lets the shell classifier measure the shapes I submitted, but it cannot show that the setting-aside always fires for legitimate inputs.

  • Shapes I could only describe, not submit:

    • git -C ~/repo commit -F - <<EOF: a global option before the subcommand fails the right-after-program test, so the body is still read as commands.
    • cat <<EOF # note: a comment on the opener line excludes it.
    • A body line ending in \: the body is kept.
    • sort/jq consumers: not on the list.

    None of these refuses more than the base does, because each falls back to the base reading. That is why they are listed here rather than claimed as covered.

  • The home discount on Windows may not apply at all. Lowercased frame text could fail to match a mixed-case home, so the bound would behave exactly as on the base. That would mean the tightening has no effect on Windows, which is a breadth question for the other lanes, not a regression. The Windows legs' verdicts are what settle it.

[SCOPE-REVIEWED] 717bba8

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Shared shell walk: heredoc bodies read as commands; HOME expansion can skip a nested payload

1 participant