Skip to content

fix(shell-walk): read a heredoc body as data only when every command just reads it - #19951

Closed
mbriones98 wants to merge 2 commits into
mainfrom
fix/shell-walk-heredoc-data-19588
Closed

mbriones98 wants to merge 2 commits into
mainfrom
fix/shell-walk-heredoc-data-19588

Conversation

@mbriones98

@mbriones98 mbriones98 commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: Make the shared shell reader treat a heredoc body the way bash does and bound its descent on structure, so every argv floor reads the same, correct command shape.

security/shell_normalizer._shell_payload_walk is the one reader every argv floor goes through. It has two readings that do not match what the shell runs:

  1. A heredoc body is read as commands whatever consumes it. The walk folds a newline into ;, so it cannot tell bash <<EOF (runs the body) from cat <<EOF or git commit -F - <<EOF (reads it as data). A commit message, PR body or spec bullet that only names a flagged command is then read as that command.
  2. A nested payload that grows past its parent after $HOME expansion is not read. The walk descends a payload only while it is shorter than its parent's text, but $HOME is expanded per token before that comparison, so on a host with a long home path a payload exceeds the line and is skipped.

Why it matters

The first reading produces false refusals of ordinary data (a commit message, a file written through a heredoc). The second makes a correctness bound depend on a value outside the command, so the same command reads differently on two hosts. Both live in the one reader, so every floor inherits them.

Not a goal

  • No floor's deny matrix changes shape; this is the shared reader only.
  • No new command spelling is added to any catalog row.
  • Multi-heredoc and other non-simple shapes keep the fail-closed reading rather than gaining new attribution.

What changed (motivation → approach → change)

The reader (security/shell_normalizer.py) folds a heredoc body into the frame as ;-separated commands, which is why a data consumer's body is scanned as if it ran. The fix attributes a body to its consumer with a positive allow-list over the whole command: the body is set aside as data only when, with it removed, every command of every frame is a listed data consumer -- a bare cat/tee/head/grep-style program, or git/gh followed directly by a message subcommand (git commit/tag/notes, gh pr/issue/release). Any other program anywhere keeps every body: a shell or evaluator, xargs, a pipe into an unlisted program, a path such as ./f, a leading assignment, an operator glued inside a word, a process substitution, or git/gh with an option before the subcommand. The boundary is read from the raw line as the shell reads it: the one heredoc operator must be unquoted and unescaped (a << in quotes or behind a backslash is text), its delimiter ends at the first unquoted blank or metacharacter (<<eof>out is the tag eof and a redirect), and some later line must close the body. The opener line must also end its command at the newline and carry no comment, substitution, subshell or ANSI-C string. Any other shape keeps the body. The frame's source drops the body together with its tokens, so token readers and source readers see the same command.

The descent bound is measured with the home path discounted, so a payload that lives inside one token of its parent stays shorter than the parent whatever the home path's length. The bound stays structural and no longer varies with $HOME.

The module spec entry for the reader is updated to describe both, and states the heredoc attribution as the one reading the reader sets aside, with why it lives in the reader. The package line budget rises for the added helpers.

This unblocks the parked chmod floor work in #19563, which can drop its local heredoc workaround and rely on the reader once this lands. #14993 rewrites the same walk for performance and will need a rebase against whichever lands first.

Backwards compatibility

Compatible: no command that is refused today becomes allowed unless every command in it is a listed data consumer; the only behaviour that changes is that a provably-inert data body is no longer read as commands, which removes false refusals. The frames the walk returns for every runner shape are the same.

Removes nothing: no reader, record, key or public function is removed; the walk's return type and callers are unchanged.

Tests

test/test_shell_walk_heredoc_and_home_bound.py locks the walk's behaviour on neutral commands: a data-consumer body is set aside (and absent from the frame source); a body any unlisted program, path, assignment, glued operator, process substitution, unlisted git/gh subcommand or -c option could run is read; a backslash-continued opener line, multi-heredoc shapes, a quoted or escaped <<, and a body no line closes keep their body; a delimiter with a glued redirect or parenthesis ends where the shell ends it; a substitution inside a data body is still its own frame; and a nested payload is descended for both a short and a long home path, with the walk terminating on deep nesting.

test/test_security_gate_liveness.py raises the package line budget to match the added helpers.

Manual verification

N/A — unit coverage sufficient; the change is in a pure text reader exercised directly by the new tests and the existing shell-normalizer / argv-floor / denied-command suites.

Related Issues

Fixes #19588
Supersedes #19892

Pattern harvest

Rule candidate: review-prompt
Pattern: a text reader that folds a data region into executable structure refuses data that merely names a flagged command; attribute the region to its consumer through a positive allow-list over the whole command, so an unrecognised program keeps the region as commands.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

…just reads it

The shared shell walk folded every heredoc body into the frame as commands,
so a message or file body that only names a command was read as that
command. Set the body aside, from both the frame's tokens and its source,
only when every command of every frame is a listed read-only data consumer
(or git/gh with a message subcommand directly after the program) and the
opener line ends at its newline; every other shape keeps the body.

Measure the walk's descent bound with the home path discounted, so a nested
payload is read whatever the length of $HOME.

Fixes #19588
@mbriones98

Copy link
Copy Markdown
Contributor Author

Sensitive change review

  • Head commit: a4ad2a1
  • What rule changed: the shared shell reader sets a heredoc body aside as data only when, with it removed, every command of every frame is a listed data consumer (bare cat/tee/head/grep-style program, or git/gh with a message subcommand right after the program) and the opener line ends at its newline. Its descent bound is measured with the home path discounted.
  • Before -> after (what was allowed/blocked, now): before, any heredoc body was read as commands, so a commit message or file body that only names a refused command was refused. After, that data body is allowed. Any other program, path, assignment, glued operator, process substitution, continued opener line or multi-heredoc shape keeps the body read as commands, as before. A nested payload under a long home path is now read where before it was skipped.
  • Worst case if wrong (breaks commands / leaks secret / locks user out): a body that does run is read as data, so a floor misses a command in it. The allow-list is limited to programs with no option that starts another program, and the decision is over the whole command, so one unlisted program anywhere restores the old reading.
  • Evidence checked (which sandbox result proves it): the reviewer's continued-line, | fish, ./f, and git -c alias shapes are refused end to end through is_denied on this head. A plain git commit -F - message is allowed.
  • No regression:
    • Tools still work (which tools/commands were run, result): git commit -F - heredoc, cat > file <<eof, tee, and gh pr create --body "$(cat <<eof ...)" read as data. Shells, xargs and evaluators still read the body.
    • Backward compatible (old config / old data / old callers still work, how checked): the walk's return type and callers are unchanged. 8,130 tests across the shell-normalizer, argv-floor, denied-rule, floor, heredoc and security suites pass locally.
    • Existing tests (which suites passed on this head): test_security*.py, test_shell*.py, test_*argv*.py, test_*denied*.py, test_*floor*.py, test_*perm_verb*.py, test_shell_walk_heredoc_and_home_bound.py. test_security_conductor_* fails identically locally without this change (environment), and CI runs it.
  • How to undo: revert the squash commit. The reader goes back to reading every heredoc body as commands.

@mbriones98

Copy link
Copy Markdown
Contributor Author

Review history carried forward from #19892 (same change, recut as one commit). Each item below was raised by the review lanes on #19892 and is already addressed in this commit; plain notes, not disposition records.

  • Narrowing in the reader vs the spec rule

Fixed. The spec entry now states the heredoc attribution as the one reading the reader sets aside, and why it lives there (a folded body is mis-read by every floor at once). It is now a positive allow-list over the whole command: a body is set aside only when every command of every frame is a listed data consumer, so any unlisted program anywhere keeps every body.

  • git/gh subcommand never checked

Fixed. git and gh now qualify only when the word right after the program is a listed message subcommand (git commit/tag/notes, gh pr/issue/release); any global option before it, or any other subcommand, keeps the body. New neutral test rows: git x <<eof, gh x <<eof.

  • arguments of a data program never examined (git -c alias)

Fixed. git -c ... puts an option before the subcommand, so it no longer qualifies and the body is kept. The same row is a new test case, and the shape is refused end to end through is_denied again.

  • Clears-when conditions

Both met in 844efd8: the spec entry explicitly states the exception and its reason, git/gh are admitted only through a positive subcommand allow-list, and a test pins that a -c alias configuration keeps the body.

  • split the home-length bound into its own PR

Kept together: the linked issue asks for both readings in one change, and the parked follow-up pins both answers in one test file that flips together. The bound change is a few lines and isolated in its own helper, so it can be reviewed on its own here.

  • source keeps the body while tokens drop it

Fixed. The walk now returns the stripped source together with the stripped tokens, so every reader of the frame, including the source-based ones, sees the same command shape. A new test pins that the set-aside body appears in no frame source.

  • shell_normalizer.py:3582 opener line continued onto the next line

Fixed. The body is set aside only when the opener line ends its command at the newline: a trailing backslash or an open quote keeps the body. The continued-line shape from this finding is refused end to end again, and a neutral test row pins the reading.

  • shell_normalizer.py:3529 short list of runners treated as complete

Fixed. The test is inverted to a positive list: a body is set aside only when every command of every frame is a bare listed data consumer (cat, tee, grep-style filters) or git/gh with a message subcommand right after the program. fish, sort, a path such as ./f, a leading assignment, a glued operator, or a process substitution all keep the body. Each of the three cited shapes is refused end to end again and has a neutral test row; git/gh with -c no longer qualify.

  • allow-list entries not each cited by the issue or a test

Kept as shipped, with reasoning. Every entry is a program whose only use of stdin is to read, print, count, filter or hash it, and none has an option that starts another program; each is a member of the existing data-consumer vocabulary, minus the members that can spawn (sed, awk, pagers, sort, jq). A heredoc into any of them is the same shape as the cat case the issue reports, so the narrowing per entry is the cat reading, not a new one. git tag -F - and git notes add -F - take a message from stdin exactly as git commit -F - does. Whole-command gating means one unlisted program anywhere keeps every body. If a maintainer prefers the minimal list (cat, tee, git commit, gh pr), dropping the rest is a one-line change with no test impact.
Existing suites: the existing denied-command, argv-floor, floor, perm-verb, shell-normalizer and security suites pass unchanged locally (8,130 passed). No existing test expected a cat, tee or git commit heredoc body to be refused, and no existing assertion was edited; the only test edits are the new walk tests and the package line budget.

@github-actions

github-actions Bot commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor

Sensitive Change Review

Sensitive files changed:

  • src/kiro_crew/security/shell_normalizer.py

  • @mbriones98 posted a complete reasoning template for aa0e7ad.

@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Oct 13, 2026
@github-actions

github-actions Bot commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ✅ PASS

UX-level review of aa0e7ad47f163a4af5f28ac6658890ca0378fe42 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

UX-Verdict: PASS

Nothing to check.

[UX-REVIEWED] aa0e7ad

Description read

[DESCRIPTION-READ] 374f81013fb947f63b55055f92ffb3e9d5bd6f4d1b78a573dd4b8edb4f664c25

That sha256 names everything this verdict read: the PR description, plus the 3 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of aa0e7ad47f163a4af5f28ac6658890ca0378fe42 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Check that the existing denied-command and argv-floor suites pass unchanged in CI. This PR narrows the reader that every floor shares, and the spec previously said narrowing never happens there.

[DESIGN-REVIEWED] aa0e7ad

Description read

[DESCRIPTION-READ] 6b86aaf622d5d770fd4436a37fe91d81ad1a2ac5110b079222f511a144d4fa91

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — 🟡 CONCERNS

Premise-level review of aa0e7ad47f163a4af5f28ac6658890ca0378fe42 — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: CONCERNS

The fix targets the right cause, but its new heredoc program list is a second copy of an existing set, and one of its parameters is never read.

Not justified as shipped

  • Item 3 — duplicate of _DATA_CONSUMER_PROGRAMS (src/kiro_crew/security/shell_normalizer.py:430): all 17 members of the new _HEREDOC_DATA_PROGRAMS literal (cat…sha256sum) are already in that set, counted by hand. perm_verb_mention.py:161 already derives its narrower list as _DATA_CONSUMER_PROGRAMS - _PERM_VERB_MENTION_EXCLUDED_PROGRAMS, so a member added to the shared set reaches it too. The new literal drifts away from the shared set instead.
    Subtraction: replace the literal with _DATA_CONSUMER_PROGRAMS - <one excluded set> (the members that can run a program or that ignore stdin).
    Clears when: _HEREDOC_DATA_PROGRAMS is derived from _DATA_CONSUMER_PROGRAMS.
  • Item 7 — zero consumers: _without_data_heredoc_body(source, tokens) throws its argument away on the first line with del tokens, and its one caller passes it anyway.
    Subtraction: drop the tokens parameter.
    Clears when: the parameter is removed.

What this change ships

Inventory (8 items) — 6 justified

Intent: stop every argv floor refusing commit messages and files written through a heredoc, and make nested-payload reading independent of $HOME length. FIX (#19588, with a measured reproduction).

  1. A heredoc body that only cat/tee/git commit/gh pr reads is no longer refused as commands — justified
  2. A nested payload is read whatever the length of the home path — justified
  3. A new list of programs that read heredocs as data — duplicate of shell_normalizer._DATA_CONSUMER_PROGRAMS
  4. An allow-list of git/gh message subcommands — justified
  5. The heredoc opener and delimiter are read from the raw line — justified
  6. The package line budget rises to 29,192 — justified
  7. A helper parameter that is accepted and then discarded — zero consumers
  8. The security spec entry for the reader is updated — justified

[FIRST-PRINCIPLES-REVIEWED] aa0e7ad

Description read

[DESCRIPTION-READ] 2162e8eed6c39261f1df48e0d9bad4c28ac5fa906f130ed06a7907c100882803

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — 🔴 changes requested (blocking)

Reviewed aa0e7ad47f163a4af5f28ac6658890ca0378fe42 — this comment is updated in place on each push.

Blocks merge: a heredoc delimiter split by a backslash-newline hides a protected-branch git push from every floor that reads the payload walk.

BLOCKING — src/kiro_crew/security/shell_normalizer.py:3652
if (lines[i].lstrip("\t") if strips_tabs else lines[i]) == tag:
cat <<eof\ne\<newline>of\ngit push origin main\neof → with an unquoted delimiter, bash joins the backslash-newline and ends the body at e\+of, then runs the push. The walker compares raw physical lines, so it takes the next real eof as the end, treats git push origin main as body, and _without_inert_heredoc_bodies drops it. _shell_payload_walk returns only [('cat <<eof\n', ['cat','<<eof',';'])], so is_denied returns None for that command, while the bare git push origin main is refused. _is_self_kill and _is_credential_mint also flip to False.
Fix: for an unquoted delimiter, return None (keep the body) when any body line before the matched close ends in \.

[BLOCK-MERGE] aa0e7ad
[OPUS-REVIEWED] aa0e7ad

Verdict parsed from the review's SHA-scoped output markers for commit aa0e7ad47f163a4af5f28ac6658890ca0378fe42.

False positive or not applicable? A repository writer can comment:
/ai-review override fable aa0e7ad47f163a4af5f28ac6658890ca0378fe42: <one-sentence reason>

@mbriones98

Copy link
Copy Markdown
Contributor Author
  • span=4853250cab09 heredoc list duplicates the data-consumer set

Kept as a positive list, on purpose. Membership in the shared data-consumer set is decided on one property: the program's ARGUMENTS are data, never a program name it runs (the comment above that set says so, and points new members only at the argument-helper exclusion list). It says nothing about what a program does with its STDIN, which is what a heredoc feeds. Several members already run or evaluate stdin-borne text (sed with the e command, awk system(), pagers with !cmd, sort --compress-program), and a member added later is vetted only for its arguments. Deriving the heredoc set by subtraction would therefore admit every future member for heredoc bodies without anyone checking its stdin grammar, which turns the fail-closed default around. A short positive list, each entry chosen for reading stdin only, keeps an unreviewed program out by default. The base set stays the single source for argument reading; this list is only the stdin subset, and its docstring states that.

@mbriones98

Copy link
Copy Markdown
Contributor Author

Agreed, and it is already gated that way: this path is on the sensitive-path list, so merging needs a maintainer approval on this head in addition to the author's template. The spec edit is called out in the description and in the Sensitive Change template as the one reading the reader sets aside, so the approving maintainer signs off on that exception explicitly.

@github-actions

github-actions Bot commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor

⚠️ Stale verdict notice (2026-10-13 22:29 UTC): the provider REFUSED to review aa0e7ad47f163a4af5f28ac6658890ca0378fe42 — the refusal is caused by the diff's own content, so a re-run is very unlikely to produce a fresh verdict. The verdict below is from an earlier completed run on an OLDER commit. A repository writer must adjudicate: /ai-review override gpt aa0e7ad47f163a4af5f28ac6658890ca0378fe42: <one-sentence reason>.

GPT 6.1 Review — 🔴 changes requested (blocking)

GPT 6.1 found at least one blocking issue that must be resolved before merging a4ad2a1227e72a4a2fddacc98c712590ca0f8342. 1 of 1 blocking finding(s) are security-class and were withheld from adjudication, so the blocking verdict stands.

This comment is updated in place on each push.

BLOCKING -- src/kiro_crew/security/shell_normalizer.py:3609 -- Misparsed heredoc boundaries bypass protected-branch denial

stripped = "\n".join(lines[: opener_line + 1] + rest)
Cron commands cat --help "<<eof"\ngit push origin main and cat <<eof>/dev/null\ndata\neof\ngit push origin main -> _vet_shell_command -> is_denied -> _shell_payload_sources deletes the executable push by mistaking quoted text for an opener or including the adjacent redirect in the delimiter; vetting allows both although the shell executes the protected-branch push.
Anchor: residual/security
Fix: Recognize active heredoc operators and exact delimiter boundaries from quote-aware raw syntax; retain the original frame whenever attribution is uncertain.

[BLOCK-MERGE] a4ad2a1
[GPT-REVIEWED] a4ad2a1

Adjudication (Opus 5.5) — is blocking on each finding proportionate?

F1 (fenced, unbounded harm: the vetter wrongly allows a protected-branch push). I confirmed both conditions in the code:

  • _heredoc_tag (shell_normalizer.py:3555-3566) has two gaps. It finds << in a token after shlex has removed the quotes, so a quoted "<<eof" still counts as an opener. It also splits the tag only on ;&|, so in <<eof>/dev/null the tag becomes eof>/dev/null.

  • In both cases no later line matches the tag, so the loop at :3604-3607 treats everything after the opener as body. The join at :3609 then drops the git push line from the vetted frame.

  • There is no recovery path: the stripped frame is what gets vetted. Both inputs are short, ordinary-looking strings an agent can write, so this is not a rare combination.

    [ADJUDICATION] a4ad2a1 total=0 uphold=0 downgrade=0
    [GPT-ADJUDICATED] a4ad2a1

    [ADJUDICATION-FENCED] a4ad2a1 fenced=1 flagged=0
    UPHOLD-FENCED F1 src/kiro_crew/security/shell_normalizer.py:3609 -- A quoted "<<eof" counts as an opener because shlex has already removed the quotes, and in a glued "</dev/null" the redirect ends up in the tag (_heredoc_tag :3555-3566); either way the tag never matches, the loop at :3604-3607 drops every line after the opener, and an easy-to-write command gets a protected-branch push past the vetter with no fallback.
    [GPT-ADJUDICATED-FENCED] a4ad2a1

False positive or not applicable? A repository writer can comment:
/ai-review override gpt a4ad2a1227e72a4a2fddacc98c712590ca0f8342: <one-sentence reason>

@github-actions

github-actions Bot commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor

Security Scope Review -- ✅ PASS

Which legitimate operations does aa0e7ad47f163a4af5f28ac6658890ca0378fe42 newly refuse? Updated in place on each push. A script-confirmed regression blocks; the model proposes candidates and never decides a refusal.

Verdict basis: adjudicated, zero confirmed regressions.

Security scope review -- adjudicated candidates

  • macos-latest (platform posix): 12 classified, 0 newly refused, 11 still allowed.
  • ubuntu-latest (platform posix): 12 classified, 0 newly refused, 11 still allowed.
  • windows-latest (platform windows): 6 classified, 0 newly refused, 5 still allowed.

No candidate flipped from allowed to refused.

Scope-Verdict: PASS

TIGHTENING: The shared shell payload walk's descent bound (_shell_payload_walk in src/kiro_crew/security/shell_normalizer.py) now measures both a nested payload and its parent with the home path collapsed to one character. Before, a payload whose ~/$HOME had been expanded to the real home path could end up longer than its parent's own text, so it was skipped and never read. Now every argv floor reads it. The threat it closes is a wrapped command hidden from the floors by $HOME length — - if len(payload) >= parent_len or payload in seen: / + if _structural_len(payload) >= parent_len or payload in seen: together with + pending: list[tuple[str, int, bool]] = [(text_lower, _structural_len(text_lower) + 1, True)]. The other half of the diff (_without_inert_heredoc_bodies) loosens rather than tightens: it removes data-consumer heredoc bodies. It was probed only to check that rebuilding each frame's source without the body does not newly match anything.

ADJUDICATED: 16 candidates, 0 confirmed newly-refused (the harness handed this review no confirmed rows; the per-leg verdicts come from the deterministic fold)

PLATFORM COVERAGE: The diff's platform assumption is the spelling and length of the home path that os.path.expanduser("~") returns. That path gets replaced back inside tokens that $HOME and ~ have already expanded.

  • posix (10 rows): bash -c/sh -c/bash -lc wrappers with inner payloads that use ~/… and $HOME/…. They cover the CLI (kirocrew status, kirocrew cron list), git (status, fetch, and a scoped push -u origin feat/example), a personal script under ~/bin, and log reads under ~/.kiro. Two of the ten are heredoc rows that write under /tmp and ~/.kiro. These include the tightest shape (bash -c ~/bin/run-tests.sh), where the expanded payload used to exceed its parent even with a short home.
  • windows (4 rows): Git Bash with $HOME and ~ expanding to a backslashed C:\Users\… home, pwsh -Command with ~\.kiro\logs, and cmd /c with %USERPROFILE%\…\kirocrew.exe. Together these probe the backslash separator, the different home length, the .exe name and the %VAR% spelling.
  • any (2 rows): a git commit -F - heredoc and a gh pr create --body "$(cat <<'EOF' …)". They check that the rebuilt frame sources stay allowed on every host.

No platform the diff assumes is left without a candidate. One note for the Windows legs: _structural_len replaces the home path case-sensitively, while the walk's text is lowercased. A literal, unexpanded home path in a command on Windows is therefore not discounted on the parent side. That makes the parent longer, so the walk descends more often, not less. It cannot cause a new refusal, so it is not a finding for this lane.

[SCOPE-REVIEWED] aa0e7ad

Find the heredoc operator in the frame's raw source with a quote- and
escape-aware scan, so a quoted or escaped `<<` is text, not an opener. Read
the delimiter as the shell does, ended by an unquoted blank or
metacharacter, so a glued redirect or parenthesis is not part of it. Keep
the body whenever no line closes it, and skip attribution when the opener
line carries a comment, substitution, subshell or ANSI-C string.
@mbriones98

Copy link
Copy Markdown
Contributor Author
  • span=c65544f87915 shell_normalizer.py:3564 delimiter read with a glued redirect or parenthesis

Fixed in aa0e7ad. The delimiter is now read from the raw opener line as the shell reads it, ended by an unquoted blank or any of ; & | < > ( ), and it must be a plain word once its quotes are removed. A body that no line closes is kept whole (the result is None) instead of running to the end of input. The <<eof>x and <<eof) shapes are refused end to end again, and neutral test rows pin that the command after the body is read.

@mbriones98

Copy link
Copy Markdown
Contributor Author
  • span=c65544f87915 shell_normalizer.py:3593 quoted or escaped << counted as an opener

Fixed in aa0e7ad. The opener is now found in the frame's raw source with a quote- and escape-aware scan. A << inside single or double quotes, behind a backslash, or as part of <<< is not an opener, and attribution needs exactly one real operator. The token reading, which has already removed quotes, is no longer consulted for the boundary. The quoted, double-quoted and escaped shapes are refused end to end again, each with a neutral test row.

@mbriones98

Copy link
Copy Markdown
Contributor Author
  • span=5e784cc06287 shell_normalizer.py:3609 heredoc boundary misread

Fixed in aa0e7ad. Both causes are addressed: the operator is found by a quote-aware scan of the raw line (so "<<eof" is text), and the delimiter ends at the first unquoted blank or metacharacter (so <<eof>/dev/null has the tag eof). Attribution is skipped when the opener line has a comment, substitution, subshell or ANSI-C string, and the frame is kept whole whenever no line closes the body. Both cited commands are refused end to end again on the new head.

@mbriones98

Copy link
Copy Markdown
Contributor Author

Sensitive change review

  • Head commit: aa0e7ad
  • What rule changed: the shared shell reader sets a heredoc body aside as data only when, with it removed, every command of every frame is a listed data consumer (bare cat/tee/head/grep-style program, or git/gh with a message subcommand right after the program) and the opener line ends at its newline. Its descent bound is measured with the home path discounted.
  • Before -> after (what was allowed/blocked, now): before, any heredoc body was read as commands, so a commit message or file body that only names a refused command was refused. After, that data body is allowed. Any other program, path, assignment, glued operator, process substitution, continued opener line or multi-heredoc shape keeps the body read as commands, as before. A nested payload under a long home path is now read where before it was skipped.
  • Worst case if wrong (breaks commands / leaks secret / locks user out): a body that does run is read as data, so a floor misses a command in it. The allow-list is limited to programs with no option that starts another program, and the decision is over the whole command, so one unlisted program anywhere restores the old reading.
  • Evidence checked (which sandbox result proves it): the reviewer shapes from both rounds are refused end to end through is_denied on this head: a continued opener line, | fish, ./f, git -c alias, a delimiter with a glued > or ), a quoted, double-quoted or escaped <<, and a body no line closes. A plain git commit -F - message is allowed.
  • No regression:
    • Tools still work (which tools/commands were run, result): git commit -F - heredoc, cat > file <<eof, tee, and gh pr create --body "$(cat <<eof ...)" read as data. Shells, xargs and evaluators still read the body.
    • Backward compatible (old config / old data / old callers still work, how checked): the walk's return type and callers are unchanged. 8,184 tests across the shell-normalizer, argv-floor, denied-rule, floor, heredoc and security suites pass locally.
    • Existing tests (which suites passed on this head): test_security*.py, test_shell*.py, test_*argv*.py, test_*denied*.py, test_*floor*.py, test_*perm_verb*.py, test_shell_walk_heredoc_and_home_bound.py. test_security_conductor_* fails identically locally without this change (environment), and CI runs it.
  • How to undo: revert the squash commit. The reader goes back to reading every heredoc body as commands.

@mbriones98

Copy link
Copy Markdown
Contributor Author

Superseded by #19986 (same change plus one more boundary case, recut as a single commit on current main).

@bolichen97
bolichen97 deleted the fix/shell-walk-heredoc-data-19588 branch November 6, 2026 07:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Shared shell walk: heredoc bodies read as commands; HOME expansion can skip a nested payload

1 participant