Skip to content

fix(security): step over ${...} expansions when matching a substitution closer - #23620

Closed
iamwhatever wants to merge 3 commits into
mainfrom
greener/pr-9241
Closed

iamwhatever wants to merge 3 commits into
mainfrom
greener/pr-9241

Conversation

@iamwhatever

@iamwhatever iamwhatever commented Oct 24, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: Make the shell-character walk expand parameters and skip comments the way a shell does, so a deny check sees the command that would actually run.

_iter_shell_chars (src/kiro_crew/security/shell_normalizer.py) models shell quote and
escape handling, but did not handle two additional bash syntax categories:

  • ${...} parameter expansion
  • # comments

A literal ) inside $(echo ${v:-}) or after a # comment arrived with
active=True, so _matching_close_paren counted it as the substitution
closer and truncated the extracted body. The nested-payload extractor then
handed every downstream scan a fragment while bash runs the whole body, so a
nested publish hides below the truncation point -- is_denied allows a command
bash executes (same shape as the documented quoted-paren truncation behind
_matching_close_paren).

Why it matters

The _iter_shell_chars generator is the single source of truth for shell
quote/escape state walked by every consumer: the boundary walk, the git-publish
border walk, the nested-payload extractor, and is_denied/is_sensitive_bash_command.
When the machine did not know about ${...} or #, those constructs were
invisible to it, and a ) inside them was treated as a bare ) -- a
false-allow of the class security reviews have caught before.

Not a goal

  • Not a new deny rule and not a change to the denied-command catalog.
  • Not a general rewrite of the shell normalizer; only the expansion/comment behaviour is corrected.
  • No change to how a verdict is decided once the command text is normalized.

What changed (motivation → approach → change)

Rebuilt on current main, where the shell reader now lives in shell_normalizer_walk.py and the # comment half is already fixed (_opens_comment in _matching_close_paren). Only the ${...} half was still open: on main, $(: ${v:-)}; printf X) still ends the body at : ${v:-.

Main handles comments inside _matching_close_paren, not in _iter_shell_chars, so this fix does the same for ${...}. The walk keeps a stack of open frames. Inside an expansion, quotes count, only a nested unpaired ${ adds a level (a bare { is text, as in bash), and parens and # are the expansion's own text. An unpaired $(, or a <( / >(, opens a new command frame with its own case count. Backtick bodies are stepped over whole. The stack replaces recursion, so deep nesting cannot exhaust the interpreter stack.

Some text cannot be read the same way by every bash: a here-document the walk reaches before the closer (${ there is data; a <<< here-string or a << after the closer does not count), $[ inside an expansion (bash builds disagree), or an unclosed expansion or backtick. For that text the walk falls back to main's paren-only span, so it is read exactly as before. _iter_shell_chars itself is unchanged.

The package line budget in test_security_gate_liveness.py goes up by 132, with a note.

Backwards compatibility

Compatible: substitution bodies only get longer, never shorter, so nothing allowed today becomes refused unless the hidden part of its body holds a refused command.

Removes nothing: no rule, key or function is removed or renamed.

Tests

test/test_security_shell_grammar.py::TestParameterExpansionHoldsItsOwnParens:

  • the body runs to the real closer for ${v:-)}, quoted }, nested ${, a bare { (${v:-{}, ${p//{/}), # inside the expansion, $$[, $${ and \$${, and a } inside $(...), <(...), >(...) or backticks; backtick bodies are opaque;
  • a verb hidden behind ${v:-)} or ${v:-$[ (}) ]} is refused;
  • every AI review repro (bare {, $$[, <(, case around a nested $(, here-documents, ${ in backticks) still refuses the trailing home wipe;
  • kill $(printf %s ${p//{/}); echo $(pgrep ...) stays allowed;
  • 3,000 nested levels return a body without a RecursionError;
  • unreadable text gets exactly the paren-only span;
  • a # <<, <<< x, '<<' or here-document after the closer, or a <<< inside the body, keeps the expansion frame and the refusal.

Every body shape was checked in bash. The 94 security, shell, argv-floor, push-gate and rm-floor test files pass locally (9,647 tests).

Proof: PROVEN test/test_security_shell_grammar.py::TestParameterExpansionHoldsItsOwnParens (head e16b4a4)

Manual verification

N/A — unit coverage sufficient: the change is a pure text walk, and each case was checked against bash.

Related Issues

Fixes #9181. Supersedes #9241: a fork PR cannot pass the Sensitive Change Review gate, so this continues it on a repository branch. Original work by @aniruddhaadak80, kept as co-author.

Sibling sweep

Swept: git grep -n "_matching_close_paren(" -- src/
Sibling: src/kiro_crew/security/shell_normalizer_walk.py:279 -- changed here (via _matching_close_paren)
Sibling: src/kiro_crew/security/argv_floor_self.py:589 -- changed here (calls _matching_close_paren, gets the fix)
Sibling: src/kiro_crew/security/rm_floor.py:96 -- changed here (calls _matching_close_paren, gets the fix)

Pattern harvest

Not generalizable: one missing shell construct in a single span walk; the comment and case rules beside it already cover the other cases.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

Contribution License Agreement

Contributions are accepted under the project's Apache-2.0 license; see the Licensing section of CONTRIBUTING.md.

@iamwhatever
iamwhatever requested review from a team and KiroCrewCaptain as code owners October 24, 2026 01:36
@iamwhatever
iamwhatever requested a review from CrysisDeu October 24, 2026 01:36
@iamwhatever

Copy link
Copy Markdown
Contributor Author

Sensitive change review

Head: 507fbbe

  • What rule changed: _matching_close_paren in shell_normalizer_walk.py now steps over a ${...} parameter expansion to its matching } before counting parens. No deny rule or catalog entry changed.
  • Before -> after: $(: ${v:-)}; printf X) used to yield the body : ${v:-; it now yields the full body, so the commands after the expansion are scanned.
  • Worst case if wrong: a body is read longer than bash runs it, which only over-scans; an unclosed expansion returns the whole remainder (fail closed).
  • Evidence checked: every new test shape was run in bash to confirm the real closer; prove.py shows the new tests fail with the fix reverted.
  • No regression:
    • Tools still work: the 94 security, shell, argv-floor, push-gate and rm-floor test files pass locally (9,620 tests), including benign expansion shapes that stay allowed.
    • Backward compatible: bodies only get longer, so nothing allowed today is refused unless its hidden part holds a refused command.
    • Existing tests: all existing shell grammar, comment and case-pattern span tests pass unchanged.
  • How to undo: revert this single commit; the walk returns to main's behaviour with no data or config to migrate.

@github-actions

github-actions Bot commented Oct 24, 2026 •

Copy link
Copy Markdown
Contributor

Sensitive Change Review

Sensitive files changed:

  • src/kiro_crew/security/shell_normalizer_walk.py

  • @iamwhatever posted a complete reasoning template for 35c1d83.

@github-actions

github-actions Bot commented Oct 24, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ✅ PASS

UX-level review of 35c1d830dd97e84a67e7c38b0161ff98b377c78c — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

UX-Verdict: PASS

Nothing to check. The diff only changes the backend shell-parsing module and its tests, and adds or changes no user-visible control, string or state.

[UX-REVIEWED] 35c1d83

Description read

[DESCRIPTION-READ] 94bac0785ed5ce9df8230f6c36ccd08dfb3488d83dc6992f3731a591c349f5f3

That sha256 names everything this verdict read: the PR description, plus the 7 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Oct 24, 2026
@github-actions

github-actions Bot commented Oct 24, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of 35c1d830dd97e84a67e7c38b0161ff98b377c78c — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

No design-level concerns: the fix models a missing shell grammar construct in the one span walk all three extractors share, and every unreadable case falls back to a span at least as long as main's.

Notes

  • _matching_close_paren now runs up to three walks (strict, paren-only, lenient) and picks a result with a reach side channel. That merge is the part most likely to regress silently. A short docstring table of the outcomes ((strict, paren, lenient) → returned span) would protect it better than the prose paragraph does now.

[DESIGN-REVIEWED] 35c1d83

Description read

[DESCRIPTION-READ] 773b6b73c6331b57cd3ee71936ce99f2346730db039a9a2aeb3b85531c2a34bb

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 24, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ✅ PASS

Premise-level review of 35c1d830dd97e84a67e7c38b0161ff98b377c78c — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

First-Principles-Verdict: PASS

The fix is at mechanism level in the one span function all three extractors call, and #9181 gives it provenance. The fallback machinery is heavier than the extractors' own fail-closed contract needs.

Notes

  • Item 4 — oversized: the strict, paren-only and lenient walks, plus reach/_gave_up, exist only to choose between spans when the expansion walk gives up. The docstring already says "for an extractor the safe reading is the whole remainder (scan more, never less)". Subtraction: when _close_paren_walk(..., True) returns None, return (len(text), False), and delete the lenient parameter, the reach list, _gave_up and the second strict walk.
  • Item 7 — undeclared: the frozen Goal says "Make the shell-character walk expand parameters and skip comments". The diff does neither: "_iter_shell_chars itself is unchanged", and comment handling was already on main. The description also says the budget "goes up by 132", but the hunk sets _PACKAGE_LINE_BUDGET = 32_701 (+175).
Inventory (7 items) — 5 justified

Intent: stop a ) inside ${...} from cutting a $(...) body short, so a deny check scans the whole command bash runs. FIX.

The other ~55 files in the patch (cron time limit, slot create, side chat, i18n, screenshots) are already on base c24c7bc8; this branch's tree does not have them, so they are base drift and are not counted.

  1. A command hidden behind ${v:-)} inside $(...) is now refused — justified
  2. A } inside a nested $(, <(, >( or backtick in an expansion no longer closes the expansion — justified
  3. Text the walk cannot read (here-document, $[, unclosed expansion) keeps the old paren-only span — justified
  4. A lenient re-walk returns the longer span, and a give-up past the paren-only closer returns the whole remainder — oversized
  5. << inside ((...)) is read as a shift, not a here-document — justified
  6. Deep nesting no longer risks a RecursionError — justified
  7. Package line budget raised by 175 — undeclared: the description says 132

[FIRST-PRINCIPLES-REVIEWED] 35c1d83

Description read

[DESCRIPTION-READ] 4812da4bdde4ab40bcc2b47a3a29b0a343b21275631d31a251086631f568493e

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Oct 24, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of 35c1d830dd97e84a67e7c38b0161ff98b377c78c and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 35c1d83

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 35c1d830dd97e84a67e7c38b0161ff98b377c78c false-positive: <reason, 20 to 500 characters>

@github-actions

github-actions Bot commented Oct 24, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — 🔴 changes requested (blocking)

Reviewed 35c1d830dd97e84a67e7c38b0161ff98b377c78c — this comment is updated in place on each push.

The new fallback in _matching_close_paren keeps the short paren-only span whenever a here-document comes before ${v:-)}. That leaves the bypass this change is meant to close still open, and is_denied allows commands that bash runs.

BLOCKING — src/kiro_crew/security/shell_normalizer_walk.py:879

if lenient is None and reach and paren[0] <= reach[0]:
    return (len(text), False)
if lenient is None or lenient == paren:
    return paren

Trigger: T=$(cat <<E\nx\nE\n: ${v:-)} && printf token); kirocrew $T or kill $(cat <<E\nx\nE\n: ${v:-)} && pgrep -f kirocrew). The path is is_denied → _substitution_bodies → _matching_close_paren. The expansion walk and the lenient walk both give up at the << (_opens_heredoc → _gave_up), and that happens before the paren-only closer. So reach[0] < paren[0], and the function returns paren, which closes at the ) inside ${v:-)}.
Outcome: The extracted body is cat <<E\nx\nE\n: ${v:- and is_denied returns None for both commands. The same commands without the here-document are refused (credential-exfil-kirocrew-token and self-protection-kill). Under bash 5.2.21 the here-document ends before the expansion and the printf tok behind it runs; it printed x\ntok.
Anchor: residual/security
Fix: When lenient is None, do not trust a paren-only span that ends after the give-up point. Return (len(text), False) whenever both expansion walks gave up (scan more, never less). The other option is to model the here-document's delimiter line so the walk can resume after it.

[OPUS-REVIEWED] 35c1d83
[BLOCK-MERGE] 35c1d83

Verdict parsed from the review's SHA-scoped output markers for commit 35c1d830dd97e84a67e7c38b0161ff98b377c78c.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 35c1d830dd97e84a67e7c38b0161ff98b377c78c false-positive: <reason, 20 to 500 characters>

@github-actions

github-actions Bot commented Oct 24, 2026 •

Copy link
Copy Markdown
Contributor

Security Scope Review -- ✅ PASS

Which legitimate operations does 35c1d830dd97e84a67e7c38b0161ff98b377c78c newly refuse? Updated in place on each push. A script-confirmed regression blocks; the model proposes candidates and never decides a refusal.

Verdict basis: adjudicated, zero confirmed regressions.

Security scope review -- adjudicated candidates

  • macos-latest (platform posix): 16 classified, 0 newly refused, 14 still allowed.
  • ubuntu-latest (platform posix): 16 classified, 0 newly refused, 14 still allowed.
  • windows-latest (platform windows): 4 classified, 0 newly refused, 3 still allowed.

No candidate flipped from allowed to refused.

Scope-Verdict: PASS

Reason: The change only makes the substitution span longer. A ) inside an unquoted ${...}, a backtick body or a <(...) no longer closes the $(...) body. Where the walk cannot read the text, it falls back to the paren-only span or to the whole remainder. No regression was confirmed. One narrowing is named but not adjudicated: PowerShell backticks inside $(...) can trigger the give-up and fallback path.

TIGHTENING: _matching_close_paren / _close_paren_walk substitution-body span walk. Its output feeds the extractors behind _substitution_bodies, rm_floor and the git-publish boundary walk. -- The one threat it refuses is a hidden verb behind a ) inside a parameter expansion: $(: ${v:-)}; ...) "does not end the body". The hunk is if expansions and ch == "$" and _opens_expansion(text, off): frames.append("{"), plus the return (max(paren[0], lenient[0]), False) and return (len(text), False) fallbacks, which never shorten the span.

ADJUDICATED: 20 candidates, 0 confirmed newly-refused

PLATFORM COVERAGE: The diff assumes POSIX/bash grammar throughout: unquoted ${, backticks as substitutions, << here-documents and ((...)) arithmetic shifts.

  • posix (16 rows): these cover each new boundary on maintainers' daily commands:
    • ${...} before the real closer, with a kirocrew mention after it (kill attribution);
    • nested defaults;
    • a push lease using $(git rev-parse ${REMOTE:-origin}/…);
    • ${...} inside an unquoted and a quoted $(cat <<EOF) here-document;
    • a substitution inside a here-document body;
    • case mixed with an expansion;
    • $(( ${N} )) arithmetic, and a sticky shift followed by a newline;
    • a backtick inside $();
    • a <(...) holding an expansion;
    • ${#...} and $(( )) inside an array subscript;
    • an rm of a temp path.
  • windows (4 rows): PowerShell uses the same ${/$( characters with other meanings, so these cover that spelling: ${env:TEMP} inside $(), ${env:ProgramFiles(x86)} (the common real case of a ) inside ${}), the .venv\Scripts\python.exe pytest subset, and a PowerShell backtick escape after a substitution closer.
  • Not covered: a PowerShell backtick escape INSIDE $(...). I could not attribute that shape to a real daily operation.

UNADJUDICATED: Two readings of the new span walk got no verdict:

  • PowerShell backtick escape inside $(...). Call path: _substitution_bodies → _matching_close_paren → _close_paren_walk. The walk reads an unquoted ` as a backtick substitution, and an odd count makes _backtick_closer fail, so it gives up. It falls back to the paren-only span; the whole remainder applies only when the give-up comes after the paren-only closer. Candidate I would have submitted: Write-Output $(git log -1 --format=%ht%s); kirocrew status` (windows).
  • shift flag never resets. Once an arithmetic << sets shift, it stays set, so a later multi-line $(( ... )) gives up on its newline. Candidate I would have submitted: x=$((1<<2)); y=$(( 1 +\n 2 )).

Both are arguable narrowings that I cannot confirm, so they do not block.

[SCOPE-REVIEWED] 35c1d83

@iamwhatever

Copy link
Copy Markdown
Contributor Author

Sensitive change review

Head: 3b0feba

  • What rule changed: _matching_close_paren in shell_normalizer_walk.py keeps a stack of paren and ${...} frames, so it steps over a parameter expansion before counting parens. No deny rule or catalog entry changed.
  • Before -> after: $(: ${v:-)}; printf X) used to yield the body : ${v:-; it now yields the full body, so the commands after the expansion are scanned.
  • Worst case if wrong: a body is read longer or shorter than bash runs it; every bash-measured shape is pinned in tests, and an unclosed expansion returns the whole remainder (fail closed).
  • Evidence checked: every new test shape was run in bash; prove.py shows the new tests fail with the fix reverted; both AI review repros are pinned as tests.
  • No regression:
    • Tools still work: the 94 security, shell, argv-floor, push-gate and rm-floor test files pass locally (9,626 tests), including benign shapes that stay allowed.
    • Backward compatible: bash-faithful spans only; a bare { is text, so ${p//{/} keeps its normal span.
    • Existing tests: all existing shell grammar, comment and case-pattern span tests pass unchanged.
  • How to undo: revert this single commit; the walk returns to main's behaviour with no data or config to migrate.

@iamwhatever

Copy link
Copy Markdown
Contributor Author

Review findings on 507fbbe, fixed in 3b0feba:

  • Opus / GPT: bare { counted as a nesting level — fixed. Only a nested ${ adds a level now; bash closes ${v:-{} at its first } and reads ${p//{/} as a literal pattern. Pinned: the rm -rf "$HOME" after ${v:-{} stays refused, and the ${p//{/} self-kill shape stays allowed.
  • GPT: mutual recursion on deep nesting — fixed. _matching_close_paren keeps an explicit frame stack; _matching_close_brace is gone. Pinned: 3,000 nested ${v:-$( levels return a body. (On base, the same 600-level command already raises RecursionError further down in is_denied; this change adds no recursion.)

@iamwhatever

Copy link
Copy Markdown
Contributor Author

Sensitive change review

Head: 04f3771

  • What rule changed: _matching_close_paren in shell_normalizer_walk.py keeps a stack of paren, ${...} and $[...] frames, so it steps over a parameter expansion before counting parens. No deny rule or catalog entry changed.
  • Before -> after: $(: ${v:-)}; printf X) used to yield the body : ${v:-; it now yields the full body, so the commands after the expansion are scanned.
  • Worst case if wrong: a body is read longer or shorter than bash runs it; every bash-measured shape is pinned in tests, and an unclosed expansion returns the whole remainder (fail closed).
  • Evidence checked: every new test shape was run in bash; prove.py shows the new tests fail with the fix reverted; all three AI review repros are pinned as tests.
  • No regression:
    • Tools still work: the 94 security, shell, argv-floor, push-gate and rm-floor test files pass locally (9,630 tests), including benign shapes that stay allowed.
    • Backward compatible: spans follow bash; a bare { is text, so ${p//{/} keeps its normal span.
    • Existing tests: all existing shell grammar, comment and case-pattern span tests pass unchanged.
  • How to undo: revert this single commit; the walk returns to main's behaviour with no data or config to migrate.

@iamwhatever

Copy link
Copy Markdown
Contributor Author

Opus finding on 3b0feba (} inside $[...] closes the expansion early) — fixed in 04f3771. Inside an expansion, $[ now opens an arithmetic frame that closes on ] and counts nested [, so ${v:-$[ (}) ]} runs to its last brace. Pinned: the body tests for $[ (}) ], $[ [}] ] and $[1+1], and the printf token shape behind $[ (}) ] stays refused.

@iamwhatever

Copy link
Copy Markdown
Contributor Author

Sensitive change review

Head: 8352eac

  • What rule changed: _matching_close_paren in shell_normalizer_walk.py keeps a stack of paren, ${...} and $[...] frames, so it steps over a parameter expansion before counting parens. No deny rule or catalog entry changed.
  • Before -> after: $(: ${v:-)}; printf X) used to yield the body : ${v:-; it now yields the full body, so the commands after the expansion are scanned.
  • Worst case if wrong: a body is read longer or shorter than bash runs it; every bash-measured shape is pinned in tests, and an unclosed expansion returns the whole remainder (fail closed).
  • Evidence checked: every new test shape was run in bash 5.2; prove.py shows the new tests fail with the fix reverted; every AI review repro so far is pinned as a test.
  • No regression:
    • Tools still work: the 94 security, shell, argv-floor, push-gate and rm-floor test files pass locally (9,635 tests), including benign shapes that stay allowed.
    • Backward compatible: spans follow bash; a bare { is text and $$ pairs off, so ordinary expansions keep their normal span.
    • Existing tests: all existing shell grammar, comment and case-pattern span tests pass unchanged.
  • How to undo: revert this single commit; the walk returns to main's behaviour with no data or config to migrate.

@iamwhatever

Copy link
Copy Markdown
Contributor Author

GPT finding on 04f3771 — fixed in 8352eac:

  • $$[ / $$( inside an expansion: frames now open only on an unpaired $ (the same pairing rule as ${), so ${v:-$$[} closes at its } as in bash.
  • <( / >( inside an expansion now open a paren frame, so the } in ${v:-<(: })} cannot close it.

Both repro commands are pinned in test_a_misread_frame_cannot_hide_a_home_wipe and are refused.

@iamwhatever

Copy link
Copy Markdown
Contributor Author

Sensitive change review

Head: 55c206d

  • What rule changed: _matching_close_paren in shell_normalizer_walk.py keeps a stack of paren and ${...} frames, so it steps over a parameter expansion before counting parens, and falls back to main's paren-only span where an expansion cannot be read. No deny rule or catalog entry changed.
  • Before -> after: $(: ${v:-)}; printf X) used to yield the body : ${v:-; it now yields the full body, so the commands after the expansion are scanned.
  • Worst case if wrong: a body is read longer or shorter than bash runs it; every bash-measured shape and every review repro is pinned, and unreadable text (here-documents, $[ inside an expansion, unclosed spans) gets exactly main's span.
  • Evidence checked: every new test shape was run in bash 5.2; prove.py shows the new tests fail with the fix reverted.
  • No regression:
    • Tools still work: the 94 security, shell, argv-floor, push-gate and rm-floor test files pass locally (9,642 tests), including benign shapes that stay allowed.
    • Backward compatible: spans follow bash; anything the walk cannot read falls back to main's paren-only span.
    • Existing tests: all existing shell grammar, comment and case-pattern span tests pass unchanged.
  • How to undo: revert this single commit; the walk returns to main's behaviour with no data or config to migrate.

@iamwhatever

Copy link
Copy Markdown
Contributor Author

Findings on 8352eac, fixed in 55c206d:

  • GPT + Opus: the case count swallows the ) of a nested $( inside ${...}. A substitution opened inside an expansion now starts with its own case count and restores the outer one when it closes.
  • Opus: ${ inside a here-document opens a frame. Any << in the text turns expansion frames off, so here-document text gets main's paren-only span.
  • Also closed: a ${ inside a backtick body (backtick bodies are now opaque, using the bash-faithful _backtick_closer), and $[ inside an expansion, where bash builds disagree; it falls back to main's span. Any unclosed expansion or backtick falls back the same way, so no new unproven span reaches rm_floor.

All the repro commands are pinned in test_a_misread_frame_cannot_hide_a_home_wipe and are refused.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 24, 2026
@iamwhatever

Copy link
Copy Markdown
Contributor Author

Sensitive change review

Head: e16b4a4

  • What rule changed: _matching_close_paren in shell_normalizer_walk.py keeps a stack of paren and ${...} frames, so it steps over a parameter expansion before counting parens, and falls back to main's paren-only span where an expansion cannot be read. No deny rule or catalog entry changed.
  • Before -> after: $(: ${v:-)}; printf X) used to yield the body : ${v:-; it now yields the full body, so the commands after the expansion are scanned.
  • Worst case if wrong: a body is read longer or shorter than bash runs it; every bash-measured shape and every review repro is pinned, and unreadable text (a here-document before the closer, $[ inside an expansion, unclosed spans) gets exactly main's span.
  • Evidence checked: every new test shape was run in bash 5.2; prove.py shows the new tests fail with the fix reverted.
  • No regression:
    • Tools still work: the 94 security, shell, argv-floor, push-gate and rm-floor test files pass locally (9,647 tests), including benign shapes that stay allowed.
    • Backward compatible: spans follow bash; anything the walk cannot read falls back to main's paren-only span.
    • Existing tests: all existing shell grammar, comment and case-pattern span tests pass unchanged.
  • How to undo: revert this single commit; the walk returns to main's behaviour with no data or config to migrate.

@iamwhatever

Copy link
Copy Markdown
Contributor Author

Opus finding on 55c206d (any later << text turned expansion frames off) — fixed in e16b4a4. The substring check is gone. The walk now falls back only when it reaches an active << here-document operator (not <<<) before the closer (_opens_heredoc). Pinned: a trailing # <<, <<< x, '<<' or here-document, and a <<< inside the body, all keep the frame and the refusal.

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Oct 24, 2026

@cruisercohen cruisercohen left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed as security scope (command-floor / shell-parsing domain), comment only — I'm not approving. I found no blocking problem; the gap closes and the change is fail-closed.

Gap confirmed real. I reproduced the documented truncation independently. For $(: ${v:-)}; printf <verb>) the extracted substitution body is : ${v:- on current main (the ) inside the default word is read as the closer), so a self-command assembled from the part after it is never scanned and is_denied returns None. On this head the body runs to the real closer and the same command is refused. Same result for the measured $$[, <(…) / >(…), a nested $(…) and a case-wrapped substitution: allowed on main, refused here.

Fail-closed / backward-compatible. Substitution bodies only get longer, never shorter, so nothing allowed today becomes refused unless the hidden part of its body holds a command this module already refuses; I saw no benign over-refusal in the suites. Where a span can't be read the way every bash reads it (a here-document body, $[ inside an expansion, an unclosed expansion/backtick) the walk returns the paren-only span exactly as before — I checked that bash, zsh and ksh genuinely disagree on these constructs, so falling back there (scan more, never less) is the right call for an extractor.

Tests have teeth. The 140 test_security_shell_grammar.py cases pass, including the new TestParameterExpansionHoldsItsOwnParens class; the broader security/shell/argv-floor/push-gate/rm-floor set passes (14,251 passed, 889 skipped on the merge with current main). The one failure I saw (test_native_skill_projection) fails identically on main without this PR, so it's environmental here.

Final sign-off is a maintainer's.

chenmingwei23
chenmingwei23 previously approved these changes Oct 24, 2026
@github-actions github-actions Bot added the merge conflict Branch has merge conflicts with its base — author must resolve before merge label Oct 24, 2026
…on closer

A ) inside a parameter expansion such as ${v:-)} is the default word,
not a closer, but the span walk closed the substitution body there and
the commands after it were never scanned. _matching_close_paren now
keeps a stack of paren and expansion frames. Inside an expansion,
quoting runs through the span, only a nested ${ adds a level (a bare {
is text, as in bash), and a } inside a nested $(...) or backtick body
belongs to that command. The stack replaces recursion, so deep nesting
cannot exhaust the interpreter stack. An unclosed expansion fails
closed.

Fixes #9181

Co-authored-by: Aniruddha Adak <aniruddhaadak80@users.noreply.github.com>
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: passed Eligible automated validation passed for the current revision merge conflict Branch has merge conflicts with its base — author must resolve before merge labels Oct 24, 2026
Opus finding on eebaf2d: a harmless arithmetic `<<` or a `$[` in front
of `${v:-)}` sent the walk to the paren-only span, which closes at the
`)` inside the expansion and hides the commands after it.

- A `<<` inside `((...))` arithmetic is a shift, not a here-document,
  unless a newline follows it before the arithmetic closes.
- Where the strict walk gives up, a lenient walk (`$[` as plain text)
  runs too; if it disagrees with the paren-only span, the longer span
  is returned unproven.
@iamwhatever

Copy link
Copy Markdown
Contributor Author

Opus finding on eebaf2d (a harmless << or $[ before ${v:-)} brings back the short span), fixed in 6c11380:

  • << inside ((...)) / $((...)) arithmetic is now read as a shift, so the expansion frame stays on. If a newline follows it before the arithmetic closes, it may be a real here-document, so the walk still falls back.
  • When the strict walk gives up, a lenient walk ($[ as plain text) also runs. If it closes somewhere other than the paren-only span, the longer span is returned unproven, per your suggestion.
  • I kept the paren-only fallback for real here-documents: returning the whole remainder there lets rm_floor stop at the << and miss the pinned rm -rf "$HOME" after it.

Pinned: both repros and the kill form in test_an_unreadable_prefix_cannot_shorten_the_body, plus a $((cat <<E home-wipe shape.

@iamwhatever

Copy link
Copy Markdown
Contributor Author

Sensitive change review

  • Head commit: 6c11380
  • What rule changed: _matching_close_paren in shell_normalizer_walk.py steps over ${...} expansions when it finds the ) that ends a substitution. New on this head: a << inside ((...)) arithmetic is a shift, not a here-document (unless a newline follows it before the arithmetic closes). Where the strict walk gives up, a lenient walk ($[ as plain text) also runs; if it disagrees with the paren-only span, the longer span is returned unproven. No deny rule or catalog entry changed.
  • Before -> after (what was allowed/blocked, now): both Opus repros on eebaf2d (an arithmetic shift prefix, and a $[ inside the expansion) were allowed; both are refused now, in the credential and self-kill forms. A benign arithmetic shift next to an expansion stays allowed.
  • Worst case if wrong (breaks commands / leaks secret / locks user out): a substitution body is read longer than bash runs it, so a few more words get scanned. The real here-document fallback (pinned home-wipe shapes) is unchanged.
  • Evidence checked (which sandbox result proves it): the new test_an_unreadable_prefix_cannot_shorten_the_body cases fail with the walk reverted to eebaf2d and pass on this head; a new home-wipe pin covers an arithmetic opener followed by a real here-document.
  • No regression:
    • Tools still work (which tools/commands were run, result): test/test_security_*.py plus argv-floor and pre-push-gate files: 5,817 passed, 2 skipped, on this head.
    • Backward compatible (old config / old data / old callers still work, how checked): spans only; the three callers (_substitution_bodies, rm_floor, argv_floor_self) already treat an unproven span as fail-closed. No config or data.
    • Existing tests (which suites passed on this head): shell grammar, gate liveness (budget raised by 26 to 32,684) and the security suites above.
  • How to undo: revert 6c11380; the walk goes back to eebaf2d behaviour.

Opus finding on 6c11380: a here-document AFTER `${v:-)}` made both
expansion walks give up, and the paren-only span closed inside the
expansion. The walk now records where it gave up; when the paren-only
closer comes before that point, the span is unproven to the end of the
text.
@iamwhatever

Copy link
Copy Markdown
Contributor Author

Sensitive change review

  • Head commit: 35c1d83
  • What rule changed: _matching_close_paren in shell_normalizer_walk.py steps over ${...} expansions when it finds the ) that ends a substitution. Where the expansion walk gives up (here-document, unclosed backtick), it now records the offset; if the paren-only closer comes before that offset, the span is unproven to the end of the text. Earlier on this PR: an arithmetic << is a shift, and a lenient walk keeps the longer span for $[. No deny rule or catalog entry changed.
  • Before -> after (what was allowed/blocked, now): the Opus repro on 6c11380 (a here-document placed after the expansion) was allowed in the credential and self-kill forms; both are refused now. The pinned here-document-before-expansion home-wipe shapes keep the paren-only span and stay refused.
  • Worst case if wrong (breaks commands / leaks secret / locks user out): a substitution body is read longer than bash runs it, so more words get scanned.
  • Evidence checked (which sandbox result proves it): the three new test_an_unreadable_suffix_cannot_shorten_the_body cases fail on 6c11380 and pass on this head.
  • No regression:
    • Tools still work (which tools/commands were run, result): test/test_security_*.py plus argv-floor and pre-push-gate files: 5,820 passed, 2 skipped, on this head.
    • Backward compatible (old config / old data / old callers still work, how checked): spans only; all three callers already treat an unproven span as fail-closed. No config or data.
    • Existing tests (which suites passed on this head): shell grammar, gate liveness (budget 32,701) and the security suites above.
  • How to undo: revert 35c1d83 and 6c11380; the walk goes back to eebaf2d behaviour.

Opus note on the double-quoted nested substitution inside an expansion: same behaviour on main, so left for a follow-up.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention merge conflict Branch has merge conflicts with its base — author must resolve before merge and removed readiness: checking Automated validation is still running labels Oct 24, 2026
@bolichen97

Copy link
Copy Markdown
Contributor

[operator: bolichen97] Continuing this in #30736; thanks for the start.

@iamwhatever

Copy link
Copy Markdown
Contributor Author

Closing as superseded: this work continued in #30736, which has merged (per bolichen97's note). Thank you!

@iamwhatever iamwhatever closed this Nov 5, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

merge conflict Branch has merge conflicts with its base — author must resolve before merge readiness: action required A blocking check or review needs attention

Projects

None yet

Development

Successfully merging this pull request may close these issues.

security: _iter_shell_chars does not model parameter expansion or comments, so a literal ) inside them truncates every substitution span

4 participants