Repository navigation
fix(security): step over ${...} expansions when matching a substitution closer - #23620
iamwhatever wants to merge 3 commits into
Conversation
Sensitive change reviewHead: 507fbbe
|
Sensitive Change ReviewSensitive files changed:
|
UX Review (Opus 5.5) — ✅ PASSUX-level review of UX-Verdict: PASS Nothing to check. The diff only changes the backend shell-parsing module and its tests, and adds or changes no user-visible control, string or state. [UX-REVIEWED] 35c1d83 Description read[DESCRIPTION-READ] 94bac0785ed5ce9df8230f6c36ccd08dfb3488d83dc6992f3731a591c349f5f3 That sha256 names everything this verdict read: the PR description, plus the 7 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
Design Review (Opus 5.5) — ✅ PASSDesign-level review of Design-Verdict: PASS No design-level concerns: the fix models a missing shell grammar construct in the one span walk all three extractors share, and every unreadable case falls back to a span at least as long as main's. Notes
[DESIGN-REVIEWED] 35c1d83 Description read[DESCRIPTION-READ] 773b6b73c6331b57cd3ee71936ce99f2346730db039a9a2aeb3b85531c2a34bb That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
First Principles Review (Opus 5.5) — ✅ PASSPremise-level review of First-Principles-Verdict: PASS The fix is at mechanism level in the one span function all three extractors call, and #9181 gives it provenance. The fallback machinery is heavier than the extractors' own fail-closed contract needs. Notes
Inventory (7 items) — 5 justifiedIntent: stop a The other ~55 files in the patch (cron time limit, slot create, side chat, i18n, screenshots) are already on base
[FIRST-PRINCIPLES-REVIEWED] 35c1d83 Description read[DESCRIPTION-READ] 4812da4bdde4ab40bcc2b47a3a29b0a343b21275631d31a251086631f568493e That sha256 names the PR description this verdict read. Recompute it with |
GPT 6.1 Review — ✅ no blocking findingsGPT 6.1 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
Opus 5.5 Review — 🔴 changes requested (blocking)Reviewed The new fallback in BLOCKING — src/kiro_crew/security/shell_normalizer_walk.py:879 Trigger: [OPUS-REVIEWED] 35c1d83 Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
Security Scope Review -- ✅ PASSWhich legitimate operations does Verdict basis: adjudicated, zero confirmed regressions. Security scope review -- adjudicated candidates
No candidate flipped from allowed to refused. Scope-Verdict: PASS Reason: The change only makes the substitution span longer. A TIGHTENING: ADJUDICATED: 20 candidates, 0 confirmed newly-refused PLATFORM COVERAGE: The diff assumes POSIX/bash grammar throughout: unquoted
UNADJUDICATED: Two readings of the new span walk got no verdict:
Both are arguable narrowings that I cannot confirm, so they do not block. [SCOPE-REVIEWED] 35c1d83 |
507fbbe to
3b0feba
Compare
Sensitive change reviewHead: 3b0feba
|
|
Review findings on 507fbbe, fixed in 3b0feba:
|
3b0feba to
04f3771
Compare
Sensitive change reviewHead: 04f3771
|
|
Opus finding on 3b0feba ( |
04f3771 to
8352eac
Compare
Sensitive change reviewHead: 8352eac
|
|
GPT finding on 04f3771 — fixed in 8352eac:
Both repro commands are pinned in |
8352eac to
55c206d
Compare
Sensitive change reviewHead: 55c206d
|
|
Findings on 8352eac, fixed in 55c206d:
All the repro commands are pinned in |
55c206d to
e16b4a4
Compare
Sensitive change reviewHead: e16b4a4
|
|
Opus finding on 55c206d (any later |
cruisercohen
left a comment
There was a problem hiding this comment.
Reviewed as security scope (command-floor / shell-parsing domain), comment only — I'm not approving. I found no blocking problem; the gap closes and the change is fail-closed.
Gap confirmed real. I reproduced the documented truncation independently. For $(: ${v:-)}; printf <verb>) the extracted substitution body is : ${v:- on current main (the ) inside the default word is read as the closer), so a self-command assembled from the part after it is never scanned and is_denied returns None. On this head the body runs to the real closer and the same command is refused. Same result for the measured $$[, <(…) / >(…), a nested $(…) and a case-wrapped substitution: allowed on main, refused here.
Fail-closed / backward-compatible. Substitution bodies only get longer, never shorter, so nothing allowed today becomes refused unless the hidden part of its body holds a command this module already refuses; I saw no benign over-refusal in the suites. Where a span can't be read the way every bash reads it (a here-document body, $[ inside an expansion, an unclosed expansion/backtick) the walk returns the paren-only span exactly as before — I checked that bash, zsh and ksh genuinely disagree on these constructs, so falling back there (scan more, never less) is the right call for an extractor.
Tests have teeth. The 140 test_security_shell_grammar.py cases pass, including the new TestParameterExpansionHoldsItsOwnParens class; the broader security/shell/argv-floor/push-gate/rm-floor set passes (14,251 passed, 889 skipped on the merge with current main). The one failure I saw (test_native_skill_projection) fails identically on main without this PR, so it's environmental here.
Final sign-off is a maintainer's.
…on closer
A ) inside a parameter expansion such as ${v:-)} is the default word,
not a closer, but the span walk closed the substitution body there and
the commands after it were never scanned. _matching_close_paren now
keeps a stack of paren and expansion frames. Inside an expansion,
quoting runs through the span, only a nested ${ adds a level (a bare {
is text, as in bash), and a } inside a nested $(...) or backtick body
belongs to that command. The stack replaces recursion, so deep nesting
cannot exhaust the interpreter stack. An unclosed expansion fails
closed.
Fixes #9181
Co-authored-by: Aniruddha Adak <aniruddhaadak80@users.noreply.github.com>
e16b4a4 to
eebaf2d
Compare
Opus finding on eebaf2d: a harmless arithmetic `<<` or a `$[` in front of `${v:-)}` sent the walk to the paren-only span, which closes at the `)` inside the expansion and hides the commands after it. - A `<<` inside `((...))` arithmetic is a shift, not a here-document, unless a newline follows it before the arithmetic closes. - Where the strict walk gives up, a lenient walk (`$[` as plain text) runs too; if it disagrees with the paren-only span, the longer span is returned unproven.
|
Opus finding on eebaf2d (a harmless
Pinned: both repros and the |
Sensitive change review
|
Opus finding on 6c11380: a here-document AFTER `${v:-)}` made both expansion walks give up, and the paren-only span closed inside the expansion. The walk now records where it gave up; when the paren-only closer comes before that point, the span is unproven to the end of the text.
Sensitive change review
Opus note on the double-quoted nested substitution inside an expansion: same behaviour on main, so left for a follow-up. |
|
[operator: bolichen97] Continuing this in #30736; thanks for the start. |
|
Closing as superseded: this work continued in #30736, which has merged (per bolichen97's note). Thank you! |
Problem / Motivation
Goal: Make the shell-character walk expand parameters and skip comments the way a shell does, so a deny check sees the command that would actually run.
_iter_shell_chars(src/kiro_crew/security/shell_normalizer.py) models shell quote andescape handling, but did not handle two additional bash syntax categories:
${...}parameter expansion#commentsA literal
)inside$(echo ${v:-})or after a#comment arrived withactive=True, so_matching_close_parencounted it as the substitutioncloser and truncated the extracted body. The nested-payload extractor then
handed every downstream scan a fragment while bash runs the whole body, so a
nested publish hides below the truncation point --
is_deniedallows a commandbash executes (same shape as the documented quoted-paren truncation behind
_matching_close_paren).Why it matters
The
_iter_shell_charsgenerator is the single source of truth for shellquote/escape state walked by every consumer: the boundary walk, the git-publish
border walk, the nested-payload extractor, and
is_denied/is_sensitive_bash_command.When the machine did not know about
${...}or#, those constructs wereinvisible to it, and a
)inside them was treated as a bare)-- afalse-allow of the class security reviews have caught before.
Not a goal
What changed (motivation → approach → change)
Rebuilt on current main, where the shell reader now lives in
shell_normalizer_walk.pyand the#comment half is already fixed (_opens_commentin_matching_close_paren). Only the${...}half was still open: on main,$(: ${v:-)}; printf X)still ends the body at: ${v:-.Main handles comments inside
_matching_close_paren, not in_iter_shell_chars, so this fix does the same for${...}. The walk keeps a stack of open frames. Inside an expansion, quotes count, only a nested unpaired${adds a level (a bare{is text, as in bash), and parens and#are the expansion's own text. An unpaired$(, or a<(/>(, opens a new command frame with its owncasecount. Backtick bodies are stepped over whole. The stack replaces recursion, so deep nesting cannot exhaust the interpreter stack.Some text cannot be read the same way by every bash: a here-document the walk reaches before the closer (
${there is data; a<<<here-string or a<<after the closer does not count),$[inside an expansion (bash builds disagree), or an unclosed expansion or backtick. For that text the walk falls back to main's paren-only span, so it is read exactly as before._iter_shell_charsitself is unchanged.The package line budget in
test_security_gate_liveness.pygoes up by 132, with a note.Backwards compatibility
Compatible: substitution bodies only get longer, never shorter, so nothing allowed today becomes refused unless the hidden part of its body holds a refused command.
Removes nothing: no rule, key or function is removed or renamed.
Tests
test/test_security_shell_grammar.py::TestParameterExpansionHoldsItsOwnParens:${v:-)}, quoted}, nested${, a bare{(${v:-{},${p//{/}),#inside the expansion,$$[,$${and\$${, and a}inside$(...),<(...),>(...)or backticks; backtick bodies are opaque;${v:-)}or${v:-$[ (}) ]}is refused;{,$$[,<(,casearound a nested$(, here-documents,${in backticks) still refuses the trailing home wipe;kill $(printf %s ${p//{/}); echo $(pgrep ...)stays allowed;# <<,<<< x,'<<'or here-document after the closer, or a<<<inside the body, keeps the expansion frame and the refusal.Every body shape was checked in bash. The 94 security, shell, argv-floor, push-gate and rm-floor test files pass locally (9,647 tests).
Proof: PROVEN test/test_security_shell_grammar.py::TestParameterExpansionHoldsItsOwnParens (head e16b4a4)
Manual verification
N/A — unit coverage sufficient: the change is a pure text walk, and each case was checked against bash.
Related Issues
Fixes #9181. Supersedes #9241: a fork PR cannot pass the Sensitive Change Review gate, so this continues it on a repository branch. Original work by @aniruddhaadak80, kept as co-author.
Sibling sweep
Swept: git grep -n "_matching_close_paren(" -- src/
Sibling: src/kiro_crew/security/shell_normalizer_walk.py:279 -- changed here (via
_matching_close_paren)Sibling: src/kiro_crew/security/argv_floor_self.py:589 -- changed here (calls
_matching_close_paren, gets the fix)Sibling: src/kiro_crew/security/rm_floor.py:96 -- changed here (calls
_matching_close_paren, gets the fix)Pattern harvest
Not generalizable: one missing shell construct in a single span walk; the comment and case rules beside it already cover the other cases.
Checklist
Contribution License Agreement
Contributions are accepted under the project's Apache-2.0 license; see the Licensing section of CONTRIBUTING.md.