Repository navigation
fix(security): read substitution bodies from the folded source (#9705) - #11134
Conversation
… payload walk The shell removes a backslash-newline while reading, before it lexes a substitution opener, so `cat <\<newline>(...)` (and the `>(` / `$(` spellings) is a substitution to bash. `_substitution_bodies` recognises its openers byte-literally and the payload walk handed it the RAW source, so with a continuation inside the opener no body was extracted and the inner command was never scanned -- a wrapper body such as `bash -c '...'` was allowed while bash ran it. The walk now reads the bodies from the same quote-aware `_fold_line_continuations` view `_self_tokens` already tokenizes, so the token and body views agree. Folding at the body walk rather than the walk's seed reshapes nothing outside a substitution, and the fold keeps single-quoted and ANSI-C spans literal, so the benign cases (real newline, `'...'`, `$'...'`, backslash-CRLF) keep their verdicts. Regression tests: three openers x (split opener, split program, split verb), a computed-verb body, a nested split opener, CRLF, real newline, single-quoted and ANSI-C literals, and the ANSI-C publish literal the issue measured flipping under a seed-level fold. Closes #9705
Design Review (Fable 5) — ✅ PASSDesign-level review of Design-Verdict: PASS Nothing to check. [DESIGN-REVIEWED] f315810 |
First Principles Review (Fable 5) — ✅ PASSPremise-level review of First-Principles-Verdict: PASS Confirm the split-opener tests fail on base — the "8 of 29 fail with the fix reverted" mutation claim is the defect's only runnable provenance. What this change shipsInventory (3 items) — 3 justifiedIntent: FIX — close #9705's remaining seam: a backslash-newline inside
Reuses existing [FIRST-PRINCIPLES-REVIEWED] f315810 |
Opus 4.8 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
GPT 5.6 Review — ✅ no blocking findingsGPT 5.6 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
Security Scope Review -- ✅ PASSWhich legitimate operations does Verdict basis: adjudicated, zero confirmed regressions. Security scope review -- adjudicated candidates
No candidate flipped from allowed to refused. Scope-Verdict: PASS TIGHTENING: ADJUDICATED: 17 candidates, 0 confirmed newly-refused PLATFORM COVERAGE: posix (13 rows) -- the fold's core assumption is bash's backslash-LF removal, so the posix rows probe every place a legitimate continuation lands: wrapped daily operations from the touched area's own corpus (gh pr view, the sanctioned pytest invocation, flake8, the lease push, eslint, push_guard), the wrap landing inside a [SCOPE-REVIEWED] f315810 |
iamwhatever
left a comment
There was a problem hiding this comment.
Approved after a full read of the diff on the current head: the change matches the linked issue and body, every load-bearing claim was spot-checked against main, tests pin the behaviour, all review lanes are clean and nothing is left undisposed on this head.
…otdev#9705) (kirodotdev#11134) * fix(security): read substitution bodies from the folded source in the payload walk The shell removes a backslash-newline while reading, before it lexes a substitution opener, so `cat <\<newline>(...)` (and the `>(` / `$(` spellings) is a substitution to bash. `_substitution_bodies` recognises its openers byte-literally and the payload walk handed it the RAW source, so with a continuation inside the opener no body was extracted and the inner command was never scanned -- a wrapper body such as `bash -c '...'` was allowed while bash ran it. The walk now reads the bodies from the same quote-aware `_fold_line_continuations` view `_self_tokens` already tokenizes, so the token and body views agree. Folding at the body walk rather than the walk's seed reshapes nothing outside a substitution, and the fold keeps single-quoted and ANSI-C spans literal, so the benign cases (real newline, `'...'`, `$'...'`, backslash-CRLF) keep their verdicts. Regression tests: three openers x (split opener, split program, split verb), a computed-verb body, a nested split opener, CRLF, real newline, single-quoted and ANSI-C literals, and the ANSI-C publish literal the issue measured flipping under a seed-level fold. Closes kirodotdev#9705 * test(security): state the folded-opener docstrings in present tense
What
_shell_payload_walknow extracts substitution bodies from the folded source (_fold_line_continuations, the same quote-aware view_self_tokensalready tokenizes) instead of the raw command text. One-line code change plus a comment, and a regression class intest/test_security.py.Why
A shell removes a backslash-newline while reading, before it lexes a substitution opener, so a continuation placed inside
<(,>(or$(still opens a substitution and bash runs the body._substitution_bodiesrecognises its openers byte-literally and the payload walk handed it the raw text, so for that spelling no body was extracted and the inner command was never scanned. Measured on current main: a wrapper body (bash -c '...', an inline interpreter import, or an assignment-then-invoke body) inside a continuation-split opener was allowed for all three parenthesised openers while bash ran it; the unsplit spelling was denied. The split-program and split-verb spellings named in #9705 are already covered on main since #9704 folded the tokenizer input; this closes the remaining opener seam and pins the whole matrix.How
Fold at the body walk, not at the walk's seed. Folding the seed reshapes the whole command and was measured (in the issue) flipping a benign ANSI-C literal; folding only the text handed to
_substitution_bodiescannot touch anything outside a substitution the tokenizer already reads folded, and_fold_line_continuationspreserves single-quoted and ANSI-C spans and does not fold backslash-CRLF (bash escapes the CR and the LF ends the line). No change to_substitution_bodiesitself, its token-fed callers, or_self_tokens.Tests
test/test_security.py::TestSubstitutionBodiesReadFoldedOpeners(29 cases):Mutation check: with the one-line fix reverted, 8 of the 29 fail (the split-opener, computed-verb and nested cases); with it, 29 pass. Full
test/test_security.py: 1016 passed. Diff-scoped black gate passes.Pattern harvest
Rule candidate: review
Pattern: two views of one command text built from different pre-lex states (one folded, one raw). Any helper that recognises shell syntax byte-literally must be fed the same folded text the tokenizer reads; when the same source string is passed to both
_self_tokensand a byte-literal scanner, the scanner sees a spelling bash never lexes.Other
Bash measurement used to pin expectations:
cat <\<newline>(echo hi)printshi;>\<newline>(and$\<newline>(behave the same;\<CR><LF>is not folded and the CR-terminated word runs as its own command.Closes #9705