Skip to content

fix(security): read substitution bodies from the folded source (#9705) - #11134

Merged
iamwhatever merged 2 commits into
mainfrom
fix/procsub-continuation-9705
Sep 16, 2026
Merged

iamwhatever merged 2 commits into
mainfrom
fix/procsub-continuation-9705

Conversation

@CrysisDeu

@CrysisDeu CrysisDeu commented Sep 15, 2026 •

Copy link
Copy Markdown
Contributor

What

_shell_payload_walk now extracts substitution bodies from the folded source (_fold_line_continuations, the same quote-aware view _self_tokens already tokenizes) instead of the raw command text. One-line code change plus a comment, and a regression class in test/test_security.py.

Why

A shell removes a backslash-newline while reading, before it lexes a substitution opener, so a continuation placed inside <(, >( or $( still opens a substitution and bash runs the body. _substitution_bodies recognises its openers byte-literally and the payload walk handed it the raw text, so for that spelling no body was extracted and the inner command was never scanned. Measured on current main: a wrapper body (bash -c '...', an inline interpreter import, or an assignment-then-invoke body) inside a continuation-split opener was allowed for all three parenthesised openers while bash ran it; the unsplit spelling was denied. The split-program and split-verb spellings named in #9705 are already covered on main since #9704 folded the tokenizer input; this closes the remaining opener seam and pins the whole matrix.

How

Fold at the body walk, not at the walk's seed. Folding the seed reshapes the whole command and was measured (in the issue) flipping a benign ANSI-C literal; folding only the text handed to _substitution_bodies cannot touch anything outside a substitution the tokenizer already reads folded, and _fold_line_continuations preserves single-quoted and ANSI-C spans and does not fold backslash-CRLF (bash escapes the CR and the LF ends the line). No change to _substitution_bodies itself, its token-fed callers, or _self_tokens.

Tests

test/test_security.py::TestSubstitutionBodiesReadFoldedOpeners (29 cases):

  • three openers x (split opener, split program, split verb), each against its unsplit twin
  • split opener around a computed-verb body; split opener nested inside another substitution
  • the payload walk itself yields the body frame for a split opener
  • backslash-CRLF in the opener and in the body is not folded (verdict matches the real-newline spelling; a mint on the following line stays denied)
  • real newline, single-quoted and ANSI-C continuations inside a body stay literal
  • the ANSI-C publish literal from the issue stays allowed

Mutation check: with the one-line fix reverted, 8 of the 29 fail (the split-opener, computed-verb and nested cases); with it, 29 pass. Full test/test_security.py: 1016 passed. Diff-scoped black gate passes.

Pattern harvest

Rule candidate: review
Pattern: two views of one command text built from different pre-lex states (one folded, one raw). Any helper that recognises shell syntax byte-literally must be fed the same folded text the tokenizer reads; when the same source string is passed to both _self_tokens and a byte-literal scanner, the scanner sees a spelling bash never lexes.

Other

Bash measurement used to pin expectations: cat <\<newline>(echo hi) prints hi; >\<newline>( and $\<newline>( behave the same; \<CR><LF> is not folded and the CR-terminated word runs as its own command.

Closes #9705

… payload walk

The shell removes a backslash-newline while reading, before it lexes a
substitution opener, so `cat <\<newline>(...)` (and the `>(` / `$(`
spellings) is a substitution to bash. `_substitution_bodies` recognises
its openers byte-literally and the payload walk handed it the RAW source,
so with a continuation inside the opener no body was extracted and the
inner command was never scanned -- a wrapper body such as `bash -c '...'`
was allowed while bash ran it.

The walk now reads the bodies from the same quote-aware
`_fold_line_continuations` view `_self_tokens` already tokenizes, so the
token and body views agree. Folding at the body walk rather than the
walk's seed reshapes nothing outside a substitution, and the fold keeps
single-quoted and ANSI-C spans literal, so the benign cases (real
newline, `'...'`, `$'...'`, backslash-CRLF) keep their verdicts.

Regression tests: three openers x (split opener, split program, split
verb), a computed-verb body, a nested split opener, CRLF, real newline,
single-quoted and ANSI-C literals, and the ANSI-C publish literal the
issue measured flipping under a seed-level fold.

Closes #9705
@CrysisDeu
CrysisDeu requested review from a team and iamwhatever September 15, 2026 23:47
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Sep 15, 2026
@github-actions

github-actions Bot commented Sep 15, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Fable 5) — ✅ PASS

Design-level review of f315810852ad9b697a45abad44877cc6bce4e778 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Nothing to check.

[DESIGN-REVIEWED] f315810

@github-actions

github-actions Bot commented Sep 15, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Fable 5) — ✅ PASS

Premise-level review of f315810852ad9b697a45abad44877cc6bce4e778 — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: PASS

Confirm the split-opener tests fail on base — the "8 of 29 fail with the fix reverted" mutation claim is the defect's only runnable provenance.

What this change ships

Inventory (3 items) — 3 justified

Intent: FIX — close #9705's remaining seam: a backslash-newline inside <(, >( or $( hid a substitution body from the deny gate while bash ran it.

  1. Continuation-split substitution openers wrapping a denied command are now refused, at every walk depth — justified
  2. Benign continuation spellings (backslash-CRLF, real newline, single-quoted, ANSI-C) keep their verdicts, pinned by a 29-case matrix — justified
  3. A comment in _shell_payload_walk stating the folded-read invariant — justified

Reuses existing _fold_line_continuations (no second spelling); grepped _substitution_bodies: 3 call sites, the other 2 already token-fed from the folded view, so 0 unfixed siblings. Cause-level: unifies the pre-lex view the byte-literal scanner reads with the one the tokenizer reads, closing the class for all three openers, not one spelling.

[FIRST-PRINCIPLES-REVIEWED] f315810

@github-actions

github-actions Bot commented Sep 15, 2026 •

Copy link
Copy Markdown
Contributor

Opus 4.8 Review — ✅ no blocking findings

Reviewed f315810852ad9b697a45abad44877cc6bce4e778 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] f315810

Verdict parsed from the review's SHA-scoped output markers for commit f315810852ad9b697a45abad44877cc6bce4e778.

False positive or not applicable? A repository writer can comment:
/ai-review override fable f315810852ad9b697a45abad44877cc6bce4e778: <one-sentence reason>

@github-actions

github-actions Bot commented Sep 15, 2026 •

Copy link
Copy Markdown
Contributor

GPT 5.6 Review — ✅ no blocking findings

GPT 5.6 completed its review of f315810852ad9b697a45abad44877cc6bce4e778 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] f315810

False positive or not applicable? A repository writer can comment:
/ai-review override gpt f315810852ad9b697a45abad44877cc6bce4e778: <one-sentence reason>

@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Sep 15, 2026
@github-actions

github-actions Bot commented Sep 15, 2026 •

Copy link
Copy Markdown
Contributor

Security Scope Review -- ✅ PASS

Which legitimate operations does f315810852ad9b697a45abad44877cc6bce4e778 newly refuse? Updated in place on each push. A script-confirmed regression blocks; the model proposes candidates and never decides a refusal.

Verdict basis: adjudicated, zero confirmed regressions.

Security scope review -- adjudicated candidates

  • macos-latest (platform posix): 13 classified, 0 newly refused, 12 still allowed.
  • ubuntu-latest (platform posix): 13 classified, 0 newly refused, 12 still allowed.
  • windows-latest (platform windows): 4 classified, 0 newly refused, 4 still allowed.

No candidate flipped from allowed to refused.

Scope-Verdict: PASS

TIGHTENING: _shell_payload_walk now hands _substitution_bodies the quote-aware fold of each frame's source instead of the raw text (_substitution_bodies(_fold_line_continuations(source))) -- the ONE threat it refuses is a substitution opener split by a backslash-newline, which bash removes while reading so that cat <\ + newline + (bash -c '<name> <verb>') runs the inner mint while the byte-literal opener scan saw no <( and the body was never walked ("measured ALLOWED for all three parenthesised openers while bash ran the mint", per the hunk's comment).

ADJUDICATED: 17 candidates, 0 confirmed newly-refused

PLATFORM COVERAGE: posix (13 rows) -- the fold's core assumption is bash's backslash-LF removal, so the posix rows probe every place a legitimate continuation lands: wrapped daily operations from the touched area's own corpus (gh pr view, the sanctioned pytest invocation, flake8, the lease push, eslint, push_guard), the wrap landing inside a <( and a >( opener with read-only bodies (the exact surface the fold newly exposes to the walk), a wrapped $(...) body whose extracted content changes shape under the fold, a single-quoted and a double-quoted continuation that bash keeps or folds literally in prose, and a nested bash -c payload exercising the fold at inner walk depth. windows (4 rows) -- the diff's second assumption is that backslash-CRLF is NOT a continuation, so one row pins the CRLF spelling of a wrapped fetch; the others pin backslash-laden Windows paths through the fold's escape-pair reading, and PowerShell's backtick and cmd's caret continuations, which the fold must pass through untouched. No assumed platform was left without a candidate: the fold reads only \+LF, and both non-bash continuation characters and the CRLF non-fold have rows. The per-leg VERDICTS are the deterministic fold's to report, per leg, after this review is captured.

[SCOPE-REVIEWED] f315810

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention labels Sep 16, 2026

@iamwhatever iamwhatever left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved after a full read of the diff on the current head: the change matches the linked issue and body, every load-bearing claim was spot-checked against main, tests pin the behaviour, all review lanes are clean and nothing is left undisposed on this head.

@iamwhatever
iamwhatever enabled auto-merge (squash) September 16, 2026 05:22
@iamwhatever
iamwhatever merged commit 6332e7e into main Sep 16, 2026
121 of 122 checks passed
@iamwhatever
iamwhatever deleted the fix/procsub-continuation-9705 branch September 16, 2026 05:23
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Sep 16, 2026
amadsalmon pushed a commit to amadsalmon/KiroCrew that referenced this pull request Sep 22, 2026
…otdev#9705) (kirodotdev#11134)

* fix(security): read substitution bodies from the folded source in the payload walk

The shell removes a backslash-newline while reading, before it lexes a
substitution opener, so `cat <\<newline>(...)` (and the `>(` / `$(`
spellings) is a substitution to bash. `_substitution_bodies` recognises
its openers byte-literally and the payload walk handed it the RAW source,
so with a continuation inside the opener no body was extracted and the
inner command was never scanned -- a wrapper body such as `bash -c '...'`
was allowed while bash ran it.

The walk now reads the bodies from the same quote-aware
`_fold_line_continuations` view `_self_tokens` already tokenizes, so the
token and body views agree. Folding at the body walk rather than the
walk's seed reshapes nothing outside a substitution, and the fold keeps
single-quoted and ANSI-C spans literal, so the benign cases (real
newline, `'...'`, `$'...'`, backslash-CRLF) keep their verdicts.

Regression tests: three openers x (split opener, split program, split
verb), a computed-verb body, a nested split opener, CRLF, real newline,
single-quoted and ANSI-C literals, and the ANSI-C publish literal the
issue measured flipping under a seed-level fold.

Closes kirodotdev#9705

* test(security): state the folded-opener docstrings in present tense
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix(security): _substitution_bodies misses a continuation-split process-sub opener/program

2 participants