Repository navigation
fix(members): scrub member payloads span-locally, keep legitimate content byte-exact - #8991
Conversation
Design Review (Fable 5, fork) — ✅ PASSDesign-level review of The design is sound. Let me verify one detail — that the fail-closed floor can't be tripped by the replacement token itself, and that the extracted merge helper is truly shared unchanged. I've confirmed from the base: Design-Verdict: PASS Real fidelity harm on the user-owned rules boundary, fixed with the repo's own span-local origin-map discipline plus a floor that can only fail toward more scrubbing. Suggestions
[DESIGN-REVIEWED] 3a0db04 |
First Principles Review (Fable 5, fork) — 🟡 CONCERNSPremise-level review of I have what I need. Base code verified by reading: First-Principles-Verdict: CONCERNS
Not justified as shipped
What this change shipsInventory (6 items) — 5 justifiedIntent: FIX — stop the member scrub from rewriting legitimate fullwidth/ZWJ/dash content when neutralizing forged authority markers (issue #8524; the base docstring pin "costs nothing" is overturned with the rule-boundary counterexample).
Sibling count: grepped the detect-on-view/inject-view pattern across the three marker scrubs — WatchThe smaller shape — SubtractionsDrop [FIRST-PRINCIPLES-REVIEWED] 3a0db04 |
Opus 4.8 Review (fork) — ✅ no blocking findingsReviewed Review detailsThe candidate discovery pass reported no candidates. I independently examined the changed code in Key checks I re-derived myself:
No reachable crash, no removed guard without compensating replacement, and no forgery admission is groundable on the changed lines. No findings. [OPUS-REVIEWED] 3a0db04 |
GPT 5.6 Review (fork) — ✅ no blocking findingsReviewed Review detailsNo findings. |
d2b7ff5 to
9d59b40
Compare
|
🤖 Kiro Crew [operator: NicholasRBowers#a942f9ca]: This PR has been inactive for 7+ days with failing CI. I've assessed the blockers and they appear resolvable — I'll push fixes directly to this branch as a co-author. Assessment: Merge conflict with main plus one Backend Tests shard red in If you'd prefer I don't touch this PR, add the |
9d59b40 to
08b1fcc
Compare
|
self-added: no 🤖 Kiro Crew [operator: NicholasRBowers#a942f9ca]
|
|
🤖 Kiro Crew [operator: NicholasRBowers#a942f9ca] Rebase + drive-to-green round 1 — head What changed and why, in full:
Verification on
|
08b1fcc to
c811708
Compare
|
🤖 Kiro Crew [operator: NicholasRBowers#a942f9ca] Round 2 — head Comment History Gate (new on main since this PR's original base) flagged 3 added comment/docstring lines carrying PR/issue numbers ( Verified locally on |
…tent byte-exact _scrub_member_payload detected forged member-authority markers on a normalized view (NFKC + Cf drop + Pd fold) but then injected the normalized copy of the WHOLE payload, so legitimate content was rewritten: a permanent rule protecting a fullwidth path reached the member naming its ASCII fold instead, and emoji ZWJ sequences and prose dashes in briefings were destroyed. Detection still runs on the normalized view, now built per character with an origin map back to original offsets (the same mechanism _structural_marker_spans already uses, generalized to NFKC), and only the matched spans are rewritten in the ORIGINAL text via the existing _apply_marker_spans. A span touching any part of a multi-char compatibility fold maps to the whole original character, so spans only over-cover (deny direction). Fail-closed floor: the span-scrubbed result is re-checked against the historical whole-string normalized view; any residual match degrades the scrub to exactly the old behavior (normalize whole payload, substitute every match), so a mapping defect can cost fidelity but never admit a forgery. The overlapping-span merge tail is extracted to _merge_overlapping_spans, shared with _structural_marker_spans unchanged. Owning doc paragraph synced. Part of kirodotdev#8524
…atches the fail-closed floor A combining mark inside a marker word (I + U+0307) composes under the whole-string NFKC floor to a char whose case fold is ASCII, but the per-character span view could not compose it: the span pass went blind, the floor fired, and the whole payload was folded - corrupting the legitimate fullwidth content the span-local rewrite exists to protect. Group each base char with its trailing combining marks, normalize the sequence as one unit, and map every emitted view char to the sequence's full original span. Residual starter-composition divergences (Hangul jamo) cannot reach the ASCII marker alphabet and still land on the unchanged fail-closed floor. Rebased onto main by Kiro Crew: main's marker machinery gained default-ignorable ranges and _MULTIBYTE_TABLE punctuation folding while this PR was open, so the member floor view inherited them via the shared helpers; the combining-sequence detection view is harmonized to mirror that same normalization, keeping detection and floor aligned as this commit intends. Original work by javenciu. Co-authored-by: Kiro Crew <noreply@kirodotdev.github.io>
c811708 to
3a0db04
Compare
|
🤖 Kiro Crew [operator: NicholasRBowers#a942f9ca] Round 3 — head Main advanced again while CI ran: Verified on |
|
🤖 Kiro Crew [operator: NicholasRBowers#a942f9ca] REVIEW-READY at head PR Readiness passed, mergeable, all 69 checks green, zero unresolved threads. All five AI review lanes clean on this head (GPT, Opus, Design, UX pass; the First Principles description-drift Watch item was fixed by the body rewrite, disposition recorded). Drive summary: rebased twice over a moving main (marker-machinery evolution, then memory-v2), reworded 3 comment lines for the Comment History Gate, and harmonized the combining-sequence detection view with the fail-closed floor's normalization — both commits keep @javenciu's authorship. Only the human maintainer review remains; auto-merge is deliberately not armed. |
cixuuz
left a comment
There was a problem hiding this comment.
Reviewed at 3a0db04. The rewrite is span-local over original text, preserves legitimate bytes, maps compatibility and combining-sequence matches conservatively, and retains a whole-view fail-closed floor with focused regression coverage.
…tent byte-exact (kirodotdev#8991) * fix(members): scrub member payloads span-locally, keep legitimate content byte-exact _scrub_member_payload detected forged member-authority markers on a normalized view (NFKC + Cf drop + Pd fold) but then injected the normalized copy of the WHOLE payload, so legitimate content was rewritten: a permanent rule protecting a fullwidth path reached the member naming its ASCII fold instead, and emoji ZWJ sequences and prose dashes in briefings were destroyed. Detection still runs on the normalized view, now built per character with an origin map back to original offsets (the same mechanism _structural_marker_spans already uses, generalized to NFKC), and only the matched spans are rewritten in the ORIGINAL text via the existing _apply_marker_spans. A span touching any part of a multi-char compatibility fold maps to the whole original character, so spans only over-cover (deny direction). Fail-closed floor: the span-scrubbed result is re-checked against the historical whole-string normalized view; any residual match degrades the scrub to exactly the old behavior (normalize whole payload, substitute every match), so a mapping defect can cost fidelity but never admit a forgery. The overlapping-span merge tail is extracted to _merge_overlapping_spans, shared with _structural_marker_spans unchanged. Owning doc paragraph synced. Part of kirodotdev#8524 * fix(members): normalize combining sequences whole so span detection matches the fail-closed floor A combining mark inside a marker word (I + U+0307) composes under the whole-string NFKC floor to a char whose case fold is ASCII, but the per-character span view could not compose it: the span pass went blind, the floor fired, and the whole payload was folded - corrupting the legitimate fullwidth content the span-local rewrite exists to protect. Group each base char with its trailing combining marks, normalize the sequence as one unit, and map every emitted view char to the sequence's full original span. Residual starter-composition divergences (Hangul jamo) cannot reach the ASCII marker alphabet and still land on the unchanged fail-closed floor. Rebased onto main by Kiro Crew: main's marker machinery gained default-ignorable ranges and _MULTIBYTE_TABLE punctuation folding while this PR was open, so the member floor view inherited them via the shared helpers; the combining-sequence detection view is harmonized to mirror that same normalization, keeping detection and floor aligned as this commit intends. Original work by javenciu. Co-authored-by: Kiro Crew <noreply@kirodotdev.github.io> --------- Co-authored-by: Kiro Crew <noreply@kirodotdev.github.io>
Problem / Motivation
_scrub_member_payload(src/kiro_crew/context.py) neutralizes forged member-authority markers by matching on a normalized view — NFKC-folded, default-ignorables dropped, punctuation and every Unicode dash folded to ASCII — and then injects the NORMALIZED copy of the whole payload. Detection needs that view; injection does not. The result is that legitimate content is rewritten on the way to the member:A.txt(fullwidth) reaches the member namingA.txt— a subtly different safety boundary than the user wrote (member payload scrub replaces the whole text with its NFKC normalization #8524's own example);👨👩👧loses its joiners);-.Issue #8524 names two candidate resolutions and calls span-level replacement "the reviewer-preferred fix", pointing at the sibling
_structural_marker_spans, which already matches on a normalized view while carrying an origin map back to original offsets.Why it matters
The rules payload is the user-owned safety boundary of the member prompt (layer 3). A scrub that silently rewrites it means the member can be steered by a boundary the user never wrote — the exact fidelity property the trust/ storage design protects everywhere else. Briefings and descriptions are lower stakes but the same contract: scrubbing is for forgeries, not for legitimate bytes.
What changed (motivation → approach → change)
Observed symptom: any member payload containing fullwidth/compatibility characters, ZWJ sequences, or Unicode dashes is rewritten wholesale, even with no forgery present (reproducer below). Root cause:
_scrub_member_payloadreturns the normalized detection view instead of the original text with matched spans replaced.Approach: generalize the origin-map mechanism
_structural_marker_spansalready uses (via the shared_marker_spanshelper). That view folds_MULTIBYTE_TABLE+ default-ignorables +Pdbut deliberately not NFKC; the member scrub requires NFKC (fullwidth bracket/letter confusables). So a member-specific sibling is added rather than the shared view widened — widening it would change which spans the per-turn context breakdown attributes, an unrelated behavioral surface.Change:
_member_marker_spans(text)— builds the member view (NFKC, then default-ignorable drops,_MULTIBYTE_TABLEpunctuation folds, andPddashes to-, mirroring_member_normalized_viewexactly) with an origin map back to original offsets, matches_MEMBER_MARKER_RESagainst it, and maps spans back to original coordinates. The view is built PER COMBINING SEQUENCE (base character plus its trailing combining marks), not per lone character: canonical composition happens ACROSS characters within one sequence (I+ U+0307 composes toİunder whole-string NFKC, which case-folds to ASCIIi), so a marker word carrying an embedded combining mark still matches. Residual divergences from the whole-string view (e.g. Hangul jamo, where starters compose with each other) yield non-ASCII characters with no ASCII case fold, so they cannot reach the marker alphabet. A match touching any part of a multi-char compatibility fold (㎢→km2) or a sequence maps to the whole original sequence: spans only ever over-cover (deny direction)._scrub_member_payloadnow rewrites span-locally in the ORIGINAL text via the existing_apply_marker_spans, then re-checks the result against the historical whole-string view (_member_normalized_view). Any residual match degrades the scrub to exactly the historical behavior — normalize the whole payload, substitute every match. A mapping defect can therefore cost fidelity, never admit a forgery: every return value either passes the whole-string detector clean or IS its output._merge_overlapping_spansand shared with_marker_spansunchanged (byte-identical logic).Alternatives rejected: (a) documenting normalized-injection semantics (issue option 2) — cheaper, but the safety-boundary corruption case remains, and the issue's own text prefers option 1; (b) reusing
_structural_marker_spansdirectly with the member patterns — its view lacks NFKC, so fullwidth forgeries would stop matching (a security regression); (c) widening the shared view to NFKC — changes span attribution for the per-turn context breakdown, an unrelated surface this fix must not move.Tests
test/test_member_prompt.py, five new tests beside the existing forgery-coverage test (which is unchanged and still passes — all 17 forgery shapes, fullwidth and zero-width included, are still neutralized):test_legitimate_fullwidth_content_survives_scrub_byte_exact— the issue's example rule, ZWJ emoji prose, and combining-mark/compat glyphs return byte-exact.test_scrub_is_span_local_around_a_neutralized_forgery— fullwidth content on both sides of a neutralized fullwidth forgery survives verbatim; a multi-char compatibility fold adjacent to a marker survives (over-cover pins the boundary).test_scrub_fails_closed_when_span_mapping_misses— with the span pass monkeypatched blind, the floor returns exactly the historical whole-payload normalized output with the forgery substituted.test_combining_mark_forgery_is_scrubbed_span_locally— a marker word carrying an embedded combining mark is detected and replaced span-locally; legitimate fullwidth content beside it survives byte-exact.test_benign_combining_marks_survive_byte_exact— benign combining sequences (including a defectively ordered one) pass through untouched.Red-before-green verified mechanically: with the production hunks reverted and the test hunks kept, the new tests fail with assertion errors; with the fix, the file's full suite passes (64 passed).
Manual verification
These exact strings are asserted by the new tests, so the transcript above is locked in by the suite rather than hand-checked once.
Screenshots / video
Why no screenshot: backend-only change to prompt scrubbing; no rendered surface changes.
Related Issues
Addresses #8524 — implements option 1, the issue's reviewer-preferred fix; the design ratification stays with maintainers, so this deliberately does not auto-close.
no linked issue closed on merge: #8524 is left open for the maintainers' design ratification, per the issue's own text.
Pattern harvest
The defect class is "detect on a transformed view, then inject the view instead of the original." The repo's own
_marker_spansdocstring states the correct discipline (span-local rewrite, origin map); this PR brings the member scrub up to it. One adjacent surface was checked and is already correct:_neutralize_structural_markersrewrites span-locally by construction.Rule candidate: when a security filter matches on a normalized/decoded/folded view of user content, the rewrite must map matches back to original coordinates and touch only those spans — returning the transformed view is a silent content-corruption bug, and the fail-closed shape is to re-check the rewritten result against the original whole-view detector and degrade to it on any residual match.
Checklist
Contribution License Agreement
Per the template placeholder (CLA text pending): offered under the same terms as my prior merged contributions to this repository (#8835).