Repository navigation
fix(memory): tolerate a non-UTF-8 byte in memory file reads - #13159
Conversation
Design Review (Fable 5) — 🟡 CONCERNSDesign-level review of Design-Verdict: CONCERNS Corruption degrades memory silently: skipped sections surface only a log warning, indefinitely, after dropping #8378's degradation-state signaling. Watch
Suggestions
[DESIGN-REVIEWED] 7b30e42 |
GPT 5.6 Review — ✅ no blocking findingsGPT 5.6 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
First Principles Review (Fable 5) — ✅ PASSPremise-level review of First-Principles-Verdict: PASS Description claims a Not justified as shipped
What this change shipsInventory (6 items) — 5 justifiedIntent: stop a single non-UTF-8 byte in a memory file from crashing the every-turn context build and the index rebuild — a FIX, with provenance (added tests fail on base).
Sibling count run: all five [FIRST-PRINCIPLES-REVIEWED] 7b30e42 |
Opus 5 Review — ✅ no blocking findingsReviewed Review detailsOne advisory: the new decode guard covers only the V1 arm, so an undecodable memory file still raises out of the every-turn context build on a V2 store. FINDING — src/kiro_crew/memory.py:1267 — the added [OPUS-REVIEWED] 7b30e42 Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
fd17c2f to
e5f225b
Compare
e5f225b to
8150ad0
Compare
8150ad0 to
7636046
Compare
b212e9c to
edb75f7
Compare
f8e0406 to
57c8fdd
Compare
57c8fdd to
1664b7b
Compare
1664b7b to
0cce84f
Compare
0cce84f to
32aa053
Compare
The pure readers (recent-history assembly and index rebuild) and the context builders decoded memory files with read_text(encoding="utf-8"): one non-UTF-8 byte raised UnicodeDecodeError up through the reader, breaking get_context / activity_index (both reached from build_session_context) and an index rebuild. Keep every existing read exactly as before (same single call, same containment) and wrap ONLY the decode failure: on UnicodeDecodeError log a warning naming the file and skip that entry/section. No new read path, no reopen, no errors="replace", no new file-opening helper. Sites: _read_recent_history_uncached (skip the day), the two rebuild_index reads (skip the file), and get_context / activity_index consumption of preferences+projects on the every-turn context path (skip the section) so context assembly cannot crash while the readers stay strict. read_preferences, read_projects and append_history keep a strict decode (unchanged): each feeds a read-modify-write path (the consolidator's CAS baseline, add_preference, the dashboard Save, the history append), so an undecodable file must raise and be left intact rather than round-tripped lossily and overwritten with no backup on the default V1 path. The context builders catch the strict reader's error at consumption instead. Trade-off: a corrupt history file is left out of the FTS index until it is repaired; V1 search will not surface its text meanwhile. This is preferred over a reopen (which would race a concurrent delete) or a lossy re-decode. Also prune the now-black-clean test/test_knowledge.py from the shrinking black baseline (.github/black-baseline.txt), which the black gate requires. Tests: the pure readers skip a corrupt file and still return clean entries; get_context survives a corrupt preferences.md and activity_index survives a corrupt projects.md; the read-modify-write readers raise and leave the corrupt file intact. Simplifies #8378: the smallest correct fix, no new config, module, or degradation-state machinery. Co-authored-by: ManoharSwamynathan <7479574+ManoharSwamynathan@users.noreply.github.com>
32aa053 to
7b30e42
Compare
buluoray
left a comment
There was a problem hiding this comment.
Approve. No lossy read reaches a write-back on 7b30e42d:
- Every read-modify-write reader (
read_preferences,read_projects,append_history) still decodes strictly, so a bad byte raises and the file is left intact; the new tests pin that. - The new tolerant sites (
_read_recent_history_uncached,activity_index,get_context,rebuild_index) only feed prompt context or the FTS index. test_memory,test_memory_no_regressionandtest_history: 447 passed.
Non-blocking: the PR body's paragraph about pruning test/test_knowledge.py from .github/black-baseline.txt is stale; the diff doesn't touch that file.
…dev#13159) The pure readers (recent-history assembly and index rebuild) and the context builders decoded memory files with read_text(encoding="utf-8"): one non-UTF-8 byte raised UnicodeDecodeError up through the reader, breaking get_context / activity_index (both reached from build_session_context) and an index rebuild. Keep every existing read exactly as before (same single call, same containment) and wrap ONLY the decode failure: on UnicodeDecodeError log a warning naming the file and skip that entry/section. No new read path, no reopen, no errors="replace", no new file-opening helper. Sites: _read_recent_history_uncached (skip the day), the two rebuild_index reads (skip the file), and get_context / activity_index consumption of preferences+projects on the every-turn context path (skip the section) so context assembly cannot crash while the readers stay strict. read_preferences, read_projects and append_history keep a strict decode (unchanged): each feeds a read-modify-write path (the consolidator's CAS baseline, add_preference, the dashboard Save, the history append), so an undecodable file must raise and be left intact rather than round-tripped lossily and overwritten with no backup on the default V1 path. The context builders catch the strict reader's error at consumption instead. Trade-off: a corrupt history file is left out of the FTS index until it is repaired; V1 search will not surface its text meanwhile. This is preferred over a reopen (which would race a concurrent delete) or a lossy re-decode. Also prune the now-black-clean test/test_knowledge.py from the shrinking black baseline (.github/black-baseline.txt), which the black gate requires. Tests: the pure readers skip a corrupt file and still return clean entries; get_context survives a corrupt preferences.md and activity_index survives a corrupt projects.md; the read-modify-write readers raise and leave the corrupt file intact. Simplifies kirodotdev#8378: the smallest correct fix, no new config, module, or degradation-state machinery. Co-authored-by: iamwhatever <iamwhatever@users.noreply.github.com> Co-authored-by: ManoharSwamynathan <7479574+ManoharSwamynathan@users.noreply.github.com>
What
A single non-UTF-8 byte in a memory file raised
UnicodeDecodeErrorup through the readers, breaking the every-turn context build (get_context, reached fromcontext.py'sbuild_session_context) and an index rebuild.get_contextreadspreferences.mdon every turn — even the startup path withinclude_activity=False— so a corrupt preferences file crashed context assembly.The fix keeps every existing read exactly as it was (same call, same containment) and wraps only the decode failure: on
UnicodeDecodeError, log a warning naming the file and skip that entry/section (treat it as absent). No new read path, noerrors="replace"round-trip, no new file-opening helper, noO_NOFOLLOWtricks. Applied at:_read_recent_history_uncachedand the tworebuild_indexreads (skip the file);get_context's own consumption ofread_preferences()/read_projects()(skip that section), so the every-turn path survives a corrupt preferences/projects file while the readers themselves stay strict.read_preferences,read_projectsandappend_historykeep a strict decode. Each feeds a read-modify-write path (the consolidator's CAS baseline,add_preference, the dashboard Save, the history append), so an undecodable file must raise and be left intact rather than round-tripped lossily and overwritten with no backup on the default V1 path —get_contextcatches the error at the point of consumption instead.Why this shape
Earlier iterations replaced the read with a tolerant re-read, which repeatedly introduced new problems (a lossy CAS overwrite, a symlink-follow, a TOCTOU race, a Windows
O_NOFOLLOWgap) — all downstream of one decision: re-reading through a new code path. Main's read security posture is already correct; the only bug is that a bad byte raises. Catching just the decode failure fixes that with no new attack surface.Trade-off: a corrupt pure-read file is skipped, not partially recovered. That is deliberate and safe for these read-only consumers.
Supersedes #8378 (simplified; authorship preserved)
#8378 fixed the crash but routed readers through the full
_guarded_entryhardening machinery plus degradation-state handling. This is the smallest correct fix. Original authorship is credited viaCo-authored-by:on the commit.Base black-baseline
Also prunes the now-black-clean
test/test_knowledge.pyfrom.github/black-baseline.txt(the shrinking-baseline gate requires removing graduated entries; one-line deletion, conductor-approved).Tests
TestNonUtf8Toleranceintest/test_memory.py: the pure readers skip a corrupt file and still return the clean entries; the read-modify-write readers raise and leave the corrupt file intact.Ran locally:
isort,flake8,black --check,mypy src/kiro_crew/memory.py— clean;pytest test/test_memory.py test/test_memory_no_regression.py test/test_history.py test/test_memory_selfheal.py→ 454 passed.Pattern harvest
Not generalizable: the fix is to catch
UnicodeDecodeErrorat the existing read sites rather than add a read path. The reusable idea — "wrap the decode, don't replace the read" — is already localized to this module.