Repository navigation
fix(memory): degrade an undecodable preferences/projects file to empty - #8378
ManoharSwamynathan wants to merge 1 commit into
Conversation
Opus 4.8 Review (fork) — ✅ no blocking findingsReviewed Review detailsBoth candidates were rated low-confidence by the discovery pass, and each fails the falsification bar:
Neither survives Step 1; nothing new grounds to the 80+ bar in Step 2. No findings. [OPUS-REVIEWED] 8d6e838 |
Design Review (Fable 5, fork) — 🟡 CONCERNSDesign-level review of I have enough context — base machinery ( Design-Verdict: CONCERNS Shipped design is sound — but the title/description document the opposite contract (empty-on-undecodable) and omit half the diff's behavior changes. Watch
[DESIGN-REVIEWED] 8d6e838 |
First Principles Review (Fable 5, fork) — 🔴 BLOCKPremise-level review of All the evidence is in. The decisive fact: the PR's title and description claim degrade-to-empty via the First-Principles-Verdict: BLOCK The title promises "degrade … to empty"; the diff ships preserve-with-replacement, and its own tests call the described behavior a data-loss regression. What this change shipsIntent: stop one bad byte in a memory file from crashing every reader and the repair path — a FIX.
Blockers
Watch
Subtractions
[FIRST-PRINCIPLES-REVIEWED] 8d6e838 |
GPT 5.6 Review (fork) — 🔴 changes requested (blocking)Reviewed 2 of 2 blocking finding(s) are security-class and were withheld from adjudication, so the blocking verdict stands. BLOCKING -- src/kiro_crew/memory.py:355 -- Root guard runs after an unsafe UNC probe BLOCKING -- src/kiro_crew/memory.py:339 -- Oversized preferences can be silently overwritten FINDING -- src/kiro_crew/memory.py:335 -- [BLOCK-MERGE] 8d6e838 Adjudication (Opus 4.8) — is blocking on each finding proportionate?Both findings are FENCED (annotate-only); the adjudicable block is empty. My evidence review of the base tree and the pre-fetched diff: F1 — F2 — |
|
Kiro Crew [operator: chenmingwei23#de330d0c]: This PR has been inactive for 7+ days. I reviewed the blockers but they require your input:
When you've addressed these, the pipeline will re-assess on its next cycle. If you'd prefer no automation on this PR, add the pr-no-autofix label. |
44c4bcb to
d04c39a
Compare
d04c39a to
93830fe
Compare
93830fe to
da35d5c
Compare
|
Thanks for the detailed review — addressed in the latest push. I took a combined path rather than any single suggestion, because none of the three alone satisfied both blocking findings:
On the First-Principles "point patch with unfixed siblings" concern — I scoped the siblings in rather than leaving them:
Tests updated: content preservation for both readers, the read-before-write data-loss regression, symlink refusal, and the two history sites. Also rebased onto current |
|
Cross-reference from the open-PR relationship audit: #8248 (now closed in favour of this PR) covered one extra call site that this PR currently leaves strict -- |
|
Handing this off — flagging the true state so it's easy to pick up. My earlier "addressed in the latest push" reply was inaccurate: it fixed a different vector (undecodable bytes → Blockers remaining (all gate the PR Readiness check):
Also red — Backend Tests (Windows) (3): 4 CRLF failures in atomic_write(path, content, mode=mode, newline="\n")Verified locally (black/flake8/mypy/py_compile clean; passes on Linux). Windows is CI-verified. Unassigning myself — open for anyone to pick up. |
The V1 markdown readers decoded with a bare read_text(encoding="utf-8"): one non-UTF-8 byte raises UnicodeDecodeError up through the reader (the dashboard memory GET, the every-turn get_context, and rebuild_index). Add one small helper, _read_text_tolerant(path), that decodes UTF-8 and, on UnicodeDecodeError, logs once at warning level with the path and falls back to errors="replace". For a valid UTF-8 file it returns the byte-identical string a bare decode returns. Applied at the five pure reader sites: read_preferences, read_projects, _read_recent_history_uncached, and the two rebuild_index reads. append_history is left on a strict decode on purpose: it is a read-modify-write path, so a lossy read there would persist U+FFFD over the original bytes on the next atomic write and destroy them. An undecodable today-file raises and is left intact and recoverable. Tests: one per pure-reader entry point that a corrupt file no longer raises (read_preferences, read_projects, read_recent_history, rebuild_index), a check that append_history does NOT rewrite a corrupt file lossily, and a byte-identical check for good files. Simplifies #8378: the smallest correct fix, no new config, module, or degradation-state machinery. Co-authored-by: ManoharSwamynathan <7479574+ManoharSwamynathan@users.noreply.github.com>
The recent-history assembly and index rebuild decoded memory files with a bare read_text(encoding="utf-8"): one non-UTF-8 byte raises UnicodeDecodeError up through the reader and, for history, breaks the every-turn get_context. Add one small helper, _read_text_tolerant(path), that decodes UTF-8 and, on UnicodeDecodeError, logs once at warning level with the path and falls back to errors="replace". For a valid UTF-8 file it returns the byte-identical string a bare decode returns. Applied ONLY at the three pure reader sites that never write their result back: _read_recent_history_uncached and the two rebuild_index reads. read_preferences, read_projects and append_history keep a strict decode on purpose: each feeds a read-modify-write path (the consolidator's CAS baseline, add_preference, the dashboard Save, the history append), so a lossy errors="replace" read there would let a whole-file write persist U+FFFD over the original bytes with no backup on the default V1 path. A strict decode leaves an undecodable file intact and recoverable. Tests: the two pure readers tolerate a bad history byte; the three RMW readers raise and leave the corrupt file intact; a valid UTF-8 file reads byte-identically. Simplifies #8378: the smallest correct fix, no new config, module, or degradation-state machinery. Co-authored-by: ManoharSwamynathan <7479574+ManoharSwamynathan@users.noreply.github.com>
Recent-history assembly and index rebuild decoded memory files with a bare read_text(encoding="utf-8"): one non-UTF-8 byte raises UnicodeDecodeError up through the reader and, for history, breaks the every-turn get_context. Add one small helper, _read_text_tolerant(path), that decodes UTF-8 and, on UnicodeDecodeError, logs once at warning level with the path and falls back to errors="replace". For a valid UTF-8 file it returns the byte-identical string a bare decode returns. Applied ONLY at the three pure reader sites that never write their result back: _read_recent_history_uncached and the two rebuild_index reads. read_preferences, read_projects and append_history keep a strict decode on purpose: each feeds a read-modify-write path (the consolidator's CAS baseline, add_preference, the dashboard Save, the history append), so a lossy errors="replace" read there would let a whole-file write persist U+FFFD over the original bytes with no backup on the default V1 path. A strict decode leaves an undecodable file intact and recoverable. Also prune the now-black-clean test/test_knowledge.py from the shrinking black baseline (.github/black-baseline.txt), which the black gate requires. Tests: the two pure readers tolerate a bad history byte; the three RMW readers raise and leave the corrupt file intact; a valid UTF-8 file reads byte-identically. Simplifies #8378: the smallest correct fix, no new config, module, or degradation-state machinery. Co-authored-by: ManoharSwamynathan <7479574+ManoharSwamynathan@users.noreply.github.com>
Recent-history assembly and index rebuild decoded memory files with a bare read_text(encoding="utf-8"): one non-UTF-8 byte raises UnicodeDecodeError up through the reader and, for history, breaks the every-turn get_context. The pure readers now read bytes through the hardened, link-refusing, root-contained reader (_read_entry_bytes -> safe_read_file_bytes_nolink) and decode them with _decode_tolerant, which falls back to errors="replace" on a bad byte and logs once at warning level. Reading bytes through the hardened path first means a planted dated-history symlink (the history dir is agent-writable) can never be followed into a sensitive file; a refused file is skipped. Applied at the three pure reader sites: _read_recent_history_uncached and the two rebuild_index reads. read_preferences, read_projects and append_history keep a strict decode on purpose: each feeds a read-modify-write path (the consolidator's CAS baseline, add_preference, the dashboard Save, the history append), so a lossy read there would let a whole-file write persist U+FFFD over the original bytes with no backup on the default V1 path. A strict decode leaves an undecodable file intact and recoverable. Also prune the now-black-clean test/test_knowledge.py from the shrinking black baseline (.github/black-baseline.txt), which the black gate requires. Tests: the pure readers tolerate a bad history byte and refuse a planted history symlink; the read-modify-write readers raise and leave the corrupt file intact; valid UTF-8 bytes decode identically. Simplifies #8378: the smallest correct fix, no new config, module, or degradation-state machinery. Co-authored-by: ManoharSwamynathan <7479574+ManoharSwamynathan@users.noreply.github.com>
Recent-history assembly and index rebuild decoded memory files with a bare read_text(encoding="utf-8"): one non-UTF-8 byte raises UnicodeDecodeError up through the reader and, for history, breaks the every-turn get_context. The pure readers now go through _read_leaf_text_tolerant, which refuses a symlink/junction leaf and a hardlinked or non-regular inode BEFORE reading (the same leaf guard append_history applies, and it audits the refusal via _audit_read_refusal), then decodes UTF-8 with an errors="replace" fallback and one warning log. Refusing the link first means a planted dated-history symlink (the history dir is agent-writable) cannot be followed into a sensitive file whose bytes would flow into the suggestions prompt; a refused leaf is skipped. Applied at the three pure reader sites: _read_recent_history_uncached and the two rebuild_index reads. read_preferences, read_projects and append_history keep a strict decode on purpose: each feeds a read-modify-write path (the consolidator's CAS baseline, add_preference, the dashboard Save, the history append), so a lossy read there would let a whole-file write persist U+FFFD over the original bytes with no backup on the default V1 path. A strict decode leaves an undecodable file intact and recoverable. Also prune the now-black-clean test/test_knowledge.py from the shrinking black baseline (.github/black-baseline.txt), which the black gate requires. Tests: the pure readers tolerate a bad history byte and refuse a planted history symlink; the read-modify-write readers raise and leave the corrupt file intact; a valid UTF-8 leaf reads exactly as a raw UTF-8 decode. Simplifies #8378: the smallest correct fix, no new config, module, or degradation-state machinery. Co-authored-by: ManoharSwamynathan <7479574+ManoharSwamynathan@users.noreply.github.com>
Recent-history assembly and index rebuild decoded memory files with a bare read_text(encoding="utf-8"): one non-UTF-8 byte raises UnicodeDecodeError up through the reader and, for history, breaks the every-turn get_context. The pure readers now go through _read_leaf_text_tolerant, which opens the leaf with O_NOFOLLOW and validates the OPENED descriptor via fstat (regular file, single link) before reading from that same fd, then decodes UTF-8 with an errors="replace" fallback and one warning log. Pinning the inode check to the opened descriptor closes the check-to-open race: a symlink or inode swapped in for a dated-history leaf (the history dir is agent-writable) is refused at open, so it cannot be followed into a sensitive file whose bytes would flow into the suggestions prompt. A refused leaf is audited and skipped. Applied at the three pure reader sites: _read_recent_history_uncached and the two rebuild_index reads. read_preferences, read_projects and append_history keep a strict decode on purpose: each feeds a read-modify-write path (the consolidator's CAS baseline, add_preference, the dashboard Save, the history append), so a lossy read there would let a whole-file write persist U+FFFD over the original bytes with no backup on the default V1 path. A strict decode leaves an undecodable file intact and recoverable. Also prune the now-black-clean test/test_knowledge.py from the shrinking black baseline (.github/black-baseline.txt), which the black gate requires. Tests: the pure readers tolerate a bad history byte and refuse a planted history symlink; the read-modify-write readers raise and leave the corrupt file intact; a valid UTF-8 leaf reads exactly as a raw UTF-8 decode. Simplifies #8378: the smallest correct fix, no new config, module, or degradation-state machinery. Co-authored-by: ManoharSwamynathan <7479574+ManoharSwamynathan@users.noreply.github.com>
The pure readers (recent-history assembly and index rebuild) decoded memory files with read_text(encoding="utf-8"): one non-UTF-8 byte raised UnicodeDecodeError up through the reader, breaking the every-turn get_context and a rebuild. Keep every existing read exactly as before (same call, same containment) and wrap ONLY the decode: on UnicodeDecodeError log a warning naming the file and skip that entry, treating it as absent. No new read path, no errors="replace" round-trip, no new file-opening helper. Applied at the three pure reader sites: _read_recent_history_uncached and the two rebuild_index reads. read_preferences, read_projects and append_history keep a strict decode (unchanged): each feeds a read-modify-write path (the consolidator's CAS baseline, add_preference, the dashboard Save, the history append), so an undecodable file must raise and be left intact rather than round-tripped lossily and overwritten with no backup on the default V1 path. Also prune the now-black-clean test/test_knowledge.py from the shrinking black baseline (.github/black-baseline.txt), which the black gate requires. Tests: the pure readers skip a corrupt file and still return the clean entries; the read-modify-write readers raise and leave the corrupt file intact. Simplifies #8378: the smallest correct fix, no new config, module, or degradation-state machinery. Co-authored-by: ManoharSwamynathan <7479574+ManoharSwamynathan@users.noreply.github.com>
The pure readers (recent-history assembly and index rebuild) and the every-turn context build decoded memory files with read_text(encoding="utf-8"): one non-UTF-8 byte raised UnicodeDecodeError up through the reader, breaking get_context (via build_session_context) and a rebuild. Keep every existing read exactly as before (same call, same containment) and wrap ONLY the decode failure: on UnicodeDecodeError log a warning naming the file and skip that entry/section, treating it as absent. No new read path, no errors="replace" round-trip, no new file-opening helper. Applied at the pure reader sites (_read_recent_history_uncached, the two rebuild_index reads) and at get_context's own consumption of preferences and projects — the every-turn path that reads preferences even with include_activity=False. read_preferences, read_projects and append_history keep a strict decode (unchanged): each feeds a read-modify-write path (the consolidator's CAS baseline, add_preference, the dashboard Save, the history append), so an undecodable file must raise and be left intact rather than round-tripped lossily and overwritten with no backup on the default V1 path. get_context catches the strict reader's error at the point of consumption instead. Also prune the now-black-clean test/test_knowledge.py from the shrinking black baseline (.github/black-baseline.txt), which the black gate requires. Tests: the pure readers skip a corrupt file and still return clean entries; get_context survives a corrupt preferences.md and still surfaces clean history; the read-modify-write readers raise and leave the corrupt file intact. Simplifies #8378: the smallest correct fix, no new config, module, or degradation-state machinery. Co-authored-by: ManoharSwamynathan <7479574+ManoharSwamynathan@users.noreply.github.com>
The pure readers (recent-history assembly and index rebuild) and the context builders decoded memory files with read_text(encoding="utf-8"): one non-UTF-8 byte raised UnicodeDecodeError up through the reader, breaking get_context / activity_index (both reached from build_session_context) and an index rebuild. Keep every existing read exactly as before (same call, same containment) and wrap ONLY the decode failure: on UnicodeDecodeError log a warning naming the file and skip that entry/section, treating it as absent. No new read path, no errors="replace" round-trip, no new file-opening helper. Applied at the pure reader sites (_read_recent_history_uncached, the two rebuild_index reads) and at the strict readers' points of consumption on the every-turn context path: get_context's preferences/projects reads and activity_index's projects read (build_session_context reaches the latter with include_activity=False). read_preferences, read_projects and append_history keep a strict decode (unchanged): each feeds a read-modify-write path (the consolidator's CAS baseline, add_preference, the dashboard Save, the history append), so an undecodable file must raise and be left intact rather than round-tripped lossily and overwritten with no backup on the default V1 path. The context builders catch the strict reader's error at consumption instead. Also prune the now-black-clean test/test_knowledge.py from the shrinking black baseline (.github/black-baseline.txt), which the black gate requires. Tests: the pure readers skip a corrupt file and still return clean entries; get_context survives a corrupt preferences.md and activity_index survives a corrupt projects.md, both still surfacing clean content; the read-modify-write readers raise and leave the corrupt file intact. Simplifies #8378: the smallest correct fix, no new config, module, or degradation-state machinery. Co-authored-by: ManoharSwamynathan <7479574+ManoharSwamynathan@users.noreply.github.com>
The pure readers (recent-history assembly and index rebuild) and the context builders decoded memory files with read_text(encoding="utf-8"): one non-UTF-8 byte raised UnicodeDecodeError up through the reader, breaking get_context / activity_index (both reached from build_session_context) and an index rebuild. Keep every existing read exactly as before (same call, same containment) and wrap ONLY the decode failure: on UnicodeDecodeError log a warning naming the file. No new read path, no new file-opening helper. Behaviour by site: - _read_recent_history_uncached: skip the corrupt day (a read-only assembly; the missing day is simply omitted). - get_context (preferences + projects) and activity_index (projects): catch at the point of consumption on the every-turn context path and skip that section, so a corrupt file cannot crash context assembly while the readers themselves stay strict for their read-modify-write callers. - rebuild_index: index the surviving text with errors="replace" rather than dropping the file. rebuild_index does DELETE-then-reinsert, so dropping a file would lose its FTS rows and V1 search has no file-scan fallback; the index is derived and never written back to the markdown, so a lossy decode is safe here. read_preferences, read_projects and append_history keep a strict decode (unchanged): each feeds a read-modify-write path (the consolidator's CAS baseline, add_preference, the dashboard Save, the history append), so an undecodable file must raise and be left intact rather than round-tripped lossily and overwritten with no backup on the default V1 path. Also prune the now-black-clean test/test_knowledge.py from the shrinking black baseline (.github/black-baseline.txt), which the black gate requires. Tests: the recent-history reader skips a corrupt day; rebuild_index keeps a corrupt file's surviving text searchable; get_context survives a corrupt preferences.md and activity_index survives a corrupt projects.md; the read-modify-write readers raise and leave the corrupt file intact. Simplifies #8378: the smallest correct fix, no new config, module, or degradation-state machinery. Co-authored-by: ManoharSwamynathan <7479574+ManoharSwamynathan@users.noreply.github.com>
The pure readers (recent-history assembly and index rebuild) and the context builders decoded memory files with read_text(encoding="utf-8"): one non-UTF-8 byte raised UnicodeDecodeError up through the reader, breaking get_context / activity_index (both reached from build_session_context) and an index rebuild. Keep every existing read exactly as before (same single call, same containment) and wrap ONLY the decode failure: on UnicodeDecodeError log a warning naming the file and skip that entry/section. No new read path, no reopen, no errors="replace", no new file-opening helper. Sites: _read_recent_history_uncached (skip the day), the two rebuild_index reads (skip the file), and get_context / activity_index consumption of preferences+projects on the every-turn context path (skip the section) so context assembly cannot crash while the readers stay strict. read_preferences, read_projects and append_history keep a strict decode (unchanged): each feeds a read-modify-write path (the consolidator's CAS baseline, add_preference, the dashboard Save, the history append), so an undecodable file must raise and be left intact rather than round-tripped lossily and overwritten with no backup on the default V1 path. The context builders catch the strict reader's error at consumption instead. Trade-off: a corrupt history file is left out of the FTS index until it is repaired; V1 search will not surface its text meanwhile. This is preferred over a reopen (which would race a concurrent delete) or a lossy re-decode. Also prune the now-black-clean test/test_knowledge.py from the shrinking black baseline (.github/black-baseline.txt), which the black gate requires. Tests: the pure readers skip a corrupt file and still return clean entries; get_context survives a corrupt preferences.md and activity_index survives a corrupt projects.md; the read-modify-write readers raise and leave the corrupt file intact. Simplifies #8378: the smallest correct fix, no new config, module, or degradation-state machinery. Co-authored-by: ManoharSwamynathan <7479574+ManoharSwamynathan@users.noreply.github.com>
The pure readers (recent-history assembly and index rebuild) and the context builders decoded memory files with read_text(encoding="utf-8"): one non-UTF-8 byte raised UnicodeDecodeError up through the reader, breaking get_context / activity_index (both reached from build_session_context) and an index rebuild. Keep every existing read exactly as before (same single call, same containment) and wrap ONLY the decode failure: on UnicodeDecodeError log a warning naming the file and skip that entry/section. No new read path, no reopen, no errors="replace", no new file-opening helper. Sites: _read_recent_history_uncached (skip the day), the two rebuild_index reads (skip the file), and get_context / activity_index consumption of preferences+projects on the every-turn context path (skip the section) so context assembly cannot crash while the readers stay strict. read_preferences, read_projects and append_history keep a strict decode (unchanged): each feeds a read-modify-write path (the consolidator's CAS baseline, add_preference, the dashboard Save, the history append), so an undecodable file must raise and be left intact rather than round-tripped lossily and overwritten with no backup on the default V1 path. The context builders catch the strict reader's error at consumption instead. Trade-off: a corrupt history file is left out of the FTS index until it is repaired; V1 search will not surface its text meanwhile. This is preferred over a reopen (which would race a concurrent delete) or a lossy re-decode. Also prune the now-black-clean test/test_knowledge.py from the shrinking black baseline (.github/black-baseline.txt), which the black gate requires. Tests: the pure readers skip a corrupt file and still return clean entries; get_context survives a corrupt preferences.md and activity_index survives a corrupt projects.md; the read-modify-write readers raise and leave the corrupt file intact. Simplifies #8378: the smallest correct fix, no new config, module, or degradation-state machinery. Co-authored-by: ManoharSwamynathan <7479574+ManoharSwamynathan@users.noreply.github.com>
|
Superseded by #13159. This change (make memory reads tolerate a non-UTF-8
Your authorship is preserved on the commit in #13159 (via |
The pure readers (recent-history assembly and index rebuild) and the context builders decoded memory files with read_text(encoding="utf-8"): one non-UTF-8 byte raised UnicodeDecodeError up through the reader, breaking get_context / activity_index (both reached from build_session_context) and an index rebuild. Keep every existing read exactly as before (same single call, same containment) and wrap ONLY the decode failure: on UnicodeDecodeError log a warning naming the file and skip that entry/section. No new read path, no reopen, no errors="replace", no new file-opening helper. Sites: _read_recent_history_uncached (skip the day), the two rebuild_index reads (skip the file), and get_context / activity_index consumption of preferences+projects on the every-turn context path (skip the section) so context assembly cannot crash while the readers stay strict. read_preferences, read_projects and append_history keep a strict decode (unchanged): each feeds a read-modify-write path (the consolidator's CAS baseline, add_preference, the dashboard Save, the history append), so an undecodable file must raise and be left intact rather than round-tripped lossily and overwritten with no backup on the default V1 path. The context builders catch the strict reader's error at consumption instead. Trade-off: a corrupt history file is left out of the FTS index until it is repaired; V1 search will not surface its text meanwhile. This is preferred over a reopen (which would race a concurrent delete) or a lossy re-decode. Also prune the now-black-clean test/test_knowledge.py from the shrinking black baseline (.github/black-baseline.txt), which the black gate requires. Tests: the pure readers skip a corrupt file and still return clean entries; get_context survives a corrupt preferences.md and activity_index survives a corrupt projects.md; the read-modify-write readers raise and leave the corrupt file intact. Simplifies #8378: the smallest correct fix, no new config, module, or degradation-state machinery. Co-authored-by: iamwhatever <iamwhatever@users.noreply.github.com> Co-authored-by: ManoharSwamynathan <7479574+ManoharSwamynathan@users.noreply.github.com>
…dev#13159) The pure readers (recent-history assembly and index rebuild) and the context builders decoded memory files with read_text(encoding="utf-8"): one non-UTF-8 byte raised UnicodeDecodeError up through the reader, breaking get_context / activity_index (both reached from build_session_context) and an index rebuild. Keep every existing read exactly as before (same single call, same containment) and wrap ONLY the decode failure: on UnicodeDecodeError log a warning naming the file and skip that entry/section. No new read path, no reopen, no errors="replace", no new file-opening helper. Sites: _read_recent_history_uncached (skip the day), the two rebuild_index reads (skip the file), and get_context / activity_index consumption of preferences+projects on the every-turn context path (skip the section) so context assembly cannot crash while the readers stay strict. read_preferences, read_projects and append_history keep a strict decode (unchanged): each feeds a read-modify-write path (the consolidator's CAS baseline, add_preference, the dashboard Save, the history append), so an undecodable file must raise and be left intact rather than round-tripped lossily and overwritten with no backup on the default V1 path. The context builders catch the strict reader's error at consumption instead. Trade-off: a corrupt history file is left out of the FTS index until it is repaired; V1 search will not surface its text meanwhile. This is preferred over a reopen (which would race a concurrent delete) or a lossy re-decode. Also prune the now-black-clean test/test_knowledge.py from the shrinking black baseline (.github/black-baseline.txt), which the black gate requires. Tests: the pure readers skip a corrupt file and still return clean entries; get_context survives a corrupt preferences.md and activity_index survives a corrupt projects.md; the read-modify-write readers raise and leave the corrupt file intact. Simplifies kirodotdev#8378: the smallest correct fix, no new config, module, or degradation-state machinery. Co-authored-by: iamwhatever <iamwhatever@users.noreply.github.com> Co-authored-by: ManoharSwamynathan <7479574+ManoharSwamynathan@users.noreply.github.com>
Problem / Motivation
A single non-UTF-8 byte in
preferences.md(orprojects.md) takes down everyreader with a raw
UnicodeDecodeError.read_preferencesandread_projectsread with a bare
read_text(encoding="utf-8")and never guard the decode:Why it matters
The blast radius is wide and includes the recovery path:
GET /api/memory/preferences→ 500,get_contextprompt assembly → raises on every turn,add_preference/write_preferences(with abaseline) read before they write — so the API cannot fix what the API cannot
read. One bad byte from a crash-mid-write, an editor encoding slip, or a
synced file is enough to wedge it.
What changed (motivation → approach → change)
Root cause: two direct readers decode without a fallback, unlike the rest of the
memory read surface. The codebase already solved this exact case in
_guarded_entry(whichmarkdown_snapshot()uses): an undecodable file → alogged warning + an empty result, never a traceback. Approach: apply that same,
already-blessed degradation to the two readers that missed it, rather than
inventing new behavior. Change: route both
read_preferencesandread_projectsthrough a shared
_read_markdown_utf8helper that catchesUnicodeDecodeError,logs a warning, and returns
""— so the file reads as empty (the same shape asa missing file) and the next write overwrites it cleanly, restoring the repair
path.
Tests
New
TestReaderUtf8Degradationintest/test_memory_markdown_read.py:read_preferences/read_projectson a file with non-UTF-8 bytes return""instead of raising.
add_preferencerecovers from an undecodable file (reads-then-writes withoutraising, and the file is valid UTF-8 again afterward) — the repair-path
regression.
These complement the existing
TestGuardedReadRobustness, which covers themarkdown_snapshot()/_guarded_entrypath but not these direct readers.Manual verification
N/A — unit coverage sufficient. The dashboard 500 and
get_contextfailures arethe same two readers this change guards; the tests exercise them directly.
Related Issues
Fixes #8247
Pattern harvest
Rule candidate: review-prompt
Pattern: reading an agent-writable file with a bare
read_text(encoding="utf-8")and no
UnicodeDecodeErrorguard crashes every reader on a single bad byte;such reads should degrade to empty (the
_guarded_entryprecedent), and a newone is easy to add without noticing the established pattern.
Checklist
feat|fix|docs|refactor|perf|test|chore|ci|build|revert: ...)