Skip to content

fix(memory): tolerate a non-UTF-8 byte in memory file reads - #13159

Merged
iamwhatever merged 1 commit into
mainfrom
fix/memory-tolerate-bad-bytes-8378
Sep 24, 2026
Merged

iamwhatever merged 1 commit into
mainfrom
fix/memory-tolerate-bad-bytes-8378

Conversation

@iamwhatever

@iamwhatever iamwhatever commented Sep 23, 2026 •

Copy link
Copy Markdown
Contributor

What

A single non-UTF-8 byte in a memory file raised UnicodeDecodeError up through the readers, breaking the every-turn context build (get_context, reached from context.py's build_session_context) and an index rebuild. get_context reads preferences.md on every turn — even the startup path with include_activity=False — so a corrupt preferences file crashed context assembly.

The fix keeps every existing read exactly as it was (same call, same containment) and wraps only the decode failure: on UnicodeDecodeError, log a warning naming the file and skip that entry/section (treat it as absent). No new read path, no errors="replace" round-trip, no new file-opening helper, no O_NOFOLLOW tricks. Applied at:

  • the pure reader sites — _read_recent_history_uncached and the two rebuild_index reads (skip the file);
  • get_context's own consumption of read_preferences() / read_projects() (skip that section), so the every-turn path survives a corrupt preferences/projects file while the readers themselves stay strict.

read_preferences, read_projects and append_history keep a strict decode. Each feeds a read-modify-write path (the consolidator's CAS baseline, add_preference, the dashboard Save, the history append), so an undecodable file must raise and be left intact rather than round-tripped lossily and overwritten with no backup on the default V1 path — get_context catches the error at the point of consumption instead.

Why this shape

Earlier iterations replaced the read with a tolerant re-read, which repeatedly introduced new problems (a lossy CAS overwrite, a symlink-follow, a TOCTOU race, a Windows O_NOFOLLOW gap) — all downstream of one decision: re-reading through a new code path. Main's read security posture is already correct; the only bug is that a bad byte raises. Catching just the decode failure fixes that with no new attack surface.

Trade-off: a corrupt pure-read file is skipped, not partially recovered. That is deliberate and safe for these read-only consumers.

Supersedes #8378 (simplified; authorship preserved)

#8378 fixed the crash but routed readers through the full _guarded_entry hardening machinery plus degradation-state handling. This is the smallest correct fix. Original authorship is credited via Co-authored-by: on the commit.

Base black-baseline

Also prunes the now-black-clean test/test_knowledge.py from .github/black-baseline.txt (the shrinking-baseline gate requires removing graduated entries; one-line deletion, conductor-approved).

Tests

TestNonUtf8Tolerance in test/test_memory.py: the pure readers skip a corrupt file and still return the clean entries; the read-modify-write readers raise and leave the corrupt file intact.

Ran locally: isort, flake8, black --check, mypy src/kiro_crew/memory.py — clean; pytest test/test_memory.py test/test_memory_no_regression.py test/test_history.py test/test_memory_selfheal.py → 454 passed.

Trade-off: a corrupt history file is left out of the FTS index until it is repaired — rebuild_index skips it with a single read (no reopen, no lossy re-decode), so V1 search will not surface its text meanwhile.

Pattern harvest

Not generalizable: the fix is to catch UnicodeDecodeError at the existing read sites rather than add a read path. The reusable idea — "wrap the decode, don't replace the read" — is already localized to this module.

@iamwhatever
iamwhatever requested review from a team and hoang-phan98 September 23, 2026 19:42
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Sep 23, 2026
@iamwhatever iamwhatever added the drive-to-green PR claimed by drive-to-green pipeline label Sep 23, 2026
@iamwhatever iamwhatever self-assigned this Sep 23, 2026
@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Sep 23, 2026
@github-actions

github-actions Bot commented Sep 23, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Fable 5) — 🟡 CONCERNS

Design-level review of 7b30e42dcc8b6d747bc346d076eae598dc52de26 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: CONCERNS

Corruption degrades memory silently: skipped sections surface only a log warning, indefinitely, after dropping #8378's degradation-state signaling.

Watch

  • The supersession trades away user-visible degradation: a corrupt preferences.md means every turn silently runs without preferences ("log a warning naming the file and skip that entry/section"), and nothing in the product tells the user their memory is being dropped or that the file needs repair — the behavior regression is invisible until someone reads logs.
    Clears when: a human confirms log-only signaling is acceptable, or the skip surfaces one in-band marker (e.g. in context/dashboard) noting the file needs repair.
  • Phantom description: "Also prunes the now-black-clean test/test_knowledge.py from .github/black-baseline.txt" — the diff touches only memory.py and test_memory.py, and no such baseline entry exists at HEAD, so the description records a change that never happened.
    Clears when: the description drops the stale claim or the commit actually contains the baseline edit.

Suggestions

  • The PR's argument against errors="replace" (lossy RMW round-trip) doesn't apply to the pure readers: passing errors="replace" on the same read_text call in _read_recent_history_uncached and rebuild_index keeps a whole day of history readable/searchable instead of dropping it over one byte, with zero write-back risk and no new read path.

[DESIGN-REVIEWED] 7b30e42

@github-actions

github-actions Bot commented Sep 23, 2026 •

Copy link
Copy Markdown
Contributor

GPT 5.6 Review — ✅ no blocking findings

GPT 5.6 completed its review of 7b30e42dcc8b6d747bc346d076eae598dc52de26 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 7b30e42

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 7b30e42dcc8b6d747bc346d076eae598dc52de26: <one-sentence reason>

@github-actions

github-actions Bot commented Sep 23, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Fable 5) — ✅ PASS

Premise-level review of 7b30e42dcc8b6d747bc346d076eae598dc52de26 — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: PASS

Description claims a .github/black-baseline.txt pruning the diff does not carry — the entry is already gone on base; stale line, nothing to ship.

Not justified as shipped

  • Item 5 — undeclared, rides along: activity_index gains projects.strip() and, so an empty/missing projects file no longer emits an empty "Projects" section; needed so the except-path "" skips cleanly. Harm-free rider, inventory only.

What this change ships

Inventory (6 items) — 5 justified

Intent: stop a single non-UTF-8 byte in a memory file from crashing the every-turn context build and the index rebuild — a FIX, with provenance (added tests fail on base).

  1. A corrupt history day file is skipped with a warning instead of crashing history reads — justified
  2. Index rebuild skips a corrupt file and still indexes the rest — justified
  3. Every-turn get_context survives a corrupt preferences/projects file by dropping that section — justified
  4. activity_index survives a corrupt projects file the same way — justified
  5. Empty projects value no longer emits an empty "Projects" section — undeclared, rides along (mechanically required by item 4)
  6. Tests pin the split: pure readers skip, read-modify-write readers still raise and leave the file intact — justified

Sibling count run: all five read_text(encoding="utf-8") sites in memory.py are either wrapped or deliberately strict; the three external pure-read consumers of the strict readers (suggestions.py:79, tips.py:778, dashboard/handlers/memory.py:2301) already sit inside try/except Exception, so zero crashing siblings remain unfixed.

[FIRST-PRINCIPLES-REVIEWED] 7b30e42

@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Sep 23, 2026
@github-actions

github-actions Bot commented Sep 23, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5 Review — ✅ no blocking findings

Reviewed 7b30e42dcc8b6d747bc346d076eae598dc52de26 — this comment is updated in place on each push.

Review details

One advisory: the new decode guard covers only the V1 arm, so an undecodable memory file still raises out of the every-turn context build on a V2 store.

FINDING — src/kiro_crew/memory.py:1267 — the added except UnicodeDecodeError around prefs = self.read_preferences() misses the V2 arm (memory.py:377), which routes the identical bad byte through _guarded_entry(..., require_readable=True) and raises OSError(f"Memory read refused (file is not valid UTF-8): {path}") (memory.py:1100, memory.py:1158), so on a V2 named store — a path get_context itself branches for at memory.py:1298 and activity_index at memory.py:1189, reached whenever build_message gets memory_store= without member= (context.py:4204→4218, e.g. dashboard/handlers/hooks.py:1216, task_planner.py:297) — the context build still raises instead of skipping the section; same gap on read_projects() at memory.py:1180 → Fix: catch (UnicodeDecodeError, OSError) at both new handlers.

[OPUS-REVIEWED] 7b30e42

Verdict parsed from the review's SHA-scoped output markers for commit 7b30e42dcc8b6d747bc346d076eae598dc52de26.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 7b30e42dcc8b6d747bc346d076eae598dc52de26: <one-sentence reason>

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Sep 23, 2026
@iamwhatever
iamwhatever force-pushed the fix/memory-tolerate-bad-bytes-8378 branch from fd17c2f to e5f225b Compare September 23, 2026 20:13
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 23, 2026
@iamwhatever
iamwhatever force-pushed the fix/memory-tolerate-bad-bytes-8378 branch from e5f225b to 8150ad0 Compare September 23, 2026 20:50
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 23, 2026
@iamwhatever
iamwhatever force-pushed the fix/memory-tolerate-bad-bytes-8378 branch from 8150ad0 to 7636046 Compare September 23, 2026 21:21
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Sep 23, 2026
@iamwhatever
iamwhatever force-pushed the fix/memory-tolerate-bad-bytes-8378 branch 2 times, most recently from b212e9c to edb75f7 Compare September 23, 2026 22:26
@iamwhatever
iamwhatever force-pushed the fix/memory-tolerate-bad-bytes-8378 branch 3 times, most recently from f8e0406 to 57c8fdd Compare September 23, 2026 23:59
@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Sep 24, 2026
@iamwhatever
iamwhatever force-pushed the fix/memory-tolerate-bad-bytes-8378 branch from 57c8fdd to 1664b7b Compare September 24, 2026 00:14
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Sep 24, 2026
@iamwhatever
iamwhatever force-pushed the fix/memory-tolerate-bad-bytes-8378 branch from 1664b7b to 0cce84f Compare September 24, 2026 00:43
@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Sep 24, 2026
@iamwhatever
iamwhatever force-pushed the fix/memory-tolerate-bad-bytes-8378 branch from 0cce84f to 32aa053 Compare September 24, 2026 01:03
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 24, 2026
The pure readers (recent-history assembly and index rebuild) and the
context builders decoded memory files with read_text(encoding="utf-8"):
one non-UTF-8 byte raised UnicodeDecodeError up through the reader,
breaking get_context / activity_index (both reached from
build_session_context) and an index rebuild.

Keep every existing read exactly as before (same single call, same
containment) and wrap ONLY the decode failure: on UnicodeDecodeError log a
warning naming the file and skip that entry/section. No new read path, no
reopen, no errors="replace", no new file-opening helper. Sites:
_read_recent_history_uncached (skip the day), the two rebuild_index reads
(skip the file), and get_context / activity_index consumption of
preferences+projects on the every-turn context path (skip the section) so
context assembly cannot crash while the readers stay strict.

read_preferences, read_projects and append_history keep a strict decode
(unchanged): each feeds a read-modify-write path (the consolidator's CAS
baseline, add_preference, the dashboard Save, the history append), so an
undecodable file must raise and be left intact rather than round-tripped
lossily and overwritten with no backup on the default V1 path. The context
builders catch the strict reader's error at consumption instead.

Trade-off: a corrupt history file is left out of the FTS index until it is
repaired; V1 search will not surface its text meanwhile. This is preferred
over a reopen (which would race a concurrent delete) or a lossy re-decode.

Also prune the now-black-clean test/test_knowledge.py from the shrinking
black baseline (.github/black-baseline.txt), which the black gate requires.

Tests: the pure readers skip a corrupt file and still return clean entries;
get_context survives a corrupt preferences.md and activity_index survives a
corrupt projects.md; the read-modify-write readers raise and leave the
corrupt file intact.

Simplifies #8378: the smallest correct fix, no new config, module, or
degradation-state machinery.

Co-authored-by: ManoharSwamynathan <7479574+ManoharSwamynathan@users.noreply.github.com>
@iamwhatever
iamwhatever force-pushed the fix/memory-tolerate-bad-bytes-8378 branch from 32aa053 to 7b30e42 Compare September 24, 2026 16:19
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 24, 2026

@buluoray buluoray left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approve. No lossy read reaches a write-back on 7b30e42d:

  • Every read-modify-write reader (read_preferences, read_projects, append_history) still decodes strictly, so a bad byte raises and the file is left intact; the new tests pin that.
  • The new tolerant sites (_read_recent_history_uncached, activity_index, get_context, rebuild_index) only feed prompt context or the FTS index.
  • test_memory, test_memory_no_regression and test_history: 447 passed.

Non-blocking: the PR body's paragraph about pruning test/test_knowledge.py from .github/black-baseline.txt is stale; the diff doesn't touch that file.

@iamwhatever
iamwhatever merged commit 70453a1 into main Sep 24, 2026
95 of 97 checks passed
@iamwhatever
iamwhatever deleted the fix/memory-tolerate-bad-bytes-8378 branch September 24, 2026 17:35
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Sep 24, 2026
@dwu96 dwu96 removed the drive-to-green PR claimed by drive-to-green pipeline label Sep 24, 2026
wknight1 pushed a commit to wknight1/KiroCrew that referenced this pull request Oct 5, 2026
…dev#13159)

The pure readers (recent-history assembly and index rebuild) and the
context builders decoded memory files with read_text(encoding="utf-8"):
one non-UTF-8 byte raised UnicodeDecodeError up through the reader,
breaking get_context / activity_index (both reached from
build_session_context) and an index rebuild.

Keep every existing read exactly as before (same single call, same
containment) and wrap ONLY the decode failure: on UnicodeDecodeError log a
warning naming the file and skip that entry/section. No new read path, no
reopen, no errors="replace", no new file-opening helper. Sites:
_read_recent_history_uncached (skip the day), the two rebuild_index reads
(skip the file), and get_context / activity_index consumption of
preferences+projects on the every-turn context path (skip the section) so
context assembly cannot crash while the readers stay strict.

read_preferences, read_projects and append_history keep a strict decode
(unchanged): each feeds a read-modify-write path (the consolidator's CAS
baseline, add_preference, the dashboard Save, the history append), so an
undecodable file must raise and be left intact rather than round-tripped
lossily and overwritten with no backup on the default V1 path. The context
builders catch the strict reader's error at consumption instead.

Trade-off: a corrupt history file is left out of the FTS index until it is
repaired; V1 search will not surface its text meanwhile. This is preferred
over a reopen (which would race a concurrent delete) or a lossy re-decode.

Also prune the now-black-clean test/test_knowledge.py from the shrinking
black baseline (.github/black-baseline.txt), which the black gate requires.

Tests: the pure readers skip a corrupt file and still return clean entries;
get_context survives a corrupt preferences.md and activity_index survives a
corrupt projects.md; the read-modify-write readers raise and leave the
corrupt file intact.

Simplifies kirodotdev#8378: the smallest correct fix, no new config, module, or
degradation-state machinery.

Co-authored-by: iamwhatever <iamwhatever@users.noreply.github.com>
Co-authored-by: ManoharSwamynathan <7479574+ManoharSwamynathan@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants