Skip to content

fix(memory): degrade an undecodable preferences/projects file to empty - #8378

Closed
ManoharSwamynathan wants to merge 1 commit into
kirodotdev:mainfrom
ManoharSwamynathan:fix/memory-non-utf8-read
Closed

ManoharSwamynathan wants to merge 1 commit into
kirodotdev:mainfrom
ManoharSwamynathan:fix/memory-non-utf8-read

Conversation

@ManoharSwamynathan

Copy link
Copy Markdown
Contributor

Problem / Motivation

A single non-UTF-8 byte in preferences.md (or projects.md) takes down every
reader with a raw UnicodeDecodeError. read_preferences and read_projects
read with a bare read_text(encoding="utf-8") and never guard the decode:

store.read_preferences()   # UnicodeDecodeError: 'utf-8' codec can't decode byte 0xff
store.add_preference('x')  # same — it reads first

Why it matters

The blast radius is wide and includes the recovery path:

  • dashboard GET /api/memory/preferences → 500,
  • get_context prompt assembly → raises on every turn,
  • and the repair path itself — add_preference / write_preferences (with a
    baseline) read before they write — so the API cannot fix what the API cannot
    read. One bad byte from a crash-mid-write, an editor encoding slip, or a
    synced file is enough to wedge it.

What changed (motivation → approach → change)

Root cause: two direct readers decode without a fallback, unlike the rest of the
memory read surface. The codebase already solved this exact case in
_guarded_entry (which markdown_snapshot() uses): an undecodable file → a
logged warning + an empty result, never a traceback. Approach: apply that same,
already-blessed degradation to the two readers that missed it, rather than
inventing new behavior. Change: route both read_preferences and read_projects
through a shared _read_markdown_utf8 helper that catches UnicodeDecodeError,
logs a warning, and returns "" — so the file reads as empty (the same shape as
a missing file) and the next write overwrites it cleanly, restoring the repair
path.

Tests

New TestReaderUtf8Degradation in test/test_memory_markdown_read.py:

  • read_preferences / read_projects on a file with non-UTF-8 bytes return ""
    instead of raising.
  • add_preference recovers from an undecodable file (reads-then-writes without
    raising, and the file is valid UTF-8 again afterward) — the repair-path
    regression.

These complement the existing TestGuardedReadRobustness, which covers the
markdown_snapshot() / _guarded_entry path but not these direct readers.

Manual verification

N/A — unit coverage sufficient. The dashboard 500 and get_context failures are
the same two readers this change guards; the tests exercise them directly.

Related Issues

Fixes #8247

Pattern harvest

Rule candidate: review-prompt
Pattern: reading an agent-writable file with a bare read_text(encoding="utf-8")
and no UnicodeDecodeError guard crashes every reader on a single bad byte;
such reads should degrade to empty (the _guarded_entry precedent), and a new
one is easy to add without noticing the established pattern.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality (new regression tests added; full suite validated by this PR's CI)
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable) (N/A — internal robustness, no documented behavior change)
  • No secrets, credentials, or internal references in the diff

@ManoharSwamynathan
ManoharSwamynathan requested review from a team and smeyffret September 4, 2026 05:09
@github-actions github-actions Bot added fork Pull request from a fork (external contributor) readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Sep 4, 2026
@github-actions

github-actions Bot commented Sep 4, 2026 •

Copy link
Copy Markdown
Contributor

Opus 4.8 Review (fork) — ✅ no blocking findings

Reviewed 8d6e8389165467468013a0f688f40012218397c6 via the fork AI-review pipeline; updated in place on each push.

Review details

Both candidates were rated low-confidence by the discovery pass, and each fails the falsification bar:

  • Candidate 1 (FileTooLargeError → "" data loss): the only trigger is a preferences.md/projects.md exceeding the 50 MB cap (MAX_FILE_BYTES = 50 * 1024 * 1024). These files grow one short line per add_preference and are never fed multi-MB content in practice — the input condition does not occur, failing bar (a). The old read_text would indeed have read the whole file, but reaching this regression requires an unrealistic 50 MB managed markdown file.
  • Candidate 2 (rebuild_index history crash): the offending line path.read_text(encoding="utf-8") in the history glob loop is unchanged context — identical in the base (memory.py:996). The PR does not touch it and explicitly documents deferring it. It is a pre-existing crash on a corrupt/planted history file, not a defect the diff introduces in changed lines.

Neither survives Step 1; nothing new grounds to the 80+ bar in Step 2.

No findings.

[OPUS-REVIEWED] 8d6e838

@github-actions

github-actions Bot commented Sep 4, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Fable 5, fork) — 🟡 CONCERNS

Design-level review of 8d6e8389165467468013a0f688f40012218397c6 via the fork AI-review pipeline — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

I have enough context — base machinery (_guarded_entry, safe_read_file_bytes_nolink, lock/CAS write paths) all verified against the diff.

Design-Verdict: CONCERNS

Shipped design is sound — but the title/description document the opposite contract (empty-on-undecodable) and omit half the diff's behavior changes.

Watch

  • Description↔diff mismatch on the central design point: the title and body say the helper "returns "" — so the file reads as empty… and the next write overwrites it cleanly," but the code decodes with errors="replace" and the tests assert content is preserved, explicitly rejecting the described behavior ("returning "" … would make the next write discard every preference"). The shipped choice is the better one; the description was written for a different revision. Anyone triaging [Bug]: non-UTF-8 byte in preferences.md crashes every reader (dashboard tab, prompt assembly, and the repair path) #8247 or reading the merge commit will expect empty semantics. Rewrite title/body to match.
  • Undocumented scope: the diff also adds symlink/hardlink/size-cap hardening to read_preferences/read_projects, guards append_history/read_recent_history, and reroutes rebuild_index — none mentioned in the description. The hardening has a named cause (agent-writable memory dir), but it changes refusal semantics on a read-then-write path and deserves its own statement, ideally its own PR.
  • The new helper empties on refusal (e.g. size-cap) while preserving on undecodable bytes; a legitimate oversized preferences.md read as "" feeds add_preference, whose next write truncates the file to one line — the exact silent-loss shape the helper's own docstring argues against. Consider raising or skipping the write on refusal in repair paths.

[DESIGN-REVIEWED] 8d6e838

@github-actions

github-actions Bot commented Sep 4, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Fable 5, fork) — 🔴 BLOCK

Premise-level review of 8d6e8389165467468013a0f688f40012218397c6 via the fork AI-review pipeline — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

All the evidence is in. The decisive fact: the PR's title and description claim degrade-to-empty via the _guarded_entry precedent, while the shipped code deliberately does the opposite (errors="replace", content preserved) and its own test docstring calls the described behavior a data-loss bug. Half the diff's call-site changes are also undeclared. Emitting the review.

First-Principles-Verdict: BLOCK

The title promises "degrade … to empty"; the diff ships preserve-with-replacement, and its own tests call the described behavior a data-loss regression.

What this change ships

Intent: stop one bad byte in a memory file from crashing every reader and the repair path — a FIX.

  1. read_preferences/read_projects survive a bad byte, keeping content as U+FFFD — the fix, but contradicts the stated approach
  2. Those two readers now refuse symlinks/hardlinks/special/oversized files — undeclared; derived (agent-writable dir boundary)
  3. add_preference repair path works on an undecodable file, preserving survivors — justified
  4. append_history no longer crashes on a bad byte in today's file — undeclared, rides along (same root cause)
  5. Every-turn recent-history read now empties bad/linked/oversized files instead of crashing — undeclared, rides along
  6. rebuild_index reads prefs/projects via the hardened readers — undeclared; removes a second spelling
  7. rebuild_index history files still crash on the same byte — symptom-level, admitted unfixed sibling
  8. New warning logs + audit refusal events on the two readers — rides along with item 2

Blockers

  • Framing contradicted by the diff. Description: the helper "returns "" — so the file reads as empty"; tests "return "" instead of raising". Diff: return data.decode("utf-8", errors="replace"), and the test asserts "survives, not blanked" with a docstring saying "Before the fix the reader returned "" and this wiped the file." The description's whole justification — "apply that same, already-blessed degradation … rather than inventing new behavior" — is negated: the docstring names its policy "the deliberate difference from _guarded_entry". Resolution: either ship the claimed mechanism (return self._guarded_entry(path)["content"], deleting the ~50-line divergent helper) or withdraw the degrade-to-empty claim so the diff and the approved description match.

Watch

  • Unfixed sibling: grepped read_text(encoding="utf-8") in memory.py — 6 pre-patch sites, 5 fixed, 1 left (rebuild_index history glob, base memory.py:996), still raising on the exact byte class this PR fixes; the deferred fix is the same one-liner shipped at the append_history site.
  • _read_markdown_utf8 re-implements _guarded_entry's five-step guard chain (root guard, leaf-link, S_ISREG, nolink read, size cap) minus the mtime-stability retry — two hardened readers with silently divergent guarantees will drift.
  • Items 4–6 are undeclared sibling fixes; correct in direction, but the description reviewed is not the change shipped.

Subtractions

  • Drop the self._preferences_file.exists() / self._projects_file.exists() pre-checks in read_preferences/read_projects — the helper's OSError branch already returns "" for a missing file; the extra stat is a redundant step.

[FIRST-PRINCIPLES-REVIEWED] 8d6e838

@github-actions

github-actions Bot commented Sep 4, 2026 •

Copy link
Copy Markdown
Contributor

GPT 5.6 Review (fork) — 🔴 changes requested (blocking)

Reviewed 8d6e8389165467468013a0f688f40012218397c6 via the fork AI-review pipeline; updated in place on each push.

2 of 2 blocking finding(s) are security-class and were withheld from adjudication, so the blocking verdict stands.

BLOCKING -- src/kiro_crew/memory.py:355 -- Root guard runs after an unsafe UNC probe
return self._read_markdown_utf8(self._preferences_file)
Untrusted Windows UNC workspace -> dashboard preferences read -> .exists() contacts the SMB host before _read_root_guard() -> credential probe.
Anchor: residual/security
Fix: Call the helper unconditionally and remove equivalent pre-guard .exists() probes.

BLOCKING -- src/kiro_crew/memory.py:339 -- Oversized preferences can be silently overwritten
return ""
Oversized preferences file -> consolidation reads an empty baseline twice -> compare-and-swap succeeds -> existing preferences are replaced.
Anchor: residual/crash-data-loss-corruption
Fix: Propagate oversized/read-refusal failures instead of representing them as empty content.

FINDING -- src/kiro_crew/memory.py:335 -- "safe_read_file_bytes_nolink" adds undeclared credential-path hardening, while the history hunks also change append/context behavior beyond the stated preferences/projects repair -> Fix: remove these unrelated hunks and submit them under a declared scope.

[BLOCK-MERGE] 8d6e838
[GPT-REVIEWED] 8d6e838

Adjudication (Opus 4.8) — is blocking on each finding proportionate?

Both findings are FENCED (annotate-only); the adjudicable block is empty. My evidence review of the base tree and the pre-fetched diff:

F1 — read_preferences (diff hunk, new memory.py:355) does if self._preferences_file.exists(): return self._read_markdown_utf8(...). The .exists() runs before _read_root_guard() (invoked only inside _read_markdown_utf8). _read_root_guard's own docstring (memory.py:611-613) states stat/glob/exists on a UNC path is itself the outbound SMB credential probe and declares the invariant that no syscall may touch an un-admitted path. The pre-guard .exists() violates that invariant on a Windows UNC workspace. Harm rung: UNBOUNDED (credential/NTLM probe, one-shot, no recovery path). Condition — Windows UNC-shaped, non-allowed workspace — is operator-producible, not extreme; the guard exists precisely for it. Not FLAG-eligible.

F2 — _read_markdown_utf8's FileTooLargeError branch returns "" (diff, new memory.py:339 region), with MAX_FILE_BYTES = 50 MB (hooks.py:2237). This is a read-then-write repair reader (add_preference memory.py:339; write_preferences CAS at memory.py:322): an oversized file reads as "", the baseline re-read inside the lock also yields "", CAS matches, and _atomic_write_text (memory.py:328) overwrites the >50 MB content. The PR chose errors="replace" over "" for undecodable bytes to avoid exactly this silent blanking, but did not apply the same reasoning to the size cap. Harm rung: UNBOUNDED (silent, unrecoverable data loss). No write-side size cap guarantees preferences stay under 50 MB; consequence is silent. When torn on an unbounded, unrecoverable class → UPHOLD-FENCED.

[ADJUDICATION] 8d6e8389165467468013a0f688f40012218397c6 total=0 uphold=0 downgrade=0
[GPT-ADJUDICATED] 8d6e8389165467468013a0f688f40012218397c6
[ADJUDICATION-FENCED] 8d6e8389165467468013a0f688f40012218397c6 fenced=2 flagged=0
UPHOLD-FENCED F1 src/kiro_crew/memory.py:355 -- Pre-guard `.exists()` fires the UNC/SMB credential probe the root guard exists to prevent; operator-producible condition, unrecoverable leak.
UPHOLD-FENCED F2 src/kiro_crew/memory.py:339 -- Oversized read returns "" on a read-then-write path, CAS matches empty baseline, and the >50MB file is silently overwritten with no recovery.
[GPT-ADJUDICATED-FENCED] 8d6e8389165467468013a0f688f40012218397c6

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention merge conflict Branch has merge conflicts with its base — author must resolve before merge and removed readiness: checking Automated validation is still running labels Sep 4, 2026
@chenmingwei23 chenmingwei23 added needs-pr-triage PR scanner: awaiting automated triage needs-author-decision PR blocked on author input and removed needs-pr-triage PR scanner: awaiting automated triage labels Sep 7, 2026
@chenmingwei23

Copy link
Copy Markdown
Contributor

Kiro Crew [operator: chenmingwei23#de330d0c]: This PR has been inactive for 7+ days. I reviewed the blockers but they require your input:

  • The core "degrade an undecodable file to empty ('')" design is disputed by all three review lanes as a data-loss amplifier: because add_preference / the consolidator read-then-write, one bad byte makes the next write replace the whole file, permanently discarding the preferences that survived around it. Before this PR the corruption was loud and recoverable from disk. The reviewers propose competing remedies -- errors="replace" to preserve most content (Design), rename the file aside to preferences.md.corrupt before treating it as empty (Design), or route through _guarded_entry(path)["content"] (GPT). These conflict, and each changes the PR's central choice, so this is a design decision for you rather than a mechanical fix.
  • GPT also flags a security angle at memory.py:302: the raw read follows a symlink, so an agent-planted credential symlink could be read by the dashboard reader. Its fix (_guarded_entry) overlaps with, but is not identical to, the data-loss remedies above -- pick one path deliberately.
  • First Principles notes the same bare-decode pattern survives at 4 more sites in this file (append_history, _read_recent_history_uncached on the every-turn get_context path, and rebuild_index which still crashes on the very byte this PR fixes), contradicting the description's "unlike the rest of the memory read surface" claim. Decide whether to scope those in or keep the PR to the 2 readers.
  • The branch also conflicts with main (needs a rebase), which I can help with once the design direction above is settled.

When you've addressed these, the pipeline will re-assess on its next cycle. If you'd prefer no automation on this PR, add the pr-no-autofix label.

@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Sep 7, 2026
@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running and removed readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention labels Sep 7, 2026
@ManoharSwamynathan

Copy link
Copy Markdown
Contributor Author

Thanks for the detailed review — addressed in the latest push. I took a combined path rather than any single suggestion, because none of the three alone satisfied both blocking findings:

  • Symlink follow (GPT, memory.py:302): read_preferences/read_projects now route through the same hardening _guarded_entry uses — the _read_root_guard admission gate, a leaf reparse-point / non-regular-file reject, and safe_read_file_bytes_nolink (O_NOFOLLOW, hardlink/sensitive-target reject, size cap) confined to the memory root. The raw read no longer follows a planted link.
  • Data loss (GPT :305 / Design): but I decode with errors="replace" instead of returning "". _guarded_entry's empty-on-undecodable is right for the read-only snapshot (leak nothing); these readers feed the read-before-write repair path, where blanking would make the next write discard the preferences that survived around the bad byte. errors="replace" preserves them and the next write re-canonicalizes the file.

On the First-Principles "point patch with unfixed siblings" concern — I scoped the siblings in rather than leaving them:

  • rebuild_index now reads preferences/projects through the hardened readers (no longer a second spelling that crashes on the very byte).
  • append_history (already link/hardlink-guarded; read-then-write) decodes with errors="replace" to preserve surviving entries.
  • _read_recent_history_uncached (read-only, on the every-turn get_context path) now reads each day through the guarded _guarded_entry, so one bad byte in any of the 181 files no longer raises.

Tests updated: content preservation for both readers, the read-before-write data-loss regression, symlink refusal, and the two history sites. Also rebased onto current main, so the merge conflict is resolved.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention labels Sep 7, 2026
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 7, 2026
@bolichen97

Copy link
Copy Markdown
Contributor

Cross-reference from the open-PR relationship audit: #8248 (now closed in favour of this PR) covered one extra call site that this PR currently leaves strict -- rebuild_index's history *.md glob read. Worth folding the same errors="replace" treatment in here so the fix is complete at every reader.

@ManoharSwamynathan

Copy link
Copy Markdown
Contributor Author

Handing this off — flagging the true state so it's easy to pick up. My earlier "addressed in the latest push" reply was inaccurate: it fixed a different vector (undecodable bytes → errors="replace") and did not resolve the two GPT findings, which is why PR Readiness is still red on 8d6e838.

Blockers remaining (all gate the PR Readiness check):

  1. GPT 5.6 🔴 F1 — read_preferences/read_projects call .exists() before _read_root_guard(). On a Windows UNC workspace that .exists() is itself an outbound SMB/NTLM probe, violating the guard's invariant. Fix: drop the pre-guard .exists() and call the reader unconditionally (its OSError branch already returns "" for a missing file). First Principles' "Subtractions" note asks for the same removal.

  2. GPT 5.6 🔴 F2 — the FileTooLargeError branch returns "". On the read-before-write CAS path an oversized (>50 MB) file reads as empty, the in-lock baseline re-reads empty, CAS matches, and the file is silently overwritten. Fix: propagate the oversized/read-refusal instead of returning "" on repair paths.

  3. First Principles 🔴 (+ Design CONCERNS) — framing mismatch: the title/description claim "degrade undecodable file to empty (return "")", but the shipped code preserves content via errors="replace" (the better choice, per both reviewers), and several call-site changes (symlink/size hardening, append_history, rebuild_index) are undeclared. Fix: rewrite the title/description to match the shipped behavior and declare the scope.

Also red — Backend Tests (Windows) (3): 4 CRLF failures in test_memory.py / test_memory_markdown_read.py. Root cause: memory writes go through atomic_write with the default newline=None, which rewrites \n to os.linesep (CRLF on Windows); reads return CRLF and the empty-default check byte-compares against LF scaffolds. One-line fix at the single write chokepoint (_atomic_write_text):

atomic_write(path, content, mode=mode, newline="\n")

Verified locally (black/flake8/mypy/py_compile clean; passes on Linux). Windows is CI-verified.

Unassigning myself — open for anyone to pick up.

@ManoharSwamynathan
ManoharSwamynathan marked this pull request as draft September 10, 2026 21:07
@github-actions github-actions Bot added the merge conflict Branch has merge conflicts with its base — author must resolve before merge label Sep 11, 2026
@iamwhatever iamwhatever self-assigned this Sep 23, 2026
@iamwhatever iamwhatever added the drive-to-green PR claimed by drive-to-green pipeline label Sep 23, 2026
iamwhatever added a commit that referenced this pull request Sep 23, 2026
The V1 markdown readers decoded with a bare read_text(encoding="utf-8"):
one non-UTF-8 byte raises UnicodeDecodeError up through the reader (the
dashboard memory GET, the every-turn get_context, and rebuild_index).

Add one small helper, _read_text_tolerant(path), that decodes UTF-8 and,
on UnicodeDecodeError, logs once at warning level with the path and falls
back to errors="replace". For a valid UTF-8 file it returns the
byte-identical string a bare decode returns. Applied at the five pure
reader sites: read_preferences, read_projects, _read_recent_history_uncached,
and the two rebuild_index reads.

append_history is left on a strict decode on purpose: it is a
read-modify-write path, so a lossy read there would persist U+FFFD over
the original bytes on the next atomic write and destroy them. An
undecodable today-file raises and is left intact and recoverable.

Tests: one per pure-reader entry point that a corrupt file no longer
raises (read_preferences, read_projects, read_recent_history,
rebuild_index), a check that append_history does NOT rewrite a corrupt
file lossily, and a byte-identical check for good files.

Simplifies #8378: the smallest correct fix, no new config, module, or
degradation-state machinery.

Co-authored-by: ManoharSwamynathan <7479574+ManoharSwamynathan@users.noreply.github.com>
iamwhatever added a commit that referenced this pull request Sep 23, 2026
The recent-history assembly and index rebuild decoded memory files with a
bare read_text(encoding="utf-8"): one non-UTF-8 byte raises
UnicodeDecodeError up through the reader and, for history, breaks the
every-turn get_context.

Add one small helper, _read_text_tolerant(path), that decodes UTF-8 and,
on UnicodeDecodeError, logs once at warning level with the path and falls
back to errors="replace". For a valid UTF-8 file it returns the
byte-identical string a bare decode returns. Applied ONLY at the three
pure reader sites that never write their result back:
_read_recent_history_uncached and the two rebuild_index reads.

read_preferences, read_projects and append_history keep a strict decode on
purpose: each feeds a read-modify-write path (the consolidator's CAS
baseline, add_preference, the dashboard Save, the history append), so a
lossy errors="replace" read there would let a whole-file write persist
U+FFFD over the original bytes with no backup on the default V1 path. A
strict decode leaves an undecodable file intact and recoverable.

Tests: the two pure readers tolerate a bad history byte; the three RMW
readers raise and leave the corrupt file intact; a valid UTF-8 file reads
byte-identically.

Simplifies #8378: the smallest correct fix, no new config, module, or
degradation-state machinery.

Co-authored-by: ManoharSwamynathan <7479574+ManoharSwamynathan@users.noreply.github.com>
iamwhatever added a commit that referenced this pull request Sep 23, 2026
Recent-history assembly and index rebuild decoded memory files with a
bare read_text(encoding="utf-8"): one non-UTF-8 byte raises
UnicodeDecodeError up through the reader and, for history, breaks the
every-turn get_context.

Add one small helper, _read_text_tolerant(path), that decodes UTF-8 and,
on UnicodeDecodeError, logs once at warning level with the path and falls
back to errors="replace". For a valid UTF-8 file it returns the
byte-identical string a bare decode returns. Applied ONLY at the three
pure reader sites that never write their result back:
_read_recent_history_uncached and the two rebuild_index reads.

read_preferences, read_projects and append_history keep a strict decode on
purpose: each feeds a read-modify-write path (the consolidator's CAS
baseline, add_preference, the dashboard Save, the history append), so a
lossy errors="replace" read there would let a whole-file write persist
U+FFFD over the original bytes with no backup on the default V1 path. A
strict decode leaves an undecodable file intact and recoverable.

Also prune the now-black-clean test/test_knowledge.py from the shrinking
black baseline (.github/black-baseline.txt), which the black gate requires.

Tests: the two pure readers tolerate a bad history byte; the three RMW
readers raise and leave the corrupt file intact; a valid UTF-8 file reads
byte-identically.

Simplifies #8378: the smallest correct fix, no new config, module, or
degradation-state machinery.

Co-authored-by: ManoharSwamynathan <7479574+ManoharSwamynathan@users.noreply.github.com>
iamwhatever added a commit that referenced this pull request Sep 23, 2026
Recent-history assembly and index rebuild decoded memory files with a
bare read_text(encoding="utf-8"): one non-UTF-8 byte raises
UnicodeDecodeError up through the reader and, for history, breaks the
every-turn get_context.

The pure readers now read bytes through the hardened, link-refusing,
root-contained reader (_read_entry_bytes -> safe_read_file_bytes_nolink)
and decode them with _decode_tolerant, which falls back to
errors="replace" on a bad byte and logs once at warning level. Reading
bytes through the hardened path first means a planted dated-history
symlink (the history dir is agent-writable) can never be followed into a
sensitive file; a refused file is skipped. Applied at the three pure
reader sites: _read_recent_history_uncached and the two rebuild_index
reads.

read_preferences, read_projects and append_history keep a strict decode on
purpose: each feeds a read-modify-write path (the consolidator's CAS
baseline, add_preference, the dashboard Save, the history append), so a
lossy read there would let a whole-file write persist U+FFFD over the
original bytes with no backup on the default V1 path. A strict decode
leaves an undecodable file intact and recoverable.

Also prune the now-black-clean test/test_knowledge.py from the shrinking
black baseline (.github/black-baseline.txt), which the black gate requires.

Tests: the pure readers tolerate a bad history byte and refuse a planted
history symlink; the read-modify-write readers raise and leave the corrupt
file intact; valid UTF-8 bytes decode identically.

Simplifies #8378: the smallest correct fix, no new config, module, or
degradation-state machinery.

Co-authored-by: ManoharSwamynathan <7479574+ManoharSwamynathan@users.noreply.github.com>
iamwhatever added a commit that referenced this pull request Sep 23, 2026
Recent-history assembly and index rebuild decoded memory files with a
bare read_text(encoding="utf-8"): one non-UTF-8 byte raises
UnicodeDecodeError up through the reader and, for history, breaks the
every-turn get_context.

The pure readers now go through _read_leaf_text_tolerant, which refuses a
symlink/junction leaf and a hardlinked or non-regular inode BEFORE reading
(the same leaf guard append_history applies, and it audits the refusal via
_audit_read_refusal), then decodes UTF-8 with an errors="replace" fallback
and one warning log. Refusing the link first means a planted dated-history
symlink (the history dir is agent-writable) cannot be followed into a
sensitive file whose bytes would flow into the suggestions prompt; a
refused leaf is skipped. Applied at the three pure reader sites:
_read_recent_history_uncached and the two rebuild_index reads.

read_preferences, read_projects and append_history keep a strict decode on
purpose: each feeds a read-modify-write path (the consolidator's CAS
baseline, add_preference, the dashboard Save, the history append), so a
lossy read there would let a whole-file write persist U+FFFD over the
original bytes with no backup on the default V1 path. A strict decode
leaves an undecodable file intact and recoverable.

Also prune the now-black-clean test/test_knowledge.py from the shrinking
black baseline (.github/black-baseline.txt), which the black gate requires.

Tests: the pure readers tolerate a bad history byte and refuse a planted
history symlink; the read-modify-write readers raise and leave the corrupt
file intact; a valid UTF-8 leaf reads exactly as a raw UTF-8 decode.

Simplifies #8378: the smallest correct fix, no new config, module, or
degradation-state machinery.

Co-authored-by: ManoharSwamynathan <7479574+ManoharSwamynathan@users.noreply.github.com>
iamwhatever added a commit that referenced this pull request Sep 23, 2026
Recent-history assembly and index rebuild decoded memory files with a
bare read_text(encoding="utf-8"): one non-UTF-8 byte raises
UnicodeDecodeError up through the reader and, for history, breaks the
every-turn get_context.

The pure readers now go through _read_leaf_text_tolerant, which opens the
leaf with O_NOFOLLOW and validates the OPENED descriptor via fstat (regular
file, single link) before reading from that same fd, then decodes UTF-8
with an errors="replace" fallback and one warning log. Pinning the inode
check to the opened descriptor closes the check-to-open race: a symlink or
inode swapped in for a dated-history leaf (the history dir is
agent-writable) is refused at open, so it cannot be followed into a
sensitive file whose bytes would flow into the suggestions prompt. A
refused leaf is audited and skipped. Applied at the three pure reader
sites: _read_recent_history_uncached and the two rebuild_index reads.

read_preferences, read_projects and append_history keep a strict decode on
purpose: each feeds a read-modify-write path (the consolidator's CAS
baseline, add_preference, the dashboard Save, the history append), so a
lossy read there would let a whole-file write persist U+FFFD over the
original bytes with no backup on the default V1 path. A strict decode
leaves an undecodable file intact and recoverable.

Also prune the now-black-clean test/test_knowledge.py from the shrinking
black baseline (.github/black-baseline.txt), which the black gate requires.

Tests: the pure readers tolerate a bad history byte and refuse a planted
history symlink; the read-modify-write readers raise and leave the corrupt
file intact; a valid UTF-8 leaf reads exactly as a raw UTF-8 decode.

Simplifies #8378: the smallest correct fix, no new config, module, or
degradation-state machinery.

Co-authored-by: ManoharSwamynathan <7479574+ManoharSwamynathan@users.noreply.github.com>
iamwhatever added a commit that referenced this pull request Sep 23, 2026
The pure readers (recent-history assembly and index rebuild) decoded
memory files with read_text(encoding="utf-8"): one non-UTF-8 byte raised
UnicodeDecodeError up through the reader, breaking the every-turn
get_context and a rebuild.

Keep every existing read exactly as before (same call, same containment)
and wrap ONLY the decode: on UnicodeDecodeError log a warning naming the
file and skip that entry, treating it as absent. No new read path, no
errors="replace" round-trip, no new file-opening helper. Applied at the
three pure reader sites: _read_recent_history_uncached and the two
rebuild_index reads.

read_preferences, read_projects and append_history keep a strict decode
(unchanged): each feeds a read-modify-write path (the consolidator's CAS
baseline, add_preference, the dashboard Save, the history append), so an
undecodable file must raise and be left intact rather than round-tripped
lossily and overwritten with no backup on the default V1 path.

Also prune the now-black-clean test/test_knowledge.py from the shrinking
black baseline (.github/black-baseline.txt), which the black gate requires.

Tests: the pure readers skip a corrupt file and still return the clean
entries; the read-modify-write readers raise and leave the corrupt file
intact.

Simplifies #8378: the smallest correct fix, no new config, module, or
degradation-state machinery.

Co-authored-by: ManoharSwamynathan <7479574+ManoharSwamynathan@users.noreply.github.com>
iamwhatever added a commit that referenced this pull request Sep 23, 2026
The pure readers (recent-history assembly and index rebuild) and the
every-turn context build decoded memory files with
read_text(encoding="utf-8"): one non-UTF-8 byte raised UnicodeDecodeError
up through the reader, breaking get_context (via build_session_context)
and a rebuild.

Keep every existing read exactly as before (same call, same containment)
and wrap ONLY the decode failure: on UnicodeDecodeError log a warning
naming the file and skip that entry/section, treating it as absent. No new
read path, no errors="replace" round-trip, no new file-opening helper.
Applied at the pure reader sites (_read_recent_history_uncached, the two
rebuild_index reads) and at get_context's own consumption of preferences
and projects — the every-turn path that reads preferences even with
include_activity=False.

read_preferences, read_projects and append_history keep a strict decode
(unchanged): each feeds a read-modify-write path (the consolidator's CAS
baseline, add_preference, the dashboard Save, the history append), so an
undecodable file must raise and be left intact rather than round-tripped
lossily and overwritten with no backup on the default V1 path. get_context
catches the strict reader's error at the point of consumption instead.

Also prune the now-black-clean test/test_knowledge.py from the shrinking
black baseline (.github/black-baseline.txt), which the black gate requires.

Tests: the pure readers skip a corrupt file and still return clean entries;
get_context survives a corrupt preferences.md and still surfaces clean
history; the read-modify-write readers raise and leave the corrupt file
intact.

Simplifies #8378: the smallest correct fix, no new config, module, or
degradation-state machinery.

Co-authored-by: ManoharSwamynathan <7479574+ManoharSwamynathan@users.noreply.github.com>
iamwhatever added a commit that referenced this pull request Sep 24, 2026
The pure readers (recent-history assembly and index rebuild) and the
context builders decoded memory files with read_text(encoding="utf-8"):
one non-UTF-8 byte raised UnicodeDecodeError up through the reader,
breaking get_context / activity_index (both reached from
build_session_context) and an index rebuild.

Keep every existing read exactly as before (same call, same containment)
and wrap ONLY the decode failure: on UnicodeDecodeError log a warning
naming the file and skip that entry/section, treating it as absent. No new
read path, no errors="replace" round-trip, no new file-opening helper.
Applied at the pure reader sites (_read_recent_history_uncached, the two
rebuild_index reads) and at the strict readers' points of consumption on
the every-turn context path: get_context's preferences/projects reads and
activity_index's projects read (build_session_context reaches the latter
with include_activity=False).

read_preferences, read_projects and append_history keep a strict decode
(unchanged): each feeds a read-modify-write path (the consolidator's CAS
baseline, add_preference, the dashboard Save, the history append), so an
undecodable file must raise and be left intact rather than round-tripped
lossily and overwritten with no backup on the default V1 path. The
context builders catch the strict reader's error at consumption instead.

Also prune the now-black-clean test/test_knowledge.py from the shrinking
black baseline (.github/black-baseline.txt), which the black gate requires.

Tests: the pure readers skip a corrupt file and still return clean entries;
get_context survives a corrupt preferences.md and activity_index survives a
corrupt projects.md, both still surfacing clean content; the
read-modify-write readers raise and leave the corrupt file intact.

Simplifies #8378: the smallest correct fix, no new config, module, or
degradation-state machinery.

Co-authored-by: ManoharSwamynathan <7479574+ManoharSwamynathan@users.noreply.github.com>
iamwhatever added a commit that referenced this pull request Sep 24, 2026
The pure readers (recent-history assembly and index rebuild) and the
context builders decoded memory files with read_text(encoding="utf-8"):
one non-UTF-8 byte raised UnicodeDecodeError up through the reader,
breaking get_context / activity_index (both reached from
build_session_context) and an index rebuild.

Keep every existing read exactly as before (same call, same containment)
and wrap ONLY the decode failure: on UnicodeDecodeError log a warning
naming the file. No new read path, no new file-opening helper. Behaviour
by site:

- _read_recent_history_uncached: skip the corrupt day (a read-only
  assembly; the missing day is simply omitted).
- get_context (preferences + projects) and activity_index (projects):
  catch at the point of consumption on the every-turn context path and
  skip that section, so a corrupt file cannot crash context assembly while
  the readers themselves stay strict for their read-modify-write callers.
- rebuild_index: index the surviving text with errors="replace" rather
  than dropping the file. rebuild_index does DELETE-then-reinsert, so
  dropping a file would lose its FTS rows and V1 search has no file-scan
  fallback; the index is derived and never written back to the markdown,
  so a lossy decode is safe here.

read_preferences, read_projects and append_history keep a strict decode
(unchanged): each feeds a read-modify-write path (the consolidator's CAS
baseline, add_preference, the dashboard Save, the history append), so an
undecodable file must raise and be left intact rather than round-tripped
lossily and overwritten with no backup on the default V1 path.

Also prune the now-black-clean test/test_knowledge.py from the shrinking
black baseline (.github/black-baseline.txt), which the black gate requires.

Tests: the recent-history reader skips a corrupt day; rebuild_index keeps
a corrupt file's surviving text searchable; get_context survives a corrupt
preferences.md and activity_index survives a corrupt projects.md; the
read-modify-write readers raise and leave the corrupt file intact.

Simplifies #8378: the smallest correct fix, no new config, module, or
degradation-state machinery.

Co-authored-by: ManoharSwamynathan <7479574+ManoharSwamynathan@users.noreply.github.com>
iamwhatever added a commit that referenced this pull request Sep 24, 2026
The pure readers (recent-history assembly and index rebuild) and the
context builders decoded memory files with read_text(encoding="utf-8"):
one non-UTF-8 byte raised UnicodeDecodeError up through the reader,
breaking get_context / activity_index (both reached from
build_session_context) and an index rebuild.

Keep every existing read exactly as before (same single call, same
containment) and wrap ONLY the decode failure: on UnicodeDecodeError log a
warning naming the file and skip that entry/section. No new read path, no
reopen, no errors="replace", no new file-opening helper. Sites:
_read_recent_history_uncached (skip the day), the two rebuild_index reads
(skip the file), and get_context / activity_index consumption of
preferences+projects on the every-turn context path (skip the section) so
context assembly cannot crash while the readers stay strict.

read_preferences, read_projects and append_history keep a strict decode
(unchanged): each feeds a read-modify-write path (the consolidator's CAS
baseline, add_preference, the dashboard Save, the history append), so an
undecodable file must raise and be left intact rather than round-tripped
lossily and overwritten with no backup on the default V1 path. The context
builders catch the strict reader's error at consumption instead.

Trade-off: a corrupt history file is left out of the FTS index until it is
repaired; V1 search will not surface its text meanwhile. This is preferred
over a reopen (which would race a concurrent delete) or a lossy re-decode.

Also prune the now-black-clean test/test_knowledge.py from the shrinking
black baseline (.github/black-baseline.txt), which the black gate requires.

Tests: the pure readers skip a corrupt file and still return clean entries;
get_context survives a corrupt preferences.md and activity_index survives a
corrupt projects.md; the read-modify-write readers raise and leave the
corrupt file intact.

Simplifies #8378: the smallest correct fix, no new config, module, or
degradation-state machinery.

Co-authored-by: ManoharSwamynathan <7479574+ManoharSwamynathan@users.noreply.github.com>
iamwhatever added a commit that referenced this pull request Sep 24, 2026
The pure readers (recent-history assembly and index rebuild) and the
context builders decoded memory files with read_text(encoding="utf-8"):
one non-UTF-8 byte raised UnicodeDecodeError up through the reader,
breaking get_context / activity_index (both reached from
build_session_context) and an index rebuild.

Keep every existing read exactly as before (same single call, same
containment) and wrap ONLY the decode failure: on UnicodeDecodeError log a
warning naming the file and skip that entry/section. No new read path, no
reopen, no errors="replace", no new file-opening helper. Sites:
_read_recent_history_uncached (skip the day), the two rebuild_index reads
(skip the file), and get_context / activity_index consumption of
preferences+projects on the every-turn context path (skip the section) so
context assembly cannot crash while the readers stay strict.

read_preferences, read_projects and append_history keep a strict decode
(unchanged): each feeds a read-modify-write path (the consolidator's CAS
baseline, add_preference, the dashboard Save, the history append), so an
undecodable file must raise and be left intact rather than round-tripped
lossily and overwritten with no backup on the default V1 path. The context
builders catch the strict reader's error at consumption instead.

Trade-off: a corrupt history file is left out of the FTS index until it is
repaired; V1 search will not surface its text meanwhile. This is preferred
over a reopen (which would race a concurrent delete) or a lossy re-decode.

Also prune the now-black-clean test/test_knowledge.py from the shrinking
black baseline (.github/black-baseline.txt), which the black gate requires.

Tests: the pure readers skip a corrupt file and still return clean entries;
get_context survives a corrupt preferences.md and activity_index survives a
corrupt projects.md; the read-modify-write readers raise and leave the
corrupt file intact.

Simplifies #8378: the smallest correct fix, no new config, module, or
degradation-state machinery.

Co-authored-by: ManoharSwamynathan <7479574+ManoharSwamynathan@users.noreply.github.com>
@iamwhatever

Copy link
Copy Markdown
Contributor

Superseded by #13159.

This change (make memory reads tolerate a non-UTF-8 preferences.md / projects.md / history file instead of crashing the context build) has been rebased onto current main and reworked in #13159, which is green:

  • The per-turn context path (get_context, activity_index) and the pure readers (recent-history read, rebuild_index) now catch UnicodeDecodeError at the point of consumption and skip the undecodable file, so one bad byte no longer aborts the build.
  • The read-modify-write callers (read_preferences / read_projects / append_history) stay strict, so a corrupt file is never lossily rewritten.
  • No new read path is introduced — the existing reads are kept exactly as main has them; only the decode is wrapped. Regression, symlink, and named-store cases are covered by tests.

Your authorship is preserved on the commit in #13159 (via Co-authored-by). Thank you for the original fix — closing this PR in favor of #13159. The branch is left intact.

@github-actions github-actions Bot removed the readiness: action required A blocking check or review needs attention label Sep 24, 2026
iamwhatever added a commit that referenced this pull request Sep 24, 2026
The pure readers (recent-history assembly and index rebuild) and the
context builders decoded memory files with read_text(encoding="utf-8"):
one non-UTF-8 byte raised UnicodeDecodeError up through the reader,
breaking get_context / activity_index (both reached from
build_session_context) and an index rebuild.

Keep every existing read exactly as before (same single call, same
containment) and wrap ONLY the decode failure: on UnicodeDecodeError log a
warning naming the file and skip that entry/section. No new read path, no
reopen, no errors="replace", no new file-opening helper. Sites:
_read_recent_history_uncached (skip the day), the two rebuild_index reads
(skip the file), and get_context / activity_index consumption of
preferences+projects on the every-turn context path (skip the section) so
context assembly cannot crash while the readers stay strict.

read_preferences, read_projects and append_history keep a strict decode
(unchanged): each feeds a read-modify-write path (the consolidator's CAS
baseline, add_preference, the dashboard Save, the history append), so an
undecodable file must raise and be left intact rather than round-tripped
lossily and overwritten with no backup on the default V1 path. The context
builders catch the strict reader's error at consumption instead.

Trade-off: a corrupt history file is left out of the FTS index until it is
repaired; V1 search will not surface its text meanwhile. This is preferred
over a reopen (which would race a concurrent delete) or a lossy re-decode.

Also prune the now-black-clean test/test_knowledge.py from the shrinking
black baseline (.github/black-baseline.txt), which the black gate requires.

Tests: the pure readers skip a corrupt file and still return clean entries;
get_context survives a corrupt preferences.md and activity_index survives a
corrupt projects.md; the read-modify-write readers raise and leave the
corrupt file intact.

Simplifies #8378: the smallest correct fix, no new config, module, or
degradation-state machinery.

Co-authored-by: iamwhatever <iamwhatever@users.noreply.github.com>
Co-authored-by: ManoharSwamynathan <7479574+ManoharSwamynathan@users.noreply.github.com>
wknight1 pushed a commit to wknight1/KiroCrew that referenced this pull request Oct 5, 2026
…dev#13159)

The pure readers (recent-history assembly and index rebuild) and the
context builders decoded memory files with read_text(encoding="utf-8"):
one non-UTF-8 byte raised UnicodeDecodeError up through the reader,
breaking get_context / activity_index (both reached from
build_session_context) and an index rebuild.

Keep every existing read exactly as before (same single call, same
containment) and wrap ONLY the decode failure: on UnicodeDecodeError log a
warning naming the file and skip that entry/section. No new read path, no
reopen, no errors="replace", no new file-opening helper. Sites:
_read_recent_history_uncached (skip the day), the two rebuild_index reads
(skip the file), and get_context / activity_index consumption of
preferences+projects on the every-turn context path (skip the section) so
context assembly cannot crash while the readers stay strict.

read_preferences, read_projects and append_history keep a strict decode
(unchanged): each feeds a read-modify-write path (the consolidator's CAS
baseline, add_preference, the dashboard Save, the history append), so an
undecodable file must raise and be left intact rather than round-tripped
lossily and overwritten with no backup on the default V1 path. The context
builders catch the strict reader's error at consumption instead.

Trade-off: a corrupt history file is left out of the FTS index until it is
repaired; V1 search will not surface its text meanwhile. This is preferred
over a reopen (which would race a concurrent delete) or a lossy re-decode.

Also prune the now-black-clean test/test_knowledge.py from the shrinking
black baseline (.github/black-baseline.txt), which the black gate requires.

Tests: the pure readers skip a corrupt file and still return clean entries;
get_context survives a corrupt preferences.md and activity_index survives a
corrupt projects.md; the read-modify-write readers raise and leave the
corrupt file intact.

Simplifies kirodotdev#8378: the smallest correct fix, no new config, module, or
degradation-state machinery.

Co-authored-by: iamwhatever <iamwhatever@users.noreply.github.com>
Co-authored-by: ManoharSwamynathan <7479574+ManoharSwamynathan@users.noreply.github.com>
@mbriones98 mbriones98 removed the drive-to-green PR claimed by drive-to-green pipeline label Oct 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

fork Pull request from a fork (external contributor) merge conflict Branch has merge conflicts with its base — author must resolve before merge needs-author-decision PR blocked on author input

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: non-UTF-8 byte in preferences.md crashes every reader (dashboard tab, prompt assembly, and the repair path)

5 participants