Repository navigation
Conversation
First Principles Review (Fable 5, fork) — 🟡 CONCERNSPremise-level review of First-Principles-Verdict: CONCERNS The "JSON paths are provably ASCII" boundary is false for agent-written JSON — the same locale-decode root cause survives at four read sites. Not justified as shipped
What this change shipsIntent: stop Windows CJK hosts from crashing (or 500ing) when a research campaign's prose is saved or read back — a FIX.
Watch
Subtractions
[FIRST-PRINCIPLES-REVIEWED] a565b6a |
Design Review (Fable 5, fork) — 🟡 CONCERNSDesign-level review of Design-Verdict: CONCERNS Sound fix with a well-defended read contract, but the "JSON is pure ASCII" boundary only holds for process-written JSON — agent-written JSON reads keep the bug. WatchThe PR's own triage rule ("ask what writes the file") flags reads its boundary statement exempts. The description exempts JSON because " [DESIGN-REVIEWED] a565b6a |
GPT 5.6 Review (fork) — ✅ no blocking findingsReviewed Review detailsNo findings. |
Opus 4.8 Review (fork) — ✅ no blocking findingsReviewed Review detailsNo findings. The fork transcode is lossless and reversible, not corruption: a legacy parent's bytes [OPUS-REVIEWED] a565b6a |
560ded6 to
205508a
Compare
|
The exact-head security The fix is the reviewer's prescription: UTF-8 first, and on The existing tests caught a real mistake in my first attempt, which is worth recording. I initially wrote the helper as The helper now stays on New tests:
Gates: The description is re-synced to describe the fallback, since the body previously described the reads as unconditionally strict UTF-8. Amended rather than stacked, to stay inside the two-commit rule. |
205508a to
2ff5d0f
Compare
|
All three exact-head The common cause: the fallback wrote.
On GPT's second option ("or migrate through a symlink-safe no-follow file operation"): I could not make that honest. Windows has no Cost of the subtraction: one extra Regressions — deterministic, one per load-bearing property, all four red on Two shaping notes, since both were weaknesses in the previous head's tests:
Green-after: 20 passed / 1 skipped in the module; 253 passed / 10 skipped across the three |
2ff5d0f to
68b8df2
Compare
|
The exact-head security The earlier finding was that the fallback's write followed an agent-planted symlink, and I closed it by removing the write. This one is about the read: Being pre-existing is not a defence here. The fix is the reviewer's prescription, unaltered: read bytes through the hooks no-link primitive constrained to the campaign tree, then decode. data = hooks.safe_read_file_bytes_nolink(str(path), str(research_dir()), allow_truncate=True)
Three consequences of reading bytes, each handled rather than absorbed:
The off-loop ratchets moved with the call, not around it. New regressions — Both link kinds are covered because they fail different checks — a symlink at the open, a hard link at
Green-after: 255 passed / 10 skipped across The description is re-synced to describe the no-link read. |
a8fa925 to
ef160e7
Compare
|
The security
The mistake was mine and it was a contract error, not an oversight. I reached for The deeper error is that one reader was serving four consumers with different needs. I classified them:
Three of the four derive new data. A prefix handed to those is a partial copy presented as the source, and the adjudication is right that a campaign directory being agent-authored makes an oversized I did not invent a helper for this; the repository already had both contracts. One reordering rides along, and it is load-bearing rather than tidying: the fork's seed read now happens before The UTF-8 half of the finding is a separate mechanism and is fixed separately. A cap can land inside a multi-byte sequence; a whole-buffer Red-before in a pristine control worktree at The third prints the defect — Two shaping notes. Green-after: 29 passed / 1 skipped in The description is re-synced to describe the split policy. Still one commit. |
ef160e7 to
9913611
Compare
|
Correction to my previous comment: the
F1 —
|
`auto_research` moves model-generated prose through `Path.read_text` / `write_text` with no `encoding=`, so the bytes on disk are whatever `locale.getpreferredencoding()` picks -- UTF-8 on POSIX, the ANSI code page on Windows. A model routinely emits characters a legacy ANSI code page cannot represent, so on a CJK Windows host writing `FINDINGS.md` raises `UnicodeEncodeError` and the cycle's finding is lost rather than saved (measured on cp950). The five free-text writes now pin `encoding="utf-8"`, and the reads that pair with them go through `_read_campaign_text`. That reader exists for one reason: pinning the reads to UTF-8 would make a campaign created BEFORE the pin -- whose findings are sitting on disk in the host code page -- raise `UnicodeDecodeError` on every report, fork, export and knowledge route, turning an upgrade into a permanent HTTP 500. So the read must not raise. It must also not GUESS. Retrying with `locale.getpreferredencoding()` treats a decode succeeding as evidence of what wrote the file, and it is not: cp950 `U+7652` is the bytes `c2 a1`, which are equally valid UTF-8 for `U+00A1`, and 899 cp950 multi-byte characters have that property, so a legacy report could be served -- and copied into a fork -- as Latin mojibake with nothing raised. It would also take provenance from ambient state, applying the READING host's code page to a file some other host wrote. Nothing on disk can break that tie: a campaign directory carries no format or version marker, `status.json` has no schema version, the campaign row records no writer generation, and in agent mode `FINDINGS.md` is written by the research agent rather than by this process at all. So the codec is not chosen by trial. `latin-1` is total and lossless, so a pre-pin file still renders, the file is never rewritten, and its characters stay recoverable from the string -- byte-exact up to the universal-newline translation both reads already applied. This is the contract `kiro_crew.knowledge.readers._decode_text_bytes` already uses for text of unknown provenance. The cost is stated rather than hidden: a pre-pin campaign whose findings really were in the host code page renders as mojibake on that host instead of correct text. That population is narrow -- the same write raised `UnicodeEncodeError` for any character outside the code page, which is the defect this fixes, so the files that survived are the ones that were entirely code-page-encodable -- and the characters are recoverable, where a wrong guess is not detectable at all. Both reads stay in text mode, so universal-newline translation is unchanged and a CRLF findings file still comes back with `\n`. Nothing on the read path writes: no migration, no re-encode of the source, no `errors="replace"` landing back on disk. `_handle_report` reads through `asyncio.to_thread`, matching the export and fork siblings that already offload this same unbounded file -- this change is what can make that read two opens on a legacy campaign. Campaign reads and writes follow links exactly as they do on `origin/main`; this change does not alter that in either direction, and hardening it belongs in its own PR (see the description). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
9913611 to
a565b6a
Compare
|
Both exact-head blockers on The production diff drops from 255 changed lines to 59, and from three changed files to two. Why reduction and not repairBoth findings live in code that was never part of the encoding fix. Classifying every production hunk made that obvious:
Everything in the third row is gone. It was not free: routing prose reads through the no-follow primitive brought its opt-in 50 MB cap along, and that cap is exactly what produced Blocker A — truncation ahead of There is no cap, no truncation and no byte read on any campaign prose path, so there is nothing that can be truncated before redaction. Blocker A is gone by construction. What survivestry:
return path.read_text(encoding="utf-8")
except UnicodeDecodeError:
return path.read_text(encoding="latin-1")Both reads stay in text mode, so universal-newline translation comes back for free and the CRLF handling, the incremental decoder and the byte-level plumbing all disappear with it. Nothing on the read path writes. No locale probing, no migration write, no lossy rewrite of the source, no CRLF churn — the four things I was told not to reintroduce, and none of them is back. Blocker B — the write pathThe finding is correct and the defect is real. After reduction, this PR's entire change to those lines is an - path.write_text(text)
+ path.write_text(text, encoding="utf-8")Link-following is byte-identical to I did look for the primitive rather than assuming there was none. The mechanism really is closed by it. The third line is why it is not a drop-in: it collides with That is in the PR description, with the measurements, so whoever picks it up starts from the primitive and the conflict rather than rediscovering both. If you would rather this PR carry it anyway, say so and I will do it properly here. Red-beforeProduction reverted to Four of those seven cannot go red on a UTF-8 CI runner — the original bug is host-conditioned — so their assertions are written as "the bytes on disk are UTF-8" rather than "the write did not raise", and I am naming which four rather than letting the count imply more than it proves. Three fail on every host, including the Green-after: 21 passed in the module; 254 passed / 9 skipped across the three Still one commit. Description rewritten for the reduced scope. |
|
🤖 Kiro Crew [operator: NicholasRBowers#a942f9ca]: This PR has been inactive for 7+ days. I reviewed the blockers but they require your input:
When you've addressed these, the pipeline will re-assess on its next cycle. |
|
Closing as superseded. Every write in The branch has been in merge conflict and idle 8+ days, and the remaining delta is the regression test plus a If you want the coverage kept, refile the regression test alone rebased onto |
Problem / Motivation
auto_researchwrites the prose an LLM research loop produces —FINDINGS.md,the per-cycle files,
guidance.txt,brief.md— withPath.write_textand noencoding:write_textwith no encoding encodes withlocale.getpreferredencoding(): UTF-8on POSIX, but the legacy ANSI code page on Windows. The content of these files
is bounded by the model's output and by nothing this code controls — the module's
own test file already says exactly that about
FINDINGS.md.A model routinely emits characters that code page cannot represent. An emoji
does not degrade to a placeholder; it raises. Measured on a
cp950host bycalling these helpers directly with a realistic finding:
Not everything non-ASCII trips it, which is what makes it look fine until it
isn't — on
cp950an em dash, curly quotes and Greek all encode successfully(Big5 covers them) and an emoji does not.
The write is not retried and not caught, so the cycle's finding is lost and the
caller surfaces a 500 for a campaign that otherwise ran correctly.
The read side has the matching half.
_copy_parent_findingsseeds a forkedcampaign by reading the parent's findings and writing them straight back, both
with the locale codec. A findings file that is UTF-8 on disk — because it was
written on a UTF-8 host, or restored from a backup — makes the fork raise
UnicodeDecodeErrorinstead.Why it matters
The failure lands after the expensive part. A research cycle has already called
the model and produced its finding; the only thing left is to save it, and that is
the step that raises. There is no retry and no partial write, so the output is
gone and the operator sees a 500 with a codec error naming a character rather than
anything about their campaign.
It is silent in review because it is host-conditioned: on POSIX and on a
UTF-8-code-page Windows box,
locale.getpreferredencoding()already returns UTF-8and every one of these calls is correct. Only a CJK Windows host — a large share of
Windows installs — sees it, and the failing character is whatever the model
happened to emit that day.
What changed (motivation → approach → change)
Root cause: the encoding of a file whose content this process does not control was
left to the host's locale instead of being chosen. The change is scoped to that
and nothing else — 59 changed lines in one production file, most of them the
reader's docstring.
Five free-text writes pin
encoding="utf-8"—_write_text,_write_new_cycle_files,_copy_parent_findings's destination,write_guidance, and thebrief.mdwrite. UTF-8 is the choice the rest of therepository already makes, and it is what these files already are wherever they
were written on POSIX.
The reads that pair with them go through
_read_campaign_text. It existsfor one reason: pinning the reads to UTF-8 would make a campaign created
before the pin — whose findings are sitting on disk in the host code page —
raise
UnicodeDecodeErroron every report, fork, export and knowledge route,turning an upgrade into a permanent HTTP 500. So the read must not raise.
It must also not GUESS, and this is the part worth reading. Retrying with
locale.getpreferredencoding()treats a decode succeeding as evidence of whatwrote the file. It is not: cp950
癒is the bytesc2 a1, which are equallyvalid UTF-8 for
¡. Enumerating the code page gives 899 cp950 multi-bytecharacters with that property, so a legacy report could be served — and copied
into a fork — as Latin mojibake with nothing raised. That branch would also take
provenance from ambient state: it applies the reading host's code page to a
file some other host wrote, so one campaign directory decodes differently on two
machines, or after a locale change on one.
Nothing on disk can break that tie, so nothing pretends to. A campaign
directory carries no format or version marker,
status.jsonhas no schemaversion, the campaign row records no writer generation, and in agent mode
FINDINGS.mdis written by the research agent itself rather than by thisprocess — so even "we pinned our own writer" does not establish the encoding of
every file on this path.
latin-1is total (never raises) and lossless,so a pre-pin file still renders, the file is never rewritten, and its characters
stay recoverable from the string — byte-exact up to the universal-newline
translation both reads already applied, which folds
\r\nto\nand changesnothing else. That is the contract
kiro_crew.knowledge.readers._decode_text_bytesalready uses for text ofunknown provenance.
The cost, stated rather than hidden: a pre-pin campaign whose findings
really were in the host code page renders as mojibake on that host instead of
correct text. That population is narrow — pre-pin, the same write raised
UnicodeEncodeErrorfor any character outside the code page, which is the verydefect this fixes, so the files that survived are the ones that were entirely
code-page-encodable — and its characters stay recoverable, where a wrong guess
is not detectable at all.
Both reads stay in text mode. Universal-newline translation is therefore
unchanged and a CRLF findings file still comes back with
\n; two tests in thetree compare findings contents after a fork and would shift otherwise.
Nothing on the read path writes. No migration, no re-encode of the source,
no
errors="replace"landing back on disk. A read path that repairs its owninput is how the original bytes stop existing.
_handle_reportreads throughasyncio.to_thread, matching the export andfork siblings that already offload this same unbounded file. This change is what
can make that read two opens on a legacy campaign, so it is the change's own
cost being paid off the loop rather than an unrelated tidy-up.
The JSON boundary, narrowed
The JSON paths in this module are deliberately NOT changed, but the rule is
narrower than "JSON is ASCII" and an earlier revision of this description
overstated it. The correct statement is about the writer, not the format:
status.json,grill_tree.json,the queue file and the workflow-run file are all written by this process
through
json.dumps, which defaults toensure_ascii=True, so their bytes arepure ASCII and decode identically under every code page. Pinning those reads
would add call sites of noise to a diff whose whole claim is "this file can
contain a character the host cannot encode" — which is not true of them.
flag.write_text(str(time.time()))is left for the same reason.finding files,
questions.jsonandemergent_questions.jsonare written by theresearch agent, so they are UTF-8 with raw non-ASCII rather than ASCII. Their
reads use a bare
read_text()— the locale codec — so on the same CJK Windowshost this PR targets, they carry the same root cause.
Which of those reads is actually exposed, measured rather than counted.
UnicodeDecodeErroris aValueError, not anOSError, so it escapes a guardwritten as
except (json.JSONDecodeError, OSError):UnicodeDecodeErrorcheck_stagnation(json.JSONDecodeError, OSError)_pending_question(json.JSONDecodeError, OSError)get_findings(json.JSONDecodeError, OSError)_ingest_emergent_questions(json.JSONDecodeError, OSError)_read_json_or_missing(OSError, ValueError)_read_finding_fileUnicodeDecodeErrorexplicitlySo four sites are exposed and two are already guarded — and the fact that
_read_finding_filenames the exception by hand is evidence this class wasalready known here.
All six are pre-existing on
origin/mainand unchanged by this PR, whichtouches no JSON read or write. They are a separate sibling class and are left
unfixed here deliberately: this change is the prose encoding fix, and the point of
its scope reduction was to stop it accumulating adjacent work.
_ingest_emergent_questionsis the one worth flagging to whoever takes it — it unlinks the file only after a
successful parse, so a raise there re-fires every cycle rather than once.
Link-following on campaign paths is unchanged, and deliberately so
Everything under a campaign directory is agent-authored, so a planted link at
FINDINGS.mdorfindings_for_knowledge.mdis a real threat — for both the readand the write. This change does not alter that in either direction: every read
and write here follows links exactly as it does on
origin/main, and the onlydifference in the diff is an
encoding=keyword.Earlier revisions of this branch did harden the reads (a no-follow descriptor
read bounded to the research root). That is removed, because it was not free: the
no-follow primitive's 50 MB cap is opt-in truncation, and routing prose reads
through it introduced a new silent-corruption surface — a truncated prefix
feeding the fork seed and the knowledge export, and a cap landing inside a UTF-8
character sending the whole retained prefix through the fallback codec. Reviewers
found both, correctly. An encoding fix should not have to carry a preview/security
contract to land, and the riders were generating findings of their own.
The write side is a real defect on
origin/mainand it should be fixed — in itsown PR. For whoever takes it, the primitive already exists:
pinned_fs.write_file_pinned, which describes itself as "THE single no-followfile-publish path" and keeps its
lstatrefusal of a planted symlink on Windowstoo. It is not a drop-in here, and the reason is measured rather than assumed:
That last line collides with an existing, test-pinned contract on
main—TestWriteHelperDeletionRace::test_write_text_does_not_resurrect_a_deleted_campaign_dir,which names
findings_for_knowledge.mdspecifically and requires that aconcurrent campaign deletion win rather than have its directory silently
recreated. Reconciling "never follow a planted link" with "never recreate a
deleted campaign" is a design decision about the write path, not a call-site swap,
and it is not something an encoding fix should decide in passing.
Tests
Added to
test/test_auto_research_onloop_fs.py— the module that already guardsthis file's filesystem discipline, and which already documents
FINDINGS.mdasmodel-sized output.
Red-before, production reverted to
origin/mainwith the new tests kept, on acp950 host:
Which of those go red on a UTF-8 CI runner, said plainly. The original bug is
host-conditioned, so four of them (
write_text_encodes_utf8,new_cycle_files_encode_utf8,findings_survive_a_fork_round_trip,bytes_valid_under_both_decodings) cannot fail wherelocale.getpreferredencoding()already returns UTF-8. Their assertions aretherefore written as "the bytes on disk are UTF-8", not "the write did not
raise", so they still pin the property everywhere. Three fail on every host:
a_legacy_code_page_campaign_reads_back_recoverableandan_undecodable_file_degrades_without_destroying_it(a strict UTF-8 read raiseson those bytes wherever it runs), and
the_read_never_consults_the_host_code_page,which is a structural ratchet.
The two that pass on both builds do so by design and are reported as guards,
not as evidence:
test_the_guard_can_actually_failasserts the payload really isundecodable under
cp950and clean under UTF-8, so it tests the oracle ratherthan the fix;
test_a_crlf_findings_file_still_comes_back_with_newlinesis ano-regression guard for text mode.
test_bytes_valid_under_both_decodings_resolve_to_the_declared_contractis theambiguous case itself —
c2 a1, valid under both candidate decodings — and itcarries its own guard-the-guard (
raw == b"\xc2\xa1"andraw.decode("utf-8") == "¡") so it cannot pass on bytes that were neverambiguous.
test_the_read_never_consults_the_host_code_pageis a ratchet(
not hasattr(h, "locale")plus a source check on the reader), because the onlyway to pick the host codec is to assume the reading host wrote the file, and that
assumption cannot come back by accident.
Green-after: 21 passed in that module; 254 passed / 9 skipped across
test_auto_research.py,test_auto_research_onloop_fs.pyandtest_auto_research_onloop_db.py.Gates:
flake8andisort --check-onlyclean;mypy --platform linux0errors in
handlers.py;scripts/check_black_formatting.pyandscripts/check_comment_history.pyboth passed, each re-run after the commitbecause both scope
origin/main...HEADand pass vacuously before it exists.Manual verification
N/A — unit coverage sufficient: the failure is a codec raise inside two helper
functions, and the tests drive those helpers directly with a payload measured to
defeat the host code page, which is exactly what a manual run would do.
Related Issues
None. Found while auditing
locale.getpreferredencodingsites for the failureclass
scripts/check_subprocess_encoding.pywas added for — the samedecode/encode reached through a file read or write rather than a subprocess pipe,
which that gate does not cover.
Pattern harvest
Rule candidate:
review-promptPattern: an unpinned
encoding=is only a real defect once you can name a writerthat emits non-ASCII — but "the model wrote it" is always such a writer. Most
unpinned reads in this repository are genuinely harmless because
json.dumpsdefaults to
ensure_ascii=True, so the bytes are ASCII whatever the code page.That is exactly why counting call sites is the wrong triage: the same call is
noise on a JSON file and a data-loss bug on a file an LLM filled. Ask what writes
the file, not how many places read it.
Second: a partial code page is worse than a missing one. On
cp950an em dashand CJK both encode fine and an emoji does not, so this class survives casual
testing with "some non-ASCII" and fails on the one character nobody tried. Pick
the payload from the complement of the host code page, and assert that choice in
the test so it cannot silently become a character the code page happens to cover.
Third, for tests of host-conditioned bugs: assert the property the fix chooses,
not the symptom the host produces. "The write did not raise" is untestable on a
runner that never raises. "The bytes on disk are UTF-8" is the same guarantee,
stated so that it holds — and fails — everywhere.
Fourth: a successful decode is not provenance. A
try: decode(A) / except UnicodeDecodeError: decode(B)chain reads like afallback and behaves like a classifier, and it is a silent one — the wrong
branch produces plausible text, not an error. Before writing one, ask what on disk
records the encoding. If the answer is "nothing", the honest move is one decoder
plus a total, lossless degradation, so the characters survive for whoever can
decide later. Where the two candidates' domains overlap by hundreds of characters,
as UTF-8 and a DBCS code page do, trial decoding should be a review smell.
Fifth, and the one this PR learned the hard way: a security rider is not free
just because it is correct. Hardening the reads here pulled in a no-follow
primitive whose opt-in size cap silently truncated the fork seed and the knowledge
export, and whose cut could land inside a UTF-8 character and mojibake the whole
retained prefix. Both were real findings, and neither existed before the rider
did. When a fix in one dimension needs a contract in another to stay correct, that
is the signal to split the PR, not to keep widening it.
Checklist
feat|fix|docs|refactor|perf|test|chore|ci|build|revert: ...)Contribution License Agreement