Skip to content

fix(auto-research): write campaign text as utf-8, not the host code page - #9247

Closed
leonlaiyc wants to merge 1 commit into
kirodotdev:mainfrom
leonlaiyc:fix/auto-research-text-utf8
Closed

leonlaiyc wants to merge 1 commit into
kirodotdev:mainfrom
leonlaiyc:fix/auto-research-text-utf8

Conversation

@leonlaiyc

@leonlaiyc leonlaiyc commented Sep 7, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

auto_research writes the prose an LLM research loop produces — FINDINGS.md,
the per-cycle files, guidance.txt, brief.md — with Path.write_text and no
encoding:

def _write_text(path: Path, text: str) -> None:
    path.write_text(text)

write_text with no encoding encodes with locale.getpreferredencoding(): UTF-8
on POSIX, but the legacy ANSI code page on Windows. The content of these files
is bounded by the model's output and by nothing this code controls — the module's
own test file already says exactly that about FINDINGS.md.

A model routinely emits characters that code page cannot represent. An emoji
does not degrade to a placeholder; it raises.
Measured on a cp950 host by
calling these helpers directly with a realistic finding:

_write_text             UnicodeEncodeError: 'cp950' codec can't encode
                        character '\U0001f600' in position 8
_write_new_cycle_files  UnicodeEncodeError: ... same

Not everything non-ASCII trips it, which is what makes it look fine until it
isn't — on cp950 an em dash, curly quotes and Greek all encode successfully
(Big5 covers them) and an emoji does not.

The write is not retried and not caught, so the cycle's finding is lost and the
caller surfaces a 500 for a campaign that otherwise ran correctly.

The read side has the matching half. _copy_parent_findings seeds a forked
campaign by reading the parent's findings and writing them straight back, both
with the locale codec. A findings file that is UTF-8 on disk — because it was
written on a UTF-8 host, or restored from a backup — makes the fork raise
UnicodeDecodeError instead.

Why it matters

The failure lands after the expensive part. A research cycle has already called
the model and produced its finding; the only thing left is to save it, and that is
the step that raises. There is no retry and no partial write, so the output is
gone and the operator sees a 500 with a codec error naming a character rather than
anything about their campaign.

It is silent in review because it is host-conditioned: on POSIX and on a
UTF-8-code-page Windows box, locale.getpreferredencoding() already returns UTF-8
and every one of these calls is correct. Only a CJK Windows host — a large share of
Windows installs — sees it, and the failing character is whatever the model
happened to emit that day.

What changed (motivation → approach → change)

Root cause: the encoding of a file whose content this process does not control was
left to the host's locale instead of being chosen. The change is scoped to that
and nothing else — 59 changed lines in one production file, most of them the
reader's docstring.

  • Five free-text writes pin encoding="utf-8" — _write_text,
    _write_new_cycle_files, _copy_parent_findings's destination,
    write_guidance, and the brief.md write. UTF-8 is the choice the rest of the
    repository already makes, and it is what these files already are wherever they
    were written on POSIX.

  • The reads that pair with them go through _read_campaign_text. It exists
    for one reason: pinning the reads to UTF-8 would make a campaign created
    before the pin — whose findings are sitting on disk in the host code page —
    raise UnicodeDecodeError on every report, fork, export and knowledge route,
    turning an upgrade into a permanent HTTP 500. So the read must not raise.

    try:
        return path.read_text(encoding="utf-8")
    except UnicodeDecodeError:
        return path.read_text(encoding="latin-1")
  • It must also not GUESS, and this is the part worth reading. Retrying with
    locale.getpreferredencoding() treats a decode succeeding as evidence of what
    wrote the file. It is not: cp950 癒 is the bytes c2 a1, which are equally
    valid UTF-8 for ¡. Enumerating the code page gives 899 cp950 multi-byte
    characters with that property, so a legacy report could be served — and copied
    into a fork — as Latin mojibake with nothing raised. That branch would also take
    provenance from ambient state: it applies the reading host's code page to a
    file some other host wrote, so one campaign directory decodes differently on two
    machines, or after a locale change on one.

    Nothing on disk can break that tie, so nothing pretends to. A campaign
    directory carries no format or version marker, status.json has no schema
    version, the campaign row records no writer generation, and in agent mode
    FINDINGS.md is written by the research agent itself rather than by this
    process — so even "we pinned our own writer" does not establish the encoding of
    every file on this path. latin-1 is total (never raises) and lossless,
    so a pre-pin file still renders, the file is never rewritten, and its characters
    stay recoverable from the string — byte-exact up to the universal-newline
    translation both reads already applied, which folds \r\n to \n and changes
    nothing else. That is the contract
    kiro_crew.knowledge.readers._decode_text_bytes already uses for text of
    unknown provenance.

    The cost, stated rather than hidden: a pre-pin campaign whose findings
    really were in the host code page renders as mojibake on that host instead of
    correct text. That population is narrow — pre-pin, the same write raised
    UnicodeEncodeError for any character outside the code page, which is the very
    defect this fixes, so the files that survived are the ones that were entirely
    code-page-encodable — and its characters stay recoverable, where a wrong guess
    is not detectable at all.

  • Both reads stay in text mode. Universal-newline translation is therefore
    unchanged and a CRLF findings file still comes back with \n; two tests in the
    tree compare findings contents after a fork and would shift otherwise.

  • Nothing on the read path writes. No migration, no re-encode of the source,
    no errors="replace" landing back on disk. A read path that repairs its own
    input is how the original bytes stop existing.

  • _handle_report reads through asyncio.to_thread, matching the export and
    fork siblings that already offload this same unbounded file. This change is what
    can make that read two opens on a legacy campaign, so it is the change's own
    cost being paid off the loop rather than an unrelated tidy-up.

The JSON boundary, narrowed

The JSON paths in this module are deliberately NOT changed, but the rule is
narrower than "JSON is ASCII" and an earlier revision of this description
overstated it. The correct statement is about the writer, not the format:

  • Process-written JSON is provably ASCII. status.json, grill_tree.json,
    the queue file and the workflow-run file are all written by this process
    through json.dumps, which defaults to ensure_ascii=True, so their bytes are
    pure ASCII and decode identically under every code page. Pinning those reads
    would add call sites of noise to a diff whose whole claim is "this file can
    contain a character the host cannot encode" — which is not true of them.
    flag.write_text(str(time.time())) is left for the same reason.
  • Agent-written JSON is NOT, and this PR does not fix it. The per-cycle
    finding files, questions.json and emergent_questions.json are written by the
    research agent, so they are UTF-8 with raw non-ASCII rather than ASCII. Their
    reads use a bare read_text() — the locale codec — so on the same CJK Windows
    host this PR targets, they carry the same root cause.

Which of those reads is actually exposed, measured rather than counted.
UnicodeDecodeError is a ValueError, not an OSError, so it escapes a guard
written as except (json.JSONDecodeError, OSError):

Site Guard UnicodeDecodeError
check_stagnation (json.JSONDecodeError, OSError) escapes
_pending_question (json.JSONDecodeError, OSError) escapes
get_findings (json.JSONDecodeError, OSError) escapes
_ingest_emergent_questions (json.JSONDecodeError, OSError) escapes
_read_json_or_missing (OSError, ValueError) already caught
_read_finding_file names UnicodeDecodeError explicitly already caught

So four sites are exposed and two are already guarded — and the fact that
_read_finding_file names the exception by hand is evidence this class was
already known here.

All six are pre-existing on origin/main and unchanged by this PR, which
touches no JSON read or write. They are a separate sibling class and are left
unfixed here deliberately: this change is the prose encoding fix, and the point of
its scope reduction was to stop it accumulating adjacent work. _ingest_emergent_questions
is the one worth flagging to whoever takes it — it unlinks the file only after a
successful parse, so a raise there re-fires every cycle rather than once.

Link-following on campaign paths is unchanged, and deliberately so

Everything under a campaign directory is agent-authored, so a planted link at
FINDINGS.md or findings_for_knowledge.md is a real threat — for both the read
and the write. This change does not alter that in either direction: every read
and write here follows links exactly as it does on origin/main, and the only
difference in the diff is an encoding= keyword.

Earlier revisions of this branch did harden the reads (a no-follow descriptor
read bounded to the research root). That is removed, because it was not free: the
no-follow primitive's 50 MB cap is opt-in truncation, and routing prose reads
through it introduced a new silent-corruption surface — a truncated prefix
feeding the fork seed and the knowledge export, and a cap landing inside a UTF-8
character sending the whole retained prefix through the fallback codec. Reviewers
found both, correctly. An encoding fix should not have to carry a preview/security
contract to land, and the riders were generating findings of their own.

The write side is a real defect on origin/main and it should be fixed — in its
own PR.
For whoever takes it, the primitive already exists:
pinned_fs.write_file_pinned, which describes itself as "THE single no-follow
file-publish path"
and keeps its lstat refusal of a planted symlink on Windows
too. It is not a drop-in here, and the reason is measured rather than assumed:

supports_pinned_walk()                     : False   (Windows)
planted hardlink -> victim file            : SECRET  (refused; the mechanism IS closed)
write into a deleted campaign directory    : parent recreated

That last line collides with an existing, test-pinned contract on main —
TestWriteHelperDeletionRace::test_write_text_does_not_resurrect_a_deleted_campaign_dir,
which names findings_for_knowledge.md specifically and requires that a
concurrent campaign deletion win rather than have its directory silently
recreated. Reconciling "never follow a planted link" with "never recreate a
deleted campaign" is a design decision about the write path, not a call-site swap,
and it is not something an encoding fix should decide in passing.

Tests

Added to test/test_auto_research_onloop_fs.py — the module that already guards
this file's filesystem discipline, and which already documents FINDINGS.md as
model-sized output.

Red-before, production reverted to origin/main with the new tests kept, on a
cp950 host:

FAILED ...::test_write_text_encodes_utf8
  UnicodeEncodeError: 'cp950' codec can't encode character '\U0001f600'
FAILED ...::test_new_cycle_files_encode_utf8
  UnicodeEncodeError: ... same
FAILED ...::test_findings_survive_a_fork_round_trip
  UnicodeDecodeError: 'cp950' codec can't decode byte 0xf0 in position 8
FAILED ...::test_a_legacy_code_page_campaign_reads_back_recoverable
  UnicodeEncodeError: 'latin-1' codec can't encode characters in position 14-16
FAILED ...::test_an_undecodable_file_degrades_without_destroying_it
  UnicodeDecodeError: 'cp950' codec can't decode byte 0xff in position 0
FAILED ...::test_bytes_valid_under_both_decodings_resolve_to_the_declared_contract
FAILED ...::test_the_read_never_consults_the_host_code_page
7 failed, 2 passed

Which of those go red on a UTF-8 CI runner, said plainly. The original bug is
host-conditioned, so four of them (write_text_encodes_utf8,
new_cycle_files_encode_utf8, findings_survive_a_fork_round_trip,
bytes_valid_under_both_decodings) cannot fail where
locale.getpreferredencoding() already returns UTF-8. Their assertions are
therefore written as "the bytes on disk are UTF-8", not "the write did not
raise", so they still pin the property everywhere. Three fail on every host:
a_legacy_code_page_campaign_reads_back_recoverable and
an_undecodable_file_degrades_without_destroying_it (a strict UTF-8 read raises
on those bytes wherever it runs), and the_read_never_consults_the_host_code_page,
which is a structural ratchet.

The two that pass on both builds do so by design and are reported as guards,
not as evidence: test_the_guard_can_actually_fail asserts the payload really is
undecodable under cp950 and clean under UTF-8, so it tests the oracle rather
than the fix; test_a_crlf_findings_file_still_comes_back_with_newlines is a
no-regression guard for text mode.

test_bytes_valid_under_both_decodings_resolve_to_the_declared_contract is the
ambiguous case itself — c2 a1, valid under both candidate decodings — and it
carries its own guard-the-guard (raw == b"\xc2\xa1" and
raw.decode("utf-8") == "¡") so it cannot pass on bytes that were never
ambiguous.

test_the_read_never_consults_the_host_code_page is a ratchet
(not hasattr(h, "locale") plus a source check on the reader), because the only
way to pick the host codec is to assume the reading host wrote the file, and that
assumption cannot come back by accident.

Green-after: 21 passed in that module; 254 passed / 9 skipped across
test_auto_research.py, test_auto_research_onloop_fs.py and
test_auto_research_onloop_db.py.

Gates: flake8 and isort --check-only clean; mypy --platform linux 0
errors in handlers.py
; scripts/check_black_formatting.py and
scripts/check_comment_history.py both passed, each re-run after the commit
because both scope origin/main...HEAD and pass vacuously before it exists.

Manual verification

N/A — unit coverage sufficient: the failure is a codec raise inside two helper
functions, and the tests drive those helpers directly with a payload measured to
defeat the host code page, which is exactly what a manual run would do.

Related Issues

None. Found while auditing locale.getpreferredencoding sites for the failure
class scripts/check_subprocess_encoding.py was added for — the same
decode/encode reached through a file read or write rather than a subprocess pipe,
which that gate does not cover.

Pattern harvest

Rule candidate: review-prompt

Pattern: an unpinned encoding= is only a real defect once you can name a writer
that emits non-ASCII — but "the model wrote it" is always such a writer.
Most
unpinned reads in this repository are genuinely harmless because json.dumps
defaults to ensure_ascii=True, so the bytes are ASCII whatever the code page.
That is exactly why counting call sites is the wrong triage: the same call is
noise on a JSON file and a data-loss bug on a file an LLM filled. Ask what writes
the file, not how many places read it.

Second: a partial code page is worse than a missing one. On cp950 an em dash
and CJK both encode fine and an emoji does not, so this class survives casual
testing with "some non-ASCII" and fails on the one character nobody tried. Pick
the payload from the complement of the host code page, and assert that choice in
the test so it cannot silently become a character the code page happens to cover.

Third, for tests of host-conditioned bugs: assert the property the fix chooses,
not the symptom the host produces.
"The write did not raise" is untestable on a
runner that never raises. "The bytes on disk are UTF-8" is the same guarantee,
stated so that it holds — and fails — everywhere.

Fourth: a successful decode is not provenance. A
try: decode(A) / except UnicodeDecodeError: decode(B) chain reads like a
fallback and behaves like a classifier, and it is a silent one — the wrong
branch produces plausible text, not an error. Before writing one, ask what on disk
records the encoding. If the answer is "nothing", the honest move is one decoder
plus a total, lossless degradation, so the characters survive for whoever can
decide later. Where the two candidates' domains overlap by hundreds of characters,
as UTF-8 and a DBCS code page do, trial decoding should be a review smell.

Fifth, and the one this PR learned the hard way: a security rider is not free
just because it is correct.
Hardening the reads here pulled in a no-follow
primitive whose opt-in size cap silently truncated the fork seed and the knowledge
export, and whose cut could land inside a UTF-8 character and mojibake the whole
retained prefix. Both were real findings, and neither existed before the rider
did. When a fix in one dimension needs a contract in another to stay correct, that
is the signal to split the PR, not to keep widening it.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable) — n/a, no user-facing docs cover campaign file encoding
  • No secrets, credentials, or internal references in the diff

Contribution License Agreement

@leonlaiyc
leonlaiyc requested review from a team and buluoray September 7, 2026 13:41
@github-actions github-actions Bot added fork Pull request from a fork (external contributor) readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Sep 7, 2026
@github-actions

github-actions Bot commented Sep 7, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Fable 5, fork) — 🟡 CONCERNS

Premise-level review of a565b6a76a7695e3ba775105cc86051afef73708 via the fork AI-review pipeline — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: CONCERNS

The "JSON paths are provably ASCII" boundary is false for agent-written JSON — the same locale-decode root cause survives at four read sites.

Not justified as shipped

  • Item 2 — duplicate of src/kiro_crew/knowledge/readers.py:31: _read_campaign_text re-spells _decode_text_bytes's exact utf-8→latin-1+newline-fold contract; its own docstring cites it.

What this change ships

Intent: stop Windows CJK hosts from crashing (or 500ing) when a research campaign's prose is saved or read back — a FIX.

  1. Saving a cycle's finding no longer raises on a CJK Windows host (five writes pin UTF-8) — justified
  2. Campaign prose reads no longer depend on the host code page (new _read_campaign_text) — duplicate of knowledge/readers.py:31
  3. A pre-pin code-page campaign renders as recoverable mojibake instead of a permanent 500 — justified
  4. The report route now reads FINDINGS.md off the event loop — rides along
  5. A test ratchet forbids host-code-page probes returning to this module — justified

Watch

  • Unfixed siblings of the root cause (decoding agent-authored files with the locale codec): grepped json.loads(...read_text()) — get_findings (handlers.py:701) and check_stagnation (handlers.py:563) don't even catch UnicodeDecodeError, plus _pending_question (667) and the emergent-questions read (2027). The description's "pure ASCII" rule holds only for json.dumps writers; the module's own comment (handlers.py:502) says the worker writes cycle_NNN.json itself, and _read_finding_file (721) already calls that file "LLM-written". 4 unfixed siblings.
    Clears when: those reads route through the same decode contract, or the description drops the "provably ASCII" claim and states these are left unfixed.

Subtractions

  • Collapse _read_campaign_text's try/except body to _decode_text_bytes(path.read_bytes()) (3 consumers: _read_text_or_missing, _copy_parent_findings, _read_report) — one contract to maintain, and BOM'd UTF-16 handling comes free.

[FIRST-PRINCIPLES-REVIEWED] a565b6a

@github-actions

github-actions Bot commented Sep 7, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Fable 5, fork) — 🟡 CONCERNS

Design-level review of a565b6a76a7695e3ba775105cc86051afef73708 via the fork AI-review pipeline — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: CONCERNS

Sound fix with a well-defended read contract, but the "JSON is pure ASCII" boundary only holds for process-written JSON — agent-written JSON reads keep the bug.

Watch

The PR's own triage rule ("ask what writes the file") flags reads its boundary statement exempts. The description exempts JSON because "json.dumps defaults to ensure_ascii=True", but that covers only files this process writes: emergent_questions.json (handlers.py:2027, commented "LLM output"), questions.json (handlers.py:667, agent-written per the brief at :1927), and the per-cycle finding JSONs (handlers.py:563, :701) are agent-authored, so their bytes are UTF-8 with raw non-ASCII, not ASCII. Their bare read_text() uses the locale codec, and the surrounding except (json.JSONDecodeError, OSError) does not catch UnicodeDecodeError — so on the same CJK Windows host this PR targets, an agent-mode campaign still raises uncaught on the very next read after the fixed writes succeed (and at :2027 the unconsumed file re-raises every cycle). Pre-existing on main, so no regression, but the fix is incomplete against its own stated failure class and rule.
Clears when: a follow-up (or this PR) pins/fallback-decodes the agent-written JSON reads, or the author shows those files are process-written ASCII.

[DESIGN-REVIEWED] a565b6a

@github-actions

github-actions Bot commented Sep 7, 2026 •

Copy link
Copy Markdown
Contributor

GPT 5.6 Review (fork) — ✅ no blocking findings

Reviewed a565b6a76a7695e3ba775105cc86051afef73708 via the fork AI-review pipeline; updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] a565b6a

@github-actions

github-actions Bot commented Sep 7, 2026 •

Copy link
Copy Markdown
Contributor

Opus 4.8 Review (fork) — ✅ no blocking findings

Reviewed a565b6a76a7695e3ba775105cc86051afef73708 via the fork AI-review pipeline; updated in place on each push.

Review details

No findings.

The fork transcode is lossless and reversible, not corruption: a legacy parent's bytes B become S = B.decode("latin-1") (every code point ≤ 255), and dst.write_text(S, encoding="utf-8") yields bytes B'. Reading the fork back via _read_campaign_text decodes B' as valid UTF-8 → S, matching the parent's read exactly, so both render identically. The original bytes are recoverable from the fork alone (B'.decode("utf-8").encode("latin-1") == B, since latin-1 is a total bijection on 0–255). The candidate's stated wrong outcome — "no longer recoverable from the fork alone" — is therefore false, and the only remaining harm (an external code-page-aware migration that re-decodes the raw file) is speculative machinery that does not exist and that the module explicitly documents as impossible (no format/version/writer marker on disk). (a)/(c) do not hold.

[OPUS-REVIEWED] a565b6a

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Sep 7, 2026
@leonlaiyc
leonlaiyc force-pushed the fix/auto-research-text-utf8 branch from 560ded6 to 205508a Compare September 7, 2026 14:13
@leonlaiyc

Copy link
Copy Markdown
Contributor Author

The exact-head security [BLOCK-MERGE] on 560ded6d3 is accepted and fixed at 205508a24. The finding is correct and it is a regression this PR introduced, not a pre-existing gap: pinning the reads to UTF-8 makes a campaign written before the fix — whose FINDINGS.md is sitting on disk in the host code page — raise UnicodeDecodeError on every report, fork, export and knowledge route. An upgrade would have turned a working campaign into a permanent HTTP 500.

The fix is the reviewer's prescription: UTF-8 first, and on UnicodeDecodeError decode with locale.getpreferredencoding() and migrate the file to UTF-8, so the repair happens once and the next read needs no fallback. Both are now behind one helper, _read_campaign_text, used by all three read sites. The migration is best-effort under contextlib.suppress(OSError) — a read-only or racing filesystem must not turn a successful read into an error — and bytes that decode under neither codec fall back to errors="replace" rather than raising, because losing a character from a report is better than losing the report.

The existing tests caught a real mistake in my first attempt, which is worth recording. I initially wrote the helper as read_bytes() + in-memory decode, to avoid a second open. That silently dropped text mode's universal-newline translation, so a CRLF findings file started coming back with \r\n. Two tests already in the tree failed on it:

FAILED test_auto_research_onloop_fs.py::TestKnowledgeExportIoOffLoop::test_findings_read_and_sanitized_write_run_off_the_event_loop
  assert ([])            <- the off-loop ratchet spies read_text, which I had stopped calling
FAILED test_auto_research.py::TestForkAndGrillTreeHTTP::test_fork_creates_child_with_parent_link
  AssertionError: '# Parent fin...' == '# Parent fin...'    <- the newline shift

The helper now stays on read_text for both the primary and the legacy decode, and only reaches for read_bytes on the unreachable-by-either-codec path. It costs one extra open on a legacy file, which the migration then makes the last one — a trade the docstring states rather than leaving implicit. The earlier docstring claimed reading bytes once was the safer choice; that claim is gone, because it was wrong.

New tests:

  • test_a_legacy_code_page_campaign_still_reads_and_is_migrated — writes FINDINGS.md in the host codec, asserts the read returns the original text AND that the file is now UTF-8 on disk. Skips itself where the host codec already IS UTF-8, since there is no legacy shape to stage there, and asserts the fixture is genuinely legacy-shaped before relying on it.
  • test_an_undecodable_file_degrades_instead_of_raising — bytes that decode under neither codec must not become a 500.

Gates: test_auto_research_onloop_fs.py + test_auto_research.py = 224 passed, 9 skipped, 0 failed. flake8, isort, mypy --platform linux (0 errors in handlers.py) and scripts/check_black_formatting.py clean.

The description is re-synced to describe the fallback, since the body previously described the reads as unconditionally strict UTF-8.

Amended rather than stacked, to stay inside the two-commit rule.

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 7, 2026
@leonlaiyc
leonlaiyc force-pushed the fix/auto-research-text-utf8 branch from 205508a to 2ff5d0f Compare September 8, 2026 02:08
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 8, 2026
@leonlaiyc

Copy link
Copy Markdown
Contributor Author

All three exact-head [BLOCK-MERGE] findings on 205508a24 are accepted and fixed at 2ff5d0fa8, in one amended commit rather than three sequential reviewer-fix pushes. They turned out to have one shared root cause, so the repair is a subtraction rather than three patches.

The common cause: the fallback wrote. _read_campaign_text migrated the file to UTF-8 in place after a legacy decode. That single write is what GPT 5.6, Opus 4.8's FINDING, and half of Opus 4.8's BLOCK were each seeing from a different side. Removing it closes all three at once, and is GPT's own first-listed prescription.

Finding Resolution
GPT 5.6 (security) — migration follows an agent-controlled symlink Migration write removed. See below on why the "or" branch was not viable.
Opus 4.8 (FINDING) — errors="replace" persists a lossy decode over the original bytes Same removal: nothing is written, so the original bytes always survive.
Opus 4.8 (BLOCK) — _handle_report runs findings I/O on the loop report = await asyncio.to_thread(_read_report, cid), matching the export/fork siblings.
First Principles (BLOCK) — undeclared whole-file CRLF conversion Reverted to LF. handlers.py's diff drops from 7211 lines to 61.

On GPT's second option ("or migrate through a symlink-safe no-follow file operation"): I could not make that honest. Windows has no O_NOFOLLOW — this repo already documents that at sage_lib/learning.py:132, where naming the flag unconditionally raised AttributeError. A CJK Windows host is the only host that has legacy-code-page campaign files, so a no-follow migration would have been unenforceable precisely where it was needed, leaving a lstat pre-check as the fallback — which is the "another pathname pre-check" the finding asks not to invent. Not writing at all is the only version that holds on the platform that matters.

Cost of the subtraction: one extra open when re-reading a pre-upgrade campaign, on a human-triggered route. The file is no longer repaired in place; it is simply read correctly every time.

Regressions — deterministic, one per load-bearing property, all four red on 205508a24:

test_a_legacy_code_page_campaign_reads_without_rewriting_the_file
  AssertionError: reading a report must not rewrite it
test_an_undecodable_file_degrades_without_destroying_it
  AssertionError: a lossy decode must not be persisted
test_a_findings_hardlink_is_never_written_through
  AssertionError: the link target was overwritten
test_the_findings_read_runs_off_the_event_loop
  AssertionError: assert (['MainThread'] and False)

Two shaping notes, since both were weaknesses in the previous head's tests:

  • The host codec is now pinned via monkeypatch rather than read. The previous legacy test read it and therefore skipped itself on a UTF-8 CI runner — it could never have run where CI runs.
  • The symlink case carries the repo's requires_symlinks marker, so it skips without the Windows privilege. Its hard-link sibling needs no privilege and runs everywhere — write_text truncates the inode, so a hard link is destroyed just as thoroughly. That is the variant that produced the red-before above.

Green-after: 20 passed / 1 skipped in the module; 253 passed / 10 skipped across the three auto_research suites. flake8, isort, mypy --platform linux (0 errors in handlers.py) and scripts/check_black_formatting.py all pass. The description is re-synced — it previously described a migration that no longer happens.

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 8, 2026
@leonlaiyc
leonlaiyc force-pushed the fix/auto-research-text-utf8 branch from 2ff5d0f to 68b8df2 Compare September 8, 2026 03:24
@leonlaiyc

Copy link
Copy Markdown
Contributor Author

The exact-head security [BLOCK-MERGE] on 2ff5d0fa8 is accepted and fixed at 68b8df29b. No rebuttal — it is correct, and the previous round's fix did not reach it.

The earlier finding was that the fallback's write followed an agent-planted symlink, and I closed it by removing the write. This one is about the read: path.read_text() follows a link just as happily, and the report route serves what it finds. Removing the write never touched that. Taking the two as one problem was my error, and the adjudication is right that the harm has no recovery path.

Being pre-existing is not a defence here. _read_report did a bare p.read_text() on origin/main too, so this PR did not introduce it. But the PR's whole subject is how campaign prose is read, it routes every one of those reads through a new helper, and scoping the defect out while editing the exact function would be disclosure standing in for a fix. So it is fixed here.

The fix is the reviewer's prescription, unaltered: read bytes through the hooks no-link primitive constrained to the campaign tree, then decode.

data = hooks.safe_read_file_bytes_nolink(str(path), str(research_dir()), allow_truncate=True)

safe_read_file_bytes_nolink opens with O_NOFOLLOW, fstats the descriptor it actually opened (rejecting hardlinks and non-regular files), and resolves that descriptor's real path against the research root — so there is no check-to-use window. Crucially it resolves the handle with GetFinalPathNameByHandleW on Windows, so containment holds on the platform that has no O_NOFOLLOW — which is exactly the gap that made a no-follow write impossible last round. The read has a way out that the write did not.

Three consequences of reading bytes, each handled rather than absorbed:

  1. Universal-newline translation was text mode's, and two tests in the tree pin it. It is now explicit (\r\n and \r → \n), with test_a_crlf_findings_file_still_comes_back_with_newlines added to say so. That test passes on the previous head too — it is a no-regression guard, not evidence of the fix.
  2. Missing vs refused must stay distinguishable. Callers degrade a missing findings file to "no findings yet", and a refused one must not take that path or the guard is served as an empty report. Missing raises FileNotFoundError (via lexists, so a dangling symlink reads as refused, not missing); refused raises PermissionError, which every existing except OSError already handles.
  3. The helper applies the repo's standard size cap. allow_truncate=True deliberately: FileTooLargeError is not an OSError, so refusing would escape every caller's existing guard as a 500 on a large report. Truncating a rendered view is the better answer, and this is a new bound on a previously unbounded read — stated here rather than left to be discovered.

The off-loop ratchets moved with the call, not around it. TestKnowledgeExportIoOffLoop and TestReportReadsOffLoop spied Path.read_text, which the findings read no longer calls. Rather than weaken them, a _spy_campaign_read helper spies hooks.safe_read_file_bytes_nolink; the property asserted (the read runs off the event loop) is unchanged.

New regressions — TestCampaignReadsRefuseAPlantedLink:

test_a_symlinked_findings_file_is_refused          (requires_symlinks)
test_a_hardlinked_findings_file_is_refused         runs everywhere
test_the_same_bytes_are_served_from_a_real_file    guard the guard
test_a_missing_file_is_still_missing_not_refused   guard the guard

Both link kinds are covered because they fail different checks — a symlink at the open, a hard link at st_nlink > 1 — and because the hard-link case needs no privilege, so it runs on the Windows hosts this PR's encoding fallback exists for. Red-before against 2ff5d0fa8, with the disclosure printed in the assertion:

FAILED ...::test_a_hardlinked_findings_file_is_refused
  AssertionError: assert 'Host *\n  Id.../id_ed25519\n' == ''

test_the_same_bytes_are_served_from_a_real_file exists because a fail-closed change invites a helper that refuses everything; it pins that the same bytes still come back from a genuine file.

Green-after: 255 passed / 10 skipped across test_auto_research.py, test_auto_research_onloop_fs.py and test_auto_research_onloop_db.py. flake8, isort, mypy --platform linux (0 errors in handlers.py) and scripts/check_black_formatting.py all pass. Still one amended commit.

The description is re-synced to describe the no-link read.

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Sep 8, 2026
@leonlaiyc
leonlaiyc force-pushed the fix/auto-research-text-utf8 branch from a8fa925 to ef160e7 Compare September 8, 2026 10:10
@leonlaiyc

Copy link
Copy Markdown
Contributor Author

The security [BLOCK-MERGE] on 8b03664f9 is accepted and fixed at ef160e770. No rebuttal — the finding is correct, the adjudication's refusal to FLAG it is correct, and I verified the defect still stood in the code before changing anything.

a8fa925bd was a rebase of 8b03664f9 (identical git patch-id --stable), so being a newer SHA cleared nothing: the same allow_truncate=True was still there and the same seed copy still truncated. I measured it rather than reasoning about it, with the cap lowered to 24 bytes:

source bytes    : 62
seed copy bytes : 24
seed has TAIL   : False
exception       : none;  fork route answered 201

The mistake was mine and it was a contract error, not an oversight. I reached for allow_truncate=True because FileTooLargeError is not an OSError and would escape every caller's existing guard as a 500 — and I wrote that reasoning into the docstring as if it settled the question. It does not. The answer to an exception that escapes a guard is to name it at the routes that must handle it, not to make the read return a prefix so nobody has to. Truncation bought silence at the cost of correctness, and the docstring defended the trade instead of noticing it.

The deeper error is that one reader was serving four consumers with different needs. I classified them:

Consumer What it does with the string Reader Oversize
report route renders it _read_campaign_preview bounded prefix
fork seed (parent_findings.md) stores a copy _read_campaign_text 413
export / artifact stores a copy _read_campaign_text 413
knowledge ingest stores a copy, served by RAG _read_campaign_text 413

Three of the four derive new data. A prefix handed to those is a partial copy presented as the source, and the adjudication is right that a campaign directory being agent-authored makes an oversized FINDINGS.md a producible input rather than an extreme one — the same threat model that motivates the link guard two rounds up.

I did not invent a helper for this; the repository already had both contracts. dashboard/handlers/artifacts.py passes allow_truncate=False with the comment "a file too large to store whole is REFUSED rather than promoted partial", for a promoted artifact copy — the same shape as the fork seed. The preview readers (prompts.py, cron.py, the artifact preview) pass True. So this splits one function into the two that already exist elsewhere and routes each consumer to the one that matches, and the routes name FileTooLargeError and answer 413.

One reordering rides along, and it is load-bearing rather than tidying: the fork's seed read now happens before create_campaign. Refusing after the child row and directory exist would leave an orphaned campaign behind — the same shape Opus 4.8 flagged for the refused-link case two rounds ago — so the refusal has to come first for 413 to mean anything.

The UTF-8 half of the finding is a separate mechanism and is fixed separately. A cap can land inside a multi-byte sequence; a whole-buffer decode("utf-8") then fails at the cut, and my latin-1 fallback sent the entire retained prefix through it — half a character at the end turning every earlier non-ASCII character into mojibake. The bounded read now decodes with codecs.getincrementaldecoder("utf-8")().decode(data, final=False), which holds an incomplete trailing sequence instead of raising, so at most the cut character is lost. A genuinely non-UTF-8 file still raises at its first invalid sequence and still lands on latin-1, so the encoding contract from the previous round is unchanged on both paths.

Red-before in a pristine control worktree at a8fa925bd with only the new test files copied in:

FAILED ...::test_a_complete_read_refuses_an_oversized_file
  Failed: DID NOT RAISE <class 'kiro_crew.hooks.FileTooLargeError'>
FAILED ...::test_the_export_and_knowledge_read_refuses_rather_than_truncating
  Failed: DID NOT RAISE <class 'kiro_crew.hooks.FileTooLargeError'>
FAILED ...::test_a_cap_inside_a_utf8_character_does_not_mojibake_the_prefix
  AssertionError: the prefix before the cut must survive intact
FAILED ...::test_forking_an_oversized_parent_refuses_instead_of_seeding_a_partial_copy
  AssertionError: assert 201 == 413

The third prints the defect — 已完成 arriving as latin-1 mojibake because one character was cut in half eight bytes later. The fourth is the derived-data case end to end, through the real fork route.

Two shaping notes. test_the_report_view_is_bounded_and_says_so passes on both heads and I am reporting it as a contract pin, not fix evidence: display was already bounded, and what it locks in is that the bounded contract now belongs to display alone. test_the_cap_fixture_really_truncates is the guard-the-guard — it asserts the lowered cap really returns a prefix under allow_truncate=True and really raises without it, so nothing else in the class can pass on a file that was never over the cap. The cap is monkeypatched rather than staged as a real 50 MB file, so this runs in milliseconds on every runner.

Green-after: 29 passed / 1 skipped in test_auto_research_onloop_fs.py; 263 passed / 10 skipped across the three auto_research suites. flake8, isort, mypy --platform linux (0 errors in handlers.py); scripts/check_black_formatting.py and scripts/check_comment_history.py both re-run after committing, since both scope origin/main...HEAD and pass vacuously before the commit exists.

The description is re-synced to describe the split policy. Still one commit.

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 8, 2026
@leonlaiyc
leonlaiyc force-pushed the fix/auto-research-text-utf8 branch from ef160e7 to 9913611 Compare September 8, 2026 10:27
@leonlaiyc

Copy link
Copy Markdown
Contributor Author

Correction to my previous comment: the a8fa925bd body carries THREE upheld blockers, not one. I read only the first and answered only F2. F1 and F4 are fixed at 991361127; F3 is NOT, and I am saying so rather than letting it look handled.

a8fa925bd was a rebase of 8b03664f9 with an identical git patch-id --stable, so nothing was cleared by the new SHA — and the re-review found two more.

F1 — p.exists() ahead of the link-hardened read — fixed

Accepted without rebuttal. The safe reader was doing its job and the stat in front of it was undoing it: Path.exists resolves the name, and on a Windows reparse point to a UNC target that resolution is an outbound SMB authentication, before the guard runs at all. _read_report now calls _read_campaign_preview directly and lets the reader answer — it already distinguishes FileNotFoundError from PermissionError, and both are OSError, which the caller already catches. A subtraction, not an addition.

Pinned as a ratchet and labelled one: the mechanism needs a reparse point to an attacker-controlled share, which a test cannot stage, so test_the_report_read_never_stats_the_path_first asserts the shape and test_a_missing_report_is_still_an_empty_report guards that dropping the stat did not turn "no findings yet" into an error.

The same exists() also runs in the export and knowledge routes and I did NOT remove those. Removing them reordered two responses (assert 503 == 404 in two existing tests) because the stat is load-bearing for the 404-before-503 ordering there. They are pre-existing, not in this PR's diff, and the finding named only _read_report — so they stay, named here rather than quietly left.

F2 — truncation corrupts derived data — fixed (detailed in my previous comment)

F4 (advisory) — the byte-exact claim — fixed

This one is mine and it is the kind I most want caught. The docstring asserted text.encode("latin-1") == data, and that is false for a legacy file containing CR, because the read folds newlines afterwards. Worse, my own test had assert b"\r" not in raw in it — I had staged a fixture that avoided the case my prose was over-claiming. The fixture now carries \r\n, the assertion is out.encode("latin-1") == raw.replace(b"\r\n", b"\n"), and the docstring says "byte-exact up to the newline fold". The code was always right; the sentence was wrong.

F3 — campaign writes still follow links — NOT fixed, and not scoped away quietly

The finding is correct and I am not disputing it. write_guidance, _write_text, _write_new_cycle_files and the brief.md write all reach the file by name, so a planted guidance.txt symlink lets a user nudge O_TRUNC an arbitrary host file. This PR touched those lines and hardened the read path against precisely this threat, so the asymmetry is fair.

The prescription is "route every changed campaign write through the existing pinned no-link writer." I tried to, and measured why it is not a call-site swap:

  • hooks.safe_write_file_nolink opens without O_CREAT by design — "a caller mirroring content back to a file it previously read has no business creating one." Campaign writes create the file on first write, so it cannot serve them as-is.

  • atomic_write does defeat the mechanism — verified, because the rename replaces the planted link instead of writing through it:

    atomic_write to a hardlinked dest : victim -> SECRET       (intact)
    write_text  to a hardlinked dest  : victim -> OVERWRITTEN  (destroyed)
    

    But it also does path.parent.mkdir(parents=True, exist_ok=True), which breaks _write_text's documented contract that a concurrent campaign deletion must win rather than be resurrected.

  • Its parent_dir_fd mode skips that mkdir — and is unsupported on Windows: pinned_parent_replace_supported() returns False, and opening a directory descriptor there fails EACCES (measured on this host). Windows is the platform this PR exists for.

So a correct fix needs a create-capable, link-refusing, parent-non-creating writer that works on Windows, and the repository does not have one. That is a design change to the write path with its own review surface. Every previous round of this PR produced its next finding by making exactly this kind of change under time pressure inside a security-fenced area, and I would rather hand you an evidenced stop than a fourth swing at a primitive I have not designed properly. Happy to do it as its own PR, or to expand this one if you would rather it ship together — your call, and I will not close this out claiming it is clean.

Green-after: 31 passed / 1 skipped in test_auto_research_onloop_fs.py; 265 passed / 10 skipped across the three auto_research suites. flake8, isort, mypy --platform linux (0 errors in handlers.py); check_black_formatting.py and check_comment_history.py both re-run after committing, since both scope origin/main...HEAD and pass vacuously otherwise.

Still one commit. Description re-synced, including the F3 disclosure above.

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 8, 2026
`auto_research` moves model-generated prose through `Path.read_text` /
`write_text` with no `encoding=`, so the bytes on disk are whatever
`locale.getpreferredencoding()` picks -- UTF-8 on POSIX, the ANSI code page on
Windows. A model routinely emits characters a legacy ANSI code page cannot
represent, so on a CJK Windows host writing `FINDINGS.md` raises
`UnicodeEncodeError` and the cycle's finding is lost rather than saved
(measured on cp950).

The five free-text writes now pin `encoding="utf-8"`, and the reads that pair
with them go through `_read_campaign_text`.

That reader exists for one reason: pinning the reads to UTF-8 would make a
campaign created BEFORE the pin -- whose findings are sitting on disk in the
host code page -- raise `UnicodeDecodeError` on every report, fork, export and
knowledge route, turning an upgrade into a permanent HTTP 500. So the read must
not raise.

It must also not GUESS. Retrying with `locale.getpreferredencoding()` treats a
decode succeeding as evidence of what wrote the file, and it is not: cp950
`U+7652` is the bytes `c2 a1`, which are equally valid UTF-8 for `U+00A1`, and
899 cp950 multi-byte characters have that property, so a legacy report could be
served -- and copied into a fork -- as Latin mojibake with nothing raised. It
would also take provenance from ambient state, applying the READING host's code
page to a file some other host wrote.

Nothing on disk can break that tie: a campaign directory carries no format or
version marker, `status.json` has no schema version, the campaign row records no
writer generation, and in agent mode `FINDINGS.md` is written by the research
agent rather than by this process at all. So the codec is not chosen by trial.
`latin-1` is total and lossless, so a pre-pin file still renders, the file is
never rewritten, and its characters stay recoverable from the string -- byte-exact
up to the universal-newline translation both reads already applied. This is the
contract `kiro_crew.knowledge.readers._decode_text_bytes` already uses for text
of unknown provenance.

The cost is stated rather than hidden: a pre-pin campaign whose findings really
were in the host code page renders as mojibake on that host instead of correct
text. That population is narrow -- the same write raised `UnicodeEncodeError`
for any character outside the code page, which is the defect this fixes, so the
files that survived are the ones that were entirely code-page-encodable -- and
the characters are recoverable, where a wrong guess is not detectable at all.

Both reads stay in text mode, so universal-newline translation is unchanged and
a CRLF findings file still comes back with `\n`. Nothing on the read path
writes: no migration, no re-encode of the source, no `errors="replace"` landing
back on disk.

`_handle_report` reads through `asyncio.to_thread`, matching the export and fork
siblings that already offload this same unbounded file -- this change is what
can make that read two opens on a legacy campaign.

Campaign reads and writes follow links exactly as they do on `origin/main`; this
change does not alter that in either direction, and hardening it belongs in its
own PR (see the description).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@leonlaiyc
leonlaiyc force-pushed the fix/auto-research-text-utf8 branch from 9913611 to a565b6a Compare September 8, 2026 14:07
@leonlaiyc

Copy link
Copy Markdown
Contributor Author

Both exact-head blockers on 991361127 are accepted, and the answer is a scope reduction rather than a third security patch. a565b6a76 removes the riders that produced them.

The production diff drops from 255 changed lines to 59, and from three changed files to two.

Why reduction and not repair

Both findings live in code that was never part of the encoding fix. Classifying every production hunk made that obvious:

required for the encoding fix five encoding="utf-8" writes; the reads that pair with them
repairs a regression caused by the above the non-raising legacy decode (a strict UTF-8 read would 500 every pre-pin campaign)
unrelated security hardening link-hardened reads, bounded preview, truncation policy, 413 derived-data behaviour, exists() pre-stat

Everything in the third row is gone. It was not free: routing prose reads through the no-follow primitive brought its opt-in 50 MB cap along, and that cap is exactly what produced Blocker A — truncation ahead of _redact_finding, plus a cut landing inside a UTF-8 character and sending the whole retained prefix through the fallback codec. The rider generated the finding. Removing the rider removes it causally, not by argument:

$ grep -n 'allow_truncate|safe_read_file_bytes_nolink|_read_campaign_preview|MAX_FILE_BYTES|FileTooLargeError|read_bytes' handlers.py
(no matches)

There is no cap, no truncation and no byte read on any campaign prose path, so there is nothing that can be truncated before redaction. Blocker A is gone by construction.

What survives

try:
    return path.read_text(encoding="utf-8")
except UnicodeDecodeError:
    return path.read_text(encoding="latin-1")

Both reads stay in text mode, so universal-newline translation comes back for free and the CRLF handling, the incremental decoder and the byte-level plumbing all disappear with it. Nothing on the read path writes. No locale probing, no migration write, no lossy rewrite of the source, no CRLF churn — the four things I was told not to reintroduce, and none of them is back.

Blocker B — the write path

The finding is correct and the defect is real. After reduction, this PR's entire change to those lines is an encoding= keyword:

-    path.write_text(text)
+    path.write_text(text, encoding="utf-8")

Link-following is byte-identical to origin/main before and after, and with the read hardening gone there is no longer an asymmetry where reads are protected and writes are not. So I am not claiming the defect away — I am saying this PR neither introduces nor changes it.

I did look for the primitive rather than assuming there was none. pinned_fs.write_file_pinned is it — it calls itself "THE single no-follow file-publish path" and keeps its lstat refusal of a planted link on Windows too, which is where my earlier round wrongly concluded no-follow writes were impossible. Measured:

supports_pinned_walk()                     : False   (Windows)
planted hardlink -> victim file            : SECRET  (refused; the mechanism IS closed)
write into a deleted campaign directory    : parent recreated

The mechanism really is closed by it. The third line is why it is not a drop-in: it collides with TestWriteHelperDeletionRace::test_write_text_does_not_resurrect_a_deleted_campaign_dir, a test already on main that names findings_for_knowledge.md — the exact file this finding cites — and requires a concurrent campaign deletion to win rather than have its directory silently recreated. Reconciling "never follow a planted link" against "never resurrect a deleted campaign" is a design decision about the write path, and an encoding fix should not make it in passing.

That is in the PR description, with the measurements, so whoever picks it up starts from the primitive and the conflict rather than rediscovering both. If you would rather this PR carry it anyway, say so and I will do it properly here.

Red-before

Production reverted to origin/main, new tests kept, on a cp950 host: 7 failed, 2 passed, including UnicodeEncodeError: 'cp950' codec can't encode character '\U0001f600' from the two writers and UnicodeDecodeError: 'cp950' codec can't decode byte 0xf0 from the fork round trip.

Four of those seven cannot go red on a UTF-8 CI runner — the original bug is host-conditioned — so their assertions are written as "the bytes on disk are UTF-8" rather than "the write did not raise", and I am naming which four rather than letting the count imply more than it proves. Three fail on every host, including the not hasattr(h, "locale") ratchet. The two that pass on both builds are the oracle test and a no-regression guard, reported as guards.

Green-after: 21 passed in the module; 254 passed / 9 skipped across the three auto_research suites. flake8, isort, mypy --platform linux (0 errors in handlers.py); check_black_formatting.py and check_comment_history.py both re-run after committing, since both scope origin/main...HEAD and pass vacuously otherwise.

Still one commit. Description rewritten for the reduced scope.

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention merge conflict Branch has merge conflicts with its base — author must resolve before merge and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 8, 2026
@NicholasRBowers NicholasRBowers added needs-pr-triage PR scanner: awaiting automated triage needs-author-decision PR blocked on author input and removed needs-pr-triage PR scanner: awaiting automated triage labels Sep 12, 2026
@NicholasRBowers

Copy link
Copy Markdown
Contributor

🤖 Kiro Crew [operator: NicholasRBowers#a942f9ca]: This PR has been inactive for 7+ days. I reviewed the blockers but they require your input:

  • main has since pinned UTF-8 at every site this PR touches, with a different legacy-file policy. fix(apps): declare every builtin's platform truthfully, and fix the Windows defects #9413 (merged 2026-09-09) changed the same read_text() / write_text() calls in src/kiro_crew/apps/builtins/auto_research/handlers.py to encoding="utf-8", errors="replace" (lossy on a pre-pin code-page file, never raises). This PR's _read_campaign_text() instead does strict UTF-8 with a lossless latin-1 fallback and argues against guessing/repairing. Every hunk now conflicts with main's version of the same line, so the rebase is not mechanical: the resolution IS the choice between errors="replace" and the latin-1 fallback for legacy campaign files. Only you can make that call: (a) accept main's policy and close this PR or reduce it to the new test_auto_research_onloop_fs.py cases that still hold against main, or (b) rebase onto main and re-cast the PR as replacing errors="replace" with _read_campaign_text() on the read paths, with the body arguing why the lossless fallback is the right contract.
  • Design Review and First Principles Review are both 🟡 CONCERNS on a565b6a76 with no disposition posted; once scope is settled, each item needs its own target=design / target=first-principles disposition comment.
  • The sole CI red, Backend Tests (Windows) (4), is test/test_work_ledger.py::test_two_conductors_binding_one_worker_at_once_yield_exactly_one_binding -- a file this diff does not touch, already fixed on main by test: second five-run hygiene pass (real-home escapes, host reds, flakes) #9614/fix(work-ledger): retry a contended record read; make a test's ids salt-free #9673; a rebase clears it. GPT and Opus are clean on the current head.

When you've addressed these, the pipeline will re-assess on its next cycle.

@iamwhatever

Copy link
Copy Markdown
Contributor

Closing as superseded.

Every write in src/kiro_crew/apps/builtins/auto_research/handlers.py already pins encoding="utf-8" on main (lines 631, 646, 660, 711, 2034), landed by #9413 ("fix(apps): declare every builtin's platform truthfully, and fix the Windows defects", merged 2026-09-09). The host-code-page defect this PR describes no longer exists at those sites.

The branch has been in merge conflict and idle 8+ days, and the remaining delta is the regression test plus a .github/ baseline row that main has since pruned differently.

If you want the coverage kept, refile the regression test alone rebased onto main, with no baseline edit. Your other encoding fixes in this family are still live -- see #7560, which I have flagged for rebase rather than closure.

@github-actions github-actions Bot removed the readiness: action required A blocking check or review needs attention label Sep 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

fork Pull request from a fork (external contributor) merge conflict Branch has merge conflicts with its base — author must resolve before merge needs-author-decision PR blocked on author input

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants