Repository navigation
fix(portability): decode the imported cron store as UTF-8, not the locale - #12855
Conversation
Opus 5 Review (fork) — ✅ no blocking findingsReviewed Review detailsNothing blocks; one advisory finding. FINDING — test/test_portability.py:2371 — the comment [OPUS-REVIEWED] 875b4ea |
Design Review (Fable 5, fork) — ✅ PASSDesign-level review of Design-Verdict: PASS Nothing to check. [DESIGN-REVIEWED] 875b4ea |
GPT 5.6 Review (fork) — ✅ no blocking findingsReviewed Review detailsNo findings. |
First Principles Review (Fable 5, fork) — ✅ PASSPremise-level review of First-Principles-Verdict: PASS Nothing to check. What this change shipsInventory (5 items) — 5 justifiedIntent: stop a Windows-locale decode from wiping or mangling an imported cron schedule — a FIX.
Verification run: grepped [FIRST-PRINCIPLES-REVIEWED] 875b4ea |
eca058b to
729b8cd
Compare
…cale
`_sanitize_imported_crons` read and wrote the imported `crons.json` with a
bare `Path.read_text()`/`write_text()`, which decode and encode with
`locale.getencoding()` — cp1252 on a Windows host. Cron job names are
operator-authored text and routinely non-ASCII, and `snapshot._merge_crons`,
which merges this same file a few lines later in `apply_import_zip`, already
pins `encoding="utf-8"` on both ends for exactly that reason. The sanitizer
runs FIRST, so it was the one path in the flow still reading the archive's
UTF-8 through the code page.
Both failure directions are real, and the second is the damaging one:
- a code page that cannot decode the bytes raises `UnicodeDecodeError`. That
IS a `ValueError`, so it lands in this function's own
`except (ValueError, OSError)` recovery arm, whose job is to replace an
unusable store with `{"jobs": []}`. A perfectly good backup is silently
reduced to an empty schedule and reported as "unreadable".
- a code page that decodes most bytes (cp1252) fails the other way: the read
succeeds with mojibake, and the sanitizer's pause rewrite persists that
mojibake to disk, so the corruption survives into the live store.
Pin `encoding="utf-8"` on the one read and the three writes. The recovery arm
itself is unchanged: bytes that are not JSON at all are still replaced.
Covered by `TestTheCronSanitizerDecodesAsUtf8`, which drives the code page
through `io.text_encoding` — the funnel `read_text`/`write_text` consult when
no encoding is named. Patching the locale function alone would not reproduce
this under `PYTHONUTF8`, which is set for successors but not for the running
gateway process.
729b8cd to
875b4ea
Compare
…cale (kirodotdev#12855) `_sanitize_imported_crons` read and wrote the imported `crons.json` with a bare `Path.read_text()`/`write_text()`, which decode and encode with `locale.getencoding()` — cp1252 on a Windows host. Cron job names are operator-authored text and routinely non-ASCII, and `snapshot._merge_crons`, which merges this same file a few lines later in `apply_import_zip`, already pins `encoding="utf-8"` on both ends for exactly that reason. The sanitizer runs FIRST, so it was the one path in the flow still reading the archive's UTF-8 through the code page. Both failure directions are real, and the second is the damaging one: - a code page that cannot decode the bytes raises `UnicodeDecodeError`. That IS a `ValueError`, so it lands in this function's own `except (ValueError, OSError)` recovery arm, whose job is to replace an unusable store with `{"jobs": []}`. A perfectly good backup is silently reduced to an empty schedule and reported as "unreadable". - a code page that decodes most bytes (cp1252) fails the other way: the read succeeds with mojibake, and the sanitizer's pause rewrite persists that mojibake to disk, so the corruption survives into the live store. Pin `encoding="utf-8"` on the one read and the three writes. The recovery arm itself is unchanged: bytes that are not JSON at all are still replaced. Covered by `TestTheCronSanitizerDecodesAsUtf8`, which drives the code page through `io.text_encoding` — the funnel `read_text`/`write_text` consult when no encoding is named. Patching the locale function alone would not reproduce this under `PYTHONUTF8`, which is set for successors but not for the running gateway process. Co-authored-by: DeryFerd <DeryFerd@users.noreply.github.com>
Problem / Motivation
_sanitize_imported_cronsinsrc/kiro_crew/portability.pyread and wrote the importedcrons.jsonwith a barePath.read_text()/Path.write_text(). With noencoding=argument, both consultlocale.getencoding(), which is cp1252 on a Windows host. Cron job names are operator-authored text and routinely non-ASCII, so that code page is the wrong decoder for this file.Two distinct failures follow, and I reproduced both against the real function before changing anything:
UnicodeDecodeError. That is aValueError, so it lands in this function's ownexcept (ValueError, OSError)recovery arm, whose job is to replace an unusable store with{"jobs": []}. A perfectly good backup is silently reduced to an empty schedule, and the import summary tells the user their store was unreadable when it was not.Why it matters
The rewrite is not an edge path. Rule 3 of this function pauses every imported job that runs a command or a script, which is the common case for a settings backup, and pausing rewrites the store. So any import on a Windows host with a non-ASCII job name goes through the write half of this bug.
The wipe direction is worse than the mojibake one, because it is silent and it lies. A user restores a backup and their entire schedule is gone, with a message blaming the archive rather than the decoder. The archive was fine.
The same function also runs first in
apply_import_zip, before the merge-or-copy branch that installs the store, so a store mangled here is what the rest of the import operates on.What changed (motivation → approach → change)
Symptom: an imported cron store with non-ASCII job names comes back mangled or empty on a Windows host. Root cause: this function was the one reader/writer in the flow that named no encoding.
The sibling is what made the fix obvious.
snapshot._merge_cronsmerges this exact file a few lines later in the sameapply_import_zipcall and already pinsencoding="utf-8"on both its read and its write, with a comment stating the reason: cron job names are operator-authored text and routinely non-ASCII, so the locale codepage is the wrong decoder for this file on any host. The sanitizer runs first, so it was the remaining path that read the archive's UTF-8 through the code page.The change is four
encoding="utf-8"arguments, one on the read and three on the writes:src/kiro_crew/portability.py— theread_text()at the top of_sanitize_imported_crons, plus the threewrite_text()calls (the two{"jobs": []}recovery writes and the pause rewrite).I considered two alternatives and did not take them:
errors="replace"would make a malformed byte decode instead of raising. That is the wrong direction for a JSON store: it would substitute a replacement character into a job name and then write it back, turning a detectable bad store into a silently altered one. Strict decoding plus the existing recovery arm is the contract this function already had, and I left it alone.Backwards compatibility
Compatible: no signature, schema, or contract changes. The four sites only gain an
encodingargument. On a host whose locale is already UTF-8, which is every POSIX host and a Windows host running withPYTHONUTF8, the bytes read and written are identical to before. The only hosts whose behaviour changes are those whose code page differs from UTF-8, and there the change is from corrupt to correct.Tests
test/test_portability.pygainsTestTheCronSanitizerDecodesAsUtf8, three tests that lock the read half, the write half, and the recovery arm:dropped == ["<the whole cron store was unreadable>"].{"jobs": []}, so the fix does not disable the recovery arm it sits next to.I proved the tests catch this rather than merely pass. Stashing the source half of the change makes two of the three fail with exactly the symptoms above; restoring it makes all three pass.
One detail worth a reviewer's attention, because it is how I got this wrong first. The tests drive the code page through
io.text_encoding, not throughlocale.getencoding.platform_compat.ensure_utf8_console()setsPYTHONUTF8for successor processes, and setting that variable in-process does not changesys.flags.utf8_modeor whatread_text()resolves to, so patching the locale function alone lets the test go green while the defect is live. The first version of this test passed for that reason and I rewrote it.Alongside the new tests:
test/test_portability.pyandtest/test_onboarding_import_coverage.py— 447 passed, 7 skipped.test/test_subprocess_encoding_gate.py,test/test_subprocess_utf8.py,test/test_issue_radar_az_encoding.py,test/test_cron.py— 171 passed, 1 skipped. These are the suites that own the encoding contract on this codebase, so I ran them even though the diff does not touch them.Manual verification
N/A, unit coverage is sufficient here: the defect is entirely in how a path is opened, and the tests assert the decoded job name and the bytes on disk, which is what a manual import would show. I did reproduce both failure directions outside pytest as well, by writing a UTF-8 store and calling the function directly under a forced code page, and the results matched what the tests now assert.
One disclosure so a red CI run does not read as ignored.
test/test_snapshot.pyhas three failures on this Windows checkout:TestSecurity::test_symlink_filtered_out(WinError 1314, the symlink privilege) and twoTestNotificationMergeWriteSideContracttests (WinError 123). I confirmed they are pre-existing rather than mine by checking out a cleanupstream/mainand running the same three, which fail identically there. They are in a file my diff does not touch, and they are Windows-only privilege failures that Linux CI does not hit.Related Issues
No dedicated issue. This is the same defect class the project has already fixed twice: pinning UTF-8 on the mcp-gateway rewriter's remaining JSON reads (#10743) and decoding
azoutput as UTF-8 rather than the host code page (#7560). Both were a text read with noencoding=decoding UTF-8 data through the code page, and this is one more instance of it.Pattern harvest
Rule candidate: lint
Pattern: a
Path.read_text()/Path.write_text()with noencoding=on a data file, thepathlibtwin of the classscripts/check_subprocess_encoding.pyalready gates forsubprocesstext mode. That gate exists because the same mistake reached users through the dashboard's file diffs (#3219, #3669) and its own header says the point is to keep the class from growing back. There is no equivalent forPath, so the two merged fixes above and this one were each found by hand. A mechanical check would want a shrink-only baseline rather than a hard zero, since a bareread_text()on a/procleaf is correct on Linux and there are several hundred of those; the subprocess gate's baseline file is the established shape for exactly that situation.Checklist
feat|fix|docs|refactor|perf|test|chore|ci|build|revert: ...) — one commit,fix(portability): decode the imported cron store as UTF-8, not the localesnapshot._merge_cronsalready uses on the same file