Skip to content

fix(portability): decode the imported cron store as UTF-8, not the locale - #12855

Merged
iamwhatever merged 1 commit into
kirodotdev:mainfrom
DeryFerd:fix/portability-cron-sanitizer-utf8
Sep 24, 2026
Merged

iamwhatever merged 1 commit into
kirodotdev:mainfrom
DeryFerd:fix/portability-cron-sanitizer-utf8

Conversation

@DeryFerd

Copy link
Copy Markdown
Contributor

Problem / Motivation

_sanitize_imported_crons in src/kiro_crew/portability.py read and wrote the imported crons.json with a bare Path.read_text() / Path.write_text(). With no encoding= argument, both consult locale.getencoding(), which is cp1252 on a Windows host. Cron job names are operator-authored text and routinely non-ASCII, so that code page is the wrong decoder for this file.

Two distinct failures follow, and I reproduced both against the real function before changing anything:

  • A code page that cannot decode the bytes raises UnicodeDecodeError. That is a ValueError, so it lands in this function's own except (ValueError, OSError) recovery arm, whose job is to replace an unusable store with {"jobs": []}. A perfectly good backup is silently reduced to an empty schedule, and the import summary tells the user their store was unreadable when it was not.
  • A code page that decodes most bytes (cp1252 has five undefined byte values) fails the other way. The read succeeds with mojibake, and the sanitizer's own pause rewrite then persists that mojibake to disk, so the corruption survives into the live store.

Why it matters

The rewrite is not an edge path. Rule 3 of this function pauses every imported job that runs a command or a script, which is the common case for a settings backup, and pausing rewrites the store. So any import on a Windows host with a non-ASCII job name goes through the write half of this bug.

The wipe direction is worse than the mojibake one, because it is silent and it lies. A user restores a backup and their entire schedule is gone, with a message blaming the archive rather than the decoder. The archive was fine.

The same function also runs first in apply_import_zip, before the merge-or-copy branch that installs the store, so a store mangled here is what the rest of the import operates on.

What changed (motivation → approach → change)

Symptom: an imported cron store with non-ASCII job names comes back mangled or empty on a Windows host. Root cause: this function was the one reader/writer in the flow that named no encoding.

The sibling is what made the fix obvious. snapshot._merge_crons merges this exact file a few lines later in the same apply_import_zip call and already pins encoding="utf-8" on both its read and its write, with a comment stating the reason: cron job names are operator-authored text and routinely non-ASCII, so the locale codepage is the wrong decoder for this file on any host. The sanitizer runs first, so it was the remaining path that read the archive's UTF-8 through the code page.

The change is four encoding="utf-8" arguments, one on the read and three on the writes:

  • src/kiro_crew/portability.py — the read_text() at the top of _sanitize_imported_crons, plus the three write_text() calls (the two {"jobs": []} recovery writes and the pause rewrite).

I considered two alternatives and did not take them:

  • Pinning only the read would fix the wipe direction and leave the mojibake direction, because the write half is what persists the damage. Both ends have to be pinned, which is what the sibling already does.
  • Adding errors="replace" would make a malformed byte decode instead of raising. That is the wrong direction for a JSON store: it would substitute a replacement character into a job name and then write it back, turning a detectable bad store into a silently altered one. Strict decoding plus the existing recovery arm is the contract this function already had, and I left it alone.

Backwards compatibility

Compatible: no signature, schema, or contract changes. The four sites only gain an encoding argument. On a host whose locale is already UTF-8, which is every POSIX host and a Windows host running with PYTHONUTF8, the bytes read and written are identical to before. The only hosts whose behaviour changes are those whose code page differs from UTF-8, and there the change is from corrupt to correct.

Tests

test/test_portability.py gains TestTheCronSanitizerDecodesAsUtf8, three tests that lock the read half, the write half, and the recovery arm:

  • A valid UTF-8 store with a non-ASCII job name is not reported as unreadable and is not rewritten, which fails today with dropped == ["<the whole cron store was unreadable>"].
  • A job that executes is paused and reported by its real name, and the rewritten store still holds that name, which is the assertion the mojibake direction fails.
  • A store whose bytes are not JSON at all is still replaced with {"jobs": []}, so the fix does not disable the recovery arm it sits next to.

I proved the tests catch this rather than merely pass. Stashing the source half of the change makes two of the three fail with exactly the symptoms above; restoring it makes all three pass.

One detail worth a reviewer's attention, because it is how I got this wrong first. The tests drive the code page through io.text_encoding, not through locale.getencoding. platform_compat.ensure_utf8_console() sets PYTHONUTF8 for successor processes, and setting that variable in-process does not change sys.flags.utf8_mode or what read_text() resolves to, so patching the locale function alone lets the test go green while the defect is live. The first version of this test passed for that reason and I rewrote it.

Alongside the new tests:

  • test/test_portability.py and test/test_onboarding_import_coverage.py — 447 passed, 7 skipped.
  • test/test_subprocess_encoding_gate.py, test/test_subprocess_utf8.py, test/test_issue_radar_az_encoding.py, test/test_cron.py — 171 passed, 1 skipped. These are the suites that own the encoding contract on this codebase, so I ran them even though the diff does not touch them.

Manual verification

N/A, unit coverage is sufficient here: the defect is entirely in how a path is opened, and the tests assert the decoded job name and the bytes on disk, which is what a manual import would show. I did reproduce both failure directions outside pytest as well, by writing a UTF-8 store and calling the function directly under a forced code page, and the results matched what the tests now assert.

One disclosure so a red CI run does not read as ignored. test/test_snapshot.py has three failures on this Windows checkout: TestSecurity::test_symlink_filtered_out (WinError 1314, the symlink privilege) and two TestNotificationMergeWriteSideContract tests (WinError 123). I confirmed they are pre-existing rather than mine by checking out a clean upstream/main and running the same three, which fail identically there. They are in a file my diff does not touch, and they are Windows-only privilege failures that Linux CI does not hit.

Related Issues

No dedicated issue. This is the same defect class the project has already fixed twice: pinning UTF-8 on the mcp-gateway rewriter's remaining JSON reads (#10743) and decoding az output as UTF-8 rather than the host code page (#7560). Both were a text read with no encoding= decoding UTF-8 data through the code page, and this is one more instance of it.

Pattern harvest

Rule candidate: lint

Pattern: a Path.read_text() / Path.write_text() with no encoding= on a data file, the pathlib twin of the class scripts/check_subprocess_encoding.py already gates for subprocess text mode. That gate exists because the same mistake reached users through the dashboard's file diffs (#3219, #3669) and its own header says the point is to keep the class from growing back. There is no equivalent for Path, so the two merged fixes above and this one were each found by hand. A mechanical check would want a shrink-only baseline rather than a hard zero, since a bare read_text() on a /proc leaf is correct on Linux and there are several hundred of those; the subprocess gate's baseline file is the established shape for exactly that situation.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|refactor|perf|test|chore|ci|build|revert: ...) — one commit, fix(portability): decode the imported cron store as UTF-8, not the locale
  • Existing tests pass and new tests added for new functionality — three new tests, proven to fail without the source change
  • Self-review completed; code follows project style guidelines — the change mirrors the encoding handling snapshot._merge_crons already uses on the same file
  • Documentation updated (if applicable) — N/A: no doc states an encoding contract for this read/write, and this PR does not change documented behaviour, it fixes the code to match what the sibling reader already does
  • No secrets, credentials, or internal references in the diff

@DeryFerd
DeryFerd requested review from a team and krishdhasmana September 22, 2026 16:50
@github-actions github-actions Bot added fork Pull request from a fork (external contributor) readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Sep 22, 2026
@github-actions

github-actions Bot commented Sep 22, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5 Review (fork) — ✅ no blocking findings

Reviewed 875b4ea1dbc93f7a035fb1dab2f50db8077a6fbc via the fork AI-review pipeline; updated in place on each push.

Review details

Nothing blocks; one advisory finding.

FINDING — test/test_portability.py:2371 — the comment # is re-encoded with the code page, so the mojibake the bare read (and the class docstring's claim that the write half is "asserted below") describes a mechanism that cannot occur: json.dumps(data, indent=2) at portability.py:747 keeps ensure_ascii=True, so write_text receives pure ASCII and cp1252 encodes it byte-identically to UTF-8 — deleting encoding="utf-8" from all three write calls (lines 668/681/747) leaves the whole added class green, so only the read is pinned → Fix: either dump the rewrite with ensure_ascii=False so the on-disk bytes actually depend on the write codec and the existing name assertion pins it, or reword the comment/docstring to say the write encoding is explicitness for consistency with snapshot._merge_crons, not asserted behaviour.

[OPUS-REVIEWED] 875b4ea

@github-actions

github-actions Bot commented Sep 22, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Fable 5, fork) — ✅ PASS

Design-level review of 875b4ea1dbc93f7a035fb1dab2f50db8077a6fbc via the fork AI-review pipeline — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Nothing to check.

[DESIGN-REVIEWED] 875b4ea

@github-actions

github-actions Bot commented Sep 22, 2026 •

Copy link
Copy Markdown
Contributor

GPT 5.6 Review (fork) — ✅ no blocking findings

Reviewed 875b4ea1dbc93f7a035fb1dab2f50db8077a6fbc via the fork AI-review pipeline; updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 875b4ea

@github-actions

github-actions Bot commented Sep 22, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Fable 5, fork) — ✅ PASS

Premise-level review of 875b4ea1dbc93f7a035fb1dab2f50db8077a6fbc via the fork AI-review pipeline — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: PASS

Nothing to check.

What this change ships

Inventory (5 items) — 5 justified

Intent: stop a Windows-locale decode from wiping or mangling an imported cron schedule — a FIX.

  1. Importing a backup with non-ASCII cron job names no longer wipes the schedule on a non-UTF-8 host — justified
  2. The pause rewrite no longer persists code-page mojibake into the live cron store — justified
  3. The recovery arm's empty-store replacement now writes UTF-8 bytes — justified
  4. The function's docstring now records the UTF-8 contract and why — justified
  5. Three tests pin the read, write, and recovery halves of that contract — justified

Verification run: grepped \.(read_text|write_text)\( in src/kiro_crew/portability.py — 4 hits, all fixed by this diff, 0 siblings left; _merge_crons in snapshot.py already pins encoding="utf-8" on all 3 of its sites.

[FIRST-PRINCIPLES-REVIEWED] 875b4ea

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Sep 22, 2026
@DeryFerd
DeryFerd force-pushed the fix/portability-cron-sanitizer-utf8 branch from eca058b to 729b8cd Compare September 24, 2026 09:29
…cale

`_sanitize_imported_crons` read and wrote the imported `crons.json` with a
bare `Path.read_text()`/`write_text()`, which decode and encode with
`locale.getencoding()` — cp1252 on a Windows host. Cron job names are
operator-authored text and routinely non-ASCII, and `snapshot._merge_crons`,
which merges this same file a few lines later in `apply_import_zip`, already
pins `encoding="utf-8"` on both ends for exactly that reason. The sanitizer
runs FIRST, so it was the one path in the flow still reading the archive's
UTF-8 through the code page.

Both failure directions are real, and the second is the damaging one:

- a code page that cannot decode the bytes raises `UnicodeDecodeError`. That
  IS a `ValueError`, so it lands in this function's own
  `except (ValueError, OSError)` recovery arm, whose job is to replace an
  unusable store with `{"jobs": []}`. A perfectly good backup is silently
  reduced to an empty schedule and reported as "unreadable".
- a code page that decodes most bytes (cp1252) fails the other way: the read
  succeeds with mojibake, and the sanitizer's pause rewrite persists that
  mojibake to disk, so the corruption survives into the live store.

Pin `encoding="utf-8"` on the one read and the three writes. The recovery arm
itself is unchanged: bytes that are not JSON at all are still replaced.

Covered by `TestTheCronSanitizerDecodesAsUtf8`, which drives the code page
through `io.text_encoding` — the funnel `read_text`/`write_text` consult when
no encoding is named. Patching the locale function alone would not reproduce
this under `PYTHONUTF8`, which is set for successors but not for the running
gateway process.
@DeryFerd
DeryFerd force-pushed the fix/portability-cron-sanitizer-utf8 branch from 729b8cd to 875b4ea Compare September 24, 2026 15:55
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 24, 2026
@iamwhatever
iamwhatever enabled auto-merge (squash) September 24, 2026 16:57
@iamwhatever
iamwhatever merged commit b923665 into kirodotdev:main Sep 24, 2026
97 checks passed
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Sep 24, 2026
wknight1 pushed a commit to wknight1/KiroCrew that referenced this pull request Oct 5, 2026
…cale (kirodotdev#12855)

`_sanitize_imported_crons` read and wrote the imported `crons.json` with a
bare `Path.read_text()`/`write_text()`, which decode and encode with
`locale.getencoding()` — cp1252 on a Windows host. Cron job names are
operator-authored text and routinely non-ASCII, and `snapshot._merge_crons`,
which merges this same file a few lines later in `apply_import_zip`, already
pins `encoding="utf-8"` on both ends for exactly that reason. The sanitizer
runs FIRST, so it was the one path in the flow still reading the archive's
UTF-8 through the code page.

Both failure directions are real, and the second is the damaging one:

- a code page that cannot decode the bytes raises `UnicodeDecodeError`. That
  IS a `ValueError`, so it lands in this function's own
  `except (ValueError, OSError)` recovery arm, whose job is to replace an
  unusable store with `{"jobs": []}`. A perfectly good backup is silently
  reduced to an empty schedule and reported as "unreadable".
- a code page that decodes most bytes (cp1252) fails the other way: the read
  succeeds with mojibake, and the sanitizer's pause rewrite persists that
  mojibake to disk, so the corruption survives into the live store.

Pin `encoding="utf-8"` on the one read and the three writes. The recovery arm
itself is unchanged: bytes that are not JSON at all are still replaced.

Covered by `TestTheCronSanitizerDecodesAsUtf8`, which drives the code page
through `io.text_encoding` — the funnel `read_text`/`write_text` consult when
no encoding is named. Patching the locale function alone would not reproduce
this under `PYTHONUTF8`, which is set for successors but not for the running
gateway process.

Co-authored-by: DeryFerd <DeryFerd@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

fork Pull request from a fork (external contributor)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants