Skip to content

fix(file-explorer): decode git and ripgrep output as utf-8, not the host codepage - #7544

Closed
leonlaiyc wants to merge 1 commit into
kirodotdev:mainfrom
leonlaiyc:fix/file-explorer-git-status-utf8
Closed

leonlaiyc wants to merge 1 commit into
kirodotdev:mainfrom
leonlaiyc:fix/file-explorer-git-status-utf8

Conversation

@leonlaiyc

@leonlaiyc leonlaiyc commented Sep 1, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

file_explorer's _git_status runs

["git", "-C", str(repo_root), "status", "--porcelain=1", "-z"]

with text=True and no encoding=, so git's bytes are decoded with
locale.getpreferredencoding(False).

-z is what makes that unsafe, and it is worth being exact about. Measured against
real git on a repo containing one untracked café-日本.txt:

$ git status --porcelain=1 | od -c
?? " c a f \ 3 0 3 \ 2 5 1 - \ 3 4 6 \ 2 2 7 ... "     <- C-quoted, pure ASCII

$ git status --porcelain=1 -z | od -c
?? c a f 303 251 - 346 227 245 346 234 254 . t x t \0  <- verbatim, raw UTF-8

The default form is ASCII whatever the filename is. -z deliberately turns that
off. So on any host whose preferred encoding is not UTF-8 — which is every Windows
console codepage — the handler decodes UTF-8 bytes with cp950 / cp932 / cp1252.

The same call is made for rev-parse --abbrev-ref HEAD, and git refnames are UTF-8
too.

Why it matters

One file with a non-ASCII name takes out the whole repository's status.
UnicodeDecodeError is neither an OSError nor a subprocess.TimeoutExpired, so
it escapes both except clauses in _git_status and the endpoint 500s — there is
no status_error fallback for it. Measured on a live cp950 host:

UnicodeDecodeError: 'cp950' codec can't decode byte 0xe6 in position 9

Where the codepage is lenient the failure is quieter and worse. cp1252 maps
every byte to something, so nothing raises: the path comes back mojibaked
(café-日本.txt → café-日本.txt), the endpoint returns 200, and that key
simply never matches its tree entry. The file silently shows no git status at all.

This is not a hypothetical class either — the repository ships a CI gate for it,
scripts/check_subprocess_encoding.py, with file_explorer/server.py already on
its baseline at 3 calls.

What changed (motivation → approach → change)

Symptom: non-ASCII paths crash or corrupt git status. Root cause: git's UTF-8 output
decoded with the host codepage. Fix: name the encoding, on both git calls in
_git_status.

 branch = run_limited(
     cmd_branch,
     capture_output=True,
     text=True,
+    encoding="utf-8",
     timeout=GIT_TIMEOUT_SEC,
     check=False,
 )

run_limited forwards every keyword to subprocess.run untouched, so this needs no
new plumbing. It is also not a new convention: platform/update_capability.py
already pins encoding="utf-8" on its own git call, with a comment naming non-ASCII
paths as the reason.

errors= is deliberately not added. errors="replace" would convert the crash
into the silent-mojibake case, which the tests below treat as equally wrong; UTF-8 is
what git actually writes, so the right move is to say so.

The subprocess-encoding baseline goes from 3 to 0 for this file, the way the
gate asks (--update-baseline), so its row is pruned rather than lowered.

The third spawn in this file is the same defect, and is fixed here too.
_search_rg runs ripgrep with --json, a stream that is UTF-8 by definition and
that carries both the matched path and the matched line — so non-ASCII is the norm
there, not the exception. Its except clause is also OSError / TimeoutExpired
only, which means a decode failure does not degrade to the _search_python fallback
the way a spawn failure does: it 500s the search.

Contention was checked before folding it in rather than assumed. Open PR #4650
rewrites _search (its server.py hunk is anchored there and adds ~170 lines) and
leaves def _search_rg as unchanged context, so the one line changed here is not a
line that PR touches.

With all three pinned, this file leaves .github/subprocess-encoding-baseline.txt
entirely — the diff to that file is the removal of its row, not a lowered count.

Tests

New file test/test_file_explorer_git_status_encoding.py — deliberately a new file
rather than an addition to test_file_explorer_app.py, which #4650 also edits.

The harness feeds _git_status git's real -z bytes through a run_limited
stand-in that decodes the way subprocess documents: the encoding= keyword when
one is given, locale.getpreferredencoding(False) otherwise. That makes the
interesting codepage a test parameter instead of a property of the runner, so these
are deterministic on a UTF-8 Linux CI box and on a cp950 laptop alike — no console
codepage to switch, no timing, no skip.

Test Locks in
..._a_non_ascii_path_survives_a_non_utf8_host_codepage[cp950] / [ascii] the raising case from both directions — a real shipped Windows DBCS codepage, and the degenerate POSIX LC_ALL=C one. Also asserts no status_error/branch_error was set and that both git calls ran, so a regression cannot pass as an empty-but-successful result
..._a_locale_that_silently_mojibakes_is_also_refused cp1252, which never raises. This is the case errors="replace" would leave in place, and asserting on the path itself catches both failure modes with one assertion
..._an_ascii_path_is_unaffected over-fix guard: the ordinary path is unchanged. Passes on both sides, which is the point
..._a_ripgrep_match_survives_a_non_utf8_host_codepage the search spawn, driven with ripgrep's own --json record. Asserts the decoded path and the matched line survive exactly, so a lenient codepage's mojibake fails it as loudly as a raising one

An autouse fixture stubs wrap_argv and cgroup_scope_argv to identity. Neither
touches decoding; wrap_argv refuses outright on a host with no OS sandbox backend
(every Windows runner), so stubbing them keeps the test measuring one thing and keeps
it portable. The same idiom is already used in test_file_explorer_app.py.

Red-before, with server.py restored to current origin/main (7308a8b30) and
nothing else changed — 3 failed, 1 passed, and the three failures are the three
distinct mechanisms:

UnicodeDecodeError: 'cp950' codec can't decode byte 0xe6 in position 9
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 6
AssertionError: assert {'café-日本.txt': '??'} == {'café-日本.txt': '??'}

The one that passed is the ASCII over-fix guard, which must pass on both sides.

The ripgrep case has its own red-before against the same tree:

FAILED ...::test_a_ripgrep_match_survives_a_non_utf8_host_codepage
  UnicodeDecodeError: 'cp950' codec can't decode byte 0xe6 in position 152

Green after: the new file plus test/test_file_explorer_app.py — 100 passed,
2 skipped
.

Gates on the touched files: scripts/check_subprocess_encoding.py passed
(239 known calls in 117 files remain, down from 242 in 118), scripts/check_black_formatting.py
passed in scope, flake8 and isort --check-only clean, git diff --check
clean, and mypy src/kiro_crew/apps/builtins/file_explorer/server.py — Success: no
issues found in 1 source file
. No repository-wide suite; server CI owns that.

Manual verification

N/A — unit coverage sufficient: the defect is a decoding choice, and the tests drive
the real _git_status with the exact bytes git emits under -z, which is closer to
the failure than opening the panel on one machine would be.

Related Issues

None. Found by auditing spawns that ask a tool for raw bytes and then decode with the
host locale, against scripts/check_subprocess_encoding.py's own baseline.

Pattern harvest

Rule candidate: review-prompt
Pattern: a flag that turns OFF a tool's ASCII-escaping (-z, --null,
core.quotePath=false, --porcelain variants) added to a call that decodes with the
host locale.

The generalizable question is narrower and more useful than "does this call pass
encoding=": did someone ask the tool for raw bytes, and is anything downstream
still assuming ASCII?
-z and core.quotePath=false exist precisely to remove the
escaping that was making the locale codec harmless, so adding one silently converts a
latent issue into a live one — and the failure lands only on hosts whose codepage is
not UTF-8, which is nobody on the CI matrix.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

Contribution License Agreement

@leonlaiyc
leonlaiyc requested review from a team and dwu96 September 1, 2026 07:05
@github-actions github-actions Bot added fork Pull request from a fork (external contributor) readiness: action required A blocking check or review needs attention labels Sep 1, 2026
…ost codepage

`_git_status` runs `git status --porcelain=1 -z`. The `-z` form is the one
porcelain shape that does NOT C-quote a path: verified against real git, the
default form returns `?? "caf\303\251-\346\227\245....txt"` (pure ASCII whatever
the filename is) while `-z` returns the same path verbatim as raw UTF-8 bytes.

Both git calls there passed `text=True` with no `encoding=`, so those bytes were
decoded with `locale.getpreferredencoding(False)`. On a Windows console codepage
that is cp950 / cp932 / cp1252, not UTF-8, and one file with a non-ASCII name
took out the whole repository's status: `UnicodeDecodeError` is neither
`OSError` nor `TimeoutExpired`, so it escaped both `except` clauses in
`_git_status` and the endpoint 500ed. Measured on a live cp950 host -- `'cp950'
codec can't decode byte 0xe6`. Where the codepage happens to be lenient the
failure is quieter and worse: cp1252 maps every byte to something, so the path
is silently mojibaked and never matches its tree entry.

`_search_rg` has the same defect with a different producer: ripgrep's `--json`
stream is UTF-8 by definition and carries both the matched path and the matched
line, so non-ASCII is the norm there rather than the exception. Its `except`
clause is also `OSError` / `TimeoutExpired` only, so a decode failure does not
degrade to the `_search_python` fallback the way a spawn failure does -- it
500s the search.

Name the encoding on all three spawns, which is what `platform/update_capability.py`
already does on its own git call for this reason. Refnames are UTF-8 too, so the
branch call gets the same treatment. That takes this file off the
subprocess-encoding baseline entirely (3 -> 0).

The regressions drive the real functions with git's `-z` bytes and ripgrep's own
JSON through a `run_limited` stand-in that decodes the way `subprocess`
documents -- the `encoding=` keyword when given, the locale codec otherwise --
so the interesting codepage is a parameter rather than a property of the runner.
cp950 and ascii cover the raising case from both directions, cp1252 covers the
silent-mojibake case (which is why `errors=` would not be a fix), and an ASCII
path pins that the ordinary case is unchanged.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@leonlaiyc
leonlaiyc force-pushed the fix/file-explorer-git-status-utf8 branch from d74b265 to 7332bed Compare September 1, 2026 07:31
@leonlaiyc leonlaiyc changed the title fix(file-explorer): decode git status output as utf-8, not the host codepage fix(file-explorer): decode git and ripgrep output as utf-8, not the host codepage Sep 1, 2026
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 1, 2026
@bolichen97

Copy link
Copy Markdown
Contributor

Open PR relationship audit

This is a consolidated, point-in-time code-level audit note. It compares complete merge-base diffs and current/merged code; it does not treat a shared topic as duplication or partial coverage as completion.

Relationship findings

  • This PR is OVERLAPPING with PR #4650. The goals differ or the implementations can complement each other; this is not a duplicate claim. Recommended action for PR #7544: KEEP. Complementary, not competing: PR #4650 does not pin any encoding and leaves the three defective spawns as-is, so landing it would not fix this defect and would leave the file's baseline row intact. Only cross-reference recorded on PR #7544's timeline. Files: src/kiro_crew/apps/builtins/file_explorer/server.py.
  • This PR is OVERLAPPING with PR #7560. The goals differ or the implementations can complement each other; this is not a duplicate claim. Recommended action for PR #7544: KEEP. Two independent instances of the same class in different modules; both should land. Sequence them and let the second rebase the one-row baseline hunk. Files: .github/subprocess-encoding-baseline.txt.

No PR, Issue, label, branch, or review state was changed by the relationship-note portion of this audit.

@github-actions github-actions Bot added the merge conflict Branch has merge conflicts with its base — author must resolve before merge label Sep 9, 2026
@NicholasRBowers NicholasRBowers added needs-pr-triage PR scanner: awaiting automated triage needs-author-decision PR blocked on author input and removed needs-pr-triage PR scanner: awaiting automated triage labels Sep 12, 2026
@NicholasRBowers

Copy link
Copy Markdown
Contributor

🤖 Kiro Crew [operator: NicholasRBowers#a942f9ca]: This PR has been inactive for 7+ days. I reviewed the blockers but they require your input:

  • main has already landed this fix. fix(apps): declare every builtin's platform truthfully, and fix the Windows defects #9413 (merged 2026-09-09) pinned encoding="utf-8", errors="replace" on the git and ripgrep subprocess.run calls in src/kiro_crew/apps/builtins/file_explorer/server.py and pruned file_explorer/server.py from .github/subprocess-encoding-baseline.txt. Both of this PR's production hunks now conflict with lines that already carry the pin, and its baseline hunk deletes a line that no longer exists. What remains unique is test/test_file_explorer_git_status_encoding.py. Your call: (a) close this PR as superseded, or (b) reduce it to the regression test alone, re-based onto main, if it still adds coverage that main's change did not bring.
  • If you keep it, drop the .github/ hunk in the rebase: Fork workflow-change guard blocks any fork PR that touches .github/** until a maintainer applies allow-fork-workflow-change, and with the baseline already pruned on main there is nothing left for this PR to prune.
  • CI reds (Backend Tests shards 3/4 on 3.10/3.12/Windows, Coverage Gate) date from a 2026-09-01 base and were not re-evaluated on a rebase; they are moot until scope is settled. No AI review lane has stamped this head.

When you've addressed these, the pipeline will re-assess on its next cycle.

@iamwhatever

Copy link
Copy Markdown
Contributor

Closing as superseded.

main already decodes both sites as UTF-8: encoding="utf-8", errors="replace" is on the _git_status git calls in the file-explorer backend, landed by #9413 ("fix(apps): declare every builtin's platform truthfully, and fix the Windows defects", merged 2026-09-09). The host-codepage mojibake this PR fixes is gone.

The branch has been in merge conflict and idle 8+ days, and the only residual delta is the regression test plus a .github/subprocess-encoding-baseline.txt row that main has since pruned on its own.

If you want to keep the coverage, refile just the regression test rebased onto main, with no baseline edit. Sibling fixes of yours in this family are still live and worth rebasing -- #7560 (az output) is one.

@github-actions github-actions Bot removed the readiness: action required A blocking check or review needs attention label Sep 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

fork Pull request from a fork (external contributor) merge conflict Branch has merge conflicts with its base — author must resolve before merge needs-author-decision PR blocked on author input

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants