Skip to content

Use UTF-8 for JSON command-line file I/O - #238

Merged
stefankoegl merged 1 commit into
stefankoegl:masterfrom
KennyMcSimpson:fix/cli-utf8-json-files
Oct 24, 2026
Merged

stefankoegl merged 1 commit into
stefankoegl:masterfrom
KennyMcSimpson:fix/cli-utf8-json-files

Conversation

@KennyMcSimpson

Copy link
Copy Markdown
Contributor

On non-UTF-8 locales, jsondiff and jsonpatch read UTF-8 JSON files using
the system encoding, which can silently change Unicode values. In-place
jsonpatch -i -u can also raise UnicodeEncodeError when writing characters
outside that encoding, even if the input files contain only ASCII escapes.

Explicitly use UTF-8 for JSON file arguments and both in-place output paths,
and document the file encoding. Standard input and standard output continue
to use their existing stream encodings.

The regression tests disable Python's UTF-8 mode and locale coercion. They
cover both programs' UTF-8 inputs and Unicode in-place output through the
atomic and fallback paths. All four cases fail against the original source
on a GBK Windows system.

Validation: Python 3.10 and 3.13 full unit suites each pass 217 tests with one
POSIX-only skip; all 20 Python 3.13 property tests pass. Critical Flake8
checks and git diff --check pass.

AI assistance: OpenAI Codex prepared the change, regression tests, and this description.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

The focused changes cover all intended file I/O paths with targeted regression tests and no identified blocking issues.

No open issues 🎉

What changed in this PR

Makes JSON command-line file I/O consistently use UTF-8, preventing locale-dependent Unicode corruption while preserving existing standard input and output encodings.

Changes:

  • Explicitly uses UTF-8 for file arguments and both in-place output paths.
  • Adds regression tests for UTF-8 inputs and Unicode output with Python’s UTF-8 mode disabled.
  • Documents the file-encoding behavior.
File Description
tests.py Adds encoding regression coverage and updates subprocess and output-mocking helpers.
jsonpatch_cli.py Sets UTF-8 explicitly for JSON file reads and in-place writes.
doc/​commandline.rst Documents UTF-8 file input and in-place output.

Review effort: Balanced

@coveralls

Copy link
Copy Markdown

Coverage Status

coverage: 100.0%. remained the same — KennyMcSimpson:fix/cli-utf8-json-files into stefankoegl:master

@stefankoegl
stefankoegl merged commit f63a4f3 into stefankoegl:master Oct 24, 2026
12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants