Skip to content

fix(config): read the .env credential store as UTF-8, not the locale - #13316

Closed
DeryFerd wants to merge 1 commit into
kirodotdev:mainfrom
DeryFerd:fix/env-credential-store-utf8
Closed

DeryFerd wants to merge 1 commit into
kirodotdev:mainfrom
DeryFerd:fix/env-credential-store-utf8

Conversation

@DeryFerd

@DeryFerd DeryFerd commented Sep 24, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: the gateway reads its credential store the way every writer writes it, so a non-ASCII credential or note can no longer fail the boot or silently corrupt the value on a Windows host.

~/.kiro/crew/.env is the credential store: the dashboard's channel settings, the setup wizard, and the WeChat QR pairing flow all write tokens into it, and load_credentials() reads it back at gateway boot. Two of its readers decode the file with the host's active code page instead of UTF-8:

  • read_env_file_credential() reads with a bare read_text() (src/kiro_crew/config/loader.py:2147), inside a try that catches only OSError
  • the load_credentials() boot loop reads with a bare read_text() too (src/kiro_crew/config/loader.py:5515), with no exception guard at all

On a Windows host whose code page is cp936 (the zh-CN install this repository was audited on), a human-written comment in the file raises UnicodeDecodeError. In load_credentials() that exception has no guard around it, so the gateway fails to start. In read_env_file_credential() the exception escapes the recovery arm, because UnicodeDecodeError is a ValueError, not an OSError, and the arm catches only OSError. A code page that decodes most bytes (cp1252) fails the other way: the read succeeds and silently mojibakes whatever non-ASCII content the file carried.

The file invites exactly this content. The setup docs show operators creating it by hand (cat > ~/.kiro/crew/.env in docs/guides/slack-setup.md), and a credential file with a note next to the tokens is the normal case, not an exotic one.

Why it matters

Every other reader and writer of this store names UTF-8 explicitly: the setup wizard (cli_setup.py), the dashboard credential writer (_write_env_updates_locked in dashboard/handlers/messaging.py, which also stores content through atomic_write, documented UTF-8 for str), the WeChat QR handler (all three of its read sites), the secrets migrator (read_bytes() plus an explicit decode("utf-8", ...)), and the service warning reader (service/common.py). The store's contract is UTF-8 on disk; these two reads are the only stragglers, so the mismatch is live on a single host with no cross-host precondition. The gateway writes a token through the dashboard, the operator later adds a comment line in their own editor, and the next boot is down.

The quiet half is the one that worries me more. On cp936 the failure is loud (boot fails). On cp1252 most UTF-8 byte sequences decode fine, so the read "succeeds" and the corruption is silent: a non-ASCII credential value or operator setting comes back as mojibake and propagates into the child environment, and nothing anywhere says so. The per-key reader's own docstring promises that an unreadable file reads as unset rather than failing, and today a decode failure is precisely the unreadable case that promise does not cover.

Not a goal

  • Teaching the store an explicit encoding marker or migration path for legacy-saved files. A store written under an old code page can also be valid UTF-8 (a cp936 一 is the bytes D2 BB, a valid UTF-8 sequence), and no decoder ordering resolves that ambiguity — the fallback keeps such stores loading and the limit is stated below. Picking a marker format is a store-format decision for maintainers, and it can ship as its own change.
  • The other bare text reads in config/loader.py (.local_secret and the credential env used by read_env_file_credential's siblings). Same defect class, different files, and a separate tranche.
  • Changing what load_credentials() does with a store neither decoder can read: it still aborts the boot, exactly as it does on main.

What changed (motivation → approach → change)

Symptom: gateway boot fails, or credentials silently corrupt, when the operator's .env carries non-ASCII bytes on a non-UTF-8 code page. Root cause: the two reads named above let Path.read_text() resolve the encoding through the host locale, while the store's writers all pin UTF-8. Change: read the store ONCE, as bytes, and decode that single snapshot: encoding="utf-8" first, then — on a decode failure — the host's ANSI code page, named explicitly through locale.getencoding(), because a store saved under a legacy code page predates the UTF-8 contract and its writer used the locale. The single snapshot is load-bearing both ways: under the gateway's PYTHONUTF8=1 launch a bare-name fallback would answer utf-8 and fail the boot a second time, and a second read would decode a file a concurrent save may have replaced. The explicit name is load-bearing: the gateway launches with PYTHONUTF8=1, so a bare fallback would answer utf-8 and fail the boot a second time. The fallback is declared, not incidental: it keeps every store that loads on main today loading after this change, and it is why this PR does not trade the mojibake bug for a boot-crash bug. Both hunks ship it; the per-key reader answers unset only when neither decoder can read, per its documented contract.

Two things I deliberately did not change:

  • The reader keeps the same codec the writers already use, utf-8 and not utf-8-sig. The writers parse the store strictly, so a BOM-prefixed first key is invisible to them: if the readers stripped the BOM while the writers cannot, a credential the dashboard cleared would survive the clear. Making writers and readers agree on BOM handling is its own change, and a BOM-prefixed store keeps main's behavior here.
  • The other bare text reads in loader.py (.local_secret, the credential env used by read_env_file_credential's siblings) are out of scope here. The .env store is one file with one writer contract, so this PR closes that file's family; the same class has other instances on other files and I would rather land them as their own tranche than blur the writer-sweep argument this one is making.

The change follows the pattern this repository has already been applying to the same defect class: #7560 (decode the az CLI's output as UTF-8, not the code page), #10743 (pin UTF-8 on the MCP gateway rewriter's remaining JSON reads), #10706 (read the Windows command line as UTF-8), and #12372 (open the backend spawn log as UTF-8). This is the same fix for the same reason, one file later.

Backwards compatibility

Compatible: for any store the current writers can produce, the rendered result is identical or better. A pure-ASCII store (every token the dashboard and setup wizard write) decodes byte-identically on every host. A UTF-8 store with non-ASCII content decodes correctly instead of mojibaked on cp1252 and raises nothing on cp936. A store saved under a legacy code page (an ANSI/GBK editor save, a PowerShell 5.1 >> redirect) decodes exactly as it does on main today, through the fallback, so nothing that loads today fails to load after this change. A BOM-prefixed store from a hand editor behaves exactly as it does on main today (its first key reads as unset); making the writers see the BOM is its own change. One limit to state plainly: for a mixed store (UTF-8 values next to locale-encoded bytes, hand-crafted only — no writer produces one) the fallback reproduces main's decode, mojibake included, rather than introducing it. A store that neither decoder can read keeps its old behavior: unset from read_env_file_credential() (its documented recovery arm) and a boot abort from load_credentials().

Tests

New file test/test_env_credential_reads_utf8.py, six tests:

  • cp936: a UTF-8 .env with a non-ASCII comment (KIRO_API_KEY=abc123 next to # 说明:生产密钥) must read the token value back exactly. On the old code this fails with UnicodeDecodeError: 'gbk' codec can't decode byte 0xa5, the escape through the OSError-only arm.
  • cp1252: the same file under a code page that can decode most bytes must still return the value intact (pinning the silent-mojibake direction).
  • legacy store: the same tokens saved under cp936 must still read back when locale.getencoding() names cp936 (the decoder the fallback uses; decode() takes the codec by name, so the store's byte snapshot is what both decoders see). This is the fallback's no-regression pin.
  • undecodable store: bytes that neither UTF-8 nor the host code page can decode must read as unset, per the recovery arm's documented promise.
  • two source assertions pinning the shape of both reads: one read_bytes() snapshot per read, raw.decode("utf-8") first, raw.decode(locale.getencoding()) as the fallback, no read_text left in either function, and the UnicodeDecodeError arm present.

The tests drive the legacy direction through locale.getencoding(), the codec the fallback must name; decode() takes the codec by name, so the tests stay deterministic on any host. The substitute passes a named encoding through and answers only the bare call, which matters on 3.13 where a bare replacement would override explicitly-named encodings too. With no bare read left in either function the funnel patch now stands in for the launch configuration: it answers utf-8 while locale.getencoding() names the ANSI page the fallback should decode with. Each behavioral assertion was proven to bite by running it against the code it guards against before the fix: the cp936 test fails with the 'gbk' codec error above, the legacy-store test fails with '' != 'abc123' against the bare fallback under the simulated PYTHONUTF8 launch, and the undecodable-store test fails with a UnicodeDecodeError through the OSError-only arm.

Related runs: the new file plus test_credential_scrub_sync.py (whose TestReadEnvFileCredential class already exercises this helper) and test_config_writers_advisory_lock.py all pass together under the repo's xdist flags (37 passed). flake8 is clean on both changed files, mypy is clean on loader.py, the repository's black gate passes, and the comment-history gate reports no history narration on the added lines.

Manual verification

Beyond the tests I probed the failure directions by hand against the old and new code: under a forced cp936 funnel the old read raises the exact error production shows, and the fixed read returns the token; under cp1252 the value comes back intact either way, which is why the value-integrity assertion exists; and a cp936-saved legacy store reads the same value before and after, which is why the fallback exists. I also swept every reader and writer of the store to confirm the five UTF-8 pins named above, since the mismatch between those pins and these two reads is what makes the UTF-8 contract the intended one rather than my preference.

One honest limit: CI's Linux hosts default to UTF-8, so a purely behavioral test would pass there with or without the fix, which is the same reason the repository's own Windows text-I/O tests (apps/builtins/auto_improvement/tests/test_windows_text_io.py) pin this class as source assertions, and I followed that grammar here. The genuinely cp936-host check would be booting the gateway on a zh-CN Windows install with a commented .env; I could not run that combination, and the funnel forcing in the tests is the closest reproduction of it.

Related Issues

No dedicated issue covers these two reads. Same defect class as the merged #7560, #10743, #10706 and #12372, and the same file family as the open #12855, which fixes the imported cron store's decode.

Pattern harvest

Rule candidate: lint
Pattern: a text-mode Path.read_text()/write_text() without encoding= in runtime sources. The subprocess twin of this defect is already gated by scripts/check_subprocess_encoding.py with a shrink-only baseline; the Path equivalent does not exist, which is why instances of this class keep turning up one file at a time.

Checklist

  • One commit, Conventional Commits title (fix(config): ...)
  • Existing tests pass, new tests added for the changed behavior
  • Self-review completed; the change follows the store's own writer contract and the repo's existing UTF-8 pin pattern
  • Documentation: N/A — no doc describes the store's encoding (the guides show the KEY=VALUE format and cat heredocs only), and the user-visible format is unchanged
  • No secrets, credentials, or internal references in the diff

@DeryFerd
DeryFerd requested review from a team and krishdhasmana September 24, 2026 09:08
@github-actions github-actions Bot added fork Pull request from a fork (external contributor) readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Sep 24, 2026
@github-actions

github-actions Bot commented Sep 24, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Fable 5, fork) — ✅ PASS

Design-level review of 84f76087158b32e15f8f79d4098948706d9dd592 via the fork AI-review pipeline — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Verify on a real Windows host that locale.getencoding() returns the ANSI page under PYTHONUTF8=1 — the tests patch it, so that premise is never executed.

Suggestions

  • The bytes→UTF-8→locale decode dance is duplicated inline at both sites, and the description already promises a tranche for .local_secret and siblings; extracting one shared helper now (and re-pointing the source-shape test pins at it) keeps the policy in one place before a third copy lands.

[DESIGN-REVIEWED] 84f7608

Description read

[DESCRIPTION-READ] 9b014ceef1d71c1275f9a03561996206d8a790958925fddfff0e15c2c32d9f96

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Sep 24, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Fable 5, fork) — ✅ PASS

Premise-level review of 84f76087158b32e15f8f79d4098948706d9dd592 via the fork AI-review pipeline — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: PASS

Confirm on one real Windows host that locale.getencoding() names the ANSI page under the gateway's PYTHONUTF8=1 launch — the fallback's one platform premise.

(Python ≥3.12 floor covers the 3.11+ API; boot cannot regress even if wrong — the fallback then raises exactly where main already does.)

What this change ships

Inventory (5 items) — 5 justified

Intent: a FIX — the two remaining locale-decoding reads of the .env credential store are pinned to the UTF-8 contract every writer already uses, so non-ASCII content no longer crashes boot or mojibakes.

  1. Gateway boot survives a UTF-8 .env with non-ASCII content on a non-UTF-8 code page — justified
  2. Per-key credential read returns the value instead of a UnicodeDecodeError escaping its OSError-only guard — justified
  3. Legacy locale-encoded stores keep loading, via a declared ANSI-page fallback on the same byte snapshot in both readers — justified
  4. A store neither decoder reads now answers unset from the per-key reader, honoring its own docstring — justified
  5. Six new tests: behavioral coverage that fails on base, plus source-shape pins on both read sites — justified

Verified against base: both bare read_text() sites exist as described (loader.py:536, loader.py:4860); writers are UTF-8-pinned (messaging.py:4606, cli_setup.py:643/756, service/common.py:337); PYTHONUTF8=1 set in platform_compat.py:55. The one in-file sibling (.local_secret, loader.py:918) is declared out of scope in ## Not a goal, so the tranche boundary is stated, not silent.

[FIRST-PRINCIPLES-REVIEWED] 84f7608

Description read

[DESCRIPTION-READ] a376acf4ad5128acd24af3ce5a0df4f02d37c0f8545f12aecb7f64c7575d46fb

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Sep 24, 2026 •

Copy link
Copy Markdown
Contributor

GPT 5.6 Review (fork) — 🔴 changes requested (blocking)

Reviewed 84f76087158b32e15f8f79d4098948706d9dd592 via the fork AI-review pipeline; updated in place on each push.

1 of 1 blocking finding(s) are security-class and were withheld from adjudication, so the blocking verdict stands.

BLOCKING -- src/kiro_crew/config/loader.py:540 -- UTF-8-first decoding corrupts ambiguous legacy stores
text = raw.decode("utf-8")
cp936 .env value KEY=一 -> load_credentials() decodes D2 BB as UTF-8 һ -> corrupted value propagates to child processes.
Anchor: residual/crash-data-loss-corruption
Fix: Add encoding metadata or migration before preferring UTF-8 for legacy stores.
[BLOCK-MERGE] 84f7608
[GPT-REVIEWED] 84f7608

Adjudication (Opus 5) — is blocking on each finding proportionate?

F1 — harm rung: silent corruption of a credential value in memory (fence's unbounded class), not exposure or on-disk loss; the store is never rewritten by either read path.

Conditions confirmed in this run: the store must be non-UTF-8-encoded, which no writer in the tree can produce — cli_setup.py:643 reads UTF-8 and rewrites via atomic_write (str → UTF-8, atomic_write.py:899), weixin_qr.py:124/147/162 read UTF-8 and write through _atomic_write (weixin_qr.py:135), and secrets/migrate.py:224 already decodes the same store UTF-8-first today; the decode at the new raw.decode("utf-8") is WHOLE-FILE, so any byte in the snapshot that is invalid UTF-8 routes the entire snapshot to the correct locale.getencoding() decode (patch line 37) — the mojibake needs a legacy file that is simultaneously wholly valid UTF-8, i.e. at most a character or two of non-ASCII each landing in the narrow cp936∩UTF-8 overlap (一 = D2 BB does; 工 = B9 A4 does not, and falls back correctly); the non-ASCII must sit in a VALUE, and CREDENTIAL_KEYS (loader.py:463-483) are tokens, IDs and PATs; and on the shipped desktop/installer launch PYTHONUTF8=1 is pinned at interpreter start (website/electron/gateway-env.js:19, scripts/smoke-windows-install.ps1:470), where the base ep.read_text() (loader.py:536, loader.py:4860) already resolves to utf-8 and the diff changes nothing — the delta exists only on a bare CLI launch under a legacy ANSI page. Recovery: none automatic (the value reaches os.environ at loader.py:4912 and children, and the remote auth fails visibly); an operator re-save through any UTF-8 writer fixes the store permanently, and such a store is in any case mojibaked-and-rewritten on main the first time weixin_qr.py:135 touches it.

Real fix cost: encoding metadata or a migration for the store — a new on-disk provenance field plus a migration path every reader and writer of .env must agree on, and a permanent ordering contract, to serve an artifact no current writer produces.

[ADJUDICATION] 84f7608 total=0 uphold=0 downgrade=0
[GPT-ADJUDICATED] 84f7608
[ADJUDICATION-FENCED] 84f7608 fenced=1 flagged=1
FLAG F1 src/kiro_crew/config/loader.py:540 -- Real, but it needs a legacy-code-page .env no writer in this tree can produce (all name UTF-8: cli_setup.py:643, weixin_qr.py:124, atomic_write.py:899) whose ENTIRE byte content is coincidentally valid UTF-8 so the whole-snapshot locale fallback never engages, with the non-ASCII inside a credential value that is a token or ID by the service's own format, on a launch that is not the PYTHONUTF8-pinned desktop/installer one where the base read already decoded utf-8 identically.
[GPT-ADJUDICATED-FENCED] 84f7608

🏷️ Fenced finding(s) machine-flagged as likely edge case

The security fence keeps these findings blocking regardless of adjudication; the only clearance path is a human override recorded by a repository writer, who must independently verify a rationale before recording it — it is machine-authored, and a wrong override on a security-class finding ships exactly the class the fence exists to stop. (This lane's comment deliberately carries no override command.)

  • F1 src/kiro_crew/config/loader.py:540 — Real, but it needs a legacy-code-page .env no writer in this tree can produce (all name UTF-8: cli_setup.py:643, weixin_qr.py:124, atomic_write.py:899) whose ENTIRE byte content is coincidentally valid UTF-8 so the whole-snapshot locale fallback never engages, with the non-ASCII inside a credential value that is a token or ID by the service's own format, on a launch that is not the PYTHONUTF8-pinned desktop/installer one where the base read already decoded utf-8 identically.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Sep 24, 2026
@github-actions

github-actions Bot commented Sep 24, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5 Review (fork) — ✅ no blocking findings

Reviewed 84f76087158b32e15f8f79d4098948706d9dd592 via the fork AI-review pipeline; updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 84f7608

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 24, 2026
@DeryFerd
DeryFerd force-pushed the fix/env-credential-store-utf8 branch from 161d97f to 6363157 Compare September 24, 2026 10:41
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 24, 2026
@DeryFerd
DeryFerd force-pushed the fix/env-credential-store-utf8 branch from 6363157 to a5bee1a Compare September 24, 2026 11:20
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 24, 2026
@DeryFerd
DeryFerd force-pushed the fix/env-credential-store-utf8 branch from a5bee1a to 6b165fd Compare September 24, 2026 12:18
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 24, 2026
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Sep 25, 2026
@DeryFerd
DeryFerd force-pushed the fix/env-credential-store-utf8 branch from fd73f3d to c5b28c0 Compare September 25, 2026 17:17
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 25, 2026
@DeryFerd
DeryFerd force-pushed the fix/env-credential-store-utf8 branch from c5b28c0 to ef91b80 Compare September 25, 2026 18:08
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 25, 2026
@DeryFerd
DeryFerd force-pushed the fix/env-credential-store-utf8 branch from ef91b80 to 79ae70e Compare September 27, 2026 06:46
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 27, 2026
@DeryFerd
DeryFerd force-pushed the fix/env-credential-store-utf8 branch from 79ae70e to ee39b9a Compare September 27, 2026 07:48
@DeryFerd
DeryFerd requested a review from a team as a code owner September 27, 2026 07:48
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 27, 2026
read_env_file_credential and load_credentials read ~/.kiro/crew/.env with
a bare read_text(), which decodes with the active code page. Every other
reader and writer of the store names UTF-8 explicitly: the setup wizard,
the dashboard credential writer, the WeChat QR handler, the secrets
migrator, and the service warning reader. On a cp936 host a UTF-8
comment in the operator's .env raises UnicodeDecodeError out of the
per-key reader's recovery arm (it is a ValueError, not an OSError) and
out of a boot loop with no guard at all, while a code page that decodes
most bytes (cp1252) silently mojibakes the file instead.

Read the store ONCE, as bytes, and decode that snapshot: UTF-8 first,
then — on a decode failure — the host's ANSI code page, named
explicitly through locale.getencoding(), the way a legacy writer encoded
it. The single snapshot matters twice: under the gateway's PYTHONUTF8
launch a bare fallback would answer utf-8 and fail the boot a second
time, and a second read would decode a file a concurrent save may have
replaced. Boot keeps working for every store that worked before, UTF-8
stores stop mojibaking on cp936/cp1252 hosts, and only a store that
neither decoder can read keeps its old behavior: unset from the per-key
reader, a boot failure from load_credentials. A BOM-prefixed store keeps
today's behavior too: the writers do not see a BOM, so the readers must
not silently strip one either.

Covered by test/test_env_credential_reads_utf8.py: the legacy direction
names its ANSI page through locale.getencoding() (decode takes the codec
by name, so no text funnel is involved), and source assertions pin one
byte snapshot per read with both decodes on it.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

fork Pull request from a fork (external contributor)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant