Repository navigation
fix(config): read a byte-order-marked .env the same way everywhere - #15596
Conversation
43ffa46 to
1ecb184
Compare
|
Comment History Gate on 43ffa46 flagged the issue number in the new test module's docstring ( |
First Principles Review (Opus 5.5, fork) — ✅ PASSPremise-level review of First-Principles-Verdict: PASS Check that BOM-less What this change shipsInventory (8 items) — 8 justifiedIntent: a Windows user's hand-edited
[FIRST-PRINCIPLES-REVIEWED] 068f74f Description read[DESCRIPTION-READ] b50c3994eb2b233be3839d2ab17e582b6fe686ddfbbb0d3cd5d2f53baa94c2cb That sha256 names the PR description this verdict read. Recompute it with |
GPT 6.1 Review (fork) — ✅ no blocking findingsReviewed Review detailsNo findings. |
Design Review (Opus 5.5, fork) — ✅ PASSDesign-level review of Design-Verdict: PASS Check that Suggestions
[DESIGN-REVIEWED] 068f74f Description read[DESCRIPTION-READ] c8faa66641fa9ecd6061c2d84c51e6957bc6bbd542c63e43ab3a74ca094469d8 That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
Opus 5.5 Review (fork) — ✅ no blocking findingsReviewed Review detailsNo findings that block the merge. FINDING — src/kiro_crew/dashboard/handlers/messaging.py:6449 — Discord and Telegram commit config before writing FINDING — src/kiro_crew/service/common.py:533 — For a UTF-16/UTF-32 [OPUS-REVIEWED] 068f74f |
1ecb184 to
28f309b
Compare
|
New head 28f309b, rebased onto main 321fd99, with one added commit for the two advisory findings on 1ecb184. GPT 5.6 Review, Opus 5 Review, Backend Tests (Windows) (2) on 1ecb184: not caused by this diff. Gates on 28f309b: |
28f309b to
ea90c71
Compare
|
New head ea90c71, rebased onto main ecdc59a. Two commits: the original change and one follow-up for the review findings. GPT 5.6 Review on 28f309b, The earlier GPT finding (BOM-less locale decode errors) and the Opus test-isolation finding, both reviewed on 1ecb184, stay fixed as described in my previous comment and are carried into the same follow-up commit. Backend Tests (Windows) (2): the two Backend Tests (Windows) (5) on 28f309b: three Opus 5 Review on 28f309b "review incomplete": that run's validation step failed with "API rate limit exceeded for installation". I pushed this head after later runs of the lane had succeeded. Gates on ea90c71: |
ea90c71 to
fe84901
Compare
|
New head 5cdefda, rebased onto main 9a59174. Still two commits; GPT 5.6 Review on a393be6, BLOCKING, Design Review on a393be6, suggestion to pin "every Opus 5 Review on a393be6, "review incomplete": the lane did not finish, so there was no finding to answer. This head re-runs it. Backend Tests (Windows) (4) on a393be6, macOS Tests / Backend Tests (macOS) (1) on a393be6, Run on this head: |
|
On 5cdefda every check passed and every review lane is PASS or no findings. Two advisory notes come back from earlier heads: First Principles Review, confirm on Windows that a BOM-less decode is unchanged under UTF-8 mode and legacy code pages: Design Review, check the degraded boot once by hand (a UTF-16 |
|
Head
Against origin/main, 61 of the 72 tests in |
|
Head
|
|
Head
|
|
Head
|
|
|
Head 26e3584: for Discord and Telegram, which keep their config write when the |
|
A .env saved by PowerShell 5.1 carries a byte-order mark. With a UTF-8 BOM the first key parsed as "\ufeffKEY", so that credential was never loaded and `secrets import` migrated it as garbage. With a UTF-16 BOM the strict decode raised UnicodeDecodeError, which escaped the OSError guard in read_env_file_credential and was unguarded in load_credentials, so the gateway could not start. Every reader and in-place rewriter of the data home's .env now decodes through one helper, read_env_text / decode_env_bytes in config/loader: a UTF-8 BOM is dropped and the rest read as UTF-8, a UTF-16 BOM raises, and a file with no BOM decodes exactly as before. The readers treat an undecodable file as empty and log one warning naming it; the rewriters let the error propagate so they never overwrite a file they could not parse. Routing the dashboard, WeChat and setup rewriters through the same helper keeps a clear able to match the key the gateway loads. Closes kirodotdev#15553
|
…ix it
The BOM-aware readers caught UnicodeDecodeError, so a BOM-less .env that
does not decode in the locale was silently loaded as empty instead of
raising as before. decode_env_bytes now raises a dedicated
EnvFileWideEncodingError subclass for a UTF-16 or UTF-32 byte-order mark,
and the credential readers catch only that. The UTF-32LE mark, which
begins with the UTF-16LE one, is matched first. Before this a UTF-32 .env
fell through to the locale decode: under a strict locale the plain
UnicodeDecodeError escaped load_credentials and read_env_file_credential
and stopped the gateway, and under cp1252 it loaded NUL-riddled keys.
The error names the detected encoding ("the .env is saved as UTF-32;
re-save it as UTF-8") instead of the base class's "'utf-8' codec can't
decode" text, which the WeChat sign-in surfaced as its error detail. It
carries only the BOM bytes in .object, so its repr and a traceback's
locals never hold the file.
A dashboard channel save against a wide-encoded .env answered with an
opaque HTTP 500. _write_env_off_loop, which every channel save goes
through, now turns the refusal into a 409 whose JSON error names the
file and encoding, asks for a UTF-8 re-save and says the .env was not
changed. The Slack, Teams, Webex, WeCom and Feishu saves still roll their
config write back on that failure; Discord and Telegram commit config
before the .env write and keep it, so their 409 also says the other
settings were saved, and a retry after the UTF-8 re-save completes the
save.
For a UTF-16 or UTF-32 .env the service-install and doctor check for a
dropped KIRO_API_KEY treats the file as defining nothing, so it prints
its one existing warning instead of raising. The gateway's own decode
warning names the file and asks for a UTF-8 re-save, so the install
warning keeps a single remedy path.
`kirocrew setup --slack` on a UTF-16 or UTF-32 .env prints how to re-save
it and leaves the file untouched instead of raising a traceback.
Every successful decode through loader.read_env_file re-arms
warn_undecodable_env for that path, so a .env that is fixed and then
broken again in the same process warns again. The secrets importer now
reads through read_env_file too (one read gives it both the bytes it
restores and the text it parses), so a successful import re-arms the
warning as well.
The service warning's own name scan treats any file it cannot decode as
defining nothing, so the warning still fires for a BOM-less file in a
legacy code page instead of raising into a handler that swallows it.
Every in-place rewriter (dashboard channel save, the WeChat helpers,
`setup --slack`) puts a UTF-8 BOM back on the text it writes, as the
importer does: the BOM is what makes the loader decode the file as UTF-8,
so dropping it on a save would hand a non-ASCII value to the locale codec
at the next boot.
The reader scan also flags a read-mode open() and a path built as
`<dir> / ".env"`, not only read_text/read_bytes on env_path().
Tests pin every byte order explicitly, and the credential and migrate
test helpers scrub every recognised credential key so an exported one
cannot change the result.
|
Head
|
Problem / Motivation
Goal: a
.envsaved by default Windows tooling loads every credential, or fails with one clear warning, instead of silently dropping the first key or keeping the gateway from starting.PowerShell 5.1 writes a byte-order mark.
Out-File -Encoding utf8writes UTF-8 with a BOM, and the default>/Out-Filewrites UTF-16LE with a BOM. With a UTF-8 BOM the first line of~/.kiro/crew/.envparses as\ufeffJIRA_API_TOKEN, so that credential is never loaded andkirocrew secrets importmigrates it as garbage. With a UTF-16 or UTF-32 BOM the decode raisesUnicodeDecodeError(or, under a single-byte code page such as cp1252, silently yields NUL-riddled keys), which escapes theexcept OSErrorinread_env_file_credentialand is unguarded inload_credentials, so gateway construction fails.Why it matters
The secrets guide tells users to edit
.envby hand. A Windows user who does that with the default shell or editor loses their first credential with only a "not a recognised credential" warning quoting a key they did not type, or gets a gateway that does not start.Not a goal
.envfiles. They are treated as unset with a warning; supporting a wide encoding for a plaintext credential file is more surface than the report asks for..envwith no BOM is decoded. The readers keep their current encoding (the locale default inconfig/loader, UTF-8 in the rewriters); the legacy-code-page question from fix(config): read the .env credential store as UTF-8, not the locale #13316 is untouched.SKILL.md/prompt.md(skills: a UTF-16 (BOM) global SKILL.md aborts the whole skill index, failing every chat turn and returning 500 from /api/skills #15490, context: a UTF-16 user prompt override (~/.kiro/crew/prompt.md) escapesexcept OSErrorand fails every default-agent chat turn on Windows #15491).What changed (motivation → approach → change)
.env, gateway load\ufeffKEY, dropped.env, gateway loadUnicodeDecodeError, startup fails (cp1252: NUL-riddled keys).env.envwas not changedkirocrew secrets importon a BOM fileThe root cause is that every
.envreader decoded bytes with no BOM handling. Fixing only the readers is not enough: the dashboard, WeChat and Slack-setup rewriters parse the same file and match keys exactly, so a reader that stripped the BOM while the rewriters did not would load a key that a dashboard clear could never remove.So there is one decoder in
config/loader.py,decode_env_bytes, and every reader and rewriter of the data home's.envdecodes through it, by way ofread_env_file(one read returning the raw bytes and the decoded text, whichsecrets/migrate.pyuses because it restores the bytes on its rewrite) or itsread_env_textwrapper. A UTF-8 BOM means UTF-8: it is dropped and the rest decoded as UTF-8. A UTF-16 or UTF-32 BOM, in either byte order, raisesEnvFileWideEncodingError, aUnicodeDecodeErrorsubclass whoseencodingis the codec the mark declares and whose message names the encoding ("the .env is saved as UTF-32; re-save it as UTF-8") rather than the base class's "'utf-8' codec can't decode" text. The UTF-32LE mark begins with the UTF-16LE one, so the BOM table matches it first. No BOM decodes exactly as before. The readers (load_credentials,read_env_file_credential,migrate_env_secretsinsecrets/migrate.py) catch onlyEnvFileWideEncodingError, treat a UTF-16 or UTF-32 file as empty and log one warning naming it throughwarn_undecodable_env._names_defined_in_env_fileinservice/common.pytreats any file it cannot decode as defining nothing, so the dropped-KIRO_API_KEYwarning still fires for a BOM-less file in a legacy code page instead of being swallowed with the exception. A later successfulread_env_fileof that path (any reader, the importer included) re-arms the warning, so a file that is fixed and then broken again in the same gateway process warns again. The warning is deduplicated per path because the gateway re-reads.envfrom many call sites for its whole life; without the re-arm that dedup would mute the path for the rest of the process, so a file broken again after a fix would read every credential as unset with no warning. For a UTF-16 or UTF-32.env, the service-install and doctor check for a droppedKIRO_API_KEYreads the file as defining nothing and prints its one existing warning instead of raising; the gateway's own decode warning names the file and asks for a UTF-8 re-save, so the install warning keeps a single remedy path. The dashboard rewriters (_write_env_updates_lockedindashboard/handlers/messaging.py, the three.envhelpers indashboard/handlers/weixin_qr.py) let the error propagate, so they never overwrite a file they could not parse._write_env_off_loop, which every channel save goes through, turns it into an HTTP 409 with a JSONerrornaming the file and encoding, asking for a UTF-8 re-save and saying the.envwas not changed. The Slack, Teams, Webex, WeCom and Feishu saves still roll their config write back on that failure; Discord and Telegram commit config before the.envwrite and keep it, so their 409 ends "The .env was not changed; your other settings were saved." and a retry after the UTF-8 re-save completes the save. The WeChat sign-in keeps itspersist_failedresponse, whosedetailis now the error's own message ("the .env is saved as UTF-16; re-save it as UTF-8"). Every rewriter puts a UTF-8 BOM back on the text it writes (env_bom_prefixinconfig/loader.py, the first three bytes only), as the importer does. The BOM is what makes the loader read the file as UTF-8, so dropping it on a save would hand a non-ASCII value to the locale codec at the next boot.kirocrew setup --slack(_setup_slack_tokensincli_setup.py) catches it, before the prompts and again under the lock, prints how to re-save the file as UTF-8 and returns without writing.Backwards compatibility
Compatible: a
.envwith no BOM is decoded with the same encoding and error handling as on main, so a BOM-less file that does not decode in the locale still raises fromload_credentialsandread_env_file_credential. A UTF-8 BOM file now loads its first key where it was dropped, and a UTF-16 or UTF-32 file is read as empty where it raised. A dashboard channel save on such a file answers 409 where it answered 500.Tests
test/test_env_file_bom.py, 66 cases (every wide-encoding case runs with explicit UTF-16LE, UTF-16BE, UTF-32LE and UTF-32BE bytes, so the big-endian branches are covered on a little-endian runner). 55 fail on origin/main and pass on this head:.envloadsJIRA_API_TOKENun-prefixed throughload_credentialsandread_env_file_credential;.envis empty for both readers and produces exactly one warning naming the file, and warns again if the file is fixed and then broken again;.env, under both a UTF-8 and a cp1252 locale, is empty for both readers, holds no NUL keys and produces exactly one warning naming the file and UTF-32;EnvFileWideEncodingErrorcarrying only the BOM bytes, so itsrepr()holds no credential, and its message names the encoding and no codec; a UTF-32LE BOM is classified as UTF-32, not UTF-16;.envin each wide byte order that holdsKIRO_API_KEYit still prints the dropped-key warning instead of raising, without the key's value;.envwas not changed and leaves the file byte-identical;.envwas not changed and the other settings were saved, keeps itsenabled: trueconfig change and leaves the file byte-identical;setup --slackrewrites of a UTF-8 BOM file keep the BOM, so a non-ASCII value still loads with an ASCII locale;secrets import --applymigrates a BOM-prefixed first key and keeps the BOM and CRLF terminators on the rewrite, and a successful import re-arms the undecodable-file warning;kirocrew setup --slackon a UTF-16 or UTF-32.envprints the remedy naming the encoding, does not prompt, does not raise and leaves the file byte-identical;EnvFileWideEncodingError;config/loader.pythat locates the.env(throughenv_path()or a<dir> / ".env"join) reads it withread_text, or withread_bytesor a read-modeopenwithout a loader decoder (an AST scan ofsrc/kiro_crew, which finds the five bare readers on origin/main).The other 11 pin behaviour that must not change, or check the scan itself: a plain UTF-8 CRLF file loads as before, a rewrite of a file with no BOM adds none, a rewriter refuses a wide-encoded file without touching it, the importer migrates nothing from one and leaves it byte-identical, and the scan flags synthetic bare readers (
read_text, rawread_bytes, read-modeopen, a<dir> / ".env"join) but not a write-modeopenor the writers'os.openlock file.Run on this head:
pytest test/test_env_file_bom.py(66 passed);test_env_file_bom.py,test_config_json_bom.py,test_user_json_bom.py,test_cli_setup_slack.py,test_cli_setup_cov80.py,test_service.py,test_cli.py,test_cli_doctor.py,test_secrets_migrate.py,test_weixin_qr.py,test_config_loader.py,test_agent_spec_hardened_reads.py,test_members_dm_thread.py,test_messaging_*.py,test_channel_env_write_off_loop.py,test_config_writers_advisory_lock.py,test_config_savers_locked_write.py,test_handlers_messaging_coverage.py,test_whatsapp_setup.pyandtest_kiro_prerequisite.pytogether: 3945 passed, 6 skipped, 2 failed. The 2 aretest_cli.pydoctor STT cases that fail on this host on every head of this PR; the diff does not touch doctor STT.scripts/check_black_formatting.py,scripts/check_comment_history.py(against origin/main),isort --check-only,flake8andmypyon the changed files are clean.Manual verification
N/A — unit coverage sufficient: the tests write real BOM bytes and drive the real readers, rewriters and importer against a temp file. Not run on a Windows host.
Related Issues
Related: #15490, #15491 (same BOM family, different readers), #13316 (closed; its reader-only BOM change was backed out for the rewriter mismatch this PR closes).
Pattern harvest
Rule candidate: review-prompt
Pattern: a file with several readers and read-modify-write rewriters decodes through one shared helper, so a normalisation added to the readers cannot leave the rewriters matching a different key.
Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)Contribution License Agreement
Closes #15553