Skip to content

fix(config): read a byte-order-marked .env the same way everywhere - #15596

Merged
bolichen97 merged 2 commits into
kirodotdev:mainfrom
jeeshofone:fix/15553-env-bom
Oct 4, 2026
Merged

bolichen97 merged 2 commits into
kirodotdev:mainfrom
jeeshofone:fix/15553-env-bom

Conversation

@jeeshofone

@jeeshofone jeeshofone commented Sep 30, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: a .env saved by default Windows tooling loads every credential, or fails with one clear warning, instead of silently dropping the first key or keeping the gateway from starting.

PowerShell 5.1 writes a byte-order mark. Out-File -Encoding utf8 writes UTF-8 with a BOM, and the default > / Out-File writes UTF-16LE with a BOM. With a UTF-8 BOM the first line of ~/.kiro/crew/.env parses as \ufeffJIRA_API_TOKEN, so that credential is never loaded and kirocrew secrets import migrates it as garbage. With a UTF-16 or UTF-32 BOM the decode raises UnicodeDecodeError (or, under a single-byte code page such as cp1252, silently yields NUL-riddled keys), which escapes the except OSError in read_env_file_credential and is unguarded in load_credentials, so gateway construction fails.

Why it matters

The secrets guide tells users to edit .env by hand. A Windows user who does that with the default shell or editor loses their first credential with only a "not a recognised credential" warning quoting a key they did not type, or gets a gateway that does not start.

Not a goal

What changed (motivation → approach → change)

Surface Before After
UTF-8 BOM .env, gateway load first key keyed as \ufeffKEY, dropped every key loads un-prefixed
UTF-16 or UTF-32 BOM .env, gateway load UnicodeDecodeError, startup fails (cp1252: NUL-riddled keys) treated as empty, one warning naming the file and encoding
Dashboard channel save on a UTF-16 or UTF-32 .env HTTP 500, no remedy HTTP 409 naming the file and encoding, asking for a UTF-8 re-save, saying the .env was not changed
Dashboard / WeChat / setup rewrite of a BOM file key match misses the first key first key matched, BOM kept on the rewrite
kirocrew secrets import on a BOM file first key not recognised first key migrates, BOM kept on the rewrite

The root cause is that every .env reader decoded bytes with no BOM handling. Fixing only the readers is not enough: the dashboard, WeChat and Slack-setup rewriters parse the same file and match keys exactly, so a reader that stripped the BOM while the rewriters did not would load a key that a dashboard clear could never remove.

So there is one decoder in config/loader.py, decode_env_bytes, and every reader and rewriter of the data home's .env decodes through it, by way of read_env_file (one read returning the raw bytes and the decoded text, which secrets/migrate.py uses because it restores the bytes on its rewrite) or its read_env_text wrapper. A UTF-8 BOM means UTF-8: it is dropped and the rest decoded as UTF-8. A UTF-16 or UTF-32 BOM, in either byte order, raises EnvFileWideEncodingError, a UnicodeDecodeError subclass whose encoding is the codec the mark declares and whose message names the encoding ("the .env is saved as UTF-32; re-save it as UTF-8") rather than the base class's "'utf-8' codec can't decode" text. The UTF-32LE mark begins with the UTF-16LE one, so the BOM table matches it first. No BOM decodes exactly as before. The readers (load_credentials, read_env_file_credential, migrate_env_secrets in secrets/migrate.py) catch only EnvFileWideEncodingError, treat a UTF-16 or UTF-32 file as empty and log one warning naming it through warn_undecodable_env. _names_defined_in_env_file in service/common.py treats any file it cannot decode as defining nothing, so the dropped-KIRO_API_KEY warning still fires for a BOM-less file in a legacy code page instead of being swallowed with the exception. A later successful read_env_file of that path (any reader, the importer included) re-arms the warning, so a file that is fixed and then broken again in the same gateway process warns again. The warning is deduplicated per path because the gateway re-reads .env from many call sites for its whole life; without the re-arm that dedup would mute the path for the rest of the process, so a file broken again after a fix would read every credential as unset with no warning. For a UTF-16 or UTF-32 .env, the service-install and doctor check for a dropped KIRO_API_KEY reads the file as defining nothing and prints its one existing warning instead of raising; the gateway's own decode warning names the file and asks for a UTF-8 re-save, so the install warning keeps a single remedy path. The dashboard rewriters (_write_env_updates_locked in dashboard/handlers/messaging.py, the three .env helpers in dashboard/handlers/weixin_qr.py) let the error propagate, so they never overwrite a file they could not parse. _write_env_off_loop, which every channel save goes through, turns it into an HTTP 409 with a JSON error naming the file and encoding, asking for a UTF-8 re-save and saying the .env was not changed. The Slack, Teams, Webex, WeCom and Feishu saves still roll their config write back on that failure; Discord and Telegram commit config before the .env write and keep it, so their 409 ends "The .env was not changed; your other settings were saved." and a retry after the UTF-8 re-save completes the save. The WeChat sign-in keeps its persist_failed response, whose detail is now the error's own message ("the .env is saved as UTF-16; re-save it as UTF-8"). Every rewriter puts a UTF-8 BOM back on the text it writes (env_bom_prefix in config/loader.py, the first three bytes only), as the importer does. The BOM is what makes the loader read the file as UTF-8, so dropping it on a save would hand a non-ASCII value to the locale codec at the next boot. kirocrew setup --slack (_setup_slack_tokens in cli_setup.py) catches it, before the prompts and again under the lock, prints how to re-save the file as UTF-8 and returns without writing.

Backwards compatibility

Compatible: a .env with no BOM is decoded with the same encoding and error handling as on main, so a BOM-less file that does not decode in the locale still raises from load_credentials and read_env_file_credential. A UTF-8 BOM file now loads its first key where it was dropped, and a UTF-16 or UTF-32 file is read as empty where it raised. A dashboard channel save on such a file answers 409 where it answered 500.

Tests

test/test_env_file_bom.py, 66 cases (every wide-encoding case runs with explicit UTF-16LE, UTF-16BE, UTF-32LE and UTF-32BE bytes, so the big-endian branches are covered on a little-endian runner). 55 fail on origin/main and pass on this head:

  • a UTF-8 BOM .env loads JIRA_API_TOKEN un-prefixed through load_credentials and read_env_file_credential;
  • a UTF-16 BOM .env is empty for both readers and produces exactly one warning naming the file, and warns again if the file is fixed and then broken again;
  • a UTF-32 BOM .env, under both a UTF-8 and a cp1252 locale, is empty for both readers, holds no NUL keys and produces exactly one warning naming the file and UTF-32;
  • every wide byte order raises EnvFileWideEncodingError carrying only the BOM bytes, so its repr() holds no credential, and its message names the encoding and no codec; a UTF-32LE BOM is classified as UTF-32, not UTF-16;
  • the service headless-auth check sees a BOM-prefixed first key, and for a .env in each wide byte order that holds KIRO_API_KEY it still prints the dropped-key warning instead of raising, without the key's value;
  • a Slack or Webex dashboard save against each wide byte order answers 409 with the remedy, says the .env was not changed and leaves the file byte-identical;
  • a Discord or Telegram dashboard save against each wide byte order answers 409 with the remedy, says the .env was not changed and the other settings were saved, keeps its enabled: true config change and leaves the file byte-identical;
  • a dashboard channel clear removes a BOM-prefixed first key and keeps the BOM;
  • the dashboard, WeChat upsert and delete, and setup --slack rewrites of a UTF-8 BOM file keep the BOM, so a non-ASCII value still loads with an ASCII locale;
  • the service headless-auth check treats a BOM-less non-UTF-8 file as defining nothing;
  • the WeChat read, upsert and delete helpers match a BOM-prefixed first key;
  • secrets import --apply migrates a BOM-prefixed first key and keeps the BOM and CRLF terminators on the rewrite, and a successful import re-arms the undecodable-file warning;
  • kirocrew setup --slack on a UTF-16 or UTF-32 .env prints the remedy naming the encoding, does not prompt, does not raise and leaves the file byte-identical;
  • a BOM-less file that does not decode in the locale still raises, and not as EnvFileWideEncodingError;
  • no function outside config/loader.py that locates the .env (through env_path() or a <dir> / ".env" join) reads it with read_text, or with read_bytes or a read-mode open without a loader decoder (an AST scan of src/kiro_crew, which finds the five bare readers on origin/main).

The other 11 pin behaviour that must not change, or check the scan itself: a plain UTF-8 CRLF file loads as before, a rewrite of a file with no BOM adds none, a rewriter refuses a wide-encoded file without touching it, the importer migrates nothing from one and leaves it byte-identical, and the scan flags synthetic bare readers (read_text, raw read_bytes, read-mode open, a <dir> / ".env" join) but not a write-mode open or the writers' os.open lock file.

Run on this head: pytest test/test_env_file_bom.py (66 passed); test_env_file_bom.py, test_config_json_bom.py, test_user_json_bom.py, test_cli_setup_slack.py, test_cli_setup_cov80.py, test_service.py, test_cli.py, test_cli_doctor.py, test_secrets_migrate.py, test_weixin_qr.py, test_config_loader.py, test_agent_spec_hardened_reads.py, test_members_dm_thread.py, test_messaging_*.py, test_channel_env_write_off_loop.py, test_config_writers_advisory_lock.py, test_config_savers_locked_write.py, test_handlers_messaging_coverage.py, test_whatsapp_setup.py and test_kiro_prerequisite.py together: 3945 passed, 6 skipped, 2 failed. The 2 are test_cli.py doctor STT cases that fail on this host on every head of this PR; the diff does not touch doctor STT. scripts/check_black_formatting.py, scripts/check_comment_history.py (against origin/main), isort --check-only, flake8 and mypy on the changed files are clean.

Manual verification

N/A — unit coverage sufficient: the tests write real BOM bytes and drive the real readers, rewriters and importer against a temp file. Not run on a Windows host.

Related Issues

Related: #15490, #15491 (same BOM family, different readers), #13316 (closed; its reader-only BOM change was backed out for the rewriter mismatch this PR closes).

Pattern harvest

Rule candidate: review-prompt
Pattern: a file with several readers and read-modify-write rewriters decodes through one shared helper, so a normalisation added to the readers cannot leave the rewriters matching a different key.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

Contribution License Agreement

Closes #15553

@jeeshofone
jeeshofone requested a review from a team as a code owner September 30, 2026 21:49
@github-actions github-actions Bot added fork Pull request from a fork (external contributor) readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Sep 30, 2026
@jeeshofone

Copy link
Copy Markdown
Contributor Author

Comment History Gate on 43ffa46 flagged the issue number in the new test module's docstring (test/test_env_file_bom.py:1). Fixed on 1ecb184: the docstring states the behaviour without the reference, and COMMENT_HISTORY_BASE_REF=origin/main scripts/check_comment_history.py passes locally. Await Fast Gate, Coverage Gate and PR Readiness were red downstream of that one gate (the heavy jobs were skipped, so Coverage Gate saw frontend-test=skipped and failed closed). The same head is rebased onto current main at e3309cd; the touched-module suites (1085 tests), black, isort, flake8 and mypy on the changed files pass on it.

@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Oct 1, 2026
@github-actions

github-actions Bot commented Oct 1, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5, fork) — ✅ PASS

Premise-level review of 068f74fb0991f98080a84d6038386a9cf14c285c via the fork AI-review pipeline — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: PASS

Check that BOM-less .env files still decode the same on Windows: decode_env_bytes now calls locale.getpreferredencoding(False) where it used to call bare read_text().

What this change ships

Inventory (8 items) — 8 justified

Intent: a Windows user's hand-edited .env loads every credential, or fails with one clear warning, instead of losing the first key or blocking gateway start. FIX.

  1. A .env saved as UTF-8 with a BOM now loads its first key without the \ufeff prefix, in the gateway and in secrets import. — justified
  2. A .env saved as UTF-16 or UTF-32 is read as empty with one warning, instead of raising during gateway construction. — justified
  3. That warning is logged once per path, and fires again after the file is fixed and then broken again. — justified
  4. A dashboard channel save on a UTF-16 or UTF-32 .env returns 409 with the fix, not 500. — justified
  5. The Discord and Telegram 409s say their other settings were saved (config_kept, 2 callers). — justified
  6. The dashboard, WeChat, setup and import rewriters now match the first key in a BOM file and keep the BOM when they rewrite it. — justified
  7. kirocrew setup --slack refuses a wide-encoded .env with a re-save instruction, both before the prompts and under the lock. — justified
  8. The service and doctor check for a dropped KIRO_API_KEY treats a file it cannot decode as defining nothing, instead of raising. — justified

[FIRST-PRINCIPLES-REVIEWED] 068f74f

Description read

[DESCRIPTION-READ] b50c3994eb2b233be3839d2ab17e582b6fe686ddfbbb0d3cd5d2f53baa94c2cb

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Oct 1, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review (fork) — ✅ no blocking findings

Reviewed 068f74fb0991f98080a84d6038386a9cf14c285c via the fork AI-review pipeline; updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 068f74f

@github-actions

github-actions Bot commented Oct 1, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5, fork) — ✅ PASS

Design-level review of 068f74fb0991f98080a84d6038386a9cf14c285c via the fork AI-review pipeline — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Check that test_no_function_reads_env_path_without_the_decoder passes on this head: it is the only thing stopping a later .env reader from skipping the shared decoder.

Suggestions

  • Decoding a UTF-16 or UTF-32 BOM is one raw.decode(codec) in decode_env_bytes. That would remove the refusal code: the 409 path, config_kept, the setup remedy, and a gateway that boots with no credentials and only a log line to say why. Rewriters would then save the file back as UTF-8.

[DESIGN-REVIEWED] 068f74f

Description read

[DESCRIPTION-READ] c8faa66641fa9ecd6061c2d84c51e6957bc6bbd542c63e43ab3a74ca094469d8

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 1, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review (fork) — ✅ no blocking findings

Reviewed 068f74fb0991f98080a84d6038386a9cf14c285c via the fork AI-review pipeline; updated in place on each push.

Review details

No findings that block the merge.

FINDING — src/kiro_crew/dashboard/handlers/messaging.py:6449 — Discord and Telegram commit config before writing .env, including the legacy bot_token purge (drop_keys=("bot_token",) if purge_legacy_token). Setup: .env is UTF-16, and this PR now loads such a file as unset, so a bot can run on the legacy config token. When the operator saves a new token, the save deletes that legacy token. _write_env_off_loop(env_updates, config_kept=True) then returns a 409 saying "your other settings were saved". The message never says the token the bot was running on is gone, and when staged is empty no other setting was saved at all. On the next restart the bot has no credential → Fix: in both handlers, check .env for a wide BOM before the _LockedSectionWrite(...).commit(), so the purge never runs ahead of a refused .env write, and pass config_kept=bool(staged).

FINDING — src/kiro_crew/service/common.py:533 — For a UTF-16/UTF-32 .env, except (OSError, UnicodeDecodeError): return set() makes headless_auth_warning print the remedy printf ... >> {target}. That appends UTF-8 bytes after the wide BOM, and decode_env_bytes still rejects the whole file. The key stays unset and the warning comes back on every install → Fix: catch loader.EnvFileWideEncodingError separately and print a "re-save it as UTF-8" remedy instead of the append command.

[OPUS-REVIEWED] 068f74f

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 1, 2026
@jeeshofone

Copy link
Copy Markdown
Contributor Author

New head 28f309b, rebased onto main 321fd99, with one added commit for the two advisory findings on 1ecb184.

GPT 5.6 Review, config/loader.py load_credentials decode handling: fixed. The finding was right: catching UnicodeDecodeError also swallowed a BOM-less .env that does not decode in the locale, which raised on main. decode_env_bytes now raises EnvFileUtf16Error, a UnicodeDecodeError subclass, only for a UTF-16 byte-order mark, and all four readers (load_credentials, read_env_file_credential, _names_defined_in_env_file, migrate_env_secrets) catch only that. Any other decode error propagates exactly as on main. New regression test_a_bomless_file_invalid_in_the_locale_still_raises forces an ASCII locale over a UTF-8 body and asserts both readers raise a plain UnicodeDecodeError; it fails on 1ecb184 ("DID NOT RAISE") and passes on 28f309b.

Opus 5 Review, test/test_env_file_bom.py whole-dict assertion: fixed. _load now delenvs every loader.CREDENTIAL_KEYS entry before calling load_credentials, so an exported credential in the runner's shell cannot add an entry to the compared dict.

Backend Tests (Windows) (2) on 1ecb184: not caused by this diff. test_native_skill_projection.py::test_leftover_alias_lock_files_are_swept_only_when_provably_residue and ::test_a_lock_file_a_launcher_still_holds_is_never_swept fail because the orphan alias-lock sweep's unlink is refused with EACCES ("being used by another process") on Windows. This PR touches no skill-projection, lock or platform code. The same two tests fail on main's own push CI at 4d07623 (run 36798364951, Windows (2) job 110167936256). The sweep and the tests came in with #15150; the fix is open as #15633. Expect that shard to stay red on this head until #15633 merges.

Gates on 28f309b: test/test_env_file_bom.py 11 passed; every test module exercising the touched readers and rewriters (42 modules) 4615 passed, 3 failed, and those 3 (two test_cli.py doctor STT cases, one test_service.py systemd case) fail identically on origin/main on this host. scripts/check_black_formatting.py, isort --check-only, flake8, mypy on the changed files and scripts/check_comment_history.py pass.

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Oct 1, 2026
@jeeshofone

Copy link
Copy Markdown
Contributor Author

New head ea90c71, rebased onto main ecdc59a. Two commits: the original change and one follow-up for the review findings.

GPT 5.6 Review on 28f309b, service/common.py:338 "UTF-16 warning recommends corrupting the credential file": fixed. The finding was right: for a UTF-16 .env, headless_auth_warning still printed the printf ... >> append, and appending UTF-8 text to a UTF-16 file leaves it mixed-encoding and still unreadable. A new _env_file_is_utf16 reads only the file's first two bytes; when it reports a UTF-16 byte-order mark, the warning says the gateway cannot read the file because it is saved as UTF-16, asks for it to be re-saved as UTF-8 with KIRO_API_KEY=<value> added, and prints only the restart command. The regression test_service_warning_for_a_utf16_file_asks_for_utf8_not_an_append asserts the text names UTF-16 and UTF-8, has no >>, and does not echo the key's value. It fails on 28f309b and passes on ea90c71.

The earlier GPT finding (BOM-less locale decode errors) and the Opus test-isolation finding, both reviewed on 1ecb184, stay fixed as described in my previous comment and are carried into the same follow-up commit.

Backend Tests (Windows) (2): the two test_native_skill_projection.py lock-sweep failures were main-side and are fixed on main by #15633. This head contains that merge, so the shard should clear.

Backend Tests (Windows) (5) on 28f309b: three test_agent_sdk_context_bridge.py::test_context_import_orders_do_not_form_a_cycle parametrizations hit their 30 s subprocess timeout together on the Windows runner. That test imports modules this PR does not touch. I found no tracking issue; this head reruns it.

Opus 5 Review on 28f309b "review incomplete": that run's validation step failed with "API rate limit exceeded for installation". I pushed this head after later runs of the lane had succeeded.

Gates on ea90c71: test/test_env_file_bom.py 12 passed; test_secrets_migrate.py, test_weixin_qr.py, test_config_loader.py and test_service.py plus the BOM tests: 1090 passed, 1 failed. The failure is the test_service.py systemd case, which fails the same way on origin/main on this host. scripts/check_black_formatting.py, isort --check-only, flake8, mypy on the changed files and scripts/check_comment_history.py pass.

@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Oct 1, 2026
@jeeshofone

Copy link
Copy Markdown
Contributor Author

New head 5cdefda, rebased onto main 9a59174. Still two commits; git range-diff shows the first = and the second carrying the change below.

GPT 5.6 Review on a393be6, BLOCKING, service/common.py:280, a UTF-32 .env got the append remedy: fixed. The UTF-16-only probe is replaced by env_file_wide_encoding in config/loader.py, which returns "UTF-16" (either byte order, the same predicate decode_env_bytes uses), "UTF-32" (either byte order) or None, still from the first four bytes only. headless_auth_warning takes the re-save branch for both wide encodings and names the one it found, so a UTF-32 file is told to re-save as UTF-8 and gets no >> line. The adjudicator is right that the old append could not destroy anything, but it left the file mixed-encoding and the hint wrong, and the fix is one probe. New test_service_warning_for_a_utf32_file_asks_for_utf8_not_an_append (UTF-32LE and UTF-32BE) and the widened test_wide_encoding_probe_agrees_with_the_decoder fail on a393be6 and pass here. UTF-32 is still not decoded, and the loader still does not classify a UTF-32LE BOM as UTF-16.

Design Review on a393be6, suggestion to pin "every .env reader goes through decode_env_bytes": done. TestEveryReaderUsesTheLoaderDecoder parses every module under src/kiro_crew outside config/loader.py and fails if a function that calls env_path() also calls read_text, or calls read_bytes without decode_env_bytes. It passes on the current tree, and test_the_scan_catches_a_bare_reader checks it flags a synthetic bare read_text and a raw read_bytes().decode() and accepts a decoded read. It looks at one function body at a time, so a path passed into a helper is only checked if that helper calls env_path() itself. That limit is in its docstring.

Opus 5 Review on a393be6, "review incomplete": the lane did not finish, so there was no finding to answer. This head re-runs it.

Backend Tests (Windows) (4) on a393be6, test_autonudge.py::test_a_terminal_watch_notifies_even_though_it_cancels_its_own_timer ("the user must be told the watch finished"): this PR does not touch autonudge. It is a flaky test with no tracking issue yet, and a separate fix is in progress.

macOS Tests / Backend Tests (macOS) (1) on a393be6, test_sandbox_interpreter_enoent_retry.py::TestACancelDuringTheShellProbeIsNotLost::test_the_cancelled_command_is_never_launched and three test_decisions_local_runtime_real_spawn.py cases (runtime stayed starting): this PR does not touch the sandbox or the local decisions runtime. Single-run reds; this head retries them.

Run on this head: test_env_file_bom.py 41 passed; with test_cli_setup_slack.py, test_cli_setup_cov80.py, test_service.py, test_cli.py, test_secrets_migrate*.py, test_weixin_qr.py, test_config_loader.py, test_members_dm_thread.py, test_session_export.py and test_messaging_*.py, 2897 passed, 5 skipped and 2 failed (the two test_cli.py doctor STT cases, which fail the same way on main on this host). The comment-history gate, isort --check-only, flake8 and mypy on the changed files are clean, and black reports nothing on the lines this PR changes.

@jeeshofone

Copy link
Copy Markdown
Contributor Author

On 5cdefda every check passed and every review lane is PASS or no findings. Two advisory notes come back from earlier heads:

First Principles Review, confirm on Windows that a BOM-less decode is unchanged under UTF-8 mode and legacy code pages: decode_env_bytes decodes a BOM-less file with raw.decode(encoding or locale.getpreferredencoding(False), errors). A bare read_text() with no encoding uses the same locale encoding: locale.getpreferredencoding(False) returns utf-8 when UTF-8 mode is on and the ANSI code page when it is off. test_a_bomless_file_invalid_in_the_locale_still_raises pins that a locale decode failure still raises. I have no Windows host to run it on, so a maintainer check there is still welcome.

Design Review, check the degraded boot once by hand (a UTF-16 .env now starts a credential-less gateway with only a log warning): that is the behaviour #15553 asks for, in place of a gateway that does not start. The warning names the file and asks for a UTF-8 re-save, kirocrew service install and doctor print the same remedy when KIRO_API_KEY would be dropped, and kirocrew setup --slack refuses to write and prints it too. If a maintainer would rather fail fast, I can change it.

@jeeshofone

Copy link
Copy Markdown
Contributor Author

Head f8c6d108e (rebased onto current main) addresses the findings from an independent review of 5cdefda83:

  • UTF-32 .env still stopped the gateway (config/loader.py, decode_env_bytes): fixed. A UTF-16 or UTF-32 byte-order mark in either byte order now raises EnvFileWideEncodingError (renamed from EnvFileUtf16Error), so load_credentials, read_env_file_credential, the importer and setup --slack all take the treat-as-unset-and-warn path. The UTF-32LE mark is matched before the UTF-16LE one. Under cp1252 the file no longer decodes into NUL-riddled keys. Covered by test_utf32_is_unset_with_one_warning_in_any_locale (UTF-8 and cp1252 locales, both byte orders).
  • Dashboard channel save answered HTTP 500 (dashboard/handlers/messaging.py, _write_env_off_loop): fixed in the shared path every channel save uses. The refusal is now a 409 whose JSON error names the file and encoding, asks for a UTF-8 re-save and says the .env was not changed; each handler's rollback still runs. Covered for Slack, Discord, Webex and Telegram across all four byte orders.
  • NIT, misleading "'utf-8' codec" text: the error's message is now "the .env is saved as UTF-16; re-save it as UTF-8" (or UTF-32), with no codec wording, so the WeChat detail reads correctly too.
  • NIT, importer did not re-arm the warning (secrets/migrate.py): the importer reads through the new loader.read_env_file, which returns the bytes and text from one read and re-arms warn_undecodable_env. Covered by test_a_successful_import_re_arms_the_undecodable_warning.
  • NIT, narrow reader scan: the AST pin now also flags a read-mode open() and a path built as <dir> / ".env", and its self-test proves a write-mode open and the writers' os.open lock file are not flagged. It finds nothing new in src/kiro_crew today.

Against origin/main, 61 of the 72 tests in test/test_env_file_bom.py fail, and all 72 pass on this head.

@jeeshofone

Copy link
Copy Markdown
Contributor Author

Head d7d1cfa1f:

  • First Principles, item 3 (the 409 body's code key had no consumer): fixed by deleting it. _wide_env_refusal now returns {"error": message}, the same body shape the other channel-save refusals use. The test asserts on the message only.
  • Backend Tests (Windows) (4) on f8c6d108e: test_resume_publishes_hydrated_slot.py::test_a_delete_then_recreate_during_the_read_does_not_overwrite_the_replacement (assert 200 == 409). That file isn't in this diff and the test doesn't touch .env handling. It passes 3/3 when run alone on Linux at this head, and I found no issue or PR that names it. I'm treating it as an untracked Windows flake. This head is the one fresh retry.

@jeeshofone

Copy link
Copy Markdown
Contributor Author

Head 244350e90:

  • First Principles, item 7 (the warning re-arm has no reported defect): keeping it. The .env is read again by every read_env_file_credential call for as long as the gateway runs, which is why the warning is deduplicated per path in the first place. With the dedup but no re-arm, the first warning silences that path for the rest of the process. Fix the file, break it again with a second UTF-16 save, and every credential silently reads as unset with nothing logged. Re-arming the warning is the reason the importer now reads through read_env_file; that was a requested fix on this PR. The same lane passed this exact re-arm on 5cdefda83. The rationale now sits at the set's definition in config/loader.py and in the PR description.
  • Backend Tests (3.12, 4) / Coverage Gate on d7d1cfa1f: test_crew_log_emit.py::test_a_predecessor_collected_during_the_deferral_costs_no_dropped_write hit OSError: [Errno 39] Directory not empty while tearing down its temp crew-log directory. That file isn't in this diff. The test passes 3/3 when run alone at this head, and I found no issue that names it, so I'm treating it as an untracked flake. The Windows (4) red from the previous head is green on d7d1cfa1f.

@jeeshofone

Copy link
Copy Markdown
Contributor Author

Head 9f1995d51, replying to the lanes on 244350e90:

  • GPT, messaging.py:5378 (rollback wording): fixed. Discord and Telegram commit config.json before calling _write_env_off_loop and don't roll it back, so the old docstring was wrong for them. The docstring now lists the saves that roll back (Slack, Teams, Webex, WeCom, Feishu). It also says Discord and Telegram keep their config change on the 409, and that a retry after the UTF-8 re-save completes the save. The PR description is corrected the same way. There's no behaviour change: the .env is still untouched on that path, so the 409's "the .env was not changed" stays accurate.
  • First Principles, "please confirm": confirmed, and intended. On main, a UTF-16 .env raised out of load_credentials and stopped the gateway. With this PR the gateway starts, treats the file as unset and logs one warning that names the file. A UTF-32 .env now gets the same treatment. Before, it raised under a strict locale and loaded NUL-filled keys under cp1252. That is the goal in secrets: a UTF-16 or UTF-8-BOM .env breaks credential loading on Windows — the first key is silently dropped, or the load path crashes #15553: such a file fails with one clear warning and does not stop the gateway.
  • First Principles items 6 and 8 (dashboard 409, setup --slack remedy): keeping both. An independent review of this PR found that a dashboard save on a UTF-16 file gave an opaque 500 with no fix text, while the CLI printed the remedy for the same file. Once the loader treats the file as unset, these two writers are where a user runs into it, so each one names the encoding and the fix and changes nothing. Neither path writes more than it did on base. Both still refuse before writing.
  • First Principles item 7 (env_file_wide_encoding in the service warning): keeping it. For a wide-encoded file, the existing dropped-KIRO_API_KEY warning printed a printf ... >> .env append. That append would leave the file mixed-encoding and still unreadable, so the warning's own advice was wrong for this case. The probe reads only the first four bytes and shares decode_env_bytes' BOM table, so the warning and the loader can't disagree, and the warning still never holds a credential value.

@jeeshofone

Copy link
Copy Markdown
Contributor Author
  • Item 7: service-install/doctor wide-encoding branch and env_file_wide_encoding

Outcome: rebutted, kept. The clear condition ("a reported case shows the append remedy misleading a user") was met during this PR's review. GPT 5.6 Review on a393be64b raised a BLOCKING finding at service/common.py:280: a UTF-32 .env was given the >> append. This branch is the fix for that finding.
The gateway warning doesn't cover this case, because the two warnings run at different times. headless_auth_warning runs during kirocrew service install and doctor, before the gateway has read the file, and it prints a command for the user to run. For a UTF-16 or UTF-32 file, that printf ... >> .env writes ASCII bytes into a wide-encoded file. The result is mixed-encoding: still unreadable, and now harder to repair by re-saving. Removing the branch would put that command back.
env_file_wide_encoding is not a second BOM sniffer. It calls the same _wide_bom table decode_env_bytes uses, so the two can't disagree, which test_wide_encoding_probe_agrees_with_the_decoder checks. It reads only the first four bytes, so the warning still never holds a credential value. Calling decode_env_bytes there instead would mean reading the whole file a second time. An earlier Opus finding on 0240af3dd rejected exactly that.
Head 67c222cb4 keeps it. That head is 9f1995d51 folded back to two commits and rebased onto main 4e0aa9b70; the diff is unchanged (git range-diff shows both commits =).

@jeeshofone

Copy link
Copy Markdown
Contributor Author

Head 26e3584: for Discord and Telegram, which keep their config write when the .env write is refused, the 409 now ends "The .env was not changed; your other settings were saved." The other five channels roll their config back and keep "The .env was not changed." test_env_file_bom.py::TestDashboardChannelSave::test_a_save_that_keeps_its_config_says_the_other_settings_were_saved pins the wording and the kept enabled: true for both channels across all four wide byte orders; it fails on the previous head.

@jeeshofone

Copy link
Copy Markdown
Contributor Author
  • Item 6: service-install/doctor wide-encoding branch and env_file_wide_encoding

Outcome: rebutted, kept. This is the same branch First Principles flagged as item 7 on 9f1995d51 (span 898f92160d38). The disposition there still holds, and head 76bcbc06d doesn't touch this code: the only change since the last passing head is the Discord/Telegram 409 wording in messaging.py, plus a rebase onto main.
The clear condition ("a report shows a user following the append advice on a UTF-16 .env") was met during this PR's review. GPT on a393be64b raised a BLOCKING finding at service/common.py:280 because a UTF-32 .env was given the printf ... >> .env append. This branch is the fix.
The gateway's "Cannot decode" log doesn't cover this case. headless_auth_warning runs during kirocrew service install and doctor, before the gateway reads the file, and the line it prints at service/common.py:320 is a command for the user to run. Appending ASCII to a UTF-16 or UTF-32 file leaves it mixed-encoding and still unreadable. Deleting the branch would bring that advice back.
env_file_wide_encoding (config/loader.py:585) reads only the first four bytes and uses the decoder's BOM table, so the two can't disagree (test_wide_encoding_probe_agrees_with_the_decoder), and the warning never holds a credential value.

A .env saved by PowerShell 5.1 carries a byte-order mark. With a UTF-8
BOM the first key parsed as "\ufeffKEY", so that credential was never
loaded and `secrets import` migrated it as garbage. With a UTF-16 BOM
the strict decode raised UnicodeDecodeError, which escaped the OSError
guard in read_env_file_credential and was unguarded in
load_credentials, so the gateway could not start.

Every reader and in-place rewriter of the data home's .env now decodes
through one helper, read_env_text / decode_env_bytes in config/loader:
a UTF-8 BOM is dropped and the rest read as UTF-8, a UTF-16 BOM raises,
and a file with no BOM decodes exactly as before. The readers treat an
undecodable file as empty and log one warning naming it; the rewriters
let the error propagate so they never overwrite a file they could not
parse. Routing the dashboard, WeChat and setup rewriters through the
same helper keeps a clear able to match the key the gateway loads.

Closes kirodotdev#15553
@jeeshofone

Copy link
Copy Markdown
Contributor Author
  • Item 7: service-install/doctor wide-encoding branch and env_file_wide_encoding

Outcome: fixed in 196f212. env_file_wide_encoding is deleted from config/loader.py, and headless_auth_warning in service/common.py is back to its single append-remedy path, line for line as on main. The PR's only change in that file is _names_defined_in_env_file, which reads through the shared decoder and treats a file it cannot decode as defining nothing.
A wide .env therefore still makes the install/doctor check warn that KIRO_API_KEY is dropped instead of raising (new test test_service_warning_still_fires_for_a_wide_file, all four byte orders; it fails on main with UnicodeDecodeError). The gateway's decode warning is what names the file and asks for a UTF-8 re-save, which is the one clear warning the Goal describes.
The tests that covered the removed branch (the probe test and the UTF-16/UTF-32 warning-text, read-once and KIROCREW_HOME cases) are gone with it. The PR body's description and test counts (66 cases, 55 failing on main) match this head. Rebased onto main f027885; still two commits.

…ix it

The BOM-aware readers caught UnicodeDecodeError, so a BOM-less .env that
does not decode in the locale was silently loaded as empty instead of
raising as before. decode_env_bytes now raises a dedicated
EnvFileWideEncodingError subclass for a UTF-16 or UTF-32 byte-order mark,
and the credential readers catch only that. The UTF-32LE mark, which
begins with the UTF-16LE one, is matched first. Before this a UTF-32 .env
fell through to the locale decode: under a strict locale the plain
UnicodeDecodeError escaped load_credentials and read_env_file_credential
and stopped the gateway, and under cp1252 it loaded NUL-riddled keys.

The error names the detected encoding ("the .env is saved as UTF-32;
re-save it as UTF-8") instead of the base class's "'utf-8' codec can't
decode" text, which the WeChat sign-in surfaced as its error detail. It
carries only the BOM bytes in .object, so its repr and a traceback's
locals never hold the file.

A dashboard channel save against a wide-encoded .env answered with an
opaque HTTP 500. _write_env_off_loop, which every channel save goes
through, now turns the refusal into a 409 whose JSON error names the
file and encoding, asks for a UTF-8 re-save and says the .env was not
changed. The Slack, Teams, Webex, WeCom and Feishu saves still roll their
config write back on that failure; Discord and Telegram commit config
before the .env write and keep it, so their 409 also says the other
settings were saved, and a retry after the UTF-8 re-save completes the
save.

For a UTF-16 or UTF-32 .env the service-install and doctor check for a
dropped KIRO_API_KEY treats the file as defining nothing, so it prints
its one existing warning instead of raising. The gateway's own decode
warning names the file and asks for a UTF-8 re-save, so the install
warning keeps a single remedy path.

`kirocrew setup --slack` on a UTF-16 or UTF-32 .env prints how to re-save
it and leaves the file untouched instead of raising a traceback.

Every successful decode through loader.read_env_file re-arms
warn_undecodable_env for that path, so a .env that is fixed and then
broken again in the same process warns again. The secrets importer now
reads through read_env_file too (one read gives it both the bytes it
restores and the text it parses), so a successful import re-arms the
warning as well.

The service warning's own name scan treats any file it cannot decode as
defining nothing, so the warning still fires for a BOM-less file in a
legacy code page instead of raising into a handler that swallows it.

Every in-place rewriter (dashboard channel save, the WeChat helpers,
`setup --slack`) puts a UTF-8 BOM back on the text it writes, as the
importer does: the BOM is what makes the loader decode the file as UTF-8,
so dropping it on a save would hand a non-ASCII value to the locale codec
at the next boot.

The reader scan also flags a read-mode open() and a path built as
`<dir> / ".env"`, not only read_text/read_bytes on env_path().

Tests pin every byte order explicitly, and the credential and migrate
test helpers scrub every recognised credential key so an exported one
cannot change the result.
@jeeshofone

Copy link
Copy Markdown
Contributor Author

Head 068f74fb0 is 196f21298 with a fresh committer date only (git diff 196f21298 068f74fb0 is empty; base is still main f02788522). It retries the one red on 196f21298, which is outside this PR's diff:

Backend Tests (Windows) (1): test/test_spawn_approval_crew_log.py::test_a_prompt_no_surface_received_is_a_host_decline_with_its_cause raised OSError: could not acquire exclusive file lock: still held after waiting 0.00s (one attempt: an acquire on the event-loop thread never waits ...) at platform_lock_compat.py:336. The test came in with #16560 (f31e2f709), and this PR touches neither that file nor the crew log. It is the same event-loop lock family as #16093. Every review lane (GPT, Opus, Design, First Principles, UX, Security Scope) passed on 196f21298.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

fork Pull request from a fork (external contributor)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

secrets: a UTF-16 or UTF-8-BOM .env breaks credential loading on Windows — the first key is silently dropped, or the load path crashes

3 participants