Skip to content

fix(env): reload .env credential changes without a gateway restart - #23491

Closed
RohanK6 wants to merge 2 commits into
mainfrom
env-hotreload-22450-final
Closed

RohanK6 wants to merge 2 commits into
mainfrom
env-hotreload-22450-final

Conversation

@RohanK6

@RohanK6 RohanK6 commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: An edit to $KIROCREW_HOME/.env takes effect on a running gateway, restarting only the affected channel, without a full gateway restart.

Edits to $KIROCREW_HOME/.env took effect only after a full gateway restart. The live config watcher fingerprinted config.json only, and nothing re-read .env after boot. The dashboard's own credential writer applied its changes live, but a hand edit or any other writer did not, so a rotated token kept reading as the old value until someone restarted the gateway.

Why it matters

Operators rotate channel credentials by editing .env. Today the edit silently does nothing until a restart, so a revoked token keeps a channel failing to authenticate, and a new token looks "saved" but is not in force. Requiring a full restart for a credential change also drops every running session.

Not a goal

  • Hot-swapping a Slack token in place. Slack's client is host-managed with no in-process restart path, so a Slack token change logs a "run kirocrew restart" notice instead.
  • Pushing a new environment into already-running children (ACP sessions, MCP servers). They keep their environment until they respawn.
  • Changing which keys .env supports or how boot reads it.

What changed (motivation → approach → change)

Symptom: a .env edit is ignored until restart. Root cause: ConfigWatch never watched .env. Change: .env is reconciled in the same poll cycle that already reloads config.json.

operator edits $KIROCREW_HOME/.env
        │
        ▼
ConfigWatch._cycle()  ── same poll that already reloads config.json
        │
        ├─ env_fingerprint(.env) unchanged → nothing to do
        │
        └─ env_fingerprint(.env) changed
                │
                ▼
        read_env_credentials_raw()       (off-loop, permission-checked)
        _prepare_path_additions()        (off-loop: PATH expand + validate)
                │
                ▼
        diff vs baseline (seeded once at boot from the raw map
        load_credentials() actually read, not a later disk read)
                │
                ├─ validate every changed key first (atomic batch)
                │
                ▼
        apply_env_credential_changes()   (on-loop: os.environ mutation only)
                │   set/pop os.environ, honouring the Docker scrub and PATH carve-outs
                │   pop only what .env itself propagated (ownership is recorded)
                │
                ├─ success → advance fingerprint + baseline
                │            dispatch credentials.<KEY> (+ Teams restart_credentials)
                │                     │
                │                     ▼
                │            restart_channel() re-reads load_credentials()
                │            (Slack: host-managed → "run kirocrew restart" notice)
                │
                └─ failure → leave fingerprint + baseline untouched
                             next tick retries the whole diff
  • After boot, every gateway-side credential read (channel reconnect, readiness badge, jira/doctor/upload-destination) uses load_credentials(propagate=False), so a worker-thread read cannot reseed a credential the watcher just removed.
  • The dashboard's credential writer now goes through the same apply_env_credential_changes instead of its own set/pop loop, so it also respects the Docker credentials scrub.
  • A .env PATH line is expanded and validated off the event loop (env.prepare_env_path_additions, run via asyncio.to_thread) and handed to the applier as path_additions. A ~otheruser entry calls pwd.getpwnam / the SAM database, and that lookup no longer runs on the loop. os.path.expanduser semantics are unchanged.
  • A live set records .env ownership only when the write takes the entry over (a new key, a rotation, an overwrite of a foreign value, or a key .env already owns). A line whose value merely matches what the process inherited from a service unit or shell is not owned, so removing it later leaves the inherited credential in place, as boot's setdefault already does.
  • The applier decides per os.environ ENTRY rather than per .env spelling. On Windows (case-insensitive os.environ) every spelling of a key (HTTPS_PROXY / https_proxy) is grouped and resolved once, to the first non-empty spelling in .env order (boot's setdefault precedence), so a rename, a partial duplicate removal, or both spellings edited in one save cannot leave the entry depending on iteration order. On POSIX each entry has one spelling, so nothing changes there.
  • The watcher's diff (changed_env_keys) also compares each entry's effective value, so a Windows save that only reorders two spellings with different values is applied instead of silently adopted as the new baseline.
  • The Weixin QR sign-in writes its token through the same apply_env_credential_changes as the dashboard channel writer, so it records .env ownership (a later removal of the line revokes it in-process) and keeps the Docker scrub guard.
  • Teams: closing the client on a credential removal also clears its bound webhook handler and returns 503 for a request already past the close check, because the inbound route stays registered independently of the client. A tenant-only edit restarts Teams through a new restart_credentials descriptor field.
  • Teams: the webhook handler (state.teams_on_activity) is bound only once a client is the accepted handle, after boot's start_channels and after a restart passes its generation check. maybe_start_teams no longer binds it before the connect completes, so a restart superseded mid-connect never exposes its handler (the route answers 503 until then).
  • Slack: the restart-needed notice matches credential keys by canonical entry name, the same rule as registry.changed_credential_keys, so a lowercase .env Slack key on Windows still gets the notice.

This supersedes #23027 and #23470 (both closed). Same reviewed change, squashed onto current main so PR Hygiene's commit limit is met without a force-push; every AI review lane passed on that tree at #23470's last head. The second commit fixes the two findings raised here.

Backwards compatibility

Compatible: boot still seeds .env with setdefault, every key .env accepted before is still accepted, and PATH entries resolve exactly as before.

Removes nothing: the diff adds a watcher input and routes existing credential writes through one applier; no key, kind, public function or stored data is removed or renamed.

Reader: src/kiro_crew/dashboard/messaging_api/channel_config_writes.py:write_channel_env -- crew page -- under _KIROCREW_CREDS_SCRUBBED a dashboard save no longer writes credential keys into os.environ, matching the boot loop; children read them from .env (test_a_landed_write_honours_the_docker_creds_scrub)

Tests

Proof: PROVEN test/test_channel_boot_hot_reload.py::TestTeamsCloseClearsWebhookHandler::test_closing_teams_clears_the_late_bound_webhook_handler, test/test_channel_boot_hot_reload.py::TestTeamsWebhookPublishedOnlyForTheAcceptedClient::test_an_accepted_start_binds_its_handler, test/test_channel_boot_hot_reload.py::TestCredentialChangeRestartsChannel::test_a_credential_change_restarts_the_owning_channel, test/test_channel_boot_hot_reload.py::TestCredentialChangeRestartsChannel::test_a_credential_change_for_a_degraded_channel_is_deferred_not_dropped, test/test_channel_boot_hot_reload.py::TestSlackCredentialChangeWarns::test_a_slack_token_change_logs_a_restart_needed_notice (head 10ae360)

  • test/test_env_credential_hot_reload.py: an env-only edit dispatches credentials.<KEY> and applies os.environ; a failed apply leaves the baseline so a repair re-applies the rotation; an undecodable or wide-encoded .env is retried rather than read as every key cleared; a removal pops only what .env propagated; a .env PATH edit is expanded on a worker thread, not the loop thread; a Windows case-duplicate keeps its entry when one spelling is removed; a live add that matches an inherited value does not take ownership, so deleting the line keeps the inherited token, while an add that writes a new value does own it and is popped on removal; both spellings edited in one save resolve to boot's precedence in either iteration order, modelled with a case-folding os.environ; a reorder of two Windows spellings with different values is a diff and installs the new winner, while a reorder of equal values is not.
  • test/test_dotenv_path_extension.py: a .env PATH line appends absolute directories behind the existing PATH, deduped; relative and unexpanded entries are dropped with a warning; ~ and ~otheruser still expand through os.path.expanduser.
  • test/test_channel_boot_hot_reload.py: a credential change restarts only the owning channel; a degraded channel's change is deferred, not dropped; a Slack token change logs the restart notice; closing Teams clears the late-bound webhook handler; a Teams restart superseded mid-connect never binds its webhook handler while an accepted one does; a lowercase Slack key on Windows still logs the notice.
  • test/test_weixin_qr.py: a first sign-in records ownership, so deleting the line afterwards revokes the token in-process.
  • test/test_teams_gateway.py: maybe_start_teams registers the transport but leaves the webhook handler unbound for the host.
  • test/test_channel_env_write_off_loop.py: the dashboard writer honours the Docker credentials scrub.

Local on head 10ae360118 (Python 3.12): 2002 passed, sequential and parallel, across the env/config/channel/Teams/Slack/jira/loader suites; black, isort, flake8 clean on all 36 changed Python files; mypy clean on the nine changed source modules; scripts/check_black_formatting.py gate passes; flake-pattern ratchet passes against the merge base.

Manual verification

N/A — the watcher cycle, applier, and channel restart dispatch are each driven end to end by the ConfigWatch._cycle() tests above against a real temp .env, which is the path a hand edit takes.

Related Issues

Fixes #22450

Sibling sweep

Swept: git grep -nE "load_credentials(|apply_env_credential_changes|os.environ[.*] *=" -- src/
Sibling: src/kiro_crew/dashboard/messaging_api/channel_config_writes.py -- changed here
Sibling: src/kiro_crew/slack/gateway_runtime/channel_lifecycle.py -- changed here
Sibling: src/kiro_crew/dashboard/source_providers/jira.py -- changed here
Sibling: src/kiro_crew/dashboard/upload_destination.py -- changed here

Pattern harvest

Not generalizable: the defect was one missing watcher input (.env alongside config.json), not a repeated code shape a rule could catch.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable) — docs/system-specs/modules/config.md
  • No secrets, credentials, or internal references in the diff

Contribution License Agreement

Contributed under the project's Apache-2.0 license.

…22450)

Edits to $KIROCREW_HOME/.env took effect only after a full gateway
restart: the live config watcher fingerprinted config.json only and
nothing re-read .env after boot. The dashboard's own credential writer
applied its changes live, but a hand edit or any non-dashboard writer
did not.

ConfigWatch now reconciles .env in the same cycle that reloads
config.json. It tracks a separate env fingerprint
(loader.env_fingerprint), kept out of _config_fingerprint because that
one is the parsed-config cache key. On an .env change it re-reads the
raw credential map off the loop, diffs it against a baseline, and
pushes changed keys into os.environ (set/pop), dispatching the same
path a synthetic credentials.<KEY> change takes so the channel-restart
applier restarts only the channels whose descriptor credentials
changed (restart_channel, which re-reads load_credentials). Slack is
host-managed with no in-process restart path, so a Slack token change
logs a restart-needed notice instead. Teams gains a restart_credentials
descriptor field for keys (the tenant id) that are restart-relevant but
not required for readiness, since folding the tenant into `credentials`
would wrongly report multi-tenant Teams as un-startable.

Diff semantics, built up across review rounds and pinned by tests:
- A removed key is popped from os.environ only when the current value
  still equals what .env last propagated, so an inherited shell/systemd
  value a removal never set is left alone.
- The overwrite warning fires only on a genuinely foreign value
  collision, not on every routine rotation.
- The watcher's diff baseline is seeded from the raw .env map the
  first boot-time load_credentials() actually read (recorded once,
  consumed once), not a later disk read, so a credential rotated in
  the boot window is still diffed and applied instead of silently
  adopted as the new baseline.
- The fingerprint and baseline only advance after an apply succeeds;
  a failed apply (e.g. a malformed `=value` line in the same write as
  a real rotation) leaves both untouched so the next tick retries the
  whole diff, and the apply validates every key before mutating
  anything so a batch is atomic.
- Every post-boot credential read (channel reconnect, readiness badge,
  jira/doctor/upload-destination readers) now calls
  load_credentials(propagate=False), so a worker-thread read can no
  longer reseed a credential the live watcher just removed. The
  dashboard's own credential writer routes through the same
  apply_env_credential_changes instead of hand-rolling os.environ
  set/pop, so it inherits the Docker-scrub skip and PATH carve-out.
- A non-BOM decode error on .env is treated as non-definitive (warned
  once, retried) instead of wedging the watcher permanently.
- Teams: a credential removal closes the live client but its inbound
  webhook route stays registered, so TeamsClient now re-checks
  "closed" after the processing awaits (not just at entry) and 503s
  once closed; closing clears the bound activity handler.

Adds loader helpers (env_fingerprint, read_env_credentials_raw,
changed_env_keys, apply_env_credential_changes,
record_boot_raw_env/take_boot_raw_env) and
registry.changed_credential_keys (case-insensitive on Windows), plus
tests for the loader helpers, the watcher's .env tracking and boot-
window baseline, the credential-driven channel restart, and the
Teams webhook-after-close race.

Later review rounds, folded into this commit:
- A .env PATH line is expanded and validated off the event loop
  (ConfigWatch._prepare_path_additions -> env.prepare_env_path_additions,
  run via asyncio.to_thread) and handed to the applier as path_additions,
  so a ~otheruser entry's pwd.getpwnam / SAM lookup never blocks the loop.
  os.path.expanduser semantics are unchanged.
- The applier decides per os.environ ENTRY, not per .env spelling: on
  Windows (case-insensitive os.environ) all spellings of a key are grouped
  and resolved once, to the first non-empty spelling in .env order (boot's
  setdefault precedence), so renames, partial duplicate removals and
  both-spellings edits are no longer order-dependent.
- Ownership is recorded only when a write takes the entry over (new key,
  rotation, overwrite of a foreign value). A line that merely matches an
  inherited service/shell value is not owned, so removing it keeps the
  inherited credential.
- Teams: the webhook handler (state.teams_on_activity) is bound only once
  a client is the accepted handle (after boot's start_channels, and after
  a restart passes its generation check), so a restart superseded
  mid-connect never exposes its handler.
- Slack: the restart-needed notice matches credential keys by canonical
  entry name, like registry.changed_credential_keys.
@github-actions

github-actions Bot commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ✅ PASS

UX-level review of 10ae360118a439a58ef83cab8beb45e58796c6de — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

UX-Verdict: PASS

Nothing to check. The diff only changes backend code and adds no dashboard control, label or string, so no screenshot was needed. The changed log lines and the Teams webhook 503 never reach the dashboard UI.

[UX-REVIEWED] 10ae360

Description read

[DESCRIPTION-READ] b18d10fd42a6903b3ff7c41981198848fcad1364230fd03b5e97ed176acd76e3

That sha256 names everything this verdict read: the PR description, plus the 3 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@RohanK6 RohanK6 reopened this Oct 23, 2026
@github-actions

github-actions Bot commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of 10ae360118a439a58ef83cab8beb45e58796c6de — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

Nothing blocks. Watching .env from the poll that already reloads config.json, and sending every credential write through one applier, fixes the cause. The patch removes nothing that a reader outside it depends on.

Notes

  • The .env ownership record now crosses an in-app restart by being written into the environment as a marker (_carry_env_ownership_across_reexec). This is one more piece of state that has to stay in step with boot's setdefault. A later change that propagates .env without calling note_env_owned_key will quietly stop a removed credential from being unset.
  • The Teams webhook binding-order change (handler bound only once the client is accepted) is a separate concurrency fix bundled into this PR. If it causes a regression, reverting it alone will be harder than if it had its own commit.

[DESIGN-REVIEWED] 10ae360

Description read

[DESCRIPTION-READ] 590accd65ecdb3c97596a08e05cf0ef33d3002b5820c1251e3e5f99ea5b48e42

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — 🔴 changes requested (blocking)

GPT 6.1 found at least one blocking issue that must be resolved before merging 10ae360118a439a58ef83cab8beb45e58796c6de. 1 of 1 blocking finding(s) are security-class and were withheld from adjudication, so the blocking verdict stands.

This comment is updated in place on each push.

BLOCKING -- website/src/apps/meetings/hooks/useNoteDraft.ts:322 -- Pagehide save can be overwritten by an older autosave

if (inFlight && (direct || mode === 'unload')) return send(text, direct)
Trigger: Reload after typing newer text while an earlier autosave awaits server-side worker execution. pagehide → saveNow bypasses the mutation queue, letting the newer PUT reach handle_put_note → _save_note first.
Outcome: When the older PUT resumes, store.write_note unconditionally replaces the newer note. The document has discarded its draft, so reopening loses the latest edits; the metadata lock serializes writes but does not enforce edit order.
Anchor: residual/crash-data-loss-corruption
Fix: Include an editor ID and increasing save sequence in note PUTs, and reject stale writes inside _save_note’s transaction while retaining the direct pagehide send.

[BLOCK-MERGE] 10ae360
[GPT-REVIEWED] 10ae360

Adjudication (Opus 5.5) — is blocking on each finding proportionate?

F1 — the harm claim holds: the user's newest note edits are lost with no warning. This file is not in this PR's diff (git diff 22c595e4 HEAD does not touch it), so this is existing behaviour, but that does not change the ruling.
Conditions confirmed: on pagehide, the save goes out even while an older save is still on the wire, and it skips the queue (useNoteDraft.ts:322). The server sends both PUTs to the thread pool (meeting_lifecycle.py:862), and the save only takes the metadata lock with no check of edit order (meeting_lifecycle.py:877-881). So if the older PUT gets the lock second, it overwrites the newer text. Nothing on the server rejects the older save, and the client cannot recover because the page is gone.
Rarity: reloading while an autosave is in flight is a normal user action, and lock order between the two pool threads is not guaranteed. Nothing about the conditions is contradictory, so this is not extreme enough for a FLAG.

[ADJUDICATION] 10ae360118a439a58ef83cab8beb45e58796c6de total=0 uphold=0 downgrade=0
[GPT-ADJUDICATED] 10ae360118a439a58ef83cab8beb45e58796c6de

[ADJUDICATION-FENCED] 10ae360118a439a58ef83cab8beb45e58796c6de fenced=1 flagged=0
UPHOLD-FENCED F1 website/src/apps/meetings/hooks/useNoteDraft.ts:322 -- A reload during an in-flight autosave sends a second PUT outside the queue, and _save_note (meeting_lifecycle.py:877) writes whichever request gets the lock last with no edit-order check, so the newest edits can be silently lost under ordinary use.
[GPT-ADJUDICATED-FENCED] 10ae360118a439a58ef83cab8beb45e58796c6de

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 10ae360118a439a58ef83cab8beb45e58796c6de: <one-sentence reason>

@github-actions

github-actions Bot commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ✅ PASS

Premise-level review of 10ae360118a439a58ef83cab8beb45e58796c6de — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

First-Principles-Verdict: PASS

Every item traces back to the reported harm: a .env edit was ignored until a restart. Two items add surface they don't need.

Notes

  • Item 3 — one consumer, generalized: I grepped restart_credentials across src/. Only one descriptor sets it (channels.py:113, Teams), and it is a tuple field on every ChannelDescriptor. Subtraction: shrink it to a single optional key, or drop the field until a second channel needs it.
  • Item 6 — undeclared: _ENV_OWNED_KEYS_REEXEC_MARKER = "_KIROCREW_ENV_OWNED_KEYS" adds a new process environment variable, serialize_env_owned_keys_for_reexec is called from both reexec_* paths, and the description never mentions either. Subtraction: defer the re-exec carry. Without it, a .env line removed after an in-app restart stays set until a full restart, which is the same behaviour as today.

What this change ships

Inventory (10 items) — 5 justified

Intent: a credential an operator edits in .env takes effect on the running gateway without a full restart. This is a FIX. The change has more than 10 items: the PATH prep off the loop, the Windows case grouping and the boot-window baseline are left out.

  1. A hand edit to .env reaches the running gateway's environment within one poll — justified
  2. A channel whose credential changed restarts on its own — justified
  3. A Teams edit that changes only the tenant restarts Teams — one consumer, generalized
  4. A Slack token change logs a "run kirocrew restart" notice — justified
  5. Removing a .env line unsets only values .env itself set — justified
  6. That ownership record survives an in-app restart through a new environment variable — undeclared
  7. A dashboard save under the Docker scrub no longer writes credentials into os.environ — rides along
  8. The Weixin QR sign-in now writes its token through the shared applier — rides along
  9. The Teams webhook answers 503 until a client is accepted, and again after it closes — rides along
  10. Credential reads after boot no longer re-seed the process environment (28 call sites) — justified

[FIRST-PRINCIPLES-REVIEWED] 10ae360

Description read

[DESCRIPTION-READ] c35d1856c4f2ec4cb149a93dd6988a5ff0fe30dc12c41000a7d79a953c53df70

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed 10ae360118a439a58ef83cab8beb45e58796c6de — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 10ae360

Verdict parsed from the review's SHA-scoped output markers for commit 10ae360118a439a58ef83cab8beb45e58796c6de.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 10ae360118a439a58ef83cab8beb45e58796c6de: <one-sentence reason>

…ough the applier

changed_env_keys compared .env keys one at a time, so on Windows (where
several spellings share one case-insensitive os.environ entry) a save that
only reordered two spellings with different values produced an empty diff,
even though it changed which value boot and the live applier install. The
watcher then advanced its baseline without applying the new winner. It now
also compares each entry's effective value (first non-empty spelling in
.env order) and reports every spelling of an entry whose value moved. On
POSIX the result is the same per-key set as before.
(GPT 6.1 Review blocking finding.)

The first Weixin QR sign-in wrote WEIXIN_TOKEN into os.environ directly and
never recorded .env ownership, so removing the line later left the revoked
token in force until a full restart. The write now goes through
apply_env_credential_changes, like the dashboard channel writer, which
records ownership and keeps the Docker scrub guard.
(Opus 5.5 Review blocking finding.)
@RohanK6 RohanK6 closed this Oct 23, 2026
@RohanK6 RohanK6 reopened this Oct 23, 2026
@github-actions github-actions Bot added the readiness: action required A blocking check or review needs attention label Oct 23, 2026
@RohanK6

RohanK6 commented Oct 23, 2026

Copy link
Copy Markdown
Contributor Author

Superseded by #23514: the same change as one commit on current main. It adds the two structural-guard registrations this PR's CI flagged (config.loader facade manifest and the gateway composition inventory). With #23335 now in the base, the out-of-diff meetings finding falls outside the review range.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

readiness: action required A blocking check or review needs attention

Projects

None yet

Development

Successfully merging this pull request may close these issues.

.env changes require a full gateway restart (no hot-reload)

1 participant