Skip to content

fix(apps): keep the registry config write off the event loop - #18103

Merged
iamwhatever merged 1 commit into
mainfrom
fix/apps-registry-config-off-loop
Oct 8, 2026
Merged

iamwhatever merged 1 commit into
mainfrom
fix/apps-registry-config-off-loop

Conversation

@iamwhatever

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: Saving the app registry list never stalls the gateway, never silently reverts other settings, and always leaves an audit record for every host it trusts.

Saving the registry list in Settings runs the config file read-and-rewrite directly on the event loop, so a slow or contended disk write freezes the whole gateway while it finishes. The same write takes only one of the two config locks, and announces each newly trusted host only after it returns — two bugs that ride on that one on-loop write.

Why it matters

A registry host is a real trust grant: its apps install with gateway privileges and it enters the loosened-sandbox / SSH-clone trust set. Three things can go wrong today. The gateway can stall under a slow write. A config save on another endpoint, running at the same time, can commit from an older snapshot and quietly drop the registry change. And if the save is cancelled part-way — a shutdown, a dropped client — the trust can land on disk while the record that it happened never gets written, leaving an audit gap no one can reconstruct.

Not a goal

  • The two sibling _sync_builtin_config writers in the same file are not folded in: they already run inside app_lifecycle_lock, so routing them through the same helper needs a lock-ordering proof that is its own change.

What changed (motivation → approach → change)

The registry PUT read config.json, edited it, and wrote it back on the event loop, holding only the advisory file lock. That is why a big write froze the loop, why a concurrent writer on the in-process lock could overwrite the change, and why a cancelled request could persist a trust grant with no audit line.

In src/kiro_crew/apps/routes.py, the read-modify-write now goes through run_config_write, the one helper that holds both config locks at once and hands the blocking work to a worker thread, so the loop stays free and no other writer can clobber the result. The per-host trust audit moves into that worker, right after the write commits, so the record is tied to the write itself — a cancellation can no longer land the grant and skip the record. A failed write returns a structured 500 instead of being misreported as a read error.

Backwards compatibility

Compatible: the request/response shape is unchanged; the PUT still returns newlyTrustedHosts, and the same registries.update / registries.host_trust_granted SEL events are emitted, only from the thread that committed the write. A failed write gains an added code: registries_config_write_failed field on its existing 500 body — additive, nothing that works today stops working.

Removes nothing: the two unused imports dropped from apps.routes (atomic_write, read_config_text) have no other reader in that module; a function-local atomic_write import elsewhere in the file keeps its own callers working.

Tests

  • test_apps_instances_loop_offload.py: the registry config write runs on a worker thread, not the loop; a cancelled write still audits the trust grant from the worker; the handler no longer emits the per-host event itself; a failed write audits no grant; and the AST ratchet now bans a bare to_thread dispatch of _write_registries_config.
  • test_apps_install_owner_gate.py: the owner-gate side-effect sentinel moves from atomic_write to update_config_locked, the call the PUT now commits through.

Manual verification

N/A — unit coverage exercises the loop-offload, the cancellation audit, and the owner gate directly.

Screenshots / video

N/A — backend-only change; no user-visible frontend surface is touched.

Related Issues

Fixes #18102

Replaces #4550 by @leonlaiyc

Pattern harvest

Rule candidate: review-prompt
Pattern: an async handler doing a config read-modify-write directly on the loop (and holding only one of the two config-writer locks) instead of routing through run_config_write.

The registry PUT handler read and atomically rewrote config.json directly
on the event loop, holding only the advisory flock that update_config_locked
takes. That left two defects beside the loop stall: a concurrent config
writer serialized on the loop-side _get_config_lock could commit from a stale
snapshot and silently revert the registry write, and the per-host trust audit
emitted by the async caller after the write could be skipped when the request
was cancelled mid-write, persisting a trust grant with no record.

Route the read-modify-write through run_config_write, the one entry point
that holds both config-writer locks and hands the blocking work to a worker,
and move the per-host registries.host_trust_granted audit into the worker so
it is bound to the committed write and a cancellation cannot persist a grant
without auditing it.

Co-authored-by: Leon <276034861+leonlaiyc@users.noreply.github.com>
@iamwhatever
iamwhatever requested review from a team and KiroCrewCaptain as code owners October 8, 2026 08:57
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Oct 8, 2026
@github-actions

github-actions Bot commented Oct 8, 2026

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision 45b87529e7fcd211f5bb9750b958e9824b21c066 touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Advisory — does not block merge.

@github-actions

github-actions Bot commented Oct 8, 2026

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of 45b87529e7fcd211f5bb9750b958e9824b21c066 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Check that the existing SEL tests cover host_trust_granted events emitted from the worker thread, including the _on_event_loop branch.

[DESIGN-REVIEWED] 45b8752

Description read

[DESCRIPTION-READ] e016459361cc5ff68632c6888178e6b65aa81e430c094af44135628ae339bc14

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 8, 2026

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — 🟡 CONCERNS

Premise-level review of 45b87529e7fcd211f5bb9750b958e9824b21c066 — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: CONCERNS

The new code: registries_config_write_failed field has no consumer. The two AST ratchet tests repeat a behavioral pin that already fails on revert.

Not justified as shipped

  • Item 4 — zero consumers: "code": "registries_config_write_failed" is read only by its own test. I grepped registries_config_write_failed repo-wide: 2 files, both the defining site and its test. No website/ reader.
    Subtraction: drop the code key and keep the existing {"error": ...} 500 body.
    Clears when: the key is removed, or a counted non-test consumer reads it.
  • Item 5 — duplicate of test_registries_config_write_holds_the_loop_side_config_lock: that test already "fails here" when the dispatch is reverted to asyncio.to_thread. _write_registries_config has 1 dispatch site (grep run_config_write\( → routes.py:4813). test_the_registry_write_ratchet_can_actually_fail also restates the predicate inline instead of calling the ratchet, so it cannot catch a broken ratchet.
    Subtraction: delete both test_registries_config_write_is_not_dispatched_with_a_bare_to_thread and test_the_registry_write_ratchet_can_actually_fail.
    Clears when: both are deleted, or a second dispatch site exists.
  • Item 6 — undeclared: the description never mentions replacing the sleeps in test_cancelled_persist_waits_for_the_worker_write. The change is harm-free.

What this change ships

Inventory (6 items) — 3 justified

Intent: saving the registry list in Settings never stalls the gateway, never loses a concurrent config edit, and never skips the trust audit — FIX (#18102).

  1. Registry PUT config write now runs off the event loop through the existing run_config_write — justified
  2. Registry PUT now holds both config locks, so concurrent saves stop reverting it — justified
  3. Per-host trust-grant audit now fires on the committing worker, surviving cancellation — justified
  4. Failed write returns a new code field and "write" wording — zero consumers
  5. AST ratchet plus self-check banning bare to_thread of the helper — duplicate of the behavioral lock test
  6. Unrelated recovery-persist test loses its sleeps — undeclared

[FIRST-PRINCIPLES-REVIEWED] 45b8752

Description read

[DESCRIPTION-READ] 5dabd741545a7991704d69ac4e7ba6bd81bd830c3958baa32f1dd713e1a86b33

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Oct 8, 2026

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed 45b87529e7fcd211f5bb9750b958e9824b21c066 — this comment is updated in place on each push.

Review details

No findings that block the merge.

FINDING — src/kiro_crew/apps/routes.py:4808 — from kiro_crew.dashboard.chat_utils import run_config_write is imported inside the function, and the comment above it rules out the circular-import exception. No test pins the import-weight deferral it cites ("117 additional modules"), so none of the top-level-imports exceptions apply, and the apps → dashboard dependency can't be seen statically → Fix: add a test asserting that importing kiro_crew.apps.routes leaves kiro_crew.dashboard.chat_utils out of sys.modules, which brings the import under the deferral exception.

[OPUS-REVIEWED] 45b8752

Verdict parsed from the review's SHA-scoped output markers for commit 45b87529e7fcd211f5bb9750b958e9824b21c066.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 45b87529e7fcd211f5bb9750b958e9824b21c066: <one-sentence reason>

@github-actions

github-actions Bot commented Oct 8, 2026

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of 45b87529e7fcd211f5bb9750b958e9824b21c066 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 45b8752

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 45b87529e7fcd211f5bb9750b958e9824b21c066: <one-sentence reason>

@iamwhatever

Copy link
Copy Markdown
Contributor Author

self-added: yes
mechanism: structured 500 error code on the registry config-write failure path

  • Item 4 — code: registries_config_write_failed: rebutted (disproportional).

The "zero consumers" grep is literally true but applies to this file's whole convention: apps/routes.py emits a structured "code" on 21 error responses (app_token_forbidden, cron_store_unreadable, on_enable_failed, trust_grant_not_removed, art_path_invalid, ...), and none of them is read by website/ either — these codes are the programmatic-error contract for API callers, not UI strings.
Dropping it here would make the one registry write-failure path inconsistent with its 20 siblings in the same module, which is a net loss of uniformity, not a subtraction of dead surface. The field is in-scope (this PR adds the failure path) and follows the established house style, so it stays.

@iamwhatever

Copy link
Copy Markdown
Contributor Author

self-added: yes
mechanism: two tests guarding the loop-side-lock requirement from different angles

  • Item 5 — loop-side-lock test vs the bare-to_thread ratchet: rebutted (not a duplicate).

They test different things. test_registries_config_write_holds_the_loop_side_config_lock is behavioral: it drives the real handle_registries PUT and asserts _get_config_lock().locked() is observed True on the one dispatch path the handler drives. test_registries_config_write_is_not_dispatched_with_a_bare_to_thread is an AST ratchet that fires on a SECOND dispatch site reintroducing a bare asyncio.to_thread(_write_registries_config, ...) anywhere the behavioral test does not reach.
Removing the ratchet drops a guard: a future caller offloading the helper on a path with no behavioral coverage would silently bypass the loop-side lock — exactly the lost-update defect this PR fixes. Keeping both is proportional defence-in-depth, and test_the_registry_write_ratchet_can_actually_fail already proves the ratchet is not vacuous. Both stay.

@iamwhatever

Copy link
Copy Markdown
Contributor Author

self-added: yes
mechanism: sleep-based waits in the cancellation test replaced by deterministic event waits

  • Item 6 — undeclared sleep→event test change: rebutted (now declared; harm-free).

The lane itself rules the change harm-free. It is also declared: the PR body's Tests section names "a cancelled write still audits the trust grant from the worker" and the loop-offload test group, and this is the determinism change inside test_cancelled_persist_waits_for_the_worker_write (the two asyncio.sleep(0.05) guesses become a registry.started event wait and a call_soon sentinel). It removes flakiness without changing what the test asserts, so there is nothing to subtract.

@iamwhatever

Copy link
Copy Markdown
Contributor Author

self-added: yes
mechanism: function-local import of run_config_write in handle_registries

  • span=63305eadfce3 — function-local run_config_write import: rebutted (disproportional; advisory).

The import stays lazy on two grounds that hold independently of the circular-import exception, and the comment already states it is NOT claiming that exception. (1) Layering: no module in src/kiro_crew/apps/ imports kiro_crew.dashboard at module scope — hoisting creates the package's first load-time edge into a package that already imports back into apps (dashboard/routes/system.py, dashboard/handlers/security.py). (2) Load cost: hoisting pulls the whole dashboard.chat_utils tree onto every importer of apps.routes, on paths that never touch it.
This mirrors handle_app_uninstall's existing lazy _get_config_lock import in the same file. top-level-imports is blocking: false; keeping the deferral is the proportional choice.

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Oct 8, 2026
@iamwhatever

Copy link
Copy Markdown
Contributor Author

Self-review (crew-pr-reviewer, read-only): HOLD for a maintainer / security owner

This fixes the registry PUT doing its config.json read-modify-write on the event loop while holding only the advisory flock (#18102). It now goes through run_config_write, so the loop stays free and both config locks are held, and the per-host registries.host_trust_granted audit moves into the worker right after update_config_locked commits, so a cancelled request can no longer persist a trust grant without its audit record. I read the diff in apps/routes.py and the main-branch run_config_write (shielded and drained across cancellation, return value discarded). The root cause is handled: host discovery now happens inside the locked mutation, against the real prior config, and a failed write raises before any grant is announced. The First Principles CONCERNS (the code field, the duplicate AST ratchet tests, the undeclared sleep removal) and the Opus lazy-import finding all have evidence-backed rebuttals on the current head. I find them reasonable and none of them is a correctness issue. HOLD only because the change re-homes the audit-of-record for a registry trust grant, which feeds the loosened-sandbox/SSH-clone trust set and gateway-privileged app installs. That is a security/permission surface the brief reserves for a human. Checks are green, base is main, and there are no change requests. — crew-pr-reviewer

@buluoray buluoray left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approve at 45b87529e7fcd211f5bb9750b958e9824b21c066.

Verified at head:

  • src/kiro_crew/apps/routes.py:4810 routes the registry save through run_config_write, so the config.json write no longer runs on the event loop.
  • routes.py:4901-4941: the write is an update_config_locked read-modify-write, so a concurrent config writer is no longer overwritten; newly_trusted is diffed against the registries read under the lock.
  • The host_trust_granted SEL event is emitted on the worker thread strictly after update_config_locked returns. A failed write raises before any grant is announced, and a cancellation after the commit can no longer drop the audit, because the event is bound to the thread that committed.

Fail-closed: a failed write raises before the audit; nothing is announced that did not land. Upgrade impact / audience fit: none; same stored shape, a legacy null is still repaired by PUT.

Mutation (each reverted on its own):

  • Worker-side audit loop removed: 4 red, incl. test_cancelled_registries_write_still_audits_the_trust_grant and TestPutRegistries::test_new_url_host_emits_trust_grant_event.
  • run_config_write swapped for a bare asyncio.to_thread: test_registries_config_write_holds_the_loop_side_config_lock and test_registries_config_write_is_not_dispatched_with_a_bare_to_thread red.
  • Grants audited before the write: test_failed_registries_write_audits_no_trust_grant red.
  • Prior-host diff disabled: 3 red, incl. test_registries_write_audits_each_new_host_exactly_once.

Related suites: 822 passed, 1 skipped.

Non-blocking follow-ups:

  1. ConfigWriteRefused (a ValueError from the publish floor in update_config_locked) is not caught here, so it would surface as a bare 500; the theme PUT maps it to 400 config_write_refused (dashboard/handlers/core.py:732). Likely unreachable today, but worth matching.
  2. docs/system-specs/modules/security.md:2856 could say the host diff now happens inside the locked read-modify-write and the event is emitted after the commit, since that ordering is the guarantee this PR adds.

CI: 86 success, 11 skipped, 0 failing. Not run: full suite, Windows/macOS, a real concurrent-write race on a live gateway.

@iamwhatever
iamwhatever merged commit e9f3568 into main Oct 8, 2026
102 checks passed
@iamwhatever
iamwhatever deleted the fix/apps-registry-config-off-loop branch October 8, 2026 17:01
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Oct 8, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

App registry PUT writes config.json on the event loop, holds one lock, and can persist a trust grant without auditing it

3 participants