Skip to content

fix(apps): serialize installed.json writers with a per-app record lock - #22796

Merged
iamwhatever merged 1 commit into
mainfrom
fix/22118-serialize-installed-writers
Oct 21, 2026
Merged

iamwhatever merged 1 commit into
mainfrom
fix/22118-serialize-installed-writers

Conversation

@iamwhatever

@iamwhatever iamwhatever commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: Two writers of one app's installed.json can no longer overlap, so a stale save cannot undo another writer's change, such as a sessionApproval consent-pending mark.

Every record writer reads the record, changes it and replaces the whole file. Nothing serialized them: a dev-mode toggle (which takes no app_lifecycle_lock) or a provenance save could read the record, a widening update_app could then write it disabled and pending, and the first writer's older copy would land on top as enabled: true, sessionApprovalConsentPending: false. app_can_manage_session_approvals then allows session control with no consent moment. The race was already on main; #22067 narrowed it but could not close it.

Why it matters

The sessionApproval grant lets an app approve tool calls and send turns in the user's own sessions. A lost pending mark grants that with no consent. The same lost-update shape can drop any other field a concurrent writer set.

Not a goal

What changed (motivation → approach → change)

Symptom: a stale record save lands after a widening update. Cause: writers read-check-replace with no exclusion. Change: one per-app, cross-process lock held across each writer's read, checks and replace.

  • manager_records.py: app_record_lock(name) takes platform_compat.file_lock (POSIX flock, Windows msvcrt.locking, bounded, fail-closed) on a sidecar, <data home>/app-record-locks/<name>.lock. Not installed.json, which os.replace swaps to a new inode, and not inside the app tree, which update_app moves aside. It is reentrant within one thread, so a transaction that calls _write_installed does not deadlock on a second descriptor. @holds_app_record_locks plus hold_app_record_lock(name) let a writer take it right before its first read and keep it until the function returns; outside such a function the call refuses. _write_installed takes it too, around the save rule and the replace.
  • Each writer takes it before its first read of the record: install_app, update_app (before its old-manifest read, across the tree swap and rollback), register_external_app (across its manifest-plus-record transaction and rollback), enable_app, disable_app, mark_session_approval_consent_pending, set_dev_mode (dev_mode.py), set_app_source, set_app_provenance (manager_lookup.py), backfill_default_on_builtins and register_builtin_apps (manager_builtins.py). manager.py lists the new names in the facade.
  • The lock is single-shot on the event-loop thread, so the async callers that ran a writer on the loop now await it in a worker thread: handle_enable_app (enable and its rollback disable) and handle_disable_app in routes_enable.py, and install_from_registry's two set_app_provenance calls in registry_pipeline/install.py. Its register_external_app call stays on the loop: a worker would keep writing after the install task is cancelled while the finally restores the old checkout; on the loop a contended lock refuses before the record is read, and the route already holds the app's lifecycle lock. Gateway first-run setup (backfill_default_on_builtins) stays on the loop too, before readiness; a contended acquire there skips the backfill until the next start.
  • docs/system-specs/modules/app-kit-platform.md describes the lock.

Backwards compatibility

Compatible: no record field, route or CLI changes. A writer that cannot take the lock within file_lock's ceiling now raises instead of writing unserialized; on the event-loop thread the acquire is single-shot, as file_lock is everywhere. Each critical section is a few file reads and one replace, except update_app, whose lock spans its tree copy.

Removes nothing: every write still happens, now one at a time per app.

Tests

test/test_app_record_lock.py:

  • test_a_dev_mode_save_cannot_undo_a_widening_update: a dev-mode save pauses at its record replace; a widening update_app starts and must wait on the record lock; after release the record ends disabled and pending and the grant is refused. Fails without the lock (the update never waits).
  • test_another_process_cannot_take_a_held_lock: a child process cannot take a held lock and can take a free one, on POSIX and Windows.
  • test_the_lock_is_reentrant_in_one_thread, test_holding_outside_a_locked_writer_refuses, test_the_lock_lives_outside_the_app_tree.
  • test_apps_routes_coverage_enable_disable.py::TestRecordWritersRunOffTheLoop: the enable and disable routes call their writers off the event loop.
  • test_every_record_writer_takes_the_lock_before_it_reads: every function in kiro_crew/apps that calls _write_installed is decorated and calls hold_app_record_lock before its first record read or write.

Determinism: the file passed 10/10 repeated runs and in reversed order; the interleave waits on events through kiro_crew.testing.wait.wait_until, not on time.

Manual verification

N/A — the cross-process behaviour is covered by the child-process test; there is no UI change.

Related Issues

Closes #22118

Pattern harvest

Rule candidate: review-prompt
Pattern: a read-modify-write of a shared state file with no lock spanning the read and the replace; lock a sidecar, never the replaced file.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@iamwhatever
iamwhatever requested review from a team and KiroCrewCaptain as code owners October 21, 2026 15:22
@iamwhatever
iamwhatever requested a review from CrysisDeu October 21, 2026 15:22
@github-actions

github-actions Bot commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision 8bcb20503bd570e149852fefd62930e8d0ca073c touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions

github-actions Bot commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ✅ PASS

Premise-level review of 8bcb20503bd570e149852fefd62930e8d0ca073c — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

First-Principles-Verdict: PASS

Nothing blocks. The fix works at the cause: no lock spanned a writer's read and its replace. It covers all 18 _write_installed call sites, and #22118 plus a test that fails on base give it provenance.

What this change ships

Inventory (7 items) — 7 justified

Intent: stop two concurrent writers of one app's installed.json from undoing each other's change, such as the sessionApproval consent-pending mark. FIX.

  1. Every writer of the record (install, update, external registration, enable, disable, dev mode, provenance, builtin sync, consent reconcile) now waits for any other writer of the same app, including across processes — justified
  2. New lock files persisted under <data home>/app-record-locks/ — justified
  3. A writer that cannot take the lock now raises instead of writing — justified
  4. The enable and disable routes now do their record writes in a worker thread instead of on the event loop — justified
  5. The two registry-install provenance saves now run in a worker thread — justified
  6. On first gateway start, a contended lock skips the default-on builtin backfill until the next start — justified
  7. New public app_record_lock, hold_app_record_lock and @holds_app_record_locks, used by 11 writers — justified

[FIRST-PRINCIPLES-REVIEWED] 8bcb205

Description read

[DESCRIPTION-READ] 4ac7534742b7ff80e917fbe02fe00beae3d2dff8a93cf921767b66aa93dc75c9

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of 8bcb20503bd570e149852fefd62930e8d0ca073c — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

No design-level concerns: one per-app sidecar lock across each writer's read and replace removes the lost-update cause, and the lock order is consistent.

Notes

  • Lock order is consistent: the lifecycle lock is taken before the record lock, which is taken before the dev-mode sentinel lock. Neither of those other locks is ever held while a record is written, so the new lock adds no deadlock path. A writer added later must keep this order; the spec could say so.
  • Two callers stay on the event loop, where the lock is single-shot: register_external_app, and the gateway first-run backfill. If another writer holds the lock, the install fails and the backfill is skipped until the next start. Both outcomes are reversible.

[DESIGN-REVIEWED] 8bcb205

Description read

[DESCRIPTION-READ] 980a9810171b2e1d3f9c0e7393d996a50ab578098ed1b9294bfa18ad30eb1e11

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Oct 21, 2026
@github-actions

github-actions Bot commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of 8bcb20503bd570e149852fefd62930e8d0ca073c and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 8bcb205

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 8bcb20503bd570e149852fefd62930e8d0ca073c: <one-sentence reason>

@github-actions

github-actions Bot commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed 8bcb20503bd570e149852fefd62930e8d0ca073c — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 8bcb205

Verdict parsed from the review's SHA-scoped output markers for commit 8bcb20503bd570e149852fefd62930e8d0ca073c.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 8bcb20503bd570e149852fefd62930e8d0ca073c: <one-sentence reason>

@github-actions github-actions Bot added the merge conflict Branch has merge conflicts with its base — author must resolve before merge label Oct 21, 2026
@iamwhatever
iamwhatever force-pushed the fix/22118-serialize-installed-writers branch from 2e5337d to 049fcdc Compare October 21, 2026 16:32
@iamwhatever

Copy link
Copy Markdown
Contributor Author

First Principles review on 2e5337d: item 6 (the single-caller _write_installed_locked split) is fixed in 049fcdc. The body is now inlined in _write_installed under with app_record_lock(name):. No other items needed action.

@github-actions github-actions Bot removed the merge conflict Branch has merge conflicts with its base — author must resolve before merge label Oct 21, 2026
@iamwhatever
iamwhatever force-pushed the fix/22118-serialize-installed-writers branch from 049fcdc to ec05e0c Compare October 21, 2026 16:59
@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Oct 21, 2026
@iamwhatever

Copy link
Copy Markdown
Contributor Author

Review-ready at ec05e0c (single commit on main cd4d536).

  • Fix for Apps: serialize installed.json writers so a stale save cannot undo a sessionApproval consent-pending mark #22118: every _write_installed caller (12 functions under apps/) now takes a per-app, cross-process lock before its first record read and holds it through the replace, including the install/update/external-registration manifest-plus-record transactions. The lock file is a sidecar at config_dir()/app-record-locks/<name>.lock, so the os.replace of installed.json never swaps the locked inode (per @xg-gh-25). It uses platform_compat.file_lock: flock on POSIX, msvcrt on Windows, bounded and fail-closed.
  • Event-loop callers (enable/disable routes, registry provenance) run the locked writers through asyncio.to_thread.
  • Regression test test/test_app_record_lock.py interleaves a dev-mode save with a widening update and asserts the record stays disabled and pending; it fails on main. It also covers child-process contention and has an AST check that each writer locks before reading.
  • CI: PR Readiness pass, 104 checks, 0 failing. GPT and Opus fresh with no blocking findings; Design and First Principles PASS. Main has since gained 4 commits that touch no file in this PR (one sibling doc in the same docs dir). The merge is clean, and docs lint plus the lock tests pass on the merged tree.

@iamwhatever
iamwhatever force-pushed the fix/22118-serialize-installed-writers branch from ec05e0c to 8bcb205 Compare October 21, 2026 18:07
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: passed Eligible automated validation passed for the current revision labels Oct 21, 2026
@iamwhatever

Copy link
Copy Markdown
Contributor Author

Rebased onto main b27a601. The new head is 8bcb205. Main did not change any file this PR touches, so main's changes and the lock are both kept as they were. Related app tests pass (6414 passed). Waiting on CI.

@buluoray buluoray left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed at 8bcb205. The review below was done at ec05e0c; this head is a rebase onto b27a601 and the PR's own diff is identical line for line.

Approve. The lost update is real on main: set_dev_mode re-read the record only under the dev sentinel lock, which update_app never takes, so a dev-mode save could land after a widening update and drop its consent-pending mark. The per-app record lock closes that.

Checked against the platform-compat rules: the lock goes through platform_compat.open_create_or_existing and platform_compat.file_lock (apps/manager_records.py:268-270), on an empty sidecar outside both installed.json and the app tree, and nothing is written through the lock descriptor, so the Windows range lock holds. Holds are released through an ExitStack on return and on error, reentrancy is thread-local so _write_installed never opens a second descriptor, and both refusal paths fail closed. Lock order is consistent: record lock then sentinel lock in set_dev_mode, no sentinel holder writes a record, the async lifecycle lock is always outside the record lock, and the multi-lock holders walk a fixed order. Readers stay lock-free, which is safe because every write is an atomic replace. Enable, disable and set_app_provenance now run off the loop; the on-loop register_external_app refuses at once under contention and the generic except plus the finally restore the checkout.

Ran the record-lock, enable/disable route, dev-mode, manager and registry-install tests (293 passed). Mutations: a no-op hold, a dropped hold in set_dev_mode or update_app, no reentrancy, the enable route back on the loop, and the lock inside the app tree each turn a test red.

Follow-ups, not blocking:

  1. Test gap: the race test also passes when hold_app_record_lock does nothing, because the dev thread pauses inside atomic_write, which is already under the lock in _write_installed. Pausing it between its re-read and its write would show the lock spans the whole transaction. Removing the lock inside _write_installed is also not caught by any test.
  2. backfill_default_on_builtins is reached from run_first_run_setup, which _init_services calls on the event loop, so its acquire is a single attempt. On contention it is skipped for that boot with a warning. Running it off the loop would let it wait.
  3. The comment above the on-loop register_external_app in registry_pipeline/install.py says a contended lock takes the refusal path below; it actually raises and is handled by the generic except.
  4. Lock sidecars are never removed on uninstall. They are empty and harmless, so this is only housekeeping.

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Oct 21, 2026
@iamwhatever
iamwhatever enabled auto-merge (squash) October 21, 2026 21:49
@iamwhatever
iamwhatever merged commit 90079d0 into main Oct 21, 2026
105 checks passed
@iamwhatever
iamwhatever deleted the fix/22118-serialize-installed-writers branch October 21, 2026 21:50
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Oct 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Apps: serialize installed.json writers so a stale save cannot undo a sessionApproval consent-pending mark

2 participants