Repository navigation
Conversation
da5ba08 to
81ec689
Compare
Every config.json update in dashboard/handlers/memory.py reads the file, changes one key and writes it all back on the event loop. The read, the JSON parse, and write_config_atomically -- a tmp-file write plus a rename, which can fsync -- all block, stalling every other session while they run. Three sites do it: api_memory_settings, _write_embed_model_config, and _set_migrated, which runs on EVERY boot while migrated is false and so lands the stall exactly when the gateway is bringing sessions up. This is the class the repo has been closing site by site (kirodotdev#4118, kirodotdev#3803, kirodotdev#4550), and kirodotdev#4946's review named this module. The whole transaction crosses over, never just the read. Offloading the read alone would leave the write on the loop and insert a suspension point between the read and the write-back while the file is unguarded on disk: an external editor, a CLI command or another process landing in that gap would be silently overwritten by a write derived from state nobody re-checked. That gap is zero today because the sequence is synchronous, and it stays zero because the worker performs the whole thing without yielding. The existing per-config lock is held across the hop, so two coroutines still cannot interleave. Two of the three sites also hand-rolled a reader this module already imports. read_config_for_update is the documented companion to write_config_atomically with 27 call sites, and api_memory_settings uses it 200 lines above; _set_migrated and _write_embed_model_config instead caught Exception around json.loads. The helper additionally refuses a non-object top level, where the hand-rolled version accepted a list and then raised AttributeError from setdefault -- a crash where a fail-closed refusal was intended. ConfigReadError is not swallowed by the helper: what to tell the user differs per site, and each keeps exactly the behaviour it had -- skip and retry next boot, raise ValueError, or answer 500 config_unreadable. api_memory_settings now validates its body before the transaction. None of that reads the config, and a 400 previously took the lock and abandoned it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
_write_env_updates stats and reads the whole .env, re-parses it line by line, then creates a 0600 temp file, chmods it, writes and renames. All synchronous file I/O, and six async channel config-save handlers call it. Three already reached it through asyncio.to_thread -- telegram, teams, wecom -- and three called it inline on the gateway loop: slack, discord and webex, stalling every other session for the duration of a token save. So this is not a missing convention but an existing one applied to half the call sites. messaging.py already uses asyncio.to_thread 29 times, and with three siblings doing it correctly nothing in the file said which half was right, or stopped the next channel from copying the wrong one. It is the class the repo has been closing site by site (kirodotdev#4118, kirodotdev#3803, kirodotdev#4550), and kirodotdev#4946's review named this module. The WHOLE call is offloaded, never a part of it: the read-modify-write is one transaction, and a suspension point between the read and the rename would let a concurrent writer's keys be dropped by a write derived from lines nobody re-read. Keeping _write_env_updates one synchronous function on one worker preserves that without depending on the caller, which is now said on the function itself so the next channel inherits the reason and not just the shape. The regression pins all six channels rather than the three that moved, since the defect was the split and not any one site. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Every config.json update in dashboard/handlers/memory.py reads the file, changes one key and writes it all back on the event loop. The read, the JSON parse, and write_config_atomically -- a tmp-file write plus a rename, which can fsync -- all block, stalling every other session while they run. Three sites do it: api_memory_settings, _write_embed_model_config, and _set_migrated, which runs on EVERY boot while migrated is false and so lands the stall exactly when the gateway is bringing sessions up. This is the class the repo has been closing site by site (kirodotdev#4118, kirodotdev#3803, kirodotdev#4550), and The whole transaction crosses over, never just the read. Offloading the read alone would leave the write on the loop and insert a suspension point between the read and the write-back while the file is unguarded on disk: an external editor, a CLI command or another process landing in that gap would be silently overwritten by a write derived from state nobody re-checked. That gap is zero today because the sequence is synchronous, and it stays zero because the worker performs the whole thing without yielding. The existing per-config lock is held across the hop, so two coroutines still cannot interleave. Two of the three sites also hand-rolled a reader this module already imports. read_config_for_update is the documented companion to write_config_atomically with 27 call sites, and api_memory_settings uses it 200 lines above; _set_migrated and _write_embed_model_config instead caught Exception around json.loads. The helper additionally refuses a non-object top level, where the hand-rolled version accepted a list and then raised AttributeError from setdefault -- a crash where a fail-closed refusal was intended. ConfigReadError is not swallowed by the helper: what to tell the user differs per site, and each keeps exactly the behaviour it had -- skip and retry next boot, raise ValueError, or answer 500 config_unreadable. api_memory_settings now validates its body before the transaction. None of that reads the config, and a 400 previously took the lock and abandoned it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Every config.json update in dashboard/handlers/memory.py reads the file, changes one key and writes it all back on the event loop. The read, the JSON parse, and write_config_atomically -- a tmp-file write plus a rename, which can fsync -- all block, stalling every other session while they run. Three sites do it: api_memory_settings, _write_embed_model_config, and _set_migrated, which runs on EVERY boot while migrated is false and so lands the stall exactly when the gateway is bringing sessions up. This is the class the repo has been closing site by site (kirodotdev#4118, kirodotdev#3803, kirodotdev#4550), and The whole transaction crosses over, never just the read. Offloading the read alone would leave the write on the loop and insert a suspension point between the read and the write-back while the file is unguarded on disk: an external editor, a CLI command or another process landing in that gap would be silently overwritten by a write derived from state nobody re-checked. That gap is zero today because the sequence is synchronous, and it stays zero because the worker performs the whole thing without yielding. The existing per-config lock is held across the hop, so two coroutines still cannot interleave. Two of the three sites also hand-rolled a reader this module already imports. read_config_for_update is the documented companion to write_config_atomically with 27 call sites, and api_memory_settings uses it 200 lines above; _set_migrated and _write_embed_model_config instead caught Exception around json.loads. The helper additionally refuses a non-object top level, where the hand-rolled version accepted a list and then raised AttributeError from setdefault -- a crash where a fail-closed refusal was intended. ConfigReadError is not swallowed by the helper: what to tell the user differs per site, and each keeps exactly the behaviour it had -- skip and retry next boot, raise ValueError, or answer 500 config_unreadable. api_memory_settings now validates its body before the transaction. None of that reads the config, and a 400 previously took the lock and abandoned it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Every config.json update in dashboard/handlers/memory.py reads the file, changes one key and writes it all back on the event loop. The read, the JSON parse, and write_config_atomically -- a tmp-file write plus a rename, which can fsync -- all block, stalling every other session while they run. Three sites do it: api_memory_settings, _write_embed_model_config, and _set_migrated, which runs on EVERY boot while migrated is false and so lands the stall exactly when the gateway is bringing sessions up. This is the class the repo has been closing site by site (kirodotdev#4118, kirodotdev#3803, kirodotdev#4550), and The whole transaction crosses over, never just the read. Offloading the read alone would leave the write on the loop and insert a suspension point between the read and the write-back while the file is unguarded on disk: an external editor, a CLI command or another process landing in that gap would be silently overwritten by a write derived from state nobody re-checked. That gap is zero today because the sequence is synchronous, and it stays zero because the worker performs the whole thing without yielding. The existing per-config lock is held across the hop, so two coroutines still cannot interleave. Two of the three sites also hand-rolled a reader this module already imports. read_config_for_update is the documented companion to write_config_atomically with 27 call sites, and api_memory_settings uses it 200 lines above; _set_migrated and _write_embed_model_config instead caught Exception around json.loads. The helper additionally refuses a non-object top level, where the hand-rolled version accepted a list and then raised AttributeError from setdefault -- a crash where a fail-closed refusal was intended. ConfigReadError is not swallowed by the helper: what to tell the user differs per site, and each keeps exactly the behaviour it had -- skip and retry next boot, raise ValueError, or answer 500 config_unreadable. api_memory_settings now validates its body before the transaction. None of that reads the config, and a 400 previously took the lock and abandoned it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Every config.json update in dashboard/handlers/memory.py reads the file, changes one key and writes it all back on the event loop. The read, the JSON parse, and write_config_atomically -- a tmp-file write plus a rename, which can fsync -- all block, stalling every other session while they run. Three sites do it: api_memory_settings, _write_embed_model_config, and _set_migrated, which runs on EVERY boot while migrated is false and so lands the stall exactly when the gateway is bringing sessions up. This is the class the repo has been closing site by site (kirodotdev#4118, kirodotdev#3803, kirodotdev#4550), and The whole transaction crosses over, never just the read. Offloading the read alone would leave the write on the loop and insert a suspension point between the read and the write-back while the file is unguarded on disk: an external editor, a CLI command or another process landing in that gap would be silently overwritten by a write derived from state nobody re-checked. That gap is zero today because the sequence is synchronous, and it stays zero because the worker performs the whole thing without yielding. The existing per-config lock is held across the hop, so two coroutines still cannot interleave. Two of the three sites also hand-rolled a reader this module already imports. read_config_for_update is the documented companion to write_config_atomically with 27 call sites, and api_memory_settings uses it 200 lines above; _set_migrated and _write_embed_model_config instead caught Exception around json.loads. The helper additionally refuses a non-object top level, where the hand-rolled version accepted a list and then raised AttributeError from setdefault -- a crash where a fail-closed refusal was intended. ConfigReadError is not swallowed by the helper: what to tell the user differs per site, and each keeps exactly the behaviour it had -- skip and retry next boot, raise ValueError, or answer 500 config_unreadable. api_memory_settings now validates its body before the transaction. None of that reads the config, and a 400 previously took the lock and abandoned it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…5059) Every config.json update in dashboard/handlers/memory.py reads the file, changes one key and writes it all back on the event loop. The read, the JSON parse, and write_config_atomically -- a tmp-file write plus a rename, which can fsync -- all block, stalling every other session while they run. Three sites do it: api_memory_settings, _write_embed_model_config, and _set_migrated, which runs on EVERY boot while migrated is false and so lands the stall exactly when the gateway is bringing sessions up. This is the class the repo has been closing site by site (#4118, #3803, #4550), and The whole transaction crosses over, never just the read. Offloading the read alone would leave the write on the loop and insert a suspension point between the read and the write-back while the file is unguarded on disk: an external editor, a CLI command or another process landing in that gap would be silently overwritten by a write derived from state nobody re-checked. That gap is zero today because the sequence is synchronous, and it stays zero because the worker performs the whole thing without yielding. The existing per-config lock is held across the hop, so two coroutines still cannot interleave. Two of the three sites also hand-rolled a reader this module already imports. read_config_for_update is the documented companion to write_config_atomically with 27 call sites, and api_memory_settings uses it 200 lines above; _set_migrated and _write_embed_model_config instead caught Exception around json.loads. The helper additionally refuses a non-object top level, where the hand-rolled version accepted a list and then raised AttributeError from setdefault -- a crash where a fail-closed refusal was intended. ConfigReadError is not swallowed by the helper: what to tell the user differs per site, and each keeps exactly the behaviour it had -- skip and retry next boot, raise ValueError, or answer 500 config_unreadable. api_memory_settings now validates its body before the transaction. None of that reads the config, and a 400 previously took the lock and abandoned it. Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
|
🤖 Kiro Crew [operator: iamwhatever]: This PR has been inactive for 7+ days with failing CI. I've assessed the blockers and they appear resolvable — I'll push fixes directly to this branch as a co-author. Assessment: Two mechanical blockers, neither in the PR's own logic. (1) The branch is If you'd prefer I don't touch this PR, add the |
81ec689 to
bf0ae7c
Compare
|
👋 Hi! This PR's description is missing some required sections from our PR template. Workflow runs won't be auto-approved until the description is updated. Missing sections:
Please update your PR description to include these sections, then push or re-save the description. The workflows will be approved on the next cycle. |
3 similar comments
|
👋 Hi! This PR's description is missing some required sections from our PR template. Workflow runs won't be auto-approved until the description is updated. Missing sections:
Please update your PR description to include these sections, then push or re-save the description. The workflows will be approved on the next cycle. |
|
👋 Hi! This PR's description is missing some required sections from our PR template. Workflow runs won't be auto-approved until the description is updated. Missing sections:
Please update your PR description to include these sections, then push or re-save the description. The workflows will be approved on the next cycle. |
|
👋 Hi! This PR's description is missing some required sections from our PR template. Workflow runs won't be auto-approved until the description is updated. Missing sections:
Please update your PR description to include these sections, then push or re-save the description. The workflows will be approved on the next cycle. |
|
👋 Hi! This PR's description is missing some required sections from our PR template. Workflow runs won't be auto-approved until the description is updated. Missing sections:
Please update your PR description to include these sections, then push or re-save the description. The workflows will be approved on the next cycle. |
4 similar comments
|
👋 Hi! This PR's description is missing some required sections from our PR template. Workflow runs won't be auto-approved until the description is updated. Missing sections:
Please update your PR description to include these sections, then push or re-save the description. The workflows will be approved on the next cycle. |
|
👋 Hi! This PR's description is missing some required sections from our PR template. Workflow runs won't be auto-approved until the description is updated. Missing sections:
Please update your PR description to include these sections, then push or re-save the description. The workflows will be approved on the next cycle. |
|
👋 Hi! This PR's description is missing some required sections from our PR template. Workflow runs won't be auto-approved until the description is updated. Missing sections:
Please update your PR description to include these sections, then push or re-save the description. The workflows will be approved on the next cycle. |
|
👋 Hi! This PR's description is missing some required sections from our PR template. Workflow runs won't be auto-approved until the description is updated. Missing sections:
Please update your PR description to include these sections, then push or re-save the description. The workflows will be approved on the next cycle. |
Open PR relationship auditThis is a consolidated, point-in-time code-level audit note. It compares complete merge-base diffs and current/merged code; it does not treat a shared topic as duplication or partial coverage as completion. Relationship findings
No PR, Issue, label, branch, or review state was changed by the relationship-note portion of this audit. |
|
👋 Hi! This PR's description is missing some required sections from our PR template. Workflow runs won't be auto-approved until the description is updated. Missing sections:
Please update your PR description to include these sections, then push or re-save the description. The workflows will be approved on the next cycle. |
7885b24 to
0eb7742
Compare
|
Rebased onto main Clean rebase — no conflicts. The only delta from your old head is one stray Gates run locally on the rebased head: Please review the rebased head. A maintainer push makes the maintainer the last pusher, so under this repo's last-push rule a second approver is needed. Reply here if anything looks wrong. |
|
👋 Hi! This PR's description is missing some required sections from our PR template. Workflow runs won't be auto-approved until the description is updated. Missing sections:
Please update your PR description to include these sections, then push or re-save the description. The workflows will be approved on the next cycle. |
1 similar comment
|
👋 Hi! This PR's description is missing some required sections from our PR template. Workflow runs won't be auto-approved until the description is updated. Missing sections:
Please update your PR description to include these sections, then push or re-save the description. The workflows will be approved on the next cycle. |
Current main already offloads the enable and disable writers from the original change. Complete the remaining registry PUT path by moving its entire read-modify-write off the loop and using the shared locked config updater. Compute newly trusted hosts from the state protected by that lock, preserve unrelated config, deduplicate grants, and emit trust events only after the write commits. config.json has two writer generations that do not exclude each other: update_config_locked takes the sidecar advisory flock, while the legacy dashboard writers (agents endpoint, updates.py, security.py, messaging.py, mcp.py, core.py STT) serialize on the loop-side _get_config_lock alone. Dispatch the registry write through dashboard.chat_utils.run_config_write, the one entry point that holds both, so a concurrent legacy PUT cannot commit a stale snapshot over it. The blocking work still runs in a worker, so the loop never stalls. Add deterministic execution-level and AST coverage for worker-thread dispatch, loop-side lock ownership across the write, locked mutation, preserved data, grant deduplication, and failed-write audit behavior.
run_config_write shields its worker and drains it across cancellation, then re-raises CancelledError and discards the return value. A gateway shutdown or a client disconnect landing while the registry PUT's write is in flight therefore committed the trust grant and never reached the caller-side audit: persisted trust with no registries.host_trust_granted record, which is exactly the reconstruction gap that event exists to close. Emit the per-host grant inside _write_registries_config, immediately after update_config_locked returns. Nothing can interrupt a thread between those two statements, so the event and the commit cannot disagree; the emission is still strictly after the write, so a failed write announces nothing; and SEL adapts to an off-loop caller and swallows filesystem errors on a non-critical event, so this cannot turn a committed write into a 500. The handler keeps the registries.update API-outcome event, which correctly belongs to the request. Three tests: a cancellation delivered while the worker is parked inside the locked write (red-before: config committed, zero grants); exactly-once auditing per newly trusted host; and no grant when the locked write raises. Also corrects the run_config_write import comment, which implied the top-level-imports circular-import exemption. There is no cycle on that edge -- every import order was checked. The real reasons are layering and load cost, both now stated and checkable: no module in apps/ imports dashboard at module scope, and hoisting pulls 117 extra modules into every importer of apps.routes (479 -> 596). The import stays lazy; the justification is now accurate. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
0eb7742 to
6024990
Compare
|
CI note on head
No push for these. A re-run after #16801 lands should clear shard 7. |
|
Thank you, @leonlaiyc, for this fix — moving the registry config write off the event loop, holding both config-writer locks, and binding the per-host trust audit to the committed write so a cancellation can't persist a grant without a record. Because the base had moved on considerably since this branch was cut, I've reopened the change as a fresh main-repo PR rebased on current |
Problem / Motivation
Goal: Saving the app-registry list no longer blocks the gateway's event loop or silently reverts another settings save, and every trust grant it commits is audited.
PUT /api/apps/registries performed its config.json read-modify-write synchronously on the asyncio event loop and without holding both config-writer lock generations. A concurrent writer could therefore lose unrelated configuration.
The trust audit also had two opposite failure modes:
Current main already contains the earlier enable/disable offload work. This PR is intentionally limited to the remaining registry PUT path.
Why it matters
config.jsonholds every user setting, not only the registry list. A registry save that interleaves with any other dashboard settings save can silently revert that other save, and nothing reports it. The blocking read and write also run on the gateway's event loop, so a slow disk stalls every chat, channel and API request for that time. A registry host is a trust grant: its apps install with gateway privileges, and it joins the loosened clone trust set. A grant that lands without itsregistries.host_trust_grantedrecord leaves incident response unable to establish when a host entered the trust set.Not a goal
config.jsonwriter. This PR moves only the registry PUT.What changed (motivation → approach → change)
Still live on
main(b43b751).handle_registriesreadsconfig.jsonwithread_config_textand publishes it with a bareatomic_write, both inline on the loop and outsideupdate_config_locked. That spelling is not oneTestEveryConfigWriterIsLockedscans for. It audits each grant before the write, so a failed write still reports the grant.Approach.
config.jsonhas two writer generations:update_config_locked(cross-process sidecar flock) and the legacy dashboard writers that serialize on the in-process_get_config_lockonly. Holding one of them still lets the other family commit a stale snapshot over this write.dashboard.chat_utils.run_config_writeis the existing bridge that holds both and runs the blocking work in a worker. It also shields and drains that worker across cancellation, then re-raises and discards the return value, so the grant audit has to live in the worker that committed the write rather than in the caller.Change. In
src/kiro_crew/apps/routes.py:_write_registries_config, anupdate_config_lockedmutation. Newly trusted hosts are computed inside the lock, against the prior on-disk registries. Main'sor []repair idiom for an explicit"registries": nullis kept.handle_registriesdispatches it throughrun_config_write. A malformed or non-objectconfig.json(ConfigReadErrorwithout anOSErrorcause) keeps main's 500 "malformed" answer, and an unreadable one keeps "cannot read config". A failed write now answers a structured 500 withregistries_config_write_failedinstead of escaping as an unhandled exception.registries.host_trust_grantedevent is emitted in that worker right after the write commits. A failed write therefore emits no grant, and a cancelled caller can no longer leave persisted trust with no audit record. The request-levelregistries.updateevent stays with the request.run_config_writeimport stays call-time. No module inapps/importskiro_crew.dashboardat module scope, and hoisting it would load the dashboard tree into every importer ofapps.routes.atomic_writeandread_config_textimports are now unused here and are removed.This rebase (onto
b43b751, 3240 commits). It resolved two textual conflicts. Inroutes.py, main had added the non-object check:update_config_locked→read_config_for_updateraisesConfigReadErrorfor exactly that case, so the PR side was taken. In the test file, main's_mark_recovered("inst", inst_double, 0)signature is kept, with this PR's event-based wait. Two newer main tests needed adapting to the moved write:test_apps_install_owner_gate.py's side-effect sentinel wrappedroutes.atomic_write. It now wrapsroutes.update_config_locked, which is where the registry PUT commits.handle_registrieswith a bare request, and main'shandle_registrieshas since gainedrequire_owner_dashboard_request. They now admit the request through that gate, which they do not test;test_apps_install_owner_gate.pycovers it.The shared
apps/routes.pychanges in #6854 and #5488 are in separate semantic regions. #6206 is a broader App Store refresh that touches the same file.Backwards compatibility
Compatible: the PUT accepts the same body and returns the same success shape (
ok,registries,newlyTrustedHosts). The malformed and unreadable config answers are unchanged. The only new answer is a structured 500 for a failed write, which previously surfaced as an unhandled exception.Reader:
apps/routes.pymodule attributeatomic_write-- app -- removed as unused here. Its one reader was thetest_apps_install_owner_gate.pysentinel, moved toupdate_config_locked(test_owner_reaches_the_side_effect[registries_put]).spec_builder/tests/test_routes.pypatchesatomic_writeon spec_builder's ownroutesmodule, not this one.Reader:
apps/routes.pymodule attributeread_config_text-- app -- removed as unused here, with no reader onorigin/main.Tests
test/test_apps_instances_loop_offload.py, 23 passed, covering:_get_config_lock;threading.Eventboundary still producing the committed config and exactly one grant audit;asyncio.to_threadregression;_mark_recoveredcancellation test, driven by events and an event-loop sentinel instead ofsleep(0.05).test/test_apps_install_owner_gate.py(owner gate on all five install routes, includingregistries_put) passes with the moved sentinel. The two files together: 40 passed.With the related suites (
test_config_rmw_preserves_settings.py, which holds theTestEveryConfigWriterIsLockedratchet, plustest_apps_routes_coverage.py,test_registries_api.py,test_file_menu_items.pyandtest_token_auth.py), the total is 710 passed, 11 skipped, 0 failed. On the changed files,mypyonapps/routes.py,flake8,isort --check-only,black --checkand the comment-history gate are all clean.Manual verification
N/A — unit coverage sufficient: the change is a backend config-write path, and the tests drive the real
handle_registriesthroughrun_config_writeandupdate_config_lockedagainst a realconfig.json, including the cancellation window.Related Issues
No issue filed; this is the registry-PUT path left after the enable/disable offload work.
Pattern harvest
Rule candidate: semgrep
Pattern: a
config.jsonwrite spelled as a bareatomic_write(config_path, ...)bypassesupdate_config_lockedand is invisible to thewrite_config_atomically-matching ratchet.Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)Contribution License Agreement
🤖 Generated with Claude Code