Skip to content

fix(dashboard): offload eager spawn's agent-binding config load - #4118

Closed
leonlaiyc wants to merge 1 commit into
kirodotdev:mainfrom
leonlaiyc:fix/eager-spawn-config-load-offloop
Closed

leonlaiyc wants to merge 1 commit into
kirodotdev:mainfrom
leonlaiyc:fix/eager-spawn-config-load-offloop

Conversation

@leonlaiyc

@leonlaiyc leonlaiyc commented Aug 17, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: Eager spawn must load its agent-binding configuration without blocking the gateway event loop.

The current speculative spawn synchronously stats, reads, merges and validates configuration before its off-loop binding resolution.

Why it matters

Slow configuration storage or validation stalls all sessions sharing the gateway loop during speculative startup.

Not a goal

Changing loader migrations, scheduler gating, binding resolution or eager-session ownership policy.

What changed (motivation → approach → change)

Integrates onto upstream main 38d5688. Offloads the single current _eager_spawn agent-binding load through asyncio.to_thread. Preserves session_key resolution, captured bindings, app self-heal, explicit Crew Member handling, memory assignment, deferred resets, V2 refusal, model resolution and pre-allocation ownership guards.

The scheduler uses live.snapshot() and no longer performs a preceding synchronous load. This change therefore does not rely on migrations having already run. The loader still writes migrations: #7937 is merged as 752d08a and is an ancestor of this base. Current _persist_config_migration re-reads the document and applies only migration-owned deltas, using update_config_locked for contained paths and deferring a contended lock. It no longer serializes the stale whole cfg snapshot through cfg.save(). Redirected foreign configs retain their existing immediate-fresh-read delta behavior without creating a lock sidecar. Existing loader global publication semantics are unchanged, including ticket-ordered autocompact publication.

A real first eager worker load is held immediately before migration persistence; a concurrent locked setting update lands after its original read. The test confirms both the unrelated setting and migration survive. The source comment records only the current delta-write invariant.

Backwards compatibility

Compatible: the same configuration snapshot feeds the same binding and ownership flow.
Removes nothing: this changes the execution thread of one load without narrowing accepted configuration or sessions.

Tests

  • Eager config, current eager spawn, scheduler config-source/backoff, agent selection and execution catalog: 246 passed.
  • Config migration ordering and backup containment: 13 passed, 1 platform skip.
  • Legacy lazy-load migration: 40 passed.
  • New tests prove off-loop load, normal allocation, and rejection before allocation after replacement, agent/model/project/store changes or a real turn starting during load.
  • Changed-file Black, isort, flake8 and mypy passed. Diff-scoped comment-history gate and git diff --check passed. Session spec updated.

Measured on Python 3.13 on Windows. Fresh CI supplies broader cross-platform and repository gate results; no whole-tree clean claim is made.

Manual verification

N/A — deterministic bounded worker handshakes exercise thread identity, stale-work refusal and a real migration/config-writer interleave.

Related Issues

Closes #4117

Retains the original eager config offload intent while incorporating the merged config write-back ordering fix.

Pattern harvest

Rule candidate: review-prompt
Pattern: offloading a configuration read requires checking its migration writes; fresh-document deltas and writer-lock ordering must preserve concurrent settings.

Checklist

  • One logical commit
  • Focused tests passed
  • Self-review and owning spec updated
  • No secrets or unrelated source changes

Contribution License Agreement

@leonlaiyc
leonlaiyc requested a review from a team August 17, 2026 12:38
@github-actions github-actions Bot added fork Pull request from a fork (external contributor) readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Aug 17, 2026
@leonlaiyc
leonlaiyc force-pushed the fix/eager-spawn-config-load-offloop branch from 5feabb2 to c58fe84 Compare August 18, 2026 06:19
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Aug 18, 2026
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Aug 18, 2026
@bolichen97

Copy link
Copy Markdown
Contributor

Hi @leonlaiyc, a maintainer nudge on this one: it is carrying the readiness: action required label and has had no activity for about 4 days, so it is not moving toward merge.

Current state:

  • Base: 356 commits behind main
  • Merge state: mergeable, but the readiness gate is still red
  • Red signals:
    • Backend Tests (Windows) (2) (cancelled)
    • PR Readiness

Could you rebase onto the latest main and push a fix?

git fetch upstream           # or: git fetch origin, if this branch lives here
git rebase upstream/main
# resolve any conflicts, run the local gates below, then update the branch with
# a force-with-lease so the rebase lands without clobbering anyone else's work

The local gates to run before updating the branch:

black src/kiro_crew test && isort src/kiro_crew test
flake8 src/kiro_crew test && mypy src/kiro_crew
python -m pytest
cd website && npm run build && npm run test

A good part of this branch's redness is likely stale rather than a real defect: it predates a lot of what is now on main, and several of these gates (the Coverage Gate, the Windows shards, the AI review lanes) have changed since the last run here. A rebase alone often clears them. If something still fails afterwards and you believe it is a false positive, say so in a comment and we will take a look. If the change is no longer needed, feel free to close the PR.

leonlaiyc added a commit to leonlaiyc/KiroCrew that referenced this pull request Aug 23, 2026
Every config.json update in dashboard/handlers/memory.py reads the file,
changes one key and writes it all back on the event loop. The read, the
JSON parse, and write_config_atomically -- a tmp-file write plus a rename,
which can fsync -- all block, stalling every other session while they run.
Three sites do it: api_memory_settings, _write_embed_model_config, and
_set_migrated, which runs on EVERY boot while migrated is false and so
lands the stall exactly when the gateway is bringing sessions up. This is
the class the repo has been closing site by site (kirodotdev#4118, kirodotdev#3803, kirodotdev#4550), and
kirodotdev#4946's review named this module.

The whole transaction crosses over, never just the read. Offloading the
read alone would leave the write on the loop and insert a suspension point
between the read and the write-back while the file is unguarded on disk: an
external editor, a CLI command or another process landing in that gap would
be silently overwritten by a write derived from state nobody re-checked.
That gap is zero today because the sequence is synchronous, and it stays
zero because the worker performs the whole thing without yielding. The
existing per-config lock is held across the hop, so two coroutines still
cannot interleave.

Two of the three sites also hand-rolled a reader this module already
imports. read_config_for_update is the documented companion to
write_config_atomically with 27 call sites, and api_memory_settings uses it
200 lines above; _set_migrated and _write_embed_model_config instead caught
Exception around json.loads. The helper additionally refuses a non-object
top level, where the hand-rolled version accepted a list and then raised
AttributeError from setdefault -- a crash where a fail-closed refusal was
intended.

ConfigReadError is not swallowed by the helper: what to tell the user
differs per site, and each keeps exactly the behaviour it had -- skip and
retry next boot, raise ValueError, or answer 500 config_unreadable.

api_memory_settings now validates its body before the transaction. None of
that reads the config, and a 400 previously took the lock and abandoned it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
leonlaiyc added a commit to leonlaiyc/KiroCrew that referenced this pull request Aug 23, 2026
_write_env_updates stats and reads the whole .env, re-parses it line by
line, then creates a 0600 temp file, chmods it, writes and renames. All
synchronous file I/O, and six async channel config-save handlers call it.

Three already reached it through asyncio.to_thread -- telegram, teams,
wecom -- and three called it inline on the gateway loop: slack, discord and
webex, stalling every other session for the duration of a token save.

So this is not a missing convention but an existing one applied to half the
call sites. messaging.py already uses asyncio.to_thread 29 times, and with
three siblings doing it correctly nothing in the file said which half was
right, or stopped the next channel from copying the wrong one. It is the
class the repo has been closing site by site (kirodotdev#4118, kirodotdev#3803, kirodotdev#4550), and
kirodotdev#4946's review named this module.

The WHOLE call is offloaded, never a part of it: the read-modify-write is
one transaction, and a suspension point between the read and the rename
would let a concurrent writer's keys be dropped by a write derived from
lines nobody re-read. Keeping _write_env_updates one synchronous function
on one worker preserves that without depending on the caller, which is now
said on the function itself so the next channel inherits the reason and not
just the shape.

The regression pins all six channels rather than the three that moved,
since the defect was the split and not any one site.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
leonlaiyc added a commit to leonlaiyc/KiroCrew that referenced this pull request Aug 24, 2026
Every config.json update in dashboard/handlers/memory.py reads the file,
changes one key and writes it all back on the event loop. The read, the
JSON parse, and write_config_atomically -- a tmp-file write plus a rename,
which can fsync -- all block, stalling every other session while they run.
Three sites do it: api_memory_settings, _write_embed_model_config, and
_set_migrated, which runs on EVERY boot while migrated is false and so
lands the stall exactly when the gateway is bringing sessions up. This is
the class the repo has been closing site by site (kirodotdev#4118, kirodotdev#3803, kirodotdev#4550), and

The whole transaction crosses over, never just the read. Offloading the
read alone would leave the write on the loop and insert a suspension point
between the read and the write-back while the file is unguarded on disk: an
external editor, a CLI command or another process landing in that gap would
be silently overwritten by a write derived from state nobody re-checked.
That gap is zero today because the sequence is synchronous, and it stays
zero because the worker performs the whole thing without yielding. The
existing per-config lock is held across the hop, so two coroutines still
cannot interleave.

Two of the three sites also hand-rolled a reader this module already
imports. read_config_for_update is the documented companion to
write_config_atomically with 27 call sites, and api_memory_settings uses it
200 lines above; _set_migrated and _write_embed_model_config instead caught
Exception around json.loads. The helper additionally refuses a non-object
top level, where the hand-rolled version accepted a list and then raised
AttributeError from setdefault -- a crash where a fail-closed refusal was
intended.

ConfigReadError is not swallowed by the helper: what to tell the user
differs per site, and each keeps exactly the behaviour it had -- skip and
retry next boot, raise ValueError, or answer 500 config_unreadable.

api_memory_settings now validates its body before the transaction. None of
that reads the config, and a 400 previously took the lock and abandoned it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@bolichen97
bolichen97 enabled auto-merge (squash) August 24, 2026 06:59
leonlaiyc added a commit to leonlaiyc/KiroCrew that referenced this pull request Aug 24, 2026
Every config.json update in dashboard/handlers/memory.py reads the file,
changes one key and writes it all back on the event loop. The read, the
JSON parse, and write_config_atomically -- a tmp-file write plus a rename,
which can fsync -- all block, stalling every other session while they run.
Three sites do it: api_memory_settings, _write_embed_model_config, and
_set_migrated, which runs on EVERY boot while migrated is false and so
lands the stall exactly when the gateway is bringing sessions up. This is
the class the repo has been closing site by site (kirodotdev#4118, kirodotdev#3803, kirodotdev#4550), and

The whole transaction crosses over, never just the read. Offloading the
read alone would leave the write on the loop and insert a suspension point
between the read and the write-back while the file is unguarded on disk: an
external editor, a CLI command or another process landing in that gap would
be silently overwritten by a write derived from state nobody re-checked.
That gap is zero today because the sequence is synchronous, and it stays
zero because the worker performs the whole thing without yielding. The
existing per-config lock is held across the hop, so two coroutines still
cannot interleave.

Two of the three sites also hand-rolled a reader this module already
imports. read_config_for_update is the documented companion to
write_config_atomically with 27 call sites, and api_memory_settings uses it
200 lines above; _set_migrated and _write_embed_model_config instead caught
Exception around json.loads. The helper additionally refuses a non-object
top level, where the hand-rolled version accepted a list and then raised
AttributeError from setdefault -- a crash where a fail-closed refusal was
intended.

ConfigReadError is not swallowed by the helper: what to tell the user
differs per site, and each keeps exactly the behaviour it had -- skip and
retry next boot, raise ValueError, or answer 500 config_unreadable.

api_memory_settings now validates its body before the transaction. None of
that reads the config, and a 400 previously took the lock and abandoned it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
leonlaiyc added a commit to leonlaiyc/KiroCrew that referenced this pull request Aug 24, 2026
Every config.json update in dashboard/handlers/memory.py reads the file,
changes one key and writes it all back on the event loop. The read, the
JSON parse, and write_config_atomically -- a tmp-file write plus a rename,
which can fsync -- all block, stalling every other session while they run.
Three sites do it: api_memory_settings, _write_embed_model_config, and
_set_migrated, which runs on EVERY boot while migrated is false and so
lands the stall exactly when the gateway is bringing sessions up. This is
the class the repo has been closing site by site (kirodotdev#4118, kirodotdev#3803, kirodotdev#4550), and

The whole transaction crosses over, never just the read. Offloading the
read alone would leave the write on the loop and insert a suspension point
between the read and the write-back while the file is unguarded on disk: an
external editor, a CLI command or another process landing in that gap would
be silently overwritten by a write derived from state nobody re-checked.
That gap is zero today because the sequence is synchronous, and it stays
zero because the worker performs the whole thing without yielding. The
existing per-config lock is held across the hop, so two coroutines still
cannot interleave.

Two of the three sites also hand-rolled a reader this module already
imports. read_config_for_update is the documented companion to
write_config_atomically with 27 call sites, and api_memory_settings uses it
200 lines above; _set_migrated and _write_embed_model_config instead caught
Exception around json.loads. The helper additionally refuses a non-object
top level, where the hand-rolled version accepted a list and then raised
AttributeError from setdefault -- a crash where a fail-closed refusal was
intended.

ConfigReadError is not swallowed by the helper: what to tell the user
differs per site, and each keeps exactly the behaviour it had -- skip and
retry next boot, raise ValueError, or answer 500 config_unreadable.

api_memory_settings now validates its body before the transaction. None of
that reads the config, and a 400 previously took the lock and abandoned it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@bolichen97

Copy link
Copy Markdown
Contributor

Open PR relationship audit

This is a consolidated, point-in-time code-level audit note. It compares complete merge-base diffs and current/merged code; it does not treat a shared topic as duplication or partial coverage as completion.

Relationship findings

  • This PR is OVERLAPPING with PR #7161. The goals differ or the implementations can complement each other; this is not a duplicate claim. Recommended action for PR #4118: KEEP. Independent goals in adjacent code; neither subsumes the other and no conflict is expected. Files: src/kiro_crew/dashboard/chat_runner.py.
  • This PR is OVERLAPPING with PR #7937. The goals differ or the implementations can complement each other; this is not a duplicate claim. Recommended action for PR #4118: KEEP. PR #7937 removes the only blocking review finding's mechanism but implements none of PR #4118's behavior: the eager-spawn load is still on the loop in current main. Files: src/kiro_crew/config/loader.py.

No PR, Issue, label, branch, or review state was changed by the relationship-note portion of this audit.

@github-actions github-actions Bot added the merge conflict Branch has merge conflicts with its base — author must resolve before merge label Sep 6, 2026
@bolichen97
bolichen97 force-pushed the fix/eager-spawn-config-load-offloop branch from 8161898 to 4a63f79 Compare September 8, 2026 19:20
@bolichen97

Copy link
Copy Markdown
Contributor

Rebased onto main 6ae74179d by a maintainer as part of the 2026-09-08 open-PR audit. Branch was 797 commits behind and mergeable_state: dirty.

Conflicts (1 file):

  • src/kiro_crew/dashboard/chat_runner.py — main's fix(chat): honour global agent.model default for auto-created slots #8834 added loaded_cfg = cfg immediately after the synchronous cfg = KiroCrewConfig.load() in _eager_spawn. Resolved by keeping this PR's cfg = await asyncio.to_thread(KiroCrewConfig.load) (with its comment block) and re-applying main's loaded_cfg = cfg right after it, so _default_session_model still receives the loaded config.

Gates run locally on changed files: black, isort, flake8 all clean; test/test_eager_spawn_config_load_off_loop.py 3 passed; test/test_eager_spawn.py 64 passed.

Please review the resolution. A maintainer push makes the maintainer the last pusher, so under the repo's last-push rule a second approver is needed. Reply if anything looks wrong.

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention merge conflict Branch has merge conflicts with its base — author must resolve before merge readiness: checking Automated validation is still running labels Sep 8, 2026
auto-merge was automatically disabled October 4, 2026 05:48

Head branch was pushed to by a user without write access

@leonlaiyc
leonlaiyc force-pushed the fix/eager-spawn-config-load-offloop branch from 4a63f79 to 01eec1a Compare October 4, 2026 05:48
@leonlaiyc
leonlaiyc requested a review from a team as a code owner October 4, 2026 05:48
@leonlaiyc
leonlaiyc requested a review from buluoray October 4, 2026 05:48
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention merge conflict Branch has merge conflicts with its base — author must resolve before merge and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Oct 4, 2026

Copy link
Copy Markdown
Contributor

PR triage (2026-10-05): DISTINCT, needs a rebase

I compared the full merge-base diff (38d5688d96...01eec1a505) with origin/main af1eb612da.

  • Not done on main yet. _eager_spawn on main still calls cfg = KiroCrewConfig.load() synchronously on the loop (src/kiro_crew/dashboard/chat_runner.py:5548). No other open PR offloads this call site. Merged fix: order the config write-back migration against concurrent writes #7937 (PR) only added the prerequisite delta-write and lock ordering in _persist_config_migration. Merged fix(chat): honour global agent.model default for auto-created slots #8834 (PR) only added loaded_cfg = cfg, and this branch already keeps that line. Neither one replaces this change.
  • Conflict. git merge-tree against current main conflicts in one place: docs/system-specs/modules/session.md. Main added the paragraph on Continue/Project slot retention (fix(dashboard): re-check slot identity at every await before continue and project act #11467) at the same spot as this PR's new eager-spawn paragraph. Keeping both paragraphs fixes it. chat_runner.py and the new test merge cleanly.
  • Checked on current main plus this patch: test/test_eager_spawn_config_load_off_loop.py and test/test_eager_spawn.py give 120 passed. If the one production line goes back to the synchronous load, 8 of the new tests fail, so the tests do catch a regression.
  • I found no breaking change and no P0 bug.

Recommendation: REBASE onto current main, resolve the doc conflict, then KEEP for maintainer review. It still closes Issue #4117.

@github-actions github-actions Bot removed the merge conflict Branch has merge conflicts with its base — author must resolve before merge label Oct 5, 2026
@iamwhatever

Copy link
Copy Markdown
Contributor

Thank you, @leonlaiyc, for this fix. To get it merged against current main (the branch had drifted far behind and _eager_spawn moved), I opened a rebased in-repo replacement, #17944, which carries your change — the KiroCrewConfig.load() offload to asyncio.to_thread — with you credited via Co-authored-by. It is green and awaiting maintainer review there, so I'm closing this one in its favor. Appreciate the contribution!

@iamwhatever iamwhatever closed this Oct 8, 2026
@github-actions github-actions Bot removed the readiness: action required A blocking check or review needs attention label Oct 8, 2026
@mbriones98 mbriones98 removed the drive-to-green PR claimed by drive-to-green pipeline label Oct 22, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

fork Pull request from a fork (external contributor)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Speculative eager spawn stats, reads and jsonschema-validates config.json on the gateway event loop

5 participants