Skip to content

fix(dashboard): settle and authorize the agent switch before it publishes - #12702

Closed
rnoack1 wants to merge 1 commit into
kirodotdev:mainfrom
rnoack1:split/agent-switch-settle-before-publish
Closed

rnoack1 wants to merge 1 commit into
kirodotdev:mainfrom
rnoack1:split/agent-switch-settle-before-publish

Conversation

@rnoack1

@rnoack1 rnoack1 commented Sep 22, 2026

Copy link
Copy Markdown
Contributor

Problem / Motivation

The agent switch resolves which session key its change should land on behind an await.
A rebind arriving inside that window is parked rather than applied, so the key the resolve
observed never moves.

The handler published the binding triple before that was settled. The result is a slot
whose slot.agent names the newly selected agent while the arm still names the key the slot
has already left. mark_retire_on_next_claim does not retro-correct a turn already in
flight, so a turn claiming the rebound key runs under an agent whose own switch can still
answer 409 and roll back underneath it.

Why it matters

The switch answers 200 and the UI shows the new agent, so nothing signals that the binding
was never armed on the key that will actually run. The failure surfaces a turn later, as an
agent running with the previous agent's arm, with nothing connecting it to the switch that
caused it.

Rolling back after the fact does not close the window: by then a concurrent send may already
have cold-started a replacement session on the new binding, and restoring the old agent would
advertise a binding nothing runs. Only ordering the settle and the authorization before the
commit removes the window.

What changed

The commit triple now follows the settle and the rebind authorization:

  • settled — the arm is transferred to the key that landed and the switch commits;
  • unsettled — nothing is published and the handler answers 409, so the caller retries
    against the current binding.

_settle_arm_target is the single place that resolves an arm target and reports whether it
settled, retracting the source arm when it does not. The workspace-root probe also moves
ahead of the commit, so an unavailable root answers 503 with nothing published to unwind —
unwinding there previously re-ran the same failed resolve and raised a 500.

Two consequences of that reordering are worth calling out for review:

  • _rollback_switch becomes async, and two call sites added since this branch point are
    now awaited. A bare call there builds a coroutine nobody awaits, so the rollback would
    silently never run.
  • the unwind guards that sit ahead of the commit hold no write of ours in this ordering, so
    committed_agent starts at the module's uncommitted sentinel and those guards stand down
    instead of unwinding a commit that has not happened.

The slot-allocation core, the arm verbs and the cleared-cwd marker are carried here
identically to their siblings (#11813, #11831, #11910, #11983, #12055) so this change
compiles and tests on its own.

Tests

Nine added cases across five new classes in test/test_chat_slot_switch_atomicity.py:

  • the settle and the gate both precede the commit triple;
  • an unsettled settle answers 409 and leaves the agent alone;
  • the arm transfer follows the commit with no await between them;
  • a never-scoped slot arms no directory but still retires the agent.

The no-await requirement is pinned twice over — once by inspecting the handler and once by a
structural scan of every commit site in the module. The scan is armed by its own synthesized
positive control, so it cannot pass vacuously on a module that happens to contain no commit
sites.

Break-armed, and the arms are each other's controls:

  • removing the pre-commit gate fails only the ordering case;
  • removing the unsettled 409 fails only the unsettled case;
  • inserting an await between the commit and the arm fails both no-await cases, so the
    inspection check and the structural scan cross-validate rather than duplicate;
  • making the helper arm the cleared default unconditionally fails only the never-scoped case.

Four existing test fixtures gain the async stub the carried pre-commit probe requires.

@github-actions github-actions Bot added the fork Pull request from a fork (external contributor) label Sep 22, 2026
@dwu96

dwu96 commented Sep 22, 2026

Copy link
Copy Markdown
Contributor

👋 Hi! This PR is currently in draft status. Workflow runs won't be auto-approved until it's marked as ready for review.

When you're ready, click "Ready for review" and the workflows will be approved on the next cycle automatically.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention merge conflict Branch has merge conflicts with its base — author must resolve before merge labels Sep 22, 2026
@rnoack1
rnoack1 force-pushed the split/agent-switch-settle-before-publish branch from bf30fa6 to 3dfed2d Compare September 22, 2026 07:26
@dwu96

dwu96 commented Sep 22, 2026

Copy link
Copy Markdown
Contributor

👋 Hi! This PR is currently in draft status. Workflow runs won't be auto-approved until it's marked as ready for review.

When you're ready, click "Ready for review" and the workflows will be approved on the next cycle automatically.

@github-actions github-actions Bot removed the merge conflict Branch has merge conflicts with its base — author must resolve before merge label Sep 22, 2026
@dwu96

dwu96 commented Sep 22, 2026

Copy link
Copy Markdown
Contributor

👋 Hi! This PR is currently in draft status. Workflow runs won't be auto-approved until it's marked as ready for review.

When you're ready, click "Ready for review" and the workflows will be approved on the next cycle automatically.

6 similar comments
@dwu96

dwu96 commented Sep 22, 2026

Copy link
Copy Markdown
Contributor

👋 Hi! This PR is currently in draft status. Workflow runs won't be auto-approved until it's marked as ready for review.

When you're ready, click "Ready for review" and the workflows will be approved on the next cycle automatically.

@dwu96

dwu96 commented Sep 22, 2026

Copy link
Copy Markdown
Contributor

👋 Hi! This PR is currently in draft status. Workflow runs won't be auto-approved until it's marked as ready for review.

When you're ready, click "Ready for review" and the workflows will be approved on the next cycle automatically.

@dwu96

dwu96 commented Sep 24, 2026

Copy link
Copy Markdown
Contributor

👋 Hi! This PR is currently in draft status. Workflow runs won't be auto-approved until it's marked as ready for review.

When you're ready, click "Ready for review" and the workflows will be approved on the next cycle automatically.

@dwu96

dwu96 commented Sep 24, 2026

Copy link
Copy Markdown
Contributor

👋 Hi! This PR is currently in draft status. Workflow runs won't be auto-approved until it's marked as ready for review.

When you're ready, click "Ready for review" and the workflows will be approved on the next cycle automatically.

@dwu96

dwu96 commented Sep 24, 2026

Copy link
Copy Markdown
Contributor

👋 Hi! This PR is currently in draft status. Workflow runs won't be auto-approved until it's marked as ready for review.

When you're ready, click "Ready for review" and the workflows will be approved on the next cycle automatically.

@dwu96

dwu96 commented Sep 24, 2026

Copy link
Copy Markdown
Contributor

👋 Hi! This PR is currently in draft status. Workflow runs won't be auto-approved until it's marked as ready for review.

When you're ready, click "Ready for review" and the workflows will be approved on the next cycle automatically.

@github-actions github-actions Bot added the merge conflict Branch has merge conflicts with its base — author must resolve before merge label Sep 24, 2026
@dwu96

dwu96 commented Sep 24, 2026

Copy link
Copy Markdown
Contributor

👋 Hi! This PR is currently in draft status. Workflow runs won't be auto-approved until it's marked as ready for review.

When you're ready, click "Ready for review" and the workflows will be approved on the next cycle automatically.

1 similar comment
@dwu96

dwu96 commented Sep 25, 2026

Copy link
Copy Markdown
Contributor

👋 Hi! This PR is currently in draft status. Workflow runs won't be auto-approved until it's marked as ready for review.

When you're ready, click "Ready for review" and the workflows will be approved on the next cycle automatically.

@bolichen97 bolichen97 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Request changes: resolved_cwd imports a function that does not exist, so every dashboard agent switch returns 503

src/kiro_crew/config/paths.py:385-387 (audited head 3dfed2d):

from kiro_crew.config.loader import session_default_cwd

return str(session_default_cwd(session_key))

This PR does not touch config/loader.py. At this head that file defines only _session_work_dir (src/kiro_crew/config/loader.py:536), and neither origin/main nor the merge base has session_default_cwd. The rename that adds it exists only in the open PRs #11722, #11813 and #11831. The description says the shared infrastructure is carried here "so this change compiles and tests on its own", but the loader rename was left out.

What breaks. api_chat_slot_agent (chat_handlers.py:6986, POST /api/chat/slots/{slot}/agent) gets past its early refusals on an ordinary local switch and always runs chat_handlers.py:7200:

cleared_arm_cwd = await state.sessions.resolve_arm_cwd(session_key, CWD_CLEARED)

CWD_CLEARED is "" (paths.py:330), so resolve_arm_cwd (session.py:1745-1756) always goes to asyncio.to_thread(resolved_cwd, "", key). The deferred import then raises ImportError: cannot import name 'session_default_cwd' from 'kiro_crew.config.loader'. The except Exception at :7201 catches that and returns 503 workspace_unavailable. Every agent switch and every same-name reset fails this way, before anything is committed, so no user can switch a slot's agent. Reviewers reproduced it on the PR tree by calling resolved_cwd("", "dashboard:chat-1") directly and through a real SessionManager.resolve_arm_cwd.

Why the tests stay green. Every fixture that drives this handler replaces the resolver with an AsyncMock, so the real resolver never runs. The fixtures are test/chat_test_helpers.py:133, test/test_chat_slot_switch_atomicity.py:82, test/test_eager_spawn.py:754 and test/test_switch_handlers_sibling_alias_busy.py:85. mypy does catch it: paths.py:385: error: Module "kiro_crew.config.loader" has no attribute "session_default_cwd". No check has failed on this head yet, so nothing has surfaced this so far.

Also a mypy error. The diff re-adds set_lifecycle_turn_active and mark_lifecycle_lease at session.py:1758/:1772. Both already exist at :1714/:1728, so mypy reports no-redef.

Smallest fix.

  1. Either bring the _session_work_dir -> session_default_cwd rename into config/loader.py, or import _session_work_dir here.
  2. Add one test that calls the real SessionManager.resolve_arm_cwd(key, "") with no mock.
  3. Remove the duplicate method definitions.

The branch also conflicts with main, so it needs a rebase anyway.

From a full open-PR audit against origin/main c66f95b (2026-09-25); findings were checked by independent reviewers reading the code. Reply here if this is wrong.

@rnoack1
rnoack1 force-pushed the split/agent-switch-settle-before-publish branch from 3dfed2d to c898993 Compare September 27, 2026 06:45
@dwu96

dwu96 commented Sep 27, 2026

Copy link
Copy Markdown
Contributor

👋 Hi! This PR is currently in draft status. Workflow runs won't be auto-approved until it's marked as ready for review.

When you're ready, click "Ready for review" and the workflows will be approved on the next cycle automatically.

@github-actions github-actions Bot added merge conflict Branch has merge conflicts with its base — author must resolve before merge and removed merge conflict Branch has merge conflicts with its base — author must resolve before merge labels Sep 27, 2026
…shes

The settle and its authorization gate now precede the binding commit, an unsettled
key answers 409 and commits nothing, and the arm transfer follows with no await.
@rnoack1
rnoack1 force-pushed the split/agent-switch-settle-before-publish branch from c898993 to 7be963c Compare September 28, 2026 03:44
@dwu96

dwu96 commented Sep 28, 2026

Copy link
Copy Markdown
Contributor

👋 Hi! This PR is currently in draft status. Workflow runs won't be auto-approved until it's marked as ready for review.

When you're ready, click "Ready for review" and the workflows will be approved on the next cycle automatically.

@github-actions github-actions Bot removed the merge conflict Branch has merge conflicts with its base — author must resolve before merge label Sep 28, 2026
@dwu96

dwu96 commented Sep 28, 2026

Copy link
Copy Markdown
Contributor

👋 Hi! This PR is currently in draft status. Workflow runs won't be auto-approved until it's marked as ready for review.

When you're ready, click "Ready for review" and the workflows will be approved on the next cycle automatically.

@dwu96

dwu96 commented Sep 28, 2026

Copy link
Copy Markdown
Contributor

👋 Hi! This PR is currently in draft status. Workflow runs won't be auto-approved until it's marked as ready for review.

When you're ready, click "Ready for review" and the workflows will be approved on the next cycle automatically.

1 similar comment
@dwu96

dwu96 commented Sep 28, 2026

Copy link
Copy Markdown
Contributor

👋 Hi! This PR is currently in draft status. Workflow runs won't be auto-approved until it's marked as ready for review.

When you're ready, click "Ready for review" and the workflows will be approved on the next cycle automatically.

@dwu96

dwu96 commented Sep 29, 2026

Copy link
Copy Markdown
Contributor

👋 Hi! This PR is currently in draft status. Workflow runs won't be auto-approved until it's marked as ready for review.

When you're ready, click "Ready for review" and the workflows will be approved on the next cycle automatically.

3 similar comments
@dwu96

dwu96 commented Sep 29, 2026

Copy link
Copy Markdown
Contributor

👋 Hi! This PR is currently in draft status. Workflow runs won't be auto-approved until it's marked as ready for review.

When you're ready, click "Ready for review" and the workflows will be approved on the next cycle automatically.

@dwu96

dwu96 commented Sep 29, 2026

Copy link
Copy Markdown
Contributor

👋 Hi! This PR is currently in draft status. Workflow runs won't be auto-approved until it's marked as ready for review.

When you're ready, click "Ready for review" and the workflows will be approved on the next cycle automatically.

@dwu96

dwu96 commented Sep 29, 2026

Copy link
Copy Markdown
Contributor

👋 Hi! This PR is currently in draft status. Workflow runs won't be auto-approved until it's marked as ready for review.

When you're ready, click "Ready for review" and the workflows will be approved on the next cycle automatically.

@github-actions github-actions Bot added the merge conflict Branch has merge conflicts with its base — author must resolve before merge label Sep 30, 2026

Copy link
Copy Markdown
Contributor

Triage (open-PR audit against origin/main 5edf2fe, head 7be963c): OVERLAPPING. Needs a rebase and a merge discussion.

Rebase. The branch point is cd783d2, 1254 commits behind main. git merge-tree reports conflicts in src/kiro_crew/dashboard/chat_handlers.py and src/kiro_crew/session_allocation.py. #16427 has since recomposed the chat handlers, and api_chat_slot_agent is now at chat_handlers.py:5667 on main. Main still has none of the arm primitives this PR carries (CWD_CLEARED, resolve_arm_cwd, mark_retire_on_next_claim, _settle_arm_target, claim_cwd), so nothing merged supersedes it.

Earlier review. The CHANGES_REQUESTED review on 3dfed2d raised two problems, and both look fixed at 7be963c:

  • paths.py:428 now imports _session_work_dir, which exists at loader.py:566.
  • set_lifecycle_turn_active and mark_lifecycle_lease are each defined once in session.py.

The second point that review asked for, a test that calls the real SessionManager.resolve_arm_cwd(key, "") without a mock, is still missing. Every fixture still uses an AsyncMock for it.

Merge discussion. This PR's own change is the agent-switch reordering: settle and authorize before the commit triple, answer 409 on an unsettled arm, and make _rollback_switch async. The shared core is in session_allocation.py, config/paths.py, session.py, chat_runner.py and state.py. That same core is carried, with small differences between copies, by the open PRs #11813, #11831, #11910, #11983 and #12055. #12055 applies the same settle-or-409 pattern to the workspace and project handlers. Please land the core once, in one PR, and stack this handler change on top of it so the copies don't drift.

Minor. The comment block at chat_handlers.py:7909 ("Commit the agent BEFORE any await…") still describes the old ordering. It now sits directly above code that deliberately commits after the awaits. Please reword it to say why a send landing in that window is safe now: the post-commit reset and the retire arm take care of it.

No breaking change and no P0 found at this head, so this is a comment, not a change request.

@iamwhatever

Copy link
Copy Markdown
Contributor

Thanks for this work, @rnoack1. Closing it because it no longer applies to current main (b42f19b).

If the arm design comes back on top of the current agent_switch.py, a fresh PR against that file would be the way to go.

@github-actions github-actions Bot removed the readiness: action required A blocking check or review needs attention label Oct 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

fork Pull request from a fork (external contributor) merge conflict Branch has merge conflicts with its base — author must resolve before merge

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants