fix(mothership): never sweep a Chat run while one of its Sim tools is executing by waleedlatif1 · Pull Request #8482 · simstudioai/sim

waleedlatif1 · 2026-09-30T22:11:27Z

Summary

Problem. The orphaned-run sweep could settle a Chat run as interrupted while one of its Sim tools was still executing, which closes tool admission under a worker that still holds the run.

Root cause. The sweep treats a leased run as orphaned once it has gone past the recovery window (ORPHANED_RUN_GRACE_MS, one hour) without a status write and no controller holds its chat lock. A Sim tool call writes nothing to its run while it executes. The only sign that it is alive is its execution lease heartbeat, and the sweep never checked it. A tool call that runs longer than the window, such as a long workflow run, therefore looked idle. The default one-hour run deadline hid this.

Fix.

The leased-run candidate predicate now excludes any run that has an unsettled, unrevoked tool execution whose lease has not expired (execution_lease_expires_at > clock_timestamp()). The same predicate guards both candidate selection and the guarded settle update.
settleRuns now locks the candidate run rows (in id order, after the chat rows, matching the controller claim's lock order) before the guarded update. Tool admission locks its run row, so an admission racing the sweep either commits a lease that the update then sees, or finds the run already settled and is refused.
settleRuns then locks the candidate runs' unsettled tool execution rows, after the run rows. A lease heartbeat writes only the tool row, so a renewal that commits first is seen by the guarded update, and a later heartbeat finds the lease expired.

Behaviour changes

A Chat run whose Sim tool holds a live execution lease is no longer swept, however long the tool runs. It becomes eligible once the lease expires, is revoked, or settles, and the run has been idle for the recovery window.
The sweep adds one row lock per candidate run inside its existing transaction. The lock order is unchanged relative to controller claims (chats first, then runs, both by id).
Legacy-run handling is unchanged.

Test plan

New integration test: a run idle past the window with a live tool execution lease is not settled (orphaned-runs.integration.ts)
New integration test: a tool admitted while the sweep waits on the run's lock is seen, and the run is not settled
New integration test: never settles a run whose Sim tool lease a heartbeat renewed as the sweep settled it
The new tests fail on the pre-fix orphaned-runs.ts and pass with the fix (confirmed red, then green)
The full orphaned-runs integration suite passes (20/20) against disposable Postgres and Redis (bun run test:integration lib/mothership/async-runs/orphaned-runs)
bun run type-check (apps/sim), biome, check:test-patterns
Staging: a Chat run with a Sim tool call running longer than an hour completes without being settled as interrupted

… executing The orphaned-run sweep settled a leased run once it had gone an hour without a status write and no controller held its chat lock. A Sim tool call writes nothing to its run while it executes; only its execution lease heartbeat shows it is alive. So a long tool call (a workflow run can take well over an hour) could have its run settled as interrupted while the worker still held the run, closing tool admission under it. The default one-hour run deadline masked this. The sweep now skips a leased run with an unsettled, unrevoked tool execution whose lease has not expired, both when it selects candidates and in the guarded update. It also locks the run rows before that update, so a tool admission (which locks its run row) either commits a lease the update then sees, or finds the run settled and is refused.

waleedlatif1 · 2026-09-30T22:11:32Z

@greptile

waleedlatif1 · 2026-09-30T22:11:33Z

@cubic-dev-ai review this PR

vercel · 2026-09-30T22:11:34Z

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment

Project	Deployment	Actions	Updated
docs	Skipped		Sep 30, 2026 10:38pm UTC

cubic-dev-ai · 2026-09-30T22:11:39Z

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

cubic-dev-ai

No issues found across 2 files

Confidence score: 5/5

Automated review surfaced no issues in the provided summaries.
No files require special attention.

_{Re-trigger cubic}

cubic-dev-ai

All reported issues were addressed across 2 files

_{Reply with feedback, questions, or to request a fix.

Fix all with cubic | Re-trigger cubic}

greptile-apps · 2026-09-30T22:15:25Z

Confidence Score: 5/5

[High risk] Changes how the system decides when to clean up stalled chat runs.

The PR appears safe to merge based on the reviewed changes.

Summary

The PR prevents the orphaned-run sweep from settling a Chat run with a live Sim tool execution lease. Since the previous review, it also locks unsettled tool rows before the guarded update, adds a heartbeat-race test, and makes the lock-holder tests release their transactions on failure.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart LR
  A[Sweep selects idle run] --> B[Lock chat row]
  B --> C[Lock run row]
  C --> D[Lock unsettled tool rows]
  D --> E{Live execution lease?}
  E -- Yes --> F[Leave run active]
  E -- No --> G[Guarded settlement]

_{Reviews (2) · Last reviewed commit: "fix(mothership): lock a run's unsettled ..."}

…eep settles it A lease heartbeat writes only the tool row, so one that passed its expiry check before the lease ran out could commit after the sweep read the old lease and settled the run. The sweep now locks those tool rows after the run rows, so the guarded update sees a committed renewal and a later heartbeat finds the lease expired. Lock-holder tests release on failure.

waleedlatif1 · 2026-09-30T22:38:47Z

@greptile

waleedlatif1 · 2026-09-30T22:38:48Z

@cubic-dev-ai review this PR

cubic-dev-ai · 2026-09-30T22:38:55Z

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

cubic-dev-ai

1 issue found across 2 files

Confidence score: 4/5

In apps/sim/lib/mothership/async-runs/orphaned-runs.integration.ts, a timed-out lock barrier can leave a claim or sweep running after the test exits, mutating shared Redis and database state while the next test starts; stop or await the background promise on timeout.

Prompt for AI agents (unresolved issues)


Check if these issues are valid — if so, understand the root cause of each and fix them. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="apps/sim/lib/mothership/async-runs/orphaned-runs.integration.ts">

<violation number="1" location="apps/sim/lib/mothership/async-runs/orphaned-runs.integration.ts:245">
P2: When a lock barrier times out, `run` throws but its claim or sweep promise keeps running after the test exits. That background sweep can update the shared Redis cursor and database while a following test is starting, contaminating later tests; track and await every queued operation in the cleanup path.</violation>
</file>

_{Reply with feedback, questions, or to request a fix.

Fix all with cubic | Re-trigger cubic}

vercel Bot temporarily deployed to Preview September 30, 2026 22:11 Inactive

cubic-dev-ai Bot reviewed Sep 30, 2026

View reviewed changes

Comment thread apps/sim/lib/mothership/async-runs/orphaned-runs.ts

Comment thread apps/sim/lib/mothership/async-runs/orphaned-runs.integration.ts Outdated

greptile-apps Bot reviewed Sep 30, 2026

View reviewed changes

Comment thread apps/sim/lib/mothership/async-runs/orphaned-runs.integration.ts Outdated

vercel Bot temporarily deployed to Preview September 30, 2026 22:38 Inactive

cubic-dev-ai Bot reviewed Sep 30, 2026

View reviewed changes

Comment thread apps/sim/lib/mothership/async-runs/orphaned-runs.integration.ts

Comment thread apps/sim/lib/mothership/async-runs/orphaned-runs.integration.ts

waleedlatif1 merged commit 55d877b into staging Oct 1, 2026
33 of 34 checks passed

waleedlatif1 mentioned this pull request Oct 1, 2026

v0.9.8: search cleanup recovery, new search providers, billing and chat reliability #8494

Merged

waleedlatif1 deleted the fix/orphaned-run-sweep-live-tool-lease branch October 1, 2026 01:24

Conversation

waleedlatif1 commented Sep 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Behaviour changes

Test plan

Uh oh!

waleedlatif1 commented Sep 30, 2026

Uh oh!

waleedlatif1 commented Sep 30, 2026

Uh oh!

vercel Bot commented Sep 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cubic-dev-ai Bot commented Sep 30, 2026

Uh oh!

cubic-dev-ai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

cubic-dev-ai Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

greptile-apps Bot commented Sep 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Confidence Score: 5/5

Summary

Uh oh!

Uh oh!

waleedlatif1 commented Sep 30, 2026

Uh oh!

waleedlatif1 commented Sep 30, 2026

Uh oh!

cubic-dev-ai Bot commented Sep 30, 2026

Uh oh!

cubic-dev-ai Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

This branch was previously deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

waleedlatif1 commented Sep 30, 2026 •

edited

Loading

vercel Bot commented Sep 30, 2026 •

edited

Loading

cubic-dev-ai Bot left a comment •

edited

Loading

greptile-apps Bot commented Sep 30, 2026 •

edited

Loading

cubic-dev-ai Bot left a comment •

edited

Loading