fix(mothership): answer a retried task wake whose turn already ran by waleedlatif1 · Pull Request #8484 · simstudioai/sim

waleedlatif1 · 2026-09-30T22:24:40Z

Summary

Problem. The worker retries a task wake under the same run ID until it sees its own run for that ID. When sim accepted the wake but the headless turn ended inside sim without reaching the worker (for example, a usage-limit refusal), the worker never saw a run. It retried the wake, sim accepted it again, and the loop never ended.

Root cause. prepareTaskWake only checked the chat stream lock. It had no idea that a turn under this run ID had already run, so every retry looked like a new wake.

Fix.

After prepareTaskWake takes the chat lock, it looks up the latest run for the wake's run ID. If a turn already ran under that ID, it answers not_found (404), and the worker drops the notification instead of retrying. The check runs while the lock is held, so a turn that is still running under that ID keeps answering busy (409).
If the lookup throws, or the wake answers not-found, the lock taken moments earlier is released. The wake turn that would normally release it never starts, and without this the chat would stay locked.
ensureHeadlessRunIdentity now logs the underlying error and keeps it as the cause of its generic "execution record is unavailable" error, so the failure can be diagnosed.

Behaviour changes

A retried wake whose turn already ended now gets 404 (it used to get 202 and run again).
A retried wake whose turn is still holding the chat still gets 409.
If the run lookup fails, the wake gets 500 and the chat lock is released.
A failed headless run-record insert is now logged with its cause. The error message returned to callers has not changed.

Test plan

prepare-wake.integration.ts runs against real PostgreSQL and Redis through the wake route. It covers the not-found retry, a busy retry while the turn still holds the chat, and releasing the lock when the lookup fails.
Red then green: with prepare-wake.ts reverted to staging, the not-found and lock-release tests fail. The busy test passes both before and after the fix; it is there to keep the new check from turning an in-flight turn's 409 into a 404.
Related unit tests pass: lib/mothership/tasks, lib/mothership/request/lifecycle, app/api/mothership/wake.
bun run type-check (apps/sim), biome on the changed files, and bun run check:test-patterns all pass.
Staging check: a wake whose turn was refused for usage limits stops being retried.

The worker retries a task wake under the same run ID until its own run appears. When sim ended that turn without reaching the worker (a usage-limit refusal), every retry reopened the turn, hit the unique stream-id constraint, and failed behind a generic message, so the worker retried forever. Under the chat lock, a wake whose run ID already has a sim run now releases the lock and answers not-found, which the worker treats as a refusal and dismisses the notification. An in-flight turn still holds the lock and answers busy. The headless run-record catch-all now logs the underlying insert error.

The retried-wake check reads copilot_runs after taking the chat lock. If that read threw, the lock stayed held until its TTL because the wake turn that releases it never started. Release with the exact lease on any throw after the acquire.

waleedlatif1 · 2026-09-30T22:24:44Z

@greptile

waleedlatif1 · 2026-09-30T22:24:45Z

@cubic-dev-ai review this PR

vercel · 2026-09-30T22:24:48Z

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment

Project	Deployment	Actions	Updated
docs	Skipped		Sep 30, 2026 10:50pm UTC

cubic-dev-ai · 2026-09-30T22:24:50Z

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

cubic-dev-ai

All reported issues were addressed across 3 files

_{Reply with feedback, questions, or to request a fix.

Fix all with cubic | Re-trigger cubic}

greptile-apps · 2026-09-30T22:28:44Z

Confidence Score: 5/5

[Medium risk] Adds retry logic to task wake handling.

The PR appears safe to merge; no outstanding finding or new actionable defect remains.

Summary

This PR makes task wakes whose turn already ran return not-found instead of starting another turn.

It preserves the busy response while a turn holds the chat lock and releases the acquired lease if the run lookup fails.
It adds integration coverage for retries and lock cleanup, and logs the cause of failed headless run-record creation.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  A[Task wake] --> B{Chat lock acquired?}
  B -- No --> C[409: retry while busy]
  B -- Yes --> D{Run lookup}
  D -- Existing run --> E[Release acquired lease; 404]
  D -- Lookup fails --> F[Release acquired lease; 500]
  D -- No run --> G[202: start wake turn]

_{Reviews (3) · Last reviewed commit: "test(mothership): stub the chat lease ge..."}

cubic-dev-ai

All reported issues were addressed across 3 files

_{Requires human review: Auto-approval blocked because this review re-detected 1 unresolved issue already reported by Cubic.

Re-trigger cubic}

…p fails

waleedlatif1 · 2026-09-30T22:36:52Z

@greptile

waleedlatif1 · 2026-09-30T22:36:54Z

@cubic-dev-ai review this PR

cubic-dev-ai · 2026-09-30T22:37:00Z

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

cubic-dev-ai

No issues found across 3 files

Confidence score: 5/5

Automated review surfaced no issues in the provided summaries.
No files require special attention.

_{Re-trigger cubic}

waleedlatif1 · 2026-09-30T22:50:50Z

@greptile

waleedlatif1 · 2026-09-30T22:50:52Z

@cubic-dev-ai review this PR

cubic-dev-ai · 2026-09-30T22:50:57Z

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

cubic-dev-ai

No issues found across 4 files

Confidence score: 5/5

Automated review surfaced no issues in the provided summaries.
No files require special attention.

_{Re-trigger cubic}

waleedlatif1 added 2 commits September 30, 2026 15:21

cubic-dev-ai Bot reviewed Sep 30, 2026

View reviewed changes

Comment thread apps/sim/lib/mothership/tasks/application/prepare-wake.ts Outdated

Comment thread apps/sim/lib/mothership/tasks/application/prepare-wake.integration.ts Outdated

greptile-apps Bot reviewed Sep 30, 2026

View reviewed changes

Comment thread apps/sim/lib/mothership/tasks/application/prepare-wake.ts Outdated

Comment thread apps/sim/lib/mothership/request/lifecycle/run.ts

cubic-dev-ai Bot reviewed Sep 30, 2026

View reviewed changes

fix(mothership): release the wake's own chat lease when the run looku…

7cb813b

…p fails

vercel Bot temporarily deployed to Preview September 30, 2026 22:36 Inactive

greptile-apps Bot reviewed Sep 30, 2026

View reviewed changes

Comment thread apps/sim/lib/mothership/tasks/application/prepare-wake.integration.ts

cubic-dev-ai Bot reviewed Sep 30, 2026

View reviewed changes

test(mothership): stub the chat lease getter in the wake unit test mock

4ed2f71

vercel Bot temporarily deployed to Preview September 30, 2026 22:50 Inactive

cubic-dev-ai Bot reviewed Sep 30, 2026

View reviewed changes

waleedlatif1 merged commit 088955f into staging Oct 1, 2026
32 checks passed

icecrasher321 mentioned this pull request Oct 1, 2026

v0.9.8: search cleanup recovery, new search providers, billing and chat reliability #8494

Merged

waleedlatif1 deleted the fix/task-wake-idempotent-delivery branch October 1, 2026 01:24

Conversation

waleedlatif1 commented Sep 30, 2026

Summary

Behaviour changes

Test plan

Uh oh!

waleedlatif1 commented Sep 30, 2026

Uh oh!

waleedlatif1 commented Sep 30, 2026

Uh oh!

vercel Bot commented Sep 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cubic-dev-ai Bot commented Sep 30, 2026

Uh oh!

cubic-dev-ai Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

greptile-apps Bot commented Sep 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Confidence Score: 5/5

Summary

Uh oh!

Uh oh!

Uh oh!

cubic-dev-ai Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

waleedlatif1 commented Sep 30, 2026

Uh oh!

waleedlatif1 commented Sep 30, 2026

Uh oh!

cubic-dev-ai Bot commented Sep 30, 2026

Uh oh!

Uh oh!

cubic-dev-ai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

waleedlatif1 commented Sep 30, 2026

Uh oh!

waleedlatif1 commented Sep 30, 2026

Uh oh!

cubic-dev-ai Bot commented Sep 30, 2026

Uh oh!

cubic-dev-ai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

This branch was previously deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

vercel Bot commented Sep 30, 2026 •

edited

Loading

cubic-dev-ai Bot left a comment •

edited

Loading

greptile-apps Bot commented Sep 30, 2026 •

edited

Loading

cubic-dev-ai Bot left a comment •

edited

Loading