fix(mothership): keep Chat stream legs alive without a wall clock by waleedlatif1 · Pull Request #8463 · simstudioai/sim

waleedlatif1 · 2026-09-30T11:40:11Z

Summary

Sim no longer ends a Chat run's stream at 60 minutes. The worker's own run deadline still applies until its flag is turned off, so on its own this PR moves the limit from Sim to the worker. It has no worker dependency.

Reconnect cap: the reconnect stream (GET /api/copilot/chat/stream) no longer ends a live turn with resume_timeout at its cap. It closes without a terminal event, and the client re-attaches from its cursor.
Worker idle timeout: a 120 s idle timeout on the worker SSE leg replaces the 60-minute wall clock on each leg.
- The worker writes a keepalive comment whenever a leg has been quiet for 10 s, so a healthy leg is never silent for more than about 25 s.
- 120 s stays well under common intermediary idle cuts.
- A silent leg fails as a retryable interruption and re-attaches with its receipt.
- A caller-set timeout (currently only Slack search) still applies; other headless callers are bounded by the worker's run deadline.
Retry budget: the retry budget for a reachable worker (three retries within 30 s) refills only after five minutes of streaming that delivers events. It never refills per event, so a deterministic failure stays bounded.
Per-operation timeouts: the tool watchdog, permission wait and client tool wait each have their own constant, with unchanged values. The delegation TTL reuses the long-running tool watchdog's cap. ORCHESTRATION_TIMEOUT_MS is removed.
Client reconnect budget: a reconnect attempt whose tail delivered new events restarts the budget at the base delay. Separate network drops hours apart no longer add up to exhaustion.
Replay buffer during a park: the chat-lock heartbeat refreshes the replay buffer's TTLs and its byte counter's TTL, so a run parked on a long tool call or approval keeps its history and its accounting. A finished stream is marked closed when its cleanup is scheduled, and a heartbeat still in flight can't extend it again.
Cursor ahead of a restarted buffer: when the buffer's numbering restarted after it expired, a reconnect cursor ahead of it is reported as replay_gap instead of waiting for events that never come.

Known limits

The usage analytics settle window (USAGE_SETTLE_MS) still assumes a stream of about an hour. It only affects cached analytics hours, not invoices or the usage gate; revisit before the worker deadline is turned off.
Worker deadline: the worker's run deadline still ends runs until its flag is turned off.
replay_gap still ends the view: a reconnect the replay ring can't serve still ends with a replay_gap error. The follow-up PR re-syncs it from the worker's durable log.
Inbox runs: Chat runs started from the inbox still run in a Trigger.dev task with a 90-minute maxDuration. That is tracked separately.
maxDuration = 3600 on the POST and execute routes: it stays, because it only takes effect on serverless hosts.

Test plan

buffer-ttl.integration.ts (real Redis): a park longer than the buffer and counter TTLs; no revival after cleanup is scheduled; a cursor ahead of a restarted buffer.
Unit and DOM tests:
- worker leg liveness (past an hour, silent leg, unanswered request);
- the retry-window refill;
- the reconnect cap;
- the client reconnect budget.
bun run lint, bun run type-check, bun run check:audits, full apps/sim vitest.

…wall clock - End the replay GET at its cap without a terminal event, so the client re-attaches from its cursor instead of ending a live turn with resume_timeout. - Replace the per-leg worker SSE wall clock with an idle timeout (WORKER_STREAM_IDLE_TIMEOUT_MS, 120 s) that fails a silent leg as a retryable interruption; a caller-set timeout still applies. - Let StreamRetryWindow run without a deadline by default and replenish its reachable budget only after five minutes of healthy streaming, never per event. - Split the tool watchdog, permission wait, client tool wait, and delegation TTL onto their own constants and remove ORCHESTRATION_TIMEOUT_MS. - Refresh the replay buffer TTLs from the chat-lock heartbeat, and report a replay gap for a cursor ahead of a buffer whose numbering restarted. - Document why maxDuration stays on the chat POST and execute routes.

…s events A reconnect attempt whose re-attached tail delivered new events now restarts the retry budget at the base delay, so separate network drops hours apart in a long turn no longer add up to the ten-attempt exhaustion.

… never revive a closed buffer - The chat-lock heartbeat now slides the owner byte counter's TTL along with the replay buffer's. Otherwise, after a park longer than an hour, the counter expired while the ring survived: the ring could grow to about twice its target, and its refunds drained the user counter. - Scheduling a finished stream's cleanup now marks it closed, and the refresh leaves a closed stream alone. A heartbeat still in flight at teardown can no longer re-extend a buffer whose cleanup was already scheduled. - Describe the worker idle timeout in terms of network intermediaries, and state that the delegation TTL reuses the long-running tool watchdog's cap.

vercel · 2026-09-30T11:40:30Z

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment

Project	Deployment	Actions	Updated
docs	Skipped		Sep 30, 2026 4:49pm UTC

greptile-apps · 2026-09-30T11:45:01Z

Confidence Score: 5/5

[Medium risk] Adjusts stream timeout logic and retry budgets for long-running chat.

The PR appears safe to merge based on the reviewed changes.

Summary

The PR replaces Sim’s one-hour Chat stream limit with per-leg idle detection and reconnect handling. It also preserves replay data during long parks, distinguishes a restarted buffer from a valid cursor, and keeps separate tool and approval timeouts.

The changes since the previous review bound stalled error-body reads, require delivered events to span the retry-budget refill window, and correct replay cleanup ordering and outcome tracing.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart LR
  Client[Chat client] --> Replay[Replay stream]
  Replay -->|cap reached without terminal| Client
  Sim[Sim stream leg] --> Worker[Worker SSE]
  Worker -->|events or keepalives| Sim
  Sim -->|idle interruption| Retry[Retry window]
  Retry -->|reattach with receipt| Worker

_{Reviews (2) · Last reviewed commit: "fix(mothership): mark a closing buffer b..."}

cubic-dev-ai

1 issue found across 22 files

Confidence score: 5/5

In apps/sim/lib/mothership/request/session/buffer-ttl.integration.ts, Test 1 parks longer than the 2-second stream TTL, so the keys may expire before the test completes and make its result unreliable; align the park duration with the TTL or refresh the keys.

Prompt for AI agents (unresolved issues)


Check if these issues are valid — if so, understand the root cause of each and fix them. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="apps/sim/lib/mothership/request/session/buffer-ttl.integration.ts">

<violation number="1" location="apps/sim/lib/mothership/request/session/buffer-ttl.integration.ts:13">
P3: Test 1's park is much longer in real time than the stream TTL it compresses. `COPILOT_STREAM_TTL_SECONDS` is 2, so the events/seq/owner-counter keys expire 2 real-time seconds after their last refresh, but each tick of the loop also spends a real `await sleep(500)` — 10 ticks ≈ 5s of real time. The assertions only pass because the abort poller's heartbeat lands `refreshBufferTtl` every ~500ms real; on a stalled CI runner a single >2s gap between heartbeats expires the keys and fails the final `getLatestSeq === 1` / `appendText === 2` / counter equality asserts. Widen the real-time margin by raising the TTL (e.g. 10-20s) while keeping the 21s fake-time jump so the park still outlasts it.</violation>
</file>

_{Reply with feedback, questions, or to request a fix.

Fix all with cubic | Re-trigger cubic}

… only on real progress - A non-OK worker response's body is read under the same idle bound as the leg, so a stalled error body can no longer block the turn forever. - The reachable retry budget refills only when a leg's delivered events span five minutes, from its first event to its latest. A leg that delivered one event and then only kept alive has made no progress and no longer refills it.

… a capped replay as ended - scheduleBufferCleanup sets the closed marker before shortening the TTLs, so a heartbeat refresh that lands mid-pipeline is either overridden or sees the marker. - A reconnect that ends at its cap is reported as ended without a terminal, not as a client disconnect: the outcome is read before the route closes its own stream. - The buffer TTL integration test keeps a 5 s TTL against a 12 s park, so a slow runner cannot expire the buffer between heartbeats.

waleedlatif1 · 2026-09-30T16:49:50Z

@greptile

waleedlatif1 · 2026-09-30T16:49:51Z

@cubic-dev-ai review this PR

cubic-dev-ai · 2026-09-30T16:49:58Z

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

cubic-dev-ai

No issues found across 22 files

Confidence score: 5/5

Automated review surfaced no issues in the provided summaries.
No files require special attention.

_{Re-trigger cubic}

waleedlatif1 added 3 commits September 30, 2026 03:24

vercel Bot temporarily deployed to Preview September 30, 2026 11:40 Inactive

greptile-apps Bot reviewed Sep 30, 2026

View reviewed changes

Comment thread apps/sim/lib/mothership/request/go/stream.ts

Comment thread apps/sim/lib/mothership/request/lifecycle/stream-retry.ts Outdated

cubic-dev-ai Bot reviewed Sep 30, 2026

View reviewed changes

Comment thread apps/sim/lib/mothership/request/session/buffer.ts Outdated

Comment thread apps/sim/app/api/copilot/chat/stream/route.ts

Comment thread apps/sim/lib/mothership/request/session/buffer-ttl.integration.ts Outdated

waleedlatif1 added 2 commits September 30, 2026 07:41

vercel Bot temporarily deployed to Preview September 30, 2026 16:49 Inactive

waleedlatif1 mentioned this pull request Sep 30, 2026

fix(mothership): re-sync Chat reconnects from the worker log and trim the replay ring by bytes #8469

Merged

3 tasks done

cubic-dev-ai Bot reviewed Sep 30, 2026

View reviewed changes

waleedlatif1 merged commit e816f01 into staging Sep 30, 2026
32 checks passed

waleedlatif1 deleted the fix/stream-leg-continuity branch September 30, 2026 21:35

icecrasher321 mentioned this pull request Oct 1, 2026

v0.9.8: search cleanup recovery, new search providers, billing and chat reliability #8494

Merged

Navigation Menu

fix(mothership): keep Chat stream legs alive without a wall clock - #8463

fix(mothership): keep Chat stream legs alive without a wall clock#8463
waleedlatif1 merged 5 commits into
stagingfrom
fix/stream-leg-continuity

waleedlatif1 commented Sep 30, 2026

Uh oh!

vercel Bot commented Sep 30, 2026 •

edited

Loading

Uh oh!

greptile-apps Bot commented Sep 30, 2026 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

cubic-dev-ai Bot left a comment •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

waleedlatif1 commented Sep 30, 2026

Uh oh!

waleedlatif1 commented Sep 30, 2026

Uh oh!

cubic-dev-ai Bot commented Sep 30, 2026

Uh oh!

cubic-dev-ai Bot left a comment

Uh oh!

Uh oh!

This branch was previously deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

waleedlatif1 commented Sep 30, 2026

Summary

Known limits

Test plan

Uh oh!

vercel Bot commented Sep 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

greptile-apps Bot commented Sep 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Confidence Score: 5/5

Summary

Uh oh!

Uh oh!

Uh oh!

cubic-dev-ai Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

waleedlatif1 commented Sep 30, 2026

Uh oh!

waleedlatif1 commented Sep 30, 2026

Uh oh!

cubic-dev-ai Bot commented Sep 30, 2026

Uh oh!

cubic-dev-ai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

This branch was previously deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

vercel Bot commented Sep 30, 2026 •

edited

Loading

greptile-apps Bot commented Sep 30, 2026 •

edited

Loading

cubic-dev-ai Bot left a comment •

edited

Loading