chore(trigger): fix queues that never serialized and schedule missing self-hosted crons by waleedlatif1 · Pull Request #8715 · simstudioai/sim

waleedlatif1 · 2026-10-07T01:22:14Z

Summary

Queues that didn't do what they said

run-data-drain: enqueue sites key runs by data-drain:<id> to serialize a drain, but with no queue limit each key got the env's full concurrency. Add concurrencyLimit: 1 (manual runs already refuse while one is in progress, so nothing needs concurrency), plus an explicit maxDuration: 5400 (same value it inherited) noting the cursor commits only at the end
table-run-dispatcher: triggered with concurrencyKey: dispatchId, so the limit is per dispatch and 8 was meaningless; 1 serializes a duplicate run of the same dispatch without throttling distinct dispatches (children run on the cell task's own queue)
lifecycle-email: no concurrency limit; cap at 10 (delayed one-off sends, no SLA)

Bounded per-run task: cleanup-table-row-ttl gets maxDuration: 600 (capped at 100 batches per run; 30-day max 30s). Every task whose work can grow with user data or fan-out stays on the global ceiling, since a run that hits maxDuration is killed and not retried

Config: logLevel: 'log' → 'info' (the worker already coerces unknown levels to info; no behavior change). Corrected comments that tied the 95-minute stale thresholds to the task ceiling — they measure time since the last heartbeat

Self-hosted crons (Helm + Compose, check:cron-parity green; chart 1.11.6 → 1.11.7): schedule cleanup-stale-executions (*/30) — without it, runs whose worker died never stop showing as running — and idempotency-key cleanup (daily; batched deletes bounded by the route's 5-minute limit; every namespace's retention is ≤ the 7-day cutoff and the long-lived namespaces are excluded). Docs table updated, including the missing fold-table-row-changes row

Type of Change

Chore

Testing

bun run lint, bun run type-check, bun run check:audits (incl. check:cron-parity), bun run docs-manifest:check — green
Tests beside changed sources (cleanup-stale-executions, cleanup-table-row-ttl, table dispatcher, data drains) — green
helm unittest helm/sim, helm template renders both new CronJobs with the expected schedules/timeouts
Config-only; no new tests

Checklist

Code follows project style guidelines
Self-reviewed my changes
Tests added/updated and passing (new tests pass the test-audit authoring gate)
No new warnings introduced
I confirm that I have read and agree to the terms outlined in the Contributor License Agreement (CLA)

…, schedule missing self-hosted crons

vercel · 2026-10-07T01:22:17Z

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
docs	Ready	Preview	Oct 7, 2026 2:01am UTC

cubic-dev-ai

1 issue found across 16 files

Confidence score: 3/5

quickbooks-webhook-ingress.ts can hit the five-minute cap before dispatching every target for a valid high-fan-out delivery; because dispatch is serial and has no continuation point, some targets may never be reached. Keep the 5,400-second ceiling or add resumable dispatch before lowering it.

Prompt for AI agents (unresolved issues)


Check if these issues are valid — if so, understand the root cause of each and fix them. When an issue isn't valid or won't be fixed in this PR, reply in its thread with the reason and then resolve the thread. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="apps/sim/background/quickbooks-webhook-ingress.ts">

<violation number="1" location="apps/sim/background/quickbooks-webhook-ingress.ts:146">
P2: This five-minute cap can stop a valid high-fan-out delivery before all targets are dispatched: the task processes them serially and does not persist a continuation point. Keep the 5,400-second ceiling or split/limit accepted work so it can finish within five minutes.</violation>
</file>

_{Reply with feedback, questions, or to request a fix.

Fix all with cubic | Turn on auto-fix | Re-trigger cubic}

greptile-apps · 2026-10-07T01:29:12Z

Confidence Score: 5/5

[High risk] Adds background job scheduling and adjusts task concurrency limits.

The PR appears safe to merge; no new blocking issue was found.

Summary

This PR makes keyed queues serialize matching runs, adds missing self-hosted cleanup schedules, and keeps growing jobs on the global time limit.

run-data-drain and table-run-dispatcher now allow one run per concurrency key.
Docker and Helm schedule stale-execution and idempotency-key cleanup.
The latest changes remove short limits from email and QuickBooks jobs and restore the larger file-cleanup machine.
All three earlier, unnumbered findings are addressed: the fork and cleanup-dispatch limits are gone, and file cleanup again uses large-1x.
No new actionable issues or mounted-rule violations were found.

_{Reviews (4) · Last reviewed commit: "chore(trigger): keep unbounded-fan-out t..." · Reviewed by Greptile}

…rim sizing comments

waleedlatif1 · 2026-10-07T01:36:38Z

@greptile

waleedlatif1 · 2026-10-07T01:36:39Z

@cubic-dev-ai review this PR

cubic-dev-ai · 2026-10-07T01:36:44Z

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

cubic-dev-ai

All reported issues were addressed across 17 files

_{Requires human review: Auto-approval blocked because this review re-detected 1 unresolved issue already reported by Cubic.

Fix all with cubic | Turn on auto-fix | Re-trigger cubic}

…e cleanup on kill

waleedlatif1 · 2026-10-07T01:51:16Z

@greptile

waleedlatif1 · 2026-10-07T01:51:18Z

@cubic-dev-ai review this PR

cubic-dev-ai · 2026-10-07T01:51:26Z

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

cubic-dev-ai

No issues found across 12 files

Confidence score: 5/5

Automated review surfaced no issues in the provided summaries.
No files require special attention.

_{Turn on auto-fix | Re-trigger cubic}

…ump chart for new cronjobs

waleedlatif1 · 2026-10-07T01:59:00Z

@greptile

waleedlatif1 · 2026-10-07T01:59:02Z

@cubic-dev-ai review this PR

cubic-dev-ai · 2026-10-07T01:59:08Z

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

cubic-dev-ai

No issues found across 11 files

Confidence score: 5/5

Automated review surfaced no issues in the provided summaries.
No files require special attention.

_{Turn on auto-fix | Re-trigger cubic}

chore(trigger): tighten task limits, fix queues that never serialized…

e20a3a4

…, schedule missing self-hosted crons

cubic-dev-ai Bot reviewed Oct 7, 2026

View reviewed changes

Comment thread apps/sim/background/fork-content-copy.ts Outdated

Comment thread apps/sim/background/quickbooks-webhook-ingress.ts Outdated

vercel Bot deployed to Preview October 7, 2026 01:25 View deployment

greptile-apps Bot reviewed Oct 7, 2026

View reviewed changes

Comment thread apps/sim/background/fork-content-copy.ts Outdated

Comment thread apps/sim/background/cleanup-dispatch.ts Outdated

Comment thread apps/sim/background/cleanup-file-versions.ts Outdated

refactor(trigger): derive the fork ceiling from STALE_ACTIVE_MS and t…

8a3bdd9

…rim sizing comments

vercel Bot temporarily deployed to Preview October 7, 2026 01:36 Inactive

cubic-dev-ai Bot reviewed Oct 7, 2026

View reviewed changes

Comment thread apps/sim/background/cleanup-file-versions.ts Outdated

Comment thread apps/sim/background/cleanup-dispatch.ts Outdated

chore(trigger): drop caps whose work grows with user data or that los…

a500cf2

…e cleanup on kill

vercel Bot temporarily deployed to Preview October 7, 2026 01:50 Inactive

waleedlatif1 changed the title ~~chore(trigger): tighten task limits, fix queues that never serialized, schedule missing self-hosted crons~~ chore(trigger): fix queues that never serialized, bound per-run tasks, schedule missing self-hosted crons Oct 7, 2026

cubic-dev-ai Bot reviewed Oct 7, 2026

View reviewed changes

chore(trigger): keep unbounded-fan-out tasks on the global ceiling; b…

7c35766

…ump chart for new cronjobs

waleedlatif1 changed the title ~~chore(trigger): fix queues that never serialized, bound per-run tasks, schedule missing self-hosted crons~~ chore(trigger): fix queues that never serialized and schedule missing self-hosted crons Oct 7, 2026

cubic-dev-ai Bot reviewed Oct 7, 2026

View reviewed changes

vercel Bot deployed to Preview October 7, 2026 02:01 View deployment

waleedlatif1 merged commit ab6b400 into staging Oct 7, 2026
39 checks passed

waleedlatif1 deleted the chore/trigger-task-config branch October 7, 2026 02:16

waleedlatif1 mentioned this pull request Oct 7, 2026

v0.9.15: mship improvements, memory improvements, nextjs bump, plane and ramp integrations, outbox hardening #8748

Merged

Conversation

waleedlatif1 commented Oct 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Type of Change

Testing

Checklist

Uh oh!

vercel Bot commented Oct 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cubic-dev-ai Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

greptile-apps Bot commented Oct 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Confidence Score: 5/5

Summary

Uh oh!

Uh oh!

Uh oh!

Uh oh!

waleedlatif1 commented Oct 7, 2026

Uh oh!

waleedlatif1 commented Oct 7, 2026

Uh oh!

cubic-dev-ai Bot commented Oct 7, 2026

Uh oh!

cubic-dev-ai Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

waleedlatif1 commented Oct 7, 2026

Uh oh!

waleedlatif1 commented Oct 7, 2026

Uh oh!

cubic-dev-ai Bot commented Oct 7, 2026

Uh oh!

cubic-dev-ai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

waleedlatif1 commented Oct 7, 2026

Uh oh!

waleedlatif1 commented Oct 7, 2026

Uh oh!

cubic-dev-ai Bot commented Oct 7, 2026

Uh oh!

cubic-dev-ai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

This branch was successfully deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

waleedlatif1 commented Oct 7, 2026 •

edited

Loading

vercel Bot commented Oct 7, 2026 •

edited

Loading

cubic-dev-ai Bot left a comment •

edited

Loading

greptile-apps Bot commented Oct 7, 2026 •

edited

Loading

cubic-dev-ai Bot left a comment •

edited

Loading