fix(billing): enforce the plan usage limit mid-run and bill runs that outlive their period by waleedlatif1 · Pull Request #8461 · simstudioai/sim

waleedlatif1 · 2026-09-30T10:05:14Z

Summary

Chat runs are moving off a fixed wall-clock deadline, so spend has to be bounded by the plan's usage limit while a run is in progress, not only when it starts. This PR adds Sim's side of mid-run enforcement and makes sure a run that outlives its billing period is still invoiced exactly once.

Mid-run usage verdict

POST /api/billing/update-cost now returns usageExceeded (and, when true, a usageUpgrade card payload) at the top level of every answer: the 200 and the duplicate 409. The verdict is read through the cached execution usage gate, so a run under its limit pays no extra ledger read on most steps. Older workers ignore the fields.
update-cost answers for direct-v1 runs too, judging the payer saved in the run's account decision. The verdict read is bounded to 1 s, well inside the worker's 5 s callback timeout, and answers not-exceeded past it.
Continuation validation (POST /api/copilot/api-keys/validate, purpose: "continuation") also reads the original payer's spend, for attributed and direct-v1 runs alike, so a worker can poll it on a cadence. An admitted direct-v1 verdict is cached like an attributed one. A continuation 402 carries a body declared in the contract:
- usage limit: { code: "USAGE_LIMIT_EXCEEDED", error, usageUpgrade }
- blocked account: { code: "BILLING_BLOCKED", error }
A new turn's 402 stays empty, as before: the worker replaces any validation 402 body with its own message.
The update-cost schema ties the fields together: usageUpgrade is present exactly when usageExceeded is true.
The lifecycle's own continuation admission reads spend through the same gate. A refused continuation shows the existing upgrade card and stops the worker run, so the user's next message after upgrading continues normally instead of being refused as busy.

Card and blocked accounts

One resolver chooses the card's action and copy from the payer's plan. Sim's own card and the payload the worker writes into its durable log both use it, so live, replayed, and reloaded chats render the same card. A member over the cap their organization set gets copy naming who can raise it.
A blocked account (payment failed, dispute) is refused as blocked and never gets the usage card. The direct-v1 gate checks the actor and the saved payer for a block before their spend, through one account block check shared with continuation validation.
Every usage-limit refusal, including a worker 402 on the first leg, goes through one handler that shows the card and stops the worker run.

Fail-open mid-run, fail-closed at admission

Mid-run checks judge the period a run's charges land in. A Stripe-period payer's charges roll forward, so it is judged against its current subscription period (cached for a minute), and a rollover or an early anchor move is judged against the period charges now land in. Any other payer (an enterprise reporting window, or the open default period) never rolls: its run stays one ledger row whose creation time falls in the admitted window, which is summed by creation time, so the admitted window is judged even after it ends.
A direct-v1 run is judged against the payer saved in its account decision, never one re-selected from the actor's current memberships.
If usage or the current period cannot be read, or the period ends while its verdict is read, the mid-run paths report "unknown" and keep the run going; the next callback judges the next period. Admission before a run still refuses when usage cannot be read.

Runs that outlive their billing period

A cumulative charge is recorded against the billing period frozen at admission. For a long run, later top-ups would land in a period whose close had already invoiced it, so that overage was never billed.
When a Stripe-period payer's subscription has moved to a later period (including a period start that moved forward inside the old one), further spend is recorded in one new row per later period, stamped with the current period. Threshold settlement follows the stamped period. This applies to attributed runs and to direct-v1 runs, whose account decision now carries the payer's subscription from admission. Decisions minted before this still parse and keep today's behaviour.
The subscription row is share-locked for every period-aware write, so a rollover or an early period-start move waits for any in-flight write to the old period and the close's sums are final.
The cost callback refuses an idempotency key containing @, so a request's period row keys never collide with another request's.
Rows are only ever created forward, under the existing per-request lock, and totals converge on the maximum cumulative cost, so retries and out-of-order callbacks stay idempotent.
Mixed versions and rollback: code that predates period rows reads only the first row. If it handles a later callback for a run that already has period rows, during a deploy or after a rollback, it can re-add those rows' amount. That can only be invoiced if it lands between the rollover and that period's close, which waits at least an hour, and only for runs spanning a rollover. The exposure is one run's post-rollover spend, cents to dollars.

Other

The analytics settle window is unchanged. Its doc now notes that a turn running past it can under-report in a cached hour; invoices, thresholds, and the usage gate read live sums.
Syncs the worker's billing contract (usageExceeded, UsageUpgrade, UsageLimitRefusal).

Worker contract

Read usageExceeded and usageUpgrade from the top level of the update-cost answer.
Poll continuation validation. On a 402 with USAGE_LIMIT_EXCEEDED, write <usage_upgrade>{usageUpgrade}</usage_upgrade> as assistant text into the durable log, then complete. On a 402 with BILLING_BLOCKED (or no body), do not show the usage card.

Test plan

Real-Postgres integration: a charge spanning two period closes is invoiced exactly once in total; out-of-order and retried callbacks; a forward-moved period start rolls; a period is never stamped backwards; a first charge after a close lands in the current period; a rollover and an early period-start move each wait for an in-flight old-period write; per-period token shares; a reporting run's top-ups after its window ends count in that window and a later run's in the next.
Real-Postgres route integration (update-cost/route.integration.ts): a direct-v1 run across a Stripe rollover records its later spend in the current period (before this, the callback failed settlement and the spend was lost).
bun run test:integration (the PostgreSQL integration job) passes locally.
Route tests for the update-cost verdict (including duplicates, unreadable ledgers, blocked accounts, member caps, ended periods, reporting windows, direct-v1 payers, and the 1 s bound) and for continuation validation (usage limit, blocked, unknown, cached polling, billing off). Tests assert verdicts and responses; each guard was reverted to confirm its test fails.
Lifecycle tests: a refused continuation shows the card and stops the worker run; unreadable ledgers and ended periods let the leg run; blocked accounts are refused as blocked.
bun run lint, bun run type-check, bun run check:audits (including check:api-validation:strict), bun run mship-contracts:check.

vercel · 2026-09-30T10:05:19Z

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
docs	Ready	Preview	Sep 30, 2026 8:24pm UTC

cubic-dev-ai

All reported issues were addressed across 26 files

_{Tip: instead of fixing issues one by one fix them all with cubic

Re-trigger cubic}

greptile-apps · 2026-09-30T10:10:17Z

Confidence Score: 5/5

[Critical risk] Billing system enforces usage limits mid-run and handles period boundaries.

The PR appears safe to merge; the remaining new finding is a non-blocking blocked-account messaging regression.

Summary

The PR adds mid-run billing verdicts and period-aware cumulative charges, while the changes since the previous review also introduce Lucid live Search and adjust validation and stream handling.

Billing callbacks and continuation checks use the admitted payer’s usage; long Stripe-period runs record later spend in a later period.
Lucid Search adds member-account MCP search and bounded structured document reads.
A blocked-account ledger failure now loses its specific refusal message.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart LR
  Run[Admitted run] --> Callback[Cost callback]
  Callback --> Ledger[Period-aware usage ledger]
  Ledger --> Gate[Mid-run usage verdict]
  Gate --> Worker[Worker continues or stops]
  Search[Lucid Search] --> MCP[Member MCP account]
  MCP --> Results[Verified search results]
  Results --> Read[Bounded document read]

_{Reviews (13) · Last reviewed commit: "Merge origin/staging into fix/midrun-usa..."}

waleedlatif1 · 2026-09-30T11:02:17Z

@greptile

waleedlatif1 · 2026-09-30T11:02:18Z

@cubic-dev-ai review this PR

cubic-dev-ai · 2026-09-30T11:02:25Z

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

cubic-dev-ai

All reported issues were addressed across 27 files

_{Reply with feedback, questions, or to request a fix.

Fix all with cubic | Re-trigger cubic}

waleedlatif1 · 2026-09-30T11:35:16Z

@greptile

waleedlatif1 · 2026-09-30T11:35:18Z

@cubic-dev-ai review this PR

cubic-dev-ai · 2026-09-30T11:35:25Z

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

cubic-dev-ai

All reported issues were addressed across 28 files

_{Tip: cubic can generate docs of your entire codebase and keep them up to date. Try it here.

Fix all with cubic | Re-trigger cubic}

waleedlatif1 · 2026-09-30T11:42:28Z

@greptile

waleedlatif1 · 2026-09-30T11:42:29Z

@cubic-dev-ai review this PR

cubic-dev-ai · 2026-09-30T11:42:36Z

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

cubic-dev-ai

No issues found across 28 files

Confidence score: 5/5

Automated review surfaced no issues in the provided summaries.
No files require special attention.

_{Tip: cubic can generate docs of your entire codebase and keep them up to date. Try it here.

Re-trigger cubic}

waleedlatif1 · 2026-09-30T11:51:06Z

@greptile

waleedlatif1 · 2026-09-30T11:51:08Z

@cubic-dev-ai review this PR

cubic-dev-ai · 2026-09-30T11:51:14Z

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

cubic-dev-ai

No issues found across 28 files

Confidence score: 5/5

Automated review surfaced no issues in the provided summaries.
No files require special attention.

_{You've manually re-run cubic several times on this PR. Each manual re-review checks the full PR again and counts toward your usage quota. To preserve your usage limits, we recommend letting cubic automatically review new commits.
Tip: cubic can generate docs of your entire codebase and keep them up to date. Try it here.

Re-trigger cubic}

waleedlatif1 · 2026-09-30T17:12:26Z

@greptile

waleedlatif1 · 2026-09-30T19:12:42Z

@cubic-dev-ai review this PR

cubic-dev-ai · 2026-09-30T19:12:47Z

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

cubic-dev-ai

All reported issues were addressed across 32 files

_{You've manually re-run cubic several times on this PR. Each manual re-review checks the full PR again and counts toward your usage quota. To preserve your usage limits, we recommend letting cubic automatically review new commits.
Tip: cubic can generate docs of your entire codebase and keep them up to date. Try it here.

Fix all with cubic | Re-trigger cubic}

…and in A reporting-window or default-period payer's charges never roll forward, so after its admitted window ends the mid-run verdict judged an empty new window and under-enforced the limit. Both the attributed and direct-v1 verdicts now judge the current period only for a Stripe payer, matching the cost callback's rollover gate, and the admitted period otherwise, even after it ends. Stripe payers keep being judged against their current period.

waleedlatif1 · 2026-09-30T19:39:38Z

@greptile

waleedlatif1 · 2026-09-30T19:39:39Z

@cubic-dev-ai review this PR

cubic-dev-ai · 2026-09-30T19:39:47Z

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

cubic-dev-ai

No issues found across 32 files

Confidence score: 5/5

Automated review surfaced no issues in the provided summaries.
No files require special attention.

_{You've manually re-run cubic several times on this PR. Each manual re-review checks the full PR again and counts toward your usage quota. To preserve your usage limits, we recommend letting cubic automatically review new commits.
Tip: cubic can generate docs of your entire codebase and keep them up to date. Try it here.

Re-trigger cubic}

…ck calls - The account block, continuation delegation, cache, billing-off and rollover tests assert the verdict or HTTP response. Where behaviour depends on an input, the fake answers by that input, as the real ledger and settlement do. - Pins against real PostgreSQL that a reporting run's top-ups after its window ends are counted in that window, where its request was first charged, and a later run's charges in the next.

waleedlatif1 · 2026-09-30T19:58:26Z

@greptile

waleedlatif1 · 2026-09-30T19:58:27Z

@cubic-dev-ai review this PR

cubic-dev-ai · 2026-09-30T19:58:34Z

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

cubic-dev-ai

All reported issues were addressed across 32 files

_{You've manually re-run cubic several times on this PR. Each manual re-review checks the full PR again and counts toward your usage quota. To preserve your usage limits, we recommend letting cubic automatically review new commits.
Tip: cubic can generate docs of your entire codebase and keep them up to date. Try it here.

Fix all with cubic | Re-trigger cubic}

…retry, and make the reporting-window test deterministic - A new turn's 402 is empty again and the stream no longer parses 402 bodies: the worker replaces any validation 402 body with its own message and only polls continuation, so the new-turn codes, USAGE_UNAVAILABLE and the server-side refusal reasons had no consumer. - The mid-run verdict reads its current period once; a period that ends during the read is left to the next callback. - The cumulative-usage '@' check left the ledger; the cost callback already refuses such keys. - The reporting-window integration test derives its boundary from the first row's created_at and waits on the database clock.

waleedlatif1 · 2026-09-30T20:22:31Z

@greptile

waleedlatif1 · 2026-09-30T20:22:33Z

@cubic-dev-ai review this PR

cubic-dev-ai · 2026-09-30T20:22:39Z

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

greptile-apps · 2026-09-30T20:30:01Z

Comments Outside Diff

These findings could not be posted inline.

Blocked accounts get upgrade cards apps/sim/lib/mothership/request/go/stream.ts:270 ▶

When the worker refuses a leg with a 402 BILLING_BLOCKED response, this handler treats it as a usage-limit refusal without checking the body. The lifecycle then shows an upgrade card and completes the turn, telling the user to raise a usage limit instead of reporting the billing block.
Blocked account message is lost apps/sim/lib/billing/calculations/usage-monitor.ts:345 ▶

When a blocked account’s usage ledger read fails, the error handler replaces the payment- or dispute-specific refusal with a generic message saying usage limits could not be determined. Execution remains blocked, but the user loses the information needed to restore access.

cubic-dev-ai

No issues found across 31 files

Confidence score: 5/5

Automated review surfaced no issues in the provided summaries.
No files require special attention.

_{You've manually re-run cubic several times on this PR. Each manual re-review checks the full PR again and counts toward your usage quota. To preserve your usage limits, we recommend letting cubic automatically review new commits.
Tip: cubic can generate docs of your entire codebase and keep them up to date. Try it here.

Re-trigger cubic}

waleedlatif1 · 2026-09-30T20:39:54Z

Re: the outside-diff P1 "Blocked accounts get upgrade cards" (apps/sim/lib/mothership/request/go/stream.ts:272).

This is not a change introduced by this PR, and no producer of a BILLING_BLOCKED body exists on this path today:

The worker discards Sim's validation body before answering its own caller: worker/apps/server/src/http/security.ts cancels response.body on any non-OK validation and throws HttpAccessError(status, "Account access or usage admission was denied"). The Go gateway likewise forwards only the status with its own error payload.
So a 402 reaching stream.ts never carries a code, and stream.ts on current staging already maps every 402 to BillingLimitError. This PR leaves that line unchanged.
An earlier revision of this PR did parse BILLING_BLOCKED/USAGE_UNAVAILABLE from that 402. It was removed deliberately because it could never fire.

Showing a distinct blocked-account message on this path needs the worker and gateway to pass Sim's refusal code through. That is a separate cross-repo change, tracked as a follow-up. The continuation path this PR adds does send and consume BILLING_BLOCKED end to end: the worker parses Sim's continuation 402 body.

waleedlatif1 · 2026-09-30T20:39:55Z

@greptile

vercel Bot temporarily deployed to Preview September 30, 2026 10:05 Inactive

cubic-dev-ai Bot reviewed Sep 30, 2026

View reviewed changes

greptile-apps Bot reviewed Sep 30, 2026

View reviewed changes

vercel Bot temporarily deployed to Preview September 30, 2026 11:01 Inactive

cubic-dev-ai Bot reviewed Sep 30, 2026

View reviewed changes

Comment thread apps/sim/lib/billing/calculations/usage-monitor.ts Outdated

Comment thread apps/sim/lib/mothership/request/tools/billing.ts

greptile-apps Bot reviewed Sep 30, 2026

View reviewed changes

Comment thread apps/sim/lib/billing/core/mid-run-usage.ts Outdated

Comment thread apps/sim/lib/billing/core/mid-run-usage.ts Outdated

vercel Bot temporarily deployed to Preview September 30, 2026 11:35 Inactive

greptile-apps Bot reviewed Sep 30, 2026

View reviewed changes

Comment thread apps/sim/lib/billing/core/mid-run-usage.ts Outdated

cubic-dev-ai Bot reviewed Sep 30, 2026

View reviewed changes

Comment thread apps/sim/lib/billing/core/mid-run-usage.ts Outdated

Comment thread apps/sim/app/api/copilot/api-keys/validate/route.ts

Comment thread apps/sim/lib/billing/calculations/usage-monitor.ts Outdated

vercel Bot temporarily deployed to Preview September 30, 2026 11:42 Inactive

cubic-dev-ai Bot reviewed Sep 30, 2026

View reviewed changes

vercel Bot temporarily deployed to Preview September 30, 2026 11:50 Inactive

cubic-dev-ai Bot reviewed Sep 30, 2026

View reviewed changes

waleedlatif1 force-pushed the fix/midrun-usage-enforcement branch from c51ff3e to 0972f66 Compare September 30, 2026 17:12

vercel Bot temporarily deployed to Preview September 30, 2026 17:12 Inactive

cubic-dev-ai Bot reviewed Sep 30, 2026

View reviewed changes

Comment thread apps/sim/lib/billing/core/billing-attribution.ts

vercel Bot temporarily deployed to Preview September 30, 2026 19:39 Inactive

greptile-apps Bot reviewed Sep 30, 2026

View reviewed changes

Comment thread apps/sim/lib/billing/core/mid-run-usage.ts

Comment thread apps/sim/lib/billing/core/billing-attribution.test.ts Outdated

cubic-dev-ai Bot reviewed Sep 30, 2026

View reviewed changes

vercel Bot temporarily deployed to Preview September 30, 2026 19:58 Inactive

cubic-dev-ai Bot reviewed Sep 30, 2026

View reviewed changes

Comment thread apps/sim/lib/billing/core/usage-log.integration.ts Outdated

greptile-apps Bot reviewed Sep 30, 2026

View reviewed changes

Comment thread apps/sim/lib/billing/core/usage-log.integration.ts Outdated

waleedlatif1 added 2 commits September 30, 2026 13:19

Merge origin/staging into fix/midrun-usage-enforcement

4ec5704

vercel Bot deployed to Preview September 30, 2026 20:24 View deployment

cubic-dev-ai Bot reviewed Sep 30, 2026

View reviewed changes

waleedlatif1 merged commit 0f218b6 into staging Sep 30, 2026
32 of 33 checks passed

waleedlatif1 deleted the fix/midrun-usage-enforcement branch September 30, 2026 21:02

icecrasher321 mentioned this pull request Oct 1, 2026

v0.9.8: search cleanup recovery, new search providers, billing and chat reliability #8494

Merged

Conversation

waleedlatif1 commented Sep 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Mid-run usage verdict

Card and blocked accounts

Fail-open mid-run, fail-closed at admission

Runs that outlive their billing period

Other

Worker contract

Test plan

Uh oh!

vercel Bot commented Sep 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cubic-dev-ai Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

greptile-apps Bot commented Sep 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Confidence Score: 5/5

Summary

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

waleedlatif1 commented Sep 30, 2026

Uh oh!

waleedlatif1 commented Sep 30, 2026

Uh oh!

cubic-dev-ai Bot commented Sep 30, 2026

Uh oh!

cubic-dev-ai Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

waleedlatif1 commented Sep 30, 2026

Uh oh!

waleedlatif1 commented Sep 30, 2026

Uh oh!

cubic-dev-ai Bot commented Sep 30, 2026

Uh oh!

Uh oh!

cubic-dev-ai Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

waleedlatif1 commented Sep 30, 2026

Uh oh!

waleedlatif1 commented Sep 30, 2026

Uh oh!

cubic-dev-ai Bot commented Sep 30, 2026

Uh oh!

cubic-dev-ai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

waleedlatif1 commented Sep 30, 2026

Uh oh!

waleedlatif1 commented Sep 30, 2026

Uh oh!

cubic-dev-ai Bot commented Sep 30, 2026

Uh oh!

cubic-dev-ai Bot left a comment

waleedlatif1 commented Sep 30, 2026 •

edited

Loading

vercel Bot commented Sep 30, 2026 •

edited

Loading

cubic-dev-ai Bot left a comment •

edited

Loading

greptile-apps Bot commented Sep 30, 2026 •

edited

Loading

cubic-dev-ai Bot left a comment •

edited

Loading

cubic-dev-ai Bot left a comment •

edited

Loading

cubic-dev-ai Bot left a comment •

edited

Loading

cubic-dev-ai Bot left a comment •

edited

Loading

greptile-apps Bot commented Sep 30, 2026 •

edited

Loading