fix(mothership): close abandoned tool meters once instead of alarming every tick

[waleedlatif1]

Summary

Problem. The service-usage replay tick logged Service usage requires reconciliation at ERROR on every tick, in every process, indefinitely. The alarm never cleared and carried no information about which meter was stuck.

Root cause. A tool meter row (_tool_execution, cost_usd IS NULL) is opened when a tool starts and closed by finishServiceUsage when it ends. When the process that owns the tool dies mid-execution, nothing ever closes that row. serviceMeteringHealth counted every such row older than 5 minutes, so a single abandoned meter produced a permanent, repeating error. The 5-minute threshold also flagged tools that were still legitimately running under the long-running watchdog.

Fix. Replace the health count with closeAbandonedServiceMeters: the replay tick closes open meters older than twice the longest tool watchdog (TOOL_WATCHDOG_LONG_RUNNING_MS), claimed with FOR UPDATE SKIP LOCKED so concurrent ticks close each meter exactly once. A pricing failure keeps its recorded error; otherwise the row records Tool execution never finished. Each closed meter is logged once, at WARN, with its stream, tool call, creation time and reason.

Behaviour changes

• The recurring Service usage requires reconciliation ERROR is gone. Each abandoned meter now yields one WARN line naming its stream and tool call, then never again.
• Open tool meters older than 2x the longest tool watchdog are marked closed (delivered_at set). Meters younger than that are left alone, so in-flight tools are no longer flagged.
• Known spend is unaffected: priced usage is stored as separate receipts, which are still claimed and delivered as before. Meters are never delivered to the worker.

Test plan

• service-store.integration.ts (real PostgreSQL 17 + pgvector, migrated with db:migrate): a new case covers an abandoned meter, an abandoned meter with a pricing error, a meter just past the watchdog that must stay open, and a priced receipt that must stay deliverable. Two concurrent closes return each abandoned meter once, a third returns nothing, and the receipt is still claimable.
• Existing receipt delivery integration case still passes with the health check removed.
• New integration case fails before the fix and passes after.
• vitest run lib/mothership/billing (unit) passes.
• bun run type-check (apps/sim) and Biome pass.

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment

Project	Deployment	Actions	Updated
docs	Skipped		Sep 30, 2026 10:34pm UTC

[waleedlatif1]

@greptile

[waleedlatif1]

@cubic-dev-ai review this PR

[cubic-dev-ai]

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

[cubic-dev-ai]

No issues found across 3 files

Confidence score: 5/5

• Automated review surfaced no issues in the provided summaries.
• No files require special attention.

_{Re-trigger cubic}

[greptile-apps]

Confidence Score: 5/5

[Critical risk] Changes how tool billing meters are closed and tracked.

The PR appears safe to merge; no new actionable issue or outstanding previous finding remains.

Summary

The PR replaces a recurring reconciliation error with one-time closure and warning logs for aged tool meters, while leaving priced receipts on their existing delivery path. The follow-up changes preserve closed-meter reasons when a tool finishes late and use the repository’s ID generator in tests.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart LR
  A[Tool meter opens] --> B{Tool finishes?}
  B -->|Yes| C[Finish meter]
  B -->|No, past age threshold| D[Close meter and warn once]
  E[Priced receipt] --> F[Claim and deliver separately]

Loading

_{Reviews (2) · Last reviewed commit: "fix(mothership): keep a closed tool mete..."}

[cubic-dev-ai]

All reported issues were addressed across 3 files

<sub>Reply with feedback, questions, or to request a fix.

Fix all with cubic | Re-trigger cubic</sub>

[waleedlatif1]

@greptile

[waleedlatif1]

@cubic-dev-ai review this PR

[cubic-dev-ai]

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

[cubic-dev-ai]

No issues found across 3 files

Confidence score: 5/5

• Automated review surfaced no issues in the provided summaries.
• No files require special attention.

_{Re-trigger cubic}