Skip to content

fix(cloud-agent): complete native container billing and recovery - #8884

Open
eshurakov wants to merge 2 commits into
fix/container-usage-atomic-capacityfrom
eshurakov/giddy-ibis
Open

eshurakov wants to merge 2 commits into
fix/container-usage-atomic-capacityfrom
eshurakov/giddy-ibis

Conversation

@eshurakov

Copy link
Copy Markdown
Contributor

Summary

Native Cloudflare Containers can lose the final usage interval during a controlled stop. Organization compute also needs member accounting, billing recovery, and customer charge visibility.

  • Settle controlled-stop usage from the recorded running boundary. Preserve settlement replay and debit integrity.
  • Apply organization member allowances and Kilo Pass accounting through shared consumption logic. Preserve the acting member for bot usage.
  • Bound billing admission outages to one minute. Preserve specific payer and retry details through the control plane, SDK, and recovery UI.
  • Show compute prices and charge history. Include compute in spend alerts and provide a scoped historical backfill.
  • Add wildcard Cloud Agent enrollment, scoped emergency rollback, and an operations runbook.

Stacked on #8274. This PR contains one implementation commit above that branch.

Verification

Prior local implementation checks recorded in this worktree:

  • Controlled stop before the first heartbeat: 120 seconds and $1.20 settled exactly once.
  • Credit exhaustion and same-chat recovery; customer price and compute charge reporting.
  • Organization accounting, Kilo Pass, alerts, wildcard rollback, and recovery after a 60-second meter outage.
  • Real local Cloud page: loading, failed options, disabled SKU, and available price states. Browser request fixtures controlled the options responses; no paid session was started for these UI checks.
  • Backfill entrypoint against an explicitly verified local script database: one payer scope rebuilt.

These checks establish local behavior. Hosted restart behavior, production timing, payment-provider behavior, and Snowflake ingestion remain unverified.

Visual Changes

Compute pricing and billing recovery now appear in Cloud Agent. Compute charges appear in usage analytics.

Before After
No baseline screenshot was captured. Local Cloud page showing compute price

Reviewer Notes

Automated results from the prior local verification report:

  • Cloud Agent integration: 647 passed. SDK provenance: 45 passed. Worker utilities: 971 passed.
  • Full Cloud Agent unit run: 5,509 passed; seven Git test timeouts passed separately or with a 30-second timeout.
  • Latest follow-up checks: 301 distinct focused tests passed, including 40 meter PostgreSQL tests and 130 sandbox integration tests.
  • Changed-package typechecks and lint passed. The full repository suite was not rerun after the follow-up repairs.
  • Local tooling had three tmux test failures attributed to the environment in the verification report.

For PR publication, pnpm format:check and git diff --check HEAD~1 HEAD passed again. Other results above come from the existing local verification report.

Review settlement replay, attribution, allowance enforcement, and admission recovery. Member limits apply to native compute in this release; SDK, Vercel, and Gastown accounting retain their existing behavior.

Release work is separate: verify migration 0297's index lock window and the production script target. Follow services/container-usage-meter/RUNBOOK.md for deployment order and immediate scoped backfill. Historical compute can leave an incomplete anomaly baseline during the deploy/backfill gap. External monitors and provider reconciliation require operations checks.

@eshurakov
eshurakov added this pull request to stack #8402 October 24, 2026 21:27
origin: normalizedOrigin(metadata.identity.billingOrigin),
...(metadata.workspace?.sandboxProvider === 'cloudflare-containers' &&
subject.type === 'org'
? computeMemberMetadata(actor, metadata.identity.userId)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

CRITICAL: Acting-member billingUserId never reaches the meter

This branch puts the acting member into metadata.billingUserId, and assertSandboxBillingAllocation now permits that key. But every meter generation is created by createBillingGeneration (services/cloud-agent-next/src/metered-billing-lifecycle.ts:921-928), which rebuilds metadata from an allow-list (container_class, durable_object_id, vcpu, memory_mib, disk_mb, origin) and never copies input.metadata.billingUserId; usageContextFromBillingContext then sends that metadata in recordStart. For subject.type === 'org' with a bot actor the meter's applyStartWithDb (services/container-usage-meter/src/postgres.ts:506) throws UsageMutationConflictError, which billingAdmissionFailureFromError maps to meter_unavailable. Org bot sessions therefore retry admission for the full one-minute window and fail billing_unavailable, and pre-upgrade org bot intervals settle with remaining = 0. The new test only asserts the buildSandboxBillingInput result, so the stored context is not covered. Copy billingUserId into the stored metadata (or stop rebuilding metadata there).

...(pin.billing === null ? {} : { billing: pin.billing }),
};
if (pin.provider === 'cloudflare-containers') {
billingAdmissionDeadlineAt = await nativeBillingAdmissionDeadlineAt(

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

WARNING: Retry create can reopen admission after the alarm already failed the sandbox

The new alarm branch above (sandbox-do.ts:1892-1905) runs failCreationRoutesNow(... 'billing_unavailable') as soon as the stored deadline passes, including while runCreate is still in its runtime-artifact/launch-env preflight (the alarm only defers on createInFlight after that block). runCreate validates the allocation only at entry (line 4240). If the alarm fires during that window it clears the key and fails the routes; this retry path then finds no preparing route, writes a fresh now + billingAdmissionMs deadline against the stale allocationId, calls armAlarm(state) and proceeds with provider.create(...), performing meter admission and launching a container for a request already reported as billing_unavailable (the post-create stopRef cleans it up, but the meter still sees a phantom start/stop interval). Re-check state.kind/allocationId and that the attempt is still preparing after the preflight, before writing the deadline or creating.

@@ -0,0 +1 @@
CREATE INDEX "IDX_compute_usage_charge_created" ON "compute_usage_charge" USING btree ("created_at"); No newline at end of file

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SUGGESTION: Partitioned index build blocks settlement inserts

compute_usage_charge is a monthly-partitioned, write-hot ledger (every paid segment inserts a row), so this parent CREATE INDEX locks the parent and each existing partition and blocks settlement inserts until commit. The RUNBOOK documents scheduling a low-traffic window, but the migration has no lock_timeout and no guarded path, so a large partition can stall the meter's settlement path for the whole build. Consider the runbook's per-partition CONCURRENTLY + ATTACH build, or at least SET LOCAL lock_timeout so a large partition fails fast.

const request = {
allocationRef: record.allocationRef,
billingFailure: billingFailureFromAdmission(context.subject, {
code: budget?.code === 'BILLING_UNAVAILABLE' ? 'meter_unavailable' : 'insufficient_credits',

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SUGGESTION: A block without code is reported as a credit shortage

enforceBudgetStop writes block.code only when it creates a new block (metered-billing-lifecycle.ts:608-616), so a block persisted by the previous release has no code. Replaying such a block through onContainerStarted/enforceBudgetStop reaches this mapping with budget.code === undefined and sends INSUFFICIENT_CREDITS to the customer even when the stop came from an unreadable meter verdict, which is exactly the case that must not claim a credit shortage. Treat an absent code as BILLING_UNAVAILABLE, or apply failureCode to an existing block too.

if (billingMode === 'paid') {
const remaining = await balanceForSubject(tx, input.subject, true);
if (remaining <= MINIMUM_REMAINING_MICRODOLLARS) {
if (isNativeOrganizationUsage(input) && !organizationMemberId(input)) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SUGGESTION: Unattributed native org start is reported as a transient meter outage

This throws UsageMutationConflictError, which billingAdmissionFailureFromError maps to meter_unavailable, so the client keeps retrying for the whole admission budget and then reports a retryable billing outage. An unattributed org actor is a permanent configuration problem (the RUNBOOK says such starts 'fail billing verification until their configuration is corrected'), so it should fail fast like the new non-retryable invalid_configuration SKU path instead of consuming create retries.

`);
let rebuiltScopes = 0;
for (const { scope_key: scopeKey } of scopes.rows) {
await sweepHourlyBuckets(database, { lookbackHours: 720, now, scopeKey });

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SUGGESTION: Unbounded backfill loop has no batching, progress or resume

scopes.rows is fully materialized and the loop has no limit, batch size or cursor, so a large backfill is all-or-nothing and an interrupted run restarts every payer from zero, re-running the 720-hour sweep each time. Chunk the scope list (and log progress) so operators can resume instead of rerunning all scopes.

asc(charge.rate_cents_per_unit)
)
.limit(COMPUTE_CHARGE_PAGE_SIZE + 1)
.offset(input.page * COMPUTE_CHARGE_PAGE_SIZE);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SUGGESTION: Offset pagination re-aggregates the range and can drift between pages

Each page re-runs the grouped/sorted aggregation over up to a 366-day range only to discard page * 100 rows (page is allowed up to 1000). Because the ledger is append-only and the order starts with desc(day), a charge written between page requests shifts groups across the offset, so rows can be duplicated or skipped. Keyset pagination on the existing order columns (or a much lower page cap) avoids both.

@kilo-code-bot

kilo-code-bot Bot commented Oct 24, 2026 •

Copy link
Copy Markdown
Contributor

Code Review Summary

Status: 7 Issues Found | Recommendation: Address before merge

Executive Summary

The acting-member billingUserId added for org bot compute is dropped before the meter, so org bot native sessions cannot be admitted; the rest are one control-plane admission race and five hardening suggestions.

Assumptions: [INFERENCE] Hosted restart behavior, mixed-version/rollback behavior, provider timing and the PostgreSQL session timezone were not exercised (no tests run). The pre-existing CURRENT_DATE vs UTC usage_date asymmetry is not reported because it predates this diff.

Overview

Severity Count
CRITICAL 1
WARNING 1
SUGGESTION 5
Issue Details (click to expand)

CRITICAL

File Line Issue
services/cloud-agent-next/src/container-usage-context.ts 289 metadata.billingUserId is built and allowed by assertSandboxBillingAllocation, but createBillingGeneration rebuilds metadata from an allow-list and drops it. Meter applyStart then rejects org bot usage as meter_unavailable, so org bot sessions retry for one minute and fail billing_unavailable; the RUNBOOK claims this metadata is retained.

WARNING

File Line Issue
services/cloud-agent-next/src/control-plane/sandbox/sandbox-do.ts 4307 The alarm can fail routes for the admission deadline while runCreate is in preflight; the retry path then writes a fresh deadline, arms an alarm and creates a container (meter start/stop) for a request already reported as billing_unavailable.

SUGGESTION

File Line Issue
packages/db/src/migrations/0297_windy_changeling.sql 1 Parent CREATE INDEX on the partitioned, write-hot compute_usage_charge blocks settlement inserts; migration has no lock_timeout or per-partition concurrent path (runbook only advises scheduling).
services/cloud-agent-next/src/sandbox-containers/SandboxContainers.ts 1541 A billing block persisted before this release has no code, so a replayed BILLING_UNAVAILABLE stop is reported to the customer as INSUFFICIENT_CREDITS.
services/container-usage-meter/src/postgres.ts 506 Unattributed native org starts throw a conflict that maps to retryable meter_unavailable instead of failing fast as invalid_configuration.
apps/web/src/lib/spend-alerts/compute-backfill.ts 24 Backfill loop over all scopes has no batching, progress or resume point.
apps/web/src/lib/cloud-agent/compute-billing.ts 118 Deep offset pagination re-aggregates the full range per page and can duplicate/skip rows as charges are appended.
Files Reviewed (106 files)
  • apps/web/src/app/api/internal/organizations/compute-balance-alert/ (route + test) - auth/scoping verified
  • apps/web/src/components/cloud-agent-next/ (billing error, compute price/rate, session panel/select/dialog + tests) - 1 issue
  • apps/web/src/components/usage-analytics/ (compute section, dashboard)
  • apps/web/src/lib/cloud-agent/ (compute-billing, compute-price) - 1 issue
  • apps/web/src/lib/spend-alerts/ (sweep, compute-backfill, tests) - 1 issue
  • apps/web/src/lib/kilo-pass-org/consumption.test.ts, apps/web/src/lib/spend-writer-audit.test.ts
  • apps/web/src/routers/ (compute-billing-router, root-router, tests)
  • apps/web/src/scripts/backfill-compute-spend-alerts.ts
  • dev/local/env-sync/plan.test.ts
  • packages/cloud-agent-sdk/src/ (normalizer, schemas, session-manager, tests)
  • packages/container-usage/src/heartbeat.ts (+ test)
  • packages/db/src/migrations/0297_windy_changeling.sql, packages/db/src/schema.ts - 1 issue
  • packages/db/src/migrations/meta/0297_snapshot.json, _journal.json (generated)
  • packages/session-ingest-contracts/src/ (cloud-agent-billing, index)
  • packages/web-shared/src/lib/kilo-pass-org/consumption.ts, microdollars.ts, organizations/organization-usage.ts (+ test)
  • packages/worker-utils/ (cloud-agent-compute, compute-member, microdollars, organization-consumption, organization-usage, package.json, tests)
  • services/cloud-agent-next/src/container-usage-context.ts (+ test) - 1 issue
  • services/cloud-agent-next/src/billing-failure.ts, metered-billing-lifecycle.ts, trpc-error.ts
  • services/cloud-agent-next/src/control-plane/sandbox/ (allocation, native-billing-admission, routes, sandbox-do, sqlite-schema, drizzle migrations + tests) - 1 issue
  • services/cloud-agent-next/src/control-plane/session/ (messages, session-do, sqlite-schema, drizzle migrations) and control-plane/generated-migrations.test.ts
  • services/cloud-agent-next/src/persistence/customer-billing-failure.ts (+ test)
  • services/cloud-agent-next/src/sandbox-containers/ (SandboxContainers, containers-billing, tests) - 1 issue
  • services/cloud-agent-next/src/sandbox-control/ (cloudflare-containers-provider, provider, tests)
  • services/cloud-agent-next/src/session/message-settlement-outbox.ts, src/shared/ (billing, protocol, timers, mirror test)
  • services/cloud-agent-next/test/ (integration, sdk-provenance)
  • services/container-usage-meter/src/ (balance-alert, billing-config, meter, postgres, tests) - 1 issue
  • services/container-usage-meter/ (RUNBOOK.md, .dev.vars.example, package.json, vitest config, worker-configuration.d.ts, wrangler.jsonc)
  • services/kilo-mcp/catalog.json, .kilo/skills/kilo-mcp/SKILL.md - read-only catalog entries, no write exposure
  • pnpm-lock.yaml (generated)

Reviewed by deepseek-flash-latest · Input: 178.7K · Output: 51.6K · Cached: 8.2M

Review guidance: REVIEW.md from base branch fix/container-usage-atomic-capacity

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant