fix(backend): end request-scoped pg pools on workerd to stop Hyperdrive slot exhaustion - #3363
posthog-eu[bot] wants to merge 10 commits into
Conversation
closeClient was a no-op on the workerd runtime, so every request-scoped pg.Pool built by getPgClient (auth-key and subkey resolution, RBAC checks, checkWriteAppAccess) was never ended. On Cloudflare Workers the unclosed pools leaked their Hyperdrive sockets until the connection slots were exhausted, and new connections failed with "Timed out while waiting for an open slot in the pool" on the bundle-upload write path. closeClient now always ends the pool, deferring end() to waitUntil via backgroundTask so it adds no request latency, and logging any end() failure instead of throwing. This mirrors the proven closeClient in plugin_runtime/utils/pg.ts. Module-scoped reused pools (file_read_cache) are never passed to closeClient, so they are unaffected. Generated-By: PostHog Desktop Task-Id: 3ea49ca0-9ec6-4083-a09f-4041420d20d1
|
Important Review skippedBot user detected. To trigger a single review, invoke the ⚙️ Run configurationConfiguration used: Organization UI Review profile: ASSERTIVE Plan: Advanced Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Merging this PR will improve performance by 82.43%
|
| Benchmark | BASE |
HEAD |
Efficiency | |
|---|---|---|---|---|
| ⚡ | /updates manifest response with metadata |
255.5 µs | 140.1 µs | +82.43% |
Tip
Curious why performance improved? Comment @codspeedbot explain why performance improved on this PR, or directly use the CodSpeed MCP with your agent.
Comparing posthog-self-driving/fixbackend-stop-leaking-per-request-8884b3 (861ebcd) with main (ea1a1ed)
Footnotes
-
2 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩
rihoarvutikonto
left a comment
There was a problem hiding this comment.
Unique: always-end() on workerd Hyperdrive Pool fights the plugin_runtime Client+skipEnd contract; the new test already expects end() to be unsupported.
Align getPgClient/closeClient with plugin_runtime: workerd Hyperdrive gets a per-request pg.Client with connect() and skipEndClients; non-Hyperdrive paths keep short-lived Pool with explicit end(). Pool max is 1 on workerd. Updates callers to await getPgClient and replaces lifecycle unit tests. Co-authored-by: Martin DONADIEU <martindonadieu@gmail.com>
…callers - Export checkoutPgClient/releasePgClient for Pool checkout vs Hyperdrive Client - Await getPgClient across backend call sites; align transaction types with PgQueryClient - Restore typecheck-clean org/channel transaction paths Co-authored-by: Martin DONADIEU <martindonadieu@gmail.com>
…rding Merge origin/main into the Hyperdrive pg lifecycle fix branch. Resolve conflicts in app put, queue consumer, onboarding login, and manifest_size while keeping async getPgClient and PgQueryClient checkout helpers. Co-authored-by: Martin DONADIEU <martindonadieu@gmail.com>
Unit tests that mock pg.ts need checkoutPgClient and releasePgClient after the Hyperdrive Client + skipEnd alignment. Share a small helper for the Pool checkout path used across affected mocks. Co-authored-by: Martin DONADIEU <martindonadieu@gmail.com>
|
@coderabbitai review |
…only Revert the full main merge that pulled CLI onboarding changes and failed the Builder TUI preview job. Keep the pg Hyperdrive lifecycle fix scoped to backend utils while adding only the migration required for the published CLI whoami RPC contract test. Co-authored-by: Martin DONADIEU <martindonadieu@gmail.com>
|
@coderabbitai review |
Add adress to typos extend-words for the published CLI migration RPC name. Point private/cli-mcp-tests at main so the TUI preview job has current goldens without pulling unrelated CLI onboarding code. Co-authored-by: Martin DONADIEU <martindonadieu@gmail.com>
|
@coderabbitai review |
Re-adding migrations from main that the merge revert removed. Supabase migrations must stay append-only for CI; this keeps the pg lifecycle PR scoped while matching main schema history. Co-authored-by: Martin DONADIEU <martindonadieu@gmail.com>
|
@coderabbitai review |
|
@coderabbitai full review |
1 similar comment
|
@coderabbitai full review |
Co-authored-by: Martin DONADIEU <martindonadieu@gmail.com>
|
@coderabbitai full review |
|
|
@coderabbitai full review |



Summary (AI generated)
closeClientwas a no-op on theworkerdruntime, so every request-scopedpg.PoolfromgetPgClientwas never ended — the Hyperdrive sockets leaked until the pool slots ran out.checkWriteAppAccess(supabase/functions/_backend/files/files.ts), which gates paid bundle uploads; the api worker leaks on the same auth/RBAC path.closeClientnow always ends the pool, deferringend()towaitUntilviabackgroundTask(no added request latency) and loggingend()failures instead of throwing.export function closeClient(c: Context, db: ReturnType<typeof getPgClient>) { - if (getRuntimeKey() !== 'workerd') - return backgroundTask(c, db.end()) - return undefined + return backgroundTask(c, Promise.resolve(db.end()).catch((error: unknown) => { + cloudlogErr({ requestId: c.get('requestId'), message: 'PG client end failed', error: serializePostgresError(error) }) + })) }Motivation (AI generated)
hono_middleware.ts), RBAC checks (rbac.ts), andcheckWriteAppAccess(files.ts). The TUS path re-runscheckWriteAppAccesson everyPATCHchunk, so one large upload could drain slots on its own.try/finally { closeClient }, so re-enablingend()onworkerdfixes all of them at one point. This is the codebase's own established pattern —plugin_runtime/utils/pg.tsalready ends its non-Hyperdrive pools this way.file_read_cache.ts'ssharedDeletedLookupPool) are never passed tocloseClient, so they keep their intended lifetime.Business Impact (AI generated)
capgo_files-prod:filesandcapgo_api-prod:api), where authorization was failing at the database step.Test Plan (AI generated)
oxlintpasses on the changed backend file and the new test.bun test:unit— newtests/pg-close-client-lifecycle.unit.test.tsassertscloseClientends the pool and swallowsend()failures. (Could not run locally: this environment has nobun/node_modules; relying on CI.)capgo_files-prod:filesandcapgo_api-prod:api.PATCH) still authorize and complete.Checklist
bun run lint:backend && bun run lint.Created with PostHog Desktop from this inbox report.
Need help on this PR? Tag
@codesmith-botwith what you need. Autofix is disabled.