Skip to content

fix(backend): end request-scoped pg pools on workerd to stop Hyperdrive slot exhaustion - #3363

Open
posthog-eu[bot] wants to merge 10 commits into
mainfrom
posthog-self-driving/fixbackend-stop-leaking-per-request-8884b3
Open

posthog-eu[bot] wants to merge 10 commits into
mainfrom
posthog-self-driving/fixbackend-stop-leaking-per-request-8884b3

Conversation

@posthog-eu

@posthog-eu posthog-eu Bot commented Sep 17, 2026 •

Copy link
Copy Markdown
Contributor

Summary (AI generated)

  • closeClient was a no-op on the workerd runtime, so every request-scoped pg.Pool from getPgClient was never ended — the Hyperdrive sockets leaked until the pool slots ran out.
  • The failing write path is checkWriteAppAccess (supabase/functions/_backend/files/files.ts), which gates paid bundle uploads; the api worker leaks on the same auth/RBAC path.
  • Fix: closeClient now always ends the pool, deferring end() to waitUntil via backgroundTask (no added request latency) and logging end() failures instead of throwing.
 export function closeClient(c: Context, db: ReturnType<typeof getPgClient>) {
-  if (getRuntimeKey() !== 'workerd')
-    return backgroundTask(c, db.end())
-  return undefined
+  return backgroundTask(c, Promise.resolve(db.end()).catch((error: unknown) => {
+    cloudlogErr({ requestId: c.get('requestId'), message: 'PG client end failed', error: serializePostgresError(error) })
+  }))
 }

Motivation (AI generated)

  • Each request minted several pools that nobody closed: api-key and subkey resolution (hono_middleware.ts), RBAC checks (rbac.ts), and checkWriteAppAccess (files.ts). The TUS path re-runs checkWriteAppAccess on every PATCH chunk, so one large upload could drain slots on its own.
  • Hyperdrive then rejected new connections with "Timed out while waiting for an open slot in the pool."
  • Every leaking call site already wraps its client in try/finally { closeClient }, so re-enabling end() on workerd fixes all of them at one point. This is the codebase's own established pattern — plugin_runtime/utils/pg.ts already ends its non-Hyperdrive pools this way.
  • Module-scoped reused pools (file_read_cache.ts's sharedDeletedLookupPool) are never passed to closeClient, so they keep their intended lifetime.

Business Impact (AI generated)

  • Restores reliability on the paid bundle-upload write path across two production workers (capgo_files-prod:files and capgo_api-prod:api), where authorization was failing at the database step.
  • Removing the per-request pool leak should cut the pool-exhaustion errors toward zero on the upload path; the nightly bundle-cleanup burst may still peak, so a residual floor is expected rather than a full elimination.

Test Plan (AI generated)

  • oxlint passes on the changed backend file and the new test.
  • bun test:unit — new tests/pg-close-client-lifecycle.unit.test.ts asserts closeClient ends the pool and swallows end() failures. (Could not run locally: this environment has no bun/node_modules; relying on CI.)
  • After deploy, confirm the "Timed out while waiting for an open slot in the pool" exception volume drops on capgo_files-prod:files and capgo_api-prod:api.
  • Confirm bundle uploads (including multi-chunk TUS PATCH) still authorize and complete.

Checklist

  • My code follows the code style of this project and passes bun run lint:backend && bun run lint.
  • My change requires a change to the documentation.
  • I have updated the documentation accordingly.
  • My change has adequate E2E test coverage.
  • I have tested my code manually, and I have provided steps how to reproduce my tests.

Created with PostHog Desktop from this inbox report.


View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.

Review in cubic

closeClient was a no-op on the workerd runtime, so every request-scoped
pg.Pool built by getPgClient (auth-key and subkey resolution, RBAC checks,
checkWriteAppAccess) was never ended. On Cloudflare Workers the unclosed
pools leaked their Hyperdrive sockets until the connection slots were
exhausted, and new connections failed with "Timed out while waiting for an
open slot in the pool" on the bundle-upload write path.

closeClient now always ends the pool, deferring end() to waitUntil via
backgroundTask so it adds no request latency, and logging any end() failure
instead of throwing. This mirrors the proven closeClient in
plugin_runtime/utils/pg.ts. Module-scoped reused pools (file_read_cache) are
never passed to closeClient, so they are unaffected.

Generated-By: PostHog Desktop
Task-Id: 3ea49ca0-9ec6-4083-a09f-4041420d20d1
@posthog-eu
posthog-eu Bot deployed to deepsec-pr September 17, 2026 09:09 Active
@coderabbitai

coderabbitai Bot commented Sep 17, 2026 •

Copy link
Copy Markdown
Contributor

Important

Review skipped

Bot user detected.

To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Advanced

Run ID: 347c8910-b824-47f1-b7bb-a2fff3d3dcd6

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@codspeed

codspeed Bot commented Sep 17, 2026 •

Copy link
Copy Markdown
Contributor

Merging this PR will improve performance by 82.43%

⚠️ Different runtime environments detected

Some benchmarks with significant performance changes were compared across different runtime environments,
which may affect the accuracy of the results.

Open the report in CodSpeed to investigate

⚡ 1 improved benchmark
✅ 42 untouched benchmarks
⏩ 2 skipped benchmarks1

Performance Changes

Benchmark BASE HEAD Efficiency
⚡ /updates manifest response with metadata 255.5 µs 140.1 µs +82.43%

Tip

Curious why performance improved? Comment @codspeedbot explain why performance improved on this PR, or directly use the CodSpeed MCP with your agent.


Comparing posthog-self-driving/fixbackend-stop-leaking-per-request-8884b3 (861ebcd) with main (ea1a1ed)

Open in CodSpeed

Footnotes

  1. 2 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

@posthog-eu
posthog-eu Bot marked this pull request as ready for review September 17, 2026 09:33

@rihoarvutikonto rihoarvutikonto left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unique: always-end() on workerd Hyperdrive Pool fights the plugin_runtime Client+skipEnd contract; the new test already expects end() to be unsupported.

Comment thread supabase/functions/_backend/utils/pg.ts Outdated
Align getPgClient/closeClient with plugin_runtime: workerd Hyperdrive
gets a per-request pg.Client with connect() and skipEndClients; non-Hyperdrive
paths keep short-lived Pool with explicit end(). Pool max is 1 on workerd.

Updates callers to await getPgClient and replaces lifecycle unit tests.

Co-authored-by: Martin DONADIEU <martindonadieu@gmail.com>
@cursor
cursor Bot deployed to deepsec-pr September 23, 2026 16:48 Active
cursoragent and others added 2 commits September 23, 2026 17:19
…callers

- Export checkoutPgClient/releasePgClient for Pool checkout vs Hyperdrive Client
- Await getPgClient across backend call sites; align transaction types with PgQueryClient
- Restore typecheck-clean org/channel transaction paths

Co-authored-by: Martin DONADIEU <martindonadieu@gmail.com>
…rding

Merge origin/main into the Hyperdrive pg lifecycle fix branch. Resolve
conflicts in app put, queue consumer, onboarding login, and manifest_size
while keeping async getPgClient and PgQueryClient checkout helpers.

Co-authored-by: Martin DONADIEU <martindonadieu@gmail.com>
@cursor
cursor Bot deployed to deepsec-pr September 23, 2026 17:21 Active
Unit tests that mock pg.ts need checkoutPgClient and releasePgClient
after the Hyperdrive Client + skipEnd alignment. Share a small helper
for the Pool checkout path used across affected mocks.

Co-authored-by: Martin DONADIEU <martindonadieu@gmail.com>
@cursor
cursor Bot deployed to deepsec-pr September 23, 2026 17:31 Active
@cursor

cursor Bot commented Sep 23, 2026

Copy link
Copy Markdown

@coderabbitai review

cursoragent and others added 2 commits September 23, 2026 18:01
…nd onboarding"

This reverts commit cc38f57, reversing
changes made to e7f3e32.
…only

Revert the full main merge that pulled CLI onboarding changes and failed
the Builder TUI preview job. Keep the pg Hyperdrive lifecycle fix scoped
to backend utils while adding only the migration required for the
published CLI whoami RPC contract test.

Co-authored-by: Martin DONADIEU <martindonadieu@gmail.com>
@cursor
cursor Bot deployed to deepsec-pr September 23, 2026 18:03 Active
@cursor

cursor Bot commented Sep 23, 2026

Copy link
Copy Markdown

@coderabbitai review

Add adress to typos extend-words for the published CLI migration RPC
name. Point private/cli-mcp-tests at main so the TUI preview job has
current goldens without pulling unrelated CLI onboarding code.

Co-authored-by: Martin DONADIEU <martindonadieu@gmail.com>
@cursor

cursor Bot commented Sep 23, 2026

Copy link
Copy Markdown

@coderabbitai review

@cursor
cursor Bot deployed to deepsec-pr September 23, 2026 18:19 Active
Re-adding migrations from main that the merge revert removed. Supabase
migrations must stay append-only for CI; this keeps the pg lifecycle PR
scoped while matching main schema history.

Co-authored-by: Martin DONADIEU <martindonadieu@gmail.com>
@cursor
cursor Bot deployed to deepsec-pr September 23, 2026 18:38 Active
@cursor

cursor Bot commented Sep 23, 2026

Copy link
Copy Markdown

@coderabbitai review

@TorichanCapgo

Copy link
Copy Markdown
Contributor

@coderabbitai full review

1 similar comment
@TorichanCapgo

Copy link
Copy Markdown
Contributor

@coderabbitai full review

Co-authored-by: Martin DONADIEU <martindonadieu@gmail.com>
@cursor
cursor Bot deployed to deepsec-pr September 23, 2026 19:06 Active
@TorichanCapgo

Copy link
Copy Markdown
Contributor

@coderabbitai full review

@sonarqubecloud

Copy link
Copy Markdown

@TorichanCapgo

Copy link
Copy Markdown
Contributor

@coderabbitai full review

This branch was successfully deployed

1 active deployment
deepsec-pr — 861ebcda Deployed Sep 23, 2026 by cursor[bot] via Scan PR changes #7768
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants