perf(webapp): avoid unindexed fileId scan in get-background-worker-by-version#4245
Conversation
…-version The endpoint loaded each file's tasks via the files.tasks relation, which queries BackgroundWorkerTask by the unindexed fileId column and sequential-scans a very large table for every request. Group task slugs by fileId in memory from the tasks that are already loaded via the indexed workerId relation, and drop the files.tasks include. The response shape is unchanged.
|
WalkthroughThe background worker version endpoint now loads files directly and builds a file ID-to-task-slug mapping from already-loaded background worker tasks. Each response file uses this mapping for its 🚥 Pre-merge checks | ✅ 3 | ❌ 2❌ Failed checks (2 warnings)
✅ Passed checks (3 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
@trigger.dev/build
trigger.dev
@trigger.dev/core
@trigger.dev/python
@trigger.dev/react-hooks
@trigger.dev/redis-worker
@trigger.dev/rsc
@trigger.dev/schema-to-json
@trigger.dev/sdk
commit: |
## Summary 2 new features, 11 improvements, 5 bug fixes. ## Breaking changes - Trigger.dev v3 is no longer supported. For self-hosted deployments, 4.5.0 is the last version we officially support for running v3; stay on 4.5.0 or upgrade to v4. v3 triggers, batch triggers, reschedules, and deploys now return a clear upgrade message instead of running. ([#4236](#4236)) ## Improvements - You can now mark environment variables synced via the `syncEnvVars` build extension as secrets. Return `{ name, value, isSecret: true }` from your callback and those variables are stored redacted in the dashboard, just like manually created secret env vars. ([#4203](#4203)) - Remove the legacy `--mcp` and `--mcp-port` options from the `dev` command. Run the dedicated `trigger mcp` command to start the Trigger.dev MCP server. ([#4246](#4246)) - Removed the unused `ResourceMonitor` export from `@trigger.dev/core/v3/serverOnly`. It was a server-side logging helper with no remaining consumers. ([#4244](#4244)) - Removed the unused `@trigger.dev/core/v3/zodNamespace` export and the legacy v3 socket message schemas. These were only used by the now-retired v3 engine and have no v4 consumers. ([#4236](#4236)) ## Bug fixes - Fix a `chat.agent` message-loss race where sending a message right after an action (such as an undo) could drop the follow-up's response from the UI until a refresh. ([#4234](#4234)) ## Server changes These changes affect the self-hosted Docker image and Trigger.dev Cloud: - Added `EVENT_REPOSITORY_POSTGRES_WRITES_DISABLED` to skip all PostgreSQL task-event writes for deployments that store task events in ClickHouse. Leave it off unless `EVENT_REPOSITORY_DEFAULT_STORE` is `clickhouse_v2`, otherwise task events are lost. ([#4242](#4242)) - Promo credits: a /promo signup landing page, redeeming a promo code when a new org selects a plan, and showing remaining credits on the usage page. ([#4138](#4138)) - Speed up retrieving a background worker by version. The endpoint no longer runs a slow lookup that scanned the full task table for large deployments; it now reuses data it already loads, so the response is the same but returns much faster. ([#4245](#4245)) - Clearer login error when an email address is blocked by the WHITELISTED_EMAILS setting: the message now explains the address isn't allowed on this instance instead of the ambiguous "This email is unauthorized". ([#4220](#4220)) - Make the native build server the default in project build settings. It's now opt-out, stored as a new `disableNativeBuildServer` key. Also clarifies in the UI that build settings apply to GitHub-triggered and native build server deployments. ([#3980](#3980)) - Optionally process high-volume telemetry ingestion in parallel for higher throughput under heavy load by setting `OTEL_TRANSFORM_WORKER_POOL_ENABLED=1`. Off by default. ([#4232](#4232)) - Add a `REALTIME_BACKEND_DEFAULT` env var to choose the default realtime backend (`electric`, `native`, or `shadow`) for environments whose org has no per-org override. Defaults to `electric`, so existing behavior is unchanged. ([#4231](#4231)) - Clarified on the Regions page that a region only affects where your runs execute, not where your data is stored. This shows as a tooltip on the Location column and in the confirmation dialog when you change your default region. ([#4226](#4226)) - Improved the reliability of how run data is read and written. ([#4237](#4237)) - Fixed stale login errors: an error from a previous login attempt (for example a rejected email address) no longer keeps reappearing on the login page and no longer makes later, successful attempts look like they failed. ([#4220](#4220)) - The Errors page now shows better details for each error. Errors that don't carry a message — such as errors thrown without a message, or values thrown that aren't `Error` objects — get a meaningful title instead of all reading "Unknown error", and are grouped by their name (or value) rather than collapsed into a single group. The error type now shows the actual error name, and stack traces now appear where previously they were missing. ([#4225](#4225)) - Return a clear client error when SSO form submissions use an unsupported content type ([#4238](#4238)) - Query page: extracting fields from a run's output with JSON functions (such as JSONExtractString or JSONExtractInt) no longer fails with an "illegal type: JSON" error. ([#4221](#4221)) <details> <summary>Raw changeset output</summary> # Releases ## @trigger.dev/build@4.5.4 ### Patch Changes - You can now mark environment variables synced via the `syncEnvVars` build extension as secrets. Return `{ name, value, isSecret: true }` from your callback and those variables are stored redacted in the dashboard, just like manually created secret env vars. ([#4203](#4203)) - Updated dependencies: - `@trigger.dev/core@4.5.4` ## trigger.dev@4.5.4 ### Patch Changes - Remove the legacy `--mcp` and `--mcp-port` options from the `dev` command. Run the dedicated `trigger mcp` command to start the Trigger.dev MCP server. ([#4246](#4246)) - Updated dependencies: - `@trigger.dev/core@4.5.4` - `@trigger.dev/build@4.5.4` - `@trigger.dev/schema-to-json@4.5.4` ## @trigger.dev/core@4.5.4 ### Patch Changes - Removed the unused `ResourceMonitor` export from `@trigger.dev/core/v3/serverOnly`. It was a server-side logging helper with no remaining consumers. ([#4244](#4244)) - Removed the unused `@trigger.dev/core/v3/zodNamespace` export and the legacy v3 socket message schemas. These were only used by the now-retired v3 engine and have no v4 consumers. ([#4236](#4236)) ## @trigger.dev/python@4.5.4 ### Patch Changes - Updated dependencies: - `@trigger.dev/sdk@4.5.4` - `@trigger.dev/core@4.5.4` - `@trigger.dev/build@4.5.4` ## @trigger.dev/react-hooks@4.5.4 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.4` ## @trigger.dev/redis-worker@4.5.4 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.4` ## @trigger.dev/rsc@4.5.4 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.4` ## @trigger.dev/schema-to-json@4.5.4 ### Patch Changes - Updated dependencies: - `@trigger.dev/core@4.5.4` ## @trigger.dev/sdk@4.5.4 ### Patch Changes - Fix a `chat.agent` message-loss race where sending a message right after an action (such as an undo) could drop the follow-up's response from the UI until a refresh. ([#4234](#4234)) - Updated dependencies: - `@trigger.dev/core@4.5.4` </details> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
What
The
GET /api/v1/projects/:projectRef/background-workers/:envSlug/:versionendpoint loaded each file's tasks through the nestedfiles.tasksrelation. Prisma resolves that as a separate query:BackgroundWorkerTask.fileIdis not indexed — the FK constraint exists, but Postgres does not auto-create an index for foreign keys — so on a large table this can only run as a sequential scan, which gets progressively slower as the table grows and was observed taking minutes per call in production.The loader already loads every task for the worker via
tasks: true, which uses the indexedworkerIdrelation, and those rows already includefileId. This PR groups task slugs byfileIdin memory from that already-loaded data and drops thefiles.tasksinclude entirely.Behavior change (latent bug fix)
The response shape is unchanged, but there is a semantic correction for source files reused across worker versions (files are de-duplicated by
@@unique([projectId, contentHash]), so one file row can be linked to many workers).file.taskscame from theBackgroundWorkerFile.tasksrelation, i.e. everyBackgroundWorkerTaskwith thatfileId— across all workers sharing the file. So a worker's manifest could list tasks it doesn't actually have.file.tasksis grouped from the queried worker's own tasks, so it reflects only that worker version's tasks.Verified on a local DB: 460 files are referenced by tasks from more than one worker; of 6819 (worker, file) pairs, 6 differ — all one file where the old union leaked a task slug (
cancellation-test) into worker versions that never had it. The new per-worker behavior is the correct one for a worker-version manifest. (Thanks to the automated review for flagging this.)Analysis
Captured the exact SQL before/after by instrumenting Prisma against real data (a worker with 62 files):
WHERE "fileId" IN (...)scan.fileIdquery is gone and the other four are identical.EXPLAIN of the two access paths:
No new index is required: the
workerIdaccess path is already covered by the existingBackgroundWorkerTask_workerId_slug_keyunique index.Testing
pnpm run typecheck --filter webapppasses.