Skip to content

fix(execution): complete background jobs on workflow failures, fault only on platform errors - #8368

Closed
waleedlatif1 wants to merge 3 commits into
stagingfrom
fix/async-execution-user-failures-not-faults
Closed

waleedlatif1 wants to merge 3 commits into
stagingfrom
fix/async-execution-user-failures-not-faults

Conversation

@waleedlatif1

@waleedlatif1 waleedlatif1 commented Sep 28, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

  • Async API and resume jobs re-threw every execution error, so Trigger.dev failed the run and alerted on user workflow failures (a block raising its own error, a missing required field). workflow-execution also checked core's finalized signal before core set it, so its guard never applied
  • One classifier in lib/workflows/executor/job-failure.ts, built on the existing classifyExecutionError: a failure core recorded and attributed to a block is the workflow's outcome, and the job completes with success: false. Everything else (engine, setup, unrecorded failures, programming errors in Sim code) still faults and alerts
  • Awaits core's post-execution work before classifying (fixes the ordering bug)
  • Failures in Sim's own code still fault even inside a block: uncaused runtime defects (TypeError/ReferenceError/RangeError with no cause) and a new SystemError (@sim/utils/errors) thrown at invariant sites (a declared internal operation with no registered handler) are carried across the executeTool flattening as ToolResponse.isSystemError. A fetch network failure (TypeError('fetch failed', { cause })) is not treated as a defect
  • Missing-required-field validation throws the existing WorkflowValidationError with block attribution, so it counts as the block's outcome (the editor console also names the block now)
  • Applied to workflow-execution, resume-execution and schedule-execution. Schedule previously swallowed every error; it now re-throws platform faults only after its own failure bookkeeping (failure count, next run, claim)
  • /api/jobs and the queue-job status fallback project a completed failure result back to failed, so pollers still see a failed run
  • Webhook is intentionally unchanged: it runs inside webhookIdempotency (retryFailures: true), so re-throwing there would let a provider redelivery re-run a partly executed workflow. Faulting webhook engine errors needs to happen outside the idempotency wrapper, so it's a follow-up

Type of Change

  • Bug fix

Testing

  • New regression tests, each confirmed to fail on the pre-fix code: workflow-execution completes on a recorded block failure, resume-execution completes on a recorded block failure, schedule-execution faults on an unrecorded failure after bookkeeping, and the serializer attributes missing required fields to their block
  • Classifier cases: late finalization, validation errors, unrecorded failures, unattributed engine errors, and programming errors both direct and flattened by a tool
  • executeTool sets isSystemError only for programming errors
  • Type-check, lint, all 51 audits and the full apps/sim suite pass (one unrelated agent-CLI test times out only under full-suite load and passes on its own)

Checklist

  • Code follows project style guidelines
  • Self-reviewed my changes
  • Tests added/updated and passing
  • No new warnings introduced
  • I confirm that I have read and agree to the terms outlined in the Contributor License Agreement (CLA)

…only on platform errors

Async API and resume jobs re-threw every execution error, so Trigger.dev marked
the run failed and alerted on user workflow failures (a block's own error, a
missing required field). The workflow-execution task also checked core's
finalized signal before core had set it, so its existing guard never applied.

- Add one classifier (lib/workflows/executor/job-failure) built on
  classifyExecutionError: a failure core recorded and attributed to a block is
  the workflow's outcome and the job completes with success: false; anything
  else (engine, setup, unrecorded, or a programming error in Sim code) faults.
- Await core's post-execution work before classifying.
- Carry programming errors thrown inside tools across the executeTool
  flattening as ToolResponse.isSystemError so they still fault the job.
- Throw WorkflowValidationError with block attribution for missing required
  fields so pre-execution validation is attributed to its block.
- Apply the rule to workflow-execution, resume-execution and schedule-execution;
  schedule re-throws platform faults only after its own failure bookkeeping.
- Project a completed failure result back to failed in /api/jobs and the
  queue-job status fallback so pollers still see a failed run.
@vercel

vercel Bot commented Sep 28, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment
Project Deployment Actions Updated
docs Skipped Skipped Sep 28, 2026 4:28pm UTC

Request Review

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 19 files

Reply with feedback, questions, or to request a fix.

Fix all with cubic | Re-trigger cubic

Comment thread apps/sim/background/resume-execution.ts
Comment thread packages/utils/src/errors.ts Outdated
Comment thread apps/sim/background/schedule-execution.ts Outdated
@greptile-apps

greptile-apps Bot commented Sep 28, 2026 •

Copy link
Copy Markdown
Contributor

RetriggerConfidence Score: 5/5

[High risk] Changes how workflow execution failures are classified and reported.

The PR appears safe to merge based on the reviewed changes.

Summary

The PR distinguishes recorded workflow failures from platform faults in background jobs, then projects completed jobs with failed workflow results as failed to pollers. It also attributes missing-field validation errors to their block. The change since the previous review types the wait-block regression-test fixture.

Diagram
%%{init: {'theme': 'neutral'}}%%
flowchart TD
  A[Workflow execution throws] --> B[Settle post-execution work]
  B --> C{Recorded workflow outcome?}
  C -->|Yes, not a system fault| D[Complete job with success false]
  C -->|No| E[Fault job]
  D --> F[Project failed status to pollers]
Loading

Reviews (3) · Last reviewed commit: "test(serializer): type the missing-requi..."

Comment thread apps/sim/lib/workflows/executor/job-failure.ts
Comment thread apps/sim/background/async-preprocessing-correlation.test.ts Outdated
…t of system faults

- Add SystemError and isSystemError to @sim/utils/errors; throw SystemError
  where an internal tool operation has no registered handler so a broken
  registry still faults the job after executeTool flattens the error.
- isProgrammingError excludes errors carrying a cause, so fetch's
  TypeError('fetch failed', { cause }) for an unreachable host is not treated
  as a defect in Sim code.
- Track the schedule job fault in a box so a thrown undefined still faults.
- Drop the partial executor-errors mock; the real hasExecutionResult suffices.
@waleedlatif1

Copy link
Copy Markdown
Collaborator Author

@greptileai review — second commit (e3d6098) addresses the prior findings.

Comment thread apps/sim/serializer/index.test.ts Outdated
@waleedlatif1

Copy link
Copy Markdown
Collaborator Author

@greptile

@waleedlatif1

Copy link
Copy Markdown
Collaborator Author

@cubic-dev review

@cubic-dev-ai

cubic-dev-ai Bot commented Sep 28, 2026

Copy link
Copy Markdown
Contributor

@cubic-dev review

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

@waleedlatif1

Copy link
Copy Markdown
Collaborator Author

Closing in favor of a stricter design: page by default and only complete quietly for failures positively tagged as user-caused at their source. The block-attribution rule here could silence platform faults inside blocks (e.g. database errors, handler invariants). Replacement PR incoming.

@waleedlatif1
waleedlatif1 deleted the fix/async-execution-user-failures-not-faults branch September 28, 2026 19:55

This branch was previously deployed

1 inactive deployment
Preview — f631e401 Deployed Sep 28, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant