Repository navigation
Closed
Labels
agent:workflowkind:defectA public product behavior is incorrectA public product behavior is incorrectpriority:P2Normal-priority product workNormal-priority product workstatus:readyReady for implementationReady for implementation
Description
Problem
Failed-run redrive (workflow#501) continues a failed run in a new linked run. It reuses completed activity results and retries the failed step, which is how users recover from a bad deploy without re-running paid or customer-visible work. Today it refuses most real runs, and users can't find it.
FailedRunRedrivePlanrefuses any history containing anything other than activity events: signals, updates, timers, side effects, version markers and children.- It requires an escaped activity failure. A failure in workflow code or on a structural limit can't be redriven.
- Runs with memos or search attributes or a parent are refused, and so is a corrected workflow class. That rules out the case where the fix belongs in workflow code.
- The Server route and the PHP, Python and Rust client methods exist, but no docs page, API reference, CLI reference or Waterline action mentions redrive. The failures guide says a failed run is terminal and lists recovery paths without it.
When a run is refused, the only remaining option is to start over, which re-executes every completed activity.
Acceptance criteria
- A failed run whose completed prefix includes signals, updates, fired timers, side effects, version markers, memos or search attributes can be redriven. The successor replays those facts in order and never re-executes completed activities or side effects, or re-waits fired timers.
- A run that failed in workflow code or on a structural limit can be redriven onto a corrected workflow definition. A prefix the new code cannot replay is refused with a typed reason before any new activity is scheduled.
- Histories that stay unsupported, such as child workflows and local activities, are refused with a stated reason and no mutation.
- Run detail and describe say whether a failed run is redrivable and why not. Waterline offers redrive on eligible runs.
- Embedded PHP and published PHP, Python and Rust workers pass the same redrive conformance cases, including cold replay of the successor.
- The failures guide, the Server API and CLI references, the SDK guides and the Waterline operator API document redrive. They cover what is reused, what runs again, and every refusal reason.
Metadata
Metadata
Assignees
Labels
agent:workflowkind:defectA public product behavior is incorrectA public product behavior is incorrectpriority:P2Normal-priority product workNormal-priority product workstatus:readyReady for implementationReady for implementation
Activity
rmcdaniel commentedon Oct 20, 2026
Additional scope from review (Slack miner cases 47836ceb and b2817412): redrive lineage is poorly exposed. The engine creates a linked successor (run_number+1, continued_from_run_id, recovery_kind redrive, and a redrive link), and Server describe returns those fields. But:
WorkflowRundrops continued_from_run_id and recovery_kind.Showing redrive lineage in Waterline, the SDK run types and the workflow-id docs fits this issue's docs and Waterline acceptance.
test: add redrive start-visibility corpus evidence (#864)
test: add redrive eligibility report corpus evidence (#864)
rmcdaniel commentedon Nov 3, 2026
Progress and A2 slicing (workflow agent, now on a new session).
can_redriveandredrive_blocked_reasonfrom the same decisionredrive()applies. Corpus fixtureredrive-eligibility-reportfails on A1's source and passes on head.A2 is large because each history family has different state, so I'm splitting it into drafts that each add a corpus fixture:
reused_from_run_idandreused_recorded_at, and replay readsreused_recorded_atsonow()after the reused step matches the source run. That adds two keys toSideEffectRecordedandVersionMarkerRecordedin the payload contract. Embedded replay paths:WorkflowExecutor,QueryStateReplayer,WorkflowFiberRunner.workflow_signals/workflow_updatesrows and receipts, so they come after A2a/A2b settle the copy mechanics.upserted_at_sequence > 0). This lifts the remainingunsupported_visibility_metadatarefusal.Service-mode SDK replay has to honor
reused_recorded_aton the same events for portability. That goes into (C) once A2a fixes the keys.test: expect the redrive decision in the describe routing contract (#864
test: give the redrive eligibility fixture its own replay history (#864)
rmcdaniel commentedon Nov 3, 2026
Progress:
reused_recorded_atonSideEffectRecorded,VersionMarkerRecordedandTimerFired, will be filed once Redrive failed runs whose completed steps include side effects or version markers #991 and Redrive failed runs whose completed steps include a fired timer #992 merge and fix those keys.test: redrive reuses one marker for a repeated patch call (#864)
rmcdaniel commentedon Nov 3, 2026
Progress (General, handing the remainder to the Workflow agent):
docs/failures-and-recovery.mdwith all translations. Released behavior goes in the main text. Redrive failed runs that have start-time memos or search attributes #985 and Report redrive eligibility in run detail and describe #990–Redrive failed runs whose completed steps include a fired timer #992 go in a "Not yet released" subsection, to fold in once the release ships. Before this PR, no docs page mentioned redrive.reused_recorded_at, despite the Redrive failed runs whose completed steps include side effects or version markers #991 note.ctx.now()in the new run uses the redrive time, not the failed run's start (replayed_started_at). That is reproduced by cold replay and is now sdk-python#232 (P2).Still open here beyond B and C:
WorkflowRundropscontinued_from_run_idandrecovery_kind.test: give the unsupported child event its own step sequence (#864)
rmcdaniel commentedon Nov 3, 2026
Status (workflow agent): A2d is draft #1001. Workflow-code memo and search-attribute upserts now redrive, with a replay corpus counterfactual. A2c (signals and updates) starts after #877 merges, because #877 changes signal and update receipts.
A3 proposal: redrive onto a corrected definition
Acceptance: a run that failed in workflow code or on a structural limit can be redriven onto corrected code, and a prefix the new code can't replay is refused with a typed reason before anything new is scheduled.
What blocks it today:
WorkflowFailedrecordsfailed_step_sequence/failed_step_kindonly for an escaped activity failure, so the plan refuses withunrecorded_failure_boundary.FailedRunRedriveEligibility::definitionMatches()requires the recorded fingerprint, soworkflow_definition_unavailable_or_changed.Proposed design:
WorkflowFailedrecordsfailed_step_sequenceas the step cursor at the throw, withfailed_step_kindset toworkflow_codeorstructural_limit. Older failures without these keys keep refusing withunrecorded_failure_boundary. No backfill.allow_definition_change: trueonredrive(), and the matching Server/CLI option later. It is never silent. With the option off, behavior is unchanged.WorkflowReplayer/QueryStateReplayerpath. If replay raises a step-history or identity mismatch, or yields a new command before the boundary, refuse withdefinition_incompatible_with_prefixand report the first mismatching step sequence and kind. Nothing is written.definition_change_requires_replay_checkuntil server#980 (B) defines a worker-side preflight. Embedded runs get the full feature.Corpus: a replay fixture where corrected code redrives a workflow-code failure, plus one where an incompatible prefix is refused without writes.
I'll start A3 after A2d unless someone objects to steps 3–5.
39 remaining items
docs: note start-time visibility carryover through redrive (durable-w…
refactor: decide redrive eligibility in one place
test: run detail and describe must report the redrive decision (durab…
fix: report redrive eligibility in run detail and describe
test: redrive must reuse side effects and version markers at their re…
test: add redrive reused-marker corpus evidence (durable-workflow#864)
test: redrive must reuse a fired timer at its recorded time (durable-…
test: add redrive reused-timer corpus evidence (durable-workflow#864)
test: update the redrive plan test for supported side effects (durabl…
fix: redrive runs whose workflow code upserted memos or search attrib…
fix: redrive runs whose completed steps include an applied signal wait (
feat: redrive a workflow-code failure onto a corrected definition (du…
fix: refuse service redrive of reused steps its SDKs can't replay yet (…
fix: redrive runs whose workflow applied an update before the failure (…
fix: report a redrive link as typed lineage (durable-workflow#864)