Skip to content

Let failed-run redrive recover ordinary workflows and document it #864

Closed

Description

@rmcdaniel

Problem

Failed-run redrive (workflow#501) continues a failed run in a new linked run. It reuses completed activity results and retries the failed step, which is how users recover from a bad deploy without re-running paid or customer-visible work. Today it refuses most real runs, and users can't find it.

  • FailedRunRedrivePlan refuses any history containing anything other than activity events: signals, updates, timers, side effects, version markers and children.
  • It requires an escaped activity failure. A failure in workflow code or on a structural limit can't be redriven.
  • Runs with memos or search attributes or a parent are refused, and so is a corrected workflow class. That rules out the case where the fix belongs in workflow code.
  • The Server route and the PHP, Python and Rust client methods exist, but no docs page, API reference, CLI reference or Waterline action mentions redrive. The failures guide says a failed run is terminal and lists recovery paths without it.

When a run is refused, the only remaining option is to start over, which re-executes every completed activity.

Acceptance criteria

  • A failed run whose completed prefix includes signals, updates, fired timers, side effects, version markers, memos or search attributes can be redriven. The successor replays those facts in order and never re-executes completed activities or side effects, or re-waits fired timers.
  • A run that failed in workflow code or on a structural limit can be redriven onto a corrected workflow definition. A prefix the new code cannot replay is refused with a typed reason before any new activity is scheduled.
  • Histories that stay unsupported, such as child workflows and local activities, are refused with a stated reason and no mutation.
  • Run detail and describe say whether a failed run is redrivable and why not. Waterline offers redrive on eligible runs.
  • Embedded PHP and published PHP, Python and Rust workers pass the same redrive conformance cases, including cold replay of the successor.
  • The failures guide, the Server API and CLI references, the SDK guides and the Waterline operator API document redrive. They cover what is reused, what runs again, and every refusal reason.

Activity

  1. added
    bugSomething isn't working
    kind:defectA public product behavior is incorrect
    priority:P2Normal-priority product work
    status:readyReady for implementation
    on Oct 20, 2026
  2. rmcdaniel commented on Oct 20, 2026

    @rmcdaniel
    MemberAuthor

    Additional scope from review (Slack miner cases 47836ceb and b2817412): redrive lineage is poorly exposed. The engine creates a linked successor (run_number+1, continued_from_run_id, recovery_kind redrive, and a redrive link), and Server describe returns those fields. But:

    • Waterline labels redrive links as plain Parent/Child diagnostic-only fallbacks.
    • Python's typed WorkflowRun drops continued_from_run_id and recovery_kind.
    • The workflow-id page lists only start and continue-as-new as ways a new run is created.

    Showing redrive lineage in Waterline, the SDK run types and the workflow-id docs fits this issue's docs and Waterline acceptance.

  3. rmcdaniel commented on Nov 3, 2026

    @rmcdaniel
    MemberAuthor

    Progress and A2 slicing (workflow agent, now on a new session).

    A2 is large because each history family has different state, so I'm splitting it into drafts that each add a corpus fixture:

    • A2a: side effects and version markers. History-only facts keyed by step sequence. The successor copies them in order. Like reused activity completions, they carry reused_from_run_id and reused_recorded_at, and replay reads reused_recorded_at so now() after the reused step matches the source run. That adds two keys to SideEffectRecorded and VersionMarkerRecorded in the payload contract. Embedded replay paths: WorkflowExecutor, QueryStateReplayer, WorkflowFiberRunner.
    • A2b: fired timers. The successor records the timer as already fired, with no new timer row to wait on.
    • A2c: signals and updates. These need their workflow_signals / workflow_updates rows and receipts, so they come after A2a/A2b settle the copy mechanics.
    • A2d: workflow-code upserts (memo and search attributes at upserted_at_sequence > 0). This lifts the remaining unsupported_visibility_metadata refusal.

    Service-mode SDK replay has to honor reused_recorded_at on the same events for portability. That goes into (C) once A2a fixes the keys.

  4. rmcdaniel commented on Nov 3, 2026

    @rmcdaniel
    MemberAuthor
  5. rmcdaniel commented on Nov 3, 2026

    @rmcdaniel
    MemberAuthor

    Progress (General, handing the remainder to the Workflow agent):

    Still open here beyond B and C:

  6. rmcdaniel commented on Nov 3, 2026

    @rmcdaniel
    MemberAuthor

    Status (workflow agent): A2d is draft #1001. Workflow-code memo and search-attribute upserts now redrive, with a replay corpus counterfactual. A2c (signals and updates) starts after #877 merges, because #877 changes signal and update receipts.

    A3 proposal: redrive onto a corrected definition

    Acceptance: a run that failed in workflow code or on a structural limit can be redriven onto corrected code, and a prefix the new code can't replay is refused with a typed reason before anything new is scheduled.

    What blocks it today:

    • WorkflowFailed records failed_step_sequence / failed_step_kind only for an escaped activity failure, so the plan refuses with unrecorded_failure_boundary.
    • FailedRunRedriveEligibility::definitionMatches() requires the recorded fingerprint, so workflow_definition_unavailable_or_changed.

    Proposed design:

    1. Record the boundary. When workflow code throws, or a structural limit fails the run, WorkflowFailed records failed_step_sequence as the step cursor at the throw, with failed_step_kind set to workflow_code or structural_limit. Older failures without these keys keep refusing with unrecorded_failure_boundary. No backfill.
    2. Plan. For those kinds, every step before the boundary is the reused prefix, and there is no failed activity step. The successor resumes the new code at the boundary.
    3. Opt in to a changed definition. A different fingerprint is accepted only when the request says so: allow_definition_change: true on redrive(), and the matching Server/CLI option later. It is never silent. With the option off, behavior is unchanged.
    4. Dry-run replay before any write. Inside the redrive transaction, build the candidate successor history in memory (start events plus reused steps) and replay it with the current definition, reusing the WorkflowReplayer/QueryStateReplayer path. If replay raises a step-history or identity mismatch, or yields a new command before the boundary, refuse with definition_incompatible_with_prefix and report the first mismatching step sequence and kind. Nothing is written.
    5. Service mode. The server can't run SDK code, so a definition change on a service-mode run is refused with definition_change_requires_replay_check until server#980 (B) defines a worker-side preflight. Embedded runs get the full feature.

    Corpus: a replay fixture where corrected code redrives a workflow-code failure, plus one where an incompatible prefix is refused without writes.

    I'll start A3 after A2d unless someone objects to steps 3–5.

  7. 39 remaining items

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions