Skip to content

Replay a redriven run from the failed run's start time - #233

Merged
rmcdaniel merged 3 commits into
mainfrom
fix/232-redrive-start-clock
Nov 3, 2026
Merged

rmcdaniel merged 3 commits into
mainfrom
fix/232-redrive-start-clock

Conversation

@rmcdaniel

Copy link
Copy Markdown
Member

Fixes #232. Part (C) of workflow#864.

Problem

Redrive creates a new run whose WorkflowStarted is recorded at redrive time. Core replays that run from the failed run's start, using payload.replayed_started_at when payload.recovery_kind is redrive (RedriveReplayClock::startedAt, WorkflowFiberRunner::workflowStartedAt). The SDK used recorded_at. So ctx.now() and ctx.uuid7() before the failed step returned redrive-time values, and arguments or branches derived from them could differ from the failed run.

Change

  • replay(): on a WorkflowStarted with recovery_kind == "redrive" and a replayed_started_at key, that value is the start clock. An invalid or naive value is refused with the existing explicit clock error, naming payload.replayed_started_at, and never falls back. Without recovery_kind: redrive the key is ignored, as in Core.
  • Cancellation replay clock: a step that carries reused_recorded_at uses it, consistent with RedriveReplayClock::recordedAt.
  • test_history_event_contract.py documents Core's redrive payload keys: resume_step_sequence, recovery_kind and replayed_started_at on WorkflowStarted, and reused_* on ActivityCompleted, TimerFired, SideEffectRecorded and VersionMarkerRecorded. They are excluded from the published-PHP fixture comparison, because ordinary runs never carry them.

Evidence (local, Python 3.13)

  • The new clock tests cover:
    • redrive start time for ctx.now() and the uuid7 timestamp, with a reused activity result;
    • the key ignored without recovery_kind;
    • five invalid values refused.
  • Counterfactual: with only workflow.py reverted, 6 of the 7 new tests fail. The ignored-without-redrive case passes either way, as intended.
  • Full suite: 3,750 passed, 134 skipped, 8 failed. The same 8 fail on unmodified main: TLS fixture certificates rejected by the host's OpenSSL (Missing Authority Key Identifier) and two boundary-scanner tests tied to the host git identity. Hosted CI is the environment check.

🤖 Generated with Claude Code

A redrive successor's WorkflowStarted is recorded at redrive time. Core
replays it from payload.replayed_started_at when recovery_kind is redrive;
the SDK used recorded_at, so ctx.now() and ctx.uuid7() before the failed
step differed from the failed run. Use the replayed start, refuse an invalid
one, and let the cancellation clock prefer reused_recorded_at.

Fixes #232

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@rmcdaniel

Copy link
Copy Markdown
Member Author

Security exact-head review of d5fbb9308dcf358a28b9328c97525caea10e3d95: not accepted yet. The WorkflowStarted redrive clock, invalid/no-fallback behavior, UUIDv7 binding, documented payload keys and new replay corpus fixture look correct. One adjacent clock claim is still wrong and untested.

WorkflowContext._observe_cancellation_replay_time() reads top-level event["timestamp"] first and consults payload.reused_recorded_at only when that value is absent. Core’s authoritative RedriveReplayClock::recordedAt() does the reverse: whenever reused_recorded_at is a string, it is the step time; only otherwise does it use the event recording time. Thus a reused redrive step that also has a top-level timestamp still advances Python’s cancellation replay clock to redrive time rather than the original step time. This can change CancellationContext.remaining() and cleanup branching during replay.

The new test_redriven_run_replays_from_the_failed_run_start_time includes reused_recorded_at, but it never observes cancellation time or calls ctx.now() after the reused step, so the new cancellation branch has no behavioral coverage. Existing cancellation fixtures commonly carry top-level timestamp, which is the important counterfactual.

Please give a string payload.reused_recorded_at precedence over both top-level timestamp and recorded_at, while preserving the existing fallback when it is absent/non-string and fail-closed behavior for an invalid selected string. Add a replay test with both a later top-level timestamp and an earlier reused_recorded_at, then prove a bound cancellation context’s remaining budget uses the earlier reused time. Reverting that precedence should fail the new test. Return the corrected exact head; the start-clock acceptance can carry forward if unchanged.

The cancellation clock read the event's top-level timestamp before
reused_recorded_at, so a step reused by redrive still advanced cleanup
time to the redrive. Prefer reused_recorded_at, as Core's
RedriveReplayClock::recordedAt does, and pin the remaining budget.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@sonarqubecloud

sonarqubecloud Bot commented Nov 3, 2026

Copy link
Copy Markdown

@rmcdaniel

Copy link
Copy Markdown
Member Author

Security re-review of exact head 8db78ed1f8658343e7363175cb04cc706657554f: accepted. This closes the cancellation-clock block from 6610968399; normal hosted checks remain merge gates.

The correction now matches Core ordering exactly: a string payload.reused_recorded_at is selected before either top-level timestamp or recorded_at. Invalid/non-string reused values do not displace the existing event-time path. The new remaining-budget test exercises the previously missing consumer path rather than only testing timestamp selection: a cancellation observed at 00:08 with deadline 00:30.123456 and a reused cleanup step re-recorded at 00:29 but originally recorded at 00:12 must retain 18.123456 seconds. The prior head returns 1.123456, so the counterfactual is exact. The redrive start-clock fail-closed behavior, ignored-without-redrive behavior and corpus fixture from the prior review remain unchanged.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Replay a redriven run with the failed run's start time

1 participant