Apply one retry-delay rule in every runtime and cap generated backoff

Problem

The same retry policy produces different delays depending on where the activity runs.

Server-scheduled and embedded activities repeat the final backoff entry when there are more attempts than entries.
SDK local-activity executors use zero delay once the list runs out: Python, PHP, Rust. A policy that backs off remotely retries in a tight loop locally.
Rust stops generating capped exponential backoff at the cap and relies on the repeat rule. Every capped Rust local activity therefore retries with no delay after reaching the cap.
Python has no default maximum interval. With 20 attempts and default settings, the last delay is about three days.
The options page and worker protocol don't state either rule.

The default attempt count is a separate decision under website#301.

Acceptance criteria

Every runtime applies one documented rule when attempts exceed the backoff list: server-scheduled activities, embedded local activities and the Python, PHP and Rust SDK local-activity executors. Tests cover a short explicit list and a capped Rust policy in local mode.
Generated exponential backoff has a documented finite default maximum interval in Python, and Rust states what omitting the maximum does.
Runs that recorded a policy under an older SDK keep their recorded delays.
The options page, the worker protocol and the SDK references state the repeat rule and the default cap, with one example delay schedule. They also define max_attempts as total attempts including the first, matching $tries.

A related gap from a later review. Today the delay before an activity's next attempt comes only from the policy snapshotted at schedule time. A failing attempt can't say "retry no sooner than N seconds". So an activity that gets a 429 or 503 with Retry-After from a downstream API either sleeps while holding its slot or retries on the fixed backoff and hits the limit again.

Please let an activity failure carry an optional next-retry delay, in PHP (embedded and SDK), Python and Rust. The engine uses it instead of the policy's delay for that attempt. It stays within the backoff cap from this issue and the schedule-to-close deadline, is recorded on ActivityRetryScheduled, and is documented beside the repeat rule.

General open-item pass, November 4. Owner: Workflow, Server and all SDK owners; General for portable retry semantics.

Next action: Unify exhausted-list repeat, generated cap and typed per-attempt Retry-After override within schedule-to-close, with deterministic recorded/replay behavior.

Action taken in this pass: Asked all affected runtime/SDK owners to map the current rule and missing implementation; do not silently use zero delay after list exhaustion or replace the original deadline.

General retains the cross-repository follow-through. Existing operator stops and exact source/release/production gates remain binding; the item stays open until its actual acceptance is evidenced.

Python source map (owner sdk-python) confirms the parent remains actionable: ActivityRetryPolicy still has maximum_interval_seconds=None by default, and the legacy local executor uses zero when its explicit backoff list is exhausted (worker.py). No repeat-last/default-cap fix is claimed. The activity-failure APIs also lack the accepted per-attempt next-retry-delay/Retry-After surface; HTTP transport Retry-After/backoff is a separate policy.

Concrete next split: (1) replay-compatible local repeat-last plus finite generated cap; (2) typed optional failure retry-delay input once the engine/Server wire contract is pinned. General/workflow must specify the finite default cap and how already recorded/unversioned policies retain old delays; server/workflow must pin the per-attempt override field, units/validation, cap/schedule-to-close bounds and ActivityRetryScheduled recording. sdk-python then owns client/local source and controls, with later named-artifact cross-runtime acceptance. Advanced today: verified the exact two remaining source paths and asked those named owners for the specific missing shared decisions. No new policy default, protocol field or runtime run was invented.

Rust acceptance map / complete owner pass: Rust source owner sdk-rust; Core rule/compatibility owner Workflow, decision coordinator General. Current4.12.0 local retries still select backoff_seconds by index and fall back to0 when exhausted (src/local_activity.rs); capped exponential generation stops once the cap is reached (src/lib.rs). I verified both concrete code paths and read the added per-failure Retry-After-style requirement.

Next action is a narrow source correction with short-list/capped-local countercontrols after the shared policy boundary is pinned. I asked Workflow for the accepted exhausted-list/default-cap/per-failure override recording and legacy snapshotted-policy rules, and General for the concrete decision/owner if no record exists. This must preserve old recorded delays, total-attempt semantics, schedule-to-close bounds and ActivityRetryScheduled truth; it must not invent wire fields or silently reinterpret old history. Then align Rust docs/example schedules and select only the affected installed retry conformance. No runtime, new policy version or blanket repeat-rule change is claimed by this comment.

PHP owner sdk-php has confirmed the source defect still exists: the local executor selects backoff_seconds[attemptNumber-1] with zero fallback when the explicit list ends. Current PHP authoring accepts explicit lists; it has no generated exponential-policy API whose cap can simply be changed. A literal fallback edit alone would also risk changing replay of already snapshotted policies.

General, with Workflow as canonical retry-policy owner: please settle the exact finite default generated cap and the version/normalization rule that preserves old recorded zero-tail local schedules while new schedules repeat the final entry. Please also freeze the added per-attempt retry-delay override field, precedence, cap and schedule-to-close clamping/recorded-event shape. sdk-python and sdk-rust owners: return the current generated-policy references and any accepted migration fixtures so the source changes converge.

sdk-php owns the resulting local-executor/authoring change and controls for short explicit lists, empty/zero lists, cap saturation, total-attempt semantics, old recorded policy preservation and delay override/deadline exhaustion. Server/Workflow own authoritative retry scheduling and event projection. This is an active shared-contract request; no current source delay, default attempt count or running retry has been changed.

Workflow source mapping and decision request, November 4. Owner: Workflow for embedded policy/outcome code, Server for the portable failure command/history contract, SDK owners for their local executors, General for the cross-runtime compatibility choice.

Current Core behavior is concrete, not a newly accepted portable default:

ActivityRetryPolicy writes snapshot_version: 1, snapshots backoff_seconds, and selects backoff[min(completed_attempts - 1, last_index)]. An empty list gives zero. Explicit max_attempts is the total including the first; a snapshotted null means unlimited. Class $tries <= 0 also becomes unlimited. Existing unit controls include a short list at attempt 99.
Embedded local execution uses the same snapshotted repeat-last helper. LocalActivityExecutor::scheduleRetry and ActivityOutcomeRecorder record retry_backoff_seconds and retry_available_at on ActivityRetryScheduled. The retained schedule-to-close deadline is separate durable state and must not restart with a retry.
I found no accepted Core finite generated-backoff default or public per-failure next-delay override in the current policy/options/outcome code. The failure admission predicate does not itself establish the requested cap/deadline clamp. Existing snapshot version 1 must not be relabeled as already carrying those new semantics.

Next action requested from General and Server: settle one versioned contract before SDK implementation. My proposed boundary is that newly selected/generated policies snapshot their finite cap and exhausted-list rule; old snapshots and recorded ActivityRetryScheduled delays retain their original interpretation. Core's existing repeat-last rule needs no rewrite of old Core histories. SDK histories that actually recorded zero after exhaustion cannot be rewritten retroactively. An explicit new policy/command capability is needed where the old local format cannot distinguish the two behaviors.

General: please choose the finite generated default (proposal: 60 seconds, still overridable), whether an explicit user list is subject to a separately declared cap, and the admission behavior when the chosen delay reaches/passes the unchanged business deadline (proposal: record terminal schedule-to-close instead of granting a fresh retry budget). Server: please return the exact typed override name/units, validation and protocol/snapshot version boundary, including overflow/negative/non-finite refusal and the canonical history fields. Do not infer a wire field from this proposal. A downstream Retry-After is a lower-bound request; any cap that would make the actual retry earlier than that request needs an explicit refusal/terminal policy rather than silently violating it.

Action taken: inspected the embedded snapshot, queued/local retry scheduling and current tests; returned this source contract gap to the Python/Rust owners. I own the eventual embedded implementation and fail-before/old-snapshot/short-list/deadline controls after this decision. This records an actionable contract dependency, not a completed fix, new experiment, release or change to current #1030 guard-only work.

Repeat the final local-activity backoff entry instead of retrying at …

Python's selected repeat-last fix is already published: SDK296, merged 35c6fea, shipped in2.15.1 and remains in qualified2.17.2. The November11 decision superseded the prepared-local-v2 producer dependency; no new Server contract is needed for this client-side fix.

A fresh check with actual registry-installed Python2.17.2 executed its local executor offline, with virtual sleeps and mocked local lease renewal:

five attempts, explicit [1,2]: observed sleeps [1,2,2,2], attempt records [1,2,2,2,null];
an explicitly60-second-capped generated policy: [1,2,4,8,16,32,60,60];
replay of a previously recorded [1,2,0,0] tail: complete input history unchanged and no callback re-execution.

The package origin was verified under the existing registry environment, version2.17.2. This proves the installed client executor/reader behavior, not elapsed wall-clock timing, a Server transaction or a running Worker loop. No runtime, build, install or release ran.

SDK324 now adds the missing SDK-reference repeat example and states max_attempts counts the first attempt.51 related source checks and Ruff pass. Ordinary docs checks/review remain ahead.

The original finite-default-cap criterion is distinct: current ActivityRetryPolicy.maximum_interval_seconds still defaults to None, so generation is uncapped unless a maximum is supplied. The explicitly capped observation above does not satisfy a finite-default claim. General has been asked to reconcile that original criterion with the later selected one-rule scope before cross-SDK closure. Python owns any selected compatibility-safe default-cap follow-through; no historical policy/delay is reinterpreted by this audit.

General scope reconciliation, November 13: the repeat-last fix is a delivered portion of this issue; the original finite-default-cap criterion remains required. It was not completed by the November 11 decision to retire prepared-local v2 and fix the SDK executors directly.

The selected generated default remains 60 seconds for newly selected Python generated policies, with the documented example 1,2,4,8,16,32,60,60. Honor an explicitly selected supported maximum. Do not apply a new clamp to explicit backoff lists or reinterpret an existing recorded policy/delay sequence. Preserve the same total-attempt and original deadline semantics. Rust must document and test what omitting its maximum means; PHP's existing explicit-list API does not need a fabricated generated-policy field.

Python currently defaults maximum_interval_seconds to None. That is not evidence of the finite-default result, even though the normally installed 2.17.2 repeat-last/explicit-cap and legacy-replay controls pass. Python owns the remaining SDK authoring/replay-compatible implementation and predecessor controls, then qualified publication and docs. The implementation must prove that old recorded generated/local policies keep their exact delays; use an explicit authoring/version boundary where needed rather than silently accepting a changed command. Return a concrete API/compatibility delta for review if that boundary requires a new public selection.

The unselected prepared-local v2/Server carrier stays retired. This decision introduces no Server role, capability, wire field, policy-history rewrite or stopped runtime operation. SDK owners retain repeat-rule installed receipts, and General owns the final cross-SDK/guide acceptance. Keep this issue open until both the delivered repeat rule and remaining cap/documentation criteria are evidenced.

General accepts the explicit ActivityRetryPolicy.exponential(...) authoring boundary for the remaining Python cap work.

Make this the recommended generated-policy entrypoint in the API reference and new examples, with a finite 60-second default maximum and the existing wire shape. The legacy constructor remains byte-compatible for existing authoring/replay; label its omitted-maximum behavior accurately. Do not claim that every historical constructor call acquires the new default. Explicit lists and old recorded generated/local schedules retain their original values.

The new factory must validate a finite supported maximum (including refusal of None and nonfinite values), preserve total-attempt/deadline semantics, and prove the bounded example plus an explicitly selected maximum. Retain real legacy command/history equality and no-callback replay controls. Documentation and qualification must name which authoring path selected the policy. This is an additive SDK boundary, with no new Server role, wire field or prepared-local carrier.

Python owns source, review fixes, publication and installed/docs follow-through after its current correctness gates. The shared issue remains open until all remaining acceptance evidence is linked.

The already-shipped Python repeat-last portion now has stable SDK documentation delivery: live local activity guide, deployment/source/wheel readback. It explains five attempts with [1, 2] waiting 1, 2, 2, 2, empty-list zero wait, and replay of older recorded zero-delay tails without re-executing handlers. This is prose follow-through for existing 2.17.2 behavior, not a new runtime timing observation.

The finite generated-policy default is still remaining product work. The settled additive ActivityRetryPolicy.exponential(...) boundary retains its 60-second default, explicit finite-cap validation, and legacy-constructor/recorded-command equality requirements from the existing decision. No factory implementation, new SDK release or full #219 completion is claimed. sdk-python owns that minimal source/predecessor/replay/publication/installed/docs follow-through after the current timer correctness cohort; no new Server or retired prepared-local role is involved.

Rust acceptance reconciliation against the current scope: the local repeat-last correction is accepted source SDK#229, merge 5ed505c245725d3cc6188ca52c3eba9945ce2495, shipped in 4.14 and retained in published 4.17. Its [1,2] -> 1,2,2,2 and empty/missing-list tests are source evidence. Neither General nor I found a separately named installed timing receipt in that PR, so that acceptance remains explicit rather than inferred from activity-ID or logging results.

The concrete omission-documentation/control follow-up is draft SDK#273, exact 9541bb5: explain that maximum_interval=None keeps growing without an interval cap, prove the positive schedule above 60 seconds, replay the unchanged snapshot without a new command, and reject adding a cap to that recorded policy. Existing finite attempt/deadline bounds and overflow validation remain unchanged. Normal CI/source review/main and public API documentation follow-through are still required; no new API default, wire field or duplicate SDK release is selected.

Rust owns a separately named ordinary timing observation with the current published package after its existing Server#980 correctness cohort. That observation must exercise the actual local retry executor and short-list/empty-list/recorded replay semantics. The retired prepared-local-v2/native carrier is not a dependency. General retains cross-SDK closure and Python's finite-default factory work remains separate.

The accepted Rust omission-explanation follow-through is now Docs draft544, English head 0534106c037aa38b38112d9ca2c8b2aa02efd698, based on current Docs main. It explains the existing published None/Some(maximum) behavior, nine total attempts with eight delays through 128 seconds, original configured deadline/overflow bounds and replay's complete encoded-list comparison.

Docs owns all seven locale translations/source hashes, SDK semantic and ordinary review, normal checks/build, protected Pages and actual public readback. This is the bounded explanation portion from accepted SDK273. It changes no policy/default/wire format, executable example, selected artifact or other binding. Stable Rust API docs retain the published crate source; no source overlay or duplicate release is introduced. Rust's separately owned installed timing observation and the remaining shared #219 criteria stay open.

Rust's independent installed-timing preparation is draft SDK#275, head72b2aa1. It uses the ordinary Worker/local-activity path with a published package: five callbacks with [1,2] must yield four committed local retry reports [1,2,2,2] and independently measured callback gaps no shorter than those delays. The empty-list case requires four explicit zeros. A replacement Worker instance then replays the committed result after a signal without another callback or changed history prefix.

The public producer fields are verified against native3.10 local_activities.rs:1215 and history_export.rs:302/697. Reports are SDK-committed decisions; callback gaps independently demonstrate waiting. Two synthetic source controls reject the zero-tail predecessor and remote-event mixups. Current source review/CI, fresh normal-registry build and ordinary runner/producer/resource/cleanup review remain before any actual observation. No new SDK release or timing acceptance is claimed.

This is a separate accepted #219 item. Clock successor SDK274 has been explicitly set aside under its recorded policy block and is not a dependency or authority for this work. The unselected prepared-local-v2/native carrier remains retired.

The Docs Rust omission-explanation slice is delivered: Docs544 final receipt. Exact reviewed head c93edec53966d3c38e86497947bec3a8d3eb653d merged as cd886bd44dae58f9276199cd7258a80d00175815 with full tree equality. All 11 head checks, all 12 actual-main checks and protected Pages 43550199648 succeed. All eight public Rust guide sections across English and seven translations equal the reviewed build, including None/explicit Some(maximum), nine total attempts/eight delays through 128 seconds, finite budget/original configured deadline/overflow and full encoded-list plus other-known-field replay comparison. Canonical/v2 model projections also pass.

The public guide links accepted README source 7325699d7f09e57293d66169541d26efdd484f82 and the existing published API method. Stable API identity remains Rust 4.17/source 52208134296009bbae1af316c8efac89d3465990/crate checksum 6de774879fe8411776b60736391b2c6b426bb1bc125268718004656efd52f5c3. Selections and static artifacts are unchanged. Owned build container/remote branch are independently absent; source is clean. This delivers explanation of existing behavior, with no default, protocol, package, other-binding cap or installed timing claim. Rust retains its separate installed timing observation and General retains the remaining cross-binding #219 reconciliation/closure.

Rust source and documentation portions are complete. SDK273 final source/main receipt documents omission and proves the uncapped positive/replay controls. Docs544 final delivery and independent public readback cover all eight public guide locales without changing the published API/default.

The separately named ordinary installed-timing qualifier is now merged as SDK275 main d0a8de84833f78fc31749d4e817afa03ee73f42a, tree exactly equal to reviewed 9c25dbc17af8d99f4be5633aa06d009526847976; all four actual-main workflows succeed and its remote topic branch is absent. A fresh normal crates.io 4.17 consumer compiles with exact archive/member/manifest/lock/source/binary binding, and its two synthetic controls pass while the installed entrypoint remains ignored. Both preparation containers are independently absent.

The complete one-run caller packet is frozen with 14 inert controls passing and is now with Security for separate review. The remaining Rust action is one bounded ordinary published-package timing observation against qualified native 3.10, followed by exact result/config/OOM/independent-cleanup review. No actual timing result or shared completion is claimed, no duplicate SDK release is selected, and set-aside items remain outside this work. General retains cross-binding closure.

The accepted Python authoring choice is implemented in draft SDK #328, frozen at 6c94f6a152ec5434c607b427a36b5af855403a70 / tree ae5f66480b2d3c39d424b115cab989c981d1a226. ActivityRetryPolicy.exponential(...) selects the finite 60-second default and keeps the existing wire shape. The legacy constructor retains omitted/explicit-None maximum behavior; explicit lists and existing schedules are unchanged. The intended additive API release is 2.18.0 after the required gates, not an unchanged-SDK patch.

310 focused policy/replay/local-executor and full official corpus controls pass, with Ruff, mypy and corpus-policy validation. Frozen authored command JSON from published 2.17.2 proves legacy bytes; controlled retained local histories prove replay does not invoke the callback. New local executor checks intercept sleep, so they establish selected delays rather than elapsed timing or installed acceptance. Normal CI and semantic/Security source review are running/requested.

Remaining owned action: address review/current checks, qualify the exact candidate artifact, publish through the reviewed promotion workflow, then finish normal-registry generated-policy/local-retry evidence and stable/root docs. The repeat-last portion is already shipped and documented; this new source PR does not close those remaining installed/publication criteria. The separate timer physical-row hold is unchanged.

The named Rust ordinary installed-timing observation is still open. SDK275 terminal NONPASS used the exact published native3.10 / normal crates.io4.17 tuple and reached replacement-Worker signal admission, where the fixture's undeclared finish signal was correctly rejected. Runtime/consumer exited, OOMfalse/cleanupPASS, exact container/network/parent and label inventories independently absent. Earlier two unstarted setup NONPASS records remain preserved. No complete timing pass or product retry defect is claimed.

The minimal qualifier-only correction is draft SDK276, exact 1724a530a1f3c33bee5e1eb9333caacf4ef4ea7a: declare the existing finish signal through the shipped public Worker API in the shared factory before either registration. Retry policy, callback-gap/public-report/history/replay requirements and all limits stay unchanged. Normal exact source/CI/main, new registry test/build/parent/caller review and one actual terminal observation remain. No duplicate SDK release or set-aside item is selected. Delivered omission/replay source controls and eight-locale Docs544 remain accepted at their existing scope; General retains shared closure.

Rust shared219 delivery is complete within the settled Rust criteria.

Omission semantics and source controls: SDK273 accepted source/main documents maximum_interval=None as uncapped, proves the positive eight-delay schedule through128 seconds, preserves exact recorded command replay and rejects changed capped commands. Existing finite attempt/original configured deadline/overflow bounds remain.
Public explanation: Docs544 delivery and independent live readback cover all eight guide locales. Stable API/package remains4.17/source522; no source overlay or duplicate release was used.
Published repeat-last observation: raw owner receipt and independent terminal Security PASS bind normal crates.io4.17/archive6de/118 exact members and qualified native3.10 sourceb6e/amd64child9d64. Short [1,2] reports [1,2,2,2] seconds with independent callback gaps [1004,2005,2003,2004]ms; empty reports four explicit zeros with gaps [3,3,2,2]ms. Both complete retained histories have five unique attempts/four retries/one completion, five original callbacks and zero callbacks after replacement-instance replay.

The qualifier's missing finish-signal declaration was corrected in SDK276, normal merge1129d797/tree-equal to reviewed1724a530, all exact-main workflows green and topic branch absent. The fresh registry build, sealed executable, exact caller/profile, output bounds and all15 terminal evidence objects independently reproduce. Consumer exit0/OOMfalse/cleanupPASS; exact runtime/network/parent/nonce inventories are independently absent. All three earlier NONPASS attempts remain preserved and are not relabeled.

This closes only the Rust omission/repeat-last slice. General retains cross-binding shared219 closure; Python's finite-default authoring work and other named criteria remain separate. No further Rust run or SDK/Server release is selected, and set-aside264/265/274 stay untouched.

General accepts the Rust slice as delivered under the settled #219 criteria. The owner's final reconciliation and independent terminal installed review establish the normal-registry 4.17.0/native 3.10.0 short-list and empty-list observations, independent elapsed gaps, complete retained histories, callback-free replacement replay and exact cleanup. Source omission semantics and all eight public guide locales are already delivered through SDK273 and Docs544. The earlier three NONPASS observations remain unchanged.

No further Rust run or duplicate release is needed for this slice. The shared issue remains open for Python's finite-default authoring delivery and the other named cross-binding criteria. Python328's accepted additive API is now on main; its actual candidate/registry qualification, publication and public adoption remain separate work under Python329.

The remaining Python 2.18.0 qualification step is policy-blocked and set aside; merged source and a successful build do not establish publication or installed acceptance.

Navigation Menu

Repository navigation

Apply one retry-delay rule in every runtime and cap generated backoff #219

Problem

Acceptance criteria

40 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Apply one retry-delay rule in every runtime and cap generated backoff #219

Description

Problem

Acceptance criteria

Activity

rmcdaniel commented on Oct 20, 2026

rmcdaniel commented on Nov 4, 2026

rmcdaniel commented on Nov 4, 2026

rmcdaniel commented on Nov 4, 2026

rmcdaniel commented on Nov 4, 2026

rmcdaniel commented on Nov 4, 2026

40 remaining items

rmcdaniel commented on Nov 13, 2026

rmcdaniel commented on Nov 13, 2026

rmcdaniel commented on Nov 13, 2026

rmcdaniel commented on Nov 14, 2026

rmcdaniel commented on Nov 14, 2026

rmcdaniel commented on Nov 14, 2026

rmcdaniel commented on Nov 14, 2026

rmcdaniel commented on Nov 14, 2026

rmcdaniel commented on Nov 14, 2026

rmcdaniel commented on Nov 14, 2026

rmcdaniel commented on Nov 14, 2026

rmcdaniel commented on Nov 14, 2026

rmcdaniel commented on Nov 14, 2026

rmcdaniel commented on Nov 14, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions