Repository navigation
Keep Rust workers polling after matching workflow-failure timeout receipts #146
Description
Delivered November 2: Rust SDK 4.10.2 is published and passes installed lifecycle acceptance against native 3.0.0-rc.3, source 04cad60ed661db2c9e961d85e7a132d82ee62e46, image index sha256:899b404b02f856cb582c5e77a95882a3f9928781083d1e7a3fe569f3ff978d6b. All three published SDKs accept matching history-wait timeout receipts for run and execution deadlines and process unrelated work afterward in the same managed process. The normally installed Rust 4.10.1 comparison exits with the matching HTTP 409 while 4.10.2 stays running. Exact acceptance, timing, package identities and scope. SQLite/amd64 and shared-token scope only. Server #794 and Workflow #947 remain open for native workflow-bound Nexus execution, which still returns 501; no PHP Server release is reopened.
Original finding and acceptance:
Rust SDK 4.10.1 exits its managed worker when a history-wait failure acknowledgment receives a valid terminal timeout receipt. A later unrelated workflow stays unprocessed. General owns the correction and installed acceptance.
Observed on November 1 at 19:46 UTC using the normally installed crates.io package, source ebad5c1b729c4f4c728b03de4e7d7d3c915fd8e1, crate SHA-256 84f511bfad4bf0d76aa1723766bdcc911bc912fd95d6d4a3354fcb70949013e8. Its 61 source/schema files match the published source. The runtime was the published Server 2.11.8 image with exactly the unreleased controller from Server #924 overlaid, SHA-256 81a5ef86dbe61695c7d2b2d42479c6f84860d9724cca98bcba97e6350ae5e7aa. Core 2.15.12 and all other Server runtime files remained pristine. This is SDK behavior against a source candidate, not acceptance of a published corrected Server image.
The real two-second run deadline closed through Core and returned HTTP 409 with reason=run_timed_out, outcome=completed, recorded=false, run_status=failed, empty created_task_ids, and the original matching task, attempt and run identities. The SDK worker returned Error::Http on that receipt. Continue-as-new, timeout closure and the next cancellation workflow had run, but the final exception-propagation workflow never entered within its original 12-second bound. The worker's timeout response identifies the timed-out workflow, not the later pending workflow.
In Worker::settle_workflow_task_decision, the command-completion branch recognizes workflow_task_completion_is_terminal_timeout. The empty-decision history-wait branch and other failure-report branches instead propagate fail_workflow_task...().await?. A previous passing lifecycle run exercised different timing and does not establish the failing branch's compatibility.
Acceptance:
- Apply the existing strict timeout identity fence to failure acknowledgments as well as completion, including history waits and ordinary workflow-task failures.
- A matching terminal receipt ends only that task's processing; the same managed worker processes a subsequent unrelated workflow.
- Preserve sticky-cache invalidation, normal failure reporting and retry semantics.
- Wrong task/run/attempt, malformed or nonterminal timeout receipts, and unrelated HTTP errors still propagate. Do not swallow arbitrary 409 responses.
- Add deterministic transport-level coverage of the affected branches, publish a qualified patch, and verify a normally installed published SDK against the corrected immutable Server artifact. Keep Workflow #947 and Server #794 open until the remaining installed lifecycle acceptance is complete.
Activity
rmcdaniel commentedon Nov 1, 2026
SDK 4.10.2 is published. PR #147 merged through protected checks as
ccf58113c2578250e11920f9a40d881e9fb25452. Its tree is identical to the qualified source head380e5b03a05912e907317e9c25843601931db538.The protected publisher succeeded and created the GitHub Release. The immutable annotated tag resolves to the merged commit. Registry, local-package and downloaded-package SHA-256 values all match:
737d844d9e12eb93a186eebfab7d75ee35a32aaefab424d149530a0f68cdd865The downloaded archive records the same clean VCS commit and matches the release checkout. The publisher also resolved a fresh lockfile for
durable-workflow = "=4.10.2"and built it with Rust 1.86. Its small release-evidence artifact records these checks.Installed lifecycle acceptance remains open: General is preparing the published worker and will test it against the corrected immutable native 3.0.0-rc.3 tuple when available. Source qualification and package installation do not establish that result. Duplicate post-merge CI remains queued. The PHP Server release stop remains in force.
rmcdaniel commentedon Nov 1, 2026
The normally installed 4.10.2 lifecycle worker is ready for the immutable native candidate. Its unchanged fixture source builds with Rust 1.90, a separate lockfile and the exact crates.io dependency. The installed crate matches the publisher's registry checksum and clean merged commit. All 61 source/schema files match tag
4.10.2.This verifies worker installation/build and provenance. It does not complete the pending lifecycle checks against native 3.0.0-rc.3. The prior 4.10.1 fixture and its historical evidence are preserved separately.
Cleanup: the merged PR #147 head branch was deleted and its absence verified with GitHub HTTP 404. Its completed clean source worktree is removed. The small published-package diagnostics and installed worker are retained only for #146/#794/#947 acceptance; they can be retired after that result is recorded.
rmcdaniel commentedon Nov 2, 2026
Published-artifact acceptance passes against native 3.0.0-rc.3. The normally installed Rust SDK 4.10.2 managed worker accepts matching terminal history-wait timeout receipts and continues processing unrelated workflows in the same process.
Exact tuple: tag
native-3.0.0-rc.3→ source04cad60ed661db2c9e961d85e7a132d82ee62e46; indexsha256:899b404b02f856cb582c5e77a95882a3f9928781083d1e7a3fe569f3ff978d6b; executed linux/amd64 manifestsha256:93d09f05a75329ed1337ccf15844207c5410a50ea3928aa72b33579a51116730. Release run succeeded and image labels match. The arm64 manifest was identified, not executed.WorkflowTimedOutwithtimeout_kind=run_timeout./failrequest for three seconds without changing its body. The published runtime returns 409,outcome=completed,recorded=false,reason=run_timed_out, failed run, empty created-task list and matching task/attempt/run identities. History has exactly one timeout of the right kind. The same worker then completes a separate continue-as-new workflow with the exact result.fail_workflowcommand through/complete; it does not establish Rust ordinary-failure-path installed coverage. The published Rust source tests cover its failure-report surfaces and negative identity/refusal controls.Error::Httpon the matching history-wait 409 and exits with code 1. Some already-polled comparison cases finish before that exit, so this is specifically a managed-process failure, not a claim that every subsequent task is immediately blocked. The 4.10.2 processes remain running after their cases.Executed November 2, 06:43–06:56 UTC in disposable containers on SQLite, with normal native maintenance enabled and a shared fixture token. This covers eight passing cells / 22 scenarios plus the failing older-SDK comparison. It does not qualify role-only authentication, other databases, arm64 execution or Cloud deployment. PHP/Python sources and the Rust package's 61 source/schema files match published tags; crate archives/checksums and worker binary identities were verified. SDKs and runtime were not patched.
The SDK correction is delivered through 4.10.2. Nexus workflow-bound execution remains a separate 501 capability gap, so Server #794 and Workflow #947 stay open. No PHP Server release is involved.
Keep managed workers polling after a non-recorded task_not_leased rec…
Keep managed workers polling after an exact run_closed receipt