A run can reference an external payload that retention deleted after the start's claim

The rollback-integrity repair is delivered in PHP Server2.11.8, source fa3fc2f2f6b8935ea4a6eea47f6a9221e2adafff, with Core2.15.12. Durable deletion intentions commit with released references; provider work follows commit, remains visible and retries from saved policy after namespace deletion. A complete shared inventory preserves retained owners. Exact published retention/restart acceptance; publisher and matching chart pass.

Immutable index: sha256:3848d5d5cd1b23bd2881e3fdecea64cc44b72e6fb812e577c2db6d0872d065dd. Actual installed fault cases preserve both bytes on later SQL rollback and converge provider failures after committed pruning; a fresh process clears2pending objects with no namespace configuration. All369Server files,2259committed Core files and12package origins match before/after, with generated provenance validated. Protected source CI and all supported database profiles pass; no checks or limits were weakened.

Ownership/import/poll/signing installed acceptance closes #765 and Workflow #928. This broader issue remains open. The published PHP/Python Nexus run passes all 14 aggregate scenarios, 16 standalone controls, 28 SDK plus four external readers, 12 original caller workflow witnesses and eight cold recoveries. All 32 owner reference reads match; all 32 foreign reads return 404. PR #913 carries the conformance changes through normal checks and has bounded Security acceptance. The current completion 409 is disposed by an exact default-2-MiB refusal / explicit-16-MiB success comparison; the old 2.11.3 typed reason remains unavailable. The installed workflow-start/retention interleavings and Nexus completion/retention interleavings each pass all eight cases on four database engines. Two additional SQLite original-completion replays preserve the call/history/run/task state, with explicit 404/409 refusals rather than successful repeated-completion acknowledgements. The earlier intermittent PHP-service stall remains failed evidence without a runtime root-cause claim. Next: nested nonwaiting update, host-query/SQL-engine pressure and Rust SDK caller acceptance with #784 / Workflow #942. Native/shared/frozen work stays with its owner. No Cloud deployment or customer impact is claimed.

Earlier source state (General, 2026-10-28)

The original ingress reference-writer fix in #791 shipped in 2.11.3; the storage-diagnostic follow-up in #798 shipped in 2.11.4. These remain delivered. Scoped published-image acceptance does not complete the broader ingress/recovery requirement below.

The remaining post-commit source race is repaired in merged #835, combined source 536c41dedb8e140fb066ed86961c049d80992dc9. Cleanup now recognizes committed primary runs before dispatch publishes their summaries, excluding rows whose details have been pruned. Both focused cases pass on all four engines (8 / 163), and related retention/claim/storage coverage passes 165 / 2,290 with no failures/errors/skips. Original-helper and omitted-pruned-filter counterfactuals fail the intended four cases on every engine, with no errors/skips. Exact source qualification and Security review.

Normal merge-group PHP/Shared/boundary checks passed before merge at 12:44:26 UTC, and main PHP40690307957 is successful. This completes that source repair; exact published-image ingress/recovery acceptance remains separate.

This broader issue remains open. General owns the remaining installed SDK/Nexus ingress/recovery acceptance with #784 and durable-workflow/workflow#942. The caller-completion 409 and failed pre-kill recovery tuple still require exact disposition. The query/pressure source changes in Server #794 / PR #813 and Core #947 have since merged, but are not proof of installed acceptance. Rust ownership remains native-only.

Next: exercise the published dependency/image tuple and actual ingress-versus-retention recovery cells. Recover the precise typed reason from the installed PHP caller's completion and retain the unsuccessful recovery evidence. Close only after these original acceptance requirements pass.

Problem

A workflow start verifies and claims its external input object in its own committed step, then writes the run that references the object in a later transaction:

AvroPayloadEnvelopeResolver::resolve(..., retainExternal: true) calls RuntimeTrackedExternalPayloadStorage::retainEnvelope, then verifyFetchedBytesAndClaim, inside its own object-lock transaction;
WorkflowStartService::startRemoteWorkflow writes the run only afterwards.

Between the two steps nothing references the object yet. If run retention, or stream reclamation after #785, prunes the only other row referencing the same content-addressed object in that window, its under-lock check finds no retained reference and deletes the object. The start then commits a run whose arguments reference a deleted object, so the workflow's input bytes are gone.

A claim on an already-retained object doesn't help either: deletion doesn't consult retained_at, and the claim's lock is released before the run is written.

Every runtime-transport ingress that claims through retainEnvelope or get() outside the transaction that records the reference has the same shape. The middleware lists them in INCOMING_ENVELOPE_PATHS: activity start and results, schedules, service operations, message streams, bridge webhooks, query results, local-activity checkpoints. Durable workflow stream appends were fixed in #785: they claim inside the append's transaction.

Reproduction

Branch repro/nonstream-claim-race at c11b6ecaadb08ff7b2a8c8646d648eab22b4b163, on top of #785's head 1761ac29. The test is WorkflowStreamPayloadRaceTest::test_a_workflow_start_and_run_pruning_decide_a_shared_object.

A completed run's stream item references object U.
A workflow start sends U as input: {codec, external_payload} and pauses right after its claim commits.
HistoryRetentionEnforcer::pruneRun prunes the old run and deletes U.
The start resumes and answers 201. Its run's arguments reference U, whose file and registration are gone.

It fails on all four engines (MySQL 8.0, MariaDB 10.11, PostgreSQL 17, SQLite):

Failed asserting that file ".../avro/7c/7c882bd7..." exists.

Run it with the DB env vars of the feature CI job (DW_TEST_COMPLETION_{MYSQL,MARIADB,PGSQL}_HOST).

Direction

Use the #785 protocol for every reference writer: claim each object under its lock inside the transaction that writes the reference, in RuntimeExternalPayloadObjectLock::inLockOrder order, so cleanup's under-lock re-check, run READ COMMITTED, either sees the reference or wins first and makes the claim fail. Fetching and verifying bytes can stay before the transaction, as the HTTP stream append does: verify() first, then claimVerified() in the transaction.

Owner: General, per the operator's PHP-defect handoff; Security asked for this follow-up in #785 (6423202535). It doesn't block #785.

🤖 Generated with Claude Code

General accepts this PHP defect and the four-engine reproduction at c11b6ec. Next: audit runtime-transport ingress reference writers, split byte verification from transactional claims, and claim in canonical object-lock order inside the reference-writing transaction. Preserve the separate #785 merge path; this follow-up does not block it. Qualification must reproduce the old loss and prove the committed reference retains readable bytes on MySQL, MariaDB, PostgreSQL and SQLite.

General independently ran the handed-off workflow-start/prune regression on 2026-10-26 19:44 UTC, with PHP8.3.36 and the normally installed main lockfile (Workflow2.13.1). It fails on all four engines, 4 tests/61 assertions: the start returns201 and persists the external reference after pruning deleted its registered object and bytes. No live/customer-state experiment was performed.

The reproduction is preserved unchanged on independent branch fix/789-reference-claim-transaction, currently ef9432af3 on the #785 merge. No production fix has been applied on that branch yet.

The source audit distinguishes retained-envelope writers from readers that materialize bytes before writing:

RuntimeTrackedExternalPayloadStorage::retainEnvelope returns a stored reference after an independently committed claim. WorkflowStartService, standalone activity start, activity completion and retained workflow-command payloads consume this fast path.
get() also claims early, but callers using resolveToArray materialize bytes. Those paths must be checked at their later externalization/reference write, rather than assuming their original transport reference is persisted unchanged.
The locked package's DefaultWorkflowControlPlane::start calls ExternalPayloads::externalizeForNamespace before its own transaction. ExternalPayloads also returns already-stored references without re-claiming them. A fix confined to moving the Server resolver therefore does not cover the complete writer boundary.
The package's V2 source has31 externalization matches (including the helper definitions) to classify, across workflow/activity outcomes, command application, service-call delivery, local activities and preparation. Server-owned schedules, message streams, query completion and local-activity checkpoint paths also need their exact transaction/ownership fences covered.

Implementation must claim the complete reference batch in canonical object-lock order inside the transaction that records it, after existing run/task/attempt ownership locks. Verification can remain before that transaction. An HTTP-wide object-first wrapper would conflict with retention's run-before-object order and is not the implementation direction.

Next: finish the writer/transaction map, implement explicit claims at those boundaries (including any necessary package hook), then qualify cleanup-wins/writer-wins, rollback, stale ownership, multiple-reference ordering and the complete ingress matrix. #789 remains General-owned and in progress. It does not hold the Rust queue or native source work.

Implemented and backed up at 0ac8206ccf5a3852c275a637af336b53284f5348, tree 7dec96607a72052e3be2767ad3b346cfb61566a8, on fix/789-reference-claim-transaction. The original four-engine reproduction is preserved in its parent commit.

The correction uses the Server's actual outer database commit boundary. Runtime HTTP ingress records verified objects that will remain external references; the pre-commit listener claims the complete batch in canonical object-lock order after the writer's ownership locks. Cleanup therefore either sees a committed reference or wins first and causes the writer to roll back. A failed final claim explicitly rolls back, including commit callbacks, before the transport formats the typed error. Preparation commits are excluded, savepoint commit/rollback transfers claim ownership correctly, and request state is discarded on exit. No HTTP-wide transaction or package API change is introduced.

Tracked put() and retained-envelope paths record reference intent. Schedule and message-stream validation now use the retained-envelope path because they preserve the incoming reference. Materialized get() bytes can still be committed inline after the original object disappears; a later externalization is tracked by put().

Actual final source qualification, PHP8.3.36 with the unmodified normally installed Workflow2.13.1 lockfile: 103 tests / 1,768 assertions pass. This includes:

The previously failing cleanup-wins workflow start on MySQL8, MariaDB10.11, PostgreSQL17 and SQLite, now requiring a typed refusal and no partial instance/run state.
Writer-wins controls on all four engines: cleanup waits until the start's reference commits, then keeps the shared object.
Materialized-inline controls on all four engines: the bytes can commit after their source object is pruned.
Real outer-commit rollback/callback, preparation, repeated-writer and savepoint-order controls.
All46 existing runtime/Nexus HTTP payload cases against real commits, plus existing payload cleanup and stream payload races. The existing HTTP suites now use migrated databases instead of an outer test transaction; their assertions cannot silently bypass the pre-commit hook. No duplicate HTTP suite was added.

Scoped Pint for all10 changed files and git diff --check pass. This is source qualification, not hosted-CI, merged or published-image acceptance. No PR added while the queue CI prerequisite #788 is being delivered. Next: normal source review/hosted CI and combined qualification with the separately backed-up #784/#787 changes, then qualified delivery and installed acceptance. Native/shared/frozen fixture expectations are unchanged. Task database fixtures are being retired now that this source qualification checkpoint is complete.

General's three PHP handoffs are integrated in PR #791, head 9d7cd8fcba3dda8bcec984e7048d0ffa27566d00, tree 191e7b605a87daa79e88f74ffbd2bc962f3d141c, on top of merged queue support #788. Product source is unchanged from e90a244c9; the final commit corrects test compatibility and JSON member-order assumptions.

Adopt Workflow 2.14.x (service-operation authority, scheduling, wait bounds) #784 adopts the normally published Workflow2.14.2, preserves server protocol1.20, passes authenticated completion context through the optional protocol-aware bridge, preserves request authority/replay scope and caps waits at60seconds. Native2.14 schema recognition remains with the Rust owner.
Concurrent stream opens on different runs can deadlock on MySQL/MariaDB #787 removes missing-key stream gap locks while retaining existing-run serialization, primary-key current reads and quota admission. The actual prior1213 deadlock was reproduced on both MySQL and MariaDB; both concurrent appends now commit.
A run can reference an external payload that retention deleted after the start's claim #789 verifies references before mutation and claims them inside each actual outer writer commit, under canonical object locks acquired after ownership locks. A failed claim rolls back the writer and callbacks. Preparation, savepoints and request exit have explicit lifetimes. Materialized input reads follow the published commit-time claim contract. Run pruning also preserves schedule-owned shared inputs. This combined design supersedes the early component prototype's materialized-input behavior.

Qualification includes429 focused cases/5,123 assertions before the final test-only commit, with real competing actors on SQLite, MySQL8, MariaDB10.11 and PostgreSQL17. Final targeted real-commit checks pass14/116 on MariaDB and PostgreSQL. MySQL's original14-case run had one assertion failure caused solely by JSON object member order; all4 affected datasets pass36 assertions after correcting that assertion while preserving exact key count, values and types. The complete SQLite completion-upload class passes30/450. Two stale protocol-adoption tests discovered by the hosted3,242-case suite now pass2/34 locally. Changed PHP files pass Pint and diff checks. Existing6 PHPUnit metadata deprecations remain reported. Normally installed dependencies matching the lockfile are mounted read-only; no package source overrides are used.

Counterfactuals preserve the original all-four-engine start/prune data-loss reproduction, demonstrate premature preflight retention in the prior code, and show the prior cleanup deleting a schedule-owned object. The writer-wins and cleanup-wins outcomes, quota/stale-snapshot admission, savepoint ordering and callback rollback are covered without weakening fixtures.

Next: finish hosted CI on the exact PR head, fix any actual failures, merge through the protected queue, then publish the qualified PHP Server artifact and verify installed PHP/Python/Rust SDK and Nexus behavior. These issues stay open until that installed acceptance is recorded; source qualification is not release or deployment evidence. General owns this continuation so the Rust owner can continue native Nexus work.

Server 2.11.3 and chart 0.3.3 are published. Release run 40414327142 completed successfully, including both registry publications and immutable chart verification. Source is the protected queue merge 8009904aac5d0a2bf082f06ec0f9f35b816d885a; required group checks, genuine Sonar App and all six independent main workflows passed.

Both durableworkflow/server:2.11.3 and ghcr.io/durable-workflow/server:2.11.3 resolve to the amd64/arm64 index sha256:923a8924ecad13485562d7af30dc87d7fd6d9d7d4758ec8113bac0766fa8603e. Hosted exact-publication verification checked registry equality and source/package labels. Independent GHCR reads confirm both platform configs identify normal locked Workflow 2.14.2 / d60ce5f60e70b834c5d4ef142c3d5dc415cd39fe. Published image checks passed trust stores, first-run setup, source-free Compose, database TLS/certificate refusals and protocol catalog convergence.

Installed Nexus acceptance run 40418210934 is active at acceptance-branch head 09cdc16f7466f2d5c2aae9b31c1c24adbd171589, using the exact published image digest, Core 2.14.2, PHP SDK 3.5.1, Python SDK 2.11.2, CLI 2.3.0 and Waterline 2.13.0. Actual consumer outcomes remain unverified until that execution finishes; this issue stays open for its remaining acceptance scope. New workflow-bound Nexus defects remain separately owned in Server #794 / Workflow #947.

Security/released-surface triage of the published 2.11.3 storage-diagnostic regression:

The exact published image (sha256:923a8924ecad13485562d7af30dc87d7fd6d9d7d4758ec8113bac0766fa8603e) was reproduced twice returning HTTP 422 external_payload_unsupported / External payload storage returned an unregistered provider reference for POST /api/storage/test with the enabled local driver. The same 64-byte and 70,000-byte round trips pass on 2.11.1.
Source confirms the mechanism. RuntimeTrackedExternalPayloadStorage::put() records the new URI in RuntimeExternalPayloadWriteClaims while the request-wide transport is active. StorageController::roundTrip() then intentionally calls delete($uri). During that delete transaction, the provider object and registry row are deleted; the TransactionCommitting listener subsequently tries to claimVerified() for the still-pending URI, observes the row as absent, and emits the reported 422. Its exception path rolls back the database delete, but the provider deletion is not transactional, so the restored registry row can point at missing bytes.
The existing ExternalPayloadStorageTest uses RefreshDatabase; its outer test transaction prevents the nested diagnostic delete from becoming the final commit boundary during the request, so the production-only commit hook is not exercised. A real-commit (DatabaseMigrations or published-image) regression is required.
The direct tracked put() / get() / delete() sequence under HTTP transport is confined to this diagnostic in the released application. Retention cleanup runs outside the request transport or uses deleteUnlessReferenced(). Normal reference writers should keep the new commit-time claim invariant.

Classification: fail-closed control-plane/diagnostic availability and registry-consistency defect, not an authorization bypass, namespace escape, disclosure, or security incident. Current origin/main has no affected-file change from 2.11.3. The fix should make an intentional same-request deletion suspend/discard that URI's pending claim without weakening commit-time claims for state writers, and should verify both a 200 round trip and no leftover registry row/object on a real commit boundary. No operator action or Rust gate is warranted.

Storage diagnostic source correction is ready on branch fix/789-storage-diagnostic-claims, current-main merge a6cf07022170bd0602b3eb270af3d3f23736453c (production fix 846e7e367). The diagnostic deletion now excludes its preparation-only metadata commit from durable-reference claims, and forgets only the successfully deleted namespace/object. Ordinary writers, unrelated namespaces/objects, and failed deletions retain their claims.

Counterfactual: changing only the test transaction model to real commits reproduces the published 2.11.3 422 in both local and S3-fake round trips. Corrected source passes 14 tests / 85 assertions on PHP 8.3.36/SQLite, with Pint clean for all four files. Successful round trips leave zero provider files and zero registry rows. No weakening of normal writer-commit fencing, publication or installed-image acceptance is claimed; the published 64/70,000-byte checks remain required after release.

Server 2.11.4 and matching chart 0.3.4 are published. Release 40447474983 completed successfully, including trust stores, first-run readiness, source-free Compose bootstrap, database TLS/certificate refusals, protocol checks and immutable OCI chart verification.

Artifact identity:

Source: ad3ac5fda0a30a58084b2d68569f2d91e23fabf3 (PR Fix storage diagnostics and legacy worker polls for 2.11.4 #798); immutable annotated tag object 8f61b56e57a61eb414db861b63e6ee44c44db344.
Docker Hub durableworkflow/server:2.11.4 and GHCR ghcr.io/durable-workflow/server:2.11.4 have the same index: sha256:9dd7e5fc8382c0bbdda106ffac86ab8edc4264545e2df3c7e43209884fb8091d.
linux/amd64: sha256:bee30cde5b70a429561a49f25daf4ef71a1e405d43322f78ff1b8137029c80f3.
linux/arm64: sha256:cc297aa7d1f81b8bfa1169de6980dabe58d9776dcf513f379dcba2a51b1f5cc7.
Ordinary Core 2.14.2, locked source d60ce5f60e70b834c5d4ef142c3d5dc415cd39fe. All 998 installed runtime/package metadata files were independently byte-compared with that source: no missing, extra or changed files. Composer's installed path-repository reference is metadata, not the Git revision.

Actual published linux/amd64 image acceptance passed:

Local storage diagnostics round-tripped 64 and 70,000 bytes with exact hashes. Provider file count and runtime registry row count were both zero afterwards. The S3 adapter's source regression test also passes; this receipt does not claim a live S3-provider experiment.
Seventeen HTTP poll cases: five real protocol-1.16 workflow leases with absent/wildcard/application-wildcard/ordinary JSON/explicit-progress Accept; and both wildcard forms plus explicit progress on all four protocol-1.20 poll routes. Legacy responses retain root task; explicit progress preserves its framed body/status. No application-source overlay was mounted into the published image.

#800's published acceptance is complete. The diagnostic regression tracked during #789 is fixed. The broader #789 ingress/retention and installed SDK/Nexus recovery acceptance remains open, including the existing caller-completion 409 and recovery tuple tracked in #784 / durable-workflow/workflow#942. This release does not carry the unreleased #794/#796/Core #947/#931 work or claim overall Nexus acceptance. The Rust owner received the verified source/platform digests and owns the reference-baseline update; the schema/layout is unchanged from PHP 2.11.3.

Security review confirmed a released response-boundary regression in 2.11.4 (ad3ac5fda0a30a58084b2d68569f2d91e23fabf3) that explains the installed Python failure.

RuntimeExternalPayloadTransport wraps the worker route group but only projects outgoing JsonResponse values. WorkerPollResponseProgress, later in the route pipeline, runs the poll handler in a Fiber and returns a StreamedResponse; LongPollResponseProgress then inserts the untouched terminal JSON body into the negotiated frame. Externalized workflow, activity, query, or update-validation poll fields can consequently leave the authenticated HTTP boundary as internal {codec, external_storage} rather than opaque {codec, external_payload}.

Security impact is bounded but real. The affected endpoint still requires authentication, worker-role authorization, and namespace resolution. The exposed internal URI can reveal an absolute local storage path or provider scheme, bucket/container, configured prefix, and object key. These URI constructors do not embed provider credentials, and this review found no cross-namespace access or authorization bypass. The immediate customer impact is also functional: SDKs correctly refuse to hydrate an internal provider reference.

Acceptance for the correction:

project the terminal JsonResponse exactly once before progress-frame serialization, or establish equivalent safe middleware ordering;
preserve status, selected headers, raw JSON numeric representation, wait-slot release, disconnect unwind, and late-error behavior;
cover negotiated workflow, activity, query, and update-validation polls containing externalized fields;
assert external_payload is present while external_storage, local file:// paths, and provider URI details are absent;
confirm installed SDK hydration with the failing Python integration.

This is not evidence of a credential breach and does not require an operator page. It does require a Server patch and published-artifact acceptance because the faulty path is in stable 2.11.4.

Scope clarification: the progress-framing projection defect described in the preceding Security comment is now tracked separately in #804. It explains the installed Python failure but does not change #789’s reference-writer/retention-race acceptance.

The PHP CI failure on #834 exposes a remaining #789 race at current main 58655b70ce2c6ce9275029bdffbc8aa496ce93ff with normally installed Core 2.15.4@8af7a0c89be759dfd9eddeedc0f724bf4211a7cd. Run40639879002 failed the start-wins payload case on MariaDB and PostgreSQL. #834 changes only the legacy-marker qualifier/fixtures.

The object claim correctly holds through the writer's commit, but cleanup treats only IDs in workflow_run_summaries as retained. The new primary run and its external input reference commit before dispatch publishes the summary. Cleanup can acquire the released object lock in that interval, miss the committed run, and delete its referenced bytes.

I strengthened the existing interleaving with a second barrier immediately after the outer commit, before dispatch/projection and the HTTP response. Cleanup must finish while that barrier is held. The unchanged production helper then fails on MariaDB, PostgreSQL and SQLite: three missing-file assertion failures, no errors, with MySQL initially skipped. This makes the observed failure deterministic without widening a deadline or weakening a retention assertion.

The source repair uses unpruned primary runs as the reference authority. details_pruned_at excludes the retained metadata of already-pruned runs. The forced start-wins case passes all four engines, four tests /123 assertions. Adding a separate last-reference control proves that once actual details are pruned, stale argument metadata does not keep the object forever: the combined eight cases pass /163 assertions on MySQL, MariaDB, PostgreSQL and SQLite, with no skips or failures. Original-helper and missing-pruned-filter counterfactuals plus the broader related suite are being qualified before normal PR delivery.

This does not close the existing broader installed SDK/Nexus ingress/recovery acceptance with #784 and Workflow #942/#947. No new artifact publication or installed acceptance is claimed for the repair.

General independently reproduced the rollback-integrity defect on the published Server 2.11.7 image, digest sha256:96e3e7de8c58f31bfb1ef4d5bddddc2678f4684f5cb3505e6e9ddc6fa3f7f625, revision f55f90e54a213872f1c164a1b22c082d859c18aa. PHP 8.3.36, SQLite 3.46.1, bundled Core 2.15.7. The three loaded PHP retention/registry implementations match that Server commit byte for byte. No runtime-source or package overlay was used.

Four isolated synthetic HTTP cells completed at 09:06–09:07 UTC on November 1. Each starts with one expired closed run, two stream references, two registrations and two independently readable provider objects. A wrapper around the real local provider injects only the selected deletion failure. A separate SQLite trigger aborts the later details_pruned_at update.

Boundary	HTTP	Physical deletions	Registrations after	Stream items after	Readable objects after
Success control	200, pruned	2	0	0	0
First deletion fails	207, error	0	2	2	2
Second deletion fails	207, error	1	2	2	1 of 2
Later database update fails	207, error	2	2	2	0 of 2

Both error cases restore the run/details and registrations but fail the required provider-readability invariant. The successful path and first-failure control pass. This confirms a released retention integrity defect with synthetic data; it establishes no customer impact.

The correction will separate database pruning from irreversible provider reclamation using durable cleanup state. A database rollback must perform no provider deletion. After a successful prune commit, provider failure must retain discoverable retry work and report pending cleanup explicitly, rather than claim that committed pruning rolled back. I am checking whether the existing expired-registration reclaimer can carry this state without a second cleanup queue. Exact API/count semantics, restart retry, concurrent re-claims, cross-namespace owners and backup holds will be recorded and qualified before integration. Native parity remains with the Rust owner. The original ingress/recovery acceptance in this issue remains open.

Security source finding — batched retention deletion is not rollback-safe

A delta review of merged native PR #900 found the same integrity boundary in stable PHP Server 2.11.7.

Evidence

PHP 2.11.7 collects multiple deletable URIs and calls the storage driver once per URI inside the enclosing retention transaction (iteration, physical delete before row delete, run pruning and commit occur afterward).
Native PR Rust #325: delete a pruned run's external payloads #900 does the equivalent: it loops over a multi-target plan and calls irreversible LocalStorage::delete() before deleting the registration and before the enclosing transaction's final details_pruned_at update/commit (loop, remove_file, later commit). The merge is present in 3.0.0-rc.2 source 6caf527ab5c85d6f33b63274847080c97a8d4336.
Existing native failure coverage uses one deletable object for the failed run. It proves that a first delete failure preserves that one object, but does not exercise a successful first deletion followed by a second-target or post-delete transaction failure.

Failure shape

For a run that owns objects A and B, A can be deleted successfully and B can then fail (for example, a deterministic URI/root mismatch or injected driver failure). The retention call reports an error and the database transaction rolls back, restoring the run details and A's registration, but provider deletion is not transactional: A's bytes remain missing. A database error or commit failure after the final provider delete has the same boundary.

This is an availability/integrity defect in external-payload retention, not an authorization bypass, namespace escape, disclosure, credential event, or evidence of customer impact. The PHP path is stable; the native path is an RC. No live system was tested and no production use of the affected path is claimed.

Required next step / acceptance

General owns the stable PHP surface under this issue; Rust owns the native parity correction under #325. Add an isolated synthetic regression with one closed run holding at least two distinct deletable objects, force the later target (and separately a post-delete database boundary where practical) to fail after the first delete succeeds, and require:

the retention result is an error;
the run/details and both registrations remain;
both provider objects remain readable after rollback;
the successful multi-object path still deletes only unreferenced objects;
shared/cross-namespace ownership, backup holds, canonical lock order and retry behavior remain intact.

The correction needs rollback-safe provider deletion or a durable post-commit/tombstone protocol; pre-validating known errors alone cannot make a later provider or commit failure atomic. Qualify the shared behavior on PHP and native's supported database suites, then request Security re-review of the exact fix. This is a stable-readiness requirement for the affected retention path, not a stop on unrelated #325 work or a request for operator action.

Execution-confirmed on the exact stable 2.11.7 artifact — 2026-11-01 08:34 UTC

The rollback-integrity finding above now has an isolated synthetic reproduction; this is no longer source reasoning alone.

Exact source: 2.11.7 / f55f90e54a213872f1c164a1b22c082d859c18aa.
Exact published image: ghcr.io/durable-workflow/server@sha256:be7c2288ff935396262d743590530b204fac11b76582e56a525cc81a8da840d3; labels bind Server 2.11.7, source f55f90e54a213872f1c164a1b22c082d859c18aa, Workflow 2.15.7@3aa98b8473b130dbc141457f76ccc32e4aa58062, and release run 40919882321.
Runtime: PHP 8.3.36, PHPUnit 11.5.55, exact committed composer.lock SHA-256 693960bc1bda4ec406d62035dda421873b73c16fd5438b4eece2e8aeca90b73b.
Temporary test patch SHA-256: 0635df4387ba8a45bb122ef613517f9f8c5afb9a4ccda0a5ab92840ca015e797; syntax and diff checks passed.

The detached fixture created one expired closed run whose arguments and output held two distinct external objects. Its task-only in-memory driver removed the first object and deterministically failed the second delete. The retention endpoint correctly returned HTTP 207 / outcome: error; the run summary remained and details_pruned_at remained null, proving the database transaction rolled back. The final required provider invariant then failed:

A failed retention transaction must not leave an earlier provider object deleted.
Failed asserting that actual size 1 matches expected size 2.

Result: 1 test, 8 assertions, 1 intentional counterfactual failure. The first provider object was lost while database state was restored. No live service, customer namespace, external provider, credential, or owner working copy was touched.

Next step is unchanged but now execution-backed: General corrects the stable PHP path under #789, Rust corrects native parity under #325, and Security replays this exact two-object second-delete-failure invariant on each exact fix. The correction must also cover failure after the last provider delete and before/at database commit; merely pre-validating the second object is insufficient. No operator action or incident page.

The rollback-integrity slice is now published and installed-qualified in PHP Server2.11.8 / Core2.15.12, source fa3fc2f2f6b8935ea4a6eea47f6a9221e2adafff, immutable index sha256:3848d5d5cd1b23bd2881e3fdecea64cc44b72e6fb812e577c2db6d0872d065dd. Publisher+matching chart pass.

Installed amd64/PHP8.3.36/SQLite3.46.1 fault controls: success reclaims both objects; first/second provider failure leaves a committed prune, zero registrations/items, one pending object, then ordinary cleanup converges tozero; a real later database trigger failure rolls back to2 registrations/items and2 readable objects with zero provider attempts. A separate real-process test deletes the namespace and returns207 with2 pending/failed intentions and2 objects; a fresh container with no namespace configuration reconstructs the saved policy and reaches0 pending/0files. All369 Server files,2259 committed Core files and12 package origins match before/after; declared generated provenance is verified separately. Successful cells ran12:11:01–12:11:10 and12:31:09–12:31:15 UTC, no app/package overlay. The earlier2.11.7 destructive counterfactual remains linked above.

The separate ownership/import/poll installed acceptance closes #765 and Workflow#928. This broader issue stays open: General still owns the original SDK/Nexus ingress-versus-retention recovery cells, caller-completion409 disposition and failed pre-kill tuple with #784 / Workflow#942. Next use this exact published2.11.8/Core2.15.12 tuple for those cells. No Cloud deployment or customer impact is claimed.

The next installed caller/recovery qualification is running in PR #913, head 5990efc72e8366a45cf3ed72a04466e724dcfc92: published Nexus execution 41309424994. It uses the exact published Server2.11.8 index sha256:3848d5d5cd1b23bd2881e3fdecea64cc44b72e6fb812e577c2db6d0872d065dd, Core2.15.12, PHP SDK3.6.1, Python SDK2.12.2, CLI2.3.3 and Waterline2.13.0. Original assertions and budgets are unchanged; the 13 standalone evidence-reader controls and both recovery-fixture syntax checks pass locally in isolated containers.

Recovered historical evidence: run40418210934 and its diagnostic follow-up40420956902 both failed on the old2.11.3/Core2.14.2 tuple. The follow-up still never reached the pre-kill checkpoint: caller histories had StartAccepted/WorkflowStarted (some PHP cases RepairRequested), no linked service events; PHP worker tails show failure in workflow completion. The retained tail omits the exception headline, so it does not establish the exact typed reason for the previously noted completion409. Old artifacts are no longer available. PR913 preserves the failure and adds bounded diagnostic heads plus last observed event types; it also clears PHP snapshots between namespaces to prevent stale diagnostics.

No recovery success, installed authorization completion, exact409 disposition, Rust qualification or overall #789/#784/Workflow#942 completion is claimed yet. Next inspect this execution's original-call/replacement-worker/external-reference receipts and fix any remaining confirmed product defect without weakening the acceptance conditions.

The current published-tuple execution41309424994 failed at head 5990efc72e8366a45cf3ed72a04466e724dcfc92. The immutable Server2.11.8/Core2.15.12/PHP SDK3.6.1/Python SDK2.12.2 tuple still has eight external cases that never reached the original-call/waiting-service pre-kill checkpoint. Four PHP authorization callers time out pending; four Python authorization callers and all eight uncaught typed-failure caller observations qualify. No recovery or PHP authorization success is claimed.

The new bounded worker heads expose Python's completion refusal: HTTP409 history_transaction_limit_exceeded, recorded:false, no target call observed. PHP external workers also exit255 after HTTP409. The fixture uses4MiB arguments and roughly8MiB results, while the package's documented default logical payload ceiling is2MiB (DW_V2_LIMIT_PAYLOAD_SIZE_BYTES). Enabling storage at the2MiB offload threshold does not raise that independent ceiling. Source inspection also shows the Server completion response retains the reason but drops Core's refusal limit/message, so the exact dimension is not established by that HTTP response alone.

Further diagnostic correction: the ordinary PHP probe reused namespace-only log files, so its authorization log could include earlier external failures. This prevents attributing an old fatal headline to the current authorization attempt. Recovery logs were already process-specific.

PR913 now has head 63da293e2ce54a358763c1a4ff7898405dcd7015: the large-value fixture explicitly sets a16MiB logical payload ceiling while retaining the2MiB offload threshold,30s recovery budget,128MiB PHP observer cap, transaction ceilings and all identity/reference assertions. PHP logs are process-specific and record bounded typed refusal status/path/reason through the installed public transport while preserving stream position. All13 standalone evidence controls and changed PHP/Python syntax checks pass in isolated containers.

Execution41311142390 is active. Next verify this explicit-configuration run, obtain PHP's actual authorization/completion reason, and fix any confirmed remaining defect. The earlier default-ceiling run remains failed evidence; increasing the fixture's declared logical ceiling is not a default-configuration recovery claim. #789/#784 and Workflow#942/#947 remain open. Only bounded hosted observations were inspected in memory; no log/archive mirror was retained locally.

Focused published PHP/Python caller acceptance passes in execution41311142390, exact PR913 head 63da293e2ce54a358763c1a4ff7898405dcd7015, published Server2.11.8 index sha256:3848d5d5cd1b23bd2881e3fdecea64cc44b72e6fb812e577c2db6d0872d065dd / Core2.15.12, normal PHP SDK3.6.1 and Python SDK2.12.2. All25 SDK/caller and4 external/authorization evidence-reader checks pass.

Eight oversized cold recovery cases (all four PHP/Python caller/service pairs, shared plus caller namespace) qualify in8.566–10.432s under the original30s budget. Original run/call/deadline identity, confirmed SIGKILL/replacement process, one start/completion, decoded Unicode/int64/null values and distinct persisted argument/result references are checked. All32 owner reference fetches have exact hashes/byte sizes; all32 foreign reference fetches are404. Replacement PHP callers exit0 and the128MiB observer controls pass.
All eight authored authorization cases qualify: original callers receive the typed failure without target dispatch; direct admin-route controls remain403. All eight uncaught typed-failure observations also qualify. Small-value recovery, database interruption, cancellation and original-deadline checks are covered by the25 SDK/caller reader checks; this is their exact hosted scope, not native/Rust acceptance.
Configuration is explicit:16MiB logical payload ceiling for4MiB arguments/roughly8MiB results;2MiB storage-offload threshold. Transaction ceilings and all30s/128MiB identity/reference assertions remain unchanged. The earlier default2MiB tuple41309424994 stays failed evidence. No default-ceiling oversized-recovery claim or product-source fix is made.
Independent published-image HTTP-kernel controls on SQLite also produce the expected result with and without idempotency keys: both allowed namespace cases complete/admit; role and namespace denials each record ServiceCallFailed with no linked target run. The six keyed controls verify all369 installed Server files and2259 committed Core files before/after. This is kernel-level supplemental evidence, not an SDK/socket qualification.

The aggregate conformance report still prints fail, even though this focused job passes. The shell returns0 after writing that report;0 alone does not mean overall Nexus acceptance. The existing retention step did not retain its aggregate findings, so the small collector-only branch test/789-conformance-findings-20261101 at 686739950 adds bounded scenario statuses/findings to retained outcomes. Its isolated workflow run will establish the remaining exact findings without restarting the PR's full matrix. The collector has passed isolated Node syntax checking; it is not yet installed-qualified or merged.

#789/#784 and Workflow#942/#947 remain open for their broader requirements, including dedicated ingress-versus-retention, nested nonwaiting update, pressure and Rust caller scope. Next inspect the aggregate findings, integrate the reviewed collector into PR913, qualify the final head and proceed through the protected merge path.

The published PHP/Python Nexus slice now passes on exact runner 4a1379445e14c18ddd4ea0d44abcd257b4d8eef5: run 41322264219, November 1, 15:39–15:49 UTC. PR #913 contains the qualified conformance changes; normal PR checks and merge remain separate.

The immutable published tuple is Server 2.11.8 at sha256:3848d5d5cd1b23bd2881e3fdecea64cc44b72e6fb812e577c2db6d0872d065dd, installed Core 2.15.12 from 7c80ffccdb6ab6e6ace8339618a9d6f1d6d9432f, PHP SDK 3.6.1, Python SDK 2.12.2, CLI 2.3.3 and Waterline 2.13.0.

All 14 aggregate scenarios pass with zero findings. Both cross-language cases retain six actual original caller workflows each: completed, typed failure and cancellation in the shared and tenant namespaces, with matching caller and linked-service histories.
All 16 standalone evidence controls, 28 SDK reader checks and four external/authorization reader checks pass, with zero failures or skips. Admission/reflection/claim-only evidence remains insufficient.
All eight oversized cold recoveries pass in 7.884–11.300 seconds with the original run/call identity, replacement worker, exact bytes/hashes, Unicode, signed 64-bit maximum and null semantics. Thirty-two owner reference reads match exactly; all 32 foreign-namespace reads return 404. PHP replacement observers stay within the 128 MiB limit.
All eight authorization and eight uncaught typed-failure cases qualify.

The large-value fixture explicitly uses a 16 MiB logical-payload ceiling and the existing 2 MiB storage-offload threshold. Recovery time, observer memory, transaction ceilings and identity assertions are unchanged. The matched default-ceiling 409 refusal remains valid negative evidence.

The earlier intermittent PHP-service stall remains a failed run. Bounded worker diagnostics were added; the following run did not reproduce it. No runtime fix or root cause is claimed. The two intervening evidence-reader mistakes were corrected to consume actual workflow proof and the published Python cancellation shape.

#789 remains open. Dedicated installed ingress/retention interleavings, nested nonwaiting update, host-query/SQL-engine pressure and Rust SDK caller acceptance remain separate original work. This result changes no runtime source, release or Cloud deployment.

Published-image workflow-start ingress versus retention now passes both controlled interleavings on all four database engines: 8/8 cases. SQLite ran November 1, 15:54:01–15:54:08 UTC; MySQL, MariaDB and PostgreSQL ran 15:59:24–15:59:57 UTC.

The artifact remains PHP Server 2.11.8, source fa3fc2f2f6b8935ea4a6eea47f6a9221e2adafff, index sha256:3848d5d5cd1b23bd2881e3fdecea64cc44b72e6fb812e577c2db6d0872d065dd, with installed Core 2.15.12 from 7c80ffccdb6ab6e6ace8339618a9d6f1d6d9432f. Database versions were SQLite 3.46.1, MySQL 8.0.46, MariaDB 10.11.19 and PostgreSQL 17.11.

Each case uses independent PHP processes and database connections, a real uploaded Avro object referenced by an old completed run's stream, the published HTTP kernel's POST /api/workflows, and the ordinary retention implementation. Scheduling follows the public workflow-start race controls, without replacing runtime source or storage behavior.

Interleaving	Observed on every engine
Verification commits; retention prunes before the writer resumes	Ingress returns 422 `external_payload_unsupported`. No new instance or run commits. The old stream reference, object registration and provider bytes are removed.
Writer holds its final claim; retention attempts the same object lock	Retention waits. Ingress returns 201. Retention then prunes the old run's stream while keeping the new run's registration and exact readable payload bytes.

The row-locking engines additionally verify the cleanup connection waiting on runtime_external_payload_object_locks in their database lock/process views. SQLite verifies that cleanup cannot finish while the writer is paused holding its write lock. All 369 Server and 2,259 committed Core files match the published source before and after every case; no package source was overlaid. All task databases and probe containers were removed and their absence checked. Existing database servers were preserved.

This is installed HTTP-kernel qualification of workflow-start ingress. It does not qualify every other ingress writer, SDK concurrency, nested update behavior, pressure behavior or Rust callers. Those remaining paths keep #789 open. The separate published PHP/Python SDK qualification is recorded in the preceding receipt.

The dedicated installed Nexus command versus retention cases also pass on SQLite 3.46.1, MySQL 8.0.46, MariaDB 10.11.19 and PostgreSQL 17.11: 8/8 interleavings, in addition to the eight workflow-start cases.

These use the same immutable PHP Server 2.11.8 / Core 2.15.12 tuple and source-file guards as the preceding receipt. SQLite ran November 1, 16:13:01–16:13:08 UTC; the other three engines ran 16:14:56–16:15:29 UTC. Two independent processes exercise the actual HTTP kernel's protocol-1.20 workflow-task completion, a keyed start_service_operation command, a workflow-bound handler, and ordinary retention of an old stream sharing the uploaded Avro object.

Retention first: after verification commits, retention removes the old registration and bytes. Nexus then reconstructs the already-verified value, creates a fresh registration with the exact hash and size, and commits the original caller's command with HTTP 200 / recorded: true. The linked service run references the same content-addressed URI and its exact bytes remain readable. The deleted opaque registration is not reused.
Writer first: retention waits on the held object lock. The command commits once with HTTP 200 / recorded: true; pruning then removes the old stream while preserving the registration and exact bytes required by the linked service run. Database lock/process views confirm the wait on all three row-locking engines; SQLite confirms cleanup cannot finish while the writer is paused.

All 369 Server and 2,259 committed Core files match before and after every case. The original task completes, one caller ServiceCallStarted event exists, and the call's caller/target namespace and linked-run identity match. No runtime source or storage implementation was replaced. Task databases and probe containers are gone; the shared fixture servers remain intact.

Two additional SQLite controls at 16:17:49–16:17:57 UTC resend the byte-identical original completion request. Both preserve the original call, caller history, linked run and task without creating a second call. Their responses are explicitly 404 external_payload_not_found after the old opaque registration was deleted, and 409 task_not_leased after the original task completed. These are no-duplicate controls, not proof that protocol 1.20 returns a successful repeated-completion acknowledgement. Published SDK recovery remains separately qualified by run 41322264219.

The initial Nexus fixture incorrectly required every retention-first operation to refuse. Inspection showed a valid fresh registration and exact restored bytes; qualification now checks those actual ownership, hash, size, linked-run and history invariants. No runtime change was needed. This completes these installed ingress/retention cells, while broader update/query-pressure and Rust caller acceptance remain open.

Closing: native Server 3.0.1 doesn't have this race, and PHP Server is frozen (operator direction, November 1).

Native evidence, at release source 923d45b3 (native-3.0.1):

The claim and the run are one transaction. start_command (rust/src/runtime/store.rs, about 422–427) begins the transaction, calls claim_external_in, then writes the run that holds the reference. A refused claim drops the transaction, so no run is committed.
The claim takes the object lock and re-reads the registration. claim_external_in (external_registry.rs, about 376–397) takes payload_object_lock, then requires the ready row to still exist (404 external_payload_not_found) and to be retained or unexpired (410 external_payload_expired). Then it sets retained_at in the same transaction.
Durable transitions are serialized. SQLite uses BEGIN IMMEDIATE, PostgreSQL pg_advisory_xact_lock (backend.rs, about 478), MySQL/MariaDB a FOR UPDATE on dw_server_schema (about 693). A start's claim and a retention prune can't interleave inside each other's transactions.
The prune re-decides under the lock. retention_payloads.rs (about 145–147) takes the same object lock and calls payload_retained again, which counts any unpruned run holding any registration of that object. Provider bytes are deleted only after commit, and only when no registration remains (payload_deletions.rs). Cleanup skips claimed (retained_at) rows (payload_cleanup.rs).
Signal, completion, activity-result, standalone-activity and update claims each run inside their own state transaction too.
Nearest tests (single-threaded): external_workflow_input_retains_one_reference_with_start_and_survives_cold_worker_completion (rust/tests/execution/external_payloads.rs), retention_deletes_objects_no_retained_run_holds and pending_deletions_wait_out_a_backup_hold_and_yield_to_a_new_registration (external_payloads/retention.rs).

Gap, not a defect: no native test runs the start and a prune concurrently, in both orders, on each engine. The guarantee above comes from transaction serialization plus the lock, not from such a test. Server will add one with the next external-payload work.

The concurrency test noted above is tracked in #960.

Rust durable-workflow#325: delete external payload objects only after…

Rust durable-workflow#325: take over published PHP Server 2.11.8 data…

Navigation Menu

Repository navigation

A run can reference an external payload that retention deleted after the start's claim #789

Earlier source state (General, 2026-10-28)

Problem

Reproduction

Direction

Security source finding — batched retention deletion is not rollback-safe

Evidence

Failure shape

Required next step / acceptance

Execution-confirmed on the exact stable 2.11.7 artifact — 2026-11-01 08:34 UTC

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

A run can reference an external payload that retention deleted after the start's claim #789

Description

Earlier source state (General, 2026-10-28)

Problem

Reproduction

Direction

Activity

rmcdaniel commented on Oct 26, 2026

rmcdaniel commented on Oct 26, 2026

rmcdaniel commented on Oct 26, 2026

rmcdaniel commented on Oct 26, 2026

rmcdaniel commented on Oct 26, 2026

rmcdaniel commented on Oct 27, 2026

rmcdaniel commented on Oct 27, 2026

rmcdaniel commented on Oct 27, 2026

rmcdaniel commented on Oct 27, 2026

rmcdaniel commented on Oct 27, 2026

rmcdaniel commented on Oct 28, 2026

rmcdaniel commented on Nov 1, 2026

rmcdaniel commented on Nov 1, 2026

Security source finding — batched retention deletion is not rollback-safe

Evidence

Failure shape

Required next step / acceptance

rmcdaniel commented on Nov 1, 2026

Execution-confirmed on the exact stable 2.11.7 artifact — 2026-11-01 08:34 UTC

rmcdaniel commented on Nov 1, 2026

rmcdaniel commented on Nov 1, 2026

rmcdaniel commented on Nov 1, 2026

rmcdaniel commented on Nov 1, 2026

rmcdaniel commented on Nov 1, 2026

rmcdaniel commented on Nov 1, 2026

rmcdaniel commented on Nov 1, 2026

rmcdaniel commented on Nov 2, 2026

rmcdaniel commented on Nov 2, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions