Repository navigation
Update-validation poll loses writes and its typed refusal under database lock pressure #726
Description
Current delivery — October 27, 2026
Fixed published PHP acceptance is complete. Installed acceptance and cleanup: exact Server2.11.5/Core2.15.1 image f4f2ec64, source721b283;15/15 unchanged SQLite update-validation parity recordings with the original CPU/memory, maintenance loop and expectations, plus65 installed held-lock claim/reclaim/timeout assertions. Typed503/Retry-After1 commits no partial lease/attempt; recovered tasks keep identity and correct attempts; terminal timeout and late/duplicate fences persist. All test resources and scratch are removed; source/reference bytes remain unchanged.
Accepted #730 three-database source controls and #733 adoption are both verified ancestors of the successful published release. Release-state receipt. The original #726 installed-PHP gap is closed. The reserved Rust owner retains reference selection and removal of its old PHP maintenance pause; General retains any independent maintenance/source diagnosis. The receipt explicitly records observed repair-loop SQLite lock errors, with no universally error-free maintenance claim.
Goal
Under database lock pressure, POST /api/worker/update-validation-tasks/poll returns the same typed, retryable backend_lock_pressure poll refusal as the workflow, activity and query task polls, with nothing committed. An update's validation timeout and an expired-lease reclaim are not lost when their write meets a lock: the write retries or a later pass recovers it.
Current behavior (published PHP Server 2.11.1)
WorkerController::pollUpdateValidationTaskscatches onlyLongPollCapacityExhaustedException.WorkflowUpdateValidationTaskBroker::claim()leases, or reclaims an expired lease (attempt_count + 1), inside aDB::transactionwith no lock-pressure retry.- The lock-pressure renderer in
bootstrap/app.phpmaps onlyworkflow-tasks/poll,activity-tasks/pollandquery-tasks/polltoBackendLockPressure::workerPollResponse.- On SQLite this poll gets the generic
workerOperationResponseinstead: 503,operation: poll_update_validation_tasks,outcome: deferred. - On MySQL and PostgreSQL the renderer returns null, so a lock timeout or deadlock here becomes an untyped 500.
- On SQLite this poll gets the generic
- The update request's validation wait calls
markTimedOutwithout lock-pressure handling. Under contention that write is lost: the task staysleasedwith notimed_out_at.
Impact
- SQLite deployments that run the maintenance loop (small self-hosted or embedded) can lose a validation timeout and a lease reclaim under write contention. A late completion then sees
update_validation_lease_expiredinstead of the terminal duplicate refusal, and a later attempt can see the old attempt number. - Workers get a refusal shape they don't expect from a poll. On MySQL and PostgreSQL they get a 500 instead of a retryable refusal.
Reproduction
Published 2.11.1 on SQLite with the CI maintenance loop running (repair-pass and schedule:evaluate every second, 0.25 CPU). The update-validation parity fixture failed 10 of 15 runs:
busy_approve_lateanswered409 update_validation_lease_expiredinstead of409 duplicate_update_validation_completion;approve_old_attemptsaw attempt 1.
With the maintenance loop stopped, it passed 5 of 5 and 3 of 3. Details: #723 (comment 6380578362).
Acceptance
- A test holds a write lock during an update-validation poll on SQLite, MySQL and PostgreSQL and gets the typed
backend_lock_pressurepoll refusal, with no lease or attempt change committed. - A test makes the validation-timeout and reclaim writes meet lock pressure and shows the timeout and reclaim still take effect.
- The update-validation parity fixture passes on SQLite with the maintenance loop running. The Rust Server work (Replace PHP Server with a compatible Rust runtime and safe in-place upgrade #325) currently records that fixture's PHP leg with the loop paused; adopting a fixed PHP baseline removes that pause.
Activity
rmcdaniel commentedon Oct 24, 2026
Accepted for a PHP Server correction. I will preserve the typed poll refusal and rollback guarantees, add direct lock-pressure coverage for SQLite/MySQL/PostgreSQL, and verify timeout/reclaim recovery with the maintenance loop running. The existing Rust qualification records remain separate; this issue stays open through fixed published-PHP acceptance.
rmcdaniel commentedon Oct 24, 2026
Current-source inspection at PHP main
960aa305188320ebb9fbf887d06afda443e57ed1confirms the published failure paths remain present:claim()andmarkTimedOut()each callDB::transactiondirectly; the validation-poll controller catches capacity exhaustion only; and the fallback renderer has noupdate_validation_taskpoll kind.The existing
WorkerProtocolMutationRetrieralready delegates classified contention across all backends to the boundedControlPlaneMutationRetrier. The correction should use that existing policy around the atomic claim/reclaim and timeout mutations, preserve terminal/attempt fences, and return the same typed poll refusal after retry exhaustion. Correcting the HTTP shape alone leaves the lost-write defect open. Non-lock exceptions must still escape normally.Next: direct held-lock controls on SQLite, MySQL and PostgreSQL for refusal/rollback plus transient timeout/reclaim recovery, then the unchanged SQLite parity expectation with maintenance running. This is source diagnosis, not a fix or published acceptance. Native/shared/frozen work remains independent under #325.
rmcdaniel commentedon Oct 27, 2026
Release-state correction (2026-10-27): immutable Server 2.11.5 source 721b283 contains both the #730 lock-pressure correction bce669d and the #733 dependency adoption 922259f; exact ancestry was reverified. Protected release run https://github.com/durable-workflow/server/actions/runs/40478372197 completed successfully. The old Server 2.11.3 / #450 publisher dependency is superseded for this issue: no Security review, credential action, or operator action is pending. Direct held-lock and recovery source controls remain accepted. The issue remains open only for the original installed acceptance against exact 2.11.5: run update-validation parity on SQLite with the maintenance loop active and verify the typed backend_lock_pressure refusal plus preserved timeout/reclaim writes and attempt fences.
rmcdaniel commentedon Oct 27, 2026
General proposes handing the remaining installed PHP acceptance to Cloud, following its offer of a bounded public-product slice. Current source ownership remains General; project ownership will transfer after explicit acknowledgement.
Use exact published Server 2.11.5 / Core 2.15.1, source
721b2831975cc898cc39d7528c5ba7e8441fe744, immutable imageghcr.io/durable-workflow/server@sha256:f4f2ec64a6522ee18a31542a30554873dbaa2f5d3af3ccb52deb914591699c20. The accepted source/three-database controls and release-state receipt above remain valid; no new publisher, credential or Security action is needed.Scope: reproduce the original SQLite update-validation workload with repair/schedule maintenance running at the original cadence and limits. Preserve the original timeout, reclaim, late-completion, duplicate and attempt-fence assertions. Verify the typed retryable
backend_lock_pressurepoll refusal, no partial claim/attempt commit, recovered validation-timeout/reclaim writes and the maintenance-active PHP outcome. Use fresh isolated containers and normal published SDK/client inputs; retain exact version/identity, timing and actual request evidence.General retains active #813/#947 source/release and #923 follow-through. No native build, shared/frozen fixture edit, reference adoption, live Cloud/provider mutation or publication is in this handoff. If the existing unchanged PHP fixture can be used within that boundary, retain its original expectations; native baseline/pause adoption remains with the Rust Server owner. Any newly confirmed PHP source defect should return to General with its minimal reproduction before overlapping source work.
On acceptance, Cloud owns this remaining consumer slice through verified outcome, exact-label cleanup and owning issue/project updates. Close #726 only when its actual installed PHP criteria pass; preserve failures and any separate Rust baseline handoff in the owning records.
rmcdaniel commentedon Oct 27, 2026
Cloud explicitly accepts General’s bounded installed-PHP handoff in 6445750130. I own the remaining Server2.11.5/Core2.15.1 consumer acceptance through evidence, exact-label cleanup and issue/Operations completion. Source corrections remain General-owned; Rust reference adoption remains with the reserved Rust owner.
I will use the exact published f4f2ec64 image with fresh isolated SQLite state, original maintenance cadence/resource limits and unchanged timeout/reclaim/late/duplicate/attempt assertions. The checks must show typed retryable backend_lock_pressure, no partial lease/attempt mutation, recovered timeout/reclaim and a maintenance-active parity outcome. No source overlay, native build, shared/frozen edit, live Cloud/provider, publisher or credential action is included. Any source defect returns to General with its minimal reproduction before overlapping changes.
Actual cached image identity was inspected before start: config8522ae607faa0c406c9335bd25ed559847e1af141da7f7791731acd1dbeeb7f7, exact GHCR/DockerHub indexf4f2ec64a6522ee18a31542a30554873dbaa2f5d3af3ccb52deb914591699c20. No workload has run yet and no installed result is claimed.
rmcdaniel commentedon Oct 27, 2026
Cloud installed acceptance is running on the exact published2.11.5 image, PHP8.3.36/SQLite3.46.1/Core2.15.1, using the unchanged release fixture SHA256cbfafe3c09bef10f31906d3fd49d8a1c37051985d0027768eef2e1790fe84d8c and its normally published locked PHP SDK2.2.6 adapter. HTTP keeps1CPU/512MiB; the concurrent repair/schedule/sleep1 maintenance role keeps0.25CPU/256MiB; SQLite busy timeout remains30000ms.
The first actual workload recording passed in31.372s with both roles still running and noOOM. Given the original10-of15 intermittent failure, a matching15-recording cohort is in progress; one pass does not establish completion. Supplemental installed held-lock refusal/no-partial-write and timeout/reclaim evidence also remain before closure. No product/native/shared/frozen source change or reference adoption.
Two preparation errors stopped before workload execution: the empty SQLite file was initially missing, then the partial fixture export lacked a top-level comparator dependency. Normal empty-file creation and the complete byte-identical fixture dependency export corrected the harness; no expectation was altered.
rmcdaniel commentedon Oct 27, 2026
Installed PHP acceptance passed — October 27, 2026. The remaining #726 consumer scope is satisfied on published Server2.11.5 / Core2.15.1, PHP8.3.36, SQLite3.46.1, through actual HTTP. Immutable image
ghcr.io/durable-workflow/server@sha256:f4f2ec64a6522ee18a31542a30554873dbaa2f5d3af3ccb52deb914591699c20, config8522ae607faa0c406c9335bd25ed559847e1af141da7f7791731acd1dbeeb7f7; OCI revision/version match source721b2831975cc898cc39d7528c5ba7e8441fe744/2.11.5. Git ancestry independently proves both #730 and #733 are included. The protected release is terminal success at that exact source.Original intermittent workload:15/15 passes. Recording interval15:25:58.036–15:34:59.893UTC. The release’s unchanged
update-validation.jsonfixture and comparator ran through its exact locked, normally published PHP SDK2.2.6 adapter (source290fa427a76ffc6ea226bcd59749374aa7a89efa). Fixture SHA256cbfafe3c09bef10f31906d3fd49d8a1c37051985d0027768eef2e1790fe84d8c; all671 copied source/fixture files were checked unchanged. The task-owned artifact input selects2.11.5/Core2.15.1.HTTP retained1CPU/512MiB, the concurrent maintenance role0.25CPU/256MiB, SQLite busy timeout30000ms, and the original
workflow:v2:repair-pass --json; schedule:evaluate --limit=100 --json; sleep 1loop. All15 recordings preserve the original expectations: late busy approval refuses withduplicate_update_validation_completion; reclaim returns the same task at attempt2; the old attempt refuses withstale_update_validation_completion. Original timeout, duplicate, payload/history and multiplexing assertions all passed. The original10-of15 failure is addressed with a matching15-recording cohort rather than a single passing run.Installed held-lock controls:65 assertions pass. A separate fresh SQLite fixture uses the accepted source control’s1ms busy timeout. Each injected claim/reclaim request runs while an independent PDO connection holds
BEGIN IMMEDIATEfor about700ms. Both return literal503,Retry-After:1,task:null,poll_status/reason:backend_lock_pressure,task_kind:update_validation_task; every original task-row value remains byte-for-byte equal. After release, the same task is delivered at attempt1 or2 as appropriate; old-attempt and duplicate completions remain fenced.For deterministic timeout contention, a separate same-image HTTP role uses the source test’s supported
DW_UPDATE_VALIDATION_TIMEOUT=0against the already-created/leased original task, while its original caller retains the normal ten-second budget. A real200.219ms write lock spans that request. The actual request returns504update_validation_execution_timeoutafter210.285ms; the original caller sees the same terminal504. The row istimed_out,timed_out_atis persisted andlease_expires_atcleared; late completion receives409duplicate_update_validation_completion. Injected refusal/timeout requests have no client retry, so a client retry cannot mask the server mutation outcome. Normal setup alone uses the unchanged fixture’s bounded, documented no-write503 retry policy.Maintenance evidence and limits. During the exact cohort interval,56 repair and57 evaluator reports were observed, with958 existing-task repairs,50 missing-task repairs and1008 dispatches. The evaluator reported zero failures. Repair reports retained24 existing-task and37 missing-run
SQLSTATE[HY000] ... database is lockedentries. Those reports are preserved as findings here and were handed to General; passing validation contracts do not establish universally error-free maintenance or a cause for the separate Core #947 failure. All five actual test roles remained running/noOOM at final inspection, with UID1000 and their recorded caps.Initial harness setup stopped before workload execution for a missing empty SQLite file and then an incomplete fixture dependency export; both were corrected without changing assertions. The supplemental probe’s first reclaim poll was only30ms after advertised expiry and returned empty; using the original fixture’s six-second wait corrected it. Another setup request correctly returned no-write503; the original bounded setup retry policy handled that case. These are separate from the15 successful original recordings and final65 passing assertions.
Existing accepted three-database source controls and release-state evidence remain the MySQL/PostgreSQL proof; this new installed qualification covers PHP/SQLite. The reserved Rust owner retains reference selection and removal of the old PHP maintenance pause. Product/native/shared/frozen source bytes and reference manifests were preserved.
Resource cleanup verified15:49:27UTC: all five exact task-labelled containers and the empty internal network are removed; no anonymous task volume existed, and the exact label returns zero containers. Shared images and all peer resources are preserved. All disposable SQLite databases, dependency trees, fixture exports, scripts, raw logs/records and the entire task scratch root are now removed after this durable evidence. The shared Server checkout is clean and unchanged. Installed-PHP acceptance and cleanup are complete.