Skip to content

Fix parallel Activity wake-ups after stale MySQL snapshots - #1053

Closed
Loosand wants to merge 2 commits into
durable-workflow:mainfrom
Loosand:codex/mysql-parallel-activity-join
Closed

Loosand wants to merge 2 commits into
durable-workflow:mainfrom
Loosand:codex/mysql-parallel-activity-join

Conversation

@Loosand

@Loosand Loosand commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Refs #1052. Parallel Activity completions can all commit successfully while leaving an embedded all() parent waiting without a resume task on MySQL REPEATABLE READ.

Activity outcome validation performs a consistent read before acquiring the shared run lock. A later lock holder can therefore retain a snapshot older than its sibling's committed completion. Enable the existing current locking history read in the Activity barrier, so the last member observes that completion and schedules the parent exactly once.

Add a deterministic regression through real job handlers and independent PDO transactions. It pauses the last outcome after its actual validation read, commits its sibling through a separate storage connection, then resumes the last outcome. The cases cover default/named storage, 2/6/25 members, nested all(), incomplete-group non-wake and duplicate delivery. Separate real Redis queue-worker cases cover six- and twenty-five-member batches.

Verification

The final committed tests are identical in the counterfactual and candidate runs:

Runtime Deterministic regression result
Unmodified main 4b2cdc5507120d9c213b0b02a85738bec96c574a 8 tests / 86 assertions / 8 intended failures, all at the missing-resume-task assertion
Candidate 33bc5d07269136b7810522a44ce73df10225f3c7 8 tests / 126 assertions, pass

These controls used MySQL 8.4.8 / REPEATABLE READ, PHP 8.4.25 and Laravel 13.40.0. The unmodified barrier blob is 84be4ac9ada051f415d2608d427685c917b1fc4a. The complete new test file also passes the normal upstream PR checks: 10 tests / 136 assertions, including two real Redis queue-worker cases. Each Activity has one attempt and each worker-driven run has exactly two workflow tasks (initial execution and final continuation).

The complete existing non-PR workflow on the fork passed against the exact candidate SHA, including the final build gate:

Check Observed result
Unit suite with coverage 5,893 tests / 80,043 assertions / 49 conditional skips; zero failures or errors, 15 PHPUnit notices
All four MySQL feature shards 2,845 tests / 28,432 assertions / 3 conditional skips; zero failures or errors
All four PostgreSQL feature shards 2,845 tests / 27,911 assertions / 40 conditional skips; zero failures or errors
MariaDB 10.5 Repository's locking/readiness/replay and bounded-dashboard smoke profiles pass
Combined V2 coverage gate 93.89% (75,949 / 80,888 lines), above the required 93.08% baseline
Quality and contracts ECS, PHPStan, repository contracts, regression corpus and HTTP tracing pass
Source upgrade paths Laravel 12 / PHP 8.2 and Laravel 13 / PHP 8.3 embedded upgrade journeys pass

PR-only jobs and optional Sonar analysis are skipped by that workflow's event/secret conditions. The optional Codecov upload was rejected because the fork has no accepted upload token; the required coverage calculation and ratchet pass, and raw reports are retained as workflow artifacts. External Codecov/Sonar publication is not claimed.

Local composer ecs, full composer stan -- --memory-limit=1G, composer unit and composer coverage also pass. Earlier container attempts had orphan-process reaping and shared MySQL connection-capacity failures; the isolated runner corrected these, without changing production behavior, assertions or test timeouts. Local full-feature precursor runs were stopped and are not counted as passes; the full-feature evidence above comes from the completed hosted matrix.

  • Behavior changes include focused regression coverage.
  • Broader automated checks are complete.

Development and validation use Codex assistance. These results qualify the source candidate; they do not claim published-package or production acceptance.

Compatibility

No public API, schema, payload-format, dependency or isolation-setting changes. The current history read runs under the existing parent-run lock. Stale-attempt validation and the task/execution/run lock order are preserved. Child and timer closure paths are unchanged.

This prevents the completion-time lost wake-up. Automatic recovery of already-stranded runs is not claimed by this change. It is independent of the signal-gap-lock deadlock corrected in #1045.

Read current barrier history under the run lock after Activity outcome validation may have established an older repeatable-read snapshot. Cover independent-PDO interleavings, named storage, nested and large groups, duplicate delivery, and real queue workers.
@Loosand
Loosand marked this pull request as ready for review November 6, 2026 04:35
@rmcdaniel

Copy link
Copy Markdown
Member

The reported lost wake-up is fixed in Workflow 2.20.5 through #1056 and its maintenance backport #1057. Qualification covers independent MySQL connections, default/named storage, two/six/twenty-five members, nested groups and duplicate completion; #1052 has the published-package results.

Closing this follow-up because that fix has shipped. This PR was not merged. The release prevents new occurrences and does not automatically repair already-stranded runs. Thank you for reporting the interleaving.

@rmcdaniel rmcdaniel closed this Nov 7, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants