Skip to content

Close activity attempts without locking the attempt after task, execution and run #1000

Closed

Description

@rmcdaniel

Problem

Lease renewal (DefaultActivityTaskBridge::observeAttempt) locks rows in the order attempt → execution → run → task.

ActivityOutcomeRecorder::record() takes the opposite order on its ordinary path: task, then execution, then run. On the stale, cancelled and terminated branches it then calls closeAttempt(), which runs ActivityAttempt::query()->lockForUpdate()->find(...). If a renewal of the same attempt is in flight, the renewal holds the attempt and waits for the execution, while the outcome holds the execution and waits for the attempt. PostgreSQL and MySQL resolve that by aborting one transaction with a deadlock error.

How reachable it is: since dec2aff, refusedExpiredAttempt() refuses most expired or settled callbacks under the run lock before taking any task or execution lock. So the ordinary stale branch is mostly reached in races where the attempt is still Running with an unexpired lease when the fence reads it. The code path is still live, and the cancelled/terminated branch closes attempts the same way.

Found during Security's review of #901, which removed the same reversal from the new audit append (ActivityOutcomeRejection::record() now reads the attempt without a lock). This issue covers the pre-existing closeAttempt() lock, which #901 does not change.

Acceptance criteria

  • No outcome path requests the attempt row lock after holding the task, execution or run lock. Either lock the attempt first, in renewal order, or close it with a guarded conditional update that needs no prior row lock.
  • A deterministic PostgreSQL and MySQL interleaving (same process harness as ActivityOutcomeRetentionTest::testLeaseRenewalAndLateOutcomeAuditCannotDeadlock in Record bounded diagnostic history for rejected late activity outcomes #901) covers renewal against the stale and cancelled outcome branches. It fails with a deadlock on the current code.

Activity

  1. added
    kind:defectA public product behavior is incorrect
    priority:P3Lower-priority product improvement
    on Nov 3, 2026
  2. rmcdaniel commented on Nov 4, 2026

    @rmcdaniel
    MemberAuthor

    Owner: Workflow, with Security reviewing the existing claim/outcome fencing boundary. Current ActivityOutcomeRecorder::closeAttempt still takes lockForUpdate after the task/execution/run locks, so the issue's reverse-order site remains live.

    Next source action is the deterministic renewal-versus-stale/cancelled outcome interleaving on PostgreSQL and MySQL, preserving the current expired-attempt fence. Prefer a guarded conditional terminal update if it can prove the exact attempt/current authority and unchanged stale-write behavior; otherwise adopt one canonical lock order across both paths. I have separated this from #1030's pre-effect profile reads, which do not solve the deadlock. No lease extension or new capability is inferred.

  3. rmcdaniel commented on Nov 11, 2026

    @rmcdaniel
    MemberAuthor

    #1030 was closed as superseded by #1076, so this issue no longer waits on it. The next source action recorded above is unchanged. Owner: Workflow. It's queued behind the open defect PRs #1077–#1083.

  4. rmcdaniel commented on Nov 12, 2026

    @rmcdaniel
    MemberAuthor

    Fixed by #1102 (General, 2026-11-12). closeAttempt() is now a guarded conditional update (WHERE status = 'running', closed_at = COALESCE(closed_at, now)) with no attempt row lock, so an outcome never requests the attempt lock after holding the task, execution and run. That removes the cycle with lease renewal. The new test asserts the lock order on the PR's PostgreSQL feature suite, and all six shards passed. It ships in the next Workflow release.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    kind:defectA public product behavior is incorrectpriority:P3Lower-priority product improvement

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions