Skip to content

go executor: prevent invalidation deadlocks#20

Merged
ChenJesse merged 2 commits into
mainfrom
codex/fix-invalidation-deadlock
Jul 14, 2026
Merged

go executor: prevent invalidation deadlocks#20
ChenJesse merged 2 commits into
mainfrom
codex/fix-invalidation-deadlock

Conversation

@ChenJesse

Copy link
Copy Markdown
Contributor

Summary

  • synchronize each task execution lifecycle across state, context, pending invalidations, and terminal-channel generations
  • split invalidation into claim, event publication, cancellation, terminal wait, state reset, and restart scheduling without holding the executor mutex across a channel wait
  • keep completing tasks alive until invalidation has registered the replacement execution, and prevent dependents from starting against an invalidating dependency
  • join the invalidation loop during teardown and make watcher failures cancel through the same scheduling barrier
  • preserve the observable Invalidated → Stopped event order and allow ShouldInvalidate callbacks to inspect task state safely

Supersedes #17 by folding its pending-invalidation cleanup into the synchronized lifecycle transition.

Root cause

The captured SIGQUIT dump showed an invalidation goroutine blocked on a task terminal channel for 122 minutes while holding Executor.mu. Thirteen task goroutines were then blocked trying to enter the next execution pass through that same mutex.

PR #15 / 59908fc introduced cross-goroutine KeepAlive lifecycle reads. PR #16 / 6472f72 then added a second unsynchronized writer that replaced ctx and terminalCh from task completion. Completion could swap in a fresh empty terminal channel just before invalidation evaluated its receive, permanently losing the signal. The pre-existing global mutex around that receive amplified one lost signal into a whole-executor deadlock.

Regression evidence

The regression test was committed first and failed under the race detector before the fix:

WARNING: DATA RACE
executor.go:669
 task_execution.go:76
testing.go:1712: race detected during execution of test

The fix also closes two adjacent teardown gaps found during review: dynamic errgroup registration during restart, and a watcher error being trapped behind a lifecycle guard.

Verification

  • go test -race . -run "^(TestKeepAliveInvalidationSynchronizesTaskCompletion|TestTaskWaitsForInvalidationCompletion|TestInvalidatingDependencyIsNotReady|TestShouldInvalidateCanReadTaskState|TestExecutorReturnsWatcherFailure)$" -count=20
  • go test -race ./... -count=1
  • two independent concurrency reviews; no remaining Critical or Important findings

@ChenJesse
ChenJesse merged commit b705c98 into main Jul 14, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant