Skip to content

test(pool): prevent failed-stop eviction regression deadlock - #948

Merged
CaliLuke merged 1 commit into
mainfrom
fix/eviction-test-stop-order
Oct 17, 2026
Merged

CaliLuke merged 1 commit into
mainfrom
fix/eviction-test-stop-order

Conversation

@CaliLuke

@CaliLuke CaliLuke commented Oct 17, 2026 •

Copy link
Copy Markdown
Owner

The daily release publisher is blocked by main CI run 38646089002, where TestEvictionStopsJobRestoredByFailedStop waits indefinitely in Worker.stop. stopJob publishes a stream event before processing the stop directly, so the event loop can acquire the test's gated callback first. The test then joins that loop before releasing its callback gate.

Join the worker runtime before publishing the controlled stop, leaving the direct caller to own the callback. Preserve the failed-stop/eviction overlap and verify that the job stays tracked while blocked, the handler failure reaches the caller, and eviction retries Stop and clears the job. Add deferred gate release and bounded completion waits. Production behavior is unchanged.

The adjacent TLA model reproduces the original circular wait and checks the corrected ordering.

Validation: 100 regression repetitions; related stop/rebalance tests repeated 10 times with the race detector; complete pulse/pool suite; pinned package lint; repository-wide lint and full root-module unit suite through the commit/push hooks. Independent agent review reported no findings. GitHub CI is running.

@coderabbitai

coderabbitai Bot commented Oct 17, 2026 •

Copy link
Copy Markdown

Review in Change Stack →

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration
  • Configuration used: Organization UI
  • Review profile: ASSERTIVE
  • Plan: Advanced
  • Run ID: 17697bab-7c60-4de8-8b8c-7b0f92e15833
📥 Commits

Reviewing files that changed from the base of the PR and between 7bd8685 and 98131b0.

📒 Files selected for processing: 5
  • pulse/pool/tla/eviction_test/README.md
  • pulse/pool/tla/eviction_test/StopEvictionTest.tla
  • pulse/pool/tla/eviction_test/current.cfg
  • pulse/pool/tla/eviction_test/quiesced.cfg
  • pulse/pool/worker_evict_test.go

Included review availability: This review used your included allowance. Your plan provides up to 1 included review per hour; 0 remain after this review.

📜 Recent review details
⏰ Context from checks skipped due to timeout: 15
  • GitHub Check: testdata-compile (grpc/codegen)
  • GitHub Check: race (generator)
  • GitHub Check: race (remaining)
  • GitHub Check: testdata-compile (expr)
  • GitHub Check: race (http)
  • GitHub Check: pulse-redis
  • GitHub Check: testdata-compile (http/codegen)
  • GitHub Check: generated-code-quality
  • GitHub Check: race (grpc)
  • GitHub Check: testdata-compile (codegen/service)
  • GitHub Check: value-contract
  • GitHub Check: testdata-compile (jsonrpc/codegen)
  • GitHub Check: openapi-contract
  • GitHub Check: Analyze (go)
  • GitHub Check: ci
🧰 Additional context used
📚 Code guidelines: 1
AGENTS.md — unknown
📓 Path-based instructions: 2
Source excerpt: **Go**: Use the stable release pinned in the root `go.mod`.

📄 CodeRabbit inference engine (AGENTS.md)

Files:

  • pulse/pool/worker_evict_test.go
Source excerpt: Write table-driven tests in `*_test.go`.

📄 CodeRabbit inference engine (AGENTS.md)

Files:

  • pulse/pool/worker_evict_test.go
🔇 Additional comments: 5
pulse/pool/worker_evict_test.go: 1

333-334: LGTM!

Also applies to: 339-347, 349-349, 359-375

pulse/pool/tla/eviction_test/StopEvictionTest.tla: 1

1-37: LGTM!

pulse/pool/tla/eviction_test/current.cfg: 1

1-3: LGTM!

pulse/pool/tla/eviction_test/quiesced.cfg: 1

1-3: LGTM!

pulse/pool/tla/eviction_test/README.md: 1

1-51: LGTM!

📝 Summary

Summary by CodeRabbit

  • Tests
    • Strengthened coverage of worker shutdown and job eviction, checking that jobs remain tracked while a stop is in progress and are evicted after a failed stop completes.
  • Documentation
    • Added guidance on the modeled shutdown scenarios, verification results, and how to run the checks.

The failed-stop eviction test now checks the in-progress state while a handler’s Stop call is blocked. A TLA+ model, two configurations, and a README describe and check callback ordering and eviction invariants.

Changes: Failed-stop eviction

Layer / File(s) Summary
Check eviction during a blocked stop
pulse/pool/worker_evict_test.go
The test stops the worker before starting stopJob. While the handler’s Stop is blocked, it checks that eviction has not returned and the job remains tracked. After release, it checks the expected stop error and eviction completion.
Model callback ordering and check invariants
pulse/pool/tla/eviction_test/StopEvictionTest.tla, pulse/pool/tla/eviction_test/current.cfg, pulse/pool/tla/eviction_test/quiesced.cfg, pulse/pool/tla/eviction_test/README.md
The model represents callback and worker-stop transitions, and checks three invariants. The configurations select different initial callback orderings. The README reports a NoJoinCycle violation for current.cfg, passing invariants for quiesced.cfg, model limits, and TLC run instructions.

Priority: ⬇️ Low

Estimated code review effort: 3 (Moderate) | ~20 minutes

Change: Other

Merge Risk: ⚪ Minimal · up to 98131

The regression test now avoids the reported deadlock while checking that eviction retries the failed stop and removes the job. No issue identified here needs resolution before merge.

Pre-merge checks: Passed 5
Check name Status Explanation
Docstring Coverage Passed Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 2 functions across 1 files. (4 skipped: 4 …
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Description check Passed The description clearly explains the regression-test deadlock, the corrected callback ordering, the added TLA model, and the validation performed.
Title check Passed The title concisely and accurately identifies the pool regression-test change and its purpose: preventing a failed-stop eviction deadlock.
Full details: Docstring Coverage

Explanation

Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 2 functions across 1 files. (4 skipped: 4 unsupported.)

✨ Finishing Touches
📝 Generate docstrings
  • Commit to this branch
  • Create a new PR
🧪 Generate unit tests (beta)
  • Commit to this branch
  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

I’m a rabbit; I watched the stop,
The tracked job stayed put in place.
Once released, the checks ran on,
Two models traced the callback race.
I thump my feet for tidy states!

Comment @coderabbitai help to get the list of available commands.

@CaliLuke
CaliLuke merged commit 98131b0 into main Oct 17, 2026
19 checks passed
@CaliLuke
CaliLuke deleted the fix/eviction-test-stop-order branch October 18, 2026 04:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant