Skip to content

Decode every lane's mbarrier address when dropping unused TRYWAITs - #564

Merged
JRPan merged 1 commit into
accel-sim:devfrom
purdue-aalp:fix-trywait-drop-lanes
Oct 18, 2026
Merged

JRPan merged 1 commit into
accel-sim:devfrom
purdue-aalp:fix-trywait-drop-lanes

Conversation

@JRPan

@JRPan JRPan commented Oct 18, 2026

Copy link
Copy Markdown
Collaborator

Fixes #561.

drop_unused_trywaits in post-traces-processing collected only the first address of each ARRIVE line. Multi-lane lines in base_stride/base_delta format carry the other lanes' mbarriers as offsets, so an mbarrier reached only through a delta lane looked un-arrived, and every TRYWAIT on it was dropped. In the #561 nvjet splitK kernel this removed all 28 TRYWAITs of rank 4's loader warp in every cluster (336 total). The loader then ran ahead, the mbarrier's pending count underflowed, and the cluster deadlocked.

The pass now decodes every active lane for all three address formats. A TRYWAIT is dropped if any of its lanes names an mbarrier with no ARRIVE.

Validation

  • The failing kernel keeps all its TRYWAITs and simulates to completion (10,856 cycles, no deadlock).
  • The reporter re-traced with this patch: all 96 kernels complete, including 16 splitK kernels.

#563 does a larger refactor that shares one address decoder between the tracer, the post-processor and the trace parser. That can land later on top of this.

🤖 Generated with Claude Code

drop_unused_trywaits only read the first address of each ARRIVE line, so
mbarriers reached only through base_stride/base_delta lanes looked
un-arrived and their TRYWAITs were dropped, deadlocking the simulator.

Fixes accel-sim#561

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@JRPan
JRPan force-pushed the fix-trywait-drop-lanes branch from 98d10cd to 1ce4870 Compare October 18, 2026 15:53
@JRPan
JRPan merged commit c5e0872 into accel-sim:dev Oct 18, 2026
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

v2.0.0: trace-driven H100 sim deadlocks on cuBLAS split-K GEMM (nvjet_sm90_*_splitK) traced from vLLM via torch_hook

1 participant