Skip to content

[12.9.x] Fix host-function callbacks in replayed graphs - #3107

Merged
Andy-Jost merged 2 commits into
NVIDIA:12.9.xfrom
Andy-Jost:ajost/backport-3061-12.9
Oct 15, 2026
Merged

Andy-Jost merged 2 commits into
NVIDIA:12.9.xfrom
Andy-Jost:ajost/backport-3061-12.9

Conversation

@Andy-Jost

Copy link
Copy Markdown
Contributor

Description

Manual backport of #3061 to 12.9.x, addressing #790. CUDA graphs containing a host function could crash on their second launch because the bindings routed callbacks through trampoline state that was freed after the first invocation. driver.cuLaunchHostFunc and runtime.cudaLaunchHostFunc now pass the callback and user data to CUDA unchanged.

  • Synchronize the host-function blocks regenerated for CUDA 12.9 after confirming their unfixed generated baselines exactly match the maintenance branch. Preserve the existing runtime Tempita layout.
  • Validate callback values, stream, status, and invocation count after synchronization, and exercise five launches of each captured graph. The driver test creates and destroys its own context for this branch.
  • Document callback and user-data lifetime requirements and add the release note.

The CUDA 12.9 source build and simulated callback/replay checks passed. The GPU regression cases could not be rerun because the GPU test host was unavailable.

Checklist

  • New or existing tests cover these changes.
  • The documentation is up to date with these changes.

Backport NVIDIA#3061 to the CUDA 12.9 maintenance branch. Regenerate the host-function wrappers for CUDA 12.9 and port callback assertions, graph replay coverage, and lifetime guidance.

Signed-off-by: Andy Jost <ajost@nvidia.com>
@Andy-Jost Andy-Jost added bug Something isn't working P0 High priority - Must do! cuda.bindings Everything related to the cuda.bindings module labels Oct 14, 2026
@Andy-Jost Andy-Jost self-assigned this Oct 14, 2026
@Andy-Jost
Andy-Jost requested review from juenglin and mdboom October 14, 2026 19:48

@mdboom mdboom left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As a backport of the generated stuff, this looks correct. I'll defer to @juenglin's review of #3061, and if that gets a ✅ , then consider this approved as well.

@Andy-Jost
Andy-Jost merged commit e399e68 into NVIDIA:12.9.x Oct 15, 2026
61 checks passed
@Andy-Jost
Andy-Jost deleted the ajost/backport-3061-12.9 branch October 15, 2026 14:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working cuda.bindings Everything related to the cuda.bindings module P0 High priority - Must do!

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants