draft: fix: GPU tensor allocation fall back to host memory

[mattwittwer]

No description provided.

[greptile-apps]

Confidence Score: 4/5

[Medium risk] Expands GPU memory fallback to include pinned host memory.

The PR appears safe to merge, though the new pinned-host fallback paths would benefit from regression coverage.

Findings

1.  Pinned fallback lacks tests ▶

Summary

This PR completes GPU-output fallback handling when Triton supplies CPU_PINNED memory.

• It places pinned-host intermediates in shared memory and treats them as host memory during copies.
• It copies those intermediates into Triton-provided buffers for ordinary and decoupled responses.

Diagram

sequenceDiagram
  participant T as Triton
  participant B as Native backend
  participant S as Python stub
  B->>T: Request output buffer for GPU tensor
  T-->>B: Return CPU or CPU_PINNED buffer
  B->>S: Share host intermediate handle
  S->>B: Fill intermediate from GPU tensor
  B->>T: Copy intermediate into output buffer

Loading

_{Reviews (1) · Last reviewed commit: "fix: Handle CPU_PINNED output buffers wh..."}

[greptile-apps]

Pinned fallback lacks tests The new CPU_PINNED branch copies GPU output through a shared-memory buffer before filling Triton’s output buffer, but there is no regression test for that transfer in either ordinary or decoupled responses. A test that forces a pinned-host output and checks the returned bytes in both modes would help catch a future change that returns incorrect output.

Knowledge Base Used:

• Inference response delivery
• Shared memory and GPU transport

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!