test: CUDA memory pool fallback in python backend

[mattwittwer]

What does the PR do?

Adds regression coverage for silent output corruption in the Python backend when the CUDA memory pool is exhausted and GPU output tensors fall back to pinned host memory (#7148). The runtime fix is python_backend#457.

The existing IOTest.test_ensemble_io already drives the ensemble_io pipeline of three chained dlpack_io_identity Python models, with per-request flags choosing which stage emits its output as a GPU tensor, and asserts exact equality against a 1000 x FP32 input. This change re-runs that test with --cuda-memory-pool-byte-size=0:1024, so every 4000-byte GPU output overflows the pool and the ensemble's response allocator falls back to pinned memory for each of them. The block also fails if the server log does not contain the core's falling back to pinned system memory warning, so it cannot pass without exercising the fallback path. No new models or Python code.

Checklist

• PR title reflects the change and is of format <commit_type>: <Title>
• Changes are described in the pull request.
• Related issues are referenced.
• Populated github labels field
• Added test plan and verified test passes.
• Verified that the PR passes existing CI.
• Verified copyright is correct on all changed files.
• Added succinct git squash message before merging.
• All template sections are filled out.
• Optional: Additional screenshots for behavior/output changes with before/after.

Commit Type:

• test

Related PRs:

• fix: GPU tensor allocation fall back to host memory  python_backend#457 (the fix this
  test guards; the new block fails until it is in the test container)

Where should the reviewer start?

• qa/L0_backend_python/io/test.sh — the new block IOTest.test_ensemble_io with GPU outputs falling back to pinned memory: model setup mirrors the existing default trial, SERVER_ARGS adds the 1024-byte CUDA pool, and the post-run grep on the server log guards against a vacuous pass.

Test plan:

• CI Pipeline ID: [71702518]

Caveats:

Background

Related Issues:

• Relates to GitHub issue: client silent failure - E0422 05:03:24.145960 1 pb_stub.cc:402] An error occurred while trying to load GPU buffers in the Python backend stub: failed to copy data: invalid argument #7148

[greptile-apps]

Confidence Score: 3/5

[Low risk] Adds a test case for GPU memory fallback behavior.

The PR is not ready to merge because the new test block can signal its process group on startup failure or stop the QA script during cleanup.

Findings

1.  Startup failure signals the test group ▶
2.  Cleanup failure skips remaining checks ▶

Summary

The PR reruns the Python-backend ensemble IO test with a 1024-byte CUDA memory pool and checks the server log for pinned-memory fallback.

• The test also checks output equality through the existing ensemble IO test.
• No new findings were identified since the previous review.

_{Reviews (2) · Last reviewed commit: "Merge branch 'main' into mwittwer/python..."}

[greptile-apps]

Startup failure signals the test group
If the server fails to start, run_server leaves SERVER_PID=0. This block records the failure but continues to kill $SERVER_PID, so kill 0 signals the test's entire process group instead of a server. That can terminate the parent QA run. Skip server cleanup when startup fails.

[greptile-apps]

Cleanup failure skips remaining checks
Cleanup runs under set -e. If the server has already exited, kill fails; if it exits with a nonzero status, wait fails. Either failure ends the script before it checks the fallback log or runs the remaining IO subtests. Handle cleanup failures without stopping those checks.