fix: keep is_ready error_string_shm alive until parent ack

[dundysm]

What does the PR do?

fixes a use-after-free in Stub::ProcessUserModelReadinessRequest that permanently hangs the python backend stub (and often the server shm pool) when user is_ready() raises or returns a non-boolean.

Ticket / issue

• python_backend: error-string shm freed before parent reads it in is_ready() readiness check -> allocator corruption -> permanent hang server#8978

Root cause

on the has_exception path, error_string_shm was declared inside the if (has_exception) block. its destructor runs at the closing brace, which frees the Boost managed shm allocation (ref_count_ 1 -> 0 -> DeallocateUnsafe) before the stub notifies the parent.

the parent then does PbString::LoadFromSharedMemory(pool, readiness_payload->error) on an already-freed handle, which corrupts the pool free-list. the next deallocate can spin holding the single pool mutex (~98% cpu on the stub); later IPCMessage::Create blocks on the same mutex. hang is permanent for that model instance.

success paths (is_ready true/false) never allocate an error string, so they never hit this. only:

• test_is_ready_raises_exception
• test_is_ready_returns_non_boolean

take the broken path. matches the reporter repro 100%.

Fix

widen std::unique_ptr<PbString> error_string_shm to function scope so it survives notify -> parent Load -> ack. same lifetime pattern already used by GetCUDAMemoryPoolAddress in this file (which explicitly comments that the wait exists so the parent can finish reading before the function returns and frees that shm).

handshake, timeout (kUserModelReadinessTimeoutMs), and parent RunUserModelReadinessCheck Load/ack path are unchanged. sticky-error caching is out of scope.

rebuild note: this change is in pb_stub.cc, which builds into triton_python_backend_stub only (not libtriton_python.so). shipping a fixed .so without a rebuilt stub leaves the bug live.

Test plan

• local: code review + clang-format on src/pb_stub.cc (done). full stub cmake build + qa/L0_backend_python/model_readiness/test.sh needs a Triton server tree / QA image; not run in this environment. please rely on CI / L0 for the hang repro.
• expected L0 coverage (server repo): qa/L0_backend_python/model_readiness/
  • test_is_ready_raises_exception and test_is_ready_returns_non_boolean should complete (assert NOT READY, then healthy model still READY) with no pegged stub cpu and no indefinite hang
  • test_is_ready_returns_true / returns_false / coroutine / timeout / concurrent as regression

Checklist

• PR title reflects the change and is of format <type>: <description>
• Changes are described in the pull request.
• Related issues are referenced.
• Added test plan (L0 left to CI; see above).

Commit Type

• fix

cc @dyastremsky @pskiran1 @Kanupriyagoyal @mc-nv @BhaskarVenkatesha

[dundysm]

@dyastremsky @pskiran1 @Kanupriyagoyal @mc-nv @BhaskarVenkatesha reviewing / ack welcome. fix is the scope widen described in the pr body; l0 model_readiness left to ci.

[greptile-apps]

Confidence Score: 4/5

[Medium risk] Adjusts lifetime of error message in inter-process communication.

The PR should not merge until the parent’s error-string read remains safe when the stub’s acknowledgement wait times out.

Findings

1.  Error string can expire early ▶

Summary

The PR extends the stub-side lifetime of a readiness error string through the parent acknowledgement wait, addressing an early shared-memory release on is_ready() errors.

• The normal parent read-and-ack path is protected.
• The timeout still permits release before a parent that has observed the response loads its error string.

_{Reviews (1) · Last reviewed commit: "fix: keep is_ready error_string_shm aliv..."}

[greptile-apps]

Error string can expire early If the parent sees the readiness response but is delayed for more than five seconds before loading its error string, the stub’s acknowledgement wait times out and destroys error_string_shm. The parent can then load a freed shared-memory handle, leaving the use-after-free possible despite this change.

Knowledge Base Used: Python stub process and IPC

[BhaskarVenkatesha]

Hi @dundysm,

Thank you for your interest in this issue and for taking the time to put this PR together.

I reported #8978 and already have a fix in progress on my side. It is currently going through our internal review before I submit it upstream. In your comment on the issue you mentioned you would stand down if I preferred to take it, and I would like to take you up on that.

Could you please close this PR? I will be opening a PR for this fix myself in the next few days and will link it to #8978 so everyone following can track it.

Thanks again for your understanding and for helping move this forward.