[SAURABHSALVE]

What this PR does / why we need it:

LOCAL_EXEC_JOB_TEMPLATE (kubeflow/trainer/backends/localprocess/constants.py) starts with set -e and ran the cleanup command ($CLEANUP_SCRIPT → rm -rf $PYENV_LOCATION, used whenever cleanup_venv=True, the default) as the last statement, after the entrypoint:

set -e
$OS_PYTHON_BIN -m venv --without-pip $PYENV_LOCATION
...
$ENTRYPOINT
$CLEANUP_SCRIPT   # <- never reached if anything above fails

If the entrypoint (the user's training function) or any earlier setup step — venv creation, ensurepip, dependency install — exited non-zero, set -e aborted the script immediately and the trailing cleanup line was never reached. So every failed LocalProcessBackend TrainJob permanently leaked its per-job virtualenv under the OS temp directory, regardless of cleanup_venv=True. Only jobs that completed successfully were ever cleaned up — exactly backwards from what matters during normal local iteration, where failures are common.

Fix: register the cleanup command as a trap ... EXIT immediately after set -e, before venv creation, instead of as a trailing statement. A bash EXIT trap always runs exactly once on any exit path and does not alter the script's own exit status (verified empirically — see below), which LocalJob.success depends on (self._success = self._process.returncode == 0).

Verification

1. Added kubeflow/trainer/backends/localprocess/utils_test.py (this module previously had no dedicated unit tests for its script-building helpers) with two regression tests on the real generated script string: the trap is registered before venv creation and the cleanup command is not a bare trailing statement after the entrypoint; and a cleanup_venv=False case still registers a harmless no-op trap.
2. Confirmed both new tests fail on unmodified main for the right reason (no trap in the script; cleanup command is a trailing statement) and pass on this branch.
3. pytest kubeflow/trainer/backends/localprocess/ → 29 passed (no regressions in the rest of the module).
4. Confirmed bash's EXIT-trap exit-code semantics empirically (set -e; trap 'echo cleanup-ran' EXIT; false → cleanup runs, script still exits 1; same with true → exits 0), then ran the real generated script end-to-end through Git Bash for both a failing and a succeeding training function with POSIX-normalized paths (this dev machine's own python -m venv layout and backslash-path handling hit two unrelated, already-tracked Windows issues — venv bin/ vs Scripts/ layout and backslash path mangling, same family as fix(trainer): resolve LocalProcessBackend Windows compatibility issues #762/bug(trainer): container paths are built with backslashes on a Windows client #848/fix(trainer): build container paths with POSIX separators #849 — so paths were normalized only to exercise the trap itself): in both the failing and succeeding case, and even when the script failed later at an unrelated step, the venv directory did not survive, confirming cleanup fires on every exit path, not just the entrypoint.
5. ruff check, ruff format --check, and ty check kubeflow/trainer/backends/localprocess on the changed files show no new issues (pre-existing diagnostics in job.py/utils.py are all on unrelated lines and present on main too).

Which issue(s) this PR fixes:

Fixes #888

Checklist:

• No user-facing docs changes needed — cleanup_venv's documented behavior (remove the venv after training) is unchanged; this fixes it to actually hold on the failure path too.

🤖 Generated with Claude Code

[google-oss-prow]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign electronic-waste for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment