[SAURABHSALVE]

What this PR does / why we need it:

wait_for_job_status in the Spark module still uses a mutable set literal as its status default, in all three places that define it: kubeflow/spark/api/spark_client.py, kubeflow/spark/backends/base.py, and kubeflow/spark/backends/kubernetes/backend.py.

status: set[SparkJobStatus] = {SparkJobStatus.COMPLETED},  # evaluated once, shared forever

This is the same anti-pattern reported in #329, but #330 and #393 (both currently open) only touch the Trainer and Optimizer modules — neither diff includes a single Spark file. #874 ("validate requested job statuses") edits these same three Spark files but only adds validation inside the function body; it doesn't touch the default-argument line either, so the mutable default survives that change too. This PR closes that gap for Spark.

Repro (see #885 for the full write-up): the default set object is created once and shared by every call that omits status. If any code mutates it in place (e.g. extends it with status.add(...) instead of copying first), every later call made without an explicit status argument silently picks up the mutated set for the rest of the process's lifetime.

Fix: default to None in all three signatures, and build a fresh {SparkJobStatus.COMPLETED} set inside spark_client.py and backends/kubernetes/backend.py when status is None — matching the fix pattern already used in #330/#393 for the other modules.

Verification

1. Added three regression tests: two in spark_client_test.py (signature-default check, and a behavioral test that mutates the set returned to one call and asserts the next default-using call is unaffected) and one in backend_test.py (signature-default check).
2. Confirmed all three new tests fail on unmodified main for the right reason (status.default is {SparkJobStatus.COMPLETED} instead of None; the mutation leaks into the second call).
3. All three pass on this branch.
4. Full targeted run: pytest kubeflow/spark/api/spark_client_test.py kubeflow/spark/backends/kubernetes/backend_test.py → 127 passed.
5. ruff check, ruff format --check, and ty check kubeflow/spark on the changed files show no new issues (the 60 pre-existing ty diagnostics in this module are all in unrelated files/lines and present on main too).

Which issue(s) this PR fixes:

Fixes #885

Checklist:

• No user-facing docs changes needed — this only changes an internal default-argument implementation detail; the documented default behavior ({SparkJobStatus.COMPLETED}) is unchanged.

🤖 Generated with Claude Code

[google-oss-prow]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign astefanutti for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment