Repository navigation
fix(runtimes): add restricted PSS security contexts - #3702
danish9039 wants to merge 5 commits into
Conversation
|
🎉 Welcome to the Kubeflow Trainer! 🎉 Thanks for opening your first PR! We're happy to have you as part of our community 🚀 Here's what happens next:
Join the community:
Feel free to ask questions in the comments if you need any help or clarification! |
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
There was a problem hiding this comment.
Pull request overview
Adds restricted-PSS settings to Torch distributed runtimes and their Helm equivalents.
Changes:
- Adds pod and container security contexts.
- Configures
/tmpas the training-node working directory. - Adds Helm rendering assertions.
Reviewed changes
Copilot reviewed 6 out of 6 changed files in this pull request and generated 6 comments.
Show a summary per file
| File | Description |
|---|---|
manifests/base/runtimes/torch_distributed.yaml |
Adds Torch runtime security settings. |
manifests/base/runtimes/data-cache/torch_distributed_with_cache.yaml |
Adds security settings to cached training jobs. |
charts/kubeflow-trainer/templates/runtimes/torch-distributed.yaml |
Mirrors settings in the Helm template. |
charts/kubeflow-trainer/templates/runtimes/data-cache/torch-distributed-with-cache.yaml |
Mirrors cached-runtime settings in Helm. |
charts/kubeflow-trainer/tests/runtimes/torch_distributed_test.yaml |
Tests rendered Torch security settings. |
charts/kubeflow-trainer/tests/runtimes/torch_distributed_with_cache_test.yaml |
Tests rendered cached-runtime security settings. |
9ecb903 to
15266dc
Compare
danish9039
changed the title
|
The CPU end-to-end failure is addressed by kubeflow/sdk#604, which keeps generated files writable without overriding custom image working directories. @andreyvelich @juliusvonkohout for review |
Signed-off-by: danish9039 <danishsiddiqui040@gmail.com>
Signed-off-by: danish9039 <danishsiddiqui040@gmail.com>
Master moved the torch runtimes to pytorch/pytorch:2.13.0-cuda13.0-cudnn9-runtime. Signed-off-by: danish9039 <danishsiddiqui040@gmail.com>
pytorch/pytorch declares no USER and owns /workspace as root, so under runAsUser 1000 any relative write from the image working directory fails with Permission denied. The Fashion-MNIST example downloads to ./data and hits exactly that. Mount an emptyDir at /workspace on the trainer node so the working directory is writable without overriding the image WORKDIR. No fsGroup is needed: emptyDir is created world-writable. Signed-off-by: danish9039 <danishsiddiqui040@gmail.com>
The kubelet can only verify runAsNonRoot against a numeric image user, so USER cache_user leaves the image dependent on an explicit runAsUser. Signed-off-by: danish9039 <danishsiddiqui040@gmail.com>
9bb6f2f to
a5364b5
Compare
There was a problem hiding this comment.
🟡 Changes recommended
The workspace mount masks custom-image contents, and generated cache pods unnecessarily force configurable images to UID 1000.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Review details
- Files reviewed: 9/9 changed files
- Comments generated: 5
- Review effort level: Balanced
| volumeMounts: | ||
| - name: workspace | ||
| mountPath: /workspace |
| volumeMounts: | ||
| - name: workspace | ||
| mountPath: /workspace |
| volumeMounts: | ||
| - name: workspace | ||
| mountPath: /workspace |
| volumeMounts: | ||
| - name: workspace | ||
| mountPath: /workspace |
| "allowPrivilegeEscalation": False, | ||
| "capabilities": {"drop": ["ALL"]}, | ||
| "runAsNonRoot": True, | ||
| "runAsUser": 1000, |
What this PR does / why we need it:
This PR makes the Torch distributed Trainer runtime manifests and matching Helm templates restricted-PSS compatible.
It adds pod and container security contexts for the Torch distributed runtime and the data-cache variant, and updates Helm unit tests to cover those fields.
The runtime containers preserve each image's configured working directory so custom images retain their existing command and relative-path behavior.
The CPU end-to-end tests also require kubeflow/sdk#604, which writes SDK-generated files to a writable location without overriding the image working directory.
Validation:
make helm-unittestpassed: 18 suites / 109 testskustomize build manifests/base/runtimeskustomize build manifests/base/runtimes/data-cachehelm templatefrom a temporary chart copy after dependency buildgit diff --checkWhich issue(s) this PR fixes (optional, in
Fixes #<issue number>, #<issue number>, ...format, will close the issue(s) when PR gets merged):Related to kubeflow/community-distribution#3487
Checklist: