Repository navigation
Conversation
Add a KEP that lets an MPI runtime declare several node groups and lets a TrainJob set the size, slots, image and resources of each group and of the launcher with one schema. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Signed-off-by: Matteo Bunino <48362942+matbun@users.noreply.github.com>
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
|
@andreyvelich I just added user stories as requested |
Update the Go types to the implemented ones (launcher as a struct and containerName as a string, as the Kube API Linter requires), add the rules the implementation enforces (no trainjob-ancestor-step label, successPolicy and failurePolicy references to disabled groups dropped, resourceClaimsPerNode rejected with node groups), and link the draft implementation. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Signed-off-by: Matteo Bunino <48362942+matbun@users.noreply.github.com>
| workers: | ||
| - jobName: highcore | ||
| numNodes: 1 | ||
| labels: {home: afs, mount-eos: "true", mount-cvmfs: "true", mount-afs: "true", rdma: "true"} | ||
| nodeSelector: |
There was a problem hiding this comment.
One topic we should discuss while reviewing this KEP is that TrainJob users usually are not aware of TrainingRuntime configuration, as they only reference the desired runtime.
Therefore, @tenzen-y proposed initial approach with the ancestor label to merge values between TrainJob and Runtime spec.
What this PR does / why we need it:
This PR adds KEP-4064, which lets one MPI TrainJob run several node groups, each with its own pod template, number of nodes and slots, so a single job can mix CPU, NVIDIA and AMD nodes. It follows the discussion in #4064 and lists the flat launcher design proposed there under Alternatives.
Which issue(s) this PR fixes (optional, in
Fixes #<issue number>, #<issue number>, ...format, will close the issue(s) when PR gets merged):Part of #4064
Checklist:
Cc @andreyvelich