Repository navigation
Conversation
…fraction In NvFractions mode the binder turns gpu-memory.portion.limit into the nvidia.com/container.<c>.gpu-memory.limit annotation at bind time and keeps gpu-fraction on the pod. ValidateGPUFractionRequest then rejected that update as "cannot combine ... limit annotation with gpu-fraction", so every bind of a pod with a portion limit failed and the pod cycled between Scheduled and BindingError. Exempt the translated limit when the portion-limit annotation is present, mirroring the existing gpu-memory.limit shorthand exemption. Ref kai-scheduler/KAI-Scheduler#2490 Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Signed-off-by: Zachary Zhao <ongarabazanadi77@gmail.com>
ZeXi0308
force-pushed
the
fix/nvfractions-portion-limit
branch
from
October 24, 2026 19:07
34a67dd to
6dfa765
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
In
NvFractionsGPU sharing mode, a pod that requestsgpu-fractiontogether withkai.scheduler/container.<c>.gpu-memory.portion.limitpasses admission and gets scheduled, but every bind fails:The two halves disagree:
nvfractionsplugin translates the portion limit intonvidia.com/container.<c>.gpu-memory.limit(portion × the node'snvidia.com/gpu.memory) at bind time and keepsgpu-fractionon the pod.ValidateGPUFractionRequestrejects any NvFractions limit next togpu-fraction. It already exempts the analogousgpu-memory+ translatedgpu-memory.limitcase (hasMemoryLimitShorthand), but had no exemption for a translated portion limit.Since
validateGpuMemoryPortionLimitAnnotationrequiresgpu-fractionto be present, a portion limit could never bind in NvFractions mode; the pod cycles betweenScheduledandBindingError.This PR exempts the translated limit when the pod carries a
gpu-memory.portion.limitannotation, mirroring the existing shorthand exemption. Thegpu-fraction.limitshorthand is covered too, because the admission mutation translates it intogpu-memory.portion.limitbefore validation runs.Related Issues
Ref kai-scheduler/KAI-Scheduler#2490
Checklist
make changelog(or applied theskip-changeloglabel). Do not editCHANGELOG.mddirectly — pending fragments are folded into it at release time.Breaking Changes
None.
Additional Notes
Tests added (
TestValidateGPUFractionRequest_TranslatedLimits):gpu-fractionwith/without a portion limit next to an NvFractions limit, plus the existinggpu-memorywith/withoutgpu-memory.limitprecedent as a control. The "without" cases still fail withcannot combine.Verified against KAI-Scheduler with a local
replace github.com/kai-scheduler/api => ../api: allpkg/admission/webhook/...packages (includingnvfractionsandgpusharing, whose "rejects nvfractions limit combined with gpu-fraction" case still passes) andpkg/binder/plugins/{nvfractions,gpusharing}pass.Reviewer note: keeping the fix in admission rather than having the binder strip
gpu-fractionseemed right, since the portion-limit validation requiresgpu-fractionand the binder should not rewrite the user's request. One consequence to be aware of: a user-suppliednvidia.com/container.<c>.gpu-memory.limitnext togpu-fraction+gpu-memory.portion.limitis now accepted at create time as well, which is the same latitude thegpu-memory+gpu-memory.limitprecedent already has. Happy to tighten this if you prefer.Once this is tagged I'll open the follow-up PR in KAI-Scheduler to bump the dependency and close the issue.
🤖 Generated with Claude Code