Skip to content

fix(api): accept the binder-translated NvFractions limit next to gpu-fraction - #28

Open
ZeXi0308 wants to merge 1 commit into
kai-scheduler:mainfrom
ZeXi0308:fix/nvfractions-portion-limit
Open

ZeXi0308 wants to merge 1 commit into
kai-scheduler:mainfrom
ZeXi0308:fix/nvfractions-portion-limit

Conversation

@ZeXi0308

Copy link
Copy Markdown

Description

In NvFractions GPU sharing mode, a pod that requests gpu-fraction together with kai.scheduler/container.<c>.gpu-memory.portion.limit passes admission and gets scheduled, but every bind fails:

admission webhook denied the request: cannot combine nvidia.com/container. limit annotation with gpu-fraction or gpu-memory annotation

The two halves disagree:

  1. The binder's nvfractions plugin translates the portion limit into nvidia.com/container.<c>.gpu-memory.limit (portion × the node's nvidia.com/gpu.memory) at bind time and keeps gpu-fraction on the pod.
  2. ValidateGPUFractionRequest rejects any NvFractions limit next to gpu-fraction. It already exempts the analogous gpu-memory + translated gpu-memory.limit case (hasMemoryLimitShorthand), but had no exemption for a translated portion limit.

Since validateGpuMemoryPortionLimitAnnotation requires gpu-fraction to be present, a portion limit could never bind in NvFractions mode; the pod cycles between Scheduled and BindingError.

This PR exempts the translated limit when the pod carries a gpu-memory.portion.limit annotation, mirroring the existing shorthand exemption. The gpu-fraction.limit shorthand is covered too, because the admission mutation translates it into gpu-memory.portion.limit before validation runs.

Related Issues

Ref kai-scheduler/KAI-Scheduler#2490

Checklist

  • Self-reviewed
  • Added/updated tests (if needed)
  • Updated documentation (if needed) — n/a
  • Added a changelog fragment via make changelog (or applied the skip-changelog label). Do not edit CHANGELOG.md directly — pending fragments are folded into it at release time.

Breaking Changes

None.

Additional Notes

Tests added (TestValidateGPUFractionRequest_TranslatedLimits): gpu-fraction with/without a portion limit next to an NvFractions limit, plus the existing gpu-memory with/without gpu-memory.limit precedent as a control. The "without" cases still fail with cannot combine.

Verified against KAI-Scheduler with a local replace github.com/kai-scheduler/api => ../api: all pkg/admission/webhook/... packages (including nvfractions and gpusharing, whose "rejects nvfractions limit combined with gpu-fraction" case still passes) and pkg/binder/plugins/{nvfractions,gpusharing} pass.

Reviewer note: keeping the fix in admission rather than having the binder strip gpu-fraction seemed right, since the portion-limit validation requires gpu-fraction and the binder should not rewrite the user's request. One consequence to be aware of: a user-supplied nvidia.com/container.<c>.gpu-memory.limit next to gpu-fraction + gpu-memory.portion.limit is now accepted at create time as well, which is the same latitude the gpu-memory + gpu-memory.limit precedent already has. Happy to tighten this if you prefer.

Once this is tagged I'll open the follow-up PR in KAI-Scheduler to bump the dependency and close the issue.

🤖 Generated with Claude Code

…fraction

In NvFractions mode the binder turns gpu-memory.portion.limit into the
nvidia.com/container.<c>.gpu-memory.limit annotation at bind time and keeps
gpu-fraction on the pod. ValidateGPUFractionRequest then rejected that update
as "cannot combine ... limit annotation with gpu-fraction", so every bind of a
pod with a portion limit failed and the pod cycled between Scheduled and
BindingError.

Exempt the translated limit when the portion-limit annotation is present,
mirroring the existing gpu-memory.limit shorthand exemption.

Ref kai-scheduler/KAI-Scheduler#2490

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Signed-off-by: Zachary Zhao <ongarabazanadi77@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant