Skip to content

ci: enforce Community GPU scope and host requirements - #1620

Merged
yifeif-nv merged 4 commits into
NVIDIA:ci/developerfrom
yifeif-nv:fix/community-gpu-owner-resources-20261008
Oct 8, 2026
Merged

yifeif-nv merged 4 commits into
NVIDIA:ci/developerfrom
yifeif-nv:fix/community-gpu-owner-resources-20261008

Conversation

@yifeif-nv

@yifeif-nv yifeif-nv commented Oct 8, 2026 •

Copy link
Copy Markdown
Collaborator

Background

The real Community runs exposed two capacity mismatches: Docker reported OOM for Nemotron-H 9B on the 64 GiB host, while Clef's complete 27B workload belongs outside the intended lightweight Community scope. Nemotron's unchanged E2E subsequently passed on a 128 GiB host with the same L4. Its image upload succeeded but the registry visibility gate rejected the response, so no image has been admitted.

Exit Criteria

Choose only qualified, bounded host profiles before allocation. Explicitly deferred owners must remain visible without being counted as model passes; a PR with only deferred owners must still run the standard GPU smoke cases. Registry diagnostics must work without allocating a VM, and published receipts must be readable by the SSH owner while credentials remain private.

Implementation

  • Read owner resource records from the protected CI commit and select one of four fixed 64/128 GiB provider types. Default owners remain at 64 GiB; cache misses retain an admitted owner's profile. No owner lock is added yet.
  • Fully deferred owners trigger the existing five-family GPU smoke suite within the original budget. Mixed selections run their active owners. Summaries require actual completed cases, and the image producer explicitly rejects deferred-only owner qualification.
  • Add a protected manual dependency-image-audit task with only package read access. It records filtered metadata and never creates a VM, publishes an image or changes visibility.
  • Preserve the producer's root-private proof and export only its nonsecret receipt to the SSH user's private directory. The restrictive-umask transfer bug was reproduced locally and is separate from the live visibility failure.

Change categories

  • CI or developer tooling

Validation

Commands and Results

  • python3 -m pytest .github/tests/test_community_dependency_image.py tools/tests/test_community_ci.py tools/tests/test_community_gpu_ci.py tools/tests/test_architecture.py -q -p no:cacheprovider: 638 passed, 1 optional real-Docker integration test skipped on the combined head.
  • Real filesystem controls under umask 077 confirmed the original SSH permission denial and verified that only the exported receipt becomes readable by its intended UID; unrelated users remain denied.
  • Ruff check/format on six changed Python files, actionlint on both workflows, and git diff --check: passed.

Hardware, Environment, and Revisions

Tested head ec3ace17f4d59d706c365d7306cb0bea7eb67d15, tree 0a753987a4d65ca080b31afc942a228a4c2fde62, based on deployed Dev cb4262ba06fbe3a19e7b68456421b81cac5ec0dc. Local controller tests used isolated Linux ARM64 CPU containers. Motivating 128 GiB L4 run used unchanged main 48765e7d: native BYOK and the selected original Nemotron E2E passed; both cleanup paths confirmed deletion.

Not Run / Remaining Gaps

Live Dev run 37795815020 passed the default 64 GiB pre-allocation profile, all 9 actual GPU baseline cases and both cleanup paths while reporting Clef explicitly deferred with no model download/container. The original source selection also included the shared smoke owners; the purely deferred-owner expansion branch is covered by local execution controls, not a separate cloud trial. The read-only registry audit successfully retrieved matching package/version metadata and then failed the private-only gate because the actual visibility is public. No image digest or 128 GiB owner lock is admitted. Live 128 GiB profile consumption and successful receipt export remain unqualified; the user's registry visibility preference is pending. No numerical criteria changed, and Nightly was not executed.

Contributor Self-Review

  • I have completed a self-review of this change.

Notes For Future Readers

Review pre-allocation admission and summary coverage first, then the metadata-only audit and receipt ownership. Resource selection conservatively uses requested owners before the PR's runtime plans exist; a future fully deferred owner may retain its larger admitted profile. This remains bounded and does not affect the currently active Nemotron case. The producer still requires original family coverage, private registry verification, and confirmed cleanup before image admission. This PR targets Dev only.

Risk level

  • Medium

The changes affect GPU admission and coverage reporting. Fixed profiles, trusted metadata, original budgets, explicit deferred results, real smoke coverage and the existing owner/backstop cleanup constrain the behavior; live verification is still required.

Read selected owners' host RAM requirements from trusted CI dependency
locks before allocating one VM. Restrict the profile to 64 or 128 GiB and
map it to four fixed single-GPU x86 instance types.

Validate larger-host qualification before create and retain the owner
profile when changed dependency inputs require a cold install. Preserve
the 64 GiB default, provider selection, and confirmed cleanup behavior.

No owner lock is admitted by this change. Activating a 128 GiB owner still
requires successful real workload qualification and confirmed release.

Signed-off-by: yifeif <277870278+yifeif-nv@users.noreply.github.com>
Report fully deferred Community owners without staging their checkpoints or launching their containers. When every requested owner is explicitly deferred, execute the existing five smoke families within the original allocation budget. Mixed selections retain only their active owners; missing cases and invalid flags still fail.

Require real active-case evidence for complete and passing summaries. Dependency-image producers use a strict coverage guard before Docker so fallback smoke cannot qualify a deferred family image. Describe Docker OOM evidence without asserting the kernel memory cause.

Validation: 591 local producer, GPU orchestration, source-contract and workflow tests passed with one existing optional Docker-image skip. Ruff, actionlint, compile and diff checks passed. Live 128-GiB producer monitoring remains on its immutable deployed source.
Signed-off-by: yifeif <277870278+yifeif-nv@users.noreply.github.com>
Keep the privileged producer proof private and export only the nonsecret
published receipt into the SSH user's existing private auth directory.
Assign that file the directory owner's UID/GID and mode 0600, then copy it
from the auth path without adding a workflow step or time budget.

Preserve qualification, immutable digest, private visibility, credential
cleanup, and confirmed VM deletion gates. The restrictive-umask transfer
defect is separate from the earlier live visibility-gate failure.

Signed-off-by: yifeif <277870278+yifeif-nv@users.noreply.github.com>
Inspect package visibility and immutable version metadata through a protected
manual Community workflow job with package read access. Preserve the filtered
evidence even when visibility fails the existing private-package requirement.

Keep audit dispatches outside PR checks and image production, and recheck the
triggering actor before requests. The audit allocates no VM and does not publish
or admit an image.

Signed-off-by: yifeif-nv <277870278+yifeif-nv@users.noreply.github.com>
@coderabbitai

coderabbitai Bot commented Oct 8, 2026

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration
  • Configuration used: Repository: NVIDIA/TensorRT-Model-Connect/.coderabbit.yaml
  • Review profile: CHILL
  • Plan: Enterprise
  • Run ID: d2c67c9e-5704-4e65-84d4-c335e0eb19e3

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
  • Autopilot · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Comment @coderabbitai help to get the list of available commands.

@yifeif-nv
yifeif-nv merged commit 13947f2 into NVIDIA:ci/developer Oct 8, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant