fix(self-managed): declare the LLM PKI issuer in one Helmfile state - #951
Conversation
The nvcf-pki release was declared in both 01-dependencies and 02-core. 01-dependencies gates it on the full issuer-ownership model. 02-core gated it only on addons.llm.pki.enabled, so an operator who pointed the stack at their own external issuer still got a ClusterIssuer created under that name, backed by an OpenBao they had disabled. The nvcf-pki chart keeps that object through uninstall and rollback, so the request-router Certificate never issues until it is removed by hand. With the managed default both states applied the same release twice. Delete the 02-core declaration and the now-dangling - cert-manager/nvcf-pki edge on llm-request-router. Ordering still holds because 01-dependencies completes before 02-core starts. The six issuer knobs the 02-core copy introduced were undocumented and referenced nowhere else. Both test scripts now assert across the whole helmfile.d directory instead of a single state file. llm-pki-release.sh checks that exactly one nvcf-pki release is declared in cert-manager, and that no state declares a needs edge on a release another state owns. check-llm-pki-issuer.sh gains cross-state declaration counts for the managed and external-issuer configs, and is now wired into make test. Closes #950 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Enterprise Run ID: 📒 Files selected for processing (4)
💤 Files with no reviewable changes (1)
Included review availability: Your plan includes up to 12 reviews per rolling hour; 11 remain after this review. 📝 WalkthroughWalkthroughThe self-managed stack removes the duplicate ChangesLLM PKI ownership
Estimated code review effort: 3 (Moderate) | ~20 minutes Merge Risk: 🔵 Low · up to The duplicate release is removed, but the validation script may overlook declarations or dependency edges if it does not recover every rendered state. The PR is mergeable with owner awareness and follow-up to make incomplete state recovery fail closed. Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Comment |
The LLM PKI profile always rendered a cert-manager Certificate when addons.llm.pki.enabled was true, so an operator who already issues the request-router server certificate had no supported way to mount it. Add an explicit existing-Secret identity mode at both layers: - llmRequestRouter.tls.mode selects certManager (default, unchanged) or existingSecret. existingSecret requires tls.secretName, tls.certPath, and tls.keyPath, and rejects certificate.enabled=true as mixed ownership. The guard runs from deployment.yaml because no Certificate renders in this mode. - addons.llm.pki.mode plumbs the same choice through the stack. existingSecret renders certificate.enabled=false, skips the OpenBao provisioning hook, and leaves the nvcf-pki release unresolved in the dependency state, so no cert-manager or issuer ownership is introduced. clusterIssuer.enabled, dnsNames, and allowedDomains only steer stack-managed issuance and now fail rendering in this mode. Worker trust is unchanged. The existing NVCA workload transport trust contract carries the public bundle and optional fingerprint, and its certificates-only validation already keeps the server private key off the compute plane. Render tests cover cert-manager mode, existing-Secret mode, mixed ownership, and each missing or conflicting value, in both the chart and the stack. The stack cases assert zero nvcf-pki declarations across the whole helmfile.d, reusing the cross-state helpers added in #951. Closes #947 Signed-off-by: Mike Camp <mcamp@nvidia.com> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Why
deploy/stacks/self-managed/declared thenvcf-pkiHelm release in twoHelmfile states that disagreed with each other.
01-dependenciesgates it onthe full issuer-ownership model (managed ClusterIssuer, external ClusterIssuer,
external namespaced Issuer) and carries
needsedges on OpenBao andcert-manager.
02-coregated it only oncondition: addons.llm.pki.enabled.makeapplies the wholehelmfile.d/directory in one command, so bothdeclarations were applied together.
With the managed default this double-applied the same release. With an
explicitly external issuer it was worse:
01-dependenciescorrectly skippedthe release, and
02-coreinstalled it anyway withclusterIssuer.enabled: true, the operator's own issuer name, and aserverpointing at an OpenBao that was not deployed. The stack created a
cluster-scoped
ClusterIssuerunder the name the operator had reserved fortheir external issuer. It never reached
Ready=True, the request-routerCertificatenever issued, and thenvcf-pkichart setshelm.sh/resource-policy: keep, so the bad object survived uninstall androllback.
Two changes landed on the same day and did not see each other. The one that
added the
02-coredeclaration described a gap that the other had alreadyclosed.
Neither test caught it.
llm-pki-release.shrendered only02-core.check-llm-pki-issuer.shasserted the ownership matrix only against01-dependencies.What changed
nvcf-pkirelease fromhelmfile.d/02-core.yaml.gotmpl, andremoved the
- cert-manager/nvcf-pkientry from thellm-request-routerrelease's
needslist, which would otherwise dangle. Ordering still holdsbecause
01-dependenciescompletes before02-corestarts, like every otherdependency in the stack. The six issuer knobs the
02-corecopy introduced(
issuerServer,issuerPath,issuerAuthMountPath,issuerAuthRole,issuerServiceAccountName,issuerAudience) were referenced nowhere else inthe repo and are gone with it.
tests/llm-pki-release.shto render the wholehelmfile.d/directory. It keeps both of its original intents and widens them: the issuer
is declared exactly once in
cert-manageracross all states, and no statedeclares a
needsedge on a release another state owns. The needs check isnow generic over every release in every state, including states pulled in
through nested helmfiles.
tests/check-llm-pki-issuer.sh: the manageddefault declares exactly one
nvcf-pkirelease acrosshelmfile.d/, and theexternal-issuer config declares zero. The count is of declarations, not
enabled releases, so a declaration that is merely conditioned off still
fails.
tests/check-llm-pki-issuer.shintomake test. It was not run by anytarget, which is part of why this slipped through.
01-dependenciesis the declaration that survives because it owns theownership model that
docs/user/llm-function-enablement.mddocuments and thatthe 549-line
check-llm-pki-issuer.shcovers, and it is the only one that canexpress real
needsedges on OpenBao and cert-manager.Customer Release Notes
Fixed a self-managed stack defect where configuring an external issuer for LLM
request-router TLS still created a stack-owned
ClusterIssuerunder thatissuer's name, pointed at an OpenBao instance that was not installed. The bad
object was retained through uninstall and rollback and had to be deleted by
hand.
Plan Summary
Removes one duplicate Helm release declaration from the self-managed stack. In
the managed default the same release is now applied once instead of twice, with
identical values. With an external issuer, no
nvcf-pkirelease is installedat all, which is the documented behavior. No chart, image, or value defaults
change.
Usage
Not applicable.
Testing
Run from
deploy/stacks/self-managed:Both new assertions were mutation-tested by restoring the stray declaration:
llm-pki-release.shreportsfound 2, andcheck-llm-pki-issuer.shreportsmanaged-defaults-all expected 1 ... got 2and, with that case skipped,external-issuer-all expected 0 ... got 1. The new cross-stateneedscheckwas mutation-tested by re-adding the
cert-manager/nvcf-pkiedge onllm-request-router, which it reports as dangling.Verified end to end before and after: the external-issuer config went from 1
declared
nvcf-pkirelease to 0, and the managed default from 2 to 1.tests/pdb-value-wiring.shfails on cleanmainwithno releases found that matches specified selector(name=cassandra)againsthelmfile 1.7.4. That is pre-existing and unrelated, so the three scripts above
were run individually rather than through
make test. No QA needed.Notes
A separate in-flight branch for a pre-created request-router TLS Secret mode
also edits
02-core.yaml.gotmpland wraps thenvcf-pkirelease plus therouter's
needsedge in a cert-manager-mode guard. Expect a conflict there.Resolve in favor of this change: drop the wrapped release and the wrapped
needsedge, keep the rest. That branch's behavior is correct either way andneeds no logic change.
References
Closes #950
Related Pull Requests
None
Dependencies
None
Summary by CodeRabbit