Skip to content

fix(self-managed): declare the LLM PKI issuer in one Helmfile state - #951

Merged
mikeyrcamp merged 1 commit into
mainfrom
fix/llm-pki-duplicate-release
Aug 18, 2026
Merged

fix(self-managed): declare the LLM PKI issuer in one Helmfile state#951
mikeyrcamp merged 1 commit into
mainfrom
fix/llm-pki-duplicate-release

Conversation

@mikeyrcamp

@mikeyrcamp mikeyrcamp commented Aug 18, 2026

Copy link
Copy Markdown
Contributor

Why

deploy/stacks/self-managed/ declared the nvcf-pki Helm release in two
Helmfile states that disagreed with each other. 01-dependencies gates it on
the full issuer-ownership model (managed ClusterIssuer, external ClusterIssuer,
external namespaced Issuer) and carries needs edges on OpenBao and
cert-manager. 02-core gated it only on condition: addons.llm.pki.enabled.
make applies the whole helmfile.d/ directory in one command, so both
declarations were applied together.

With the managed default this double-applied the same release. With an
explicitly external issuer it was worse: 01-dependencies correctly skipped
the release, and 02-core installed it anyway with
clusterIssuer.enabled: true, the operator's own issuer name, and a server
pointing at an OpenBao that was not deployed. The stack created a
cluster-scoped ClusterIssuer under the name the operator had reserved for
their external issuer. It never reached Ready=True, the request-router
Certificate never issued, and the nvcf-pki chart sets
helm.sh/resource-policy: keep, so the bad object survived uninstall and
rollback.

Two changes landed on the same day and did not see each other. The one that
added the 02-core declaration described a gap that the other had already
closed.

Neither test caught it. llm-pki-release.sh rendered only 02-core.
check-llm-pki-issuer.sh asserted the ownership matrix only against
01-dependencies.

What changed

  • Deleted the nvcf-pki release from helmfile.d/02-core.yaml.gotmpl, and
    removed the - cert-manager/nvcf-pki entry from the llm-request-router
    release's needs list, which would otherwise dangle. Ordering still holds
    because 01-dependencies completes before 02-core starts, like every other
    dependency in the stack. The six issuer knobs the 02-core copy introduced
    (issuerServer, issuerPath, issuerAuthMountPath, issuerAuthRole,
    issuerServiceAccountName, issuerAudience) were referenced nowhere else in
    the repo and are gone with it.
  • Rewrote tests/llm-pki-release.sh to render the whole helmfile.d/
    directory. It keeps both of its original intents and widens them: the issuer
    is declared exactly once in cert-manager across all states, and no state
    declares a needs edge on a release another state owns. The needs check is
    now generic over every release in every state, including states pulled in
    through nested helmfiles.
  • Added two cross-state cases to tests/check-llm-pki-issuer.sh: the managed
    default declares exactly one nvcf-pki release across helmfile.d/, and the
    external-issuer config declares zero. The count is of declarations, not
    enabled releases, so a declaration that is merely conditioned off still
    fails.
  • Wired tests/check-llm-pki-issuer.sh into make test. It was not run by any
    target, which is part of why this slipped through.

01-dependencies is the declaration that survives because it owns the
ownership model that docs/user/llm-function-enablement.md documents and that
the 549-line check-llm-pki-issuer.sh covers, and it is the only one that can
express real needs edges on OpenBao and cert-manager.

Customer Release Notes

Fixed a self-managed stack defect where configuring an external issuer for LLM
request-router TLS still created a stack-owned ClusterIssuer under that
issuer's name, pointed at an OpenBao instance that was not installed. The bad
object was retained through uninstall and rollback and had to be deleted by
hand.

Plan Summary

Removes one duplicate Helm release declaration from the self-managed stack. In
the managed default the same release is now applied once instead of twice, with
identical values. With an external issuer, no nvcf-pki release is installed
at all, which is the documented behavior. No chart, image, or value defaults
change.

Usage

Not applicable.

Testing

Run from deploy/stacks/self-managed:

tests/check-llm-pki-issuer.sh          passed
tests/llm-pki-release.sh               passed
tests/llm-router-worker-address.sh     passed
git diff --check                       clean

Both new assertions were mutation-tested by restoring the stray declaration:
llm-pki-release.sh reports found 2, and check-llm-pki-issuer.sh reports
managed-defaults-all expected 1 ... got 2 and, with that case skipped,
external-issuer-all expected 0 ... got 1. The new cross-state needs check
was mutation-tested by re-adding the cert-manager/nvcf-pki edge on
llm-request-router, which it reports as dangling.

Verified end to end before and after: the external-issuer config went from 1
declared nvcf-pki release to 0, and the managed default from 2 to 1.

tests/pdb-value-wiring.sh fails on clean main with
no releases found that matches specified selector(name=cassandra) against
helmfile 1.7.4. That is pre-existing and unrelated, so the three scripts above
were run individually rather than through make test. No QA needed.

Notes

A separate in-flight branch for a pre-created request-router TLS Secret mode
also edits 02-core.yaml.gotmpl and wraps the nvcf-pki release plus the
router's needs edge in a cert-manager-mode guard. Expect a conflict there.
Resolve in favor of this change: drop the wrapped release and the wrapped
needs edge, keep the rest. That branch's behavior is correct either way and
needs no logic change.

References

Closes #950

Related Pull Requests

None

Dependencies

None

Summary by CodeRabbit

  • Deployment
    • Removed the self-managed stack’s standalone PKI release configuration.
    • Updated the LLM request router’s deployment dependency.
  • Bug Fixes
    • Improved validation of PKI issuer ownership across Helmfile states.
    • Added checks to ensure valid release dependencies and namespace placement.
    • Ensured PKI is declared exactly once when enabled and omitted when externally managed.

The nvcf-pki release was declared in both 01-dependencies and 02-core.
01-dependencies gates it on the full issuer-ownership model. 02-core gated
it only on addons.llm.pki.enabled, so an operator who pointed the stack at
their own external issuer still got a ClusterIssuer created under that name,
backed by an OpenBao they had disabled. The nvcf-pki chart keeps that object
through uninstall and rollback, so the request-router Certificate never
issues until it is removed by hand. With the managed default both states
applied the same release twice.

Delete the 02-core declaration and the now-dangling
- cert-manager/nvcf-pki edge on llm-request-router. Ordering still holds
because 01-dependencies completes before 02-core starts. The six issuer
knobs the 02-core copy introduced were undocumented and referenced nowhere
else.

Both test scripts now assert across the whole helmfile.d directory instead
of a single state file. llm-pki-release.sh checks that exactly one nvcf-pki
release is declared in cert-manager, and that no state declares a needs edge
on a release another state owns. check-llm-pki-issuer.sh gains cross-state
declaration counts for the managed and external-issuer configs, and is now
wired into make test.

Closes #950

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@mikeyrcamp
mikeyrcamp requested a review from a team as a code owner August 18, 2026 17:44
@mikeyrcamp
mikeyrcamp requested a review from along-2017 August 18, 2026 17:44
@coderabbitai

coderabbitai Bot commented Aug 18, 2026

Copy link
Copy Markdown

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: b3f31397-32de-4a6a-978f-4b698a92a786

📥 Commits

Reviewing files that changed from the base of the PR and between 1badcd4 and 3cb7794.

📒 Files selected for processing (4)
  • deploy/stacks/self-managed/Makefile
  • deploy/stacks/self-managed/helmfile.d/02-core.yaml.gotmpl
  • deploy/stacks/self-managed/tests/check-llm-pki-issuer.sh
  • deploy/stacks/self-managed/tests/llm-pki-release.sh
💤 Files with no reviewable changes (1)
  • deploy/stacks/self-managed/helmfile.d/02-core.yaml.gotmpl

Included review availability: Your plan includes up to 12 reviews per rolling hour; 11 remain after this review.


📝 Walkthrough

Walkthrough

The self-managed stack removes the duplicate nvcf-pki release from the core Helmfile state. It updates routing dependencies and extends tests to validate release ownership, dependency targets, namespaces, and uniqueness across all Helmfile states.

Changes

LLM PKI ownership

Layer / File(s) Summary
Remove duplicate release and update routing dependency
deploy/stacks/self-managed/helmfile.d/02-core.yaml.gotmpl
The core state no longer declares nvcf-pki. llm-request-router now depends on nvcf/api.
Validate cross-state release ownership
deploy/stacks/self-managed/tests/check-llm-pki-issuer.sh, deploy/stacks/self-managed/Makefile
The ownership test renders all Helmfile states and checks managed and external issuer declaration counts. The test runs from the self-managed test target.
Validate rendered release graph
deploy/stacks/self-managed/tests/llm-pki-release.sh
The test renders all states, validates same-state needs targets and the cert-manager namespace, and requires exactly one nvcf-pki declaration.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Merge Risk: 🔵 Low · up to 3cb77

The duplicate release is removed, but the validation script may overlook declarations or dependency edges if it does not recover every rendered state. The PR is mergeable with owner awareness and follow-up to make incomplete state recovery fail closed.

Suggested reviewers: along-2017

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title uses the required Conventional Commits format and accurately describes the duplicate LLM PKI declaration fix.
Linked Issues check ✅ Passed The changes satisfy issue #950 by removing the duplicate declaration, fixing the dangling dependency, and expanding tests for managed and external issuer cases.
Out of Scope Changes check ✅ Passed All changes support issue #950 by modifying Helmfile ownership and related validation tests; no unrelated changes are present.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch fix/llm-pki-duplicate-release

Comment @coderabbitai help to get the list of available commands.

@mikeyrcamp
mikeyrcamp added this pull request to the merge queue Aug 18, 2026
Merged via the queue into main with commit 46b37f4 Aug 18, 2026
15 checks passed
@mikeyrcamp
mikeyrcamp deleted the fix/llm-pki-duplicate-release branch August 18, 2026 18:58
mikeyrcamp added a commit that referenced this pull request Aug 18, 2026
The LLM PKI profile always rendered a cert-manager Certificate when
addons.llm.pki.enabled was true, so an operator who already issues the
request-router server certificate had no supported way to mount it.

Add an explicit existing-Secret identity mode at both layers:

- llmRequestRouter.tls.mode selects certManager (default, unchanged) or
  existingSecret. existingSecret requires tls.secretName, tls.certPath, and
  tls.keyPath, and rejects certificate.enabled=true as mixed ownership. The
  guard runs from deployment.yaml because no Certificate renders in this mode.
- addons.llm.pki.mode plumbs the same choice through the stack. existingSecret
  renders certificate.enabled=false, skips the OpenBao provisioning hook, and
  leaves the nvcf-pki release unresolved in the dependency state, so no
  cert-manager or issuer ownership is introduced. clusterIssuer.enabled,
  dnsNames, and allowedDomains only steer stack-managed issuance and now fail
  rendering in this mode.

Worker trust is unchanged. The existing NVCA workload transport trust contract
carries the public bundle and optional fingerprint, and its certificates-only
validation already keeps the server private key off the compute plane.

Render tests cover cert-manager mode, existing-Secret mode, mixed ownership,
and each missing or conflicting value, in both the chart and the stack. The
stack cases assert zero nvcf-pki declarations across the whole helmfile.d,
reusing the cross-state helpers added in #951.

Closes #947

Signed-off-by: Mike Camp <mcamp@nvidia.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix(self-managed): nvcf-pki release is declared in two Helmfile states

2 participants