Repository navigation
security(kyverno): system namespaces have had no Kyverno policy coverage since June #4228
Description
🤖 Generated by the Agentic Engineer
Evidence
- Kyverno's chart-default resource filters exclude the
kube-system,kube-public,kube-node-leaseandkyvernonamespaces from admission. Until fix(kyverno): curb reports-controller memory growth via resource filters #2134 (2026-06-17) background scans ignored those filters and still reported on those namespaces; fix(kyverno): curb reports-controller memory growth via resource filters #2134 made background scans honour them to cut report volume. - Read-only on prod, 2026-09-27: every PolicyReport in those four namespaces (64) carries results from 2026-06-17, and none of the 56 kube-system pods created since then has a report. So since June, no Kyverno policy has evaluated anything in these namespaces, at admission or in the background. The compliance dashboard does not show this: those namespaces look quiet, not uncovered.
- docs(kyverno): record that the system namespaces are never evaluated, and prove it #4227 clears the frozen June reports and documents the gap. It does not restore coverage.
Audience and problem
The maintainer reads the compliance aggregate as "the cluster's policy state". It silently excludes the namespaces that run the cluster's core system workloads.
Hypothesis
If background scans cover the system namespaces again, without adding admission enforcement there, the aggregate becomes a true statement about the whole cluster, at a report-volume cost that stays within the reports controller's memory budget.
Success signal
Every workload in the four namespaces that a background policy matches has a current report (under 2 hours old), and the reports controller's memory stays within its current limit for a week.
Smallest useful change
Decide first, with measurements. Kyverno has one filter list shared by admission and background scans, so the options differ in cost:
- Drop the four namespace entries from the filters, scoped by reviewing which admission policies would start applying there (a
failurePolicy: Failpolicy could then block a system pod). - Keep admission as is and cover those namespaces from another scanner (Kubescape already scans them), stating in the docs that Kyverno does not.
- Accept the gap, and say so on the dashboard.
Acceptance criteria
- The decision and its reasoning are recorded here.
- If coverage is restored, a policy that matches a kube-system resource produces a current report there, and no admission policy newly applies to those namespaces without a recorded review.
Metadata
Metadata
Assignees
Labels
Type
Fields
Projects
- Status
Activity
fix(kyverno): drop the unevaluated-namespace prune; the cleanup contr…
devantler commentedon Sep 28, 2026
Measurements for the decision (2026-09-28,
mainand prod read-only)1. Option 2's premise does not hold. The in-cluster Kubescape operator does not scan these namespaces either: its declared scope excludes
kube-system,kube-public,kube-node-leaseandkubescape(#3923, pinned by #3926; #3924 declined addingkube-system). Only the CI manifest scan sees them, and it sees manifests, not the live objects Talos and KSail create there. So "cover them from another scanner" is not available today without reopening #3924.2. Admission is gated by the webhooks, not only by the resource filters. Read live: every Kyverno resource webhook (
validate.kyverno.svc-fail,-ignore, the image-verify webhook, and both mutating ones) carries anamespaceSelectorofkubernetes.io/metadata.name NotIn [kube-system, kyverno]. The API server never sends those two namespaces' objects to Kyverno at admission, whatever the resource filters say. So dropping the filter entries would change admission only forkube-publicandkube-node-lease, which hold acluster-infoConfigMap and node Leases (Leases are already filtered by[Lease,*,*]).3. What would newly apply is background work. Of the 41 ClusterPolicies on
main, 12 exclude the system namespaces explicitly (among themvalidate-pod-security,validate-host-restrictions,add-security-context,add-default-deny,prefer-baseline-nodes). Among the rest, the ones that could act on existing system workloads in the background, rather than only report, are the generate and mutate-existing ones with no explicit exclusion found in a static read:auto-vpa,add-ns-quota,insert-pod-antiaffinity,spread-pods. Each needs a scoped review before the filters change. TheAudit-mode validators (validate-replica-*,validate-pdb-drain-safe,disallow-latest-tag) are what would restore reporting.Proposed decision
Option 1, sequenced: first give the four background-capable policies above an explicit system-namespace exclusion (or confirm one exists through a precondition), then drop the four namespace entries from the filters, then measure the reports controller's memory for a week against the success signal. Admission risk is limited by finding 2. Option 3 remains the fallback if the memory budget does not hold.
devantler commentedon Sep 28, 2026
A correction to finding 3 above. That static read looked only at the policy files and missed the kustomize patches in
cluster-policies/kustomization.yaml:add-ns-quotaalready excludes all four system namespaces, plus others.insert-pod-antiaffinityandspread-podsexcludekube-system. Both are admission-time mutations. The webhooknamespaceSelectoralready keepskube-systemandkyvernoout, andkube-publicandkube-node-leasehold no Deployments or StatefulSets.auto-vparelied on the filters alone. In prod it would have collided with the Flux-owned static VPAs of the same names. fix(kyverno): auto-vpa must exclude the system namespaces explicitly before the resource filters drop #4231 / fix(kyverno): exclude the system namespaces from auto-vpa explicitly #4232 add its explicit exclusion.With #4232 merged, the prerequisite in the proposed decision is met. The next step is to drop the four namespace entries from the resource filters and measure the reports controller's memory for a week.
devantler commentedon Sep 28, 2026
A second correction, found while implementing the next step. The earlier review checked only namespace-scoped excludes.
add-default-denyandadd-default-limitrangematchNamespaceobjects and exclude by name: they listkube-system,kube-publicandkube-node-lease, but notkyverno. Dropping thekyvernofilter entry would make Kyverno generate a default-deny network policy in its own namespace, and adefault-limitrangethat collides with the Flux-owned LimitRange of the same name that already lives there.So the work is split:
kyvernofiltered. The throwaway-cluster proof now requires a report in each of the three and none inkyverno.kyvernoneeds its own decision first: either exclude it from those two generate policies by name, or accept that Kyverno's own namespace stays unevaluated. That remains open here.devantler commentedon Sep 28, 2026
#4235 deployed — first read-only check (2026-09-28 ~08:15Z)
kyvernoConfigMap's resource filters now keep only[*/*,kyverno,*]among the namespace entries.Measurement date: 2026-10-05. The #4234 success signal is that the reports controller stays within its limit for a week; compare restarts and memory then. Still open here: whether to cover the
kyvernonamespace (needs the two generate policies to exclude it by name first).devantler commentedon Oct 2, 2026
Decision on the
kyvernonamespace: keep Kyverno out, rely on Kubescape (option 2)This answers the open question in the last update. The other three namespaces are already covered by #4235, and their memory measurement is still due 2026-10-05.
Decision: Kyverno keeps filtering its own namespace. Coverage there comes from Kubescape, which is option 2 in the issue body.
Why:
failurePolicy: Failrule could then block Kyverno's own pods from rolling out. Kyverno's documentation recommends excluding its own namespace for this reason.kyverno(it does includekube-system,kube-publicandkube-node-lease).kyvernoby name, which is extra change for a namespace that already has a scanner.What stays visible: the
kyvernonamespace still holds 29 PolicyReports whose newest result is from 2026-06-17 (Deployments, ReplicaSets, a PDB and the HelmRelease). Asdocs/policy-reports.mdalready says, nothing can rewrite or prune them while Kyverno filters the namespace, so they show a past result, not current coverage.Remaining for this issue: the 2026-10-05 measurement of the reports controller's restarts and memory for #4234. If it stays within its limit, the acceptance criteria are met and this can close.
devantler commentedon Oct 3, 2026
Decision: retain the three Kubernetes system namespaces' restored Kyverno background coverage from #4235, and retain Kyverno's own namespace filter to protect it from self-targeting generate policies.
The admission review distinguishes webhook selectors from resource filters. Resource webhooks exclude kube-system and kyverno; kube-public can match Enforce RBAC validation, and its existing Role has a passing result. Node Leases remain filtered. Generate and mutate-existing rules exclude the three Kubernetes system namespaces. The documentation correction changes no policy action, selector, filter or RBAC exclusion.
Read-only verification on 2026-10-03 joined fresh kube-system policy results to current resources and confirmed the observed, Ready policy-engine release and resource webhook scope. This supplies the conditional current-report evidence. The decision deliberately does not claim complete four-namespace Kyverno coverage or a week of reports-controller memory stability.
Kubescape provides separate posture visibility for Kyverno workloads; stored results do not prove their freshness or equivalent admission enforcement. Its exclusions also mean it cannot replace the other system namespaces' restored Kyverno background coverage. The repository documentation correction will make this scope and limitation explicit.
chore: agent-claim #4228 nonce=ec54e4b6a58c7aab030e27afdc8f3e88