Refresh stale Kyverno reports after policy exemptions #3155

Activity

1. devantler

   added the

   Bug

   issue type on Aug 15, 2026
2. github-project-automation

   added this to 🌊 Project Boardon Aug 15, 2026
3. coderabbitai commented on Aug 15, 2026

   

   coderabbitaibot

   on Aug 15, 2026 – with coderabbitai

   Contributor

   More actions

   ⚠️ Possible Duplicate Issue(s)

   • Prune stale Kyverno results after policies stop matching #2573

   🔗 Related PRs

   #2985 - fix(k8s): gate deploy health on a serving CNPG cluster instead of a fully-ready one [merged]
   #3037 - fix(cilium): refuse a zero-count gate release and correct the runbook order [merged]
   #3109 - fix(deploy): avoid redundant GHCR node reboots [merged]

   ---

   📝 Issue Planner

   _{Check the box below or use the @coderabbitai plan command to generate an implementation plan and prompts that you can use with your favorite coding assistant.}

   • Create Plan

   ---

   🧪 Issue enrichment is currently in open beta.

   You can configure auto-planning by selecting labels in the issue_enrichment configuration.

   To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

   issue_enrichment:
     auto_enrich:
       enabled: false

   💬 Have feedback or questions? Drop into our discord!
4. devantler

   added a parent issue on Aug 15, 2026

   • roadmap: Kubescape security stack → 100% and hold (posture · CVE · runtime) #2447
5. devantler

   moved this to 📥 Backlog in 🌊 Project Boardon Aug 15, 2026
6. devantler

   added a commit that references this issue on Aug 22, 2026

   chore: agent-claim #3155 nonce=d6defbd0da2e25976969b9c0cd212818

   7809f69
7. devantler commented on Aug 22, 2026

   

   devantler

   on Aug 22, 2026

   ContributorAuthor

   More actions

   🤖 Generated by the Agentic Engineer

   Production recheck — 2026-08-22

   The read-only production aggregate still reports 1 fail, 0 warn across 685 policy reports. The sole failure remains validate-replica-floor/require-replica-floor; there are no non-ready nodes, Deployments, StatefulSets, or DaemonSets in the accompanying health snapshot.

   This confirms the hard-zero signal is still blocked by the stale result described here, rather than by a current workload availability regression. No report was deleted or otherwise mutated. The issue remains actionable with its existing declarative-refresh acceptance criteria.
8. devantler

   self-assigned this

   on Sep 23, 2026
9. github-project-automation

   moved this from 📥 Backlog to 👀 In Review in 🌊 Project Boardon Sep 23, 2026
10. devantler commented on Sep 27, 2026

    

    devantler

    on Sep 27, 2026

    ContributorAuthor

    More actions

    🤖 Generated by the Agentic Engineer

    Read-only production recheck, 2026-09-27. The stale result is still there, and the cause is narrower than the exemption.

    • The exemption works. I exported the live validate-replica-floor ClusterPolicy (now generation 19) and the live cluster-autoscaler-hetzner-cluster-autoscaler Deployment, then ran kyverno apply 1.19.1 (the cluster's version) on them locally. The result was skip: 1, with no fail. So the policy no longer fails this workload.
    • The report has not been rewritten since June. Both of its results are from June: require-replica-floor fail at 2026-06-16T15:20Z and cap-workload-replicas pass at 2026-06-17T23:35Z. The policy has changed generation several times since then, and background scanning is on (--backgroundScan=true, --backgroundScanInterval=1h, 2 workers). A background scan would rewrite at least the timestamps, so this resource is not being re-scanned at all.
    • It is not only this report. Grouped by each report's newest result: 538 reports are under a day old, 75 are 1–7 days old, and 206 are more than 30 days old. Some of those may just hold unchanged results, but this one provably should have changed, so a stale aggregate is a real possibility beyond this single workload.

    The platform-wide failure count is still 1: this result, and nothing current. The next step is to find out why the reports controller skips re-scanning this resource. Candidates are its per-resource policy-hash bookkeeping (ephemeral reports) and the controller's resource filters. Relevant to #3156, whose coverage gap may share this cause. Nothing was mutated or deleted.
11. devantler

    added a commit that references this issue on Sep 27, 2026

    chore: agent-claim #3155 nonce=7802c84e01d58df46dba437db506c102

    16b2d21
12. devantler commented on Sep 27, 2026

    

    devantler

    on Sep 27, 2026

    ContributorAuthor

    More actions

    🤖 Generated by the Agentic Engineer

    Root cause found: nothing scans kube-system at all, so this report can never be rewritten.

    • The reports controller runs with --skipResourceFilters=false (fix(kyverno): curb reports-controller memory growth via resource filters #2134, merged 2026-06-17 23:12Z), so background scans honour Kyverno's resource filters, and the chart's defaults include [*/*,kube-system,*] (plus kube-public, kube-node-lease and kyverno). Those filters also apply at admission.
    • Read-only on prod: all 35 kube-system reports and all 29 kyverno-namespace reports have their newest result at 2026-06-17 23:35Z, the last scan before fix(kyverno): curb reports-controller memory growth via resource filters #2134 rolled out. 56 kube-system pods and one Deployment created since then have no report at all.
    • So the exemption was never the problem (it evaluates to skip locally, as noted above), and no scan interval will fix it.

    #4227 adds a declarative prune for reports in those four namespaces once nothing has written to them for a day (a report that is written again is never touched), corrects the docs that said those reports are written at admission, and extends the kind proof to check that a kube-system resource gets no report before relying on it. Simulated against live reports: it removes exactly those 64 reports, including the platform's one failing result, and nothing else.

    Restoring Kyverno coverage of the system namespaces is a separate decision, filed as #4228.
13. devantler

    added a sub-issue on Sep 27, 2026

    • docs(kyverno): policy-reports.md says system-namespace reports are written at admission; nothing writes them #4229
14. devantler commented on Sep 27, 2026

    

    devantler

    on Sep 27, 2026

    ContributorAuthor

    More actions

    🤖 Generated by the Agentic Engineer

    Correction to my comment above: the prune does not work, and #4227 no longer ships it.

    The kind proof's positive control settled it. With the DeletingPolicy's namespace list extended to the test namespace, a day-old report there was deleted on the next run, while the identical report in kube-system survived five minutes of per-minute runs. Kyverno's cleanup controller skips the same resource-filtered namespaces as admission and background scans, so no Kyverno policy can clear these reports.

    What remains true: the 64 reports in those namespaces are June snapshots, and this issue's failing result is one of them. Manual deletion is not a recovery mechanism here, so the result clears when Kyverno covers the namespace again, which is #4228's decision. #4227 now corrects the docs and adds the proof that a kube-system resource gets no report (#4229).
15. devantler

    added

    blocked

    on Sep 27, 2026
16. devantler

    marked this as blocked on Sep 27, 2026

    • security(kyverno): system namespaces have had no Kyverno policy coverage since June #4228
17. devantler

    assigned

    devantler

    and unassigned

    devantler

    on Oct 2, 2026
18. devantler

    added 2 commits that reference this issue on Oct 2, 2026

    chore: agent-claim #3155 nonce=1c021b569868c39af03d013d132020b2

    7a2cd3a

    chore: agent-claim #3155 nonce=06811f6b33baa751025f9765c8a55f78

    63f9fc4
19. devantler

    closed this as completedin #4373on Oct 2, 2026