Repository navigation
fix(kyverno): background reporting has been frozen cluster-wide since the 3.9.0 upgrade #3760
Description
🤖 Generated by the Agentic Engineer
Evidence
Read-only production read-back on 2026-09-13, around 05:55–06:05Z:
- No Kyverno report has been refreshed since 2026-08-27. None of the 499 PolicyReports that carry results has a result newer than 7 days. The newest result across all PolicyReports is
2026-08-27T20:11:42Z, and across ClusterPolicyReports it is2026-08-27T20:13:28Z. Background scan is configured to run hourly. - The stall is not limited to the failures Prune stale Kyverno results after policies stop matching #2573 and Refresh stale Kyverno reports after policy exemptions #3155 describe. Whole reports are frozen, including their unrelated passing results. For example, the report for the
policy-reporterDeployment holds nothing newer than 2026-08-27, and the cluster-autoscaler report nothing newer than 2026-06-17. - The Kyverno chart moved to 3.9.0 (reports controller
v1.19.0) on 2026-08-27 (chore(deps): update helm release kyverno to v3.9.0 #3402). The leading reports-controller replica restarted at2026-08-27T19:28:32Zand has held the leader lease since then. Reports stopped roughly one scan pass after that restart. - In the last hour the leader's only errors were the resource report controller failing to list, and therefore to start watchers for, two kinds that background-scanned policies match:
httproutes.gateway.networking.k8s.io(84 occurrences) andciliumnetworkpolicies.cilium.io(42), bothforbiddenfor the reports-controller service account. Policies matching those kinds includerestrict-homepage-service-groups(HTTPRoute) andadd-default-deny(CiliumNetworkPolicy). - The reports controller's aggregated ClusterRole has read grants for KEDA, RBAC and Node kinds (
kyverno:reports-controller:read-*), but none for Gateway API or Cilium kinds. The Kyverno docs state that creating reports for a resource requires the reports controller to haveget,listandwatchon it.cluster-role-reports-controller-read-keda.yamlalready records the same failure shape for KEDA kinds.
Audience and impact
Everyone reading Kyverno compliance, on the Policy Reporter dashboard or via the report objects, has been reading 17-day-old data. A broken scanner reads exactly like a compliant cluster: a regression introduced since 2026-08-27 cannot show up, and fixed findings never clear. #2573 and #3155 are symptoms of this.
Expected behaviour
Background scans refresh every report within the configured interval, and the reports controller runs without permission errors for any kind a background policy matches.
Acceptance criteria
- The reports controller has
get,list,watchon HTTPRoute and CiliumNetworkPolicy, granted with the existingrbac.kyverno.io/aggregate-to-reports-controllerpattern, and its log shows noforbiddenerrors after rollout. - A live read-back after deploy shows report results newer than the rollout, across namespaces rather than a single report.
- If reports are still frozen once the grant is live, the remaining cause is diagnosed and recorded here before this issue closes, rather than assumed fixed by the grant.
- Prune stale Kyverno results after policies stop matching #2573's six stale failures are re-checked against refreshed reports.
Metadata
Metadata
Assignees
Labels
No labels
Type
Fields
Priority
None yet
Effort
None yet
Projects
- Status
Activity
chore: agent-claim #3760 nonce=a64b706b8176d3087f10bafb13db39fc
devantler commentedon Sep 13, 2026
Partial live evidence (not yet acceptance). The #3762 merge-group deploy briefly rolled the grant out before it was evicted for an unrelated handoff-fence failure (details on #3762). Read-only production reads:
kyverno:reports-controller:read-network-routeswas created at 06:42:09Z and its HTTPRoute rule appeared in the aggregatedkyverno:reports-controllerClusterRole.forbiddenerror (httproutes.gateway.networking.k8s.io) was at 06:40:51Z. It logged nothing at all afterwards; at 06:47Z its newest log line was still that one. Before the grant it logged about two such errors a minute.The heal job is redeploying
main, which will likely remove the role again. The acceptance criteria stay open until the grant is merged and a later read-back shows refreshed reports.devantler commentedon Sep 13, 2026
Control result: the permission errors track the grant. After #3762's evicted deploy,
🩹 Heal Prodredeployedmain(without the grant). Read-only production reads at 06:57Z:kyverno:reports-controller:read-network-routesforbiddenerrors from the leading reports controllerSo the grant is what stops this error class. Whether it also unfreezes the reports is still open: that needs the grant to stay live for at least one hourly background scan. #3762 is re-queued for that.
devantler commentedon Sep 13, 2026
Grant merged and deployed; the refresh criterion is still to observe. #3762 merged at 07:14:59Z at its reviewed head
3d56b1e0(merge commit29252078). Its second merge-group run passed🚀 Deploy to Prod(07:14:27Z).Read-only production reads at 07:16Z:
kyverno:reports-controller:read-network-routeshas been live since 07:11:51Z, and the aggregatedkyverno:reports-controllerrole again contains the HTTPRoute rule.forbiddenerror was at 07:07:20Z, before the grant landed. Without the grant (after heal), it logged 360 such errors in 20 minutes.Before closing: after the next background scan (at or after 08:12Z), a read-back must show PolicyReport results newer than 07:11:51Z across more than one namespace, no new
forbiddenerrors, and #2573's six stale failures re-checked. If reports are still frozen then, the remaining cause gets diagnosed here. The other lead is the reports-controller restart on 2026-08-27 at 19:28Z.devantler commentedon Sep 13, 2026
The grant fixed the permission errors; the reports controller is still not producing reports (criterion 3)
Read-only production read-back, 2026-09-13 08:04–08:50Z. The grant has been live since 07:11:51Z.
Criterion 1 holds. The leading reports controller (
…-lktth) has logged zeroforbiddenlines since the grant. Its last log line of any kind is at 07:07:20Z, and it has logged nothing at all in the 90+ minutes since. The log read itself works: the pre-grant lines come back normally.Criterion 2 does not hold, and the stall is wider than stale timestamps.
2026-08-27T20:11:42ZSo the controller is not just skipping hourly re-scans of old resources: it is not reporting on new resources either, even though the admission controller is admitting them normally. Background scan stays configured (
--backgroundScan=true,--backgroundScanInterval=1h,--admissionReports=true,--aggregateReports=true,--policyReports=true). The leader's previous container stopped cleanly (exit 0) at 2026-08-27T19:28:31Z. The current one has run since then without restarting, which is the whole 17-day window.Conclusion: after the grant, the leader is up and silent but idle. Its report pipeline stopped during the long permission-error period and has not resumed now that the errors are gone. The missing permissions explain how it stopped; they do not explain why it has not recovered.
Next discriminating step. Restart the reports controller once, then repeat this read-back. If new pods get reports within minutes and results refresh within an hour, the cause is a wedged controller and a restart after RBAC changes becomes part of the procedure. If it stays idle after a clean start, the cause is in v1.19.0 itself. Chart 3.9.1 (reports controller v1.19.1) is already published and no update PR exists yet, but bumping it restarts the pods too, so it would not tell the two causes apart. Restart first, then bump.
Not closing: criteria 2 and 4 remain open.
chore: agent-claim #3760 nonce=bda50c826e2f11c0aae6f67569795c4e
fix(kyverno): re-approve the EKS authorization surface for the restar…
chore: agent-claim #3760 nonce=0761d425f6f946abb5e9ca3cbaec3778
devantler commentedon Oct 2, 2026
All acceptance criteria are now met; closing the background-reporting incident.
The read grants are present in the effective deployed role. Complete reads of both current controller containers for the last two hours succeeded with zero permission errors; pod identities and restart counts were unchanged across the readback.
Reporting is active across the platform. The complete collections contain 1,229 reports and 3,931 results. Of these, 907 reports contain fresh results across 39 namespaces plus cluster scope. All 311 current Pods created after the grant, outside the deliberately filtered namespace, have fresh reports joined to their resource UIDs. Newly created controller workloads are also covered. Replica-floor results are 42 pass, 49 skip and zero failures; the stale KEDA and autoscaler findings have been rechecked as current skips.
The remaining recovery diagnosis is now recorded: the grant alone left the process idle, as the earlier readback established. #3779 rolled the controller through a declarative restart annotation while retaining chart 3.9.0. Its protected deployment ran on September 13 at 20:37–20:43 UTC, and retained reports across 31 namespaces began being created at 20:42. That recovery precedes the later chart upgrade by several days. The evidence supports an operationally wedged process recovered by the recorded rollout; it does not identify an internal watcher or goroutine defect.
Verified on 2026-10-02 with normal reader access. Admission-only historical results and deliberately filtered resources are not presented as fresh background scans. The current two-hour log read does not assert error absence over every historical hour.
Recovery delivery: #3779
Protected deployment: https://github.com/devantler-tech/platform/actions/runs/34781052141/job/103788791104