Kubescape scan records outlive deleted workloads, so the scan set is not an existence test

🤖 Generated by the Agentic Engineer

Why

Kubescape config-scan records are not pruned when the scanned object is deleted, so the scan set
retains entries for workloads that no longer exist. That makes presence in the scan set unusable as an
existence test — which is exactly how a reader naturally uses it.

Measured on the live cluster 2026-09-09 while verifying C-0013 for #3223: 7 tofu/tf-runner-named
scan records persist whose live counterparts are all absent. tofu-controller has no Deployment,
StatefulSet, DaemonSet or HelmRelease, and its ServiceAccounts, Role, RoleBinding and cache-encryption
Secret were each individually verified gone (its retirement landed via #3480). The scan records outlived
all of it.

Impact

This is a fail-open in the direction of "still there", and it lands on exactly the reasoning security
work depends on:

A sweep that answers "does workload X still exist?" from scan records concludes YES for a workload
that was deleted. On Six flux-system controllers fail C-0013 on a missing runAsGroup, blocking the C-0013 narrowing #3223 the whole blocker was "is tofu-controller gone?" — a scan-record-based
check would have answered wrongly and kept a Security issue parked. It stayed parked for 22 days on a
blocker whose dependency had in fact shipped.
It inflates finding counts against phantom workloads. Any posture or compliance number computed
over the scan set counts controls for objects that cannot be remediated, because there is nothing
left to remediate. Work aimed at driving those numbers to 100% then chases records rather than
workloads.
It compounds the existing trap in this surface. A LIST of workloadconfigurationscans already
returns .spec.controls as null, so only a single-object get populates controls; and a config
scan on this cluster was once dead for 53 days while every surface read healthy. This adds a third
way for a confident-looking read of the same surface to be wrong — and unlike the other two, it is
invisible to a liveness check, because the stale records sit inside an otherwise healthy scan.

⚠️ Note this is orthogonal to scan liveness. At the time of measurement the scan was LIVE — last
success 2026-09-08T09:12:36Z, three consecutive daily successes, objects still being written. A fresh
scan does not remove a dead record.

What

Reconcile the scan set against live objects so a record cannot outlive its subject. Options, cheapest
first — the point is that something must own deletion:

Prefer a scanner-side setting if one exists to prune or expire records for absent objects.
Otherwise a reconciling sweep that deletes scan records whose owning object no longer resolves,
keyed on the object reference the record already carries. Deleting a record is not a posture change,
so this is safe in a way that changing a control is not.
Failing both, stop treating the scan set as an object inventory: derive existence from the live API
and use scan records only for the verdicts of objects independently confirmed to exist. This is the
fallback, not the fix — it leaves the inflated counts in place.

Acceptance criteria

A scan record for a deleted workload does not survive reconciliation, proved by deleting a throwaway
workload and observing its record go away.
Posture and compliance counts are computed only over workloads that currently exist, so a deletion
reduces the denominator rather than leaving a permanently-unremediable finding.
Whatever mechanism is chosen, a negative control shows the sweep does not delete a record whose
object still exists.
The 7 known tofu/tf-runner orphans are gone.

Rough size: S–M.

chore: agent-claim #3697 nonce=890c104f09251d9c786bdd5cba1cbf84

fix(kubescape): class a record no list can confirm as unknown

🤖 Generated by the Agentic Engineer

Root cause found: the scanner's own cleanup deliberately skips every record that is not about a pod or a workload controller, so these records can never be removed by it. No setting changes that.

Re-measured on prod, 2026-10-05. The orphans are still there, four weeks on: 12 tofu / tf-runner records out of 2,741 scan records, the oldest created 2026-06-22. Every one of them describes a Role, RoleBinding, ServiceAccount or Secret.

Why they survive. The storage component runs a periodic cleanup. For scan records it decides by the kind named in the record's workload id, and it returns "keep" for any kind outside a fixed list:

Kind in the record	Cleanup behaviour
pod, deployment, statefulset, daemonset, replicaset, job, cronjob	removed when the object is gone
anything else (role, rolebinding, serviceaccount, secret, configmap, …)	always kept ("skipping unknown kind")

Read in the released source of kubescape/storage at v0.0.297, which is the version our storage image is built from (pkg/registry/file/cleanup.go, deleteByWlid and isCleanableWlidKind; the list is in discovery.go, marked FIXME put in a configmap). The newest release, v0.0.348, has the same code, so an upgrade does not fix it.

What this means for the options in this issue

"A scanner-side setting" does not exist. That option is closed.
The fix has to add deletion for the skipped kinds. Two places can own it:
1. Our storage image. The platform already builds this component with its own patch set, so the cleanup can be extended there to look up the skipped kinds and remove a record whose object is gone. This fixes the counts at the source and is the route I intend to take.
2. A small scheduled job in the cluster that deletes scan records whose object no longer exists. Simpler to reason about, but it is a second owner for the same data and needs delete rights on scan records.
Offering the same change to the upstream project is worth doing afterwards, and needs the maintainer's approval as a third-party contribution. It is not required for our fix.

Safety property to prove either way (the negative control in the acceptance criteria): a record is removed only after its object was looked up and positively found absent. A failed or partial lookup must keep everything.

Next step: extend the cleanup in the patched storage image, with a test for each skipped kind and for the failed-lookup case, then the throwaway-workload proof on a cluster.

chore: agent-claim #3697 nonce=0fe551d1c3b7da73162a98f33172bfff

🤖 Generated by the Agentic Engineer

The fix is open as a draft: #4493. It extends the cleanup in the platform's own build of the storage component, the first of the two routes in the previous comment.

How it decides. For a record about anything other than a pod or a workload controller, the cleanup now asks the cluster for that one object and removes the record only when the cluster answers that the named object does not exist. A denied request, a timeout, a server error, an object kind the cluster does not serve, or a record that does not identify exactly one object all keep the record.

Something the first design would have got wrong. Measured on production today (2,741 records): the name label on a record is cut off at 63 characters, while object names can be longer. Looking an object up by that label would have reported live objects as gone. The cleanup therefore trusts the labels only when the record's full workload id spells the same kind and name, which a cut-off label never does. Records that fail that test are kept.

What this covers, and what it does not

Records	After the fix
Roles, bindings, service accounts, config maps and similar	removed once the object is gone
The roughly 300 records that describe a subject, its role and the binding joining them	removed once the binding is gone
Secrets	kept for now, see below
Host and identity entries that are not cluster objects (node details, users, groups)	kept, as before

Secrets are deliberately left out of the first rollout. Checking whether a secret exists needs the same permission as reading its contents; the cluster has no narrower one. The storage component cannot read secrets today, and I do not think removing a few stale records justifies giving it that. So the known leftovers that describe a secret will remain after this fix, and the acceptance criterion "the known orphans are gone" will be met for every kind except secrets. A separate way to retire those (for example by age, once I have confirmed how often a live record is rewritten) is the follow-up I will file when the rollout is verified.

Delivery is two steps, because production pins the component by its signed digest

fix(kubescape): remove scan records whose object is positively gone #4493 builds, publishes and signs the new version. Nothing running changes.
A follow-up switches production to that version and grants the component permission to look up the object kinds in the table (read-only, one object at a time, no secrets). Then the proof from the acceptance criteria: delete a throwaway object and watch its record go, with a live object's record staying as the control.

The cleanup runs once a day, so the proof in step 2 needs up to a day, or a restart of the component to start a pass.

fix(kubescape): scope the object lookup to scan records and harden it

fix(kubescape): bind a binding lookup to its own namespace and stop r…

chore: agent-claim #3697 nonce=8e2ac1201bae28944528cf5a75c9bee7

fix(kubescape): leave ConfigMaps out of the lookup grant and list the…

fix(kubescape): approve the storage lookup grant in the authorization…

🤖 Generated by the Agentic Engineer

Live and verified on production. The cleanup removed 162 stale scan records in its first pass, and every one of them belonged to an object that no longer exists.

Delivered in #4493 (the fix, built and signed) and #4500 (production switched to it, with read-only permission to look objects up).

What happened on the cluster, 2026-10-05

	Before (03:56Z)	After the first pass (04:24Z)
Scan records	2,741	2,579
Records named after the retired tofu-controller	21	6

The pass took 77 seconds. Platform health read OK after the deploy.

Was anything removed that should not have been? No.

I looked up the object behind each of the 162 removed records on the live cluster: 162 absent, 0 still present, 0 lookups failed.
As the control, I looked up the objects behind 20 records that were kept, spread across roles, service accounts, bindings, volumes and endpoint slices: all 20 exist.
Three of the six remaining tofu-named records turned out not to be leftovers at all. A role and a role binding named tf-runner-leases-hotfix still exist in unifi, so their records were correctly kept. A name match is not an existence test, which is the point of this issue.

What was removed, by kind: 37 cluster role bindings (15 of them relation records), 17 volumes, 15 volume claims, 15 jobs, 14 endpoint slices, 13 storage capacity entries, 12 service accounts, 7 role bindings, 6 services, 5 disruption budgets, 5 nodes, 4 roles, 3 routes, 3 endpoints, 2 namespaces, 2 autoscalers, 1 ingress, 1 network policy.

Acceptance criteria

A record for a deleted object does not survive. Met, on 162 real deletions rather than one throwaway object. I did not create and delete a test object, because that is a write to production and the real deletions already show the same thing at larger scale.
A control shows a record whose object still exists is kept. Met, see above.
Counts cover only objects that exist. Met for every kind except Secrets and ConfigMaps.
The known tofu leftovers are gone. Met for all 15 that are not about a Secret. The three that remain describe Secrets.

What is left, and why it is a separate issue. The storage component is deliberately not allowed to ask about Secrets or ConfigMaps: the cluster cannot grant "does it exist" for those without granting "show me its contents". Their records (834 of 2,579 today) are therefore still kept after deletion. Retiring them needs a different mechanism, tracked in #4505.

One thing for anyone reading scan records later: the component runs this cleanup every six hours, so a deleted object's record can remain for up to that long.

Closing as delivered, with the Secret and ConfigMap remainder in #4505.

Navigation Menu

Repository navigation

Kubescape scan records outlive deleted workloads, so the scan set is not an existence test #3697

Why

Impact

What

Acceptance criteria

Metadata

Assignees

Labels

Type

Fields

Projects

Milestone

Relationships

Development

Kubescape scan records outlive deleted workloads, so the scan set is not an existence test #3697

Description

Why

Impact

What

Acceptance criteria

Activity

devantler commented on Oct 5, 2026

devantler commented on Oct 5, 2026

devantler commented on Oct 5, 2026

Metadata

Metadata

Assignees

Labels

Type

Fields

Projects

Milestone

Relationships

Development

Issue actions