fix(security): a Kubescape framework-load failure still reports zero findings instead of failing

Evidence

The in-cluster Kubescape configuration scan is aborting on every run. In the current pod's ~45h
log window there are 4 scan attempts and 0 successes, including both scheduled runs:

2026-08-02T09:12:30Z  scanning failed: open /home/nonroot/.kubescape/security.json: no such file or directory
2026-08-03T09:12:19Z  scanning failed: open /home/nonroot/.kubescape/security.json: no such file or directory

each preceded by a failed to list frameworks error. The newest stored scan object is from
2026-08-01, so posture data has been frozen for ~2 days. The CI posture gate is unaffected and
stays green, which is exactly the divergence that lets this go unnoticed.

Root cause

Read-only diagnosis, from the deployed manifest:

the policy-artifact cache (/home/nonroot/.kubescape) is backed by an emptyDir, so it is
wiped whenever the pod restarts
KS_OFFLINE=true then prevents re-downloading the framework artifacts
KS_DOWNLOAD_ARTIFACTS=true is set at the same time, which contradicts the offline flag
nothing seeds the cache — there is no init container, ConfigMap, or persistent volume behind it

The pod restarted ~45h ago. From that moment the cache was empty, no framework could be loaded, and
every scheduled scan has evaluated zero controls.

Why this is Security rather than a Chore

A broken scanner and a compliant cluster read identically. With no framework loaded, every
control reports zero findings — so the posture surface presents as clean while nothing is actually
being evaluated. Any consumer that trusts those numbers inherits a false all-clear, and a real
regression introduced in the last two days would be invisible.

This already has a concrete downstream victim: #2855 asks for "zeroed" controls to be graduated to
Kyverno Enforce. Under the current state every control looks zeroed, so a candidate list built
now would graduate controls that were never evaluated. That issue cannot safely proceed until this
is fixed.

Expected behaviour

The scheduled scan loads its framework and evaluates controls on every run, and a run that cannot
load its framework fails loudly rather than reporting zero findings.

Acceptance criteria

The framework artifacts survive a pod restart — seeded deliberately rather than left to an
emptyDir that offline mode cannot refill
KS_OFFLINE and KS_DOWNLOAD_ARTIFACTS express one coherent intent
A scheduled scan completes successfully, evidenced by a scan object newer than the fix
A framework-load failure surfaces as a failure, not as a zero-finding clean result — this
is the part that keeps the next occurrence from being silent, and it is worth more than the
immediate repair

Notes for whoever picks this up

⚠️ Do not derive control state from a LIST. get <kind> -A -o json returns objects with the
spec stripped: a workload that reads all-zero across 0 controls via LIST shows real per-severity
counts across 60 controls when fetched directly by name. A candidate list built from a LIST would
report every control as zeroed even with a perfectly healthy scanner. Fetch by name.

Also, spec.controls[].name is null in the stored objects — the cluster carries control IDs
only, so names have to come from the framework definition.

Rough size: S–M for the repair, and the fail-loudly criterion is the durable half.

🔗 Related PRs

#2316 - fix(kubescape): make ClusterSecurityException posture exceptions take effect [merged]
#2443 - fix(kubescape): keep scheduled posture scans local so results persist [merged]
#2452 - fix(kubescape): bump posture scanner to v4.0.10 so exception CRs apply offline [merged]
#2830 - feat(ci): publish Kubescape posture findings to Code Scanning [merged]
#2876 - feat(security): derive themed, deduped backlog entries from live Kubescape findings [merged]

📝 Issue Planner

_{Check the box below or use the @coderabbitai plan command to generate an implementation plan and prompts that you can use with your favorite coding assistant.}

Create Plan

🧪 Issue enrichment is currently in open beta.

You can configure auto-planning by selecting labels in the issue_enrichment configuration.

To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

issue_enrichment:
  auto_enrich:
    enabled: false

💬 Have feedback or questions? Drop into our discord!

Live re-verification, and a correction to the root cause that changes the fix

Confirmed still failing, and re-derived from the live cluster rather than inherited from this issue.

Still broken. The current pod has 0 restarts and started 2026-08-01T17:17:50Z. Since then: 4 scan attempts, 0 successes — two immediately after start, then both scheduled runs (2026-08-02T09:12:30Z, 2026-08-03T09:12:19Z). Confirmed live: KS_OFFLINE=true and KS_DOWNLOAD_ARTIFACTS=true are both set on the deployment, and the cache volume is kubescape-volume: {"emptyDir":{}}.

The correction

This issue says KS_OFFLINE=true "prevents re-downloading the framework artifacts". That is right about the outcome but understates it in a way that misdirects the fix: no download is ever attempted at all. The scan path in the logs is

Kubescape scanner initializing... → Initialized scanner → Loading policies... → open /home/nonroot/.kubescape/security.json: no such file or directory

There is no download log line, no retry, and no network error — the scanner goes straight to the local cache and fails.

That matters because the repo's own cilium-network-policy.yaml already allows github.com and release-assets.githubusercontent.com specifically for the regolibrary bundle, and its comment names this exact error string as the symptom of that egress being absent:

So the error is ambiguous between two very different causes — egress blocked, and fetch never attempted — and here it is the second. Anyone triaging this error against that comment would widen the network policy and see no change. Worth fixing that comment alongside the repair.

A second mechanism worth checking before choosing the fix

The mount is:

kubescape-volume (emptyDir{}) -> mountPath /home/nonroot/.kubescape  subPath config.json

With an empty emptyDir, the kubelet materializes that subPath as a directory (the upstream mount bug already described in helm-release.yaml). The consequence is that /home/nonroot/.kubescape is an empty directory that masks whatever the image ships at that path — so seeding is not only about surviving a restart, it is about not being hidden by this mount in the first place. The first two failures name allcontrols.json and the later two security.json, i.e. every framework the scanner asks for.

What this implies for the acceptance criteria

Criterion 2 ("KS_OFFLINE and KS_DOWNLOAD_ARTIFACTS express one coherent intent") is the load-bearing one, and the choice is now explicit:

Seed deliberately (init container / mounted artifact bundle) and keep the scanner offline — additive, leaves the documented offline-submit behaviour untouched.
Let the artifact fetch run while keeping the submit local. Note kubescapeScheduler.requestBody.scanV1.keepLocal: true is already what forces Submit=false — helm-release.yaml says so directly ("kubescapeOffline ALONE does NOT keep the posture SCAN local"). So kubescapeOffline may now be carrying little except the artifact-fetch suppression that breaks this. This needs verifying, not assuming — that flag also toggles the config.json mountPath and clusterData.keepLocal, and getting it wrong re-triggers the "is a directory" abort.

I claimed this issue to work it, but I am not shipping a speculative change to a live prod security control in the same run I diagnosed it: the deploy path is a merge queue that promotes to production, and neither option above can be verified pre-merge. I have unassigned it so it is immediately available rather than sitting behind a claim I am not acting on — the evidence above is the contribution, so the fix gets designed against the real mechanism.

⚠️ Note for whoever picks it up: scripts/kubescape-backlog-bridge is not scheduled anywhere yet — CI only runs go test against it — so it is not currently consuming this frozen data. The stored objects are stale, not empty, so its close-when-gone path is not silently closing live findings today. That changes once it is activated, which is worth sequencing after this fix rather than before.

Still failing, and confirmed independently today

Picked this up as the oldest actionable Security issue. Re-verified live rather than inheriting the
diagnosis — it reproduces, and the streak is now longer than the issue records:

2026-08-01T17:20:49Z  scanning failed: open .../allcontrols.json: no such file or directory
2026-08-02T09:12:30Z  scanning failed: open .../security.json:    no such file or directory
2026-08-03T09:12:19Z  scanning failed: open .../security.json:    no such file or directory
2026-08-04T09:12:17Z  scanning failed: open .../security.json:    no such file or directory
2026-08-05T09:12:18Z  scanning failed: open .../security.json:    no such file or directory

Every scheduled scan since the pod restart has aborted at policy load. The diagnosis in the issue body
holds — I am adding three things it does not yet have.

1. The trigger path is healthy, so nothing upstream of the scanner looks wrong

The scheduler CronJob completes successfully every run and its log ends response: ok — it POSTs the
kubescapeScan command and the operator accepts it. The scanner then logs Initialized scanner →
Loading policies... → scanning failed. So the failure is invisible to anything watching the
CronJob: the Job is Completed, not Failed. That is worth knowing for acceptance criterion 4 — a
loud failure has to come from the scan result, not from Job state.

2. It is not only stale numbers — new workloads have zero coverage

Sampling workloads created after the freeze, none has a WorkloadConfigurationScan object at all. One
public-product workload running since 2026-07-26 sits in a namespace with no scan objects whatsoever.
So the surface is not merely reporting old verdicts; anything deployed since the freeze has never been
evaluated once, and reads as clean because there is no record to be non-clean.

3. ⚠️ A measurement trap for whoever fixes this — and for #2946

The storage aggregated API strips spec.controls on LIST calls and only populates it on a
single-object GET. I hit this while verifying:

get workloadconfigurationscans -A -o json → spec.controls: null for all 1542 objects
get workloadconfigurationscan <name> -n <ns> -o json → fully populated

A list-based read therefore returns zero control entries for every object — which looks exactly like
"the scanner produced nothing", and would have had me confirm a broken scanner from an artifact of the
API rather than from the scanner. It also means any verification of criterion 3 ("a scan object newer
than the fix") must read a specific object, not a list, or it will read as still-broken after a
successful fix. Same caveat applies to #2946, which needs to read applied exceptions off these objects.

Related: a per-object sample (21 objects, 243 control entries) shows appliedIgnoreRules empty
throughout — consistent with the operator-side exception behaviour described in #2823/#2969.

Not claiming the fix

I am leaving this unassigned rather than holding a lease I cannot discharge this run. The remedy is
a real design call between seeding the artifacts, giving the cache persistence, or scoping egress for
the scanner, and it should not be rushed into the tail of a run. The evidence above is the part that
was missing.

Re-measured on the live cluster — the root cause above is stale, and the freeze is much older than recorded

Verified against admin@prod before picking this up. The issue is still real and still open, but
both its root-cause section and its freshness numbers are now wrong, so anyone working from them
would fix the wrong thing.

1. The original root cause is already fixed

KS_OFFLINE is no longer set on the scanner at all — capabilities.kubescapeOffline: disable
landed, and the HelmRelease now documents at length why. So the "offline flag prevents re-downloading
the wiped cache" mechanism described above no longer applies, and the
KS_OFFLINE/KS_DOWNLOAD_ARTIFACTS contradiction (acceptance criterion 2) is resolved.

2. The scan still aborts — but on a different error

The current scanner pod's full log window shows 4 scan attempts, 4 failures, 0 successes, all
with one reason:

failed to scan. reason: 'chmod /home/nonroot/.kubescape: operation not permitted'

Not the no such file or directory recorded above. operation not permitted is EPERM, not the
EROFS (read-only file system) that the HelmRelease comments already anticipate for the
Submit-mode path — so this is a third, currently-undocumented failure mode.

3. The freeze is 53 days, not ~2

The newest stored workloadconfigurationscan is 2026-06-27. This issue recorded 2026-08-01 when
it was filed, so posture has been frozen substantially longer than the "~2 days" above — and the
"newest object" figure has gone backwards since, which is itself unexplained.

4. Control — this is the configuration scan, not the storage layer

Ruling out the obvious confounder, since all three CRs are served by the same aggregated storage API:

CR	Newest object
`workloadconfigurationscans`	2026-06-27
`vulnerabilitymanifestsummaries`	2026-08-17
`applicationprofiles`	2026-08-19 (same day as this measurement)

The storage component is healthy and writing. Only the configuration scan is dead.

What I could NOT establish

Stated so the next reader does not take these as settled:

The 4 failures are not the scheduled scan. The scanner pod is younger than the last scheduler
run, so its window contains no scheduled scan. What triggers those 4 is unidentified —
continuousScan is disable, and keepLocal: true is present in the scheduler request-body
ConfigMap, so the documented Submit-mode explanation does not obviously cover them.
The ownership cause is inferred, not observed. /home/nonroot/.kubescape is an emptyDir
subPath mount and the container drops ALL capabilities as uid 65532, which would make chmod
on a root-owned directory fail exactly this way — but exec is not available under read-only
access, so the directory's actual ownership is unverified.

Consequence for acceptance criteria

Criteria 1 and 2 are met. Criterion 3 (a scan object newer than the fix) is not, which is why
this stays open. Criterion 4 — a framework-load failure surfaces as a failure rather than a
zero-finding clean result — is the half this measurement most vindicates: the scan has been dead for
53 days and nothing anywhere alerted. That criterion is being split out so it can ship
independently of the chmod repair, which needs evidence read-only access cannot get.

docs(security): note the posture surface is not currently producing

Still failing 2026-08-19 — the original cause is FIXED and the abort has MOVED

This is not the same failure any more. The offline/artifact cause this issue records is gone —
KS_OFFLINE is no longer set on the Deployment, and the scan now loads policies and evaluates
controls (Scan coverage score: 75% (227/243 controls evaluated), compliance 83). It still aborts,
for a new reason:

failed to generate account ID   reason: chmod /home/nonroot/.kubescape: operation not permitted
scanning failed                 error:  failed to scan. reason: 'chmod /home/nonroot/.kubescape: operation not permitted'

5 aborts in the retained 48h window (08-18 15:05, 15:40, 21:49, 22:24; 08-19 08:05).

Mechanism

/home/nonroot/.kubescape is the kubescape-volume emptyDir. The pod sets fsGroup: 65532 and
runAsUser: 65532, so the directory is group-owned by 65532 and group-writable — but it is owned by
root. chmod(2) requires ownership, not write permission, so the scanner's attempt to harden its
own state directory returns EPERM and the scan aborts before submitting.

Writability is therefore not the fix: the directory is already writable. Either the scanner must not
need to own it, or it must run as the owner.

Measured impact — posture is 53 days stale

Newest stored scan across 1542 workloadconfigurationscans: 2026-06-27T14:50:44Z.

creationTimestamp alone would be weak evidence, since an in-place patch would not move it and the
aggregated API populates neither managedFields nor comparable resourceVersions. So the decisive
measurement is coverage split at the cutoff, over every Deployment in the cluster:

Deployments	count	with a scan object
created before 2026-06-27	79	65
created after 2026-06-27	11	0

Nothing created in 53 days has ever been scanned — crossplane-sync-exporter, doggy-countdown,
descheduler, opencost-usage-scraper, open-feature-operator-controller-manager among them.

Two things worth separating

The CVE path is healthy. storage is serving and kubevuln is actively writing
vulnerabilitymanifestsummaries. Only the configuration/posture path is dead, so "kubescape is
running" is true and misleading at once.
The scheduler cannot show this. kubescape-scheduler ran 9h ago and is Completed; the abort
is in the scanner it triggers. Consistent with this issue's premise — and it is why a monitor must
key on data age, never on scheduler status or an error string, since the string has now changed
twice while the outcome held.

Note on the hardening interaction

readOnlyRootFilesystem: true, runAsNonRoot, drop: [ALL] and the uid/gid are the platform's own
security-context posture. This is a case where hardening silently disabled the thing that measures
hardening — the two-sided test in AGENTS.md (raise the floor without taxing the path) applies to the
fix: whatever restores the scan must not simply hand the container back ownership of a root-owned
directory if a narrower option exists. Related: #3023 (previous abort cause), #3239 (security-context
mutation scope).

Not fixed here — this run had no external review capacity left (all three lanes rate/usage
limited), so this is recorded rather than patched.

Where the Submit-mode scans come from — and what is still unexplained

The aborting scans are dispatched by the operator's risk-acceptance watch, not the scheduler.
Operator log, matching each abort:

dispatched cluster posture rescan for security exception change
action completed successfully  command=kubescapeScan  wlid=wlid://cluster-prod/namespace-

That string is not in this repository — it is upstream operator code, reached because
capabilities.riskAcceptance: enable makes it watch the SecurityException/ClusterSecurityException
CRs. Like the continuousScan path this repo already documents in helm-release.yaml, that request
carries no scanV1.keepLocal, so the scan runs Submit-mode, reaches cloud account-ID generation,
and dies on the chmod. Upstream's offline fix (helm-charts#862) only patched the scheduler
request body, which is why the scheduled scan is the one that survives.

Verified not-drift along the way: ks-capabilities live reads continuousScan: disable,
configurationScan: enable, kubescapeOffline: disable, matching the HelmRelease; the release is
Ready/UpgradeSucceeded at 1.40.3; and the live kubescape-scheduler ConfigMap does carry
{"scanV1":{"keepLocal":true}}. So neither config drift nor a missing scheduler flag explains this.

What is NOT yet explained — stated plainly rather than patched over

The scheduled scan completed cleanly today (09:12 dispatch → Scan results saved 09:48 →
done scanning 10:07, 227/243 controls, compliance 83) and still produced no posture writes:
storage logged zero workloadconfigurationscan activity in 24h, only
openvulnerabilityexchangecontainers. So stopping the Submit-mode aborts is necessary but, on this
evidence, not sufficient — a scan that does not abort still persists nothing.

Corrected coverage measurement, excluding the seven namespaces in KS_EXCLUDE_NAMESPACES (an earlier
65-of-79 figure was confounded by them):

Deployments in scannable namespaces	count	with a scan object
created before 2026-06-27	65	65
created after 2026-06-27	10	0

A clean 100%/0% split. Note what this does and does not establish: it proves no new objects are
created. It does not establish whether the 1542 existing objects are still being updated in
place — these CRs carry no scan timestamp, and creationTimestamp would not move under a patch. So
"53 days stale" is proven for new workloads and unproven for existing ones; that distinction
changes the severity and should be settled before anyone acts on the number.

Hypotheses tried and refuted

Recorded so the next run does not re-spend them: a check-ordering race (refuted — other PRs' aggregate
checks complete before their analyses and still pass), continuousScan drift (refuted — live config
matches), and a missing scheduler keepLocal (refuted — present in the live ConfigMap).

No PR yet, deliberately. Three hypotheses died on contact with evidence, and the remaining gap is
why a successful scan persists nothing. A speculative manifest change to the component that measures
cluster security — particularly one that would hand a container ownership of a root-owned directory,
or re-open the cloud-submit path — risks weakening posture while appearing to fix it. The next
concrete step is to settle the update-in-place question (capture kubescape.io/sync-checksum on a
sample, force a keepLocal scan, diff), which decides whether this is a total outage or new-workload-
only.

chore: agent-claim #2933 nonce=735db406450577e0357da5adec2ddb0b

The premise is stale: the config scan is running and producing findings

Re-measured against admin@prod today. Criteria 1–3 are satisfied by the current state; criterion
4 — the silent-failure guard — is not, and it is the part worth keeping. Narrowing this issue to it
rather than closing, because the original failure mode can recur and would still be invisible.

Recording this in detail because the stale premise is expensive: the next run to pick this up would
have spent a cycle fixing an emptyDir/KS_OFFLINE root cause that is no longer present.

What was measured

Check	Result
`kubescape-scheduler` CronJob	not suspended, schedule `12 9 * * `, last run 2026-08-20T09:12:00Z*
Today's scheduler Job	`succeeded=1`, `failed=none`, completed 09:12:30Z, trigger returned `response: ok`
`workloadconfigurationscansummaries` created today	24, and fresh objects every day since 2026-08-11 (2358 total)
Newest summary, read by single-object GET	`{"critical":0,"high":0,"low":0,"medium":3,"unknown":0}` — real findings, not a zero-shell
`scanning failed` / `security.json` / `failed to list frameworks` in current logs	absent
`kubescape` pod	Ready, 0 restarts since 2026-08-18T14:26Z

The severities were read with a single-object GET on purpose: a LIST strips .spec.controls, so
a list-based check reports empty content for every object and would have "confirmed" the outage that
is no longer happening.

The stated root cause does not match the live deployment

The diagnosis rested on KS_OFFLINE=true contradicting KS_DOWNLOAD_ARTIFACTS=true. Live, the
deployment sets KS_DOWNLOAD_ARTIFACTS=true and no KS_OFFLINE at all — the contradiction is
gone, which is consistent with the scan working. KS_OFFLINE appears nowhere in the repo except one
explanatory comment in helm-release.yaml.

So criterion 2 is satisfied, and criterion 1 is moot in its stated form: whatever the artifacts are
backed by, they are surviving well enough that scans have completed daily for over a week.

What remains — criterion 4, unchanged and still right

Nothing in the recovery added that guard. A broken scanner and a compliant cluster still read
identically, which is the property that let the original outage run unnoticed, and the reason this
was filed as Security rather than a Chore. The recovery removed the symptom, not the blind spot.

The guard should key on data age, not on an error string: the failure modes seen here produced
different messages (and the current one produces none at all), whereas "no fresh scan result" is true
under every one of them.

Separate anomaly found while measuring — filed as its own issue

workloadconfigurationscans (the detail objects, distinct from the summaries above) number 1542
and are all dated 2026-06-17 → 2026-06-27 — newest 2026-06-27, i.e. 54 days stale — while
summaries flow daily. Anything reading control-level detail is reading two-month-old data. That is a
different surface with a different failure, so it is filed separately rather than folded in here.

Disposition

Retitled to what is actually left. Original evidence preserved above; criteria 1–3 are met by the
current state, criterion 4 stands.

Closing — both children are now closed and this issue's own criteria are met, verified live against
admin@prod on 2026-08-20T22:0xZ.

security(kubescape): the offline capability blocks the scanner's policy-artifact fetch, so every scan aborts #3023 (the offline capability starving the policy-artifact fetch) — fixed in 34a2a07e
(fix(kubescape): let the posture scanner fetch its policy artifacts #3025) and verified this run: KS_OFFLINE is unset on the deployed scanner, and its most recent
scheduled run evaluated 227 of 243 controls with an overall compliance score of 83. Full
evidence table on that issue.
security(kubescape): a framework-load failure must fail loudly, not read as a zero-finding clean result #3024 (a framework-load failure must fail loudly rather than read as zero findings) — delivered
as the posture-staleness alert in feat(observability): alert when Kubescape posture data goes stale #3237. It is deployed and demonstrably working: it currently
reports the residual detail-surface staleness every 6h, which is exactly the silent-zero case this
issue said was worth more than the immediate repair.

The scan data criterion is satisfied on workloadconfigurationscansummaries, whose newest object is
2026-08-20T10:07:15Z and which moves daily.

Not closed by this: the control-level detail surface (workloadconfigurationscans) remains
frozen at 2026-06-27. That is a separate cause tracked in #3275, where this run recorded the
diagnosis — a working scanner with one surface stopping, not the abort described here. #3275 itself
records that this issue is resolved.

fix(security): a Kubescape framework-load failure still reports zero findings instead of failing #2933

Description

Evidence

Root cause

Why this is Security rather than a Chore

Expected behaviour

Acceptance criteria

Notes for whoever picks this up

Activity

coderabbitai commented on Aug 3, 2026

devantler commented on Aug 3, 2026

Live re-verification, and a correction to the root cause that changes the fix

The correction

A second mechanism worth checking before choosing the fix

What this implies for the acceptance criteria

devantler commented on Aug 5, 2026

Still failing, and confirmed independently today

1. The trigger path is healthy, so nothing upstream of the scanner looks wrong

2. It is not only stale numbers — new workloads have zero coverage

3. ⚠️ A measurement trap for whoever fixes this — and for #2946

Not claiming the fix

devantler commented on Aug 19, 2026

Re-measured on the live cluster — the root cause above is stale, and the freeze is much older than recorded

1. The original root cause is already fixed

2. The scan still aborts — but on a different error

3. The freeze is 53 days, not ~2

4. Control — this is the configuration scan, not the storage layer

What I could NOT establish

Consequence for acceptance criteria

devantler commented on Aug 19, 2026

Still failing 2026-08-19 — the original cause is FIXED and the abort has MOVED

Mechanism

Measured impact — posture is 53 days stale

Two things worth separating

Note on the hardening interaction

devantler commented on Aug 19, 2026

Where the Submit-mode scans come from — and what is still unexplained

What is NOT yet explained — stated plainly rather than patched over

Hypotheses tried and refuted

devantler commented on Aug 20, 2026

The premise is stale: the config scan is running and producing findings

What was measured

The stated root cause does not match the live deployment

What remains — criterion 4, unchanged and still right

Separate anomaly found while measuring — filed as its own issue

Disposition

devantler commented on Aug 20, 2026

Metadata

Metadata

Assignees

Labels

Type

Fields

Projects

Milestone

Relationships

Development

Issue actions