Repository navigation
Reconcile Kubescape vulnerability summaries with the containers they describe #4263
Description
🤖 Generated by the Agentic Engineer
Problem
The 2026-08-22 production recheck in #3156 found current, coherent vulnerability summaries, but they could not be matched to the workloads they describe. There were about 286 active containers and 138 summaries, and summary references resolved their payloads only in the scanner's own namespace. So no cluster-wide critical, high or reachable total can be stated, and no one can tell which workloads have no result at all.
Expected outcome
Every in-scope long-lived container has a current, paired vulnerability result, and cluster-wide totals can be derived from those results.
Acceptance criteria
- Each in-scope container image is reconciled to exactly one current summary and its paired manifest, or appears on a named list of missing results.
- Every missing result is either restored, or excluded with a narrow, evidence-backed reason recorded next to the exclusion.
- Cluster-wide critical, high and reachable totals are derived from the reconciled set.
- The check from the coverage-denominator sub-issue reports zero missing vulnerability results.
Part of #3156.
Metadata
Metadata
Assignees
Labels
No labels
Type
Fields
Priority
None yet
Effort
None yet
Projects
- Status
Activity
devantler commentedon Oct 1, 2026
Live baseline from the #4278 check, and a likely root cause for the stale results (2026-10-01 ~23:30Z, read-only)
Check result (
check-kubescape-result-coverage.shagainst prod, exit 1):The 13 stale vulnerability results were last written between 2026-08-09 and 2026-08-28, and none has been rewritten since — although the daily scan does run. The
kubevuln-schedulerCronJob fired and succeeded at 01:12Z today, and kubevuln re-scanned those workloads within minutes. What fails is the write:the object has been modified; please apply your changes to the latest version). Zero summary writes succeeded.So the summaries that look current are the ones first created within the last 7 days; every update of an existing summary is refused. That turns every long-running image stale after 7 days, which matches the observed spread of stale dates.
This is despite the storage configuration that already serializes this kind to a single worker (
kindQueues.vulnerabilitymanifestsummaries.workerCount: 1). That setting stops two storage workers from racing each other; it cannot help if kubevuln itself submits an update against a resourceVersion that is already out of date, which is what a 100% failure rate suggests. Next step for this issue: find where kubevuln's resourceVersion comes from (a cached read vs. a fresh get before update) on the pinned kubevuln and storage versions, and whether a retry-on-conflict path exists upstream, before deciding between an upstream fix, an image bump, or a local storage patch.The eight missing results and the runtime gaps are not explained by this and need their own look; the runtime half belongs to #4264 (kubevuln also logs many "container profile is partial (workload restart required)" warnings there).
devantler commentedon Oct 1, 2026
Narrowing the conflict, from kubevuln v0.3.430's released source (
repositories/apiserver.go,createOrUpdate): an update of an existing summary is already wrapped inretry.RetryOnConflict(retry.DefaultRetry, …), and every attempt re-reads the object first withGetOptions{ResourceVersion: "metadata"}— the storage server's metadata-only read, served from the object's separate metadata row. So a 100% failure rate means five consecutive fresh reads each returned a resourceVersion the following update rejected. That points at the storage side, not at kubevuln racing itself: the metadata-only read and the update's version check disagree about the object's current version.Hypothesis to test next: in the storage build this platform runs (the v0.0.297-based compatibility image with the SQLite contention changes), the metadata row's resourceVersion is not kept in step with the version the update path checks. A read-only way to test it without a cluster change: for one affected summary, compare the resourceVersion from a normal
getwith the one in the metadata-only response. If they differ, the fix belongs in the storage patch.devantler commentedon Oct 1, 2026
Hypothesis confirmed (read-only, 2026-10-01 ~23:45Z). For one of the summaries whose update fails on every attempt, the two reads disagree:
get(full object)get ?resourceVersion=metadata(metadata row)A
listreports 504 as well. kubevuln re-reads with the metadata-only form before each update, gets 504, submits an update the server checks against 505, and is refused; all five retries repeat the same stale read. So the root cause is in the storage server this platform runs: a write advanced the full object to 505 without updating its metadata row, and from then on no update of that object can succeed.That points at the storage compatibility image's write path (the autocommitted derived writes / source-row finalization changes on top of v0.0.297) or at v0.0.297 itself. Next step: reproduce with the image's own tests — a write that bumps the object but not its metadata row — and fix it there, plus a one-off repair for objects already split (any update path that rewrites both rows, or recreating the affected summaries).
devantler commentedon Oct 1, 2026
Mechanism and scope (read-only, ~23:55Z).
get, 31 of the 126 summaries are split — the stored object is ahead of its metadata row. Each of those can never be updated again, so it goes stale and stays stale.pkg/registry/file/storage.gosaveObject(upstream code, unchanged by the platform's compatibility patch as far as this path goes): it bumps the resourceVersion, writes the payload file first, then writes the metadata row to SQLite. If the SQLite write fails — exactly the busy/contended-writer failure the compatibility image was built to reduce — the function returns an error, but the payload already holds the new version. From then on the metadata-only read (and everylist) reports the old version, the update path checks against the payload's newer one, and every update conflicts.Fix, in two parts:
The same split can affect any kind this storage serves; the other kinds should be counted the same way before part 2 is scoped.
devantler commentedon Oct 1, 2026
Correction to the priority of the two fix parts. The last payload write on every one of the 31 split summaries falls between 2026-08-09 and 2026-08-29; none split in the 33 days since. That matches the window before the compatibility image's 60-second SQLite busy timeout, which removes the failing metadata write the split needs. So:
chore: agent-claim #4263 nonce=ae73d96bb6179a480d4299e2338bfdfa
devantler commentedon Oct 2, 2026
The normal-reader coverage check was rerun on 2026-10-02 and correctly exited 1. For 79 expected image identities it reports 57 current results, nine missing and thirteen stale. The independent listed-versus-stored version comparison still reports 31 split summaries across the observed summary set. This does not establish complete paired vulnerability coverage or a trustworthy cluster-wide severity total.
No summary was deleted and no storage write was performed in this round. A split-repair operation still needs reviewed write semantics, exact object preconditions, and post-rescan readback; the shipped detector is not that repair. Raw resource identities and the collector receipt remain private. Keep this issue open.
devantler commentedon Oct 2, 2026
Prepared direction for the existing split-result repair: preserve the intact stored summary and reconcile only its lagging metadata. The checked-in compatibility patch shows that the Delete path discards supplied preconditions, so a delete-and-rescan recipe cannot supply the required concurrency guarantee. Child #4363 corrects that operator guidance; it does not repair storage or close this coverage issue.
The proposed implementation is a bounded, summary-specific reconciliation in the Platform-owned compatibility image. It must read payloads without a cleanup-capable GET path; verify identity, creation time and a proven newer payload version; derive metadata through the existing typed projection; and compare-and-update the exact observed metadata row atomically. Preserve payload bytes, UID and scan timestamp. Refuse a replacement, concurrent metadata write, invalid version, missing/corrupt payload, unsafe path or unproven drift direction. Leave other kinds untouched and report incomplete repair explicitly.
Validation must reproduce a payload write followed by a failed metadata write, then prove agreement on both reads and a subsequent successful scanner update. Include concurrent-write refusal, identity mismatch, metadata-ahead, missing/corrupt data, interruption and idempotence controls. Rollout follows the existing signed publisher and a separate immutable image-pin PR, followed by ordinary-reader comparison and fresh scan/coverage readback. Metadata coherence alone cannot clear the missing/stale image set.
Source inspection and tests are still required before this design becomes an implementation. No scanner object was mutated or deleted.
devantler commentedon Oct 2, 2026
Fresh read-only coverage recheck on 2026-10-02 at approximately 15:30 UTC still reports 79 expected vulnerability results, 57 current, 9 missing and 13 stale, with 31 metadata/payload version splits. The collector exits 1, so this is a failed coverage check, not cluster-wide vulnerability clearance. Posture coverage is 100/100 in the same read; it does not establish vulnerability coverage.
PR #4367 corrects the unsafe routine-deletion guidance and preserves results pending a reviewed repair. It changes no detector or storage behavior and does not resolve these gaps. The non-destructive repair design recorded above still needs implementation, concurrency/recovery proof, successful scanner updates and a complete coverage readback before this issue can close.
chore: agent-claim #4263 nonce=e14741d825993978acd2bc10a85fa7d3
devantler commentedon Oct 4, 2026
Approved, read-only inspection of the deployed compatibility baseline now confirms the failure mechanism the repair must reproduce. The upstream
saveObjectincrements the object version, overwrites the payload file, and only then writes SQLite metadata. A metadata-write failure returns without restoring the previous payload. The update path also skips persistence when the proposed object equals the stored object, so a later unchanged result need not repair an existing split. This establishes a possible mechanism; it does not prove the historical cause of every stored split.The fresh coverage check still exits 1: 80 expected vulnerability results, 57 current, 10 missing and 13 stale, with 31 listed/stored version splits. Complete paired coverage and a cluster-wide severity total remain unproven.
The non-destructive repair design above remains the acceptance contract: reproduce the write failure, preserve payload bytes and identity, compare-and-update only the exact lagging metadata, refuse concurrent or ambiguous changes, and verify both reads plus a subsequent scanner update. Metadata coherence alone cannot resolve the missing or stale image results. No production result was changed or deleted.
Public source evidence: payload-before-metadata save path, unchanged-object fast path.