Complete Kubescape runtime profile pairs and prove runtime alert delivery #4264

Activity

1. github-project-automation

   added this to 🌊 Project Boardon Sep 30, 2026
2. devantler

   added the

   Bug

   issue type on Sep 30, 2026
3. devantler

   added a parent issue on Sep 30, 2026

   • Restore complete Kubescape result coverage #3156
4. devantler

   moved this to 📥 Backlog in 🌊 Project Boardon Sep 30, 2026
5. devantler

   marked this as blocked on Sep 30, 2026

   • Define the expected Kubescape result set per surface and fail on missing or stale results #4262
6. devantler commented on Oct 2, 2026

   

   devantler

   on Oct 2, 2026

   ContributorAuthor

   More actions

   🤖 Generated by the Agentic Engineer

   The normal-reader coverage check was rerun on 2026-10-02 and correctly exited 1. It reports 80 in-scope workload revisions: 71 have complete runtime pairs, four fail (one missing pair and three partial pairs), and five remain within their learning period. The latter accounts for ten pending profile objects; those are not silently counted as complete or as expired failures.

   Posture-result presence is complete for the observed workload set (100/100), but that does not prove runtime coverage, result freshness, or alert delivery. No deliberate runtime alert has been triggered or observed end to end in this round. Detailed workload identities and the collector receipt remain private. Keep this issue open for the missing/partial-pair repair and durable alert-delivery proof.
7. devantler

   self-assigned this

   on Oct 4, 2026
8. devantler commented on Oct 4, 2026

   

   devantler

   on Oct 4, 2026

   ContributorAuthor

   More actions

   🤖 Generated by the Agentic Engineer

   Root cause of the remaining runtime gaps (read-only check, 2026-10-04 17:30Z)

   The coverage check was re-run against production and exited 1: 82 expected workload revisions, 74 complete, 5 failing (one missing pair, four partial pairs), 3 still inside their learning period. The five failures have two different causes, and neither is a fault in the scanner. Workload identities stay in the private notes.

   1. Partial pairs (4 revisions): the pods were replaced before the 24-hour learning period ended.

   • The runtime agent's own log shows it for one of them: it stopped recording with the reason "container has terminated" about 20 minutes after the revision was created. The profile is then stored as finished but partial.
   • Three of the four belong to one application that is redeployed many times a day. Of its 22 stored revisions, 20 are partial; the only two complete ones are from the one revision that ran longer than 24 hours.
   • The fourth is a per-node agent whose revision was 10 hours old when every node was restarted for an upgrade on 2026-09-23. It has run uninterrupted for 11 days since and is still marked partial, so a later full learning period does not appear to upgrade a partial profile.
   • Nothing is wrong with the agents themselves: all four have run without a restart since 2026-09-23.

   2. Missing pair (1 revision): the runtime agent does not run on the nodes where this workload runs.

   • The agent is scheduled on the four worker nodes only. This workload runs only on the other node group, so it can never get a profile. A workload that runs on both groups gets its profile from its worker pods, which is why only this one shows up.
   • Running the agent on those nodes would cost roughly 370–430 MiB each (measured on the workers today; its limit is 1 GiB). Those nodes are at 60–69% memory already, so this is a real trade-off and not a free fix.

   What this means for the acceptance criteria

   • Criterion 1 ("if the cause is profile learning time rather than a fault, the in-scope window is defined"): the cause is learning time. The check currently cannot tell "cut short by a redeploy" from "the agent started watching late", because both are stored as partial.
   • Next measurement: 2026-10-05 after 02:04Z. The frequently redeployed application's current pods started at 2026-10-04 02:04Z in an already-partial revision. If that revision is complete after 24 hours, a full period does repair a partial profile and the per-node agent is a separate boot-order case. If it is still partial, a partial profile is permanent for that revision and the check needs a rule for it.
   • After that reading, the change is in scripts/check-kubescape-result-coverage.sh: report a revision whose pods have not yet lived a full learning period, and a workload that runs only where the agent does not, as their own named classes with the evidence, instead of as scanner failures. Whether to run the agent on the other node group instead is a separate decision with the memory cost above.

   No alert-delivery test was run in this round.
9. devantler commented on Oct 5, 2026

   

   devantler

   on Oct 5, 2026

   ContributorAuthor

   More actions

   🤖 Generated by the Agentic Engineer

   The follow-up observation was completed on 2026-10-05 after 02:04 UTC. The per-node agent revision remains completed/partial in both runtime profiles; its current pods have run since 2026-09-23 without replacement. A later uninterrupted period has therefore not repaired this particular revision.

   The frequently redeployed application changed revision before the planned observation. Its current pod started at 00:31 UTC today, and its current profile pair is still learning. The earlier revision is no longer present in the current profile collection, so this reading cannot answer whether that same revision would have upgraded after 24 hours. The previous proposed binary conclusion is unsupported; do not waive partial profiles or infer permanence for every revision from this sample.

   The normal-reader check earlier this round reports 82 expected runtime revisions: 74 complete, one missing pair, one partial pair and six still learning. Result presence, completion and the alert-delivery test remain separate acceptance gates. The missing pair still belongs to a workload outside the runtime agents’ scheduled node set. No runtime alert was deliberately triggered, and no security coverage policy was relaxed. Keep this issue open for a repair that handles interruption and late observation explicitly, plus end-to-end alert delivery.
10. devantler

    moved this from 📥 Backlog to 🧊 Icebox in 🌊 Project Boardon Oct 8, 2026