Repository navigation
fix(continuousscanning): resume watches from the last version and stop on error events - #487
Conversation
…p on error events SelfHealingWatch reconnected with an empty resourceVersion, so after every watch timeout the API server replayed every object as a synthetic ADDED event and continuous scanning triggered a scan for each of them again. A watch error event was also forwarded, and its *metav1.Status object then failed the e.Object.(metav1.Object) assertion in the service and panicked. Keep the version of the last object received, forget it when the server says it has expired, and end the watch on an error event instead of forwarding it. This reuses the helpers the watcher package got in kubescape#439, exported as ResumeFrom and DropGoneVersion. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: itsvishalyadav <vishalyadav87695@gmail.com>
|
Warning Review limit reachedYou've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository. Next included review available in 59 minutes. View limit details
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. |
blacksoxx
left a comment
There was a problem hiding this comment.
No blocking finding from a read of 3759d81 against main 47e9ae6. ResumeFrom keeps the version on a non-410 error event and forgets it on Expired or Gone, and Run drops it when the watch request itself comes back 410, so the three test cases match what the helper does. I had the same panic on my list; stopping the watch on the error event fixes it where it starts, and the resume removes the replay, which a guard in the service alone would not have.
Two P3s, your call:
- inline, on the resume: bookmarks.
service.go:52and:68still asserte.Object.(metav1.Object)without the ok form. Error events no longer reach them, so the panic path is closed; a comma-ok there would keep it closed if another event source is ever plugged into that channel.
Not run: the PR's tests, only read; the helpers on main and the fixtures were checked in code.
| return nil | ||
| } | ||
| // Without a version the next watch replays every object as ADDED. | ||
| watcher.ResumeFrom(&w.opts, event) |
There was a problem hiding this comment.
[P3] The watch is created with ListOptions{} and no AllowWatchBookmarks. On a quiet GVR the resumed version can age past the server's window between reconnects, which turns the next reconnect into a 410, DropGoneVersion and a full replay anyway. AllowWatchBookmarks: true in listen keeps the version fresh, and ResumeFrom already records Bookmark events.
matthyx
left a comment
There was a problem hiding this comment.
Approve at 3759d81 (base main 47e9ae6).
- Need: #486 is real in the code.
SelfHealingWatchreconnects with an emptyresourceVersion, so each reconnect replays every object as ADDED and rescans it. Forwardedwatch.Errorevents carry a*metav1.Statusand would panic at themetav1.Objectassertion inservice.go. - Fix: it reuses the helpers from #439, so
ResumeFromrecords the version andDropGoneVersionhandles a 410 on watch creation. An error event stops the watch without being forwarded, and the version is kept unless the server says it expired. Thewatcherpackage has no dependency oncontinuousscanning, so there is no import cycle. The export renames are mechanical. - Validation:
go build,go vetandgo test ./continuousscanning/pass at the head. The newTestSelfHealingWatchReconnectpasses with the fix. Withwatchbuilder.goreverted to main it fails with the expected"8"/""and"7"/"". CI is green. I did not run it on a live cluster, and I did not re-run./watchertests. - No duplicate PRs found. #439 is the earlier fix for the
watcherpackage and is complementary. - No blockers. The two P3s from blacksoxx (
AllowWatchBookmarks, comma-ok inservice.go) are optional follow-ups.Runalso retries without backoff when watch creation fails, but that is pre-existing.
Overview
This PR fixes #486
Signed Commits
SelfHealingWatchreconnected with an emptyresourceVersion, so after every watch timeout the API server replayed every object as a syntheticADDEDevent and continuous scanning scanned each of them again. A watch error event was also forwarded, and its*metav1.Statusthen failed thee.Object.(metav1.Object)assertion inservice.goand panicked.RunUntilWatchClosesrecords the version of each event's object withwatcher.ResumeFrom, so the next watch carries on from there.ResumeFromalready forgets the version when the event says it has expired, andRuncallswatcher.DropGoneVersionwhen creating the watch fails with 410 Gone, so the watch then starts over from the current state.resumeFromanddropGoneVersionin thewatcherpackage are exported asResumeFromandDropGoneVersion; their two callers there are renamed and nothing else changes.How to Test
TestSelfHealingWatchReconnecthas three cases: resuming from the last object, an error event not being forwarded (and keeping the version), and an expired version being forgotten. On main they fail withexpected: "8" actual: "",expected: "7" actual: ""andshould have 1 item(s), but has 2; with the fix they pass.go test ./continuousscanning/ -count=1andgo test ./watcher/ -skip TestRegistryCommandWatch -count=1pass (TestRegistryCommandWatchneeds Docker and fails on Windows on main too),go buildandgo vetof./watcher/... ./continuousscanning/... ./mainhandler/...are clean, andgolangci-lintreports nothing new. I haven't run it on a live cluster.Checklist before requesting a review
🤖 Generated with Claude Code