docs(module-b): add GSoC 2026 mid-evaluation blog - #1004
Conversation
Summary by CodeRabbit
WalkthroughAdds a GSoC 2026 mid-evaluation blog post about the Module B noise and relevance filter, including its design, evaluation process, benchmark results, lessons learned, and planned production integration. Updates ChangesGSoC 2026 Module B blog
Estimated code review effort: 2 (Simple) | ~10 minutes Possibly related PRs
Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 3
🧹 Nitpick comments (1)
docs/Mid_eval_blog_gsoc2026/GSoC_2026_Mid_Evaluation_Blog_Draft.md (1)
95-101: 🗄️ Data Integrity & Integration | 🔵 Trivial | ⚡ Quick winMake the runtime label names explicit.
application/utils/noise_filter/schemas.pydefinesClassifyResult.labelasKNOWLEDGE,NOISE, orUNCERTAIN. This document usesKEEPandDROPin the classifier and scorecard sections. Define these as benchmark aliases and use the runtime names in contract-facing descriptions. Otherwise, readers can mistakeKEEPorDROPfor valid production payload values.Also applies to: 146-174
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@docs/Mid_eval_blog_gsoc2026/GSoC_2026_Mid_Evaluation_Blog_Draft.md` around lines 95 - 101, Update the document’s classifier and scorecard sections to state that KEEP and DROP are benchmark aliases for the runtime labels KNOWLEDGE and NOISE. Use KNOWLEDGE, NOISE, and UNCERTAIN in contract-facing descriptions and examples, while retaining the aliases only where explicitly discussing benchmark terminology.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@docs/Mid_eval_blog_gsoc2026/GSoC_2026_Mid_Evaluation_Blog_Draft.md`:
- Line 137: Update the “Where things stood at the mid evaluation” heading to use
the hyphenated compound form “Where things stood at the mid-evaluation,”
matching the document title.
- Line 11: Update the sentence in the evaluation blog draft by adding an em dash
or sentence-ending punctuation after “all the time” before “someone,” while
preserving the surrounding examples and meaning.
- Around line 155-163: Update the confusion-matrix discussion to explicitly
state that zero UNCERTAIN examples means uncertainty classification and
human-review handoff were not evaluated. If the benchmark is intended to
validate all three labels, add representative UNCERTAIN cases and update the
matrix and related results accordingly.
---
Nitpick comments:
In `@docs/Mid_eval_blog_gsoc2026/GSoC_2026_Mid_Evaluation_Blog_Draft.md`:
- Around line 95-101: Update the document’s classifier and scorecard sections to
state that KEEP and DROP are benchmark aliases for the runtime labels KNOWLEDGE
and NOISE. Use KNOWLEDGE, NOISE, and UNCERTAIN in contract-facing descriptions
and examples, while retaining the aliases only where explicitly discussing
benchmark terminology.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yml
Review profile: CHILL
Plan: Pro Plus
Run ID: 1d4aa745-ab68-43c5-b6d6-4a9a352b084f
📒 Files selected for processing (2)
.gitignoredocs/Mid_eval_blog_gsoc2026/GSoC_2026_Mid_Evaluation_Blog_Draft.md
|
|
||
| If you haven't heard of OpenCRE before, it's OWASP's **Common Requirement Enumeration** — think of it as a huge map that connects security standards, cheat sheets, testing guides, and tools together. Instead of jumping between different security resources, OpenCRE tries to bring all those connections into one place. | ||
|
|
||
| The problem the maintainers wanted to solve this year sounds simple, but it's actually pretty tricky. Security knowledge across OWASP projects keeps changing all the time someone adds a better SSRF example, someone fixes a testing guide for subdomain takeovers, someone updates a CSRF recommendation. All of that is valuable, but until now there wasn't an automated way to detect those changes and bring them into OpenCRE. |
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win
Add punctuation before “someone”.
The sentence currently reads “keeps changing all the time someone adds”. Add an em dash or a sentence break before “someone”.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs/Mid_eval_blog_gsoc2026/GSoC_2026_Mid_Evaluation_Blog_Draft.md` at line
11, Update the sentence in the evaluation blog draft by adding an em dash or
sentence-ending punctuation after “all the time” before “someone,” while
preserving the surrounding examples and meaning.
|
|
||
| --- | ||
|
|
||
| ## Where things stood at the mid evaluation |
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win
Hyphenate “mid-evaluation”.
Use the same compound form as the document title: Where things stood at the mid-evaluation.
🧰 Tools
🪛 LanguageTool
[grammar] ~137-~137: Use a hyphen to join words.
Context: .... --- ## Where things stood at the mid evaluation After several rounds of test...
(QB_NEW_EN_HYPHEN)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs/Mid_eval_blog_gsoc2026/GSoC_2026_Mid_Evaluation_Blog_Draft.md` at line
137, Update the “Where things stood at the mid evaluation” heading to use the
hyphenated compound form “Where things stood at the mid-evaluation,” matching
the document title.
Source: Linters/SAST tools
Un-ignore the mid-eval blog (targeted .gitignore exception; *.md stays ignored otherwise) and commit it as module_B_mideval_blog.md. Covers the Noise/Relevance Filter build-out, the recall-first decision, and mid-eval results (93% agreement, 100% KNOWLEDGE recall, 0 leakage) verified against the eval-run logs.
202385a to
4a43bec
Compare
There was a problem hiding this comment.
🧹 Nitpick comments (1)
docs/Mid_eval_blog_gsoc2026/module_B_mideval_blog.md (1)
71-73: 🗄️ Data Integrity & Integration | 🔵 Trivial | ⚡ Quick winMark
KEEPandDROPas evaluation aliases.The production
ClassifyResult.labelcontract inapplication/utils/noise_filter/schemas.pyat Lines [127]-[165] acceptsKNOWLEDGE,NOISE, andUNCERTAIN. This document presentsKEEPandDROPas emitted labels. State the mapping clearly at the interface description, or readers may implement payloads that fail schema validation.Also applies to: 97-97
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@docs/Mid_eval_blog_gsoc2026/module_B_mideval_blog.md` around lines 71 - 73, Update the Module B interface description to identify KEEP and DROP as evaluation aliases for the production ClassifyResult.label values KNOWLEDGE and NOISE, respectively; retain UNCERTAIN unchanged and make clear that emitted production payloads must use the schema-supported labels.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Nitpick comments:
In `@docs/Mid_eval_blog_gsoc2026/module_B_mideval_blog.md`:
- Around line 71-73: Update the Module B interface description to identify KEEP
and DROP as evaluation aliases for the production ClassifyResult.label values
KNOWLEDGE and NOISE, respectively; retain UNCERTAIN unchanged and make clear
that emitted production payloads must use the schema-supported labels.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yml
Review profile: CHILL
Plan: Pro Plus
Run ID: 239fe626-b603-4937-98a4-08d334f18cec
📒 Files selected for processing (2)
.gitignoredocs/Mid_eval_blog_gsoc2026/module_B_mideval_blog.md
🚧 Files skipped from review as they are similar to previous changes (1)
- .gitignore
1 participant
What
Adds my GSoC 2026 mid-evaluation blog for Module B (Noise/Relevance Filter) under
docs/Mid_eval_blog_gsoc2026/.To commit it,
.gitignoregets one targeted exception —!docs/Mid_eval_blog_gsoc2026/GSoC_2026_Mid_Evaluation_Blog_Draft.md. The blanket*.mdignore rule is otherwise unchanged, so no other docs start getting tracked.Content
The post walks through the first half of the project:
Mid-eval numbers in the post are cross-checked against the eval-run logs: 93% agreement, 100% KNOWLEDGE recall, 0 knowledge leakage on the 100-example benchmark, including the confusion matrix and the score progression (82% → 87% → 91%/3-lost → 93%).
Scope
Docs only — no code or schema changes.