Repository navigation
feat(review): per-agent accuracy scoring (citation-verified rate) - #145
Merged
Merged
Conversation
getappz
added
enhancement
📝 WalkthroughWalkthroughAdds persistent per-agent citation accuracy scoring, with round recording and aggregate score queries exposed through new CLI subcommands and MCP tools. ChangesReview accuracy scoring
Estimated code review effort: 3 (Moderate) | ~20 minutes Sequence Diagram(s)sequenceDiagram
participant CLI_or_MCP
participant ReviewLedger
participant GitDiff
participant ReviewScoring
CLI_or_MCP->>ReviewLedger: load review findings
CLI_or_MCP->>GitDiff: compute base..head diff
GitDiff-->>CLI_or_MCP: changed lines
CLI_or_MCP->>ReviewScoring: record_round(findings, changed lines)
ReviewScoring->>ReviewLedger: upsert score_events
CLI_or_MCP->>ReviewScoring: scores(repo scope)
ReviewScoring->>ReviewLedger: aggregate recorded events
ReviewScoring-->>CLI_or_MCP: ranked AgentScore values
Possibly related issues
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Comment |
Records each finder's per-round tally (total findings, and how many cited a real
changed line) into a score_events table in agentflare.db, keyed PRIMARY KEY
(repo, pr, agent) so re-recording a round replaces its contribution rather than
double-counting. review scores aggregates verified/total across rounds into a
per-agent accuracy rate, ranked, so an orchestrator can weight who to trust.
Accuracy = citation-verified rate only (does the agent point at real changed
lines); corroboration is a separate metric, left out on purpose so a valuable
UNIQUE finding others missed isn't penalized. Surface: MCP review_record /
review_scores, CLI agentflare review {record|scores}.
Closes #144
getappz
force-pushed
the
feat/review-scoring
branch
from
July 11, 2026 09:15
60b3fb7 to
ac496a1
Compare
getappz
force-pushed
the
feat/review-consensus
branch
from
July 11, 2026 09:15
18a5eab to
b690436
Compare
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (1)
src/cli/review.rs (1)
141-153: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winExtract shared round-loading helper in each file to remove the Record/Consensus duplication. Both files now have two near-identical "load findings → compute_diff → changed_lines" blocks (one in Consensus/review_consensus, one in the newly-added Record/review_record), which will keep drifting apart as more scoring actions are added.
src/cli/review.rs#L141-L153: extract a privateload_round_context(conn, repo, pr, base, head) -> (Vec<StoredFinding>, HashMap<String, HashSet<u32>>)helper (usingfail()internally) and call it from both theConsensusandRecordarms.src/mcp_server.rs#L918-L934: extract a privateSelf::load_round_context(conn, repo, pr, base, head) -> Result<(Vec<StoredFinding>, HashMap<String, HashSet<u32>>), ErrorData>helper and call it from bothreview_consensusandreview_record.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/cli/review.rs` around lines 141 - 153, Extract shared round-loading helpers for the duplicated findings-load, diff-compute, and changed-lines logic: in src/cli/review.rs lines 141-153, add private load_round_context(conn, repo, pr, base, head) returning the requested findings and changed-line maps while using fail(), then call it from both Consensus and Record; in src/mcp_server.rs lines 918-934, add Self::load_round_context with the corresponding Result and ErrorData handling, then call it from both review_consensus and review_record.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@src/review.rs`:
- Around line 176-194: Update record_round around the per_agent upsert loop to
replace the entire score_events snapshot for the current (repo, pr): remove
existing agent rows before inserting current findings, including when findings
is empty, within the same transaction. Extend the idempotency test to confirm
omitted agents and an empty-agent recording no longer appear in scores().
---
Nitpick comments:
In `@src/cli/review.rs`:
- Around line 141-153: Extract shared round-loading helpers for the duplicated
findings-load, diff-compute, and changed-lines logic: in src/cli/review.rs lines
141-153, add private load_round_context(conn, repo, pr, base, head) returning
the requested findings and changed-line maps while using fail(), then call it
from both Consensus and Record; in src/mcp_server.rs lines 918-934, add
Self::load_round_context with the corresponding Result and ErrorData handling,
then call it from both review_consensus and review_record.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: 3e913f29-8d64-4a0a-9f30-f1c0677b3565
📒 Files selected for processing (3)
src/cli/review.rssrc/mcp_server.rssrc/review.rs
Comment on lines
+176
to
+194
| let mut per_agent: HashMap<&str, (u32, u32)> = HashMap::new(); | ||
| for sf in findings { | ||
| let entry = per_agent.entry(sf.agent.as_str()).or_default(); | ||
| entry.0 += 1; | ||
| if changed.get(&sf.finding.file).is_some_and(|s| s.contains(&sf.finding.line)) { | ||
| entry.1 += 1; | ||
| } | ||
| } | ||
| for (agent, (total, verified)) in &per_agent { | ||
| conn.execute( | ||
| "INSERT INTO score_events (repo, pr, agent, total, verified, recorded_at) | ||
| VALUES (?1, ?2, ?3, ?4, ?5, ?6) | ||
| ON CONFLICT(repo, pr, agent) DO UPDATE SET | ||
| total = excluded.total, | ||
| verified = excluded.verified, | ||
| recorded_at = excluded.recorded_at", | ||
| params![repo, pr, agent, total, verified, now], | ||
| )?; | ||
| } |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win
Remove stale agents when replacing a recorded round.
record_round only upserts agents present in the current findings. If an earlier recording contained agentB and a later recording for the same (repo, pr) omits agentB—including an empty finding set—the old row remains, so scores() reports phantom findings and rounds. Replace the entire round snapshot atomically, and extend the idempotency test to verify removed and empty-agent cases.
Also applies to: 629-640
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@src/review.rs` around lines 176 - 194, Update record_round around the
per_agent upsert loop to replace the entire score_events snapshot for the
current (repo, pr): remove existing agent rows before inserting current
findings, including when findings is empty, within the same transaction. Extend
the idempotency test to confirm omitted agents and an empty-agent recording no
longer appear in scores().
3 tasks done
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Step 3 of the GitHub-across-agents roadmap: per-agent accuracy scoring.
Tracks how many of each finder's findings survived mechanical verification, so
an orchestrator can weight future dispatch toward agents that cite accurately.
Builds on step 2's (#143) same
file:lineverification.How
score_eventstable inagentflare.db,PRIMARY KEY (repo, pr, agent)→(total, verified, recorded_at). Recordinga round upserts one row per agent, so re-recording the same round replaces
its contribution (idempotent — no double-count).
review recordcomputes the diff, tallies each agent'stotal/verified findings, upserts. (Read-only
consensusstays side-effectfree; recording is the explicit mutating step.)
review scoresaggregatesSUM(total)/SUM(verified)/roundcount per agent across rounds → accuracy = verified/total, ranked.
review_record/review_scores; CLIagentflare review {record|scores} [--all-repos].What "accuracy" is (and isn't)
Citation-verified rate only — "does this agent point at real changed lines."
Corroboration (how often a finding is CONFIRMED by others) is deliberately a
separate metric, so a valuable UNIQUE finding others missed isn't penalized.
Tests
3 scoring unit tests (per-agent verified/total tally + ranking, re-record
replaces not double-counts, aggregate across distinct rounds). Full suite:
281 passed, 0 failed.
Driven end-to-end via CLI: two finders (one with an off-diff citation) →
review record→review scoresshowsclaude-code 100% (2/2)/opencode 50% (1/2), ranked; a second record kept it at 1 round (nodouble-count).
Closes #144 · related #143 (step 2), #141 (db)
Summary by CodeRabbit