Skip to content

[Bugfix] Fix agnews query4 grading of incidental Africa mentions - #117

Open
Jason-lang-ui wants to merge 2 commits into
ucbepic:mainfrom
Jason-lang-ui:fix/agnews-query4-grading
Open

Jason-lang-ui wants to merge 2 commits into
ucbepic:mainfrom
Jason-lang-ui:fix/agnews-query4-grading

Conversation

@Jason-lang-ui

Copy link
Copy Markdown

The agnews/query4 validator accepts any answer containing Africa, including one that ranks North America first and Africa third. Require an affirmative first-place selection, allowing Africa to share first place.

Use a deterministic grammar for explicit answers, supported prose, numbered rankings and article-count tables. Cross-check winners, denials, ties, ranks and counts; fully parse numbers and table columns; preserve year/category/metric scope across blank lines and explicit answer transitions. Reject uncertain alternatives, retracted conclusions, malformed evidence and unsupported competing assertions. Unicode normalization and comparison-tail checks prevent conflicting claims from being silently dropped.

Validation: python3 -m unittest discover -s tests -v passes all 16 test methods, including the official controls over 104 validators and 1,658 answer-level regression cases. The matrices cover region substitutions, count ordering, ties, all three-column table permutations, malformed values, conflicting claims and heading scope. On the five observed answers, runs 0/1/3/4 pass and run 2 fails; run 0 retains its explicit first-place tie. The patch also applies cleanly to the upstream baseline and reproduces the tested files.

This is a bounded English grammar, not a complete semantic judge: correct unfamiliar wording can be rejected and may require manual review. The validate(str) -> (bool, str) interface is unchanged, with no new dependencies. If semantic judgment is required, it may be necessary to integrate LLM.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant