Repository navigation
[Bugfix] Fix agnews query4 grading of incidental Africa mentions - #117
Open
Jason-lang-ui wants to merge 2 commits into
Open
Jason-lang-ui wants to merge 2 commits into
Jason-lang-ui wants to merge 2 commits into
Conversation
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The agnews/query4 validator accepts any answer containing
Africa, including one that ranks North America first and Africa third. Require an affirmative first-place selection, allowing Africa to share first place.Use a deterministic grammar for explicit answers, supported prose, numbered rankings and article-count tables. Cross-check winners, denials, ties, ranks and counts; fully parse numbers and table columns; preserve year/category/metric scope across blank lines and explicit answer transitions. Reject uncertain alternatives, retracted conclusions, malformed evidence and unsupported competing assertions. Unicode normalization and comparison-tail checks prevent conflicting claims from being silently dropped.
Validation:
python3 -m unittest discover -s tests -vpasses all 16 test methods, including the official controls over 104 validators and 1,658 answer-level regression cases. The matrices cover region substitutions, count ordering, ties, all three-column table permutations, malformed values, conflicting claims and heading scope. On the five observed answers, runs 0/1/3/4 pass and run 2 fails; run 0 retains its explicit first-place tie. The patch also applies cleanly to the upstream baseline and reproduces the tested files.This is a bounded English grammar, not a complete semantic judge: correct unfamiliar wording can be rejected and may require manual review. The
validate(str) -> (bool, str)interface is unchanged, with no new dependencies. If semantic judgment is required, it may be necessary to integrate LLM.