A physics domain: light, wind and speed, with every constant stated - #34
Merged
Conversation
The battery's rule is that the truth is computed by the same code that renders the question, and physics threatens the rule because it leans on constants. The resolution: every constant is IN the question text. The model is never asked to remember the speed of light or the distance to Mars -- it is asked to compose figures it has been handed, across units, which is the failure mode this project has measured models into and built machinery for. Four groups: light travel time to the Moon, the Sun, Mars and Jupiter; wind distance in mixed units; time around the equator; mph to metres per second. Answers are exact Fractions; most do not terminate as decimals, and matches() grades those at the places a sensible answer states. The tests re-derive every answer from the question's own text, and that discipline caught the author twice before any model was graded: both mile-based answers were a factor of a thousand off, because MILE_KM is already kilometres per mile and the generator multiplied by 1000 again. Two test assumptions were also wrong where the code was right -- "One mile" is spelled out, not a digit, and a test that waits for the RNG to draw the Moon tests the RNG. Measured live against qwen3: 8/8, pinned.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
New battery domain per request: miles, wind speed, light travel to the Moon/Mars/Sun, around the Earth.
The rule that keeps physics inside the battery's discipline
Physics leans on constants, and constants are recall. So every constant is stated in the question text — the model is never asked to remember the speed of light or a planetary distance, it is asked to compose figures it has been handed, across units. That is precisely the failure mode this project measured models into (the exponent-guessing Fermi assembly) and built machinery for.
light_travelwind_distancearound_earthspeed_conversionSelf-grading, and it caught the author first
The tests re-derive every answer from the question's own text. That discipline fired twice before any model was graded: both mile-based answers were a factor of a thousand off (MILE_KM is already km per mile; the generator multiplied by 1000 again). Two test assumptions were also wrong where the code was right — "One mile" is spelled out, not a digit, and a test that waits for the RNG to draw the Moon tests the RNG.
Measured live against qwen3
8/8 — pinned in
check_numbers.py.14 new tests, 439 in all.