The battery oddity was two holes, and both are closed - #33
Merged
Conversation
…odel speaks The battery oddity -- unit_chain 0/2 while the units domain went 8/8 -- was recorded as "no conclusion drawn, n=2". Chased properly, it was not noise. It was two distinct holes, one behind the other. THE FIRST: the synthesis does sums, and nothing checked them. A failing answer read "378 x 86,400 = 32,356,800 seconds" -- right expression, wrong product -- and the prose gate only ever ran on step content. repair_sums() now runs on the synthesised answer: the claim regex catches the shape, the evaluator knows the exact value, and every rendering of the wrong number is replaced with it, digit boundaries respected so 1.42 is never touched by a repair of 42. Measured: when it fires it decides -- 2 of 2 repaired runs correct. THE SECOND is the dominant one: on twelve seconds-in-N-weeks questions the model asserted a bare unchecked number EIGHT times -- "There are 1,612,800 seconds", no expression, no calc, no convert. A bare assertion gives every gate in the pipeline nothing to hold. So a question that is ITSELF one conversion is now settled before the first model call: the exact value goes into the facts the synthesis prefers and the selection can offer, as a step-0 convert event, and the reasoning runs as usual around an anchor instead of a hope. Measured on the same twelve questions, same seed: before 4/12 (2 of them rescued by the repair) with the anchor 12/12 zero repairs needed The shape is kept deliberately narrow -- "how many X in N Y" and nothing else. Almost every question is not a plain conversion, and question_conversion returns None for all of those, tested against prose, arithmetic and word problems.
A two-question group rerun overwrote the pinned twenty-question baseline -- the second time a filename has nearly destroyed the number it was to be compared against. The baseline was restored from git and the pinned claims re-checked.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The full-battery run recorded an oddity:
unit_chain0/2 while the units domain went 8/8 — "same task, different phrasing, n=2, no conclusion drawn." Chased properly, it was not noise. It was two distinct holes, one behind the other.Hole 1: the synthesis does sums, and nothing checked them
A failing answer read
378 × 86,400 = 32,356,800 seconds— right expression, wrong product — and the prose gate only ever ran on step content.repair_sums()now runs on the synthesised answer: the claim regex catches the shape, the evaluator knows the exact value, and every rendering of the wrong number is replaced with it (digit boundaries respected —1.42is never touched by a repair of42).Measured: when it fires, it decides — 2 of 2 repaired runs correct.
Hole 2, the dominant one: bare assertion
On twelve seconds-in-N-weeks questions the model asserted a bare unchecked number eight times — "There are 1,612,800 seconds", no expression, no calc, no convert. A bare assertion gives every gate in the pipeline nothing to hold.
So a question that is itself one conversion is settled before the first model call: the exact value goes into the facts the synthesis prefers and the selection can offer, as a step-0 convert event. The reasoning runs as usual — around an anchor instead of a hope.
Measured
And the original two battery failures, re-run: 0/2 → 2/2 with the machinery, still 0/2 without it — the mechanism, not luck.
The shape stays deliberately narrow:
how many X in N Yand nothing else;question_conversionreturnsNonefor prose, arithmetic and word problems, tested.Also: the bench filename now carries
--group, after a two-question rerun overwrote the pinned twenty-question baseline — the second near-miss of that kind. Restored from git, claims re-checked.11 new tests, 426 in all.