Skip to content

The battery oddity was two holes, and both are closed - #33

Merged
punnerud merged 2 commits into
mainfrom
fix/synthesis-arithmetic
Aug 12, 2026
Merged

The battery oddity was two holes, and both are closed#33
punnerud merged 2 commits into
mainfrom
fix/synthesis-arithmetic

Conversation

@punnerud

Copy link
Copy Markdown
Owner

The full-battery run recorded an oddity: unit_chain 0/2 while the units domain went 8/8 — "same task, different phrasing, n=2, no conclusion drawn." Chased properly, it was not noise. It was two distinct holes, one behind the other.

Hole 1: the synthesis does sums, and nothing checked them

A failing answer read 378 × 86,400 = 32,356,800 seconds — right expression, wrong product — and the prose gate only ever ran on step content. repair_sums() now runs on the synthesised answer: the claim regex catches the shape, the evaluator knows the exact value, and every rendering of the wrong number is replaced with it (digit boundaries respected — 1.42 is never touched by a repair of 42).

Measured: when it fires, it decides — 2 of 2 repaired runs correct.

Hole 2, the dominant one: bare assertion

On twelve seconds-in-N-weeks questions the model asserted a bare unchecked number eight times — "There are 1,612,800 seconds", no expression, no calc, no convert. A bare assertion gives every gate in the pipeline nothing to hold.

So a question that is itself one conversion is settled before the first model call: the exact value goes into the facts the synthesis prefers and the selection can offer, as a step-0 convert event. The reasoning runs as usual — around an anchor instead of a hope.

Measured

twelve identical questions, same seed correct
before 4/12 (2 rescued by the repair)
with the anchor 12/12, zero repairs needed

And the original two battery failures, re-run: 0/2 → 2/2 with the machinery, still 0/2 without it — the mechanism, not luck.

The shape stays deliberately narrow: how many X in N Y and nothing else; question_conversion returns None for prose, arithmetic and word problems, tested.

Also: the bench filename now carries --group, after a two-question rerun overwrote the pinned twenty-question baseline — the second near-miss of that kind. Restored from git, claims re-checked.

11 new tests, 426 in all.

…odel speaks

The battery oddity -- unit_chain 0/2 while the units domain went 8/8 -- was
recorded as "no conclusion drawn, n=2". Chased properly, it was not noise. It
was two distinct holes, one behind the other.

THE FIRST: the synthesis does sums, and nothing checked them. A failing answer
read "378 x 86,400 = 32,356,800 seconds" -- right expression, wrong product --
and the prose gate only ever ran on step content. repair_sums() now runs on the
synthesised answer: the claim regex catches the shape, the evaluator knows the
exact value, and every rendering of the wrong number is replaced with it, digit
boundaries respected so 1.42 is never touched by a repair of 42. Measured: when
it fires it decides -- 2 of 2 repaired runs correct.

THE SECOND is the dominant one: on twelve seconds-in-N-weeks questions the
model asserted a bare unchecked number EIGHT times -- "There are 1,612,800
seconds", no expression, no calc, no convert. A bare assertion gives every gate
in the pipeline nothing to hold. So a question that is ITSELF one conversion is
now settled before the first model call: the exact value goes into the facts
the synthesis prefers and the selection can offer, as a step-0 convert event,
and the reasoning runs as usual around an anchor instead of a hope.

Measured on the same twelve questions, same seed:

  before             4/12   (2 of them rescued by the repair)
  with the anchor   12/12   zero repairs needed

The shape is kept deliberately narrow -- "how many X in N Y" and nothing else.
Almost every question is not a plain conversion, and question_conversion
returns None for all of those, tested against prose, arithmetic and word
problems.
A two-question group rerun overwrote the pinned twenty-question baseline -- the
second time a filename has nearly destroyed the number it was to be compared
against. The baseline was restored from git and the pinned claims re-checked.
@punnerud
punnerud merged commit b8cc059 into main Aug 12, 2026
6 checks passed
@punnerud
punnerud deleted the fix/synthesis-arithmetic branch August 12, 2026 16:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant