Measure the reasoning, then fix what the measurement found: 55% to 91% - #9
Open
punnerud wants to merge 4 commits into
Open
Measure the reasoning, then fix what the measurement found: 55% to 91%#9punnerud wants to merge 4 commits into
punnerud wants to merge 4 commits into
Conversation
Measured, 20 questions x 2-3 runs against llama3.2:3b, on the groups a
substring grader can actually score:
baseline 27/51 52.9 %
synthesis 33/34 97.1 %
simple 10/18 -> 12/12
multi_step 11/18 -> 12/12
malformed 6/15 -> 9/10
The baseline's answer to "What is the capital of France?" did not contain the
word Paris. It was a footnote about regional capitals -- because the final
answer was literally the text of whatever step happened to be last, and a
system prompt that rewards exploring alternatives and considering that you may
be wrong naturally ends on a caveat. The graph was drawn and then ignored.
Now the strongest path through the graph is extracted, and one call turns that
thread into an answer. This is the graph earning its place: of everything the
model said, these are the steps that hang together, and they are what the
answer gets written from.
Costs one extra model call per run (6.2 -> 7.9) and 27 % more prompt tokens
(2704 -> 3446), for 44 percentage points. If synthesis fails the run falls
back to the last step, because losing the whole answer to one failed call
would be a bad trade.
THE AMBIGUOUS GROUP IS NO LONGER GRADED, and that is a correction to the
measurement rather than to the code. Substring matching cannot score "Is a hot
dog a sandwich?" -- "A hot dog is a sandwich." is a perfectly good answer and
matches no keyword list worth writing, while a rambling hedge scores well by
accident because it contains the word "depends". The baseline beat the
synthesis arm 67 % to 33 % there on exactly that artefact. Those questions
still run and their answers are still saved; they are reported separately
instead of being averaged into a headline they would move for the wrong
reason.
Runs are now bounded by a clock rather than only by a step count. A step cap
stops a run that is still getting somewhere and lets a slow one run forever;
two minutes of wall clock bounds the wait a person actually experiences.
One real bug found by an existing test: when synthesis failed and the run fell
back to the last step, it added a second node holding identical text, which
produces an edge of exactly 1.00 between them. The relabel branch now keys on
the text being the same, not on why.
Measured, 20 questions x 2 runs each, gradeable groups only:
arm correct steps prompt tok sec repeats caught
baseline 52.9 % 6.2 2704 7.5 0
synthesis 97.1 % 6.9 3446 9.5 0
novelty 97.1 % 3.5 1391 4.6 0
no_floor_only 91.2 % 3.6 1451 4.8 0
Same accuracy as the previous arm at 40 % of the prompt tokens and half the
wall clock. Runs went from 6.9 steps to 3.5.
THE FLOOR WAS THE WHOLE EFFECT, AND THE REPEAT DETECTOR CONTRIBUTED NOTHING
MEASURABLE. min_steps=5 forced every question to five steps whether or not it
had five steps' worth of thinking in it -- a reported transcript reached its
answer at step 4 and was pushed on with "you have given 4 of 5 steps",
producing three more that added nothing. Removing it is what halved the cost.
The ablation arm says so plainly. novelty scores 33/34 against no_floor_only's
31/34, but repeats_caught is ZERO in both: a mechanism that never ran cannot
have caused a difference, so those two answers are sampling noise at two runs
per question, not evidence for the detector.
Which also revises the finding that motivated the detector. A quarter of runs
used to contain a repeated step; at 3.5 steps instead of 6.4 that largely
stops happening, because the repeats lived in steps 5 to 8 -- the ones the
floor was manufacturing. We were detecting a problem we were causing.
The detector stays, described as what it is: insurance for the pathological
case, not a contributor to these numbers. It is cheap, it is tested against a
model that repeats one move forever, and it bounds the case actually reported.
When it does fire it hands back a coverage map rather than a prohibition --
"you have already covered X, Y, Z; go somewhere none of those go" -- because
a list of what is done leaves the unexplored part by subtraction, while "do
not repeat yourself" gives nothing to aim at. Two refusals in a row end the
run.
novelty.py computes the value it compares against a threshold as
2*atan2(|a-b|, |a+b|) rather than from the dot product. Both 2-2*dot and
acos(dot) lose their precision exactly at dot ~ 1, which is where duplicates
live. The dot product is still fine for ranking.
Also fixed: a model that answered the synthesis prompt in the step schema --
having been asked for JSON on every previous turn -- had its raw JSON shown as
the answer.
One test changed meaning rather than breaking. It asserted that a model
wanting to stop early still produced at least five nodes, which was the floor
doing its damage. It now asserts the opposite: a one-step answer must not be
padded to five.
I expected the opposite. That prompt is 233 tokens of shouting resent on every
call -- most of the prompt cost once runs dropped to 3.5 steps -- and it opens
with "USE AS MANY REASONING STEPS AS POSSIBLE", which fights the change that
removed the step floor. It looked like free savings.
Measured, 20 questions x 2 runs, gradeable groups only:
arm correct steps prompt tok
long (kept) 97.1 % 3.5 1391
short_min3 88.2 % 3.7 1084
explore 79.4 % 1.6 399
short 82.4 % 1.2 281
The confound is controlled. Terse prompts made the model stop after one or two
steps, so at first the loss looked like "less reasoning" rather than "worse
instructions". short_min3 puts the floor back at three: 3.7 steps against the
long prompt's 3.5, and it still loses nine points. IT IS THE WORDING, NOT THE
AMOUNT OF REASONING.
What it buys is arithmetic. Under the short prompt the failures were 10080
minutes in a fortnight (that is a week), 6.00 change from a 20 note on 13.50 of
goods, and a wrong area comparison. "USE AT LEAST 3 METHODS TO DERIVE THE
ANSWER" is apparently doing work.
The short version stays in the file as SHORT_SYSTEM_PROMPT so the result is
reproducible instead of becoming folklore.
AND THE EVAL HAD A BUG THAT INVERTED ONE QUESTION. A circle of radius 3 has
area 28.27 against a square of side 5 at 25, so the circle is larger -- the
expectation said "square". Arms that answered correctly were being marked
wrong. Every arithmetic expectation has now been checked by hand, and every
arm was regraded from its saved answers rather than re-run, which is what
saving them was for.
Modularised: backends.py passed 500 lines and src/ passed 2000, so it is now a
package -- what every backend shares, the real one, and the fakes the tests run
against. Everything is re-exported, so `from mpe_lkg import backends` still
reaches all of it.
That split broke seven tests in an instructive way: they monkeypatched
`backends.list_models`, but health() resolves that name inside the module it
lives in, so the patch stopped reaching it while still appearing to work. The
fixtures now patch both names, with a comment saying why.
The two-run arms read 97.1 %. Four runs per question, 80 runs in all, read
91.2 %. The shipped figure is the four-run one, because a number measured once
is not measured -- and that gap is the whole reason for running it again.
arm correct steps prompt tok sec
what this project shipped 54.9 % 6.2 2704 7.5
now, validated over 80 runs 91.2 % 3.9 1649 5.2
simple 100 % (24/24)
multi_step 92 % (22/24)
malformed 80 % (16/20)
Fewer steps, fewer tokens, faster, and thirty-six points more of the questions
answered.
Six claims pinned in check_numbers, with a wide tolerance on the headline on
purpose: pinning 91.2 % to three decimals would be pinning the sampling noise
of a stochastic model rather than its behaviour. The comparison against the
baseline is pinned tightly instead, because that is the part far outside the
noise -- same questions, same model, same grader.
docs/development.md now records what was tried and measured WORSE alongside
what worked, so none of it gets rediscovered: the shorter system prompt, the
repeat detector as a contributor rather than as insurance, and grading
ambiguous questions by substring at all.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Experimental work against a measured baseline. Every arm is in
docs/claims/eval.jsonwith every answer saved, so the grading can be checked and old arms regraded without re-running the model.55 % → 91 %, validated over 80 runs
Fewer steps, fewer tokens, faster, and thirty-six points more of the questions answered. Two changes account for it, and neither is the one that was planned.
The answer is now written from the graph
It used to be the text of whatever step came last. So the answer to "What is the capital of France?" did not contain the word Paris — it was a footnote about regional capitals, because a prompt that rewards exploring alternatives naturally ends on a caveat. The graph was drawn and then ignored.
Now the strongest path through the graph is extracted and one call turns that thread into an answer.
The step floor was the other half
min_steps = 5forced every question to five steps whether or not it had five steps' worth of thinking in it. Removing it halved the step count and the token cost at no cost in accuracy.It also removed most of the repetition the loop detector was built for — those repeats lived in steps 5 to 8, the ones the floor was manufacturing. We had been detecting a problem we were causing.
Measured worse, and recorded so it is not rediscovered
Two bugs in my own measurement
The eval had an inverted expectation. A circle of radius 3 has area 28.27 against a square of side 5 at 25, so the circle is larger — the expectation said "square". Arms answering correctly were marked wrong. Every arithmetic expectation has since been checked by hand and every arm regraded from its saved answers.
A two-run arm read 97.1 % where four runs read 91.2 %. The shipped figure is the four-run one.
Also
Runs are bounded by a two-minute clock rather than only a step count.
backends.pypassed 500 lines andsrc/passed 2000, so backends is now a package — that split broke seven tests instructively: they monkeypatchedbackends.list_modelswhilehealth()resolves that name inside its own module, so the patch stopped reaching it while still appearing to work.138 tests, green on Linux 3.10–3.13, macOS and Windows.