Skip to content

Measure whether the model actually loops, and shorten the README - #8

Merged
punnerud merged 1 commit into
mainfrom
measure/loop-rate
Aug 10, 2026
Merged

Measure whether the model actually loops, and shorten the README#8
punnerud merged 1 commit into
mainfrom
measure/loop-rate

Conversation

@punnerud

Copy link
Copy Markdown
Owner

The gate the plan put in front of the loop detector. It changed two things.

The gate passes — but the threshold I proposed was wrong by 4×

Over 153 steps in 24 questions (simple, multi-step, malformed, ambiguous):

repeat — says something an earlier step already said 4.6 %
drift — new words, no new subject matter 3.9 %
neither 91.5 %

Drift does not dominate, which is the condition that would have stopped the work — drift is the failure a repeat detector cannot see by construction. A near-identical study elsewhere found the opposite (2 repeats in 41 steps, stateless drift at 49 %), so this had to be checked rather than inherited.

But the plan's 0.90 threshold flags 21 % of steps. The median step already sits at 0.846 cosine to some earlier step and the 75th percentile at 0.900 — so that bar catches the more-similar quarter of ordinary steps. Inspection settled it:

cos 0.9013
  step1: To find 17% of 250, we need to calculate a percentage of a given number.
  step2: We can use the formula: (percentage / 100) * number. In this case (17 / 100) * 250.

That is progress, not repetition. At 0.96 the flags are restatements — two of them character-identical to an earlier step:

cos 1.0000
  step1: The formula for the area of a circle is A = pi*r^2. Substituting r = 3 ... 28.27.
  step3: The formula for the area of a circle is A = pi*r^2. Substituting r = 3 ... 28.27.

The title bar needs no such care: exactly the same three steps fire anywhere from 0.95 to 0.999.

Per step it looks ignorable; per question it is not

Repeats cluster. A quarter of questions produce at least one, and one run produced four. That is the number the work is justified by, not the 4.6 %.

The script audits itself first

Elsewhere a first run reported a 58 % loop rate that was entirely the step extractor mistaking a code fence for a step and seeing it again — a measurement of the harness that nearly became a finding about the model. So the normalisation is checked for manufacturing duplicates before its numbers are used, and that check is pinned.

Also measured, because it decides whether a similarity threshold is a coherent idea at all: the mutual-kNN step graph does not percolate at k≤4 (largest component 5.2 %). Where that collapses, "the same" cannot be a threshold and has to be an exact canonical form instead.

Every raw step is committed and --reanalyse recomputes from them, so a threshold can be revisited without model time. Six new claims, written as guards and comparisons rather than point values — one battery run has no error bar, and pinning a rate to three decimals from it would be precision this has not earned.

Also: the README is a fifth of its former length

248 → 72 lines. The program that knows what is installed is better placed than a README to say what is missing, so startup now reports the models it found — or names the one command that fixes what it did not:

  Local Knowledge Graph

  Ollama did not answer at http://localhost:11434.

  This app runs a language model on your own machine through Ollama.
  1. Install it from https://ollama.com
  2. Pull a model:   ollama pull llama3.2:3b
  3. Start this again.

The detail moved to docs/: models and configuration, internal layers, design, development.

This is the gate the plan put in front of the loop detector, and it changed
two things.

THE GATE PASSES, but not for the reason expected. Over 153 steps in 24
questions: repeat 4.6 %, drift 3.9 %, neither 91.5 %. Drift does *not*
dominate -- which is the condition that would have stopped the work, since
drift is the failure a repeat detector cannot see by construction. A
near-identical study elsewhere found the opposite (2 repeats in 41 steps,
with stateless drift at 49 %), so this had to be checked rather than
inherited.

THE THRESHOLD I PROPOSED WAS WRONG BY 4x. The plan said cosine >= 0.90 for a
restatement, derived from two transcripts. Over 24 runs that flags 21 % of
steps -- because the median step already sits at 0.846 similarity to some
earlier step and the 75th percentile at 0.900, so the bar was catching the
more-similar quarter of ordinary steps. Inspection settled it: a step saying
the problem needs a percentage calculated, followed by one giving the formula
(17/100)*250, scored 0.9013 -- and that is plain progress. At 0.96 the flags
are restatements, two of them character-identical to an earlier step. The
title bar needs no such care: exactly the same three steps fire anywhere from
0.95 to 0.999.

Per step, 4.6 % sounds ignorable. Per question it is not -- repeats cluster,
so a QUARTER of questions produce at least one and a single run produced
four. That is the number the work is justified by.

The script audits itself before reporting anything. Elsewhere a first run
reported a 58 % loop rate that was entirely the step extractor seeing its own
output twice; a measurement of the harness nearly became a finding about the
model. So the normalisation is checked for manufacturing duplicates, and that
check is pinned rather than assumed.

Also measured, because it decides whether a similarity threshold is a
coherent idea at all: the mutual-kNN step graph does not percolate at k<=4
(largest component 5.2 %). Where that collapses, "the same" cannot be a
threshold and has to be an exact canonical form instead.

Every raw step is committed, and --reanalyse recomputes from them, so a
threshold can be revisited without model time and the numbers can be checked
rather than taken. Six new claims in check_numbers, written as guards and
comparisons rather than point values: one battery run has no error bar, and
pinning a rate to three decimals from it would be precision this has not
earned.

Separately, the README is a fifth of its former length. The program that
knows what is installed is better placed than a README to say what is
missing, so startup now reports the models it found, or names the one command
that fixes what it did not. The detail moved to docs/.
@punnerud
punnerud merged commit 7829c04 into main Aug 10, 2026
6 checks passed
@punnerud
punnerud deleted the measure/loop-rate branch August 10, 2026 21:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant