Measure whether the model actually loops, and shorten the README - #8
Merged
Conversation
This is the gate the plan put in front of the loop detector, and it changed two things. THE GATE PASSES, but not for the reason expected. Over 153 steps in 24 questions: repeat 4.6 %, drift 3.9 %, neither 91.5 %. Drift does *not* dominate -- which is the condition that would have stopped the work, since drift is the failure a repeat detector cannot see by construction. A near-identical study elsewhere found the opposite (2 repeats in 41 steps, with stateless drift at 49 %), so this had to be checked rather than inherited. THE THRESHOLD I PROPOSED WAS WRONG BY 4x. The plan said cosine >= 0.90 for a restatement, derived from two transcripts. Over 24 runs that flags 21 % of steps -- because the median step already sits at 0.846 similarity to some earlier step and the 75th percentile at 0.900, so the bar was catching the more-similar quarter of ordinary steps. Inspection settled it: a step saying the problem needs a percentage calculated, followed by one giving the formula (17/100)*250, scored 0.9013 -- and that is plain progress. At 0.96 the flags are restatements, two of them character-identical to an earlier step. The title bar needs no such care: exactly the same three steps fire anywhere from 0.95 to 0.999. Per step, 4.6 % sounds ignorable. Per question it is not -- repeats cluster, so a QUARTER of questions produce at least one and a single run produced four. That is the number the work is justified by. The script audits itself before reporting anything. Elsewhere a first run reported a 58 % loop rate that was entirely the step extractor seeing its own output twice; a measurement of the harness nearly became a finding about the model. So the normalisation is checked for manufacturing duplicates, and that check is pinned rather than assumed. Also measured, because it decides whether a similarity threshold is a coherent idea at all: the mutual-kNN step graph does not percolate at k<=4 (largest component 5.2 %). Where that collapses, "the same" cannot be a threshold and has to be an exact canonical form instead. Every raw step is committed, and --reanalyse recomputes from them, so a threshold can be revisited without model time and the numbers can be checked rather than taken. Six new claims in check_numbers, written as guards and comparisons rather than point values: one battery run has no error bar, and pinning a rate to three decimals from it would be precision this has not earned. Separately, the README is a fifth of its former length. The program that knows what is installed is better placed than a README to say what is missing, so startup now reports the models it found, or names the one command that fixes what it did not. The detail moved to docs/.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The gate the plan put in front of the loop detector. It changed two things.
The gate passes — but the threshold I proposed was wrong by 4×
Over 153 steps in 24 questions (simple, multi-step, malformed, ambiguous):
Drift does not dominate, which is the condition that would have stopped the work — drift is the failure a repeat detector cannot see by construction. A near-identical study elsewhere found the opposite (2 repeats in 41 steps, stateless drift at 49 %), so this had to be checked rather than inherited.
But the plan's 0.90 threshold flags 21 % of steps. The median step already sits at 0.846 cosine to some earlier step and the 75th percentile at 0.900 — so that bar catches the more-similar quarter of ordinary steps. Inspection settled it:
That is progress, not repetition. At 0.96 the flags are restatements — two of them character-identical to an earlier step:
The title bar needs no such care: exactly the same three steps fire anywhere from 0.95 to 0.999.
Per step it looks ignorable; per question it is not
Repeats cluster. A quarter of questions produce at least one, and one run produced four. That is the number the work is justified by, not the 4.6 %.
The script audits itself first
Elsewhere a first run reported a 58 % loop rate that was entirely the step extractor mistaking a code fence for a step and seeing it again — a measurement of the harness that nearly became a finding about the model. So the normalisation is checked for manufacturing duplicates before its numbers are used, and that check is pinned.
Also measured, because it decides whether a similarity threshold is a coherent idea at all: the mutual-kNN step graph does not percolate at k≤4 (largest component 5.2 %). Where that collapses, "the same" cannot be a threshold and has to be an exact canonical form instead.
Every raw step is committed and
--reanalyserecomputes from them, so a threshold can be revisited without model time. Six new claims, written as guards and comparisons rather than point values — one battery run has no error bar, and pinning a rate to three decimals from it would be precision this has not earned.Also: the README is a fifth of its former length
248 → 72 lines. The program that knows what is installed is better placed than a README to say what is missing, so startup now reports the models it found — or names the one command that fixes what it did not:
The detail moved to
docs/: models and configuration, internal layers, design, development.