Skip to content

Classify divergences by shape and by draws - #41

Merged
rjnienaber merged 1 commit into
mainfrom
feat/classify-divergences
Sep 7, 2026
Merged

rjnienaber merged 1 commit into
mainfrom
feat/classify-divergences

Conversation

@rjnienaber

Copy link
Copy Markdown
Owner

Closes #17. Files #40 for the part that cannot be done here.

Each finding in the comparison now carries what could be worked out about it without a person reading it, as fixed tokens beside the failure mode:

step 1 of 1 (exited-early, formatting, same-draws)
  reference     " 7             807 "
  spacing       "7 807"
                 ^ first difference

step 1 of 1 (exited-early, semantic, no-draws)
  reference     " 7             807 "
  nodraw        "SEVEN 807"

step 1 of 1 (exited-early, semantic, different-draws)
  reference     " 7             807 "
  restructured  "7 807 extra"

(That is real output, from four Ruby programs played against one tape.)

Axis 2 — formatting or semantic

Collapse whitespace on both sides, compare again. BASIC's PRINT puts a space before a non-negative number and another after it, and , tabs to fourteen-column zones; ports get this wrong constantly and identically, so the class belongs in one report per language while a semantic divergence stays with its program.

The received lines are joined before normalising, so a port that broke one logical line across two prints is formatting — where a line breaks is layout by any reading. Case is untouched: a port that shouts where the reference whispers changed the text, not its layout. A redacted step gets neither token, because the expected side is a pattern and there is nothing to normalise.

Axis 1 — draws

prescript play --tape now sets PRESCRIPT_TAPE_USAGE for itself and reads the count back onto the Outcome. A classification that only happens when somebody remembered to set a variable is one that mostly does not happen. A run that names the variable itself keeps its own path.

The shims now rewrite the count after every draw rather than once on the way out. The number that matters is how much had been drawn when the run stopped agreeing with the transcript, and a run that diverged by hanging is killed — which fires no exit handler in Ruby, Python or Node. TestTheDrawCountSurvivesARunThatIsKilled covers exactly that. Rewriting in place needs no truncation: the count only grows, so its decimal form only gets longer and each write covers the last.

They also write it before the first draw. Unknown is not zero — a port that drew nothing has said randomness is not implicated, and one that cannot say has said nothing — so Outcome.Draws is a *int and the two never look alike.

What this cannot do

The reference interpreter does not report its draws. Vintage BASIC has no shim — a tape is recorded from it rather than replayed into it — so on a corpus run the reference's count is unknown and same-draws is out of reach. What survives is no-draws (which needs only the port) and disagreement between ports in one finding, which is itself a finding. The full table applies when the reference is a taped port.

An unknown axis is left out of the report rather than named. A token on every line carries nothing.

docs/comparison.md states the gap plainly, and #40 tracks closing it: a second patch to BasicMonad.hs, gated on a new VINTBAS_RELEASE build and two pin updates.

Aggregation

The issue asks for formatting divergences to be aggregated one-per-language and semantic ones one-per-game. That is across scripts, and prescript plays one script per invocation — it belongs with the generated bug reports in #18, which now has the classification to key on.

Checks

make format, make lint, PRESCRIPT_REQUIRE_RUNNERS=1 make test, make build_dev, make examples pass. New unit tests cover both axes and their interaction; new integration tests cover collection with a tape, absence without one, and survival of a killed run, across all three taped runners.

🤖 Generated with Claude Code

https://claude.ai/code/session_019f9gkZVUEo3jtv51GSb4vJ

Not every divergence is worth filing, and two things about one can be
worked out without a person reading it. Both now appear as fixed tokens
beside the failure mode -- `step 5 of 12 (no-match, formatting,
no-draws)` -- so a corpus of a few hundred runs can be counted and
sorted by them.

Shape: collapse whitespace on both sides and compare again. BASIC's
PRINT puts a space before a non-negative number and another after it,
and `,` tabs to fourteen-column zones; ports get this wrong constantly
and identically, and filing each separately would mean several hundred
near-identical issues against one repository, which would rightly be
ignored. Told apart, the class goes in one report per language while a
semantic divergence stays with its program. The received lines are
joined before normalising, so a port that broke one logical line in two
is layout rather than meaning; case is not touched, because a port that
shouts where the reference whispers changed the text.

Draws: prescript now sets PRESCRIPT_TAPE_USAGE itself on any taped run
and reads the count back, so the classification happens without anyone
remembering to ask for it. A port that had drawn nothing cannot be
diverging over the numbers it drew; one that drew what the reference
drew asked in the same places and disagreed about the answer; one that
drew a different amount consumes randomness differently, and no report
can say whether that is a bug.

The shims rewrite the count after every draw rather than once on the way
out, because the number that matters is what had been drawn when the run
stopped -- and a run that diverged by hanging is killed, which fires no
exit handler in any of these languages. They also write it before the
first draw, so that a port which drew nothing says so: unknown is not
zero, and the two are not allowed to look alike.

The reference interpreter cannot answer this axis -- a tape is recorded
from Vintage BASIC rather than replayed into it -- so `same-draws` is
out of reach on a corpus run, and the report leaves an unknown axis out
rather than naming it on every line. #40 tracks closing that gap.

Closes #17

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019f9gkZVUEo3jtv51GSb4vJ
@rjnienaber
rjnienaber merged commit 86fa312 into main Sep 7, 2026
5 checks passed
@rjnienaber
rjnienaber deleted the feat/classify-divergences branch September 7, 2026 21:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Classify divergences by draw count and formatting vs semantic

1 participant