Repository navigation
Classify divergences by shape and by draws - #41
Merged
Merged
Conversation
Not every divergence is worth filing, and two things about one can be worked out without a person reading it. Both now appear as fixed tokens beside the failure mode -- `step 5 of 12 (no-match, formatting, no-draws)` -- so a corpus of a few hundred runs can be counted and sorted by them. Shape: collapse whitespace on both sides and compare again. BASIC's PRINT puts a space before a non-negative number and another after it, and `,` tabs to fourteen-column zones; ports get this wrong constantly and identically, and filing each separately would mean several hundred near-identical issues against one repository, which would rightly be ignored. Told apart, the class goes in one report per language while a semantic divergence stays with its program. The received lines are joined before normalising, so a port that broke one logical line in two is layout rather than meaning; case is not touched, because a port that shouts where the reference whispers changed the text. Draws: prescript now sets PRESCRIPT_TAPE_USAGE itself on any taped run and reads the count back, so the classification happens without anyone remembering to ask for it. A port that had drawn nothing cannot be diverging over the numbers it drew; one that drew what the reference drew asked in the same places and disagreed about the answer; one that drew a different amount consumes randomness differently, and no report can say whether that is a bug. The shims rewrite the count after every draw rather than once on the way out, because the number that matters is what had been drawn when the run stopped -- and a run that diverged by hanging is killed, which fires no exit handler in any of these languages. They also write it before the first draw, so that a port which drew nothing says so: unknown is not zero, and the two are not allowed to look alike. The reference interpreter cannot answer this axis -- a tape is recorded from Vintage BASIC rather than replayed into it -- so `same-draws` is out of reach on a corpus run, and the report leaves an unknown axis out rather than naming it on every line. #40 tracks closing that gap. Closes #17 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019f9gkZVUEo3jtv51GSb4vJ
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #17. Files #40 for the part that cannot be done here.
Each finding in the comparison now carries what could be worked out about it without a person reading it, as fixed tokens beside the failure mode:
(That is real output, from four Ruby programs played against one tape.)
Axis 2 — formatting or semantic
Collapse whitespace on both sides, compare again. BASIC's
PRINTputs a space before a non-negative number and another after it, and,tabs to fourteen-column zones; ports get this wrong constantly and identically, so the class belongs in one report per language while a semantic divergence stays with its program.The received lines are joined before normalising, so a port that broke one logical line across two prints is
formatting— where a line breaks is layout by any reading. Case is untouched: a port that shouts where the reference whispers changed the text, not its layout. A redacted step gets neither token, because the expected side is a pattern and there is nothing to normalise.Axis 1 — draws
prescript play --tapenow setsPRESCRIPT_TAPE_USAGEfor itself and reads the count back onto theOutcome. A classification that only happens when somebody remembered to set a variable is one that mostly does not happen. A run that names the variable itself keeps its own path.The shims now rewrite the count after every draw rather than once on the way out. The number that matters is how much had been drawn when the run stopped agreeing with the transcript, and a run that diverged by hanging is killed — which fires no exit handler in Ruby, Python or Node.
TestTheDrawCountSurvivesARunThatIsKilledcovers exactly that. Rewriting in place needs no truncation: the count only grows, so its decimal form only gets longer and each write covers the last.They also write it before the first draw. Unknown is not zero — a port that drew nothing has said randomness is not implicated, and one that cannot say has said nothing — so
Outcome.Drawsis a*intand the two never look alike.What this cannot do
The reference interpreter does not report its draws. Vintage BASIC has no shim — a tape is recorded from it rather than replayed into it — so on a corpus run the reference's count is unknown and
same-drawsis out of reach. What survives isno-draws(which needs only the port) and disagreement between ports in one finding, which is itself a finding. The full table applies when the reference is a taped port.An unknown axis is left out of the report rather than named. A token on every line carries nothing.
docs/comparison.mdstates the gap plainly, and #40 tracks closing it: a second patch toBasicMonad.hs, gated on a newVINTBAS_RELEASEbuild and two pin updates.Aggregation
The issue asks for formatting divergences to be aggregated one-per-language and semantic ones one-per-game. That is across scripts, and prescript plays one script per invocation — it belongs with the generated bug reports in #18, which now has the classification to key on.
Checks
make format,make lint,PRESCRIPT_REQUIRE_RUNNERS=1 make test,make build_dev,make examplespass. New unit tests cover both axes and their interaction; new integration tests cover collection with a tape, absence without one, and survival of a killed run, across all three taped runners.🤖 Generated with Claude Code
https://claude.ai/code/session_019f9gkZVUEo3jtv51GSb4vJ