Repository navigation
Compare implementations aligned by step - #39
Merged
Merged
Conversation
A script with more than one run is asking where the implementations stop agreeing, and until now it answered with a stack of separate failure reports and a tally of how many failed. Doing the comparison was left to whoever read them. It now ends with the comparison itself: every port lined up against the first run at the step where it left it, with both sides quoted under one column so the difference is read down the page. The divergence becomes a value rather than a string built where the failure happened, which is what makes aligning several runs possible at all -- a step has to be a number something else can group by. Ports that left at the same step, the same way, printing the same thing are one finding: a dozen ports of the same BASIC program missing a leading space is one bug repeated twelve times, and a report that says it twelve times is one nobody reads to the end. Three things are deliberately not divergences. A port that plays a different transcript is not compared at all, because step 7 of one transcript has nothing to do with step 7 of another. A port that never ran has not disagreed with anything. And when the reference no longer matches its own transcript, nothing is compared: the environment has moved, and every port's divergence would be measured against something that is itself wrong. A run that could not be started now says so on stderr rather than only through a logger that is off by default -- the comparison names it as one that could not be played, which invites the question. Closes #16 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019f9gkZVUEo3jtv51GSb4vJ
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #16.
A script with more than one run is asking where the implementations stop agreeing. Until now it answered with a stack of separate failure reports and a tally of how many failed, leaving the actual comparison to whoever read them. It now ends with the comparison:
What changed
Divergenceis a value (internal/play/divergence.go), not a string built where the failure happened. Aligning three ports at the step where each left the reference is only possible if that step is a number something else can group by.FailureReportis now a rendering of it, so the single-run report and the comparison cannot drift.Runreturns anOutcomerather than an exit code, soRunAllhas something to compare.Compare/Comparison(internal/play/comparison.go) groups ports that left at the same step, the same way, printing the same thing. A dozen ports missing BASIC's leading space is one bug repeated twelve times; grouping it is what makes it a finding instead of a list. This is also what Classify divergences by draw count and formatting vs semantic #17's per-language aggregation needs.What is deliberately not a divergence
Received output
The comparison shows the whole block a port printed since it last agreed with the transcript, not one chosen line. Which of several lines was meant to be the step is exactly what a divergence puts in question, and
receivedLine's heuristic answers it one way for a prompt and the wrong way for a transposed line. The caret is printed only when there is a single line, where there is nothing left to assert.Also
A run that could not be started now says so on stderr instead of only through a logger that is off by default. The comparison names it as one that could not be played, which invites the question.
Not done as proposed
The issue suggested a reporter alongside
SingleRunReporterininternal/reporters. That package no longer exists, and the comparison has to be callable fromruns.go— a separate package would importinternal/playandinternal/playwould import it back. It lives ininternal/play.Checks
make format,make lint,PRESCRIPT_REQUIRE_RUNNERS=1 make test,make build_dev,make examplesall pass.internal/play/comparison_test.gocovers grouping, ordering, the single-line marker, a failing reference, an incomparable transcript, an unplayable port, and column alignment; the report shapes were also checked end to end against a five-run script.🤖 Generated with Claude Code
https://claude.ai/code/session_019f9gkZVUEo3jtv51GSb4vJ