Skip to content

Compare implementations aligned by step - #39

Merged
rjnienaber merged 1 commit into
mainfrom
feat/diff-reporter
Sep 7, 2026
Merged

rjnienaber merged 1 commit into
mainfrom
feat/diff-reporter

Conversation

@rjnienaber

Copy link
Copy Markdown
Owner

Closes #16.

A script with more than one run is asking where the implementations stop agreeing. Until now it answered with a stack of separate failure reports and a tally of how many failed, leaving the actual comparison to whoever read them. It now ends with the comparison:

=== comparison against basic ===

step 1 of 2 (exited-early)
  basic         "YOU HAVE 3 GUESSES LEFT"
  ruby, python  "You have 3 guesses left"
                "TRY AGAIN"

matched: perl
could not be played: go

What changed

  • Divergence is a value (internal/play/divergence.go), not a string built where the failure happened. Aligning three ports at the step where each left the reference is only possible if that step is a number something else can group by. FailureReport is now a rendering of it, so the single-run report and the comparison cannot drift.
  • Run returns an Outcome rather than an exit code, so RunAll has something to compare.
  • Compare / Comparison (internal/play/comparison.go) groups ports that left at the same step, the same way, printing the same thing. A dozen ports missing BASIC's leading space is one bug repeated twelve times; grouping it is what makes it a finding instead of a list. This is also what Classify divergences by draw count and formatting vs semantic #17's per-language aggregation needs.
  • Findings are ordered by step, and share one column width across the whole report.

What is deliberately not a divergence

  • A port playing a different transcript is not compared. Step 7 of one transcript has nothing to do with step 7 of another, and lining them up reports a difference that is an artefact of the script.
  • A port that never ran has not disagreed with anything.
  • When the reference itself diverged, nothing is compared — the environment has moved, so every port's divergence would be measured against something that is itself wrong.

Received output

The comparison shows the whole block a port printed since it last agreed with the transcript, not one chosen line. Which of several lines was meant to be the step is exactly what a divergence puts in question, and receivedLine's heuristic answers it one way for a prompt and the wrong way for a transposed line. The caret is printed only when there is a single line, where there is nothing left to assert.

Also

A run that could not be started now says so on stderr instead of only through a logger that is off by default. The comparison names it as one that could not be played, which invites the question.

Not done as proposed

The issue suggested a reporter alongside SingleRunReporter in internal/reporters. That package no longer exists, and the comparison has to be callable from runs.go — a separate package would import internal/play and internal/play would import it back. It lives in internal/play.

Checks

make format, make lint, PRESCRIPT_REQUIRE_RUNNERS=1 make test, make build_dev, make examples all pass. internal/play/comparison_test.go covers grouping, ordering, the single-line marker, a failing reference, an incomparable transcript, an unplayable port, and column alignment; the report shapes were also checked end to end against a five-run script.

🤖 Generated with Claude Code

https://claude.ai/code/session_019f9gkZVUEo3jtv51GSb4vJ

A script with more than one run is asking where the implementations stop
agreeing, and until now it answered with a stack of separate failure
reports and a tally of how many failed. Doing the comparison was left to
whoever read them.

It now ends with the comparison itself: every port lined up against the
first run at the step where it left it, with both sides quoted under one
column so the difference is read down the page.

The divergence becomes a value rather than a string built where the
failure happened, which is what makes aligning several runs possible at
all -- a step has to be a number something else can group by. Ports that
left at the same step, the same way, printing the same thing are one
finding: a dozen ports of the same BASIC program missing a leading space
is one bug repeated twelve times, and a report that says it twelve times
is one nobody reads to the end.

Three things are deliberately not divergences. A port that plays a
different transcript is not compared at all, because step 7 of one
transcript has nothing to do with step 7 of another. A port that never
ran has not disagreed with anything. And when the reference no longer
matches its own transcript, nothing is compared: the environment has
moved, and every port's divergence would be measured against something
that is itself wrong.

A run that could not be started now says so on stderr rather than only
through a logger that is off by default -- the comparison names it as
one that could not be played, which invites the question.

Closes #16

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019f9gkZVUEo3jtv51GSb4vJ
@rjnienaber
rjnienaber merged commit d815086 into main Sep 7, 2026
5 checks passed
@rjnienaber
rjnienaber deleted the feat/diff-reporter branch September 7, 2026 21:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Diff reporter: compare implementations aligned by step

1 participant