Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 13 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -283,7 +283,19 @@ could not be played: go
```

A port that plays a different transcript is not compared, and neither is one
that never ran — neither of them has disagreed with anything. See
that never ran — neither of them has disagreed with anything.

Each finding is classified by two things that can be worked out without reading
it, and they appear as fixed tokens beside the failure mode:

```
step 5 of 12 (no-match, formatting, no-draws)
```

`formatting` means the two sides agree once whitespace is collapsed — BASIC's
spacing around numbers, which ports get wrong constantly and which belongs in
one report per language rather than one per program. `no-draws` means the port
had taken nothing from the tape, so randomness is not involved. See
[docs/comparison.md](docs/comparison.md).

### Development
Expand Down
62 changes: 62 additions & 0 deletions docs/comparison.md
Original file line number Diff line number Diff line change
Expand Up @@ -98,6 +98,68 @@ after all 12 steps (hung)
ruby the executable had not exited after 30s
```

## Classification

Not every divergence is a bug worth filing, and two things about one can be
worked out without a person reading it. Both appear as fixed tokens in the
finding's header, next to the failure mode:

```
step 5 of 12 (no-match, formatting, no-draws)
```

They are tokens rather than prose so that a few hundred runs can be counted and
sorted by them, and what is *not* known is left out rather than named — a token
that appears on every line carries nothing.

### Formatting or semantic

Collapse every run of whitespace to one space, trim the ends, and compare
again. If the two sides now agree, the divergence is `formatting`; otherwise it
is `semantic`.

This is worth separating for a reason beyond tidiness. BASIC's `PRINT` puts a
space before a non-negative number and another after it, and `,` tabs to
fourteen-column zones. Ports get this wrong constantly and identically, and
filing each one separately would mean several hundred near-identical issues
against the same repository — which would be ignored, reasonably. Formatting
divergences belong together, one report per language; a semantic one is about a
particular program and belongs on its own.

The received lines are joined before normalising, so a port that broke one
logical line across two prints is `formatting` — where a line breaks is layout
by any reading. Case is not touched: a port that shouts where the reference
whispers has changed the text, not its layout, and is `semantic`.

A step matched through a [redaction](script-format.md#redactions) gets neither
token. The expected side is a pattern rather than a line, so there is nothing
to normalise and no honest answer to give.

### What the tape says

When a run is played with `--tape`, prescript asks the shim to record how many
values it drew, and reads the count back afterwards — no variable to set, and
the number is what had been drawn when the run *stopped*, not when it exited,
so a run killed for hanging still reports one.

| Token | What it means | What to do |
| --- | --- | --- |
| `no-draws` | the ports had taken nothing from the tape | randomness is not involved; file it |
| `same-draws` | the ports and the reference drew the same amount | they asked in the same places and disagreed about the answer; file it |
| `different-draws` | known counts that disagree, with the reference or with each other | the port consumes randomness in a different order; a person has to judge it |
| *absent* | not enough counts are known | — |

Two ports that printed the same thing at the same step having drawn different
amounts is itself a finding, and says so without the reference.

**The reference interpreter does not report its draws.** Vintage BASIC has no
shim — a tape is *recorded from* it rather than replayed into it — so on a
corpus run the reference's count is unknown and `same-draws` is out of reach.
What survives is `no-draws`, which needs only the port, and disagreement
between ports. The full table applies when the reference is itself a taped
port, such as comparing Ruby against Python. Instrumenting the interpreter is
[#40](https://github.com/rjnienaber/prescript/issues/40).

## Ports that were not compared

Three things are not divergences, and saying they were would be the kind of
Expand Down
25 changes: 21 additions & 4 deletions docs/determinism.md
Original file line number Diff line number Diff line change
Expand Up @@ -207,10 +207,27 @@ finding — the thing seeding hides behind numbers that are stable but different
Starting the tape again would turn a clear answer into a transcript that merely
looks wrong somewhere further down.

For the same reason a shim writes down how much it used, when asked. Set
`PRESCRIPT_TAPE_USAGE` to a path and it holds the number of values drawn after
the program exits, which is what separates "this port has a logic bug" from
"this port draws in a different order".
For the same reason a shim writes down how much it used. `PRESCRIPT_TAPE_USAGE`
names a path, and the file holds the number of values drawn — which is what
separates "this port has a logic bug" from "this port draws in a different
order", and is one of the two axes a divergence is
[classified](comparison.md#what-the-tape-says) by.

`prescript play --tape` sets that variable for itself, so the count is
collected on every taped run rather than only when somebody remembered to ask.
A run that names the variable itself keeps its own path.

The count is rewritten after every draw rather than once on the way out. What
matters is how much had been drawn when the run stopped agreeing with the
transcript, and a run that diverged by hanging is killed — which fires no exit
handler in any of these languages. Rewriting in place needs no truncation: the
count only grows, so its decimal form only gets longer and each write covers
the last.

The file is also written before the first draw, so a port that drew nothing
says so. Nothing at all is written when there is no tape, and unknown must not
be read as zero: a port that drew nothing has said randomness is not implicated
in its divergence, and one that cannot say has said nothing.

### What a tape does not fix

Expand Down
170 changes: 170 additions & 0 deletions internal/play/classification.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,170 @@
package play

import "strings"

// Shape says whether a divergence is about what the program computed or only
// about how it laid the answer out.
//
// The distinction decides where a finding goes rather than how bad it is.
// BASIC's PRINT puts a space before a non-negative number and another after
// it, and `,` tabs to fourteen-column zones; ports get this wrong constantly
// and identically. Filing each one separately would mean several hundred
// near-identical issues against the same repository, which would be ignored,
// reasonably. Formatting divergences belong together, one report per language;
// a semantic one is about a particular program and belongs on its own.
type Shape string

const (
// Formatting: the two lines are the same once every run of whitespace is
// collapsed. Nothing the program computed is in dispute.
Formatting Shape = "formatting"

// Semantic: they still differ. Different words, different numbers,
// different case -- something other than layout.
Semantic Shape = "semantic"

// UnknownShape: there is nothing to normalise. The step matched by
// pattern, so the expected side is a regular expression rather than a
// line, or the run got through every step and went wrong afterwards.
UnknownShape Shape = ""
)

// DrawUse says what the tape can be made to tell us about whether randomness
// is involved in a divergence at all.
//
// A port that had drawn nothing when it diverged cannot be diverging because
// of the numbers it drew, whatever else is wrong with it. One that drew
// exactly what the reference drew asked for randomness in the same places, so
// the difference is in what it did with the values. One that drew a different
// amount has restructured how it consumes randomness, and no report can say
// whether that is a bug or a legitimate way to write the same program.
type DrawUse string

const (
// NoDraws: every port in the finding had taken nothing from the tape.
NoDraws DrawUse = "no-draws"

// SameDraws: the ports and the reference had drawn the same amount.
SameDraws DrawUse = "same-draws"

// DifferentDraws: known counts that disagree -- with the reference, or
// with each other.
DifferentDraws DrawUse = "different-draws"

// UnknownDraws: not enough of the counts are known to say. No tape was
// played, or the runtime has no shim to replay one.
UnknownDraws DrawUse = ""
)

// Classification is what can be worked out about a divergence without a person
// reading it. Both axes are computed, neither is a verdict: they say which
// findings can be filed with confidence and which need somebody to look.
type Classification struct {
Shape Shape
Draws DrawUse
}

// Tokens is the classification as it appears in a report, in a fixed
// vocabulary and a fixed order, so a corpus of a few hundred runs can be
// counted and sorted by it. What is not known is left out rather than named as
// unknown, because on most runs it is not known and a report that says so on
// every line has stopped carrying information.
func (classification Classification) Tokens() []string {
var tokens []string
if classification.Shape != UnknownShape {
tokens = append(tokens, string(classification.Shape))
}
if classification.Draws != UnknownDraws {
tokens = append(tokens, string(classification.Draws))
}
return tokens
}

// classify works out both axes for one finding. reference is what the
// reference run drew, and ports what each of the ports in this finding drew;
// either may be nil, meaning it could not say.
func (group DivergenceGroup) classify(reference *int, ports []*int) Classification {
return Classification{Shape: group.shape(), Draws: drawUse(reference, ports)}
}

// shape normalises both sides and compares them again.
//
// The received lines are joined before normalising, so a port that broke one
// logical line across two prints is still formatting rather than semantic --
// where the line breaks is layout by any reading.
func (group DivergenceGroup) shape() Shape {
if group.Matched || group.Redacted {
return UnknownShape
}

if normalise(group.Expected) == normalise(strings.Join(group.Received, " ")) {
return Formatting
}
return Semantic
}

// normalise collapses every run of whitespace to a single space and trims the
// ends, which is the whole of what "formatting" means here. Case is not
// touched: a port that shouts where the reference whispers has changed the
// text, not its layout.
func normalise(line string) string {
return strings.Join(strings.Fields(line), " ")
}

func drawUse(reference *int, ports []*int) DrawUse {
drawn, known := agreed(ports)
if !known {
// Ports in one finding printed the same thing at the same step, so
// counts that disagree mean they got there by drawing differently --
// which is the thing this axis exists to notice, whether or not the
// reference can say what it drew.
if disagree(ports) {
return DifferentDraws
}
return UnknownDraws
}

if drawn == 0 {
// True without the reference: a port that has drawn nothing is not
// diverging over the numbers it drew.
return NoDraws
}

if reference == nil {
return UnknownDraws
}

if *reference == drawn {
return SameDraws
}
return DifferentDraws
}

// agreed is the count every port in the finding reported, and whether they all
// reported one and all reported the same.
func agreed(ports []*int) (int, bool) {
if len(ports) == 0 || ports[0] == nil {
return 0, false
}

for _, drawn := range ports[1:] {
if drawn == nil || *drawn != *ports[0] {
return 0, false
}
}
return *ports[0], true
}

func disagree(ports []*int) bool {
var seen *int
for _, drawn := range ports {
if drawn == nil {
continue
}
if seen != nil && *seen != *drawn {
return true
}
seen = drawn
}
return false
}
39 changes: 33 additions & 6 deletions internal/play/comparison.go
Original file line number Diff line number Diff line change
Expand Up @@ -61,6 +61,15 @@ type DivergenceGroup struct {
Matched bool

Ports []string

// Classification is what could be worked out about the finding without a
// person reading it: whether it is only about layout, and what the tape
// says about whether randomness is involved.
Classification Classification

// portDraws is what each port in Ports drew, in the same order, kept only
// long enough to classify the finding once every port has joined it.
portDraws []*int
}

// Compare lines up what each run did against the reference's transcript. It
Expand Down Expand Up @@ -88,10 +97,18 @@ func Compare(runs []script.Run, outcomes []Outcome) Comparison {
case outcome.Failure == nil:
comparison.NotPlayed = append(comparison.NotPlayed, outcome.Name)
default:
comparison.add(index, outcome.Name, *outcome.Failure)
comparison.add(index, outcome, *outcome.Failure)
}
}

// Classified once every port has joined its finding, because both axes are
// about the finding rather than about one port: ports that printed the
// same thing having drawn different amounts is itself the answer.
for at := range comparison.Groups {
group := &comparison.Groups[at]
group.Classification = group.classify(outcomes[0].Draws, group.portDraws)
}

// Earliest step first, and within a step the order the ports appear in the
// script. A later divergence is often a consequence of an earlier one --
// the port that got the prompt wrong is the port that then answered the
Expand All @@ -104,7 +121,7 @@ func Compare(runs []script.Run, outcomes []Outcome) Comparison {
return comparison
}

func (comparison *Comparison) add(index map[string]int, port string, divergence Divergence) {
func (comparison *Comparison) add(index map[string]int, outcome Outcome, divergence Divergence) {
group := DivergenceGroup{
StepIndex: divergence.StepIndex,
StepCount: divergence.StepCount,
Expand All @@ -121,11 +138,13 @@ func (comparison *Comparison) add(index map[string]int, port string, divergence
key := fmt.Sprintf("%d\x00%s\x00%s", group.StepIndex, group.Mode,
strings.Join(group.Received, "\x00"))
if at, seen := index[key]; seen {
comparison.Groups[at].Ports = append(comparison.Groups[at].Ports, port)
comparison.Groups[at].Ports = append(comparison.Groups[at].Ports, outcome.Name)
comparison.Groups[at].portDraws = append(comparison.Groups[at].portDraws, outcome.Draws)
return
}

group.Ports = []string{port}
group.Ports = []string{outcome.Name}
group.portDraws = []*int{outcome.Draws}
index[key] = len(comparison.Groups)
comparison.Groups = append(comparison.Groups, group)
}
Expand Down Expand Up @@ -184,6 +203,14 @@ func (comparison Comparison) Report() string {
return report.String()
}

// headline is what went wrong followed by what could be worked out about it,
// in one parenthesised list of fixed tokens: `no-match, formatting, no-draws`.
// One line rather than a second one under it, because a reader scanning a
// report with twenty findings in it is deciding which to stop at.
func (group DivergenceGroup) headline() string {
return strings.Join(append([]string{string(group.Mode)}, group.Classification.Tokens()...), ", ")
}

func (comparison Comparison) labelWidth() int {
width := len(comparison.Reference)
for _, group := range comparison.Groups {
Expand Down Expand Up @@ -211,12 +238,12 @@ func (group DivergenceGroup) report(reference string, width int) string {

var report strings.Builder
if group.Matched {
fmt.Fprintf(&report, "after all %d steps (%s)\n", group.StepCount, group.Mode)
fmt.Fprintf(&report, "after all %d steps (%s)\n", group.StepCount, group.headline())
fmt.Fprintf(&report, " %-*s %s\n", width, ports, strings.Join(group.Received, " "))
return report.String()
}

fmt.Fprintf(&report, "step %d of %d (%s)\n", group.StepIndex+1, group.StepCount, group.Mode)
fmt.Fprintf(&report, "step %d of %d (%s)\n", group.StepIndex+1, group.StepCount, group.headline())
if group.Redacted {
fmt.Fprintf(&report, " %-*s %q (with redactions applied)\n", width, reference, group.Expected)
} else {
Expand Down
Loading