diff --git a/README.md b/README.md index 3e1c2dc..e0aaa32 100644 --- a/README.md +++ b/README.md @@ -283,7 +283,19 @@ could not be played: go ``` A port that plays a different transcript is not compared, and neither is one -that never ran — neither of them has disagreed with anything. See +that never ran — neither of them has disagreed with anything. + +Each finding is classified by two things that can be worked out without reading +it, and they appear as fixed tokens beside the failure mode: + +``` +step 5 of 12 (no-match, formatting, no-draws) +``` + +`formatting` means the two sides agree once whitespace is collapsed — BASIC's +spacing around numbers, which ports get wrong constantly and which belongs in +one report per language rather than one per program. `no-draws` means the port +had taken nothing from the tape, so randomness is not involved. See [docs/comparison.md](docs/comparison.md). ### Development diff --git a/docs/comparison.md b/docs/comparison.md index 14e1ad1..01ce449 100644 --- a/docs/comparison.md +++ b/docs/comparison.md @@ -98,6 +98,68 @@ after all 12 steps (hung) ruby the executable had not exited after 30s ``` +## Classification + +Not every divergence is a bug worth filing, and two things about one can be +worked out without a person reading it. Both appear as fixed tokens in the +finding's header, next to the failure mode: + +``` +step 5 of 12 (no-match, formatting, no-draws) +``` + +They are tokens rather than prose so that a few hundred runs can be counted and +sorted by them, and what is *not* known is left out rather than named — a token +that appears on every line carries nothing. + +### Formatting or semantic + +Collapse every run of whitespace to one space, trim the ends, and compare +again. If the two sides now agree, the divergence is `formatting`; otherwise it +is `semantic`. + +This is worth separating for a reason beyond tidiness. BASIC's `PRINT` puts a +space before a non-negative number and another after it, and `,` tabs to +fourteen-column zones. Ports get this wrong constantly and identically, and +filing each one separately would mean several hundred near-identical issues +against the same repository — which would be ignored, reasonably. Formatting +divergences belong together, one report per language; a semantic one is about a +particular program and belongs on its own. + +The received lines are joined before normalising, so a port that broke one +logical line across two prints is `formatting` — where a line breaks is layout +by any reading. Case is not touched: a port that shouts where the reference +whispers has changed the text, not its layout, and is `semantic`. + +A step matched through a [redaction](script-format.md#redactions) gets neither +token. The expected side is a pattern rather than a line, so there is nothing +to normalise and no honest answer to give. + +### What the tape says + +When a run is played with `--tape`, prescript asks the shim to record how many +values it drew, and reads the count back afterwards — no variable to set, and +the number is what had been drawn when the run *stopped*, not when it exited, +so a run killed for hanging still reports one. + +| Token | What it means | What to do | +| --- | --- | --- | +| `no-draws` | the ports had taken nothing from the tape | randomness is not involved; file it | +| `same-draws` | the ports and the reference drew the same amount | they asked in the same places and disagreed about the answer; file it | +| `different-draws` | known counts that disagree, with the reference or with each other | the port consumes randomness in a different order; a person has to judge it | +| *absent* | not enough counts are known | — | + +Two ports that printed the same thing at the same step having drawn different +amounts is itself a finding, and says so without the reference. + +**The reference interpreter does not report its draws.** Vintage BASIC has no +shim — a tape is *recorded from* it rather than replayed into it — so on a +corpus run the reference's count is unknown and `same-draws` is out of reach. +What survives is `no-draws`, which needs only the port, and disagreement +between ports. The full table applies when the reference is itself a taped +port, such as comparing Ruby against Python. Instrumenting the interpreter is +[#40](https://github.com/rjnienaber/prescript/issues/40). + ## Ports that were not compared Three things are not divergences, and saying they were would be the kind of diff --git a/docs/determinism.md b/docs/determinism.md index 7378686..b4bf4b5 100644 --- a/docs/determinism.md +++ b/docs/determinism.md @@ -207,10 +207,27 @@ finding — the thing seeding hides behind numbers that are stable but different Starting the tape again would turn a clear answer into a transcript that merely looks wrong somewhere further down. -For the same reason a shim writes down how much it used, when asked. Set -`PRESCRIPT_TAPE_USAGE` to a path and it holds the number of values drawn after -the program exits, which is what separates "this port has a logic bug" from -"this port draws in a different order". +For the same reason a shim writes down how much it used. `PRESCRIPT_TAPE_USAGE` +names a path, and the file holds the number of values drawn — which is what +separates "this port has a logic bug" from "this port draws in a different +order", and is one of the two axes a divergence is +[classified](comparison.md#what-the-tape-says) by. + +`prescript play --tape` sets that variable for itself, so the count is +collected on every taped run rather than only when somebody remembered to ask. +A run that names the variable itself keeps its own path. + +The count is rewritten after every draw rather than once on the way out. What +matters is how much had been drawn when the run stopped agreeing with the +transcript, and a run that diverged by hanging is killed — which fires no exit +handler in any of these languages. Rewriting in place needs no truncation: the +count only grows, so its decimal form only gets longer and each write covers +the last. + +The file is also written before the first draw, so a port that drew nothing +says so. Nothing at all is written when there is no tape, and unknown must not +be read as zero: a port that drew nothing has said randomness is not implicated +in its divergence, and one that cannot say has said nothing. ### What a tape does not fix diff --git a/internal/play/classification.go b/internal/play/classification.go new file mode 100644 index 0000000..3258f07 --- /dev/null +++ b/internal/play/classification.go @@ -0,0 +1,170 @@ +package play + +import "strings" + +// Shape says whether a divergence is about what the program computed or only +// about how it laid the answer out. +// +// The distinction decides where a finding goes rather than how bad it is. +// BASIC's PRINT puts a space before a non-negative number and another after +// it, and `,` tabs to fourteen-column zones; ports get this wrong constantly +// and identically. Filing each one separately would mean several hundred +// near-identical issues against the same repository, which would be ignored, +// reasonably. Formatting divergences belong together, one report per language; +// a semantic one is about a particular program and belongs on its own. +type Shape string + +const ( + // Formatting: the two lines are the same once every run of whitespace is + // collapsed. Nothing the program computed is in dispute. + Formatting Shape = "formatting" + + // Semantic: they still differ. Different words, different numbers, + // different case -- something other than layout. + Semantic Shape = "semantic" + + // UnknownShape: there is nothing to normalise. The step matched by + // pattern, so the expected side is a regular expression rather than a + // line, or the run got through every step and went wrong afterwards. + UnknownShape Shape = "" +) + +// DrawUse says what the tape can be made to tell us about whether randomness +// is involved in a divergence at all. +// +// A port that had drawn nothing when it diverged cannot be diverging because +// of the numbers it drew, whatever else is wrong with it. One that drew +// exactly what the reference drew asked for randomness in the same places, so +// the difference is in what it did with the values. One that drew a different +// amount has restructured how it consumes randomness, and no report can say +// whether that is a bug or a legitimate way to write the same program. +type DrawUse string + +const ( + // NoDraws: every port in the finding had taken nothing from the tape. + NoDraws DrawUse = "no-draws" + + // SameDraws: the ports and the reference had drawn the same amount. + SameDraws DrawUse = "same-draws" + + // DifferentDraws: known counts that disagree -- with the reference, or + // with each other. + DifferentDraws DrawUse = "different-draws" + + // UnknownDraws: not enough of the counts are known to say. No tape was + // played, or the runtime has no shim to replay one. + UnknownDraws DrawUse = "" +) + +// Classification is what can be worked out about a divergence without a person +// reading it. Both axes are computed, neither is a verdict: they say which +// findings can be filed with confidence and which need somebody to look. +type Classification struct { + Shape Shape + Draws DrawUse +} + +// Tokens is the classification as it appears in a report, in a fixed +// vocabulary and a fixed order, so a corpus of a few hundred runs can be +// counted and sorted by it. What is not known is left out rather than named as +// unknown, because on most runs it is not known and a report that says so on +// every line has stopped carrying information. +func (classification Classification) Tokens() []string { + var tokens []string + if classification.Shape != UnknownShape { + tokens = append(tokens, string(classification.Shape)) + } + if classification.Draws != UnknownDraws { + tokens = append(tokens, string(classification.Draws)) + } + return tokens +} + +// classify works out both axes for one finding. reference is what the +// reference run drew, and ports what each of the ports in this finding drew; +// either may be nil, meaning it could not say. +func (group DivergenceGroup) classify(reference *int, ports []*int) Classification { + return Classification{Shape: group.shape(), Draws: drawUse(reference, ports)} +} + +// shape normalises both sides and compares them again. +// +// The received lines are joined before normalising, so a port that broke one +// logical line across two prints is still formatting rather than semantic -- +// where the line breaks is layout by any reading. +func (group DivergenceGroup) shape() Shape { + if group.Matched || group.Redacted { + return UnknownShape + } + + if normalise(group.Expected) == normalise(strings.Join(group.Received, " ")) { + return Formatting + } + return Semantic +} + +// normalise collapses every run of whitespace to a single space and trims the +// ends, which is the whole of what "formatting" means here. Case is not +// touched: a port that shouts where the reference whispers has changed the +// text, not its layout. +func normalise(line string) string { + return strings.Join(strings.Fields(line), " ") +} + +func drawUse(reference *int, ports []*int) DrawUse { + drawn, known := agreed(ports) + if !known { + // Ports in one finding printed the same thing at the same step, so + // counts that disagree mean they got there by drawing differently -- + // which is the thing this axis exists to notice, whether or not the + // reference can say what it drew. + if disagree(ports) { + return DifferentDraws + } + return UnknownDraws + } + + if drawn == 0 { + // True without the reference: a port that has drawn nothing is not + // diverging over the numbers it drew. + return NoDraws + } + + if reference == nil { + return UnknownDraws + } + + if *reference == drawn { + return SameDraws + } + return DifferentDraws +} + +// agreed is the count every port in the finding reported, and whether they all +// reported one and all reported the same. +func agreed(ports []*int) (int, bool) { + if len(ports) == 0 || ports[0] == nil { + return 0, false + } + + for _, drawn := range ports[1:] { + if drawn == nil || *drawn != *ports[0] { + return 0, false + } + } + return *ports[0], true +} + +func disagree(ports []*int) bool { + var seen *int + for _, drawn := range ports { + if drawn == nil { + continue + } + if seen != nil && *seen != *drawn { + return true + } + seen = drawn + } + return false +} diff --git a/internal/play/comparison.go b/internal/play/comparison.go index 11b8c8b..cadded6 100644 --- a/internal/play/comparison.go +++ b/internal/play/comparison.go @@ -61,6 +61,15 @@ type DivergenceGroup struct { Matched bool Ports []string + + // Classification is what could be worked out about the finding without a + // person reading it: whether it is only about layout, and what the tape + // says about whether randomness is involved. + Classification Classification + + // portDraws is what each port in Ports drew, in the same order, kept only + // long enough to classify the finding once every port has joined it. + portDraws []*int } // Compare lines up what each run did against the reference's transcript. It @@ -88,10 +97,18 @@ func Compare(runs []script.Run, outcomes []Outcome) Comparison { case outcome.Failure == nil: comparison.NotPlayed = append(comparison.NotPlayed, outcome.Name) default: - comparison.add(index, outcome.Name, *outcome.Failure) + comparison.add(index, outcome, *outcome.Failure) } } + // Classified once every port has joined its finding, because both axes are + // about the finding rather than about one port: ports that printed the + // same thing having drawn different amounts is itself the answer. + for at := range comparison.Groups { + group := &comparison.Groups[at] + group.Classification = group.classify(outcomes[0].Draws, group.portDraws) + } + // Earliest step first, and within a step the order the ports appear in the // script. A later divergence is often a consequence of an earlier one -- // the port that got the prompt wrong is the port that then answered the @@ -104,7 +121,7 @@ func Compare(runs []script.Run, outcomes []Outcome) Comparison { return comparison } -func (comparison *Comparison) add(index map[string]int, port string, divergence Divergence) { +func (comparison *Comparison) add(index map[string]int, outcome Outcome, divergence Divergence) { group := DivergenceGroup{ StepIndex: divergence.StepIndex, StepCount: divergence.StepCount, @@ -121,11 +138,13 @@ func (comparison *Comparison) add(index map[string]int, port string, divergence key := fmt.Sprintf("%d\x00%s\x00%s", group.StepIndex, group.Mode, strings.Join(group.Received, "\x00")) if at, seen := index[key]; seen { - comparison.Groups[at].Ports = append(comparison.Groups[at].Ports, port) + comparison.Groups[at].Ports = append(comparison.Groups[at].Ports, outcome.Name) + comparison.Groups[at].portDraws = append(comparison.Groups[at].portDraws, outcome.Draws) return } - group.Ports = []string{port} + group.Ports = []string{outcome.Name} + group.portDraws = []*int{outcome.Draws} index[key] = len(comparison.Groups) comparison.Groups = append(comparison.Groups, group) } @@ -184,6 +203,14 @@ func (comparison Comparison) Report() string { return report.String() } +// headline is what went wrong followed by what could be worked out about it, +// in one parenthesised list of fixed tokens: `no-match, formatting, no-draws`. +// One line rather than a second one under it, because a reader scanning a +// report with twenty findings in it is deciding which to stop at. +func (group DivergenceGroup) headline() string { + return strings.Join(append([]string{string(group.Mode)}, group.Classification.Tokens()...), ", ") +} + func (comparison Comparison) labelWidth() int { width := len(comparison.Reference) for _, group := range comparison.Groups { @@ -211,12 +238,12 @@ func (group DivergenceGroup) report(reference string, width int) string { var report strings.Builder if group.Matched { - fmt.Fprintf(&report, "after all %d steps (%s)\n", group.StepCount, group.Mode) + fmt.Fprintf(&report, "after all %d steps (%s)\n", group.StepCount, group.headline()) fmt.Fprintf(&report, " %-*s %s\n", width, ports, strings.Join(group.Received, " ")) return report.String() } - fmt.Fprintf(&report, "step %d of %d (%s)\n", group.StepIndex+1, group.StepCount, group.Mode) + fmt.Fprintf(&report, "step %d of %d (%s)\n", group.StepIndex+1, group.StepCount, group.headline()) if group.Redacted { fmt.Fprintf(&report, " %-*s %q (with redactions applied)\n", width, reference, group.Expected) } else { diff --git a/internal/play/comparison_test.go b/internal/play/comparison_test.go index b45be0e..8b2356a 100644 --- a/internal/play/comparison_test.go +++ b/internal/play/comparison_test.go @@ -109,7 +109,7 @@ func TestReportShowsBothSidesAgainstTheReference(t *testing.T) { report := Compare(runs, outcomes).Report() assert.Contains(t, report, "=== comparison against basic ===") - assert.Contains(t, report, "step 2 of 2 (no-match)") + assert.Contains(t, report, "step 2 of 2 (no-match, formatting)") assert.Contains(t, report, ` basic " 7 807 "`) assert.Contains(t, report, ` ruby "7 807"`) assert.Contains(t, report, "^ first difference") @@ -240,3 +240,145 @@ func TestColumnsLineUpAcrossFindings(t *testing.T) { } assert.Len(t, columns, 1) } + +func drew(count int) *int { return &count } + +// BASIC's PRINT puts a space before a non-negative number and another after +// it, and ports get that wrong constantly. Told apart from a wrong number, the +// whole class can be filed once per language instead of once per program. +func TestWhitespaceOnlyDivergenceIsFormatting(t *testing.T) { + t.Parallel() + + runs := []script.Run{comparableRun("basic"), comparableRun("ruby")} + outcomes := []Outcome{matchedOutcome("basic"), divergedOutcome("ruby", atStep(1, "7 807"))} + + comparison := Compare(runs, outcomes) + + assert.Equal(t, Formatting, comparison.Groups[0].Classification.Shape) +} + +// Where a line breaks is layout by any reading, so a port that split one +// logical line across two prints is not accused of computing something else. +func TestALineBrokenInTwoIsStillFormatting(t *testing.T) { + t.Parallel() + + runs := []script.Run{comparableRun("basic"), comparableRun("ruby")} + outcomes := []Outcome{matchedOutcome("basic"), divergedOutcome("ruby", atStep(1, "7", "807"))} + + comparison := Compare(runs, outcomes) + + assert.Equal(t, Formatting, comparison.Groups[0].Classification.Shape) +} + +// Case is text, not layout. A port that shouts where the reference whispers +// has changed what it printed. +func TestADifferentNumberOrDifferentCaseIsSemantic(t *testing.T) { + t.Parallel() + + for _, received := range []string{" 7 806 ", " 7 eight-oh-seven "} { + runs := []script.Run{comparableRun("basic"), comparableRun("ruby")} + outcomes := []Outcome{matchedOutcome("basic"), divergedOutcome("ruby", atStep(1, received))} + + comparison := Compare(runs, outcomes) + + assert.Equal(t, Semantic, comparison.Groups[0].Classification.Shape, received) + } +} + +// A pattern is not a line, so there is nothing to normalise and no honest +// answer to give. +func TestARedactedStepIsNotClassifiedByShape(t *testing.T) { + t.Parallel() + + divergence := atStep(1, "7 807") + divergence.Redacted = true + runs := []script.Run{comparableRun("basic"), comparableRun("ruby")} + outcomes := []Outcome{matchedOutcome("basic"), divergedOutcome("ruby", divergence)} + + comparison := Compare(runs, outcomes) + + assert.Equal(t, UnknownShape, comparison.Groups[0].Classification.Shape) + assert.NotContains(t, comparison.Report(), "semantic") +} + +// A port that had drawn nothing cannot be diverging over the numbers it drew, +// and that is true whether or not the reference can say what it drew. +func TestAPortThatDrewNothingSaysSoWithoutTheReference(t *testing.T) { + t.Parallel() + + ruby := divergedOutcome("ruby", atStep(1, "seven")) + ruby.Draws = drew(0) + runs := []script.Run{comparableRun("basic"), comparableRun("ruby")} + outcomes := []Outcome{matchedOutcome("basic"), ruby} + + comparison := Compare(runs, outcomes) + + assert.Equal(t, NoDraws, comparison.Groups[0].Classification.Draws) + assert.Contains(t, comparison.Report(), "(no-match, semantic, no-draws)") +} + +// Same numbers asked for in the same places: the difference is in what the +// port did with them, which is a logic bug worth filing. +func TestDrawingAsMuchAsTheReferenceIsALogicBug(t *testing.T) { + t.Parallel() + + reference := matchedOutcome("basic") + reference.Draws = drew(12) + ruby := divergedOutcome("ruby", atStep(1, "seven")) + ruby.Draws = drew(12) + + comparison := Compare([]script.Run{comparableRun("basic"), comparableRun("ruby")}, + []Outcome{reference, ruby}) + + assert.Equal(t, SameDraws, comparison.Groups[0].Classification.Draws) +} + +func TestDrawingADifferentAmountFromTheReferenceNeedsAHuman(t *testing.T) { + t.Parallel() + + reference := matchedOutcome("basic") + reference.Draws = drew(12) + ruby := divergedOutcome("ruby", atStep(1, "seven")) + ruby.Draws = drew(9) + + comparison := Compare([]script.Run{comparableRun("basic"), comparableRun("ruby")}, + []Outcome{reference, ruby}) + + assert.Equal(t, DifferentDraws, comparison.Groups[0].Classification.Draws) +} + +// Two ports printing the same thing at the same step having drawn different +// amounts is itself the finding, and it does not need the reference to say it. +func TestPortsInOneFindingThatDrewDifferentAmounts(t *testing.T) { + t.Parallel() + + ruby := divergedOutcome("ruby", atStep(1, "seven")) + ruby.Draws = drew(4) + python := divergedOutcome("python", atStep(1, "seven")) + python.Draws = drew(9) + + comparison := Compare( + []script.Run{comparableRun("basic"), comparableRun("ruby"), comparableRun("python")}, + []Outcome{matchedOutcome("basic"), ruby, python}) + + assert.Len(t, comparison.Groups, 1) + assert.Equal(t, DifferentDraws, comparison.Groups[0].Classification.Draws) +} + +// The reference interpreter does not report its draws, so most real runs +// cannot answer this axis. Saying "unknown" on every line of every report +// would be noise, and a token that is always there carries nothing. +func TestAnUnknownDrawCountIsLeftOutRatherThanNamed(t *testing.T) { + t.Parallel() + + ruby := divergedOutcome("ruby", atStep(1, "seven")) + ruby.Draws = drew(9) + + comparison := Compare([]script.Run{comparableRun("basic"), comparableRun("ruby")}, + []Outcome{matchedOutcome("basic"), ruby}) + + assert.Equal(t, UnknownDraws, comparison.Groups[0].Classification.Draws) + report := comparison.Report() + assert.Contains(t, report, "(no-match, semantic)") + assert.NotContains(t, report, "unknown") +} diff --git a/internal/play/divergence.go b/internal/play/divergence.go index e5405f3..406580d 100644 --- a/internal/play/divergence.go +++ b/internal/play/divergence.go @@ -71,6 +71,13 @@ type Outcome struct { Name string ExitCode int Failure *Divergence + + // Draws is how many values this run took from the tape by the time it + // stopped, or nil when that is unknown -- no tape, or a runtime with no + // shim to replay one. Unknown is not zero: a port that drew nothing has + // told us randomness is not implicated in its divergence, and one that + // cannot say has told us nothing. + Draws *int } // Report explains a divergence to whoever is watching, in a form meant to be diff --git a/internal/play/play.go b/internal/play/play.go index 734ab18..0b8f4ac 100644 --- a/internal/play/play.go +++ b/internal/play/play.go @@ -6,6 +6,8 @@ import ( "os" "os/exec" "path/filepath" + "strconv" + "strings" "time" cfg "github.com/rjnienaber/prescript/internal/config" @@ -63,6 +65,9 @@ func getTerminal(config cfg.PlayConfig, run script.Run) (utils.Terminal, error) // every port. const tapeVariable = "PRESCRIPT_TAPE" +// usageVariable is where a shim writes how many values it drew. +const usageVariable = "PRESCRIPT_TAPE_USAGE" + // childEnvironment is the environment the executable is started with, plus the // tape if one was named. // @@ -76,7 +81,7 @@ const tapeVariable = "PRESCRIPT_TAPE" // The path is made absolute because the value outlives this process's idea of // where it is: it is read by a shim, inside an interpreter, started by a // runner that may have been written anywhere. -func childEnvironment(config cfg.PlayConfig, run script.Run) ([]string, error) { +func childEnvironment(config cfg.PlayConfig, run script.Run, usage string) ([]string, error) { environment := run.Environment(os.Environ()) if config.Tape == "" { return environment, nil @@ -97,7 +102,65 @@ func childEnvironment(config cfg.PlayConfig, run script.Run) ([]string, error) { if environment == nil { environment = os.Environ() } - return append(environment, tapeVariable+"="+tape), nil + environment = append(environment, tapeVariable+"="+tape) + + if usage != "" { + environment = append(environment, usageVariable+"="+usage) + } + return environment, nil +} + +// tapeUsagePath is where the shim is told to write how much of the tape it +// drew. Prescript asks for it rather than waiting to be told, because the +// count is what separates a port with a logic bug from one that consumes +// randomness in a different order, and a classification that only happens when +// somebody remembered to set a variable is one that mostly does not happen. +// +// A run that names its own path keeps it: a script or a test that wants the +// file afterwards has said where to leave it. +func tapeUsagePath(config cfg.PlayConfig, run script.Run) (string, func()) { + nothing := func() {} + if config.Tape == "" { + return "", nothing + } + + for _, declared := range run.Environment(os.Environ()) { + if strings.HasPrefix(declared, usageVariable+"=") { + return strings.TrimPrefix(declared, usageVariable+"="), nothing + } + } + + file, err := os.CreateTemp("", "prescript-draws-*") + if err != nil { + // Not worth failing a run over. The classification is a convenience + // and the comparison is the point; without the count a divergence is + // reported unclassified rather than not at all. + return "", nothing + } + + path := file.Name() + _ = file.Close() + return path, func() { _ = os.Remove(path) } +} + +// drawsFrom reads back what the shim wrote. Nil means unknown -- no tape, no +// shim, or a runtime with no shim to load one -- which is a different thing +// from a port that drew nothing, and the two are not allowed to look alike. +func drawsFrom(usage string) *int { + if usage == "" { + return nil + } + + contents, err := os.ReadFile(usage) + if err != nil { + return nil + } + + drawn, err := strconv.Atoi(strings.TrimSpace(string(contents))) + if err != nil { + return nil + } + return &drawn } // reportFailure explains a failed run to whoever is watching. It writes to @@ -115,6 +178,19 @@ func reportFailure(message string) { // unreadable tape -- and a run that did not happen is not a run that // disagreed with anything. func Run(config cfg.PlayConfig, run script.Run, logger utils.Logger) Outcome { + usage, cleanUp := tapeUsagePath(config, run) + defer cleanUp() + + outcome := play(config, run, logger, usage) + + // Read after the run rather than during it, and after the executable has + // been killed on a timeout, so the count is what had been drawn when the + // run stopped rather than what was drawn while it was being read. + outcome.Draws = drawsFrom(usage) + return outcome +} + +func play(config cfg.PlayConfig, run script.Run, logger utils.Logger, usage string) Outcome { executablePath, err := getExecutableFilePath(config, run) if err != nil { return notPlayed(run, utils.USER_ERROR) @@ -126,7 +202,7 @@ func Run(config cfg.PlayConfig, run script.Run, logger utils.Logger) Outcome { return notPlayed(run, utils.USER_ERROR) } - environment, err := childEnvironment(config, run) + environment, err := childEnvironment(config, run, usage) if err != nil { reportFailure(err.Error()) return notPlayed(run, utils.USER_ERROR) diff --git a/internal/play/runner_integration_test.go b/internal/play/runner_integration_test.go index 639d455..5903bda 100644 --- a/internal/play/runner_integration_test.go +++ b/internal/play/runner_integration_test.go @@ -272,3 +272,68 @@ func TestAMissingTapeIsReportedBeforeTheRun(t *testing.T) { assert.Equal(t, utils.USER_ERROR, exitCode) assert.Contains(t, report, "could not read the tape") } + +// Nobody has to ask. The classification is only useful if it happens on every +// run, and a count that appears when somebody remembered to set a variable is +// one that mostly does not appear. +func TestPrescriptCollectsTheDrawCountItself(t *testing.T) { + for _, language := range tapedRunners { + t.Run(language.runner, func(t *testing.T) { + t.Parallel() + runner := loadRunner(t, language.runner) + runs := script.ApplyRunner([]script.Run{drawingRun(t, language.program, referenceDraws)}, runner) + + outcome := Run(tapedConfig(t), runs[0], &utils.CustomLogger{}) + + assert.Equal(t, 0, outcome.ExitCode) + if assert.NotNil(t, outcome.Draws) { + assert.Equal(t, 5, *outcome.Draws) + } + }) + } +} + +// Unknown is not zero. Without a tape there is no shim keeping count, and a +// run that cannot say what it drew must not be read as one that drew nothing. +func TestWithoutATapeTheDrawCountIsUnknown(t *testing.T) { + for _, language := range seededRunners { + t.Run(language.runner, func(t *testing.T) { + t.Parallel() + runner := loadRunner(t, language.runner) + runs := script.ApplyRunner([]script.Run{drawingRun(t, language.program, language.drawn)}, runner) + + outcome := Run(drawingConfig(), runs[0], &utils.CustomLogger{}) + + assert.Equal(t, 0, outcome.ExitCode) + assert.Nil(t, outcome.Draws) + }) + } +} + +// A run that is killed for hanging runs no exit handler, so a count written +// only on the way out would be missing from exactly the divergences that need +// classifying. The shims write it as they go. +func TestTheDrawCountSurvivesARunThatIsKilled(t *testing.T) { + for _, language := range tapedRunners { + t.Run(language.runner, func(t *testing.T) { + t.Parallel() + runner := loadRunner(t, language.runner) + + // One step short of what the program prints, so prescript is still + // waiting when the timeout kills it. + run := drawingRun(t, language.program, referenceDraws) + run.Steps = append(run.Steps, script.Step{Line: "never printed"}) + runs := script.ApplyRunner([]script.Run{run}, runner) + + config := tapedConfig(t) + config.Timeout = getTimeout(2000) + + outcome := Run(config, runs[0], &utils.CustomLogger{}) + + assert.NotEqual(t, 0, outcome.ExitCode) + if assert.NotNil(t, outcome.Draws) { + assert.Equal(t, 5, *outcome.Draws) + } + }) + } +} diff --git a/runners/node/random.js b/runners/node/random.js index ee4d6b1..5bf42f4 100644 --- a/runners/node/random.js +++ b/runners/node/random.js @@ -34,6 +34,29 @@ if (tapePath) { let drawn = 0; + // How much of the tape a port used is the number that separates a real logic + // bug from a port that restructured its draws, so it is written down rather + // than left to be inferred from the transcript. + // + // It is rewritten after every draw rather than once on exit, because the + // number that matters is how much had been drawn when the run stopped + // agreeing with the transcript -- and a run that diverged by hanging is + // killed, which fires no exit handler at all. Rewriting at offset 0 is safe + // without truncating: the count only ever grows, so its decimal form only + // ever gets longer and each write covers the last. + const usagePath = process.env.PRESCRIPT_TAPE_USAGE; + const usageFd = usagePath ? fs.openSync(usagePath, "w") : null; + + const writeUsage = () => { + if (usageFd !== null) { + fs.writeSync(usageFd, `${drawn}\n`, 0); + } + }; + + // Written from the start, so that a port which drew nothing at all says so + // rather than saying nothing. + writeUsage(); + Math.random = function random() { // Running off the end is an error rather than a wrap. A port that draws // more than the reference did has restructured how it consumes @@ -51,16 +74,10 @@ if (tapePath) { process.exit(1); } - return values[drawn++]; + const value = values[drawn++]; + writeUsage(); + return value; }; - - // How much of the tape a port used is the number that separates a real logic - // bug from a port that restructured its draws, so it is written down rather - // than left to be inferred from the transcript. - const usagePath = process.env.PRESCRIPT_TAPE_USAGE; - if (usagePath) { - process.on("exit", () => fs.writeFileSync(usagePath, `${drawn}\n`)); - } } else { // mulberry32: short enough to read in one sitting, and its whole state is a // single 32-bit word, which is what makes the sequence identical everywhere. diff --git a/runners/python/sitecustomize.py b/runners/python/sitecustomize.py index c58de72..4800a95 100644 --- a/runners/python/sitecustomize.py +++ b/runners/python/sitecustomize.py @@ -20,7 +20,6 @@ all of which have generators of their own. """ -import atexit import os import random @@ -39,6 +38,30 @@ def _install_tape(path): values = _read_tape(path) drawn = 0 + # How much of the tape a port used is the number that separates a real + # logic bug from a port that restructured its draws, so it is written down + # rather than left to be inferred from the transcript. + # + # It is rewritten after every draw rather than once at exit, because the + # number that matters is how much had been drawn when the run stopped + # agreeing with the transcript -- and a run that diverged by hanging is + # killed, which runs no atexit handler at all. Rewriting in place is safe + # without truncating: the count only ever grows, so its decimal form only + # ever gets longer and each write covers the last. + usage = os.environ.get("PRESCRIPT_TAPE_USAGE") + usage_file = open(usage, "w", encoding="utf-8") if usage else None + + def write_usage(): + if usage_file is None: + return + usage_file.seek(0) + usage_file.write("%d\n" % drawn) + usage_file.flush() + + # Written from the start, so that a port which drew nothing at all says so + # rather than saying nothing. + write_usage() + def next_value(): # Running off the end is an error rather than a wrap. A port that draws # more than the reference did has restructured how it consumes @@ -54,6 +77,7 @@ def next_value(): value = values[drawn] drawn += 1 + write_usage() return value instance = random._inst @@ -69,13 +93,6 @@ def next_value(): # so the one that was captured has to be replaced by name as well. random.random = next_value - # How much of the tape a port used is the number that separates a real - # logic bug from a port that restructured its draws, so it is written down - # rather than left to be inferred from the transcript. - usage = os.environ.get("PRESCRIPT_TAPE_USAGE") - if usage: - atexit.register(lambda: open(usage, "w", encoding="utf-8").write("%d\n" % drawn)) - _tape = os.environ.get("PRESCRIPT_TAPE") if _tape: diff --git a/runners/ruby/random.rb b/runners/ruby/random.rb index 47073ad..e8c80cd 100644 --- a/runners/ruby/random.rb +++ b/runners/ruby/random.rb @@ -37,12 +37,33 @@ def self.next_value value = @values[@drawn] @drawn += 1 + write_usage value end def self.drawn @drawn end + + # The count is rewritten after every draw rather than written once at exit, + # because the number that matters is how much had been drawn when the run + # stopped agreeing with the transcript -- and a run that diverged by hanging + # is killed, which runs no at_exit handler at all. + # + # Rewriting in place is safe without truncating: the count only ever grows, + # so its decimal form only ever gets longer, and each write covers the last. + def self.record_usage(path) + @usage = File.open(path, "w") + @usage.sync = true + write_usage + end + + def self.write_usage + return unless @usage + + @usage.seek(0) + @usage.write("#{@drawn}\n") + end end if (tape = ENV["PRESCRIPT_TAPE"]) && !tape.empty? @@ -69,9 +90,10 @@ def rand(limit = nil) # How much of the tape a port used is the number that separates a real logic # bug from a port that restructured its draws, so it is written down rather - # than left to be inferred from the transcript. + # than left to be inferred from the transcript. Written from the start, so + # that a port which drew nothing at all says so rather than saying nothing. if (usage = ENV["PRESCRIPT_TAPE_USAGE"]) && !usage.empty? - at_exit { File.write(usage, "#{PrescriptTape.drawn}\n") } + PrescriptTape.record_usage(usage) end else srand(Integer(ENV.fetch("PRESCRIPT_SEED", "0")))