A script file describes what an interactive program prints and what should be
typed back at it. prescript record writes one; prescript play replays it.
It may be written in JSON or in YAML — see File formats.
Every script declares a version, of the form MAJOR.MINOR:
{
"version": "0.1",
"runs": [ ... ]
}MINOR increases for an additive change — typically a new optional field. Scripts written against an earlier minor of the same major keep working with no edits, so there is nothing to migrate.
MAJOR increases when existing scripts have to be rewritten to keep working. When that happens, the older major is dropped from the list of readable versions and the error names the last release that could read it.
prescript reads a fixed list of versions (knownVersions in
internal/script/version.go) and rejects anything else, including a version
newer than its own. It does not warn and continue.
That is the important half of the rule, and it is deliberate. The fields a minor version adds are precisely the ones that change what a run means: an environment variable, a redaction rule, a second run to compare against. A reader that skipped what it did not recognise would not fail — it would quietly do something else and report success. Confidently reporting a wrong answer is the failure this tool exists to catch in other people's programs, so it is not one prescript may commit itself.
A newer version says what to do about it:
Script validation errors:
version: script is written for format 9.9, but this build of prescript reads up to 0.2; upgrade prescript
which is a considerably more useful thing to read than the list of unrecognised field names the same script would otherwise produce.
While MAJOR is 0 the format is not stable, and a minor may take a field away
as well as add one -- which is what 0.x means everywhere else, and what 0.9
did to isRegex. Once 1.0 is cut, removing a field becomes a major bump.
In the same change that alters the schema:
- Append the new version to
knownVersions.CurrentVersionfollows the last entry, andrecordwrites it. - Add the field to
script_schema.jsonas optional, and to the Go type. - Leave older versions in the list. They are still readable, and removing one is a major bump.
Do not add a version ahead of the change that needs it: a version that exists but means nothing is worse than no version at all.
| Version | Change |
|---|---|
0.1 |
Initial format. |
0.2 |
Added env and inheritEnv to a run. |
0.3 |
A script may hold more than one run. |
0.4 |
Added redactions, superseding isRegex. |
0.5 |
Added runner files. |
0.6 |
Added terminal, and made a pty the default. |
0.7 |
Added a per-step timeout. |
0.8 |
A runner may use ${runnerDir} where it names a path. |
0.9 |
Removed isRegex; redactions are the only way to match by pattern. |
The schema validates the union of all fields across known minor versions, so a
script declaring 0.1 that uses a field introduced in 0.2 is accepted rather
than rejected. Version-specific schemas would catch it, at a cost in machinery
that is not yet worth paying. The version declares intent; the schema checks
shape.
A run may declare the environment its program is started with:
{
"version": "0.2",
"runs": [{
"executable": "vintbas",
"arguments": ["dice.bas"],
"env": { "VINTBAS_SEED": "0", "TZ": "UTC" },
"exitCode": 0,
"steps": [ ... ]
}]
}The rule is that declaring an environment declares all of it:
env |
child process gets |
|---|---|
| omitted | prescript's own environment, unchanged |
{ "A": "1" } |
exactly A=1, and nothing else |
{ "A": "1" } with "inheritEnv": true |
prescript's environment, with A=1 set over it |
{} |
an empty environment |
Omitting env is what every script written before 0.2 does, so those keep
behaving exactly as they did. Anything that declares one gets the reproducible
reading: a run that names TZ and VINTBAS_SEED is not also quietly depending
on the LANG that happened to be exported in the shell that started it.
inheritEnv is the escape hatch for a program that needs the caller's
environment — a compiler toolchain, say — plus a couple of variables of its
own. Declared values are applied over inherited ones.
A run that replaces its environment does not necessarily leave the child with no
PATH: execvp and most shells fall back to a compiled-in default when the
variable is unset, so #!/usr/bin/env bash generally still resolves and so do
the common tools.
That fallback is not something to rely on, because its contents differ by
platform — on Linux it is typically just /bin:/usr/bin, which excludes
/usr/local/bin, where a tool the program shells out to may well live. If the
program needs to find anything, set PATH in env explicitly. Depending on a
per-platform default is the machine-dependence this feature exists to remove.
Resolving the executable named by the script is unaffected either way: prescript
looks it up on its own PATH before starting the child, so a script naming
vintbas finds the same binary whether or not the run declares an environment.
A program is started under a pseudo-terminal by default, which is what an interactive program is normally run under and what it asks about before deciding how to behave:
version: "0.6"
runs:
- executable: ./menu.py
arguments: []
exitCode: 0
terminal: pipes # the exception, not the rule
steps: [ ... ]terminal |
the program's standard streams are |
|---|---|
omitted, or pty |
a pseudo-terminal |
pipes |
ordinary pipes |
--terminal pty and --terminal pipes override whatever the file said, which
is how a script is tried the other way round without editing it.
The question a program actually asks is isatty(), and it asks it to decide
how to buffer its output. Under a terminal, C and Python line-buffer: a prompt
reaches prescript the moment it is written. Under a pipe they switch to a 4KB
block buffer, and a prompt with no trailing newline can sit in that buffer
until the program either fills it or exits — which it will not do, because it
is waiting for the answer to the prompt it has not yet delivered. The result is
a timeout against a program that is working perfectly.
Buffering is only the most common of the things a program decides this way. Colour, progress bars, pagination and "are you sure?" prompts are all commonly switched off when the output is not a terminal, so a fixture recorded through a pipe describes a program in a mode nobody runs it in.
The pty is put into raw mode before the program starts. Two of the flags that clears are the ones that would otherwise show up in a script:
- echo, which would send every character prescript types back down the same stream it is matching against, so a step would have to expect its own input.
- newline translation, which would turn each
\nthe program writes into\r\n, so every expected line would have to end in a carriage return.
What is left is a stream of bytes identical to what a pipe would have delivered, from a program that can nonetheless see it is talking to a terminal. A script recorded under one and played under the other generally matches, which is the point: the pty changes what the program decides, not what prescript reads.
As a backstop, a \r\n written by the program itself is read as \n. A
carriage return that is not followed by a newline is left alone, because it
means "back to the start of the line" and a program that writes one meant it.
terminal: pipes is for the case where the pipe is the thing being tested — a
program whose non-interactive mode is the subject of the script, or one that
misbehaves under a terminal in a way worth pinning down. It is also the answer
on a platform with no pty to offer.
A recording made with --terminal pipes writes terminal: pipes into the
script, so replaying it does not quietly hand the program a terminal it was
never recorded against.
runs is an array, and a script may hold more than one. This is how two
implementations of the same program are compared: each run names its own
executable, and they share the steps they are expected to produce.
{
"version": "0.3",
"runs": [
{ "name": "reference", "executable": "vintbas", "arguments": ["dice.bas"], ... },
{ "name": "port", "executable": "python", "arguments": ["dice.py"], ... }
]
}Every run is played, even after one has failed. Stopping at the first failure
would withhold the comparison the script was written to make: knowing the first
of three ports is wrong says nothing about the other two. prescript exits with
the first non-zero code.
The first run is the reference, and a script with more than one ends with a comparison: every port lined up against that reference at the step where it left it, with ports that got the same thing wrong grouped into one finding. See comparison.md.
Each run has a name, which is what identifies it in the output. A run without
one is called run N after its position. Two runs may not end up with the same
name, and generated names count: a run explicitly called run 1 collides with
the unnamed run at index 1.
--exec and the arguments after -- apply to every run. That is meaningful
when the runs differ in their steps and a mistake when they differ in their
executable, and prescript cannot tell which was meant, so it warns rather than
refusing.
--timeout bounds how long the program may go on saying nothing — it is a
limit on the wait for the next character, not on how long a step may take in
total. It defaults to 30 seconds.
One slow moment should not set that limit for the whole script, so a step can name its own:
version: "0.7"
runs:
- executable: ./primes
arguments: ["1000000"]
exitCode: 0
steps:
- line: "How many? "
input: "1000000"
- line: "Done."
timeout: "5m"A step's timeout replaces the run's while that step is the one being waited
for, and is written as a Go duration — "500ms", "90s", "5m". It always
wins, including over --timeout: a step that says it is slow is a more
specific statement than a flag applied to everything.
A duration that does not parse, or one that is not positive, is a validation error reported with the regexes, before the run starts. A corpus run should find out that a script cannot be played before it has spent twenty minutes getting to the step that cannot play.
A failed run names what went wrong before it says anything else:
no-match: step 3 of 12 did not match (nothing matched it within 30s)
expected "HOW MANY ROLLS? "
received "HOW MANY ROLES? "
^ first difference
| Mode | What happened | Where to look |
|---|---|---|
no-match |
the program is still running, and printing something the step does not expect | the implementation's output |
exited-early |
the program finished with steps still outstanding | a crash, a rejected argument, or a script that outlasts the program |
hung |
every step matched and the program never exited | input the script does not go on to supply |
wrong-exit-code |
it printed everything expected of it and disagreed only about how it finished | the program's ending, not its output |
read-failed |
prescript could not read from the program | prescript or the machine, not the program |
Those are different bugs with different places to go looking. Naming them in the same words every time is what makes a few hundred failing scripts something that can be sorted and counted rather than something that has to be read one by one.
A report contains fixed strings, what the script says, and what the program printed — never how long something actually took. A timeout reports the limit it was given, not the time it waited. Two runs of the same divergence therefore produce the same bytes, and a report that changes between runs means the program's behaviour changed, not that the machine was busier.
Whichever way a timeout is reported, the program is killed before the report is written, along with anything it started: under a pty it is a session leader, so one signal reaches the interpreter and the program it launched. The alternative is one abandoned process per timeout, which a corpus run cannot afford.
Under terminal: pipes the program shares prescript's own process group, so
only the program itself is signalled — a group signal there would take
prescript with it — and a child it started of its own may outlive it.
prescript play reads a script as YAML when the file ends in .yaml or .yml,
and as JSON otherwise. prescript record writes JSON.
YAML is converted to JSON and then validated by the same schema against the same
Go types. There is one description of the format and one set of error messages,
so the two notations cannot drift apart, and a YAML script is rejected in the
same terms — including the same runs.0.exitCode style paths — as the JSON it
converts to.
YAML is the better authoring format, and the fixtures are meant to be written in it. It has comments, and the expected output is not buried in quoting.
steps:
- line: "HOW MANY ROLLS? "
input: "5000"YAML strips trailing whitespace from an unquoted scalar. A prompt almost always ends in a space, and a step that differs from the program's real output only in that space is the most annoying possible failure to read: the two look identical everywhere except in the report, which quotes them precisely because of this.
Quote input too. An unquoted 5000 is a number, an unquoted N is a string,
and an unquoted y, no or on is a boolean in YAML 1.1 readers — none of
which is what a program reading from a terminal receives.
YAML allows a mapping key of any type; JSON does not. A script that uses one is rejected, rather than reaching the schema as something it cannot describe.
A line that contains something different on every run cannot be matched
literally. redactions names the patterns that stand in for those parts, and a
step refers to one as {{name}}:
version: "0.4"
redactions:
elapsed: '[0-9]+'
path: '/[^ ]+'
runs:
- executable: ./build.sh
arguments: []
exitCode: 0
steps:
- line: "wrote {{path}} in {{elapsed}}ms"Everything outside a placeholder is matched literally — (, . and ? in the
expected output are those characters and not regular expression syntax — and the
line is anchored at both ends, so a redacted line stands for the whole line the
program printed, exactly as a literal one does.
Redactions are declared once for the whole script rather than per run, so two runs being compared against each other normalise the same things in the same way. Normalising differently is how a comparison quietly stops comparing.
Until 0.9 a step could set isRegex and match the whole line as a regular
expression. To mask one volatile number with it you had to write a pattern for
everything around that number too:
- line: '^wrote /[^ ]+ in [0-9]+ms$'
isRegex: truewhich no longer reads as the output it stands for, and no longer says which part
was expected to vary or why. The version with {{elapsed}} says both.
isRegex was removed in 0.9, and a step's fields are closed, so a script still
carrying one is rejected rather than quietly matched literally:
Script validation errors:
runs.0.steps.0: Additional property isRegex is not allowed
A placeholder naming a redaction the script does not declare is rejected, and the message lists the ones it does declare. A redaction whose pattern does not compile is reported once, against the declaration rather than against every step that refers to it.
A runner says how to start an implementation; a script says which program to start and what it should print. Splitting them is what lets one script be played against every port of the same program:
scripts/33-dice.yaml # one per program, shared across all implementations
runners/ruby.yaml # one per language, shared across all programs
runners/haskell.yaml
prescript play scripts/33-dice.yaml --runner runners/ruby.yaml -- 33_Dice/ruby/dice.rb
A hundred programs in a dozen languages is then a hundred-odd files rather than twelve hundred.
# runners/ruby.yaml
version: "0.6"
executable: ruby
arguments: ["-W0"]
env:
RUBYOPT: "--disable-gems"A runner file is read as YAML or JSON by the same rule as a script, declares a
version from the same list, and is rejected in the same terms. Its name
defaults to the file's own name.
| result | |
|---|---|
executable |
the runner's, replacing whatever the script named |
arguments |
the runner's first, then the script's (or the command line's) |
env |
merged; the script wins where both name the same variable |
inheritEnv |
set if either sets it |
terminal |
the script's, falling back to the runner's |
The argument order is the point: the runner's arguments are the interpreter's
own flags, and everything after them names the program to feed it. That is also
why the two halves stay separate rather than being flattened — the arguments
after -- replace the script's half and leave the runner's alone, so
prescript play 33-dice.yaml --runner runners/ruby.yaml -- 33_Dice/ruby/dice.rb
runs ruby -W0 33_Dice/ruby/dice.rb.
--exec still wins over a runner's executable, and says so on stderr when both
are given.
Wherever a runner names a path — in executable, in arguments, or in an
env value — it may write ${runnerDir}, which stands for the directory the
runner file itself was read from:
# runners/ruby.yaml
version: "0.8"
executable: ruby
env:
RUBYOPT: "-r${runnerDir}/ruby/random.rb"It resolves to an absolute path, so a runner and the files it points at travel together whatever directory prescript happens to be run from. Written relative instead, the path would hold only while prescript is run from one place, which is the one thing a corpus run does not do; written absolute, only on the machine that wrote it.
Nothing else is expanded. Env values are handed to the child exactly as
written, with no shell anywhere to expand anything, so a ${...} that is not
${runnerDir} is a rejection rather than a literal:
Script validation errors:
env.RUBYOPT: unknown placeholder ${runnerDr}, the only one is ${runnerDir}
The alternative is a misspelling reaching the interpreter unchanged and being reported as a missing file, a long way from the runner that named it.
This is what makes a shim addressable, which is what makes a random program scriptable at all. See docs/determinism.md.