Skip to content

Generate ready-to-file bug reports - #42

Merged
rjnienaber merged 4 commits into
mainfrom
feature/bug-reports
Sep 7, 2026
Merged

rjnienaber merged 4 commits into
mainfrom
feature/bug-reports

Conversation

@rjnienaber

Copy link
Copy Markdown
Owner

Closes #18. The last issue in milestone 5.

Why

The usual answer to a cross-implementation bug report is "works for me", and it is usually fair — the two people ran different interpreter builds, in different locales, against different random numbers. A report that does not pin all of them down cannot be acted on, only argued with. Pinning them down by hand is ten minutes of copying version strings out of terminals, so nobody does it twice.

Everything in the report is something the run already knew about itself.

What --bug-report <file> writes

# ruby: no-match at step 5 of 12

In `dice.yaml`, against `basic`.
Classified `formatting, same-draws`.

## The divergence

```
step 5 of 12 (no-match, formatting, same-draws)
  basic  " 7             807 "
  ruby   "7 807"
          ^ first difference
```

This is the first divergence; 7 later steps were never reached.

## Reproducing it

```sh
make pull_image PUBLISHED_REF=ghcr.io/rjnienaber/prescript-env@sha256:2a1f...
prescript play dice.yaml --tape tapes/vintbas-seed0.tape
```

## The environment

| | |
| --- | --- |
| prescript | `86fa312b39df` |
| script | `dice.yaml` |
| command | `ruby ports/ruby/dice.rb` |
| ruby | `ruby 3.3.5 (2024-09-03 revision ef084cc8f4) [x86_64-linux]` |
| image | `ghcr.io/rjnienaber/prescript-env@sha256:2a1f...` |
| tape | `tapes/vintbas-seed0.tape` |
| draws | `5` |
| terminal | `pty, 80x24` |
| timeout | `30s` |
| LC_ALL | `C.UTF-8` |
| TZ | `UTC` |

Plus the tape's own # header quoted verbatim, and the image's build manifest when there was one.

Decisions worth reviewing

  • The first divergence only. A run that went wrong at step 5 was not really running the same program from step 6 onwards. For a multi-run script that is the earliest finding of the comparison (so one bug in twelve ports is one report naming twelve ports); for a script whose reference failed it is the reference's own divergence, which is the more urgent finding; for a single run it is that run's failure, reported against the transcript rather than naming an implementation that was never played.
  • Only an executable named by name is asked for its version. One named by a path (./dice) is the program under test rather than the toolchain under it, and running somebody else's program with an argument it never expected is not a thing to do behind their back. Bounded at 5s and omitted on failure, so it cannot become a second way for a failed run to hang.
  • Only a registry digest earns a make pull_image line. A locally built image is identified by a bare config digest, which says which bytes ran but cannot be fetched — named in the table, left out of the commands.
  • A run outside the container says so in the table, in those words, rather than leaving a gap. It is the likeliest reason a reader cannot reproduce what they are shown.
  • argv[0] becomes prescript and absolute paths inside the working directory become relative, so the command works from a fresh checkout and a report can be posted in public without a home directory in it. Paths outside are left alone; shortening them would produce a command that does not work.
  • It files nothing and changes no exit code. A report that could not be written is said out loud on stderr — the run it described will not be repeated for free — but that is not itself a failure of the run.

Groundwork

Two facts prescript could not previously state about itself:

  • utils.TerminalSize(), moved out of the //go:build unix pty file so it is reachable everywhere (GOOS=windows go build ./... still passes).
  • utils.Version() from debug.ReadBuildInfo(). This needed the Makefile to build ./cmd/prescript rather than cmd/prescript/main.go — a package path is the only form go build stamps VCS info onto — and Config.Handled, so --version exits 0 instead of being treated as a missing subcommand.

Testing

15 unit tests in internal/play/bug_report_test.go, plus one integration test that gathers the facts from a real Ruby run against the real tape, so the interpreter probe, the draw count and the tape header are exercised rather than described.

make format, make lint, PRESCRIPT_REQUIRE_RUNNERS=1 make test, make build_dev, make examples all clean.

Docs: new docs/bug-reports.md, a ### Filing what it found section in the README, the --bug-report row in the play options table, and a pointer at the end of docs/comparison.md.

🤖 Generated with Claude Code

https://claude.ai/code/session_019f9gkZVUEo3jtv51GSb4vJ

rjnienaber and others added 4 commits September 7, 2026 22:54
The usual answer to a cross-implementation bug report is "works for me",
and it is usually fair: the two people ran different interpreter builds,
in different locales, against different random numbers. Pinning all of
that down by hand is ten minutes of copying version strings out of
terminals, so nobody does it twice and by the tenth report they are not
being filed at all.

--bug-report writes the first divergence to a file as Markdown, with
everything the run already knew about itself: both sides and the
classification, the interpreter version, the container digest, the tape
and its provenance plus how much of it the port drew, LC_ALL/LANG/TZ,
the terminal and its size, prescript's own revision, and one command
that reproduces the lot.

The first divergence only. A run that went wrong at step 5 was not
really running the same program from step 6 onwards, and listing every
later difference asks the reader to work out which one was the cause.

Two facts prescript could not previously state about itself had to
exist first: TerminalSize(), moved out of the unix-tagged pty file, and
Version() from the binary's own build info -- which also required the
Makefile to build a package path rather than a file path, since that is
the only form go build stamps VCS info onto. --version then needed
Config.Handled so cobra answering the command line itself exits zero
rather than as a usage error.

Only an executable named by name is asked for its version. One named by
a path is the program under test rather than the toolchain under it.
Only a registry digest earns a `make pull_image` line; a locally built
image's config digest is named but not pasted as a command nobody can
run. Absolute paths inside the working directory are made relative, so
a report can be posted in public without a home directory in it.

Closes #18

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019f9gkZVUEo3jtv51GSb4vJ
The repository is mounted from the host, so its files belong to a user
the container does not have, and git refuses to read a repository it
considers somebody else's. That was harmless until the binary started
carrying its own revision: `go build` asks git for it and treats the
refusal as an error rather than as "no VCS here", so every build inside
the image failed with `error obtaining VCS status: exit status 128`.

Trusted for the one directory that is ever mounted, rather than turning
VCS stamping off -- the version in a bug report is the whole point, and
a report written inside the pinned environment is the one most likely
to be filed.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019f9gkZVUEo3jtv51GSb4vJ
`should not contain "807"` was meant to say the later finding is not in
the report. Inside the container it also matched the image's own digest,
which is hex and long enough to contain any three digits sooner or
later. Says what it meant instead: the port that diverged later is not
named, and neither is its step.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019f9gkZVUEo3jtv51GSb4vJ
Twice now a NotContains over the whole report has matched something the
machine put there rather than something the report got wrong: the image
digest is hex long enough to contain any three digits, and the image
manifest names every language installed, python included.

Asserted against the divergence section alone, which is the part the
test is about.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019f9gkZVUEo3jtv51GSb4vJ
@rjnienaber
rjnienaber merged commit 00092e8 into main Sep 7, 2026
5 checks passed
@rjnienaber
rjnienaber deleted the feature/bug-reports branch September 7, 2026 22:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Generate ready-to-file bug reports with a full repro block

1 participant