Repository navigation
pin the environment a comparison run happens in - #35
Merged
Merged
Conversation
Randomness is the loudest source of transcript churn but not the largest. Interpreter versions, locale, timezone, terminal size, hash seeds and address layout all change what a program prints without changing what it does, and a difference between two ports means nothing if the two ran in different worlds. docker/ builds that world. The base image is pinned by digest; Go, Node and vintbas are fetched from upstream and checked against a published SHA-256; Ruby and Python come from Debian stable, which is what pinning means for a distribution. GOTOOLCHAIN=local is in there for the same reason as the rest -- without it the go command fetches a newer toolchain when a module asks for one, quietly unpinning the Go above it. amd64 only, deliberately. The reference interpreter is published for linux x86_64, and comparing a port against a reference built for another architecture folds floating-point and standard-library differences into every comparison. Terminal size turned out to be prescript's to fix rather than the image's. A pty opened without a size reports zero rows and columns, so a program asking how wide its terminal is got an answer that was neither what a real terminal gives nor what the next machine gives. Every child now gets 80x24 whatever terminal prescript itself was started from. Address-space randomisation is left on by default. Docker's default seccomp profile permits personality() with a handful of arguments and ADDR_NO_RANDOMIZE is not among them, so holding the layout still means dropping the profile entirely -- worth keeping until a run actually needs it, given the code being compared is other people's and the repository is mounted writable. PRESCRIPT_FIX_ADDRESS_LAYOUT=1 asks for it, and the entrypoint says on stderr when addresses will move. What a bug report needs is readable from inside a run: PRESCRIPT_IMAGE names the image, and $PRESCRIPT_ENVIRONMENT names a file listing the base digest and every toolchain version, written at build time from the binaries that were actually installed rather than from the arguments that asked for them. Issue #18 folds both into a repro block. CI builds the image and runs the suite inside it on every change, because an image nobody builds is an image that has stopped building. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019f9gkZVUEo3jtv51GSb4vJ
rjnienaber
force-pushed
the
feat/pinned-environment
branch
from
September 7, 2026 14:25
8f0e957 to
9daff0a
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #15. Stacked on #34 (
feat/rng-tape) — rebase tomainonce that merges.Randomness is the loudest source of transcript churn but not the largest. Interpreter versions, locale, timezone, terminal size, hash seeds and address layout all change what a program prints without changing what it does, and a difference between two ports means nothing if the two ran in different worlds.
What is pinned
debian:bookworm-slimby digestLANG=LC_ALL=C.UTF-8TZ=UTCPYTHONHASHSEED=0setarch --addr-no-randomize, opt-inGOTOOLCHAIN=localis in there for the same reason as everything else: without it thegocommand fetches a newer toolchain when a module asks for one, which would quietly unpin the Go that was so carefully pinned above it.amd64 only, deliberately — the reference interpreter is published for linux x86_64, and comparing against a reference built for another architecture folds floating-point and standard-library differences into every comparison.
Terminal size turned out to be prescript's job
Not the image's.
pty.Opengives a pty with zero rows and columns, so a program asking how wide its terminal is got an answer that is neither what a real terminal gives nor what the next machine gives. Every child now gets 80×24 whatever terminal prescript itself was started from, checked byTestThePtyIsAlwaysTheSameSizeagainst a fixture that runsstty size.Address-space randomisation is opt-in, and here is why
Docker's default seccomp profile permits
personality()with a handful of arguments andADDR_NO_RANDOMIZE(0x40000) is not among them, sosetarch -Ris refused under the default profile — I verified this rather than assuming the entrypoint would just work. Holding the layout still therefore means dropping the profile entirely, which is not something to do by default when the code being compared is other people's and the repository is mounted writable:Verified both ways — with it the stack lands at
0x7ffffffde000on every run, without it it moves. The entrypoint says on stderr when addresses will move rather than failing, because a container that will not start is worse than one that is slightly less repeatable.What a bug report needs
Both readable from inside a run, for #18 to fold into a repro block:
$PRESCRIPT_ENVIRONMENTnames a file listing the base digest and every toolchain version, written at build time from the binaries actually installed rather than from the arguments that asked for them — so it cannot claim a version the image does not have.$PRESCRIPT_IMAGEnames the image: its registry digest when pulled, its config digest when built locally.docker/run.shsets it, since an image cannot know its own digest at build time.Verified
Built and run locally (amd64 under emulation on an arm64 Mac). The full suite passes inside it with
PRESCRIPT_REQUIRE_RUNNERS=1, andmake examplesthere produces the same dice histogram as the host — same interpreter, same numbers, which is the property the image exists to guarantee.tmp/is deliberately not shared with the host:make build_devin the container produces a linux binary, and dropping that on top of the host's turns the nextmake examplesinto "cannot execute binary file". I hit exactly that before mounting a volume over it.CI gains a
containerjob that builds the image and runs the suite inside it on every change, because an image nobody builds is an image that has stopped building.Not done
The image is not published anywhere, so CI and a developer each build it from the same pinned Dockerfile rather than pulling the same bytes. That is enough to make the toolchains agree and not enough to make the layers byte-identical. Publishing it to GHCR is the obvious follow-up — I left it out because it creates a public artifact under your account, which seemed like your call rather than mine. Happy to add it.
🤖 Generated with Claude Code
https://claude.ai/code/session_019f9gkZVUEo3jtv51GSb4vJ