Skip to content

pin the environment a comparison run happens in - #35

Merged
rjnienaber merged 1 commit into
mainfrom
feat/pinned-environment
Sep 7, 2026
Merged

rjnienaber merged 1 commit into
mainfrom
feat/pinned-environment

Conversation

@rjnienaber

Copy link
Copy Markdown
Owner

Closes #15. Stacked on #34 (feat/rng-tape) — rebase to main once that merges.

Randomness is the loudest source of transcript churn but not the largest. Interpreter versions, locale, timezone, terminal size, hash seeds and address layout all change what a program prints without changing what it does, and a difference between two ports means nothing if the two ran in different worlds.

make image        # build it
make docker_test  # run the suite inside it
make docker_shell # look around

What is pinned

How
Base debian:bookworm-slim by digest
Ruby, Python Debian stable, a fixed series taking security fixes only
Go, Node upstream tarballs, checked against a published SHA-256
vintbas the same release asset the Makefile installs, checked the same way
Locale LANG=LC_ALL=C.UTF-8
Timezone TZ=UTC
Terminal 80×24, on the pty prescript gives the child
Hash seed PYTHONHASHSEED=0
Address layout setarch --addr-no-randomize, opt-in

GOTOOLCHAIN=local is in there for the same reason as everything else: without it the go command fetches a newer toolchain when a module asks for one, which would quietly unpin the Go that was so carefully pinned above it.

amd64 only, deliberately — the reference interpreter is published for linux x86_64, and comparing against a reference built for another architecture folds floating-point and standard-library differences into every comparison.

Terminal size turned out to be prescript's job

Not the image's. pty.Open gives a pty with zero rows and columns, so a program asking how wide its terminal is got an answer that is neither what a real terminal gives nor what the next machine gives. Every child now gets 80×24 whatever terminal prescript itself was started from, checked by TestThePtyIsAlwaysTheSameSize against a fixture that runs stty size.

Address-space randomisation is opt-in, and here is why

Docker's default seccomp profile permits personality() with a handful of arguments and ADDR_NO_RANDOMIZE (0x40000) is not among them, so setarch -R is refused under the default profile — I verified this rather than assuming the entrypoint would just work. Holding the layout still therefore means dropping the profile entirely, which is not something to do by default when the code being compared is other people's and the repository is mounted writable:

PRESCRIPT_FIX_ADDRESS_LAYOUT=1 make docker_test

Verified both ways — with it the stack lands at 0x7ffffffde000 on every run, without it it moves. The entrypoint says on stderr when addresses will move rather than failing, because a container that will not start is worse than one that is slightly less repeatable.

What a bug report needs

Both readable from inside a run, for #18 to fold into a repro block:

  • $PRESCRIPT_ENVIRONMENT names a file listing the base digest and every toolchain version, written at build time from the binaries actually installed rather than from the arguments that asked for them — so it cannot claim a version the image does not have.
  • $PRESCRIPT_IMAGE names the image: its registry digest when pulled, its config digest when built locally. docker/run.sh sets it, since an image cannot know its own digest at build time.
base=debian:bookworm-slim@sha256:88200866dfff7ea7f5cbcb6ec7c8a701889efe6fe859fe64d6990e4b07ea4171
arch=x86_64
go=go1.24.13
node=v24.20.0
python=3.11.2
ruby=3.1.2p20
vintbas=vintbas-1.0.3-1
locale=C.UTF-8
timezone=UTC

Verified

Built and run locally (amd64 under emulation on an arm64 Mac). The full suite passes inside it with PRESCRIPT_REQUIRE_RUNNERS=1, and make examples there produces the same dice histogram as the host — same interpreter, same numbers, which is the property the image exists to guarantee.

tmp/ is deliberately not shared with the host: make build_dev in the container produces a linux binary, and dropping that on top of the host's turns the next make examples into "cannot execute binary file". I hit exactly that before mounting a volume over it.

CI gains a container job that builds the image and runs the suite inside it on every change, because an image nobody builds is an image that has stopped building.

Not done

The image is not published anywhere, so CI and a developer each build it from the same pinned Dockerfile rather than pulling the same bytes. That is enough to make the toolchains agree and not enough to make the layers byte-identical. Publishing it to GHCR is the obvious follow-up — I left it out because it creates a public artifact under your account, which seemed like your call rather than mine. Happy to add it.

🤖 Generated with Claude Code

https://claude.ai/code/session_019f9gkZVUEo3jtv51GSb4vJ

Base automatically changed from feat/rng-tape to main September 7, 2026 14:25
Randomness is the loudest source of transcript churn but not the largest.
Interpreter versions, locale, timezone, terminal size, hash seeds and
address layout all change what a program prints without changing what it
does, and a difference between two ports means nothing if the two ran in
different worlds.

docker/ builds that world. The base image is pinned by digest; Go, Node
and vintbas are fetched from upstream and checked against a published
SHA-256; Ruby and Python come from Debian stable, which is what pinning
means for a distribution. GOTOOLCHAIN=local is in there for the same
reason as the rest -- without it the go command fetches a newer toolchain
when a module asks for one, quietly unpinning the Go above it.

amd64 only, deliberately. The reference interpreter is published for
linux x86_64, and comparing a port against a reference built for another
architecture folds floating-point and standard-library differences into
every comparison.

Terminal size turned out to be prescript's to fix rather than the
image's. A pty opened without a size reports zero rows and columns, so a
program asking how wide its terminal is got an answer that was neither
what a real terminal gives nor what the next machine gives. Every child
now gets 80x24 whatever terminal prescript itself was started from.

Address-space randomisation is left on by default. Docker's default
seccomp profile permits personality() with a handful of arguments and
ADDR_NO_RANDOMIZE is not among them, so holding the layout still means
dropping the profile entirely -- worth keeping until a run actually needs
it, given the code being compared is other people's and the repository is
mounted writable. PRESCRIPT_FIX_ADDRESS_LAYOUT=1 asks for it, and the
entrypoint says on stderr when addresses will move.

What a bug report needs is readable from inside a run: PRESCRIPT_IMAGE
names the image, and $PRESCRIPT_ENVIRONMENT names a file listing the base
digest and every toolchain version, written at build time from the
binaries that were actually installed rather than from the arguments that
asked for them. Issue #18 folds both into a repro block.

CI builds the image and runs the suite inside it on every change, because
an image nobody builds is an image that has stopped building.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019f9gkZVUEo3jtv51GSb4vJ
@rjnienaber
rjnienaber force-pushed the feat/pinned-environment branch from 8f0e957 to 9daff0a Compare September 7, 2026 14:25
@rjnienaber
rjnienaber merged commit 31c8e73 into main Sep 7, 2026
5 checks passed
@rjnienaber
rjnienaber deleted the feat/pinned-environment branch September 7, 2026 14:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Docker test environment with pinned toolchains

1 participant