build: a fresh worktree bootstraps its own drivers and venv - #804
Conversation
Two gates stop `make verify` in a checkout that has never been built, and both have now cost several people the same half hour. drivers/*.lua is a gitignored snapshot fetched from the pin, so a fresh clone or worktree has none. The presence guard stopped the build to name the command that fetches them; it now runs it. That costs one `ls` and no network on every run after the first, which is every run in a checkout that already has them, and a failed fetch still says what to do by hand. `go test` already downloads its modules on a fresh checkout -- this is the same bargain, once. The optimizer venv was built with whatever answers to python3. On macOS that is 3.9 with pip 21.2, which cannot do a PEP 660 editable install and reports it as "File setup.py or setup.cfg not found" -- a message that reads like a packaging fault here rather than the wrong interpreter. The choice moves into scripts/optimizer-venv.sh: $PYTHON first so a machine where it already worked is unaffected, then a python3.N at or above the floor declared in optimizer/pyproject.toml, then uv, which fetches an interpreter when the machine has none. uv stays optional and is never consulted while a suitable python exists. No interpreter and no uv is an error naming both remedies. A venv left behind by the old failure is replaced rather than reused. Verified in a worktree created from master with an empty drivers/ and no venv: `make verify` failed at the editable install before, and passes in 65s after, fetching 37 drivers and building the environment on the way. The uv path was exercised with a PATH holding no python 3.11+, and the venv it produces passes optimizer-test and the core-optimizer contract test unchanged. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
frahlg
added
the
no-changeset
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 76a0e43146
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| # that could not do the install. Keep the environment when its own interpreter | ||
| # qualifies -- reinstalling into it is what an unchanged machine wants -- and | ||
| # otherwise throw it away and choose again. | ||
| if [ -x "${VENV}/bin/python" ] && satisfies_floor "${VENV}/bin/python"; then |
There was a problem hiding this comment.
When optimizer/.venv already contains any qualifying interpreter, this branch runs before candidate selection, so make optimizer-install PYTHON=/path/to/python silently reuses the existing interpreter. With an existing Python 3.12 environment and a requested Python 3.11 binary, the requested binary is never invoked, breaking the advertised override when a developer needs to switch or repair the environment; reuse should account for an explicit PYTHON selection.
Useful? React with 👍 / 👎.
|
|
||
| if [ -n "${CHOSEN}" ]; then | ||
| echo "optimizer: building .venv with ${CHOSEN} ($("${CHOSEN}" --version 2>&1))" | ||
| "${CHOSEN}" -m venv "${VENV}" |
There was a problem hiding this comment.
On a host where the first version-qualified interpreter cannot create virtual environments, this command exits immediately under set -e, without trying another candidate or the advertised uv fallback. Reproducing a Python 3.12 candidate that passes the version check but fails -m venv, with uv available on PATH, leaves uv uninvoked and the fresh-worktree bootstrap still fails; unsuccessful creation should continue to another candidate or fall back to uv.
Useful? React with 👍 / 👎.
1 participant
make verifydoes not run in a checkout that has never been built. Fivepeople have now found this out one at a time, each losing about half an hour
to the same two gates, and the shared workaround — copy 37
.luafiles fromanother checkout by hand — is a thing we should stop asking people to know.
What stopped
The drivers gate.
drivers/*.luais a gitignored snapshot fetched fromthe commit pinned in
drivers/BUNDLED_SOURCE.json, so a fresh clone orgit worktree addhas none, anddrivers-presentstopped the build to namethe command that fetches them. It now runs it. On every later run — which is
every run in a checkout that already has the snapshot — the guard is still
one
lsand no network; measured at 44 ms. A fetch that fails still printswhat to do by hand, including the copy.
go testalready downloads itsmodules on a fresh checkout; this is the same bargain, paid once.
The optimizer venv.
optimizer-installbuilt the environment withwhatever answers to
python3. On macOS that is 3.9 with pip 21.2.4, whichcannot do a PEP 660 editable install of a pyproject-only package and reports
it as:
which reads like a packaging fault in this repository rather than the wrong
interpreter. Choosing the interpreter is now the whole job of
scripts/optimizer-venv.sh:$PYTHON— what the old recipe used, so a machine where it already workedbuilds exactly the venv it built before;
python3.NonPATHat or above the floor declared inoptimizer/pyproject.toml, starting with the 3.12 thatDockerfile.optimizerand CI run;uv, which fetches an interpreter when the machine has none.uvstays optional and is never consulted while a suitable python exists.Neither one present is an error naming both remedies. A venv left behind by
the old failure — 3.9, half-built — is replaced rather than reused, which is
the state the second attempt actually starts from.
Evidence
A worktree created from
master, no drivers, no venv:make verifydrivers/*.luaAlso exercised: the
uvroute, with aPATHholding no python 3.11+ — thevenv it produces (3.12,
--seed, so it has pip like any other) passesmake optimizer-testand the core↔optimizer contract test unchanged; the"no python, no uv" error; and a fetch failure with
curlbroken.The golden corpus is untouched and no record moved — this change reaches no
control code, and
TestGoldenCorpusReplaypasses at the same 0.01 Wtolerance in every run above.
Notes for review
Makefileandscripts/are outside the allowlist inchangeset-check.yml, so the gate asks for one; nothing here ships to agateway (the release tarballs carry neither file) and the change is local
developer tooling, which
.changeset/README.mdputs under theno-changesetlabel. Label applied.drivers-from-pinintest.ymlchecked thatdrivers-presentfails on an emptydrivers/.It now checks that the guard materialises all 37 — the same job, asking
the fresh-clone question the guard now answers.
bash -nlist incompose-migration-test, because ci: notice when a new Debian stable leaves the pinned suite behind #733 is appending tothat exact line and has right of way. Worth adding once it lands.
Co-Authored-By: Claude Opus 5 noreply@anthropic.com