Skip to content

fix: pin shared-package workspaces with the root lake manifest - #1807

Merged
kim-em merged 5 commits into
mainfrom
fix/workspace-root-manifest
Sep 26, 2026
Merged

kim-em merged 5 commits into
mainfrom
fix/workspace-root-manifest

Conversation

@kim-em

@kim-em kim-em commented Sep 26, 2026 •

Copy link
Copy Markdown
Collaborator

This PR makes scripts/evaluate_submission.py copy the benchmark checkout's root lake-manifest.json into each per-submission workspace whose .lake/packages is symlinked to the shared lean-eval/.lake/packages, instead of running lake update and lake exe cache get there, and makes the evaluation workflows download TauCeti's build outputs instead of compiling them. Running lake update against the shared directory can re-resolve transitive dependencies and check out different revisions inside it, corrupting the tree for every other workspace and for the benchmark root; the root manifest already pins every package any generated workspace requires, and Lake ignores entries a workspace does not require. This matches what lean-eval's own CI and reuseRootPackages do, and is needed for leanprover/lean-eval#649 (feat: add the classification of finite simple groups to v2), whose workspace requires TauCeti as well as Mathlib.

Because Lake only warns when a require's rev differs from the manifest and then uses the manifest, the copy is refused unless every [[require]] of the workspace's lakefile.toml is a git require whose name, URL (up to a trailing .git), rev and subDir match the root manifest's entry, and unless the workspace's lean-toolchain equals the root's (a different toolchain would recompile every shared dependency inside the sandbox). Either mismatch fails the evaluation with an error naming it, so stale generated workspaces are caught before anything is scored.

In submission.yml and public-replay-smoke.yml, lean-action no longer builds the benchmark root. A trusted step runs the new scripts/build_benchmark_root.py lean-eval instead. For a benchmark whose manifest pins TauCeti it requires scripts/fetch_dependency_caches.sh (added by leanprover/lean-eval#650, chore: download TauCeti's build outputs instead of compiling them), runs it without $GITHUB_ENV so the Lake cache settings stay inside that step, then runs lake build TauCeti (the whole library, so a submission may import any TauCeti module) and lake build with LAKE_RESTORE_ARTIFACTS=true. Every downloaded output is copied into the shared package build directories, and the first Built TauCeti... line stops the build and fails it, so a cache miss is an error rather than a silent compile. Later workspace builds, including comparator's sandboxed build (whose environment is reduced to PATH, HOME and LEAN_ABORT_ON_PANIC), find TauCeti up to date and need neither the artifact cache nor the network. Benchmarks that pin no TauCeti get a plain lake build as before. lake exe cache get is dropped on the shared workspace path because Mathlib's decompressed cache also lives in the shared tree, which lean-action primes at the root.

The manifest comes only from --repo-root (the trusted benchmark checkout). Submitter content is still limited to Submission.lean and Submission/**/*.lean, so a submission cannot supply a lake-manifest.json, and a missing, non-regular or symlinked root manifest fails closed. Workspaces that do not share packages (including when sharing fails) keep the existing lake update + lake exe cache get priming, and --preprimed-workspaces replays keep their baked manifests.

🤖 Prepared with Claude Code

kim-em and others added 5 commits September 26, 2026 05:01
Per-submission workspaces symlink .lake/packages to the benchmark root's
package directory. Running `lake update` in such a workspace can re-resolve
transitive dependencies and check out different revisions inside the
shared directory, corrupting it for every other workspace and for the
root. Now that a generated workspace can require a second package besides
Mathlib (TauCeti), the workspace's own resolution no longer necessarily
agrees with the root's pins.

When packages are shared, copy the trusted benchmark checkout's root
lake-manifest.json into the workspace instead of running `lake update`.
It pins every package any generated workspace requires, and Lake ignores
entries a workspace does not require. `lake exe cache get` is also
skipped: the shared tree, including Mathlib's decompressed build cache,
is primed at the benchmark root before evaluation. Workspaces that do
not share packages (including when sharing fails) keep running
`lake update` and `lake exe cache get`, and preprimed replay workspaces
keep their baked manifests.

The manifest is read only from --repo-root; submitter content is still
limited to Submission.lean and Submission/**/*.lean, so a submission
cannot supply a lake-manifest.json. A missing, non-regular or symlinked
root manifest fails closed.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
TauCeti must never be compiled during evaluation. lean-action's default
`lake build` of the benchmark root ran before anything configured
TauCeti's Lake artifact cache, so it compiled TauCeti from source.

Turn off lean-action's build, test and lint in the evaluation and public
replay workflows, and add a trusted step that runs the benchmark's
scripts/fetch_dependency_caches.sh when present (older pinned benchmarks
lack it and need no extra cache) and then builds the root with
LAKE_RESTORE_ARTIFACTS=true. The cache settings are evaluated inside that
step instead of being appended to $GITHUB_ENV, and restoring copies every
downloaded output into the shared package build directories. Later
workspace builds, including comparator's sandboxed build whose
environment is reduced to PATH, HOME and LEAN_ABORT_ON_PANIC, then find
TauCeti up to date there and need neither the artifact cache nor network.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Lake resolves a require whose rev differs from the manifest by warning and
using the manifest's rev. Installing the benchmark root's manifest into a
shared-packages workspace could therefore silently evaluate against
different dependencies than the workspace pins, for example Mathlib
db1c574 from the root while the generated lakefile still says d13f23b.

Before installing the manifest, require every [[require]] of the
workspace's lakefile.toml to be a git require whose name, URL (up to a
trailing .git), rev and subDir match the root manifest's entry, and fail
closed with an error naming the mismatch otherwise.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…ages

Every trace in the shared package tree comes from the benchmark root's
Lean toolchain. A generated workspace still pinning another toolchain
(for example v4.34.1 while the root moved to v4.34.0) recompiles all of
Mathlib and TauCeti inside comparator's sandbox, offline. Fail closed
before evaluation unless the workspace's lean-toolchain equals the
root's.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…miss

The trusted root build restored only the TauCeti modules the benchmark
imports, so a submission importing any other TauCeti module would compile
it offline inside the sandbox. A benchmark pinning TauCeti without
scripts/fetch_dependency_caches.sh, or a cache miss, also compiled
TauCeti silently.

Move the trusted step into scripts/build_benchmark_root.py. For a
benchmark whose manifest pins TauCeti it requires the fetch script, runs
it without $GITHUB_ENV so the Lake cache settings stay in this step,
then runs `lake build TauCeti` (the whole library) and `lake build` with
LAKE_RESTORE_ARTIFACTS=true, stopping and failing on the first
`Built TauCeti...` line. Benchmarks without TauCeti get a plain
`lake build` as before.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@kim-em
kim-em merged commit 8fe9cb3 into main Sep 26, 2026
9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant