Repository navigation
fix: pin shared-package workspaces with the root lake manifest - #1807
Merged
Merged
Conversation
Per-submission workspaces symlink .lake/packages to the benchmark root's package directory. Running `lake update` in such a workspace can re-resolve transitive dependencies and check out different revisions inside the shared directory, corrupting it for every other workspace and for the root. Now that a generated workspace can require a second package besides Mathlib (TauCeti), the workspace's own resolution no longer necessarily agrees with the root's pins. When packages are shared, copy the trusted benchmark checkout's root lake-manifest.json into the workspace instead of running `lake update`. It pins every package any generated workspace requires, and Lake ignores entries a workspace does not require. `lake exe cache get` is also skipped: the shared tree, including Mathlib's decompressed build cache, is primed at the benchmark root before evaluation. Workspaces that do not share packages (including when sharing fails) keep running `lake update` and `lake exe cache get`, and preprimed replay workspaces keep their baked manifests. The manifest is read only from --repo-root; submitter content is still limited to Submission.lean and Submission/**/*.lean, so a submission cannot supply a lake-manifest.json. A missing, non-regular or symlinked root manifest fails closed. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
TauCeti must never be compiled during evaluation. lean-action's default `lake build` of the benchmark root ran before anything configured TauCeti's Lake artifact cache, so it compiled TauCeti from source. Turn off lean-action's build, test and lint in the evaluation and public replay workflows, and add a trusted step that runs the benchmark's scripts/fetch_dependency_caches.sh when present (older pinned benchmarks lack it and need no extra cache) and then builds the root with LAKE_RESTORE_ARTIFACTS=true. The cache settings are evaluated inside that step instead of being appended to $GITHUB_ENV, and restoring copies every downloaded output into the shared package build directories. Later workspace builds, including comparator's sandboxed build whose environment is reduced to PATH, HOME and LEAN_ABORT_ON_PANIC, then find TauCeti up to date there and need neither the artifact cache nor network. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Lake resolves a require whose rev differs from the manifest by warning and using the manifest's rev. Installing the benchmark root's manifest into a shared-packages workspace could therefore silently evaluate against different dependencies than the workspace pins, for example Mathlib db1c574 from the root while the generated lakefile still says d13f23b. Before installing the manifest, require every [[require]] of the workspace's lakefile.toml to be a git require whose name, URL (up to a trailing .git), rev and subDir match the root manifest's entry, and fail closed with an error naming the mismatch otherwise. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…ages Every trace in the shared package tree comes from the benchmark root's Lean toolchain. A generated workspace still pinning another toolchain (for example v4.34.1 while the root moved to v4.34.0) recompiles all of Mathlib and TauCeti inside comparator's sandbox, offline. Fail closed before evaluation unless the workspace's lean-toolchain equals the root's. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…miss The trusted root build restored only the TauCeti modules the benchmark imports, so a submission importing any other TauCeti module would compile it offline inside the sandbox. A benchmark pinning TauCeti without scripts/fetch_dependency_caches.sh, or a cache miss, also compiled TauCeti silently. Move the trusted step into scripts/build_benchmark_root.py. For a benchmark whose manifest pins TauCeti it requires the fetch script, runs it without $GITHUB_ENV so the Lake cache settings stay in this step, then runs `lake build TauCeti` (the whole library) and `lake build` with LAKE_RESTORE_ARTIFACTS=true, stopping and failing on the first `Built TauCeti...` line. Benchmarks without TauCeti get a plain `lake build` as before. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR makes
scripts/evaluate_submission.pycopy the benchmark checkout's rootlake-manifest.jsoninto each per-submission workspace whose.lake/packagesis symlinked to the sharedlean-eval/.lake/packages, instead of runninglake updateandlake exe cache getthere, and makes the evaluation workflows download TauCeti's build outputs instead of compiling them. Runninglake updateagainst the shared directory can re-resolve transitive dependencies and check out different revisions inside it, corrupting the tree for every other workspace and for the benchmark root; the root manifest already pins every package any generated workspace requires, and Lake ignores entries a workspace does not require. This matches what lean-eval's own CI andreuseRootPackagesdo, and is needed for leanprover/lean-eval#649 (feat: add the classification of finite simple groups to v2), whose workspace requires TauCeti as well as Mathlib.Because Lake only warns when a require's rev differs from the manifest and then uses the manifest, the copy is refused unless every
[[require]]of the workspace'slakefile.tomlis a git require whose name, URL (up to a trailing.git), rev andsubDirmatch the root manifest's entry, and unless the workspace'slean-toolchainequals the root's (a different toolchain would recompile every shared dependency inside the sandbox). Either mismatch fails the evaluation with an error naming it, so stale generated workspaces are caught before anything is scored.In
submission.ymlandpublic-replay-smoke.yml, lean-action no longer builds the benchmark root. A trusted step runs the newscripts/build_benchmark_root.py lean-evalinstead. For a benchmark whose manifest pins TauCeti it requiresscripts/fetch_dependency_caches.sh(added by leanprover/lean-eval#650, chore: download TauCeti's build outputs instead of compiling them), runs it without$GITHUB_ENVso the Lake cache settings stay inside that step, then runslake build TauCeti(the whole library, so a submission may import any TauCeti module) andlake buildwithLAKE_RESTORE_ARTIFACTS=true. Every downloaded output is copied into the shared package build directories, and the firstBuilt TauCeti...line stops the build and fails it, so a cache miss is an error rather than a silent compile. Later workspace builds, including comparator's sandboxed build (whose environment is reduced toPATH,HOMEandLEAN_ABORT_ON_PANIC), find TauCeti up to date and need neither the artifact cache nor the network. Benchmarks that pin no TauCeti get a plainlake buildas before.lake exe cache getis dropped on the shared workspace path because Mathlib's decompressed cache also lives in the shared tree, which lean-action primes at the root.The manifest comes only from
--repo-root(the trusted benchmark checkout). Submitter content is still limited toSubmission.leanandSubmission/**/*.lean, so a submission cannot supply alake-manifest.json, and a missing, non-regular or symlinked root manifest fails closed. Workspaces that do not share packages (including when sharing fails) keep the existinglake update+lake exe cache getpriming, and--preprimed-workspacesreplays keep their baked manifests.🤖 Prepared with Claude Code