Repository navigation
build(evals): run evals on the shipping Bazel binary - #4323
Open
ScriptedAlchemy wants to merge 19 commits into
Open
ScriptedAlchemy wants to merge 19 commits into
ScriptedAlchemy wants to merge 19 commits into
Conversation
…/retire-crate-distribution # Conflicts: # .github/workflows/ci.yml # scripts/BUILD.bazel
…te-distribution-update
…/retire-crate-distribution
Also runs the eval binary script test in the hermetic eval workflow.
|
ScriptedAlchemy
enabled auto-merge
October 18, 2026 17:06
ScriptedAlchemy
disabled auto-merge
October 18, 2026 18:07
ScriptedAlchemy
enabled auto-merge
October 18, 2026 18:37
ScriptedAlchemy
disabled auto-merge
October 18, 2026 19:18
…on' into simplify/retire-crate-distribution-update
…6-eval-bazel-cutover
2 of 6 tasks
2 of 6 tasks
Open
6 tasks
3 of 6 tasks
7 tasks
6 tasks
6 tasks
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #4206.
Depends on #4266, which removes the unpublished-crate distribution battery. This branch includes that dependency until it merges.
All three live eval harnesses now resolve their CLI through
scripts/build-eval-binary.sh, which builds the shipping Bazel target with--config=releaseby default (TRACEDECAY_EVAL_BAZEL_CONFIG=distselects release LTO/stripping). An explicitTRACEDECAY_BINselects an existing executable. Both paths log version and SHA-256; failed builds never reuse an older artifact. Remove Cargo builds, developer target directories, PATH fallbacks, hermetic debug builds, and stale-build overrides.The resolver selects
.exeon Windows shells. Agent-adoption isolation passes the outer resolved path into the child throughTRACEDECAY_BIN, avoiding a build under the disposable cache. Memory evals check both model-cost consent flags before resolving or building the CLI, and blocked reports explicitly leave the binary unset. Wire the existing memory-runner tests into Bazel and isolate their output directory.The dependency also fixes the shipped LSP smoke tool's Python import root and adds its helper checks as a Bazel test. The exported MCP smoke source is already executable through
bazel run.Verification:
bazel test //scripts:build_eval_binary_test //scripts:bazel_dependency_patches_test //evals/... --test_output=errors: the six existing suites passed; the newly wired//evals/memory:real_model_runner_testalso passed..exeselection, isolated re-entry without another build, explicit overrides, and refusal to reuse a stale artifact after build failure.//scripts:packaged_lsp_bridge_testpassed; both shipped transport smoke entrypoints start through Bazel and reach argument validation.bash -n scripts/build-eval-binary.shandgit diff --checkpassed.Live model turns were not run. Hosted CI runs production transport journeys against the shipped binary.