Skip to content

build(evals): run evals on the shipping Bazel binary - #4323

Open
ScriptedAlchemy wants to merge 19 commits into
masterfrom
fix/4206-eval-bazel-cutover
Open

ScriptedAlchemy wants to merge 19 commits into
masterfrom
fix/4206-eval-bazel-cutover

Conversation

@ScriptedAlchemy

@ScriptedAlchemy ScriptedAlchemy commented Oct 18, 2026 •

Copy link
Copy Markdown
Owner

Fixes #4206.

Depends on #4266, which removes the unpublished-crate distribution battery. This branch includes that dependency until it merges.

All three live eval harnesses now resolve their CLI through scripts/build-eval-binary.sh, which builds the shipping Bazel target with --config=release by default (TRACEDECAY_EVAL_BAZEL_CONFIG=dist selects release LTO/stripping). An explicit TRACEDECAY_BIN selects an existing executable. Both paths log version and SHA-256; failed builds never reuse an older artifact. Remove Cargo builds, developer target directories, PATH fallbacks, hermetic debug builds, and stale-build overrides.

The resolver selects .exe on Windows shells. Agent-adoption isolation passes the outer resolved path into the child through TRACEDECAY_BIN, avoiding a build under the disposable cache. Memory evals check both model-cost consent flags before resolving or building the CLI, and blocked reports explicitly leave the binary unset. Wire the existing memory-runner tests into Bazel and isolate their output directory.

The dependency also fixes the shipped LSP smoke tool's Python import root and adds its helper checks as a Bazel test. The exported MCP smoke source is already executable through bazel run.

Verification:

  • bazel test //scripts:build_eval_binary_test //scripts:bazel_dependency_patches_test //evals/... --test_output=errors: the six existing suites passed; the newly wired //evals/memory:real_model_runner_test also passed.
  • Final resolver rerun passed, covering Windows .exe selection, isolated re-entry without another build, explicit overrides, and refusal to reuse a stale artifact after build failure.
  • Memory-runner tests cover blocked reports with neither or only one consent flag, without binary resolution.
  • Dependency //scripts:packaged_lsp_bridge_test passed; both shipped transport smoke entrypoints start through Bazel and reach argument validation.
  • bash -n scripts/build-eval-binary.sh and git diff --check passed.

Live model turns were not run. Hosted CI runs production transport journeys against the shipped binary.

@changeset-bot

changeset-bot Bot commented Oct 18, 2026 •

Copy link
Copy Markdown

⚠️ No Changeset found

Latest commit: a70bf58

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

devin-ai-integration[bot]

This comment was marked as resolved.

devin-ai-integration[bot]

This comment was marked as resolved.

@ScriptedAlchemy ScriptedAlchemy changed the title Run every eval on the shipping Bazel binary; share asupersync patches with the distribution build build(evals): run evals on the shipping Bazel binary Oct 18, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Finish eval and platform build cutover with shipping artifact and dependency patch parity

1 participant