Repository navigation
fix(ci): stop the nightly 100K Rust recall gate from timing out - #1302
Closed
cyberlife-coder wants to merge 1 commit into
Closed
cyberlife-coder wants to merge 1 commit into
cyberlife-coder wants to merge 1 commit into
Conversation
Every run of "Scheduled Rust 100K recall (Fast/Balanced/Accurate/Perfect)" since it was added on 2026-06-24 has been cancelled at its 60-minute timeout — 8 consecutive nightly failures, confirmed via the Actions API. Two compounding causes: - `scale_recall_100k.rs` built a fresh 100K-vector HNSW index from scratch in each of 4 separate #[test] fns (one per quality mode) instead of sharing one fixture across all four recall checks, despite the fixture's own doc comment claiming it was "shared across tests". - The CI step ran `cargo test` without `--release`; debug-mode HNSW insert/search at 100K vectors is an order of magnitude slower, and no other perf-sensitive job in this workflow makes that mistake. Fix: merge the four tests into one that builds the fixture once and asserts all four thresholds, and add --release to the CI invocation. Verified locally in release mode: 112s total (previously never completed in 60 min), all four thresholds still pass at the same values (Fast 0.944, Balanced 0.974, Accurate 0.997, Perfect 1.000). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JRKmKy3vWHN55DSaPFkotE
Contributor
|
Thank you for your contribution to VelesDB! Before this PR can be merged, please sign the Contributor License Agreement by posting the following comment:
This grants the project a license to use your contribution under the VelesDB Core License 1.0 and future licensing terms. I have read the CLA Document and I hereby sign the CLA You can retrigger this bot by commenting recheck in this Pull Request. Posted by the CLA Assistant Lite bot. |
Up to standards ✅🟢 Issues
|
4 tasks done
Owner
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This session's branch is platform-assigned as
claude/admiring-fermi-vp1ou7, which doesn't match this repo's Git-Flow prefix allow-list (fix/*,ci/*, etc.) enforced bypr-governance.yml, and the commit's author identity (noreply@anthropic.com, fixed by the execution environment) will trip the "Reject AI-attributed commits" check. Both gates will likely fail automatically — flagging this now so you can cherry-pick the two-file diff onto a compliant branch/identity if you want it merged as-is, same as with #1289.Description
No open PRs or open issues existed in the repo at the start of this run, so I audited CI health on
developand found: every run of the nightly "Scheduled Rust 100K recall (Fast/Balanced/Accurate/Perfect)" job has been cancelled at its 60-minute timeout since it was added on 2026-06-24 — 8 consecutive nightly failures (runs #307–#351), confirmed via the Actions API. This gate exists to validate HNSW recall@10 across all four quality modes at 100K scale; it has never once produced a pass/fail signal.Fixes # (no tracking issue existed for this — found via CI audit)
Root cause (two compounding issues)
scale_recall_100k.rsbuilt a fresh 100K-vector HNSW index from scratch in each of 4 separate#[test]fns (one per quality mode), despite the fixture helper's own doc comment claiming it was "shared across tests" — it was only a shared function, never shared state. Index construction dominates the gate's runtime, so this alone was a 4x cost multiplier.cargo testwithout--release. Debug-mode HNSW insert/search at 100K vectors is an order of magnitude slower, and it's the only perf-sensitive job in this workflow that made that mistake — its sibling Python 100K job's own comment already states "Release build is mandatory for a perf gate."Changes
crates/velesdb-core/tests/scale_recall_100k.rs: merged the four#[test]fns into one (scale_100k_recall_all_qualities) that builds the fixture once and asserts all four per-mode thresholds, collecting failures instead of stopping at the first..github/workflows/perf-gate-e2e.yml: added--releaseto thecargo testinvocation for this job.Type of Change
How Has This Been Tested?
cargo fmt --check -p velesdb-core— cleancargo clippy -p velesdb-core --test scale_recall_100k --release --features persistence -- -D warnings -D clippy::pedantic— cleancargo check -p velesdb-core --tests --features persistence(dev profile) — compilesyaml.safe_loadTest Configuration:
Checklist
Additional Notes
No functional/behavioral change to what's validated — same four thresholds, same brute-force ground truth, same single-threaded execution constraint (shared filesystem state per CLAUDE.md). Purely a runtime-efficiency and CI-config fix so the gate can actually run to completion.