Skip to content

fix: restore identifier retrieval and preserve keyword cache data - #478

Merged
Helweg merged 6 commits into
mainfrom
fix-retrieval-quality
Oct 30, 2026
Merged

Helweg merged 6 commits into
mainfrom
fix-retrieval-quality

Conversation

@Helweg

@Helweg Helweg commented Oct 29, 2026 •

Copy link
Copy Markdown
Owner

Final reviewed merge candidate

c7d6894 is the corrected candidate. The user explicitly authorized review and merge; no tag, publication or installed-plugin update is included.

Source review found a P2 lifecycle bug missing from V8 coverage: an already-used structural writer could overwrite an unsupported future keyword snapshot after freshness refresh. A new same-Indexer regression failed on78f1f7a with outcome ready instead of InvalidArg. The correction validates the persisted version under the existing mutation lease without replacing the active keyword map. A first unconditional-reload attempt broke failed-publication retry and was rejected; both failures remain preserved.

Corrected-head evidence

  • 88 focused tests passed, including fresh/reused hybrid and structural writers, exact unsupported-version error, zero document embeddings, protected DB/WAL/keyword/generation bytes, non-mutating native validation, and retry after a failed staged save.
  • Exact-commit Rust formatting, Clippy, Rust tests, build, typecheck, lint and full Vitest gate: 155 passed test files / 3,101 passed tests / 7 skipped. All16 source pins stayed unchanged; four captured gate-log hashes were independently verified.
  • Fresh CI: all15checks successful at this exact head: https://github.com/Helweg/open-codebase-index/actions/runs/41024655327 . Includes Node22/24 coverage, package consumers, evaluation smoke/pre-edit gate, exact minimum Node22.22 smoke, CodeQL and macOS/Linux/Windows path semantics.
  • Actual SDK MCP session using the final rebuilt addon: ordinary index, checked current status, replace the snapshot with tokenizer version2, checked unreadable status, then ordinary indexing rejects in the same server process. Four actual tool calls; protected keyword/DB/WAL/generation hashes unchanged; future version2 remains after close; child PID positively exited. Exact same-Indexer object reuse is proven by the permanent regression, not inferred from process identity alone.
  • Final actually loaded native SHA256: b8e25775add24c433b773a4495db93648320682261d993df4b92d945f29b4c30. All16maps/160project source identities, including131MCPworker sources, match the final compiled code.

Private evidence: /tmp/ocbi-retrieval-repair-QsU27J/validation-v9/receipt.json and review-merge-v9/exact-head-review.json, same-writer-mcp-final/receipt.json, retained failing logs.

Provenance and limits

The broader V8 real-provider/mock/installed-upgrade/omp studies below describe78f1f7a and remain immutable historical supporting evidence, not re-labeled measurements or native hashes for this corrected head. Current-head proof is the focused regressions, full gate, fresh CI and actual rebuilt-native MCP scenario above. Ranking/body/coverage/install-hook limitations remain disclosed below. The three historical real-provider misses are not silently declared fixed by this safety correction.

Prior offline auditors V3/V4 ran before the worker consumed review/HOLD instructions. They were not retrospectively authorized. V4 is supporting evidence only; acceptance rests on independent sealed-input and actual-runtime receipt checks.


Preserved V8 qualification (superseded by the review correction)

Previously qualified for review on 78f1f7a. All unchanged acceptance gates pass on the exact final V8 artifact. No merge, tag, publication or installed-user-plugin update is performed by this qualification.

The earlier V7 genuinely failed installed default-hybrid future-cache rejection. Its commands, inputs and failure remain immutable; this PR repairs the shared freshness/coordinator shortcut and qualifies newly frozen V8 bytes, not relabeled V7 results.

Changes

  • Match natural-language words within camelCase/PascalCase/acronym identifiers in BM25 and lexical definition ranking. Preserve whole-identifier query isolation, Unicode behavior and original-case cache semantics.
  • Keep legacy keyword snapshots readable. Ordinary active-branch refresh upgrades tokenization without reembedding unchanged chunks; persist inactive-branch pending state. Reject unsupported future cache versions without deleting catalog/vector/keyword data.
  • Report degraded optional keyword reads as keyword-index-unreadable, not healthy/current. Keep valid semantic retrieval available while preserving source/configuration/HEAD/compatibility precedence. Manual indexing cannot skip unsupported-format validation through the current-index shortcut.
  • Prefer source implementations for explicit code requests without overriding explicit documentation/test/configuration intent.
  • Explicitly watch selected and local-module configuration dependencies in Chokidar, including explicit host config paths, preserving transitive alias-graph refresh.
  • Reconcile edits made before initial/promoted-leader watcher readiness through one ordinary coordinated catch-up when automatic indexing is enabled; preserve watcher-only and stopped behavior.
  • Ignore late completion of an already-interrupted MCP diagnostic operation, preventing post-shutdown persistence and runtime-directory recreation.

Exact final-source verification

Check Exercised result
Rust cargo fmt --check, cargo clippy -- -D warnings, cargo test: PASS
Canonical full gate Build, typecheck, lint and full tests: 155 passed test files / 3,098 passed tests / 7 skipped
Public CI All 15 reported checks successful on this exact commit; Node 22/24 tests, CodeQL and macOS/Linux/Windows path-semantics jobs
Real-provider representative retrieval Same original corpus: 11/16 before → 12/16 after Hit@5, meeting the unchanged 12/16 floor. Final-source corpus: 13/16; source drift is a separate comparison
Unchanged mock gates All six pass on first final-artifact attempts; 38 actual MCP responses and complete per-query/reference/rendered loss audits retained
Installed upgrade Both identities × structural/default-hybrid = four lanes, upgrading actual registry 0.38.2 with exact Node 22.22.0
Migration/data protection Cold exact-byte/catalog equality, zero document-embedding POSTs for unchanged keyword refresh, partial two-branch migration, stable caller/test embedding BLOBs and vector bytes
Installed interfaces 96 exact native query assertions, six actual SDK MCP sessions, both CLI aliases, all four installed lazy workers; 79 native / 103 interpreter receipts
Future cache Preferred package ordinary CLI rejects tokenizer version 2 with exit 1 in both modes, zero refresh embeddings and unchanged protected bytes. Hybrid remains readable/noncurrent and retrieves current source through real vector fallback
Actual omp Three normal-profile sessions, 10 native tool calls: current/cached-vs-fresh, legacy readable/noncurrent, and missing index; natural source retrieval, linked test/caller evidence and working-tree impact

Public CI: https://github.com/Helweg/open-codebase-index/actions/runs/40900758957

The strengthened real-native coordinated-cache regression failed before the fix. Its passing result also covers auto-index-enabled semantic readiness and stale-source precedence; 64 focused freshness/coordinator/cache tests passed. Final full tests and installed probes cover the final rebuild rather than stale native bytes.

Frozen artifact and evidence

  • V8 pin SHA256: 11ca2b6848466d15daa38dc2b84507dfc147a7cfc0bd902a542ac7ce36741d6e.
  • Actually loaded final native SHA256: e331f44f197cefbff761323ef93b720fe32051dcca4f5874641f2843cb0e8259.
  • All 758 frozen files, 160 compiled project-source identities including 131 MCP-worker sources, and final corpus/Git-blob identity were verified. Final runtime was rehashed after actual host/consumer execution.
  • Private evidence: /tmp/ocbi-retrieval-repair-QsU27J/. Key records: validation-v8/receipt.json, real-gate-v8-audit.json, mock-verification-v8/report.md, upgrade-verification-v8/report.md, omp-verification-v8/host-summary.json and parent reviews.
  • Parent independently verified all 305 mock report artifacts, all 1,320 sealed consumer inputs, final report manifests, actual installed native paths and critical raw upgrade/query/future-cache receipts. All owned runtime PIDs exited and temporary provider ports closed; both settled workers were released. User-owned retained terminals were preserved.

Limits and preserved failures

  • Not perfect/global ranking dominance. Final real retrieval still misses crashed-index-lock recovery, duplicate-definition path disambiguation and embedding-failure recovery. All prior positive real targets remain; the negative control remains negative.
  • Mock audits retain original evaluation-budget/indexer representative/top10 losses, source-watcher graded rank 7→8 and six ancillary V7 reference removals. No previously positive Hit@1/3/5/10, full expected target/definition/caller or budget criterion is lost. These bounded ranking tradeoffs are disclosed, not a failed acceptance gate waived. Full individual old/new references remain in the audit.
  • Four selected callee destinations are not rendered as destination implementation bodies. Rank, named hubs and static test linkage are not complete bodies, graph coverage or executed tests; one evaluation target is explicitly budget-truncated. Single-run latency is not performance proof.
  • Original historical rendered text/wire bytes were never captured and remain unknown. Reconstructed historical cited-source-span hashes are static source evidence only, not regenerated historical consumer output; missing hashes were not counted as changed bodies.
  • Installed consumer qualification is local macOS ARM64 / exact Node 22.22.0 and used --ignore-scripts: install lifecycle hooks were not qualified. Cross-platform CI is separate evidence, not a claim that this consumer matrix ran on every OS.
  • SQLite SHM is transient coordination memory, not database content. Actual omp retains raw before/after sidecars and exact changed offsets separately; durable DB/WAL/vector/keyword/source/config bytes remain unchanged. Installed cold/future comparisons retained complete exact snapshots with no sidecar exception.
  • Every original product/harness failure and revised probe is retained, with no unchanged rerun to obtain green. Three consumer offline-auditor assumptions failed and were corrected in new retained source versions. Offline V3/V4 executed before the worker consumed a coordinator review HOLD; this breach is disclosed, not retrospectively authorized. Parent subsequently reviewed sources and independently verified sealed inputs/critical actual-runtime receipts; V4 is supporting evidence only, with no runtime/auditor replay after review.
  • The frozen pin's predecessor annotation self-references V8; a separate immutable lineage addendum identifies actual V7 ancestry without changing the pin or execution authority.

Original qualification and user checkout/native binary remain untouched. Installed plugin stays at 0.38.2; no provider/model/service, global configuration or auth edits were made.

@Helweg Helweg added bug Bug fix semver:patch Bug fix or patch labels Oct 29, 2026
@Helweg
Helweg marked this pull request as ready for review October 29, 2026 16:21
@Helweg
Helweg marked this pull request as draft October 30, 2026 07:49
@Helweg

Helweg commented Oct 30, 2026

Copy link
Copy Markdown
Owner Author

Merge held during the requested source review. A new same-Indexer regression reproduced a structural-writer lifecycle bug at head78f1f7a: after replacing the keyword snapshot with tokenizer version2, coordinated indexing returned ready rather than InvalidArg. The existing tests covered a new reader, not writer reuse. The correction reloads the keyword snapshot under the existing mutation lease before the structural-mode/vector-store early return. Fresh/reused writer tests now cover both modes, explicit unsupported-version error, zero document embeddings, and protected DB/WAL/keyword/generation bytes. Source gate and exact corrected artifact qualification are pending; V8 receipts remain unchanged and are not relabeled.

@Helweg Helweg left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed exact head c7d6894.

Two bounded static review slices covered native/cache/freshness and watcher/MCP lifecycle; parent also reviewed ranking/intent/contracts. The cache review found a P2 reused-structural-writer data-loss path. It is now corrected with non-mutating native version validation under the mutation lease, preserving pending keyword state needed after failed publication. Same-Indexer fresh/reused tests cover both modes and failed-save retry; the before-fix failure and rejected unconditional-reload attempt are retained.

No merge-blocking findings remain.88focused tests and the exact-head full gate pass (3101passed,7skipped); all15fresh CI checks are successful. An actual SDK MCP process with the final addon rejects future-version indexing and preserves protected bytes; source/native/map identity and child exit are verified.

Prior V8 studies remain supporting evidence at78f1f7a, not relabeled as this head. The prior offline V3/V4 checkpoint breach is explicitly disclosed: not retrospectively authorized; V4 supporting only; parent independently checked sealed inputs/runtime receipts.

Proceeding with the user's authorized squash merge, fenced to this exact head; no release/tag/plugin update.

@Helweg
Helweg marked this pull request as ready for review October 30, 2026 08:28
@Helweg
Helweg merged commit 6bbdc6d into main Oct 30, 2026
15 checks passed
@Helweg

Helweg commented Oct 30, 2026

Copy link
Copy Markdown
Owner Author

Review and merge completed. Squash commit6bbdc6de939a10110c2fba61c50a9196a9a4c207 is main; its treee2bd9dbcdb6b8e05c40f5b732ebdee2dcb7c7bde exactly matches reviewed headc7d6894. The P2 reused-structural-writer defect was reproduced and repaired before merging;3101local tests, Rust/build/typecheck/lint, all15PR checks and final-native actual MCP data-protection proof passed. Automatic post-merge main CI is also successful: https://github.com/Helweg/open-codebase-index/actions/runs/41026721571 . No release/tag/publication, installed-plugin update, local checkout synchronization or source-branch deletion performed. Historical qualification limits and offline-auditor authorization caveat remain as documented.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Bug fix semver:patch Bug fix or patch

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant