feat(semantic): local hybrid semantic_search MCP tool (+ filters, docs, deps) - #89
Merged
Merged
Conversation
- Add parseDateFilter + filterByDateRange to LocalRepositoryDataSource: single day (YYYY-MM-DD) and inclusive range (YYYY-MM-DD..YYYY-MM-DD) with validation for malformed/reversed ranges and array inputs - Replace naive '>=' comparison that only supported open-ended dates - Improve list_advisories schema descriptions (date format, examples, defaults) - Add 14 unit tests covering parse/filter edge cases Cleanly re-applies the feature from #25 onto current main (post-#76); supersedes that stale branch.
) list_advisories/search ecosystem filter used exact string match against OSV data, so only 'npm' matched; composer/pip/maven/rust/etc. silently returned 0. - Add ECOSYSTEM_ALIASES (GitHub enum -> OSV name) + case-insensitive ecosystemMatches() - Apply at both filter sites (listAdvisories + filterResults/search) - Add unit tests for all 12 ecosystems; make e2e assertions real regression guards (drop vacuous length>0 guard, use ecosystemMatches instead of fragile [0]===name) Fixes #78
cwes filter did options.cwes.includes(cwe.cwe_id) i.e. checked if bare input like '89' contains 'CWE-89' - always false. Documented bare-number form never matched. - Add normalizeCwe() + cweFilterMatches(): accept bare (89) or prefixed (CWE-89), comma-separated and/or array, case-insensitive; match if any requested CWE is present - Add unit tests (test/unit/cwe-filter.test.ts) Fixes #80
The MCP tools read ADVISORY_API_BASE (hardcoded :18005) while the server binds ADVISORY_API_PORT, so a custom API port broke every tool with 'fetch failed', and two servers collided on 18005 (serving each other's data - the source of the E2E flakiness). - Derive base URL from ADVISORY_API_HOST/PORT; keep ADVISORY_API_BASE as override - CI E2E now runs on a non-default API port (18055) to guard against regressions Fixes #81
Distinct from the test defaults (18005/18006) so the dev MCP server and the test suite can run simultaneously without colliding. Relies on the tool port fix in this PR.
…ral rerank) Local-only, advisory-specific prototype on branch proto/semantic-search. No external engine/service. - Local ONNX embeddings via @huggingface/transformers (MiniLM 384-dim), offline (reuse a cached model dir) - Compact Okapi BM25 with identifier-preserving tokenizer - RRF fusion + field-aware rerank (exact GHSA/CVE/package/CWE/phrase boosts) - Temporal-aware rerank: parse a period from the query, run recall on the residual text, boost by publish-date proximity (in-window 1.0, exp decay half-life 45d) - File-backed index (embeddings.bin/bm25.json/docs.json/meta.json), CLI build+query. Not wired into MCP tools yet.
Registers semantic_search in createAdvisoryServer (stdio + HTTP): hybrid local search with optional web_app_only/severity/ecosystem/cwes post-filters and temporal reranking. Returns build instructions if the index is absent.
…CI feasibility) Covers what/why to redistribute, git-lfs channel, cross-platform/ABI portability (LE + model pinning), and GitHub Actions scheduling feasibility (full rebuild ~borderline in 30m; incremental = seconds).
…ribution, weekly refresh, sparse checkout Covers the semantic_search tool + pipeline, empirical build/size numbers (35k in ~22m local; index ~96MB), git-lfs on a dedicated semantic-index branch, weekly scheduled refresh, and sparse-checkout/partial-clone recipes to avoid clone bloat.
- 82 unit tests across bm25, temporal, document, store (save/load round-trip incl. embeddings byte-exactness), and hybrid (RRF + field/temporal rerank via injected query embedding — no model needed) - Add @vitest/coverage-v8 + test:coverage script; per-file coverage thresholds for the deterministic semantic modules - Hygiene: store index dir resolved at runtime (indexDir()) not import time; hybridSearch accepts an injectable query embedding for tests; drop unused import
# Conflicts: # package-lock.json # package.json # src/datasources/local-repository.ts # src/tools/advisories.ts
- Add root AGENTS.md: build/bootstrap/run/test contract for agents (generic, no internal infra) - CONTRIBUTING: Node 20+, branch from main (no dev branch), unit-test command - README: correct advisory counts (~370K/~35K), CI Node matrix 20.x/22.x, main-only triggers, drop stale CI notes
- Move raw REST/MCP JSON-RPC recipes to docs/http-api.md and orchestrator/rate-limit examples to docs/integration.md; README 428->341 lines (M2) - Scope 'npm test' to unit; add test:all; skip Azure integration suite via describe.skipIf when AZURE_OPENAI_ENDPOINT unset (M3) - src/semantic/README.md: scope to how-to-run and link canonical docs/semantic-search-design.md; fix stale 'not wired in' note (M4)
@huggingface/transformers pins a vulnerable sharp ^0.34.1; add an overrides entry (sharp>=0.35.4) to resolve GHSA-rgj7-g3m4-5g8c (libheif) and GHSA-f88m-g3jw-g9cj (libvips). npm audit: 0 vulnerabilities. Rationale documented in src/semantic/embeddings.ts (our usage is text-only).
…dk/azure) Lockfile regenerated on public npm (ai 7.0.99, zod 4.6.5, @types/node 26.5.1, @ai-sdk/azure 4.0.70), keeping sharp>=0.35.4 override + transformers/coverage-v8. Validated by CI build+unit tests; avoids downgrading main on merge. NOTE: this device's CFS proxy still lags these versions, so local npm ci needs the proxy to catch up; build/tests run on the already-installed modules.
Dependency ReviewThe following issues were found:
|
Adds .github/workflows/semantic-index.yml: weekly (cron 0 6 * * 1) + manual rebuild of the reviewed-tier index, published as the rolling 'semantic-index-latest' Release asset (~100 MB/week). Release (not LFS-to-main) because main is protected; keeps the blob off clones. Docs updated with consumer steps.
…edule/release yet) workflow_dispatch only; authorize job restricts to admin/maintain; uploads .semantic-index as a downloadable artifact (contents: read, no repo write, no Release). Weekly schedule + distribution deferred and documented. Addresses: run on demand, download as artifact without releasing, and maintainer-only trigger.
sam270411
approved these changes
Sep 14, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a local, offline
semantic_searchMCP tool for advisories, plus thereview-tier / web-app filters it builds on, and a round of doc/dependency hygiene.
Everything runs on-device — no external search engine or service.
What's included
Feature —
semantic_search(prototype)@huggingface/transformers, MiniLM384-dim) + Okapi BM25, fused with Reciprocal Rank Fusion, then field-aware +
temporal reranking (a period in the query — "SSRF in August 2026", "recent" —
is parsed out and used to weight by publish-date proximity).
createAdvisoryServer(), alongsidelist_advisories/get_advisory. Index loads lazily; if absent, the tool returnsbuild instructions (non-fatal).
web_app_only,severity,ecosystem,cwes.src/semantic/; build CLIdist/semantic/build-index.js.Datasource filters (used by the tool + REST)
typereview tier (revieweddefault /unreviewed/all, gated byADVISORY_INCLUDE_UNREVIEWED) andweb_app_only(curated web-app CWE set).Tests
thresholds.
npm testscoped totest/unit; Azure integration suite nowdescribe.skipIf(!AZURE_OPENAI_ENDPOINT)so it skips cleanly without creds.Docs / hygiene
AGENTS.md(build/bootstrap/run/test contract).README.mdtrimmed 428→~340 lines; low-level REST/MCP recipes →docs/http-api.md,orchestrator/rate-limit examples →
docs/integration.md; stale counts/CI facts fixed.docs/semantic-search-design.md+docs/semantic-index-distribution.md;src/semantic/README.mdpoints to them.Security / deps
overrides: { "sharp": ">=0.35.4" }clears two transitive high-severity advisories(
GHSA-rgj7-g3m4-5g8c,GHSA-f88m-g3jw-g9cj) pulled via@huggingface/transformers;npm audit= 0.main(ai 7.0.99, zod 4.6.5, @types/node 26.5.1,@ai-sdk/azure 4.0.70) — lockfile regenerated on public npm, no downgrade on merge.
Testing
npm run buildclean;npx vitest run test/unit→ 82/82.Semantic-index build workflow (manual, maintainer-gated)
.github/workflows/semantic-index.ymlis a manual (workflow_dispatch) build thatrebuilds the index and uploads it as a downloadable artifact (
semantic-index,14-day retention). It writes nothing to the repo (
contents: read, no Release, nobranch-protection interaction, no clone bloat). An
authorizejob restricts triggeringto admins/maintainers — on top of the write access
workflow_dispatchalreadyrequires; optional hard enforcement via an Environment with required reviewers is
documented. The weekly
scheduleis deferred (not enabled yet);docs/semantic-index-distribution.mddescribes how to turn it on and how to distributea scheduled build's output (Release asset or a git-lfs branch).
Notes
semantic_searchis a prototype; scaling beyond the ~35k reviewed corpus to the full~370k needs ANN (hnswlib) — noted in the design doc.