feat(semantic): local hybrid semantic_search MCP tool (+ filters, docs, deps)

.github/workflows/build.yml

@@ -34,7 +34,7 @@ jobs:
       run: ls -lh dist/
 
     - name: Run unit tests
-      run: npx vitest run test/unit
+      run: npx vitest run test/unit --coverage
 
   e2e:
     runs-on: ubuntu-latest

.github/workflows/semantic-index.yml

@@ -0,0 +1,95 @@
+name: Semantic Index (manual build)
+
+# Manually-triggered rebuild of the local semantic-search index. Produces a
+# downloadable workflow ARTIFACT (`semantic-index`) — no Release, no repo write.
+# Restricted to maintainers/admins via the `authorize` gate below. The weekly
+# `schedule` is intentionally NOT enabled yet — see
+# docs/semantic-index-distribution.md for how to turn it on and lock it down.
+
+on:
+  workflow_dispatch:
+    inputs:
+      limit:
+        description: "Advisories to embed (0 = all reviewed ~35k; e.g. 2000 for a quick test)"
+        required: false
+        default: "2000"
+
+permissions:
+  contents: read # read repo + upload artifact only; never writes to the repo
+
+concurrency:
+  group: semantic-index
+  cancel-in-progress: false
+
+jobs:
+  # Only maintainers/admins may proceed. workflow_dispatch already requires write
+  # access; this narrows it to maintain/admin.
+  authorize:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Require maintainer or admin
+        env:
+          GH_TOKEN: ${{ github.token }}
+        run: |
+          PERM=$(gh api "repos/${{ github.repository }}/collaborators/${{ github.actor }}/permission" --jq '.permission' 2>/dev/null || echo "unknown")
+          echo "Actor '${{ github.actor }}' permission: $PERM"
+          case "$PERM" in
+            admin|maintain) echo "Authorized." ;;
+            *) echo "::error::Only maintainers/admins may run this workflow (actor has '$PERM')."; exit 1 ;;
+          esac
+
+  build-index:
+    needs: authorize
+    runs-on: ubuntu-latest
+    timeout-minutes: 60 # a full ~35k rebuild fits comfortably (~30 min budget)
+    steps:
+      - uses: actions/checkout@v7
+
+      - uses: actions/setup-node@v7
+        with:
+          node-version: 22.x
+          cache: npm
+
+      - name: Install dependencies
+        run: npm ci
+
+      - name: Build TypeScript
+        run: npm run build
+
+      - name: Clone advisory database (shallow)
+        run: |
+          git clone --depth=1 --branch=main \
+            https://github.com/github/advisory-database.git external/advisory-database
+
+      # Cache the ONNX model weights between runs (miss -> downloaded from HF).
+      - name: Cache embedding model
+        uses: actions/cache@v4
+        with:
+          path: ~/.cache/semantic-model
+          key: semantic-model-minilm-v1
+
+      - name: Build semantic index
+        env:
+          ADVISORY_REPO_PATH: ./external/advisory-database
+          SEMANTIC_MODEL_CACHE: ~/.cache/semantic-model
+          SEMANTIC_ALLOW_REMOTE: "true" # CI runner may fetch the model (no corp block)
+        run: |
+          LIMIT="${{ github.event.inputs.limit }}"
+          LIMIT="${LIMIT:-2000}"
+          if [ "$LIMIT" = "0" ]; then
+            node dist/semantic/build-index.js
+          else
+            node dist/semantic/build-index.js --limit "$LIMIT"
+          fi
+
+      - name: Index stats
+        run: |
+          ls -lh .semantic-index
+          echo "---- meta.json ----"; cat .semantic-index/meta.json
+
+      - name: Upload index artifact
+        uses: actions/upload-artifact@v4
+        with:
+          name: semantic-index
+          path: .semantic-index/
+          retention-days: 14

.gitignore

@@ -34,3 +34,7 @@ Thumbs.db
 
 # External advisory database (git submodule/clone)
 external/
+
+# Semantic-search prototype: local index + downloaded model weights
+.semantic-index/
+models/

AGENTS.md

@@ -0,0 +1,117 @@
+# AGENTS.md
+
+Guidance for AI coding agents (and humans) working in this repository. Keep changes
+minimal, typed, and covered by tests. For contribution policy see
+[CONTRIBUTING.md](CONTRIBUTING.md); for user-facing docs see [README.md](README.md).
+
+## What this is
+
+An MCP server that serves GitHub Security Advisories from a local clone of
+`github/advisory-database`. Two-tier design: **MCP server (stdio or HTTP) → local
+Express REST API → `LocalRepositoryDataSource`** (reads advisory JSON from disk).
+
+Core MCP tools: `list_advisories`, `get_advisory` (full parameters in the README).
+
+## Prerequisites
+
+- **Node.js 20+** (CI runs 20.x and 22.x)
+- Git
+- The server runs from `dist/` — **rebuild after every source change**.
+
+## Build
+
+```bash
+npm install      # or: npm ci  (installs from package-lock.json)
+npm run build    # tsc -> dist/
+```
+
+## Bootstrap the advisory database (needed to run/serve)
+
+The server reads advisory JSON from `ADVISORY_REPO_PATH` (default
+`./external/advisory-database`). Clone it once:
+
+```bash
+./scripts/setup-advisory-database.sh
+# or manually:
+git clone --depth=1 https://github.com/github/advisory-database.git external/advisory-database
+```
+
+`external/` is git-ignored. If absent, the server also auto-clones on first tool call.
+
+## Run
+
+stdio (what `.vscode/mcp.json` launches — for VS Code / agent use):
+
+```bash
+ADVISORY_REPO_PATH=./external/advisory-database node dist/index.js
+```
+
+HTTP streaming:
+
+```bash
+MCP_PORT=18006 ADVISORY_API_PORT=18005 \
+ADVISORY_REPO_PATH=./external/advisory-database node dist/http-server.js
+# health: http://localhost:18006/health
+```
+
+## Test
+
+- **Unit (fast, hermetic — start here):** `npx vitest run test/unit`
+- **E2E (spawns the server; needs the advisory DB):** `npm run test:e2e`
+  - The first `list_advisories` cold-builds an in-memory index over the whole DB;
+    if it flakes on the default 30s timeout, raise it:
+    `npx vitest run test/e2e --testTimeout=120000`.
+- **Integration (Azure OpenAI):** `npm run test:integration` — requires Azure
+  credentials; skip unless you're specifically exercising the AI SDK path.
+- `npm test` runs **all** suites (including integration). Prefer `test/unit` for
+  routine work.
+
+## Project layout
+
+| Path | Role |
+|------|------|
+| `src/index.ts` | stdio MCP entry (also starts the local REST API) |
+| `src/http-server.ts` | HTTP streaming MCP entry + REST API |
+| `src/local-server.ts` | local Express REST API the tools call |
+| `src/server.ts` | shared MCP tool registration |
+| `src/datasources/local-repository.ts` | advisory indexing + filters |
+| `src/tools/` | MCP tool schemas/handlers |
+| `test/{unit,e2e,integration}/` | test suites |
+
+## Conventions
+
+- TypeScript, ES modules, strict `tsc` build. Add/extend Zod schemas in `src/tools/`
+  for any new tool input — inputs are validated at the boundary.
+- Rebuild (`npm run build`) and restart the server after edits — it runs from `dist/`.
+- Don't commit `dist/`, `external/`, or machine-specific `.vscode/mcp.json` tweaks
+  (all git-ignored).
+
+## Key environment variables
+
+| Variable | Default | Purpose |
+|----------|---------|---------|
+| `ADVISORY_REPO_PATH` | `./external/advisory-database` | Advisory DB location |
+| `MCP_PORT` | `18006` | HTTP MCP port |
+| `ADVISORY_API_PORT` | `18005` | Local REST API port |
+| `ADVISORY_API_HOST` | `127.0.0.1` | Local REST API host |
+| `ADVISORY_REFRESH_ON_START` | `true` | `git pull` the DB on startup (`false` to skip) |
+
+## Semantic search (prototype)
+
+An optional `semantic_search` tool (local hybrid embeddings + BM25) lives under
+`src/semantic/`. Build its index once, then it loads lazily on first call:
+
+```bash
+npm run build
+node dist/semantic/build-index.js --limit 5000   # omit --limit for the full reviewed corpus
+```
+
+Relevant env: `SEMANTIC_MODEL` (default `Xenova/all-MiniLM-L6-v2`),
+`SEMANTIC_MODEL_CACHE` (point at a local transformers cache to run offline),
+`SEMANTIC_ALLOW_REMOTE=true` (allow model download), `SEMANTIC_INDEX_DIR`.
+
+## Network note
+
+Installs use the public npm registry (`registry.npmjs.org`). If your environment
+proxies or mirrors npm, configure a **git-ignored** `.npmrc` pointing at your mirror;
+the committed `package-lock.json` keeps canonical public registry URLs.

CONTRIBUTING.md

@@ -33,7 +33,7 @@ This project has adopted the [Microsoft Open Source Code of Conduct](https://ope
 - **OpenTelemetry** - Comprehensive instrumentation
 
 ### Prerequisites
-- Node.js 18+
+- Node.js 20+
 - Git
 - VS Code (recommended)
 
@@ -62,8 +62,9 @@ This project has adopted the [Microsoft Open Source Code of Conduct](https://ope
 
 5. **Run tests**
    ```bash
-   npm test
+   npx vitest run test/unit   # fast, hermetic unit tests
    ```
+   `npm test` runs every suite, including the Azure-credentialed integration tests.
 
 ### Submission Guidelines
 
@@ -75,10 +76,10 @@ Before submitting an issue, please search the existing issues to avoid duplicate
 - Environment details (Node.js version, OS, etc.)
 
 #### Pull Requests
-1. Create a new branch from `dev`
+1. Create a new branch from `main`
 2. Make your changes with clear, descriptive commit messages
 3. Add tests for new functionality
-4. Ensure all tests pass (`npm test`)
+4. Ensure all tests pass (`npx vitest run test/unit`)
 5. Update documentation as needed
 6. Submit a pull request to the `dev` branch