Prism

Prism is a multimodal semantic indexing and decoding system. It maps text, images, audio, video, and documents into shared semantic geometry, organizes that geometry into graph communities, and translates it back into grounded human-readable labels or summaries.

A runnable MVP of the six-stage hybrid retrieval/generation blueprint:

1. deterministic or OpenAI embeddings;
2. SVD PCA compression and percentile reconstruction-error gating;
3. BM25 isolation for structural outliers;
4. cosine kNN graph, deterministic community detection, and centroids;
5. continuous-vector-to-LLM prefix projection;
6. routed retrieval, drafting, and grounded verification.

Gemini Embedding 2 is the default production embedding backend, using 768-dimensional MRL output and asymmetric question-answering/document prefixes. A signed feature hashing backend remains available for offline tests. Drafting defaults to an extractive baseline; a Hugging Face causal-LM adapter is included.

Quick start

python -m venv .venv
. .venv/bin/activate             # Windows: .venv\Scripts\Activate.ps1
pip install -e '.[dev,gemini]'
# GOOGLE_API_KEY must be available to the Google Gen AI SDK.
gvd build examples/corpus.jsonl .index --pca-dimensions 4 --outlier-percentile 90
gvd stats .index
gvd query .index "How are vector prefixes mapped into a language model?"
pytest

Windows PowerShell with uv

No system Python installation is required when uv is available:

$env:GOOGLE_API_KEY = "your-key"
uv run --extra dev --extra gemini gvd build examples/corpus.jsonl .index `
  --pca-dimensions 4 --outlier-percentile 90
uv run --extra gemini gvd query .index "How are vectors projected into the model?"
uv run --extra dev --extra gemini pytest -q

Do not commit the API key. For persistent local configuration, use your preferred secret manager or a user-scoped environment variable outside the repository.

Corpus embedding uses four concurrent requests by default. Set GVD_EMBED_WORKERS=1 for strict serial quotas. For large offline ingestion jobs, prefer Gemini's asynchronous Batch API for its throughput and pricing.

For an offline smoke test, add --embedding-provider hashing --embedding-dimensions 384 to gvd build.

The query prefix defaults to task: question answering | query: .... Override it with --embedding-task-name "code retrieval" (or another Gemini-supported task) when building the index; the choice is persisted with the index.

Input is JSONL with a required text, optional id, and optional metadata:

{"id":"doc-1","text":"Document body","metadata":{"source":"manual"}}

Architecture

documents -> embed -> PCA -> reconstruction residual
                              |                |
                         <= threshold      > threshold
                              |                |
                    cosine kNN graph       BM25 index
                              |
                 communities + centroids
                              |
query -> sparse-route gate OR nearest centroid -> evidence -> drafter -> verifier

GraphVectorDrafter.fit() builds both paths. At query time, a sufficiently strong BM25 match against the isolated set wins; otherwise the query is projected through the fitted PCA model, routed to its nearest community centroids, and globally reranked across their members. --community-probes controls the recall/latency tradeoff. The default probes up to 32 communities because narrow routing proved too brittle on long books; smaller values are an explicit latency optimization. Sparse outlier routing is opt-in: residual outliers must also carry metadata.is_outlier=true or matching metadata.outlier_terms. PCA novelty alone cannot hijack normal queries.

Learned drafter

Install the ML extra:

pip install -e '.[ml]'

build_torch_projection(input_dim, hidden_dim, prefix_tokens) returns the Phase-3 MLP. Its output shape is [batch, prefix_tokens, hidden_dim]. The HuggingFaceVectorDrafter concatenates that output before prompt token embeddings. training.projection_training_step implements frozen-core cross-entropy training; after projection warm-up, apply PEFT LoRA/QLoRA to selected attention layers.

For a real training run, pairs should contain the exact fitted PCA coordinate and a grounded target text. Split by source/document—not randomly by chunk—to prevent leakage. Version the embedding model, PCA matrix, graph, tokenizer, adapter, and corpus together; their coordinate systems are coupled.

Evaluation harness

Evaluate the performance of a built GVD index against a dataset of labeled queries using the CLI:

gvd evaluate <path-to-index> <path-to-labeled-queries.jsonl> --output report.json

Labeled Queries Format

The queries file must be in JSONL format, where each line specifies a query string and its list of relevant ground-truth document IDs:

{"query": "How does PCA compress embeddings?", "relevant_doc_ids": ["pca", "residual"]}

Metrics & Honest Interpretations

The evaluation report is printed as a JSON object containing the following metric groups:

1. End-to-End Query Retrieval: Calculates Recall@1, Recall@3, Recall@5, MRR, and nDCG@5.
   • Honest Interpretation: Measures how successfully the pipeline finds relevant documents. Too few community probes can cap recall; exhaustive fan-out is appropriate for small corpora, while large indexes must tune the probe count against latency.
2. Community Clustering Quality: Computes Purity, Normalized Mutual Information (NMI), and Adjusted Rand Index (ARI) against document topic metadata labels.
   • Honest Interpretation: Measures how well unsupervised community boundaries align with human-defined topics. High purity shows cohesive clusters, but low NMI/ARI is common and does not automatically translate to poor search relevance.
3. Clustering Stability: Computes the mean pairwise ARI of community partitions generated across 5 different random seeds.
   • Honest Interpretation: Evaluates if the community partitions are robust. If the stability ARI is low (e.g. < 0.6), the graph representation contains loose, overlapping, or "fuzzy" community boundaries.
4. PCA Cosine Retrieval Ablation: Compares the metrics (Recall, MRR, nDCG) and top-5 document overlap (Jaccard) of retrieval in raw embedding space versus PCA-compressed space.
   • Honest Interpretation: Quantifies the search quality lost to SVD compression. A low top-5 Jaccard overlap or a large drop in Recall/nDCG indicates that pca_dimensions is set too low and is discarding critical semantic variance.
5. Outlier Gating Performance: Calculates Precision, Recall, and F1-score of PCA outlier detection against ground-truth is_outlier metadata labels.
   • Honest Interpretation: Evaluates the reconstruction-error threshold. Because SVD residuals reflect linear reconstructions, the gate may flag highly unique or longer documents as outliers even if they are semantically normal.

Speculative decoding evaluation

Simulate speculative verification speeds and token acceptance rates on on-policy target outputs using the CLI:

gvd draft-evaluate <path-to-on-policy-corpus.jsonl> --output report.json

On-Policy Corpus Format

The corpus file must contain JSONL records, each providing a prompt and the generated target text:

{"prompt": "The quick brown", "target_text": "fox jumps over the lazy dog."}

The command deterministically splits records into train/test sets, fits a backoff token n-gram model on the training target outputs, and runs verification sweeps over a range of draft lengths $K \in {1, 2, 3, 5, 7}$.

Metrics Evaluated

• Mean accepted tokens/verify: The average number of drafted tokens accepted by the target model per step.
• Tokens/verify: The average progress made per verification step (including the target model's correction token).
• Accepted fraction: The total proportion of drafted tokens that were successfully accepted.
• Records may include source_id or session_id; related records are grouped on the same side of the deterministic train/test split to prevent leakage.
• Exact target-token preservation: Ensures the simulated speculative output matches the target text exactly.
• Best K: The draft length $K$ that maximizes throughput (Tokens/verify).

On-policy corpus generation

Generate on-policy target model completions for speculative evaluation using the CLI:

gvd generate-on-policy <path-to-index> <path-to-queries.jsonl> <path-to-output-on-policy.jsonl> \
  [--model <model-name>] [--api-base <api-base-url>] [--timeout 180] \
  [--max-evidence-chars 24000] [--max-tokens 1024] [--resume]

Configurable Parameters

• Target Endpoint: Configured via the --api-base flag or the GVD_TARGET_API_BASE environment variable (default: http://localhost:22343/v1).
• Target Model: Configured via the --model flag or the GVD_TARGET_MODEL environment variable (default: target-model).
• Authorization: Configured via the GVD_TARGET_API_KEY environment variable. API keys are handled securely and never stored or logged.
• Labeled relevant_doc_ids are used as gold evidence by default, avoiding retrieval noise and embedding API credentials during target distillation. Pass --retrieve-evidence when intentionally generating from production retrieval.
• Timeout: Per-completion timeout in seconds; local first-token latency can be high while the model warms up.
• Evidence limit: Bounds prompts so retrieval context plus generation fits the target's context window.
• Resume: Appends only unfinished source_id records, preserving completed target calls across interruptions.

The generator runs GVD queries to retrieve relevant context documents, builds ordinary RAG prompts, executes greedy deterministic target completions (temperature = 0), and writes output records matching the {source_id, prompt, target_text, target_model, evidence_ids} schema directly.

Public-domain book corpus

The reproducible ingestion script downloads William James Sidis's The Animate and the Inanimate and Project Gutenberg ebook 1184, Alexandre Dumas's The Count of Monte Cristo, then produces chapter-bounded overlapping JSONL chunks:

pip install -e '.[books]'
python scripts/ingest_books.py
gvd build data/processed/corpus.jsonl data/processed/books-index \
  --pca-dimensions 64 --outlier-percentile 99 --community-probes 32
gvd evaluate data/processed/books-index data/processed/labeled_queries.jsonl \
  --output data/processed/books-evaluation.json

Every record preserves title, author, chapter, chunk index, work ID, and source URL. The included labels test retrieval, not factual correctness of either work.

Alignment projection training

Train the vector-to-text alignment projection adapter on GVD index documents using the CLI:

gvd train-vector-drafter <path-to-index> <path-to-output-dir> \
  [--model-name <hf-model>] [--epochs 3] [--batch-size 4] \
  [--lr 1e-4] [--gradient-accumulation-steps 1] [--prefix-tokens 8] \
  [--seed 42] [--mock-lm]

Key Training Features

• Data Filtering: Exclusively filters and trains on The Count of Monte Cristo documents from the index (metadata.work_id == dumas-monte-cristo or metadata.title == "The Count of Monte Cristo").
• Leakage Prevention: Deterministically splits documents into train/validation sets grouped by chapter value, preventing adjacent overlapping chunks from leaking across splits.
• Model Freezing: Freezes the target Hugging Face Causal LM parameters completely and only optimizes the DenseAlignmentProjection module.
• Completion-Only Causal Loss: Standard causal loss calculated only over the target sequence tokens (prefix coordinates and padding are masked using -100 label values).
• Checkpoint & Resume: Supports saving state checkpoint files (projection_checkpoint.pt) and resuming training metrics smoothly if interrupted.
• Mock Mode: Pass the --mock-lm flag to run the entire training sequence offline using a small, simulated PyTorch model and tokenizer without downloading any Hugging Face models.

Evaluate the best checkpoint on deterministic held-out chapters before deployment:

python scripts/evaluate_vector_drafter.py \
  data/processed/books-index data/processed/dumas-vector-drafter-final \
  data/processed/dumas-vector-drafter-evaluation.json

The evaluator records generated samples, lexical overlap, repetition, and a fail-closed quality gate. A checkpoint that fails the gate must not replace the grounded extractive drafter.

Fidelity stress testing

After producing a safe embedding NPZ with crossmodal-evaluate, measure geometry, neighborhood, semantic-decoding, quantization, noise, PCA, and native MRL fidelity:

gvd fidelity-evaluate data/multimodal-cifar10/fixtures.json \
  data/processed/cifar10-embeddings.npz \
  data/processed/cifar10-fidelity.json

The split is stratified within every label/modality group. PCA is fitted only on the training portion. Reports include Wilson intervals, cosine distortion, top-1/5/10 neighbor retention, prediction agreement, residual outlier rates, float16/int8 quantization, and Gaussian-noise stress curves.

See the Lume integration analysis for the boundary between Prism's research harness and Lume's Rust retrieval/runtime layer.

The 10K fidelity report documents the scalable blocked evaluation, corpus provenance, measured results, and limitations.

Production notes and honest limits

• PCA outliers are linear-subspace residuals, not necessarily semantic anomalies. Tune the percentile on held-out routing data, and consider a minimum outlier count.
• The built-in graph label propagation is dependency-free, not Leiden. For large corpora, replace the exact similarity matrix with FAISS/HNSW and Leiden/Infomap.
• A centroid alone loses substantial information. The drafter receives centroid coordinates plus retrieved member evidence; do not generate from a centroid alone.
• The local verifier is a conservative lexical check, not factual proof. A target LLM can perform grounded critique, but cannot guarantee zero hallucinations.
• True speculative decoding requires compatible draft/target tokenizers and an inference engine that verifies token probabilities. Text post-validation is a different mechanism and does not deliver speculative-decoding latency claims.
• Saved indexes use JSON + NPZ and rebuild the deterministic graph on load. Never load untrusted model weights without reviewing their serialization format.

Package map

• embeddings.py: Gemini Embedding 2 retrieval prefixes, MRL dimensions, local fallback
• pca.py: compression, inverse reconstruction, residual threshold
• sparse.py: isolated BM25 index
• graph.py: kNN topology, communities, centroids, dense search
• evaluation.py: end-to-end retrieval metrics, clustering purity/NMI/ARI, seed stability, PCA ablation, and outlier prec/rec/F1
• speculative.py: backoff n-gram drafter and speculative verification simulation sweep
• generator.py: on-policy target corpus generation via GVD query retrieval and OpenAI completions
• train_cli.py: chapter-split dataset preparation and frozen-LM alignment projection training loop
• drafter.py: extractive baseline and PyTorch/Hugging Face vector-prefix adapter
• training.py: frozen-core projection training primitive
• verifier.py: grounded acceptance/fallback contract
• pipeline.py: build, route, query, save, and load
• cli.py: command-line orchestration