Semantic Search Engine

A lightweight, extensible toolkit for preparing, indexing, and querying textual data for semantic search and downstream NLP tasks.

Project Overview
Architecture Diagram
Directory Layout
Prerequisites
Installation
Configuration
Data Ingestion & Preparation
Semantic Indexing (Milvus)
- Embedding & Reranking Models
- Indexing Scripts
Search API
Conversational RAG (Chat) Service
Python Client (sse_lib)
Management & Administrative Tools
- Django Management Commands
- Shell Scripts
Testing & Evaluation
Extending the Engine
Contribution Guidelines
License

Project Overview

The Semantic Search Engine provides a full‑stack solution for building semantic search applications:

Layer	Purpose	Main Modules
Data Ingestion	Convert raw text or annotation exports into a uniform JSON/JSONL format, optionally chunk, clean, and detect language.	`any_text_to_json.py`, `doccano_converter.py`
Storage	Persist raw documents and metadata in PostgreSQL (relational) and Milvus (vector) databases.	`data.models`, `engine.models`, `semantic_search_engine/aws_handler.py`
Embedding & Reranking	Compute dense vector representations and optionally re‑rank using cross‑encoders.	`embedders_rerankers.py`, `engine.controllers.embedders_rerankers`
Search	Retrieve results using Hybrid Search (Milvus vector + PostgreSQL text) with optional RRF merging, metadata filters, and templates.	`engine.controllers.search`, `engine.controllers.database.relational_db`
RAG (Retrieval‑Augmented Generation)	Combine retrieved snippets with LLMs (local or cloud-hosted) to generate context‑aware answers.	`chat.controllers`, `engine.controllers.models`
API Layer	Expose all functionality via a clean, versioned REST API built on Django Rest Framework.	`chat.api`, `data.api`, `engine.api`, `system.api`
Administration	User, organisation, and group management, plus collection lifecycle utilities.	`system.controllers`, `system.models`

The repository is deliberately modular: you can use the data preparation scripts standalone, or run the full Django‑powered service for end‑to‑end pipelines.

Architecture Diagram

+----------------------+      +---------------------+      +-------------------+
|  Raw Text / Doccano  | ---> |  Ingestion Scripts  | ---> |  PostgreSQL (RDB) |
+----------------------+      +---------------------+      +-------------------+
                                   |                               |
                                   v                               v
                           +-------------------+          +-------------------+
                           |  Chunking /       |          |  Metadata &      |
                           |  Language Detect  |          |  Relations        |
                           +-------------------+          +-------------------+
                                   |                               |
                                   v                               v
                           +-------------------+          +-------------------+
                           |  JSON/JSONL Files |          |  Document Models  |
                           +-------------------+          +-------------------+
                                   |                               |
                                   +---------------+---------------+
                                                   |
                                                   v
                                         +-------------------+          +-------------------+
                                         |   Milvus (Vector) | <------> |  PostgreSQL (FTS) |
                                         +-------------------+          +-------------------+
                                                   |                              |
                                                   +--------------+---------------+
                                                                  |
                                                                  v
                                                        +-------------------+
                                                        |   Hybrid Search   |
                                                        |   (RRF Merge)     |
                                                        +-------------------+
                                                                  |
                                                                  v
                                                        +-------------------+
                                                        |   RAG (LLM)       |
                                                        +-------------------+
                                                                  |
                                                                  v
                                                        +-------------------+
                                                        |   REST API (Django)|
                                                        +-------------------+

Directory Layout

semantic-search-engine/
├── src/
│   ├── sse_api/                  # Django project (the REST API server)
│   │   ├── config/               # settings, urls, wsgi, asgi, celery
│   │   ├── core/                 # shared utilities (errors, validation, AWS, constants)
│   │   ├── chat/                 # Conversational (RAG) service
│   │   ├── data/                 # Document, collection, query‑template models
│   │   ├── engine/               # Search & generative model orchestration
│   │   ├── system/               # Organisation & user model
│   │   ├── authorization/        # Keycloak / OAuth integration
│   │   └── content_supervisor/   # Web content sanitisation
│   └── sse_tools/                # Standalone CLI tools
│       ├── admin/                # Users, organisations, collections
│       ├── evaluation/           # Dataset generators & search evaluators
│       ├── indexing/             # Bulk indexing helpers
│       └── installed/            # File & dataset converters
├── configs/                      # Runtime configuration (*.example.* templates tracked)
├── sse_lib/                      # Standalone, pip‑installable REST API client
│   ├── pyproject.toml
│   ├── README.md
│   └── src/sse_lib/              # client, transport, options, models, endpoints
├── scripts/                      # Thin entry points (bash) into sse_tools (admin/ + not_tested/)
├── nginx/                        # Reverse‑proxy configuration
├── Dockerfile
├── docker-compose.yml
├── requirements.txt              # Server runtime dependencies
├── requirements-dev.txt          # Tests, linters, type checkers
├── pyproject.toml                # Build + tool configuration (black, mypy, bandit, pylint)
├── setup.cfg                     # flake8 options
├── run.sh                        # One‑command local startup
├── check-dev.sh                  # Local quality gate
└── README.md                     # <-- you are reading this file!

Prerequisites

Component	Minimum Version	Why it matters
Python	3.11 (rc1 acceptable)	All scripts and Django run on 3.11+.
pip	latest	To install dependencies from `requirements.txt`.
PostgreSQL	12+	Relational store for documents, users, collections.
Milvus	2.3+	Vector database for semantic embeddings.
Django	4.x	Core web framework for the API layer.
boto3	any	Required only if you enable AWS S3 storage.
radlab_data package	≥ 0.2.0	Provides `DirectoryFileReader`, `TextUtils`, and other utilities used by the conversion scripts.
CUDA (optional)	11+	If you want to run embeddings / LLMs on GPU.
Deepl API key (optional)	–	Enables answer translation.

All third‑party Python dependencies are listed in requirements.txt.

Installation

1. Clone the repository

git clone https://github.com/radlab-dev-group/semantic-search-engine.git
cd semantic-search-engine

2. Create a virtual environment (highly recommended)

python3 -m venv .venv
source .venv/bin/activate

3. Install Python dependencies

pip install -r requirements.txt

Tip: If you encounter a pip version conflict with system packages, set export PIP_BREAK_SYSTEM_PACKAGES=1 before installing (as done in full-install.sh).

4. Set up PostgreSQL

Create a database and a user (example for a local dev setup):

CREATE
DATABASE semantic_search;
CREATE
USER sse_user WITH PASSWORD 'sse_pass';
GRANT ALL PRIVILEGES ON DATABASE
semantic_search TO sse_user;

5. Run Django migrations

python -m sse_api.manage migrate

6. (Optional) Install Milvus locally

If you don’t have a remote Milvus cluster, you can run a Docker container:

# Milvus >= 3.0 requires the standalone deploy mode for an embedded etcd
docker run -d --name milvus-standalone \
  --security-opt seccomp:unconfined \
  -e DEPLOY_MODE=STANDALONE \
  -e ETCD_USE_EMBED=true \
  -e ETCD_DATA_DIR=/var/lib/milvus/etcd \
  -e COMMON_STORAGETYPE=local \
  -v milvus_data:/var/lib/milvus \
  -p 19530:19530 -p 19121:19121 \
  milvusdb/milvus:v3.0.3 \
  milvus run standalone

Make sure the Milvus connection details match the values in configs/milvus_config.json.

Configuration

Configuration files live under the configs/ directory. The most important ones are:

File	Description
`configs/milvus_config.json`	Milvus connection, index settings, and embedding model references.
`configs/aws_config.json`	S3 credentials and bucket name (used by `AwsHandler`).
`configs/embedders.json`	Registry of available embedding models (paths, vector size, device, `max_model_length`).
`configs/rerankers.json`	Registry of cross‑encoder reranker models (paths, device, `max_model_length`).
`configs/generative-models.json`	Mapping of generative model names to LLMRouter instances.
`configs/query-templates.json`	Definition of query‑template grammars and per‑template metadata filters.

Django Settings

sse_api/config/settings.py loads configuration through the SystemSettingsHandler (see sse_api/core/settings.py). The handler reads values from:

JSON config files (django-config.json by default).
Environment variables (prefixed with ENV_).

You can override any setting by exporting the corresponding environment variable. For example:

export ENV_DEBUG=true
export ENV_DB_HOST=localhost
export ENV_USE_AWS=1
export ENV_USE_KC_AUTH=0

Key Django Settings

Setting	Source	Default
`DEBUG`	`ENV_DEBUG` or `django-config.json`	`false`
`DATABASES`	`ENV_DB_*` or `django-config.json`	PostgreSQL config
`MAIN_API_URL`	Computed from `api` section in config	`/api/`
`ALLOWED_HOSTS`	`ENV_ALLOWED_HOSTS` or config	`[]`
`MAIN_LOGGER`	Configured via `logger` section	Python `logging` module

AWS S3 Integration

The AwsHandler expects a JSON file (default: configs/aws_config.json). Example:

{
  "aws": {
    "REGION_NAME": "us-east-1",
    "ENDPOINT_URL": "https://s3.amazonaws.com",
    "ACCESS_KEY_ID": "<your-access-key>",
    "SECRET_ACCESS_KEY": "<your-secret>",
    "STORAGE_BUCKET_NAME": "my-semsearch-bucket"
  }
}

All required fields are validated on AwsHandler initialization; missing values raise an AssertionError.

Model Registry (`models.json`)

semantic_search_engine/models.json holds model metadata used by the embedding and reranker loaders. A minimal entry looks like:

{
  "denoiser": {
    "model_name": "radlab/polish-denoiser-t5-base",
    "model_path": "/opt/models/denoiser/t5-base",
    "device": "cuda:0"
  }
}

You can add new models by extending the JSON and updating embedders.json / rerankers.json accordingly.

`max_model_length`

Both registries may declare max_model_length, the largest number of tokens the model takes in: the input text of an embedder, the whole query + text pair of a reranker.

A chunk longer than the input of the embedder is truncated by the model silently, so the API refuses such chunking with CHUNKING_TOO_LONG — see Chunking of a collection below.
A chunk which does not fit the pair budget of a reranker is only reported in the server log, because the query already occupies part of that budget and the truncation depends on its length.
An entry without max_model_length is treated as unlimited, so configurations written before this key existed keep working.

The values are exposed by GET embedders / GET rerankers in models_info.

Environment Variables (summary)

Variable	Meaning
`ENV_DEBUG`	Turn Django debug mode on/off.
`ENV_DB_NAME` / `ENV_DB_HOST` / `ENV_DB_PORT` / `ENV_DB_ENGINE` / `ENV_DB_USERNAME` / `ENV_DB_PASSWORD`	Override PostgreSQL connection.
`ENV_USE_AWS`	Enable the `AwsHandler` S3 wrapper.
`ENV_USE_KC_AUTH`	Use Keycloak authentication (requires `rdl_authorization`).
`ENV_USE_OAUTH_V1_AUTH` / `ENV_USE_OAUTH_V2_AUTH`	Use OAuth v1/v2 authentication.
`ENV_USE_INTROSPECT`	Enable token introspection for OAuth.
`ENV_PUBLIC_API_AVAILABLE`	Expose public API endpoints without authentication.
`ENV_USE_CELERY`	Index in a Celery queue instead of in the upload request.
`ENV_CELERY_BROKER_URL`	URL of the Celery broker (RabbitMQ, Redis, etc.).
`ENV_CELERY_RESULT_BACKEND`	Result backend of Celery; empty (default) keeps results out.
`ENV_CELERY_TASK_TIME_LIMIT`	Hard time limit of one indexing task, in seconds (14400).
`ENV_DEFAULT_LANGUAGE`	Default language for API responses (`pl` or `en`).
`DEEPL_AUTH_KEY`	API key for DeepL translation service.

Data Ingestion & Preparation

The engine expects documents to be stored in a normalized JSON/JSONL format. Two primary CLI utilities help you get there.

Any‑Text‑to‑JSON Converter

any_text_to_json.py walks a directory tree, reads plain‑text files, and emits a single JSON document (or JSONL) containing:

filepath, relative_filepath
category (derived from folder hierarchy)
pages → list of {page_number, page_content, metadata}
Optional language detection, chunking, overlapping tokens, and denoising.

Basic usage

python any_text_to_json.py \
  -d /path/to/raw_texts \
  -o /tmp/dataset.json

Advanced options

Flag	Description
`--proper-pages`	Merge all texts belonging to the same logical page before chunking.
`--merge-document-pages`	Collapse an entire document into a single page (useful for short docs).
`--clear-texts`	Run the radlab‑data denoiser on each chunk.
`--split-to-max-tokens-in-chunk N`	Limit each chunk to N tokens (uses the tokenizer of the chosen embedder).
`--overlap-tokens-between-chunks M`	Overlap M tokens between successive chunks (must accompany `--split-to-max-tokens-in-chunk`).
`--check-text-language`	Detect language with `fasttext-langdetect` and store it in metadata.
`--processes K`	Parallelise the pipeline with K worker processes.

Output format is automatically chosen: if the output file ends with .json, a single JSON object containing a "documents" list is written; otherwise a JSONL stream is produced (one document per line).

Doccano Export Converter

doccano_converter.py transforms Doccano annotation exports (JSONL) into datasets ready for sequence classification or token classification training.

Sequence Classification Example

python doccano_converter.py \
  -I /data/doccano/seq_export \
  -e .jsonl \
  --show-class-labels-histogram \
  -O ./prepared_datasets/seq/20231208 \
  --sequence-classification

Token Classification Example

python doccano_converter.py \
  -I /data/doccano/token_export \
  -e .jsonl \
  --show-class-labels-histogram \
  -O ./prepared_datasets/token/20231208 \
  --token-classification

Key flags:

Flag	Meaning
`-I / --input-dir`	Directory with Doccano export files.
`-e / --dataset-extension`	File extension of source files (`.jsonl` default).
`-O / --output-dir`	Destination directory for the prepared dataset.
`--sequence-classification`	Produce a dataset for text‑level classification.
`--token-classification`	Produce a dataset for token‑level labeling.
`--show-class-labels-histogram`	Print a histogram of class label frequencies (useful for sanity checks).
`--save-excel-annotations`	Export label statistics to an Excel file.
`--save-iob-standard`	Save the token‑level dataset in IOB format.
`--mapping-file`	Optional CSV mapping from source label names to canonical names.

Upload & Index Pipeline

The high‑level CLI add_files_from_dir.py bundles the previous steps: it uploads a directory to the relational DB, runs optional preprocessing, and then triggers semantic indexing into Milvus.

python add_files_from_dir.py \
  -d /path/to/texts \
  -c MyCollection \
  --clear-texts \
  --merge-document-pages \
  --split-to-max-tokens-in-chunk=200 \
  --overlap-tokens-between-chunks=20 \
  --processes=10

Internally the command:

Creates a temporary upload folder (or uses an existing UploadedDocuments record).
Calls DirectoryFileReader → RelationalDBController.add_uploaded_documents_to_db.
Instantiates DBSemanticSearchController for the target collection and indexes all resulting DocumentPageText objects.

Indexing Queue & Progress

Indexing can run in the request which uploaded the files, or in a Celery queue with RabbitMQ as the broker. ENV_USE_CELERY (or "use_celery": true in django-config.json) decides, and both modes execute the same pipeline and report the same progress:

Mode	What happens	When
`ENV_USE_CELERY=0`	The request stores the files, splits them and embeds them before it answers.	single-machine setups, no broker to run
`ENV_USE_CELERY=1` (default)	The request stores the files, writes an `IndexingJob` and answers at once; a worker does the rest.	anything bigger than a few documents

The unit of work is one document. prepare_indexing_job reads the files of one upload, stores Document / DocumentPage / DocumentPageText rows and asks for one index_document per document; a broken document is retried three times and then reported as failed without stopping the others. Both tasks acknowledge late, so a killed worker loses no message.

Progress lives in the relational database, not in the result backend of Celery, so it survives a restart of a worker and is readable from several of them at the same time:

Field	Meaning
`IndexingJob.status`	`queued` → `preparing` → `indexing` → `completed` / `partial` / `failed`
`IndexingJob.total_documents`, `documents_indexed`, `documents_failed`, `documents_skipped`	counters of the job, updated with `F()` expressions
`IndexingJob.current_document`	the document a worker is embedding right now
`Document.indexing_status`	`pending` / `indexing` / `indexed` / `failed` / `skipped`; `NULL` for documents indexed before this feature

GET api/indexing_status reads it, by job_id or by collection_name (the newest job of the collection), and answers with the job, its documents with their statuses, and how many documents of the collection are still waiting. The upload endpoints answer with the same job under indexing_job, plus "async": true when the work was only queued.

curl -H "Authorization: Token $TOKEN" \
  "$HOST/api/indexing_status?collection_name=MyCollection"

{"status": true, "body": {
  "job": {"id": 12, "status": "indexing", "total_documents": 30,
           "documents_indexed": 12, "documents_failed": 1,
           "documents_pending": 17, "percent": 43,
           "current_document": "regulation.pdf", "finished": false},
  "documents": [{"id": 41, "name": "regulation.pdf", "indexing_status": "indexed"}],
  "pending_documents": 17}}

Running it (docker compose keeps the queue behind a profile, so a stack without ENV_USE_CELERY=1 is untouched):

# .env: ENV_USE_CELERY=1 — broker, user and password have localhost defaults
docker compose --profile indexing up -d     # api + worker + rabbitmq

# ... or locally with run.sh, which needs no extra configuration on localhost:
./run.sh infra    # PostgreSQL + Milvus + RabbitMQ (127.0.0.1:5672, UI 15672)
./run.sh worker   # the indexing worker in the foreground
./run.sh start    # the API; with ENV_USE_CELERY=1 a worker starts with it
./run.sh status   # containers, worker, ports 5471/19530/19121/5672/8271
./run.sh down     # stops the worker and removes the containers; the data volumes are kept
./run.sh clean    # containers + volumes (postgres_data, milvus_data, rabbitmq_data) — deletes the data

# ... or the worker by hand, with the dependencies of the API installed:
celery -A sse_api.config worker -Q indexing -l INFO

The pool of the worker is solo (CELERY_WORKER_POOL, the default of run.sh and of docker compose). Reading the uploaded files runs in a multiprocessing.Pool of its own, while the tasks of a prefork worker are daemonic children, which Python does not allow to have children of their own: such a worker answers AssertionError: daemonic processes are not allowed to have children instead of indexing anything. A job which still lands on a daemonic process reads its files in one process, with a warning, rather than failing.

The embedder of a collection is loaded once per worker process, so CELERY_CONCURRENCY is a question of memory; keep it at 1 unless every process can hold its own model. A worker killed for good leaves its job running; python -m sse_api.manage reset_stale_indexing_jobs closes everything which is still running after an hour (--minutes, --dry-run).

Semantic Indexing (Milvus)

Embedding & Reranking Models

The system supports dynamic model loading via the JSON registries in configs/embedders.json and configs/rerankers.json. Each entry contains:

{
  "name": "sentence-transformers/all-MiniLM-L6-v2",
  "path": "/opt/models/all-MiniLM-L6-v2",
  "vector_size": 384,
  "device": "cuda:0"
}

Embedders produce dense vectors for each text chunk.
Rerankers (cross‑encoders) optionally re‑score the top‑K retrieved vectors using a second model, improving relevance.

Indexing Scripts

Script	Description
`index_collection_to_milvus.py`	Indexes all `DocumentPageText` objects from a relational collection into a Milvus collection.
`index_to_milvus.sh`	Convenience wrapper that sets the index type (`HNSW` or `IVF_FLAT`) and runs the Python script.
`index_to_postgresql.sh`	Uses `add_files_from_dir.py` to ingest raw files and then indexes them.

All scripts accept the following arguments (common to both):

--from-collection – name of the relational collection containing the raw chunks.
--to-collection – name of the Milvus collection that will be created.
--chunk-type – the text_chunk_type field used to differentiate between plain, cleaned, or overlapped chunks.
--index-name – one of the supported Milvus index types (HNSW, IVF_FLAT).

Note: The Milvus handler automatically creates the collection if it does not exist, using the embedding dimensions defined in the model registry.

Search API

The REST API follows a versioned pattern (/api/vX.Y/…). All endpoints are defined in the Django urls.py files of each app.

REST Endpoints

Method	URL (example)	Description
`POST`	`/api/new_collection/`	Create a new document collection (embedder, reranker, index type, chunking).
`GET`	`/api/collections/`	List all collections visible to the authenticated user.
`POST`	`/api/upload_and_index_files/`	Upload files (multipart) and trigger indexing.
`POST`	`/api/search_with_options/`	Perform a Hybrid Search (vector + text) with rich filtering options.
`POST`	`/api/generative_answer/`	Generate a RAG answer for a previously stored query response.
`GET`	`/api/question_templates/`	List available query‑template collections.
`GET`	`/api/filter_options/`	Filter metadata for UI drop‑downs (URLs, categories, `chunking_defaults`).
`GET`	`/api/embedders/`	Configured embedders: names and `models_info` (vector size, `max_model_length`).
`GET`	`/api/rerankers/`	Configured rerankers: names and `models_info` (`max_model_length` of a pair).
`GET`	`/api/categories/`	List distinct categories present in a collection.
`GET`	`/api/documents/`	List document names within a collection (used for UI selectors).
`GET`	`/api/document_content/`	Whole text of one document (preview), page by page (`truncated`, `next_page`).
`GET`	`/api/indexing_status/`	Progress of the indexing of a job (`job_id`) or of a collection.
`POST`	`/api/delete_texts/`	Delete text chunks (`chunk_ids[]`) from PostgreSQL and their vectors from Milvus.
`POST`	`/api/delete_documents/`	Delete documents (`documents[]` names and/or `document_ids[]`) with their pages and chunks.
`POST`	`/api/delete_collection/`	Delete the whole collection with all of its content from both databases.
`POST`	`/api/set_documents_use_in_search/`	Enable/disable documents in the search (`use_in_search` flag).
`GET`	`/api/search_history/`	Search history of the authenticated user (optionally one query by id).
`GET`	`/api/get_search_history/`	A single search history entry by its id.
`POST`	`/api/delete_search_history/`	Delete stored queries (`query_ids[]` and/or `query_response_ids[]`) with their responses, answers and RAG states.
`GET`	`/api/chats/`	Retrieve all chats for the current user.
`POST`	`/api/add_user_message/`	Append a user message to a chat, optionally run RAG and return assistant reply.
`POST`	`/api/save_chat/`	Mark a chat as saved and optionally set it to read‑only.
`GET`	`/api/get_chat_by_hash/`	Retrieve a saved chat by its hash (read‑only).

All endpoints require an Authorization header (Token <jwt>), unless ENV_PUBLIC_API_AVAILABLE is set.

Deleting content

The three deletion endpoints accept POST and always touch both databases: the relational rows are removed first, the embeddings of the removed texts afterwards.

Endpoint	Parameters	Removed
`/api/delete_texts/`	`collection_name`, `chunk_ids[]`	Text chunks (and the pages left without any chunk)
`/api/delete_documents/`	`collection_name` and `documents[]` and/or `document_ids[]`	Documents with their pages and text chunks
`/api/delete_collection/`	`collection_name`	Collection, its documents, pages, text chunks and the Milvus collection

Answers look like:

{
  "status": true,
  "body": {
    "deleted": {"texts": 128, "pages": 12, "documents": 3},
    "deleted_vectors": 128,
    "dropped_collection": false,
    "not_found": ["missing.pdf"]
  }
}

A collection may be deleted only by its owner — a collection which is merely shared with the groups of the user organisation is reported as COLLECTION_NOT_FOUND.
Vectors are matched through the metadata field written during indexing (external_text_id, external_document_id), so re-indexing is never required for a deletion.
Chats and search history referencing a deleted collection are removed with it, because their foreign keys are protected against cascading deletion.
Content which other rows still reference is refused with DELETE_PROTECTED, and identifiers which match nothing with DOCUMENT_NOT_FOUND / CHUNK_NOT_FOUND. Names or ids which do not exist are additionally listed in not_found.

Deleting the search history

POST delete_search_history removes the queries of the authenticated user. The entries are addressed with the identifiers which the history endpoints serve, and at least one of them is required:

Parameter	Meaning
`query_ids[]`	The `query_id` of a query
`query_response_ids[]`	The `query_response_id` of a response
`query_id`, `query_response_id`	The same, a single id

The responses, the generated answers and the RAG states of the chats which reference the queries are removed with them; nothing of the collections themselves is touched. The state of a chat message is detached rather than removed, so a message keeps its life and only loses the reference to the deleted RAG state.

{
  "status": true,
  "body": {
    "deleted": {"answers": 1, "responses": 2, "queries": 2, "rag_states": 1},
    "not_found": {"query_ids": [99], "query_response_ids": []}
  }
}

Only the history of the authenticated user can be deleted — an identifier of another user is reported in not_found and stays untouched. A request whose identifiers match nothing at all fails with QUERY_NOT_FOUND, history which other rows still reference with DELETE_PROTECTED, and a request without any identifier with INVALID_PARAMS. DELETE is accepted as an alias of POST.

Document visibility

POST set_documents_use_in_search enables or disables documents of a collection in the search, i.e. sets the use_in_search flag of the Document rows. Documents are addressed by their names (documents[]) and/or by their ids (document_ids[]); use_in_search defaults to false, so the call deactivates them. Only the owner of the collection may change its documents.

{
  "status": true,
  "body": {
    "updated_documents": 1,
    "updated_vectors": 128,
    "use_in_search": false,
    "not_found": []
  }
}

Deactivation is reversible and nothing is removed: the pages, the text chunks and the embeddings of a disabled document stay in both databases, so the document may be enabled again (with use_in_search: true) without any re-indexing. The embeddings are not deleted either — their is_active flag in Milvus follows the relational one, so the state is consistent for every consumer of the vector collection, not only for the search of this engine. While disabled, its chunks are not returned by search_with_options — neither from the vector search nor from the full-text side.

Chunking of a collection

How the texts of a collection are split into chunks is decided once, when the collection is created, so that one collection does not drift between uploads:

Where	What
`POST new_collection/`	optional `chunking` (`{"max_tokens_in_chunk": 500, "number_of_overlap_tokens": 64}`) and `strict_chunking`; the answer reports both
`POST upload_and_index_files/`, `POST add_and_index_texts/`	`max_tokens_in_chunk` / `number_of_overlap_tokens` of `indexing_options` are optional — without them the settings of the collection are used
`GET collections/`	every collection carries `metadata_json`, i.e. its `chunking` and `strict_chunking`
`GET filter_options/`	`chunking_defaults` — the values a UI form can be prefilled with
`GET embedders/`, `GET rerankers/`	`models_info` with `max_model_length`, the ceiling of `max_tokens_in_chunk`

resolve_chunking (data/controllers/chunking.py) applies, in this order:

A request which declares the values wins; the collection is not rewritten.
Otherwise the settings of the collection are used.
When neither side declares anything, the global defaults (500 tokens, no overlap) are used and remembered on the collection, so the next call does not have to pass anything.
A collection created with strict_chunking: true refuses indexing options which differ from its own — CHUNKING_MISMATCH (000006_DATA).
A chunk longer than max_model_length of the embedder of the collection is refused with CHUNKING_TOO_LONG (000005_DATA) — a longer text would be truncated by the model without any warning. number_of_overlap_tokens always has to stay inside [0, max_tokens_in_chunk).

Chunk sizes are not frozen: a collection may hold chunks of several lengths (the vectors stay comparable, only the thresholds and the context windows get fuzzier), which is why a request may override the settings at all. The hard invariant of a collection is its embedder, its vector size and the distance metric. What is already in the database can be audited with

python -m sse_api.manage check_collection_chunking --verify
python -m sse_api.manage check_collection_chunking --fix --dry-run

--verify reports the settings of every collection and measures the stored chunks with the tokenizer of the embedder, pointing at the chunks longer than declared. --fix repairs the declarations (defaults bound onto a collection which has none, a chunk clamped to the input of the embedder, an impossible overlap corrected) and --dry-run only shows that plan. No text is ever re-split: chunks which are already in the database can be corrected by indexing them again, and --collection NAME / --limit / --json select and format the report.

Search Options & Filters

The search_with_options endpoint expects a JSON payload with:

{
  "collection_name": "MyCollection",
  "query_str": "Jakie są najnowsze regulacje podatkowe?",
  "options": {
    "hybrid_search": true,
    "max_results": 20,
    "categories": [
      "Prawo",
      "Finanse"
    ],
    "documents": [
      "regulacje_2023.pdf"
    ],
    "relative_paths": [
      "/2023/"
    ],
    "templates": [
      1,
      5
    ],
    "only_template_documents": false,
    "max_results": 40,
    "min_similarity": 0.5,
    "rerank_results": true,
    "return_with_factored_fields": false,
    "relative_path_contains": [
      "https://gov.pl"
    ],
    "metadata_filters": [
      {
        "operator": "eq",
        "field": {
          "metadata_json.type": "regulation"
        }
      }
    ],
    "use_and_operator": true
  }
}

Key concepts:

Categories – map to the category field of Document.
Templates – IDs of QueryTemplate objects; the system will filter documents that satisfy the template’s data_filter_expressions.
Metadata filters – low‑level JSON‑logic style filters (in, eq, gt, hse, …) applied directly on the metadata_json column.
use_and_operator – determines whether multiple filter lists are intersected (true) or unioned (false).
min_similarity – cosine similarity cutoff for the vector search (default 0.5); hits scoring below this value are dropped, so a query can legitimately return zero results.

The response contains:

stats – aggregated per‑document statistics. Every chunk score is first normalized onto [0, 1] (cosine similarity, RRF value or reranker logits depending on the pipeline) before the document weights are computed:

key	meaning
`score`	mean of the raw chunk scores (original metric, e.g. cosine)
`score_normalized`	mean chunk relevance normalized to `[0, 1]`
`score_weighted`	document mass – `score_normalized * hits * pages_count`
`score_weighted_scaled`	share of the total mass; all shares sum up to `1.0`
`weight_normalized`	mass relative to the best document (best = `1.0`)

The cumulative score_weighted_scaled of the top documents drives percentage_rank_mass in the generative options, so it stays a proper percentage of the total relevance mass. stats also carries similarity, the mean cosine of the document's hits (null for a hybrid search, see below).

detailed_results – list of matching chunks with left/right context. Each chunk has:

score – the score that ordered the results. Its meaning depends on the pipeline and is reported in results.score_scale:

`score_scale`	`score` is
`cosine`	cosine similarity of the vector search (`hybrid_search` off)
`rrf`	Reciprocal Rank Fusion value from the hybrid search (small, around `1 / (60 + rank)` per list)
`reranker`	cross-encoder logit

So a well-matched chunk of a hybrid search may show score: 0.03 — that is a top fused rank, not a low similarity.

score_normalized – the same relevance on the [0, 1] scale, comparable across pipelines;
similarity – cosine similarity of the chunk; present (not null) only when the vector search is the single ranking criterion (hybrid_search off). In hybrid mode the ranking comes from the fused ranks and full-text-only chunks have no similarity at all.

score_scale – which of the above scales the score values use.
detailed_results – list of matching chunks with left/right context.
structured_results – optional structured output generated by a template that requests it.
template_prompts – any system prompts extracted from the used templates (useful for LLM instruction).

Query Templates & Grammar

Query templates are defined in configs/query-templates.json. They consist of:

Grammar – token set and alphabet used to generate natural‑language queries.
Data connector – static key‑value pairs that must match Document.metadata_json.
Data filter expressions – dynamic Python expressions evaluated against document metadata (e.g., date range checks).
Structured response flag – when true, the engine extracts specified fields from the document metadata and returns them alongside the text snippets.

The QueryTemplateController automatically validates a document against its grammar (use_document_in_sse) and filter expressions (use_document_in_sse), ensuring only relevant documents participate in the search.

Conversational RAG (Chat) Service

The Chat app adds a conversational layer on top of the semantic search engine. It stores chat history, supports content supervision, and integrates with generative models.

Chat Lifecycle

Create a new chat – POST /api/new_chat/ (optional collection and search options).
Send a user message – POST /api/add_user_message/ with chat_id, user_message, and generation options.
System processes the message:
- Detects whether the message is a RAG query (user_msg_is_rag_question flag).
- Optionally extracts an explicit question (pytanie:) and/or instruction.
- Runs the semantic search using the collection linked to the chat (or the global default).
- If use_rag_supervisor is enabled, the system creates a RAGMessageState linking the query, response, and generated answer.
- If use_content_supervisor is enabled, URLs in the user text are fetched and their content added to the prompt.
Generate assistant response – either via a local or cloud, with optional translation via DeepL.
Persist messages – Message objects store role (user, assistant, system), text, timestamps, and optional MessageState.

All chats are tied to an OrganisationUser for permission checks. The read_only flag prevents further message addition.

Content Supervisor

Implemented in ChatController._prepare_content_supervisor_state. It uses radlab_content_supervisor to:

Detect URLs (URLRegexProcessor).
Fetch remote content (HTML) and store it in ContentSupervisorState.www_content.
Append fetched snippets to the user message before sending it to the LLM.

This feature is useful for dynamic knowledge retrieval (e.g., fetching the latest news article).

Generative Model Integration

The GenerativeModelController orchestrates calls to: model APIs – defined in configs/generative-models.json with custom endpoints (/generate).

Both paths support:

Generation options (top_k, temperature, etc.).
System prompt – static instruction that can be injected from a query template.
Answer translation – using Deepl if translate_answer is true and a key is present.

Generated answers are stored in UserQueryResponseAnswer and can be rated via the SetRateForQueryResponseAnswer endpoint.

Python Client (`sse_lib`)

sse_lib/ is a standalone, pip-installable wrapper around the REST API of this repository. It is a deliberately thin layer: the Django controllers, views and serializers stay as they are, and the library only turns Python calls into those HTTP endpoints (and their JSON answers into small dataclasses). Its single runtime dependency is requests.

pip install ./sse_lib

from sse_lib import SSEClient, SearchOptions, GenerativeOptions

with SSEClient("http://localhost:8271", token="...") as client:
    client.create_collection(
        "my_docs",
        "My Documents",
        "Test collection",
        embedder="radlab/polish-bi-encoder-mean",
        reranker="radlab/polish-cross-encoder",
        index_type="HNSW",
    )
    client.upload_files(
        "my_docs",
        ["invoice.pdf"],
        max_tokens_in_chunk=128,
        number_of_overlap_tokens=0,
    )
    client.index_documents(
        "my_docs",
        ["Some text to index."],
        category="Notes",
        max_tokens_in_chunk=128,
        number_of_overlap_tokens=0,
    )

    found = client.search("my_docs", "data retention policy", SearchOptions(max_results=20))
    answer = client.generate_answer(
        found.query_response_id,
        GenerativeOptions(
            generative_model="radlab/pLLama-3-8B-DPO-L",
            percentage_rank_mass=40,
        ),
    )
    print(answer.answer)

Chunking belongs to the collection: declare it once with client.create_collection(..., chunking={"max_tokens_in_chunk": 128, "number_of_overlap_tokens": 0}) and leave both keys out of the indexing calls; client.embedders_info() reports the max_model_length of each embedder, which is the ceiling of a chunk. Passing the values per call (or once per client with default_indexing_options) still works, and a split which cannot exist (overlap >= chunk) raises SSEValueError before the request goes out. percentage_rank_mass of generate_answer stays required.

Coverage: collections, file and raw-text indexing, hybrid search, RAG answers and their ratings, embedder/reranker/ generative model listings, chats, and authentication (static token, username/password login, Keycloak code exchange).
API quirks encoded once: options/query_options/indexing_options sent as JSON strings where the views call json.loads(), files[]/texts[] form fields, endpoint paths without a trailing slash, and the {"status": false, "errors": [...]} envelope translated into typed exceptions (SSEAPIError, SSEAuthenticationError, ...).
Configuration through the constructor or SSE_API_HOST, SSE_API_TOKEN, SSE_API_PREFIX, SSE_API_LANGUAGE, SSE_API_TIMEOUT; versioned deployments need api_prefix="api/v1".
Escape hatch: client.request("GET", "some_endpoint") reaches any endpoint, and every returned object keeps the raw payload in .raw.

Full documentation lives in sse_lib/README.md.

Management & Administrative Tools

Django Management Commands

All standard Django commands are available (runserver, migrate, createsuperuser).
Additional project‑specific commands are provided by the sse_tools scripts:

Command	Description
`python -m sse_api.manage runserver 0.0.0.0:8271`	Starts the development server (default port 8271 in `run-api.sh`).
`python -m sse_api.manage migrate`	Applies database migrations (including the generated migration files under `chat/migrations`).
`python -m sse_api.manage createsuperuser`	Creates a Django superuser for the admin UI (if `ENV_SHOW_ADMIN_WINDOW` is true).
`python -m sse_api.manage reset_stale_indexing_jobs`	Closes indexing jobs and documents which were left running by a killed worker (`--minutes 60`, `--dry-run`).
`python -m sse_api.manage check_collection_chunking`	Reports the chunking of every collection; `--verify` measures the stored chunks with the tokenizer of the embedder, `--fix` repairs the declarations of the collections and `--dry-run` shows that plan (`--collection NAME`, `--limit`, `--json`).

Shell Scripts

full-install.sh – End‑to‑end installer that sets up the virtual environment, installs required packages, creates the database, runs migrations, prepares the Milvus collection, adds a default organisation/user, and loads query templates.
run-api.sh – Convenience wrapper that sets common environment variables (CUDA device, data upload limits, API keys) and starts the Django dev server.
index_to_milvus.sh & index_to_postgresql.sh – Quick wrappers around the Python indexing scripts for ad‑hoc runs.

All scripts are idempotent where possible; they log progress to stdout and rely on the underlying Python modules for error handling.

Testing & Evaluation

The evaluator package (src/sse_tools/evaluation/evaluator) provides utilities for:

Loading test configurations (JSON files defining test cases).
Running semantic search with various parameters (max results, rerank, etc.).
Generating answers using configured generative models.
Computing metrics (BLEU, ROUGE) via torchmetrics.

The main entry point is tests_loader.py. It writes results to Excel (.xlsx) files for downstream analysis. Example usage:

python src/sse_tools/evaluation/evaluator/src/tests_loader.py \
  -u alice \
  -c ./tests/config.json \
  -o results.xlsx \
  -p results_chunks.xlsx

The output contains:

Aggregated statistics per collection, test case, and model.
Human‑annotated answer comparisons (precision/recall style).
Per‑chunk detailed results for error analysis.

Extending the Engine

The codebase is deliberately plug‑and‑play:

Add a new embedding model – Append an entry to configs/embedders.json and place the model files where model_path points. No code changes required.
Add a new reranker – Same procedure with configs/rerankers.json.
Create a custom query template – Edit configs/query-templates.json (or add a new file and load it via QueryTemplatesLoaderController). Define data_filter_expressions using Python syntax; the engine evaluates them safely with eval.
Expose a new API endpoint – Create a view inheriting from APIView, add the route to the appropriate urls.py, and use existing decorators (required_params_exists, get_organisation_user).
Swap the LLM backend – Implement a new subclass of GenerativeModelControllerApi.LocalModelAPI with custom request payload/response handling, then reference the new model name in configs/generative-models.json.

All extensions respect the existing permission model (token‑based authentication) and will automatically appear in the OpenAPI/Swagger UI if django-rest-swagger is enabled.

Contribution Guidelines

Fork the repository and create a feature branch (git checkout -b feature/your‑idea).
Write tests for any new functionality (use the existing src/sse_tools/evaluation/evaluator test harness as a reference).
Run the full test suite: pytest (install via pip install pytest).
Format code with black and lint with flake8.
Update documentation – add a new section to this README, or improve docstrings.
Submit a Pull Request targeting the main branch. Ensure CI passes (if configured).

License

This project is licensed under the Apache 2.0 License – see the LICENSE file for full details.

Happy coding! 🚀

Name		Name	Last commit message	Last commit date
.github/todo/workflows		.github/todo/workflows
configs		configs
nginx		nginx
scripts		scripts
src		src
sse_lib		sse_lib
.bandit		.bandit
.dockerignore		.dockerignore
.env.example		.env.example
.gitignore		.gitignore
.pylintrc		.pylintrc
.version		.version
CHANGELOG.md		CHANGELOG.md
DEPENDENCIES.md		DEPENDENCIES.md
Dockerfile		Dockerfile
EVALUATE.md		EVALUATE.md
LICENSE		LICENSE
README-pl.md		README-pl.md
README.md		README.md
SETUP.md		SETUP.md
check-dev.sh		check-dev.sh
conftest.py		conftest.py
docker-compose.yml		docker-compose.yml
pyproject.toml		pyproject.toml
requirements-dev.txt		requirements-dev.txt
requirements.txt		requirements.txt
run.sh		run.sh
setup.cfg		setup.cfg

Latest commit

History

Folders and files

Repository files navigation