AISysRev - LLM-based Tool for Title-abstract Screening

Important

AISysRev is a Minimum Viable Product (MVP) with core functionality working. Some features are missing and there might be bugs. You can also checkout command line alternative AISysRevCmdLine

Table of contents

• AISysRev - LLM-based Tool for Title-abstract Screening
  • Introduction
  • Features
    • Title-abstract screening
    • Full-text screening
      • Uploading PDFs
      • Screening modes
  • Showcase
  • Getting started
    • Data
    • LLMs Access
    • LLM screening speed
  • System and software requirements
    • Verifying Docker setup and environment
    • Running the AISysRev application
      • MacOS, Linux and Windows (WSL)
      • Windows (non-WSL)
  • Technology
    • Front-end
    • Back-end
    • System design
  • Development requirements
  • Running in development mode
    • MacOS, Linux and Windows (WSL)
    • Windows (non-WSL)
    • Getting started with development
  • Other documentation
  • Mock data
  • Tests
    • Client
    • Server
  • Makefile Commands
    • Development
    • Database Migrations (Alembic)
  • Supported LLMs
  • BibTeX Citation
  • AI disclaimer
  • License
  • References

Introduction

The AISysRev web-application offers research-backed and AI-based support for Systematic Literature Reviews. Currently, the main supported step is title–abstract screening (see the feature table). Although the application runs in a web browser, all data is stored locally on your machine. LLMs are accessed through OpenRouter, OpenAI or through a local provider (OpenAI SDK). Data for screening can be imported from two sourced as a CSV from: 1) Scopus and from 2) Github. For Github data see our tool Github Query Tool that queries and collects github repository data.

Scopus data screening works like normal title-abstract screening for academic papers. For GitHub we apply the same logic. Title from GitHub project becomes the title and abstract is the project description + README.md contents. GitHub screening can be useful if one is interested in finding software tools that satisfy inclusion and exclusion rules. Our paper on tool discovery with LLMs.

Features

AISysRev focuses on study selection. The table below shows which steps of the systematic review process the application supports.

Systematic review step	Status	Notes
SR protocol definition	❌ Not supported	Inclusion/exclusion criteria are defined per screening task, but there is no protocol editor.
Automated search	❌ Not supported	Import search results as CSV from Scopus or from our Github Query Tool.
Study selection (title-abstract)	✅ Supported	LLM-based and manual screening. See Title-abstract screening.
Study selection (full-text)	✅ Supported	LLM screening using the passages of an uploaded PDF that best match each criterion. See Full-text screening and pdf-screening.md.
Quality assessment	❌ Not supported
Data extraction	❌ Not supported
Meta-analysis	❌ Not supported
Final report	❌ Not supported	Screening results can be exported to CSV.

In the future, we plan to expand the feature set to cover more steps of the systematic review process.

The application is based on our research papers on this topic. Please consider citing if you use the application 1–3.

Title-abstract screening

The application allows you to:

• Import a CSV file with paper titles and abstracts. You can also use our Demo CSV file
• Specify include/exclude criteria for paper screening
• Evaluate papers against the criteria using multiple LLMs
• Receive LLM evaluations as binary decisions (include/exclude), ordinal ratings (1-7), or inclusion probabilities (0–1)
• Perform manual evaluation of titles and abstracts alongside LLM evaluations
• Export evaluation results to CSV for further analysis in Microsoft Excel, Google Sheets, R, Python, etc. All of the above steps are also done for Github repositories.

Full-text screening

AISysRev supports full-text screening of PDFs in addition to title-abstract screening. For each criterion, the most relevant passage of the PDF is retrieved and given to the LLM. Full-text screening requires the OpenAI or OpenRouter provider, because the passages are found with embeddings.

Uploading PDFs

PDFs can be attached to papers in two ways:

• Manual - on a paper's card, click Upload full text and select a PDF file to upload.
• Bulk upload with Zotero/EndNote XML
  1. Click Download papers missing full text to export a RIS file containing all papers in the project that don't have a PDF attached.
  2. Import the RIS file into Zotero and use Find Full Text to retrieve full-text PDFs.
  3. Export the Zotero collection with the automatically retrieved PDFs in EndNote XML format with Export notes and Export files checked.
  4. In AISysRev, click Import full text (Zotero Export Folder) and select the exported folder. PDFs are attached to papers based on DOI.

Screening modes

When creating a screening task, choose which screening mode to use:

• Abstract - the paper's title and abstract are given to the LLM.
• PDF - excerpts from the paper's full-text PDF are given to the LLM.
• Automatic - uses PDF screening mode for papers with a PDF attached and Abstract for papers without a PDF attached.

See pdf-screening.md for details on how PDF screening mode works, including its limitations.

Showcase

Figure 1: Main view shows LLM screening tasks.

Figure 2: Manual evaluation view, with LLM evaluations (binary, ordinal, probability) alongside manual review.

Figure 3: Manual evaluation list view, with papers sorted by inclusion probability according to all executed LLMs.

Getting started

Data

The tool has been developed and tested with CSV data exported from Scopus and Github with data exported by our custom Github Query Tool. Support for Web of Science can be achieved by editing the columns headers to match the ones from Scopus. The minimum required fields are: Document title, DOI, Abstract, Authors, and Source title.

LLMs Access

The application is integrated with OpenRouter, which supports multiple LLMs ranging from very affordable to top-tier models like OpenAI’s ChatGPT, Google’s Gemini, Anthropic’s Claude, Meta's LLama, and Mistral. To use the models, you need to provide an OpenRouter key. You can set spending limits for each key directly on the OpenRouter website.

LLM screening speed

LLM calls are parallelized, and you should achieve a screening speed exceeding 100 papers per minute when using OpenRouter. The screening speed depends on the model used.

System and software requirements

• Docker, with Compose and buildx plugins installed.
• uv Python package and project manager: https://docs.astral.sh/uv/getting-started/installation/
• Enough RAM (At least 8GB recommended)
• Enough disk space (Around 8GB)
• Network connection

Tip

See https://docs.docker.com/desktop/ for Docker installation instructions. Docker Desktop includes Docker Compose, Docker Buildx, Docker Engine and the Docker CLI.

Note

If Docker Desktop did not include the Buildx plugin, see: https://github.com/docker/buildx

Verifying Docker setup and environment

1. Run docker info to verify you have Docker installed
   • Docker 26.0.0 has been tested as working. For MacOS computers with Colima, Docker version 28.5.1 confirmed to be working.
2. Run docker buildx version to verify you have Docker Buildx installed. For MacOS computers, Buildx plugin version 0.29.1 confirmed to be working.
3. Run docker compose version to verify you have Compose installed. For MacOS computers, Compose plugin version 2.40.3 confirmed to be working.
   • Version 2.33.1 has been tested as working, newer versions should also work.
   • Note: Older versions of Compose use docker-compose as the compose command. We don't provide support for legacy Compose versions.

Running the AISysRev application

First, clone the repository to your local computer.

git clone https://github.com/EvoTestOps/AISysRev.git

move to correct directory

cd AISysRev

MacOS, Linux and Windows (WSL)

Start the application in production mode:

make start-prod

If you want to develop the app, run:

make start-dev

Note

The startup of the app may take a while due to the download of corresponding Docker images & services, application dependencies and building of the application.

After startup, open the application:

If you ran start-prod, navigate to https://localhost:3000 (the Caddy server's root CA is by default untrusted. You can bypass the browser warning).

If you used make start-dev, navigate to https://localhost:3001 (the dev server uses a self-signed certificate, so you can bypass the browser warning).

Windows (non-WSL)

If you do not have Windows Subsystem for Linux (WSL), start the application with

./start-prod.bat

Technology

The AISysRev tool is built on bleeding front-end and back-end technology, with industry-demonstrated stability. We follow the best known coding conventions and toolkit.

Front-end

TypeScript, React, Tailwind CSS, Vite, Wouter, Zod, Redux

Back-end

Python, FastAPI, PostgreSQL, SQLAlchemy, Alembic

System design

See architecture.md

Development requirements

• Node.js v24 LTS
• Python 3.14
• Docker, with Compose plugin installed
• UV v0.12.17 or later: https://docs.astral.sh/uv/getting-started/installation/
  • Updating uv: uv self update

Running in development mode

MacOS, Linux and Windows (WSL)

make start-dev

Windows (non-WSL)

./start-dev.bat

Getting started with development

Open up the client: https://localhost:3001

Note

/api is internally proxied to the backend container, e.g. http://localhost:3001/api/v1/health will be proxied to http://localhost:8080/api/v1/health.

API: https://localhost:3001/api/v1

API docs: https://localhost:3001/docs

Adminer GUI: http://localhost:8081/?pgsql=postgres&username=your_username&db=your_database_dev&ns= password: your_password

Generating API types

Generate TypeScript types and Zod schemas from the backend's OpenAPI spec by running, in the repository root:

make openapi

This doesn't need the server or containers running. It imports the FastAPI app locally (with uv), writes app.openapi() to client/openapi.json, and then runs npm run generate:api in client/ to generate client/src/services/api/ (client, types, fetcher) from that spec. The app's startup hooks aren't run, so nothing connects to the database or Redis; the connection URLs are set to placeholder values only because the settings require them. The spec is exported with APP_ENV=dev, so it matches the routes of the dev server.

Alternatively, with make start-dev running, you can run the steps separately in client/:

• npm run get:openapi-spec — fetches the spec from https://localhost:3001/openapi.json into client/openapi.json.
• npm run generate:api — generates client/src/services/api/ from that spec.

client/src/services/api/ and the openapi.json are committed to git. Re-run make openapi whenever backend endpoints or schemas change.

A smoke test is in place, which instructs you to update openapi.json then the spec changes.

Other documentation

• docs/README.md: index of all documentation
• CONTRIBUTING.md: development workflow and documentation rules
• docs/architecture.md: system design and ports
• docs/pdf-screening.md: how PDF screening works
• docs/local-models.md: using local LLMs
• fuzzing/README.md: API fuzzing tools
• docs/maintenance.md: upgrading dependencies and Docker images

Mock data

Mock data is located in data/mock -folder.

Tests

Client

Run in client/:

• npm test for unit and component tests
• npm run test:e2e for e2e tests

Server

Run in repository root:

• make backend-unit for backend unit tests only
• make backend-test for all backend tests
• make backend-test-html for all backend tests and an HTML coverage report

Windows (non-WSL) users need to run the equivalent docker compose commands from the Makefile manually, since only start-dev.bat and start-prod.bat are provided.

Makefile Commands

The project includes a Makefile for common development and database operations:

Development

Command	Description
make start-dev	Start dev containers with live reloading and build on startup (default setup)
make start-dev-debug	Same as start-dev, with the backend debugger enabled
make start-test	Start test containers and rebuild images (isolated test environment)
make start-test-debug	Same as start-test, with the backend debugger enabled
make start-prod	Start production container and rebuild images
make backend-unit	Run backend unit tests
make backend-test	Run all backend tests
make backend-test-html	Run all backend tests and create an HTML coverage report
make extract-caddy-ca	Copy the production Caddy root CA to ./caddy-local-root.crt so you can trust it locally

Note: Run all commands from the project root.
Containers are isolated by environment using the Docker Compose -p flag.

Database Migrations (Alembic)

Command	Description
make m-create m="Message"	Create a new migration with an autogenerated diff (replace Message)
make m-up	Apply all pending migrations (upgrade to latest)
make m-hist	Show the full migration history with details
make m-current	Display the current migration version in the database

Supported LLMs

Currently, we support models provided via Openrouter, OpenAI or via a local provider (OpenAI SDK). The list of supported LLMs is continuously evolving and changing, which is why we don't provide a comprehensive list.

BibTeX Citation

Please use the following BibTeX citation to cite our work:

@inproceedings{huotalaAISysRevLLMbasedTool2026, 
  address={Concordia University Montreal QC Canada}, 
  title={AISysRev - LLM-based Tool for Title-abstract Screening}, 
  ISBN={9798400726361}, 
  url={https://dl.acm.org/doi/10.1145/3803437.3806408}, 
  DOI={10.1145/3803437.3806408}, 
  booktitle={Proceedings of the 34th ACM International Conference on the Foundations of Software Engineering}, 
  publisher={ACM}, 
  author={Huotala, Aleksi and Kuutila, Miikka and Turtio, Olli-Pekka and Sipilä, Simo and Mäntylä, Mika}, 
  year={2026},
  month=july, 
  pages={142–146}, 
  language={en} }

AI disclaimer

Parts of AISysRev may have been developed with AI-assisted coding tools, including "vibe coding". Where AI assistance was used, the correctness and code style of the resulting code were reviewed.

License

MIT

References

[1] Huotala, A., Kuutila, M., Ralph, P., & Mäntylä, M. (2024). The promise and challenges of using llms to accelerate the screening process of systematic reviews. Proceedings of the 28th International Conference on Evaluation and Assessment in Software Engineering, 262–271. https://doi.org/10.1145/3661167.3661172

[2] Huotala A, Kuutila M, Mäntylä M. SESR-Eval: Dataset for Evaluating LLMs in the Title-Abstract Screening of Systematic Reviews. In Proceedings of the The 19th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM) 2025 Oct (pp. 1-12) https://arxiv.org/abs/2507.19027

[3] Huotala A, Kuutila M, Turtio Olli-Pekka, Sipilä S, Mäntylä M. AISysRev: LLM-based Tool for Title-abstract Screening. In FSE Companion '26: Proceedings of the 34th ACM International Conference on the Foundations of Software Engineering 2026 Jul (pp. 142-146) https://doi.org/10.1145/3803437.3806408