SitRep BVD DRC — Extraction Pipeline

AI-assisted development: this pipeline and report were developed with GitHub Copilot and the Anthropic Claude API (claude-sonnet-4-6), with human review.

⚠️ Automated extraction — verification required. Case counts are extracted by an AI vision model without systematic manual verification. Values should be checked against the original INSP DRC PDFs before any operational or clinical use. For research and situational awareness only.

Extracts key epidemiological tables from INSP DRC Ebola SitRep PDFs with Claude, then builds standardised CSV outputs and an HTML report, including cross-source validation against INRB-UMIE manual extraction (local fallback: outputs/sitrep_report.html).

Figure: Cross-source validation of cumulative confirmed cases, suspect/probable cases, and outbreak deaths by health zone, comparing automated extraction with INRB-UMIE manual extraction.

Author: Billy J Quilty (Charité Berlin, LSHTM & MSF Epicentre)

Quick Start

Prerequisites:

• Python 3.10+
• R (for report generation)
• Quarto CLI (for rendering HTML report)
• Anthropic API key

Setup:

python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
cp .env.example .env

Add your key to .env (ANTHROPIC_API_KEY or CLAUDE_API_KEY).

Daily Workflow

source .env
python3 scripts/fetch_sitreps.py
python3 scripts/extract_sitrep.py --update

What this does:

• Downloads new SitRep PDFs to data/raw/<sitrep_name>/
• Skips already processed files using data/processed/processed.json
• Appends new rows to data/processed/master_combined_counts.csv

Other Common Commands

Single PDF:

source .env
python3 scripts/extract_sitrep.py path/to/SitRep.pdf

Batch PDFs:

python3 scripts/extract_sitrep.py SitRep_001.pdf SitRep_002.pdf SitRep_006.pdf

Render report:

quarto render sitrep_report.qmd

Run tests:

pytest tests/ -v

Outputs

Per SitRep (data/processed/epicentre_format/<sitrep_name>/):

• new_cases_counts.csv
• cumulative_counts.csv
• combined_counts.csv

Raw extraction JSONs and PDFs (data/raw/<sitrep_name>/, gitignored):

• raw_extraction.json
• <sitrep_name>.pdf
• manifest.json (at data/raw/ root)

Master aggregate files (data/processed/):

• master_combined_counts.csv
• master_response_counts.csv
• master_poe_counts.csv
• processed.json

INRB-UMIE format files (data/processed/insp_format/):

• insp_sitrep__*.csv (20 per-metric files)

Report (outputs/):

• sitrep_report.html

Core Configuration

• ANTHROPIC_API_KEY (preferred)
• CLAUDE_API_KEY (fallback)
• ANTHROPIC_MODEL (optional, default: claude-sonnet-4-6)

Notes

• count_type values in combined tables are Nouveaux and Cumules.
• ND values from source PDFs are stored as blanks.
• Subtotals/totals are preserved when present in source tables.
• SitReps 015 and 016 (29–30 May 2026) were not published on the INSP website at the time of extraction. Their PDFs were sourced from the INRB-UMIE/Ebola_DRC_2026 repository and are recorded in data/raw/manifest.json with a source_note field. These entries will be replaced with direct INSP downloads once the PDFs are published.

Experimental: GitHub Copilot Agent Mode

This feature is experimental. It requires VS Code with GitHub Copilot and the agent chat mode.

The pipeline can be orchestrated end-to-end via a custom GitHub Copilot agent defined in .github/agents/sitrep-orchestrator.agent.md. The agent calls the same underlying scripts but handles fetch, extract, INRB-UMIE submodule update, render, and deploy autonomously — with reactive error diagnosis at each stage.

Setup: no additional installation is required beyond the standard prerequisites above. The agent uses the skills in .github/skills/ for domain-specific guidance.

Usage: open GitHub Copilot Chat in VS Code, switch to Agent mode, select sitrep-orchestrator, and send a prompt such as:

check for updates
run full pipeline
render report
compare against INRB data

The agent runs fetch_sitreps.py, extract_sitrep.py --update, updates the INRB-UMIE submodule, renders the Quarto report, and pushes to origin/main — prompting you before any destructive step.

License

MIT. See LICENSE.