LettRAGraph@LLMs4OL

LettRAGraph workspace for the LLMs4OL challenge. The repository runs an extraction pipeline that turns source texts into ontology triples, then measures quality against gold triples.

Strategy Overview

1. Prepare consistent text inputs from challenge JSON.
2. Optionally run Ontology Toolkit (OTK) (private repo) to generate auxiliary ontology files.
3. Run text2graph (private repo) extraction with experiment-specific LLM settings.
4. Normalize and clean graph triples into a comparable tuple format.
5. Optionally ensemble multiple experiment outputs by intersection (precision-oriented).
6. Evaluate final output with graph-level similarity metrics.

Execution Order (Requested)

Run scripts in this order from repository root:

1. pre-process.py
2. ot_sdk_usage.py (optional OTK step; sometimes referred to as "otk_sdk_usage")
3. run.sh (or run.py through run.sh)
4. post-process.py
5. ensemble.py (optional, when combining two experiments)
6. evaluate.py

Prerequisites

• Python 3.11+ recommended
• uv installed (used by run.sh)
• Dependencies from requirements.txt
• Install Ontology Toolkit (optional) and text2graph following the respective documentation
• Access to required model/provider endpoints configured in environment and YAML files

Install dependencies:

pip install -r requirements.txt

The dataset for the challenge can be found in the LLMs4OL repository.

Configuration

Main files:

• config.yml: base runtime configuration
• experiments.yml: per-experiment overrides
• var.env: environment variables (API keys, log level, default experiment) as per example.env

Load environment variables before running:

set -a
source var.env
set +a

Step-by-Step Commands

1) Pre-process

Builds input/*.txt from challenge JSON and computes property statistics.

python pre-process.py

Useful options:

python pre-process.py -i challenge-source/TaskA-Flagship/train_task_a.json -o input

2) OTK SDK Usage (Optional)

Generates auxiliary ontology TTL files in onto/ from input/. Use this only if you want ontology-aware enrichment before extraction/post-processing.

python ot_sdk_usage.py --input-dir input --output onto

Optional concatenated mode:

python ot_sdk_usage.py --input-dir input --output onto --concat

3) Run Extraction

Recommended entrypoint:

./run.sh <experiment_name>

Examples:

./run.sh gemma-4-31B
./run.sh qwen3.6-35B

This writes experiment-scoped outputs under:

• experiments/<experiment_name>/graphs
• experiments/<experiment_name>/cache

4) Post-process

Converts generated TTL graphs into normalized tuple text files.

python post-process.py

Common options:

python post-process.py --structure-only
python post-process.py --involve-onto
python post-process.py --submission

Output location:

• experiments/<experiment_name>/output/*.txt
• optionally experiments/<experiment_name>/output/submission.json

5) Ensemble (Optional)

Intersects triples from two completed experiment outputs. Use when you want higher precision by keeping only triples confirmed by both runs.

python ensemble.py <exp_a> <exp_b>

Example:

python ensemble.py gemma-4-31B-100e gemma-4-31B-100d --destination ensemble-gemma

If you want to evaluate this ensemble next, set:

export EXPERIMENT_NAME=ensemble-gemma

6) Evaluate

Computes graph similarity metrics against the gold triples.

python evaluate.py

Optional flags:

python evaluate.py --structure-only
python evaluate.py --debug

Main artifact:

• experiments/<experiment_name>/evaluation_stats.json

Typical Workflows

Single Experiment

python pre-process.py
./run.sh gemma-4-31B
python post-process.py
python evaluate.py

Two-Model Ensemble Then Evaluate

python pre-process.py
./run.sh expA
./run.sh expB
python post-process.py
python ensemble.py expA expB --destination ensemble-expA-expB
export EXPERIMENT_NAME=ensemble-expA-expB
python evaluate.py

Repository Structure (Key Files)

• pre-process.py: creates input text files and property stats
• ot_sdk_usage.py: optional Ontology Toolkit batch processing
• run.sh: experiment launcher (loads env + runs run.py with uv)
• run.py: config-driven extraction execution per input file
• post-process.py: graph normalization and filtering
• ensemble.py: intersection-based ensemble
• evaluate.py: metric computation and summary export
• experiment_paths.py: experiment path and config resolution helpers

Notes and Troubleshooting

• Run commands from repository root so relative paths resolve correctly.
• If run.sh fails with permissions:

chmod +x run.sh

• If evaluation shows no output files, verify EXPERIMENT_NAME points to an experiment that already has output/*.txt.
• Keep secrets out of version control; do not commit real API keys.