CreditQA

Camera-ready code repository for the CreditQA benchmark and experiments.

This repo contains the inference, RAG, and evaluation code used for the paper. The larger dataset artifacts are kept outside git; place them in the expected local folders before running the scripts.

Contents

• inference/: direct agreement-context inference scripts.
• rag/: RAG index construction, retrieval, and RAG inference scripts.
• evaluation/: answer extraction, answer standardization, accuracy evaluation, and RAG comparison scripts.
• scripts/: shell wrappers for running model inference.
• outputs/: generated predictions, extracted answers, standardized answers, and RAG outputs.

Data Layout

The scripts expect the following local files and folders:

questions/
  formatted_questions_final.json
  Updated_Questions.csv
  hard_questions_updated.csv
  dataset_audit_summary.txt

agreements/
  *.md

The questions/ and agreements/ folders are ignored by git so the repository can stay lightweight. The local copies used during setup were moved to ../CreditQA_data/.

Setup

python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

Set the relevant API key before running model calls:

export TOGETHER_API_KEY=...
export OPENAI_API_KEY=...

TOGETHER_API_KEY is used by inference/run_inference.py. OPENAI_API_KEY is used by the default RAG embedding configuration.

Direct Inference

python3 inference/run_inference.py questions meta-llama/Llama-3.1-70B-Instruct-Turbo COT

Outputs are written under outputs/raw/.

RAG

Build the index:

python3 rag/build_index.py

Run RAG inference:

python3 rag/run_rag_inference.py

RAG outputs are written under outputs/.

Evaluation

The evaluation scripts preserve the original workflow. See evaluation/README.md for a brief description of each file.

For POT:

python3 evaluation/evaluate_pot.py
python3 evaluation/standardize_answers_pot.py
python3 evaluation/evaluate_accuracy.py

For COT:

python3 evaluation/extract_cot.py
python3 evaluation/standardize_answers_cot.py
python3 evaluation/evaluate_accuracy.py

Some scripts assume specific subfolders under outputs/; adjust the constants at the top of the script if evaluating a different dataset split or prompt type.

Two larger evaluation helper files are also kept outside git by default:

../CreditQA_data/evaluation/mapping.json
../CreditQA_data/evaluation/merged_questions_revision_2_standardized.json