Taxoble

Taxoble is an automated framework for constructing taxonomies for a domain from the given term set.

The current examples focus on biomedical domains such as nephrology and rheumatology, but the pipeline can be adapted to any domain with a list of candidate terms or article keywords.

Repository Layout

.
├── construct_taxonomy.ipynb          # Main notebook for taxonomy construction
├── download_data_from_pubmed.ipynb   # Optional PubMed data download notebook
├── preprocess_input.py               # Candidate term preprocessing script
├── requirements.txt                  # Python dependencies
├── LICENSE
├── .gitignore
├── data/
│   └── processed/
│       ├── nephrology/
│       │   └── jasn_pubmed_1990_2025_w_keywords.csv
│       ├── rheumatology/
│       │   └── ard_pubmed_1939_2025_w_keywords.csv
│       └── snomed_autoimmune_disease/
│           └── README.md             # SNOMED CT licensing; data not redistributed
├── results/
│   ├── nephrology/
│   │   └── experiment1/
│   │       └── final_taxonomy.txt
│   ├── rheumatology/
│   │   └── experiment1/
│   │       └── final_taxonomy.txt
│   └── snomed_autoimmune_disease/
│       └── experiment1/
│           └── final_taxonomy.txt
└── taxonomy/
    ├── __init__.py                   # Public package exports
    ├── config.py                     # Pipeline configuration dataclasses
    ├── llm_client.py                 # Ollama/OpenAI chat model adapter
    ├── embeddings.py                 # Sentence-transformer embedding helpers
    ├── nmf_pipeline.py               # NMF topic grouping utilities
    ├── llm_taxonomy.py               # LLM prompts for taxonomy construction
    ├── construction_pipeline.py      # Fine and broad taxonomy passes
    ├── graph_utils.py                # Hierarchy and DAG cleanup helpers
    ├── cycle_utils.py                # Cycle detection and repair
    ├── merge_utils.py                # Similar-node merging
    └── io_utils.py                   # Pickle and logging helpers

Example input CSVs live under data/processed/<folder_name>/. Running the pipeline adds keyword_mapping.pkl, cache/, logs/, and experiment artifacts under the same folder. Final hierarchies are written to results/<folder_name>/<experiment_name>/.

Installation

Clone the repository and install the dependencies in a fresh environment.

git clone <your-repo-url>
cd Taxoble

python -m venv .venv
source .venv/bin/activate
pip install --upgrade pip
pip install -r requirements.txt

The notebooks were developed with Python 3.10. Other recent Python 3 versions may work, but Python 3.10 is recommended for reproducibility.

LLM Setup

Taxoble supports either a local Ollama-compatible model or the OpenAI API.

Option 1: Ollama

Install and run Ollama, pull the model you want to use, and make sure the base_url in the configuration matches your Ollama server.

LLMConfig(
    provider="ollama",
    base_url="http://localhost:11435",
    model="gpt-oss:20b",
    temperature=0,
    timeout=20,
)

Option 2: OpenAI

Set your API key before running the preprocessing script or construction notebook.

export OPENAI_API_KEY="your-api-key"

Then configure:

LLMConfig(
    provider="openai",
    model="<openai-model-name>",
    api_key=os.getenv("OPENAI_API_KEY"),
    temperature=0,
    timeout=60,
    max_output_tokens=4096,
)

Data Layout

The default workflow expects processed data under:

data/
└── processed/
    └── <folder_name>/
        ├── <input-file>.csv
        ├── keyword_mapping.pkl
        ├── logs/
        └── <experiment_name>/

The preprocessing script expects a CSV column containing semicolon-separated keywords, for example:

title,abstract,keywords
"Article title","Article abstract","chronic kidney disease; dialysis; proteinuria"

The taxonomy construction notebook reads keyword_mapping.pkl from data/processed/<folder_name>/ and uses the "final" field from each entry as the candidate term list.

Workflow

1. Optional: Download PubMed Data

Use download_data_from_pubmed.ipynb to fetch PubMed records for selected journals and year ranges. The notebook saves CSV files under data/raw/.

You can skip this step if you already have a CSV file with candidate keywords.

2. Preprocess Candidate Terms

Edit the configuration block at the bottom of preprocess_input.py for your domain, input file, and keyword column.

Then run:

python preprocess_input.py

This creates keyword_mapping.pkl and supporting cache files in data/processed/<folder_name>/.

Preprocessing modes:

• canonicalize: expands abbreviations and merges strict synonyms.
• canonicalize_and_filter_by_domain: also removes terms that are not relevant to the configured domain.

3. Construct the Taxonomy

Open construct_taxonomy.ipynb and edit the TaxonomyConfig cell.

Run the notebook cells in order. The notebook performs:

1. Candidate term loading from keyword_mapping.pkl.
2. Fine-grained taxonomy construction.
3. One or more broad taxonomy construction passes.
4. Final term (keyword)-node (topic) mapping and taxonomy export.

Outputs

Intermediate artifacts are saved under:

data/processed/<folder_name>/<experiment_name>/

The readable final hierarchy is written to:

results/<folder_name>/<experiment_name>/final_taxonomy.txt

Edges are represented as (child, parent) tuples.

Resuming Long Runs

Many steps can be resumed. The preprocessing script writes cache files under data/processed/<folder_name>/cache/, and the taxonomy passes save state after each topic group. In construct_taxonomy.ipynb, keep resume=True when calling:

run_fine_grained_taxonomy_pass(...)
run_broad_taxonomy_pass(...)

If you want to restart a pass from scratch, remove the corresponding generated pickle files from the experiment directory first.

Notes

• LLM calls can be slow and may incur API costs if using a hosted provider.
• The pipeline relies on LLM. Review final outputs before using them in downstream analyses.
• Some cycle resolution may require manual input when the LLM cannot confidently repair a graph; in this case, an interactive prompt will appear and wait for the user to choose which edge to remove.
• Large keyword sets can require substantial memory for embedding and similarity computations.

Citation

If you use this code in a publication or shared project, please cite the repository.