🏔️ aspect

aspect is a Python library and command-line tool for reproducible featurisation of tabular machine-learning datasets.

It lets you define independent transformation pipelines for different input columns, including molecular structures, taxonomy, categorical variables, and numerical measurements. Pipelines can be serialised, applied to new datasets, checkpointed, and restored.

Aspect uses Providence for dataset loading, source provenance, and checkpointing, and Hugging Face Datasets for Arrow-backed data processing and caching.

The core idea:

Input dataset
    │
    ├── SMILES ──────→ Molecular fingerprints
    ├── Species ─────→ Taxonomic fingerprints
    ├── Assay ───────→ Categorical encoding
    ├── MW ──────────→ Log transformation
    └── Description ─→ Language-model embeddings
    │
    ▼
Aspect DataPipeline
    │
    ├── Reusable feature specification
    ├── Reproducible dataset checkpoint
    └── Optional source materialisation
    │
    ▼
Providence DataResolver
    │
    ├── Hugging Face Dataset
    ├── Export: CSV / TSV / JSON / Parquet
    └── Model-specific collation / DataLoader

Feature representations are stored in Arrow-compatible columns. Model-specific collators can convert these stored values into tensors, molecular graphs, or other runtime representations without embedding model-specific logic in dataset preprocessing.

Quick start

1. Featurise a dataset

Create a small CSV:

cat > compounds.csv <<'CSV'
compound_id,assay,mwt,pIC50
cmpd-1,MIC,46.07,5.0
cmpd-2,MIC,60.10,6.2
cmpd-3,IC50,78.11,4.8
CSV

Generate numeric and categorical features:

aspect featurize compounds.csv \
    --features mwt:log@log_mwt assay:hash@assay_hash pIC50@target \
    --extras compound_id \
    --output features.parquet

The feature specification defines the transformations:

Specification	Meaning
`mwt:log@log_mwt`	Apply natural logarithm to `mwt`
`assay:hash@assay_hash`	Hash the assay category into a fixed-length vector
`pIC50@target`	Copy `pIC50` into a feature named `target`
`--extras compound_id`	Explicitly retain the compound identifier

By default, Aspect also retains original input columns.

2. Save a reproducible checkpoint

The same command can save a checkpoint alongside the exported table:

aspect featurize compounds.csv \
    --features mwt:log@log_mwt assay:hash@assay_hash \
    --checkpoint features.as \
    --output features.parquet

The checkpoint records the feature specification, data source, and information needed to reconstruct the transformed dataset.

3. Restore and export

Reconstruct the transformed dataset from its checkpoint:

aspect write features.as \
    --output restored.parquet

Export it in another format:

aspect write features.as \
    --output restored.csv.gz

4. Python API

from aspect.data import DataPipeline

data = {
    "compound_id": ["cmpd-1", "cmpd-2", "cmpd-3"],
    "assay": ["MIC", "MIC", "IC50"],
    "mwt": [46.07, 60.10, 78.11],
}

pipe = DataPipeline(
    column_transforms={
        "log_mwt": ("mwt", "log"),
        "assay_hash": (
            "assay",
            {"name": "hash", "kwargs": {"ndim": 8}},
        ),
    },
    columns_to_keep=["compound_id"],
)

resolved = pipe(data, drop_unused_columns=True)

# Access the underlying Hugging Face Dataset
features = resolved.data

print(features.column_names)
print(features[:2])

The transformed dataset contains:

['compound_id', 'log_mwt', 'assay_hash']

log_mwt contains a numeric feature with shape (n, 1), while assay_hash contains an (n, 8) feature matrix.

Save and reload the pipeline:

pipe.save("features.as")

restored = DataPipeline.load("features.as")
print(restored.data_out.data[:2])

Installation

Aspect requires Python 3.11 or later.

Basic installation

pip install aspect-data

Optional dependencies

Install only the features you need:

Extra	Functionality	Installation
`chem`	Molecular fingerprints and descriptors via Schemist	`pip install "aspect-data[chem]"`
`bio`	Taxonomic fingerprints via Vectome	`pip install "aspect-data[bio]"`
`torch`	PyTorch integration	`pip install "aspect-data[torch]"`
`chemprop`	Chemprop molecular features	`pip install "aspect-data[chemprop]"`
`deep`	Transformer-based embeddings	`pip install "aspect-data[deep]"`

Install from source

git clone https://github.com/scbirlab/aspect.git
cd aspect
pip install -e ".[dev]"

Aspect depends on Providence (>=0.0.2), which provides the common dataset-resolution and checkpointing infrastructure.

Command-line interface

Aspect provides three commands:

aspect serialize
aspect featurize
aspect write

View available arguments with:

aspect --help
aspect featurize --help
aspect serialize --help
aspect write --help

`aspect serialize`

Create a reusable feature specification without processing a dataset:

aspect serialize \
    --features mwt:log@log_mwt assay:hash@assay_hash \
    --extras compound_id \
    --output feature-spec.as

The specification can then be applied to other datasets:

aspect featurize new-compounds.csv \
    --config feature-spec.as \
    --output new-features.parquet

`aspect featurize`

Apply a feature specification or saved pipeline to a dataset.

aspect featurize compounds.csv \
    --features mwt:log@log_mwt assay:hash@assay_hash \
    --output features.parquet

A saved specification can be supplied with --config:

aspect featurize compounds.csv \
    --config feature-spec.as \
    --output features.parquet

Checkpoint without exporting

aspect featurize compounds.csv \
    --config feature-spec.as \
    --checkpoint features.as

Save source and transformed data

aspect featurize compounds.csv \
    --config feature-spec.as \
    --checkpoint features.as \
    --force-write-data

This requests materialisation of both source and transformed data inside the checkpoint, making it less dependent on the original dataset remaining available.

Without this option, Providence can retain immutable remote datasets by reference, avoiding unnecessary duplication. Local inputs are generally packaged by the current implementation.

Process a subset of rows

aspect featurize large-table.parquet \
    --start 1000 \
    --end 2000 \
    --features mwt:log@log_mwt \
    --checkpoint subset.as

--start is inclusive and --end is exclusive. The example selects rows 1000–1999.

Configure the cache

aspect featurize compounds.csv \
    --features assay:hash@assay_hash \
    --cache .cache \
    --output features.parquet

`aspect write`

Export data from a saved Aspect checkpoint:

aspect write features.as --output features.csv
aspect write features.as --output features.csv.gz
aspect write features.as --output features.parquet
aspect write features.as --output features.hf

The checkpoint is loaded through DataPipeline.load() and the output is delegated to Providence.

If the checkpoint contains only a feature specification and no source dataset, there is no transformed table to export.

Python API

The main class is DataPipeline.

from aspect.data import DataPipeline

A pipeline maps input columns to named output features.

pipe = DataPipeline({
    "log_mwt": ("mwt", "log"),
    "assay_hash": (
        "assay",
        {"name": "hash", "kwargs": {"ndim": 16}},
    ),
})

Input data

Pipelines can be applied to Python mappings, Pandas DataFrames, Hugging Face datasets, local files, or Providence resolvers.

# Dictionary
resolved = pipe({
    "mwt": [46.07, 60.10],
    "assay": ["MIC", "IC50"],
})

# Pandas DataFrame
resolved = pipe(df)

# Local file
resolved = pipe("compounds.csv")

# Hugging Face Hub dataset
resolved = pipe(
    "hf://datasets/scbirlab/fang-2023-biogen-adme~scaffold-split:train"
)

Providence integration

Aspect uses Providence's DataResolver to manage source datasets and transformed results.

from providence import DataResolver
from aspect.data import DataPipeline

source = DataResolver("compounds.csv")

pipe = DataPipeline({
    "log_mwt": ("mwt", "log"),
})

result = pipe(source)

# Underlying datasets
input_ds = pipe.data_in.data
output_ds = result.data

# Source metadata
source_info = pipe.data_in.data_source

print(output_ds.column_names)

DataPipeline.__call__() returns a DataResolver, rather than a raw Hugging Face dataset.

The underlying dataset remains available through .data, and provenance is tracked through the resolver.

Exporting results

pipe.write_table("features.parquet")
pipe.write_table("features.csv.gz")

This delegates serialization to Providence.

Selecting rows

For explicit row selection, create and slice the resolver before passing it to Aspect:

source = DataResolver("compounds.csv")
source = source.select_rows(start=100, end=200)

result = pipe(source)

Reusing transformations

The same feature specification can be applied to multiple datasets:

pipe = DataPipeline({
    "log_mwt": ("mwt", "log"),
})

train = pipe("train.csv")
test = pipe("test.csv")

Each call returns the corresponding transformed resolver. The pipeline's data_in and data_out attributes refer to the most recent call.

Feature specifications

Aspect supports both structured Python specifications and compact CLI strings.

Python specifications

The basic format is:

{
    "output_column": ("input_column", "transform_name")
}

Example:

pipe = DataPipeline({
    "log_mwt": ("mwt", "log"),
})

Parameterised transforms

Pass a dictionary containing the transform name and keyword arguments:

pipe = DataPipeline({
    "assay_hash": (
        "assay",
        {
            "name": "hash",
            "kwargs": {
                "ndim": 32,
                "seed": 123,
            },
        },
    ),
})

Chained transforms

Pass a sequence of transformations:

pipe = DataPipeline({
    "scaled_log_mwt": (
        "mwt",
        [
            "log",
            {
                "name": "scale",
                "kwargs": {"factor": 0.1},
            },
        ],
    ),
})

The scale transform in this example is a custom transform that must be registered first.

Chains execute in the supplied order. Each transform must accept the representation produced by the preceding transform.

CLI specifications

The CLI format is:

input_column:transform@output_column

Syntax	Behaviour
`column`	Retain an input column
`column@new_name`	Identity transformation with a new name
`column:transform`	Apply a transform and automatically name the output
`column:transform@new_name`	Apply a transform with an explicit output name
`column:t1:t2@new_name`	Chain transformations
`column:transform(arg=value)@name`	Pass simple transform arguments

Example:

aspect featurize compounds.csv \
    --features \
        mwt:log@log_mwt \
        assay:hash(ndim=32)@assay_hash \
        pIC50@target \
    --output features.parquet

Simple integers, floats, and booleans are parsed automatically. For complex or nested parameters, use the Python API.

Available transforms

Base transforms

Available with the standard installation:

Transform	Input	Output
`identity`	Any supported column	Identity representation
`log`	Numeric	Natural logarithm
`hash`	Categorical/string	Deterministic fixed-length vector
`one-hot`	Categorical	One-hot vector

Example:

pipe = DataPipeline({
    "log_mwt": ("mwt", "log"),
    "assay_hash": (
        "assay",
        {"name": "hash", "kwargs": {"ndim": 16}},
    ),
    "assay_onehot": (
        "assay",
        {
            "name": "one-hot",
            "kwargs": {"categories": ["MIC", "IC50"]},
        },
    ),
})

Chemistry transforms

Install with:

pip install "aspect-data[chem]"

Transform	Input	Output
`morgan-fingerprint`	SMILES	Molecular fingerprint
`descriptors-2d`	SMILES	2D molecular descriptors
`descriptors-3d`	SMILES	3D molecular descriptors

Example:

data = {
    "compound_id": ["ethanol", "ethylamine", "benzene"],
    "smiles": ["CCO", "CCN", "c1ccccc1"],
}

pipe = DataPipeline(
    {
        "morgan": ("smiles", "morgan-fingerprint"),
        "desc2d": ("smiles", "descriptors-2d"),
    },
    columns_to_keep=["compound_id"],
)

features = pipe(data, drop_unused_columns=True).data

Taxonomic transforms

Install with:

pip install "aspect-data[bio]"

The vectome-fingerprint transform encodes species names or taxonomic identifiers as fixed-length taxonomic representations.

data = {
    "species": [
        "Mycobacterium tuberculosis",
        "Escherichia coli",
        "Staphylococcus aureus",
    ],
}

pipe = DataPipeline({
    "species_fp": (
        "species",
        {
            "name": "vectome-fingerprint",
            "kwargs": {"ndim": 128},
        },
    ),
})

features = pipe(data).data

Deep-learning transforms

Install with:

pip install "aspect-data[deep]"

The hf-bart transform supports transformer-derived text embeddings.

data = {
    "description": [
        "cell wall inhibitor",
        "protein synthesis inhibitor",
        "DNA gyrase inhibitor",
    ],
}

pipe = DataPipeline({
    "text_embedding": (
        "description",
        {
            "name": "hf-bart",
            "kwargs": {
                "ref": "facebook/bart-base",
                "aggregator": ["mean", "max"],
            },
        },
    ),
})

features = pipe(data).data

Model weights may be downloaded on first use.

Chemprop transforms

Install with:

pip install "aspect-data[chemprop]"

The chemprop-mol transform prepares molecular datapoints for Chemprop-style graph models.

Molecular graphs generally require model-specific batching. Aspect separates dataset storage from runtime collation so graph representations can be reconstructed when needed.

Saving and loading pipelines

Aspect supports reusable feature specifications and provenance-aware dataset checkpoints.

Save a feature specification

Create a pipeline without applying it to any dataset:

pipe = DataPipeline({
    "log_mwt": ("mwt", "log"),
    "assay_hash": (
        "assay",
        {"name": "hash", "kwargs": {"ndim": 16}},
    ),
})

pipe.save("feature-spec.as")

Restore and apply it later:

from aspect.data import DataPipeline

pipe = DataPipeline.load("feature-spec.as")

result = pipe({
    "mwt": [46.07, 60.10],
    "assay": ["MIC", "IC50"],
})

Checkpoint a transformed dataset

pipe = DataPipeline({
    "log_mwt": ("mwt", "log"),
})

result = pipe("compounds.csv")
pipe.save("features.as")

The checkpoint records the pipeline configuration and a Providence checkpoint describing the input dataset.

Load it again:

restored = DataPipeline.load("features.as")

features = restored.data_out.data

By default, when transformed data has not been explicitly saved, loading reconstructs it by applying the feature pipeline to the restored source dataset.

Materialise transformed data

pipe.save(
    "features.as",
    save_transformed=True,
)

This stores a transformed-data checkpoint alongside the source checkpoint.

Package source data

pipe.save(
    "features.as",
    save_source_data=True,
    save_transformed=True,
)

This requests packaging of the input data and transformed data.

For remote datasets with immutable revisions, Providence can instead retain source references, allowing checkpoints to be substantially smaller.

Local sources are normally materialised to preserve reproducibility.

Load without applying transformations

pipe = DataPipeline.load(
    "features.as",
    skip_call=True,
)

This restores the specification and available checkpoint artefacts without automatically running the transformation pipeline.

Checkpoint layout

A checkpoint may contain:

features.as/
├── config.json
├── data.pv/
└── transformed.pv/

config.json contains the pipeline specification and checkpoint metadata.

data.pv contains the Providence source checkpoint.

transformed.pv is included when transformed data is explicitly checkpointed.

The contents of Providence checkpoints depend on whether datasets are retained by reference or materialised.

Compatibility

The legacy methods save_checkpoint() and load_checkpoint() remain available as compatibility aliases, but new code should use save() and load().

Supported input and output formats

Input types

Aspect accepts the input types supported by Providence, including:

Input	Example
Python mapping	`{"smiles": ["CCO", "CCN"]}`
Pandas DataFrame	`pipe(df)`
Hugging Face Dataset	`pipe(dataset)`
Hugging Face IterableDataset	`pipe(iterable_dataset)`
Local file	`pipe("data.csv")`
Hugging Face dataset reference	`pipe("hf://datasets/org/repo~config:split")`
Providence DataResolver	`pipe(resolved)`

Local files

Commonly supported formats include:

CSV: .csv, .csv.gz
Tab-separated: .tsv, .tsv.gz, .txt, .txt.gz
JSON: .json
Parquet: .parquet
Arrow: .arrow
XML: .xml
Saved Hugging Face datasets: .hf

Output formats

The CLI and write_table() use Providence to select an output format from the filename.

Extension	Format
`.csv`	CSV
`.csv.gz`	Gzip-compressed CSV
`.tsv`, `.txt`	Tab-separated text
`.tsv.gz`, `.txt.gz`	Gzip-compressed tab-separated text
`.json`	JSON
`.parquet`	Parquet
`.hf`	Hugging Face dataset directory
Unrecognised	Hugging Face dataset directory with `.hf` appended

Hugging Face Hub references

Providence supports versioned dataset references:

hf://datasets/<owner>/<repo>~<config>:<split>@<revision>

For example:

hf://datasets/scbirlab/fang-2023-biogen-adme~scaffold-split:train

The revision, configuration, and split are optional where supported by the resolver.

Pinning a dataset to an immutable commit revision improves reproducibility.

Working with output columns

By default, Aspect retains original columns and adds transformed features.

out = pipe(data)

To retain only transformed features and explicitly selected metadata:

pipe = DataPipeline(
    {
        "log_mwt": ("mwt", "log"),
        "assay_hash": ("assay", "hash"),
    },
    columns_to_keep=["compound_id"],
)

out = pipe(
    data,
    drop_unused_columns=True,
)

Extra columns can also be specified at call time:

out = pipe(
    data,
    drop_unused_columns=True,
    keep_extra_columns=["compound_id", "target"],
)

Access the underlying output dataset using:

dataset = out.data
print(dataset.features)
print(dataset.column_names)

Custom transforms

Custom transforms can be registered through Aspect's transform registry.

A transform factory receives configuration arguments and returns a callable that operates on data batches.

import numpy as np

from aspect.data import DataPipeline
from aspect.transform.registry import register_function


@register_function("square")
def Square():
    def _square(data, input_column):
        x = np.asarray(data[input_column], dtype=float)
        return x ** 2

    return _square


pipe = DataPipeline({
    "mwt_squared": ("mwt", "square"),
})

out = pipe({
    "mwt": [46.07, 60.10, 78.11],
}).data

Parameterised transforms

@register_function("scale")
def Scale(factor=1.0):
    def _scale(data, input_column):
        x = np.asarray(data[input_column], dtype=float)
        return x * factor

    return _scale


pipe = DataPipeline({
    "scaled_mwt": (
        "mwt",
        {"name": "scale", "kwargs": {"factor": 0.001}},
    ),
})

Custom transforms must be registered in the Python process before loading pipelines that reference them.

Model integration

Aspect separates stored feature representations from model-specific runtime inputs.

The output dataset may contain numeric arrays, chemical fingerprints, taxonomic encodings, or other Arrow-compatible values.

For simple numerical models, extract the dataset:

features = pipe(data).data

For PyTorch models, use the built-in DataLoader integration:

loader = pipe.dataloader(
    batch_size=32,
    shuffle=True,
)

for batch in loader:
    # Batched, collated features
    pass

This requires the torch extra.

Transforms requiring specialised runtime representations can provide collators. These are applied when constructing model batches, allowing features such as molecular graphs to be stored independently from the framework-specific objects consumed by a model.

Custom collators can be provided through the collators argument:

loader = pipe.dataloader(
    batch_size=32,
    collators={
        "custom_feature": custom_collator,
    },
)

Mixed molecular and biological features

Aspect can combine multiple representations in one dataset:

data = {
    "compound_id": ["cmpd-1", "cmpd-2", "cmpd-3"],
    "smiles": ["CCO", "CCN", "c1ccccc1"],
    "assay": ["solubility", "clearance", "microsome"],
    "species": [
        "Escherichia coli",
        "Mycobacterium tuberculosis",
        "Staphylococcus aureus",
    ],
    "mwt": [46.07, 45.08, 78.11],
    "label": [0.1, 0.4, 0.8],
}

pipe = DataPipeline(
    {
        "morgan": ("smiles", "morgan-fingerprint"),
        "desc2d": ("smiles", "descriptors-2d"),
        "assay_hash": (
            "assay",
            {"name": "hash", "kwargs": {"ndim": 32}},
        ),
        "species_fp": (
            "species",
            {"name": "vectome-fingerprint", "kwargs": {"ndim": 64}},
        ),
        "log_mwt": ("mwt", "log"),
        "y": ("label", "identity"),
    },
    columns_to_keep=["compound_id"],
)

features = pipe(
    data,
    drop_unused_columns=True,
).data

This requires the chemistry and taxonomy extras.

Downstream models can independently consume molecular fingerprints, descriptors, categorical features, and taxonomic encodings, or combine suitable dense features into a common representation.

Development

Install the development dependencies:

pip install -e ".[dev]"

Run the test suite:

pytest

Run module doctests:

pytest aspect --doctest-modules

Run the CLI integration tests:

bash test/scripts/run-tests.sh

Aspect maintains separate GitHub Actions workflows for Python tests and CLI smoke tests.

Related packages

Aspect is part of a small ecosystem for reproducible molecular machine-learning workflows:

Providence — Dataset loading, source provenance, and checkpointing.
Eluent — Chemistry-aware grouping, splitting, and cross-validation.
Schemist — Molecular featurisation and cheminformatics.
Vectome — Taxonomic and biological representations.

The packages can be used independently or composed into reproducible workflows from source data to model-ready features and evaluation splits.

Issues and documentation

Bug reports, feature requests, and contributions are welcome through the GitHub issue tracker.

Documentation: aspect.readthedocs.org

Source code: github.com/scbirlab/aspect

License: MIT.

Name		Name	Last commit message	Last commit date
.github/workflows		.github/workflows
aspect		aspect
docs		docs
test		test
.gitignore		.gitignore
.readthedocs.yml		.readthedocs.yml
LICENSE		LICENSE
README.md		README.md
pyproject.toml		pyproject.toml

Latest commit

History

Folders and files

Repository files navigation

🏔️ aspect

Contents

Quick start

1. Featurise a dataset

2. Save a reproducible checkpoint

3. Restore and export

4. Python API

Installation

Basic installation

Optional dependencies

Install from source

Command-line interface

aspect serialize

aspect featurize

Checkpoint without exporting

Save source and transformed data

Process a subset of rows

Configure the cache

aspect write

Python API

Input data

Providence integration

Exporting results

Selecting rows

Reusing transformations

Feature specifications

Python specifications

Parameterised transforms

Chained transforms

CLI specifications

Available transforms

Base transforms

Chemistry transforms

Taxonomic transforms

Deep-learning transforms

Chemprop transforms

Saving and loading pipelines

Save a feature specification

Checkpoint a transformed dataset

Materialise transformed data

Package source data

Load without applying transformations

Checkpoint layout

Compatibility

Supported input and output formats

Input types

Local files

Output formats

Hugging Face Hub references

Working with output columns

Custom transforms

Parameterised transforms

Model integration

Mixed molecular and biological features

Development

Related packages

Issues and documentation

About

Topics

Resources

Stars

Watchers

Forks

Releases

Contributors

Languages

`aspect serialize`

`aspect featurize`

`aspect write`