DimPred

DimPred predicts, for any image, its values on the SPoSE dimensions of human mental object representations (the 66 dimensions of Hebart et al., 2023, or the 49 dimensions of Hebart et al., 2020), and from these dimensions the perceived similarity between images. A deep neural network (by default the image encoder of AligNet SigLIP2-B, Muttenthaler et al., 2025) turns each image into a feature vector. One linear regression per dimension, trained on the 1854 THINGS reference images, maps the features to the dimension values. Heatmaps (RISE) show which parts of an image drive its predicted dimensions. The method and its evaluation are described in

Kaniuth, P., Mahner, F. P., Perkuhn, J., & Hebart, M. N. (2025). A high-throughput approach for the efficient prediction of perceived similarity of natural objects. eLife 14:RP105394. https://doi.org/10.7554/eLife.105394

This repository contains a small Python package and a MATLAB version that apply the trained models, the trained models themselves, the training code of the DimPred paper, and the tests.

If you used an earlier version: the training code (fit.py, call.py) is in training/, and version 1.0.0 is the tag v1.0.0. What changed in 1.1.0: docs/changes.md.

Contents: Installation | Quick start | Models | More | Citation | Credits | License

Installation

Python 3.9 or newer. From a clone of this repository:

git clone https://github.com/ViCCo-Group/dimpred.git
cd dimpred
pip install -e .              # predictions and similarity from features (installs numpy and scipy)
pip install -e ".[extract]"   # in addition features and heatmaps from images (torch, open_clip_torch, timm, pillow)
pip install -e ".[test]"      # pytest, for running the tests

pip install -e . alone is enough to predict the dimensions and the similarity from features you already have. To get features or heatmaps from images, install ".[extract]" as well. The default model downloads the weights of its network (378 MB) once into ~/.cache/dimpred.

MATLAB (R2016b or newer): add the folder matlab/ to the path (addpath, see MATLAB) and keep it inside the repository, where its functions find the models. Everything runs in MATLAB alone except dimpred_extract_features and dimpred_rise, which run the Python command line tool. They need Python with numpy, scipy, torch, open_clip_torch, timm (1.0.15 or newer) and pillow; dimpred itself does not have to be installed there.

Quick start

Python

import dimpred

files = dimpred.find_images("my_images")     # image files in this folder, sorted by name
features = dimpred.extract_features(files)   # one row per image, in the order of files
embedding = dimpred.predict(features)        # images x 66 dimensions (default model)
S = dimpred.similarity(embedding)            # images x images, predicted similarity
labels = dimpred.load_model()["labels"]      # names of the 66 dimensions

With another model, the features have to come from the network of that model. extract_features takes the network from the model you give it:

print(dimpred.list_models())
model = dimpred.load_model("rn50x64_49d_ridge")     # the model of the DimPred paper
features = dimpred.extract_features(files, model)   # RN50x64 features
embedding = dimpred.predict(features, model)        # images x 49 dimensions

If the features do not fit the model (e.g. AligNet features and an RN50x64 model), predict stops with an error that names the network of the model. Features you extracted before (same network and layer, see Feature extraction) can be passed as an array.

Heatmaps with RISE (default model, 6000 masks: 49 s per image on the GPU of an Apple M1 Max; burrito.jpg is in tests/fixtures/images):

result = dimpred.rise("burrito.jpg")             # maps of each dimension and the relevance map
top = result["embedding"][0].argmax()            # the dimension with the largest predicted value
print(result["labels"][top])                     # food-related
from PIL import Image
from dimpred.rise import overlay
Image.fromarray(overlay(result["view"][0], result["relevance"][0])).save("burrito_relevance.png")

MATLAB

addpath('/path/to/dimpred/matlab')
files = dimpred_find_images('my_images');     % cell column of full paths, sorted by name
features = dimpred_extract_features(files);   % runs Python, one row per image
embedding = dimpred_predict(features);        % images x 66 dimensions (default model)
S = dimpred_similarity(embedding);            % images x images, predicted similarity
model = dimpred_load_model;                   % default model, model.labels holds the dimension names

The functions that run Python use the Python in cfg.python if given, else the one in the environment variable DIMPRED_PYTHON, else python3 (on Windows, set cfg.python or DIMPRED_PYTHON):

cfg.python = '/path/to/env/bin/python';   % a Python with numpy, scipy, torch, open_clip_torch, timm and pillow
cfg.device = 'cpu';                       % optional, default: cuda, then mps, then cpu
[features, files] = dimpred_extract_features(files, 'rn50x64_49d_ridge', cfg);
embedding = dimpred_predict(features, 'rn50x64_49d_ridge');
cfg.device = '';                          % heatmaps on the GPU if there is one
result = dimpred_rise('burrito.jpg', [], cfg);   % heatmaps with the default model, 6000 masks
[~, top] = max(result.embedding(1, :));   % result.labels{top} is the dimension with the largest value
imagesc(squeeze(result.relevance(1, :, :))), axis image off, colormap(jet)

Command line

python -m dimpred my_images                           # all images in the folder, writes dimpred_predictions.csv
python -m dimpred a.jpg b.jpg --model rn50x64_49d_ridge --out predictions.mat
python -m dimpred my_images --features-only --out features.mat
python -m dimpred --features features.mat --out predictions.csv
python -m dimpred --rise burrito.jpg --png burrito_maps   # heatmaps, writes dimpred_heatmaps.mat and PNG files

python -m dimpred --help lists all options. All functions of Python and MATLAB side by side, and more on the command line: docs/details.md.

Models

Which model to use:

• To reproduce Contier et al. (2024): vitb32_66d_elastic, the rebuilt model of that paper (the default of 1.0.0).
• To reproduce the DimPred paper: rn50x64_49d_ridge, the published RN50x64 model of the paper (its heatmaps and fMRI encoding). The code of the paper is in https://github.com/ViCCo-Group/dimpred_paper, its data and all 53 models on OSF, https://osf.io/jtekq (more).
• For the best prediction of the dimensions: the default model, alignet_siglip2b_66d_ridge.
• For heatmaps: rn50x64_66d_ridge (a convolutional network; of the 66d models, its maps were the closest to Figure 7 of the paper), or the default, which is much faster (6000 masks on the GPU of an Apple M1 Max: about 10 min per image instead of 49 s; 2000 masks still give stable maps).
• Use the same model for all images you want to compare (the values of 49d and 66d models cannot be mixed).

name	network	features	dims	regression	use
alignet_siglip2b_66d_ridge	AligNet SigLIP2-B (pre_logits)	768	66	ridge	default, best prediction of the dimensions
vitb32_66d_elastic	OpenAI CLIP ViT-B/32 (ViT-B-32-quickgelu)	512	66	elastic net	rebuilt model of Contier et al. (2024)
rn50x64_49d_ridge	OpenAI CLIP RN50x64 (RN50x64)	1024	49	fractional ridge	model of the DimPred paper
rn50x64_66d_ridge	OpenAI CLIP RN50x64	1024	66	ridge	heatmaps (convolutional network)
rn50x64_66d_elastic	OpenAI CLIP RN50x64	1024	66	elastic net	RN50x64 66d with elastic net

All models were trained on the 1854 THINGS reference images (the images of the odd-one-out experiments) with the training code in training/. rn50x64_49d_ridge is not a refit but the published fit of the paper.

How well the models work on new images

Mean r per dimension (10-fold cross-validation on the 1854 training images), and Pearson r between predicted and human similarity, averaged over 10 image sets never used for training and over the 8 of them without 48new and 48nonref:

model	per dimension	10 test sets	8 test sets
alignet_siglip2b_66d_ridge	0.810	0.634	0.572
vitb32_66d_elastic	-	0.598	0.535
rn50x64_49d_ridge	0.717 (49d)	0.562	0.496
rn50x64_66d_ridge	0.779	0.583	0.517
rn50x64_66d_elastic	-	0.578	0.511

AligNet's teacher was fitted to the THINGS odd-one-out data, so its per-dimension r and its values on 48new and 48nonref are probably too optimistic; to test predictions against THINGS odd-one-out data, use vitb32_66d_elastic. The test sets and all tables: docs/models.md.

More

• Images and limitations, how the predictions are computed, similarity
• Feature extraction, image order, heatmaps (RISE)
• Command line, file formats, MATLAB
• The models, which model was used where
• Training and reproducing the DimPred paper
• Tests, repository layout, changes in 1.1.0

Citation

Please cite

• the DimPred paper (below),
• the paper of the dimensions: Hebart et al. (2023), eLife (66d), or Hebart et al. (2020), Nature Human Behaviour (49d),
• with the default model also Muttenthaler, L., Greff, K., Born, F., Spitzer, B., Kornblith, S., Mozer, M. C., Müller, K.-R., Unterthiner, T., & Lampinen, A. K. (2025). Aligning machine and human visual representations across abstraction levels. Nature 647, 349-355, https://doi.org/10.1038/s41586-025-09631-6,
• with vitb32_66d_elastic also Contier, O., Baker, C. I., & Hebart, M. N. (2024). Distributed representations of behaviour-derived object dimensions in the human visual system. Nature Human Behaviour 8, 2179-2193, https://doi.org/10.1038/s41562-024-01980-y,
• with heatmaps also Petsiuk, V., Das, A., & Saenko, K. (2018). RISE: Randomized input sampling for explanation of black-box models. BMVC.

@article{Kaniuth_2025,
	author={Kaniuth, Philipp and Mahner, Florian P and Perkuhn, Jonas and Hebart, Martin N},
	title={A high-throughput approach for the efficient prediction of perceived similarity of natural objects},
	journal={eLife},
	volume={14},
	pages={RP105394},
	year={2025},
	DOI={10.7554/eLife.105394},
	url={https://doi.org/10.7554/eLife.105394},
	publisher={eLife Sciences Publications, Ltd}
}

Credits

Method and training code: Philipp Kaniuth. Model export with dimension means (MATLAB) and the heatmaps of the DimPred paper: Florian Mahner. The wrapper this package builds on: Luca Kämmer. Contier model: Oliver Contier. AligNet: Lukas Muttenthaler and colleagues. Package, MATLAB version, AligNet port, heatmaps and tests: Hebartlab.

License

GNU Affero General Public License v3.0 (AGPL-3.0), see LICENSE. The AligNet weights that the default model downloads come with the license of the AligNet models (training/alignet).