Language Guidance for Supervised Vision Training

An empirical study of how pretrained language models can improve generalization in standard supervised vision pipelines — without paired image-text data, contrastive pretraining, or prompt engineering.

---

Overview

Standard supervised vision training uses one-hot labels, which carry no information about semantic relationships between classes. This limited supervision leaves models vulnerable to distribution shifts, shortcut learning, texture bias, and catastrophic forgetting.

We study two lightweight mechanisms for injecting language-derived structure into vision training using off-the-shelf pretrained language models (PLMs):

Explicit Language Guidance (ExLG) — A frozen language encoder maps class-level text descriptions to embeddings. A similarity-preserving auxiliary loss aligns the visual feature space to match the relational structure of the language embedding space. The language model is used only at training time; inference cost is unchanged.

Implicit Language Guidance (ImLG) — A frozen pretrained language transformer block is inserted directly into the vision backbone (after the final ResNet layer, before the classifier). Trainable linear projections map visual features into and out of the language block. The backbone learns to route features through a semantically structured transformation.

Both methods are lightweight: they add minimal parameters, require no paired image-text data, and work with any standard vision architecture.

---

Results

Shortcut and Spurious Correlation Robustness

Language-guided models are substantially more robust to shortcut cues (color, background, hair color) that mislead baseline models.

Table 5: Shortcut learning on Tinted-CIFAR10, Skewed-CelebA, and Waterbirds.

Method	Tint-CIF10	CelebA Overall	NonBlonde-M	Blonde-F	Blonde-M	NonBlonde-F
Baseline	16.45±1.81	61.28±1.21	94.71±0.08	92.21±1.02	56.38±0.30	27.74±2.29
+ExLG	18.24±0.60	72.11±1.28	96.29±0.30	95.18±0.31	68.33±0.98	47.67±2.31
+ImLG	18.51±1.04	75.90±1.79	97.85±0.65	96.81±0.11	69.77±1.03	53.84±3.38

Method	Waterbirds Overall	Landbird/land	Waterbird/water	Landbird/water	Waterbird/land
Baseline	62.60±0.13	95.29±0.45	76.95±2.15	40.77±0.60	12.53±2.59
+ExLG	64.34±0.32	96.89±0.45	83.45±2.18	47.64±0.75	18.16±2.46
+ImLG	65.22±0.29	96.68±0.14	86.84±2.62	46.13±0.91	20.16±3.79

Activation Maps (Grad-CAM)

Language-guided models attend to semantically meaningful regions — facial features rather than hair color, bird body rather than background.

Activation maps on Skewed-CelebA (Blonde Men / Non-Blonde Women) and Waterbirds. The baseline fixates on spurious cues (hair color, background). ExLG and ImLG redirect attention toward the actual object.

---

Installation

conda env create -f env.yml
conda activate vlm_dytox

Datasets (CIFAR-10, CelebA, TinyImageNet, DN4IL) should be downloaded separately and their paths passed via --dataset_dir.

---

Usage

Choose --model as either "exlg" or "imlg"

python main.py \
  --model exlg \
  --arch resnet18mam \
  --dataset seq-cifar10 \
  --dataset_dir /path/to/cifar10 \

---