A Computer Vision and Deep Learning framework for five-class oral histopathology classification using Swin Transformer, CLIP-guided semantic fusion, and Explainable AI.
ORAL AI is a research-oriented Computer Vision system developed during my internship at the International Center for Chemical and Biological Sciences (ICCBS).
The project explores the application of modern Deep Learning and Vision Transformer architectures to oral histopathology image classification.
The primary objective was to develop a model capable of learning discriminative visual patterns from histopathological images and classifying them into five oral histopathology categories, while also providing visual explanations for its predictions.
The project covers the complete AI workflow:
Data → Preprocessing → Model Development → Experimentation → Evaluation → Explainability → Deployment
What initially began as a research problem evolved into an end-to-end AI project involving both experimental research and practical implementation.
Medical imaging presents a particularly challenging Computer Vision problem.
Unlike many conventional image classification tasks, histopathological images contain complex biological structures, subtle visual differences, considerable variation in tissue morphology, and changes caused by staining and image acquisition.
In oral histopathology, visually similar patterns may correspond to significantly different pathological conditions.
This creates several challenges:
- Subtle inter-class differences
- High intra-class variation
- Complex tissue structures
- Staining and illumination variation
- Fine-grained visual patterns
- Large and information-dense histopathological images
- Need for reliable model interpretation
Therefore, high classification accuracy alone is not sufficient. Understanding why a model reaches a particular prediction is also important, especially in medical imaging research.
This motivated the incorporation of Explainable AI through Grad-CAM alongside the classification pipeline.
The final architecture combines visual representation learning with semantic guidance.
Input Histopathology Image
│
▼
┌───────────────────┐
│ Image Processing │
│ & Augmentation │
└─────────┬─────────┘
│
▼
┌───────────────────┐
│ Swin Transformer │
│ Visual Backbone │
└─────────┬─────────┘
│
Multi-Stage Features
│
▼
┌───────────────────┐
│ CLIP Semantic │
│ Guidance │
└─────────┬─────────┘
│
▼
┌───────────────────┐
│ Multi-Stage │
│ Cross-Attention │
└─────────┬─────────┘
│
▼
┌───────────────────┐
│ Feature Fusion │
└─────────┬─────────┘
│
▼
┌───────────────────┐
│ U-Net Inspired │
│ Decoder │
└─────────┬─────────┘
│
▼
┌───────────────────┐
│ 5-Class │
│ Classification │
└───────────────────┘
│
▼
Prediction
│
▼
Grad-CAM
The Swin Transformer serves as the primary visual backbone.
It provides hierarchical visual representations and enables the model to capture both local and broader contextual information from histopathological images.
The multi-stage structure is particularly useful for medical images where meaningful patterns can exist at different spatial scales.
CLIP is incorporated to introduce semantic information into the visual learning process.
Rather than relying solely on image features, the architecture explores the interaction between visual representations and semantic information.
This provides additional guidance for distinguishing visually similar pathological classes.
Cross-attention is used to allow information from different representations to interact.
The multi-stage design enables semantic information to influence visual representations at different levels of the feature hierarchy.
This was an important part of the experimentation process and the final architecture.
The architecture combines complementary representations through multi-stage feature fusion.
The objective is to preserve useful information from different feature levels while producing a stronger representation for classification.
A U-Net-inspired decoder is incorporated to progressively refine and combine feature representations.
Although the primary task is classification, the decoder structure provides a mechanism for retaining richer spatial information within the architecture.
To improve interpretability, Grad-CAM (Gradient-weighted Class Activation Mapping) was incorporated into the pipeline.
Grad-CAM generates a visual heatmap highlighting image regions that contributed most strongly to a model prediction.
This allows us to move beyond:
"What did the model predict?"
toward:
"Which regions influenced the model's prediction?"
This is particularly important in medical imaging research, where interpretability can provide additional insight into model behavior.
The final system performs classification across five oral histopathology categories:
| Class | Description |
|---|---|
| MDOSCC | Moderately Differentiated Oral Squamous Cell Carcinoma |
| NORMAL | Normal Oral Tissue |
| OSMF | Oral Submucous Fibrosis |
| PDOSCC | Poorly Differentiated Oral Squamous Cell Carcinoma |
| WDOSCC | Well Differentiated Oral Squamous Cell Carcinoma |
The final model achieved the following performance on the test set:
| Metric | Result |
|---|---|
| Test Accuracy | 95.73% |
| Macro F1-Score | 96.20% |
| Macro ROC-AUC | 0.9976 |
These results demonstrate the model's ability to discriminate between the five oral histopathology classes within the evaluated dataset.
Important: These metrics represent experimental results on the evaluated test set and should not be interpreted as clinical validation or evidence of diagnostic performance in real-world clinical settings.
Grad-CAM was used to investigate the visual regions influencing model predictions.
These visualizations are intended for research and interpretability analysis and should not be considered a substitute for expert pathological assessment.
The trained model was integrated into an interactive application that allows users to:
- Upload a histopathology image
- Generate an AI prediction
- View prediction confidence
- View class probability distribution
- Generate Grad-CAM visualizations
- Compare the original image with model attention maps
Launch ORAL AI on Streamlit Cloud →
ICCBS-Intership-Project/
│
├── Oral_Histopathology_Backend.ipynb
│ └── Model development, training, evaluation,
│ Swin Transformer + CLIP architecture,
│ preprocessing, and experimentation
│
├── Oral_Histopathology_Frontend.ipynb
│ └── Model inference, Grad-CAM explainability,
│ prediction pipeline, and Gradio interface
│
├── app.py
│ └── Production-level Streamlit web dashboard with interactive
│ predictions, confidence visualization, and Grad-CAM overlays
│
├── README.md
Contains the experimental pipeline including:
- Dataset preparation and organization
- Image preprocessing
- Image augmentation
- Dataset splitting
- Swin Transformer backbone
- CLIP-based semantic guidance
- Multi-stage feature extraction
- Cross-attention
- Feature fusion
- U-Net-inspired architecture
- Model training and fine-tuning
- Validation and testing
- Performance evaluation
- Classification metrics
Contains the inference and explainability pipeline including:
- Trained model loading
- Histopathology image preprocessing
- Five-class prediction
- Prediction confidence
- Class probability distribution
- Grad-CAM generation
- Original / heatmap / overlay visualization
- Interactive Streamlit dashboard
- Python
- PyTorch
- Torchvision
- timm
- Image preprocessing
- Image augmentation
- Feature extraction
- Image classification
- Vision Transformers
- Attention mechanisms
- Grad-CAM
- Swin Transformer
- CLIP
- Multi-stage feature extraction
- Cross-attention
- Semantic feature fusion
- U-Net-inspired decoder
- Grad-CAM
- Streamlit
- Kaggle / Jupyter-based experimentation
This project was developed as part of my research-oriented internship at ICCBS.
The work explores how semantic guidance from CLIP can be combined with hierarchical visual representations from Swin Transformer architectures for oral histopathology classification.
The project has also laid the foundation for a research paper exploring CLIP-guided Swin Transformer architectures for oral histopathology.
The research focuses on:
- Computer Vision for medical imaging
- Vision Transformers
- Multimodal semantic guidance
- Cross-attention
- Feature fusion
- Explainable AI
- Oral histopathology classification
One of the most valuable aspects of this project was the learning process itself.
I started the internship without a deep understanding of Computer Vision or histopathology. Much of the knowledge developed throughout the project came from implementing ideas, reading research papers, analyzing failures, debugging experiments, and iterating on architectural decisions.
The project provided practical exposure to the challenges involved in moving from:
Research Problem
↓
Data
↓
Experimentation
↓
Architecture Design
↓
Model Training
↓
Evaluation
↓
Explainability
↓
Application
This project is intended strictly for research and educational purposes.
The model has not been clinically validated and is not intended to replace professional pathological examination, medical diagnosis, or clinical decision-making.
The reported performance represents experimental evaluation on the dataset used in this research project and should not be interpreted as clinical performance.
Institution: International Center for Chemical and Biological Sciences (ICCBS)
Project Area: Artificial Intelligence, Computer Vision & Healthcare
Focus: Oral Histopathology Image Classification
A research paper based on this work is currently in progress, focusing on:
CLIP-Guided Swin Transformer Architectures for Oral Histopathology
Publication details will be added here once available.
Mahrukh Baig
This repository is intended primarily for research and educational purposes.
Please review the dataset's original licensing and usage terms before using or redistributing any associated data.
To run the Streamlit dashboard locally, follow these steps:
-
Clone the repository:
git clone https://github.com/MahrukhBaig/ICCBS-Intership-Project.git cd ICCBS-Intership-Project -
Set up a virtual environment & install dependencies:
python -m venv venv # On Windows: .\venv\Scripts\activate # On macOS/Linux: source venv/bin/activate pip install -r requirements.txt
-
Add the model checkpoint: Place your
best_orchid_hybrid_stage2.pthinside themodels/directory (or configureMODEL_URLinapp.pyto auto-download it). -
Launch the dashboard:
streamlit run app.py