FacialNet

FacialNet is a deep learning Facial Emotion Recognition (FER) system. It classifies a face into one of seven basic emotions (anger, disgust, fear, happiness, sadness, surprise and neutral) and runs both on still images and on live webcam or video streams.

The project compares three ImageNet-pretrained CNN backbones, ResNet-18, DenseNet-121 and EfficientNet-B0, fine-tuned with transfer learning on two benchmark datasets: FER2013 (grayscale) and RAF-DB (color). A DnCNN denoiser makes the models robust to noisy camera images, and a conditional GAN generates new faces showing any of the seven emotions.

Quick Start

Run the web UI with the trained RAF-DB models in four commands (macOS or Linux, Python 3.10+):

cd FacialNet
python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
python webui/server.py

Then open http://localhost:8000 in Chrome, Safari or Firefox. The model weights must be in place first (see Model Weights); the submission zip already contains them. On Windows, activate the environment with .venv\Scripts\activate instead.

I want to…	Run (from `FacialNet/`, with `.venv` active)	Needs
Use the web UI (webcam, images, benchmarks)	`python webui/server.py`	RAF-DB weights; RAF-DB test set for the sample faces
Run the OpenCV webcam demo	`cd "Video Predicition" && python video_inference_efficientnet.py`	RAF-DB weights, webcam
Run it on a video file with denoising	`cd "Video Predicition" && python video_inference_efficientnet.py clip.mp4 --denoise`	RAF-DB weights, DnCNN weights
Retrain a RAF-DB model	`cd "Video Predicition" && jupyter nbconvert --to notebook --execute --inplace efficientnet.ipynb --ExecutePreprocessor.timeout=-1`	RAF-DB dataset, a GPU recommended
Reproduce the denoising study	`cd Denoising && jupyter nbconvert --to notebook --execute --inplace denoising.ipynb --ExecutePreprocessor.timeout=-1`	RAF-DB dataset and the three RAF-DB weights
Train the GAN and generate faces	`cd GAN && jupyter nbconvert --to notebook --execute --inplace gan.ipynb --ExecutePreprocessor.timeout=-1`	RAF-DB dataset and the three RAF-DB weights

Each row is explained in detail in Getting Started, Training, Inference and Web UI.

Highlights

Three backbones, one recipe: ResNet-18, DenseNet-121 and EfficientNet-B0 trained with the same data pipeline and hyperparameters, so results compare directly.
Real-time video emotion tracking: OpenCV face detection, per-face emotion classification with softmax confidence, temporal smoothing and an FPS counter.
Augmentation for real-world faces: random zoom, flips, rotation, brightness and contrast changes and random occlusion keep clean RAF-DB accuracy the same, but make the models 2–9 points more accurate on dark, low-contrast, partly covered or tightly cropped faces.
Noise robustness with DnCNN: Gaussian noise cuts average RAF-DB accuracy from 87.2% to 34.8% (σ = 50); a DnCNN denoiser fine-tuned on faces restores it to 78.4%, and can clean every face in the live video pipeline.
Emotional faces from a conditional GAN: a GAN trained on RAF-DB draws new 64×64 faces for any of the seven emotions, can show one synthetic person with every emotion or morph smoothly between two, and is checked by the three classifiers.
Web UI: a local browser app with live webcam tracking, photo upload, side-by-side model comparison, an interactive noise and denoising demo, a live GAN face generator and interactive benchmark charts. It needs no packages beyond requirements.txt.
Static image prediction: single-image scripts for the FER2013 models.
Runs anywhere PyTorch does: the RAF-DB notebooks pick NVIDIA CUDA, Apple Silicon (mps) or CPU automatically, and work unchanged on Kaggle or locally.
Notebooks with results: the training notebooks are saved with their outputs, including per-epoch loss and accuracy and training curves; the RAF-DB notebooks also report per-class test accuracy.

Results

RAF-DB (color, 7 classes, 3,068 test images)

Trained for 20 epochs on an Apple M4 MacBook Air using the mps backend, with the augmentation described in Preprocessing and augmentation. These are the weights in Video Predicition/ that the video scripts and the web UI use by default. Time per epoch is the median wall-clock time of one training and test pass.

Model	Parameters	Weights file	Time per epoch	Final train accuracy	Test accuracy
ResNet-18	11.2M	45 MB	2.1 min	96.6%	87.22%
DenseNet-121	7.0M	28 MB	5.0 min	97.2%	88.01%
EfficientNet-B0	4.0M	16 MB	3.6 min	98.1%	87.06%

DenseNet-121 is about 0.8 points ahead of the other two, which is close to the run-to-run noise of about 1 point (see Effect of data augmentation). EfficientNet-B0 matches ResNet-18 with about a third of its parameters.

Training for 20 instead of 12 epochs changed test accuracy by +0.19 (ResNet-18), +0.56 (DenseNet-121) and 0.00 points (EfficientNet-B0). The 12-epoch models are kept in Video Predicition/aug12_weights/ and can be selected in the web UI.

Per-class test accuracy:

Emotion (test images)	ResNet-18	DenseNet-121	EfficientNet-B0
Surprise (329)	87.8%	86.9%	88.1%
Fear (74)	56.8%	58.1%	58.1%
Disgust (160)	62.5%	60.6%	65.0%
Happy (1,185)	93.8%	94.9%	92.8%
Sad (478)	87.9%	85.4%	89.1%
Anger (162)	80.9%	85.2%	79.0%
Neutral (680)	85.6%	88.8%	85.3%

Fear and Disgust are the hardest classes for every model. They are also the least represented in the training set (see Datasets).

FER2013 (grayscale, 7 classes)

Trained for 15 epochs on Kaggle (NVIDIA Tesla T4) with mixed precision.

Model	Test accuracy
ResNet-18	69.53%
DenseNet-121	71.58%

FER2013 is considerably harder than RAF-DB: its images are 48×48 grayscale with noisier labels.

All numbers above are the test accuracy after the final epoch, as printed in the notebooks.

Effect of data augmentation (RAF-DB)

This comparison uses two 12-epoch runs. The first version of the RAF-DB models (Video Predicition/baseline_weights/) was trained with only a horizontal flip and ±15° rotation. The second (Video Predicition/aug12_weights/) adds random zoom, brightness and contrast changes and random occlusion (see Preprocessing and augmentation); everything else in the recipe is unchanged. Both versions were evaluated on the same test images, so each comparison is paired: the confidence interval is a bootstrap over test images, and the p-value is an exact McNemar test on the images that only one version classifies correctly.

On the clean test set, the extra augmentation makes no significant difference:

Model	Flip + rotation	Full augmentation	Difference (95% CI)	McNemar p
ResNet-18	86.90%	87.03%	+0.13 (−0.91 to +1.17)	0.85
DenseNet-121	87.48%	87.45%	−0.03 (−1.04 to +0.98)	1.00
EfficientNet-B0	87.42%	87.06%	−0.36 (−1.37 to +0.62)	0.52

The two versions of each model disagree on 240–265 of the 3,068 test images, even though their accuracies are almost equal. Differences of about 1 point between single training runs, including between the three backbones, are therefore within run-to-run noise.

The extra augmentation does reduce overfitting: final training accuracy falls from 98.2–98.8% to 92.8–96.0%. It also makes every model clearly more robust when the test faces are changed at test time:

Test condition	ResNet-18	DenseNet-121	EfficientNet-B0
Clean	86.9% → 87.0%	87.5% → 87.5%	87.4% → 87.1%
Dark (brightness × 0.5)	85.3% → 86.9%	85.1% → 87.0%	85.2% → 87.1%
Low contrast (contrast × 0.5)	84.7% → 86.7%	85.1% → 87.2%	85.3% → 87.2%
Occluded (random box covering 5–15% of the face)	80.5% → 84.2%	80.0% → 85.0%	82.8% → 85.4%
Tight crop (central 80% of the face)	73.9% → 83.4%	79.3% → 84.8%	82.7% → 85.2%

Every change in the four harder conditions is significant (McNemar p < 0.005). Without the extra augmentation the models lose up to 13 points under these conditions; with it they lose at most 4. The halved brightness and contrast and the 80% crop are stronger than anything seen in training, and all four conditions are common in webcam frames: poor lighting, hands or hair over the face, and detector boxes that crop differently from RAF-DB's alignment.

Noise robustness with DnCNN (RAF-DB)

Gaussian noise at σ = 15, 25 and 50 (0–255 scale) was added to the 3,068 RAF-DB test faces, which were then classified as-is and after denoising with DnCNN (see Denoising).

Denoising quality, PSNR (dB) / SSIM against the clean faces:

Noise σ	Noisy input	Pretrained DnCNN	DnCNN fine-tuned on RAF-DB
15	24.82 / 0.572	34.94 / 0.943	34.99 / 0.945
25	20.53 / 0.384	32.09 / 0.909	32.50 / 0.916
50	14.99 / 0.185	27.54 / 0.833	28.76 / 0.848

Emotion accuracy on noisy faces, before and after the fine-tuned DnCNN:

Model	Clean	σ = 15 noisy → denoised	σ = 25 noisy → denoised	σ = 50 noisy → denoised
ResNet-18	86.86%	81.49% → 85.40%	73.79% → 84.16%	38.43% → 78.29%
DenseNet-121	87.42%	82.43% → 86.21%	75.95% → 84.65%	49.45% → 78.62%
EfficientNet-B0	87.22%	56.36% → 85.85%	18.84% → 84.35%	16.56% → 78.19%

Noise hurts every model, and EfficientNet-B0 most of all: it falls to 18.84% at σ = 25.
Denoising first brings all three models back to within 1.5 points of clean accuracy at σ = 15, within 3 points at σ = 25, and to about 78% at σ = 50.
Fine-tuning DnCNN on faces adds 0.4 dB PSNR at σ = 25 and 1.2 dB at σ = 50 over the authors' pretrained model.
These numbers were measured with the first version of the RAF-DB models (baseline_weights/, flip and rotation only; see Effect of data augmentation). The clean column resizes images on the GPU, so it differs from those models' PIL-resize accuracy by up to 0.2 points.

Conditional GAN (RAF-DB, 64×64)

A conditional DCGAN trained for 10,000 iterations (63 minutes on an Apple M4) generates new faces for a requested emotion (see Conditional GAN). Each row below is one emotion; none of these people exist:

Does the face show the requested emotion? Each classifier labelled 500 generated faces per emotion. As a reference, the same classifiers were run on the real RAF-DB test faces downscaled to the GAN's 64×64 size:

Classifier	Surprise	Fear	Disgust	Happy	Sad	Anger	Neutral	Mean
ResNet-18, real 64×64	87.2%	52.7%	59.4%	94.2%	79.1%	76.5%	88.4%	76.8%
ResNet-18, generated	80.4%	22.8%	40.2%	97.0%	44.6%	46.4%	77.6%	58.4%
DenseNet-121, real 64×64	86.9%	60.8%	55.6%	94.6%	78.5%	82.7%	88.8%	78.3%
DenseNet-121, generated	77.4%	20.8%	26.4%	96.8%	39.0%	56.8%	78.6%	56.5%
EfficientNet-B0, real 64×64	88.8%	51.4%	65.6%	93.8%	82.4%	79.6%	85.3%	78.1%
EfficientNet-B0, generated	80.8%	17.4%	36.0%	97.0%	50.0%	56.0%	74.0%	58.7%

Generated Happy faces are recognised as happy 97% of the time, more often than real ones; Surprise and Neutral reach 74–81%.
Anger (46–57%) and Sad (39–50%) are recognised in about half of the faces, Disgust (26–40%) and Fear (17–23%) less often. These are the rarest classes (Fear has 17× fewer training faces than Happy).
Training for 10,000 instead of 3,000 iterations raised the mean from 43–44% to 56–59%, with the largest gains for Anger (from 7–11%) and Sad (from 25–26%).

Realism and diversity. The Fréchet distance between 3,068 generated faces and the 3,068 real test faces, in EfficientNet-B0 feature space, is 4.9 (9.1 after 3,000 iterations); between real training and real test faces it is 2.7, the floor a perfect generator would reach. The last raw weights, without weight averaging, score 9.1. During training, the distance to 2,000 training faces fell at every checkpoint, from 36.7 at iteration 1,000 to 8.8 at 10,000, so the final checkpoint was kept. These values use RAF-DB EfficientNet-B0 features, so they are not comparable with FID numbers in papers.

Dataset bias. The emotion label also changes the photo style: some faces turn grayscale when rendered as Fear, Sad or Anger. This mirrors RAF-DB, where 46% of the Fear training images are grayscale, against 23% for Sad, 20% for Anger and 8% for Happy.

No copying. For every emotion, the three closest training faces in pixel space are different people (GAN/gan_nearest_neighbours.png), so the GAN generates new faces rather than reproducing training images.

Keeping the random input fixed and changing only the label keeps the face and changes its expression; the change is clearest for Happy:

How It Works

Datasets

	FER2013	RAF-DB
Images	48×48 grayscale	100×100 color, aligned faces
Train / test	28,709 / 7,178	12,271 / 3,068
Folder labels	`angry`, `disgust`, `fear`, `happy`, `neutral`, `sad`, `surprise`	`1`–`7` (see below)
Used in	`Image prediction/`, root notebooks	`Video Predicition/`

RAF-DB folder numbers map to emotions as follows. The training-set counts show the strong class imbalance:

Folder	Emotion	Train images	Test images
`1`	Surprise	1,290	329
`2`	Fear	281	74
`3`	Disgust	717	160
`4`	Happy	4,772	1,185
`5`	Sad	1,982	478
`6`	Anger	705	162
`7`	Neutral	2,524	680

Both datasets use the ImageFolder layout (train/<class>/*.jpg, test/<class>/*.jpg), so class indices follow the sorted folder names.

Preprocessing and augmentation

All models take 224×224, 3-channel input normalized with ImageNet mean and standard deviation, matching the pretrained backbones.

Step	FER2013	RAF-DB
Resize	224×224	224×224
Color	Grayscale copied to 3 channels	Native RGB
Train augmentation	Horizontal flip, rotation ±10° (DenseNet-121 also translates up to 10%)	See below
Normalization	ImageNet mean/std	ImageNet mean/std

The RAF-DB training images are augmented on the fly, so every epoch sees a different version of each face. Test images are only resized and normalized.

RAF-DB augmentation	Setting	Simulates
Random zoom (`RandomResizedCrop`)	Crop 85–100% of the area, aspect ratio 0.9–1.1, resize to 224×224	Face detector boxes of different sizes
Horizontal flip	50% of images	Left and right sides of the face
Rotation	Up to ±15°	Head tilt
Brightness and contrast (`ColorJitter`)	±30% each	Lighting and camera exposure
Random erasing	25% of images, a box covering 2–33% of the image	Hands, hair or glasses covering part of the face

Models

Each backbone is loaded with torchvision ImageNet weights, and only its final classification layer is replaced with a 7-way head:

Backbone	Replaced layer	RAF-DB head	FER2013 head
ResNet-18	`model.fc`	`Dropout(0.5)` → `Linear(512, 7)`	`Linear(512, 7)`
DenseNet-121	`model.classifier`	`Dropout(0.5)` → `Linear(1024, 7)`	`Linear(1024, 7)`
EfficientNet-B0	`model.classifier`	`Dropout(0.5)` → `Linear(1280, 7)`	`Dropout(0.2)` → `Linear(1280, 7)`

The whole network is fine-tuned; no layers are frozen.

Training recipe

Setting	RAF-DB notebooks	FER2013 notebooks (`Image prediction/`)
Optimizer	AdamW, lr 1e-4, weight decay 0.01	AdamW, lr 1e-4, weight decay 0.01
LR schedule	OneCycleLR, max lr 1e-3	OneCycleLR, max lr 1e-3
Loss	Cross-entropy, label smoothing 0.1	Cross-entropy
Batch size	64	64
Epochs	20 (12 for the models in `aug12_weights/` and `baseline_weights/`)	15
Random seed	42 (`torch.manual_seed`)	Not set
Mixed precision	On CUDA only (float32 on `mps`/CPU)	CUDA AMP

The scheduler only steps when the mixed-precision grad scaler actually applied an optimizer step, which keeps OneCycleLR in sync when AMP skips a step.

The root notebooks resnet_model.ipynb and densenet_model.ipynb are the original FER2013 baselines: SGD (lr 1e-3, momentum 0.9), batch size 32, 10 epochs, no scheduler.

Denoising

Denoising/denoising.ipynb adds noise to RAF-DB and trains the denoiser:

Noise model: additive white Gaussian noise, $y = \mathrm{clip}(x + n, 0, 1)$ with $n \sim \mathcal{N}(0, \sigma^2)$, independent for every pixel and RGB channel. Noise is added on the fly, so the dataset on disk stays clean.
Training noise: a random σ from 0 to 55 for every patch, so one blind model handles all noise levels.
Test noise: σ = 15, 25 and 50 with a fixed random seed, so every model sees the same noisy test set.
Model: DnCNN (Zhang et al., IEEE TIP 2017): 20 convolution layers with 64 3×3 filters and ReLU, with Batch Normalization merged into the convolutions in the released weights (668,227 parameters). It uses residual learning: the network predicts the noise map, which is subtracted from the noisy input.
Weights: starts from the authors' color blind model dncnn_color_blind.pth from KAIR (downloaded automatically), then fine-tunes on 40×40 RAF-DB face patches for 8 epochs (Adam, learning rate 1e-5 with a 50-step warmup and cosine decay, 7 minutes on an Apple M4). A learning rate of 1e-4 destroyed the pretrained weights within a few steps.

Conditional GAN

GAN/gan.ipynb trains a class-conditional GAN on the RAF-DB training faces, resized to 64×64:

Generator (DCGAN): a 128-dimensional random vector $z$ goes through five transposed convolutions, 4×4×512 → 8×8×256 → 16×16×128 → 32×32×64 → a 64×64 RGB face (3.8M parameters). The emotion label enters every layer through conditional BatchNorm: each emotion has its own BatchNorm scale and shift. Blending two emotions' scale and shift morphs a face smoothly from one emotion to the other.
Discriminator (DCGAN): four strided convolutions with BatchNorm and LeakyReLU, and a projection output (Miyato & Koyama, 2018) that scores both realism and whether the face matches its label (2.8M parameters).
Training: binary cross-entropy with the non-saturating generator loss, one-sided label smoothing (real faces → 0.9), DCGAN initialisation, Adam (learning rate 2e-4, $\beta = (0.5, 0.999)$), batch size 64, 10,000 iterations with random horizontal flips. Every 1,000 iterations the averaged generator is scored with the Fréchet distance to 2,000 training faces (the test set is not used), and the best checkpoint is kept. A first version with residual blocks, spectral normalization, hinge loss and DiffAugment never learned on this budget: its discriminator's output swung by about ±10 for real and generated faces alike after each update.
Sampling: an exponential moving average of the generator weights (decay 0.999) is saved, with its BatchNorm statistics recomputed from its own outputs.
Evaluation: the three RAF-DB classifiers judge whether each generated face shows the requested emotion, compared with real test faces downscaled to 64×64. A Fréchet distance in EfficientNet-B0 feature space measures realism and diversity, and a nearest-neighbour search checks that the GAN does not copy training images.

Real-time video pipeline

flowchart LR
    A["Webcam or video frame"] --> B["Haar cascade face detection<br/>(on grayscale frame)"]
    B --> C["Pad face box<br/>and crop color region"]
    C --> D["Resize to 224x224<br/>ImageNet normalization"]
    D --> E["CNN: ResNet-18 /<br/>DenseNet-121 / EfficientNet-B0"]
    E --> F["Softmax: emotion<br/>and confidence"]
    F --> G["Majority vote over<br/>last 10 predictions"]
    G --> H["Draw box, label,<br/>confidence and FPS"]

Detect: OpenCV's haarcascade_frontalface_default.xml (scaleFactor=1.3, minNeighbors=5) finds faces in a grayscale copy of the frame.
Crop: each face box is padded (5% for DenseNet-121 and EfficientNet-B0, 20% for ResNet-18), clipped to the frame and cropped from the color frame.
Classify: the crop is converted BGR→RGB, preprocessed like the test set and passed through the network; softmax gives the class probabilities.
Smooth: the most frequent emotion among the last 10 predictions is displayed, which removes frame-to-frame flicker.
Display: a bounding box, the smoothed emotion, the current frame's confidence and an FPS counter are drawn on the frame.

Getting Started

Requirements

Python 3.10 or newer with pip
An NVIDIA GPU (CUDA), an Apple Silicon Mac (mps) or a CPU. Inference runs fine on a CPU; a GPU is strongly recommended for training
A webcam for live video inference
A modern browser (Chrome, Safari, Firefox or Edge) for the web UI
Disk space for the Python packages (about 1 GB), the datasets (RAF-DB is about 60 MB) and the model weights (about 90 MB for the three RAF-DB models)

Tested with Python 3.14, PyTorch 2.14, torchvision 0.29 and OpenCV 4.14 on macOS (Apple M4). The original notebooks were run on Kaggle.

Installation

From the submission zip: unzip it and open a terminal in the FacialNet folder. The zip contains the trained weights, so you can skip straight to the web UI after installing the packages.

From GitHub: the weights are not in the repository (see Model Weights).

git clone https://github.com/SabariLRM/FacialNet.git
cd FacialNet

Then create a virtual environment and install the dependencies:

python3 -m venv .venv
source .venv/bin/activate        # Windows: .venv\Scripts\activate
pip install -r requirements.txt

Check that PyTorch sees your accelerator (prints cuda, mps or cpu):

python -c "import torch; print('cuda' if torch.cuda.is_available() else 'mps' if torch.backends.mps.is_available() else 'cpu')"

All commands below assume the virtual environment is active and the terminal is in the FacialNet folder.

opencv-python is pinned below 5.0 because OpenCV 5 removed cv2.CascadeClassifier, which the video scripts use for face detection.

Download the datasets

The datasets are not committed to the repository; /data is gitignored.

RAF-DB (needed for the Video Predicition/ notebooks) goes into data/raf-db:

python -c "import kagglehub, shutil; p = kagglehub.dataset_download('shuvoalok/raf-db-dataset'); shutil.copytree(p + '/DATASET', 'data/raf-db')"

FER2013 (needed for the Image prediction/ notebooks) is available on Kaggle as msambare/fer2013. It contains train/ and test/ folders. On Kaggle, attach the dataset to the notebook; to run locally, download it and point data_dir at the folder.

Training

RAF-DB models (ResNet-18, DenseNet-121, EfficientNet-B0)

The three notebooks in Video Predicition/ share the same setup code. They pick CUDA, mps or CPU automatically, and read data from Kaggle's input folder when it exists, otherwise from ../data/raf-db.

Open a notebook in Jupyter and run all cells, or execute it from the command line:

cd "Video Predicition"
jupyter nbconvert --to notebook --execute --inplace efficientnet.ipynb --ExecutePreprocessor.timeout=-1

Replace efficientnet.ipynb with resnet-model.ipynb or densenet.ipynb for the other models. Each notebook:

Loads RAF-DB and prints the detected classes and image counts.
Builds the pretrained backbone with a new 7-way head.
Trains for 20 epochs, printing train/test loss, accuracy and epoch time.
Plots loss and accuracy curves and saves the weights next to the notebook (for example efficientnet_b0_rafdb_final.pth).
Prints overall and per-class test accuracy.

DnCNN denoiser

Run Denoising/denoising.ipynb after training the three RAF-DB classifiers (it loads their weights from Video Predicition/ for the accuracy evaluation):

cd Denoising
jupyter nbconvert --to notebook --execute --inplace denoising.ipynb --ExecutePreprocessor.timeout=-1

It downloads the pretrained DnCNN, fine-tunes it, and saves dncnn_rafdb_finetuned.pth, results.json and the figures noise_levels.png, denoising_examples.png and fer_accuracy_vs_noise.png. Set max_train_batches and max_test_batches in the data cell to small numbers for a quick test run.

Conditional GAN

Run GAN/gan.ipynb after training the three RAF-DB classifiers (it uses them to evaluate the generated faces):

cd GAN
jupyter nbconvert --to notebook --execute --inplace gan.ipynb --ExecutePreprocessor.timeout=-1

Training took 63 minutes on an Apple M4 MacBook Air (mps), including the checkpoint scoring. A web UI server that is already running loads the new generator on its next request. The notebook saves cgan_rafdb_generator.pth (the generator used by the web UI), results.json and the figures gan_samples.png, gan_real_vs_generated.png, gan_progress.png, gan_emotion_transfer.png, gan_interpolation.png, gan_classifier_check.png, gan_nearest_neighbours.png and gan_training_curves.png. The distance was still falling slightly at 10,000 iterations, so a longer run may help a little.

FER2013 models

Image prediction/resnet-model.ipynb and Image prediction/densenet.ipynb train on FER2013. They were written for Kaggle GPU notebooks: data_dir points at /kaggle/input/datasets/msambare/fer2013, and training uses CUDA mixed precision. Running them elsewhere needs a CUDA GPU and an updated data_dir.

Inference

For the browser app, see Web UI. The scripts below are the original OpenCV and matplotlib tools.

Live webcam or video file

Run the video scripts from inside Video Predicition/, because they load the weights from the current directory:

cd "Video Predicition"
python video_inference_efficientnet.py

Press q to quit. On macOS, allow camera access for your terminal the first time.

The EfficientNet script also accepts a video file, and --denoise cleans each detected face with the fine-tuned DnCNN before classifying it (this roughly halves the frame rate):

python video_inference_efficientnet.py path/to/video.mp4
python video_inference_efficientnet.py --denoise

On a test video with σ = 25 noise, the plain model mislabeled the faces at low confidence, while --denoise gave the same emotions as on the clean video.

Script	Weights	Devices	Face padding
`video_inference_efficientnet.py`	`efficientnet_b0_rafdb_final.pth`	CUDA, `mps`, CPU	5%
`video_inference_densnet.py`	`densenet121_rafdb_final.pth`	CUDA, CPU	5%
`video_inference_resnet.py`	`resnet18_rafdb_final.pth`	CUDA, CPU	20%

For video_inference_resnet.py and video_inference_densnet.py, change VIDEO_SOURCE = 0 in the script to a file path to use a video file.

Single image (FER2013 models)

The scripts in Image prediction/ load FER2013 weights, predict the emotion of one image and show it with matplotlib. Put the weights file and your image in Image prediction/, change the file name in the last line of the script (predict('image copy.png') or predict_emotion('image copy.png')), then run:

cd "Image prediction"
python resnet.py

Web UI

webui/ is a local browser app for the three RAF-DB models. It is a single Python file (webui/server.py) built on the standard library, plus a static HTML/CSS/JavaScript front end, so it needs no packages beyond requirements.txt and no build step.

Running it

python webui/server.py

Wait for FacialNet web UI running at http://localhost:8000, then open that address in a browser. Stop the server with Ctrl+C.

Option	Default	Description
`--port`	`8000`	Port to listen on; change it if 8000 is taken
`--host`	`127.0.0.1`	Use `0.0.0.0` to open the UI from another device on your network

Browsers only allow camera access on http://localhost or HTTPS. If you open the UI from another device with --host 0.0.0.0, the Image and Benchmarks tabs work, but the webcam does not.

The first prediction with each model takes a few seconds while its weights load; later frames take about 30–70 ms on an Apple M4.

Tabs

Live analyses your webcam or a local video file in real time.

Click Start camera (allow access when the browser asks) or Open video file.
Every face gets a colored box with its emotion and confidence, and a thin strip under it showing all seven probabilities.
The side panel shows the dominant emotion of the largest face and its probability bars.
The emotion timeline stacks the probabilities of the last 60 seconds, with the share of time spent in each emotion underneath.
Pause, Stop, Snapshot (saves the frame with its overlay as a JPEG), a camera picker when several cameras are connected, and a Mirror toggle.
The HUD shows frames per second, server latency and the number of faces.

Image classifies a single picture.

Drag and drop, paste (Cmd/Ctrl+V) or upload a photo.
Random sample loads a random RAF-DB test face; the colored chips load a test face of a specific emotion. The result line shows the true label and whether the prediction matches it. This needs the RAF-DB test set in data/raf-db/test (see Download the datasets).
Each face gets a result card with the probability bars. With Compare all three models on, the card shows each model's full probability distribution and whether they agree.
With noise or the denoiser on, the card shows every stage: input face → noisy face → DnCNN output.

GAN generates faces with the conditional GAN (needs GAN/cgan_rafdb_generator.pth).

Pick an emotion (or All emotions), the number of faces and a seed, or click New faces. The same seed gives the same people, so switching the emotion keeps the faces and changes their expressions.
Under each face, a ✓ means the model selected in the sidebar recognises the requested emotion; ✗ shows what it sees instead. Click a face to open it in the Image tab with the full analysis.
Emotion morph blends two emotions for one face; drag the slider to watch the face and the classifier's probabilities change.
One face, every emotion renders one random face with each of the seven labels.
The charts at the bottom show the classifier agreement and Fréchet distance from GAN/results.json.

Benchmarks shows interactive charts of the results in this README and Denoising/results.json: model summary cards, per-class accuracy, robustness gained from augmentation, accuracy under Gaussian noise with and without DnCNN, denoising PSNR/SSIM, and the class balance of the training set. Hover over a chart for exact values.

Settings sidebar

Setting	What it does
Model	EfficientNet-B0, DenseNet-121 or ResNet-18; the number is its test accuracy
Weights	Current (20 epochs) from `Video Predicition/`, Full aug, 12 epochs from `aug12_weights/` or Flip + rotation, 12 epochs from `baseline_weights/`. Sets whose files are missing are hidden
Compare all three models	Image tab only: runs every model on each face
Face detection	Auto detects faces and falls back to the whole image when none are found; Detect only classifies detected faces; Off treats the whole image as one face (best for tight face crops such as RAF-DB images)
Box padding	Extra margin around each detected face, 0–30% (5% matches the video scripts)
Add Gaussian noise	Adds noise with σ = 0–75 (0–255 scale) to each face before classification, like the denoising study
DnCNN denoiser	Cleans each face with the fine-tuned DnCNN before classification (needs `Denoising/dncnn_rafdb_finetuned.pth`)
Temporal smoothing	Live tab only: blends each face's probabilities with its previous frames to remove flicker (0% = raw per-frame output)

The moon icon in the top bar switches between dark and light themes.

How it works

flowchart LR
    A["Browser<br/>(webcam frame or image)"] -- "JPEG, POST /api/predict" --> B["Haar cascade<br/>face detection"]
    B --> C["Pad and crop<br/>each face"]
    C --> D["Optional: Gaussian noise,<br/>then DnCNN"]
    D --> E["ResNet-18 / DenseNet-121 /<br/>EfficientNet-B0"]
    E -- "JSON: boxes and 7 probabilities" --> F["Browser draws boxes, bars,<br/>timeline; smooths over time"]

The browser scales each webcam frame to at most 640 px wide and sends it as a JPEG. It sends the next frame as soon as the previous answer arrives, so the frame rate adapts to the speed of your machine.
The server uses the same face detector, preprocessing (224×224, ImageNet normalization), model heads and RAF-DB label order as the video scripts. It picks CUDA, mps or CPU automatically and keeps every loaded model in memory.
In the Live tab, faces are matched between frames by box overlap, so each face keeps its own smoothed probabilities.
Nothing leaves your computer: the server only listens on localhost unless you pass --host.

API

The front end uses four JSON endpoints, which can also be called directly:

Endpoint	Description
`GET /api/info`	Device, class names, available models and weight sets, and `Denoising/results.json`
`GET /api/sample?label=3`	A random RAF-DB test image (as a data URL) with its label index; `label` (0–6) is optional
`GET /api/generate?mode=label&label=3&n=8&seed=42`	Faces generated by the conditional GAN, each classified by `model` (default `efficientnet`)
`POST /api/predict?models=efficientnet&weights=current`	Classifies the faces in the image sent as the request body (JPEG or PNG, up to 25 MB)

Query parameters of /api/predict: models (comma-separated efficientnet, densenet, resnet), weights (current, aug12, baseline), detect (auto, on, off), pad (0–0.5), noise (σ, 0–100), denoise (1 or 0), crops (1 to return the face images of each stage) and max_faces (default 6).

curl -s -X POST --data-binary @face.jpg "http://localhost:8000/api/predict?models=efficientnet,densenet&detect=auto"

The response lists every face with its box [x, y, w, h] in image pixels and, for each model, the seven probabilities in the order Surprise, Fear, Disgust, Happy, Sad, Anger, Neutral, the top label and its confidence.

Query parameters of /api/generate: mode (label: n faces of emotion label, 0–6 or all; emotions: one face with each of the seven emotions; morph: one face blended from emotion from to emotion to in steps steps), seed (the same seed gives the same faces), model and weights (the classifier that checks each face). Every face in the response has its image (a 128×128 JPEG data URL, upscaled from 64×64), the requested emotion mix, and the classifier's seven probabilities, top label and confidence.

Troubleshooting

Problem	Fix
`Address already in use`	Another program uses port 8000: run `python webui/server.py --port 8001`
`FileNotFoundError` for a `.pth` file on start	The weights are missing; see Model Weights
The top bar says server offline	The server stopped or the page was opened from a file; start the server and open `http://localhost:8000`
The camera does not start	Allow camera access for the browser (on macOS: System Settings → Privacy & Security → Camera) and make sure no other app is using it
No face is detected	The Haar detector needs a roughly frontal, well-lit face; for tight face crops set face detection to Auto or Off
Random sample buttons are missing	Download RAF-DB into `data/raf-db`
The GAN tab says the weights are missing	Run `GAN/gan.ipynb` to create `GAN/cgan_rafdb_generator.pth`
DnCNN switch is disabled	`Denoising/dncnn_rafdb_finetuned.pth` is missing

Repository Structure

FacialNet/
├── .github/workflows/
│   └── ci_pipeline.yml               # Lint and notebook conversion checks on push / PR
├── Image prediction/                 # FER2013 (grayscale) models
│   ├── resnet-model.ipynb            # ResNet-18 training on Kaggle (15 epochs, AMP)
│   ├── densenet.ipynb                # DenseNet-121 training on Kaggle (15 epochs, AMP)
│   ├── resnet.py                     # Single-image prediction with ResNet-18
│   ├── densenet.py                   # Single-image prediction with DenseNet-121
│   └── efficientnet.py               # Single-image prediction with EfficientNet-B0
├── Video Predicition/                # RAF-DB (color) models and real-time inference
│   ├── resnet-model.ipynb            # ResNet-18 training
│   ├── densenet.ipynb                # DenseNet-121 training
│   ├── efficientnet.ipynb            # EfficientNet-B0 training
│   ├── video_inference_resnet.py     # Webcam / video emotion recognition, ResNet-18
│   ├── video_inference_densnet.py    # Webcam / video emotion recognition, DenseNet-121
│   ├── video_inference_efficientnet.py  # Webcam / video emotion recognition, EfficientNet-B0 (optional --denoise)
│   ├── *_rafdb_final.pth             # Current weights, 20 epochs (gitignored)
│   ├── aug12_weights/                # Full augmentation, 12 epochs (gitignored)
│   └── baseline_weights/             # Flip + rotation only, 12 epochs (gitignored)
├── Denoising/                        # Noise robustness study
│   ├── denoising.ipynb               # Gaussian noise, DnCNN fine-tuning and evaluation
│   ├── results.json                  # PSNR/SSIM and accuracy numbers
│   └── *.png                         # Noise levels, denoising examples, accuracy vs noise
├── GAN/                              # Conditional GAN that generates emotional faces
│   ├── gan.ipynb                     # Training, generated faces and evaluation
│   ├── results.json                  # Classifier agreement, Fréchet distance, training log
│   └── *.png                         # Samples, morphs, training progress, evaluation figures
├── webui/                            # Browser app (see Web UI)
│   ├── server.py                     # HTTP server, face detection and model inference
│   └── static/                       # index.html, style.css, app.js
├── resnet_model.ipynb                # Original FER2013 baseline, ResNet-18 (SGD)
├── densenet_model.ipynb              # Original FER2013 baseline, DenseNet-121 (SGD)
├── data/                             # Datasets, created locally (gitignored)
├── requirements.txt                  # Python dependencies
└── .gitignore                        # Ignores /data, *.pth weights and .venv/

Model Weights

Trained weights (*.pth) are gitignored and not published in the repository. The submission zip includes the files needed to run the web UI and the video scripts: the three current RAF-DB models and dncnn_rafdb_finetuned.pth. Otherwise, train them with the notebooks or place previously trained files in the folder shown.

File	Folder	Produced by	Used by
`resnet18_rafdb_final.pth`	`Video Predicition/`	`resnet-model.ipynb`	`video_inference_resnet.py`, web UI
`densenet121_rafdb_final.pth`	`Video Predicition/`	`densenet.ipynb`	`video_inference_densnet.py`, web UI
`efficientnet_b0_rafdb_final.pth`	`Video Predicition/`	`efficientnet.ipynb`	`video_inference_efficientnet.py`, web UI
Same three file names	`Video Predicition/aug12_weights/`, `Video Predicition/baseline_weights/`	Earlier 12-epoch runs of the notebooks	Web UI (optional weight sets)
`dncnn_color_blind.pth`	`Denoising/`	Downloaded from KAIR by `denoising.ipynb`	`denoising.ipynb`
`dncnn_rafdb_finetuned.pth`	`Denoising/`	`denoising.ipynb`	`video_inference_efficientnet.py --denoise`, web UI
`cgan_rafdb_generator.pth`	`GAN/`	`gan.ipynb`	Web UI (GAN tab)
`resnet18_fer2013_optimized.pth`	`Image prediction/`	`resnet-model.ipynb`	`resnet.py`
`densenet121_fer2013.pth`	`Image prediction/`	`densenet.ipynb`	`densenet.py` (expects `densenet121_fer2013(1).pth`)
`efficientnet_b0_fer2013_amp.pth`	`Image prediction/`	Not included in this repository	`efficientnet.py`

The FER2013 DenseNet-121 and EfficientNet-B0 scripts strip the _orig_mod. prefix from checkpoint keys, so they also load weights saved from a torch.compiled model.

Continuous Integration

.github/workflows/ci_pipeline.yml runs on every push and pull request to main:

Sets up Python 3.9 and installs requirements.txt.
Converts resnet_model.ipynb and densenet_model.ipynb to Python scripts with nbconvert.
Runs flake8. The build fails on syntax errors and undefined names (E9, F63, F7, F82); all other style issues are reported as warnings.

CI checks that the code is valid Python; it does not train or evaluate models.

Limitations and Known Issues

Limitations

Class imbalance: RAF-DB has 17× more Happy than Fear training images, and Fear and Disgust are the weakest classes for every model. Class-weighted loss or oversampling would likely help.
No separate validation split: the test set is evaluated after every epoch. Reported numbers are final-epoch results, not the best epoch, but there is no held-out validation set.
Face detection: the Haar cascade only finds roughly frontal faces and is sensitive to lighting, head pose and occlusion. When it misses a face, no emotion is shown.
Crop mismatch: RAF-DB faces are tightly aligned, while webcam crops come from a detector box. More padding adds background and lowers accuracy. Random zoom during training reduces the effect of tighter crops (see Effect of data augmentation), but looser crops with extra background were not tested.
Synthetic noise only: the denoiser is trained and evaluated on additive Gaussian noise. Real camera noise also includes compression artifacts, blur and signal-dependent noise, which were not tested.
GAN quality: the GAN makes 64×64 faces and was trained for 10,000 iterations, so its faces are still blurry and sometimes distorted, and Fear and Disgust are recognised in only 17–40% of generated faces. It follows the RAF-DB class balance, so it has seen 17× more Happy than Fear faces. Its Fréchet distance uses RAF-DB EfficientNet-B0 features and is not comparable with FID values in papers.
Confidence display: the label shown in the video scripts is smoothed over 10 frames, but the percentage next to it is the current frame's confidence.

Known issues

FER2013 label order in Image prediction/: resnet.py, densenet.py, efficientnet.py and the visualization cell of densenet.ipynb use ['Angry', 'Disgust', 'Fear', 'Happy', 'Sad', 'Surprise', 'Neutral']. ImageFolder sorts the FER2013 folders alphabetically (angry, disgust, fear, happy, neutral, sad, surprise), so Neutral, Sad and Surprise predictions are displayed under the wrong names. The list should follow the alphabetical order.
Weights file name: Image prediction/densenet.py loads densenet121_fer2013(1).pth, but Image prediction/densenet.ipynb saves densenet121_fer2013.pth.
Missing notebook: there is no training notebook for the FER2013 EfficientNet-B0 model that Image prediction/efficientnet.py loads.
Video script devices: video_inference_resnet.py and video_inference_densnet.py use CUDA or CPU only, so they run on the CPU on Apple Silicon.
ResNet video script: video_inference_resnet.py labels its predictions "DenseNet" and uses 20% face padding, which reduced accuracy noticeably in testing compared with 5%.
Root baseline notebooks: densenet_model.ipynb saves its weights as resnet18_fer2013.pth, which is the same file name resnet_model.ipynb uses.

Datasets and Acknowledgements

RAF-DB: Li, S., Deng, W. and Du, J. Reliable Crowdsourcing and Deep Locality-Preserving Learning for Expression Recognition in the Wild. CVPR 2017. RAF-DB is released for non-commercial research purposes only. This project uses the Kaggle mirror shuvoalok/raf-db-dataset.
FER2013: Goodfellow, I. J. et al. Challenges in Representation Learning: A report on three machine learning contests. ICML 2013 Workshop. This project uses the Kaggle version msambare/fer2013.
Pretrained backbones: ImageNet weights from torchvision.
DnCNN: Zhang, K., Zuo, W., Chen, Y., Meng, D. and Zhang, L. Beyond a Gaussian Denoiser: Residual Learning of Deep CNN for Image Denoising. IEEE Transactions on Image Processing, 2017. Official code: cszn/DnCNN; PyTorch implementation and pretrained weights: cszn/KAIR.
Face detection: OpenCV Haar cascade classifier.
GAN: Goodfellow, I. et al. Generative Adversarial Nets. NeurIPS 2014. Radford, A., Metz, L. and Chintala, S. Unsupervised Representation Learning with Deep Convolutional GANs (DCGAN). ICLR 2016. Miyato, T. and Koyama, M. cGANs with Projection Discriminator. ICLR 2018. Brock, A., Donahue, J. and Simonyan, K. Large Scale GAN Training for High Fidelity Natural Image Synthesis (BigGAN, conditional BatchNorm in the generator). ICLR 2019.

Based on the original FacialNet project by Sanjjiiev S (sanjjiiev/FacialNet). The repository does not include a license file yet.

Name		Name	Last commit message	Last commit date
.github/workflows		.github/workflows
.ipynb_checkpoints		.ipynb_checkpoints
.vscode		.vscode
Denoising		Denoising
GAN		GAN
Image prediction		Image prediction
Video Predicition		Video Predicition
webui		webui
.gitignore		.gitignore
README.md		README.md
densenet_model.ipynb		densenet_model.ipynb
requirements.txt		requirements.txt
resnet_model.ipynb		resnet_model.ipynb

Latest commit

History

Folders and files

Repository files navigation

FacialNet

Contents

Quick Start

Highlights

Results

RAF-DB (color, 7 classes, 3,068 test images)

FER2013 (grayscale, 7 classes)

Effect of data augmentation (RAF-DB)

Noise robustness with DnCNN (RAF-DB)

Conditional GAN (RAF-DB, 64×64)

How It Works

Datasets

Preprocessing and augmentation

Models

Training recipe

Denoising

Conditional GAN

Real-time video pipeline

Getting Started

Requirements

Installation

Download the datasets

Training

RAF-DB models (ResNet-18, DenseNet-121, EfficientNet-B0)

DnCNN denoiser

Conditional GAN

FER2013 models

Inference

Live webcam or video file

Single image (FER2013 models)

Web UI

Running it

Tabs

Settings sidebar

How it works

API

Troubleshooting

Repository Structure

Model Weights

Continuous Integration

Limitations and Known Issues

Limitations

Known issues

Datasets and Acknowledgements

About

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages