Multilingual Sentiment Analysis — QLoRA Fine-tuned Gemma-3 (NPPE1 DLP 2026)

Overview

Fine-tuned Google's Gemma-3 (1B-IT) model for binary sentiment classification (Positive/Negative) on multilingual text spanning 13 Indian languages, as part of an IIT Madras community prediction competition (NPPE1_DLP_2026_Term1). Used QLoRA (4-bit quantization + LoRA) for parameter-efficient fine-tuning combined with a 5-fold stratified cross-validation ensemble and OOF (out-of-fold) threshold tuning to maximize macro F1 score.

Result: Ranked #50 on the private leaderboard (jumped 98 positions from the public leaderboard) with a private score of 0.9375 and public score of 0.8718.

Problem Statement

• Binary sentiment classification (Positive / Negative) on short text sentences
• Text spans 13 Indian languages
• Dataset: 900 training samples, 100 test samples
• Evaluation metric: Macro F1

Approach

1. Base Model: Gemma-3 (1B-IT), loaded via Hugging Face Transformers as a sequence classification model (classification head on top of the base LM)
2. Quantization: 4-bit NF4 quantization via bitsandbytes (BitsAndBytesConfig) to fit training on a single Tesla T4 GPU
3. Fine-tuning: LoRA adapters (r=16, alpha=32) applied to attention and MLP projection layers (q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj), trained via peft's prepare_model_for_kbit_training
4. Cross-validation: 5-fold stratified K-fold — a fresh LoRA model trained per fold, with early stopping (patience=3) and cosine LR scheduling
5. Ensembling: Averaged test-set class probabilities across all 5 folds
6. Threshold tuning: Tuned the classification decision threshold on out-of-fold (OOF) predictions (searched 0.20–0.80) to maximize macro F1, rather than using the default 0.5 cutoff
7. Regularization: Label smoothing (0.05), weight decay (0.05), gradient clipping

Tech Stack

• Model: Google Gemma-3 (1B-IT)
• Fine-tuning: LoRA, QLoRA (4-bit NF4), PEFT
• Libraries: Hugging Face Transformers, datasets, evaluate, bitsandbytes, accelerate, scikit-learn
• Training: Hugging Face Trainer API, 5-fold StratifiedKFold cross-validation
• Compute: Tesla T4 GPU (Kaggle Notebooks)

Training Process

Sample fold training log (loss and F1 improve steadily across epochs before early stopping kicks in around epoch 7-8, avoiding overfitting):

Epoch	Training Loss	Validation Loss	F1	Accuracy
1	1.5215	0.5581	0.7235	0.7333
2	0.8025	0.4370	0.8056	0.8056
3	0.3130	0.4389	0.8317	0.8333
4	0.2961	0.4401	0.8443	0.8444
5	0.2807	0.4935	0.8444	0.8444
6	0.2388	0.4477	0.8444	0.8444
7	0.2366	0.4509	0.8388	0.8389

Results

Metric	Score
Public Leaderboard	0.8718
Private Leaderboard	0.9375
Final Rank	50 (private leaderboard, calculated on ~66% of test data)

Sample Output

Final predictions were saved as submission.csv with a near-balanced class split (52 Negative / 48 Positive on the test set):

Repository Structure

.
├── notebooks/
│   └── sentiment_analysis_qlora_ensemble.ipynb   # Main training + inference notebook
├── results/
│   ├── private_leaderboard_score.png
│   ├── training_progress_fold_metrics.png
│   └── final_predictions_submission_preview.png
├── README.md
├── requirements.txt
└── .gitignore

How to Run

This notebook was built for Kaggle's environment (GPU-enabled, with the competition dataset and Gemma-3 model pre-mounted under /kaggle/input/). To reproduce:

1. Open the notebook in a Kaggle Notebook environment with GPU (T4 or better) enabled
2. Attach the competition dataset and the Gemma-3-1B-IT model as data sources
3. Run all cells — the script installs required libraries, trains the 5-fold ensemble, and writes submission.csv

Dataset

Dataset provided as part of a private IIT Madras community prediction competition (NPPE1_DLP_2026_Term1) and is not redistributed here due to competition terms.

Notes

• The competition guidelines listed LLaMA 3.1-8B-Instruct as the suggested model; this solution instead used Gemma-3 (1B-IT), which gave better validation performance while being significantly lighter to fine-tune under the available GPU/time constraints.