A real-time sign language recognition system using YOLOv5 + CNN + OpenCV, designed to translate hand gestures into readable text.
The project helps bridge the communication gap between sign language users and verbal speakers, promoting inclusivity in healthcare, education, and public services.
| Name | Reg. No. | Contribution |
|---|---|---|
| Aayush Amritesh | 21BCE2331 | Data Preprocessing, Model Training & Testing, Evaluation |
| Dev Chandrakar | 21BCE2960 | Literature Survey, Dataset Collection, Documentation |
| Aditya Kumar Jha | 21BCE3759 | Model Selection, UI/Backend Integration |
🧑🏫 Guided by: Dr. Viswanathan A., Associate Professor (Sr.)
School of Computer Science and Engineering, VIT
Develop a real-time ASL Recognition System that:
- Detects gestures using YOLOv5
- Classifies gestures using CNN
- Translates gestures to text through a simple user interface
- Supports low‐cost consumer hardware like webcams & laptops
Over 70M people rely on sign language. Most verbal speakers don’t understand it, causing a communication barrier in critical areas such as hospitals, classrooms, and public services.
Deep learning enables accessible, affordable, and real-time solutions for gesture recognition.
Modules:
- 📷 Data Acquisition – Webcam or image input
- 🧹 Preprocessing (OpenCV) – Background removal, segmentation
- 🎯 Gesture Detection (YOLOv5) – Identify hand gesture
- 🧠 Classification (CNN) – Recognize gesture category
- 💬 Sign-to-Text Translation – Output real-time prediction
- 🖥️ User Interface – Minimal & user-friendly
| Category | Tools |
|---|---|
| Programming | Python |
| Deep Learning | YOLOv5, CNN (TensorFlow/PyTorch) |
| Image Processing | OpenCV |
| UI/Development | Jupyter/Colab/VS Code |
| Hardware | Standard webcam, i5/8GB RAM laptop |
| Feature | Value |
|---|---|
| Image Size | 224×224 |
| Classes | 6 (Hello, Yes, No, Thanks, Please, I Love You) |
| Model | YOLOv5s + Custom CNN |
| Optimizer | Adam |
| Latency | < 100ms real-time |
✔ Real-time gesture detection
✔ Works effectively in low-light & diverse backgrounds
✔ Runs smoothly on consumer-grade hardware
✔ Achieved 95% accuracy (static) & 90% dynamic in adequate lighting conditions
The system successfully bridges the communication gap by converting American Sign Language (ASL) gestures to readable text in real time using a hybrid deep learning model (YOLOv5 + CNN). It is affordable, portable, and scalable, making it suitable for deployment in schools, hospitals, and public service centers. Future work includes sentence-level recognition, multi-language support, and mobile deployment.
- 📌 Support for complete sentence-level gesture recognition
- 🌎 Extension to multiple sign languages (e.g., ISL, BSL, etc.)
- 📱 Mobile application deployment on Android/iOS
- 🎙️ Voice output for recognized text (sign-to-speech)
- 🧠 Use of Transformer-based or sequence models for better temporal gesture understanding