A real-time Indian Sign Language interpreter that recognizes hand gestures from a phone camera and converts them into text and speech. A Flutter app captures the gesture, a Python inference server extracts hand landmarks with MediaPipe and classifies them with trained deep-learning models, and the recognized sign is displayed and spoken aloud via TTS.
Academic mini-project (2025) — full write-up in
docs/Report_Mini_Project.pdf.
Flutter app — capture a sign, get it back as text and speech.
The app talks to the inference server over HTTP: single frames go to /predict
(MobileViT), short clips go to /predict-video (LSTM). The server extracts
MediaPipe landmarks, classifies them, and returns the predicted gesture.
- Static path — a single frame → hand landmarks → MobileViT → letter/number.
- Dynamic path — a short clip → 30 sampled frames of landmarks → LSTM → word.
| Task | Model | Classes | Accuracy |
|---|---|---|---|
| Static signs | MobileViT (on hand landmarks) | 35 — A–Z, 1–9 | 98.82% |
| Dynamic signs | 2-layer LSTM (30-frame sequences) | 6 — Doctor, Help, Hot, Lose, Pain, Thief | 95.16% |
Trained on a custom-built dataset (~1,400 images + augmentation). MediaPipe extracts 21 hand landmarks (x, y, z) per hand, which are fed to the classifiers instead of raw pixels — making inference lightweight and robust to background.
| Static signs (MobileViT) | Dynamic signs (LSTM) |
|---|---|
    |
  |
| A · G · H · X | Lose · Thief |
ISL-interpreter/
├── app/ # Flutter mobile app (camera capture, HTTP, TTS)
│ ├── lib/ # main.dart, screens/home_screen.dart
│ ├── android/ # Android config (build artifacts are git-ignored)
│ └── assets/
├── server/ # Flask inference server (MediaPipe + Keras)
│ ├── app.py # /predict (static) + /predict-video (dynamic)
│ └── requirements.txt
├── notebooks/ # Model training (see docs for provenance)
│ ├── train_static_mobilevit.ipynb
│ ├── train_dynamic_lstm.ipynb
│ └── train_static_vit_prototype.ipynb
├── docs/
│ ├── images/ # diagrams + app/sign screenshots
│ ├── PROJECT_DOCUMENTATION.md # detailed design notes & API reference
│ └── Report_Mini_Project.pdf # full academic report
└── README.md
Model weights (
*.h5) are not committed — place your trained files next toserver/app.py(git-ignored) or point the server at them with theISL_STATIC_MODEL/ISL_DYNAMIC_MODELenv vars. Train your own from the notebooks innotebooks/.
Provenance. The reported metrics come from the project report (
docs/Report_Mini_Project.pdf), the record of the completed project. The original training notebooks were lost; thetrain_static_mobilevitandtrain_dynamic_lstmnotebooks are faithful reconstructions of the report's architectures (run them to reproduce), andtrain_static_vit_prototypeis the surviving earlier prototype with its real output.
cd server
python -m venv .venv
source .venv/bin/activate # Windows: .venv\Scripts\activate
pip install -r requirements.txt
python app.py # serves on http://0.0.0.0:5000The server exposes POST /predict (static signs) and POST /predict-video
(dynamic signs), each returning { gesture, confidence }, plus a /network-info
endpoint the app uses to discover the server on the local network.
cd app
flutter pub get
flutter runIn the app's settings, set the server IP so your device can reach the machine running the inference server (they must be on the same network).
| Layer | Technologies |
|---|---|
| Mobile | Flutter, Dart, camera, flutter_tts, http |
| Inference | Python, Flask, Flask-CORS |
| ML / CV | TensorFlow / Keras, MediaPipe, OpenCV, NumPy |
| Training | Jupyter, MobileViT, LSTM |
- Fixed gesture vocabulary (no custom-gesture training yet).
- Dynamic recognition uses a fixed 30-frame window.
- Requires the app and server to share a local network.
- Next: on-device TFLite inference, sentence-level recognition, larger dataset.
See docs/PROJECT_DOCUMENTATION.md for the full
API reference, model I/O shapes, and design notes.
A team academic mini-project (2025) by:
- Albrin T B
- KM Aravind Krishna
- Deva Nanda Gopi
- Goutham Krishna K S — @silverdawn-gtm
Released under the MIT License.