Pill OCR App

A Flutter-based mobile OCR application optimized for reading pill bottle labels on Android tablets. Uses 100% native Paddle Lite C++ inference with no Python runtime dependencies.

Features

• Native OCR Engine: Paddle Lite C++ (PP-OCRv4/v5) optimized for ARM64
• Drug Name Recognition: 20,070 drugs with 108,468 synonyms using fuzzy matching
• Multi-Image Stitching: Combine multiple images for better accuracy
• Field Extraction: Automatically extracts drug name, dosage, RX number, directions, etc.
• Offline Operation: All processing happens on-device

Architecture

Flutter UI (Dart)
    ↓
Kotlin/JNI Bridge
    ↓
C++ OCR Engine (ocr_bridge.cpp)
    ↓
Paddle Lite ARM64 + OpenCV

Runtime: 100% Python-free native execution Detailed docs: See ARCHITECTURE.md

Technology Stack

Core Components

Layer	Technology	Purpose
UI	Flutter/Dart	Cross-platform mobile UI
Drug Matching	Dart	Drug name extraction (20K drugs)
Platform Bridge	Kotlin + JNI	Android integration
OCR Engine	C++	Native inference orchestration
ML Framework	Paddle Lite v2.10	ARM64 optimized OCR models
Image Processing	OpenCV 4.1.0	Preprocessing & stitching

OCR Models

• Detection: PP-OCRv4 mobile (detects text regions)
• Recognition: PP-OCRv4 English mobile (extracts text)
• Classification: PP-OCRv2 (handles rotated text)
• Format: Paddle Lite NaiveBuffer (.nb) - optimized for mobile

Performance

Device: Samsung Galaxy Tab A9+ (SM-X210) Inference Time: ~500-1000ms per image APK Size: +55 MB (native libraries + models) Memory: ~50-100 MB peak during inference

Getting Started

Prerequisites

• Flutter SDK 3.35.6+
• Android NDK r20.1.5948944 (critical - see note below)
• Android Studio with CMake
• Paddle Lite library (see setup below)

NDK Version Requirement

You MUST use NDK r20.1.5948944. The Paddle Lite library was compiled with this NDK version.

Using NDK 27 or other versions will cause linker errors:

ld.lld: error: invalid local symbol '__bss_start' in global part of symbol table

Check your NDK version:

cat ~/Library/Android/sdk/ndk/YOUR_NDK_DIR/source.properties | grep "Pkg.Revision"

Update if needed:

// android/app/build.gradle.kts
android {
    ndkVersion = "20.1.5948944"
}

# android/local.properties
ndk.dir=/path/to/Android/sdk/ndk/20.1.5948944

Paddle Lite Setup

1. Download Paddle Lite v2.10 for Android ARM64 with OpenCV:

   # Visit https://github.com/PaddlePaddle/Paddle-Lite/releases
   # Download: inference_lite_lib.android.armv8.gcc.c++_shared.with_extra.with_cv

2. Update CMakeLists.txt with your local path:

   # android/app/src/main/cpp/CMakeLists.txt
   set(PADDLE_LITE_DIR "/your/path/to/inference_lite_lib.android.armv8.gcc.c++_shared.with_extra.with_cv")

Installation

1. Clone the repository:

   git clone <repository-url>
   cd pill_ocr_app

2. Install Flutter dependencies:

   flutter pub get

3. Build and run:

   flutter run

   Or build APK:

   flutter build apk

Project Structure

pill_ocr_app/
├── lib/
│   ├── main.dart                    # App entry point
│   ├── screens/                     # UI screens
│   └── core/
│       ├── ocr_channel.dart         # Platform channel bridge
│       ├── drug_matcher.dart        # Drug name matching (Dart)
│       ├── field_extractor.dart     # Field extraction logic
│       └── models/
│           ├── ocr_result.dart
│           └── drug.dart
├── android/
│   └── app/
│       ├── build.gradle.kts         # Gradle build config
│       ├── src/main/
│       │   ├── kotlin/.../MainActivity.kt  # Kotlin JNI bridge
│       │   ├── assets/
│       │   │   ├── models/          # OCR models (.nb files)
│       │   │   ├── config.txt       # OCR parameters
│       │   │   └── en_dict.txt      # Character dictionary
│       │   └── cpp/                 # Native C++ code
│       │       ├── CMakeLists.txt
│       │       ├── ocr_bridge.cpp   # JNI implementation
│       │       ├── ocr_bridge.h
│       │       ├── image_stitcher.cpp
│       │       └── paddle_ocr/      # PaddleOCR processing
├── assets/
│   └── drugs.json                   # Drug database (8.5 MB)
├── dev_tools/                       # Optional Python utilities
│   ├── preprocessing_pipeline.py
│   ├── optimize_ocr_params.py
│   ├── pharmaceutical_corrections.py
│   └── test_stitching.py
├── ARCHITECTURE.md                  # Detailed architecture docs
├── DEV_TOOLS.md                     # Development utilities guide
├── OPTIMIZATION_GUIDE.md            # OCR tuning guide
└── TESTING.md                       # Testing procedures

Development Tools

The dev_tools/ directory contains optional Python utilities for development and testing. These are NOT required for the app to run.

See DEV_TOOLS.md for details on:

• Image preprocessing utilities
• OCR parameter optimization
• Pharmaceutical text corrections
• Image stitching tests

Usage

1. Single Image OCR:

   • Open app
   • Tap camera button or select from gallery
   • Wait for OCR processing (~1 second)
   • View extracted fields

2. Multi-Image OCR:

   • Take 2-5 photos of the same label from different angles
   • App automatically stitches images
   • Better accuracy for curved labels

Configuration

OCR Parameters

Edit android/app/src/main/assets/config.txt:

# Detection
det_db_thresh 0.25              # Detection threshold (0.2-0.3)
det_db_unclip_ratio 1.6         # Text box expansion (1.5-2.0)
det_limit_side_len 1280         # Max image side length

# Recognition
rec_image_height 48             # Recognition height (32 or 48)
use_direction_classify 1        # Enable text rotation detection

After editing: Increment ASSETS_VERSION in MainActivity.kt to force asset recopy.

Model Version

Switch between PP-OCRv4 and PP-OCRv5:

// MainActivity.kt
private val MODEL_VERSION = "v4"  // or "v5"

Drug Database

Location: assets/drugs.json Size: 8.5 MB Drugs: 20,070 with 108,468 unique synonyms Source: drug_named_entity_recognition Python library

Update database:

1. Export new drug data to JSON
2. Replace assets/drugs.json
3. Rebuild app (no native code changes needed)

Troubleshooting

Build Issues

"Library not found: libpaddle_light_api_shared.so"

• Update PADDLE_LITE_DIR in CMakeLists.txt

"NDK linker errors: invalid symbol table"

• Use NDK r20.1.5948944 exactly
• Check build.gradle.kts and local.properties

"OCR initialization failed"

• Check model files exist in assets/models/
• Verify ASSETS_VERSION incremented after asset changes

Runtime Issues

"No text detected"

• Ensure good lighting and focus
• Try multiple angles
• Adjust det_db_thresh in config.txt

"Low confidence results"

• Use image preprocessing (see dev_tools)
• Increase image resolution
• Try multi-image mode

Testing

# Run Dart tests
flutter test

# Run drug matcher tests
flutter test test/core/drug_matcher_test.dart

# Build and test on device
flutter run --release

See TESTING.md for comprehensive testing guide.

Performance Optimization

See OPTIMIZATION_GUIDE.md for detailed tuning strategies:

• Parameter optimization
• Image preprocessing
• Model selection
• Hardware acceleration

Contributing

1. Fork the repository
2. Create a feature branch
3. Make changes
4. Test thoroughly on real devices
5. Submit pull request

License

[Your license here]

Acknowledgments

• PaddleOCR - OCR models
• Paddle Lite - Mobile inference engine
• drug_named_entity_recognition - Drug database
• OpenCV - Image processing

Support

For issues or questions:

• Check ARCHITECTURE.md for technical details
• Review TROUBLESHOOTING.md for common problems
• Open an issue on GitHub

---

Note: This app processes all data on-device. No data is sent to external servers.