Dubify

AI-powered video dubbing — from YouTube URL to English-dubbed MP4 in one command.

Overview

Dubify is a fully automated multilingual video dubbing pipeline. Given a YouTube URL, it:

Downloads the video using yt-dlp
Extracts the audio track via FFmpeg
Transcribes speech in any language using Faster-Whisper (auto-detects source language)
Adapts the script into natural spoken English using Google Gemini 2.5 Flash
Synthesises English audio with Microsoft Edge TTS neural voices
Time-aligns each clip to the original speaker's timing via FFmpeg's atempo filter
Exports the finished MP4 — original video quality preserved, audio replaced

No re-encoding of the video stream. No per-segment API calls for translation. One command.

Architecture

                 +--------------------+
                 |  YouTube Video     |
                 +---------+----------+
                           |
                           v
                 +--------------------+
                 | Download (yt-dlp)  |
                 +---------+----------+
                           |
                           v
                 +--------------------+
                 | Audio Extraction   |
                 | (FFmpeg → WAV)     |
                 +---------+----------+
                           |
                           v
                 +--------------------+
                 | Faster-Whisper     |
                 | Transcription      |
                 +---------+----------+
                           |
                           v
                 +--------------------+
                 | Gemini Script      |
                 | Adaptation         |
                 +---------+----------+
                           |
                           v
                 +--------------------+
                 | Edge-TTS           |
                 | + atempo Alignment |
                 +---------+----------+
                           |
                           v
                 +--------------------+
                 | Audio Merge        |
                 | (adelay + amix)    |
                 +---------+----------+
                           |
                           v
                 +--------------------+
                 | Final Dubbed Video |
                 +--------------------+

Module responsibilities

Module	Class	Responsibility
`src/models.py`	`Segment`, `VideoProject`, `PipelineStage`	Shared data contracts — no logic
`config.py`	`Config`	All env vars in one place — injected everywhere
`src/downloader.py`	`VideoDownloader`	yt-dlp download with Rich progress bar
`src/extractor.py`	`AudioExtractor`	FFmpeg → 16 kHz mono WAV for Whisper
`src/transcriber.py`	`Transcriber`	Faster-Whisper speech-to-text, auto language detection
`src/translator.py`	`BaseTranslator`, `GeminiTranslator`	Batch dubbing script adaptation via Gemini 2.5 Flash
`src/synthesizer.py`	`BaseSynthesizer`, `EdgeTTSSynthesizer`	Edge TTS + atempo time-stretching
`src/audio_composer.py`	`AudioComposer`	adelay + amix timeline, copy-mux with video
`src/pipeline.py`	`DubbingPipeline`	Orchestration, timing, caching, progress display
`src/reporter.py`	—	Terminal summary and `processing_report.json`
`src/utils.py`	—	Logging, file helpers, cleanup
`main.py`	—	Thin CLI only — zero business logic

Features

🌐 Automatic language detection — Whisper identifies the source language without configuration
📦 Single batch translation — all segments sent to Gemini in one API call (10–50× cheaper than per-segment calls)
⚡ Concurrent TTS synthesis — up to 8 Edge-TTS requests in-flight simultaneously (semaphore-limited)
🎛️ Per-segment time alignment — each clip is speed-adjusted via atempo to match original timing
🗃️ Stage caching & resumability — skip completed stages on reruns with --resume (default)
🔁 Retry logic — transient Edge-TTS failures retried 3× with exponential back-off
📊 Processing reports — human-readable terminal summary + machine-readable processing_report.json
🎬 Lossless video quality — video stream copied with -c:v copy, only audio replaced

Installation

Prerequisites

Python 3.11+
FFmpeg (must be on PATH or configured via .env)
- Windows: winget install ffmpeg or download from ffmpeg.org
- macOS: brew install ffmpeg
- Linux: sudo apt install ffmpeg
Google Gemini API key — get one free at aistudio.google.com

Setup

# 1. Clone the repository
git clone https://github.com/your-username/Dubify.git
cd Dubify

# 2. Create and activate a virtual environment
python -m venv .venv
.venv\Scripts\activate        # Windows
# source .venv/bin/activate   # macOS / Linux

# 3. Install dependencies
pip install -r requirements.txt

# 4. Configure environment variables
cp .env.example .env
# Edit .env — at minimum set GEMINI_API_KEY

Usage

Basic

python main.py

Dubify will prompt for a YouTube URL interactively.

Pass URL directly

python main.py "https://www.youtube.com/watch?v=dQw4w9WgXcQ"

CLI flags

Flag	Description
`url`	YouTube video URL (positional, optional — prompts if omitted)
`--resume`	Use cached stage results (default behaviour)
`--force`	Ignore all caches and rerun the full pipeline
`--from STAGE`	Restart from a specific stage: `download` / `extract` / `transcribe` / `translate` / `synthesize` / `compose`

Examples

# Force full rerun, ignoring all caches
python main.py "https://youtube.com/watch?v=..." --force

# Resume from the TTS stage (re-synthesise but reuse transcription/translation)
python main.py "https://youtube.com/watch?v=..." --from synthesize

Output directories

downloads/    Downloaded source videos
temp/         Intermediate files (auto-cleaned unless KEEP_TEMP=True)
outputs/      Final dubbed videos + processing_report.json + segments.json
logs/         Daily log files  (YYYY-MM-DD.log)

Configuration

All settings are read from a .env file (copy .env.example to get started).

Variable	Default	Description
`GEMINI_API_KEY`	(required)	Google Gemini API key
`WHISPER_MODEL`	`base`	Whisper model: `tiny` / `base` / `small` / `medium` / `large` / `large-v3`
`GEMINI_MODEL`	`gemini-2.5-flash`	Gemini model identifier
`EDGE_TTS_VOICE`	`en-US-AndrewNeural`	Edge TTS voice — run `edge-tts --list-voices` for all options
`OUTPUT_FORMAT`	`mp4`	Output container format
`KEEP_TEMP`	`False`	Set `True` to keep intermediate files (useful for debugging)
`FFMPEG_BIN`	`ffmpeg`	Path to `ffmpeg` binary
`FFPROBE_BIN`	`ffprobe`	Path to `ffprobe` binary
`MAX_CONCURRENT_TTS`	`8`	Max simultaneous Edge-TTS requests
`MIN_ATEMPO`	`0.75`	Lower bound for TTS time-stretching ratio
`MAX_ATEMPO`	`1.35`	Upper bound for TTS time-stretching ratio
`SKIP_DURATION`	`0.15`	Drop segments shorter than this (seconds)
`TARGET_SEGMENT_DURATION`	`1.5`	Target merged-segment minimum duration (seconds)
`MAX_SEGMENT_DURATION`	`8.0`	Hard cap on a merged segment group (seconds)
`MAX_SEGMENT_GAP`	`0.5`	Gap threshold for starting a new merge group (seconds)

Project Structure

Dubify/
├── main.py                  # CLI entry point (no business logic)
├── config.py                # Environment-based configuration
├── requirements.txt
├── .env.example
│
├── src/
│   ├── models.py            # Segment, VideoProject, PipelineStage
│   ├── pipeline.py          # Stage orchestration and timing
│   ├── reporter.py          # Terminal summary + JSON report generation
│   ├── downloader.py        # yt-dlp video download
│   ├── extractor.py         # FFmpeg audio extraction
│   ├── transcriber.py       # Faster-Whisper transcription
│   ├── translator.py        # Gemini batch translation
│   ├── synthesizer.py       # Edge TTS + atempo time-stretching
│   ├── audio_composer.py    # adelay + amix + mux
│   ├── cache.py             # Stage caching and resumability
│   └── utils.py             # Logging, file helpers, text utilities
│
├── downloads/               # Downloaded source videos
├── temp/                    # Intermediate files (auto-cleaned)
├── outputs/                 # Final dubbed videos + reports
└── logs/                    # Daily log files

Technology Stack

Component	Technology
Video download	yt-dlp
Audio extraction	FFmpeg
Transcription	Faster-Whisper (CTranslate2-optimised)
Translation / Adaptation	Google Gemini 2.5 Flash
Text-to-Speech	Microsoft Edge TTS (neural voices, free)
Time alignment	FFmpeg `atempo` filter (chained for arbitrary ratios)
Audio composition	FFmpeg `adelay` + `amix` filtergraph
CLI / progress	Rich
Configuration	python-dotenv

Design Decisions

Single VideoProject state object — every stage reads from and writes to the same object rather than passing tuples or dictionaries between functions. Adding a new field (e.g. speaker) requires zero changes to stage signatures.

Batch translation — all segments are sent to Gemini in a single JSON round-trip, not one call per segment. This is 10–50× cheaper on longer videos and avoids rate limits.

Dubbing script adaptation, not literal translation — Gemini is prompted as a dubbing expert, not a translator. It is instructed to produce English that sounds like it was originally spoken in English and naturally occupies the same duration as the original segment. This dramatically reduces the amount of atempo stretching needed downstream.

Per-segment atempo time-stretching — after TTS generation, each clip is compared to the original segment duration and speed-adjusted via FFmpeg's atempo filter. This is the single biggest factor in dubbing quality and synchronisation.

No video re-encoding — the composer uses -c:v copy to preserve original video quality. Only the audio stream is replaced.

Abstract provider interfaces — BaseTranslator and BaseSynthesizer decouple the pipeline from specific vendors. Swap Gemini for DeepL, or Edge TTS for ElevenLabs, by implementing one class and changing one constructor argument.

Chunked audio composition — segments are mixed in groups of 60 to stay within FFmpeg's input limits and Windows command-line length constraints on videos with many segments.

Stage caching — each pipeline stage serialises its output to disk. On subsequent runs for the same video, completed stages are restored from cache instantly, enabling rapid iteration on any single stage.

Known Limitations

Translation accuracy on very long videos — batching 200+ segments in a single Gemini call may occasionally produce misaligned translations. Consider splitting very long videos at chapter boundaries.
Atempo range — FFmpeg's atempo accepts [0.5, 2.0] per filter instance. Dubify chains multiple filters automatically, but extreme ratios (very fast or very slow speakers) may still sound unnatural.
CPU Whisper is slow — the base model takes ~2× real-time on CPU. Use large-v3 on a GPU for production-quality transcription at faster-than-real-time speed.
Edge TTS requires internet — the TTS stage makes network calls to Microsoft's Edge speech service. Offline operation requires switching to a local TTS provider.
Single target language — the current pipeline always targets English. Multi-target dubbing requires additional configuration.

Future Improvements

The following features are architecturally prepared (abstract base classes in place) but not yet implemented:

Feature	Notes
Speaker diarisation	`Segment.speaker` field is already present; add pyannote.audio
Voice cloning (XTTS)	Implement `XTTSSynthesizer(BaseSynthesizer)`
ElevenLabs TTS	Implement `ElevenLabsSynthesizer(BaseSynthesizer)`
Subtitle generation	Export `.srt` / `.vtt` from `project.segments`
Batch processing	Accept a list of URLs; `VideoProject` is fully serialisable
GPU acceleration	Pass `device="cuda"` to Whisper via config
Multiple translation providers	`DeepLTranslator`, `IndicTranslator`, etc.
Web UI	`DubbingPipeline.run()` is already async-friendly

Performance

Benchmarked on a 10-minute Hindi YouTube video (CPU only, base Whisper model):

Stage	Time
Download	~8s
Audio Extraction	~3s
Transcription (CPU, base)	~2m 30s
Translation (Gemini batch)	~45s
Speech Synthesis (8 concurrent)	~3m 00s
Compose & Merge	~45s
Total	~7m 10s

Realtime factor: ~0.7× (faster than real-time on a 10-minute video). Using large-v3 on a CUDA GPU reduces transcription to under 1 minute.

Sample Terminal Output

╭──────────────────────────────────────────────────────────╮
│    🎬  Dubify  —  AI Video Dubbing System                │
╰──────────────────────────────────────────────────────────╯

──────────────────── Pipeline ─────────────────────

  ✓ Downloading video                (8s)  — video_title.mp4
  ✓ Extracting audio                 (3s)  — video_title_audio.wav
  ▸ Transcribing speech…
    ████████████████████ 212 / 212  0:02:42
  ✓ Transcribing speech             (2m 42s)  — HI · 212 segments
  ✓ Translating & adapting script   (1m 12s)  — 212 segments → English
  ▸ Synthesizing speech…
    ██████████████████░░ 191 / 212  0:06:38
  ✓ Synthesizing speech             (6m 38s)  — 212 clips generated
  ✓ Merging dubbed audio & exporting (51s)  — outputs/video_title_dubbed.mp4

╭─────────────────── ✓  Processing Complete ───────────────────╮
│                                                              │
│  Input Video        https://youtube.com/watch?v=...          │
│  Output             outputs/video_title_dubbed.mp4           │
│                                                              │
│  Video Duration     11m 02s                                  │
│  Processing Time    12m 31s                                  │
│  Realtime Factor    1.13x                                    │
│                                                              │
│  ── Stage Times ──                                           │
│    Download         8s                                       │
│    Audio Extraction 3s                                       │
│    Transcription    2m 42s                                   │
│    Translation      1m 12s                                   │
│    Speech Synthesis 6m 38s                                   │
│    Compose & Merge  51s                                      │
│                                                              │
│  ── Segments ──                                              │
│    Total Segments   212                                      │
│    Translated       212                                      │
│    Synthesized      212                                      │
│    Tempo Adjusted   174                                      │
│    Avg Tempo Ratio  0.91                                     │
│                                                              │
│  Detected Language  HI                                       │
╰──────────────────────────────────────────────────────────────╯

📄 Report saved: outputs/processing_report.json

Sample Processing Report

outputs/processing_report.json:

{
    "input_video": "https://www.youtube.com/watch?v=...",
    "output_video": "outputs/video_title_dubbed.mp4",

    "video_duration_seconds": 662.0,
    "processing_time_seconds": 751.4,
    "realtime_factor": 1.1349,

    "stage_times": {
        "download": 8.1,
        "extract_audio": 3.2,
        "transcription": 162.4,
        "translation": 72.1,
        "tts": 398.3,
        "time_alignment": 0.0,
        "merge": 51.4
    },

    "segments": {
        "total": 212,
        "translated": 212,
        "synthesized": 212,
        "tempo_adjusted": 174,
        "average_ratio": 0.9147
    }
}

License

MIT

Name		Name	Last commit message	Last commit date
src		src
tests		tests
.env.example		.env.example
.gitignore		.gitignore
README.md		README.md
config.py		config.py
main.py		main.py
requirements.txt		requirements.txt
temp_plain.mp3		temp_plain.mp3
temp_ssml.mp3		temp_ssml.mp3
temp_test.mp3		temp_test.mp3
temp_test.py		temp_test.py
test_audit_ssml.py		test_audit_ssml.py
test_variations.py		test_variations.py
var_a.mp3		var_a.mp3
var_b.mp3		var_b.mp3
var_c.mp3		var_c.mp3

Navigation Menu

Repository navigation

About

Repository files navigation

Dubify

Overview

Architecture

Module responsibilities

Features

Installation

Prerequisites

Setup

Usage

Basic

Pass URL directly

CLI flags

Examples

Output directories

Configuration

Project Structure

Technology Stack

Design Decisions

Known Limitations

Future Improvements

Performance

Sample Terminal Output

Sample Processing Report

License

About

Releases

Packages

Contributors

Languages

Latest commit

History

Folders and files

Repository files navigation

Dubify

Overview

Architecture

Module responsibilities

Features

Installation

Prerequisites

Setup

Usage

Basic

Pass URL directly

CLI flags

Examples

Output directories

Configuration

Project Structure

Technology Stack

Design Decisions

Known Limitations

Future Improvements

Performance

Sample Terminal Output

Sample Processing Report

License

About

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages