Voice Cloning Engine

A local voice cloning engine built on XTTS-v2: given a short reference recording of a speaker (with their authorization) and some text, it synthesizes speech in that speaker's voice. It also includes the dataset preparation and fine-tuning pipeline used to adapt XTTS-v2 to a specific speaker.

This repository contains the reusable engine and tooling only. It does not include any voice recordings, model weights, or trained checkpoints — you supply your own reference audio and download the XTTS-v2 model yourself (see Model Setup).

Overview

XTTS-v2 is a multilingual zero-shot voice cloning text-to-speech model released by Coqui. It can synthesize speech in a target voice from just a few seconds of reference audio ("zero-shot"), and can also be fine-tuned on a larger, curated dataset of one speaker to more closely match that speaker's voice.

This engine wraps XTTS-v2 with:

• Reproducible checkpoint loading — handles the different container shapes checkpoints show up in, strips training-time key prefixes, and verifies the checkpoint is actually architecture-compatible before trusting it (see Limitations and docs/TRAINING_NOTES.md for a real bug this catches).
• A synthesis API usable from Python, a CLI, or a persistent local server (a Unix socket process that loads the model once and serves repeated requests, for callers that shouldn't import torch themselves).
• A dataset preparation pipeline for turning raw reference recordings into a fine-tuning dataset (segmentation is left to you; transcription, manifest review, validation, and format conversion are provided).
• A fine-tuning script for adapting XTTS-v2's GPT component to a specific speaker, with the architecture-fidelity safety checks mentioned above.

Architecture

Reference Voice (yours, with consent)
      |
Audio Preprocessing            <- ffmpeg: mono, 24kHz, 16-bit (scripts/prepare_dataset.py)
      |
Speaker Conditioning           <- model.get_conditioning_latents() (voice_engine/engine.py)
      |
XTTS-v2 (GPT + HiFi-GAN decoder)
      |
Generated Speech               <- NaN/Inf/silence sanity checks (voice_engine/audio.py)
      |
WAV Output

Fine-tuning follows a parallel path: raw recordings are segmented, transcribed (Whisper), manually reviewed, converted to a fine-tuning dataset, and used to further train the GPT component against the reference speaker's voice (scripts/train.py).

Features

• Zero-shot voice cloning from a single reference .wav using a stock pretrained XTTS-v2 checkpoint.
• Loading and running your own fine-tuned XTTS-v2 checkpoint.
• Checkpoint compatibility verification that catches a real class of fine-tuning bug (mismatched GPT architecture / corrupted speaker conditioning) before it produces bad audio silently.
• A CLI (voice-engine generate, voice-engine serve) and a Python API (VoiceCloningEngine).
• A persistent local synthesis server (Unix domain socket, newline-delimited JSON protocol) for callers that want to avoid importing ML dependencies.
• A dataset preparation pipeline: transcription, manifest-based manual review, validation, and LJSpeech-format conversion.
• A fine-tuning script with the architecture-fidelity fix described in docs/TRAINING_NOTES.md.

Not included: a web UI, real-time streaming synthesis, multi-speaker switching in a single process, or any hosted/cloud component. Everything here runs locally.

Requirements

• Python 3.10 (developed and tested against 3.10.20)
• ffmpeg on PATH — used by the dataset preparation pipeline to convert audio (not required for inference alone)
• macOS, Linux, or Windows. Tested on macOS (CPU and Apple Silicon mps); not verified on CUDA or Windows, though nothing in the code is macOS-specific except the audio playback step in scripts/review_dataset.py (afplay).
• ~2 GB free disk space for Python dependencies (torch, coqui-tts, and their transitive dependencies), plus however much space the XTTS-v2 checkpoint(s) you download need (the pretrained model is roughly 1.8 GB).

Installation

git clone <this-repo-url>
cd voice-cloning-engine

python3.10 -m venv .venv
source .venv/bin/activate        # Windows: .venv\Scripts\activate

pip install -r requirements.txt
pip install -e .                 # installs the `voice_engine` package + `voice-engine` CLI

# Optional, only needed for the dataset-preparation scripts:
pip install -r requirements-dev.txt   # adds pytest, for running the test suite

Model Setup

This repository does not include XTTS-v2's weights. XTTS-v2 is distributed by Coqui under the Coqui Public Model License (CPML), a non-commercial license that also governs the model's generated output — see Licensing before using it for anything beyond personal, non-commercial use.

Option A — let coqui-tts download it for you (simplest):

# Accepts the CPML terms non-interactively; omit this and you'll be
# prompted for [y/n] the first time instead.
export COQUI_TOS_AGREED=1

python -c "from TTS.api import TTS; TTS('tts_models/multilingual/multi-dataset/xtts_v2')"

This downloads the model into the OS-appropriate cache directory (macOS: ~/Library/Application Support/tts/, Linux: ~/.local/share/tts/, Windows: %LOCALAPPDATA%\tts\) — resolved by voice_engine.config.default_model_cache_dir(), not hardcoded anywhere in this codebase. Inside that directory you'll find tts_models--multilingual--multi-dataset--xtts_v2/ containing config.json, vocab.json, and model.pth.

Option B — download manually from Hugging Face:

huggingface-cli download coqui/XTTS-v2 --local-dir /path/to/xtts-v2

Either way, note the resulting paths to config.json, vocab.json, and model.pth — every script and the CLI take these as explicit arguments (or VOICE_ENGINE_CONFIG / VOICE_ENGINE_VOCAB / VOICE_ENGINE_CHECKPOINT environment variables). Nothing in this engine assumes a fixed install location.

If you plan to fine-tune (scripts/train.py), you'll also need the DVAE training-support checkpoint and mel-stats file, distributed alongside the main model in the same Hugging Face repo (dvae.pth, mel_stats.pth).

Reference Audio

You need one short .wav recording of the voice you want to clone — a voice you own, or one you have explicit, informed consent to clone. See Responsible Use.

Recommended characteristics (not strictly enforced, but they matter for quality):

• 6–30 seconds, a single speaker, minimal background noise or music
• Mono, 16-bit PCM, ideally 22050–24000 Hz (the engine resamples via model.get_conditioning_latents, but starting clean helps)
• Natural, conversational speech rather than singing, whispering, or heavy emotional performance

For fine-tuning on a larger dataset of the same speaker, see Project Structure → dataset scripts below; the pipeline expects several minutes of segmented, transcribed clips.

Usage

Generate speech from text (zero-shot, or with a fine-tuned checkpoint):

voice-engine generate \
  --config path/to/config.json \
  --vocab path/to/vocab.json \
  --checkpoint path/to/model.pth \
  --reference path/to/your/reference.wav \
  --text "Hello, this is a test." \
  --output output/hello.wav

Equivalently, from Python:

from voice_engine import EngineConfig, VoiceCloningEngine

config = EngineConfig(
    xtts_config="path/to/config.json",
    xtts_vocab="path/to/vocab.json",
    checkpoint="path/to/model.pth",
    reference_audio="path/to/your/reference.wav",
)
engine = VoiceCloningEngine(config).load()
engine.synthesize_to_file("Hello, this is a test.", "output/hello.wav")

See also examples/quickstart.py for a complete minimal script.

Run the persistent local server (loads the model once, then serves requests over a Unix socket):

voice-engine serve \
  --config path/to/config.json --vocab path/to/vocab.json \
  --checkpoint path/to/model.pth --reference path/to/your/reference.wav

A caller sends one newline-delimited JSON object and reads one back — see src/voice_engine/server.py's module docstring for the protocol.

Dataset preparation pipeline (for fine-tuning on your own voice):

# 1. Put your raw recordings in data/raw/, then segment them into short
#    clips yourself (e.g. via a DAW or ffmpeg silence-splitting) into
#    data/segments/*.wav — segmentation is not automated by this project.

# 2. Auto-transcribe each segment with Whisper
python scripts/transcribe_dataset.py --data-dir data

# 3. Build a review manifest
python scripts/create_dataset_manifest.py --data-dir data

# 4. Manually review/correct transcripts, mark KEEP/REJECT
python scripts/review_dataset.py --data-dir data

# 5. Check basic audio/transcript quality
python scripts/validate_dataset.py --data-dir data

# 6. Convert KEPT clips into the final LJSpeech-format training dataset
python scripts/prepare_dataset.py --data-dir data

Fine-tune XTTS-v2 on the prepared dataset:

python scripts/train.py \
  --dataset-dir data/xtts_dataset \
  --pretrained-checkpoint path/to/model.pth \
  --pretrained-config path/to/config.json \
  --vocab path/to/vocab.json \
  --dvae-checkpoint path/to/dvae.pth \
  --mel-stats path/to/mel_stats.pth \
  --output-dir output/xtts_training \
  --epochs 8

Read docs/TRAINING_NOTES.md first — it documents a real architecture bug this pipeline guards against, and why the gpt_use_perceiver_resampler check in scripts/train.py will refuse to start training if it can't verify architecture fidelity.

Compare checkpoints side-by-side:

python scripts/evaluate_checkpoints.py \
  --config path/to/config.json --vocab path/to/vocab.json \
  --reference path/to/your/reference.wav \
  --checkpoint pretrained=path/to/model.pth \
  --checkpoint finetuned=output/xtts_training/.../best_model.pth

Output

Generated .wav files are written wherever you point --output / output_path — by default under output/, which is git-ignored. Nothing is written outside the path you specify.

Testing

pip install -r requirements-dev.txt
pytest

The test suite covers configuration resolution, checkpoint key-cleaning and compatibility logic, audio analysis, CLI argument parsing, and engine pre-load state — all with synthetic data, so it runs without downloading any model or providing any audio. It does not exercise real XTTS-v2 inference; there's no automated test that loads an actual checkpoint. If you want to verify inference end-to-end, run examples/quickstart.py against a real checkpoint and reference recording and listen to the result.

Project Structure

voice-cloning-engine/
├── src/voice_engine/
│   ├── __init__.py       # package exports
│   ├── config.py         # EngineConfig — path/device resolution, no hardcoded paths
│   ├── checkpoint.py      # state-dict cleaning + compatibility verification
│   ├── audio.py            # waveform sanity checks + WAV I/O
│   ├── engine.py            # VoiceCloningEngine — load / condition / synthesize
│   ├── server.py             # persistent Unix-socket synthesis server
│   └── cli.py                 # `voice-engine generate|serve`
├── scripts/
│   ├── transcribe_dataset.py       # Whisper auto-transcription
│   ├── create_dataset_manifest.py  # build the review manifest
│   ├── create_transcript_review.py # plain-text transcript review sheet
│   ├── review_dataset.py           # interactive KEEP/REJECT review
│   ├── validate_dataset.py         # audio/transcript QA
│   ├── prepare_dataset.py          # -> final LJSpeech-format dataset
│   ├── train.py                    # XTTS-v2 GPT fine-tuning
│   ├── evaluate_checkpoints.py     # multi-checkpoint comparison
│   └── inspect_tokenizer.py        # tokenizer debugging utility
├── examples/
│   └── quickstart.py     # minimal end-to-end example
├── tests/                 # unit tests (no model/network required)
├── docs/
│   └── TRAINING_NOTES.md  # a real architecture bug found during fine-tuning
├── requirements.txt
├── requirements-dev.txt
├── pyproject.toml
├── LICENSE
└── README.md

Limitations

• Not benchmarked for speed. No latency/throughput numbers are published here because none have been rigorously measured across hardware; expect CPU inference to be well below real-time. This engine has not been tested for real-time or interactive use cases.
• Fine-tuning was developed against a small, single-speaker dataset (tens of short clips). The training script's defaults (batch size 1, conservative learning rate) reflect that; scaling to a larger dataset may need different hyperparameters.
• English-only testing. XTTS-v2 itself is multilingual, but this engine's scripts and defaults (language="en") have only been exercised with English text and reference audio.
• scripts/review_dataset.py uses macOS's afplay for audio playback; on Linux/Windows, substitute your platform's player or use the [s] skip option and listen to clips separately.
• Segmentation of raw recordings into clips is not automated — the dataset pipeline starts from already-segmented clips.
• No automated test exercises real XTTS-v2 inference (see Testing) — the unit tests validate the engine's logic, not audio quality.
• This is a personal/portfolio-grade engine, not a hardened production service. There's no authentication, rate limiting, or multi-tenant isolation in server.py — it's designed for a single trusted local caller.

Responsible Use

Voice cloning can be used to impersonate real people. Only clone a voice you own, or one you have explicit, informed consent to clone. Do not use this engine to generate speech in someone else's voice without their permission, to impersonate someone, or to create misleading or deceptive audio. You are responsible for how you use the audio this engine produces.

Licensing

This repository's own source code (src/, scripts/, examples/, tests/) is licensed under the MIT License.

That license does not extend to two things you'll use alongside it:

• XTTS-v2 (the model architecture and pretrained weights) is distributed by Coqui under the Coqui Public Model License 1.0.0 (CPML) — non-commercial use only, and the restriction explicitly covers the model's output, not just the weights themselves. Read the full text at https://huggingface.co/coqui/XTTS-v2/blob/main/LICENSE.txt (verified current as of this writing; the license's own canonical coqui.ai/cpml.txt link no longer resolves) before using this engine for anything beyond personal, non-commercial purposes.
• coqui-tts (the Python library this engine depends on — a community-maintained fork of the original, now-unmaintained Coqui TTS package) is licensed under the Mozilla Public License 2.0 (MPL-2.0), confirmed against its source at https://github.com/idiap/coqui-ai-TTS.

There is currently no commercial licensing path for XTTS-v2. Coqui, Inc. shut down in January 2024, and a Coqui maintainer has stated publicly that "there is no way of obtaining a commercial license" as a result (see coqui-ai/TTS discussion #4304). In practice this means XTTS-v2 is non-commercial-only with no paid upgrade option — if your use case is commercial, use a differently-licensed TTS model instead. This repository's MIT license covers only the integration code, not the model it drives, and does not change any of the above.