feat(kokoro): add multilingual GGUF synthesis and optimized CPU inference

[mirek190]

Summary

Add the Kokoro 82M integration to main with optimized CPU inference, standalone multilingual Q8/BF16 GGUF packages, and the shared eSpeak-ng phonemizer introduced in #502.

Kokoro is registered through the current selectable-model CMake system and generated runtime registry. It can be included in normal CLI/server builds or selected with AUDIOCPP_MODEL_SET=custom and AUDIOCPP_MODELS=kokoro_tts.

Model and package support

• Offline synthesis from an extracted model directory or locally prepared Kokoro GGUF.
• All 54 source preset voices and nine language variants: American/British English, Spanish, French, Hindi, Italian, Japanese, Brazilian Portuguese, and Mandarin Chinese.
• Q8 and BF16 conversion with embedded voice and pronunciation resources. Q8 packages retain BF16/F32 for weights that are unsuitable for quantization.
• Native English pronunciation, shared eSpeak for Spanish/French/Hindi/Italian/Portuguese, MeCab/UniDic for Japanese, and dictionary/DAG/HMM processing for Chinese.
• Model-local optimized F32 CPU paths for convolution preparation, Snake activation, and AdaIN, including predictor normalization. Unsupported layouts retain the existing execution path.
• Reproducible warm benchmarks, CPU-kernel tests, multilingual comparison scripts, GGUF quality comparisons, and synthesis smoke tests.

These are locally prepared Kokoro packages; arbitrary third-party GGUF layouts are not supported. See preparation and usage instructions. Very long inputs and Japanese/Chinese normalization edge cases still need broader coverage.

Shared eSpeak integration

Kokoro now calls engine::audio::EspeakPhonemizer instead of maintaining a separate eSpeak runtime. Initialization, voice selection, returned-buffer copying, and runtime switching share the process-wide mutex used by SanoTTS and Inflect v2. Kokoro retains its caret-tied IPA mode, punctuation restoration, and phoneme-to-token mapping.

With AUDIOCPP_STATIC_ESPEAK=ON, CLI and server use statically linked eSpeak and automatically discover espeak-ng-data.bin beside their executable. Data extraction uses the shared cache from #502. The default dynamic build remains supported.

AUDIOCPP_ESPEAK_LIBRARY retains explicit library selection and the model-local data default. AUDIOCPP_ESPEAK_DATA can select an unpacked data directory or a shared .bin/.gguf data package. Japanese retains its separate MeCab dependency and AUDIOCPP_MECAB_LIBRARY override.

Static eSpeak is opt-in and retains the distribution requirements documented in the shared component.

CPU performance

Previously measured on Windows, Ryzen 9 7950X3D, 8 CPU threads, MSVC Release /arch:AVX512, F32 Kokoro, af_heart, seed 1234. Optimized results are the mean of two quiet runs. Timings exclude initial model load/session preparation and WAV writing. These measurements predate this shared-eSpeak migration and were not rerun as new performance claims.

Workload	Original baseline	Optimized	Improvement
Short, first request (4.475 s audio)	1.718 s	0.930 s	1.85x faster
Short, repeat 1	1.682 s	0.886 s	1.90x faster
Short, repeat 2	1.612 s	0.886 s	1.82x faster
Second prompt (4.925 s audio)	1.923 s	1.052 s	1.83x faster
Long-form (29.5 s audio)	11.558 s	6.390 s	1.81x faster
Long-form repeat	11.726 s	6.376 s	1.84x faster
Short after long-form	1.738 s	0.962 s	1.81x faster

All seven F32 benchmark WAVs matched the original baseline byte-for-byte. This is not a claim that quantized Q8/BF16 output matches F32 bit-for-bit.

Validation after integration with main

• Built the Windows Release CPU CLI and server with static eSpeak enabled.
• Built and exercised the Kokoro frontend with static eSpeak disabled.
• Shared dynamic mode: 12/12 multilingual pronunciation sequences match the pre-migration frontend exactly.
• Shared static mode with automatic data-package discovery: 12/12 sequences match exactly.
• Separate default dynamic build: 12/12 sequences match exactly.
• Q8 and BF16 Spanish CLI synthesis: both WAVs match their respective pre-migration precision's WAV byte-for-byte.
• Kokoro CPU-kernel, shared eSpeak runtime, and eSpeak data-package tests: 3/3 pass. CPU-kernel coverage checks 1,098 cases at 1 and 8 threads.
• The generated runtime registry lists kokoro_tts; CLI and server link successfully through the new model selection system.
• Diff against main passes whitespace checks.

The earlier multilingual GGUF validation produced non-silent 24 kHz audio for all 18 language/precision combinations. Current local validation is Windows CPU; this change does not claim new CUDA/Vulkan or Linux/macOS runtime coverage.

[0xShug0]

@mirek190 Bro you’re moving too fast 😄 I haven’t even rebased or cleaned up the code yet. That implementation is from pre-release 0.1 and is pretty messy...

[0xShug0]

Give me some time...Wrapping up vibevoice asr streaming 7B now, and also a bunch of ggml kernel changes and framework refactor will be tested and merged next

[mirek190]

@mirek190 Bro you’re moving too fast 😄 I haven’t even rebased or cleaned up the code yet. That implementation is from pre-release 0.1 and is pretty messy...

It WAS messy ;)

[mirek190]

Give me some time...Wrapping up vibevoice asr streaming 7B now, and also a bunch of ggml kernel changes and framework refactor will be tested and merged next

No problem
I am working on gguf implementation now for it ....
When you do your work I just adapt model to changes

[0xShug0]

I even tried to impl g2p in kokoro, while the better way (for now) is just like inflect and sanoTTS that load espeak during runtime...

[mirek190]

I even tried to impl g2p in kokoro, while the better way (for now) is just like inflect and sanoTTS that load espeak during runtime...

Currently is working this way

Kokoro text
↓
Kokoro language selection and basic text handling
↓
Dynamically loaded eSpeak-ng library
↓
IPA phonemes
↓
Kokoro-compatible IPA cleanup and mapping
↓
Kokoro token IDs
↓
Kokoro TTS model

So we should implement eSpeak-ng to audiocpp and use it to other models if they need it ( like sanoTTS, inflect_v2TTS or korokoTTS )

[mirek190]

I have already good eSpeak-ng implementation
I just isolate that component (make new PR) for reusing to other models.

[0xShug0]

I have already good eSpeak-ng implementation I just isolate that component (make new PR) for reusing to other models.

Great, thanks!

[mirek190]

Kokoro CPU, CUDA, and Vulkan performance

I ran the same seven-request resident-session benchmark across all three backends and both standalone GGUF precisions.

Test system: Ryzen 9 7950X3D, RTX 3090 24 GB, Windows Release builds, af_heart, seed 1234. Times are seconds per request and exclude model loading, standalone-resource extraction, session preparation, and WAV writing. CPU uses eight threads. CUDA values are the mean of two fresh sessions without an explicit warmup, so the first-request result includes CUDA graph capture. Vulkan values are the mean of two sessions after the driver pipeline cache was populated.

Workload	CPU Q8	CUDA Q8	Vulkan Q8	CPU BF16	CUDA BF16	Vulkan BF16
Short, first request (about 4.5 s audio)	0.947 s	0.339 s	0.200 s	0.906 s	0.352 s	0.226 s
Short, repeat 1	0.900 s	0.180 s	0.112 s	0.858 s	0.176 s	0.111 s
Short, repeat 2	0.901 s	0.095 s	0.109 s	0.868 s	0.095 s	0.112 s
Second prompt (about 4.9 s audio)	1.044 s	0.198 s	0.201 s	1.030 s	0.223 s	0.202 s
Long-form (about 29.5 s audio)	6.309 s	1.153 s	0.976 s	6.240 s	1.247 s	0.984 s
Long-form repeat	6.347 s	1.115 s	0.974 s	6.219 s	1.200 s	0.968 s
Short after long-form	0.948 s	0.175 s	0.164 s	0.929 s	0.189 s	0.169 s

The first Vulkan Q8 process on an empty driver cache took 4.090 s for the first short request while compiling pipelines; the next fresh process took 0.202 s. I report that one-time driver setup separately rather than mixing it into resident inference.

All ten retained CUDA/Vulkan WAVs were valid and non-silent, with mean volume between -26.8 and -26.9 dB. These results describe this machine and driver stack and should not be treated as general backend rankings.

[mirek190]

Ready to merge

[0xShug0]

@mirek190 Merged! No issue found during rutime validaiton. RTF ~0.2 on AMD Ryzen 9 7900X. I will do some follow-ups to remove the old g2p_en / misaki_en hacks and wire Kokoro into the UI.

[mirek190]

@mirek190 Merged! No issue found during rutime validaiton. RTF ~0.2 on AMD Ryzen 9 7900X. I will do some follow-ups to remove the old g2p_en / misaki_en hacks and wire Kokoro into the UI.

sure