Higgs_Audio_TTS voice cloning not working and Higgs TTS generation reached max_tokens before EOC (Arabic, CPU)

[user6654126]

Environment:

OS:Linux Mint 22.3 - Cinnamon 64-bit
CPU:AMD Ryzen 5 8400F
GPU: RTX 5060 8G VRAM
RAM: 6000MHz 16GB RAM
Model: higgs-audio-v3-tts-4b-q8_0.gguf
version: latest v0.7.2

• branch main -> FETCH_HEAD
  bf3315f..49b0322 main -> origin/main
  Updating bf3315f..49b0322

Symptom:

when i use this model with backend cpu sometime it work , if the TTS script is small meaning less than 2 minutes.
but the problem it doesn't even use reference voice in voice cloning instead it use different and random voices and i noticed it switch into different voice after whatever limit i set in --text-chunk-size 300
so how i make it use the reference voice for voice cloning

Commands arabic test:

./build/linux-cpu-release/bin/audiocpp_cli
--task tts
--family higgs_audio_tts
--session-option "higgs_audio.mem_saver=true"
--model /home/xxxx/audio.cpp/build/linux-cpu-release/bin/models/Higgs-Audio-v3-TTS-4B-GGUF/higgs-audio-v3-tts-4b-q8_0.gguf
--backend cpu
--threads 10
--text "$(cat /home/xxxx/audio.cpp/inputs/gemini-arabic-mini.txt)"
--voice-ref /home/xxxx/audio.cpp/inputs/stoic_10seconds.wav
--reference-text "$(cat /home/xxxx/audio.cpp/inputs/stoic_10seconds.txt)"
--out /home/xxxx/audio.cpp/outputs/higgs_ara_1.wav
--text-chunk-size 300
--metrics
--log

Logs:
higgs_ara_1 log.txt

also if i removed --session-option "higgs_audio.mem_saver=true" \ and --text-chunk-size 300 \ i get Higgs TTS generation reached max_tokens before EOC
and if script is long +5 minutes i get EOC so how to fix that

Commands arabic test:

./build/linux-cpu-release/bin/audiocpp_cli
--task tts
--family higgs_audio_tts
--model /home/xxxx/audio.cpp/build/linux-cpu-release/bin/models/Higgs-Audio-v3-TTS-4B-GGUF/higgs-audio-v3-tts-4b-q8_0.gguf
--backend cpu
--threads 10
--text "$(cat /home/xxxx/audio.cpp/inputs/gemini-arabic-mini.txt)"
--voice-ref /home/xxxx/audio.cpp/inputs/stoic_10seconds.wav
--reference-text "$(cat /home/xxxx/audio.cpp/inputs/stoic_10seconds.txt)"
--out /home/xxxx/audio.cpp/outputs/higgs_ara_1.wav
--metrics
--log

Logs:
higgs_ara_2 log.txt

gemini-arabic-mini.txt
stoic_10seconds.txt
higgs_ara_1.wav
stoic_10seconds.wav

[0xShug0]

@user6654126 Thanks for the detailed report! I'll reproduce and look into the issue. Quick question: Have you tested english intput?

[user6654126]

@user6654126 Thanks for the detailed report! I'll reproduce and look into the issue. Quick question: Have you tested english intput?

I have but i am not interested in using it for English as there are better alternative which is BreezeTTS 2 and i used it successfully using cuda and it's perfect in English so i want the best tts model in Arabic so that's why I am using higgs audio v3 tts for Arabic

[0xShug0]

@user6654126 Thanks for the detailed report! I'll reproduce and look into the issue. Quick question: Have you tested english intput?

I have but i am not interested in using it for English as there are better alternative which is BreezeTTS 2 and i used it successfully using cuda and it's perfect in English so i want the best tts model in Arabic so that's why I am using higgs audio v3 tts for Arabic

Sorry I should make it clear: This is just to help me find the root cause. At least a quick test on CUDA on my machine, English and Chinese work correctly without EOC under chunking and produce consisent voices. Arabic EOC on both CPU and CUDA.

So if English works on your machine on CPU, we can rule out backend/environment quirks. The likely issue is Arabic input processing.

higgs_english_chunk300_unseeded_cuda.wav

higgs_chinese_long_chunk300_unseeded_cuda.wav

[user6654126]

@user6654126 Thanks for the detailed report! I'll reproduce and look into the issue. Quick question: Have you tested english intput?

I have but i am not interested in using it for English as there are better alternative which is BreezeTTS 2 and i used it successfully using cuda and it's perfect in English so i want the best tts model in Arabic so that's why I am using higgs audio v3 tts for Arabic

Sorry I should make it clear: This is just to help me find the root cause. At least a quick test on CUDA on my machine, English and Chinese work correctly without EOC under chunking and produce consisent voices. Arabic EOC on both CPU and CUDA.

So if English works on your machine on CPU, we can rule out backend/environment quirks. The likely issue is Arabic input processing.

higgs_english_chunk300_unseeded_cuda.wav

higgs_chinese_long_chunk300_unseeded_cuda.wav

ok thank you @0xShug0 for explaining .. so i tested higgs model in english and the voice is not consistent and it didn't use reference voice but different voices as i said before

./build/linux-cpu-release/bin/audiocpp_cli
--task tts
--session-option "higgs_audio.mem_saver=true"
--family higgs_audio_tts
--model /home/xxxx/audio.cpp/build/linux-cpu-release/bin/models/Higgs-Audio-v3-TTS-4B-GGUF/higgs-audio-v3-tts-4b-q8_0.gguf
--backend cpu
--threads 10
--text-chunk-size 300
--text "$(cat /home/xxxx/audio.cpp/inputs/eng_test.txt)"
--voice-ref /home/xxxx/audio.cpp/inputs/stoic_10seconds.wav
--reference-text "$(cat /home/xxxx/audio.cpp/inputs/stoic_10seconds.txt)"
--out /home/xxxx/audio.cpp/outputs/higgs_eng_1.wav
--metrics
--log
eng_test.txt
higgs_eng_1.wav
higgs_eng_1 log.txt

[0xShug0]

@user6654126 I’ll focus on the consistency issue first. I ran your test case on CPU, and it produced a consistent voice. Are you using the prebuilts, or did you build from commit 49b0322?

Another minor issue, caused by stale docs on my side: --session-option "higgs_audio.mem_saver=true" is not a valid option. It should be higgs_audio_tts.mem_saver. But in the test, I used your option exactly as written so it isn't relevent.

higgs_eng_user_chunk300_mem_saver_cpu_e75c235.wav

[user6654126]

@user6654126 I’ll focus on the consistency issue first. I ran your test case on CPU, and it produced a consistent voice. Are you using the prebuilts, or did you build from commit 49b0322?

Another minor issue, caused by stale docs on my side: --session-option "higgs_audio.mem_saver=true" is not a valid option. It should be higgs_audio_tts.mem_saver. But in the test, I used your option exactly as written so it isn't relevent.

higgs_eng_user_chunk300_mem_saver_cpu_e75c235.wav

@0xShug0 i build it with these commands for more info see last update logs.txt
git pull origin main
rm -rf build/
./scripts/build_linux.sh --backend cuda --cuda-arch native --native-model-manager --target audiocpp_cli --target audiocpp_server
./scripts/build_linux.sh --backend cpu --native-model-manager --target audiocpp_cli --target audiocpp_server

last update logs.txt

why is the voice is not consistent but keep changing especially at the end in your sample
higgs_eng_user_chunk300_mem_saver_cpu_e75c235.wav

[0xShug0]

why is the voice is not consistent but keep changing especially at the end in your sample

You are right. I missed the ending part. Now it looks like the CPU backend issue and I can debug on my machine.

Linux/CUDA seems to be consisent (hopefully I didn't miss anything).

higgs_eng_user_chunk300_mem_saver_cuda_e75c235.wav

Update: Linux/Vulkan seems fine too.

higgs_eng_user_chunk300_mem_saver_vulkan_e75c235.wav

Metal also seems fine.
higgs_eng_user_chunk300_mem_saver_metal_e75c235.wav

[0xShug0]

For anyone interested in this issue:

This may not be caused by missing cross-chunk reference reuse. Higgs already reuses the encoded reference audio codes across chunks, and it also has an AR reference-prefix KV reuse path when the prefix matches. Disabling prefix KV reuse on CPU did not fix the drift using the user provided test case (whille it passed a shorter text test).

The real concern is narrower: with the same Higgs Audio TTS chunked voice-clone path, CUDA/Vulkan/Metal keep the reference voice reasonably consistent across chunks, while CPU can drift into noticeable different voices between chunks. That suggests a CPU-side instability or state/lifecycle issue when repeatedly running the same graph/code path across chunks.

Failed attempts: CPU AR weight fp16 / precision tweak; Codec-only precision tweak; Using eager attention.

[user6654126]

@0xShug0 thank you for your help. Since the issue is clearly is with using cpu the higgs model doesn't use the reference voice provided in arabic and use random and non consistent voices . Is there is a way to use this model on my rtx 5060 8gb vram as i already tried even with chunk 300 and chunk 500 everytime i get oom error. By the way i noticed similar issue with using cpu in voxcpm2 in arabic which i will post about soon the voxcpm2 model use the reference voice and voice is consistent however when converting longer tts script more than 5 minutes i get EOC error.

[0xShug0]

Higgs: I made a correctness fix to a framework module, and it seems to have fixed the Arabic EOC issue on my side.

Voxcpm2: This is more of an upstream model issue. Please check their issues. You can find discussions about noises, tag control and long-form quality there. The maintainer also mentioned that long text can indeed cause quality degradation, and the issue hasn’t been fully resolved yet.

You can check the Python + CUDA output attached. It seems to start hallucinating very early and produces more noise over time.
voxcpm_latest_direct_cuda_english_64k.mp3

I also dded a new option --request-option voxcpm2.chunk_strategy=stateless as a workaround of the noises issue. However, it does not provide cross-chunk continuity, because each chunk is synthesized independently from the same original reference/prompt. The mode is intended for text without tags, since tags can make independently synthesized chunks diverge more from each other.

[user6654126]

Higgs: I made a correctness fix to a framework module, and it seems to have fixed the Arabic EOC issue on my side.

Voxcpm2: This is more of an upstream model issue. Please check their issues. You can find discussions about noises, tag control and long-form quality there. The maintainer also mentioned that long text can indeed cause quality degradation, and the issue hasn’t been fully resolved yet.

You can check the Python + CUDA output attached. It seems to start hallucinating very early and produces more noise over time. voxcpm_latest_direct_cuda_english_64k.mp3

I also dded a new option --request-option voxcpm2.chunk_strategy=stateless as a workaround of the noises issue. However, it does not provide cross-chunk continuity, because each chunk is synthesized independently from the same original reference/prompt. The mode is intended for text without tags, since tags can make independently synthesized chunks diverge more from each other.

@0xShug0 thanks . i am waiting for fix for Higgs model as it's currently is the best opensource tts model for arabic can you please tell me about the fix you made did you fix EOC issue for arabic on cpu and the issue of non consistent voices and the model ignore the reference voice and use random voice on its own and when it will be avaliable so i can update audio.cpp build and use it
by the way i have crazy idea i don't know if it may fix that issue with higgs model but what if i use q5 or q6 instead of q8 on cuda can you please convert or tell me convert commands as i already tried that and the model didnt work

[0xShug0]

Arabic EOC issue. The fix has already been merged into the current main branch.