[Feature Request] new model: AuK

[kooks28]

"AuK is a 1.5B foundation model for speech generation and editing. Trained on millions of hours of diverse audio data, AuK supports zero-shot and instruction-based TTS, content and acoustic editing, paralinguistic editing, speech enhancement, and source separation through a unified natural-language instruction interface."
https://github.com/Tencent-Hunyuan/AuK
https://huggingface.co/tencent/AuK
https://huggingface.co/tencent/AuK-Flash

[0xShug0]

@kooks28 Thanks for the pointer. There is a catch: the model uses Qwen2.5-Omni-3B as MLLM encoder so the actual size is much larger. I also had a bad experience with a Tencent’s model (SongGeneration 2 / LeVo 2 model), which apparently was published just for KPI and turned out to be a complete disaster. Let’s hold off and see more feedback on AuK.

[HaujetZhao]

It's not for KPI. It's a revolutionary model! @0xShug0

Go check out https://auk-project.github.io/

[0xShug0]

@HaujetZhao How does Auk compare to FireRed Audio? FireRedAudio? FireRedAudio is in audio.cpp.

[HaujetZhao]

I think you should try them both yourself; there's a significant difference between them.

• https://huggingface.co/spaces/tencent/AuK
• https://huggingface.co/spaces/hugging-apps/fireredaudio-tts-demo

In my opinion, AuK excels in editing, to the point where the voice actors no longer need to re-record when lines changes; they simply need to modify the voice using AuK. That's why I consider it revolutionary. It's not a TTS (Text-to-Speech) model, but an audio editing model capable of various adjustments, such as modifying dialogue, extracting clear human voices (removing background noise), isolating specific speakers, and converting regular speech into whispers. I had been eagerly anticipating an open-source version of Adobe Podcast Speech Enhance, and now AuK has arrived, and the ability to extract and enhance voices is just one of its many features.

Firered's advantage lies in its ability to understand long audio content and generate text using ASR, but this comes at the cost of a 9B parameter model. With only 8GB of VRAM, I cannot run it smoothly. Auk, with a 1.5B parameter model, combined with Qwen2.5 omni 3B, totals less than 5B, making it feasible to run on consumer-grade GPUs. Even with only 4GB of VRAM, you can first load Qwen2.5 omni 3B and then unload it after getting latent, followed by loading Auk 1.5B for audio generation. Essentially, there's no need to run a 9B model solely for ASR. However, there are indeed users who require a powerful audio editing model, and one that doesn't exceed 5B in size.

[0xShug0]

@HaujetZhao Both FireRed Audio and FireRed TTS3 provide speech editing. So does Aux’s speech editing have any unique advantages?

Sorry for being paranoid. I was burned pretty badly by Tencent’s LeVo2 (I think they actually took down the code and model). That’s why I want to see more user feedback instead of just trusting their (Tencent) demos.

[HaujetZhao]

I tried some tricks with Claude Code to run AUK on a local RTX 5050 with 8GB of VRAM. I loaded Omni 3B to get the latent, then unloaded it, and used Auk 1.5B for audio generation. But I couldn't try FireRed Audio—I could only use it on Hugging Face Space with a limited number of attempts. FireRed Audio is too large and consumes too much of the Hugging Face Space quota. I did a few speech editing tests, and FireRed Audio always seemed unable to understand what I meant, changing the wrong parts, whereas AUK could accurately perform segment edits. With only a limited number of attempts, I can only give this limited feedback. But based on my experience, I am satisfied with AUK.

If available, I would suggest you to use it yourself on Hugging Face space. Especially for voice editing, speaker isolating, and voice quality refining.

[HaujetZhao]

AuK has one known flaw so far: it cannot estimate on its own how long the generated audio needs to be. Instead, it requires an external tool or a human to set a target audio duration, and then it generates the audio accordingly.

[kooks28]

AuK has one known flaw so far: it cannot estimate on its own how long the generated audio needs to be. Instead, it requires an external tool or a human to set a target audio duration, and then it generates the audio accordingly.

This was an issue here as well trying to get something to calculate it good was nightmare... the model produces good quality other than this problem