Add support for Yue2

[Lastofthefirst]

A new music generation model was released today:

https://map-yue2.github.io/

[0xShug0]

ah checked a few days ago and the last update was a year ago. I will take this.

[tianshuo]

The demos sounds amazing, esp. Suno V6 is restricting.

[WilliamLiao2015]

Add the GitHub repo link of YuE2 here in case anyone doesn't find it: https://github.com/multimodal-art-projection/YuE

[0xShug0]

Call it a day and continue optimizing tomorrow. Now RTF is cpp 0.3 vs python 0.31. ~14G VRAM. using the official example https://huggingface.co/m-a-p/YuE2-3B/blob/main/examples/tonight-awake.json.

Update: in dev now.

yue2_demo.mp4

cpp_current_128k.mp3

[0xShug0]

I think Q4_0 is safe. Added Q4_0 demos to the HF repo. Current perf:

Combo	Audio dur	Wall	RTF	Peak VRAM
BF16 main + F32 VAE	224.96s	60.46s	0.2688	12535 MiB
Q8_0 main + F16 VAE	194.84s	38.81s	0.1992	8867 MiB
Q4_0 main + F16 VAE	221.12s	44.15s	0.1997	7755 MiB

[0xShug0]

@IIIIIllllIIIIIlllll Let me know if it can run on AMD :). This model is pretty impressive.

Check this out: heavy metal jingle bells
Jingle_bells.wav

[IIIIIllllIIIIIlllll]

@IIIIIllllIIIIIlllll Let me know if it can run on AMD :). This model is pretty impressive.

Check this out: heavy metal jingle bells Jingle_bells.wav

no problem ;)

[prookyon]

Do you also plan to add support for https://huggingface.co/m-a-p/SheetSage2 - their model for creating ABC sheets to support cover functionality?

[overallbit]

Call it a day and continue optimizing tomorrow. Now RTF is cpp 0.3 vs python 0.31. ~14G VRAM. using the official example https://huggingface.co/m-a-p/YuE2-3B/blob/main/examples/tonight-awake.json.

Update: in dev now.
yue2_demo.mp4

cpp_current_128k.mp3

is this on windows? can you upload it?

[0xShug0]

@tianshuo @prookyon Yue2 on dev already supports ABC input: you can pass an existing score.abc with abc_file or inline abc when using cot=melody / cot=full. ABC export/edit/re-import is more of a UI-side workflow and hasn’t been implemented yet. Happy to support this once the main model impl is stabilized.

SheetSage 2 inference has been implemented https://github.com/0xShug0/audio.cpp/blob/dev/src/framework/midi/sheetsage2_runtime.cpp. It will be exposed as a standalone model later.

[0xShug0]

@overallbit Building now! Once finished, you can find them in the “Artifacts” section on this page: https://github.com/0xShug0/audio.cpp/actions/runs/34642627647 (default step is 32. 8 is good enough)

[zestybaby]

Love this one so far using Windows, but if I use ABC score feature to generate a song, my system drive's remaining space would shrunk to less than 200 MB from 8 gigs left in usual.

Also, can you adapt the Cover feature from YuE 2's original WebUI into this project with SheetSage2 and MERT2?

[IIIIIllllIIIIIlllll]

@IIIIIllllIIIIIlllll Let me know if it can run on AMD :). This model is pretty impressive.

Check this out: heavy metal jingle bells Jingle_bells.wav

The same prompt: https://huggingface.co/m-a-p/YuE2-3B/blob/main/examples/tonight-awake.json

BF16 weight + F32 vae

RTF: ~0.5

It's so big I can't upload. Are there any other prompts I can test?