Repository navigation
5GB memory allocated up front #103
Description
Activity
Which model?
Does #105 help? It makes the model weights disk-backed, so that they may swap to disk if RAM is insufficient. Otherwise, consider choosing a smaller model such as
Kokoro_espeak_Q4.gguf, which uses merely ~1.4GB max for a small prompt.Reacted by aPaleBlueDotI tried the q5 of parler mini, although I decided against using Parler after reading in an issue on this repo that it mumbles. Yes, mmap would immensely help, and could be acceptable with a few of GB of swap.
Yeah about that, Kokoro is the only practical model at this time. The others seem academic because they are too large or unreliable.
I've had discussions about this spread across PRs and issues. I'll updateREADME.mdto make this fact more visible.Some users caught Kokoro saying gibberish for out of training data, so I can't risk it.
Which model are you using now?
Undecided, still searching.
With Parler and your settings, I'm getting 2.5455017 GiB RssAnon in the middle of
dac_runner::run. This might be a problem because a comment sayssplitting this out from the primary graph so that we can better manage streaming (i.e. sentence chunks are better performed this way)but I don't see any streaming implemented.
Possibly this isn't an issue, but I could not figure out how to not have 5GB memory allocated up front, despite tweaking several numbers.