Persistent ASR + STT

[ser]

I am loading two models with docker with server --load-all:

{
  "host": "0.0.0.0",
  "port": 5552,
  "backend": "cuda",
  "device": 0,
  "threads": 8,
  "lazy_load": false,
  "max_loaded_models": 4,
  "log_request_body": false,
  "max_request_body_bytes": 2147483648,
  "voice_dir": "/voices",
  "models": [
    {
      "id": "parakeet-tdt",
      "family": "parakeet_tdt",
      "path": "/app/models/Parakeet-TDT-0.6B-v3-GGUF/",
      "task": "asr",
      "mode": "streaming"
    },
    {
      "id": "omnivoice",
      "family": "omnivoice",
      "path": "/app/models/OmniVoice-GGUF/",
      "task": "tts",
      "mode": "offline"
    }
  ]
}

How can I persist these two models in memory forever, to be hot-active all times?

[0xShug0]

@ser I think your config already does this. max_loaded_models: 4 is enough for two models, so it should not evict either one.

To be safer maybe use

{
  "lazy_load": false,
  "max_loaded_models": 0,
  "idle_unload_ms": 0
}

[ser]

Thanks! I will carefully test it report back.

[ser]

Hello, I still can see that models are not hot-ready after a while - I did an ASR at the evening and in the morning it took about 20 seconds to start another ASR. I do not understand the mechanism which unloads "unused" models.

Should we investigate it together or better choice is to write a cron which hits the audio.cpp instance regularly to keep the models hot?

[0xShug0]

it took about 20 seconds to start another ASR.

Hi @ser Could you share the logs? I’m trying to tell whether this is an actual unload/reload, a container restart, or just a cold prepare()/CUDA warmup path while the model is still resident.

[ser]

I am attaching whole logs from docker, the container itself does not restart.

[TRACE ts=20260907-165850] runtime.model.supports_style_condition 0
[TRACE ts=20260907-165850] runtime.model.supports_timestamps 1
[TIMING ts=20260907-165857] parakeet_tdt.weights.upload_ms 7419.362109
[TRACE ts=20260907-165858] server.model.id omnivoice
[TRACE ts=20260907-165858] server.model.path /app/models/OmniVoice-GGUF/
[TRACE ts=20260907-165858] server.model.family omnivggml_cuda_graph_set_enabled: disabling CUDA graphs due to GPU architecture
oice
[TRACE ts=20260907-165858] server.model.task tts
[TRACE ts=20260907-165858] server.model.mode offline
[TRACE ts=20260907-165858] server.model.backend cuda
[TRACE ts=20260907-165858] server.model.device 0
[TRACE ts=20260907-165858] server.model.threads 8
ggml_cuda_graph_set_enabled: disabling CUDA graphs due to GPU architecture
ggml_cuda_graph_set_enabled: disabling CUDA graphs due to GPU architecture
ggml_cuda_graph_set_enabled: disabling CUDA graphs due to GPU architecture
ggml_cuda_graph_set_enabled: disabling CUDA graphs due to GPU architecture
ggml_cuda_graph_set_enabled: disabling CUDA graphs due to GPU architecture
[TRACE ts=20260907-165858] server.model.session_option_count 0
[TRACE ts=20260907-165900] runtime.model.family omnivoice
[TRACE ts=20260907-165900] runtime.model.variant omnivoice
[TRACE ts=20260907-165900] runtime.model.root /tmp/audiocpp-gguf/1c5e726ee95fde33
[TRACE ts=20260907-165900] runtime.model.discovered_config_count 6
[TRACE ts=20260907-165900] runtime.model.discovered_weight_count 2
[TRACE ts=20260907-165900] runtime.model.task_count 1
[TRACE ts=20260907-165900] runtime.model.language_count 1
[TRACE ts=20260907-165900] runtime.model.supports_speaker_reference 1
[TRACE ts=20260907-165900] runtime.model.supports_style_condition 1
[TRACE ts=20260907-165900] runtime.model.supports_timestamps 0
[TRACE ts=20260907-165907] omnivoice.perf_mode off
audiocpp_server listening on http://0.0.0.0:5552
:ggml_cuda_graph_set_enabled: disabling CUDA graphs due to GPU architecture
ggml_cuda_graph_set_enabled: disabling CUDA graphs due to GPU architecture
ggml_cuda_graph_set_enabled: disabling CUDA graphs due to GPU architecture
ggml_cuda_graph_set_enabled: disabling CUDA graphs due to GPU architecture
ggml_cuda_graph_set_enabled: disabling CUDA graphs due to GPU architecture
ggml_cuda_graph_set_enabled: disabling CUDA graphs due to GPU architecture
ggml_cuda_graph_set_enabled: disabling CUDA graphs due to GPU architecture
ggml_cuda_graph_set_enabled: disabling CUDA graphs due to GPU architecture
ggml_cuda_graph_set_enabled: disabling CUDA graphs due to GPU architecture
ggml_cuda_graph_set_enabled: disabling CUDA graphs due to GPU architecture
ggml_cuda_graph_set_enabled: disabling CUDA graphs due to GPU architecture
[TRACE ts=20260908-023630] parakeet_tdt.encoder.graph_optimizer.nodes_before 2349
[TRACE ts=20260908-023630] parakeet_tdt.encoder.graph_optimizer.nodes_after 2243
[TIMING ts=20260908-023630] parakeet_tdt.encoder.graph_alloc_ms 9.770534
[TIMING ts=20260908-023630] parakeet_tdt.encoder.pos_upload_ms 14.525637
[TIMING ts=20260908-023640] parakeet_tdt.encoder.pos_compute_ms 10025.641404
[TIMING ts=20260908-023640] parakeet_tdt.encoder.pos_copy_ms 0.404922
[TIMING ts=20260908-023640] parakeet_tdt.encoder.graph_build_ms 10066.826662
[TIMING ts=20260908-023640] parakeet_tdt.encoder.graph_rebuild_ms 10066.826662
[TRACE ts=20260908-023640] parakeet_tdt.encoder.graph_cache_hit 0
[TRACE ts=20260908-023640] parakeet_tdt.encoder.graph_input_frames 1401
[TRACE ts=20260908-023640] parakeet_tdt.encoder.graph_encoded_frames 176
[TIMING ts=20260908-023640] parakeet.decoder.graph_build_ms 0.380452
[TIMING ts=20260908-023640] parakeet.decoder.joint_graph_build_ms 0.033151
[TIMING ts=20260908-023640] parakeet_tdt.prepare_ms 10068.546423
[TIMING ts=20260908-023640] parakeet.frontend_ms 26.665416
[TRACE ts=20260908-023640] parakeet.frontend.frames 137
[TRACE ts=20260908-023640] parakeet.frontend.valid_frames 136
[TRACE ts=20260908-023640] parakeet.frontend.center 1
[TRACE ts=20260908-023640] parakeet_tdt.encoder.graph_optimizer.nodes_before 2349
[TRACE ts=20260908-023640] parakeet_tdt.encoder.graph_optimizer.nodes_after 2243
[TIMING ts=20260908-023640] parakeet_tdt.encoder.graph_alloc_ms 0.607638
[TIMING ts=20260908-023640] parakeet_tdt.encoder.pos_upload_ms 0.439453
[TIMING ts=20260908-023640] parakeet_tdt.encoder.pos_compute_ms 0.803324
[TIMING ts=20260908-023640] parakeet_tdt.encoder.pos_copy_ms 0.183736
[TIMING ts=20260908-023640] parakeet_tdt.encoder.graph_build_ms 9.093252
[TIMING ts=20260908-023640] parakeet_tdt.encoder.graph_rebuild_ms 9.093252
[TRACE ts=20260908-023640] parakeet_tdt.encoder.graph_cache_hit 0
[TRACE ts=20260908-023640] parakeet_tdt.encoder.graph_input_frames 137
[TRACE ts=20260908-023640] parakeet_tdt.encoder.graph_encoded_frames 18
[TIMING ts=20260908-023643] parakeet_tdt.encoder.graph.compute_ms 3145.223057
[TIMING ts=20260908-023643] parakeet_tdt.encoder_ms 3155.179424
[TRACE ts=20260908-023643] parakeet_tdt.encoder.valid_frames 17
[TIMING ts=20260908-023708] parakeet.decoder_ms 24380.823668
[TRACE ts=20260908-024527] parakeet_tdt.encoder.graph_optimizer.nodes_before 2349
[TRACE ts=20260908-024527] parakeet_tdt.encoder.graph_optimizer.nodes_after 2243
[TIMING ts=20260908-024527] parakeet_tdt.encoder.graph_alloc_ms 1.081835
[TIMING ts=20260908-024527] parakeet_tdt.encoder.pos_upload_ms 15.47591
[TIMING ts=20260908-024527] parakeet_tdt.encoder.pos_compute_ms 13.937364
[TIMING ts=20260908-024527] parakeet_tdt.encoder.pos_copy_ms 0.882561
[TIMING ts=20260908-024527] parakeet_tdt.encoder.graph_build_ms 35.623828
[TIMING ts=20260908-024527] parakeet_tdt.encoder.graph_rebuild_ms 35.623828
[TRACE ts=20260908-024527] parakeet_tdt.encoder.graph_cache_hit 0
[TRACE ts=20260908-024527] parakeet_tdt.encoder.graph_input_frames 1401
[TRACE ts=20260908-024527] parakeet_tdt.encoder.graph_encoded_frames 176
[TIMING ts=20260908-024527] parakeet_tdt.prepare_ms 35.644829
[TIMING ts=20260908-024527] parakeet.frontend_ms 32.535156
[TRACE ts=20260908-024527] parakeet.frontend.frames 111
[TRACE ts=20260908-024527] parakeet.frontend.valid_frames 110
[TRACE ts=20260908-024527] parakeet.frontend.center 1
[TRACE ts=20260908-024527] parakeet_tdt.encoder.graph_optimizer.nodes_before 2349
[TRACE ts=20260908-024527] parakeet_tdt.encoder.graph_optimizer.nodes_after 2243
[TIMING ts=20260908-024527] parakeet_tdt.encoder.graph_alloc_ms 0.778508
[TIMING ts=20260908-024527] parakeet_tdt.encoder.pos_upload_ms 0.443181
[TIMING ts=20260908-024527] parakeet_tdt.encoder.pos_compute_ms 0.895791
[TIMING ts=20260908-024527] parakeet_tdt.encoder.pos_copy_ms 0.170914
[TIMING ts=20260908-024527] parakeet_tdt.encoder.graph_build_ms 8.332313
[TIMING ts=20260908-024527] parakeet_tdt.encoder.graph_rebuild_ms 8.332313
[TRACE ts=20260908-024527] parakeet_tdt.encoder.graph_cache_hit 0
[TRACE ts=20260908-024527] parakeet_tdt.encoder.graph_input_frames 111
[TRACE ts=20260908-024527] parakeet_tdt.encoder.graph_encoded_frames 14
[TIMING ts=20260908-024527] parakeet_tdt.encoder.graph.compute_ms 19.214896
[TIMING ts=20260908-024527] parakeet_tdt.encoder_ms 27.691973
[TRACE ts=20260908-024527] parakeet_tdt.encoder.valid_frames 14
[TIMING ts=20260908-024527] parakeet.decoder_ms 2.066728
[TRACE ts=20260908-024556] parakeet_tdt.encoder.graph_optimizer.nodes_before 2349
[TRACE ts=20260908-024556] parakeet_tdt.encoder.graph_optimizer.nodes_after 2243
[TIMING ts=20260908-024556] parakeet_tdt.encoder.graph_alloc_ms 2.492007
[TIMING ts=20260908-024556] parakeet_tdt.encoder.pos_upload_ms 18.422524
[TIMING ts=20260908-024556] parakeet_tdt.encoder.pos_compute_ms 13.334097
[TIMING ts=20260908-024556] parakeet_tdt.encoder.pos_copy_ms 0.89014
[TIMING ts=20260908-024556] parakeet_tdt.encoder.graph_build_ms 45.376055
[TIMING ts=20260908-024556] parakeet_tdt.encoder.graph_rebuild_ms 45.376055
[TRACE ts=20260908-024556] parakeet_tdt.encoder.graph_cache_hit 0
[TRACE ts=20260908-024556] parakeet_tdt.encoder.graph_input_frames 1401
[TRACE ts=20260908-024556] parakeet_tdt.encoder.graph_encoded_frames 176
[TIMING ts=20260908-024556] parakeet_tdt.prepare_ms 45.399755
[TIMING ts=20260908-024556] parakeet.frontend_ms 17.506353
[TRACE ts=20260908-024556] parakeet.frontend.frames 401
[TRACE ts=20260908-024556] parakeet.frontend.valid_frames 400
[TRACE ts=20260908-024556] parakeet.frontend.center 1
[TRACE ts=20260908-024556] parakeet_tdt.encoder.graph_optimizer.nodes_before 2349
[TRACE ts=20260908-024556] parakeet_tdt.encoder.graph_optimizer.nodes_after 2243
[TIMING ts=20260908-024556] parakeet_tdt.encoder.graph_alloc_ms 0.658935
[TIMING ts=20260908-024556] parakeet_tdt.encoder.pos_upload_ms 0.735047
[TIMING ts=20260908-024556] parakeet_tdt.encoder.pos_compute_ms 2.419446
[TIMING ts=20260908-024556] parakeet_tdt.encoder.pos_copy_ms 0.301867
[TIMING ts=20260908-024556] parakeet_tdt.encoder.graph_build_ms 11.24582
[TIMING ts=20260908-024556] parakeet_tdt.encoder.graph_rebuild_ms 11.24582
[TRACE ts=20260908-024556] parakeet_tdt.encoder.graph_cache_hit 0
[TRACE ts=20260908-024556] parakeet_tdt.encoder.graph_input_frames 401
[TRACE ts=20260908-024556] parakeet_tdt.encoder.graph_encoded_frames 51
[TIMING ts=20260908-024556] parakeet_tdt.encoder.graph.compute_ms 26.762587
[TIMING ts=20260908-024556] parakeet_tdt.encoder_ms 39.009799
[TRACE ts=20260908-024556] parakeet_tdt.encoder.valid_frames 50
[TIMING ts=20260908-024556] parakeet.decoder_ms 2.018347
[TIMING ts=20260908-024556] parakeet.frontend_ms 30.469252
[TRACE ts=20260908-024556] parakeet.frontend.frames 601
[TRACE ts=20260908-024556] parakeet.frontend.valid_frames 600
[TRACE ts=20260908-024556] parakeet.frontend.center 1
[TRACE ts=20260908-024556] parakeet_tdt.encoder.graph_optimizer.nodes_before 2349
[TRACE ts=20260908-024556] parakeet_tdt.encoder.graph_optimizer.nodes_after 2243
[TIMING ts=20260908-024556] parakeet_tdt.encoder.graph_alloc_ms 0.737867
[TIMING ts=20260908-024556] parakeet_tdt.encoder.pos_upload_ms 0.948672
[TIMING ts=20260908-024556] parakeet_tdt.encoder.pos_compute_ms 1.338141
[TIMING ts=20260908-024556] parakeet_tdt.encoder.pos_copy_ms 0.251846
[TIMING ts=20260908-024556] parakeet_tdt.encoder.graph_build_ms 8.830613
[TIMING ts=20260908-024556] parakeet_tdt.encoder.graph_rebuild_ms 8.830613
[TRACE ts=20260908-024556] parakeet_tdt.encoder.graph_cache_hit 0
[TRACE ts=20260908-024556] parakeet_tdt.encoder.graph_input_frames 601
[TRACE ts=20260908-024556] parakeet_tdt.encoder.graph_encoded_frames 76
[TIMING ts=20260908-024556] parakeet_tdt.encoder.graph.compute_ms 32.11524
[TIMING ts=20260908-024556] parakeet_tdt.encoder_ms 41.934026
[TRACE ts=20260908-024556] parakeet_tdt.encoder.valid_frames 75
[TIMING ts=20260908-024556] parakeet.decoder_ms 0.574024
[TIMING ts=20260908-024556] parakeet.frontend_ms 45.790895
[TRACE ts=20260908-024556] parakeet.frontend.frames 801
[TRACE ts=20260908-024556] parakeet.frontend.valid_frames 800
[TRACE ts=20260908-024556] parakeet.frontend.center 1
[TRACE ts=20260908-024556] parakeet_tdt.encoder.graph_optimizer.nodes_before 2349
[TRACE ts=20260908-024556] parakeet_tdt.encoder.graph_optimizer.nodes_after 2243
[TIMING ts=20260908-024556] parakeet_tdt.encoder.graph_alloc_ms 0.746997
[TIMING ts=20260908-024556] parakeet_tdt.encoder.pos_upload_ms 1.238699
[TIMING ts=20260908-024556] parakeet_tdt.encoder.pos_compute_ms 1.479874
[TIMING ts=20260908-024556] parakeet_tdt.encoder.pos_copy_ms 0.277716
[TIMING ts=20260908-024556] parakeet_tdt.encoder.graph_build_ms 10.86721
[TIMING ts=20260908-024556] parakeet_tdt.encoder.graph_rebuild_ms 10.86721
[TRACE ts=20260908-024556] parakeet_tdt.encoder.graph_cache_hit 0
[TRACE ts=20260908-024556] parakeet_tdt.encoder.graph_input_frames 801
[TRACE ts=20260908-024556] parakeet_tdt.encoder.graph_encoded_frames 101
[TIMING ts=20260908-024556] parakeet_tdt.encoder.graph.compute_ms 33.306408
[TIMING ts=20260908-024556] parakeet_tdt.encoder_ms 45.13964
[TRACE ts=20260908-024556] parakeet_tdt.encoder.valid_frames 100
[TIMING ts=20260908-024556] parakeet.decoder_ms 0.630505
[TIMING ts=20260908-024556] parakeet.frontend_ms 16.5165
[TRACE ts=20260908-024556] parakeet.frontend.frames 815
[TRACE ts=20260908-024556] parakeet.frontend.valid_frames 814
[TRACE ts=20260908-024556] parakeet.frontend.center 1
[TRACE ts=20260908-024556] parakeet_tdt.encoder.graph_optimizer.nodes_before 2349
[TRACE ts=20260908-024556] parakeet_tdt.encoder.graph_optimizer.nodes_after 2243
[TIMING ts=20260908-024556] parakeet_tdt.encoder.graph_alloc_ms 0.765508
[TIMING ts=20260908-024556] parakeet_tdt.encoder.pos_upload_ms 1.158357
[TIMING ts=20260908-024556] parakeet_tdt.encoder.pos_compute_ms 1.488234
[TIMING ts=20260908-024556] parakeet_tdt.encoder.pos_copy_ms 0.279887
[TIMING ts=20260908-024556] parakeet_tdt.encoder.graph_build_ms 9.395547
[TIMING ts=20260908-024556] parakeet_tdt.encoder.graph_rebuild_ms 9.395547
[TRACE ts=20260908-024556] parakeet_tdt.encoder.graph_cache_hit 0
[TRACE ts=20260908-024556] parakeet_tdt.encoder.graph_input_frames 815
[TRACE ts=20260908-024556] parakeet_tdt.encoder.graph_encoded_frames 102
[TIMING ts=20260908-024556] parakeet_tdt.encoder.graph.compute_ms 33.292447
[TIMING ts=20260908-024556] parakeet_tdt.encoder_ms 43.649426
[TRACE ts=20260908-024556] parakeet_tdt.encoder.valid_frames 102
[TIMING ts=20260908-024556] parakeet.decoder_ms 0.895851
[TIMING ts=20260908-024556] parakeet.frontend_ms 25.859306
[TRACE ts=20260908-024556] parakeet.frontend.frames 815
[TRACE ts=20260908-024556] parakeet.frontend.valid_frames 814
[TRACE ts=20260908-024556] parakeet.frontend.center 1
[TIMING ts=20260908-024556] parakeet_tdt.encoder.graph_rebuild_ms 0
[TRACE ts=20260908-024556] parakeet_tdt.encoder.graph_cache_hit 1
[TIMING ts=20260908-024556] parakeet_tdt.encoder.graph.compute_ms 33.043491
[TIMING ts=20260908-024556] parakeet_tdt.encoder_ms 33.333578
[TRACE ts=20260908-024556] parakeet_tdt.encoder.valid_frames 102
[TIMING ts=20260908-024556] parakeet.decoder_ms 0.111612

[0xShug0]

The log shows parakeet_tdt.weights.upload_ms only during server startup, and there is no later unload/reload entry. The 10s is from here and it is graph-build cold-start time

[TIMING ts=20260908-023640] parakeet_tdt.encoder.pos_compute_ms 10025.641404
[TIMING ts=20260908-023640] parakeet_tdt.encoder.pos_copy_ms 0.404922
[TIMING ts=20260908-023640] parakeet_tdt.encoder.graph_build_ms 10066.826662
[TIMING ts=20260908-023640] parakeet_tdt.encoder.graph_rebuild_ms 10066.826662

lazy_load=false keeps the model weights/session resident, but it does not fully warm all request-shaped graphs. Parakeet builds/warmups those graphs during real requests, so if you need it hot after idle/startup, you should warm it with representative ASR requests.