support funaudiochat ST2T mode inference-only in vLLM, speed up around 20x+ in short audios (1-30s) ,and around 50x+ in long audios

[nemoramo]

vllm-project/vllm#33058

[Mousaic]

必须赞一个

[nemoramo]

@lxchtan Can you formally submit a PR to transformers (adding the model FunAudioChat-8B), so as the vLLM modelconfig type can be auto-recognized in the coming transformers version.

[lxchtan]

Thank you for your contribution. I will submit a PR to the transformers library in the next few days.

[nemoramo]

vllm-project/vllm@36d450e already merged in vLLM! Enjoy it.

[BadDeveloper2022]

@nemoramo 感谢分享，我想问下 多轮对话的时候怎么处理？ 文字+音频混合发过去吗

[BadDeveloper2022]

@nemoramo 比如我想实现 ： 一个语音聊天机器人，有上下文的记忆功能，通过VAD切出来之后，AI说话和用户说话的历史记录，怎样给vllm传过去最好，性能最好

[tuyen-tran1]

I would like to ask whether this version of vLLM can be used to run the conversation demo.
I have integrated it into the FunAudio Chat web demo, but the assistant’s responses are incorrect and noticeably less accurate than usual.

[nemoramo]

I would like to ask whether this version of vLLM can be used to run the conversation demo. I have integrated it into the FunAudio Chat web demo, but the assistant’s responses are incorrect and noticeably less accurate than usual.

can you share the comparison between the original version and vllm version. For same sdpa backend, these two versions share nearly same accuracy, but it's not that complete match since the vllm uses its own module. However you may share the way you get this incorrectness, so as i can help.

[DDJ-1Ray]

@nemoramo Hi!

How can I get the audio token with VLLM? It just gives me text token. This is my test code. `def test_with_audio():
url = f"{BASE_URL}/v1/chat/completions"

with open("ck7vv9ag.wav", "rb") as audio_file:
    audio_data = base64.b64encode(audio_file.read()).decode('utf-8')
payload = {
    "model": MODEL_NAME,
    "messages": [{
        "role": "system", "content": SPOKEN_S2M_PROMPT
    },{
        "role": "user",
        "content": [
            {"type": "text", "text": "请转录这段音频"},
            {
                "type": "input_audio",  # input_audio
                "input_audio": {
                    "data": audio_data,
                    "format": "wav"
                }
            }
        ]
        },{"role": "assistant", "content": None}],
    "max_tokens": 200,
    "skip_special_tokens": False,
}
response = requests.post(url, json=payload)
return response.json()

`

[nemoramo]

@nemoramo Hi!

How can I get the audio token with VLLM? It just gives me text token. This is my test code. `def test_with_audio(): url = f"{BASE_URL}/v1/chat/completions"

with open("ck7vv9ag.wav", "rb") as audio_file:
    audio_data = base64.b64encode(audio_file.read()).decode('utf-8')
payload = {
    "model": MODEL_NAME,
    "messages": [{
        "role": "system", "content": SPOKEN_S2M_PROMPT
    },{
        "role": "user",
        "content": [
            {"type": "text", "text": "请转录这段音频"},
            {
                "type": "input_audio",  # input_audio
                "input_audio": {
                    "data": audio_data,
                    "format": "wav"
                }
            }
        ]
        },{"role": "assistant", "content": None}],
    "max_tokens": 200,
    "skip_special_tokens": False,
}
response = requests.post(url, json=payload)
return response.json()

`

Sorry this is a ST2T mode only inference engine. Currently I am not working on TTS branch adapting.

[BadDeveloper2022]

v0.15.0 no releases 33058

[nemoramo]

v0.15.0 no releases 33058

v0.16.0rc0 should be in this tag right now

[nemoramo]

@lxchtan vllm-project/vllm-omni#1748 s2s mode supported in vllm-omni

[nemoramo]

@nemoramo Hi!

How can I get the audio token with VLLM? It just gives me text token. This is my test code. `def test_with_audio(): url = f"{BASE_URL}/v1/chat/completions"

with open("ck7vv9ag.wav", "rb") as audio_file:
    audio_data = base64.b64encode(audio_file.read()).decode('utf-8')
payload = {
    "model": MODEL_NAME,
    "messages": [{
        "role": "system", "content": SPOKEN_S2M_PROMPT
    },{
        "role": "user",
        "content": [
            {"type": "text", "text": "请转录这段音频"},
            {
                "type": "input_audio",  # input_audio
                "input_audio": {
                    "data": audio_data,
                    "format": "wav"
                }
            }
        ]
        },{"role": "assistant", "content": None}],
    "max_tokens": 200,
    "skip_special_tokens": False,
}
response = requests.post(url, json=payload)
return response.json()

`

hi maybe you can check this pr out. vllm-project/vllm-omni#1748

[eyree]

@nemoramo Hi!
How can I get the audio token with VLLM? It just gives me text token. This is my test code. `def test_with_audio(): url = f"{BASE_URL}/v1/chat/completions"

with open("ck7vv9ag.wav", "rb") as audio_file:
    audio_data = base64.b64encode(audio_file.read()).decode('utf-8')
payload = {
    "model": MODEL_NAME,
    "messages": [{
        "role": "system", "content": SPOKEN_S2M_PROMPT
    },{
        "role": "user",
        "content": [
            {"type": "text", "text": "请转录这段音频"},
            {
                "type": "input_audio",  # input_audio
                "input_audio": {
                    "data": audio_data,
                    "format": "wav"
                }
            }
        ]
        },{"role": "assistant", "content": None}],
    "max_tokens": 200,
    "skip_special_tokens": False,
}
response = requests.post(url, json=payload)
return response.json()

`

hi maybe you can check this pr out. vllm-project/vllm-omni#1748

Hello, I am trying to use your implementation from vllm-project/vllm-omni#1748 to accelerate the S2S functionality of funaudiochat, but I have encountered some issues. Could you please help take a look?

INFO 04-13 21:17:56 [orchestrator.py:810] [Orchestrator] Stage 1 shut down
ERROR 04-13 21:17:56 [async_omni_engine.py:579] [AsyncOmniEngine] Orchestrator thread crashed
ERROR 04-13 21:17:56 [async_omni_engine.py:579] Traceback (most recent call last):
ERROR 04-13 21:17:56 [async_omni_engine.py:579]   File "/ssd1/di/vllm-omni/vllm_omni/engine/async_omni_engine.py", line 575, in _bootstrap_orchestrator
ERROR 04-13 21:17:56 [async_omni_engine.py:579]     loop.run_until_complete(_run_orchestrator())
ERROR 04-13 21:17:56 [async_omni_engine.py:579]   File "/ssd1/miniconda3/lib/python3.12/asyncio/base_events.py", line 687, in run_until_complete
ERROR 04-13 21:17:56 [async_omni_engine.py:579]     return future.result()
ERROR 04-13 21:17:56 [async_omni_engine.py:579]            ^^^^^^^^^^^^^^^
ERROR 04-13 21:17:56 [async_omni_engine.py:579]   File "/ssd1/di/vllm-omni/vllm_omni/engine/async_omni_engine.py", line 572, in _run_orchestrator
ERROR 04-13 21:17:56 [async_omni_engine.py:579]     await orchestrator.run()
ERROR 04-13 21:17:56 [async_omni_engine.py:579]   File "/ssd1/di/vllm-omni/vllm_omni/engine/orchestrator.py", line 168, in run
ERROR 04-13 21:17:56 [async_omni_engine.py:579]     await asyncio.gather(request_task, output_task)
ERROR 04-13 21:17:56 [async_omni_engine.py:579]   File "/ssd1/di/vllm-omni/vllm_omni/engine/orchestrator.py", line 220, in _orchestration_output_handler
ERROR 04-13 21:17:56 [async_omni_engine.py:579]     await self._orchestration_loop()
ERROR 04-13 21:17:56 [async_omni_engine.py:579]   File "/ssd1/di/vllm-omni/vllm_omni/engine/orchestrator.py", line 293, in _orchestration_loop
ERROR 04-13 21:17:56 [async_omni_engine.py:579]     await self._route_output(stage_id, output, req_state, stage_metrics)
ERROR 04-13 21:17:56 [async_omni_engine.py:579]   File "/ssd1/di/vllm-omni/vllm_omni/engine/orchestrator.py", line 373, in _route_output
ERROR 04-13 21:17:56 [async_omni_engine.py:579]     await self._forward_to_next_stage(req_id, stage_id, output, req_state)
ERROR 04-13 21:17:56 [async_omni_engine.py:579]   File "/ssd1/di/vllm-omni/vllm_omni/engine/orchestrator.py", line 496, in _forward_to_next_stage
ERROR 04-13 21:17:56 [async_omni_engine.py:579]     next_inputs = next_client.process_engine_inputs(
ERROR 04-13 21:17:56 [async_omni_engine.py:579]                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 04-13 21:17:56 [async_omni_engine.py:579]   File "/ssd1/di/vllm-omni/vllm_omni/engine/stage_engine_core_client.py", line 124, in process_engine_inputs
ERROR 04-13 21:17:56 [async_omni_engine.py:579]     return self.custom_process_input_func(
ERROR 04-13 21:17:56 [async_omni_engine.py:579]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 04-13 21:17:56 [async_omni_engine.py:579]   File "/ssd1/di/vllm-omni/vllm_omni/model_executor/stage_input_processors/funaudiochat.py", line 58, in funaudiochat2code2wav
ERROR 04-13 21:17:56 [async_omni_engine.py:579]     raise ValueError("Stage-0 FunAudioChat output does not contain `speech_ids` or `audio_token_ids`.")
ERROR 04-13 21:17:56 [async_omni_engine.py:579] ValueError: Stage-0 FunAudioChat output does not contain `speech_ids` or `audio_token_ids`.

My code is:

from vllm_omni import AsyncOmni
from vllm_omni.outputs import OmniRequestOutput
async def run_funaudiochat_inference(
    omni: AsyncOmni,
    input_audio: np.ndarray,
    sample_rate: int,
    max_tokens: int = 2048,
    temperature: float = 0.0,
) -> OmniRequestOutput:
    """
    Run FunAudioChat speech-to-speech inference.
    
    Args:
        omni: AsyncOmni instance
        input_audio: Input audio as numpy array
        sample_rate: Sample rate of input audio
        max_tokens: Maximum tokens to generate
        temperature: Sampling temperature
    
    Returns:
        OmniRequestOutput with generated audio
    """
    # Prepare input prompt with audio
    prompt = "<|audio_bos|><|AUDIO|><|audio_eos|>"
    
    # Prepare multi-modal data
    inputs = {
        "prompt": prompt,
        "multi_modal_data": {
            "audio": input_audio,
        },
    }
    
    # Sampling parameters (for each stage)
    from vllm import SamplingParams
    
    sampling_params_list = [
        # Stage 0: Speech-to-speech generation
        SamplingParams(
            temperature=temperature,
            top_p=1.0,
            top_k=-1,
            repetition_penalty=1.2,
            max_tokens=max_tokens,
            stop_token_ids=[151645],  # Stop token for FunAudioChat
        ),
        # Stage 1: Code2wav (audio token decoding)
        SamplingParams(
            temperature=0.0,
            top_p=1.0,
            top_k=-1,
            max_tokens=1,
        ),
    ]
    
    logger.info("Starting speech-to-speech inference...")
    
    # Run inference
    request_id = "funaudiochat_s2s_request"
    final_output = None
    
    async for request_output in omni.generate(
        prompt=inputs,
        request_id=request_id,
        sampling_params_list=sampling_params_list,
    ):
        final_output = request_output
    
    return final_output


def extract_audio_from_output(output: OmniRequestOutput) -> tuple[np.ndarray | None, int | None]:
    """
    Extract generated audio from OmniRequestOutput.
    
    Args:
        output: OmniRequestOutput from AsyncOmni
    
    Returns:
        Tuple of (audio_array, sample_rate) or (None, None) if no audio
    """
    if not hasattr(output, 'outputs') or not output.outputs:
        logger.warning("No outputs in OmniRequestOutput")
        return None, None
    
    for out in output.outputs:
        multimodal_output = getattr(out, "multimodal_output", {})
        if multimodal_output and "audio" in multimodal_output:
            audio_data = multimodal_output["audio"]
            
            # Handle different audio data formats
            if isinstance(audio_data, dict):
                audio_array = audio_data.get("data")
                sample_rate = audio_data.get("sample_rate", 24000)
            elif isinstance(audio_data, (list, tuple)) and len(audio_data) == 2:
                sample_rate, audio_array = audio_data
            elif isinstance(audio_data, np.ndarray):
                audio_array = audio_data
                sample_rate = 24000  # Default for CosyVoice3
            else:
                logger.warning(f"Unexpected audio format: {type(audio_data)}")
                continue
            
            if audio_array is not None:
                # Convert torch.Tensor to numpy if needed
                if TORCH_AVAILABLE and torch is not None and isinstance(audio_array, torch.Tensor):
                    audio_array = audio_array.cpu().numpy()
                elif hasattr(audio_array, 'cpu') and hasattr(audio_array, 'numpy'):
                    # Fallback for torch-like objects
                    audio_array = audio_array.cpu().numpy()
                
                logger.info(f"Extracted audio: shape={audio_array.shape}, sr={sample_rate}")
                return audio_array, sample_rate
    
    logger.warning("No audio found in outputs")
    return None, None


async def main():
    parser = argparse.ArgumentParser(
        description="FunAudioChat Speech-to-Speech Inference",
        formatter_class=argparse.RawDescriptionHelpFormatter,
    )
    parser.add_argument(
        "--input-audio",
        type=str,
        required=True,
        help="Input audio file path",
    )
    parser.add_argument(
        "--output-audio",
        type=str,
        default="output_funaudiochat.wav",
        help="Output audio file path (default: output_funaudiochat.wav)",
    )
    parser.add_argument(
        "--stage-config",
        type=str,
        default=None,
        help="Path to stage config YAML file (default: auto-detect from model)",
    )
    parser.add_argument(
        "--max-tokens",
        type=int,
        default=2048,
        help="Maximum tokens to generate (default: 2048)",
    )
    parser.add_argument(
        "--temperature",
        type=float,
        default=0.0,
        help="Sampling temperature (default: 0.0 for greedy)",
    )
    parser.add_argument(
        "--gpu-memory-utilization",
        type=float,
        default=0.8,
        help="GPU memory utilization (default: 0.8)",
    )
    args = parser.parse_args()
    
    # Check FUN_AUDIO_CHAT_HOME environment variable
    if "FUN_AUDIO_CHAT_HOME" not in os.environ:
        logger.warning(
            "FUN_AUDIO_CHAT_HOME not set. Please set it to the Fun-Audio-Chat repository path.\n"
            "Example: export FUN_AUDIO_CHAT_HOME=/path/to/Fun-Audio-Chat"
        )
    
    # Determine stage config path
    if args.stage_config:
        stage_config = args.stage_config
    else:
        # Use default config from repository
        repo_root = Path(__file__).resolve().parents[3]
        stage_config = str(repo_root / "vllm_omni" / "model_executor" / "stage_configs" / "funaudiochat.yaml")
    
    if not Path(stage_config).exists():
        raise FileNotFoundError(f"Stage config not found: {stage_config}")
    
    logger.info(f"Using stage config: {stage_config}")
    
    # Load input audio
    input_audio, input_sr = load_audio(args.input_audio)
    
    # Initialize AsyncOmni engine with stage config
    logger.info("Initializing AsyncOmni...")
    omni = AsyncOmni(
        model=stage_config,
        stage_configs_path=stage_config,
        trust_remote_code=True,
        log_stats=True,
    )
    
    try:
        # Run inference
        output = await run_funaudiochat_inference(
            omni=omni,
            input_audio=input_audio,
            sample_rate=input_sr,
            max_tokens=args.max_tokens,
            temperature=args.temperature,
        )
        
        # Extract generated audio
        audio_array, audio_sr = extract_audio_from_output(output)
        
        if audio_array is not None and audio_sr is not None:
            # Save output audio
            save_audio(audio_array, audio_sr, args.output_audio)
            logger.info("✓ Speech-to-speech inference completed successfully!")
        else:
            logger.error("Failed to extract audio from output")
            
            # Print debug information
            logger.info("Output structure:")
            if hasattr(output, 'outputs'):
                for idx, out in enumerate(output.outputs):
                    logger.info(f"  Output {idx}:")
                    logger.info(f"    - text: {out.text[:100] if out.text else 'None'}")
                    logger.info(f"    - multimodal_output keys: {list(getattr(out, 'multimodal_output', {}).keys())}")
    
    finally:
        # Cleanup
        logger.info("Shutting down AsyncOmni...")
        omni.shutdown()


if __name__ == "__main__":
    asyncio.run(main())