[易用性] 测试url时没有host_ip和host_port也必须要填 #609

操作系统及版本

Ubuntu 24.04.5 LTS

安装工具的python环境

docker容器中的python环境

python版本

3.11

AISBench工具版本

3.1.20260630

AISBench执行命令

ais_bench --models vllm_api_general_chat --datasets gpqa_gen_0_shot_cot_chat_prompt

模型配置文件或自定义配置文件内容

from ais_bench.benchmark.models import VLLMCustomAPIChat
from ais_bench.benchmark.utils.postprocess.model_postprocessors import extract_non_reasoning_content

models = [
dict(
attr="service",
type=VLLMCustomAPIChat,
abbr="vllm-api-general-chat",
path="",
model="MiniMax-M3.1-Flash",
stream=True,
request_rate=0,
use_timestamp=False,
retry=2,
api_key="",
host_ip="",
host_port="",
url="模型对外的url链接",
max_out_len=128000,
batch_size=4,
trust_remote_code=False,
generation_kwargs=dict(
temperature=0.6,
top_p=0.95,
chat_template_kwargs=dict(reasoning_effort="max"),
),
pred_postprocessor=dict(type=extract_non_reasoning_content),
)
]

预期行为

成功跑完测试
但我随意输入host_ip和 host_port 就行了
from ais_bench.benchmark.models import VLLMCustomAPIChat
from ais_bench.benchmark.utils.postprocess.model_postprocessors import extract_non_reasoning_content

models = [
dict(
attr="service",
type=VLLMCustomAPIChat,
abbr="vllm-api-general-chat",
path="",
model="MiniMax-M3.1-Flash",
stream=True,
request_rate=0,
use_timestamp=False,
retry=2,
api_key="",
host_ip="9.9.9.9",
host_port=12345,
url="模型对外的url链接",
max_out_len=128000,
batch_size=4,
trust_remote_code=False,
generation_kwargs=dict(
temperature=0.6,
top_p=0.95,
chat_template_kwargs=dict(reasoning_effort="max"),
),
pred_postprocessor=dict(type=extract_non_reasoning_content),
)
]

实际行为

报错

/usr/local/python3.11.10/lib/python3.11/site-packages/triton/language/extra/cann/extension/math_ops.py:13: FutureWarning: triton.language.extra.cann.extension.atan2 is deprecated and will be removed in the next release; use triton.language.extra.cann.libdevice.atan2 instead.
@_deprecated(replacement="triton.language.extra.cann.libdevice.atan2")
/usr/local/python3.11.10/lib/python3.11/site-packages/triton/language/extra/cann/extension/math_ops.py:35: FutureWarning: triton.language.extra.cann.extension.isfinited is deprecated and will be removed in the next release; use triton.language.extra.cann.libdevice.isfinited instead.
@_deprecated(replacement="triton.language.extra.cann.libdevice.isfinited")
/usr/local/python3.11.10/lib/python3.11/site-packages/triton/language/extra/cann/extension/math_ops.py:49: FutureWarning: triton.language.extra.cann.extension.finitef is deprecated and will be removed in the next release; use triton.language.extra.cann.libdevice.finitef instead.
@_deprecated(replacement="triton.language.extra.cann.libdevice.finitef")
[2026-10-22 21:49:22,219] [ais_bench] [�[32mINFO�[0m] Task [vllm-api-general-chat/GPQA_diamond]
[2026-10-22 21:49:22,221] [ais_bench] [�[33mWARNING�[0m] Model config validation failed for VLLMCustomAPIChat: {'host_ip': 'host_ip must be a valid IPv4 or IPv6 address', 'host_port': 'host_port must be a valid port number in the range (0, 65536)'}
Traceback (most recent call last):
File "/usr/local/python3.11.10/lib/python3.11/site-packages/ais_bench/benchmark/tasks/openicl_api_infer.py", line 704, in
raise e
File "/usr/local/python3.11.10/lib/python3.11/site-packages/ais_bench/benchmark/tasks/openicl_api_infer.py", line 701, in
inferencer.run(task_state_manager)
File "/usr/local/python3.11.10/lib/python3.11/site-packages/ais_bench/benchmark/tasks/openicl_api_infer.py", line 477, in run
self.inferencer:BaseApiInferencer = ICL_INFERENCERS.build(self.inferencer_cfg)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/python3.11.10/lib/python3.11/site-packages/mmengine/registry/registry.py", line 570, in build
return self.build_func(cfg, *args, **kwargs, registry=self)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/python3.11.10/lib/python3.11/site-packages/mmengine/registry/build_functions.py", line 121, in build_from_cfg
obj = obj_cls(**args) # type: ignore
^^^^^^^^^^^^^^^
File "/usr/local/python3.11.10/lib/python3.11/site-packages/ais_bench/benchmark/openicl/icl_inferencer/icl_gen_inferencer.py", line 45, in init
super().init(
File "/usr/local/python3.11.10/lib/python3.11/site-packages/ais_bench/benchmark/openicl/icl_inferencer/icl_base_api_inferencer.py", line 68, in init
super().init(model_cfg, batch_size, output_json_filepath)
File "/usr/local/python3.11.10/lib/python3.11/site-packages/ais_bench/benchmark/openicl/icl_inferencer/icl_base_inferencer.py", line 60, in init
self.model: BaseModel = build_model_from_cfg(model_cfg) # type: ignore
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/python3.11.10/lib/python3.11/site-packages/ais_bench/benchmark/utils/config/build.py", line 134, in build_model_from_cfg
raise AISBenchConfigError(
ais_bench.benchmark.utils.logging.exceptions.AISBenchConfigError: [�[1m�[41m�[33mUTILS-CFG-002�[0m]�[1m�[31mmodel config validate failed�[0m. VLLMCustomAPIChat build failed with the following errors: {'host_ip': 'host_ip must be a valid IPv4 or IPv6 address', 'host_port': 'host_port must be a valid port number in the range (0, 65536)'} | Visit �[4m�[35mhttps://ais-bench-benchmark.readthedocs.io/zh-cn/latest/faqs/error_codes.html#utils-cfg-002�[0m for further help.
[2026-10-22 21:49:22,721] [ais_bench] [�[33mWARNING�[0m] Task state is error, exit loop
[ERROR] 2026-10-22-21:49:22 (PID:7185, Device:-1, RankID:-1) ERR99999 UNKNOWN applicaiton exception

前置检查

• 我已读懂主页文档的快速入门，无法解决问题
• 我已检索过FAQ，无重复问题
• 我已搜索过现有Issue，无重复问题
• 我已更新到最新版本，问题仍存在