diff --git a/a2a-t-corpus/README.md b/a2a-t-corpus/README.md index b6db1f02..2643226f 100644 --- a/a2a-t-corpus/README.md +++ b/a2a-t-corpus/README.md @@ -9,14 +9,14 @@ It currently covers **Task-T**, **Negotiation-T** (RAN energy-saving target / in Three things to know before use: - **Test-only**: lives under `a2a-t-corpus/`, is not packaged into the `a2a_t` wheel, and is deselected in CI with `-m 'not live'`. -- **A real LLM is required**: when the three required `A2AT_LLM_*` settings are missing it fails with an actionable error; the `.env` keys match the project-root `env.example`. +- **A real LLM is required**: when the three required `A2AT_LLM_*` settings are missing it fails with an actionable error; the `.env` keys (`A2AT_LANGUAGE`, `A2AT_LLM_*`) reuse the project-root `env.example` names. - **The case format is shared with the Java repository**: case files, the schemas under `schemas/`, and the tools under `tools/` are kept identical across both repositories. ## 2. Module directory ``` a2a-t-corpus/ -├── env.example configuration template, with the three required A2AT_LLM_* entries +├── env.example configuration template, with A2AT_LANGUAGE and the three required A2AT_LLM_* entries ├── .env local configuration (secrets, gitignored; copied from env.example) ├── README.md English guide ├── README_zh.md Chinese guide @@ -220,6 +220,7 @@ python a2a-t-corpus/tools/inputCsvToJson.py --reverse --csv a2a-t-corpus/suites/ - **Only `input_case_*.json` is committed**: the CSV design table is for human review, `input_case_*.json` is what the code runs; they hold the same content, but only the JSON is committed. - **Adding a case needs no new code**: just create a scenario directory (`suites//resources//`) and put the `input_case_*.json` in it; the suite discovers scenarios at collection time. Afterwards run the matching `test_*_self_guard.py` for a structural check. +- **Non-default languages**: set `A2AT_LANGUAGE` in `a2a-t-corpus/.env` (or the process environment) and place the translated file next to the Chinese one as `input_case_from_text_.json` / `input_case_from_data_.json`, for example `input_case_from_text_en-US.json`. Keep the existing `caseDimension` / `caseCategory` vocabulary (for example `正常` / `异常`) so the report statistics and the self-guard category contract stay consistent. When no case file exists for the configured language, the workflow suites skip with an actionable message; no code change is needed once the file is added. ### 4.3 Case file format @@ -254,6 +255,8 @@ All commands run from the repository root. First prepare the LLM configuration ( cp a2a-t-corpus/env.example a2a-t-corpus/.env ``` +`A2AT_LANGUAGE` selects the prompt-resource language and the case-file variant. It defaults to `zh-CN`; set `en-US` to use the English templates. The corpus reads `a2a-t-corpus/.env` and the process environment (the project-root `.env` is not read). English cases live next to the Chinese ones as `input_case_from_text_en-US.json` / `input_case_from_data_en-US.json`; when no case file exists for the configured language, the workflow suites skip with an actionable message. Non-default languages write `output_result__.json` and `summary_report__.json`, so the Chinese run results are not overwritten. + **Run every case**: ``` @@ -288,7 +291,7 @@ uv run pytest a2a-t-corpus/suites/task/test_task_t_from_text_workflow.py -m live - For Negotiation-T / Notification-T, just replace the suite path with `a2a-t-corpus/suites/negotiation` or `a2a-t-corpus/suites/notification`; the scenario name supports wildcards too. - Every executed case prints to the console as it completes; failed or crashed cases additionally print their full call record. Result files reflect only the current run, and a filtered run overwrites only the matched cases. - `--corpus-output-dir=` redirects the result files and the summary report to a given directory; by default result files write back into each scenario directory and the summary report goes to `a2a-t-corpus/.corpus/`. -- Each scenario produces `output_result_from_text.json` / `output_result_from_data.json`, aggregated into `summary_report_.json` and a console table (pass rate overall / by scenario / by dimension / by category / by API step, with the 95% target marked, plus latency p50/p95 and token totals). +- Each scenario produces `output_result_.json` (for non-default languages `output_result__.json`), aggregated into `summary_report_.json` and a console table (pass rate overall / by scenario / by dimension / by category / by API step, with the 95% target marked, plus latency p50/p95 and token totals). - Without an LLM, run the structural self-guards first (these run in CI): ``` diff --git a/a2a-t-corpus/README_zh.md b/a2a-t-corpus/README_zh.md index c3e984e9..fda94956 100644 --- a/a2a-t-corpus/README_zh.md +++ b/a2a-t-corpus/README_zh.md @@ -9,14 +9,14 @@ a2a-t-corpus 是 A2A-T SDK 的数据驱动准确性验证模块。它按用例 使用前需了解三点： - **仅用于测试**：位于 `a2a-t-corpus/`，不打包进 `a2a_t`，CI 中以 `-m 'not live'` 排除。 -- **需要真实 LLM**：三个必填配置 `A2AT_LLM_*` 缺失时会直接报错并给出提示；`.env` 的键名与项目根 `env.example` 一致。 +- **需要真实 LLM**：三个必填配置 `A2AT_LLM_*` 缺失时会直接报错并给出提示；`.env` 的键名（`A2AT_LANGUAGE`、`A2AT_LLM_*`）与项目根 `env.example` 一致。 - **用例格式与 Java 仓库共用**：用例文件、`schemas/` 下的 Schema、`tools/` 下的工具，在两个仓库中保持一致。 ## 2. 模块目录 ``` a2a-t-corpus/ -├── env.example 配置模板，含三个必填 A2AT_LLM_* 项 +├── env.example 配置模板，含 A2AT_LANGUAGE 与三个必填 A2AT_LLM_* 项 ├── .env 本地配置（密钥，gitignore；由 env.example 复制） ├── README.md 英文说明 ├── README_zh.md 中文说明 @@ -220,6 +220,7 @@ python a2a-t-corpus/tools/inputCsvToJson.py --reverse --csv a2a-t-corpus/suites/ - **代码仓库只提交 `input_case_*.json`**：CSV 设计表供人评审用，`input_case_*.json` 供代码运行用，两者内容一致，但只需要提交 JSON。 - **新增用例不需要新增代码**：只需新建测试场景目录（`suites/<扩展>/resources/<场景>/`）并放入 `input_case_*.json`，套件在收集阶段会自动发现场景。新增后可用对应扩展的 `test_*_self_guard.py` 做一次结构检查。 +- **非默认语言**：在 `a2a-t-corpus/.env`（或进程环境变量）中设置 `A2AT_LANGUAGE`，并把翻译后的用例文件与中文文件放在同一场景目录，命名为 `input_case_from_text_<语言>.json` / `input_case_from_data_<语言>.json`，例如 `input_case_from_text_en-US.json`。`caseDimension` / `caseCategory` 建议沿用现有词表（如 `正常` / `异常`），以保持报表统计口径和 self-guard 分类契约一致。若配置语言下没有用例文件，workflow 套件会以明确的 skip 信息提示；补齐文件后无需改代码即可运行。 ### 4.3 用例文件格式 @@ -254,6 +255,8 @@ python a2a-t-corpus/tools/inputCsvToJson.py --reverse --csv a2a-t-corpus/suites/ cp a2a-t-corpus/env.example a2a-t-corpus/.env ``` +`A2AT_LANGUAGE` 同时决定提示词资源语言和用例文件变体，默认 `zh-CN`；设置为 `en-US` 后使用英文模板。corpus 读取的是 `a2a-t-corpus/.env` 和进程环境变量（不会读取项目根目录的 `.env`）。英文用例与中文用例放在同一场景目录，命名为 `input_case_from_text_en-US.json` / `input_case_from_data_en-US.json`；若配置语言下没有对应用例文件，workflow 套件会以明确的 skip 信息提示。非默认语言的结果文件为 `output_result__<语言>.json`、汇总为 `summary_report__<语言>.json`，不会覆盖中文运行结果。 + **执行全部用例**： ``` @@ -288,7 +291,7 @@ uv run pytest a2a-t-corpus/suites/task/test_task_t_from_text_workflow.py -m live - Negotiation-T / Notification-T 只需把套件路径换成 `a2a-t-corpus/suites/negotiation` 或 `a2a-t-corpus/suites/notification`，场景名同样支持通配符。 - 每条用例执行完成会立即打印到控制台；失败或崩溃的用例会额外打印完整的调用记录。结果文件只反映本次运行，过滤运行时只覆盖命中的用例。 - `--corpus-output-dir=<目录>` 可把结果文件和汇总报表写到指定目录；默认结果文件写回各场景目录，汇总报表写到 `a2a-t-corpus/.corpus/`。 -- 每个场景生成 `output_result_from_text.json` / `output_result_from_data.json`，并汇总为 `summary_report_.json` 与控制台表格（总体 / 按场景 / 按维度 / 按分类 / 按 API 步的通过率，标注 95% 目标，含耗时 p50/p95 与 Token 合计）。 +- 每个场景生成 `output_result_.json`（非默认语言为 `output_result__<语言>.json`），并汇总为 `summary_report_.json` 与控制台表格（总体 / 按场景 / 按维度 / 按分类 / 按 API 步的通过率，标注 95% 目标，含耗时 p50/p95 与 Token 合计）。 - 未配置 LLM 时，可先运行结构自检（此测试进 CI）： ``` diff --git a/a2a-t-corpus/conftest.py b/a2a-t-corpus/conftest.py index b6dd4bff..fd03673b 100644 --- a/a2a-t-corpus/conftest.py +++ b/a2a-t-corpus/conftest.py @@ -9,6 +9,8 @@ fails fast with an actionable error exactly when someone actually tries to run the live suites. - ``pytest_generate_tests``: parametrizes the workflow suite modules over the discovered scenario cases (test id = ``/``), the ``@TestFactory`` counterpart. +- ``A2AT_LANGUAGE`` (default ``zh-CN``) selects the prompt language and the case-file variant, for + example ``input_case_from_text_en-US.json``; a language without any committed case file skips. Suite modules declare four constants: ``EXTENSION_FOLDER``, ``INPUT_CASE_FILE``, ``FLOW_TYPE`` and ``RUNTIME_KIND`` (``"task"``, ``"negotiation"`` or ``"notification"``); ``API_NAMES`` defaults to the @@ -21,7 +23,8 @@ from typing import Any import pytest -from engine.constants import TASK_API_NAMES +from engine.config import resolve_language +from engine.constants import DEFAULT_LANGUAGE, TASK_API_NAMES, localized_case_file from engine.discover import ScenarioScanner from engine.loader import CaseFileLoader @@ -52,6 +55,12 @@ def _runtime_kind(request: pytest.FixtureRequest) -> str: return getattr(request.module, "RUNTIME_KIND", "task") +@pytest.fixture(scope="session") +def corpus_language() -> str: + """The configured prompt and case-file language, resolved before the LLM runtime loads.""" + return resolve_language() + + @pytest.fixture(scope="session") def task_workflow_runtime() -> Any: """The shared Task-T SDK runtime, assembled lazily on first use. @@ -128,14 +137,19 @@ def workflow_engine(request: pytest.FixtureRequest, workflow_runtime: Any) -> An @pytest.fixture(scope="module") -def workflow_context(request: pytest.FixtureRequest, summary_output_dir: Path | None) -> Any: +def workflow_context( + request: pytest.FixtureRequest, + summary_output_dir: Path | None, + corpus_language: str, +) -> Any: """The result collector of one suite module; the transcript writes on teardown.""" from engine.suite import WorkflowRunContext flow_type = getattr(request.module, "FLOW_TYPE", None) if not flow_type: pytest.fail("workflow suite modules must declare FLOW_TYPE", pytrace=False) - context = WorkflowRunContext(flow_type) + run_flow_type = flow_type if corpus_language == DEFAULT_LANGUAGE else f"{flow_type}_{corpus_language}" + context = WorkflowRunContext(run_flow_type) def finalize() -> None: context.write_outputs(output_dir_override=summary_output_dir, summary_dir_override=summary_output_dir) @@ -149,16 +163,28 @@ def pytest_generate_tests(metafunc: pytest.Metafunc) -> None: if not _WORKFLOW_FIXTURES.issubset(metafunc.fixturenames): return extension_folder = getattr(metafunc.module, "EXTENSION_FOLDER", None) - flow_file_name = getattr(metafunc.module, "INPUT_CASE_FILE", None) - if not extension_folder or not flow_file_name: + base_flow_file = getattr(metafunc.module, "INPUT_CASE_FILE", None) + if not extension_folder or not base_flow_file: return + language = resolve_language() + flow_file_name = localized_case_file(base_flow_file, language) scenario_filter = metafunc.config.getoption("--corpus-scenario") case_filter = metafunc.config.getoption("--case-filter") - scenarios = ScenarioScanner.filter_scenarios( - ScenarioScanner.discover(extension_folder, flow_file_name), - scenario_filter, - ) + scenarios = ScenarioScanner.discover_optional(extension_folder, flow_file_name) + if not scenarios: + reason = ( + f"no {language} corpus case file found: expected " + f"suites/{extension_folder}/resources//{flow_file_name}; add the " + f"language-specific case file or set A2AT_LANGUAGE={DEFAULT_LANGUAGE}" + ) + metafunc.parametrize( + ("scenario", "case"), + [pytest.param(None, None, marks=pytest.mark.skip(reason))], + ids=["none"], + ) + return + scenarios = ScenarioScanner.filter_scenarios(scenarios, scenario_filter) loader = CaseFileLoader() parameters: list[tuple[Any, Any]] = [] ids: list[str] = [] diff --git a/a2a-t-corpus/engine/assembler.py b/a2a-t-corpus/engine/assembler.py index 9c67d908..cd023ba9 100644 --- a/a2a-t-corpus/engine/assembler.py +++ b/a2a-t-corpus/engine/assembler.py @@ -2,9 +2,9 @@ Every component is produced by the production builders; the sole test seam is the recording client injected through the ``llm_client`` parameter. The unified ``A2ATConfig`` is constructed directly -from the corpus test configuration (packaged zh-CN prompt resources, corpus attempt limit), so the -module never reads the production ``.env`` loading paths. A separate runtime is provided for each --T extension the corpus exercises (Task-T / Negotiation-T / …). +from the corpus test configuration (packaged prompt resources of the configured language, corpus +attempt limit), so the module never reads the production ``.env`` loading paths. A separate +runtime is provided for each -T extension the corpus exercises (Task-T / Negotiation-T / …). """ from __future__ import annotations @@ -45,9 +45,6 @@ "task_runtime", ] -#: Prompt language fixed by the corpus (the packaged zh-CN resources serve the business payloads). -_LANGUAGE = "zh-CN" - #: Fixed LLM client-config entries the corpus bridge does not override. _HISTORY_WINDOW = 10 _SESSION_MAX_TOTAL = 300 @@ -87,7 +84,7 @@ def task_runtime() -> Runtime: recorder = _new_recorder(env) config = A2ATConfig( - prompt=PromptRuntimeConfig(language=_LANGUAGE, source_type="packaged"), + prompt=PromptRuntimeConfig(language=env.language, source_type="packaged"), input_limits=InputLimitConfig(), llm=LlmRuntimeConfig(max_attempts=env.max_attempts), ) @@ -120,7 +117,7 @@ def negotiation_runtime() -> NegotiationRuntime: recorder = _new_recorder(env) orchestrator = NegotiationGenerationOrchestratorBuilder( - language=_LANGUAGE, + language=env.language, llm_client=recorder, max_attempts=env.max_attempts, ).build() @@ -150,7 +147,7 @@ def notification_runtime() -> NotificationRuntime: recorder = _new_recorder(env) config = A2ATConfig( - prompt=PromptRuntimeConfig(language=_LANGUAGE, source_type="packaged"), + prompt=PromptRuntimeConfig(language=env.language, source_type="packaged"), input_limits=InputLimitConfig(), llm=LlmRuntimeConfig(max_attempts=env.max_attempts), ) diff --git a/a2a-t-corpus/engine/config.py b/a2a-t-corpus/engine/config.py index 86f46d86..603a01b1 100644 --- a/a2a-t-corpus/engine/config.py +++ b/a2a-t-corpus/engine/config.py @@ -1,8 +1,11 @@ -"""Unified LLM configuration of the accuracy verification corpus (port of the Java ``CorpusEnvConfig``). +"""Unified runtime configuration of the accuracy verification corpus (port of the Java ``CorpusEnvConfig``). -The configuration keys reuse the project-root ``env.example`` naming (the ``A2AT_LLM_*`` family), -so the corpus ``env.example`` is a tuned subset of the root template and one set of names serves -both production and testing. One shared configuration serves every -T extension suite. +The configuration keys reuse the project-root ``env.example`` naming (``A2AT_LANGUAGE`` and the +``A2AT_LLM_*`` family), so the corpus ``env.example`` is a tuned subset of the root template and +one set of names serves both production and testing. One shared configuration serves every -T +extension suite. ``A2AT_LANGUAGE`` selects both the prompt-resource language and the language- +specific case-file variant (``input_case_from_*_en-US.json``); it defaults to ``zh-CN`` so the +committed Chinese corpus keeps working unchanged. Resolution order: process environment variables > the ``.env`` file next to this module (Python has no ``-D`` system-property channel; the environment is the override channel). Blank @@ -21,11 +24,13 @@ from typing import Final from dotenv import dotenv_values +from engine.constants import DEFAULT_LANGUAGE, SUPPORTED_LANGUAGES -__all__ = ["CorpusEnvConfig"] +__all__ = ["CorpusEnvConfig", "resolve_language"] #: Environment-file keys (also accepted as process environment variables); names mirror the #: project-root ``env.example``. +ENV_LANGUAGE: Final[str] = "A2AT_LANGUAGE" ENV_PROVIDER: Final[str] = "A2AT_LLM_PROVIDER" ENV_BASE_URL: Final[str] = "A2AT_LLM_BASE_URL" ENV_API_KEY: Final[str] = "A2AT_LLM_API_KEY" @@ -47,11 +52,53 @@ def _env_file() -> Path: return Path(__file__).resolve().parent.parent / ".env" +def _language_problem(language: str) -> str | None: + """Return the validation problem of one prompt language, or ``None`` when it is supported.""" + if language in SUPPORTED_LANGUAGES: + return None + supported = ", ".join(sorted(SUPPORTED_LANGUAGES)) + return f"- {ENV_LANGUAGE} must be one of: {supported} (currently \"{language}\")" + + +def _resolve_language(env_file_values: dict[str, str | None]) -> str: + """Resolve the prompt language from the environment over the ``.env`` file.""" + return _resolve(ENV_LANGUAGE, env_file_values) or DEFAULT_LANGUAGE + + +def resolve_language(env_path: Path | None = None) -> str: + """Resolve only the corpus prompt language, without validating the LLM configuration. + + The corpus selects its language-specific case files at pytest collection time, before the + lazy :class:`CorpusEnvConfig` runtime loading. This helper therefore touches only + ``A2AT_LANGUAGE``; it keeps the CI collection path free of the fail-fast LLM checks. + + Args: + env_path: optional ``.env`` path; defaults to the corpus ``.env`` file. + + Returns: + the configured language, or ``zh-CN`` when unset. + + Raises: + RuntimeError: when the configured language is not supported by the bundled resources. + """ + values = dotenv_values(env_path or _env_file()) + language = _resolve_language(values) + problem = _language_problem(language) + if problem: + source = env_path or _env_file() + raise RuntimeError( + f"a2a-t-corpus has an invalid prompt-language configuration in {source}:\n" + f"{problem}\nSet {ENV_LANGUAGE} in the environment or the corpus .env file." + ) + return language + + @lru_cache(maxsize=1) def _create() -> CorpusEnvConfig: """Resolve and validate the configuration exactly once per process.""" env_file_values = dotenv_values(_env_file()) + language = _resolve_language(env_file_values) provider = _resolve(ENV_PROVIDER, env_file_values) if not provider: provider = DEFAULT_PROVIDER @@ -60,6 +107,9 @@ def _create() -> CorpusEnvConfig: model = _resolve(ENV_MODEL, env_file_values) problems: list[str] = [] + language_problem = _language_problem(language) + if language_problem: + problems.append(language_problem) if provider != DEFAULT_PROVIDER: problems.append(f"- {ENV_PROVIDER} must be \"{DEFAULT_PROVIDER}\" (currently \"{provider}\")") if not base_url: @@ -78,6 +128,7 @@ def _create() -> CorpusEnvConfig: ) return CorpusEnvConfig( + language=language, provider=provider, base_url=base_url, api_key=api_key, @@ -137,9 +188,10 @@ def _resolve_max_attempts(env_file_values: dict[str, str | None]) -> int: @dataclass(frozen=True, slots=True) class CorpusEnvConfig: - """Resolved corpus test-LLM configuration. + """Resolved corpus test runtime configuration. Attributes: + language: prompt-resource and case-file language, ``zh-CN`` or ``en-US``. provider: LLM provider, always ``openai``. base_url: OpenAI-compatible endpoint base URL. api_key: endpoint API key. @@ -150,6 +202,7 @@ class CorpusEnvConfig: max_tokens: optional maximum number of completion tokens. """ + language: str provider: str base_url: str api_key: str diff --git a/a2a-t-corpus/engine/constants.py b/a2a-t-corpus/engine/constants.py index 248e7be3..e3c38c4e 100644 --- a/a2a-t-corpus/engine/constants.py +++ b/a2a-t-corpus/engine/constants.py @@ -30,8 +30,18 @@ "FLOW_FROM_TEXT", "INPUT_CASE_FROM_DATA", "INPUT_CASE_FROM_TEXT", + "DEFAULT_LANGUAGE", + "SUPPORTED_LANGUAGES", + "case_file_variants", + "localized_case_file", ] +#: Default prompt and case-file language of the corpus (the committed cases are zh-CN). +DEFAULT_LANGUAGE: Final[str] = "zh-CN" + +#: Languages the SDK bundles prompt resources for and the corpus accepts as case-file variants. +SUPPORTED_LANGUAGES: Final[frozenset[str]] = frozenset({"zh-CN", "en-US"}) + #: Case file consumed by the natural-language workflow suites. INPUT_CASE_FROM_TEXT: Final[str] = "input_case_from_text.json" @@ -44,6 +54,42 @@ #: Flow tag of the structured-input suites. FLOW_FROM_DATA: Final[str] = "from_data" + +def localized_case_file(base_name: str, language: str) -> str: + """Return the case-file name of one language. + + The committed zh-CN cases keep the historical ``input_case_from_*.json`` names; every other + supported language appends its locale tag before the extension, for example + ``input_case_from_text_en-US.json``. + + Args: + base_name: language-neutral case-file name such as ``input_case_from_text.json``. + language: locale identifier such as ``zh-CN`` or ``en-US``. + + Returns: + the language-specific case-file name. + """ + if language == DEFAULT_LANGUAGE: + return base_name + stem, separator, suffix = base_name.rpartition(".") + if not separator: + return f"{base_name}_{language}" + return f"{stem}_{language}.{suffix}" + + +def case_file_variants(base_name: str) -> tuple[str, ...]: + """Return the default case file followed by every optional language variant. + + Args: + base_name: language-neutral case-file name such as ``input_case_from_text.json``. + + Returns: + the default-language name first, then one name per non-default supported language in + sorted order. + """ + others = (language for language in sorted(SUPPORTED_LANGUAGES) if language != DEFAULT_LANGUAGE) + return (base_name, *(localized_case_file(base_name, language) for language in others)) + # ---------- Task-T ---------- #: Task-T client generation API (natural language -> task prompt). diff --git a/a2a-t-corpus/engine/discover.py b/a2a-t-corpus/engine/discover.py index 13602061..714ae72e 100644 --- a/a2a-t-corpus/engine/discover.py +++ b/a2a-t-corpus/engine/discover.py @@ -36,8 +36,12 @@ class ScenarioScanner: """Discovers corpus scenarios by directory convention and filters them by glob.""" @staticmethod - def discover(extension_folder: str, flow_file_name: str) -> list[Scenario]: - """List every scenario under ``suites//resources/`` carrying the requested case file.""" + def discover_optional(extension_folder: str, flow_file_name: str) -> list[Scenario]: + """List matching scenarios, returning an empty list when the case file exists nowhere. + + The optional variant serves the language-specific case selection: a language without any + committed case file is a visible skip, not a collection failure. + """ source_tree = corpus_root() / "suites" / extension_folder / "resources" scenarios: list[Scenario] = [] if source_tree.is_dir(): @@ -45,6 +49,13 @@ def discover(extension_folder: str, flow_file_name: str) -> list[Scenario]: if directory.is_dir() and (directory / flow_file_name).is_file(): scenarios.append(Scenario(directory.name, directory)) scenarios.sort(key=lambda scenario: scenario.name) + return scenarios + + @staticmethod + def discover(extension_folder: str, flow_file_name: str) -> list[Scenario]: + """List every scenario under ``suites//resources/`` carrying the requested case file.""" + source_tree = corpus_root() / "suites" / extension_folder / "resources" + scenarios = ScenarioScanner.discover_optional(extension_folder, flow_file_name) if not scenarios: raise RuntimeError( "No corpus scenario directories found: expected " diff --git a/a2a-t-corpus/env.example b/a2a-t-corpus/env.example index b5ef6dfd..afc3975d 100644 --- a/a2a-t-corpus/env.example +++ b/a2a-t-corpus/env.example @@ -1,7 +1,14 @@ -# Corpus test LLM runtime (a2a-t-corpus/.env template; secrets never committed) -# Uses the same A2AT_LLM_* key names as the project-root env.example, so the corpus -# template is a tuned subset of the root one. Copy this file to .env and fill in the -# required entries; extra keys copied over from the root template are ignored. +# Corpus test runtime (a2a-t-corpus/.env template; secrets never committed) +# Uses the same A2AT_LANGUAGE and A2AT_LLM_* key names as the project-root env.example, so the +# corpus template is a tuned subset of the root one. Copy this file to .env and fill in the +# required entries; other A2AT_* keys copied over from the root template are ignored. + +# Prompt runtime +# Language of the prompt resources and the case-file variant: zh-CN (default) or en-US. +# The corpus reads its own .env (or process environment variables); the project-root .env is not +# read here. English cases live next to the Chinese ones as input_case_from_*_en-US.json; when +# they are absent, the workflow suites skip with an actionable message. +A2AT_LANGUAGE=zh-CN # LLM runtime # Selects the LLM provider used by the shared client. Only "openai" is supported. diff --git a/a2a-t-corpus/suites/negotiation/test_negotiation_self_guard.py b/a2a-t-corpus/suites/negotiation/test_negotiation_self_guard.py index bfbaca7a..f3e0962b 100644 --- a/a2a-t-corpus/suites/negotiation/test_negotiation_self_guard.py +++ b/a2a-t-corpus/suites/negotiation/test_negotiation_self_guard.py @@ -1,14 +1,20 @@ """Negotiation-T corpus self-guard (mirror of the Task-T ``CorpusSelfGuardTest``). -Validates every Negotiation-T scenario case file structurally without any LLM call: strict loading -plus the category/expectation contract. This gate runs in CI alongside the Task-T one. +Validates every Negotiation-T scenario case file structurally without any LLM call, including the +committed non-default-language variants (``input_case_from_*_en-US.json``) when present: strict +loading plus the category/expectation contract. This gate runs in CI alongside the Task-T one. """ from __future__ import annotations from typing import Final -from engine.constants import INPUT_CASE_FROM_DATA, INPUT_CASE_FROM_TEXT, NEGOTIATION_API_NAMES +from engine.constants import ( + INPUT_CASE_FROM_DATA, + INPUT_CASE_FROM_TEXT, + NEGOTIATION_API_NAMES, + case_file_variants, +) from engine.discover import ScenarioScanner from engine.loader import CaseFileLoader, InputCase @@ -19,7 +25,7 @@ "test_negotiation_scenario_case_files_are_structurally_valid", ] -#: Categories carrying a cross-step expectation contract (case data is Chinese). +#: Categories carrying a cross-step expectation contract (the report/self-guard vocabulary is Chinese). CATEGORY_NORMAL: Final[str] = "正常" CATEGORY_EXCEPTION: Final[str] = "异常" @@ -28,8 +34,9 @@ def test_negotiation_scenario_case_files_are_structurally_valid() -> None: """Load every Negotiation-T case file strictly and enforce the category/expectation contract.""" problems: list[str] = [] - for flow_file in (INPUT_CASE_FROM_TEXT, INPUT_CASE_FROM_DATA): - for scenario in ScenarioScanner.discover("negotiation", flow_file): + flow_files = (*case_file_variants(INPUT_CASE_FROM_TEXT), *case_file_variants(INPUT_CASE_FROM_DATA)) + for flow_file in flow_files: + for scenario in ScenarioScanner.discover_optional("negotiation", flow_file): cases = CaseFileLoader().load(scenario.input_file(flow_file), NEGOTIATION_API_NAMES) for input_case in cases: check_category_contract(input_case, problems) diff --git a/a2a-t-corpus/suites/notification/test_notification_self_guard.py b/a2a-t-corpus/suites/notification/test_notification_self_guard.py index bd382803..fee614ab 100644 --- a/a2a-t-corpus/suites/notification/test_notification_self_guard.py +++ b/a2a-t-corpus/suites/notification/test_notification_self_guard.py @@ -1,15 +1,21 @@ """Notification-T corpus self-guard (mirror of the Task-T ``CorpusSelfGuardTest``). -Validates every Notification-T scenario case file structurally without any LLM call: strict loading -plus the category/expectation contract. This gate runs in CI alongside the Task-T and Negotiation-T -ones. +Validates every Notification-T scenario case file structurally without any LLM call, including the +committed non-default-language variants (``input_case_from_*_en-US.json``) when present: strict +loading plus the category/expectation contract. This gate runs in CI alongside the Task-T and +Negotiation-T ones. """ from __future__ import annotations from typing import Final -from engine.constants import INPUT_CASE_FROM_DATA, INPUT_CASE_FROM_TEXT, NOTIFICATION_API_NAMES +from engine.constants import ( + INPUT_CASE_FROM_DATA, + INPUT_CASE_FROM_TEXT, + NOTIFICATION_API_NAMES, + case_file_variants, +) from engine.discover import ScenarioScanner from engine.loader import CaseFileLoader, InputCase @@ -20,7 +26,7 @@ "test_notification_scenario_case_files_are_structurally_valid", ] -#: Categories carrying a cross-step expectation contract (case data is Chinese). +#: Categories carrying a cross-step expectation contract (the report/self-guard vocabulary is Chinese). CATEGORY_NORMAL: Final[str] = "正常" CATEGORY_EXCEPTION: Final[str] = "异常" @@ -29,8 +35,9 @@ def test_notification_scenario_case_files_are_structurally_valid() -> None: """Load every Notification-T case file strictly and enforce the category/expectation contract.""" problems: list[str] = [] - for flow_file in (INPUT_CASE_FROM_TEXT, INPUT_CASE_FROM_DATA): - for scenario in ScenarioScanner.discover("notification", flow_file): + flow_files = (*case_file_variants(INPUT_CASE_FROM_TEXT), *case_file_variants(INPUT_CASE_FROM_DATA)) + for flow_file in flow_files: + for scenario in ScenarioScanner.discover_optional("notification", flow_file): cases = CaseFileLoader().load(scenario.input_file(flow_file), NOTIFICATION_API_NAMES) for input_case in cases: check_category_contract(input_case, problems) diff --git a/a2a-t-corpus/suites/task/test_task_self_guard.py b/a2a-t-corpus/suites/task/test_task_self_guard.py index 774d2bde..255a08f1 100644 --- a/a2a-t-corpus/suites/task/test_task_self_guard.py +++ b/a2a-t-corpus/suites/task/test_task_self_guard.py @@ -1,17 +1,23 @@ """Corpus self-guard (port of the Java ``CorpusSelfGuardTest``). -Validates every Task-T scenario case file structurally without any LLM call. Unknown keys, -misaligned expectations, unknown api names, out-of-catalog error codes and category/expectation -contradictions all fail here — this is the fast gate CI runs even though the live suites are -deselected. Id uniqueness is enforced per file by the loader; ids may repeat across the two flow -files of the same scenario. +Validates every Task-T scenario case file structurally without any LLM call, including the +committed non-default-language variants (``input_case_from_*_en-US.json``) when present. Unknown +keys, misaligned expectations, unknown api names, out-of-catalog error codes and +category/expectation contradictions all fail here — this is the fast gate CI runs even though the +live suites are deselected. Id uniqueness is enforced per file by the loader; ids may repeat +across the two flow files of the same scenario. """ from __future__ import annotations from typing import Final -from engine.constants import INPUT_CASE_FROM_DATA, INPUT_CASE_FROM_TEXT, TASK_API_NAMES +from engine.constants import ( + INPUT_CASE_FROM_DATA, + INPUT_CASE_FROM_TEXT, + TASK_API_NAMES, + case_file_variants, +) from engine.discover import ScenarioScanner from engine.loader import CaseFileLoader, InputCase @@ -22,7 +28,7 @@ "test_task_scenario_case_files_are_structurally_valid", ] -#: Categories carrying a cross-step expectation contract (case data is Chinese). +#: Categories carrying a cross-step expectation contract (the report/self-guard vocabulary is Chinese). CATEGORY_NORMAL: Final[str] = "正常" CATEGORY_EXCEPTION: Final[str] = "异常" @@ -31,8 +37,9 @@ def test_task_scenario_case_files_are_structurally_valid() -> None: """Load every Task-T case file strictly and enforce the category/expectation contract.""" problems: list[str] = [] - for flow_file in (INPUT_CASE_FROM_TEXT, INPUT_CASE_FROM_DATA): - for scenario in ScenarioScanner.discover("task", flow_file): + flow_files = (*case_file_variants(INPUT_CASE_FROM_TEXT), *case_file_variants(INPUT_CASE_FROM_DATA)) + for flow_file in flow_files: + for scenario in ScenarioScanner.discover_optional("task", flow_file): cases = CaseFileLoader().load(scenario.input_file(flow_file), TASK_API_NAMES) for input_case in cases: check_category_contract(input_case, problems)