Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 6 additions & 3 deletions a2a-t-corpus/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,14 +9,14 @@ It currently covers **Task-T**, **Negotiation-T** (RAN energy-saving target / in
Three things to know before use:

- **Test-only**: lives under `a2a-t-corpus/`, is not packaged into the `a2a_t` wheel, and is deselected in CI with `-m 'not live'`.
- **A real LLM is required**: when the three required `A2AT_LLM_*` settings are missing it fails with an actionable error; the `.env` keys match the project-root `env.example`.
- **A real LLM is required**: when the three required `A2AT_LLM_*` settings are missing it fails with an actionable error; the `.env` keys (`A2AT_LANGUAGE`, `A2AT_LLM_*`) reuse the project-root `env.example` names.
- **The case format is shared with the Java repository**: case files, the schemas under `schemas/`, and the tools under `tools/` are kept identical across both repositories.

## 2. Module directory

```
a2a-t-corpus/
├── env.example configuration template, with the three required A2AT_LLM_* entries
├── env.example configuration template, with A2AT_LANGUAGE and the three required A2AT_LLM_* entries
├── .env local configuration (secrets, gitignored; copied from env.example)
├── README.md English guide
├── README_zh.md Chinese guide
Expand Down Expand Up @@ -220,6 +220,7 @@ python a2a-t-corpus/tools/inputCsvToJson.py --reverse --csv a2a-t-corpus/suites/

- **Only `input_case_*.json` is committed**: the CSV design table is for human review, `input_case_*.json` is what the code runs; they hold the same content, but only the JSON is committed.
- **Adding a case needs no new code**: just create a scenario directory (`suites/<extension>/resources/<scenario>/`) and put the `input_case_*.json` in it; the suite discovers scenarios at collection time. Afterwards run the matching `test_*_self_guard.py` for a structural check.
- **Non-default languages**: set `A2AT_LANGUAGE` in `a2a-t-corpus/.env` (or the process environment) and place the translated file next to the Chinese one as `input_case_from_text_<language>.json` / `input_case_from_data_<language>.json`, for example `input_case_from_text_en-US.json`. Keep the existing `caseDimension` / `caseCategory` vocabulary (for example `正常` / `异常`) so the report statistics and the self-guard category contract stay consistent. When no case file exists for the configured language, the workflow suites skip with an actionable message; no code change is needed once the file is added.

### 4.3 Case file format

Expand Down Expand Up @@ -254,6 +255,8 @@ All commands run from the repository root. First prepare the LLM configuration (
cp a2a-t-corpus/env.example a2a-t-corpus/.env
```

`A2AT_LANGUAGE` selects the prompt-resource language and the case-file variant. It defaults to `zh-CN`; set `en-US` to use the English templates. The corpus reads `a2a-t-corpus/.env` and the process environment (the project-root `.env` is not read). English cases live next to the Chinese ones as `input_case_from_text_en-US.json` / `input_case_from_data_en-US.json`; when no case file exists for the configured language, the workflow suites skip with an actionable message. Non-default languages write `output_result_<flow>_<language>.json` and `summary_report_<flow>_<language>.json`, so the Chinese run results are not overwritten.

**Run every case**:

```
Expand Down Expand Up @@ -288,7 +291,7 @@ uv run pytest a2a-t-corpus/suites/task/test_task_t_from_text_workflow.py -m live
- For Negotiation-T / Notification-T, just replace the suite path with `a2a-t-corpus/suites/negotiation` or `a2a-t-corpus/suites/notification`; the scenario name supports wildcards too.
- Every executed case prints to the console as it completes; failed or crashed cases additionally print their full call record. Result files reflect only the current run, and a filtered run overwrites only the matched cases.
- `--corpus-output-dir=<dir>` redirects the result files and the summary report to a given directory; by default result files write back into each scenario directory and the summary report goes to `a2a-t-corpus/.corpus/`.
- Each scenario produces `output_result_from_text.json` / `output_result_from_data.json`, aggregated into `summary_report_<flow>.json` and a console table (pass rate overall / by scenario / by dimension / by category / by API step, with the 95% target marked, plus latency p50/p95 and token totals).
- Each scenario produces `output_result_<flow>.json` (for non-default languages `output_result_<flow>_<language>.json`), aggregated into `summary_report_<flow>.json` and a console table (pass rate overall / by scenario / by dimension / by category / by API step, with the 95% target marked, plus latency p50/p95 and token totals).
- Without an LLM, run the structural self-guards first (these run in CI):

```
Expand Down
9 changes: 6 additions & 3 deletions a2a-t-corpus/README_zh.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,14 +9,14 @@ a2a-t-corpus 是 A2A-T SDK 的数据驱动准确性验证模块。它按用例
使用前需了解三点:

- **仅用于测试**:位于 `a2a-t-corpus/`,不打包进 `a2a_t`,CI 中以 `-m 'not live'` 排除。
- **需要真实 LLM**:三个必填配置 `A2AT_LLM_*` 缺失时会直接报错并给出提示;`.env` 的键名与项目根 `env.example` 一致。
- **需要真实 LLM**:三个必填配置 `A2AT_LLM_*` 缺失时会直接报错并给出提示;`.env` 的键名(`A2AT_LANGUAGE`、`A2AT_LLM_*`)与项目根 `env.example` 一致。
- **用例格式与 Java 仓库共用**:用例文件、`schemas/` 下的 Schema、`tools/` 下的工具,在两个仓库中保持一致。

## 2. 模块目录

```
a2a-t-corpus/
├── env.example 配置模板,含三个必填 A2AT_LLM_* 项
├── env.example 配置模板,含 A2AT_LANGUAGE 与三个必填 A2AT_LLM_* 项
├── .env 本地配置(密钥,gitignore;由 env.example 复制)
├── README.md 英文说明
├── README_zh.md 中文说明
Expand Down Expand Up @@ -220,6 +220,7 @@ python a2a-t-corpus/tools/inputCsvToJson.py --reverse --csv a2a-t-corpus/suites/

- **代码仓库只提交 `input_case_*.json`**:CSV 设计表供人评审用,`input_case_*.json` 供代码运行用,两者内容一致,但只需要提交 JSON。
- **新增用例不需要新增代码**:只需新建测试场景目录(`suites/<扩展>/resources/<场景>/`)并放入 `input_case_*.json`,套件在收集阶段会自动发现场景。新增后可用对应扩展的 `test_*_self_guard.py` 做一次结构检查。
- **非默认语言**:在 `a2a-t-corpus/.env`(或进程环境变量)中设置 `A2AT_LANGUAGE`,并把翻译后的用例文件与中文文件放在同一场景目录,命名为 `input_case_from_text_<语言>.json` / `input_case_from_data_<语言>.json`,例如 `input_case_from_text_en-US.json`。`caseDimension` / `caseCategory` 建议沿用现有词表(如 `正常` / `异常`),以保持报表统计口径和 self-guard 分类契约一致。若配置语言下没有用例文件,workflow 套件会以明确的 skip 信息提示;补齐文件后无需改代码即可运行。

### 4.3 用例文件格式

Expand Down Expand Up @@ -254,6 +255,8 @@ python a2a-t-corpus/tools/inputCsvToJson.py --reverse --csv a2a-t-corpus/suites/
cp a2a-t-corpus/env.example a2a-t-corpus/.env
```

`A2AT_LANGUAGE` 同时决定提示词资源语言和用例文件变体,默认 `zh-CN`;设置为 `en-US` 后使用英文模板。corpus 读取的是 `a2a-t-corpus/.env` 和进程环境变量(不会读取项目根目录的 `.env`)。英文用例与中文用例放在同一场景目录,命名为 `input_case_from_text_en-US.json` / `input_case_from_data_en-US.json`;若配置语言下没有对应用例文件,workflow 套件会以明确的 skip 信息提示。非默认语言的结果文件为 `output_result_<flow>_<语言>.json`、汇总为 `summary_report_<flow>_<语言>.json`,不会覆盖中文运行结果。

**执行全部用例**:

```
Expand Down Expand Up @@ -288,7 +291,7 @@ uv run pytest a2a-t-corpus/suites/task/test_task_t_from_text_workflow.py -m live
- Negotiation-T / Notification-T 只需把套件路径换成 `a2a-t-corpus/suites/negotiation` 或 `a2a-t-corpus/suites/notification`,场景名同样支持通配符。
- 每条用例执行完成会立即打印到控制台;失败或崩溃的用例会额外打印完整的调用记录。结果文件只反映本次运行,过滤运行时只覆盖命中的用例。
- `--corpus-output-dir=<目录>` 可把结果文件和汇总报表写到指定目录;默认结果文件写回各场景目录,汇总报表写到 `a2a-t-corpus/.corpus/`。
- 每个场景生成 `output_result_from_text.json` / `output_result_from_data.json`,并汇总为 `summary_report_<flow>.json` 与控制台表格(总体 / 按场景 / 按维度 / 按分类 / 按 API 步的通过率,标注 95% 目标,含耗时 p50/p95 与 Token 合计)。
- 每个场景生成 `output_result_<flow>.json`(非默认语言为 `output_result_<flow>_<语言>.json`),并汇总为 `summary_report_<flow>.json` 与控制台表格(总体 / 按场景 / 按维度 / 按分类 / 按 API 步的通过率,标注 95% 目标,含耗时 p50/p95 与 Token 合计)。
- 未配置 LLM 时,可先运行结构自检(此测试进 CI):

```
Expand Down
44 changes: 35 additions & 9 deletions a2a-t-corpus/conftest.py
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,8 @@
fails fast with an actionable error exactly when someone actually tries to run the live suites.
- ``pytest_generate_tests``: parametrizes the workflow suite modules over the discovered scenario
cases (test id = ``<scenario>/<case id>``), the ``@TestFactory`` counterpart.
- ``A2AT_LANGUAGE`` (default ``zh-CN``) selects the prompt language and the case-file variant, for
example ``input_case_from_text_en-US.json``; a language without any committed case file skips.

Suite modules declare four constants: ``EXTENSION_FOLDER``, ``INPUT_CASE_FILE``, ``FLOW_TYPE`` and
``RUNTIME_KIND`` (``"task"``, ``"negotiation"`` or ``"notification"``); ``API_NAMES`` defaults to the
Expand All @@ -21,7 +23,8 @@
from typing import Any

import pytest
from engine.constants import TASK_API_NAMES
from engine.config import resolve_language
from engine.constants import DEFAULT_LANGUAGE, TASK_API_NAMES, localized_case_file
from engine.discover import ScenarioScanner
from engine.loader import CaseFileLoader

Expand Down Expand Up @@ -52,6 +55,12 @@ def _runtime_kind(request: pytest.FixtureRequest) -> str:
return getattr(request.module, "RUNTIME_KIND", "task")


@pytest.fixture(scope="session")
def corpus_language() -> str:
"""The configured prompt and case-file language, resolved before the LLM runtime loads."""
return resolve_language()


@pytest.fixture(scope="session")
def task_workflow_runtime() -> Any:
"""The shared Task-T SDK runtime, assembled lazily on first use.
Expand Down Expand Up @@ -128,14 +137,19 @@ def workflow_engine(request: pytest.FixtureRequest, workflow_runtime: Any) -> An


@pytest.fixture(scope="module")
def workflow_context(request: pytest.FixtureRequest, summary_output_dir: Path | None) -> Any:
def workflow_context(
request: pytest.FixtureRequest,
summary_output_dir: Path | None,
corpus_language: str,
) -> Any:
"""The result collector of one suite module; the transcript writes on teardown."""
from engine.suite import WorkflowRunContext

flow_type = getattr(request.module, "FLOW_TYPE", None)
if not flow_type:
pytest.fail("workflow suite modules must declare FLOW_TYPE", pytrace=False)
context = WorkflowRunContext(flow_type)
run_flow_type = flow_type if corpus_language == DEFAULT_LANGUAGE else f"{flow_type}_{corpus_language}"
context = WorkflowRunContext(run_flow_type)

def finalize() -> None:
context.write_outputs(output_dir_override=summary_output_dir, summary_dir_override=summary_output_dir)
Expand All @@ -149,16 +163,28 @@ def pytest_generate_tests(metafunc: pytest.Metafunc) -> None:
if not _WORKFLOW_FIXTURES.issubset(metafunc.fixturenames):
return
extension_folder = getattr(metafunc.module, "EXTENSION_FOLDER", None)
flow_file_name = getattr(metafunc.module, "INPUT_CASE_FILE", None)
if not extension_folder or not flow_file_name:
base_flow_file = getattr(metafunc.module, "INPUT_CASE_FILE", None)
if not extension_folder or not base_flow_file:
return

language = resolve_language()
flow_file_name = localized_case_file(base_flow_file, language)
scenario_filter = metafunc.config.getoption("--corpus-scenario")
case_filter = metafunc.config.getoption("--case-filter")
scenarios = ScenarioScanner.filter_scenarios(
ScenarioScanner.discover(extension_folder, flow_file_name),
scenario_filter,
)
scenarios = ScenarioScanner.discover_optional(extension_folder, flow_file_name)
if not scenarios:
reason = (
f"no {language} corpus case file found: expected "
f"suites/{extension_folder}/resources/<scenario>/{flow_file_name}; add the "
f"language-specific case file or set A2AT_LANGUAGE={DEFAULT_LANGUAGE}"
)
metafunc.parametrize(
("scenario", "case"),
[pytest.param(None, None, marks=pytest.mark.skip(reason))],
ids=["none"],
)
return
scenarios = ScenarioScanner.filter_scenarios(scenarios, scenario_filter)
loader = CaseFileLoader()
parameters: list[tuple[Any, Any]] = []
ids: list[str] = []
Expand Down
15 changes: 6 additions & 9 deletions a2a-t-corpus/engine/assembler.py
Original file line number Diff line number Diff line change
Expand Up @@ -2,9 +2,9 @@

Every component is produced by the production builders; the sole test seam is the recording client
injected through the ``llm_client`` parameter. The unified ``A2ATConfig`` is constructed directly
from the corpus test configuration (packaged zh-CN prompt resources, corpus attempt limit), so the
module never reads the production ``.env`` loading paths. A separate runtime is provided for each
-T extension the corpus exercises (Task-T / Negotiation-T / …).
from the corpus test configuration (packaged prompt resources of the configured language, corpus
attempt limit), so the module never reads the production ``.env`` loading paths. A separate
runtime is provided for each -T extension the corpus exercises (Task-T / Negotiation-T / …).
"""

from __future__ import annotations
Expand Down Expand Up @@ -45,9 +45,6 @@
"task_runtime",
]

#: Prompt language fixed by the corpus (the packaged zh-CN resources serve the business payloads).
_LANGUAGE = "zh-CN"

#: Fixed LLM client-config entries the corpus bridge does not override.
_HISTORY_WINDOW = 10
_SESSION_MAX_TOTAL = 300
Expand Down Expand Up @@ -87,7 +84,7 @@ def task_runtime() -> Runtime:
recorder = _new_recorder(env)

config = A2ATConfig(
prompt=PromptRuntimeConfig(language=_LANGUAGE, source_type="packaged"),
prompt=PromptRuntimeConfig(language=env.language, source_type="packaged"),
input_limits=InputLimitConfig(),
llm=LlmRuntimeConfig(max_attempts=env.max_attempts),
)
Expand Down Expand Up @@ -120,7 +117,7 @@ def negotiation_runtime() -> NegotiationRuntime:
recorder = _new_recorder(env)

orchestrator = NegotiationGenerationOrchestratorBuilder(
language=_LANGUAGE,
language=env.language,
llm_client=recorder,
max_attempts=env.max_attempts,
).build()
Expand Down Expand Up @@ -150,7 +147,7 @@ def notification_runtime() -> NotificationRuntime:
recorder = _new_recorder(env)

config = A2ATConfig(
prompt=PromptRuntimeConfig(language=_LANGUAGE, source_type="packaged"),
prompt=PromptRuntimeConfig(language=env.language, source_type="packaged"),
input_limits=InputLimitConfig(),
llm=LlmRuntimeConfig(max_attempts=env.max_attempts),
)
Expand Down
Loading
Loading