From e86d45bf9febf719b132deb71545e8c5accfe608 Mon Sep 17 00:00:00 2001 From: yangts-hw Date: Mon, 19 Oct 2026 11:32:05 +0800 Subject: [PATCH] docs(corpus): align corpus READMEs with the Python module guide Rewrite a2a-t-corpus/README.md and README_zh.md following the structure of the Python repository's corpus guide, adjusted to this module: - Update the directory tree to the real layout (pom.xml, src/test engine/suites, actual task scenarios: private-line-complaint, ran-energy-saving-modify, backup-battery endurance create/modify) - Move Notification-T (service-recovery, subscribe-incident) from "later phases" into the current coverage - Replace run commands with Maven equivalents (-Dcorpus.scenario, -Dcase.filter, -Dcorpus.output.dir) and document LLM config resolution (system properties > .env > environment) - Drop the non-existent tools/*.sh wrappers and update the summary report path to target/corpus/ - Present the case JSON contract as a top-level array, matching the schemas, and use private-line-complaint in the walkthrough Signed-off-by: yangts-hw --- a2a-t-corpus/README.md | 401 ++++++++++++++++++++++++++------------ a2a-t-corpus/README_zh.md | 363 +++++++++++++++++++++++++--------- 2 files changed, 556 insertions(+), 208 deletions(-) diff --git a/a2a-t-corpus/README.md b/a2a-t-corpus/README.md index de34abde..f2998f35 100644 --- a/a2a-t-corpus/README.md +++ b/a2a-t-corpus/README.md @@ -1,143 +1,302 @@ -# a2a-t-corpus - A2A-T Accuracy Verification Corpus +# a2a-t-corpus — A2A-T accuracy verification corpus module -Data-driven accuracy verification for the A2A-T SDK: workflow cases written as JSON run against a **real LLM** -through the production SDK assembly path, with per-step API/LLM transcripts, latency and token metrics, and 95%+ accuracy -target reporting for prompt tuning. +## 1. Module introduction -Current extensions: **Task-T** (private-line complaint diagnosis) and **Negotiation-T** (RAN energy-saving target / -information / feasibility negotiation), each as the benchmark pilot; Notification-T / Authorization-T follow the same -structure once their reference scenarios stabilize (suite class + `resources/` + registry entries, zero fork). +a2a-t-corpus is the data-driven accuracy verification module of the A2A-T SDK. It calls the real SDK (against a real LLM) following the steps defined by each case, records the complete request and response, latency and tokens of every step, and reports the pass rate, used to judge whether the prompts reach the 95% target. -- **Pure test module**: no `src/main`, not in the BOM, excluded from release artifacts - (`maven.deploy.skip/source.skip/javadoc.skip=true`) and from CI (`mvn ... -pl '!a2a-t-corpus'`). -- **A real LLM is mandatory**: running without the three required `A2AT_LLM_*` values fails fast with guidance. - The `.env` keys reuse the project-root `env.example` naming, so the corpus template is a tuned subset of the - root one. +It currently covers **Task-T** (private-line complaint, RAN energy-saving modify, backup-battery endurance create/modify), **Negotiation-T** (RAN energy-saving target / information / feasibility negotiation) and **Notification-T** (service recovery / incident subscription notification). Authorization-T will be added the same way once its reference scenarios stabilize. -## Layout +Three things to know before use: + +- **Test-only**: lives under `a2a-t-corpus/` with only `src/test` and no `src/main`, is not in the BOM and is excluded from release artifacts (`maven.deploy.skip/source.skip/javadoc.skip=true`), and is deselected in CI with `-pl '!a2a-t-corpus'`. +- **A real LLM is required**: when the three required `A2AT_LLM_BASE_URL` / `A2AT_LLM_API_KEY` / `A2AT_LLM_MODEL` settings are missing it fails with an actionable error; the `.env` keys match the project-root `env.example`. +- **The case format is shared with the Python repository**: case files, the schemas under `schemas/`, and the tools under `tools/` are kept identical across both repositories. + +## 2. Module directory ``` a2a-t-corpus/ -├── pom.xml / env.example / .env (secrets, gitignored) / README.md / README_zh.md -├── schemas/ input-case.schema.json, output-result.schema.json (first-hand format definitions) -├── tools/ inputCsvToJson(.py/.sh), outputJsonToCsv(.py/.sh), csv-templates/ +├── pom.xml Maven build: pure test module, release/CI excluded +├── env.example configuration template, with the three required A2AT_LLM_* entries +├── .env local configuration (secrets, gitignored; copied from env.example) +├── README.md English guide +├── README_zh.md Chinese guide +├── schemas/ +│ ├── input-case.schema.json case format definition +│ └── output-result.schema.json result format definition +├── tools/ +│ ├── inputCsvToJson.py CSV design table <-> input case file +│ ├── outputJsonToCsv.py result file -> CSV review table +│ └── csv-templates/ +│ └── input-case-template.csv CSV design-table template (one example row) └── src/test/java/net/openan/a2at/sdk/corpus/ - ├── engine/ framework core: config/loader/registry/engine/llm/assertion/report/discover + CorpusWorkFlowSuite - └── suites/ the per-extension workflow suites - ├── task/ TaskTFromTextWorkFlowTest, TaskTFromDataWorkFlowTest, TaskSelfGuardTest, - │ └── resources// input_case_from_text.json, input_case_from_data.json - │ (+ output_result_*.json written back by each run) - └── negotiation/ NegotiationTFromTextWorkFlowTest, NegotiationTFromDataWorkFlowTest, - NegotiationSelfGuardTest - └── resources// ran-energy-saving-target-negotiation, - ran-energy-saving-information-negotiation, - ran-energy-saving-feasibility-negotiation -``` - -`suites//resources//` holds `input_case_from_text.json` and `input_case_from_data.json` -(case design, hand-authored); `output_result_*.json` files are written back by the engine after each run. - -## Quick start - -```bash -cp a2a-t-corpus/env.example a2a-t-corpus/.env # fill A2AT_LLM_BASE_URL / API_KEY / MODEL -mvn -pl a2a-t-corpus -am test -Dtest=TaskTFromTextWorkFlowTest -mvn -pl a2a-t-corpus test -Dtest=TaskTFromTextWorkFlowTest -Dcorpus.scenario=private-line-complaint -mvn -pl a2a-t-corpus test -Dtest=TaskTFromTextWorkFlowTest -Dcorpus.scenario='private-*' -Dcase.filter='TC0000000*' -# Negotiation-T scenarios (RAN energy-saving) -mvn -pl a2a-t-corpus test -Dtest=NegotiationTFromTextWorkFlowTest -Dcorpus.scenario='ran-energy-saving-*' + ├── engine/ framework code: config, loader, registry, engine, llm, + │ assertion, report, discover, util + CorpusWorkFlowSuite + └── suites/ + ├── task/ + │ ├── TaskTFromTextWorkFlowTest.java natural-language case suite + │ ├── TaskTFromDataWorkFlowTest.java structured-input case suite + │ ├── TaskSelfGuardTest.java structural self-guard (no LLM, runs in CI) + │ └── resources/ + │ ├── private-line-complaint/ private-line complaint + │ │ ├── input_case_from_text.json natural-language cases, hand-written + │ │ └── input_case_from_data.json structured-input cases, hand-written + │ ├── ran-energy-saving-modify/ RAN energy-saving task modify + │ │ ├── input_case_from_text.json + │ │ └── input_case_from_data.json + │ ├── ran-intelligent-backup-battery-endurance-create/ backup-battery endurance create + │ │ ├── input_case_from_text.json + │ │ └── input_case_from_data.json + │ └── ran-intelligent-backup-battery-endurance-modify/ backup-battery endurance modify + │ ├── input_case_from_text.json + │ └── input_case_from_data.json + ├── negotiation/ + │ ├── NegotiationTFromTextWorkFlowTest.java natural-language case suite (propose / accept / reject / abort) + │ ├── NegotiationTFromDataWorkFlowTest.java structured-input case suite + │ ├── NegotiationSelfGuardTest.java structural self-guard (no LLM, runs in CI) + │ └── resources/ + │ ├── ran-energy-saving-target-negotiation/ target negotiation + │ │ ├── input_case_from_text.json + │ │ └── input_case_from_data.json + │ ├── ran-energy-saving-information-negotiation/ information negotiation + │ │ ├── input_case_from_text.json + │ │ └── input_case_from_data.json + │ └── ran-energy-saving-feasibility-negotiation/ feasibility negotiation + │ ├── input_case_from_text.json + │ └── input_case_from_data.json + └── notification/ + ├── NotificationTFromTextWorkFlowTest.java natural-language case suite + ├── NotificationTFromDataWorkFlowTest.java structured-input case suite + ├── NotificationSelfGuardTest.java structural self-guard (no LLM, runs in CI) + └── resources/ + ├── service-recovery/ service recovery notification + │ ├── input_case_from_text.json + │ └── input_case_from_data.json + └── subscribe-incident/ incident subscription notification + ├── input_case_from_text.json + └── input_case_from_data.json ``` -- `-Dcorpus.scenario`: scenario name glob (`*` wildcard, comma-separated); `-Dcase.filter`: case id glob. -- `-Dtest` filters apply reactor-wide under `-am`; the parent POM configures surefire - `failIfNoSpecifiedTests=false`, so upstream modules without a matching test class do not fail the build. -- Every executed case is printed to the console as it completes; failed/crashed cases print their full interaction - trace there too. The transcript file reflects exactly this run (a filtered run overwrites it with the matching - cases only). -- `-Dcorpus.output.dir=`: redirect outputs instead of writing back into the scenario directories - (summaries default to `target/corpus/`). -- Structure-only gates without an LLM: `mvn -pl a2a-t-corpus test -Dtest=TaskSelfGuardTest,NegotiationSelfGuardTest`. +Inside each scenario directory, `output_result_from_text.json` / `output_result_from_data.json` are result files written back by a run (gitignored) and need not be authored; the summary report is written to `a2a-t-corpus/target/corpus/` (gitignored). -## Case JSON contract (v1, first-hand definitions in `schemas/`) +## 3. Tools -```json -{ - "id": "TC00000001", "caseDimension": "01-意图清晰参数完整", "caseDesc": "标准专线业务中断投诉", - "caseCategory": "正常", - "input": [ - { "api": "generateTaskPromptFromText", - "args": { "text": "发生专线业务中断，接入端口名称为P781-……", "templateUri": "Task-T/network-layer/private-line-complaint/v1" } }, - { "api": "validateTaskPromptAndDataFilling", - "args": { "schema": { "…": "JSON Schema" }, "templateUri": "…", "promptText": { "$fromStep": 1, "$field": "promptText" } } } - ], - "expected": [ - { "result": "success", "data": {}, "error": { "errCode": "", "errMessage": "" } }, - { "result": "success", "data": { "accessPort": "P781-……" }, "error": { "errCode": "", "errMessage": "" } } - ] -} -``` - -- `id` uses the `TC` (TestCase) prefix, e.g. `TC00000001`; ids are unique within one case file. -- `input` / `expected` are equal-length arrays; order is the execution order; any step count is allowed and the same - API may appear repeatedly. -- `expected.result`: `success` / `error`; `data` is a subset assertion (optional `dataExact` tightens to full - equality); `error.errCode` must be a code from the SDK error catalog. -- `{"$fromStep": N, "$field": "promptText"}` copies a field from the response of an earlier step (1-based); - literal values always win. -- `caseCategory` is an open enumeration, e.g. `正常` / `异常` / `边界` / `模糊语义` ... (aggregation key only, - never coupled to assertions). -- Loading is strict: unknown keys, missing expectations, mismatched array lengths, unregistered api names and - out-of-catalog error codes all fail at load time with aggregated diagnostics. -- Business payloads (input texts, schema descriptions, expected slot values) and the case metadata - (`caseDimension` / `caseDesc` / `caseCategory`) stay in Chinese because they describe business cases for the - zh-CN prompt resources; everything else (tooling, messages, code) is English. - -## Outputs and metrics - -- Per scenario: `output_result_from_text.json` / `output_result_from_data.json` with one record per case whose - interactions interleave **API steps and their underlying LLM calls**, carrying complete requests/responses - (error steps dump every exception property plus the cause chain), latency and tokens - complete output, never - truncated. -- `a2a-t-corpus/target/corpus/summary_report_.json` + console table: accuracy overall / per scenario / per - dimension / per category / per API with the **95% target line annotated**, latency p50/p95, token totals. - -## Tooling loop - -```bash -python a2a-t-corpus/tools/inputCsvToJson.py --template --out my-cases.csv # design table (with an example row) -# fill the table, then convert with structural validation on by default -python a2a-t-corpus/tools/inputCsvToJson.py --csv my-cases.csv \ - --out a2a-t-corpus/src/test/java/net/openan/a2at/sdk/corpus/suites/task/resources//input_case_from_text.json -# run a suite, then review (one row per case, full per-step request/response) -python a2a-t-corpus/tools/outputJsonToCsv.py \ - --json a2a-t-corpus/src/test/java/net/openan/a2at/sdk/corpus/suites/task/resources//output_result_from_text.json \ - --out review.csv -# optional: fill input JSON back into the design table +Cases are designed in CSV and run as JSON; two scripts convert between the two. All commands run from the repository root (the project root that contains `pom.xml` and `a2a-t-corpus/`). + +**Generate the case design-table template** (one example row, opens directly in Excel): + +``` +python a2a-t-corpus/tools/inputCsvToJson.py --template --out my-cases.csv +``` + +**Convert a CSV design table into an input case file** (structural validation on by default; the output lands in `input_case_from_text.json` or `input_case_from_data.json`): + +``` +python a2a-t-corpus/tools/inputCsvToJson.py --csv my-cases.csv --out a2a-t-corpus/src/test/java/net/openan/a2at/sdk/corpus/suites/task/resources//input_case_from_text.json +``` + +**Convert an input case file back into a CSV design table** (reverse direction, for review; corresponds to step ⑧): + +``` python a2a-t-corpus/tools/inputCsvToJson.py --reverse --csv --out back.csv ``` -Run the Python scripts directly on any platform; `tools/*.sh` are thin bash wrappers that forward every argument 1:1, -so any command above works by swapping `python a2a-t-corpus/tools/.py` for `tools/.sh`: +**Convert a run-result file into a CSV review table** (one row per case, with the complete request / response of every step; corresponds to step ⑤): + +``` +python a2a-t-corpus/tools/outputJsonToCsv.py --json --out review.csv +``` + +CSV design-table columns: the fixed columns `id`, `caseDimension`, `caseDesc`, `caseCategory`, `scenario`, then the per-step groups `stepN_api`, `stepN_args`, `stepN_expect_result`, `stepN_expect_data`, `stepN_expect_errCode`, `stepN_expect_errMessageContains`; the number of steps is unbounded. + +## 4. Adding a case + +The overall flow is below, followed by a walkthrough with a concrete case. All commands below run from the repository root. + +```mermaid +flowchart TD + S1["① Fill the CSV design table"] + S2{"② Review the CSV design table"} + S3["③ Generate input_case_*.json"] + S4["④ Run the case"] + S5["⑤ Convert the result file to CSV"] + S6{"⑥ Review the case result"} + S7["⑦ Adjust the case or the code"] + S8["⑧ Convert the case file back to CSV"] + S1 --> S2 + S2 -- approved --> S3 + S2 -- rejected --> S1 + S3 --> S4 + S4 --> S5 + S5 --> S6 + S6 -- not as expected --> S7 + S7 --> S8 + S8 --> S1 + S6 -- as expected --> Done["Case done"] +``` + +### 4.1 Example: add a text case to the private-line-complaint scenario + +Goal: under the Task-T suite's `private-line-complaint` scenario, add a natural-language case `TC00000001` ("private-line business interruption complaint"). + +**① Fill the CSV design table** -```bash -tools/inputCsvToJson.sh --template --out my-cases.csv -tools/inputCsvToJson.sh --csv my-cases.csv --out /input_case_from_text.json -tools/outputJsonToCsv.sh --json /output_result_from_text.json --out review.csv -tools/inputCsvToJson.sh --reverse --csv --out back.csv +Generate the template first: + +``` +python a2a-t-corpus/tools/inputCsvToJson.py --template --out my-cases.csv ``` -The wrappers (Git Bash / WSL / Linux / macOS) are **not** meant to be double-clicked: double-clicking starts them -without arguments, so they only print the usage text (and the terminal may close right away). Run them from a terminal -with the arguments shown above, or run `python a2a-t-corpus/tools/...` directly when no bash shell is available. The -wrappers resolve `python3` (falling back to `python`) from the terminal PATH and fail with a clear message when no -Python interpreter is installed. +Each row of the template is one case. How to fill the columns for this case: + +| Column | Meaning | Value for this case | +| --- | --- | --- | +| `id` | case id, `TC` prefix, unique within the file | `TC00000001` | +| `caseDimension` | case dimension, used for statistics | `01-意图清晰参数完整` | +| `caseDesc` | case description | `基础档-专线业务中断-接入端口名称-标准` | +| `caseCategory` | case category: `正常` / `异常` / `边界` | `正常` | +| `scenario` | owning scenario directory name | `private-line-complaint` | +| `step1_api` | API called by step 1 (Java facade name) | `generateTaskPromptFromText` | +| `step1_args` | step 1 arguments (JSON) | `{"text":"发生专线业务中断，接入端口名称为P781-……","templateUri":"Task-T/network-layer/private-line-complaint/v1"}` | +| `step1_expect_result` | step 1 expected result: `success` / `error` | `success` | +| `step1_expect_data` | step 1 expected data (subset assertion, may be empty) | `{}` | +| `step1_expect_errCode` | expected error code (required when `error`) | empty | +| `step1_expect_errMessageContains` | expected message substring (optional) | empty | +| `step2_api` | API called by step 2 | `validateTaskPromptAndDataFilling` | +| `step2_args` | step 2 arguments; to pass a field from step 1's response use `{"$fromStep":1,"$field":"promptText"}` | `{"schema":{…},"templateUri":"Task-T/network-layer/private-line-complaint/v1","promptText":{"$fromStep":1,"$field":"promptText"}}` | +| `step2_expect_result` | step 2 expected result | `success` | +| `step2_expect_data` | fields expected to be extracted at step 2 (response subset) | `{"accessPort":"P781-……"}` | +| `step2_expect_errCode` | expected error code (required when `error`) | empty | +| `step2_expect_errMessageContains` | expected message substring (optional) | empty | -## Adding a scenario (zero Java changes) +For more than two steps, keep adding `step3_*`, `step4_*`, …; the same API may appear multiple times. -Create `suites//resources//` with the two input JSON files - the `@TestFactory` suites discover -scenarios at runtime. Run the matching `*SelfGuardTest` (no LLM) as the structural gate first. +**② Review the CSV design table** + +Review by hand whether the case covers the target scenario and whether the expectations are reasonable. Once approved, go to ③; otherwise go back to ①. + +**③ Generate the input case file** + +Convert the design table into a case file: + +``` +python a2a-t-corpus/tools/inputCsvToJson.py --csv my-cases.csv --out a2a-t-corpus/src/test/java/net/openan/a2at/sdk/corpus/suites/task/resources/private-line-complaint/input_case_from_text.json +``` + +The conversion validates the format (unique ids, equal-length expectations and steps, known APIs and error codes, …) and fails on any problem. The generated case file can be run directly. + +**④ Run the case** + +Run this case through the suite (see "Running cases" for the full command reference): + +``` +mvn -pl a2a-t-corpus -am test -Dtest=TaskTFromTextWorkFlowTest -Dcorpus.scenario=private-line-complaint -Dcase.filter=TC00000001 +``` + +**⑤ Convert the run result to CSV** + +Turn the `output_result_from_text.json` written back by the previous step into a review table: + +``` +python a2a-t-corpus/tools/outputJsonToCsv.py --json a2a-t-corpus/src/test/java/net/openan/a2at/sdk/corpus/suites/task/resources/private-line-complaint/output_result_from_text.json --out review.csv +``` -## Later phases (after the reference scenarios stabilize) +**⑥ Review the case result** -For Notification-T / Authorization-T: add `suites//` suites plus their `resources/`, -register the extension facade methods in `ApiRegistry`, and reuse the framework and tools as-is. +Open `review.csv` and compare each step's expectation with the actual result (`result`, `response`, error code, …). If it matches, the case is done; otherwise go to ⑦. + +**⑦ Adjust the case or the code** + +Decide whether the issue is in the case or the implementation: fix the case if the design or expectation is wrong, fix the code if the implementation is wrong. + +**⑧ Convert the case file back to CSV** + +If the case file was changed, convert it back into the design table to keep the two in sync, then go back to ①: + +``` +python a2a-t-corpus/tools/inputCsvToJson.py --reverse --csv a2a-t-corpus/src/test/java/net/openan/a2at/sdk/corpus/suites/task/resources/private-line-complaint/input_case_from_text.json --out my-cases.csv +``` + +### 4.2 Notes on adding cases + +- **Only `input_case_*.json` is committed**: the CSV design table is for human review, `input_case_*.json` is what the code runs; they hold the same content, but only the JSON is committed. +- **Adding a case needs no new code**: just create a scenario directory (`suites//resources//`, i.e. `src/test/java/net/openan/a2at/sdk/corpus/suites//resources//`) and put the `input_case_*.json` in it; the suite discovers scenarios at collection time (`ScenarioScanner` + `@TestFactory`). Afterwards run the matching `*SelfGuardTest` for a structural check. + +### 4.3 Case file format + +```json +[ + { + "id": "TC00000001", "caseDimension": "01-意图清晰参数完整", "caseDesc": "标准专线业务中断投诉", + "caseCategory": "正常", + "input": [ + { "api": "generateTaskPromptFromText", + "args": { "text": "发生专线业务中断，接入端口名称为P781-……", "templateUri": "Task-T/network-layer/private-line-complaint/v1" } }, + { "api": "validateTaskPromptAndDataFilling", + "args": { "schema": { "…": "JSON Schema" }, "templateUri": "…", "promptText": { "$fromStep": 1, "$field": "promptText" } } } + ], + "expected": [ + { "result": "success", "data": {}, "error": { "errCode": "", "errMessage": "" } }, + { "result": "success", "data": { "accessPort": "P781-……" }, "error": { "errCode": "", "errMessage": "" } } + ] + } +] +``` + +- One file is a JSON array whose elements are the cases; `id` uses a `TC` prefix (e.g. `TC00000001`) and is unique within one case file. +- `input` and `expected` are equal-length; the order is the execution order, the same API may appear multiple times, and the number of steps is unbounded. +- `expected.result` is `success` / `error`; `data` is a subset assertion (the optional `dataExact` tightens it to whole-map equality); `error.errCode` must be a defined SDK error code. +- `{"$fromStep": N, "$field": "promptText"}` references a field of an earlier step's response by step number (1-based); explicit literal values win. +- Loading is strict: unknown fields, missing expectations, length mismatches, unregistered APIs, undefined error codes and illegal `$fromStep` all fail at load time. + +## 5. Running cases + +All commands run from the repository root. First prepare the LLM configuration (the three required entries `A2AT_LLM_BASE_URL` / `A2AT_LLM_API_KEY` / `A2AT_LLM_MODEL`): + +``` +cp a2a-t-corpus/env.example a2a-t-corpus/.env +``` + +Values are resolved in the order system properties (`-Da2at.llm.*`) > module-root `.env` (`A2AT_LLM_*`) > process environment variables (`A2AT_LLM_*`). + +**Run every case** (all workflow suites of the three extensions plus the self-guards; add `-am` to build the upstream modules when they are not yet installed in the local repository): + +``` +mvn -pl a2a-t-corpus -am test +``` + +**Run a given scenario** (`-Dcorpus.scenario` is a scenario name, supports the `*` glob wildcard, and accepts several comma-separated): + +``` +mvn -pl a2a-t-corpus test -Dtest=TaskTFromTextWorkFlowTest -Dcorpus.scenario=ran-energy-saving-modify +``` + +**Run every text case of a given scenario**: + +``` +mvn -pl a2a-t-corpus test -Dtest=TaskTFromTextWorkFlowTest -Dcorpus.scenario=private-line-complaint +``` + +**Run every data case of a given scenario**: + +``` +mvn -pl a2a-t-corpus test -Dtest=TaskTFromDataWorkFlowTest -Dcorpus.scenario=private-line-complaint +``` + +**Run a given case id of a given scenario** (`-Dcase.filter` is a case id, supports the `*` glob wildcard): + +``` +mvn -pl a2a-t-corpus test -Dtest=TaskTFromTextWorkFlowTest -Dcorpus.scenario=ran-energy-saving-modify -Dcase.filter=TC00000001 +``` + +- A case's test id is `/`; you can also target a single case by its `@TestFactory` dynamic-test name in the IDE. +- For Negotiation-T / Notification-T, just swap `-Dtest` to the matching suite class (`NegotiationTFromTextWorkFlowTest`, `NotificationTFromTextWorkFlowTest`, …); the scenario name supports wildcards too. +- `-Dtest` filters apply reactor-wide under `-am`; the parent POM configures surefire `failIfNoSpecifiedTests=false`, so upstream modules without a matching test class do not fail the build. +- Every executed case prints to the console as it completes; failed/crashed cases additionally print their full call record. Result files reflect only the current run, and a filtered run overwrites only the matched cases. +- `-Dcorpus.output.dir=` redirects the result files and the summary report to a given directory; by default result files write back into each scenario directory and the summary report goes to `a2a-t-corpus/target/corpus/`. +- Each scenario produces `output_result_from_text.json` / `output_result_from_data.json`, aggregated into `summary_report_.json` and a console table (pass rate overall / by scenario / by dimension / by category / by API step, with the 95% target marked, plus latency p50/p95 and token totals). +- Without an LLM, run the structural self-guards first (these run in CI): + +``` +mvn -pl a2a-t-corpus test -Dtest=TaskSelfGuardTest,NegotiationSelfGuardTest,NotificationSelfGuardTest +``` diff --git a/a2a-t-corpus/README_zh.md b/a2a-t-corpus/README_zh.md index fde0b222..e2d7447d 100644 --- a/a2a-t-corpus/README_zh.md +++ b/a2a-t-corpus/README_zh.md @@ -1,113 +1,302 @@ -# a2a-t-corpus — A2A-T 准确性验证语料 +# a2a-t-corpus — A2A-T 准确性验证语料模块 -A2A-T SDK 的数据驱动准确性验证：把 JSON 工作流用例经生产 SDK 装配链路对**真实 LLM** 执行，产出逐步 API/LLM 完整转录、时延与 Token 指标，并以 95%+ 准确率达标线支撑提示词调优。 +## 1. 模块介绍 -当前实现扩展：**Task-T**（专线投诉诊断）与 **Negotiation-T**（无线节能目标/信息/可行性协商），均为标杆迭代；Notification-T / Authorization-T 待参考场景稳定后按同结构扩展（套件类 + `resources/` + 注册表条目，零分叉）。 +a2a-t-corpus 是 A2A-T SDK 的数据驱动准确性验证模块。它按用例定义的步骤调用真实 SDK（对接真实 LLM），记录每一步的完整请求与响应、耗时和 Token，并按用例通过率统计准确率，用来判断提示词效果是否达到 95% 的目标。 -- **纯测试模块**：无 `src/main`，不进 BOM、不参与发布（`maven.deploy.skip/source.skip/javadoc.skip=true`），且从 CI 排除（`mvn ... -pl '!a2a-t-corpus'`）。 -- **必须配置真实 LLM**：缺少三个必填 `A2AT_LLM_*` 配置时 fail-fast 并给出配置指引。`.env` 键名与项目根 `env.example` 保持一致，corpus 模板即根模板的精调子集。 +当前覆盖 **Task-T**（专线业务投诉、无线节能修改、智能备电续航创建/修改）、**Negotiation-T**（无线节能目标 / 信息 / 可行性协商）与 **Notification-T**（业务抢通 / 故障事件订阅通知）。Authorization-T 等待参考场景稳定后按同样方式接入。 -## 目录 +使用前需了解三点： + +- **仅用于测试**：位于 `a2a-t-corpus/`，只有 `src/test`、无 `src/main`，不进 BOM、不参与发布（`maven.deploy.skip/source.skip/javadoc.skip=true`），CI 中以 `-pl '!a2a-t-corpus'` 排除。 +- **需要真实 LLM**：三个必填配置 `A2AT_LLM_BASE_URL` / `A2AT_LLM_API_KEY` / `A2AT_LLM_MODEL` 缺失时会直接报错并给出提示；`.env` 的键名与项目根 `env.example` 一致。 +- **用例格式与 Python 仓库共用**：用例文件、`schemas/` 下的 Schema、`tools/` 下的工具，在两个仓库中保持一致。 + +## 2. 模块目录 ``` a2a-t-corpus/ -├── pom.xml / env.example / .env（密钥，gitignore）/ README.md / README_zh.md -├── schemas/ input-case.schema.json、output-result.schema.json（格式一手定义） -├── tools/ inputCsvToJson(.py/.sh)、outputJsonToCsv(.py/.sh)、csv-templates/ +├── pom.xml Maven 配置：纯测试模块，跳过发布、CI 排除 +├── env.example 配置模板，含三个必填 A2AT_LLM_* 项 +├── .env 本地配置（密钥，gitignore；由 env.example 复制） +├── README.md 英文说明 +├── README_zh.md 中文说明 +├── schemas/ +│ ├── input-case.schema.json 用例格式定义 +│ └── output-result.schema.json 结果格式定义 +├── tools/ +│ ├── inputCsvToJson.py CSV 设计表 <-> input 用例文件 +│ ├── outputJsonToCsv.py 结果文件 -> CSV 检视表 +│ └── csv-templates/ +│ └── input-case-template.csv CSV 设计表模板（含一行示例） └── src/test/java/net/openan/a2at/sdk/corpus/ - ├── engine/ 框架核心：config/loader/registry/engine/llm/assertion/report/discover + CorpusWorkFlowSuite - └── suites/ 各扩展的准确性验证套件 - ├── task/ TaskTFromTextWorkFlowTest、TaskTFromDataWorkFlowTest、TaskSelfGuardTest、 - │ └── resources/<场景>/ input_case_from_text.json 与 input_case_from_data.json - │ （用例设计，人工构造）；output_result_*.json 由引擎每次运行写回 - └── negotiation/ NegotiationTFromTextWorkFlowTest、NegotiationTFromDataWorkFlowTest、 - NegotiationSelfGuardTest - └── resources/<场景>/ ran-energy-saving-target-negotiation、 - ran-energy-saving-information-negotiation、 - ran-energy-saving-feasibility-negotiation + ├── engine/ 框架代码：config、loader、registry、engine、llm、 + │ assertion、report、discover、util + CorpusWorkFlowSuite + └── suites/ + ├── task/ + │ ├── TaskTFromTextWorkFlowTest.java 自然语言用例套件 + │ ├── TaskTFromDataWorkFlowTest.java 结构化输入用例套件 + │ ├── TaskSelfGuardTest.java 结构自检（不调 LLM，进 CI） + │ └── resources/ + │ ├── private-line-complaint/ 专线业务投诉 + │ │ ├── input_case_from_text.json 自然语言用例，人工编写 + │ │ └── input_case_from_data.json 结构化输入用例，人工编写 + │ ├── ran-energy-saving-modify/ 无线节能任务修改 + │ │ ├── input_case_from_text.json + │ │ └── input_case_from_data.json + │ ├── ran-intelligent-backup-battery-endurance-create/ 智能备电续航任务创建 + │ │ ├── input_case_from_text.json + │ │ └── input_case_from_data.json + │ └── ran-intelligent-backup-battery-endurance-modify/ 智能备电续航任务修改 + │ ├── input_case_from_text.json + │ └── input_case_from_data.json + ├── negotiation/ + │ ├── NegotiationTFromTextWorkFlowTest.java 自然语言用例套件（propose / accept / reject / abort） + │ ├── NegotiationTFromDataWorkFlowTest.java 结构化输入用例套件 + │ ├── NegotiationSelfGuardTest.java 结构自检（不调 LLM，进 CI） + │ └── resources/ + │ ├── ran-energy-saving-target-negotiation/ 目标协商 + │ │ ├── input_case_from_text.json + │ │ └── input_case_from_data.json + │ ├── ran-energy-saving-information-negotiation/ 信息协商 + │ │ ├── input_case_from_text.json + │ │ └── input_case_from_data.json + │ └── ran-energy-saving-feasibility-negotiation/ 可行性协商 + │ ├── input_case_from_text.json + │ └── input_case_from_data.json + └── notification/ + ├── NotificationTFromTextWorkFlowTest.java 自然语言用例套件 + ├── NotificationTFromDataWorkFlowTest.java 结构化输入用例套件 + ├── NotificationSelfGuardTest.java 结构自检（不调 LLM，进 CI） + └── resources/ + ├── service-recovery/ 业务抢通通知 + │ ├── input_case_from_text.json + │ └── input_case_from_data.json + └── subscribe-incident/ 故障事件订阅通知 + ├── input_case_from_text.json + └── input_case_from_data.json ``` -`suites/<扩展>/resources/<场景>/` 下为 `input_case_from_text.json` 与 `input_case_from_data.json`（用例设计，人工构造）；`output_result_*.json` 由引擎在每次运行后写回。 +各场景目录下的 `output_result_from_text.json` / `output_result_from_data.json` 为运行后写回的结果文件（gitignore），不需要手写；汇总报表写在 `a2a-t-corpus/target/corpus/`（gitignore）。 -## 快速开始 +## 3. 工具介绍 -```bash -cp a2a-t-corpus/env.example a2a-t-corpus/.env # 填写 A2AT_LLM_BASE_URL / API_KEY / MODEL -mvn -pl a2a-t-corpus -am test -Dtest=TaskTFromTextWorkFlowTest -mvn -pl a2a-t-corpus test -Dtest=TaskTFromTextWorkFlowTest -Dcorpus.scenario=private-line-complaint -mvn -pl a2a-t-corpus test -Dtest=TaskTFromTextWorkFlowTest -Dcorpus.scenario='private-*' -Dcase.filter='TC0000000*' -# 无线节能协商场景 -mvn -pl a2a-t-corpus test -Dtest=NegotiationTFromTextWorkFlowTest -Dcorpus.scenario='ran-energy-saving-*' +用例用 CSV 设计、用 JSON 运行，两个脚本负责两者之间的转换。命令均在仓库根目录下执行（即包含 `pom.xml` 和 `a2a-t-corpus/` 的项目根目录）。 + +**生成用例设计表模板**（含一行示例，Excel 可直接打开）： + +``` +python a2a-t-corpus/tools/inputCsvToJson.py --template --out my-cases.csv ``` -- `-Dcorpus.scenario`：场景名 glob（`*` 通配，逗号分隔）；`-Dcase.filter`：用例 id glob。 -- `-Dtest` 过滤在 `-am` 下作用于整个反应堆；父 POM 已配置 surefire `failIfNoSpecifiedTests=false`，上游模块无同名测试类不会导致构建失败。 -- 每条已执行用例完成即打印到控制台；失败/崩溃用例同时打印完整 interaction 轨迹；转录文件只反映本次执行（过滤运行会以命中的用例整体覆盖文件）。 -- `-Dcorpus.output.dir=`：把输出重定向到指定目录（缺省写回场景目录；summary 默认落 `target/corpus/`）。 -- 无 LLM 时先跑结构自守卫：`mvn -pl a2a-t-corpus test -Dtest=TaskSelfGuardTest,NegotiationSelfGuardTest`。 +**把 CSV 设计表转换为 input 用例文件**（默认做格式校验，产物写入 `input_case_from_text.json` 或 `input_case_from_data.json`）： + +``` +python a2a-t-corpus/tools/inputCsvToJson.py --csv my-cases.csv --out a2a-t-corpus/src/test/java/net/openan/a2at/sdk/corpus/suites/task/resources/<场景>/input_case_from_text.json +``` -## 用例 JSON 契约（v1，一手定义见 `schemas/`） +**把 input 用例文件回填为 CSV 设计表**（反向转换，用于对照检视，对应新增用例步骤 ⑧）： -```json -{ - "id": "TC00000001", "caseDimension": "01-意图清晰参数完整", "caseDesc": "标准专线业务中断投诉", - "caseCategory": "正常", - "input": [ - { "api": "generateTaskPromptFromText", - "args": { "text": "发生专线业务中断，接入端口名称为P781-……", "templateUri": "Task-T/network-layer/private-line-complaint/v1" } }, - { "api": "validateTaskPromptAndDataFilling", - "args": { "schema": { "…": "JSON Schema" }, "templateUri": "…", "promptText": { "$fromStep": 1, "$field": "promptText" } } } - ], - "expected": [ - { "result": "success", "data": {}, "error": { "errCode": "", "errMessage": "" } }, - { "result": "success", "data": { "accessPort": "P781-……" }, "error": { "errCode": "", "errMessage": "" } } - ] -} -``` - -- `id` 使用 `TC`（TestCase）前缀，如 `TC00000001`；id 在单个用例文件内唯一。 -- `input` / `expected` 为**等长数组**，顺序即执行序；步数不限，同一 API 可多次出现。 -- `expected.result`：`success` / `error`；`data` 为子集断言（可选 `dataExact` 收紧为全量相等）；`error.errCode` 必须位于 SDK 错误码目录。 -- `{"$fromStep": N, "$field": "promptText"}` 按步序（1-based）引用更早步骤响应中的字段；显式字面值优先。 -- `caseCategory` 为开放枚举：`正常` / `异常` / `边界` / `模糊语义` …（仅作统计与组织维度，断言与之间无耦合）。 -- 加载为严格解析：未知键、缺期望、数组不等长、api 未注册、errCode 不在目录 → 加载期汇总报错。 -- 业务载荷（输入文本、schema 描述、期望槽位值）与用例元数据（`caseDimension` / `caseDesc` / `caseCategory`）保留中文，因为它们描述的是面向 zh-CN 提示词资源的业务用例；其余（工具、消息、代码）一律英文。 - -## 输出与指标 - -- 每场景：`output_result_from_text.json` / `output_result_from_data.json`，每个用例一条记录，其 interactions 交错记录 **API 步与其底层 LLM 调用**，携带完整请求/响应（异常步 dump 每个属性 + cause 链）、时延、Token——**完整输出，不截断**。 -- `a2a-t-corpus/target/corpus/summary_report_.json` + 控制台表：总体/按场景/按维度/按分类/按 API 的准确率（**95% 达标线标注**）、时延 p50/p95、Token 合计。 - -## 工具闭环 - -```bash -python a2a-t-corpus/tools/inputCsvToJson.py --template --out my-cases.csv # 用例设计表（含一行示例） -# 填表后转换（默认开启产物结构校验） -python a2a-t-corpus/tools/inputCsvToJson.py --csv my-cases.csv \ - --out a2a-t-corpus/src/test/java/net/openan/a2at/sdk/corpus/suites/task/resources/<场景>/input_case_from_text.json -# 跑套件后审视（一行=一个用例，含每步完整请求/响应） -python a2a-t-corpus/tools/outputJsonToCsv.py \ - --json a2a-t-corpus/src/test/java/net/openan/a2at/sdk/corpus/suites/task/resources/<场景>/output_result_from_text.json \ - --out review.csv -# 可选：把 input JSON 回填为设计表 +``` python a2a-t-corpus/tools/inputCsvToJson.py --reverse --csv --out back.csv ``` -各平台直接运行 Python 脚本；`tools/*.sh` 为 bash 薄包装（Git Bash / WSL / Linux / macOS），参数 1:1 透传，把上面命令中的 `python a2a-t-corpus/tools/<名称>.py` 换成 `tools/<名称>.sh` 即可： +**把运行结果文件转换为 CSV 检视表**（一行一个用例，含每一步完整请求 / 响应，对应新增用例步骤 ⑤）： -```bash -tools/inputCsvToJson.sh --template --out my-cases.csv -tools/inputCsvToJson.sh --csv my-cases.csv --out <场景>/input_case_from_text.json -tools/outputJsonToCsv.sh --json <场景>/output_result_from_text.json --out review.csv -tools/inputCsvToJson.sh --reverse --csv --out back.csv +``` +python a2a-t-corpus/tools/outputJsonToCsv.py --json --out review.csv +``` + +CSV 设计表列：固定列 `id`、`caseDimension`、`caseDesc`、`caseCategory`、`scenario`，之后按步骤成组 `stepN_api`、`stepN_args`、`stepN_expect_result`、`stepN_expect_data`、`stepN_expect_errCode`、`stepN_expect_errMessageContains`，步数不限。 + +## 4. 新增用例步骤 + +整体流程如下，下面再以一个具体用例逐步展开。以下所有命令均在仓库根目录下执行。 + +```mermaid +flowchart TD + S1["① 回填 CSV 设计表"] + S2{"② CSV 设计表评审"} + S3["③ 生成 input_case_*.json"] + S4["④ 执行用例"] + S5["⑤ 结果文件转换为 CSV"] + S6{"⑥ 检视用例结果"} + S7["⑦ 调整用例或代码"] + S8["⑧ 用例文件回填 CSV"] + S1 --> S2 + S2 -- 评审通过 --> S3 + S2 -- 评审不通过 --> S1 + S3 --> S4 + S4 --> S5 + S5 --> S6 + S6 -- 不符合预期 --> S7 + S7 --> S8 + S8 --> S1 + S6 -- 符合预期 --> Done["用例完成"] ``` -两个 sh 脚本**不是双击直接运行的**：双击启动时没有参数，只会打印用法说明（且终端可能立刻关闭）。请在终端中带参数运行；没有 bash 环境时直接 `python a2a-t-corpus/tools/...` 运行 Python 脚本。脚本会从终端 PATH 中查找 `python3`（找不到则回退 `python`），两者都缺失时给出明确报错。 +### 4.1 示例：在 private-line-complaint 场景新增一条 text 用例 + +目标：在 Task-T 套件的 `private-line-complaint` 场景下，新增一条自然语言用例 `TC00000001`（“专线业务中断投诉”）。 + +**① 回填 CSV 设计表** + +先生成模板： + +``` +python a2a-t-corpus/tools/inputCsvToJson.py --template --out my-cases.csv +``` + +模板每行是一个用例，本用例各列的填法： + +| 列名 | 含义 | 本用例取值 | +| --- | --- | --- | +| `id` | 用例编号，`TC` 前缀，文件内唯一 | `TC00000001` | +| `caseDimension` | 用例维度，用于统计 | `01-意图清晰参数完整` | +| `caseDesc` | 用例描述 | `基础档-专线业务中断-接入端口名称-标准` | +| `caseCategory` | 用例分类：`正常` / `异常` / `边界` | `正常` | +| `scenario` | 所属场景目录名 | `private-line-complaint` | +| `step1_api` | 第 1 步调用的 API（Java 门面名） | `generateTaskPromptFromText` | +| `step1_args` | 第 1 步入参（JSON） | `{"text":"发生专线业务中断，接入端口名称为P781-……","templateUri":"Task-T/network-layer/private-line-complaint/v1"}` | +| `step1_expect_result` | 第 1 步期望结果：`success` / `error` | `success` | +| `step1_expect_data` | 第 1 步期望数据（子集断言，可留空） | `{}` | +| `step1_expect_errCode` | 期望错误码（`error` 时必填） | 空 | +| `step1_expect_errMessageContains` | 期望错误信息包含的子串（可选） | 空 | +| `step2_api` | 第 2 步调用的 API | `validateTaskPromptAndDataFilling` | +| `step2_args` | 第 2 步入参：要把第 1 步响应中的字段传进来，用 `{"$fromStep":1,"$field":"promptText"}` | `{"schema":{…},"templateUri":"Task-T/network-layer/private-line-complaint/v1","promptText":{"$fromStep":1,"$field":"promptText"}}` | +| `step2_expect_result` | 第 2 步期望结果 | `success` | +| `step2_expect_data` | 第 2 步期望抽出的字段（响应子集） | `{"accessPort":"P781-……"}` | +| `step2_expect_errCode` | 期望错误码（`error` 时必填） | 空 | +| `step2_expect_errMessageContains` | 期望错误信息包含的子串（可选） | 空 | + +步骤多于两步时，继续补 `step3_*`、`step4_*` …即可，同一 API 也可以重复出现。 + +**② 评审 CSV 设计表** + +人工评审这条用例的设计是否覆盖目标场景、期望值是否合理。评审通过后进入 ③；不通过则回到 ① 继续修改。 + +**③ 生成 input 用例文件** + +把设计表转换为用例文件： + +``` +python a2a-t-corpus/tools/inputCsvToJson.py --csv my-cases.csv --out a2a-t-corpus/src/test/java/net/openan/a2at/sdk/corpus/suites/task/resources/private-line-complaint/input_case_from_text.json +``` + +转换时会做格式校验（编号唯一、期望与步骤等长、api 与错误码合法等），不通过会直接报错。生成的用例文件即可直接运行。 + +**④ 执行用例** + +用套件执行这条用例（执行命令详解见「运行用例步骤」）： + +``` +mvn -pl a2a-t-corpus -am test -Dtest=TaskTFromTextWorkFlowTest -Dcorpus.scenario=private-line-complaint -Dcase.filter=TC00000001 +``` + +**⑤ 把运行结果转换为 CSV** + +用上一步写回的 `output_result_from_text.json` 生成检视表： + +``` +python a2a-t-corpus/tools/outputJsonToCsv.py --json a2a-t-corpus/src/test/java/net/openan/a2at/sdk/corpus/suites/task/resources/private-line-complaint/output_result_from_text.json --out review.csv +``` -## 新增场景（零 Java 改动） +**⑥ 检视用例结果** -新建 `suites/<扩展>/resources/<场景>/` 并放入两个 input JSON 即可——`@TestFactory` 套件运行时扫描发现场景。结构门禁先用对应扩展的 `*SelfGuardTest`（无 LLM）跑一遍。 +打开 `review.csv`，逐条对照每一步的期望与实际结果（`result`、`response`、错误码等）。符合预期则用例完成；不符合预期进入 ⑦。 -## 后续阶段（参考场景稳定后） +**⑦ 调整用例或代码** -Notification-T / Authorization-T：新增 `suites/<扩展>/` 套件与对应 `resources/`，在 `ApiRegistry` 登记该扩展 facade 方法，框架与工具链原样复用。 \ No newline at end of file +判断问题出在用例还是实现：用例设计或期望写错了就改用例，实现有问题就改代码。 + +**⑧ 用例文件回填 CSV** + +如果改了用例文件，用反向转换把它导回设计表，保持两边一致，再回到 ①： + +``` +python a2a-t-corpus/tools/inputCsvToJson.py --reverse --csv a2a-t-corpus/src/test/java/net/openan/a2at/sdk/corpus/suites/task/resources/private-line-complaint/input_case_from_text.json --out my-cases.csv +``` + +### 4.2 新增用例注意 + +- **代码仓库只提交 `input_case_*.json`**：CSV 设计表供人评审用，`input_case_*.json` 供代码运行用，两者内容一致，但只需要提交 JSON。 +- **新增用例不需要新增代码**：只需新建测试场景目录（`suites/<扩展>/resources/<场景>/`，即 `src/test/java/net/openan/a2at/sdk/corpus/suites/<扩展>/resources/<场景>/`）并放入 `input_case_*.json`，套件在收集阶段（`ScenarioScanner` + `@TestFactory`）会自动发现场景。新增后可用对应扩展的 `*SelfGuardTest` 做一次结构检查。 + +### 4.3 用例文件格式 + +```json +[ + { + "id": "TC00000001", "caseDimension": "01-意图清晰参数完整", "caseDesc": "标准专线业务中断投诉", + "caseCategory": "正常", + "input": [ + { "api": "generateTaskPromptFromText", + "args": { "text": "发生专线业务中断，接入端口名称为P781-……", "templateUri": "Task-T/network-layer/private-line-complaint/v1" } }, + { "api": "validateTaskPromptAndDataFilling", + "args": { "schema": { "…": "JSON Schema" }, "templateUri": "…", "promptText": { "$fromStep": 1, "$field": "promptText" } } } + ], + "expected": [ + { "result": "success", "data": {}, "error": { "errCode": "", "errMessage": "" } }, + { "result": "success", "data": { "accessPort": "P781-……" }, "error": { "errCode": "", "errMessage": "" } } + ] + } +] +``` + +- 一个文件是一个 JSON 数组，数组元素即用例；`id` 使用 `TC` 前缀（如 `TC00000001`），在单个用例文件内唯一。 +- `input` 与 `expected` 等长，顺序就是执行顺序，同一 API 可多次出现，步数不限。 +- `expected.result` 为 `success` / `error`；`data` 为子集断言（可选 `dataExact` 收紧为全量相等）；`error.errCode` 必须是 SDK 已定义的错误码。 +- `{"$fromStep": N, "$field": "promptText"}` 按步骤序号（从 1 开始）引用更早步骤响应中的字段；显式字面值优先。 +- 加载时严格校验：未知字段、缺期望、数组不等长、api 未注册、错误码未定义、`$fromStep` 非法都会在加载阶段统一报错。 + +## 5. 运行用例步骤 + +以下命令均在仓库根目录下执行。先准备 LLM 配置（三个必填项 `A2AT_LLM_BASE_URL` / `A2AT_LLM_API_KEY` / `A2AT_LLM_MODEL`）： + +``` +cp a2a-t-corpus/env.example a2a-t-corpus/.env +``` + +配置值按 系统属性（`-Da2at.llm.*`）> 模块根 `.env`（`A2AT_LLM_*`）> 进程环境变量（`A2AT_LLM_*`）的优先级解析。 + +**执行全部用例**（三个扩展的全部 work flow 套件 + 结构自检；上游模块尚未安装到本地仓库时需带 `-am` 一并构建）： + +``` +mvn -pl a2a-t-corpus -am test +``` + +**执行指定场景用例**（`-Dcorpus.scenario` 为场景名，支持 glob 通配符 `*`，可用逗号分隔多个）： + +``` +mvn -pl a2a-t-corpus test -Dtest=TaskTFromTextWorkFlowTest -Dcorpus.scenario=ran-energy-saving-modify +``` + +**执行指定场景下的全部 text 用例**： + +``` +mvn -pl a2a-t-corpus test -Dtest=TaskTFromTextWorkFlowTest -Dcorpus.scenario=private-line-complaint +``` + +**执行指定场景下的全部 data 用例**： + +``` +mvn -pl a2a-t-corpus test -Dtest=TaskTFromDataWorkFlowTest -Dcorpus.scenario=private-line-complaint +``` + +**执行指定场景下指定用例编号的用例**（`-Dcase.filter` 为用例编号，支持 glob 通配符 `*`）： + +``` +mvn -pl a2a-t-corpus test -Dtest=TaskTFromTextWorkFlowTest -Dcorpus.scenario=ran-energy-saving-modify -Dcase.filter=TC00000001 +``` + +- 用例的测试编号形如 `<场景名>/<用例编号>`，也可以直接在 IDE 里按 `@TestFactory` 的动态测试名定位单条用例。 +- Negotiation-T / Notification-T 只需把 `-Dtest` 换成对应套件类（`NegotiationTFromTextWorkFlowTest`、`NotificationTFromTextWorkFlowTest` 等），场景名同样支持通配符。 +- `-Dtest` 过滤在 `-am` 下作用于整个反应堆；父 POM 已配置 surefire `failIfNoSpecifiedTests=false`，上游模块无同名测试类不会导致构建失败。 +- 每条用例执行完成会立即打印到控制台；失败或崩溃的用例会额外打印完整的调用记录。结果文件只反映本次运行，过滤运行时只覆盖命中的用例（整体覆写）。 +- `-Dcorpus.output.dir=<目录>` 可把结果文件和汇总报表写到指定目录；默认结果文件写回各场景目录，汇总报表写到 `a2a-t-corpus/target/corpus/`。 +- 每个场景生成 `output_result_from_text.json` / `output_result_from_data.json`，并汇总为 `summary_report_.json` 与控制台表格（总体 / 按场景 / 按维度 / 按分类 / 按 API 步的通过率，标注 95% 目标，含耗时 p50/p95 与 Token 合计）。 +- 未配置 LLM 时，可先运行结构自检（此测试进 CI）： + +``` +mvn -pl a2a-t-corpus test -Dtest=TaskSelfGuardTest,NegotiationSelfGuardTest,NotificationSelfGuardTest +```