From 4210bab89dea2d787e5c57b66af2e53cb6d35a51 Mon Sep 17 00:00:00 2001 From: Christopher Tso Date: Sat, 4 Jul 2026 00:46:52 +0200 Subject: [PATCH] docs: place quickstart concurrency in evaluate options Entire-Checkpoint: 0847deb64d0a --- README.md | 14 +++++++------- .../docs/docs/next/targets/configuration.mdx | 2 +- .../features/readme-quickstart/.agentv/config.yaml | 2 -- examples/features/readme-quickstart/README.md | 2 -- .../readme-quickstart/evals/my-eval.eval.yaml | 4 ++-- 5 files changed, 10 insertions(+), 14 deletions(-) diff --git a/README.md b/README.md index 1a3069a24..1a435d096 100644 --- a/README.md +++ b/README.md @@ -19,7 +19,7 @@ Test AI targets on real repo tasks and measure what actually works. - **Workspace / fixtures / graders** are task-owned context: repos, setup scripts, files, fixtures, isolation, deterministic checks, and LLM grading prompts. - **Target** is the system under test: an agent, provider, gateway, replay target, CLI wrapper, transcript provider, or future app/service wrapper. Each eval selects one `target` by configured target `id` or with an eval-local target object. - **Tags** are run/result grouping labels. `tags.experiment` is the default experiment namespace, such as `with-skills` or `without-skills`; keep suite/category and target/model names out of that tag. -- **Execution** configures runner-level behavior such as general suite concurrency with `execution.max_concurrency`; `evaluate_options` holds eval behaviors such as repeat policy and budgets. +- **Evaluate options** configure eval run behavior such as `max_concurrency`, repeat policy, and budgets. - **Default test** configures inherited per-test defaults such as score `threshold`. - **Run** is one concrete execution of a tagged eval against a resolved target that writes portable artifacts for readers such as Dashboard, compare, and trend. @@ -58,7 +58,7 @@ defaults: grader: local-openai-grader ``` -**3. Create shared test defaults** in `evals/default-test.yaml`. This is a promptfoo-style partial test config that AgentV applies to each test: +**3. Create shared test defaults** in `evals/default-test.yaml`. This is a partial test config that AgentV applies to each test: ```yaml threshold: 0.8 @@ -83,8 +83,8 @@ description: Code generation quality tags: experiment: with-skills target: local-openai -execution: - max_concurrency: 1 +evaluate_options: + max_concurrency: 2 default_test: file://./default-test.yaml @@ -109,8 +109,8 @@ Plain assertion strings are short-form rubric criteria: AgentV groups them into `llm-rubric` and writes each criterion to `grading.json.assertion_results` for the Dashboard. Use explicit `type: llm-rubric` when you need weights, required flags, or `score_ranges`, or when you need a custom grader prompt, grader target, or -preprocessing; use string `value` for promptfoo-compatible free-form rubric -checks. Executable graders use `type: script`. +preprocessing; use string `value` for free-form rubric checks. Executable +graders use `type: script`. The target can be an eval-local object when this eval needs target settings of its own: @@ -142,7 +142,7 @@ tests: `target: local-openai` resolves the configured target id from `.agentv/config.yaml` and uses its provider, model, hooks, and provider settings. The object form above defines a full eval-local target and must include enough provider configuration to run. AgentV records the resolved target information in run artifacts so results can be audited and replayed. The `tags.experiment` label stays `with-skills` because the condition is unchanged; the model/provider variation belongs to the resolved target metadata. -Use `default_test.threshold` for the inherited per-test pass cutoff. `default_test` can also point at a shared file, matching promptfoo's external defaults pattern: +Use `default_test.threshold` for the inherited per-test pass cutoff. `default_test` can also point at a shared file: ```yaml default_test: file://{{ env.AGENTV_REPO_ROOT }}/.agentv/default-test.yaml diff --git a/apps/web/src/content/docs/docs/next/targets/configuration.mdx b/apps/web/src/content/docs/docs/next/targets/configuration.mdx index c66fc5a6c..2b09b75ab 100644 --- a/apps/web/src/content/docs/docs/next/targets/configuration.mdx +++ b/apps/web/src/content/docs/docs/next/targets/configuration.mdx @@ -1,6 +1,6 @@ --- title: Targets Configuration -description: Configure execution targets for providers and agents +description: Configure targets for providers and agents sidebar: order: 1 --- diff --git a/examples/features/readme-quickstart/.agentv/config.yaml b/examples/features/readme-quickstart/.agentv/config.yaml index 18be46b82..15cd0c876 100644 --- a/examples/features/readme-quickstart/.agentv/config.yaml +++ b/examples/features/readme-quickstart/.agentv/config.yaml @@ -2,5 +2,3 @@ targets: file://targets.yaml graders: file://graders.yaml tests: file://tests.yaml defaults: file://defaults.yaml -execution: - max_concurrency: 1 diff --git a/examples/features/readme-quickstart/README.md b/examples/features/readme-quickstart/README.md index 6866c6f46..4ebd3a55d 100644 --- a/examples/features/readme-quickstart/README.md +++ b/examples/features/readme-quickstart/README.md @@ -10,8 +10,6 @@ targets: file://targets.yaml graders: file://graders.yaml tests: file://tests.yaml defaults: file://defaults.yaml -execution: - max_concurrency: 1 ``` Each referenced file contains that field's value directly, such as a bare target diff --git a/examples/features/readme-quickstart/evals/my-eval.eval.yaml b/examples/features/readme-quickstart/evals/my-eval.eval.yaml index 86a1700d6..0668f847d 100644 --- a/examples/features/readme-quickstart/evals/my-eval.eval.yaml +++ b/examples/features/readme-quickstart/evals/my-eval.eval.yaml @@ -2,8 +2,8 @@ description: Code generation quality tags: experiment: with-skills target: local-openai -execution: - max_concurrency: 1 +evaluate_options: + max_concurrency: 2 default_test: file://./default-test.yaml