Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion .github/requirements-ci.txt
Original file line number Diff line number Diff line change
Expand Up @@ -8,4 +8,5 @@ picklescan==1.0.5
pytest==9.1.1
pytest-xdist==3.8.0
pyyaml==6.0.3
referencing==0.37.0
referencing==0.37.0
requests==2.32.5
Original file line number Diff line number Diff line change
Expand Up @@ -50,6 +50,48 @@ pwsh -NoProfile -ExecutionPolicy Bypass -File .github/skills/discovery-services-

The runner skips completed stages based on `run-state.json` and continues from the next incomplete stage.

## Validation evidence and approvals

Validation retries repeat the original test; they never replace it with an
`OK` connectivity prompt. Each poll saves the full response to
`validation-response-<response-id>.json`, while `validation-result.json`
records the conversation, response, investigation, original prompt, verdict,
and diagnostic reason. Failed and timed-out investigations are retained even
when successful-test cleanup is enabled.

For tool-backed agents, success requires a correlated call and successful
output from the deployed tool, not just an assistant message. Code environments
must additionally print this receipt in tool stdout after executing assertions
and verifying that the requested output files exist and have nonzero sizes:

```text
DISCOVERY_VALIDATION_RESULT={"checks":[{"name":"slope","actual":2.1,"expected":2.1,"tolerance":1e-8}],"artifacts":[{"path":"/output/coefficients.csv","bytes":123}]}
```

The numbers, tolerances, paths, and sizes must be measured, not copied from this
example. The validator checks equality/tolerances, rejects nonfinite numbers and
empty artifacts, and requires a platform output-resource URI. Native tools
without code environments require successful tool output but no script receipt.
No-tool agents retain their response-only smoke test. This is execution evidence
for the supplied checks, not a general semantic grader of scientific analyses.

Pending approval tools, API-required actions, Discovery
`discoveryProperties.confirmations` in `AwaitingUserInput` state, and completed replies asking for
confirmation produce `VALIDATION_APPROVAL_INPUT_REQUIRED=true` and
`TASK_STATUS=validate:input_required`. Do not mark validation done or disable
confirmation. Complete platform approvals in the Discovery UI, then rerun
`-Stage validate -RunDir <RunDir>` to poll the same response. For a completed
confirmation-only reply, use `ask_user` to obtain the user's answer, then pass
that answer as `-ValidationPrompt` with the same run directory. The runner sends
the continuation in the preserved conversation and retains the original test.
Do not infer approval or automatically submit function outputs.

Offline regression checks:

```powershell
python -m unittest discover -s .\.github\tests -p test_agent_deployer_validation.py -v
```

## Stage TODO status

When the assistant environment exposes a native task/TODO UI, such as the VS Code GitHub Copilot prompt-window TODO list, use that UI as the primary progress tracker. First validate every requested agent exists under `agents/microsoft/<agent>` or `agents/partners/<partner-name>/<agent>`. If any agent is missing or ambiguous, do not create deployment TODOs; stop and ask the user to fix the agent name or specify `-PublisherName`.
Expand Down Expand Up @@ -152,4 +194,3 @@ Validation requires the response to complete and not report an execution failure
- `local`: use local Docker, then push to ACR.

Use `-WhatIfPlan` to inspect repo discovery and planned stages without Azure calls.

Original file line number Diff line number Diff line change
Expand Up @@ -11,5 +11,8 @@
| Tool provisioning does not finish | Rerun with `-Resume <RunDir>`; the deploy stage can continue polling existing in-flight resources. |
| Agent deployment fails | Check the temp `agent.yaml` and `agent-deploy-config.json` in the run folder. |
| Validation times out | Rerun with `-Resume <RunDir>` or deploy with `-SkipValidation` if the user only needs deployment. |
| Validation asks for confirmation | Do not self-approve. Complete pending approvals in Discovery and rerun the validation stage. For a completed confirmation-only reply, collect the user's answer with `ask_user` and pass it as `-ValidationPrompt` in the same run directory. |
| Validation reports no execution evidence | Inspect the saved response snapshot and `validation-result.json`. A plan, an `OK` reply, or an assistant-only success marker is not a tool test. |
| Code executed but receipt validation fails | Check runtime stdout for `DISCOVERY_VALIDATION_RESULT=` with measured checks, tolerances, and nonzero output-file sizes, plus the platform's output-resource URI. Do not fabricate the receipt in assistant text. |

The runner disables Azure CLI telemetry because some CLI versions can emit telemetry cleanup crashes after otherwise successful commands.
Loading
Loading