Repository navigation
feat(ci): add prompt-agent e2e tests for copilot harness toolbox and skill samples - #1127
Merged
Zhijie Huang (hund030) merged 4 commits intoNov 10, 2026
Merged
Conversation
…skill samples - extend the prompt-agent CI runner and recovery script to manage plural `toolboxes:`, endpoint-scoped toolbox MCP connections, and `azure.ai.skill` resources - add `test-spec.yml` contracts and enable `04-foundry-toolbox`, `05-skills`, and `microsoft-copilot-harness/quickstart-web-research` in the managed E2E matrix
Zhijie Huang (hund030)
temporarily deployed
to
L4-validation
November 2, 2026 03:38 — with 
GitHub Actions
Inactive
GitHub Actions
Inactive
…hecks - verify preflight absence and post-deploy metadata for endpoint-scoped toolbox MCP connections, and accept `null` from empty `azd ai connection list` - enforce mutually exclusive toolbox vs. skill topology in `verify_adopted()` and reject symlinks in source skill bundles before workspace copy
Zhijie Huang (hund030)
deployed
to
L4-validation
November 9, 2026 08:34 — with
GitHub Actions
Active
GitHub Actions
Active
…ble regex - rebind the verified toolbox MCP connection to `project-managed-identity` before CI invocation so app-only service principal callers do not fail on `UserEntraToken` OBO exchange - use word-boundary anchors in `quickstart-web-research/test-spec.yml` so labeled fields match when streamed immediately after a pre-tool preamble
Zhijie Huang (hund030)
deployed
to
L4-validation
November 10, 2026 02:35 — with
GitHub Actions
Active
GitHub Actions
Active
Zhijie Huang (hund030)
marked this pull request as ready for review
November 10, 2026 03:06
Zhijie Huang (hund030)
deployed
to
L4-validation
November 10, 2026 03:06 — with
GitHub Actions
Active
GitHub Actions
Active
Zhijie Huang (hund030)
deployed
to
L4-validation
November 10, 2026 03:06 — with
GitHub Actions
Active
GitHub Actions
Active
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
The manifest rewrite can follow an absolute symlink and overwrite a shared source manifest, while a crash before ownership is saved can block recovery cleanup and leave the toolbox deployed.
2 open issues
What changed in this PR
Extends prompt-agent cloud E2E CI to cover the Copilot harness toolbox, skills, and web-research samples introduced in #1075.
Changes:
- Adds test contracts and enables CI discovery for three previously excluded samples.
- Extends deployment, attachment verification, and cleanup to owned skills and toolbox MCP connections.
- Updates provisioning compatibility, extension requirements, lifecycle tests, and cancellation time.
| File | Description |
|---|---|
samples/cli/prompt-agents/microsoft-copilot-harness/quickstart-web-research/azure.yaml |
Requires the skills extension. |
samples/cli/prompt-agents/AGENTS.md |
Removes file-based skills from unsupported capabilities. |
.azure-pipelines/scripts/prompt-agents/tests/test_prompt_agent_recovery.py |
Covers skill and connection recovery safeguards. |
.azure-pipelines/scripts/prompt-agents/tests/test_prompt_agent_pipeline.py |
Updates cancellation-budget expectations. |
.azure-pipelines/scripts/prompt-agents/tests/test_prompt_agent_lifecycle.py |
Covers managed-resource deployment and cleanup. |
.azure-pipelines/scripts/prompt-agents/tests/test_prompt_agent_helpers.py |
Tests validation, ownership, and skill-bundle handling. |
.azure-pipelines/scripts/prompt-agents/run-prompt-agent.sh |
Deploys and cleans up skills and toolbox connections. |
.azure-pipelines/scripts/prompt-agents/README.md |
Documents expanded adapter capabilities and cleanup. |
.azure-pipelines/scripts/prompt-agents/provision-prompt-agent.sh |
Allows documentation-only infrastructure directories. |
.azure-pipelines/scripts/prompt-agents/prompt_agent_helpers.py |
Adds managed-resource validation and isolated skill naming. |
.azure-pipelines/scripts/prompt-agents/cleanup-prompt-agent.sh |
Recovers owned skills and toolbox connections. |
.azure-pipelines/scripts/prompt-agents/blacklist.json |
Enables the three samples for CI discovery. |
.azure-pipelines/prompt-agents-samples-ci.yml |
Increases cancellation time for cleanup. |
.azure-pipelines/prompt-agent-tests/cli/microsoft-copilot-harness/quickstart-web-research/test-spec.yml |
Adds official-source research assertions. |
.azure-pipelines/prompt-agent-tests/cli/05-skills/test-spec.yml |
Checks required travel-guide sections. |
.azure-pipelines/prompt-agent-tests/cli/04-foundry-toolbox/test-spec.yml |
Adds web-search response assertions. |
Review effort: Balanced
Zhijie Huang (hund030)
enabled auto-merge (squash)
November 10, 2026 04:27
Hui Miao (huimiu)
approved these changes
Nov 10, 2026
Zhijie Huang (hund030)
deleted the
feat/prompt-agents-copilot-harness-e2e
branch
November 10, 2026 05:03
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Enable prompt-agent cloud E2E CI coverage for the Microsoft Copilot harness toolbox and skill samples introduced in #1075 (
04-foundry-toolbox,05-skills, andmicrosoft-copilot-harness/quickstart-web-research).Changes
.azure-pipelines/scripts/prompt-agents/prompt_agent_helpers.py: support pluraltoolboxes:,microsoft_copilot_toolset_preview/web_searchtools, isolatedazure.ai.skillarchive renaming (ci-e2e-sk-<nonce>), and endpoint-scoped toolbox MCP connection verification (toolbox_connection_name,connection_absent,skill_absent)..azure-pipelines/scripts/prompt-agents/run-prompt-agent.sh&cleanup-prompt-agent.sh: enableAZD_ALPHA_ENABLE_AGENTS_MANAGED_HARNESS=true, deploy and verify ownedazure.ai.skillservices, and delete/verify absence of owned skills and auto-created toolbox MCP connections during teardown and recovery..azure-pipelines/prompt-agent-tests/cli/**&blacklist.json: addtest-spec.ymlcontracts and remove04-foundry-toolbox,05-skills, andmicrosoft-copilot-harness/quickstart-web-researchfromblacklist.json.samples/cli/prompt-agents/microsoft-copilot-harness/quickstart-web-research/azure.yaml&provision-prompt-agent.sh: pinazure.ai.skills: '>=1.0.0-beta.8'soazd initinstalls the skills extension, and allow doc-onlyinfra.pathdirectories during disposable provisioning.Manual Validation
run-prompt-agent.shandcleanup-prompt-agent.shlive against a Sweden Central Foundry project (gpt-5.4-mini) forsamples/cli/prompt-agents/04-foundry-toolbox,samples/cli/prompt-agents/05-skills, andsamples/cli/prompt-agents/microsoft-copilot-harness/quickstart-web-research; all three deployed, passed theirtest-spec.ymlassertions, and verifiedabsentcleanup for the temporary agent, toolbox, skill, and toolbox MCP connection.