Skip to content

feat: improve azure-diagnostics from evaluations - #3519

Draft
ghcp4a-bot[bot] wants to merge 52 commits into
mainfrom
skill-improvement/azure-diagnostics-43345242602
Draft

feat: improve azure-diagnostics from evaluations#3519
ghcp4a-bot[bot] wants to merge 52 commits into
mainfrom
skill-improvement/azure-diagnostics-43345242602

Conversation

@ghcp4a-bot

@ghcp4a-bot ghcp4a-bot Bot commented Nov 13, 2026

Copy link
Copy Markdown
Contributor

Automated candidate from skill improvement run 43345242602.

Workflow run: https://github.com/microsoft/GitHub-Copilot-for-Azure/actions/runs/43345242602

This pull request is intentionally a draft and requires human review.

@KarishmaGhiya
test: add live Functions VNet routing diagnostics
1e82730 
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 277bdab8-4b39-4631-92e7-b6ec8d849424
@KarishmaGhiya
test: avoid leaking Functions routing hypotheses
e44febd 
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 277bdab8-4b39-4631-92e7-b6ec8d849424
@KarishmaGhiya
test: scope Functions failure to observed evidence
cfbdae2 
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 277bdab8-4b39-4631-92e7-b6ec8d849424
@KarishmaGhiya
test: make Functions routing prompts user-like
e5ff0ea 
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 277bdab8-4b39-4631-92e7-b6ec8d849424
@KarishmaGhiya
test: simplify Functions eval safety constraint
a4ee2de 
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 277bdab8-4b39-4631-92e7-b6ec8d849424
@KarishmaGhiya
fix: read private endpoint IP from its NIC
1627d35 
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 277bdab8-4b39-4631-92e7-b6ec8d849424
@KarishmaGhiya
fix: wait for Flex Function host readiness
95a4272 
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 277bdab8-4b39-4631-92e7-b6ec8d849424
@KarishmaGhiya
fix: resolve Flex app hostname reliably
bdbf3b5 
Use the Function App hostNames collection with the deterministic Azure Websites hostname fallback because defaultHostName is absent for some Flex Consumption resource responses.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 277bdab8-4b39-4631-92e7-b6ec8d849424
@KarishmaGhiya
fix: retry transient fixture deployments
719c72d 
Retry bounded Azure deployment failures when a dependent network operation is still in progress, while preserving immediate failure for all other ARM errors.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 277bdab8-4b39-4631-92e7-b6ec8d849424
@KarishmaGhiya
test: make Functions VNet prompts realistic
2f1a3fd 
Replace test-authored diagnostic instructions with symptom-first customer reports and add a temporary baseline-only specification for targeted validation.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 277bdab8-4b39-4631-92e7-b6ec8d849424
@KarishmaGhiya
chore: remove temporary VNet validation spec
724f507 
The baseline-only workflow completed successfully, so remove the temporary targeted run specification from the permanent branch.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 277bdab8-4b39-4631-92e7-b6ec8d849424
@KarishmaGhiya
test: add Flex SNAT diagnostic evaluations
09f8882 
Seed deterministic outbound connection-pressure telemetry on the live Flex topology, add customer-style diagnosis and ZIP support-bundle evals, and include them in Azure Diagnostics hill climbing.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 277bdab8-4b39-4631-92e7-b6ec8d849424
@KarishmaGhiya
test: require complete VNet diagnostic reports
2d3684e 
Make critical Flex topology, causal evidence, remediation, and recovery details explicit final-answer gates so tool-only evidence cannot satisfy the routing evals.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 277bdab8-4b39-4631-92e7-b6ec8d849424
@KarishmaGhiya
fix: allow delayed Functions telemetry ingestion
96296fc 
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 277bdab8-4b39-4631-92e7-b6ec8d849424
@KarishmaGhiya
fix: preserve eval workspaces for deferred grading
511bfc3 
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 277bdab8-4b39-4631-92e7-b6ec8d849424
@KarishmaGhiya
fix: rewrite persisted workspace paths before grading
a5d14cb 
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 277bdab8-4b39-4631-92e7-b6ec8d849424
@KarishmaGhiya
test: accept equivalent support bundle schemas
cf0d8d8 
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 277bdab8-4b39-4631-92e7-b6ec8d849424
@KarishmaGhiya
Revert "test: add Flex SNAT diagnostic evaluations"
b5bc14a 
This reverts commit 09f8882.
@KarishmaGhiya
test: add temporary VNet routing validation
0f9cc57 
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 277bdab8-4b39-4631-92e7-b6ec8d849424
@KarishmaGhiya
test: compare VNet routing baseline conditions
906c9d1 
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 277bdab8-4b39-4631-92e7-b6ec8d849424
@KarishmaGhiya
test: advance VNet routing fixture version
2f77281 
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 277bdab8-4b39-4631-92e7-b6ec8d849424
@KarishmaGhiya
test: enable VNet routing improvement iterations
d6ac01d 
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 277bdab8-4b39-4631-92e7-b6ec8d849424
@KarishmaGhiya
test: allow accepted VNet candidate draft PR
b69e85e 
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 277bdab8-4b39-4631-92e7-b6ec8d849424
@KarishmaGhiya
chore: remove temporary VNet improvement spec
62a251d 
The targeted baseline and three-iteration improvement runs are complete, so remove the temporary workflow specification from the PR branch.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 277bdab8-4b39-4631-92e7-b6ec8d849424
@KarishmaGhiya
fix: strengthen Function networking diagnosis
8766f36 
Route Function networking failures to concise evidence-driven guidance that covers topology, causality, impact scope, minimal remediation, and recovery verification without encoding grader-specific wording.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 277bdab8-4b39-4631-92e7-b6ec8d849424
@KarishmaGhiya
fix: clarify Function network paths
6d66e10 
Make the concise diagnosis framework distinguish source and destination topology and explain unaffected platform dependencies without encoding scenario-specific grader language.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 277bdab8-4b39-4631-92e7-b6ec8d849424
@KarishmaGhiya
chore: remove concise guidance validation spec
2695490 
The two focused baseline runs are complete, so remove the temporary workflow specification while retaining the concise diagnostic guidance and fixture identity.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 277bdab8-4b39-4631-92e7-b6ec8d849424
@KarishmaGhiya
test: focus VNet critical diagnosis gate
19dafeb 
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 277bdab8-4b39-4631-92e7-b6ec8d849424
@KarishmaGhiya
test: add focused VNet improvement run
4bec113 
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 277bdab8-4b39-4631-92e7-b6ec8d849424
@KarishmaGhiya
test: compare VNet arms without routing score
16f6723 
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 277bdab8-4b39-4631-92e7-b6ec8d849424
KarishmaGhiya and others added 22 commits November 10, 2026 12:13
@KarishmaGhiya
test: remove read-only hint from VNet prompts
afb8e49 
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 277bdab8-4b39-4631-92e7-b6ec8d849424
@KarishmaGhiya
test: add discriminating Function network scenarios
7db602e 
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 277bdab8-4b39-4631-92e7-b6ec8d849424
@KarishmaGhiya
test: keep VNet diagnostics evals read-only
049698c 
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 277bdab8-4b39-4631-92e7-b6ec8d849424
@KarishmaGhiya
test: add VNet baseline validation spec
b6b1e47 
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 277bdab8-4b39-4631-92e7-b6ec8d849424
@KarishmaGhiya
test: add Functions VNet SNAT live evals
8d0cb7f 
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 277bdab8-4b39-4631-92e7-b6ec8d849424
@KarishmaGhiya
test: require downloadable diagnostics bundle link
625b1d7 
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 277bdab8-4b39-4631-92e7-b6ec8d849424
@KarishmaGhiya
test: add focused SNAT baseline run spec
a75b4b1 
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 277bdab8-4b39-4631-92e7-b6ec8d849424
@KarishmaGhiya
test: add isolated Functions route repair eval
60e88fa 
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 277bdab8-4b39-4631-92e7-b6ec8d849424
@KarishmaGhiya
test: add focused route repair baseline spec
6e9e6e7 
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 277bdab8-4b39-4631-92e7-b6ec8d849424
@KarishmaGhiya
fix: preserve workspaces for deferred grading
fdb3a05 
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 277bdab8-4b39-4631-92e7-b6ec8d849424
@KarishmaGhiya
fix: restore Vally workspaces for deferred grading
b4f3e00 
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 277bdab8-4b39-4631-92e7-b6ec8d849424
@KarishmaGhiya
test: add mutable Functions NAT capacity eval
d04cc35 
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 277bdab8-4b39-4631-92e7-b6ec8d849424
@KarishmaGhiya
Merge origin/main into kaghiya/diagnostics-functions-vnet-live-evals
53598ab 
Resolve the sdk-agent-executor test import overlap in the merged tree.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 277bdab8-4b39-4631-92e7-b6ec8d849424
@KarishmaGhiya
test: accept equivalent SNAT bundle formats
681d6e7 
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 277bdab8-4b39-4631-92e7-b6ec8d849424
@KarishmaGhiya
test: add NAT capacity baseline run spec
29ffdeb 
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 277bdab8-4b39-4631-92e7-b6ec8d849424
@KarishmaGhiya
test: add focused Functions VNet hill climb spec
1cf4cef 
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 277bdab8-4b39-4631-92e7-b6ec8d849424
@KarishmaGhiya
test: add Functions held-out hill climb coverage
9777cdb 
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 277bdab8-4b39-4631-92e7-b6ec8d849424
@KarishmaGhiya
fix: retry NAT fixture private blob verification
da01075 
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 277bdab8-4b39-4631-92e7-b6ec8d849424
@KarishmaGhiya
test: make NAT capacity evaluation tool-agnostic
6c9984b 
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 277bdab8-4b39-4631-92e7-b6ec8d849424
@KarishmaGhiya
fix: ignore read-only NAT routing inspection
a0d1702 
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 277bdab8-4b39-4631-92e7-b6ec8d849424
@KarishmaGhiya
fix: prevent stale connections in routing fixture
2c8ea83 
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 277bdab8-4b39-4631-92e7-b6ec8d849424
@github-actions
feat: improve azure-diagnostics from evaluations
3099336 
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@github-actions

github-actions Bot commented Nov 13, 2026

Copy link
Copy Markdown
Contributor
Details# 🔍 Token Analysis Report

@github-copilot-for-azure/scripts@1.0.0 tokens
node --import tsx src/tokens/cli.ts compare --base origin/main --head HEAD --markdown

fatal: path 'evals/azure-skills/azure-diagnostics/fixture/functions-vnet-routing/README.md' exists on disk, but not in 'origin/main'

📊 Token Change Report

Comparing origin/main → HEAD

Summary

Metric Value
📈 Total Change +1,663 tokens (+42%)
Before 4,001 tokens
After 5,664 tokens
Files Changed 3

Changed Files

File Before After Change
evals/azure-skills/azure-diagnostics/fixture/functions-vnet-routing/README.md - 1,261 +1261
plugins/azure-skills/skills/azure-diagnostics/references/functions/triggers.md 2,129 2,461 +332 (+16%)
plugins/azure-skills/skills/azure-diagnostics/SKILL.md 1,872 1,942 +70 (+4%)

@github-copilot-for-azure/scripts@1.0.0 tokens
node --import tsx src/tokens/cli.ts check --markdown plugins/azure-skills/skills/azure-diagnostics/SKILL.md

📊 Token Limit Check Report

Checked: 1 files
Exceeded: 0 files

✅ All files within token limits

@github-copilot-for-azure/scripts@1.0.0 tokens
node --import tsx src/tokens/cli.ts check --markdown plugins/azure-skills/skills/azure-diagnostics/references/functions/triggers.md

📊 Token Limit Check Report

Checked: 1 files
Exceeded: 0 files

✅ All files within token limits

Automated token analysis. See skill authoring guidelines for best practices.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

Copilot code review Copilot
Copilot will automatically review once the pull request is marked ready for review

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@KarishmaGhiya