test: add live Functions VNet routing diagnostics by KarishmaGhiya · Pull Request #3481 · microsoft/GitHub-Copilot-for-Azure

KarishmaGhiya · 2026-11-03T23:34:03Z

Description

Adds live Azure Functions Flex Consumption VNet diagnostics and remediation
coverage for routing faults, outbound connection pressure, support-bundle
generation, and persistent outbound-capacity pressure.

Provisions a Flex Function App, delegated integration subnet, separate
private-endpoint subnet, private DNS, keyless host/test storage, route table,
Application Insights, and Log Analytics.
Adds read-only routing scenarios that correlate Function symptoms, private
endpoint DNS, route changes, and Activity Log evidence.
Adds a fresh-resource mutable route-repair scenario that removes only the
causal /32 -> None route and verifies recovery with an actual blob read.
Adds SNAT/connection-pressure diagnosis and downloadable support-bundle
scenarios, including deterministic checks for required files, structured
evidence, redaction, and a clickable ZIP link.
Adds a fresh-resource mutable NAT-capacity scenario that requires application
connection improvements first, then provisions a Standard public IP and NAT
Gateway, associates only the Function integration subnet, and verifies that
the observed outbound IP matches the NAT public IP under recovery load.
Preserves Vally workspaces across split generation and deferred grading while
retaining Vally's required original workDir, then removes restored and
snapshot workspaces after grading.
Adds fixture coordination fallbacks, Microsoft.App registration, JSONL-safe
logging, bounded provisioning timeouts, and targeted retries for Azure
Network AnotherOperationInProgress failures.
Includes the suites in Azure Diagnostics hill climbing with a validated plan
of 44 development prompts, 10 held-out prompts, and 608 maximum answer/judge
calls.

Validation

npm run build — passed.
cd tests && npm test -- --run vally/__tests__/sdk-agent-executor.test.ts skill-improvement/__tests__/evaluation.test.ts — passed, 27/27 tests.
cd tests && npm test -- --run skill-improvement/__tests__/config.test.ts — passed, 74/74 tests.
cd tests && npm run typecheck — passed.
cd tests && npm run lint -- --quiet — passed.
cd scripts && npm run vally validate-stimulus — passed.
Strict Vally lint for functions-vnet-snat.eval.yaml,
functions-vnet-snat-comparative.eval.yaml, and
functions-vnet-nat-capacity.eval.yaml — passed.
JavaScript syntax checks for the outbound and client-IP echo functions —
passed.
cd tests && npm run skill-improvement -- validate --config ../tests/skill-improvement/specs/azure-diagnostics.yaml — passed; 44 development prompts, 10 held-out prompts, and 608 maximum calls.
Initial live fixture run 41934541342 — passed all 152 baseline generations/judgments; both conditions passed 38/38 prompts and the VNet suite passed 8/8 raw trials.
Realistic routing prompt run 41940768805 — passed 8/8 targeted trials.
Mutable route-repair run 42945073201 — completed successfully across all four baseline arms with fresh resources per trial.
Corrected SNAT baseline run 43118444840 — completed 16/16 answer generations and 16/16 judge calls. All workspace-backed file graders executed successfully. Artifact replay verified the corrected link and structured-evidence patterns against all 8/8 support-bundle outputs; two weak-evidence prompt-judge failures remain substantive rather than infrastructure failures.
Latest PR checks — all passed, including CodeQL, ESLint, Tests, Scripts,
skill structure, token analysis, line endings, shell permissions, markdown
references, and evaluations.

Checklist

Tests pass locally (cd tests && npm test)
Title has one of the prefixes: fix:, feat:, feature:, chore:, misc:, test:, eval:
If modifying skill descriptions: verified routing correctness with integration tests (In tests/, npm run test:vally -- --plugin <plugin-dirname> --skill <skill>) — Not applicable; no skill description changed.

Related Issues

Extends the merged Azure Diagnostics evaluation work in #3374 and the live
diagnostics fixture work in #3477.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 277bdab8-4b39-4631-92e7-b6ec8d849424

github-actions · 2026-11-03T23:35:43Z

Details

# 🔍 Token Analysis Report

@github-copilot-for-azure/scripts@1.0.0 tokens
node --import tsx src/tokens/cli.ts compare --base origin/main --head HEAD --markdown

fatal: path 'evals/azure-skills/azure-diagnostics/fixture/functions-vnet-routing/README.md' exists on disk, but not in 'origin/main'

📊 Token Change Report

Comparing origin/main → HEAD

Summary

Metric	Value
📈 Total Change	+1,564 tokens (+39%)
Before	4,001 tokens
After	5,565 tokens
Files Changed	3

Changed Files

File	Before	After	Change
`evals/azure-skills/azure-diagnostics/fixture/functions-vnet-routing/README.md`	-	1,261	+1261
`plugins/azure-skills/skills/azure-diagnostics/references/functions/triggers.md`	2,129	2,362	+233 (+11%)
`plugins/azure-skills/skills/azure-diagnostics/SKILL.md`	1,872	1,942	+70 (+4%)

@github-copilot-for-azure/scripts@1.0.0 tokens
node --import tsx src/tokens/cli.ts check --markdown plugins/azure-skills/skills/azure-diagnostics/SKILL.md

📊 Token Limit Check Report

Checked: 1 files
Exceeded: 0 files

✅ All files within token limits

@github-copilot-for-azure/scripts@1.0.0 tokens
node --import tsx src/tokens/cli.ts check --markdown plugins/azure-skills/skills/azure-diagnostics/references/functions/triggers.md

📊 Token Limit Check Report

Checked: 1 files
Exceeded: 0 files

✅ All files within token limits

Automated token analysis. See skill authoring guidelines for best practices.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 277bdab8-4b39-4631-92e7-b6ec8d849424

Use the Function App hostNames collection with the deterministic Azure Websites hostname fallback because defaultHostName is absent for some Flex Consumption resource responses. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 277bdab8-4b39-4631-92e7-b6ec8d849424

Retry bounded Azure deployment failures when a dependent network operation is still in progress, while preserving immediate failure for all other ARM errors. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 277bdab8-4b39-4631-92e7-b6ec8d849424

Copilot

🟡 Changes recommended

Windows shell parsing can break resource discovery, and deployment retries can continue after known lease loss, risking failed provisioning and orphaned deployments.

4 open issues

1 file not reviewed

evals/azure-skills/azure-diagnostics/fixture/functions-vnet-routing/app/package-lock.json: Generated file

What changed in this PR

Extends the existing live Azure Diagnostics evaluations with a temporary Flex Consumption fixture that isolates private-storage routing failures while keeping the Function host healthy.

Changes:

Adds key-protected health and blob-read endpoints, validates healthy private connectivity, then injects a narrow /32 blackhole route.
Adds two read-only diagnostic evaluations and includes them in the skill-improvement plan with a 536-call ceiling.
Updates fixture provisioning and workflows with coordination fallbacks, configurable timeouts, deployment retries, and resource-provider registration.

File	Description
`tests/skill-improvement/specs/azure-diagnostics.yaml`	Includes the routing suite and adjusts call budgets.
`tests/skill-improvement/__tests__/workflow.test.ts`	Checks coordination fallbacks and provider registration.
`tests/skill-improvement/__tests__/config.test.ts`	Checks plan budgets and fixture constraints.
`tests/azure-fixtures/provision-fixture.ts`	Adds timeout overrides, deployment retries, and stderr progress logging.
`tests/azure-fixtures/fixture-common.ts`	Defines the optional provisioning timeout.
`tests/azure-fixtures/__tests__/provision-workflow-fixture.test.ts`	Covers coordination timing, retries, and logging.
`tests/azure-fixtures/__tests__/provision-budget.test.ts`	Covers timeout overrides and invalid values.
`evals/azure-skills/azure-diagnostics/functions-vnet-routing.eval.yaml`	Adds symptom-first and change-correlation evaluations.
`evals/azure-skills/azure-diagnostics/fixture/functions-vnet-routing/README.md`	Documents topology, fault injection, and safety constraints.
`evals/azure-skills/azure-diagnostics/fixture/functions-vnet-routing/post-provision.ts`	Deploys the app, verifies baseline connectivity, and injects the fault.
`evals/azure-skills/azure-diagnostics/fixture/functions-vnet-routing/main.bicep`	Defines Functions, networking, keyless storage, and telemetry resources.
`evals/azure-skills/azure-diagnostics/fixture/functions-vnet-routing/azure-manifest.json`	Configures temporary reuse and a 40-minute provisioning budget.
`evals/azure-skills/azure-diagnostics/fixture/functions-vnet-routing/app/src/functions/read-blob.js`	Adds bounded managed-identity blob reads and diagnostic logging.
`evals/azure-skills/azure-diagnostics/fixture/functions-vnet-routing/app/src/functions/health.js`	Adds a storage-independent health endpoint.
`evals/azure-skills/azure-diagnostics/fixture/functions-vnet-routing/app/package.json`	Declares the runtime and application dependencies.
`evals/azure-skills/azure-diagnostics/fixture/functions-vnet-routing/app/package-lock.json`	Locks application dependency versions.
`evals/azure-skills/azure-diagnostics/fixture/functions-vnet-routing/app/host.json`	Disables telemetry sampling for diagnostic evidence.
`.github/workflows/test-all-integration.yml`	Adds coordination settings and Microsoft.App registration.
`.github/workflows/skill-improvement.yml`	Adds coordination settings and Microsoft.App registration.

_{Review effort: Balanced}

+  try {
+    return execFileSync(AZ_COMMAND, args, {
+      encoding: "utf8",
+      shell: process.platform === "win32",


+            `${FIXTURE_COMPONENT_TAG}=${config.resourceGroupNameBase}`,
+            retentionTag,
+          ],
+          dependencies.runAz,


+    return execFileSync(AZ_COMMAND, args, {
+      encoding: "utf8",
+      shell: process.platform === "win32",
+      timeout,


Replace test-authored diagnostic instructions with symptom-first customer reports and add a temporary baseline-only specification for targeted validation. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 277bdab8-4b39-4631-92e7-b6ec8d849424

The baseline-only workflow completed successfully, so remove the temporary targeted run specification from the permanent branch. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 277bdab8-4b39-4631-92e7-b6ec8d849424

Seed deterministic outbound connection-pressure telemetry on the live Flex topology, add customer-style diagnosis and ZIP support-bundle evals, and include them in Azure Diagnostics hill climbing. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 277bdab8-4b39-4631-92e7-b6ec8d849424

Make critical Flex topology, causal evidence, remediation, and recovery details explicit final-answer gates so tool-only evidence cannot satisfy the routing evals. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 277bdab8-4b39-4631-92e7-b6ec8d849424

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 277bdab8-4b39-4631-92e7-b6ec8d849424

Resolve the sdk-agent-executor test import overlap in the merged tree. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 277bdab8-4b39-4631-92e7-b6ec8d849424

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 277bdab8-4b39-4631-92e7-b6ec8d849424

test: add live Functions VNet routing diagnostics

1e82730

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 277bdab8-4b39-4631-92e7-b6ec8d849424

KarishmaGhiya added 4 commits November 3, 2026 16:12

test: avoid leaking Functions routing hypotheses

e44febd

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 277bdab8-4b39-4631-92e7-b6ec8d849424

test: scope Functions failure to observed evidence

cfbdae2

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 277bdab8-4b39-4631-92e7-b6ec8d849424

test: make Functions routing prompts user-like

e5ff0ea

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 277bdab8-4b39-4631-92e7-b6ec8d849424

test: simplify Functions eval safety constraint

a4ee2de

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 277bdab8-4b39-4631-92e7-b6ec8d849424

KarishmaGhiya had a problem deploying to skill-improvement-evals November 4, 2026 00:38 — with GitHub Actions Failure

fix: read private endpoint IP from its NIC

1627d35

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 277bdab8-4b39-4631-92e7-b6ec8d849424

KarishmaGhiya had a problem deploying to skill-improvement-evals November 4, 2026 00:57 — with GitHub Actions Failure

fix: wait for Flex Function host readiness

95a4272

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 277bdab8-4b39-4631-92e7-b6ec8d849424

KarishmaGhiya had a problem deploying to skill-improvement-evals November 4, 2026 01:38 — with GitHub Actions Failure

KarishmaGhiya had a problem deploying to skill-improvement-evals November 4, 2026 23:17 — with GitHub Actions Failure

KarishmaGhiya temporarily deployed to skill-improvement-evals November 4, 2026 23:32 — with GitHub Actions Inactive

KarishmaGhiya marked this pull request as ready for review November 5, 2026 00:07

KarishmaGhiya requested a review from a team as a code owner November 5, 2026 00:07

Copilot AI balanced review requested due to automatic review settings November 5, 2026 00:07

Copilot started reviewing on behalf of KarishmaGhiya November 5, 2026 00:07

View session

Copilot AI reviewed Nov 5, 2026

View reviewed changes

KarishmaGhiya temporarily deployed to skill-improvement-evals November 5, 2026 00:21 — with GitHub Actions Inactive

github-actions Bot mentioned this pull request Nov 5, 2026

[repo-status] Weekly Status: Oct 30 – Nov 5, 2026 #3486

Open

KarishmaGhiya had a problem deploying to skill-improvement-evals November 5, 2026 21:36 — with GitHub Actions Failure

KarishmaGhiya added 2 commits November 5, 2026 13:46

fix: allow delayed Functions telemetry ingestion

96296fc

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 277bdab8-4b39-4631-92e7-b6ec8d849424

KarishmaGhiya had a problem deploying to skill-improvement-evals November 5, 2026 21:50 — with GitHub Actions Failure

fix: preserve eval workspaces for deferred grading

511bfc3

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 277bdab8-4b39-4631-92e7-b6ec8d849424

test: add VNet baseline validation spec

b6b1e47

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 277bdab8-4b39-4631-92e7-b6ec8d849424

KarishmaGhiya temporarily deployed to skill-improvement-evals November 11, 2026 00:59 — with GitHub Actions Inactive

KarishmaGhiya added 3 commits November 10, 2026 17:40

test: add Functions VNet SNAT live evals

8d0cb7f

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 277bdab8-4b39-4631-92e7-b6ec8d849424

test: require downloadable diagnostics bundle link

625b1d7

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 277bdab8-4b39-4631-92e7-b6ec8d849424

test: add focused SNAT baseline run spec

a75b4b1

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 277bdab8-4b39-4631-92e7-b6ec8d849424

KarishmaGhiya had a problem deploying to skill-improvement-evals November 11, 2026 01:53 — with GitHub Actions Failure

KarishmaGhiya added 2 commits November 10, 2026 18:11

test: add isolated Functions route repair eval

60e88fa

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 277bdab8-4b39-4631-92e7-b6ec8d849424

test: add focused route repair baseline spec

6e9e6e7

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 277bdab8-4b39-4631-92e7-b6ec8d849424

KarishmaGhiya temporarily deployed to skill-improvement-evals November 11, 2026 02:18 — with GitHub Actions Inactive

fix: preserve workspaces for deferred grading

fdb3a05

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 277bdab8-4b39-4631-92e7-b6ec8d849424

KarishmaGhiya had a problem deploying to skill-improvement-evals November 11, 2026 22:51 — with GitHub Actions Failure

fix: restore Vally workspaces for deferred grading

b4f3e00

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 277bdab8-4b39-4631-92e7-b6ec8d849424

KarishmaGhiya temporarily deployed to skill-improvement-evals November 11, 2026 23:15 — with GitHub Actions Inactive

KarishmaGhiya added 4 commits November 11, 2026 15:17

test: add mutable Functions NAT capacity eval

d04cc35

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 277bdab8-4b39-4631-92e7-b6ec8d849424

Merge origin/main into kaghiya/diagnostics-functions-vnet-live-evals

53598ab

Resolve the sdk-agent-executor test import overlap in the merged tree. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 277bdab8-4b39-4631-92e7-b6ec8d849424

test: accept equivalent SNAT bundle formats

681d6e7

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 277bdab8-4b39-4631-92e7-b6ec8d849424

test: add NAT capacity baseline run spec

29ffdeb

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 277bdab8-4b39-4631-92e7-b6ec8d849424

KarishmaGhiya had a problem deploying to skill-improvement-evals November 12, 2026 03:20 — with GitHub Actions Failure

KarishmaGhiya added 3 commits November 11, 2026 19:46

test: add focused Functions VNet hill climb spec

1cf4cef

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 277bdab8-4b39-4631-92e7-b6ec8d849424

test: add Functions held-out hill climb coverage

9777cdb

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 277bdab8-4b39-4631-92e7-b6ec8d849424

fix: retry NAT fixture private blob verification

da01075

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 277bdab8-4b39-4631-92e7-b6ec8d849424

KarishmaGhiya temporarily deployed to skill-improvement-evals November 12, 2026 07:23 — with GitHub Actions Inactive

github-actions Bot mentioned this pull request Nov 12, 2026

[repo-status] Weekly Status – week of Nov 6–12, 2026 #3518

Open

test: make NAT capacity evaluation tool-agnostic

6c9984b

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 277bdab8-4b39-4631-92e7-b6ec8d849424

KarishmaGhiya temporarily deployed to skill-improvement-evals November 12, 2026 19:21 — with GitHub Actions Inactive

fix: ignore read-only NAT routing inspection

a0d1702

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 277bdab8-4b39-4631-92e7-b6ec8d849424

KarishmaGhiya had a problem deploying to skill-improvement-evals November 12, 2026 22:36 — with GitHub Actions Failure

KarishmaGhiya had a problem deploying to skill-improvement-evals November 12, 2026 23:28 — with GitHub Actions Failure

fix: prevent stale connections in routing fixture

2c8ea83

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 277bdab8-4b39-4631-92e7-b6ec8d849424

KarishmaGhiya deployed to skill-improvement-evals November 13, 2026 01:29 — with GitHub Actions Active

Navigation Menu

Repository navigation

test: add live Functions VNet routing diagnostics - #3481

test: add live Functions VNet routing diagnostics#3481
KarishmaGhiya wants to merge 51 commits into
mainfrom
kaghiya/diagnostics-functions-vnet-live-evals

KarishmaGhiya commented Nov 3, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Nov 3, 2026 •

edited

Loading

📊 Token Change Report

Summary

Changed Files

📊 Token Limit Check Report

✅ All files within token limits

📊 Token Limit Check Report

✅ All files within token limits

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

This branch was successfully deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

KarishmaGhiya commented Nov 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Validation

Checklist

Related Issues

Uh oh!

github-actions Bot commented Nov 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

📊 Token Change Report

Summary

Changed Files

📊 Token Limit Check Report

✅ All files within token limits

📊 Token Limit Check Report

✅ All files within token limits

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Uh oh!

This branch was successfully deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

KarishmaGhiya commented Nov 3, 2026 •

edited

Loading

github-actions Bot commented Nov 3, 2026 •

edited

Loading