diff --git a/evals/azure-skills/azure-diagnostics/response-quality.eval.yaml b/evals/azure-skills/azure-diagnostics/response-quality.eval.yaml index 065c6b325..a3bb46107 100644 --- a/evals/azure-skills/azure-diagnostics/response-quality.eval.yaml +++ b/evals/azure-skills/azure-diagnostics/response-quality.eval.yaml @@ -512,3 +512,24 @@ stimuli: - Verification confirms checkpoint blobs advance, restarts resume near the last processed offset, authorization errors stop, and replayed events do not create duplicate business side effects. tags: *quality-tags graders: *quality-graders + + - name: "Resource details do not establish Azure Resource Health" + prompt: | + Evaluate this synthetic, offline diagnostic claim. Do not query or modify Azure. + + Evidence: + - `az resource show --ids ` returned the resource document + - The document contains `provisioningState: Succeeded` + - No Azure Resource Health response, workload probe, metric, or application + request result was collected + + Can this evidence establish that Azure Resource Health is Available or that + the workload is healthy? Explain the distinction and give the supported + read-only Resource Health lookup that an authorized operator could run. + rubric: + - The response states that `az resource show` provides resource configuration and provisioning metadata and cannot establish Azure Resource Health or workload health. + - The response identifies the current `Microsoft.ResourceHealth/availabilityStatuses/current` lookup, such as through `az rest`, and does not run it in this evidence-only scenario. + - The response notes that the lookup requires `Microsoft.ResourceHealth/AvailabilityStatuses/current/read` and applies only to supported resource types. + - 'The response distinguishes Azure platform availability states from application, guest, dependency, and end-to-end health; it does not infer `Available` or healthy from `provisioningState: Succeeded`.' + tags: *quality-tags + graders: *quality-graders diff --git a/plugins/azure-skills/skills/azure-diagnostics/SKILL.md b/plugins/azure-skills/skills/azure-diagnostics/SKILL.md index 0f071c3a2..7c9366c1b 100644 --- a/plugins/azure-skills/skills/azure-diagnostics/SKILL.md +++ b/plugins/azure-skills/skills/azure-diagnostics/SKILL.md @@ -83,7 +83,7 @@ Activate this skill when user wants to: ### Common Diagnostic Commands ```bash -# Check resource health +# Inspect resource configuration and provisioning metadata (not Resource Health) az resource show --ids RESOURCE_ID # View activity log az monitor activity-log list -g RG --max-events 20 @@ -141,13 +141,21 @@ mcp_azure_mcp_resourcehealth ### Using CLI ```bash -# Check specific resource health -az resource show --ids RESOURCE_ID +# Query the current Azure Resource Health availability status +RESOURCE_ID="/subscriptions//resourceGroups//providers///" +az rest --method get \ + --url "${RESOURCE_ID}/providers/Microsoft.ResourceHealth/availabilityStatuses/current?api-version=2025-05-01" # Check recent activity az monitor activity-log list -g RG --max-events 20 ``` +`az resource show` returns resource configuration and provisioning metadata; it does not query Azure Resource Health and cannot by itself establish that a resource is healthy or `Available`. + +The Resource Health request requires Azure authentication and the `Microsoft.ResourceHealth/AvailabilityStatuses/current/read` operation at the applicable scope. It applies only to resource types supported by Azure Resource Health. Interpret `availabilityState` as Azure platform availability evidence (`Available`, `Unavailable`, `Degraded`, or `Unknown`), not as application or workload health. `Available` does not prove the guest, application, dependencies, or user path are healthy, and `Unknown` must not be treated as healthy. + +If the resource type is unsupported, the status is unavailable, permission is denied, or the health query fails, report that limitation and leave Resource Health availability unestablished; do not substitute provisioning success or invent a healthy result. + --- ## References