Skip to content

Add ARM activity validation for Terraform quickstarts - #501

Merged
Steven Ma (stevenjma) merged 8 commits into
masterfrom
stema-microsoft-improve-validation-workflow
Nov 10, 2026
Merged

Steven Ma (stevenjma) merged 8 commits into
masterfrom
stema-microsoft-improve-validation-workflow

Conversation

@stevenjma

@stevenjma Steven Ma (stevenjma) commented Aug 18, 2026 •

Copy link
Copy Markdown
Contributor

Why

The current quickstart E2E workflow runs PR code with repository-owned Azure credentials and cannot run for forks.

This PR changes the model: contributors deploy the sample in their own subscription, record an ARM correlation ID, and a maintainer verifies the resulting ARM activity with /validate.

How it works

  1. The contributor deploys one changed quickstart with a new ARM_CORRELATION_REQUEST_ID.
  2. metadata.json records the correlation ID and UTC start time.
  3. The automatic pre-check validates Terraform and the metadata.
  4. A maintainer comments /validate.
  5. Trusted code from master queries the three approved regional ARMProd stores.
  6. GitHub publishes terraform-deployment-validation on the exact PR commit.

Security boundary

  • /validate never executes quickstart code from the PR.
  • The OIDC application is read-only, subscriptionless, and has no client secret.
  • Correlation values are passed to Kusto as query parameters.
  • Validation fails if a regional query fails or telemetry does not stabilize.
  • PRs changing the validation contract cannot run /validate; those changes must merge and be canary-tested separately.
  • Changes under test/** retain a separate internal E2E workflow. Forks do not receive repository OIDC credentials.

What the check proves

The contributor-supplied correlation ID has:

  • at least one completed or ARM-accepted Terraform write; and
  • no observed final HTTP write failures.

It does not prove:

  • the correlation ID belongs to this PR or applied source;
  • terraform apply exited successfully; or
  • a 202 Accepted operation eventually provisioned successfully.

These are explicit limits of the good-faith evidence model. Code review and static checks remain required.

Reviewer guide

Area Files
/validate authorization and trusted checkout .github/workflows/e2e.yaml
ARMProd query and result handling .github/scripts/validate-arm-correlation.py
Metadata freshness and format .github/scripts/validate-quickstart-metadata.py, .github/schemas/quickstart-metadata.schema.json
Automatic PR checks .github/workflows/pr-check.yaml
Internal test-harness E2E .github/workflows/e2e-pr.yaml
Contributor flow and limitations quickstart/CONTRIBUTE.md

Validation completed

  • Python validator suites run in the required pre-check.
  • Workflow YAML and metadata schema parse successfully.
  • Live ARMProd test queried all three regions and returned 14 completed writes, 0 failures.
  • ARMProd access was approved under IcM 865765936.
  • Deep adversarial review completed; confirmed findings were fixed.

Rollout

issue_comment workflows load from the default branch, so the final canary happens after merge on a fresh single-quickstart PR with valid metadata.

Steven (stemaMSFT) and others added 5 commits August 18, 2026 11:21
Add correlation-backed metadata validation and a maintainer-triggered /validate workflow for Terraform quickstarts.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Filter changed-files output to existing quickstart sample directories before invoking metadata and Terraform validation.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Restore PR coverage for E2E test changes and scope metadata freshness checks to deployable or metadata updates.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Configure the validation workflow for a dedicated corporate OIDC application with no Azure subscription roles.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Query the three regional ARM request stores with stable fail-closed snapshots and distinguish synchronous completion from asynchronous acceptance.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@stevenjma

Copy link
Copy Markdown
Contributor Author

@microsoft-github-policy-service agree company="Microsoft"

Steven Ma (stevenjma) and others added 2 commits October 8, 2026 13:18
Block validation-contract changes from /validate, run validator tests in CI, align timestamp contracts, and state the contributor-supplied evidence boundary precisely.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Tighten contributor documentation and reviewer-facing workflow names without changing validation behavior.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@stevenjma Steven Ma (stevenjma) changed the title Improve quickstart deployment validation Add ARM activity validation for Terraform quickstarts Oct 21, 2026

@Vieran Chunyan (Vieran) left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for opening this!

Comment thread .github/workflows/e2e.yaml Resolved
Comment thread .github/workflows/pr-check.yaml Resolved
Require repository write permission for /validate, include deleted quickstart files in change detection, and fail closed on Kusto partial query failures.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@stevenjma

Copy link
Copy Markdown
Contributor Author

#sign-off

@stevenjma
Steven Ma (stevenjma) merged commit 83a3384 into master Nov 10, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants