Skip to content

Harden unskip closed tests workflow - #11694

Merged
Amaury Levé (Evangelink) merged 6 commits into
microsoft:mainfrom
Evangelink:dev/amauryleve/harden-unskip-workflow
Oct 2, 2026
Merged

Amaury Levé (Evangelink) merged 6 commits into
microsoft:mainfrom
Evangelink:dev/amauryleve/harden-unskip-workflow

Conversation

@Evangelink

@Evangelink Amaury Levé (Evangelink) commented Oct 1, 2026 •

Copy link
Copy Markdown
Member

Summary

  • replace prompt-only grep/edit trust with the immutable dotnet/skills Unskip Closed Tests package from 57e48d3619bf44b9307a3197239d398b0287b25c (dotnet/skills#1254)
  • bind every candidate to the exact commit, blob, source span, syntax-derived declaration/test identities, and canonical GitHub references; revalidate before editing and publishing
  • add a cross-platform PowerShell TestFX verification hook that bootstraps the pinned toolchain, packs acceptance projects when needed, maps source to one test project, runs every exact FQN, and writes each requested TRX
  • add PowerShell CI coverage for request validation, duplicate/fabricated identities, project mapping, portable target selection, exact command construction, stale revisions, missing TRX, acceptance packing, and replacement of the packaged fail-closed placeholder

The workflow now fails closed for open, not-planned, inaccessible, ambiguous, stale, malformed, skipped, zero-result, mismatched, or non-passing candidates. The read-only model can only select trusted manifest IDs; a deterministic safe-output job owns edits, verification, branch freshness, and creation/readback of at most one draft PR.

Validation

  • python agentic-workflows\unskip-closed-tests\tests\run_tests.py in dotnet/skills@57e48d3619bf44b9307a3197239d398b0287b25c — 22 passed
  • dotnet restore UnskipClosedTests.Tool.csproj --locked-mode -bl:{} — passed
  • dotnet build UnskipClosedTests.Tool.csproj --no-restore -bl:{} — passed, 0 warnings / 0 errors
  • pinned gh-aw v0.89.21 compile unskip-closed-tests --strict — passed
  • native actionlint v1.7.12 on the generated lock and helper workflow — passed
  • python .github/scripts/check_source_bom.py — passed
  • pwsh -NoLogo -NoProfile -File .github/scripts/test_unskip_closed_tests_verify.ps1 — 10 passed
  • python .github/scripts/check_action_pins.py — passed, 2545 references across 54 files
  • packaged PowerShell placeholder — failed closed with exit code 2
  • live TestFX inventory/resolve — 1 candidate, 0 eligible; Progress remains on screen when output is written from test #3491 resolved to closed/not_planned, Testing Platform overwriting/removing user console output #4425 to open, and source stayed unchanged
  • real PowerShell exact-FQN verification — 1 executed, 1 passed, 0 skipped with the requested TRX containing MSTest.Analyzers.Test.IgnoreShouldHaveJustificationAnalyzerTests.WhenTestMethodHasIgnoreWithMessage_NoDiagnostic
  • stale-revision and malformed PowerShell requests — rejected without producing TRX

The package regression suite also covers repeated Ignore text at distinct sites, fabricated anchors/FQNs, false nesting, class-level ambiguity/inheritance/partial declarations, malformed GitHub evidence, inaccessible/not-planned/open references, revision changes, zero/all-skipped/mismatched execution, and post-verification eligibility changes.

Install the verified dotnet/skills workflow package at 7bdab53812ed1ce4fe2cb6b0cf84e17c8ff0f097 and add TestFX-specific exact-FQN TRX verification.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot AI balanced review requested due to automatic review settings October 1, 2026 20:29

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

The verifier invocation and TRX path checks currently reject every candidate, while generated titles bypass duplicate-PR detection.

Review effort: Balanced
Findings: 2 High severity · 1 Medium severity

Open (3)
What changed in this PR

Replaces the prompt-driven unskip workflow with a pinned, deterministic workflow package and TestFX-specific verification.

Changes:

  • Adds source-bound candidate collection and GitHub eligibility validation.
  • Adds guarded edits with exact TRX execution verification.
  • Adds the TestFX verification hook, CI tests, and workflow documentation.
File Description
.github/​workflows/​unskip-closed-tests.md Configures the hardened workflow.
.github/​workflows/​unskip-closed-tests.lock.yml Compiles the workflow definition.
.github/​workflows/​unskip-closed-tests.config.json Defines inventory and verification settings.
.github/​workflows/​unskip-closed-tests-verify.sh Provides a fail-closed default hook.
.github/​workflows/​unskip-closed-tests-prepare.md Collects candidates and publishes verified edits.
.github/​workflows/​unskip-closed-tests-shared.md Restricts agent tools and outputs.
.github/​workflows/​test-unskip-closed-tests.yml Runs verification-hook tests.
.github/​workflows/​README.md Updates the workflow catalog.
.github/​agents/​unskip-closed-tests.agent.md Defines the read-only planner.
.github/​scripts/​unskip_closed_tests_verify.py Builds and runs exact TestFX tests.
.github/​scripts/​test_unskip_closed_tests_verify.py Tests the repository hook.
.github/​workflows/​unskip-closed-tests-tool/​ApplyEngine.cs Applies, verifies, and reports edits.
.github/​workflows/​unskip-closed-tests-tool/​ConfigLoader.cs Validates workflow configuration.
.github/​workflows/​unskip-closed-tests-tool/​GitRepository.cs Validates Git source identity.
.github/​workflows/​unskip-closed-tests-tool/​InventoryEngine.cs Inventories syntax-bound Ignore sites.
.github/​workflows/​unskip-closed-tests-tool/​IssueResolver.cs Resolves GitHub eligibility.
.github/​workflows/​unskip-closed-tests-tool/​JsonSupport.cs Handles canonical JSON and digests.
.github/​workflows/​unskip-closed-tests-tool/​ManifestValidator.cs Validates trusted manifests.
.github/​workflows/​unskip-closed-tests-tool/​Models.cs Defines workflow data contracts.
.github/​workflows/​unskip-closed-tests-tool/​PathRules.cs Enforces safe repository paths.
.github/​workflows/​unskip-closed-tests-tool/​Program.cs Implements tool commands and exits.
.github/​workflows/​unskip-closed-tests-tool/​TrxVerifier.cs Validates exact passing TRX results.
.github/​workflows/​unskip-closed-tests-tool/​UnskipClosedTests.Tool.csproj Defines the isolated helper tool.
.github/​workflows/​unskip-closed-tests-tool/​Directory.Build.props Isolates inherited build properties.
.github/​workflows/​unskip-closed-tests-tool/​Directory.Build.targets Isolates inherited build targets.
.github/​workflows/​unskip-closed-tests-tool/​Directory.Packages.props Disables repository central versions.
.github/​workflows/​unskip-closed-tests-tool/​global.json Pins the helper SDK family.
.github/​workflows/​unskip-closed-tests-tool/​packages.lock.json Locks helper dependencies.

💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread .github/scripts/unskip_closed_tests_verify.py Outdated
Comment thread .github/workflows/unskip-closed-tests.config.json
Comment thread .github/workflows/unskip-closed-tests-tool/ApplyEngine.cs Outdated
@microsoft-github-policy-service microsoft-github-policy-service Bot added the state/needs-review Awaiting review from the team. label Oct 1, 2026
@github-actions github-actions Bot removed the state/needs-review Awaiting review from the team. label Oct 1, 2026
@Evangelink
Amaury Levé (Evangelink) marked this pull request as ready for review October 2, 2026 07:43
@github-actions github-actions Bot added the state/needs-review Awaiting review from the team. label Oct 2, 2026
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Copilot AI balanced review requested due to automatic review settings October 2, 2026 07:44
@github-actions github-actions Bot added state/needs-review Awaiting review from the team. and removed state/needs-review Awaiting review from the team. labels Oct 2, 2026
@Evangelink
Amaury Levé (Evangelink) enabled auto-merge (squash) October 2, 2026 07:44
Allow verification output under the dedicated Git metadata directory and align generated PR titles with duplicate detection.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@github-actions github-actions Bot added state/needs-review Awaiting review from the team. and removed state/needs-review Awaiting review from the team. labels Oct 2, 2026
@microsoft-github-policy-service microsoft-github-policy-service Bot added state/needs-review Awaiting review from the team. and removed state/needs-review Awaiting review from the team. labels Oct 2, 2026
@github-actions github-actions Bot added state/needs-review Awaiting review from the team. and removed state/needs-review Awaiting review from the team. labels Oct 2, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

Unresolved security and correctness gaps can expose credentials or publish unverified edits.

Review effort: Balanced
Findings: 2 High severity

Open (2)
Resolved since last review (3)
Previously missed (2)

In code that hasn't changed since last review

Medium severity PR base commit is not revalidated after branch advance

.github/​workflows/​unskip-closed-tests-prepare.md:270

The default branch can advance after the check at line 239 but before PR creation. In that race, this readback verifies only the branch name, leaving a PR published against a newer base even though its edit and test evidence came from EXPECTED_COMMIT. Read back baseRefOid as well and close the PR/delete its branch when it differs from the verified commit.

Medium severity Issue URL parser accepts malformed numeric suffixes

.github/​workflows/​unskip-closed-tests-tool/​InventoryEngine.cs:574

This accepts a malformed URL such as issues/123abc as issue 123. If issue 123 is completed, unrelated text can make the candidate eligible and remove its Ignore; require a non-identifier boundary after the captured number.

This issue also appears in the following locations of the same file:

  • line 579
  • line 584

Comment thread .github/workflows/unskip-closed-tests-prepare.md Outdated
Comment thread .github/workflows/unskip-closed-tests-tool/ApplyEngine.cs Outdated
@microsoft-github-policy-service microsoft-github-policy-service Bot added state/needs-review Awaiting review from the team. and removed state/needs-review Awaiting review from the team. labels Oct 2, 2026
@github-actions github-actions Bot added state/needs-review Awaiting review from the team. and removed state/needs-review Awaiting review from the team. labels Oct 2, 2026
@microsoft-github-policy-service microsoft-github-policy-service Bot removed the state/needs-review Awaiting review from the team. label Oct 2, 2026
@github-actions github-actions Bot added the state/approved Proposal approved; ready for implementation. label Oct 2, 2026
@microsoft-github-policy-service microsoft-github-policy-service Bot added state/needs-review Awaiting review from the team. and removed state/needs-review Awaiting review from the team. labels Oct 2, 2026
@github-actions github-actions Bot removed state/needs-review Awaiting review from the team. state/approved Proposal approved; ready for implementation. labels Oct 2, 2026
@microsoft-github-policy-service microsoft-github-policy-service Bot added the state/needs-review Awaiting review from the team. label Oct 2, 2026
@github-actions github-actions Bot added state/approved Proposal approved; ready for implementation. and removed state/needs-review Awaiting review from the team. state/approved Proposal approved; ready for implementation. labels Oct 2, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

Verification currently exposes write-scoped checkout credentials to executed test code, and its publication job lacks an overall timeout.

Review effort: Balanced
Findings: 1 High severity · 1 Medium severity

Open (2)
Resolved since last review (4)

Comment thread .github/workflows/unskip-closed-tests-prepare.md Outdated
Comment thread .github/workflows/unskip-closed-tests-prepare.md
@microsoft-github-policy-service microsoft-github-policy-service Bot added the state/needs-review Awaiting review from the team. label Oct 2, 2026
@github-actions github-actions Bot added state/approved Proposal approved; ready for implementation. and removed state/needs-review Awaiting review from the team. state/approved Proposal approved; ready for implementation. labels Oct 2, 2026
@Evangelink
Amaury Levé (Evangelink) enabled auto-merge (squash) October 2, 2026 09:14
Run repository code in a bounded read-only verification job, package the verified edits by content hash, and apply them only in a fresh authenticated publisher checkout.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot AI balanced review requested due to automatic review settings October 2, 2026 09:22
@microsoft-github-policy-service microsoft-github-policy-service Bot added state/needs-review Awaiting review from the team. and removed state/needs-review Awaiting review from the team. labels Oct 2, 2026
@Evangelink
Amaury Levé (Evangelink) merged commit 4da1a36 into microsoft:main Oct 2, 2026
41 of 45 checks passed
@Evangelink
Amaury Levé (Evangelink) deleted the dev/amauryleve/harden-unskip-workflow branch October 2, 2026 09:23
@microsoft-github-policy-service microsoft-github-policy-service Bot removed state/needs-review Awaiting review from the team. state/approved Proposal approved; ready for implementation. labels Oct 2, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

Publication freshness, OS-conditioned target coverage, and the publisher timeout remain incomplete.

Review effort: Balanced
Findings: 1 High severity · 1 Medium severity

Open (2)
Resolved since last review (4)

Comment on lines +323 to +329
all(.retained_candidates[];
. as $retained |
any($manifest[0].candidates[];
.candidate_id == $retained.candidate_id and
.path == $retained.path and
.decision.eligible == true and
((.owner.test_fqns | sort) == ($retained.test_fqns | sort)))) and
} else {
[string] $properties.TargetFrameworks
}
return @(Select-TargetFrameworks -Frameworks @($frameworkText -split ';' | Where-Object { $_ }))
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

state/needs-review Awaiting review from the team.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants