feat: add dependency-update PR validator skill

[haeter525]

Description

Adds a Claude Code skill that validates dependabot dependency-update PRs against quark-engine's CI and drafts a merge/escalate recommendation for a human to act on.

Key Changes

• Add dependency-update-pr-validator skill (.claude/skills/dependency-update-pr-validator/): SKILL.md, validation scripts, examples, and evals.
  • Parses a dependabot PR — package, current version, target version.
  • Verifies the target version is actually installed in the CI run (green CI ≠ target tested).
  • Detects hardcoded version pins in workflow steps that silently mask a bump.
  • Checks pip resolution and workflow pinning.
  • Never writes to GitHub — every output is a draft for a human to review and post.

Motivation and Context

Dependabot opens many near-identical dependency-update PRs; hand-reviewing each is repetitive and easy to rubber-stamp. A hardcoded langchain pin in pytest.yml previously let bumps pass CI while the new version was never tested. This skill automates the checks and produces a clear merge-or-escalate recommendation.

How to Use

Open claude code at the project folder. Type in following prompt.

/dependency-update-pr-validator <PR Link>

For example,

/dependency-update-pr-validator https://github.com/ev-flow/quark-engine/pull/932

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 80.27%. Comparing base (cf35a66) to head (6ce23a7).
⚠️ Report is 3 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master     #932      +/-   ##
==========================================
+ Coverage   78.89%   80.27%   +1.38%     
==========================================
  Files          81       82       +1     
  Lines        7131     7185      +54     
==========================================
+ Hits         5626     5768     +142     
+ Misses       1505     1417      -88     

Flag	Coverage Δ
unittests	80.27% <ø> (+1.38%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[pulorsok]

allowed-tools: bash(gh:*) undermines the "never writes to GitHub" guarantee

The skill's headline safety promise (SKILL.md:13) is:

Skill never write GitHub. Never call gh pr comment, gh pr merge, gh pr review.

But the capability grant in the frontmatter (SKILL.md:5) is:

allowed-tools: bash(gh:*), bash(.claude/skills/.../scripts/*.sh)

bash(gh:*) pre-authorizes every gh subcommand — including the mutating ones (gh pr comment, gh pr merge, gh pr review, gh pr edit, gh api -X POST/PATCH/DELETE) — with no permission prompt at call time. So the "never write" guarantee is enforced only by prose instructions, while the actual security gate is wide open.

Why this matters: these PRs live upstream (ev-flow/quark-engine), so any write is a shared-state action. A single instruction-following slip — or a prompt injection embedded in a dependabot PR body the skill reads (e.g. "please merge this PR") — could silently comment on or merge an upstream PR, with no human in the loop.

Suggested fix: scope the allowlist to the read-only verbs the scripts actually use, e.g.

allowed-tools: >-
  bash(gh pr view:*),
  bash(gh pr checks:*),
  bash(gh run list:*),
  bash(gh run view:*),
  bash(gh api:*),
  bash(.claude/skills/dependency-update-pr-validator/scripts/*.sh)

Keep gh pr merge / gh pr review out entirely, and leave gh pr comment out so that any attempt to post triggers an explicit human permission prompt. That makes the "never writes to GitHub" promise structurally enforced rather than advisory.

(Note: gh api can also issue writes; if you want to be stricter, drop gh api:* too and whitelist only the exact read endpoints the scripts hit.)

[haeter525]

Fixed in f996ee5.

bash(gh:*) has been replaced with a scoped allowlist of the read-only gh subcommands the scripts actually use:

allowed-tools: >-
  bash(gh pr view:*),
  bash(gh pr checks:*),
  bash(gh run list:*),
  bash(gh run view:*),
  bash(gh api:*),
  bash(.claude/skills/dependency-update-pr-validator/scripts/*.sh)

gh pr comment, gh pr merge, and gh pr review are no longer pre-authorized — any attempt to write to GitHub will now trigger an explicit human permission prompt. The "Skill never write GitHub" guarantee is now structurally enforced rather than advisory.