fix(evals): stage contracts and repair graders for advisory agent-behavior failures

[Jamie Kim (jkim323)]

Pull Request

Description

Related Issue(s)

Type of Change

Select all that apply:

Code & Documentation:

• Bug fix (non-breaking change fixing an issue)
• New feature (non-breaking change adding functionality)
• Breaking change (fix or feature causing existing functionality to change)
• Documentation update

Infrastructure & Configuration:

• GitHub Actions workflow
• Linting configuration (markdown, PowerShell, etc.)
• Security configuration
• DevContainer configuration
• Dependency update

AI Artifacts:

• Reviewed contribution with hve-builder and addressed all actionable findings
• Copilot instructions (.github/instructions/*.instructions.md)
• Copilot prompt (.github/prompts/*.prompt.md)
• Copilot agent (.github/agents/*.agent.md)
• Copilot skill (.github/skills/*/SKILL.md)
• Copilot hook (.github/hooks/*/*.json)
• Eval spec added/updated for changed AI artifacts (evals/)

Note for AI Artifact Contributors:

• Agents: Research, indexing/referencing other project (using standard VS Code GitHub Copilot/MCP tools), planning, and general implementation agents likely already exist. Review .github/agents/ before creating new ones.
• Skills: Must include both bash and PowerShell scripts. See Skills.
• Model Versions: Contributions MUST target models listed in the model catalog (scripts/linting/model-catalog.json) whose provider appears in providerAllowlist and whose status is ga or preview. Run npm run lint:models to validate references.
• See Agents Not Accepted and Model Version Requirements.

Other:

• Script/automation (.ps1, .sh, .py)
• Other (please describe):

Sample Prompts (for AI Artifact Contributions)

User Request:

Execution Flow:

Output Artifacts:

Success Indicators:

For detailed contribution requirements, see:

• Common Standards: docs/contributing/ai-artifacts-common.md - Shared standards for XML blocks, markdown quality, RFC 2119, validation, and testing
• Agents: docs/contributing/custom-agents.md - Agent configurations with tools and behavior patterns
• Prompts: docs/contributing/prompts.md - Workflow-specific guidance with template variables
• Instructions: docs/contributing/instructions.md - Technology-specific standards with glob patterns
• Skills: docs/contributing/skills.md - Task execution utilities with cross-platform scripts

Testing

Checklist

Required Checks

• Documentation is updated (if applicable)
• Files follow existing naming conventions
• Changes are backwards compatible (if applicable)
• Tests added for new functionality (if applicable)

AI Artifact Contributions

• Used hve-builder review mode to review contribution
• Addressed all actionable findings from the hve-builder review
• Verified contribution follows common standards and type-specific requirements

Required Local Checks

The following local-safe validation commands must pass before merging:

• Local validation aggregate: npm run validate:local
• Documentation validation (if docs changed): npm run validate:docs
• Spell checking: npm run spell-check
• Link validation: npm run lint:md-links

Security Considerations

• This PR does not contain any sensitive or NDA information
• Any new dependencies have been reviewed for security issues
• Security-related scripts follow the principle of least privilege

Additional Notes

[Codecov Comments Bot (codecov-commenter)]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 81.86%. Comparing base (ee3df32) to head (6715491).

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3242      +/-   ##
==========================================
- Coverage   81.87%   81.86%   -0.01%     
==========================================
  Files         110      110              
  Lines       17467    17467              
==========================================
- Hits        14301    14300       -1     
- Misses       3166     3167       +1     

Flag	Coverage Δ
pester	86.45% <ø> (-0.01%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.
see 1 file with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[github-actions]

Eval Execution

❌ Status: Failed — 1 spec(s) block merge

• Artifacts evaluated: 7
• Specs run: 7
• Assertions passed: 320
• Assertions failed (blocking): 4
• Assertions failed (advisory): 82
• Trial dips tolerated by per-stimulus aggregate: 14
• Failed specs (merge-blocking): 1

Artifact	Kind	Status	Specs	Passed	Failed (blocking)	Failed (advisory)	Trial dips (tolerated)	Input tokens / trial	Cache-read tokens / trial
accessibility-framework-assessor	agent	⚠️ advisory-fail	1	22	0	8	0	90,392	60,316
code-review	agent	⚠️ advisory-fail	1	109	0	9	12	61,819	36,067
data-science-engineering-coach	agent	❌ fail	1	76	4	59	1	245,528	218,352
rai-planner	agent	✅ pass	1	45	0	0	0	195,798	160,319
security-planner	agent	⚠️ advisory-fail	1	36	0	3	1	139,186	111,711
sssc-planner	agent	⚠️ advisory-fail	1	14	0	1	0	75,546	55,499
sssc-reviewer	agent	⚠️ advisory-fail	1	18	0	2	0	189,690	155,988

Legend — ✅ clean · ⚠️ advisory failures only (non-blocking) · ⏭️ skipped · ❌ merge-blocking failure

Only Failed specs (merge-blocking) gates this PR. Advisory assertion failures are signal-quality checks captured during iteration; review them, but they do not block merge and may be acceptable.

Token columns are advisory means per trial reported by the model provider. They vary between trials and never gate this PR.