Run Engraph research evidence gates in CI#3
Conversation
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 463f5e611b
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| with: | ||
| python-version: "3.12" | ||
| - name: Verify shared manifest contract | ||
| run: python3 research-evidence/contracts/v2/verify_contract.py |
There was a problem hiding this comment.
The contract verifier this step invokes calls shutil.which("jsonschema") and raises reference JSON Schema validator unavailable when that executable is missing (research-evidence/contracts/v2/verify_contract.py:88), but the workflow only checks out the repo and runs actions/setup-python before this command. On a clean ubuntu-latest run, setup-python supplies the interpreter but does not install this CLI, so the first gate exits 1 before any integrity checks run; add an explicit validator install or use an in-repo/Python validator before invoking it.
Useful? React with 👍 / 👎.
| - name: Verify claim durability | ||
| run: python3 scripts/verify_claim_durability.py | ||
| - name: Verify authentic P3 correctness receipt | ||
| run: python3 scripts/research_receipt.py verify research-evidence/runs/p3-actor-parity-v2-20260717-a2.receipt.json |
There was a problem hiding this comment.
research_receipt.py verify still validates the receipt's HOME-relative model artifact (.engraph/models/...) and reports model artifact unavailable when that private file is absent (scripts/research_receipt.py:1255-1271); this workflow has no cache restore or model download and the commit claims none should occur. A standard GitHub runner therefore fails this gate, and the following audit, even though all committed receipt files are present, so the CI needs either a committed/offline-only verification mode or provisioning of the bound artifact.
Useful? React with 👍 / 👎.
| - name: Audit the authentic P3 attempt | ||
| run: python3 scripts/research_receipt.py audit-plans research-evidence/runs --attempt-id 076b69d2-3a1f-40d6-9555-9c7cc6c69302 | ||
| - name: Run integrity tests | ||
| run: python3 -m unittest discover -s scripts -p "test_*.py" -q |
There was a problem hiding this comment.
These tests include test_command_evidence_rejects_secrets_without_false_tokenizer_match, which asserts that /Users/d/private/file redacts as $HOME/private/file; on ubuntu-latest, HOME is not /Users/d, so _safe_command_argv returns $ABS_PATH_SHA256/... instead. Once earlier gates are fixed, this step fails on the hosted Ubuntu runner unless the test is made platform-independent or the workflow uses a runner/environment matching that hard-coded home path.
Useful? React with 👍 / 👎.
| paths: | ||
| - models/** | ||
| - research-evidence/** |
There was a problem hiding this comment.
The claim-durability verifier binds live claim surfaces from README.md and scans the whole README for unregistered high-risk claims (scripts/verify_claim_durability.py:481-513), but README.md is not in either paths list. A PR or push on feat/engraph-p3-perf that changes only README claims will skip this workflow, leaving the live excerpt hashes and high-risk-claim guard unenforced in the path-scoped gate; include README.md in both filters.
Useful? React with 👍 / 👎.
1 participant
Summary
Local verification
Claim boundary
CI supports only the committed a2 correctness receipt and control behavior. It does not establish portable reproduction, the historical 1.67x performance claim, model acquisition history, or redistribution rights.
Reviewer focus