feat(guardian): give the finder diff-check's dependants as context (flagged, pre-registered)

Why

In a local trial on a private application corpus, the production guardian (Haiku medium, no skeptic) was run on 5 merged PRs that each later needed a fix PR for a bug the original PR introduced:

guardian caught 1 of 5 and came near on 1;
cgis diff-check --data-keys named the exact sites of 2 of the 5 that guardian missed, and pointed at 2 of the 4 sites of a third;
one bug was missed by both.

The blind spots differ. Guardian reasons about logic inside the diff. diff-check sees code outside the diff that shares a key or a callee with it. The question is whether handing the finder diff-check's items as leads turns those into findings, without drowning it.

Only aggregates from the private corpus are reported here. Per-PR details stay local.

Change (separate PR, off by default)

Flag: diff_check in GUARDIAN_FEATURES.
Context: the collector runs diff-check in-process on the review's graph and adds a section "Code outside this diff that depends on what it changed (leads, not findings)".
- Capped at 15 items, callers before key sites, tests excluded.
- Each item carries its file:line and its reason line.
Ingest: the workflow ingests with --data-keys when the flag is on. Without data keys, the section says so and lists callers only.
Prompt: one line telling the finder that a lead is worth a finding only once it has read the site and can name what breaks.

Pre-registered evaluation

Corpus.

Dev: the 5 trial PRs. Tuning is allowed on these, and they are not used for the verdict.
Holdout: a fresh draw of 15 merged PRs from the same application, using the same mining. Each is a merged feat/fix/refactor PR referenced by a later fix PR whose body says it repairs the earlier one. The dev 5 are excluded.
Sealed sites: before any review runs, the bug site of each holdout PR (file and symbol, taken from its fix PR) is written down and sealed locally.

Arms. Production config with and without diff_check. Both arms use the same graph, built with data keys, and run 3 reps per PR.

Hit. A finding counts as a hit if it points at the sealed site and names the mechanism the fix repaired.

It is judged by a blind auditor, a Claude Code subagent with read-only clones, with the arm hidden.
A PR counts as caught by an arm when ≥ 2 of 3 reps hit.

Noise.

Findings per review.
The share of WRONG among 20 sampled non-hit findings per arm, audited with the J protocol and 4 planted false findings.

Decision: adopt (flag on in production) if all of these hold:

caught(diff_check) − caught(base) ≥ 2 PRs out of 15;
at most 1 PR caught by base is lost;
findings per review rise by no more than 30%;
the audited WRONG share rises by no more than 10 pp.

Validity. Planted WRONG ≥ 3/4. At most 1 review per arm may be unparseable.

Regression guard on public data. On B's 19 Martian PRs, 1 rep, both arms, Haiku judge: recall must not drop by more than 0.05.

Budget. About $4 on the private corpus (15 × 2 × 3 reviews) and about $1.5 on Martian. Cap $8. Audits are done by subagents and cost nothing.

Out of scope

Wiring guardian into the private application's CI. That comes after this verdict, and its fetch needs a token.

🤖 Generated with Claude Code

Two design notes from the review of #736

Both are recorded before any run.

The skeptic does not see section 8. judge_all gets only the diff (core.py:143-148), so a skeptic would judge a lead-based finding without its lead. The registered arms run with the skeptic off, as production does, so the result is unaffected. A later skeptic-on comparison would have to pass the section to the skeptic first, or count this as a confound.
A graph built with --data-keys adds DATA_KEY nodes and JSON/YAML files. Impact traversal excludes key edges, so the impact section is unchanged. Drift is not used in either arm. To keep the arms different in section 8 alone, both arms use the same data-key graph, as already registered ("Both arms use the same graph, built with data keys").

feat(guardian): give the finder diff-check's dependants as leads (fla…

Reopened. GitHub closed this from #736's description, whose text "does not close #733" contains the closing keyword. The feature has landed off by default (f7546cc). The pre-registered evaluation above has not run yet, and this issue stays open until its verdict.

Amendment: stratify by where the bug lies

The maintainer agreed to this before any review run.

Why

diff_check adds context outside the diff, so it can only plausibly help where the sealed bug site lies outside the diff. In the first sealed batch only 1 of 6 qualifying PRs had such a site. With so few, a flat "+2 of 15" rule mostly measures the sample, not the feature.

Changed

Strata. Each sealed PR is marked in_diff or outside, recorded at sealing time and before any run.
Decision (adopt) uses the outside stratum:
- caught(diff_check) − caught(base) ≥ 2;
- at most 1 outside PR caught by base is lost.
  The in_diff stratum is a regression check: at most 1 PR caught by base is lost there.
  The noise rules are unchanged: findings per review up by ≤ 30%, audited WRONG share up by ≤ 10 pp.
Power. If the outside stratum has fewer than 6 PRs, the verdict is INCONCLUSIVE (underpowered), not FAIL. The next step is then to mine more history, not to close this.
No enrichment. PRs are drawn as they qualify, not chosen for being outside, and the report gives the outside share of all qualifying fixes as a fact about the corpus.

Unchanged

3 reps per arm, and a PR counts as caught when ≥ 2 of 3 reps hit.
Blind audit, planted findings, the Martian guard, and the $8 cap.
One PR whose fix pairs with a dev-set PR was excluded from the holdout as already seen.

Holdout sealed, before any run

Sealing: 157 original/fix pairs were sealed by blind subagents. 31 qualify; one more was excluded because it pairs with a dev-set fix.
Strata: 9 outside the diff and 22 in_diff. Outside is 29% of qualifying fixes.
Size: all 31 are used, with no sub-draw. This is more than the registered 15, and it gives the outside stratum 9 ≥ 6. Every other rule stands as amended.
Integrity: the sealed file's sha256 prefix is 6ef5857f8308fa58. The file stays local because the corpus is private.
Cost: at Haiku 5.5's $0.10 / $0.50 per MTok, 31 × 2 arms × 3 reps comes to about $1, inside the $8 cap.

Result: `diff_check` is not adopted. Production is unchanged and the flag stays off.

Pre-registered here, amended to stratify by bug site, with the holdout sealed before any run.

The corpus is private, so only aggregates are reported.

Spend: about $1.9 in total, against a $8 cap: $1.48 on the private corpus and about $0.4 on Martian. The audits were done by subagents at no API cost.

Holdout

157 original/fix pairs were sealed blind, and 31 qualify: 9 with the bug site outside the diff and 22 in_diff.
Each qualifying PR was reviewed under both arms × 3 reps, with Haiku 5.5 at medium effort, skeptic off, and one data-key graph per PR.
That makes 186 reviews, with 0 failures and 0 parse failures. All 864 findings were audited blind, arm hidden, against the sealed site and mechanism.

stratum	n	caught by base (≥2/3 reps)	caught with diff_check	gained	lost
outside	9	0	1	1	0
in_diff	22	6	6	2	2

With HIT+PARTIAL counted, the picture is the same: outside 0 → 1, and in_diff 8 → 7 (+2 / −3).

Rules

rule	result
power: outside n ≥ 6	✅ 9
outside gain ≥ 2	❌ 1
outside lost ≤ 1	✅ 0
in_diff lost ≤ 1	❌ 2, both 2/3 → 1/3 rep splits, consistent with sampling noise
findings/review ≤ +30%	✅ 4.60 → 4.69 (+2%)
audited WRONG share ≤ +10 pp	not run: the verdict was already decided by the rules above
Martian recall drop ≤ 0.05	✅ 0.557 → 0.574 (19 PRs, 1 rep, Haiku judge, both arms on data-key graphs)

Why, from a free diagnostic (not pre-registered)

For the 9 outside PRs, the leads section was rebuilt and checked for the sealed site:

2 of 9 contained it. In one, the site was the only lead, and that is the one PR diff_check caught (2/3 reps). In the other, the site was one of 15 leads, and the finder missed it.
7 of 9 did not contain it. In 6 of those, diff-check had 14 to 172 more items than the cap of 15 shows. The site was either cut by the ranking or not reachable through calls or keys.

So the finder uses a lead when the lead is precise. The bottleneck is diff-check's ranking and recall on a real application, not the prompt wiring.

A tail issue: rank or filter diff-check items for review use before revisiting this, for example by proximity to the changed lines or by key specificity. The test is whether the sealed site enters the top 15. That can be measured for free on these same 31 sealed PRs before any review spend.
The flag remains available for experiments.

Closed with the verdict above: not adopted, flag off. Follow-up is the tail on ranking.

Navigation Menu

Repository navigation

feat(guardian): give the finder diff-check's dependants as context (flagged, pre-registered) #733

Why

Change (separate PR, off by default)

Pre-registered evaluation

Out of scope

Two design notes from the review of #736

Amendment: stratify by where the bug lies

Why

Changed

Unchanged

Holdout sealed, before any run

Result: `diff_check` is not adopted. Production is unchanged and the flag stays off.

Holdout

Rules

Why, from a free diagnostic (not pre-registered)

Next

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

feat(guardian): give the finder diff-check's dependants as context (flagged, pre-registered) #733

Description

Why

Change (separate PR, off by default)

Pre-registered evaluation

Out of scope

Activity

zaebee commented on Oct 22, 2026

Two design notes from the review of #736

zaebee commented on Oct 22, 2026

zaebee commented on Oct 22, 2026

Amendment: stratify by where the bug lies

Why

Changed

Unchanged

zaebee commented on Oct 22, 2026

Holdout sealed, before any run

zaebee commented on Oct 22, 2026

Result: diff_check is not adopted. Production is unchanged and the flag stays off.

Holdout

Rules

Why, from a free diagnostic (not pre-registered)

Next

zaebee commented on Oct 22, 2026

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions

Result: `diff_check` is not adopted. Production is unchanged and the flag stays off.