guardian: cross-model skeptic to cut recall-lean noise without killing recall

Follow-up to the recall-lean finder work (PR feat/guardian-universal-prompt).

The recall-lean finder (Tier-1/2) now surfaces real defects the old prompt missed (pr-144 recall 0/5 → median 3/5), but at the cost of high finder-stage noise (16-26 findings/run on pr-140/143). The skeptic is supposed to trim that.

Problem: a same-model mistral-medium skeptic is binary — it either refutes everything (recall→0) or nothing (noise stays ~finder-only). Empirically:

old skeptic (refute-heavy): recall destroyed (pr-140 0.0/0.1).
reconciled skeptic (commit afae100, refute-light): recall held (pr-140 0.2/0.3, pr-143 0.7/0.5) but noise NOT cut (12-17).
- few-shot false-positive examples: swung mistral back to refute-all (pr-143 0.7→0.0) → reverted.

Hypothesis / task: the designed default is a cross-model skeptic (gemini for a mistral finder). A different model should hold the nuance "refute vague/ghost/unreachable, keep concrete-low-confidence" that same-model mistral cannot. Run the bench with GUARDIAN_SKEPTIC=gemini over a mistral finder on pr-140/143/144 and measure recall+precision+noise vs the reconciled mistral skeptic. If it cuts noise while holding recall, the two-stage delivers; few-shot-FP may then work cross-model (it does NOT on same-model mistral).

feat(guardian): recall-lean universal finder + skeptic reconciliation (…

feat(guardian): Ollama provider for local/colab inference — finder + …

Update: the hybrid (cloud finder + local cross-family skeptic) works — this is the answer

Tested the core hypothesis of this issue empirically (mistral-medium finder + local Ollama llama3.1:8b skeptic, cross-family, on bench pr-143):

config	recall	noise	notes
mistral finder-only	~0.8	15–26	recall-lean, noisy by design
mistral + same-model mistral skeptic	holds	12–17 (barely cut)	binary — can't nuance
+ few-shot false-positives (same-model)	→0	—	swung to refute-everything
mistral + local `llama3.1:8b` (cross-family)	0.67	7	✅ cuts noise, holds recall

The cross-family local skeptic refuted 15 of 26 findings with differentiated verdicts (9 confirmed / 15 refuted / 1 unruled) — exactly the nuance a same-model mistral skeptic could not produce (it was binary: either refute-all → recall 0, or refute-light → noise uncut).

Key takeaways:

A different model family for the skeptic is what unlocks differentiated filtering — same-model is the trap.
The local skeptic is free (Ollama, no API cost) — the precision half costs nothing.
Pure-local 8B finders stay weak/inconsistent (qwen3/granite ~0/6), so the finder should stay a strong (cloud) model.

Killer config: cloud strong recall-lean finder + free local cross-family Ollama skeptic. Unblocked by #256 (Ollama provider, now merged).

Caveat: n=1, single PR — worth a few more runs across pr-140/143 to confirm, but the mechanism is clear.

docs(plans): implementation plan for guardian skeptic scoring (#246)

docs(specs): guardian skeptic — per-finding judgement + impact scoring (

fix(guardian): address SonarCloud findings on the skeptic scoring PR

bench(guardian): phase 1-2 results for the per-finding skeptic (#246)

Answered: cross-family is the whole effect, and isolation is what makes it safe

Implementation in #269; measurements below (raw rows in benchmarks/guardian/results.jsonl, spec docs/specs/2026-07-29-guardian-skeptic-scoring-design.md).

Method: one frozen finder pass per PR (mistral-medium), every skeptic variant judging that same set, so the skeptic is the only variable.

The original question — does a cross-model skeptic cut noise without killing recall?

Yes, but only per-finding. Same model, same frozen set, three PRs:

	mean recall	mean noise
finder alone	0.667	15.3
gemini, per-finding	0.600	10.3
gemini, batch	0.422	7.0
codestral (same family), either arm	0.667	15.3 (unchanged)

Same-family contributes nothing. codestral judged 28/28 on pr-143 and refuted zero. The measured history ("same-family skeptic is binary") now holds with the mechanism isolated as a variable.
Cross-family in a batch over-refutes. gemini batched refuted 14 of 28 on pr-143 and took 2 ground-truth findings with them — the pathology that was reverted once before, reproduced under controlled conditions.
Cross-family per-finding is the usable operating point: −33% noise for −10% recall, versus batch's −54% for −37%.

Recommended prod change: GUARDIAN_SKEPTIC=gemini (repo variable). GUARDIAN_SKEPTIC_MODEL=gemini-2.5-flash verified working. GUARDIAN_IMPACT_THRESHOLD should stay unset (0) — see below.

Honest limits

n=1 run per PR, 3 PRs. pr-140 lost a ground-truth finding under per-finding gemini too (recall 0.40 → 0.20); the recall protection is comparative, not absolute.
The impact-score axis under-delivers. Pooled separation is 3.0 against a 3.0 gate — no margin — and per-PR it swings 1.5 / 7.0 / 2.0. On pr-143 the separation among survivors is 0.0: gemini's low scores sit on findings it already refuted, so the axis is largely collinear with the verdict rather than orthogonal to it.
Per-finding costs +69% skeptic tokens (221k vs 131k across the three PRs). The spec assumed narrow per-call context would make N calls cheaper; that is refuted — findings cluster by file, so a file's hunks are re-sent once per finding in it.

Follow-ups filed

bench(guardian): metric for findings the skeptic KILLED (matched_gt cannot express it) #270 — a metric for findings the skeptic killed. The current matched_gt cannot express it: it is derived from the post-refutation visible list, so a refuted finding can never carry the flag. That made my first read of these results tautological.
guardian: does impact_score earn its cost? (collinear with verdict, +69% tokens) #271 — decide whether impact_score earns its cost, or ships as observe-only.

🤖 Generated with Claude Code

feat(guardian): per-finding skeptic judgement + impact scoring (#246) (…

Field data: the same-family pathology is what's running in production right now

First production runs since Guardian came back up (it had been failing at Install dependencies since 7bd74f9/#230 moved the guardian deps to [project.optional-dependencies] while the workflow kept --group — fixed in #300). So there was no recent prod data; there is now.

The recommended change from the comment above was never applied

Current repo Actions variables:

GUARDIAN_PROVIDER = mistral      GUARDIAN_MODEL         = mistral-medium-latest
GUARDIAN_SKEPTIC  = mistral      GUARDIAN_SKEPTIC_MODEL = codestral-latest

That is precisely the same-family arm this issue measured as contributing nothing — "codestral judged 28/28 on pr-143 and refuted zero". The recommendation (GUARDIAN_SKEPTIC=gemini, per-finding) is still unset in prod.

It reproduces outside the bench

Two real PRs, reviewed today:

PR	findings	skeptic judged	refuted	posted
#297	25	25/25	0	25 inline
#298	6	6/6	0	5 inline + 1 out-of-diff

Zero refutations is read from the absence of the _N findings refuted by the skeptic pass._ note (render.py:92), not from the skeptic N/M signature — that pair is skeptic_judged/skeptic_total, a completeness indicator, so 25/25 means "every finding was successfully judged", not "every finding was confirmed". Worth stating because the signature reads like a confirm ratio and it isn't.

So the skeptic stage ran to completion, cost tokens, and changed nothing. The controlled bench result now has a production counterpart.

A precision datapoint the bench can't produce

The bench measures recall against curated ground truth. This is the other axis: I hand-checked all 25 findings on #297 against the source. The bulk assert "unvalidated dictionary access" on code that validates explicitly one line earlier:

"Unvalidated dictionary access for min key" — guarded by if not isinstance(constraint, dict) or "min" not in constraint: return None immediately above.
"Unvalidated type for reference before string operations" and "unvalidated string slicing" — both guarded by isinstance(reference, str) and reference.startswith("$").
"Potential KeyError if section is missing" — both dicts are built by comprehension over _TOLERANCE_SECTIONS, so every key exists by construction.
"Floating-point comparison may be flaky" — the line carries an explicit + 1e-9 epsilon.
"Prohibited use of Any" — dict[str, Any] is used throughout drift.py; mypy strict is green.

Two were fair, both minor (an asymmetric type guard, and a float() without a guard — though the input is a module constant). On #298 the two "double/over-escaping" findings were wrong: both cited sites escape exactly once.

The one finding across all three reviews that changed my code came from the LGTM review on #299 — it noted the diff carried "a version bump in uv.lock" alongside the docs. It was right; a stray lockfile change had been swept into two PRs by git add -A. Removed from both.

Caveats

n=2 PRs, no ground truth. This says nothing about recall — only that the skeptic refuted nothing and that finder precision on these two diffs was poor.
I am judging findings on my own PRs, which is a real bias in the false-positive assessment above. The specific claims are checkable against the diffs; the aggregate "mostly false" is my judgement.
The docs: add the chunked-review lab note (a negative result worth keeping) #299 catch is a reminder that the noisy finder isn't uniformly wrong — the useful observation came from the run with the fewest findings.

Suggested next step

Flip GUARDIAN_SKEPTIC=gemini (+ GUARDIAN_SKEPTIC_MODEL=gemini-2.5-flash, both already verified working in #269) and re-run these same two PRs. They now serve as a ready-made before/after pair with the finder output already recorded.

🤖 Generated with Claude Code

Follow-up: the prod switch is done, and the mechanism holds — but the finder swamps the measurement

GUARDIAN_SKEPTIC=gemini + GUARDIAN_SKEPTIC_MODEL=gemini-2.5-flash are now set as repo variables (was mistral/codestral-latest). Re-ran the same two PRs from the comment above.

PR	skeptic	findings	posted	refuted
#297	`codestral-latest` (same-family)	25	25	0
#297	`gemini-2.5-flash` (cross-family)	23	14	9 — 39%
#298	`codestral-latest` (same-family)	6	6	0
#298	`gemini-2.5-flash` (cross-family)	46	9	37 — 80%

The behavioural split is unambiguous and reproduces this issue's central claim in production: same-family refuted nothing in two runs out of two, cross-family refutes. Runs 30662722029 / 30662721279; metrics artifacts confirm skeptic_status=ok, judged == total in all four.

Reading refutations from an inline review — a trap

The _N findings were refuted by the skeptic pass._ note lives in render_report (render.py:92), which produces guardian_report.md — and that file is only posted when inline publishing fails (if: steps.guardian.outputs.posted_inline != 'true'). An inline review never carries the note, so its absence means nothing. I misread exactly this and briefly concluded gemini had refuted zero.

The usable signals are findings_total from the metrics artifact versus the count of inline comments actually posted. Worth surfacing since the whole point of #270 is that refutation is hard to observe.

The measurement is not controlled — and the finder, not the skeptic, dominates

CI has no frozen-finder replay: --record-finder writes the pass for offline re-scoring, but nothing replays it, so each trigger runs a fresh finder. On the identical #298 diff the finder produced 6 findings in one run and 46 in the next — an 8× swing.

So the refutation percentages above cannot be cleanly attributed to the skeptic; the finder's own variance is larger than the effect being measured. The bench methodology in the comment above (one frozen finder pass, every skeptic variant judging the same set) is exactly what CI lacks. Filed as #302.

Precision of the survivors: mixed, and worse where it matters

Noise volume fell; survivor quality did not follow.

drift trust batch: shallow-depth false violation, tricode cross-validation, tolerance ratchet #297 — no improvement. The 14 survivors still contain the same false-positive classes as before: "Prohibited use of Any" (used throughout drift.py, strict mypy green), "Unvalidated string slicing on reference" (guarded by isinstance(reference, str) and reference.startswith("$") one line above), "floating-point precision issue" (the line carries an explicit + 1e-9). The skeptic cut volume without cutting the wrong class.
fix(cli): escape data interpolated into Rich console output #298 — genuinely better. All 9 survivors are one coherent claim: escape(str(x)) where x is already a str. That is a fair point about the sweep in that PR, and gemini refuted both of run 1's invented "double-escaping" findings. Still 2 of the 9 are plainly wrong — str(count) and str(out_degree) are called on ints, where str() is required, not redundant.

Cost: the #298 gemini run took 198s and 19.7K tokens.

Caveats

n=2 PRs, no ground truth, fresh finder per run — this measures refutation behaviour, not recall. Nothing here says whether the 9 and 37 refutations included true findings.
Same self-review bias as the previous comment: I am judging findings on my own diffs. The individual claims are checkable; the aggregate judgement is mine.

Recommendation: keep GUARDIAN_SKEPTIC=gemini (it is strictly better than a skeptic that refutes nothing), but treat any further skeptic A/B as unmeasurable until #302 lands.

🤖 Generated with Claude Code

Recorded in #305 as one of the two triggers for revisiting whether cgis ships as an installable review agent: a settled finder+skeptic operating point.

Correction to my comment above

I closed my follow-up with "treat any further skeptic A/B as unmeasurable until #302 lands." That is wrong, and #302 is now closed as redundant.

Frozen-finder replay already exists, in guardian_bench.py:

--replay-finder   Judge a recorded finder pass instead of running the finder.

load_finder_recording strips verdict/skeptic_note/impact_score on read, so a recording taken from a skeptic-bearing run cannot leak those verdicts into the next arm — which is the property that makes the comparison valid. It is the same machinery the controlled comparison in this thread already used.

I had checked scripts/guardian_review.py, found no replay flag, and generalised from that to "CI has no replay" without looking at the bench. The 6 → 46 → 36 finder variance I reported is real, but it is variance in production reviews of a live PR — which is not where a skeptic A/B should be run, so it does not make A/B unmeasurable.

Standing recommendation is unchanged: keep GUARDIAN_SKEPTIC=gemini. What changes is the caveat — a controlled A/B can be run today with guardian_bench.py --replay-finder against any PR that has a pr-*.yaml fixture. For PR #313 that fixture is #315.

Evidence for an ensemble rather than a better single reviewer (2026-08-13)

Three reviewers were run over the same diff (PR #377, the salvage_findings change). Recording it here because it bears directly on the skeptic design.

reviewer	real defects found	quality of its reasoning
SonarCloud	0	—
Guardian (gemini-2.5-flash finder + gemini skeptic)	1	conclusion right, cause wrong twice
Fable	1	mechanism right, threshold wrong by 10×

The findings did not overlap. Guardian found that salvaged findings bypassed _sanitize_finder_result, so a model-hallucinated verdict: "refuted" would delete a finding from the report silently. Fable found that json.raw_decode raises RecursionError — a RuntimeError, not a ValueError — so deeply nested openers escaped the except ValueError in a function documented as never raising, on a call path (chunked._single_pass → scripts/guardian_review.py) with no guard above it.

Both were genuine and both were fixed (225582f, be3efed).

Two things worth carrying into the skeptic work

1. Confirming a finding and confirming its rationale are different checks, and we do neither on the second axis.

Guardian's skeptic returned confirmed and restated the finder's reasoning verbatim — including the false claim that _sanitize_finder_result "includes dedup_findings logic". It does not; it resets skeptic-owned fields. And dedup_findings is called only from axes.py and chunked.py, so the strict finder path does not dedup either — the stated difference between the two paths did not exist.

Applying that finding's suggested diff unexamined would have produced the right code for the wrong reason, and the test written alongside it would have pinned an invariant that isn't there. The skeptic's verdict axis asks "is this finding true"; nothing asks "is this explanation true", and a finding can be worth acting on while its rationale is fabricated.

2. Neither model's error was in its conclusion — both were in the supporting detail.

Guardian named the wrong function; Fable named 2,000 nested openers where the real threshold is ~20,000 (verified by reproducing across depths). A reviewer that is right about what and wrong about why is the common case here, not the exception, which argues against treating a single reviewer's rationale as evidence.

Related: the same session's finding that a same-family Mistral skeptic refutes 6.6% (not ~0% as the older codestral note implied) — see docs/specs/2026-08-11-guardian-code-review-bench.md, Phase 3.

fix(guardian): pin the scheme and keep setup scripts out of the dep t…

Experiment G result: "misdirected" is not confirmed on the holdout. Production stays as it is

Run as pre-registered (issuecomment-6319613841) at main 95d26d3, on B's frozen Martian reviews: the gemini-2.5-flash finder, 57 reviews and 242 candidates, scored on B's Haiku-judge grids.

Baseline: 51 matched goldens, noise 191, kill tolerance 5.1.

Validity

All 8 passes are valid: 57 of 57 reviews were scored in each, with 0 unruled. So the worst case for unruled findings is identical to the normal count.

Results

arm	rep	noise cut (≥ 30%)	killed (≤ 5.1)	`misdirected` given (on matched candidates)	gate
haiku-5-5 medium, base	1 / 2 / 3	19.9% / 20.4% / 20.4%	0 / 0 / 0	—	FAIL ×3
haiku-5-5 medium, misdirected	1 / 2 / 3	13.1% / 14.7% / 15.7%	1 / 0 / 0	23 (3) / 25 (1) / 23 (3)	FAIL ×3
sonnet-5-5 low, base	1	19.9%	1	—	FAIL
sonnet-5-5 low, misdirected	1	16.8%	1	23 (3)	FAIL

Decision, as registered: not confirmed

Rule 1 fails. misdirected passes the gate in 0 of 3 reps.
Rule 2 fails. Mean killed is 0.33 higher with misdirected; the rule needed it to be ≥ 1 lower.
Rule 3 fails. The mean noise cut is 5.8 pp lower, just past the −5 pp limit.

The worst-case count gives the same result. The production skeptic is unchanged, and context_requests and misdirected both stay off.

What the holdout shows

Here the recall cost the verdict was built to fix does not exist. The base skeptic hides 0–1 matched goldens out of 51 in every pass. The kills on A's corpus that E and F were about belong to that corpus, which is small, single-repository, and full of findings with a right pointer but a wrong argument. They are not a general property of the skeptic.
On new data the verdict mostly works as a hedge on noise. About 90% of the 23–25 misdirected verdicts per pass fell on candidates that match no golden. Those findings stay visible, so the noise cut drops in every pass: by 4.7–6.8 pp for Haiku and 3.1 pp for Sonnet.
Run-to-run spread is small here. Across the base reps, the noise cut stays within 19.9–20.4% and killed is 0 each time. That contrasts with the E-vs-F swing on A's corpus.
The binding half of the gate on Martian is noise. Every skeptic tried cuts only about 20%.
- The noise measure has a caveat: a "noise" candidate is one that matches no Martian golden comment, and golden sets are known to miss real defects. So part of that 191 may be real issues, and the absolute 30% bar may not fit this corpus.
- The comparison between the base and misdirected arms is unaffected by this, since both are scored the same way.

#246 so far, in one line

Claude as finder helps (B, provisional).
Neither a context mechanism (C) nor a bigger skeptic model (D, E) moves the gate.
A fourth verdict that passed on its own development set (F) fails on a holdout (G).

Cost

G cost $4.98 of the $6.50 cap.
guardian: cross-model skeptic to cut recall-lean noise without killing recall #246 total is ≈ $24.0. The cap has been extended by the maintainer during the run.

Experiment H pre-registration: how much of the Martian "noise" is real

Written before any audit label exists. No API spend: the auditors are Claude Code subagents, which read the code themselves.

Question

G found that on Martian the skeptic cuts about 20% of noise and kills almost no matched goldens. "Noise" means a candidate that the maximum matching leaves unused against the golden comments (profile core). If the golden sets under-count real defects, part of that noise is real findings. Then both the 30% noise bar and "the skeptic cut noise" mean less than they appear to.

Free breakdown (stored grids, already computed)

The noise is 191 candidates from B's 57 gemini reviews.

19 (10%) are duplicates of a golden that another candidate already matched. The judge said "match", but the 1:1 matching had already used that golden.
172 match no golden at all.
By project: cal.com 141, sentry 34, grafana 16.
Goldens outside core are rare (7 distinct: style and speculative), so they cannot explain the 172.

Audit sample

Draw: seed 246008. 54 of the 172 no-match candidates, uniformly at random.
Blind controls: 12 of the 51 matched candidates, shuffled in.
Batches: 66 items in 3 batches of 22, one per auditor subagent.

Auditors see:

the finding (file, line, severity, category, title, evidence, problem, fix);
the PR URL;
base and head SHA;
a read-only clone.

They do not see goldens, match status, skeptic verdicts or this hypothesis. Each item gets one label:

REAL_BUG: the claim holds in the head code, and it is a defect a maintainer should fix (correctness, security, data, performance, broken contract), in code the PR changed or newly relies on.
REAL_MINOR: the claim holds, but its value is low: a test gap, a restated TODO, style, defensive hardening, a hypothetical with no reachable trigger.
WRONG: the code contradicts the claim.
UNDECIDABLE: the clone cannot settle it.

Reported

REAL_BUG share among the no-match noise, with a Wilson 95% CI, and the implied count out of 172. Also REAL_BUG+REAL_MINOR.
Validity: the controls must come out REAL_BUG or REAL_MINOR in ≥10 of 12; otherwise the audit is reported INVALID as a measure. UNDECIDABLE above 25% of the sample is reported as a limit.
Cross-tab with G's Haiku-medium base skeptic (3 reps) on the sampled items: the refute rate on REAL_BUG items versus WRONG items. If the skeptic refutes REAL_BUG at a rate close to WRONG, its "noise cut" is not separating anything.

Interpretation, fixed now

REAL_BUG ≥ 25%: the golden sets materially under-count. The Martian noise bar should be re-specified against audited labels. I will propose that; the bands move only on the user's decision.
REAL_BUG ≤ 10%: the noise is genuine, and the skeptic's ~20% cut is the real gap.
Between the two: report it as partial under-count, with no rule change proposed.

Data stays in the scratchpad. Public text gets aggregates only. The repos are public OSS, so individual examples may be quoted.

Experiment H result: the Martian noise is genuine, not under-counted goldens

Pre-registration: issuecomment-6324705240. Three blind Opus auditor subagents read the clones read-only. They labelled 66 items with 0 UNDECIDABLE. There was no API spend.

Validity

The blind controls came out REAL in 11 of 12, so the audit is VALID. 8 controls were labelled REAL_BUG and 3 REAL_MINOR.

The one control labelled WRONG is a grafana device-limit finding that the golden comment also flags. The auditor reads that behaviour as intended: an expired device is excluded from the count, so reviving it is a new device.

The noise (191)

19 (10%) are duplicates of an already-matched golden. These are a matching artefact, not noise.
The other 172 match no golden. In the 54 sampled from them:

label	n/54	share [Wilson 95%]	implied of 172
REAL_BUG	3	5.6% [1.9, 15.1]	~10 [3–26]
REAL_MINOR	24	44.4%	~76
WRONG	27	50.0% [37.1, 62.9]	~86 [64–108]

By project:
- cal.com: 2 bug / 15 minor / 24 wrong;
- sentry: 0 / 5 / 2;
- grafana: 1 / 4 / 1.
The 3 real bugs the goldens miss (all logic):
- cal.com#10967: team destination calendars are dropped when the base destinationCalendar is null (?.push on null).
- cal.com#11059: Salesforce bypasses refreshOAuthTokens, which the other migrated apps use.
- grafana#106778: "Duplicate" is reported as supported for a non-existent Grafana rule.
What REAL_MINOR is: 23 of 24 are in PR scope. They are true, but nobody would block on them:
- test gaps and restated TODOs, mostly tests;
- unguarded JSON.parse behind a secret-gated webhook;
- a broad except Exception;
- a missing command -v in a dev script.

Verdict, as pre-registered: REAL_BUG ≤ 10%. The golden sets do not materially under-count, the 30% noise bar stands, and the gap is the skeptic's.

Cross-tab with G's Haiku-medium base skeptic (3 reps per item)

audit label	refuted	uncertain	confirmed	refute rate
REAL_BUG (noise)	0	3	6	0%
controls (any)	0	12	24	0%
REAL_MINOR	3	24	45	4%
WRONG	30	32	19	37%

The skeptic does discriminate: it never refutes a real bug and refutes wrong claims about 9× more often than minor ones. Two things cap its cut at ~20%:

About half the noise is true but low-value. The skeptic is asked about truth, so it confirms these, and it is right to.
It refutes only 37% of WRONG claims. It confirms 23% of them, and the rest go to uncertain.

Implication (descriptive, not a gate)

The next lever is not a better truth-skeptic. It is a value filter: a finder or skeptic instruction that drops true-but-low-value findings (test-gap and TODO restatements, defensive hardening with no reachable trigger). That would be a separate pre-registered experiment. Its risk is the 8 controls labelled REAL_BUG, and test_gap is a core golden category, so it needs a kill check.

A secondary target is the 63% of WRONG claims that the skeptic leaves standing.

Production is unchanged.

H: a limit of the instrument (added after the result)

The blind controls test one direction only: whether an auditor misses real findings (11 of 12 were caught). No control tests whether an auditor accepts a plausible but false claim.

The pre-registered verdict is not affected. A lenient auditor inflates REAL_BUG. The observed rate is 5.6%, so leniency can only mean the true rate is lower, and the "REAL_BUG ≤ 10%" verdict stands. The other failure, missing real bugs, is the one the controls cover.
What it does affect is the REAL_MINOR / WRONG split (44% / 50%). The "half the noise is true but low-value" reading rests on that split, and so does the value-filter lever I proposed from it.
What I will do first: before any value-filter pre-registration, re-audit with negative controls. The plan is planted findings that the code contradicts, plus a fresh auditor on a REAL_MINOR subsample. Until then the split is provisional.

H2 pre-registration: a negative control for the H auditors

This follows issuecomment-6324912970 and is written before any H2 label exists. The API spend is $0.

Planted findings

A separate "planter" subagent writes 10 findings on 10 distinct sampled PRs. It does not see the audit labels.

Each finding mimics the finder's style, using real findings from the same PR as examples.
Each makes a specific, plausible claim that the head code contradicts.
The planter records, for each finding, the file:line that refutes it.

Batch

One fresh auditor gets the same prompt as in H and a blind batch of 30, shuffled with seed 246009:

the 10 planted findings;
15 of H's 24 REAL_MINOR, seed 246009;
5 of H's 27 WRONG, seed 246009.

Rules, fixed now

Leniency check: the planted findings must be labelled WRONG in ≥8 of 10.
- If fewer, the auditor is lenient. H's REAL_MINOR/WRONG split is then reported as unreliable, and the value-filter lever is not built on it.
Split re-check: the share of the 15 REAL_MINOR that the fresh auditor relabels WRONG.
- ≥ 30%: the split is revised, with "true but low-value" reported as overstated.
- < 30%: the split stands.
The 5 WRONG items are reported descriptively, as agreement in the other direction.

H's own pre-registered verdict (REAL_BUG ≤ 10%) is not re-opened by H2.

H2 result: the auditors are not lenient, and H's split stands

The pre-registration is issuecomment-6325568858. One fresh blind auditor labelled 30 items, none of them UNDECIDABLE. API spend: $0.

item	label from the fresh auditor	rule
planted false findings	WRONG 10/10	needed ≥8/10 → PASS, not lenient
H's REAL_MINOR (15)	REAL_MINOR 13, WRONG 2 (13%)	needed <30% → split stands
H's WRONG (5)	WRONG 5/5	agreement, descriptive

The planted findings were made to look like a competent reviewer's work. They quoted verbatim head code and made specific claims, for example a dropped attendee list, a missing migration default, or a commit skipped on empty results. The head code refuted each claim at a named line, and the fresh auditor caught all ten.

Both REAL_MINOR→WRONG flips are types findings. Both are borderline: an inferred return type, and an override array that goes to a different callee.

What follows from H and H2 together:

The Martian noise is about half true-but-low-value and half wrong, with about 5% real bugs. The skeptic correctly declines to refute the true-but-low-value half.
The next lever (a value filter) has a measured basis. It still needs its own pre-registration with a kill check, because test_gap is a core golden category.

Production is unchanged.

Experiment I pre-registration: an impact threshold as the value filter (holdout)

Written before any I call. The instrument is unchanged: scripts/martian_skeptic_replay.py on main. The skeptic already scores impact_score 0–10, and production hides below GUARDIAN_IMPACT_THRESHOLD, which defaults to 0 (hide nothing). So "a value filter" needs no code. It is a threshold choice.

Dev (free, already seen; G's Haiku-medium base verdicts on the gemini-finder reviews)

Curve: the threshold that hides impact_score < t, alongside refuted (noise cut / killed, per rep; tolerance 5.1).
- t=3: 26–27% / 0;
- t=4: 41–43% / 2–3, PASS 3/3;
- t=5: 61–64% / 9–10.
- Sonnet-low has a different scale: it passes only at t=3 (30.4% / 1).
Against H's audit labels at t=4:
- blind controls hidden 0/36;
- REAL_MINOR 35/72;
- WRONG 40/81;
- REAL_BUG 3/9. So t=4 does cost some real-but-ungolden bugs. At t=3 that cost is 0/9.

The threshold was chosen on this data, so it is not evidence.

Holdout

Population: B's Haiku-medium-finder reviews of the same 19 PRs, 3 runs. That is 57 reviews and 405 candidates, a different finder population from the one the threshold was tuned on. They are scored on B's existing Haiku-judge grids, profile core.
Baseline (dry-checked): matched 99, noise 306, kill tolerance 9.9.
Arm: claude-haiku-5-5, effort medium, base skeptic, × 3 reps.
Fixed now: threshold t=4.
Cap: $1.50.

Decision rule

PASS: noise cut ≥ 30% and killed ≤ 9.9 in all 3 reps, at t=4. Worst case: unruled verdicts are counted as hidden-if-matched (G's rule), and validity requires ≤ 5% unruled.
If it passes: propose GUARDIAN_IMPACT_THRESHOLD=4 for a Haiku-medium skeptic. The user decides, and the threshold is only valid for that model.
Descriptive: the whole t=0..7 curve on the holdout, and refuted-only (t=0), as the G-equivalent baseline for this finder.

Experiment I result: t=4 misses the holdout bar narrowly. NOT confirmed.

The pre-registration is issuecomment-6325826722. The holdout is B's Haiku-medium-finder reviews: 57/57 scorable, 405 candidates, 99 matched, noise 306, tolerance 9.9. The skeptic is claude-haiku-5-5 at effort medium, base, × 3. Unruled verdicts: 1 / 0 / 0. Cost $0.63.

threshold	rep 1 cut / killed	rep 2	rep 3
t=0 (refuted only)	3.3% / 3	2.9% / 5	3.9% / 4
t=3	10.8% / 4	11.1% / 6	12.7% / 5
t=4 (pre-registered)	31.4% / 7 PASS	29.4% / 7	29.1% / 8
t=5	50.7% / 14	50.3% / 11	51.6% / 13

Verdict, as pre-registered: FAIL. It passed 1 rep of 3. Two reps missed the 30% cut by 0.6 and 0.9 pp, while kills stayed inside tolerance in every rep. The bar is not moved after the fact; that would be the user's decision.

What it says, descriptively:

The threshold is where a value filter lives. On this finder, the truth-only skeptic (t=0) cuts about 3% of the noise. Haiku's findings are rarely refutable. t=4 cuts about 30%, roughly ×9, at the cost of 3–4 extra kills.
The curve is steep and the cliff is one step away. At t=5, kills jump to 11–14 on the holdout and 9–10 on dev. A threshold one point above the edge of the bar is the edge of what the integer score can resolve.
It is model- and finder-specific. Sonnet-low passes only at t=3 on dev. On the Haiku finder the same skeptic sits at the margin.

Production is unchanged and GUARDIAN_IMPACT_THRESHOLD stays 0. Total #246 spend is ≈ $24.6.

Experiment J pre-registration: does the Claude finder's advantage survive a code-grounded audit?

Written before any J label. No API spend: the auditors are Claude Code subagents with read-only clones, as in H/H2.

Why

B's finder result (Haiku-medium recall 0.54 vs gemini 0.28) was judged by Haiku alone, and the gemini judge died on 402 (#719). A Claude judge may favour Claude findings. J checks the claim two ways that do not rest on that judge's taste.

Populations

B's frozen reviews of 19 Martian PRs, 3 runs, Haiku-judge grids, profile core:

finder	matched	dup-of-matched	no-match
gemini	51	19	172
haiku-medium	99	18	288

Part A: real bugs per review, golden-free

Batches: 3 blind batches for 3 auditors.
Draw: 45 Haiku no-match and 15 Haiku matched, seed 246010.
Planted: 8 of H2's 10 planted false findings. These auditors have never seen them.
Instructions: same prompt and labels as H.
Gemini side: H's labels (no-match REAL_BUG 3/54; matched controls REAL_BUG 8/12).
Estimate per finder: real bugs per review = (matched+dup)·p_matched + no_match·p_nomatch, over 57 reviews. Dup is labelled like matched; this is the conservative choice for neither side.
Validity: planted WRONG ≥ 7/8; Haiku matched labelled REAL ≥ 12/15.

Part B: judge self-preference

Draw: one blind auditor gets 20 Haiku-matched and 20 gemini-matched (finding, golden) pairs, seed 246010, shuffled with the finder hidden.
The question: is this the same issue as the golden comment? SAME / PARTIAL / DIFFERENT, checked against the code.
Bias flag: the Haiku SAME rate is ≥ 15 pp below gemini's.

Decision

B's advantage holds if two things are true:

Audited REAL_BUG per review for Haiku is ≥ 1.5× gemini's. B's recall ratio was about 1.9×.
Part B raises no bias flag.

If either fails, B is reported as inflated, and the recall is recomputed with the audited SAME rates. Either way this only informs a proposal: switching the production guardian to the Claude finder is a separate PR and is the user's call.

Experiment J result: the Claude finder's advantage holds under a code-grounded audit

The pre-registration is issuecomment-6326340042. Part A had 3 blind auditors on 68 items, Part B one blind auditor on 40 pairs. There were 0 UNDECIDABLE labels and $0 of API spend.

Validity

Planted false findings: WRONG in 8/8 (needed ≥7).
Haiku-matched findings: REAL in 12/15, which is exactly the minimum (needed ≥12). All 3 WRONG labels come from one of the three auditors. That auditor was also the harshest on no-match (8 WRONG of 14), so this is auditor variance, not a property of either finder.

Part A: audited real bugs per review (57 reviews per finder)

finder	findings/review	REAL_BUG	REAL_MINOR	WRONG
gemini (labels from H)	4.25	0.99	1.65	1.61
haiku-medium	7.11	2.15	2.74	2.21

Ratio: REAL_BUG per review is 2.18× (needed ≥1.5×).
Where it comes from: Haiku's no-match findings are real bugs more often than gemini's: 7/45 (16%) vs 3/54 (6%). Haiku also matches more goldens, 117 vs 70 including duplicates.
Robustness: the matched stratum alone gives 1.67×.

Part B: judge self-preference

On the (finding, golden) pairs the Haiku judge accepted:

finder	SAME	PARTIAL	DIFFERENT
haiku-medium	17/20 (85%)	3	0
gemini	18/20 (90%)	2	0

The difference is 5 pp, below the 15 pp threshold, so no bias flag. The Haiku judge's matches hold up equally for both finders.

Limit: Part B had no negative control (no planted non-matching pair). Its auditor also checked little code (6 tool calls), because these pairs are mostly settled by the text. "0 DIFFERENT" therefore bounds only the judge's leniency toward both finders equally, which is what the bias question needs.

Verdict, as pre-registered: B's advantage HOLDS

The Claude Haiku-medium finder surfaces about 2.2× the real bugs per review that gemini-2.5-flash does.

The cost is about 1.7× the findings. That includes about 1.4× the wrong ones, 2.21 vs 1.61 per review, so a reader sees more to dismiss. On this finder the base skeptic removes little of it: experiment I measured about 3% at t=0.

Production is unchanged. Switching the guardian's finder is a separate PR, and the call is the user's.

ci(guardian): pass the Anthropic key and effort to the review job (#727)

Experiment K pre-registration: Haiku finder against the finder production actually runs (Mistral)

Written before any K review is generated.

Why

B and J compared the Haiku finder with gemini. But the production finder is mistral-medium-latest; gemini only runs there as the skeptic. So the case for switching the finder (#727 merged, variables not flipped yet) rests on an untested assumption: that Mistral is no better than gemini. K tests that assumption.

Arm

Finder: GUARDIAN_PROVIDER=mistral, model mistral-medium-latest, GUARDIAN_SKEPTIC=off, --slice graph.
Inputs: the same 19 Martian PRs, 3 runs, PR-major. This is B's design unchanged.
Cap: $3 at price mistral=0.40:2.00.
Judge: Haiku low, profile core, cap $1. This is B's judge. J found no self-preference against a non-Claude finder.

Measures

B-style recall against Haiku-medium (0.54) and gemini (0.28). Also findings per review.
J Part A audit, unchanged, done by fresh Claude Code subagents with read-only clones:
- Draw: 45 Mistral no-match and 15 Mistral matched, seed 246011.
- Planted: the same 8 planted false findings.
- Batches: 3 blind batches.
- Estimate: real bugs per review = (matched+dup)·p_matched + no_match·p_nomatch, over the judgeable reviews.
- Validity: planted WRONG ≥ 7/8; Mistral matched REAL ≥ 12/15.

Decision

Switch supported: audited REAL_BUG per review for Haiku-medium (2.15, from J) is ≥ 1.5× Mistral's. Then the maintainer may flip GUARDIAN_PROVIDER=anthropic.
Switch not supported: the ratio is < 1.5×. Then the finder stays on Mistral, and the skeptic question (gemini credits are exhausted) is settled separately.
Invalid: either validity check fails. Then nothing is decided from K.

Experiment K amendment: noise enters the decision

This amendment is written before any K judge grid or audit label exists. Only one finder run has finished so far.

What I had already seen: the smoke review of cal.com #10600 produced 22 findings. For comparison, Haiku-medium averages about 7.1 per review and gemini about 4.3. The rule as registered counts only REAL_BUG per review. A finder that writes three times as many findings can win on that count while burying the review in noise, and with gemini's credits gone there is no skeptic left to filter that noise out.

Added measures, from the same audit

WRONG per review.
Findings per review.
Precision, REAL_BUG / findings.

For Haiku-medium, J already gives 2.15 REAL_BUG, 2.21 WRONG and 7.11 findings per review.

Decision, replacing the one registered earlier

Let r = Haiku REAL_BUG/review ÷ Mistral REAL_BUG/review.

Switch supported in either of two cases:
- r ≥ 1.5, which is the rule as registered; or
- r ≥ 0.8 (Haiku is not materially worse at finding bugs) and Mistral WRONG/review ≥ 1.5 × Haiku's (≥ 3.3).
Switch not supported: r < 0.8. Then Mistral finds clearly more real bugs, and it stays whatever its noise level. If its WRONG/review is ≥ 3.3, a tail is filed for a non-gemini skeptic in front of Mistral.
Otherwise: no clear winner. The finder stays on Mistral and the cost difference decides later.

The validity checks are unchanged.

Experiment K result: Mistral finder (production) vs Haiku-medium

Pre-registration: issuecomment-6334475949. Amendment: -6334905374.

Spend: finder $1.05 and judge $0.51. The audit was done by 3 Opus subagents at no API cost.

Recall against goldens (Haiku judge, profile core)

finder	recall	precision	findings/review
gemini	0.279	0.211	4.25
mistral-medium	0.364	0.065	18.04
haiku-medium	0.541	0.244	7.11

55 of 57 Mistral reviews were judgeable (176 goldens; 183 for the others).

Code-grounded audit, labels for 68 items

stratum	REAL_BUG	REAL_MINOR	WRONG
no-match (45)	0	10	35
matched (15)	5	6	4
planted (8)	–	–	8

Per review, estimated

finder	REAL_BUG	REAL_MINOR	WRONG	findings
haiku-medium (J)	2.15	2.74	2.21	7.11
mistral-medium	0.63	4.34	13.06	18.04

r = 3.42.
Mistral's WRONG is 13.06 per review, against a threshold of 3.31.

Verdict by the registered rule: INVALID

Planted WRONG was 8/8, which passes. But Mistral matched REAL was 11/15, one short of the required 12. As registered, nothing is decided from K.

Not pre-registered: sensitivity and reading

The validity check exists to catch a harsh auditor. The four matched items labelled WRONG look like real misreadings by Mistral rather than auditor harshness. Each one points at the golden's location and gets the consequence wrong:

Prisma updateMany({}) "resets fields": it writes nothing.
An email mapping called "inverted" is the intended mapping.
retryCount > 1 is reachable only for SMS reminders.
A Go device-limit path returns the result it is meant to return.

This is the same "real pointer, wrong argument" pattern seen in E.

Worst case for Haiku: every Mistral matched or dup finding counts as REAL_BUG, which is 104/55 = 1.89 per review. Then r = 1.14 ≥ 0.8, and WRONG stays ≥ 12.6 per review. Under the amended rule the switch would still be supported. On noise, Mistral writes about 6× as many false findings per review as Haiku.

Whether to flip GUARDIAN_PROVIDER=anthropic on this reading is the maintainer's decision. K formally decides nothing.

Production switched (2026-10-22)

The production guardian now runs:

Finder: GUARDIAN_PROVIDER=anthropic. The model is the default claude-haiku-5-5, with GUARDIAN_ANTHROPIC_EFFORT=medium.
Skeptic: GUARDIAN_SKEPTIC=off.

GUARDIAN_MODEL (which held mistral-medium-latest) and GUARDIAN_SKEPTIC_MODEL are deleted. The secret was passed by #727.

The maintainer made this call on B, J and K:

Haiku finds 2.2× the real bugs per review that gemini does (J).
Against the Mistral finder it replaces, the ratio is 3.4×, with about 6× fewer WRONG findings. K is formally INVALID by one item, but the result is robust in the worst case.

The skeptic-tuning line (A, D, E, F, G, I) ends with no Claude skeptic passing the gate, so the skeptic stays off. Open follow-ups: #719, #721, #728.

guardian: cross-model skeptic to cut recall-lean noise without killing recall #246

Description

Activity

zaebee commented on Jun 14, 2026

Update: the hybrid (cloud finder + local cross-family skeptic) works — this is the answer

zaebee commented on Jul 29, 2026

Answered: cross-family is the whole effect, and isolation is what makes it safe

The original question — does a cross-model skeptic cut noise without killing recall?

Honest limits

Follow-ups filed

zaebee commented on Jul 31, 2026

Field data: the same-family pathology is what's running in production right now

The recommended change from the comment above was never applied

It reproduces outside the bench

A precision datapoint the bench can't produce

Caveats

Suggested next step

zaebee commented on Jul 31, 2026

Follow-up: the prod switch is done, and the mechanism holds — but the finder swamps the measurement

Reading refutations from an inline review — a trap

The measurement is not controlled — and the finder, not the skeptic, dominates

Precision of the survivors: mixed, and worse where it matters

Caveats

zaebee commented on Jul 31, 2026

zaebee commented on Aug 1, 2026

Correction to my comment above

zaebee commented on Aug 13, 2026

Evidence for an ensemble rather than a better single reviewer (2026-08-13)

Two things worth carrying into the skeptic work

47 remaining items

zaebee commented on Oct 21, 2026

Experiment G result: "misdirected" is not confirmed on the holdout. Production stays as it is

Validity

Results

Decision, as registered: not confirmed

What the holdout shows

#246 so far, in one line

Cost

zaebee commented on Oct 21, 2026

Experiment H pre-registration: how much of the Martian "noise" is real

Question

Free breakdown (stored grids, already computed)

Audit sample

Reported

Interpretation, fixed now

zaebee commented on Oct 21, 2026

Experiment H result: the Martian noise is genuine, not under-counted goldens

Validity

The noise (191)

Cross-tab with G's Haiku-medium base skeptic (3 reps per item)

Implication (descriptive, not a gate)

zaebee commented on Oct 21, 2026

H: a limit of the instrument (added after the result)

zaebee commented on Oct 21, 2026

H2 pre-registration: a negative control for the H auditors

Planted findings

Batch

Rules, fixed now

zaebee commented on Oct 21, 2026

H2 result: the auditors are not lenient, and H's split stands

zaebee commented on Oct 21, 2026

Experiment I pre-registration: an impact threshold as the value filter (holdout)

Dev (free, already seen; G's Haiku-medium base verdicts on the gemini-finder reviews)

Holdout

Decision rule

zaebee commented on Oct 21, 2026

Experiment I result: t=4 misses the holdout bar narrowly. NOT confirmed.

zaebee commented on Oct 21, 2026

Experiment J pre-registration: does the Claude finder's advantage survive a code-grounded audit?

Why

Populations

Part A: real bugs per review, golden-free

Part B: judge self-preference

Decision

zaebee commented on Oct 21, 2026

Experiment J result: the Claude finder's advantage holds under a code-grounded audit

Validity

Part A: audited real bugs per review (57 reviews per finder)

Part B: judge self-preference

Verdict, as pre-registered: B's advantage HOLDS