[vbabenko97]

This branch fixes problems a code and methods review found in the validators, the reporter, and the docs. It also switches the reporter to Claude Opus 5.5 at low effort. Python computes every metric, flag, and the risk level; Claude only writes prose from the payload.

Validators (clinval_validators/checks.py)

• validate_predictions rejects a CSV with a missing column, no train or test rows, a label outside {0, 1}, a one-class test set, or a probability outside [0, 1]. The CLI prints the problems and exits 2; the UI shows them and stops.
• compute_risk sets the level: RED when any patient is in both train and test, YELLOW when a subgroup is flagged, GREEN otherwise. The prompt used to ask Claude to choose the level from an undefined "large" gap.
• Each subgroup is compared with the rest of the table instead of the pooled AUC, which contains the degraded rows. A subgroup is flagged when its AUC is at least 0.10 below the rest and the upper end of a patient-level bootstrap 95% interval is below zero. Density and decade age bands are evaluated when those columns exist. On subgroup_density_degradation, density C (0.670 against 0.792 on the rest) was not flagged before and is flagged now.
• The split audit fields are renamed to test_split_as_given, patient_disjoint_sample, and sample_range95. Its patient sample includes training rows and it rescores fixed predictions, so it cannot measure inflation. It no longer enters the risk rule. On a small table a patient sample can contain one class, where AUC is undefined; such seeds are left out of sample_range95, and patient_disjoint_sample and gap are null when the seed-0 sample is one of them. The payload no longer contains NaN, which is not valid JSON.
• The payload adds subgroup_summary (evaluated, flagged, and intervals entirely below or above zero) and pooled_metric.rows: "all". Live reports had miscounted subgroups and called the pooled AUC a test-set AUC.

Reporter (reporter/report.py, prompts/)

• Model claude-opus-5-5, effort low, max_tokens 4096, with server-side refusal fallback (beta server-side-fallback-2026-07-01). The choice comes from a blind review of 24 live reports for each of seven configurations; CLAUDE.md records the results under Gotchas.
• The reporter joins every text block, because a model that thinks puts a thinking block first. It raises ReportNotRenderedError when the API request fails (no credits, bad key, rate limit, no connection), or when the reply stops at max_tokens, is refused, or has no text. The CLI then prints the payload and exits 1, and the UI shows the reason instead of a traceback.
• The UI passes the typed API key into the reporter. It used to write the key into os.environ, which every Streamlit session in the process shares. The key field also used to be prefilled with the server's ANTHROPIC_API_KEY, which sent that key to every browser; it now starts empty, and an empty field uses the server key on the server.
• The prompts move from prompts/opus-4-8/ to prompts/. They add rules for mistakes seen in live reports: the split audit is not evidence for or against inflation, a calibration gap is stated with both numbers, and the critique gives exactly one recommendation.

Tests and CI

• The fresh-seed subgroup test uses generator seeds 7–11 and asserts the measured outcomes, misses and false alarms included. Over seeds 7–36, 6 of 30 clean tables get a flag, the M030 drop is missed on 4 of 30, and the density drop is flagged on 21 of 30.
• A GitHub Actions workflow runs pytest on Python 3.11 and 3.13 with actions/checkout@v7 and actions/setup-python@v7 (Node 24). It uses no secrets and makes no API call.

Docs

README, CLAUDE.md, and the data and scenario READMEs describe the current code. CLAUDE.md records why a one-sided Bonferroni correction was measured and not adopted. The July 2026 planning notes move to docs/archive/.

Verification

• python -m pytest -q: 59 passed on anthropic 0.116.0 and on 1.15.0, the version a fresh install gets. ruff check: clean.
• An offline run against a mock transport sent the same request on both SDK versions: model, effort, the fallback beta header, and fallbacks: "default".
• A Streamlit AppTest run with a stand-in server key and a client that returns 403 showed an empty key field, no server key in the widget tree, and the API error as an on-page message.
• With an API key, clinval-validate --predictions claude-science-artifacts/scenarios/optimistic_site_shortcut.csv returned a complete RED report whose numbers match the payload.

Known limit: risk reports from every model tested sometimes run past the prompt's 300-word limit.

[vbabenko97]

Code review

Found 1 issue:

1. sample_range95 is [NaN, NaN] when any of the 200 patient samples has one class. roc_auc_score returns NaN for that seed, and np.percentile keeps the NaN, so one undefined sample replaces the whole range. validate_predictions still accepts the table, because it only requires both classes on the test split. A 200-patient table with 12 positive images (1.5% of rows) passed validation and returned a numeric seed-0 AUC with sample_range95: [NaN, NaN]. The subgroup bootstrap drops replicates that lack a class before it takes a percentile. (bug due to audit_split_quality writing every seed into sample_aucs and passing that array to np.percentile)

clinval-validator/clinval_validators/checks.py

Lines 186 to 191 in bf99252

	splitter = GroupShuffleSplit(n_splits=1, test_size=0.15, random_state=seed)
	_, sample_idx = next(splitter.split(df, groups=groups))
	sample_aucs[seed] = roc_auc_score(y_true[sample_idx], y_prob[sample_idx])
	low, high = np.percentile(sample_aucs, [2.5, 97.5])
	patient_disjoint_sample = float(sample_aucs[0])

🤖 Generated with Claude Code

_{- If this code review was useful, please react with 👍. Otherwise, react with 👎.}