[jeremy-wayland]

Rebased onto main after #2 squash-merged (2114f9d). That merge's tree is identical to #2's last tip, so the branch's final tree did not change in the rebase.

Adds single-target regression models to Pasteur's simulation and evaluation. Binary, multiclass and multilabel output is unchanged: in a full before/after run, all 36 output files are identical (see Verification).

What changes

Model contract (metadata.json)

• New task_type: "regression".
• output.name defaults to variable, which is what sklearn-onnx regressors emit, then predictions.
• output.classes is optional. Its one entry names the prediction column; without it the column is called prediction.
• These are checked when the model loads, and rejected:
  • an output width other than 1;
  • output.thresholds;
  • --positive-class-index.

Targets

• Classifiers are labelled by cohort membership, which can't hold continuous values. Regression instead reads --targets <parquet> --target-col <name>, joined to the cohort by ID as text (--targets-id-col, default node_id).
• A cohort row with no target, a duplicate ID, or a missing or non-finite value stops the run with counts and example IDs. A constant target is also an error.

Metrics

Field	Regression meaning
baselines	rmse, mae, r2 on the clean cohort
metric_based.r2.resiliency	blackout R² ÷ clean R²: the share of skill over the mean predictor that survives. 1.0 means blackout cost nothing; 0 means no better than the mean; negative means worse; neutral 1.0 when clean R² ≤ 0
jitter_stability	the existing 1/(1+100v) squash, with v divided by the target's variance so the score doesn't depend on the unit
flipper_stability	where the prediction crosses a clinical cutoff along the path between two patients

• Flipper: each pair is one patient whose true target is below --flip-threshold and one at or above it. Scoring reuses the binary flip finder.
• The same cutoff must be passed to simulate and to evaluate. A grid whose pairs don't straddle the cutoff is rejected. Without a cutoff, simulate skips flipper.
• A new optional regression section reports the same behaviour in target units: target_sd, blackout_rmse, blackout_r2, jitter_prediction_sd, flip_threshold and flipper_never_flipped. It is absent for classifiers.

CLI

• New --task regression, with --targets, --target-col and --targets-id-col.
• --labels with regression is an error, and so is --targets with a classifier.
• evaluate/compare --labels is optional at parse time and checked per task.
• --flip-threshold keeps its 0.5 default for classifiers. Regression has no default.
• compare --predictions-out writes:
  • target, which is null on synthetic flipper rows;
  • one prediction column per model;
  • all_agree: whether every model puts the patient on the same side of the cutoff (null without a cutoff).

Commits

1. core: the regression task, metrics, flipper simulation and scoring.
2. model: the contract and output reading, with two new fixtures.
3. cli: --task regression and --targets.
4. docs: data contracts, metrics, CLI, outputs and CHANGELOG.
5. core: resiliency becomes the R² ratio, after sandbox validation (see the end).

Adding a TaskType variant breaks the exhaustive matches downstream until their layer handles it. So, as in #2:

• pasteur-core builds and passes its tests at commit 1 (47 tests);
• pasteur-model builds from commit 2 (81 tests);
• the whole workspace builds from commit 3 (110 tests).

Every new match names each variant; none uses a wildcard.

Breaking (library):

• TaskType gains Regression.
• EvaluationResult gains regression: Option<RegressionScores>.
• EvaluateArgs.labels/CompareArgs.labels and flip_threshold become Options.

Verification

• cargo fmt --check, cargo clippy --workspace -D warnings (with and without --all-targets) and cargo test --workspace --exclude pypasteur-bindings all pass at the final commit: 110 tests, 32 of them new. A strict (-W) Sphinx build of docs/ passes too.
• Classifier output is unchanged. I ran a synthetic 300-row cohort through Multiclass and multilabel model evaluation #2's tip at the time (dbb8669) and through this branch, for binary, multiclass and multilabel. Each task went through simulate, evaluate (blackout, jitter, flipper) and compare --predictions-out. All 36 outputs match:
  • 15 JSON results, identical apart from created_at;
  • 21 parquet files (simulation bundles and predictions), with identical schemas and identical rows after sorting.
• Regression end to end. I ran the same 300-row cohort, with synthetic HbA1c-like targets and a 6.5 cutoff, through every command using the linear and forest fixtures. RMSE, MAE, R² and blackout error match scikit-learn's mean_squared_error/mean_absolute_error/r2_score on onnxruntime's predictions to within 2.2e-16.
• New tests:
  • core: 6 unit and 9 integration tests (hand-computed RMSE/MAE/R², resiliency, unit-free jitter, the flipper crossing, a mismatched cutoff, constant or missing targets, binary JSON keeping its shape);
  • model: 3 contract and 4 ONNX tests against onnxruntime's outputs;
  • cli: 4 target-join and 1 all_agree unit tests, plus 5 integration tests (end-to-end, cutoff required and checked, cohort labels and classifier models refused, targets missing for part of the cohort).
• Fixtures. generate.py gains LinearRegression and a 5-tree RandomForestRegressor, fit on random data. It now takes an output directory. Regenerating every fixture reproduces each existing fixture's expected values exactly, and only the two new .onnx files and the new expected.json entries are committed.

Known limits

• The cutoff isn't stored in the grid. A mismatch between simulate and evaluate is caught only when some pair doesn't straddle the new cutoff.
• A regression grid evaluated with --task binary isn't detected. The binary path is deliberately untouched.
• Normalised jitter depends on the cohort's target variance, so compare models only on the same cohort.
• Only single-target models are supported. Multi-target regression is a follow-up that can reuse Multiclass and multilabel model evaluation #2's per-label multi machinery.
• generalizability is still a placeholder.

All fixtures and test data are synthetic. No patient data.

Sandbox validation (real models, synthetic HbA1c targets)

Seven regressors were trained: ridge, gradient boosting, random forest, MLP, 1-NN, ridge without glucose, and intercept-only.

Oracle agreement:

• RMSE, MAE and R² match sklearn.
• Ridge's flipper stability matches the closed-form crossing (c − f(a)) / (f(b) − f(a)).

Unit invariance: R², resiliency, jitter and flipper are identical in % and mmol/mol (cutoffs 6.5% and 47.54 mmol/mol).

Validity:

• 1-NN is less jitter-stable than ridge.
• Dose-response is monotone in both noise scale and blackout rate.

Changed after validation: resiliency is blackout R² ÷ clean R²

Clean RMSE ÷ blackout RMSE rewarded models for having little accuracy to lose. When a model loses a feature entirely and falls back to the mean, that ratio is sqrt(1 − R²). With glucose blacked out for every patient:

model	R² clean → blackout	RMSE ratio	R² ratio
ridge	0.659 → 0.274	0.685	0.415
gbr	0.657 → 0.280	0.690	0.426
mlp	0.667 → 0.264	0.673	0.396
1-NN	0.262 → −0.092	0.822	−0.350

Under the RMSE ratio, 1-NN looked the most resilient after falling below a mean predictor. Under the R² ratio, it ranks last.

• Key and field: metric_based is now keyed r2, and the regression section gains blackout_r2.
• No-skill models: a model with clean R² ≤ 0 has no skill to keep and is reported at the neutral 1.0, matching how Multiclass and multilabel model evaluation #2 skips labels at chance.

Docs: flipper never-flipped share

Regression models shrink towards the mean. With the cutoff in the tail (8% at or above 6.5%), 57–68% of pairs never cross for well-fitted models, and those pairs dominate flipper_stability. metrics.rst now says so.

After rebasing onto main (5ed3bc8), these all pass at commit 5: fmt, clippy (--all-targets -D warnings), the workspace tests, the strict Sphinx build, and all 78 sandbox checks.

🤖 Generated with Claude Code