Single-target regression model evaluation

CHANGELOG.md

@@ -22,14 +22,28 @@
 - Multiclass flipper finds where the pairwise margin `p[b] − p[a]` crosses 0;
   multilabel flipper samples pairs per label (new `pair_label` grid column)
   and crosses each label's own threshold.
+- **Single-target regression models.** `task_type: "regression"` in
+  `metadata.json` (default output `variable`, then `predictions`), and
+  `--task regression` with true values from `--targets <parquet>
+  --target-col <name>`, joined by ID as text. `baselines` are `rmse`, `mae`
+  and `r2`; resiliency is blackout R² ÷ clean R² (the share of skill kept)
+  under `metric_based.r2`; jitter variance is scaled by the target's variance.
+  Flipper pairs patients across a clinical cutoff (`--flip-threshold`, in
+  target units, passed to both `simulate` and `evaluate`) and finds where the
+  prediction crosses it. A new `regression` section reports these in target
+  units; classifier JSON is unchanged.
 
 ### Changed
 
 - **Breaking (library):** `Model::predict_proba` returns a `DataFrame` with
   one column per output, `SimulationVariant.y` is a `DataFrame` label matrix,
   `EvaluationConfig` gains `task`, and `OnnxModel::from_file` takes
-  `positive_class_index: Option<usize>`. CLI usage for binary models is
-  unchanged.
+  `positive_class_index: Option<usize>`. `TaskType` gains `Regression` and
+  `EvaluationResult` gains an optional `regression` field. CLI usage for
+  binary models is unchanged.
+- `evaluate`/`compare` `--labels` is no longer required at parse time; it is
+  required for classifiers and refused for regression. `--flip-threshold`
+  still defaults to 0.5 for classifiers.
 - **ROC AUC handles tied scores.** Ties count half (Mann–Whitney) instead of
   depending on row order. Results without ties agree with the previous
   implementation to floating-point rounding.

cli/README.md

@@ -48,7 +48,9 @@ pasteur-cli simulate \
 | Flag | Default | Description |
 |------|---------|-------------|
 | `--input` | *(required)* | Local clean parquet |
-| `--labels` | — | Group-membership parquet ([schema](../docs/source/data-contracts.rst)) for flipper pair sampling (omit to skip flipper) |
+| `--labels` | — | Group-membership parquet ([schema](../docs/source/data-contracts.rst)) for flipper pair sampling (omit to skip flipper). Classifiers only |
+| `--targets`, `--target-col` | — | Regression: targets parquet and its target column ([schema](../docs/source/data-contracts.rst)) |
+| `--targets-id-col` | `node_id` | Regression: ID column of `--targets`, matched as text |
 | `--output` | `output` | Root for sim-type folders |
 | `--dataset-name` | — | Provenance label for logs |
 | `--id-col` | `node_id` | Row id column (becomes `source_row_id`) |
@@ -58,7 +60,8 @@ pasteur-cli simulate \
 | `--jitter-scale` | `1.0` | Jitter noise scale |
 | `--jitter-iters` | `3` | Number of `jitter_<i>` variants |
 | `--positive-group-id` | `103` | `group_id` for positive cohort (flipper). Repeat once per class/label for `--task multiclass`/`multilabel` |
-| `--task` | `binary` | `binary`, `multiclass`, or `multilabel`: how flipper pairs are sampled |
+| `--task` | `binary` | `binary`, `multiclass`, `multilabel`, or `regression`: how flipper pairs are sampled |
+| `--flip-threshold` | — | Regression only: clinical cutoff (target units) that flipper pairs straddle; omit to skip flipper |
 | `--flipper-pairs` | `200` | Cross-label pairs to sample |
 | `--flipper-steps` | `20` | Interpolation steps per pair |
 | `--random-state` | `42` | RNG seed |
@@ -114,7 +117,9 @@ pasteur-cli evaluate blackout \
 |------|---------|-------------|
 | `sim_type` | *(positional)* | e.g. `blackout`, `jitter`, `flipper` |
 | `--sim-root` | *(required)* | Local bundle with `clean/` and `<sim_type>/` |
-| `--labels` | *(required)* | Local `groups.parquet` for label derivation |
+| `--labels` | *(classifiers)* | Local `groups.parquet` for label derivation |
+| `--targets`, `--target-col` | *(regression)* | Local targets parquet and its target column |
+| `--targets-id-col` | `node_id` | ID column of `--targets`, matched as text |
 | `--model` | *(required)* | Local `.onnx` path |
 | `--contract` | — | Explicit `metadata.json` path (default: next to `--model`) |
 | `--input-name` | `input` | ONNX input tensor name |
@@ -123,7 +128,7 @@ pasteur-cli evaluate blackout \
 | `--positive-group-id` | `103` | `group_id` for positive cohort; repeat once per model output (in `output.classes` order) for multiclass/multilabel |
 | `--task` | `binary` | Must match the model's `metadata.json` `task_type` |
 | `--dataset-name` | `simulation` | Internal dataset key in JSON |
-| `--flip-threshold` | `0.5` | Decision threshold for flipper stability (multilabel: labels without a contract threshold; unused for multiclass) |
+| `--flip-threshold` | `0.5` | Decision threshold for flipper stability (multilabel: labels without a contract threshold; unused for multiclass). Regression: clinical cutoff in target units, no default, required for `flipper` |
 | `-o, --output` | stdout | Write `EvaluationResult` JSON to file |
 
 ---
@@ -148,7 +153,7 @@ Same flags as `evaluate`, except:
 - `--predictions-out` writes a local per-patient predictions parquet
 - A single `--contract` applies to every model
 
-**Predictions schema:** `source_row_id`, `variant_name`, `label`, one f64 column per model (stem of model path), `all_agree` (all models on the same side of 0.5).
+**Predictions schema:** `source_row_id`, `variant_name`, `label`, one f64 column per model (stem of model path), `all_agree` (all models on the same side of 0.5). Regression writes `target` instead of `label`, and `all_agree` compares sides of `--flip-threshold` (null without one).
 
 The `--output` directory must already exist.
 
@@ -181,7 +186,7 @@ pasteur-cli cache clear --kind simulations
 
 Defined in [`core/src/schema.rs`](../core/src/schema.rs):
 
-- **`EvaluationResult`**: `baselines`, `evaluations.metric_invariant`, `evaluations.metric_based`, `created_at`
+- **`EvaluationResult`**: `baselines`, `evaluations.metric_invariant`, `evaluations.metric_based`, `created_at`, plus optional `multi` (multiclass/multilabel) or `regression` sections
 - **`ComparisonResult`**: `evaluations: [{ model_label, evaluation }]`
 
 ---

cli/src/args.rs

@@ -36,6 +36,8 @@ pub enum TaskArg {
     Multiclass,
     /// One cohort per label; rows may belong to any number.
     Multilabel,
+    /// One continuous target per row, from --targets; the model predicts it.
+    Regression,
 }
 
 impl From<TaskArg> for TaskType {
@@ -44,10 +46,28 @@ impl From<TaskArg> for TaskType {
             TaskArg::Binary => TaskType::Binary,
             TaskArg::Multiclass => TaskType::Multiclass,
             TaskArg::Multilabel => TaskType::Multilabel,
+            TaskArg::Regression => TaskType::Regression,
         }
     }
 }
 
+/// Where a regression model's true values come from. Classifiers take
+/// `--labels` instead.
+#[derive(clap::Args, Clone)]
+pub struct TargetArgs {
+    /// Regression only: local parquet with one row per patient, holding an ID
+    /// column and the true target value.
+    #[arg(long)]
+    pub targets: Option<PathBuf>,
+    /// Regression only: the target column in --targets (e.g. `hba1c`).
+    #[arg(long)]
+    pub target_col: Option<String>,
+    /// Regression only: the ID column in --targets, matched as text to the
+    /// clean input's --id-col values.
+    #[arg(long, default_value = "node_id")]
+    pub targets_id_col: String,
+}
+
 #[derive(clap::Args)]
 pub struct CacheArgs {
     #[command(subcommand)]
@@ -122,18 +142,26 @@ pub struct SimulateArgs {
     pub jitter_iters: u32,
     #[arg(long, default_value_t = 42)]
     pub random_state: u64,
-    /// Group/cohort membership table used to derive labels for flipper pair sampling.
+    /// Group/cohort membership table used to derive labels for flipper pair
+    /// sampling. Classification only; regression takes --targets.
     #[arg(long)]
     pub labels: Option<PathBuf>,
+    #[command(flatten)]
+    pub targets: TargetArgs,
     /// Cohort `group_id` defining the labels. Binary takes one (the positive
     /// cohort); multiclass and multilabel take one per model output, repeated
-    /// in the model's `output.classes` order.
+    /// in the model's `output.classes` order. Unused for regression.
     #[arg(long = "positive-group-id", default_values_t = [103u32])]
     pub positive_group_ids: Vec<u32>,
-    /// `binary`, `multiclass`, or `multilabel`; must match each model's
-    /// `metadata.json` `task_type`.
+    /// `binary`, `multiclass`, `multilabel`, or `regression`; must match each
+    /// model's `metadata.json` `task_type`.
     #[arg(long, value_enum, default_value_t = TaskArg::Binary)]
     pub task: TaskArg,
+    /// Regression only: the clinical cutoff, in the target's units, that
+    /// flipper pairs are sampled across (one patient below it, one at or
+    /// above). Without it the flipper grid is skipped for regression.
+    #[arg(long)]
+    pub flip_threshold: Option<f64>,
     #[arg(long, default_value_t = 200)]
     pub flipper_pairs: usize,
     #[arg(long, default_value_t = 20)]
@@ -171,8 +199,12 @@ pub struct EvaluateArgs {
     /// Local root with `clean/` and `<sim_type>/` subfolders.
     #[arg(long)]
     pub sim_root: PathBuf,
+    /// Group/cohort membership table. Required for classification; regression
+    /// takes --targets.
     #[arg(long)]
-    pub labels: PathBuf,
+    pub labels: Option<PathBuf>,
+    #[command(flatten)]
+    pub targets: TargetArgs,
     #[arg(long)]
     pub model: PathBuf,
     /// Model contract sidecar (`metadata.json`). Defaults to
@@ -193,20 +225,22 @@ pub struct EvaluateArgs {
     pub positive_class_index: Option<usize>,
     /// Cohort `group_id` defining the labels. Binary takes one (the positive
     /// cohort); multiclass and multilabel take one per model output, repeated
-    /// in the model's `output.classes` order.
+    /// in the model's `output.classes` order. Unused for regression.
     #[arg(long = "positive-group-id", default_values_t = [103u32])]
     pub positive_group_ids: Vec<u32>,
-    /// `binary`, `multiclass`, or `multilabel`; must match each model's
-    /// `metadata.json` `task_type`.
+    /// `binary`, `multiclass`, `multilabel`, or `regression`; must match each
+    /// model's `metadata.json` `task_type`.
     #[arg(long, value_enum, default_value_t = TaskArg::Binary)]
     pub task: TaskArg,
     #[arg(long, default_value = "simulation")]
     pub dataset_name: String,
     /// Decision threshold for flipper crossings: the positive-class
     /// probability for binary models, every label without a contract
     /// `output.thresholds` for multilabel. Unused for multiclass (argmax).
-    #[arg(long, default_value_t = 0.5)]
-    pub flip_threshold: f64,
+    /// Defaults to 0.5. For regression it is the clinical cutoff in the
+    /// target's units, has no default, and must match `simulate`'s.
+    #[arg(long)]
+    pub flip_threshold: Option<f64>,
     #[arg(short, long)]
     pub output: Option<PathBuf>,
 }
@@ -216,8 +250,12 @@ pub struct CompareArgs {
     pub sim_type: String,
     #[arg(long)]
     pub sim_root: PathBuf,
+    /// Group/cohort membership table. Required for classification; regression
+    /// takes --targets.
     #[arg(long)]
-    pub labels: PathBuf,
+    pub labels: Option<PathBuf>,
+    #[command(flatten)]
+    pub targets: TargetArgs,
     #[arg(long = "model", required = true)]
     pub models: Vec<PathBuf>,
     /// Model contract sidecar (`metadata.json`). Defaults to
@@ -238,20 +276,22 @@ pub struct CompareArgs {
     pub positive_class_index: Option<usize>,
     /// Cohort `group_id` defining the labels. Binary takes one (the positive
     /// cohort); multiclass and multilabel take one per model output, repeated
-    /// in the model's `output.classes` order.
+    /// in the model's `output.classes` order. Unused for regression.
     #[arg(long = "positive-group-id", default_values_t = [103u32])]
     pub positive_group_ids: Vec<u32>,
-    /// `binary`, `multiclass`, or `multilabel`; must match each model's
-    /// `metadata.json` `task_type`.
+    /// `binary`, `multiclass`, `multilabel`, or `regression`; must match each
+    /// model's `metadata.json` `task_type`.
     #[arg(long, value_enum, default_value_t = TaskArg::Binary)]
     pub task: TaskArg,
     #[arg(long, default_value = "simulation")]
     pub dataset_name: String,
     /// Decision threshold for flipper crossings: the positive-class
     /// probability for binary models, every label without a contract
     /// `output.thresholds` for multilabel. Unused for multiclass (argmax).
-    #[arg(long, default_value_t = 0.5)]
-    pub flip_threshold: f64,
+    /// Defaults to 0.5. For regression it is the clinical cutoff in the
+    /// target's units, has no default, and must match `simulate`'s.
+    #[arg(long)]
+    pub flip_threshold: Option<f64>,
     #[arg(short, long)]
     pub output: Option<PathBuf>,
     /// Local path for per-patient predictions parquet (agent uploads with `hf`).

cli/src/commands/compare.rs

@@ -5,19 +5,26 @@ use pasteur_model::OnnxModel;
 use crate::args::CompareArgs;
 use crate::data::{
     build_evaluation_inputs, build_predictions_table, flipper_feature_order_from_inputs,
+    resolve_flip_threshold, LabelSource,
 };
 use crate::io::{print_or_write, write_predictions_parquet};
 use crate::model::{load_model, model_label};
 
 pub fn run(args: CompareArgs) -> Result<()> {
+    let task = args.task.into();
+    let source = LabelSource::required(
+        task,
+        args.labels.as_deref(),
+        &args.positive_group_ids,
+        &args.targets,
+    )?;
     let inputs = build_evaluation_inputs(
         &args.sim_root,
         &args.sim_type,
-        &args.labels,
-        &args.positive_group_ids,
-        args.task.into(),
+        &source,
+        task,
         &args.dataset_name,
-        args.flip_threshold,
+        resolve_flip_threshold(task, &args.sim_type, args.flip_threshold)?,
     )?;
 
     let feature_order = flipper_feature_order_from_inputs(&inputs, &args.sim_type);

cli/src/commands/evaluate.rs

@@ -2,19 +2,27 @@ use anyhow::Result;
 use pasteur_core::Evaluator;
 
 use crate::args::EvaluateArgs;
-use crate::data::{build_evaluation_inputs, flipper_feature_order_from_inputs};
+use crate::data::{
+    build_evaluation_inputs, flipper_feature_order_from_inputs, resolve_flip_threshold, LabelSource,
+};
 use crate::io::print_or_write;
 use crate::model::load_model;
 
 pub fn run(args: EvaluateArgs) -> Result<()> {
+    let task = args.task.into();
+    let source = LabelSource::required(
+        task,
+        args.labels.as_deref(),
+        &args.positive_group_ids,
+        &args.targets,
+    )?;
     let inputs = build_evaluation_inputs(
         &args.sim_root,
         &args.sim_type,
-        &args.labels,
-        &args.positive_group_ids,
-        args.task.into(),
+        &source,
+        task,
         &args.dataset_name,
-        args.flip_threshold,
+        resolve_flip_threshold(task, &args.sim_type, args.flip_threshold)?,
     )?;
 
     let feature_order = flipper_feature_order_from_inputs(&inputs, &args.sim_type);
@@ -25,7 +33,7 @@ pub fn run(args: EvaluateArgs) -> Result<()> {
         args.positive_class_index,
         args.null_fill,
         args.contract.as_deref(),
-        args.task.into(),
+        task,
     )?;
 
     let evaluator = Evaluator::new();