Skip to content

feat(dspymator): TypeSafe decision LMs, predict_proba, sklearn scorers - #14

Draft
crowdcent-centaur wants to merge 2 commits into
mainfrom
centaur/dspymator-typesafe
Draft

crowdcent-centaur wants to merge 2 commits into
mainfrom
centaur/dspymator-typesafe

Conversation

@crowdcent-centaur

Copy link
Copy Markdown
Collaborator

Lets DSPyMator run on TypeSafe's non-generative decision models (dspy.experimental.TypeSafe, Jev) and gives every decision-typed DSPyMator real probabilities.

Changes

  • lm accepts any DSPy BaseLM or decision client. Previously anything that wasn't a dspy.LM got re-wrapped as dspy.LM(model_name), which broke TypeSafe.
  • predict_proba and classes_ for a single target typed bool, Literal[...], or Noul/Choice/Score. Probabilities come from DSPy 3.4's record_evidence(), one context per row (a ContextVar, so async rows can't mix). This works with generative LMs too, not just Jev.
  • Classifier tag for decision targets, so sklearn scorers (scoring="roc_auc", neg_log_loss) work in cross_val_score / GridSearchCV.
  • predict/transform unwrap Noul/Choice/Score to plain values.
  • Numpy scalars are converted to Python before they go into DSPy examples or inputs, because TypeSafe's JSON request can't serialize numpy types.
  • uv.lock: dspy 3.0.3 → 3.4.0. The >=2.6.24 floor is unchanged; the new paths are import-guarded.
  • Offline tests with a fake decision client (tests/test_dspymator.py); user-guide section added.

Live check (Jev jev-1.13.0, 300 balanced Onion-vs-HuffPost headlines)

  • cross_val_score(..., scoring="roc_auc"), 3-fold: 0.856 / 0.919 / 0.889. 120 rows × 4 folds runs in about 6s.
  • GEPA with Jev as the student and gpt-5-mini as the reflection model runs end to end (about 300 rollouts, about 3.5 min). It did not beat the seed prompt on this task, in two runs (the second told the reflection model that the student is a non-generative classifier). The rewritten prompts are long, and Jev prefers short instructions. The mechanics work; the lift is unproven.

Gotchas for users

  • Use dspy.Predict, not ChainOfThought: Jev can't write a reasoning field.
  • Every decision output needs OutputField(desc=...).
  • On the Centaur VM, pass api_key="PROXY_TYPESAFE_KEY" (the egress proxy swaps in the real key).

Draft until the demo notebook/video lands.

crowdcent-centaur and others added 2 commits September 25, 2026 23:54
- lm accepts any dspy BaseLM or decision client (dspy.experimental.TypeSafe)
- predict/transform unwrap Noul/Choice/Score values
- predict_proba + classes_ for a single decision target via dspy record_evidence
- tag as classifier so sklearn scorers (roc_auc, neg_log_loss) work
- lock dspy 3.4.0; offline tests with a fake decision client
…zer); fix lint

Co-authored-by: Cursor <cursoragent@cursor.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant