Repository navigation
Foundation: decision backends with typed answers, replay, and scripted modes #63
Copy link
Copy link
Closed
Labels
priority: p0Blocks everything else; do firstBlocks everything else; do firstsize: MAbout a dayAbout a daytype: foundationShared code, tooling, CI, or docs that recipes depend onShared code, tooling, CI, or docs that recipes depend on
Milestone
Description
Goal
One small interface that every notebook calls, so a recipe runs offline from fixtures and live against Jev with the same question definitions.
Scope
- Question definitions for
Choice,Noul, andScorethat mirror the TypeSafe request shape:instructions, andcriteriaas a mapping for Choice or an ordered list for Score. Read S02 and the SDK reference before designing these. - Typed answer objects that mirror the response shape: Choice has
choice,probabilities,confidence; Score hasscore,probabilities,confidence,legend; Noul hasnoulonly. - A
Backendprotocol with one call takingstateand a mapping of named questions, returning named answers plusmodelandusage. ReplayBackend: answers from recorded fixtures keyed by a stable hash of state and questions. A miss raises an error that names the missing key. It never fabricates an answer.ScriptedBackend: a deterministic, seeded stand-in driven by a function, for closed-loop recipes where the next request depends on earlier answers. Its answers are always marked synthetic.get_backend()that returns the replay or scripted backend by default and the live backend only whenJEV_COOKBOOK_LIVE=1.- Every answer carries provenance:
syntheticorrecorded, and for recorded answers the model string and date.
Acceptance criteria
- Unit tests cover hashing stability, replay hits and misses, scripted determinism under a seed, and provenance on every answer
- A request mixing all three question types round-trips through
ReplayBackend - Importing the package and using the offline backends works with
typesafe-sdknot installed - Public API is documented with docstrings and one short usage example
- Lint and tests pass in CI, or locally with the output pasted in the pull request if CI does not exist yet
Dependencies
- Blocked by Foundation: project scaffolding and Python package skeleton #61 (project scaffolding and Python package skeleton)
- Unblocks Foundation: opt-in live Jev backend and response recorder #64, Foundation: fixture specification and validator #65, Foundation: recipe template and scaffolder #68, Foundation: simulation and state primitives #70
- Tracked in Epic: Foundation #74
References
- Recipe contract
- Catalog
- TypeSafe documentation index for tools: https://docs.typesafe.ai/llms.txt
- Python SDK: https://docs.typesafe.ai/sdk/python
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
priority: p0Blocks everything else; do firstBlocks everything else; do firstsize: MAbout a dayAbout a daytype: foundationShared code, tooling, CI, or docs that recipes depend onShared code, tooling, CI, or docs that recipes depend on
Activity
Foundation: reconcile recipe catalog integration and orchestration ru…
Foundation: evaluation toolkit (#83)
Foundation: notebook style, plotting theme, and run-mode header (#85)
Foundation: fixture specification and validator (#94)