Repository navigation
Assessment BATCH: Clean Up Submission Config #1154
Copy link
Copy link
Closed
Labels
Description
Is your feature request related to a problem?
For BATCH assessments, the prompt template is tied to the API request instead of the config, which leads to runtime errors from typos and inefficient handling of input schemas. This matters because it causes confusion, potential data corruption, and increases the risk of errors during assessments.
Describe the solution you'd like
- Move the prompt template into the config as
submission. - Hoist
input_schemato the top level ofconfig_blob. - Validate all placeholders in
submissionagainstinput_schemakeys at config-save. - Remove
queryfrom the request; interpolatesubmissionserver-side. - Eliminate
duplicate_detectionfrom the API-client pipeline.
Original issue
Problem
For BATCH assessments, the per-row prompt template lives on the API request (BatchInput.query), separate from the input_schema it interpolates (buried in config_blob.assessment.params). Consequences:
- A typo'd
{column}in the template is caught only at run time — the literal{placeholder}leaks into the prompt sent to the provider, silently corrupting the assessment. No early feedback to the config author. - Every caller resends the full template on each request, though it belongs to the config.
input_schemasits inside the assessment provider params even though it describes the shared input rows (it's already popped before the provider call — proof it's not a provider param) and pre-filters have to reach cross-block to validate against it.- The
duplicate_detectionpre-filter in the API-client config adds confusion and duplicates a legacy-RUN concept.
Proposed change (BATCH only)
- Move the prompt template into the config as
submission— mandatory onassessment.params, optional on each pre-filter'sparams. Holds the{column}template. - Hoist
input_schemato theconfig_blobtop level (flat sibling ofpre_filters/assessment). One data contract, read by both consumers.json_output_schemastays assessment-specific. - Validate at config-save: every
{placeholder}in everysubmissionmust resolve against the top-levelinput_schemakeys; an unknown column rejects the save. Runtime only validates the structureddatarows against the schema. - Remove
queryfrom the request (BatchInput); server interpolatessubmissionwith each row's values. - Remove
duplicate_detectionfrom the API-client pipeline (legacy RUN pipeline untouched).
RESPONSE is out of scope — no proper wiring today; its query removal is cosmetic.
Target config shape
{
"config_blob": {
"input_schema": {
"submission_id": { "type": "text" },
"level": { "type": "text" },
"artwork_image": { "type": "image", "format": "url" }
},
"pre_filters": {
"topic_relevance": { "provider": "openai", "params": { "instructions": "...", "submission": "grade {submission_id} at {level}" } }
},
"assessment": {
"provider": "openai",
"params": { "instructions": "...", "submission": "assess {submission_id} {level} {artwork_image}", "json_output_schema": { } }
}
}
}Acceptance criteria
-
submissionmandatory on assessment, optional on pre-filter. -
input_schemais a mandatory, non-empty top-levelconfig_blobfield. - Config save rejects a
submissionwith an undeclared{column}(assessment and pre-filter). -
queryrejected onBatchInput(extra=forbid); prompt sourced from configsubmission. - Runtime validates
datarows against the top-levelinput_schema. -
duplicate_detectiongone from API config/pipeline/results; legacy RUN unaffected.
Notes / impact
- Breaking: clients sending
querynow 422; ASSESSMENT configs now require top-levelinput_schema+submission. - No DB migration (
config_blobis JSONB); existing configs re-validate only on next save. - Frontend (kaapi-frontend) config CRUD + submit form need a follow-up.
Implemented in #1153.
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
Type
Projects
- StatusShow more project fieldsClosed
Activity
[-]Assessment BATCH: move prompt into config `submission`, hoist `input_schema`, validate at config-save[/-][+]Batch: Improve prompt template config[/+][-]Batch: Improve prompt template config[/-][+]Assessment BATCH: Clean Up Submission Config[/+]github-actions commentedon Aug 26, 2026
🎉 This issue has been resolved in version 1.5.0-main.9 🎉
The release is available on GitHub release
Your semantic-release bot 📦🚀