Repository navigation
quest: quest eval keeps each set in Kixel Quest #789
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,38 @@ | ||
| # [S] quest eval keeps each set in Kixel Quest | ||
|
|
||
| ## Goal | ||
|
|
||
| `quest eval add` writes a case to Kixel Quest and `quest eval run` reads the | ||
| repository's set from it; `review-eval.json` and `EVAL_FILE` are gone, and so | ||
| are the audits' baselines in this tree, which move to Kixel Quest too. The | ||
| [mistake audit](/quest/m0/mistake-audit.md) and | ||
| [review-dependents](/quest/m0/review-dependents.md) add their cases there, | ||
| for every repository, instead of to a repository's own set. | ||
|
|
||
| Non-goals: changing how a case is found, located, or scored. | ||
|
|
||
| ## Plan | ||
|
|
||
| Decided 2026-11-19 (maintainer, pick round of | ||
| [moq-dev/moq#9207](https://github.com/moq-dev/moq/pull/9207)): "Kixel quest | ||
| for any historical information. Git is for current agents." ✅ Apply it here | ||
| too, once Kixel Quest stores eval sets (rejected: moq only; drop the | ||
| follow-up). Confirmed 2026-11-20 (maintainer, `/quest-plan` after a | ||
| `/quest-run` in moq-dev/moq). | ||
|
|
||
| - Talk to Kixel Quest as `quest records` does, with its record token; a | ||
| missing token fails `eval` with how to get one. No local-file fallback: this | ||
| tree's Goal allows depending on Kixel Quest. | ||
| - Import this repository's `review-eval.json` before deleting it, and check | ||
| that `quest eval run` scores the same before and after. | ||
|
Comment on lines
+26
to
+27
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Specify how every repository's existing The plan deletes
|
||
| - Move each audit's baseline (such as the mistake audit's "Baseline from the | ||
| 2026-11-18 run") into Kixel Quest's records the same way, and have each | ||
| audit read its baseline from there. | ||
| - [review-dependents](/quest/m0/review-dependents.md) keeps its 14-case set | ||
| in a clone until this lands, then adds it here. | ||
|
|
||
| ## Waiting | ||
|
|
||
| - kixel-ai/kixel.quest: its quest `quest/m1/eval-sets.md` lands, so Kixel | ||
| Quest stores and serves eval sets. Check that the file is gone from that | ||
| repository's tree. | ||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Specify how
quest eval runselects audit cases.The Goal says
quest eval runreads the repository's set. It also says the mistake audit and review-dependents add their cases to a set shared "for every repository." The Plan does not state how a run includes those shared cases, or how an audit's cases are kept separate from another audit's. Add that selection rule to the Plan before implementation. Without it, an implementer cannot tell whether an audit's cases run for each repository, and the plan's Goal and Plan can conflict.