Skip to content

Bound generation calls and publish the paired evaluation - #9

Merged
shipitdev merged 2 commits into
mainfrom
codex/bounded-generation-v6
Nov 10, 2026
Merged

shipitdev merged 2 commits into
mainfrom
codex/bounded-generation-v6

Conversation

@shipitdev

Copy link
Copy Markdown
Owner

Changes

  • Bound standalone generation to one writer and one verifier.
  • Format existing valid writer-selected citations in code, preserving claims and verifying the resulting answer.
  • Strengthen scope, completeness, source-support and version instructions. Withhold factual rejections without repair calls.
  • Publish the fixed-cohort comparison and expose provider/model labels in saved dashboard outputs.

Evaluation

  • Same-model 28-case comparison released 14 to 18 answers and reduced citation-format failures from six to zero.
  • Mean model stages fell from 2.91 to 2.00 across 11 shared released answers.
  • Two separately labeled TokenHarbor fallback cases complete the 30-question cohort. One Groq quota failure remains.
  • Paired reference correctness declined on four cases. Reference alignment and changed withholding require review. No broad quality-gain or held-out release claim.
  • Further judge calls stopped on user direction. Cache-only assembly used zero model requests. Relevancy was not rescored.

Validation

  • 160 Python tests and three subtests passed, with two optional skips.
  • Separate RAGAS environment checks passed.
  • Asserted unchanged evidence/references, provider separation, budgets and redaction.
  • Saved results published to the existing TechQA dataset.

@shipitdev
shipitdev merged commit 2cd229b into main Nov 10, 2026
2 checks passed
@shipitdev
shipitdev deleted the codex/bounded-generation-v6 branch November 10, 2026 19:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant