HUB-047 stage 2: question set, nugget reference and annotation prompts - #17
Merged
Conversation
…mpts
Stage 2 of HUB-047: the artefacts an annotator and a scorer both consume.
- tests/fixtures/answer_eval_questions.json -- 15 questions, 8 job-scoped and
7 over bounded topic/tag filters.
- tests/fixtures/answer_eval_nuggets.json -- q-nginx-buffering annotated, 15
nuggets (7 vital), each carrying the document span it came from.
- tests/prompts/{nugget_extraction,nugget_assignment}.md -- the two prompts,
so the remaining 14 questions are a mechanical run.
- tests/validate_answer_eval_nuggets.py -- verifies every span verbatim
against the corpus. All 15 verify.
Annotation used a model from a different family than the qwen3.5:9b under
test, mined from documents resolved by scope, at zero metered judge cost.
Three findings the design had not anticipated. Unfiltered corpus questions
cannot be annotated at all: the scope is 679 documents and any sampling of it
reintroduces the relevance judgement the document-scope rule exists to
exclude, so corpus coverage is measured over bounded filters and the exclusion
is recorded in the fixture rather than dropped silently. Span verification
needs Unicode folding to be usable -- curly quotes and em dashes defeat literal
comparison on text that is genuinely present -- without weakening the rule that
the span must appear word for word. And the first annotated scope is 39%
noise: four "consequence" dictionary entries, a boilerplate stub and a 404
page among 18 retained documents, which is a screening finding worth its own
look.
The nugget set is machine-annotated and not yet spot-checked; the fixture says
so, and a test asserts that disclosure stays present. A second test asserts
the validator imports nothing from app/, so nuggets can never be built from
what retrieval surfaced.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The artefacts you asked to have committed, so an LLM can score from them.
What's in
tests/fixtures/answer_eval_questions.jsontests/fixtures/answer_eval_nuggets.jsonq-nginx-bufferingannotated: 15 nuggets, 7 vital, each with its source spantests/prompts/nugget_extraction.mdtests/prompts/nugget_assignment.mdtests/validate_answer_eval_nuggets.pyAll 15 spans verify. Annotated by a different model family than the
qwen3.5:9bunder test, mined from documents resolved by scope, zero metered judge calls.Three things the design didn't anticipate
excluded_scopes, not dropped silently.aa216228retained four dictionary/thesaurus entries for the word "consequence", an API-gateway boilerplate stub and a 404 page: 7 of 18 documents carry nothing on Nginx. That's a screening finding worth its own look — how did "consequence" enter the query plan? It also makes this question a genuine test of finding signal in a noisy scope.Honesty guards, asserted by test
validate_answer_eval_nuggets.pyimports nothing fromapp/— asserted by AST inspection, so nuggets can never be built from what retrieval surfaced. That self-reference is what mechanically produced perfect recall for one published configuration (arXiv:2608.03860).Verification
422 tests + 622 subtests green in-container (17 new). Span validation run read-only against the live corpus: 15 verified, 0 failures. No deployed container recreated.
🤖 Generated with Claude Code