Skip to content

HUB-047 stage 2: question set, nugget reference and annotation prompts #17

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
funsaized merged 1 commit into from
Aug 14, 2026
Merged

HUB-047 stage 2: question set, nugget reference and annotation prompts #17

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
35 changes: 33 additions & 2 deletions backlog.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1539,8 +1539,39 @@ The instrument ranks **configurations**, not reports: nugget-style scoring
agrees with human judgment at the system level and is noisy per report
(arXiv:2504.15068, arXiv:2509.26184). It may never gate a single report.

**Next stage:** build the 15-question set against that design — the operator
hand-verification pass is the gating cost and needs scheduling, not code.
**Stage 2 opened 2026-08-13: artefacts committed, 1 of 15 questions annotated.**
`tests/fixtures/answer_eval_questions.json` (15 questions, 8 job-scoped and 7
over bounded topic/tag filters), `tests/fixtures/answer_eval_nuggets.json`
(`q-nginx-buffering`, 15 nuggets, 7 vital), both prompts under
`tests/prompts/`, and `tests/validate_answer_eval_nuggets.py`, which verifies
every span verbatim against the corpus. **15 of 15 spans verify.** Annotation
was done by a model from a different family than the `qwen3.5:9b` under test,
at zero metered judge cost.

Three things the design did not anticipate, found while annotating:

- **Unfiltered corpus questions cannot be annotated at all** under the
document-scope rule: the scope is 679 documents / 44 MB, too large to read
exhaustively, and any sampling of it reintroduces exactly the relevance
judgement the rule exists to exclude. Corpus-scope coverage is therefore
measured over bounded topic and tag filters, which exercise the same
corpus-wide path (HUB-043) with an annotatable scope. Recorded in the
question set as `excluded_scopes`, not silently dropped.
- **Span verification needs Unicode folding to be usable.** Curly quotes, em
dashes and non-breaking spaces defeat literal comparison on text that is
genuinely present. Folding them is not a loosening — the span must still
appear word for word — but without it the guard rejects correct nuggets and
would be switched off.
- **The first annotated scope is 39% noise.** Job `aa216228` retained four
dictionary/thesaurus entries for the word "consequence", an API-gateway
boilerplate stub and a 404 page: 7 of 18 documents carry nothing on the
subject. That is a screening finding, not a defect in the reference, and it
makes this question a real test of whether the pipeline finds signal in a
noisy scope. Worth a look at how "consequence" entered the query plan.

**Next:** annotate the remaining 14 with the committed prompt, spot-check ~25
nuggets by hand to convert the annotator's error rate from an assumption into
a number, then baseline the deployed configuration and decide HUB-049's flag.

### HUB-048 — Knowledge-graph go/no-go, decided by measurement

Expand Down
135 changes: 135 additions & 0 deletions research-hub/tests/fixtures/answer_eval_nuggets.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,135 @@
{
"schema_version": "1.0.0",
"provenance": "HUB-047 nugget reference. Annotated 2026-08-13 by Claude Opus 5 -- a different model family from the qwen3.5:9b under evaluation, so the annotator is not the system being scored. Nuggets were mined from the documents resolved by scope (documents_for_job / documents_matching), never from ScopedRetrievalService output and never from a generated report. Every source_span is verified verbatim against the corpus by tests/validate_answer_eval_nuggets.py.",
"annotator": {
"model_family": "claude",
"generator_under_test": "qwen3.5:9b",
"human_spot_check": "not yet performed -- see limitations.spot_check"
},
"limitations": {
"spot_check": "No human has verified a sample of these nuggets. Until that happens the error rate of the annotator is unknown, and every score computed from this file inherits that unknown. The design (PRDs/hub-047-answer-completeness-scoring.md) calls for ~25 hand-checked nuggets to turn this into a measured number.",
"coverage": "Recall is measured against a self-authored reference. A fact absent from these lists is invisible to the metric, however important it was.",
"single_annotator": "One annotator, no inter-assessor agreement to report."
},
"questions": [
{
"question_id": "q-nginx-buffering",
"question": "Nginx reverse proxy buffering and timeout tuning",
"scope_note": "18 documents in job scope, of which 7 carry no material on this subject (four 'consequence' dictionary/thesaurus entries, an API-gateway stub, a 404 page). Nuggets were drawn from the remaining substantive documents.",
"nuggets": [
{
"id": "N1",
"text": "proxy_read_timeout defaults to 60 seconds.",
"importance": "vital",
"source_document_id": "38973c38-1e9e-51ea-aa1f-664368ce2851",
"source_span": "The default `proxy_read_timeout` is 60 seconds."
},
{
"id": "N2",
"text": "proxy_read_timeout bounds the interval between two successive reads from the upstream, not the total time the backend may take.",
"importance": "vital",
"source_document_id": "7293527e-511b-5630-87ac-5d2c12f60ef0",
"source_span": "`proxy_read_timeout`: After connection, time Nginx waits for backend data. Interval between two read operations exceeding this is timeout."
},
{
"id": "N3",
"text": "A 502 means Nginx could not connect to the backend, while a 504 means it connected but the backend did not respond in time.",
"importance": "vital",
"source_document_id": "7293527e-511b-5630-87ac-5d2c12f60ef0",
"source_span": "502 means can't connect to backend (service down/port blocked), 504 means connected but response timed out"
},
{
"id": "N4",
"text": "proxy_connect_timeout bounds establishing the TCP connection to the upstream, and exceeding it produces a 502 rather than a 504.",
"importance": "vital",
"source_document_id": "7293527e-511b-5630-87ac-5d2c12f60ef0",
"source_span": "Example: `proxy_connect_timeout 10s`, if backend takes 15s to accept connection, Nginx returns 502."
},
{
"id": "N5",
"text": "proxy_buffer_size holds the response headers, one buffer per request, while proxy_buffers is the array used for the response body.",
"importance": "vital",
"source_document_id": "7293527e-511b-5630-87ac-5d2c12f60ef0",
"source_span": "`proxy_buffer_size`: buffer for response headers, one per request"
},
{
"id": "N6",
"text": "proxy_busy_buffers_size may not exceed half of the total buffer allocation.",
"importance": "vital",
"source_document_id": "7293527e-511b-5630-87ac-5d2c12f60ef0",
"source_span": "buffers currently sending to client"
},
{
"id": "N7",
"text": "Server-Sent Events and similar streaming responses require proxy_buffering off, because buffering delays delivery to the client.",
"importance": "vital",
"source_document_id": "7293527e-511b-5630-87ac-5d2c12f60ef0",
"source_span": "backend continuously pushes event stream. If Nginx buffers, client gets delayed messages."
},
{
"id": "N8",
"text": "Buffering exists to decouple a fast backend from a slow client, freeing the backend to serve the next request sooner.",
"importance": "okay",
"source_document_id": "7293527e-511b-5630-87ac-5d2c12f60ef0",
"source_span": "Concept: Nginx doesn"
},
{
"id": "N9",
"text": "An 'upstream sent too big header' error is resolved by raising proxy_buffer_size.",
"importance": "okay",
"source_document_id": "7293527e-511b-5630-87ac-5d2c12f60ef0",
"source_span": "If backend response headers are huge (lots of cookies), you might see"
},
{
"id": "N10",
"text": "The same read-timeout mechanism applies to FastCGI and uWSGI upstreams under fastcgi_read_timeout and uwsgi_read_timeout.",
"importance": "okay",
"source_document_id": "38973c38-1e9e-51ea-aa1f-664368ce2851",
"source_span": "The same mechanism applies to FastCGI upstreams (`fastcgi_read_timeout`) and uWSGI upstreams (`uwsgi_read_timeout`)."
},
{
"id": "N11",
"text": "Timeouts should be raised at the narrowest scope that needs it, because a global raise conceals slow backends.",
"importance": "okay",
"source_document_id": "38973c38-1e9e-51ea-aa1f-664368ce2851",
"source_span": "Set timeouts at the most specific scope possible. A global raise hides slow backend problems; a per-location raise is intentional and auditable."
},
{
"id": "N12",
"text": "proxy_request_buffering off streams a large request body to the upstream instead of buffering it first.",
"importance": "okay",
"source_document_id": "7293527e-511b-5630-87ac-5d2c12f60ef0",
"source_span": "makes Nginx stream directly"
},
{
"id": "N13",
"text": "Without HTTP keepalive to the upstream, every request opens a new TCP connection, exhausting ephemeral ports under high concurrency.",
"importance": "okay",
"source_document_id": "38973c38-1e9e-51ea-aa1f-664368ce2851",
"source_span": "Without HTTP keepalive between Nginx and the upstream, every request opens a new TCP connection. Under high concurrency this exhausts ephemeral ports"
},
{
"id": "N14",
"text": "max_fails and fail_timeout remove a failing upstream server from rotation automatically.",
"importance": "okay",
"source_document_id": "38973c38-1e9e-51ea-aa1f-664368ce2851",
"source_span": "Use `max_fails` and `fail_timeout` to automatically remove a sick upstream from rotation:"
},
{
"id": "N15",
"text": "proxy_next_upstream decides when a request is retried against the next server, covering connection errors, timeouts and 502/503/504 responses.",
"importance": "okay",
"source_document_id": "7293527e-511b-5630-87ac-5d2c12f60ef0",
"source_span": "defines when to retry next server: connection error, timeout, or 502/503/504."
}
]
}
],
"pending_questions": [
"q-redis-appendfsync", "q-transformer-efficiency", "q-kafka-rebalancing",
"q-p-hacking", "q-microservices-tradeoffs", "q-postgres-logical-replication",
"q-kubernetes-autoscaling", "q-corpus-tag-llm", "q-corpus-topic-redis",
"q-corpus-topic-kubernetes", "q-corpus-topic-microservices",
"q-corpus-topic-kafka", "q-corpus-topic-transformers", "q-corpus-topic-nginx"
]
}
123 changes: 123 additions & 0 deletions research-hub/tests/fixtures/answer_eval_questions.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,123 @@
{
"schema_version": "1.0.0",
"provenance": "HUB-047 answer-completeness evaluation set, opened 2026-08-13 against the 679-document corpus (758 observations, 71125 lexical rows). Job-scoped questions use the job's own topic, which is the only query a report's retrieval ever issues. Corpus-scoped questions use bounded filters (topic or tag) rather than the unfiltered corpus -- see excluded_scopes.",
"corpus_state": {
"documents": 679,
"job_sources": 758,
"chunk_fts_rows": 71125,
"qdrant_points": 68072
},
"excluded_scopes": {
"unfiltered_corpus": "A question whose scope is the whole corpus cannot be given a nugget reference under this design. Nuggets must be mined from the documents in scope (never from retrieval output), and the unfiltered scope is 679 documents / 44 MB of markdown -- too large to read exhaustively, and any sampling of it would reintroduce the relevance judgement the rule exists to keep out. Corpus-scoped coverage is therefore measured over bounded topic and tag filters, which exercise the same corpus-wide retrieval path (HUB-043) with an annotatable scope."
},
"questions": [
{
"id": "q-nginx-buffering",
"scope": "job",
"job_id": "aa216228-e1d1-46e6-a892-d0fff815e3a0",
"question": "Nginx reverse proxy buffering and timeout tuning",
"documents_in_scope": 18,
"note": "Smallest job scope (481 KB). 7 of its 18 retained documents are off-topic: four dictionary/thesaurus entries for the word 'consequence', one API-gateway boilerplate stub, one 404 page, and one general Nginx encyclopedia entry. Annotated first partly because that makes it a real test of whether the pipeline finds the signal in a noisy scope."
},
{
"id": "q-redis-appendfsync",
"scope": "job",
"job_id": "d81a18e0-d0ad-4827-982f-54090df95227",
"question": "Redis AOF appendfsync configuration options",
"documents_in_scope": 23
},
{
"id": "q-transformer-efficiency",
"scope": "job",
"job_id": "f56e42bf-748d-48b0-9c25-64a648fbf235",
"question": "Transformer efficiency improvements",
"documents_in_scope": 19
},
{
"id": "q-kafka-rebalancing",
"scope": "job",
"job_id": "8505df01-0a77-4c3a-a297-30523ec1c9e3",
"question": "Kafka consumer group rebalancing strategies",
"documents_in_scope": 20
},
{
"id": "q-p-hacking",
"scope": "job",
"job_id": "a3f7ecd0-f556-41b6-8896-34993b64edb9",
"question": "Statistical methods for detecting p-hacking in published research",
"documents_in_scope": 19,
"note": "Academic PDFs rather than documentation pages; the job whose candidate pool reaches fewest sources (8 of 19, HUB-044)."
},
{
"id": "q-microservices-tradeoffs",
"scope": "job",
"job_id": "bc3e5297-a641-4f21-8be6-b952369251ae",
"question": "Microservices versus monolithic architecture tradeoffs",
"documents_in_scope": 22,
"note": "The HUB-044 motivating case: 15 chunks from 8 of 22 sources reach the model."
},
{
"id": "q-postgres-logical-replication",
"scope": "job",
"job_id": "70a0de86-16b9-4ba6-a8e3-3e08097b9238",
"question": "Postgres logical replication for major version upgrades with minimal downtime",
"documents_in_scope": 25,
"note": "The only job whose candidate pool reaches every embedded source (24 of 24)."
},
{
"id": "q-kubernetes-autoscaling",
"scope": "job",
"job_id": "48c9247e-35bf-4d69-84e0-9792edf225c9",
"question": "Kubernetes pod autoscaling and resource management",
"documents_in_scope": 56,
"note": "Largest job scope (2.6 MB); expensive to annotate."
},
{
"id": "q-corpus-tag-llm",
"scope": "corpus",
"tags": ["llm"],
"question": "What techniques reduce the compute cost of transformer attention?",
"documents_in_scope": 9,
"note": "Smallest bounded corpus scope; annotate this one second."
},
{
"id": "q-corpus-topic-redis",
"scope": "corpus",
"topic_filter": "Redis AOF appendfsync configuration options",
"question": "What durability and latency tradeoffs distinguish the Redis appendfsync settings?"
},
{
"id": "q-corpus-topic-kubernetes",
"scope": "corpus",
"topic_filter": "Kubernetes pod autoscaling and resource management",
"question": "How should horizontal pod autoscaler metrics and scaling behaviour be configured?",
"documents_in_scope": 122,
"note": "Spans several jobs on one topic; the largest bounded scope."
},
{
"id": "q-corpus-topic-microservices",
"scope": "corpus",
"topic_filter": "Microservices versus monolithic architecture tradeoffs",
"question": "When does splitting a monolith into microservices cost more than it returns?"
},
{
"id": "q-corpus-topic-kafka",
"scope": "corpus",
"topic_filter": "Kafka consumer group rebalancing strategies",
"question": "How do Kafka rebalancing protocols reduce consumer group downtime?"
},
{
"id": "q-corpus-topic-transformers",
"scope": "corpus",
"topic_filter": "Transformer efficiency improvements",
"question": "Which transformer efficiency techniques trade accuracy for throughput, and by how much?"
},
{
"id": "q-corpus-topic-nginx",
"scope": "corpus",
"topic_filter": "Nginx reverse proxy buffering and timeout tuning",
"question": "How do reverse proxy buffer sizes and timeouts interact to produce 502 and 504 responses?",
"note": "Deliberately overlaps q-nginx-buffering so the job and corpus paths are comparable on one subject."
}
]
}
59 changes: 59 additions & 0 deletions research-hub/tests/prompts/nugget_assignment.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,59 @@
# Nugget assignment prompt (HUB-047)

Run once per (report, nugget) pair when scoring. This is a matching task over
text supplied in the prompt, so it runs on the **local** model and costs no
metered judge calls — that is a design constraint, not an optimization: an
evaluation that spends metered budget per run cannot be run often enough to be
useful.

Assignment must be calibrated against blind human labels before any score
computed from it is quoted. Lexical matching is not sufficient — it silently
misses paraphrased-but-correct coverage — which is why this is a model and why
its agreement has to be measured.

---

```text
You are given a REPORT and one FACT. Decide whether the report states that
fact. Judge only against the report text; do not use outside knowledge, and do
not reward a report for being about the right topic.

Answer with one label:
"supported" — the report states this fact, in any wording.
"partial" — the report gestures at it but omits what makes it the fact
(e.g. names the mechanism but not the condition or value).
"absent" — the report does not state it.

Output JSON only: {"label": "...", "evidence": "<quote from the report, or null>"}
```

---

## Scoring

Per question:

- `vital_nugget_recall` = vital nuggets labelled `supported` / all vital
nuggets. **The primary number.**
- `weighted_nugget_recall` = vital counted double, okay counted once.
- `all_nugget_recall` = unweighted over every nugget.
- `partial_rate` = reported separately. `partial` counts as **zero** in every
recall figure; crediting half a fact would let a report that gestures at
everything beat one that establishes anything.

A nugget counts as `supported` only when the report text carrying it belongs to
a claim that **passed the claim gate**. Otherwise completeness could be bought
with unverified assertions, which is the failure the gate exists to prevent.

Report these beside `citation_validity` and never blended into it. The two axes
trade against each other, and a single blended score hides the trade a change
needs to be judged on.

## What this instrument may be used for

Ranking **configurations** — is `EVIDENCE_PACKING=marginal_gain` better than
`rank`, does late chunking help, does a knowledge graph earn its keep.

Not for judging a single report. Nugget scoring tracks human judgment at the
system level and is noisy per topic (arXiv:2504.15068, arXiv:2509.26184), so a
per-report number from it is noise and must never gate publication.
Loading
Loading