feat: let an analysis cite from an explicitly versioned corpus

docs/DEVELOPMENT.md

@@ -69,6 +69,36 @@ uv run pytest -q
 
 真实 UltiCode HTTP / 模型 e2e 仍是显式 opt-in；`e2e_sourced_analysis.py` 使用 agent-authored synthetic Markdown corpus（不是提交、DTO 或用户授权材料），分析输入则是 authenticated user 的 validated read-only submission projection，且不调用真实模型。`e2e_sourced_analysis_model.py` / `e2e_model_qa.py` 是真实模型入口，调用形如 `DEEPSEEK_API_KEY=... DEEPSEEK_MODEL=<model> uv run python e2e_sourced_analysis_model.py`，仍需显式提供现有环境和 `DEEPSEEK_API_KEY` 与 `DEEPSEEK_MODEL`；不得把本地开发账号密码、Cookie、源码、检索文本或模型回答写入日志。可执行题集和当前评估状态见 `services/agent/data/keyword_cases.json` 与对应 Linear 任务。
 
+`e2e_answer_evaluation.py` evaluates generated answers on the development split only. Its answer
+pass receives the case question and retrieved evidence, not expected/allowed/forbidden outcomes;
+it must return explicit retrieved chunk IDs in `citations`, and judging checks only those citations.
+Holdouts remain sealed. Export the existing `DEEPSEEK_API_KEY` before this opt-in call; the entry
+requires `DEEPSEEK_MODEL`. The maximum-call ceiling is `CALLS_PER_CASE × development case count`,
+where each case has two logical passes and up to three billed attempts per pass. The default remains
+64; retries are billed again and counted in each row's `model_calls`. Usage is emitted from the model
+session even when a later answer or judge pass aborts, with unknown token totals labelled `unknown`.
+
+The runner treats the generated answer as one untrusted JSON string value and tells the judge to
+ignore directives inside it (a prompt boundary, not proof of injection immunity), reserves the
+verdict and metadata-sidecar destinations in a consistent lock order before the first billed call
+and never overwrites an existing artifact, snapshots
+the corpus and case file once before the calls so the artifact identifies the material actually
+judged, and writes results under the user state directory without printing answer text. Before
+opening the model session, it checks all case queries against retrieval's limit and preflights each
+answer prompt and every possible citation subset's judge prompt using a 1,000-Unicode-character
+worst-case answer; accepted answers are limited to 1,000 Unicode code points, so an over-limit
+response aborts before its judge call. Calculate the required ceiling from the executable config and
+loaded case file, then run:
+
+```bash
+cd services/agent
+DEEPSEEK_MAX_CALLS="$(
+  uv run python -c 'from e2e_answer_evaluation import CALLS_PER_CASE, development_cases; from keyword_evaluation import _CASES_PATH, load_cases; print(CALLS_PER_CASE * len(development_cases(load_cases(_CASES_PATH))))'
+)" \
+ULTICODE_ANSWER_EVAL=1 DEEPSEEK_MODEL=<model> \
+uv run python e2e_answer_evaluation.py
+```
+
 真实模型入口的调用形式（`DEEPSEEK_MODEL` 为必填，脚本不再继承任何默认模型标识。`DEEPSEEK_MAX_TOKENS` 默认值按入口不同：`e2e_sourced_analysis_model.py`（每次运行仅 1 次调用）默认 2000——实测 300 时该次决策被输出上限截断（`finish_reason=length`、`content_len=150`）而报 `ModelProtocolError`，2000 时同一 prompt 通过；`e2e_model_qa.py`（最多 8 次调用）保持 300，真实运行在 300 下即通过，不应无证据地抬高其每次上限）：
 
 ```bash
@@ -94,7 +124,20 @@ uv run python e2e_citation_support_model.py
 
 `DEEPSEEK_API_KEY` 只在真正进入判定阶段才需要：检索、完整性门禁与引用条数不足（`insufficient_citations`）都在**只读预检**里完成，因此没有凭据也能看到材料缺口。
 
-它只判定**分析实际发出的引用**：每条引用一次模型调用，只问「片段是否支持结论」（`supports` / `derivable`）；`exists` 始终取自确定性完整性门禁，不由模型决定。verdict 写盘后经 `load_verdicts` 读回再汇总，因此仍按重算 id 绑定到具体 claim/quote。少于 `ULTICODE_CITATION_REQUIRED_ROWS`（默认 3）报 `insufficient_citations`；未过**完整性门禁**的引用在**发起任何模型调用之前**就报 `citation_integrity_failed`；verdict 汇总阶段发现不支持的引用报 `citation_gate_failed`。这些都以至退出码 1 结束，不报「低分通过」。输出行标注 `judge=model` 与 `human_review=not_performed`，以便与将来的人工复核记录区分。verdict 默认写到**状态目录**（`$XDG_STATE_HOME` 为绝对路径时用它，否则 `~/.local/state`）下的 `ulticode/citation-verdicts-<随机>.json`：每次运行独立，且不落在 checkout 里，可用 `ULTICODE_CITATION_VERDICTS` 改路径；无论哪种，目标位置都会在**付费调用之前**被独占占位，不可写或已被占用即 `verdict_destination_unusable`；写盘同时生成 `<path>.meta.json` 附属文件，记录判定者（`judge=model`）、模型标签、语料、阈值与提交事实摘要，使 verdict 脱离本次会话仍可解释。密钥只从环境读取，不得写入日志或仓库；上面两条真实模型入口示例中的 `DEEPSEEK_API_KEY=...` 只是占位符，实际运行同样应先在环境中导出。
+它只判定**分析实际发出的引用**：每条引用一次模型调用，只问「片段是否支持结论」（`supports` / `derivable`）；`exists` 始终取自确定性完整性门禁，不由模型决定。执行前会按固定顺序预留 verdict 与 metadata sidecar 路径，避免并发评测互相覆盖；verdict 写盘后经 `load_verdicts` 读回再汇总，因此仍按重算 id 绑定到具体 claim/quote。可选地用仓库外的语料替换默认样本（**两个变量都要或都不要**，否则 `FAIL reason=corpus_source_incomplete`，且不会回退到默认语料）：
+
+```bash
+# DEEPSEEK_API_KEY 必须已由操作者导出或由密钥存储注入到环境，绝不出现在本命令行；
+# 缺失时以 deepseek_api_key_required 结束。
+ULTICODE_CITATION_SUPPORT=1 DEEPSEEK_MODEL=<model> \
+ULTICODE_CITATION_CORPUS_DIR=/绝对路径/语料目录 \
+ULTICODE_CITATION_CORPUS_MANIFEST=/绝对路径/manifest.json \
+uv run python e2e_citation_support_model.py
+```
+
+声明一律以 manifest 为准（缺失/不可读/非法或嵌套过深 JSON/校验不过 → `corpus_manifest_unusable`），每条必须逐字等于源码里钉死的**材料类别**——`ACCEPTED_PERMISSION` / `ACCEPTED_SCOPE` / `ACCEPTED_SAMPLE_KIND` / `ACCEPTED_ACCESS_SCOPE`（授权材料时四个一起改）（`corpus_declaration_unsupported`）；manifest 的祖先路径也逐段通过 no-follow 目录描述符打开，拒绝祖先符号链接；manifest 必须是以 `O_NOFOLLOW|O_NONBLOCK` 打开的普通文件，按最多 1 MiB 有界读取并确认 EOF（符号链接、FIFO、非普通文件及超限均报 `corpus_manifest_unusable`）；manifest **只读一次**，其条目、对应文档与钉死类别作为同一份不可变快照贯穿检索、worksheet 与 verdict 元数据，运行中途替换文件不会让它们描述不同材料；根目录必须是真实目录而非符号链接（`corpus_root_unusable`），manifest 至少声明一条（`corpus_empty`）；根路径从 `/`（绝对路径）或 cwd 描述符（相对路径）开始逐段以 `O_DIRECTORY|O_NOFOLLOW` 打开，祖先符号链接也被拒绝，条目相对该描述符打开（根与条目在检查与读取之间被换成链接都会被内核拒绝），按 manifest 自身 `source_path` 的**纯文件名**解析（绝对路径或外部路径不是实际打开的文件，报 `corpus_entry_path_not_relative`；引用里记录的 `source_path` 即该已验证文件名），一条对应一个文件，符号链接（`corpus_entry_escapes_root`）、缺失（`corpus_entry_missing`）、两个条目指向同一个**文件**（含同一 inode 的两个硬链接名，`corpus_entry_duplicate_source`；副本按规范化文本比较，Unicode NFC 规范等价、CRLF/LF 或无关尾随空白差异也算同一片段；NFC 仅用于去重，不改变原文、原始摘要与行号 → `corpus_entry_duplicate_content`；条目用 `O_NOFOLLOW|O_NONBLOCK` 单次打开，非普通文件先经 `fstat` 拒绝，FIFO 无 writer 也不会阻塞预检，检查与读取之间被换成符号链接也由内核拒绝——总之单片段不得被计成多条引用）、不可读/非 UTF-8/为空/超 `MAX_SOURCE_CHARS`/有界读取未到 EOF（前缀过检而后缀未读）（`corpus_entry_unusable`）都直接失败；空 manifest 列表单列为 `corpus_empty`（区别于解析不了的 `corpus_manifest_unusable`）；`source_position` 由**原始文件**推导（首末非空物理行，仅 LF、CRLF、CR 算换行，前导空行也计入），与 manifest 不符即 `corpus_entry_position_mismatch`。不设这两个变量时行为与默认完全一致。证据行与 verdict 元数据里的 `corpus=` 取自钉死的 permission，因此非 synthetic 材料不会被标成 synthetic。契约细节见 `services/agent/README.md`。
+
+少于 `ULTICODE_CITATION_REQUIRED_ROWS`（默认 3）报 `insufficient_citations`；阈值**高于检索上限**（`MAX_RESULTS=3`，任何语料都够不到）直接报 `citation_threshold_above_retrieval_limit`，不冒充材料缺口；preflight 阶段就做 manifest↔文件绑定，摘要或 `chunk_id` 对不上报 `corpus_entry_unbound`（在登录之前，不进半程失败）；`source_path` 含 NUL 字节报 `corpus_entry_unusable`（`os.open` 抛 `ValueError`，已归一）；未过**完整性门禁**的引用在**发起任何模型调用之前**就报 `citation_integrity_failed`；verdict 汇总阶段发现不支持的引用报 `citation_gate_failed`。这些都以至退出码 1 结束，不报「低分通过」。citation judge 的 CLAIM、QUOTE、SUBMISSION_FACTS 作为同一 JSON 对象里的独立数据值序列化，契约明确忽略其中的评分指令及伪造字段标签；该输入边界不等于已证明模型免疫注入。输出行标注 `judge=model` 与 `human_review=not_performed`，以便与将来的人工复核记录区分。verdict 默认写到**状态目录**（`$XDG_STATE_HOME` 为绝对路径时用它，否则 `~/.local/state`）下的 `ulticode/citation-verdicts-<随机>.json`：每次运行独立，且不落在 checkout 里，可用 `ULTICODE_CITATION_VERDICTS` 改路径；artifact claim 逐段 no-follow 打开并保留父目录 fd，锁、占用检查、临时文件、发布、回读及清理均相对该目录 fd 执行；模型调用期间或发布时父目录被替换不会重定向输出，显示路径失配则写入失败。发布以原子、不覆盖的硬链接完成，付费调用期间晚出现的目标也会被保留并报写入失败；失败清理仅对本次成功发布的目标核验 inode 后删除（核验与删除不是原子操作，不保证抵抗主动替换已发布文件的 writer）。无论哪种，目标位置都会在**付费调用之前**被独占占位，不可写或已被占用（包括 dangling symlink）即 `verdict_destination_unusable`；写盘同时生成 `<path>.meta.json` 附属文件，记录判定者（`judge=model`）、模型标签、语料、阈值、提交事实摘要，以及 `validated_corpus`——manifest 的 SHA-256 加上每条被判定文档的 id、version、已验证文件名、位置与内容摘要，使 verdict 与判定时的确切材料绑定、脱离本次会话仍可解释。密钥只从环境读取，不得写入日志或仓库；上面两条真实模型入口示例中的 `DEEPSEEK_API_KEY=...` 只是占位符，实际运行同样应先在环境中导出。
 
 关键词 vs 向量的最小对照是**评测专用**的，不切换主路径，且需要一次性单机 Qdrant 与 `eval` 依赖组：
 

services/agent/README.md

@@ -48,6 +48,12 @@ owner-reported total, so an account with a long submission history issues one re
 request per page. The smoke scripts emit only fixed status labels and item
 counts. They do not print response bodies, cookie names or values, tokens, submission source,
 usernames, roles, tool names, source text, or model answer content.
+
+### Answer-level evaluation (development split only)
+
+For the answer-evaluation contract, dynamic call-budget calculation, opt-in command and artifact
+behavior, see the canonical [Development and testing guide](../../docs/DEVELOPMENT.md).
+
 ## U02 boundary
 
 U02 is preparing authorized-corpus retrieval and sourced analysis. The checked-in corpus is an
@@ -62,28 +68,14 @@ the deterministic sample slice only. The executable keyword evaluation is versio
 module; authorized-corpus, vector-retrieval, real-model evaluation, and isolation evidence are
 tracked in the U02 Linear tasks.
 
-`data/keyword_cases.json` annotates every case with `required_evidence`, `answerable`,
-`expected_behavior` (`cite`, `no_evidence`, or `refuse`), `allowed_behavior`, and
-`forbidden_behavior`; the loader rejects a case missing any of them. `refuse` marks questions the
-corpus cannot answer, and each one forbids its own specific claim — locating a code line and naming
-a runtime cause are different errors, so they are not collapsed into one rule.
-
-`src/keyword_evaluation.py` records one entry per case: retrieval hit, citation traceability,
-retrieval outcome, expected versus observed behavior, fabrication risk, tool calls, and elapsed
-time. Any hit on a `refuse` case is recorded as a fabrication risk, and a case with no hit is not
-counted as traceable because it has no citation to trace.
-
-Retrieval facts and answer judgements are kept apart on purpose. `citation_support` and
-`answer_completion` are recorded as `DEFERRED`: a traceable source id proves where a fragment came
-from, not that it supports a conclusion, and deciding that needs the model or human pass tracked in
-DAV-58.
-
-`src/retrieval.py` provides bounded keyword retrieval and source metadata. `src/sourced_analysis.py`
-separates observed submission facts from hypotheses and only cites retrieved fragments. Java services
-remain the authority for identity, ownership, publication, and submission facts. The tool projections
-validate scalar types and bounded string lengths before returning data to the model.
-
 U02 and later phases must preserve U01's regression coverage for unknown tools, invalid arguments,
 tool failure, timeout, cancellation, and the absence of raw source/error data in captured model
 messages. Do not add write tools until the corresponding DAV-53 isolation and confirmation gates are
 satisfied.
+
+
+### Corpus override for the acceptance entry point
+
+The corpus-override command and validation contract for
+`e2e_citation_support_model.py` are documented in the canonical
+[Development and testing guide](../../docs/DEVELOPMENT.md).