feat(benchmark): add evidence-bound trajectory assessment

[loopx-agent]

Goal And Delivered Outcome

Long-horizon research can accumulate useful numerical diagnostics while never testing its proposed task benefit. Score-only views miss that distinction, while an ungrounded “research quality” score can reward it.

This adds an opt-in, source-bound trajectory-assessment producer to benchmark-toolkit. A declared episode/evidence pack now yields six independently displayed judgments with citations, reasons, unknown coverage and fixed denominators. benefit_test distinguishes a completed objective-utility test from prerequisite-only work. Preview makes no model call; explicit execution invokes local Codex, stores a versioned sidecar and separate analysis usage, and supports identical-input reuse, cooperative cancellation and explicit failure retry. Trusted registration binds a fixed pack to its owner Goal after verifying real source hashes, literal excerpts and contained paths. The workbench host discovers it through opaque identifiers and preserves historical results when sources change. It never changes solver state, Explore facts or benchmark scores.

• Outcome basis: maintainer-requested trajectory process assessment for the existing Goal research-history journey; S5/S11 research measurement refinement.
• Intended base: main; merge base 71096c16bfbf12ed4305883b4242028ff1504a6c.

Author Declaration

• Written by: model_agent — OpenAI Codex.
• Specification: docs/architecture/rfcs/long-horizon-harness-benchmark-research-program-v0.md, “Evidence-based episode evaluation”, at the merge base above; the concrete producer contract is docs/reference/trajectory-assessment.md in this change.

Criterion	Disposition	Owner / evidence
Episode evidence, native outcomes and inference remain distinct	implemented	Typed assessment owner; no score or Explore writer; actual archived-case execution
Fixed eligibility, unknowns, sample scope and source references	implemented	TypeScript prepare/qualify; omitted, forged, duplicate and changed-input negatives
Full cost accounting and independently calibrated scientific assessment	deferred	Separate analysis usage is implemented; full solver/evaluator cost and independent label calibration remain research acceptance
Authorized fixed-pack registration/discovery and source return	implemented	Real six-source registration; cross-owner/key/path/excerpt negative tests; source changes disable new execution but retain historical captured citations
Existing Goal one-click analysis journey	candidate validated in companion	Packaged Goal UI triggered an actual model run; refresh during execution, cited original readback and reopening the result reused the same attempt. Companion visual approval, publication and installation remain open

Scope And Continuation

This is the producer and authorized pack-provider stage, not a claim that the entire product capability is shipped. The independently testable boundary is fixed source registration → scoped discovery → actual model → qualified artifact/reuse/cancellation/source return. The existing research-history consumer integrates that same owner in a companion change.

The same existing research/product work continues with the frontend trigger/status/cancel/retry, unavailable and historical states, and inline return to original evidence; no new Todo or alternate analysis owner is required. There is no arbitrary renderer path/command interface here. No new analysis service, scheduler, canonical graph or Goal authority is introduced.

Placement: existing built-in benchmark-toolkit; TypeScript owns the shared method and validation, Python owns Codex process/file adaptation. Browser consumers import Node-free shared data types; the original owner re-exports them without a duplicate schema. The six metric names and versioned sidecar are a new bounded shared analysis contract, not aliases for Todo/replan lifecycle states. Future-facing pass: input/key/prompt/schema are re-derived at the I/O boundary, and consumers reuse the same typed reduction rather than a second Python decision owner.

Validation

• Tested revision: c840751d178f622351f19100b001d4ab98197536 (checks performed on its final source tree).
• Run state: finished.
• Input classes: synthetic; authorized_private_read_only.

Check kind	Result	Evidence / limitation
unit / integration	passed	5 typed contract tests; 10 real-file/child-process/CLI tests covering reuse, forged request key, rejected incomplete output, retained failure/retry, concurrent execution and cancellation, cross-owner isolation, source hashes/excerpts, path containment and historical citations, verified old-method result preservation and tamper rejection
unit	passed	39 existing behavior-finding tests; this is adjacent regression coverage, not a causal efficacy test
real_entrypoint	passed	Source CLI with actual Codex 0.162.0-alpha.17.2, requested gpt-6.1-sol/max/fast on an authorized archived Graph episode; v2 completed in 97.83s, 60,623 input / 6,315 output tokens, including 5,178 reasoning tokens. Subsequent same-key CLI readback reused the result without a model call
manual	passed	Original six artifact SHA-256 values and literal excerpts checked through the registration CLI; output citations range-checked. Numeric prerequisite progress coexists with benefit_test=no; source texts and traces are deliberately not published
static	passed	Full control-plane typecheck, focused Ruff, semantic changed-diff advisory and full tracked-tree vocabulary smoke, whitespace and public/private scan
real_backend / product	passed in local companion candidate	Packaged frontend → scoped HTTP → this provider → actual Codex → result/cited original. Execution took 148.39s, 54,066 input / 9,607 output tokens; refreshing mid-run and closing/reopening retained attempt 1. Six sources remain available and unchanged. This UI is outside this PR, with preview/publication/installation still pending. No PostgreSQL/authority changes

Coverage limits: model labels remain explicitly uncalibrated, citation validity is not entailment, this purpose-selected episode does not estimate whole-run proportions or treatment effects, and requested fast tier is not independently observed (served_service_tier=null). Crashed-adapter locks require operator inspection; no PID-based automatic lock stealing. CI is not green: documentation has two missing legacy recovery-source links, the existing history browser test observed HTTP 503, and a Windows runtime test observed startup timeout instead of lock timeout. Independent exact-head review remains pending. No self-merge.

Frontend / Visual Evidence

• UI impact: none in this producer PR. The existing research-history UI is a separate consumer integration with a locally verified end-to-end candidate; visual approval, publication and installation remain outstanding. CLI presence alone is not frontend completion.
• Source data: none in public UI artifacts. No first-screen edits.

Type of Change

• New feature
• Documentation update
• Test update

LoopX Area

• Benchmark boundary (evidence)
• Capability or extension

Technical Direction

S5/S11 evidence-based long-horizon research assessment. Shared-authority migration fixture impact: N/A; this does not mutate or route Goal authority.

Boundary Checklist

• No private state, credentials, raw traces, verifier output, internal links or local machine paths in this diff or PR body.
• Reuses the existing research/product work and owning capability.
• Scope is the independently testable producer stage; remaining product/scientific acceptance is explicit.
• UI impact is marked none for this change.
• Every commit includes DCO sign-off.

[loopx-agent]

Approval conclusion (author-owned PR; GitHub blocks formal self-approval)

APPROVE — 对精确头 c840751d178f622351f19100b001d4ab98197536 的完整 14 路径、+1623/-0 独立评审。结论只覆盖 producer 与固定包 provider 阶段，不认证科学判断、打包前端或合并资格。比较基线 71096c16bfbf12ed4305883b4242028ff1504a6c。

动机

长期研究的复盘者需要区分真正检验任务收益的实验与只完成数值前提的工作。

此前原生分数和诊断可以持续增长，却没有固定样本、未知覆盖和可回查引用；本头增加显式执行的证据包分析，复用同一结果并保留来源变化后的历史。

本轮独立验证了固定六项判断的传输、固定分母、逐条引用与来源校验，以及注册、复用、取消和历史回读；没有验证模型判断的正确率。

不改变 solver、原生评分、Explore 事实、研究预算或执行权；不把分析标签当科学资格。

既有研究历史页面的完整打包交互和独立人工/模型校准仍未完成，继续由现有产品与研究 owner 推进。

先按接受前的 docs/architecture/rfcs/long-horizon-harness-benchmark-research-program-v0.md（71096c16bfbf12ed4305883b4242028ff1504a6c，Evidence-based episode evaluation）判断设计，再看实现。Native outcome：原生结果权威保持；Information：固定样本、未知与引用分开；Cost and coverage：分析用量单列且不补造总体比例；Qualification starts：独立人工编码与 held-out 校准 deferred；Product presentation：既有产品 companion deferred。新增 reference 文档是接口说明，不作为证明自身正确的独立规格。

改动思路

保留现有 benchmark-toolkit：TypeScript 只解释分析合同，Python 只适配本地进程与文件；来源绑定侧车不成为第二个研究图或评分权威。

本 PR 的可交付边界是固定包注册、按 owner 发现、显式分析、资格化结果、同键复用及原文返回；前端与科学验收保留在既有后续工作。

最强反对理由是：可重复地生成六个数字，仍可能把前提工作误称任务收益，或把某个精选案例冒充整段研究。这里没有新总分或自动决策；benefit_test 明确区分 prerequisite 与原目标效用测试，redundancy=yes 反而表示问题，结果固定标为 uncalibrated_model。引用合法也只证明给定行存在。允许这个有界前置阶段，不等于让 CLI 替代普通用户完整前端；后者仍由现有 owner 交付。

触发	行为与边界
preview / discover	读取并准备，不启动模型
显式 execute	按完整 input/config/rubric 重新派生 key，只启动持锁的本地 child
同键成功复用	重新校验原输出，返回原结果和原成本，不新增调用
来源或方法变化	停止新执行；按原 input/receipt/digest 保留历史结果和引用

这是源码推导与已执行反例的文字表，不把流程图或作者报告当实验资格。

具体改动

1. parseTrajectoryInput / qualifyTrajectoryAssessment 负责唯一 source/episode、预先固定的 eligible pairs、完整输出和行范围，分母不由模型选择。空资格分母为 null，未知仍留在样本中；非穷尽且总体未知的样本不能声称总体率。
2. execute 在 I/O 边界重新生成 prompt/schema/key，不相信调用方塞入的 prepared prompt。真实独立子进程 probe 验证了 ephemeral、忽略配置/规则、禁 shell/web 的 argv 与重派生；该 synthetic child 只验证进程和合同，不验证真实模型推理质量。
3. inspect_pack / read_citation 复用已有文件锁和 typed owner，核对实际文件摘要、literal excerpt、路径 containment、owner/pack/key。新注册 3 个案例不会自动分析或继承旧 key；显式旧 ID 在默认选择变化后仍读回同一结果。方法升级只读原证明，不以新 rubric 改判旧结果。

独立验证：5 个 TS 测试、10 个真实文件/子进程/CLI 测试通过；完整 control-plane typecheck、Ruff、whitespace、changed-diff advisory 后的全树 semantic smoke 通过。39 个原 behavior-finding 用例在不可变 base/head 均通过。额外六项独立 oracle 得到 benefit_test=no、redundancy=yes、其余 unknown；来源变化阻止调用但历史引用可回读，恢复原来源后不新增调用。这些是合同与状态证据，不是标签有效性实验。

对主干的风险

共享 dispatcher 只有显式新命令分支，effect runtime 只有 lazy 的 prepare/qualify 注册；旧指令、调度、Todo 与评分入口没有改写。默认关闭不仅靠文档：preview 不创建 job，旧路径两头同样39通过，复用与跨 scope 反例的进程计数保持不变。新六项词表是 benchmark-toolkit 的有界共享分析语义，由一个 browser-safe TS 定义拥有，不套用 Todo 生命周期；advisory 实际发现这项新增，完整 smoke 不等于词义正确或动态路径全覆盖。

execution.lock 的崩溃遗留仍需 host 检查，未实现自动偷锁；取消只处理 adapter 自己的 child。真实 Codex 账户执行、模型 entailment/误判率、独立人工校准、整个研究成本和 packaged UI 在本轮未独立资格化。作者及 companion 的真实模型声明未被继承为本评审证据。服务层仍须用现有认证/来源 audience 限定 runtime/Goal，provider 不是认证服务。遵循当前 scoped wait_for_ci=false，没有查询或等待 CI，也没有启动 benchmark 或付费推理。

我的整体评价

保留现有 benchmark-toolkit：TypeScript 只解释分析合同，Python 只适配本地进程与文件；来源绑定侧车不成为第二个研究图或评分权威。

本 PR 的可交付边界是固定包注册、按 owner 发现、显式分析、资格化结果、同键复用及原文返回；前端与科学验收保留在既有后续工作。

本轮没有发现 producer 边界内的阻塞缺陷。相关 future-facing pass 已体现为 Node-free 共享 types、单一 TS 资格判断、输入重派生和历史证明复用；没有理由另加通用分析服务或移入 solver lifecycle。审批结论不关闭父研究/产品验收，不授予 merge、admin bypass 或对旧评审的处置权；本次不执行合并。

English verdict: APPROVE - The exact head delivers a source-bound, opt-in assessment producer with independently verified transport, scope, fixed-denominator, reuse and historical-readback invariants. Model validity, scientific calibration and the packaged frontend remain explicitly separate.

Reviewer: model_agent; model=gpt-6.1-sol; provider=OpenAI; reasoning_effort=max; declaration_source=runtime_reported.