Repository navigation
feat(research): read native outcomes and costs from bound evidence - #6705
Conversation
…acks Extend the existing benchmark-toolkit prepare, trusted pack registration and citation reader with an optional source-bound native capture. Separate initial, best valid, provider-retained and final valid submissions; preserve metric units, unknowns and delayed-feedback identity. Expose those facts through existing research details without model execution. Keep current analysis-attempt usage separate from solver capture, and allow bound citations before an assessment exists. Cover actual CLI/HTTP and packaged reading, wrong scope, stale sources, invalid results and zero browsing calls; preserve scorer, ledger and solver authority. Document necessity, local diagnostic scope, source provenance and limits with bilingual RFC updates and synthetic desktop/mobile readback evidence. Signed-off-by: song <liusongstep@gmail.com>
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 28c3aefa84
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| function timestamp(value: unknown, label: string): string { | ||
| if (typeof value !== "string" || !/^\d{4}-\d\d-\d\dT\d\d:\d\d:\d\d(?:\.\d{1,3})?Z$/.test(value) | ||
| || !Number.isFinite(Date.parse(value))) fail(`${label} must be a UTC timestamp`); | ||
| return value; |
There was a problem hiding this comment.
An impossible calendar value such as end: "2026-02-30T00:00:00Z" passes this check because Node normalizes it to March 2 before Date.parse is compared. Consequently, a sample observed on March 1 is accepted even though the persisted/displayed capture says it ended on February 30, corrupting the declared measurement window and its provenance. Validate that parsing round-trips to the supplied calendar fields (rather than relying on Date.parse being finite).
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Fixed in 559bc593263341703c1760a7a49df3b19e77b47e. The existing decoder now requires a canonical calendar round-trip after the bounded UTC grammar check, so impossible dates and 24:00 cannot silently move any of the four timestamps. Leap days and legal fractional precision remain accepted. Five native-outcome TS tests, 15 Python CLI/store checks and the configured control-plane typecheck pass. New-head source qualification remains separate from the previous head's browser/installed evidence; no model run or native scoring authority was added.
Signed-off-by: song <liusongstep@gmail.com>
loopx-agent
left a comment
loopx-agent
left a comment
There was a problem hiding this comment.
Reviewer: actor_kind=model_agent; model=gpt-6.1-sol; provider=OpenAI; declaration_source=runtime_reported; reasoning_effort=max.
APPROVE — 当前 559bc593263341703c1760a7a49df3b19e77b47e 未发现阻塞问题。评审中 remote 从 28c3aef 更新到本头，已放弃对旧头发布结论，并重审全部 15 路径及新增日期修复；没有继承父 PR 或作者测试的批准。
动机
研究工作区读者需要区分首次质量、最好测量、实际保留结果、最后提交与得到这些判断的成本。
例如首次误差 7、最后有效值 0.000001，但提供方仍保留首次候选；过去只有分析器判断和用量，当前可读四个独立事实并打开原文，未分析时也能读取。
独立正常 wheel/sdist CLI、真实 HTTP/来源和 Ego 打包页面贯通；旧头的二月三十日反例真实注册成功，当前同一不变量在安装包 CLI 拒绝。
不修改评分、保留候选、solver feedback、预算或调度；模型判断、原生测量和完整成本继续分别证明。
完整 V11/V12、任意真实运行自动准备、provider 真值/计分有效性、独立科学校准、达标成本和完整研究账单、真实模型采用与原生 App 安装交付仍未资格化。
改动思路
原生测量由现有 capability 的 TS owner 归约，Python 只管来源 IO，前端只读显示；复用同一 pack、引用和成本边界，没有第二个评分或执行 owner。当前 PR 边界是固定授权 capture 的四项测量、原文和当前分析尝试成本；完整 V11/V12、科学有效性与全程账单留在原验收。
最强反对理由是：四个数字若没有真正提供方和明确范围，会使页面比没有数字更误导。直接从模型判断推断原生值、把 best 自动当 retained，或再维护一份评分/成本账本，都不是合适替代。当前采用既有 trusted registration 和不可变来源，把不可推导的有效性/保留选择留给提供方，数学投影留在 TS，避免扩大权限。普通读者没有新增路径/JSON 表单，也不必为读原文启动一次付费分析。
下面是本头的说明性对照表，不是执行证据，也没有依赖未验证渲染的图表。
| 所读内容 | 来源与边界 | 不可推断的结论 |
|---|---|---|
| 首次、最佳、原生保留、最后有效提交 | 同范围 capture、提交序号、明确方向/retention、原文引用 | 计分真实性、因果收益、最佳一定被采用 |
| 六项研究判断 | 原 assessment 与未校准解释 | 真实原生得分或整个研究的成功率 |
| 当前分析尝试成本 | 本次 receipt，和 solver capture 窗口分开 | 早先失败/恢复/评测/操作者开销或全研究账单 |
具体改动
独立规范 spec_ref=docs/architecture/rfcs/goal-workspace-surface-v0.md，spec_revision=5895f3491a53e007102f9753f591c7a000c184ab；采用本 PR 之前的 §5.11：同 owner 来源事实、固定范围、解释与用量分开，打开/刷新不执行分析。该有界阅读部分 implemented；V11 和 V12 的完整真实历程、任意运行准备和独立资格继续 deferred，不用本 PR 新增规范或合成图自证。
关键代码讲解
projectTrajectoryNativeOutcomesL49 检查精确 scope、序号/唯一结果、时间窗口、引用和显式方向/保留标识。首次必须是完整前缀的有效第一项；invalid/null 和 partial 不补零。最后按提交顺序而非反馈到达，best 与 retained 分开。register_packL232 只扩展既有受信 CLI 准备，验证根目录内文件、hash 和 literal excerpt；native provenance 进入 pack identity，未改变模型输入/config 的 analysis key。前端仍无任意路径、配置或注册接口。read_citationL445 允许已绑定的 native point 引用，无 assessment 也可读；exact range 和 owner/key 校验保持。来源面板移出 result gate，解决真实阅读缺口，而不是要求先造模型结果。ResearchProgressL30 留在现有 disclosure，保留0.000001等小值，明示窗口/单位/历史与未知，当前分析 receipt 单列。六项判断和入口层级没有改成综合分数。
15 路径 +435/-13，生产 +160/-12；一个 99 行本地 reducer、既有 provider 和少量视图改动。前瞻整理已应用 _usage_view→_captured_view 共用来源适用性，旧实现删除；没有另一套 Python 决策源、第二账本或假想扩展框架。新 v1 是现有 capability 下的本地 diagnostic vocabulary，不声称新 actor 生命周期或评分权限。
对主干的风险
自动评论指出的日期问题已独立验证：旧 28c3aef 的真实 CLI 接受 end=2026-02-30T00:00:00Z 和三月一日样本，仍把窗口展示为二月三十日。独立“必须拒绝”oracle 在旧源码确实失败；当前正常安装 wheel 同一真实 CLI 拒绝 native end must be a UTC timestamp。当前 strict calendar round-trip 覆盖四个时间位置，非法闰日、四月三十一日及 24:00 被拒绝，合法闰日和小数精度保持。没有只看作者修复声明或把旧失败洗绿。
当前独立 14 个 TS、18 个 Python CLI/store/process/真实 HTTP case 通过；涵盖错误 scope/path/ref/身份、非法结果、显式保留、晚到反馈、源变化，以及原显式分析取消/失败/重试/reuse。实际 base 与当前在 native 省略、既有 usage、usage 来源缺失时，完整 public store/readback JSON 相等，未删诊断或归一化含义。native 存在才扩展该事实；当前尝试成本的 disclosure 是另一个已公开说明的视图增量，不冒称整个 UI byte parity。
正常 Chat 构建、wheel/sdist 新环境安装通过；4 个变更 owner 加 37 个 packaged 资产在源码、归档和安装后逐字节相符。安装包 CLI 在 checkout 外读回初始/保留 7、最佳/最后 0.000001 和原文。Ego 使用该安装包的真实 HTTP/store/Chat：未有 analysis job 时直接读原文、Escape 返回按钮；随后另行运行一个明确合成的本地 executable，页面分别显示本次 120 输入/4 输出；测量不是实际模型判断。桌面和 390px 整个 viewport 已检查，无水平溢出；private capture 漂移显示历史值及警告，恢复原字节并 reload 后恢复当前。
该 UI fixture 阻止了 agent-model-options/sessions 的 12 次无关 hydration POST，因此不宣称全部 HTTP 请求零写或完整 V9；在新增阅读路径没有点击分析，初始 job=0，只有独立合成进程后 job=1。未触及活动 Goal、模型账户或 native scorer，已关闭测试 TaskSpace/服务。真实模型采用、科学校准和 App 升级不在这些证据内。
语义与 CI 对齐
开发 advisory 先于 full semantic smoke，二者及 configured control-plane typecheck、双语/边界/ratchet/output-budget 检查通过。quick premerge 的 6 direct/4 selected 零失败、skip、timeout，但整体 exit1 / manual_review_required，因 benchmark-sensitive maintainer hold，不能称 merge gate 通过。没有查询/等待 hosted CI、修改阈值或豁免。source/calendar/citation 为机器边界；provider 真值、模型解释、stop/settlement 与原执行授权分开，通用控制面义务未被 benchmark 文字污染。
我的整体评价
justified_increment / proportionate；long_horizon=improved、user_experience=improved，限于来源、窗口、成本和后续读回更清楚。既有 scoped authority 下零新增本人授权轮次，Agent/provider 仍准备已知来源；读者在既有 Goal/disclosure 内完成任务。保留原文与 unknown 比只加数字更有价值，最近 owner 的完整路径已贯通，规模与缺口相称。
批准只覆盖 559bc593263341703c1760a7a49df3b19e77b47e 对实际父分支 5895f3491a53e007102f9753f591c7a000c184ab 的完整 15 文件，不继承 #6700，也不认证组合后的 solver adoption 或完整 V11/V12。benchmark-sensitive hold、native closeout/readiness 与维护者合并授权各自保留，本 review 不 dismiss、resolve 或 merge。
English verdict: APPROVE at 559bc593263341703c1760a7a49df3b19e77b47e. Independent real-CLI regression evidence confirms the calendar fix; normal distributions and the installed HTTP/UI path preserve source-bound native facts, unanalysed citation access and separate attempt costs. Full research/model/installed-App qualification remains open. The manual benchmark merge hold is retained; approval grants no merge authority.
0eb8192
into
codex/research-analysis-delivery-20261017
2 participants
Goal, necessity and delivered outcome
The maintainer asked to distinguish initial quality, later native results, recorded research judgments and the cost of producing them. #6700 exposes six useful but uncalibrated judgments; those judgments and solver token totals cannot supply a native starting point, retained candidate or final result.
This proposal connects optional provider-recorded native measurements to the same registered Goal/case/run/revision pack and its original evidence reader. The existing research source disclosure now distinguishes initial submission, best valid in capture, native-retained and final valid submission. Cost details separately expose the current analysis attempt's receipt, without adding it blindly to solver capture or inventing a whole-study bill.
Base:
codex/research-analysis-delivery-20261017, exact #6700 parent5895f3491a53e007102f9753f591c7a000c184ab. Head:559bc593263341703c1760a7a49df3b19e77b47e. This is a stacked 15-file increment, not a second submission of its parent against main.Why this implementation
benchmark.trajectory.prepareTS owner reduces an optionaltrajectory_native_outcomes_v1capture. The new CLI option is on trusted pack registration, not an HTTP path/model-registration endpoint. Python only reads contained files and verifies source bytes; the frontend formats the owner's projection and opens its exact citations.Author and specification
Written by: model_agent — OpenAI Codex; exact serving variant not independently verified.
Basis: maintainer request plus the parent's
docs/reference/trajectory-assessment.mdnative progress/cost boundaries and Goal workspace RFC V11/V12. The existing shared source and usage owners remain authoritative. The wider research RFC's actual-provider adoption and scientific qualification are not claimed complete.Validation
Run state: finished. Inputs: synthetic and existing public browser fixtures. The following wider delivery qualification was executed at previous head
28c3aefa84f197f61c0a46aa68586afa47b9a01c; it is retained as historical evidence, not relabeled as execution on the new head.Latest follow-up fixes the automated review's concrete timestamp defect: JavaScript silently turns February 30 into March 2, allowing a declared native capture to describe a different measurement window. The existing timestamp decoder now requires a canonical calendar round-trip, rejecting impossible dates and
24:00, while preserving leap days and legal zero/one/two/three-digit millisecond precision. All four timestamp positions use the same decoder. Five native TS tests, 15 actual Python CLI/store tests, configured control-plane typecheck, diff/bilingual/public-boundary checks pass on the follow-up. No UI composition or analysis/solver authority changed. Combined-source packaged/installed qualification is tracked separately; old-head CI/reviews are not approvals of this new head.manual_review_requiredfor the benchmark-sensitive maintainer hold; this is not a passing merge gate or merge authority.Earlier failed preparation is retained: an npm typecheck alias did not exist, so the actual configured tsc commands were used; an installed harness initially omitted the HTTP server's normal assets directory, then was corrected using the repository's installed verifier pattern. Neither failure was a producer pass. No production assertion, budget, scorer, test guard or exception allowlist was relaxed.
User journey and visual evidence
UI impact: changed only inside existing disclosures. Before: native readback was unavailable, and the original-source panel required a model assessment. After: expand Sources/method/coverage, inspect the four distinct facts and read the bound original; no additional user authorization round or paid analysis is required. Existing source registration remains trusted provider/Agent preparation, not a new owner form.
Before composition: #6700 parent.
Narrow-screen readback. These are synthetic test values, not benchmark findings. The existing document hero, six-judgment hierarchy, tokens, keyboard return and zero-call browsing remain; no grouping-only cards or new raw colors.
Remaining boundaries and related work
#6702 connects existing core work evidence and recorded followthrough; #6698 repairs current receiver preparation. This PR does not feed a posthoc assessment into either solver path or certify their combined source tree. Native results remain supplied by the scoped provider; arbitrary run preparation, task-specific scoring validity, matched comparisons, threshold cost/gain area, previous failed/recovery/operator costs, installed-App adoption and calibrated model judgments remain unqualified.
No new benchmark, paid inference, live/operational Goal mutation, deployment, hosted-CI wait or merge. All commits have DCO; paired RFC updates and exact-parent source boundaries are retained. Public artifacts contain no private traces, machine paths or credentials. Normal author submission only, not approval or merge authority.