Repository navigation
fix(benchmark): make warm preparation readback revision-consistent - #6706
Conversation
Bracket the actual guard with the exact preparation Todo authority revision. Refuse interleaved completion or transfer, and expose the observed Todo, revision and current replan identity in the existing context receipt. Preserve phase deadlines, normal worker admission and owed first review. Cover real File/SQLite interleavings and the actual native warm resume entry without paid inference; retain the parent proposal's other recovery boundaries. Signed-off-by: song <liusongstep@gmail.com>
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
…eady Signed-off-by: song <liusongstep@gmail.com>
loopx-agent
left a comment
loopx-agent
left a comment
There was a problem hiding this comment.
Reviewer: model_agent; model=gpt-6.1-sol; provider=OpenAI; runtime_reported; reasoning_effort=max
完整评审 6706@79c00981c72491851034d6f4e1cf5b471e6ac714，实际父分支 codex/checkpoint-continuation-intent-20261017@d72cd3dcdbb9450a5d37b51aaf4c989ff7c8d2f9。APPROVE 仅针对当前五文件的有界修复，不继承或代替父 PR #6698 的整体结论。
动机
显式恢复已保留研究状态的操作者，需要知道目标准备任务现在是否仍属于当前 Agent。
准备任务在 quota 返回后完成或转交，旧版仍通过；相同 File/SQLite 真实 CLI 交错下，当前头重新读状态与修订并拒绝 ready，保留已发生的原状态变更。
独立四例拒绝 oracle 在父修订全部 DID NOT RAISE、当前头全部通过；54 项当前相关用例及一个真实 Codex fork/resume 准备入口通过。
不续预算、不改评分/冷却、不重放历史权威、不新增模型权限或通用恢复 UI。
完整安装版 SForge 恢复、再次捕获、任意采用后重试及真实模型/科研净收益仍未资格化；当前原生 Codex 用的是确定性本地响应。
本 PR 的完整边界是准备状态的一致读回、原创建操作的准确重放、拒绝前不改 cadence，以及移除新增的历史截止解释段；不认证父提案整体或科研采用。
改动思路
复用既有规范 Todo create/receipt 与 quota owner，在 benchmark provider 的 Python IO 组合边界夹读真实修订；不建立第二个就绪账本、租约或 TypeScript 决策镜像。
本 PR 的完整边界是准备状态的一致读回、原创建操作的准确重放、拒绝前不改 cadence，以及移除新增的历史截止解释段；不认证父提案整体或科研采用。
保留现状会留下已经复现的假 ready；单改提示或重新发起操作既不能固定当前事实，也可能复制准备任务。当前选择同一 create owner 的原始回执与真实修订，继续由原 owner 校验完整请求摘要；不把“文件存在”“历史创建成功”或模型收到说明当作当前许可。全局修订在无关并发编辑时可能保守拒绝，这比用过时 open 快照继续安全，但不是对长期可用率的证明。
具体改动
完整净提案五路径、+202/-28，其中生产改动集中在已有 warm provider；README 双语说明、共享测试 fixture 的 hard/soft mode 参数及真实原生回归与其同行交付。
关键代码讲解
_preparation_todo（benchmark/runtime/sforge_warm_start.py:283）先读准确的checkpoint-preparationcreate receipt，取原 intent 的 title，识别当前短任务或一个已接受旧长任务，再交回原 create owner 验证整个请求。短/长恢复均不创建第二个 Todo；不同请求拒绝。SHA 是本地历史 intent 兼容，不是用文字猜权限。prepare_context（311 行）保留真实 native/context owner、当前 acquire/adopt/release 顺序，将 readiness 放在 cadence=1 写入之前。拒绝时注册配置不变；后续配置答复不确定仍须原操作恢复，不能盲目回滚欠下的 review。_read_preparation（361 行）用规范 revision 夹住实际 quota guard，再核相同 Todo 的 open/当前 owner，返回修订、selected Todo 与 replan 身份。它是一致观测，之后 worker 仍须 fresh admission；原 phase deadline 在前后检查，没有新时长。
实现前固定的规格为 benchmark/README.md@d72cd3dcdbb9450a5d37b51aaf4c989ff7c8d2f9，并保留 stride RFC §7.7 的完整继续与科研边界。explicit-warm-current-work、original-operation-recovery、original-allowance-and-stops 在当前 helper/真实 CLI 反例中实现；§7.7-full-continuation 仍 deferred。当前 PR 文档不是自证整个父提案已通过。
对主干的风险
最强反例用同一个独立四例 oracle 在不可变父修订及本头执行：File/SQLite × quota 返回后完成/转交。父修订四例均因未拒绝而失败，当前四例均通过，实际 canonical done/foreign claim 原样保留。当前54项相关用例覆盖稳态重复、guard 前后竞态、gate/expiry、短长原请求及外来请求、拒绝前 cadence 不变。真实 Codex fork/resume 准备入口另有1 pass/40 deselected，使用本地确定性 Responses peer；不调用付费模型，不证明研究理解或完整安装版 SForge 运行。
这次没有把额外探索当全部绿色：ordinary cold CLI 在 base/head 都缺少 sforge；扩大到原有 task-entry 模块时当前35 pass/22 fail，原基线独立环境为35 pass/22 skip。差异来自当前环境存在 Harbor namespace 但缺少所需 agents 包，不是相同依赖条件下的比较；再用同一解释器/依赖分别载入真实 base/head，代表性失败都在未改的 harbor.agents 导入处复现。初次把两份环境当相同的归因断言失败已保留并纠正。不把空 stdout 相等当 CLI parity，不把 skipped 计作通过。七个 core/冷入口 owner 在 base/head 逐字相同，受影响 warm 实际链已执行；完整两个研究 adapter 的环境资格继续开放。
语义与 CI 对齐
原任务中的 evidence/no-submit/settlement/first-replan 要求保留；新接收端不再获得解释旧 Host deadline 的段落，旧已接受长请求只为原操作恢复保留。当前 Goal、题目时限、cooldown、gate 和 stop 仍约束执行。preparation_readback 是局部派生投影；共享 canonical 状态仍归既有 typed owner，没有 substring/prose classifier 或新的通用义务。
Ruff、diff、advisory 后的完整 semantic smoke 均通过。advisory 不扫描 benchmark provider、输出零项不证明无语义变化；判断还依赖逐条指导语义与真实 IO 对照。当前 native quick premerge 的5 direct+4 selected 全部执行、零失败/跳过/超时，但整体 exit1 的 benchmark-sensitive 人工 hold 保留，不能称绿色 merge gate。未查询、轮询或等待 CI。初次独立 probe 的导入错误只导致 collection failure，修正测试路径后才得到上述父红/头绿；未为此修改产品。
我的整体评价
复用既有规范 Todo create/receipt 与 quota owner，在 benchmark provider 的 Python IO 组合边界夹读真实修订；不建立第二个就绪账本、租约或 TypeScript 决策镜像。
本 PR 的完整边界是准备状态的一致读回、原创建操作的准确重放、拒绝前不改 cadence，以及移除新增的历史截止解释段；不认证父提案整体或科研采用。
long_horizon=improved：避免过时准备快照借后来 review 继续，原创建身份仍可恢复，后续工作须重新准入。 user_experience=improved：有既有授权时继续走原命令，不要求用户换 ID、手改路径或再确认；错误保留可检查的接收端。 相关 future-facing pass 以现有 receipt/title/revision 收敛恢复知识，并删除多余新提示；不引入泛化框架。兼容保留只因已经接受的持久操作不能换摘要，原 create owner 仍拒绝其它主体或请求。
当前完整五路径可以 APPROVE；原父提案、完整安装恢复/再次捕获、实际科研效用及环境/人工 hold 继续分别判断。公开评审、closeout/readiness 也不授予 merge、dismissal、升级或运行研究的权限。
English verdict: APPROVE — 79c0098. The complete five-path follow-up reuses canonical receipt/create and revision facts to reject stale preparation, preserve exact original operation replay and defer cadence mutation until readback succeeds. The same real File/SQLite race oracle fails all four cases at the parent and passes at this head;54 focused cases and one real Codex warm entry with deterministic local inference pass. Wider adapter prerequisites and the benchmark manual hold remain explicit; full recovery and research utility are unqualified.
f7c27b1
into
codex/checkpoint-continuation-intent-20261017
3 participants
Goal, necessity and delivered outcome
The maintainer requested reliable continuation of retained research, with inspectable current work rather than a restored-history or readiness boolean alone. This follow-up targets #6698's concrete inconsistent-read defect: a preparation Todo could finish or transfer after its first read, while an unrelated current review still let setup report
preparation_actionable=true.Base:
codex/checkpoint-continuation-intent-20261017, exact parentd72cd3dcdbb9450a5d37b51aaf4c989ff7c8d2f9. Proposed head:79c00981c72491851034d6f4e1cf5b471e6ac714. This is a five-path follow-up, not a second copy of the parent's diff against main.Why this repair
The existing canonical Todo reader already supplies an authority revision. Bracket the actual quota/guard with that revision and re-read the exact Todo's open/owner state. If canonical state changed, refuse readiness and preserve the receiver for inspection. Another ledger, inferred permission from conversation text or retry budget would not fix inconsistent observations.
The existing context receipt now returns
preparation_readback: observed authority revision, preparation/selected Todo and current replan identity. The operator can inspect the basis of readiness. It is an observation, not an execution lease; ordinary worker admission, first current replan, original phase and stop rules still apply.Python remains this benchmark provider's IO composition; existing TypeScript Todo/quota/replan owners retain decisions. No new countdown, grant, score rule, capture format or generic forced continuation. Native history/candidate/retained evidence are not rewritten.
The latest repair also addresses the parent's three remaining concerns:
checkpoint-preparationchanged the canonical request digest and stranded an already-created receiver. Read the original canonical create receipt, recognize the original or historical longer preparation task, then replay its exact intent through the existing create owner, which still checks the entire request digest. The owner's priority-decoratedtextis not the original intent; use its existingtitle. Do not mint a second operation/Todo or infer current permission from historical acceptance. A different request still rejects.configure-goal. A rejected preparation no longer commits cadence one. Do not blindly roll back an ambiguous write, which could erase an owed review; fresh worker admission and the joint continuation gate remain required.Author and specification
Written by: model_agent — OpenAI Codex; exact serving variant not independently verified.
Implemented against the maintainer's demonstrated readiness invariant and the parent revision's
benchmark/README.mdexplicit warm worker contract / stride RFC §7.7. Current preparation/review must be available under original authority; imported history does not grant it. Real research/model benefit and the wider installed restoration journey are explicitly outside this bounded repair.Validation
Run state: finished. Inputs: synthetic. Latest-head checks below are distinct from the earlier wider qualification.
DID NOT RAISE); current production helper rejects it.2600db4manual_review_requiredfor the benchmark-sensitive hold. This is not latest-head or merge qualification. Normal npm setup had corrected a missing TypeScript dependency.A broader latest attempt had 61 pass / 6 explicit skips / 43 deselected, then failed an obsolete assertion requiring the removed paragraph. Updating that assertion to require the original task and forbid the removed paragraph produced the decisive native pass above. Skips and the earlier zero-selection attempt are not passing coverage. No new benchmark, paid inference, deployment, hosted CI wait or merge was performed.
Scope and remaining parent concerns
This fixes the readback, partial-create replay and rejected-readiness cadence defects, and removes the disputed paragraph. It does not dismiss reviews or certify the whole #6698 proposal. Cross-version replay here covers partial creation only: arbitrary retries after context adoption/worker execution retain their original revision, lease and operation recovery rules. Full installed SForge restoration/recapture and actual model/research adoption remain separately unqualified. Ambiguous configuration writes require inspection rather than an unconditional rollback.
Related #6702 reconstructs research evidence/followthrough in core; #6700 owns the user-facing analysis reader. Neither makes this receiver's model adoption or scientific utility proven.
UI impact: none; existing operator context receipt and documentation only. No authority-store migration or provider promotion. Public material excludes private traces/paths; all commits have DCO. Normal author submission only; no approval or merge authorization claimed.