Skip to content

fix(benchmark): keep partial native scores out of completion markers - #7072

Merged
loopx-agent merged 1 commit into
mainfrom
codex/edgebench-partial-results
Oct 22, 2026
Merged

loopx-agent merged 1 commit into
mainfrom
codex/edgebench-partial-results

Conversation

@loopx-agent

@loopx-agent loopx-agent commented Oct 22, 2026 •

Copy link
Copy Markdown
Collaborator

A failed native Goal could retain a score in final_result.json, which the existing SForge scanner and page interpret as a completed attempt. The adapter now atomically preserves the full score and native/runtime outcome in partial_result.json for nonterminal native outcomes, keeping the completion marker for the existing handled terminal path. This repairs the reproduced post-merge finding in #7061.

The existing CLI regression now covers failed, usage-limited, cancelled, missing, complete, blocked, paused and budget-exhausted outcomes through ordinary SForge discovery and its actual HTTP/Jinja page. Documentation agrees on new-attempt source pinning, unchanged frozen histories and rollback. The writer remains with the existing adapter; no new classifier, scorer or provider abstraction is added.

Validation at this branch commit:

  • Related native worker, adapter, feedback and report suites: 489 passed, 11 skipped. The first invocation lacked the existing tests/ import path; the corrected invocation passed without source changes.
  • Same synthetic scoring input, pinned SForge source 193cecda05661380575b16c9fcc88a8c16ece9f3: all four abnormal cases changed from 1/1 done to 0/1 done / aborted, preserving their full scores; healthy control stayed 1/1 done. Sixteen complete non-native result payload pairs stayed equal.
  • Regressions run against base 722bcc582043bdc2ab60746e4b0dc7a57130b716: four abnormal cases fail and four handled-stop controls pass; all eight pass with the repair.
  • Ruff, diff hygiene and RFC bilingual staged gate pass. Public-boundary scans of all four files are clean; the broader global loopx check still reports unrelated legacy Goal authority problems.
  • Repository standard premerge: four direct checks, Python compile, ten catalog canaries, eight risk-profile smokes and the public-boundary check pass. Its overall manual_review_required is solely benchmark_sensitive; the owner-authorized exact-head self-review is published, with native merge readiness still required. CI is not queried or waited for under this managed Goal policy.

The explicit Docker cases remain unqualified on this host: the existing provider's stream close raises OSError: ENOTCONN, independently reproduced without LoopX. No Linux Docker pass, installed rollout, live inference, benchmark scoring/submission or historical reclassification is claimed. This patch changes only result-file destination after collection, exercised with the real pinned consumer. Existing Docker assertions are preserved and adapted to that destination.

Bounded refactoring review: the existing writer, score serializer and atomic replace already own this boundary; adding a new abstraction would not improve it. Owner-authorized self-review and native exact-head merge readiness remain required before the authorized merge.

Signed-off-by: LoopX Agent <337587101+loopx-agent@users.noreply.github.com>

@loopx-agent loopx-agent left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewer: actor_kind=model_agent; model=gpt-6.1-sol; provider=OpenAI; declaration_source=runtime_reported; reasoning_effort=max

Approval conclusion (author-owned PR; GitHub blocks formal self-approval)

精确提交 e41becf868425bd24a905a10e5202bf78a6665f5,不可变基线 722bcc582043bdc2ab60746e4b0dc7a57130b716。本记录为作者自评,不冒充独立维护者或 GitHub 正式批准。

动机

运行 native Goal 并通过既有 SForge 页面查看结果的维护者,需要同时保留失败结论与已取得的分数。

同一失败分数输入在基线显示绿色 done、1/1 done;修复后显示 aborted、0/1 done,完整评分仍保存在部分结果文件中。

真实消费者已不再把失败运行计为完成,健康完成对照及普通非 native 结果保持原行为。

本执行仍无 Linux Docker 全链路通过和已安装采用证明;既有外部 stream.close 故障及无关 canonical fixture 失败保留。

不更改评分、任务语义、native continuation 或外部执行权限,不改写冻结历史,也不把合成验证称为安装、Linux Docker 或长程模型收益资格。

依据是 #7061 的已复现审计、修复前 README 的失败不产生完成文件约定,以及固定 SForge 193cecda05661380575b16c9fcc88a8c16ece9f3 的真实消费者。未把本 PR 新增说明或作者测试结论当成独立规范。该修复减少错误完成数对后续判断的干扰;没有测量长程模型净收益。

改动思路

保留现有结果写入函数、RunResult 序列化及原子 replace,只按原有 terminal 状态选择产物名;没有第二套状态机或消费者兼容层。

本次交付界限是既有结果生产者与固定消费者之间的兼容修复;原状态分类、评分来源、归档和容器执行继续由原所有者负责。

不修改外部 SForge 是有依据的范围选择:它早已依赖最终文件判断完成。额外增加一个它不读取的标签不能修复问题;删除整个失败分数又会丢掉原需求。由既有投影所有者选择部分结果文件,保留完整序列化与原子写入,就是更小且可回滚的修复。相关 #6902 提交恢复仍归其原路径,本次不复制它的机制。

具体改动

完整四文件 +66/-20:一个生产函数 +5/-3,其余为原有回归与两处说明;无生成物、私有运行数据、状态框架或新 CLI。

关键代码讲解

  1. _write_native_final_result 保留原 eligibility、完整 RunResult.to_dict()、原生标签和 best-only 字段。第101行仅按既有 terminal 选择 final_result.json,其余已经具备写入条件的 native 结果进入 partial_result.json,第102至104行仍以临时文件原子替换。运行前失败和零运行时的旧边界不扩展。
  2. main 的结果交接 仍先根据实际退出/原生 outcome 判定状态,再将同一结果交给写入函数;runtime-receipt.json 和0/130/1退出码不变。正常新 attempt 使用原有排他建目录,修复不清理或改名旧成绩。
  3. 原 CLI 回归扩展到 failed、usage_limited、cancelled、缺失 outcome、complete、blocked、paused、budget_exhausted;实际执行正常目录发现与原生 HTTP/Jinja 页面,断言异常为 aborted/零完成,正常停止保留 final。另核完整失败 score 字段与原子临时文件消失;既有 Docker 断言保留并适配真实产物名。
  4. README 与 RUNTIME 统一说明部分结果、非完成标记、资格边界、新 pin、新 attempt、冻结历史与回滚;没有压缩掉 continuation、预算或权限义务。

正向路径是:真实终态 -> 原写入 -> 原最终文件 -> 正常 scanner -> 既有页面计完成。异常路径是:原分类器输出失败/取消 -> 同一完整评分写部分文件 -> scanner 无 final -> 页面 aborted/0 done;同一 runtime receipt 和非零退出仍可观察。部分文件的存在不授予评分验收、真实实验或模型调用权限。

逐项规范映射:benchmark/edgebench/README.md 的 completion-marker-compatibility 由异常不产生 final 与实际零完成读回满足;preserve-score-and-outcome 由完整评分字段、部分文件及原生标签读回满足;historical-and-nonnative-isolation 由新 attempt 排他目录、冻结证据保留与16组普通路径对照满足。

对主干的风险

验证与失败归因: 当前头六组相关测试 489通过、11跳过;首条命令遗漏既有 tests/ 导入路径导致5个导入失败,补齐测试调用路径后无需产品修改即可通过,保留原失败记录。Ruff、语义词汇 advisory、双语 RFC 与 diff 检查通过。仓库 standard premerge 的4项直接检查、编译、10项 catalog canary、8项风险 smoke及公开边界检查通过;整体仍返回 manual_review_required,唯一原因是 benchmark_sensitive 的显式维护者审查,不能说成工具全绿。当前具名修复已有维护者自修复/自合并授权,本精确头作者评审记录履行该审查,随后仍须原生 readiness。

反例灵敏度: 在不可变基线上运行这八个新 CLI 用例,四个异常分支失败,四个正常停止对照通过;当前八个都通过。同一外部消费者、同一合成评分、同一真实 HTTP 页面再次验证,四类异常从 done1变0,healthy保持1,完整分数不丢失;16组普通非 native 完整载荷逐项相同。未归一化状态、分数或完成计数。消费者测试用合成 scorer/backend,真实 writer、scanner、server 与模板均实际运行;它不等于真实模型或 Docker 全链路。

环境边界: 既有显式 Docker 三例此前实际执行,在外部 SDK stream.close 报 ENOTCONN;不加载 LoopX、只执行 printf 的原 SForge 后端也复现。相关 provider、backend、worker 因果源码在此前观测、基线及当前头一致;本次按原环境开关跳过,不声称 Linux Docker 已通过。此前独立 base/head 相同 canonical authority fixture 红灯亦保留,本次没有借此修改无关 authority。全局临时 loopx check 仍报告别的旧 Goal authority 问题;本 diff 的隔离 premerge 公开扫描通过。按管理 Goal 策略未查询、轮询或等待远端 CI。

语义对齐与隔离: 文件名是 benchmark 内部派生投影,复用既有 typed outcome,不从报错正文猜类别,不新增通用领域义务。异常默认产物的变化已披露;既有 terminal、非 native、提示词、continuation、权限、评分和冻结证据边界保留。正常 blocked/paused/budget 是合法停止,保持旧 final 行为,不借此声称任务完成或分数验收。风险是旧页面仍不展示完整 partial 内容;操作者可读取全量文件与现有 runtime summary,页面至少不再误报完成。新来源 pin 仅用于新 attempt,回滚同理。

有界重构判断:现有 writer、serializer、atomic replace 已形成足够清晰的 seam,继续抽象会增加维护成本,本次不新增。无需 UI 配置、新存储、外部 provider 修改或历史迁移。

我的整体评价

APPROVE(作者自评结论)。该提案完整修复已审计的结果交接回退,保留失败分数、实际失败读回和原成功/普通路径,范围与问题相称。当前没有未解决的代码阻断项;保留 Docker、旧 fixture、安装与真实研究资格边界。允许合并与检查通过是分开的门:仅在头仍为上述提交且原生 merge-readiness 为 ready 后,使用本次已授予的自合并权限。

English verdict: APPROVE at e41becf868425bd24a905a10e5202bf78a6665f5 (author-owned COMMENTED self-review, not formal GitHub approval). The existing writer preserves full abnormal native scores in partial_result.json while restoring the final marker's consumer-compatible meaning. Matched real scanner/HTTP rendering changes four abnormal runs from done1 to aborted/done0; healthy and 16 ordinary payload pairs remain unchanged. 489 related tests pass with 11 optional skips; native local checks pass with the explicit benchmark manual-review hold recorded. Existing Docker environment and historical fixture limitations remain disclosed; no installed, Linux Docker or live benchmark qualification is claimed. Merge still requires unchanged-head native readiness and owner authorization.

@loopx-agent
loopx-agent merged commit 2babbdb into main Oct 22, 2026
18 of 23 checks passed
@loopx-agent
loopx-agent deleted the codex/edgebench-partial-results branch October 22, 2026 13:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant