Skip to content

fix(benchmark): preserve the original task objective in native Goals - #6646

Closed
loopx-agent wants to merge 1 commit into
mainfrom
codex/native-original-objective-20261016
Closed

loopx-agent wants to merge 1 commit into
mainfrom
codex/native-original-objective-20261016

Conversation

@loopx-agent

Copy link
Copy Markdown
Collaborator

Native non-game trials currently initialize the LoopX Goal with a generic continuation reminder. Although the complete task remains in the startup prompt, structured replan context therefore omits the task’s stated priorities. Initialize new Goals from the original task agent_query, keeping the existing research guidance in the startup prompt and workspace instructions.

This changes the default objective for new native trials. It does not migrate existing Goals or restored checkpoints, infer acceptance criteria from referenced files, or change scoring, submission opportunities, cooldowns, budgets, game mode, blind or best-only prompts. The shared runtime validates that the objective belongs to the supplied native task prompt.

Validation covers the actual adapter CLI handoff, all worker prompt profiles, isolated real Goal bootstrap and typed replan readback with ordered priorities and multiline constraints, invalid objective rejection, task entry and warm-start paths. No live model or grader was launched; improved research decisions or scores remain unproven. Existing experiment trajectories were not modified.

The bounded ownership pass renamed the internal continuation helper to distinguish research guidance from the task-owned objective; no new core decision owner or protocol vocabulary was introduced. This is benchmark-provider plumbing; no frontend/Lark entrypoint or setting is added.

Checks: 380 passed across test_edgebench_prompts.py, test_sforge_runtime.py, test_sforge_warm_start.py and test_task_entry.py; Ruff and git diff --check passed. The semantic advisory found no supported new vocabulary carriers in its scan scope; the full semantic-vocabulary drift smoke exited successfully. Earlier development runs exposed assertions encoding the previous default and two errors in the new test harness; these were corrected before the final run.

Signed-off-by: LoopX Agent <337587101+loopx-agent@users.noreply.github.com>
@loopx-agent

Copy link
Copy Markdown
Collaborator Author

Author follow-up: real restored-context validation found a remaining blocker at afbc0d62b74079fe6c96fbcf7ce74ab82faf065f.

The original query is correctly persisted in the new Goal. However, replan_context_codec._goal_facts.goal_text() passes the entire objective through public_safe_compact_text(). An ordinary task-file or scratch-file path causes that helper to return None, so the typed core_goal reports the objective as missing. Both an inline problem statement with a scratch-file example and a query pointing to a workspace instruction file reproduce this after actual checkpoint adoption from an installed wheel. This is distinct from the earlier private verification-fixture errors, which have been retained and corrected.

The existing 380 passing checks do not cover this input. I am extending the delivery to handle Goal source/readback and public evidence privacy through their appropriate boundaries, with negative coverage for secret-bearing evidence. The original query must remain actionable without silently making public projections disclose private locators or credentials. No live workers were upgraded and no score benefit is claimed. Please hold approval until the revised exact head and installed-path readback are available.

@loopx-agent loopx-agent left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewer: model_agent | gpt-6.1-sol | OpenAI | runtime_reported | effort=max

Request changes conclusion (author-owned PR; GitHub blocks formal self-review)

动机

精确 head:afbc0d62b74079fe6c96fbcf7ce74ab82faf065f。新建 native 非 game 试验的操作者需要在后续 replan 中继续看到原任务及优先级。此前 Goal 使用通用“继续研究”提醒;本 PR 将原始 agent_query 传入 Goal,同时把研究提醒保留在启动 prompt 和工作区 AGENTS。这个方向合理,但当前完整路径仍有 P1 缺口:原题引用绝对任务文件路径时,Goal 保存成功,typed replan 的整个 objective 却变成 null。

验收依据:benchmark/edgebench/README.md 的 Feedback protocol (new-run default: native),以及 docs/architecture/rfcs/hierarchical-agent-stride-control-v0.md 的 7.4;均以改动前 47d0f31482303ffa514a3e45cf34099ce275a069 为准。前者要求原始任务继续作为规划输入,后者保留原 Goal/vision 的方向权威。 本次不要求解析任务文件生成 acceptance,不迁移旧 Goal/checkpoint,也不以此证明模型研究收益或新执行权限。

改动思路

沿用 run.main → SForgeWorker → BenchmarkCodex → bootstrap → replan。provider 层负责把原题和提醒分开;已有 Goal/typed context owner 负责读回。独立反例走了真实启动和读回,失败位于 _goal_facts.goal_text → public_safe_compact_text:普通绝对任务路径导致完整文本被过滤。只把原题写入前置字段,尚未交付后续决策可用的目标。

具体改动

已检查全部六个文件:run.py 选择原题;sforge.py 将 objective 的归属校验从 workspace guidance 改为完整 native prompt;prompts.py 将 helper 更名为 native_research_guidance,提醒本身的继续研究、评分、cooldown 和提交恢复条款不变;两个测试文件更新真实 bootstrap/replan 及 CLI handoff;README 披露新试验默认行为和旧 checkpoint 不迁移。

P1:修正带任务文件路径的原题读回。 位置为 benchmark/edgebench/run.py:235 新传入值进入的真实路径。无 credential-like 内容的合成原题,只要包含普通绝对任务文件引用,两个 heartbeat profile 的 core_goal.objective 都为 null;相同入口的多行优先级任务正常。请在既有 Goal/context owner 内修复任务输入与公开证据输出的边界,保留隐私保护,并加入真实任务文件引用的原生回归;不要用删除路径、改题、推测 acceptance 或关闭过滤替代修复。

对主干的风险

独立重跑现有四模块 380 passed、0 skipped。额外真实 run.main + bootstrap + task-plan + typed replan 探针为 8 passed、2 failed;失败正是两个 heartbeat 模式的绝对文件引用。相同探针在改动前十例都停在通用 objective,说明验证能区分前后变化。现有 Goal 再准备仍保留已捕获目标;blind/best-only、game、native grader、预算和 cooldown 的分支及已有回归已核对。风险 canary4项直接检查通过,19项全部执行:18过、1项继承 DSH advisory;总 canary 仍为 quality_receipt_missing,另有1项人工边界 hold。Ruff 和完整 semantic 检查通过;首次缺 Harbor 的环境失败及私有探针搭建错误单独保留,未当作产品结论。未运行真实模型、grader 或新 benchmark 任务,未查询或等待 CI。

我的整体评价

REQUEST_CHANGES。 原题传输是必要的有界修复,但真实任务文件路径仍会让 replan 丢失目标,不能仅凭380项通过批准。面向下一步的边界检查结论是:复用既有 context owner 做小范围补修足够,无需另建解析器、配置开关或迁移框架;新 head 需完整重验。研究效果、已安装 worker、旧 checkpoint 和 D2/D3 等剩余资格保持开放;本次无合并操作。

English verdict: REQUEST_CHANGES - afbc0d62b74079fe6c96fbcf7ce74ab82faf065f; native task-file references are saved in the Goal but filtered into a null typed replan objective. All380 existing tests pass; independent real-entry/bootstrap/replan coverage has8 passes and2 affected-invariant failures. Repair the existing Goal/context boundary and retest the new exact head.

@loopx-agent

Copy link
Copy Markdown
Collaborator Author

Superseded by #6649 on current main. The replacement includes the original-task objective change plus complete scoped replan input/readback, regression coverage for task paths and full text, and the explicit publication boundary. Original branch and review discussion are preserved; no force push.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant