diff --git a/benchmark/README.md b/benchmark/README.md index d1cba0dace..b57d1a7bac 100644 --- a/benchmark/README.md +++ b/benchmark/README.md @@ -46,7 +46,7 @@ by itself establish a C2 uplift claim. ## Current work -- [Execution observations and research preset](runtime/RESEARCH_LOOP.md) +- [Execution observations through ordinary Goal planning](runtime/EXECUTION_OBSERVATIONS.md) describes two experiment choices: independent local measurements with ordinary planning/stride selection, and an explicit preset with retained research successor work. It also covers both entries' complete initial Goal objective diff --git a/benchmark/runtime/EXECUTION_OBSERVATIONS.md b/benchmark/runtime/EXECUTION_OBSERVATIONS.md new file mode 100644 index 0000000000..06edc94773 --- /dev/null +++ b/benchmark/runtime/EXECUTION_OBSERVATIONS.md @@ -0,0 +1,186 @@ +# Execution observations through ordinary Goal planning + +## Selecting local observations + +For heartbeat work that needs local execution measurements, set +`execution_observation="local_probe"` on `BenchmarkCodex`. Keep the full task +in the existing Goal/Todo planning input and choose `task_entry` and +`stride_strategy` independently. Merge these settings into the experiment's +existing agent configuration, retaining its model, task inputs and budgets: + +```yaml +agents: + - import_path: benchmark.runtime.harbor:BenchmarkCodex + kwargs: + execution_mode: heartbeat + task_entry: loopx-planned + execution_observation: local_probe + stride_strategy: "off" +``` + +`evidence_conditioned_v1` remains an independent, explicit strategy choice. +With `loopx-planned`, the ordinary planner determines the work; with +`seeded-todo`, the adapter seeds only the ordinary task and its full acceptance +criteria. No research preset creates candidate/research stages, a successor +Todo, a Vision patch or a forced stride strategy. Work after an initial +candidate follows the original Goal, accepted requirements, ordinary +Todo/continuation/replan and remaining useful authorized work. Candidate +completion does not settle other requested work. User Stop, original budgets +and genuine blockers remain authoritative; no work is created just to fill time. + +This option supports heartbeat only. Omission or `None` leaves it off and +preserves the ordinary task-entry, stride and probe-installation defaults. +When selected, the adapter installs `loopx-research-probe` and passes +`LOOPX_EXECUTION_OBSERVATION=local_probe` to the worker. Planning and heartbeat +instructions request reuse of applicable observations from an execution that +matches the actual candidate and input revision. Wrap a needed execution only +when decision-critical measurements are missing; do not repeat an experiment +merely to produce an attachment. Configuration and readback invoke neither a +probe nor a model. Runtime receipts record the selected producer, which does +not attest execution, adoption or improvement. + +## Retiring the bundled research treatment + +`research_loop` and `LOOPX_RESEARCH_LOOP` have been removed. Any presence of +the adapter kwarg (including `False`) or worker environment variable (including +`0`) fails before runner setup or wake creation with a migration error. Remove +both fields from new configurations; choose observation and stride separately +as above. The fixed two-stage initializer and its additional research +instructions are removed rather than retained as a compatibility mode. + +Use a new trial for a changed treatment. Historical configurations, scores, +trajectories and receipts remain pinned to their original product/runner +revisions and are not rewritten or relabeled as ordinary-planning evidence. +A retirement error does not migrate or reopen an active trial. + +## Preserving the original Goal + +When Harbor creates a Goal, an omitted `task_objective` uses the first complete +task instruction with leading/trailing whitespace removed (`instruction.strip()`). +An explicit `task_objective` takes precedence. Later phases update task input +without replacing the existing Goal objective, clearing waits or overriding Stop. +This initialization repair applies with observations omitted, `None` or +`"local_probe"`; it intentionally replaces the older generic objective. +It is independent of measurement selection and does not change the Core Goal +contract. Pin both product and runner revisions in comparisons, even with +observations off; unchanged defaults do not imply whole-treatment parity with +historical versions. + +## Validation and comparison boundary + +The [execution integration fixtures](../../tests/control_plane/test_research_execution_loop.py) +exercise the adapter, real local process, File/SQLite authorities, Explore +writeback, evidence-driven replan and subsequent changed execution. They also +cover independent observation with stride off and preservation of waits and Stop. +The [ordinary stage-continuity fixture](../../tests/control_plane/test_research_stage_continuity.py) +uses accepted requirements, the real scheduler/subprocess Host and exactly-once +settlement to verify that a completed stage cannot hide already requested work, +and that an accepted owner revision can withdraw that work. + +These are controlled engineering checks with deterministic actors, not evidence +that a model independently chooses productive research or that retirement +improves long-horizon outcomes. For that claim, compare the same complete task, +model, verifier and budgets at pinned revisions; record actual task structure, +strategy and instructions. Measure first-candidate quality, later validated +improvements and complete costs including observations and control overhead. +A combined-treatment comparison does not isolate a component's causal effect. + +## Measurement and attribution + +The adapter uses the staged runner/Python for `loopx-research-probe`. The +producer is also callable as `python -m benchmark.runtime.research_probe`: + +```sh +loopx-research-probe --candidate-file candidate.py --output probe-001.json \ + --input-revision public-case-v1 --planned-method-id scheduler \ + --executed-method-id mask-selection --change-reason 'The control isolated mask selection first.' \ + --evidence-ref probe:local-001 --limit-seconds 30 --diagnostics probe-diagnostics.json \ + -- python3 candidate.py +``` + +The solver may write `{"search_iterations":2,"stop_reason":"search_cap"}` to +`LOOPX_RESEARCH_DIAGNOSTICS`. The host measures elapsed time, exit/timeout and +the selected candidate-file digest before/after. A changed candidate rejects +attribution. Solver counters and reasons remain reports; missing counters +remain null. Low resource use does not impose a mandatory time floor. +Malformed optional diagnostics preserve reliable host measurements and retain +a bounded diagnostic-error receipt. Evidence paths must be fresh; raw output +stays local. The probe neither submits nor accesses hidden evaluation. + +The adjacent `.candidate.json` preserves selected relative paths and hashes. +`candidate_sha256` hashes the sorted JSON file manifest, not native archive +container bytes. Pins cover only declared files at the probe boundaries, not +transitive imports or independent writers. Result attribution must still prove +that the evaluated snapshot matches the selected candidate and input revision. + +Attach the emitted `execution_observation` through normal Explore result +writeback. Its input revision and evidence refs must belong to the enclosing +finding; released results must identify the candidate actually evaluated. +Finding refs, observation `evidence_refs` and `result_ref` share the existing +Explore reference syntax, so an existing `validation:run/1` can be retained. +Method ids keep their separate identifier rule. Producer selection leaves +[the Goal's Explore mode](../../loopx/capabilities/explore/README.md#explore-harness-modes) +unchanged; enable ordinary evidence writeback through that existing setting. +Different planned/executed method ids require a change reason. Findings, +replan and research-history details retain this same source. They do not +certify method benefit or Goal completion. Missing observations preserve old +events; a reader for the optional field is required for new retained findings. + +Result followthrough uses existing material delivery classifications. +`surface_only` and unknown historical outcomes remain readable but cannot +consume a cited work result or create a new unresolved experiment. Useful +negative results and concrete blockers remain eligible evidence. + +This adapter retains existing Host admission. They do not install C1's +per-provider-request guard or C3's native submission/feedback recovery. +Those guarantees require their separate entries and qualification; C1's +request-admission entry requires a configured Responses-compatible endpoint. +Local process and integration checks do not establish initial-quality parity, +long-horizon benefit or complete-cost efficiency. + +## 中文交付边界 + +执行观测复用普通 Goal/Todo/Explore：heartbeat 可独立选择 +`execution_observation="local_probe"`，`task_entry` 和 `stride_strategy` 分别配置。 +`loopx-planned` 由普通 planner 决定任务；`seeded-todo` 只预置保留完整验收要求的 +普通任务。省略观测或设为 `None` 时，不安装探针、不追加观测指引，任务入口和 +stride 默认选择保持不变。观测不创建候选/研究两阶段、研究后继或 Vision 补丁， +不强制 `evidence_conditioned_v1`。初始候选之后，按原目标、已接受要求、普通 +Todo/replan/Explore 及剩余有价值工作接续；候选完成不能关闭其他已请求工作。 +Stop、原预算和真实阻塞继续有效，不为填满时长制造任务。 + +`research_loop` 和 `LOOPX_RESEARCH_LOOP` 已直接退役，不保留兼容模式。新配置 +必须移除这两个字段；即使值为 `False` 或 `0`，仍会在 runner 初始化或 wake 创建 +之前明确报迁移错误，防止旧配置静默改变实验含义。需要测量时，按上方 YAML +独立选择 producer 和策略。处理组合变化须建立新 trial，不能原地转换进行中的 +工作；历史版本、配置、轨迹、分数和回执保持原归属，不改写为普通规划的验证。 + +新建 Harbor Goal 时，未提供 `task_objective` 则保存首次完整指令的 +`instruction.strip()`；显式值优先。后续 phase 更新输入，不替换原 objective， +也不清除等待或 Stop。这项初始化修复在未选观测、`None` 和 `"local_probe"` +下均生效，有意替换旧泛化目标，不改变 Core Goal 合同。即使未开启观测，对比 +也须固定 product/runner revision，不能将默认选择不变宣称为完整历史处理等价。 + +独立入口使用同一 `loopx-research-probe`。优先复用与实际候选和输入相符的已有 +执行证据；缺少影响决策的测量时，才在必要实验的现有预算内使用探针，不为补 +附件重复实验。配置和读回不调用模型或探针；producer 回执只记录配置，不证明 +发生执行、采用或收益。探针记录真实耗时、退出和超时状态，固定所选文件前后 +哈希；可选 solver 计数与宿主测量分开，缺测保持未知，坏诊断不能抹去可靠测量。 +证据路径必须新建，原始输出留在本地；不提交原生评测，不读取隐藏评测。候选 +变化拒绝归因；文件清单哈希不是原生归档哈希，只覆盖明确选择的文件及探针 +边界，不覆盖传递依赖或独立写入者，结果仍须证明对应实际评测快照和输入版本。 + +观测通过既有 Explore 写回、replan 和研究历程保留，输入版本与引用必须属于 +该 finding，计划/实际方法不同须记录理由。finding 引用及观测的 `evidence_refs`、 +`result_ref` 共用既有语法，可保留 `validation:run/1`；方法 ID 仍使用独立规则。 +选择 producer 不改变 Goal 的 Explore 模式，写回仍通过既有设置开启。旧事件 +缺失观测仍可读，新字段需兼容 reader。`surface_only` 和未知历史 delivery +不能消费实质结果或创建未解决实验；有用负结果和真实阻塞仍是有效证据。 + +真实进程和 File/SQLite 集成检查覆盖普通任务接续、执行观测进入 Explore、 +基于证据的 replan、后续变更执行以及 Stop/等待保留；普通阶段接续还覆盖真实 +scheduler/Host、验收要求和一次结算。确定性 actor 的工程检查不证明模型自主 +研究收益；效果和效率结论仍需对齐任务、模型、评测器、预算及完整测量/控制成本 +的实验，整体组合对照不能归因单项因果效果。既有 Host 准入不变；本入口不安装 +C1 逐次 provider 请求守卫或 C3 原生提交/反馈恢复，这些组合保证须分别验证， +C1 仍需配置兼容 Responses 的端点。 diff --git a/benchmark/runtime/RESEARCH_LOOP.md b/benchmark/runtime/RESEARCH_LOOP.md deleted file mode 100644 index 4de0ea7e45..0000000000 --- a/benchmark/runtime/RESEARCH_LOOP.md +++ /dev/null @@ -1,308 +0,0 @@ -# Execution observations and research preset - -## Choosing the experiment - -The heartbeat adapter supports both an independent observation producer and -an explicit research preset. They answer different experimental questions and -share the existing Explore evidence path: - -| Experiment intent | Adapter choice | Task structure and strategy | -| --- | --- | --- | -| Validate an existing candidate, reproduce a result, or measure ordinary work without prescribing its next stage. | `execution_observation="local_probe"` | Existing Goal/Todo planning determines the work; `task_entry` and `stride_strategy` remain independent. | -| Test a declared optimization task as initial candidate plus retained follow-up research, keeping the best validated artifact. | `research_loop=True` | The preset creates canonical candidate/research Todos and selects `evidence_conditioned_v1` with the research instructions. | - -The research preset remains a supported benchmark combination. Its explicit -follow-up work is useful to test whether candidate completion leads to further -research under the original goal and budget. Independent observations also -serve tasks that do not need that fixed structure. Neither choice introduces -a global LoopX product mode or establishes better outcomes by configuration. - -## Independent execution observations - -For heartbeat work that needs measurements with its own task structure, set -`execution_observation="local_probe"` on `BenchmarkCodex`. Keep the original -task in the existing Goal/Todo planning input and choose `task_entry` and -`stride_strategy` independently. For example, merge these adapter settings -into the experiment's existing agent configuration, retaining its model, -task inputs and budgets: - -```yaml -agents: - - import_path: benchmark.runtime.harbor:BenchmarkCodex - kwargs: - execution_mode: heartbeat - task_entry: loopx-planned - execution_observation: local_probe - stride_strategy: "off" -``` - -`evidence_conditioned_v1` remains an explicit strategy choice. With -`loopx-planned`, the ordinary planner owns the task structure; with -`seeded-todo`, the ordinary seed retains the task's full acceptance criteria. -Selecting this producer creates no initial-candidate/research stages and -adds no obligation to continue optimizing a completed task. Follow-up work -depends on the original goal, evidence, remaining useful authorized work and -the existing Stop, budget and blocking boundaries. - -This option supports `heartbeat` execution only. Omission or `None` leaves it -off; omitting both it and `research_loop` leaves the defaults for probe -installation, task stages and stride selection unchanged. Goal -initialization is intentionally repaired as described below. -The adapter installs the existing `loopx-research-probe` command and -passes `LOOPX_EXECUTION_OBSERVATION=local_probe` to its worker. Planning and -heartbeat instructions request evidence when relevant: reuse an existing -observation that matches the actual execution, candidate and input; use the -probe during a needed experiment only when a decision-critical measurement -is missing. Do not repeat an experiment merely to produce a new receipt. -Configuration and ordinary readback do not invoke the probe or a model. - -Runtime receipts record the resolved producer and retain the research preset's -`research_loop` marker when applicable. Those fields describe configuration; -they do not attest that a probe ran, that evidence was adopted or that a -candidate improved. Keep the resolved settings and task input in the -experiment's existing records; no separate product mode is introduced. - -## Original Goal at the adapter entry - -When Harbor creates a Goal, in either ordinary entry or the research preset, -an omitted `task_objective` uses the first complete task instruction with -leading/trailing whitespace removed (`instruction.strip()`). An explicit -`task_objective` takes precedence. Later phases update the task input without -replacing the existing Goal's objective, retaining the original goal text. - -This adapter initialization repair applies with `execution_observation` -omitted, `None` or `"local_probe"`, and with `research_loop=True`. It is -independent of measurement selection and intentionally changes the earlier -generic Goal objective in both entries. The preset's two stages, fixed stride -and additional research instructions are retained; its default Goal input has -changed, so the complete treatment is not identical to historical runs. -This does not change the Core Goal contract or prove that a model will -autonomously continue correctly. Compare experiments at pinned product and -runner revisions, including the Goal initialization change in the treatment -definition even when no observation producer is selected. - -## Research preset - -Select the preset explicitly in the same experiment configuration: - -```yaml -agents: - - import_path: benchmark.runtime.harbor:BenchmarkCodex - kwargs: - execution_mode: heartbeat - task_entry: loopx-planned - research_loop: true -``` - -`research_loop=True` defines a complete benchmark treatment: -it prepares an initial-candidate Todo and a deferred research successor, -selects `evidence_conditioned_v1`, installs the same probe and delivers the -original research instructions. It still prepares these stages when -`task_entry="loopx-planned"` is explicitly selected. An explicitly -conflicting stride strategy, including `"off"`, remains rejected. - -Combining `research_loop=True` with `execution_observation="local_probe"` -is rejected before work begins because the preset already selects that -producer. To study independent planning or a different stride strategy, select -the independent observation configuration instead. Use a new trial when -changing treatments; neither entry is an in-place conversion of active work. - -The preset uses existing `todo_done` and successor semantics: the retained -research Todo is canonical deferred work with `resume_when=todo_done:` -and an explicit successor relation. Candidate completion makes that work due -for the ordinary resume/replan path; it cannot by itself close the research -Todo. Goal instructions alone do not precreate this work or its dependency. -Research retains the best candidate, meaningful negative results and reasoned -closeout against the original objective. User Stop, the original budget and -genuine blockers remain authoritative. - -Preparation writes ordinary Agent Vision scope as `surface_only` before adding -the successor edge. With the default `hard_lease` handoff, it observes the -candidate Todo's lease version, acquires its own short-lived grant with -compare-and-swap, supplies that grant's exact proof to the Todo update and -releases it in `finally`. This uses the existing ownership guard. It produces -real lease events and version changes, with control overhead, but invokes no -model, runs no experiment and spends no work Turn. `surface_only` does not -claim research progress. - -An existing successor edge makes preparation return without changing Vision -or acquiring another lease. A missing edge is repaired only when this Agent's -research Todo remains deferred on exactly that initial Todo's completion; -the retry reuses the retained Todo, publishes scope before the edge and uses -a fresh lease grant after a released failure. Research that has moved to -another status or wait, or belongs to an earlier initial Todo, is returned -without repair. This is bounded initialization recovery; it does not reopen -research or clear a genuine wait. - -## Comparison and validation boundary - -The [supplied-candidate validation fixture](../../tests/control_plane/test_research_execution_loop.py) -checks the independent entry's real process, Explore readback and preservation -of waits and Stop. It does not demonstrate that ordinary planning replaces -the preset's research continuation, that a model will choose productive -follow-up work, or that this choice improves efficiency. The separate -candidate-to-research fixture checks a synthetic continuation chain; it also -does not establish model research benefit. - -For a matched comparison, supply the same complete explicit `task_objective` -to both entries and retain the same task, model, verifier and budget. Record -each entry's resolved task structure, strategy and instructions. Comparing -the whole preset with an independent arm estimates the combined treatment; -isolating a component requires an appropriate additional comparison. Measure -initial-candidate quality, subsequent validated improvements and complete -costs, including observation and control overhead. - -The preset has no predetermined retirement. Revisit its scope against actual -usage and maintenance costs. Replacing it with ordinary planning requires -evidence that the requested research continuation and task coverage are retained; -claims about better outcomes or efficiency require matched comparisons. -The existence of a Goal contract or one successful validation is insufficient. -Historical studies remain pinned to their original product and runner -revisions, task inputs and treatment definitions; frozen configurations and -results are not rewritten or relabeled as evidence for another entry. - -## Measurement and attribution - -Both entries use the staged runner/Python for `loopx-research-probe`. The -producer is also callable as `python -m benchmark.runtime.research_probe`: - -```sh -loopx-research-probe --candidate-file candidate.py --output probe-001.json \ - --input-revision public-case-v1 --planned-method-id scheduler \ - --executed-method-id mask-selection --change-reason 'The control isolated mask selection first.' \ - --evidence-ref probe:local-001 --limit-seconds 30 --diagnostics probe-diagnostics.json \ - -- python3 candidate.py -``` - -The solver may write `{"search_iterations":2,"stop_reason":"search_cap"}` to -`LOOPX_RESEARCH_DIAGNOSTICS`. The host measures elapsed time, exit/timeout and -the selected candidate-file digest before/after. A changed candidate rejects -attribution. Solver counters and reasons remain reports; missing counters -remain null. Low resource use does not impose a mandatory time floor. -Malformed optional diagnostics preserve reliable host measurements and retain -a bounded diagnostic-error receipt. Evidence paths must be fresh; raw output -stays local. The probe neither submits nor accesses hidden evaluation. - -The adjacent `.candidate.json` preserves selected relative paths and hashes. -`candidate_sha256` hashes the sorted JSON file manifest, not native archive -container bytes. Pins cover only declared files at the probe boundaries, not -transitive imports or independent writers. Result attribution must still prove -that the evaluated snapshot matches the selected candidate and input revision. - -Attach the emitted `execution_observation` through normal Explore result -writeback. Its input revision and evidence refs must belong to the enclosing -finding; released results must identify the candidate actually evaluated. -Finding refs, observation `evidence_refs` and `result_ref` share the existing -Explore reference syntax, so an existing `validation:run/1` can be retained. -Method ids keep their separate identifier rule. Producer selection leaves -[the Goal's Explore mode](../../loopx/capabilities/explore/README.md#explore-harness-modes) -unchanged; enable ordinary evidence writeback through that existing setting. -Different planned/executed method ids require a change reason. Findings, -replan and research-history details retain this same source. They do not -certify method benefit or Goal completion. Missing observations preserve old -events; a reader for the optional field is required for new retained findings. - -Result followthrough uses existing material delivery classifications. -`surface_only` and unknown historical outcomes remain readable but cannot -consume a cited work result or create a new unresolved experiment. Useful -negative results and concrete blockers remain eligible evidence. - -These entries retain existing Host admission. They do not install C1's -per-provider-request guard or C3's native submission/feedback recovery. -Those guarantees require their separate entries and qualification; C1's -request-admission entry requires a configured Responses-compatible endpoint. -Local process and integration checks do not establish initial-quality parity, -long-horizon benefit or complete-cost efficiency. - -## 中文交付边界 - -保留 `research_loop=True` 作为明确的 benchmark 研究实验预设，同时提供独立 -`execution_observation="local_probe"`。两者共享既有 Explore 证据链，但实验 -目的不同：已有候选的单次验证、复现实验或普通工程测量适合独立入口；原任务 -明确要求持续优化，并需要研究“初始候选之后是否继续产生收益”时，可选择研究 -预设。预设的固定组合便于验证这一接续策略；观测独立则允许不需要该结构的任务 -复用测量能力。两种选择均有用途，不将其升级为 LoopX 全局产品模式，也不因配置 -成功就断言效果更好。 - -两份 YAML 示例可分别放入现有 `agents[].kwargs` 配置，保留原模型、任务输入 -和预算。独立入口的 `task_entry` 和 `stride_strategy` 分别选择， -`"off"` 须在 YAML 中保留引号。 -`loopx-planned` 继续由既有 planner 决定任务结构，`seeded-todo` 只预置保留 -完整验收要求的普通任务。采集执行证据不会预建初始候选/研究阶段，也不要求 -完成普通任务后继续研究；后续工作由原 Goal、当前证据和剩余有价值工作决定。 - -独立参数目前只支持 heartbeat；省略或设为 `None` 不启用该入口，同时省略 -`research_loop` 时，探针安装、任务阶段和 stride 选择的默认行为不变。 -两入口的 Goal 初始化均包含下面说明的有意修复。两入口使用同一个 `loopx-research-probe` -命令，独立入口通过 `LOOPX_EXECUTION_OBSERVATION=local_probe` 传给 worker。 -独立入口的规划和 heartbeat 指引要求优先复用与实际执行、候选和输入相符的现有观测； -缺少影响决策的关键测量时，才在需要进行的实验中使用探针，不为补一份回执 -重复实验。配置和普通读回不触发探针或模型调用。运行回执记录解析后的 producer， -研究预设保留 `research_loop` 标识；这些配置记录不证明发生了执行或实质采用。 - -Harbor 新建 Goal 时，普通入口和研究预设均在未显式提供 `task_objective` 时， -将首次完整任务指令去除首尾空白后的 `instruction.strip()` 保存为 objective； -显式值优先。后续 phase 更新任务输入，不替换已有 Goal 的 objective，保留原目标 -文本。这项适配器入口修复在未选观测、`None`、`"local_probe"` 和 -`research_loop=True` 下均生效，有意替换两入口过去的泛化目标文案。预设的 -两阶段、固定 stride 和追加研究指引保留，但默认 Goal 输入已经改变，不能宣称 -整个处理与历史版本完全一致。这不是测量参数附带的研究语义，也未修改 Core -Goal 合同或证明模型已经能够自主正确接续。即使未启用观测，实验对比也须固定 -product/runner revision，并记录 Goal 初始化的变化。 - -`research_loop=True` 表示完整研究处理：预建候选 Todo 和研究后继，使用 -`evidence_conditioned_v1`、同一探针和原研究指引。即使显式选择 `loopx-planned` -也保留原阶段初始化；显式 `"off"` 等冲突策略仍会被拒绝。预设已选择本地 producer， -因此不能同时指定 `execution_observation="local_probe"`；需要独立规划或其他 -stride 时，选择独立入口的配置。变更处理时建立新 trial，不原地改写进行中的工作。 - -研究预设沿用既有 Todo 依赖、successor 和 Vision：它实际创建 canonical deferred -研究后继及 `resume_when=todo_done:`，候选完成使其进入普通 resume/replan -接续判断，不等于研究 Todo 完成。普通 Goal 的持续推进指引不会自动创建同一条 -持久化后继及依赖，因此两者不能只凭合同文字就视为等价。保留最好候选和有意义 -的负结果，依据原目标及剩余工作解释研究结束；Stop、原预算和真实阻塞保持 -有效，不要求填满时长或改变原生评测规则。 - -准备阶段先以 `surface_only` 写入 Agent Vision 范围，再添加后继边。默认 -`hard_lease` 下，先读取候选 Todo 的 lease 版本，经 compare-and-swap 取得 -独立短暂授权，以该授权的精确 proof 更新 Todo，最后在 `finally` 中释放同一 -授权，沿用既有所有权 guard。它会产生真实 lease 事件、版本变化和控制成本， -但不调用模型、不执行实验、不消耗工作 Turn；`surface_only` 不声称研究进展。 - -已有完整后继边时，准备动作直接返回，不改 Vision，也不再取得 lease。只在本 -Agent 的研究 Todo 仍为 deferred 且精确等待当前初始 Todo 完成时，才补缺失 -的边；重试复用同一研究 Todo，先写 scope 再写边，先前失败释放后使用新授权。 -研究已经转入其他状态、改变等待条件或属于先前初始 Todo 时，原样返回而不修补。 -这是有限的初始化恢复，不重开研究或清除真实等待。 - -现有单次候选验证测试覆盖独立入口的真实进程、Explore 读回以及 Stop/等待保留， -未证明它能替代预设的研究接续，也未证明模型会自主产生有效后续实验或提高效率。 -另一个候选到研究的测试验证 synthetic 接续链，同样不等于模型研究收益。 -对照实验应给两入口同一个完整显式 `task_objective`，对齐任务、模型、评测器和 -预算，记录任务结构、策略与提示的实际组合。整体组合对照只能评价整体处理； -归因单项能力需要额外设计相应对照。除初始质量和后续可验证改善外，还需计算 -探针与控制成本在内的完整消耗。 - -不预定删除研究预设，依据实际用途和维护成本重新判断其范围。若以普通规划替代， -需证明所要求的研究接续及任务覆盖仍然成立；若声称效果或效率更好，则需可比实验。 -存在 Goal 合同、通过一次验证都不足以证明它应退役。历史实验继续固定原 product/runner -版本、输入和处理定义，冻结配置及结果不改标为另一入口的验证。 - -本地探针固定所选文件，记录真实耗时、退出与超时状态；可选 solver 报告与宿主 -测量分开，缺失计数保持未知，坏诊断不能抹去可靠测量。证据路径必须新建，原始 -输出留在本地。候选变化拒绝归因；文件清单哈希不是原生归档哈希，只覆盖明确 -选择的文件及探针前后边界，不涵盖传递依赖或独立写入者。结果仍须对应实际评测 -快照、候选和输入版本，计划/实际方法不同须记录理由。 - -同一可选观测通过既有 Explore 写回、replan 和研究历程保留；输入版本与引用 -必须属于该 finding，不认证方法收益或 Goal 完成。finding 引用与观测的 -`evidence_refs`、`result_ref` 共用既有 Explore 引用语法，可直接保留 -`validation:run/1` 这类原引用；方法 ID 仍使用独立规则。选择 producer 不改变 -Goal 的 Explore 模式；需要写回时，仍通过[既有设置](../../loopx/capabilities/explore/README.md#explore-harness-modes) -开启普通证据能力。旧事件缺失观测仍可读， -新事件需要支持该可选字段的 reader。`surface_only` 和未知历史 delivery -不能消费实质结果或创建未解决实验；有用负结果和真实阻塞继续可作为证据。 - -两入口均沿用现有 Host 准入，不安装 C1 逐次 provider 请求守卫，也不交付 C3 -原生提交/反馈恢复。组合 C1 需其独立入口及兼容 Responses 的端点；各组合保证 -须分别验证。本地执行和工程回归不证明首次质量持平、长程收益或完整成本效率。 diff --git a/benchmark/runtime/harbor.py b/benchmark/runtime/harbor.py index 157c9235e6..0ca3c5f22d 100644 --- a/benchmark/runtime/harbor.py +++ b/benchmark/runtime/harbor.py @@ -73,13 +73,18 @@ def __init__( turn_envelope=False, native_goal_continue=False, task_objective=None, - research_loop=False, execution_observation=None, goal_id=_GOAL_ID, agent_id=_AGENT_ID, defer_codex_home=False, **kwargs, ): + if "research_loop" in kwargs: + raise ValueError( + "research_loop has been removed; remove it from the adapter kwargs. " + "Use ordinary Goal/Todo planning, choose stride_strategy independently, " + "and select execution_observation='local_probe' only when needed." + ) if isinstance(validation_command, str): raise ValueError("validation_command must be an argv list, not shell text") self.execution = Execution( @@ -114,9 +119,6 @@ def __init__( self.replan_after_todos = int(3 if replan_after_todos is None else replan_after_todos) if not 1 <= self.replan_after_todos <= 5: raise ValueError("replan_after_todos must be between 1 and 5") - from .research_loop import research_stride_strategy - - stride_strategy = research_stride_strategy(research_loop, stride_strategy) if stride_strategy is not None and not self.execution.uses_loopx: raise ValueError("stride_strategy requires a LoopX execution mode") self.stride_strategy = (normalize_stride_strategy(stride_strategy) @@ -127,14 +129,9 @@ def __init__( if not self.execution.uses_loopx: raise ValueError("planning_horizon_goal requires a LoopX execution mode") self.planning_horizon_goal = planning_horizon_goal - if not isinstance(research_loop, bool) or (research_loop and self.execution.mode != "heartbeat"): - raise ValueError("research_loop requires heartbeat execution") - self.research_loop = research_loop from .research_probe import resolve_execution_observation - self.execution_observation = resolve_execution_observation( - execution_observation, research_loop=research_loop, - ) + self.execution_observation = resolve_execution_observation(execution_observation) if self.execution_observation is not None and self.execution.mode != "heartbeat": raise ValueError("execution_observation requires heartbeat execution") self.checkpoint_request_path = None @@ -355,7 +352,6 @@ async def install(self, environment: BaseEnvironment) -> None: "execution_mode": self.execution.mode, "iteration_context": self.execution.context, "task_entry": self.execution.task_entry, - **({"research_loop": True} if self.research_loop else {}), **({"execution_observation": self.execution_observation} if self.execution_observation is not None else {}), **self._turn_envelope_receipt(), @@ -518,12 +514,8 @@ async def _prepare_phase( cwd=cwd, ) - if self.execution.task_entry == "seeded-todo" or self.research_loop: + if self.execution.task_entry == "seeded-todo": await self._seed_phase(environment, cwd=cwd) - if self.research_loop: - from .research_loop import prepare_research_continuation - - await prepare_research_continuation(self, environment, cwd=cwd) cadence = await self._loopx( environment, ["configure-goal", "--goal-id", self.goal_id], cwd=cwd, @@ -549,13 +541,6 @@ async def _seed_phase(self, environment: BaseEnvironment, *, cwd: str) -> None: "implement and validate against the task's full requirements and acceptance " "criteria. Keep unmet requirements explicit when judging task completion." ) - if self.research_loop: - text = ( - f"[P0] Build and evaluate the strongest feasible initial candidate for {self._task_document}. " - "Check the actual interface, correctness and resource use before the first submission. " - "Preserve the tested candidate and released feedback. Completing this candidate stage " - "does not complete the separate original research work." - ) if self._seeded_todo_id: listed = await self._loopx(environment, [ "todo", "list", "--goal-id", self.goal_id, "--role", "agent", @@ -617,9 +602,8 @@ def _worker_env(self, *, cwd: str) -> dict[str, str]: "LOOPX_SHARED_SKILLS": _SHARED_SKILLS, "LOOPX_EXECUTION_MODE": self.execution.mode, "LOOPX_TASK_ENTRY": self.execution.task_entry, - **({"LOOPX_RESEARCH_LOOP": "1"} if self.research_loop else {}), **({"LOOPX_EXECUTION_OBSERVATION": self.execution_observation} - if self.execution_observation is not None and not self.research_loop else {}), + if self.execution_observation is not None else {}), **({"LOOPX_CHECKPOINT_REQUEST": self.checkpoint_request_path} if self.checkpoint_request_path is not None else {}), **({"LOOPX_TURN_ENVELOPE": "1"} if self.execution.turn_envelope else {}), diff --git a/benchmark/runtime/planning.py b/benchmark/runtime/planning.py index b721aaeaf6..b65f7f236b 100644 --- a/benchmark/runtime/planning.py +++ b/benchmark/runtime/planning.py @@ -36,7 +36,7 @@ def render_planning_prompt(packet: dict, env: dict[str, str]) -> str: value = float(env[key]) if not math.isfinite(value) or value <= 0: raise ValueError(f"{key} must be finite and positive") - from .research_loop import runtime_execution_guidance + from .research_probe import runtime_execution_guidance return ("$loopx\n\nBenchmark research planning guidance:\n" + _RESEARCH_PLANNING_GUIDANCE + runtime_execution_guidance(env) diff --git a/benchmark/runtime/research_loop.py b/benchmark/runtime/research_loop.py deleted file mode 100644 index 64031a2675..0000000000 --- a/benchmark/runtime/research_loop.py +++ /dev/null @@ -1,120 +0,0 @@ -"""An explicit benchmark research preset using existing Goal and Todo owners.""" -from __future__ import annotations - -from uuid import uuid4 - - -def runtime_execution_guidance(env: dict[str, str]) -> str: - """Select the research preset or independent observation guidance.""" - from .research_probe import execution_observation_guidance, resolve_execution_observation - - research_preset = env.get("LOOPX_RESEARCH_LOOP") == "1" - producer = resolve_execution_observation( - env.get("LOOPX_EXECUTION_OBSERVATION"), research_loop=research_preset, - ) - if research_preset: - return research_execution_guidance() - return execution_observation_guidance() if producer is not None else "" - - -def research_stride_strategy(enabled, strategy): - """One treatment definition shared by native CLI and programmatic entries.""" - if not enabled: - return strategy - if strategy not in (None, "evidence_conditioned_v1"): - raise ValueError("research_loop requires evidence_conditioned_v1, not a disabled or different review strategy") - return "evidence_conditioned_v1" - - -async def prepare_research_continuation(runtime, environment, *, cwd): - """A candidate stage cannot close a separately retained research outcome. - - The deferred successor contains actual optimization work, not a recurring - instruction to replan. Its existing resume owner releases it when the - initial stage is done; budget, Stop and evidence-backed closeout stay native. - """ - initial = runtime._seeded_todo_id - if not initial: - raise ValueError("research continuation requires its initial candidate stage") - listed = await runtime._loopx(environment, ["todo", "list", "--goal-id", runtime.goal_id, - "--role", "agent"], cwd=cwd) - initial_todo = next((todo for todo in listed["todos"] if todo["todo_id"] == initial), None) - if initial_todo is None: - raise ValueError("research continuation requires its retained initial candidate stage") - matches = [todo for todo in listed["todos"] if todo.get("target_key") == "research-continuation"] - if len(matches) > 1: - raise ValueError("research continuation identity is ambiguous") - if matches: - research = matches[0] - if research.get("claimed_by") != runtime.agent_id: - raise ValueError("research continuation belongs to another Agent") - if (research["todo_id"] in initial_todo.get("successor_todo_ids", []) - or research.get("status") != "deferred" - or research.get("resume_when") != "todo_done:" + initial): - return research # Never reopen work, clear waits or replace later Vision. - else: - research = await runtime._loopx(environment, [ - "todo", "add", "--goal-id", runtime.goal_id, "--role", "agent", - "--claimed-by", runtime.agent_id, "--task-class", "advancement_task", - "--action-kind", "benchmark_task", "--target-key", "research-continuation", - "--status", "deferred", "--resume-when", "todo_done:" + initial, - "--text", f"[P1] Improve the original research objective in {runtime._task_document} through " - "complete, evidence-driven experiments. Preserve the best validated candidate. Use released " - "results and actual resource diagnostics to choose between a local repair and a different " - "mechanism; retain negative results. Candidate correctness and a successful submission settle " - "only that candidate stage. Close this research work only with evidence against the original " - "objective and remaining useful authorized work. Explicit user Stop, the original budget limit " - "and genuine blockers remain valid stops; do not manufacture work or run merely to fill time.", - "--execute"], cwd=cwd) - binding = ["--goal-id", runtime.goal_id, "--todo-id", initial] - mode = await runtime._loopx(environment, ["handoff-mode", "show", "--goal-id", - runtime.goal_id], cwd=cwd) - lease = None - proof = [] - if mode["handoff_mode"] == "hard_lease": - observed = await runtime._loopx(environment, ["task-lease", "inspect", *binding], cwd=cwd) - version = observed["lease"]["version"] if observed.get("lease") else 0 - # A failed update releases this attempt. A retry must acquire a fresh - # grant, never borrow another execution's proof or reuse a retired key. - key = "research-preparation-" + uuid4().hex - acquired = await runtime._loopx(environment, ["task-lease", "acquire", *binding, - "--owner", runtime.agent_id, "--idempotency-key", key, - "--expected-version", str(version)], cwd=cwd) - lease = acquired["lease"] - proof = ["--task-lease-idempotency-key", key, - "--task-lease-expected-version", str(lease["version"])] - try: - # Publish scope before the edge: an interrupted preparation with a - # missing edge reuses this Todo, while a complete edge preserves later Vision. - await runtime._loopx(environment, ["refresh-state", "--goal-id", runtime.goal_id, - "--agent-id", runtime.agent_id, "--classification", "research_scope_initialized", - "--delivery-batch-scale", "single_surface", "--delivery-outcome", "surface_only", - "--vision-state", "active", "--vision-advancement-policy", "as_needed", - "--vision-summary", "Build a strong initial candidate, then execute evidence-driven improvements while retaining the best validated artifact.", - "--vision-acceptance", "Candidate validity settles its stage, not the original research objective. Respect user Stop, original budget and genuine blockers; justify research closeout from remaining useful authorized work.", - "--no-global-sync", "--suppress-external-sinks"], cwd=cwd) - await runtime._loopx(environment, ["todo", "update", *binding, - "--agent-id", runtime.agent_id, "--successor-todo-id", research["todo_id"], - *proof, "--execute"], cwd=cwd) - finally: - if lease is not None: - await runtime._loopx(environment, ["task-lease", "release", *binding, - "--owner", runtime.agent_id, "--idempotency-key", lease["idempotency_key"], - "--expected-version", str(lease["version"])], cwd=cwd) - return research - - -def research_execution_guidance() -> str: - return ( - '\nResearch-loop execution evidence:\nThe initial candidate and ongoing research are separate canonical Todos. ' - 'Do not close the research successor merely because the candidate passes tests. Preserve the original ' - 'objective and existing Stop/budget/blocked conditions. Before the first native submission, run a relevant ' - 'local candidate check and retain actual resource observations, including unknown measurements. Use ' - '`loopx-research-probe --help` for the local probe wrapper: it pins explicit candidate files, measures ' - 'runtime and enforces the selected local limit. A solver can write search_iterations and stop_reason as JSON ' - 'to LOOPX_RESEARCH_DIAGNOSTICS; absent values remain unknown, and solver reports are not independently ' - 'verified. This is not another evaluation or a requirement to use every available second. Attach the emitted ' - 'execution_observation in the normal Explore result writeback. Preserve its input revision and evidence ' - 'refs. Record changed method ids and the reason when actual implementation differs from the proposal. ' - 'Resolve this evidence at the next replan.\n' - ) diff --git a/benchmark/runtime/research_probe.py b/benchmark/runtime/research_probe.py index 01cf8c0bfb..9b4f87025c 100644 --- a/benchmark/runtime/research_probe.py +++ b/benchmark/runtime/research_probe.py @@ -17,17 +17,25 @@ import time -def resolve_execution_observation(producer, *, research_loop=False): - """Resolve a producer; the research preset already includes local_probe.""" +def resolve_execution_observation(producer): + """Select measurements without prescribing task structure or strategy.""" if producer not in (None, "local_probe"): raise ValueError("execution_observation must be omitted or 'local_probe'") - if research_loop and producer is not None: + return producer + + +def execution_observation_from_env(env: dict[str, str]): + if "LOOPX_RESEARCH_LOOP" in env: raise ValueError( - "research_loop already includes the local_probe producer; " - "omit execution_observation for the research preset, or omit research_loop " - "to choose task_entry and stride_strategy independently" + "LOOPX_RESEARCH_LOOP has been removed; remove it from the worker environment. " + "Use ordinary Goal/Todo planning and select LOOPX_EXECUTION_OBSERVATION=local_probe " + "only when needed." ) - return "local_probe" if research_loop else producer + return resolve_execution_observation(env.get("LOOPX_EXECUTION_OBSERVATION")) + + +def runtime_execution_guidance(env: dict[str, str]) -> str: + return execution_observation_guidance() if execution_observation_from_env(env) else "" def execution_observation_guidance() -> str: diff --git a/benchmark/runtime/worker.py b/benchmark/runtime/worker.py index 440bcc6494..f41a74c240 100644 --- a/benchmark/runtime/worker.py +++ b/benchmark/runtime/worker.py @@ -88,7 +88,7 @@ def heartbeat_body(env: dict[str, str], turn_id: str, *, native_goal: bool, body = payload["task_body"].replace( "$HOME/.codex/loopx/registry.global.json", env["LOOPX_REGISTRY"] ) - from .research_loop import runtime_execution_guidance + from .research_probe import runtime_execution_guidance return body + runtime_execution_guidance(env) @@ -380,12 +380,9 @@ def run_once(env: dict[str, str]) -> dict: stage = env.get("LOOPX_TASK_STAGE", "execute") if stage not in {"plan", "execute"} or (stage == "plan" and execution.task_entry != "loopx-planned"): raise ValueError("invalid task-entry stage") - from .research_probe import resolve_execution_observation + from .research_probe import execution_observation_from_env - research_preset = env.get("LOOPX_RESEARCH_LOOP") == "1" - observation_producer = resolve_execution_observation( - env.get("LOOPX_EXECUTION_OBSERVATION"), research_loop=research_preset, - ) + observation_producer = execution_observation_from_env(env) if observation_producer is not None and execution.mode != "heartbeat": raise ValueError("execution_observation requires heartbeat execution") turn_id = f"wake-{time.time_ns()}-{uuid.uuid4().hex[:12]}" @@ -402,7 +399,6 @@ def run_once(env: dict[str, str]) -> dict: "task_entry": execution.task_entry, **({"execution_observation": observation_producer} if observation_producer is not None else {}), - **({"research_loop": True} if research_preset else {}), **({"turn_envelope": True, "turn_envelope_source": "explicit"} if execution.turn_envelope else {}), "stage": stage, diff --git a/benchmark/tests/test_task_entry.py b/benchmark/tests/test_task_entry.py index 7521fd02fc..4bcc494003 100644 --- a/benchmark/tests/test_task_entry.py +++ b/benchmark/tests/test_task_entry.py @@ -51,7 +51,6 @@ def test_execution_observation_keeps_task_entry_and_stride_independent(tmp_path, env = agent._worker_env(cwd=str(tmp_path)) assert agent.execution.task_entry == entry assert agent.stride_strategy == strategy - assert agent.research_loop is False assert env["LOOPX_EXECUTION_OBSERVATION"] == "local_probe" assert "LOOPX_RESEARCH_LOOP" not in env @@ -66,7 +65,6 @@ def test_execution_observation_keeps_task_entry_and_stride_independent(tmp_path, {"execution_observation": "local_probe", "execution_mode": "loopx-goal"}, {"execution_observation": "local_probe", "execution_mode": "turn", "validation_command": ["python", "validate.py"]}, - {"execution_observation": "local_probe", "research_loop": True}, ]) def test_execution_observation_rejects_ambiguous_or_unsupported_treatments(tmp_path, settings): pytest.importorskip("harbor") @@ -76,47 +74,59 @@ def test_execution_observation_rejects_ambiguous_or_unsupported_treatments(tmp_p BenchmarkCodex(logs_dir=tmp_path, model_name="fixture", **settings) -def test_research_preset_and_ordinary_default_remain_distinct(tmp_path): +@pytest.mark.parametrize("retired", [True, False, None, "true"]) +def test_retired_research_setting_fails_before_adapter_setup(tmp_path, monkeypatch, retired): + pytest.importorskip("harbor") + from benchmark.runtime.harbor import BenchmarkCodex, CodexOffline + + def unexpected_setup(*args, **kwargs): + pytest.fail("A retired treatment cannot initialize the underlying runner") + + monkeypatch.setattr(CodexOffline, "__init__", unexpected_setup) + with pytest.raises(ValueError, match="research_loop has been removed"): + BenchmarkCodex(logs_dir=tmp_path, model_name="fixture", research_loop=retired) + + +def test_observation_guidance_is_optional_without_a_research_stage(tmp_path): pytest.importorskip("harbor") from benchmark.runtime.harbor import BenchmarkCodex from benchmark.runtime.planning import render_planning_prompt - from benchmark.runtime.research_loop import research_execution_guidance + from benchmark.runtime.research_probe import execution_observation_guidance ordinary = BenchmarkCodex(logs_dir=tmp_path / "ordinary", model_name="fixture") plain_env = ordinary._worker_env(cwd=str(tmp_path)) assert "LOOPX_RESEARCH_LOOP" not in plain_env assert "LOOPX_EXECUTION_OBSERVATION" not in plain_env assert ordinary.stride_strategy is None - preset = BenchmarkCodex(logs_dir=tmp_path / "preset", model_name="fixture", research_loop=True) - preset_env = preset._worker_env(cwd=str(tmp_path)) - assert preset_env["LOOPX_RESEARCH_LOOP"] == "1" - assert preset.stride_strategy == "evidence_conditioned_v1" + observed = BenchmarkCodex(logs_dir=tmp_path / "observed", model_name="fixture", + execution_observation="local_probe") + observed_env = observed._worker_env(cwd=str(tmp_path)) + assert observed.execution.task_entry == ordinary.execution.task_entry + assert observed.stride_strategy == ordinary.stride_strategy packet = {"input_digest": "original", "text": "Test the supplied candidate."} ordinary_prompt = render_planning_prompt(packet, plain_env) - preset_guidance = research_execution_guidance() - preset_prompt = render_planning_prompt(packet, preset_env) - assert preset_prompt.count(preset_guidance) == 1 - assert preset_prompt.replace(preset_guidance, "") == ordinary_prompt - with pytest.raises(ValueError, match="research_loop"): - BenchmarkCodex(logs_dir=tmp_path / "conflicting", model_name="fixture", - research_loop=True, stride_strategy="off") + observed_prompt = render_planning_prompt(packet, observed_env) + guidance = execution_observation_guidance() + assert guidance not in ordinary_prompt + assert observed_prompt.count(guidance) == 1 + assert observed_prompt.replace(guidance, "") == ordinary_prompt @pytest.mark.parametrize("selection", [ {"LOOPX_EXECUTION_OBSERVATION": "unknown"}, {"LOOPX_EXECUTION_OBSERVATION": "local_probe", "LOOPX_EXECUTION_MODE": "plain"}, + *({"LOOPX_RESEARCH_LOOP": value} for value in ("1", "0", "")), {"LOOPX_EXECUTION_OBSERVATION": "local_probe", "LOOPX_RESEARCH_LOOP": "1"}, ]) def test_raw_worker_rejects_invalid_observation_before_creating_a_wake(tmp_path, selection): wakes = tmp_path / "wakes" - with pytest.raises(ValueError, match="execution_observation|LOOPX_EXECUTION_OBSERVATION"): + with pytest.raises(ValueError, match="execution_observation|LOOPX_EXECUTION_OBSERVATION|LOOPX_RESEARCH_LOOP has been removed"): run_once({"LOOPX_EXECUTION_MODE": "heartbeat", "LOOPX_WAKE_LOG_DIR": str(wakes), **selection}) assert not wakes.exists() -@pytest.mark.parametrize("settings,research_preset", [({}, False), ({"execution_observation": "local_probe"}, False), - ({"research_loop": True}, True)]) -def test_install_records_and_exposes_only_the_selected_observation_producer(tmp_path, monkeypatch, settings, research_preset): +@pytest.mark.parametrize("settings", [{}, {"execution_observation": "local_probe"}]) +def test_install_records_and_exposes_only_the_selected_observation_producer(tmp_path, monkeypatch, settings): pytest.importorskip("harbor") from benchmark.runtime.harbor import BenchmarkCodex, CodexOffline @@ -150,9 +160,9 @@ async def task_exec(**kwargs): command = next(command for command in commands if "> /logs/agent/loopx-install.json" in command) receipt = json.loads(shlex.split(command)[2]) assert receipt.get("execution_observation") == ("local_probe" if selected else None) - assert receipt.get("research_loop", False) is research_preset + assert "research_loop" not in receipt assert receipt["task_entry"] == "loopx-planned" - assert receipt.get("stride_strategy") == ("evidence_conditioned_v1" if research_preset else None) + assert receipt.get("stride_strategy") is None @pytest.fixture @@ -445,17 +455,15 @@ async def no_pending(**kwargs): ) -@pytest.mark.parametrize("producer,objective,research_preset", [ - pytest.param(None, None, False, id="ordinary-default"), - pytest.param("local_probe", None, False, id="ordinary-observed"), - pytest.param("local_probe", "Validate the supplied candidate once.", False, - id="ordinary-objective"), - pytest.param(None, None, True, id="preset-default"), - pytest.param(None, "Compare a validated alternative against the supplied candidate.", True, - id="preset-objective"), +@pytest.mark.parametrize("producer,objective", [ + pytest.param(None, None, id="ordinary-default"), + pytest.param("local_probe", None, id="ordinary-observed"), + pytest.param(None, "Validate the supplied candidate once.", id="explicit-objective"), + pytest.param("local_probe", "Compare a validated alternative against the supplied candidate.", + id="observed-objective"), ]) -def test_goal_bootstrap_preserves_original_objective_and_selected_task_structure( - tmp_path, monkeypatch, producer, objective, research_preset +def test_goal_bootstrap_preserves_original_objective_without_inventing_stages( + tmp_path, monkeypatch, producer, objective ): pytest.importorskip("harbor") from benchmark.runtime import harbor @@ -479,16 +487,9 @@ def test_goal_bootstrap_preserves_original_objective_and_selected_task_structure monkeypatch.setattr(harbor, key, str(value)) env = dict(os.environ) | {"HOME": str(home), "CODEX_HOME": str(home / "codex"), "PYTHONPATH": str(Path(__file__).resolve().parents[2]), "LOOPX_USAGE_PING": "0"} - interrupt_successor = research_preset and objective is not None class Environment: async def exec(self, command, *, env=None, cwd=None, **kwargs): - nonlocal interrupt_successor - if interrupt_successor and "--successor-todo-id" in shlex.split(command): - interrupt_successor = False - # The two Todos are real durable writes; only this transport - # attempt fails before the edge update reaches the CLI. - raise RuntimeError("Simulated successor-update transmission failure") result = subprocess.run(["/bin/bash", "-c", command], env=env, cwd=cwd, text=True, capture_output=True, timeout=90) return SimpleNamespace(return_code=result.returncode, stdout=result.stdout, stderr=result.stderr) @@ -497,94 +498,32 @@ async def upload_file(self, source, target): Path(target).write_bytes(Path(source).read_bytes()) agent = harbor.BenchmarkCodex(logs_dir=tmp_path / "logs", model_name="fixture", - research_loop=research_preset, execution_observation=producer, - stride_strategy=None if research_preset else "off", task_objective=objective) + execution_observation=producer, stride_strategy="off", task_objective=objective) monkeypatch.setattr(agent, "_profile_env", lambda: env) task = ("Start from the supplied solution.py and compare one alternative against it.\n" "First preserve correctness, then minimize operations. Retain useful negative results.\n" "The final comparison must keep both priorities and this trailing requirement.") - def call(*args): - return asyncio.run(agent._loopx(Environment(), list(args), cwd=str(project))) - try: - if research_preset and objective is not None: - with pytest.raises(RuntimeError, match="successor-update transmission failure"): - asyncio.run(agent._prepare_phase(Environment(), task, cwd=str(project))) - partial = call("todo", "list", "--goal-id", agent.goal_id, "--role", "agent") - interrupted_todos = {todo["todo_id"]: todo for todo in partial["todos"]} - assert len(interrupted_todos) == 2 - assert not interrupted_todos[agent._seeded_todo_id].get("successor_todo_ids") - [interrupted_research] = [todo for todo in interrupted_todos.values() - if todo.get("target_key") == "research-continuation"] - assert interrupted_research["status"] == "deferred" - interrupted_lease = call("task-lease", "inspect", "--goal-id", agent.goal_id, - "--todo-id", agent._seeded_todo_id)["lease"] - assert interrupted_lease["status"] == "released" asyncio.run(agent._prepare_phase(Environment(), task, cwd=str(project))) registry = json.loads(Path(harbor._REGISTRY).read_text()) state = Path(registry["goals"][0]["state_file"]) if not state.is_absolute(): state = project / state assert split_state_frontmatter(state.read_text())[0]["objective"] == (objective or task) - worker_env = agent._worker_env(cwd=str(project)) - packet = task_plan_packet(worker_env, [str(cli), "--format", "json", - "--registry", harbor._REGISTRY, "--runtime-root", harbor._LOOPX_RUNTIME]) + prefix = [str(cli), "--format", "json", "--registry", harbor._REGISTRY, + "--runtime-root", harbor._LOOPX_RUNTIME] + packet = task_plan_packet(agent._worker_env(cwd=str(project)), prefix) assert packet["text"].endswith(task + "\n") - if research_preset: - assert call("handoff-mode", "show", "--goal-id", agent.goal_id)["handoff_mode"] == "hard_lease" - listed = call("todo", "list", "--goal-id", agent.goal_id, "--role", "agent") - todos = {todo["todo_id"]: todo for todo in listed["todos"]} - assert len(todos) == 2 - initial = todos[agent._seeded_todo_id] - [research] = [todo for todo in todos.values() - if todo.get("target_key") == "research-continuation"] - assert initial["status"] == "open" - assert initial["successor_todo_ids"] == [research["todo_id"]] - assert research["claimed_by"] == agent.agent_id - assert research["status"] == "deferred" - assert research["resume_when"] == "todo_done:" + initial["todo_id"] - assert agent._task_document in research["text"] - linked_lease = call("task-lease", "inspect", "--goal-id", agent.goal_id, - "--todo-id", initial["todo_id"])["lease"] - assert linked_lease["status"] == "released" - if objective is not None: - assert todos.keys() == interrupted_todos.keys() - assert research["todo_id"] == interrupted_research["todo_id"] - assert linked_lease["version"] > interrupted_lease["version"] - assert linked_lease["idempotency_key"] != interrupted_lease["idempotency_key"] - resume_when = "resume_at:2099-01-01T00:00:00Z" - call("todo", "update", "--goal-id", agent.goal_id, "--agent-id", agent.agent_id, - "--todo-id", initial["todo_id"], "--status", "blocked", "--resume-when", resume_when, - "--reason", "Wait for the scheduled external observation.", "--execute") - call("configure-goal", "--goal-id", agent.goal_id, - "--waiting-on", "external_evidence", "--execute") - else: - assert packet["existing_todos"] == [] + assert packet["existing_todos"] == [] + assert agent._seeded_todo_id is None assert packet["execution_handoff"]["spends_quota"] is False agent._phase_number += 1 feedback = "Released feedback is available; retain the original priorities and constraints." asyncio.run(agent._prepare_phase(Environment(), feedback, cwd=str(project))) - next_packet = task_plan_packet(agent._worker_env(cwd=str(project)), [str(cli), "--format", "json", - "--registry", harbor._REGISTRY, "--runtime-root", harbor._LOOPX_RUNTIME]) + next_packet = task_plan_packet(agent._worker_env(cwd=str(project)), prefix) assert next_packet["text"].endswith(feedback + "\n") - if research_preset: - next_listed = call("todo", "list", "--goal-id", agent.goal_id, "--role", "agent") - next_todos = {todo["todo_id"]: todo for todo in next_listed["todos"]} - assert next_todos.keys() == todos.keys() - assert next_todos[initial["todo_id"]]["successor_todo_ids"] == [research["todo_id"]] - next_research = next_todos[research["todo_id"]] - for field in ("target_key", "claimed_by", "status", "resume_when", "text"): - assert next_research[field] == research[field] - next_lease = call("task-lease", "inspect", "--goal-id", agent.goal_id, - "--todo-id", initial["todo_id"])["lease"] - assert next_lease["status"] == "released" and next_lease["version"] == linked_lease["version"] - if objective is not None: - assert next_todos[initial["todo_id"]]["status"] == "blocked" - assert next_todos[initial["todo_id"]]["resume_when"] == resume_when - assert json.loads(Path(harbor._REGISTRY).read_text())["goals"][0]["waiting_on"] == "external_evidence" - else: - assert next_packet["existing_todos"] == [] + assert next_packet["existing_todos"] == [] assert split_state_frontmatter(state.read_text())[0]["objective"] == (objective or task) assert candidate.read_bytes() == before finally: @@ -593,32 +532,6 @@ def call(*args): restart_effect_runtime(wait_for_exit=True) -@pytest.mark.parametrize("status,resume_when", [ - pytest.param("open", None, id="research-already-open"), - pytest.param("done", None, id="research-completed"), - pytest.param("deferred", "resume_at:2099-01-01T00:00:00Z", id="research-waiting"), - pytest.param("deferred", "todo_done:prior-initial", id="different-initial-stage"), -]) -def test_research_preset_reentry_preserves_existing_research_without_relinking(status, resume_when): - from benchmark.runtime.research_loop import prepare_research_continuation - - research = {"todo_id": "retained-research", "target_key": "research-continuation", - "claimed_by": "fixture-agent", "status": status, "resume_when": resume_when} - calls = [] - - async def call(environment, args, *, cwd): - calls.append(args) - assert args[:2] == ["todo", "list"], "Existing research cannot trigger preparation effects" - return {"todos": [{"todo_id": "current-initial", "status": "open", - "claimed_by": "fixture-agent", "successor_todo_ids": []}, research]} - - owner = SimpleNamespace(_seeded_todo_id="current-initial", _loopx=call, - goal_id="fixture-goal", agent_id="fixture-agent", _task_document="task-phase-001.md") - retained = asyncio.run(prepare_research_continuation(owner, None, cwd=".")) - assert retained == research - assert calls == [["todo", "list", "--goal-id", owner.goal_id, "--role", "agent"]] - - @pytest.mark.parametrize("status", ["ready", "blocked"]) def test_planning_budget_and_blocked_handoff_use_the_real_adapter_run( tmp_path, monkeypatch, status diff --git a/loopx/capabilities/explore/README.md b/loopx/capabilities/explore/README.md index 106b7c9da5..1846d972b8 100644 --- a/loopx/capabilities/explore/README.md +++ b/loopx/capabilities/explore/README.md @@ -10,7 +10,7 @@ are explicit nulls. Its refs and input revision must belong to the enclosing finding. Released `result_ref` requires a matching `result_candidate_sha256`. This records producer evidence, not verified execution or utility. Findings, replan context and the existing research-history detail retain it; omission -preserves old events. See the [local probe producer](../../../benchmark/runtime/RESEARCH_LOOP.md). +preserves old events. See the [local probe producer](../../../benchmark/runtime/EXECUTION_OBSERVATIONS.md). ## At a Glance diff --git a/loopx/capabilities/explore/README.zh-CN.md b/loopx/capabilities/explore/README.zh-CN.md index 2d643c1110..dfa4597444 100644 --- a/loopx/capabilities/explore/README.zh-CN.md +++ b/loopx/capabilities/explore/README.zh-CN.md @@ -7,7 +7,7 @@ 结束原因与证据引用。未测量项显式为 null；版本和引用必须与所属 finding 一致。 已释放结果的 `result_ref` 必须有匹配的 `result_candidate_sha256`。该记录是来源证据， 不认证执行真实性或收益；finding、replan 和既有研究历程详情保留相同信息，缺省仍兼容 -旧事件。参见[本地探针生产路径](../../../benchmark/runtime/RESEARCH_LOOP.md)。 +旧事件。参见[本地探针生产路径](../../../benchmark/runtime/EXECUTION_OBSERVATIONS.md)。 ## 概览 diff --git a/tests/control_plane/test_research_execution_loop.py b/tests/control_plane/test_research_execution_loop.py index f96abe67c9..aa65635492 100644 --- a/tests/control_plane/test_research_execution_loop.py +++ b/tests/control_plane/test_research_execution_loop.py @@ -12,7 +12,6 @@ import pytest -from benchmark.runtime.research_loop import prepare_research_continuation from benchmark.runtime.research_probe import run_probe from canonical_authority_fixture import isolate_sqlite_runtime import test_quota_settlement_cli as fixture @@ -110,7 +109,7 @@ def test_probe_cli_preserves_a_failed_candidate_exit_and_its_observation(tmp_pat @pytest.mark.parametrize("provider", ["file", "sqlite"]) -def test_candidate_stage_leaves_research_work_and_execution_reaches_next_context(tmp_path, monkeypatch, provider): +def test_requested_followup_uses_ordinary_todos_and_execution_reaches_next_replan(tmp_path, monkeypatch, provider): isolate_sqlite_runtime(tmp_path, monkeypatch) project, runtime, registry = fixture._write_fixture(tmp_path) configuration = json.loads(registry.read_text()) @@ -125,18 +124,17 @@ def call(*args, expected=0): "ok", "error", "error_code", "effective_action", "reason", "blocker", "autonomous_replan_obligation")}) return result - async def cli(_environment, args, **_kwargs): - return call(*args) - call("configure-goal", "--goal-id", fixture.GOAL_ID, "--execution-replan-after-turns", "2", "--execution-stride-strategy", "evidence_conditioned_v1", "--execute") - owner = SimpleNamespace(_seeded_todo_id=fixture.TODO_ID, _loopx=cli, - goal_id=fixture.GOAL_ID, agent_id=fixture.AGENT_ID, _task_document="TASK.md") - root = asyncio.run(prepare_research_continuation(owner, None, cwd=str(project))) - original = registry.read_bytes() - again = asyncio.run(prepare_research_continuation(owner, None, cwd=str(project))) - assert again["todo_id"] == root["todo_id"] - assert registry.read_bytes() == original + # The actor records requested work through ordinary Todo operations. The + # observation producer neither creates these stages nor imposes a strategy. + root = call("todo", "add", "--goal-id", fixture.GOAL_ID, "--role", "agent", + "--claimed-by", fixture.AGENT_ID, "--task-class", "advancement_task", + "--action-kind", "validate", "--status", "deferred", + "--resume-when", "todo_done:" + fixture.TODO_ID, + "--text", "Compare the requested local mechanism after validating the supplied candidate.") + call("todo", "update", "--goal-id", fixture.GOAL_ID, "--agent-id", fixture.AGENT_ID, + "--todo-id", fixture.TODO_ID, "--successor-todo-id", root["todo_id"], "--execute") call("todo", "complete", "--goal-id", fixture.GOAL_ID, "--agent-id", fixture.AGENT_ID, "--todo-id", fixture.TODO_ID) binding = ("--goal-id", fixture.GOAL_ID, "--agent-id", fixture.AGENT_ID)