This research adapter calls SForge's native run_agent. Task images, prompts,
submission cooldowns, judge isolation and scoring remain owned by the pinned
EdgeBench checkout. Best-only adds host capture/admission scheduling; native and
blind retain native automatic evaluation. The CLI defaults to the shared
EdgeBench integration branch,
maintained jointly by huangruiteng and loopx-agent. The canonical repository
and branch live in source-default.json.
Run the controller on Linux with Docker and native SForge
iptables permissions; a macOS Docker socket alone does not provide that boundary.
The first CLI entry for a log directory fetches the selected branch into
.edgebench-source/checkout and atomically records its exact commit in
.edgebench-source/receipt.json. Concurrent worker/judge startup shares that lock.
Later entries reuse the frozen checkout without fetching, even if the branch or
repository default changes. Dirty/moved source, conflicting explicit selection,
and existing runs without a source receipt fail before model/container entry.
Use a new log directory to adopt a newer integration revision; never repin an
active or historical cohort.
Prepare dependencies in an isolated environment before starting a study:
python -m benchmark.edgebench.source --log-dir /data/private-runs
python -m pip install /data/private-runs/.edgebench-source/checkout harborThe bootstrap selects this checkout before importing SForge, so an unrelated
installed package or ambient PYTHONPATH cannot silently supply the provider.
Worker runtime-receipt.json includes the verified sforge_source. This records
code selection, not enabled task/scoring options or benchmark qualification.
No dependency installation is performed automatically.
Start the native judge and workers with the same log directory:
SFORGE_TASKS_DIR=/data/tasks python -m benchmark.edgebench.serve \
--log-dir /data/private-runs --port 8080
python -m benchmark.edgebench.run \
--task TASK_ID --tasks-dir /data/tasks --log-dir /data/private-runs \
--run-id UNIQUE_ATTEMPT --worker native-goal --model MODEL --effort high \
--feedback native --submission-cooldown 1800 --eval-interval 300 \
--judge-url http://HOST:8080For an alternative source, pass --sforge-repository URL with --sforge-ref REF
or a full --sforge-revision SHA on the first entry (including preparation).
For example, --sforge-repository https://github.com/ByteDance-Seed/EdgeBench.git --sforge-ref refs/heads/main selects upstream instead. These source options also
apply to the online judge CLI; SFORGE_LOG_DIR is equivalent to --log-dir there.
Use an explicit SHA to replay a known revision. A server in another log directory
is a separate cohort: the launcher cannot attest a remote server's loaded code.
These defaults apply to the LoopX EdgeBench CLI. Direct sforge serve and imported
Python APIs remain caller-managed compatibility entrypoints; use them only with
an explicitly pinned provider environment. Keep offline backfill pinned to the
original scorer, or record regrading under a different source separately.
Maintainers update the shared branch through reviewed integration work and keep
its PR manifest current. Changing the default repository/branch requires updating
the single source file and this protocol; adding a second contributor-specific
default would split the source of truth again.
The five worker profiles are official, single, native-goal,
heartbeat-resume, and heartbeat-explore. official uses SForge's Codex Stop
hook and outer recovery. single executes once with neither. native-goal uses
the shared native Codex Goal transport. The heartbeat profiles use the shared
LoopX scheduler and resume store. heartbeat-explore additionally enables the
existing Explore Graph and analysis-only Explore Harness gates with the
adaptive-resilient profile; this grants no child-agent execution authority.
All profiles use the same explicitly staged Codex binary and model/effort. This permits an updated Codex version with the native scaffold; it is not a claim of identical settings to a published leaderboard. Preserve the version receipt and report hardware, emulation and every protocol deviation with results.
Use the offline runtime payloads documented in
benchmark/runtime/RUNTIME.md. CODEX_AUTH_JSON_PATH
opts into a private local Codex login; it is copied into each disposable trial
and excluded from artifact collection. Host network isolation additionally
permits the OpenAI OAuth endpoint for refresh during long runs.
If the Linux host requires an HTTP proxy, start the restricted TLS relay with
python -m benchmark.runtime.connect_proxy --host BRIDGE_IP --port 9090 --allow-host chatgpt.com --allow-host auth.openai.com, with the host upstream
in HTTPS_PROXY. Pass --api-proxy-url http://BRIDGE_IP:9090 to the trial.
The relay rejects other hosts and ports and never decrypts TLS. Check both an
allowed TLS handshake and a denied external destination before admission;
stop the relay process after the campaign. Do not expose an unrestricted host
proxy to isolated tasks. This transport does not grant general internet access.
LOOPX_INSTALL_HTTPS_PROXY may separately provide the trusted install phase's
proxy when the pinned source installer needs to build frontend assets. It is
passed only to setup subprocesses and cleared before solver execution and native
network isolation. It is never placed in the container's persistent environment.
When the product must run from main while the adapter is under development, set
LOOPX_SRC_DIR / LOOPX_EXPECTED_COMMIT to the clean product checkout and
LOOPX_RUNNER_SRC_DIR / LOOPX_EXPECTED_RUNNER_COMMIT to the clean runner checkout.
Only benchmark/runtime is overlaid from the runner archive; installed LoopX
code remains at the product pin. Both revisions are recorded. Without separate
runner settings, the existing single-checkout contract applies.
For an explicitly selected best-only trial, start the dedicated online judge with the same task
and private log roots. It requires a pinned SForge revision with bounded
asynchronous judge capacity (judge_max_concurrent / judge_max_pending).
Each slot reserves one solver (4 CPUs/16 GiB) and one evaluator (4 CPUs/8 GiB):
SFORGE_TASKS_DIR=/data/tasks SFORGE_LOG_DIR=/data/private-runs \
python -m benchmark.edgebench.online_judge --slots 2 --port 8080The preflight counts existing Docker container limits and retains 2 CPUs/4 GiB
of host headroom. Unlimited containers or insufficient capacity block admission.
Run this in an operator-controlled Docker pool: the host lock coordinates this
adapter's online/offline processes, not arbitrary outside Docker launches. Keep
that reserved capacity available for the cohort. Online policy v2 releases a
run's slot after verified worker removal. An evaluation still in flight keeps
that slot until its native result is terminal; stopping a solver alone does not
free evaluator capacity. Native histories, archives and original registration
identities remain readable, and released registrations reject new captures.
Native/blind use python -m benchmark.edgebench.serve instead. Existing v1 cohorts remain
frozen: this does not upgrade or reclaim slots in an already-running server.
An operator may explicitly pass --allow-resource-overcommit for a monitored
shared Docker pool. Container CPU/memory ceilings then remain enforced, but are
not treated as exclusive reservations. Startup requires available memory for
each evaluator's ceiling, up to 4 GiB per worker, and 4 GiB host headroom; unknown
container limits still block admission. The admission receipt explicitly records
zero exclusive reservation and the monitoring requirement. Monitor actual memory,
CPU pressure and grading latency throughout the cohort and stop affected trials
if sustained pressure makes operation unreliable. This mode preserves per-run
evaluation lanes but does not guarantee dedicated compute or equal latency;
record shared-pool contention when comparing experiments. For a larger explicitly
shared cohort, --shared-startup-memory-gib 16 may replace the per-slot startup
estimate. It requires --allow-resource-overcommit, cannot be below one evaluator,
one worker startup allowance and host headroom (16 GiB with current limits), and
still fails on insufficient available memory or unbounded containers. This is a
startup margin, not sustained capacity qualification: retain continuous load and
grading-latency monitoring and stop affected trials on sustained pressure. Strict
admission and the shared mode without this explicit option retain their defaults.
Then run:
python -m benchmark.edgebench.run \
--task TASK_ID --tasks-dir /data/tasks --log-dir /data/private-runs \
--run-id UNIQUE_ATTEMPT --worker heartbeat-resume \
--feedback best-only \
--model MODEL --effort xhigh --judge-url http://HOST:8080 \
--api-proxy-url http://PROXY_IP:9090Trial timeouts use explicit --timeout → task defaults
→ 64,800 seconds (18 hours). Portfolio Risk Calibration defaults to
43,200 seconds (12 hours); Lean Analysis Proofs defaults to
43,200 seconds (12 hours). Both task defaults apply to every worker and
feedback profile; other tasks retain the 18-hour fallback. These are total trial
budgets, including planning, not per-turn limits.
New trials disable LoopX's background usage sender with LOOPX_USAGE_PING=0.
The shared runtime sets it for installation/control commands and explicitly in
Codex's shell environment policy for every worker profile. Install and worker
receipts record loopx_usage_ping_enabled: false; this records the configured
policy, not a claim of measured model adoption or throughput. This research
setting avoids an unnecessary detached statistics process per CLI invocation.
It does not change product defaults, scoring or feedback, and does not replace
container-provider orphan reaping. Existing trials retain their pinned settings.
SForge loopx-planned has no independent planning timeout. Planning and
execution share the original absolute trial deadline; planning consumes that
budget and a process resume cannot reset it. Runtime and worker receipts record
planning_timeout_seconds: null. The existing 160-second startup/settlement
reserve still controls admission of execution wakes. Shared Harbor retains its
300-second planning default.
The adapter also records the actual solver command's exit code, timeout and
elapsed time in execution-receipt.json. A positive runtime alone no longer
qualifies completion. An expired deadline before execution entry fails; an inner
GNU timeout after verified execution entry is a normal budget stop. Other nonzero
or unknown exits, including an early exit with code 124, are runner_failed and
do not publish final_result.json. Keep their captures and failure evidence;
do not count an initial artifact's score as a solver outcome.
For native Goals with positive runtime, partial_result.json preserves the full
scoring result and its runtime_status / native_goal_outcome without making
SForge's visualizer count the attempt as done. Handled terminal stops retain
final_result.json; its presence does not independently qualify a score. Use
new source pins for new attempts and leave frozen histories unchanged; see the
native outcome contract for rollback and classification.
New native archive-submission trials default to no periodic auto-evaluation
(--eval-interval 0) and 3,600 seconds between accepted active submissions.
The first submission is immediately eligible. The native judge measures cooldown
from acceptance, not result arrival; unused intervals do not bank extra submissions.
These defaults apply equally to all five workers. The startup prompt describes
the resolved cooldown and keeps submission paths runnable as the current promising
candidate, with a separate checkpoint of the best supported version. It does not
require every local proxy metric to improve before exposing a candidate.
The same policy body is appended to the task workspace's AGENTS.md before model
entry, preserving the task's existing instructions. The runner reads it back as
the ordinary worker and records workspace-AGENTS.md in the private trial log.
Resume does not rewrite later task edits. A conflicting/partial policy block,
symlink, non-file target, write failure or readback mismatch stops entry. This
provides durable instructions across continuation and context compression;
file delivery is not evidence that the model read or followed them.
Blind and best-only retain explicit --eval-interval → task defaults →
300 seconds, and their existing 120-second native submission setting. Best-only
requires a positive interval. Native/blind permit zero; negative intervals or
cooldowns are rejected before creating an attempt directory. Explicit values
override these defaults and are passed to SForge and recorded in the runtime
receipt. Native terminal evaluation and native final selection remain unchanged.
An explicitly enabled periodic sampler can add hidden evaluations to native final
selection; it is not a feedback-budget control or an independent holdout.
These are research-adapter defaults. The upstream Codex leaderboard experiment
configuration
uses 1,800 seconds for these tasks. A native-agent official arm does not imply
that all leaderboard settings are reproduced. Existing attempts keep their
recorded sampling interval; new defaults never rewrite a running attempt or
create historical snapshots. Sampling cadence does not set evaluator
concurrency or replace the submission cooldown.
--timeout,
--eval-interval, and --submission-cooldown support explicitly recorded
qualification runs. The shared Harbor defaults are unchanged. Native task
internet policy is retained. Each attempt requires a new output directory.
Heartbeat and native Goal workers have no independent per-call time limit.
The shared worker uses the remaining trial budget, retaining 160 seconds for
startup/cleanup; natural completion determines continuation boundaries. A second
scheduler wake alone does not prove resume: verify another actual model
invocation with the same session identity. Native Goal owns its continuation
without an outer resume loop. Heartbeat profiles now also disable SForge outer
recovery: their LoopX scheduler owns repeated wakes, error backoff and terminal
exit. Once it exits, SForge collects the final artifacts instead of restarting
the scheduler. This changes the heartbeat transport, not LoopX's decision to
continue or end a lane; scheduler exit alone does not prove task success.
The private execution receipt distinguishes wrapper startup from actual work
entry. For LoopX workers, both inner deadline expiry and Docker's outer timeout
must have an execution-entry marker before the runner writes a final result;
a timeout during planning remains a failed attempt. Direct native Codex commands
enter execution without that planning wrapper. Cancellation remains separate.
The official profile retains native outer recovery, and single/native Goal
behavior is unchanged. Record a new runner revision for new attempts; do not
rewrite earlier outer_resume receipts. Explicit total timeouts can support diagnostics,
but short probes are not a prerequisite for running the intended protocol.
--feedback native is the default for new attempts across all five workers,
replacing best-only. This changes new-run defaults, not existing trials or pinned
study manifests, and does not establish a score improvement. Native feedback
retains the provider's complete public result; a score-only API/artifact filter
is not implemented. Freeze feedback content, cooldown, terminal selection and
resource policy in the study manifest before launching a new comparison.
Use explicit --feedback best-only to retain strict-improvement publication, or
--feedback blind as an evaluator-feedback-free control. Harbor is unchanged.
For native archive submission, start python -m benchmark.edgebench.serve and use the runner
without --feedback to select the defaults. Explicit
--feedback native --submission-cooldown 120 --eval-interval 300 restores the
previous native timing for a new attempt. Do not edit running receipts. Game
tasks retain the provider's game prompt and API; the archive-submission cooldown
does not limit game API calls.
The pinned native judge's cooldown is in-memory session state. Server restart or fresh registration does not prove cooldown continuity or safe retry after an ambiguous response. Its capacity bound is aggregate, not a reserved evaluation lane per run. Qualify restart/reconciliation, per-run contention and infrastructure failure accounting before admitting a campaign; a one-hour cooldown alone does not establish fairness. This adapter change does not repair those provider gaps or change scoring, accepted-submission accounting, or final selection.
The shared native task policy also describes accepted-result recovery. Check the
pinned provider's generated sforge-submit --help: clients that support
--result ID can use sforge-submit --result ID --details to read or poll an
already accepted submission without a new evaluation or cooldown reset. If the
acceptance reply was lost, inspect sforge-submit --list and reconcile the
submission identity before retrieving it. Plain --details still
creates a new submission. Older clients may lack result lookup; unavailable
commands, missing sessions and missing results remain unresolved feedback,
and never justify blind resubmission. This does not add restart persistence.
New non-game native trials describe the task, submission tools, feedback and
evaluation rules in the startup prompt and workspace AGENTS.md. The wrapper no
longer prescribes incremental implementation or a best-validated-candidate
workflow. A brief objective/completion reminder retains continued optimization:
implementation, passing checks and submission alone do not complete the objective.
Useful in-scope improvements remain the goal. The wrapper does not repeat a
generic stop/budget/blocker reminder; the original task and runner still own
those boundaries. The wrapper adds no local-test pass or
prior-improvement prerequisite for submission; task-owned requirements still
apply. New LoopX Goals retain the original agent_query as their objective,
including references to task files. Existing Goals and restored checkpoints keep
their captured objective.
Native submission prompts omit the host's periodic sampling explanation, both when the interval is positive and when it is zero. The interval still reaches the runner unchanged; it no longer changes the native prompt. Host capture and selection behavior are unchanged. The prompt retains on-disk manual/terminal submission, recoverable candidate and result/source identity guidance without adding a local-test pass or prior-improvement requirement.
Blind and best-only prompts still describe automatic capture using the resolved runner interval: positive intervals name the independent on-disk sampling and runnable-candidate requirement; zero reports periodic evaluation as disabled. The shared observation describes capture, not feedback access. Blind exposes no official feedback; best-only retains its strictly improving snapshot channel.
Shared loopx-planned entry also drops the adapter's research-strategy prefix.
The product planning packet, including Core first-delivery guidance, is unchanged
and remains a separate experimental variable. Optional execution observations,
stride settings and checkpoint storage guidance retain their explicit selection.
Budgets and deadlines stay with the existing runner. Opted-in native Goal
continuation retains the original task and evaluation rules without requiring a
prior improvement claim before submission.
This is a prompt-policy behavior change for new native attempts across all five workers. Grading, cooldown, selection, blind/best-only task wrappers, game prompts and frozen runs are unchanged. Pin the new runner revision for new trials; cross-version comparisons also change prompt policy and cannot isolate feedback alone. Instruction readback establishes neither earlier feedback nor score gains; those require matched studies.
| Mode | Agent-visible evaluator feedback | Evaluation access |
|---|---|---|
| native | Exact score, pass rate, counts, summary, metrics and failed names; --details exposes per-check messages; --list shows active-submission history |
Agent may submit within the native cooldown/budget; automatic samples are hidden |
| blind | None; public task files, local tests and compiler feedback remain available | Host evaluates fixed automatic samples; agent has no judge route or credentials |
| best-only | Latest strict improvement's complete official result API response and corresponding submitted-source checkpoint; baseline, ties, regressions and unsuccessful evaluations stay silent | Fixed capture cadence; one evaluator and latest pending capture per run; agent cannot request extra evaluations |
Best-only supports non-game, offline tasks with score_first,
valid_then_score or pass_rate_first selection, including maximizing and minimizing scores. It
requires the explicit API-only proxy and a positive sampling interval. Unsupported
selection policies fail with an actionable error instead of silently changing
the task's ranking. Native grading and score selection remain unchanged. The
publisher requires the online judge and runner to share their native log root,
including each submission's original source archive. Ordinary native judges fail
the best-only admission preflight before a solver is launched.
Online scheduling keeps one evaluation in flight and one latest pending capture per run. Every capture is archived with its original capture time and digest; new captures supersede only the pending online candidate. A run cannot fill a shared FIFO with stale samples, and a second run has its own reserved opportunity. This guarantees capacity opportunity, not identical grading duration or equal numbers of improvements. Submission retries reuse an epoch-bound capture identity; a lost response cannot create duplicate evaluations. An unexpected judge restart holds that attempt for reconciliation instead of silently replaying it.
The publisher accepts only that sampler's admitted submission/round identities
and verifies the original source digest. Before publication it fetches the
selected submission's official result, verifies task/submission/score/pass-rate
against native history, and checks the admission epoch and evaluator provenance.
Provenance includes native Python source and task-specification digests, the judge
image key, selection policy and score direction; the image key is not an immutable
image-content attestation. Result-body digests bind hook delivery to that response.
An unavailable or mismatched response preserves the incumbent for retry.
The response is returned in full as provided by the native API, including any
summary, metrics and diagnostics the task supplies. This does not expand access
to raw judge output, hidden files, evaluator code or unrelated trials.
Offline/history-only results cannot
establish the baseline or change the online incumbent. The solver command's exit
pauses capture/delivery; official outer resume reuses the same publisher and lane.
After native worker cleanup, the host sends an epoch-bound release for the exact
run/task, including failures before feedback startup. The private
online-captures/release.json records acknowledgement without credentials.
Release is idempotent and does not cancel an evaluation or erase native history.
If cleanup or release acknowledgement fails, treat release as unconfirmed and
reconcile the exact registration before replacement admission; do not restart
the live judge.
After the entire online cohort ends, stop its judge and backfill all captures:
python -m benchmark.edgebench.offline_scoring \
--trial /data/private-runs/runs/UNIQUE_ATTEMPT/TASK_ID --tasks-dir /data/tasksThe offline command requires terminal solver state, the original task digest and
a final capture. It acquires the same exclusive pool lock, runs native grading
sequentially, and writes a separate offline-scoring/result.json. Superseded and
final captures remain visible at their original times. Failed scores remain
failed; a rerun resumes saved results without silently retrying or rewriting them.
This first implementation reevaluates online captures too, favoring a complete
uniform offline record over evaluation reuse. The original native final result
covers online submissions only; use the complete offline result for post-run
qualification. Neither report by itself certifies integrity or score countability.
No offline result is routed to the worker.
The first completed valid finite score establishes a silent baseline. The task's
native selection policy is the sole improvement criterion among valid scored
snapshots. Only a strictly better native rank updates /opt/edgebench-feedback/latest.json; ties,
regressions, invalid/non-finite results and errors do not update it. Multiple
completed improvements observed together coalesce to the best one. Out-of-order
results compete against the best observed native rank, never against the last result.
For pass_rate_first, a higher pass rate can be an improvement even when its
scalar score is lower; a scalar gain with a lower native rank is silent.
When that policy returns no winner (for example, all pass rates are zero),
polling stays silent and continues normally. A later native winner can improve
the already established baseline; no scalar fallback or evaluator change is used.
Notifications describe the named evaluated snapshot, not the current workspace.
The source archive is the agent's own original submission, with its SHA-256; it
contains no judge output. The adapter never restores files automatically.
Before publishing an improvement, the adapter makes only the disclosed copy
readable and verifies its digest as the ordinary agent worker. A read or digest
failure retains the previous notification and incumbent for retry. Private host
archives retain their original permissions; evaluator and secret directories
are not made public. The hook script and system hook configuration are also
checked for ordinary-worker readability before handoff.
All five workers receive the source-bound official result directly through managed
Codex PostToolUse, SessionStart and UserPromptSubmit hooks. A short synchronous
reader adds additionalContext before the next model request, preserving the
original tool result. It does not force a wake, interrupt active reasoning or
change Stop/continuation behavior. The official worker keeps its native Stop
hook. Delivery is serialized and deduplicated per Codex session; a fresh session
receives the latest checkpoint, while resume receives only a new one. The
reader neither queries the evaluator nor interprets arbitrary packet prose.
Official report text is evaluation data, not executable instructions or authority.
Qualification uses a real native judge and publisher, staged Codex 0.160.0, the actual system hook in a disposable container and a synthetic Responses endpoint: a complete official result arriving during a tool call enters the next model request; a subsequent result enters a resumed request. Earlier Codex versions must be qualified before admission. Inspect session input and subsequent checkpoint adoption separately: injection proves model visibility, not use or score improvement. A missing notification says nothing about failure: evaluation or delivery may still be pending.
A notification looks like this (digest abbreviated for illustration):
{
"schema_version": "edgebench_best_feedback_v2",
"latest": {
"kind": "new_best",
"snapshot_id": "auto-7",
"source_sha256": "<SHA-256>",
"source_archive": "/opt/edgebench-feedback/auto-7-<SHA-256>.tar.gz",
"run_id": "example-run",
"task_id": "example-task",
"online_epoch": "<judge-process-epoch>",
"evaluator": {
"native_source_sha256": "<SHA-256>",
"task_spec_sha256": "<SHA-256>",
"judge_image_key": "example.judge.task:version",
"selection": "score_first",
"score_direction": "maximize"
},
"result_sha256": "<SHA-256 of the canonical official_result JSON>",
"official_result": {
"submission_id": "example-submission",
"status": "completed",
"error": null,
"report": {
"task_id": "example-task",
"submission_id": "example-submission",
"valid": true,
"score": 3,
"pass_rate": 0.75,
"summary": "Task-provided official summary",
"metrics": {"coverage": 0.75}
}
},
"message": "This evaluated snapshot strictly improved the task's native ranking among valid scored snapshots. It may differ from your current files; keep using local validation."
}
}Before the first improvement, latest is null. Native outer resume preserves the
same publisher and session cursor; it does not reset the incumbent. Delivery
failures retry before committing an improvement. The host-only best-only-host
artifacts record file publication and health; never mount or copy them into the
worker. Worker-local session hook receipts under /logs/agent/best-feedback-delivery
record emission, not model acknowledgement.
Each hook reads the current packet while holding its session delivery lock, so a
waiting reader cannot overwrite a newer delivery cursor and replay old feedback.
Treat missing archive/delivery evidence as an unqualified treatment, not as a
successful best-only trial. Public notifications do not include these errors.
The delivery hook uses the runner's staged LOOPX_PORTABLE_PYTHON interpreter,
including for Official and Single workers. A task image does not need to provide
python3 on its PATH. Native and blind plain workers retain their installation
path without this additional staging.
Sparse feedback reduces disclosed information but still supports adaptive tuning;
it does not prove protection against evaluator overfitting or generalization.
Keep native/blind comparisons and any independent final evaluation separate.
First qualify the real network/judge boundary with the opt-in Docker smoke in
benchmark/tests/test_edgebench_feedback_docker.py, then compare matched tasks,
models, budgets and sampling schedules. Synthetic transport validation alone
cannot establish solver uptake or score improvement.
--feedback native preserves native evaluator feedback.
--feedback blind requires a non-game task with native internet isolation and
an explicit API-only proxy IP/port. Its host firewall omits the judge route;
native registration sets the agent submission allowance to zero, task environments
omit judge credentials, and the submission command is removed. Its prompt keeps
the task query, deliverable paths, incremental iteration, runnable files,
best-version objective and no penalty for failed attempts. Local validation
replaces official feedback instructions; submission limits and host evaluation
details are omitted. Host auto-evaluation is unchanged, and neither its existence,
schedule nor results enter the blind prompt. agent_prompt.md records the actual
overridden prompt, including for native Goal and heartbeat workers.
This provider-specific prompt variant lives with the Python SForge adapter; it does not change LoopX goal authority or the official native wrapper. Earlier blind trials used a minimal task-only wrapper and are a different protocol: restart with a new attempt id rather than splicing their curves. Report actual runtime separately from the time limit, and distinguish the agent's final artifact from the evaluator-selected historical best. Prompt parity does not establish feedback isolation or matched experimental validity by itself. Qualify actual denied judge reads/submissions and successful host evaluation before admitting blind trial results; a prompt or unit test alone is insufficient. Configuration receipts prove requested startup settings; actual session model, resume continuity, capability use, evaluator completion and integrity need runtime/post-run qualification before any score is countable. Raw trial outputs and credentials belong outside the public repository.
Full controller capture requires a filesystem that preserves case-sensitive and Unicode-normalization-sensitive filenames. A Linux controller using a shared host volume may not have those properties. New controller-enabled launches probe the export volume before native registration; incompatible storage fails early instead of losing a later replan checkpoint. Capture-disabled runs do not probe it.
Use --checkpoint-export-root /data/private-checkpoints to place full worker
exports on a separately provisioned compatible volume. The root must already
exist and be a physical absolute directory. Each run uses
ROOT/RUN_ID/TASK_ID/checkpoints, which must not already exist; its private runtime receipt records
checkpoint_export_directory. Without the option, the original
LOG_DIR/runs/RUN_ID/TASK_ID/checkpoints location remains in use. The probe checks
filename fidelity, not free space or retention capacity: reserve enough space for
the source export and sealed copy. Credentials and native evaluator storage keep
their existing owners. No worker path, scoring, deadline or capture opportunity
changes. Select this option only for a new run; it does not relocate a running
bank or recover a missed point.
Default change: new controller-enabled native runs automatically keep only
the newest complete point plus explicit pins. After publishing a successful
capture reply, a separate storage worker applies the policy saved at
checkpoints/.checkpoint-retention/policy.json. Pass
--checkpoint-retention-policy PRIVATE_POLICY_JSON to override that policy, or
--no-checkpoint-retention to keep every point without starting storage
maintenance. These options are mutually exclusive. Capture itself remains
explicitly enabled through its existing declaration and controller; runs without
that path do not start retention. A storage-maintenance failure is reported separately in
checkpoints/retention-error.json; it does not turn a successful capture into
a failure or cancel source research. Inspect that error before assuming space
was reclaimed. Successful maintenance writes retention-last.json and the
bank's durable deletion receipts.
Use the checkpoint retention policy and preview to choose uniform or tiered history, protect recent publications and pin every selected continuation/dependency. The policy controls the outer inert artifact bank only; native submissions, archives, counters and cooldown keep their owners. Use separate policy files for separate lineages. Do not import a donor's pins into a different receiver's new bank.
For frozen running donors, use the documented standalone storage watcher rather
than editing or restarting the research runtime. Stop that watcher to disable
its rotation; use --no-checkpoint-retention for new runs. Removing the custom
policy option reselects the default. The existing CLI/Host
qualification remains separate from frontend delivery and scientific utility.
New heartbeat-resume and heartbeat-explore trials replan after 6 settled
effective work Turns by default. Use --replan-after-turns N to set a different
count (1–6); --replan-after-turns 3 retains the previous benchmark default.
Use --replan-after-todos 3 for the previous completed-Todo cadence
as an explicit ablation. The two options are mutually exclusive and require a
heartbeat profile. This default is independent of --task-entry, feedback mode,
and --turn-envelope. Official, single and native-goal profiles are unchanged.
The shared control plane counts settled work Turns, not tool calls or idle wakes; other replan reasons may trigger sooner. Both runtime and worker-profile receipts record the resolved cadence. Only newly launched attempts use the new default; keep existing runs and archived settings pinned.
New heartbeat-resume and heartbeat-explore trials include the same-source
original Goal and acceptance in preselection planning context by default.
--no-planning-horizon-goal disables it for rollback or comparison;
--planning-horizon-goal explicitly enables it. Plain and native-Goal workers reject
this setting. It does not enable stride, Explore or TurnEnvelope.
The adapter writes the shared execution_profile.planning_horizon_goal through
configure-goal, checks the effective readback before work, and records an
resolved boolean in both worker profile and runtime receipt, including when
the command-line option is omitted. Restore follows the same path: the new
attempt's resolved setting overrides the donor setting and must read back as
that exact boolean before work. To preserve an older off treatment, explicitly
pass --no-planning-horizon-goal. Historical receipts without this field remain absent;
report export rejects mismatched or malformed settings.
Compare on/off from the same qualified pre-replan checkpoint with equal remaining budget, source and other settings. Input fidelity does not establish better research or scores. See the shared Goal configuration and rollback.
Add --stride-strategy evidence_conditioned_v1 to a new heartbeat-resume
or heartbeat-explore trial to opt into the existing Goal review strategy.
The runner writes the option together with the resolved cadence and checks
the real Goal configuration before planning or execution. The install,
worker-profile and runtime receipts record an explicitly selected version.
Omitting the option preserves ordinary off commands and receipt shapes;
--stride-strategy off explicitly selects the disabled treatment. Plain and
native-goal profiles reject this option before creating a trial. Unknown
versions also fail before registration or execution.
The Goal review contract defines structured proposals and the three independently controlled axes. Strategy configuration grants no new execution authority, evaluator feedback, budget or submissions. Freeze it with exact sources under a new run ID; do not change live workers. Receipts qualify requested configuration, while actual review, proposal adoption and task gains require trajectory evidence.
Add --turn-envelope to an otherwise identical heartbeat-resume or
heartbeat-explore command to use the product's short normal packet and local
same-invocation full detail. Other workers reject the option before creating a
trial. The default remains off; the runtime and worker-profile receipts record
it explicitly. Use a new run ID and freeze this as a separate context treatment,
keeping objective guidance, task/scorer version, model and budgets matched.
Omit the flag on the next run to disable it. No active experiment is changed by
installing this code. Inspect wake receipts and the saved decisions before
claiming adoption; score countability still requires the usual integrity review.
The Portfolio ten-arm retrospective report
contains recorded scores and per-arm runner settings, with explicit evaluator
cohorts and qualification limits. Use python -m benchmark.edgebench.export_report
to reduce an explicitly selected terminal collection or verify its settings/data
hash bindings. The adapter does not launch jobs, qualify integrity, read sessions,
or upload to a network service. Supplemental settings need publication review;
missing provenance must remain explicit. Canonical experiment-board rows continue
to use benchmark-toolkit's upload-envelope and readback contracts.
For a new export using native ranking, add "native_ranking" to each selected
run's reviewed settings, with exactly these two fields:
{"native_ranking": {"selection": "pass_rate_first", "score_direction": "maximize"}}Copy the effective values from the pinned native task's judge configuration:
selection is score_first, valid_then_score, or pass_rate_first;
score_direction is maximize or minimize. Task names never select a policy.
Unknown/incomplete metadata and a contradictory best_score.higher_is_better
fail export. The adapter binds native history and final best_round,
best_score, and best_pass_rate, then checks the selected sample's score and
pass rate. It exports numeric sample pass rates and the selected pass-rate metric;
it does not rerank submissions or call a scorer. A selected score may therefore
be below another sample even when scores are maximized.
Without this metadata, the historical scalar-maximum verification path remains;
immutable Portfolio snapshots need no migration. Neither path requires SForge
for export or standalone offline verification. --verify checks file bindings,
direction consistency, and selected-sample witnesses, not ranking optimality or
the truth of supplied policy metadata. The recorded task/source hashes do not
prove that metadata matches the private task configuration; that provenance
still needs operator review. This is an S11/E2 terminal-report repair, with no
change to native scoring, integrity qualification, or study promotion.