diff --git a/benchmark/README.md b/benchmark/README.md index 7cf8ce709f..61a27b4067 100644 --- a/benchmark/README.md +++ b/benchmark/README.md @@ -113,3 +113,628 @@ The [DeepSWE GPT xhigh v1 archive note](deepswe-gptxhigh-versions.md) registers preserved for inspection; the companion notice governs their current evidence status. Neither archive category is the architecture for new work or grants runner, verifier, submission, or publication authority. + +## Inert checkpoint capture + +This is a developer prerequisite for the [research checkpoint contract](../docs/architecture/rfcs/hierarchical-agent-stride-control-v0.md#77-research-state-branching-and-matched-continuation), +not a qualified research continuation. The EdgeBench CLI accepts an explicit +`--checkpoint-request /path/to/request.json` **host** file and stages its exact +bytes into the heartbeat worker's protected control directory. The declaration +is validated before trial creation/registration and its digest appears in the +runtime receipt. The programmatic `BenchmarkCodex` and heartbeat `SForgeWorker` +adapters also accept `checkpoint_request_path` to an already staged normalized +absolute **guest** file. Ambient host variables do not enable capture; omitting +these options disables the hook. Official and native-goal profiles reject them. + +The request contains exactly these fields; select only authorized private +artifacts and exclude credentials and hidden evaluation material: + +```json +{ + "destination": "/logs/agent/checkpoints/research-point", + "components": [ + {"id": "code", "source_path": "/workspace/solution.py"}, + {"id": "evidence", "source_path": "/control/source-evidence.json"} + ], + "point_metadata": {"purpose": "inert source capture"} +} +``` + +Caller component IDs are preserved. Automatic LoopX-state, research-context and +session-binding participants choose an unused local ID when their preferred +name is already declared; source-path deduplication and byte witnesses still apply. + +The destination must be a new direct child of the trial log owner's +`checkpoints` directory. Capture runs after a successful execute wake's original +settlement, before the next wake. It uses the remaining phase budget with a +30-second capture ceiling, preserves SQLite's committed WAL through the existing +backup owner, rejects observed source changes/symlinks and never overwrites a +destination. Capture failures are recorded separately in the wake receipt; +they do not replay work or change its settled outcome. The ceiling bounds +publication eligibility and RPC waits, not an operating-system I/O cancellation +guarantee. Capture cost remains part of the trial's elapsed time. + +For readback, call the existing `effect_runtime_result` adapter with method +`runtime.checkpoint_artifacts`, action `verify`, the captured `root` and the +receipt's `expected_sha256`. Action `materialize` with those fields and a new +`destination` verifies and copies the unit into isolation. Neither action adopts +state or starts execution. The new `loopx_checkpoint_artifact_set_v0` manifest +is an internal artifact contract; `point_metadata` is unverified evidence. +`research_point_qualified`, `execution_authority_granted` and `privacy_certified` +remain false. Integrity checks cover the declared members and observed stable +source boundary, not adversarial filesystem mutation or a cross-owner transaction. +Native session reconstruction, effect reconciliation, fresh authority admission +and evaluator cooldown/opportunity inheritance must pass before matched continuation. + +For pre-decision capture at a **fresh replan projected at the start of a heartbeat +wake**, add `"trigger": "fresh_replan"` and set `destination` to the trial's +`checkpoints` directory itself. The Host preloads that wake's original quota guard +once, the typed owner validates its committed semantic replan identity, and the adapter +captures from the latest successful, settled work wake before invoking the model. +Work selection may still be owed on this new guard: the exact registered Agent, +Goal, Turn and semantic obligation must match, but inert capture does not invent +a causal settlement identity or authorize delivery before selection. An already +selected guard retains its existing settlement-identity checks. +The model reads the complete preloaded guard and follows its existing selection, +required-read, recovery and settlement contract. Each obligation has one immutable +destination; a repeated read verifies that point instead of replacing it with later +work. The after-wake hook cannot substitute a post-decision state. Capture failures +remain explicit and do not settle or complete the new work. +The first attempt is also retained: if it fails and the model proceeds, retries +cannot substitute a later state for that missed pre-decision point. +Losing, denying access to or corrupting the staged declaration records a capture +failure while the ordinary wake continues. The Host still obtains and delivers +the original quota guard; an unavailable admission owner stops work as before. +If that guard projects a fresh replan, the missed first attempt is retained even +without readable declaration bytes, so repairing the file cannot capture later +state under the same obligation. Subsequent distinct replans remain eligible. + +A Host can now explicitly hold a **mid-wake quota response** using +`quota should-run ... --decision-output-root CAPTURE_ROOT +--decision-handoff-socket SOCKET`. The socket must already be listening in a +private physical directory owned by the same user (Unix only). After writing the +complete original decision, the CLI sends one newline-delimited JSON object +`{"path": ".../decision.json", "sha256": "..."}` and withholds successful output +until it receives exactly `{"released": true, "sha256": "..."}` for those bytes. +The existing selection command preserves both transport options. Ordinary CLI +output and admission remain unchanged when the option is absent; this option +does not enable TurnEnvelope. + +`benchmark.runtime.checkpoint.held_quota_decision` is the matching local Host +context: it validates the declared capture root/digest and asks the existing +TypeScript replan selector for capture intent, then releases the original +response on normal context exit. A Host exception, disconnected/lost reply, +changed capture or the 120-second transport deadline fails the call while +retaining the original admission and captured decision. Recover that original +identity; do not obtain a replacement guard to reconstruct a missed point. + +An explicit native CLI caller can pass its live `child_process` handle as +`native_process` to `held_quota_decision`. The Host interrupts its owned process +group, requires graceful leader exit, and reaps descendants before calling the +selector or entering the capture body. Failed interruption stops that capture; +it does not turn an unfinished native history into a completed cursor. After a +successful context exit, `continuation_body` contains the original guard for an +explicit resume of **that source session**. The killed tool remains aborted in +native history; the packet is supplied on a new native turn and is never rerun. +A dead source socket needs no acknowledgement in this path. Omit the process +argument to retain the original output-only transport. + +A real native CLI regression holds this interrupted boundary beyond the tool's +yield interval and verifies retained context, explicit original-packet delivery +and one guard receipt. This qualifies interruption/recovery after the Host +receives the packet, not absence of earlier inference, independent writer +quiescence, or an automatically captured research checkpoint. + +The heartbeat worker now installs that listener when its explicit checkpoint +request uses `fresh_replan`. The renderer carries the private handoff endpoint +into guard, selection and reentry commands; this path also uses the existing +captured TurnEnvelope. With checkpointing off the renderer, process lifecycle +and model-visible output stay unchanged. Ordinary guards and repeated replan +obligations are released without restarting the native CLI. On a first replan, +the worker selects through the typed owner, interrupts the source, records its +session, and captures explicitly declared inert files and optional native history. Failed optional capture +still resumes the source with its original packet; altered packet bytes or lost +session identity fail closed. All resumptions share the original wake deadline. + +This is **not a qualified complete research checkpoint**. Interrupted native +capture is qualified on the tested Linux profile. On Darwin, a first mid-wake +replan with `native_session` is explicitly refused **before** signalling the +native CLI: its original tool result continues and the missed point is recorded +once. The observed Darwin Host can omit the pending command from its interrupted +history; delayed reads and leader-only signalling did not recover that cursor. +Completed-wake capture remains separate. No history/output predicate is relaxed. +On the qualified profile, the actual worker can seal an explicit interrupted cursor: all retained +pages, the failed final guard command with no released output or later visible +item, its parsed CLI routes/Turn identity, and the original held decision bytes. +A second matching guard is ambiguous and rejects. The exact renderer Turn +assignment is supported; other shell programs and expansions are unsupported +by this conservative provider profile. History is +rechecked against the byte witness before publication. A native fork reconstructs +that entire captured prefix, including the aborted tool, without using a completed +Turn as a substitute; completed-cursor checks stay unchanged. Lost fork replies +reuse the existing one-attempt recovery path. + +Full capture uses the external controller path below. Native-only history plus +declared files remains an inert subset. An interrupted cursor can now enter the +existing explicit session-binding admission for a distinct registered destination +Goal/Agent. Its pending source guard remains historical input; the receiver must +obtain a new work admission and lease. Binding alone grants no execution or +evaluation opportunity. +Remove the checkpoint request to disable automatic handoff; a direct caller can +omit the socket option to disable only the transport. + +Add `"native_session": true` to include Codex's completed native cursor, all +retained history pages, rollout/compaction records and native index/history SQLite +stores. The adapter verifies the native session's actual workspace, reads its full +cursor before the byte probe and reads it again after that probe. A changed cursor, +configuration, source session binding or participant inventory rejects before publication; the artifact +owner then verifies the same physical witness through capture. This prevents later +Turns on the selected source session from being sealed under an earlier cursor. The adapter excludes +credential files, logs, memories and the symlinked skill installation. Record the +effective configuration digest and Host version; tools/skills and credentials need +their existing installation/authentication owners. Native state is opaque and its +storage layout remains a pinned provider qualification. + +`benchmark.runtime.native_checkpoint.fork_native_checkpoint` verifies a unit and +reconstructs a distinct native session in a **new isolated environment preserving +the captured guest paths**, with matching configuration and Host version. It checks +the full history digest and cursor, starts no model and registers no LoopX binding. +A failed or partial reconstruction is not an admitted session. This is a session +participant prerequisite, not full research continuation or arbitrary path import. + +An explicit controller can then prepare that participant through the existing +registry and session owners. Prepare a fresh governed benchmark Goal and +distinct Agent identity in the isolated guest; set its destination `LOOPX_REGISTRY`, +`LOOPX_RUNTIME_ROOT`, `LOOPX_GOAL_ID` and `LOOPX_AGENT_ID`, plus the qualified +`CODEX_BIN`, `CODEX_HOME`, `LOOPX_PROJECT`, `MODEL_NAME` and `REASONING_EFFORT`. +The captured session profile must match the destination, and native automatic +Goals must remain explicitly disabled. Fork deferral preserves native Goal data; +it does not remove historical authority. The controller environment and explicit +configuration are not exported as authority or credentials. + +```sh +python -m benchmark.runtime.native_checkpoint \ + --root /private/captured-point --expected-sha256 CAPTURE_DIGEST \ + --config-path /private/qualified-config.toml --sandbox read-only +``` + +This command starts no model and claims no work. It stores the verified fork for +the registered destination Goal/Agent, without replacing an existing binding. +The registry owner's fence guards the configuration check and binding write; +this preserves the existing profile's lifetime semantics. The lifecycle-only +`source_session_v1` profile is rejected by its owner because it cannot run +governed benchmark work. The binding atomically retains the captured artifact digest. +If the binding committed but its controller reply was lost, repeat this exact +command: it reads the original binding and verifies current Goal/Agent/profile, +Host/configuration and complete native history, returning the same admission +without another fork or model call. Changed or unmarked bindings reject; this +cannot retrofit recovery onto older bindings. Once native work has advanced, use +the existing continuation rather than retrying admission at the old cursor. +Provenance survives ordinary observations of the same session, but does not follow +a replacement session, profile or Goal lifetime. Before dispatching the native +fork, the controller durably records the artifact/destination basis and original +native thread identities under the existing process lock. If the fork reply is +lost **before binding**, retry the same command in the same isolated environment. +It admits only one new native identity whose native parent, workspace and complete +history match the captured source. It never dispatches another fork on recovery. +Missing or multiple outcomes, changed destinations/configuration and advanced +history remain unresolved and require reconciliation; do not erase the attempt. +At the qualified Host version, the interactive `thread/list` omits a fresh fork +until its first user event. Recovery therefore locates opaque IDs through a +read-only native SQLite index, then verifies them through `thread/read` and +`thread/turns/list`; it never edits native rows or treats the index as authority. +Omit +this command to leave the unit inert. Code/research authority, unresolved effects, +native evaluator admission and remaining whole-trial budget still require their +owners before executing a research continuation. This source-checkout controller +entrypoint is a partial participant journey; it does not ship a frontend restore +operation or qualify a complete research point. + +The explicit real-Host tests below also qualify this CLI and the worker's native +resume transport: a local static provider receives the retained conversation, +runs a read-only verification of materialized code/negative evidence and completes +a new native Turn plus its original LoopX guard/writeback/spend. Later source +candidate/evidence and original native participants remain unchanged. These tests +swap fixed paths only after their synthetic source is quiescent; they do not prove +complete environment isolation. Goal configuration change, unsupported lifecycle +profile, profile change and duplicate binding are rejected. This proves protocol adoption +and synthetic continued work, not model research utility or native score gains. + +For the native feedback protocol with **no automatic evaluations**, start an +explicit checkpoint-aware native judge: + +```sh +python -m benchmark.edgebench.checkpoint_judge --checkpoint-capture \ + --tasks-dir /path/to/tasks --log-dir /path/to/private-run-logs +``` + +This adapter's supported SForge source is the public compatibility fork at +[`0e20fb499ee248f012d6467c00b3dec00a29fe85`](https://github.com/huangruiteng/EdgeBench/commit/0e20fb499ee248f012d6467c00b3dec00a29fe85). +It exposes the existing native judge/admission owner; the upstream `sforge==1.1.0` +distribution does not provide that seam. Enabling capture against an unsupported +provider fails before adding routes. This pin qualifies the capture prerequisite, +not an upstream release or restored admission. From this LoopX checkout, prepare +an isolated Python 3.13 environment and verify the actual imported source: + +```sh +CKPT_PROVIDER_ROOT=$(mktemp -d) +git clone https://github.com/huangruiteng/EdgeBench.git "$CKPT_PROVIDER_ROOT/provider" +git -C "$CKPT_PROVIDER_ROOT/provider" checkout --detach 0e20fb499ee248f012d6467c00b3dec00a29fe85 +uv venv --python 3.13 "$CKPT_PROVIDER_ROOT/venv" +uv pip install --python "$CKPT_PROVIDER_ROOT/venv/bin/python" \ + -e '.[test]' -e "$CKPT_PROVIDER_ROOT/provider" 'httpx>=0.28,<1' +"$CKPT_PROVIDER_ROOT/venv/bin/python" -c 'import sforge,loopx; print(sforge.__file__); print(loopx.__file__)' +"$CKPT_PROVIDER_ROOT/venv/bin/python" -m pytest -q benchmark/tests/test_checkpoint_judge.py +# Linux with an owned disposable Docker daemon; grades only synthetic fixtures. +LOOPX_EDGEBENCH_DOCKER_SMOKE=1 "$CKPT_PROVIDER_ROOT/venv/bin/python" -m pytest -q \ + benchmark/tests/test_edgebench_feedback_docker.py::test_checkpoint_opportunity_uses_native_admission_and_real_grade +``` + +The Docker journey checks a real grade, remaining rounds/cooldown, incumbent bytes, +and the cold-registration counterexample; it does not restore evaluator authority. +Native session qualification is separate: the tested Host is `codex-cli 0.160.0`. +Retain the complete paginated history/compaction and actual native stores from an +authorized completed source, then run capture and fork in isolation with that +same binary and fixed guest paths. Compare every retained Turn ID and the full +history digest, including exclusion of later history; mocked pages alone do not +qualify this participant. No model call or LoopX registration is implied by fork. + +The current cursor/workspace regressions also exercise `codex-cli 0.162.0-alpha.2` +through real native writer/readback and the actual CLI/typed artifact owners. A +static localhost response stream supplies synthetic source Turns; this tests the +storage protocol and capture boundary, not model behavior or research utility. +Run with an explicit native binary; ordinary runs keep the qualification skip: + +```sh +LOOPX_NATIVE_CHECKPOINT_BIN=/path/to/pinned/codex python -m pytest -q \ + tests/control_plane/test_research_checkpoint_capture.py::test_real_native_cursor_must_match_sealed_store_basis \ + benchmark/tests/test_native_checkpoint.py::test_real_native_workspace_readback_rejects_foreign_code_basis +``` + +Its default-off factory otherwise preserves the native routes. With +`"evaluation_opportunity": true` in the worker declaration, the capture hook +reads that registration's released history and complete public result details, +next round counters, accepted submission clock, remaining cooldown, task/native +source identity and original trial deadline. It also preserves the native-selected +incumbent's opaque submission archive, separate from the current working candidate. +Automatic participants prefer `evaluation_opportunity` and `evaluation_incumbent` +and allocate unused IDs when a caller already owns either name. The sealed +`point_metadata.evaluation_opportunity` maps `component_id` and +`incumbent_component_id` to the retained evidence and archive; the latter is null +when no native incumbent exists. The source witness follows this same mapping. +Native ranking chooses its original released round; +the result identity, archive digest and byte count bind its download to the same +judge epoch and consumed-round state. Missing or moved selected bytes reject +capture; no selected result is explicitly recorded as absent. The transport never +extracts archive members, overwrites a prior participant or silently accepts a +truncated response. Its checked file witness must match the descriptor before +the artifact owner publishes the unit. The provider uses the existing +native admission/history locks and rejects an outstanding grade, unreconciled +consumed round, missing result details, automatic-evaluation history or closed +registration. It never returns tokens or hidden task material and never grades, +registers or refills an opportunity. Download is limited to the same registration +and a matching capture digest; source movement requires a new owned boundary. +The local `sforge_checkpoint_opportunity_v0` +artifact records these facts; it grants no restored admission. Disabling capture +means omitting both the judge flag and worker declaration option on a new run. + +For an isolated single-Goal trial, `"loopx_state": true` adds the existing +state-backup owner's inventory: registered active state, project research state, +runtime authority/effect/quota journals and registry. The declared worker and +physical routes must match that registry; missing registered state, incomplete +inventory, symlink routes and narrower overlapping components reject capture. +File and SQLite authority providers remain inert artifacts after materialization; +copying their old leases or settlement records grants no new execution authority. + +With this option, capture also retains the **complete paged canonical Todo read** +at one provider revision when its source coordination is promoted, as a private +artifact. That context also preserves the complete validated Explore event log +(nodes, edges and findings, including dead ends) and all indexed vision/checkpoint +records, including their original stride decisions. These are historical inputs, +not imported destination graph, vision or effect authority. Empty histories are +explicit; malformed rows and incomplete appends reject capture. The original +source-file digests/absence must match the artifact owner's source witness, so an +append between the context read and sealing cannot mix research points. It rechecks the canonical head and exported bytes before sealing; +missing promoted authority or a moved revision fails this optional capture while +ordinary work continues. An unpromoted source keeps the previous inert inventory +capture and explicitly records canonical context as unavailable; it cannot use +this context-adoption path. Leases are excluded from the read, while historical +claim/status fields remain evidence. Declarations without `loopx_state` keep +their previous path. + +After separately admitting the checkpoint's native session, an operator can +adopt this retained context on an **already claimed and leased destination Todo** +whose note is empty: + +```sh +python -m benchmark.runtime.research_checkpoint \ + --root /private/checkpoints/point --expected-sha256 CHECKPOINT_SHA256 \ + --source-todo-id SOURCE_TODO --todo-id DESTINATION_TODO \ + --operation-id CONTEXT_OPERATION --expected-revision CURRENT_REVISION \ + --lease-key CURRENT_LEASE_KEY --lease-version CURRENT_LEASE_VERSION +``` + +Supply the destination `LOOPX_PROJECT`, `LOOPX_REGISTRY`, `LOOPX_RUNTIME_ROOT`, +`LOOPX_GOAL_ID`, `LOOPX_AGENT_ID`, `CODEX_BIN`, `CODEX_HOME`, `MODEL_NAME`, +`REASONING_EFFORT` and `LOOPX_CODEX_SANDBOX` explicitly. The Goal and Agent must +differ from the captured source. The session owner must admit the distinct +current native binding and matching execution profile. Before prepare and adopt, +readback must match the captured complete native history, Host and configuration; +a branch that has already continued cannot adopt context from this older point. The adapter uses the existing continuation prepare/adopt owners, +then reads back the canonical note and Markdown projection. It retains the full +source read in a private workspace-relative file, with its checkpoint and source +revision references; current requirements, status, claim and lease stay governed +by their existing owners. The operation starts no model and submits no candidate. +The option is disabled by omitting the command; an adopted note is ordinary +current Todo context and can be changed through its owner. + +This is **Todo context adoption**, not canonical graph/vision/effect reactivation +or a complete research continuation. A deterministic local-provider qualification +uses the real native Host fork and resume, CLI and canonical continuation owner: +a tool reads the retained candidate, negative Todo/Explore evidence and vision, excludes +later source evidence, then settles current governed work. This proves mechanical +adoption and use, not model judgment or research benefit. It does not replace +current Todo requirements with source requirements, reactivate source claims, or +certify Host budget and evaluator participants as part of the same point. If prepare/adopt commits but +its response/projection is uncertain, repeat this command with the **same original +operation ID, expected revision, checkpoint and lease proof**. It replays the +existing prepare/adopt receipts and reconciles projection; it does not create a +new note or reset execution authority. Changed notes, artifacts, native history, +leases or request identity still refuse recovery. Do not substitute a new session, +claim, revision or context operation to conceal uncertainty. This qualifies only +context-command recovery; joint participant recovery and the packaged frontend +journey remain open. + + +The composed local qualification also seals the native evaluator capsule with the +conversation/code/context artifact. An actual resumed Codex tool reads captured +research and public feedback, computes a synthetic candidate from captured code, +and submits through the native HTTP owner. It retains consumed rounds and the +active cooldown, rejects a further submission at the inherited limit, and enters +through the existing SForge Host phase check with the captured remaining duration. +Later source evidence stays excluded and the source service remains unchanged. +The fixture quiesces participants itself, uses a local deterministic model and +synthetic grading, and does not activate a destination Explore graph or old +vision/effects. It qualifies this composed mechanical path, not an automatic +cross-owner barrier, ambiguous activation recovery, isolated Linux solver restore, +or research utility. Those remain requirements before scientific continuation. + +Declare authorized workspace code/research files, effective configuration and +installed source/profile pins as non-overlapping components. The native and +opportunity participants do not imply that omitted state was saved. Every source +remains private; inspect the verified inventory before qualifying a complete point. +Automatic native-store participants allocate unused artifact IDs without changing +caller IDs. The sealed `native_session.store_members` mapping binds those IDs to +the existing allowlisted native filenames. Reconstruction accepts earlier fixed-ID +artifacts, rejects duplicate/undeclared mappings and keeps source-history checks. + +Fresh replan can arise at Host preflight or inside a running native wake. The +explicit handoff retains the original guard and rejects replacement by a later +point. Full-state capture additionally requires the controller writer barrier +below. Capturing that point does not admit a destination session or prove useful +continued research. + +### Controller-only native capsule transport + +The read-only participant above does **not** implement the newer native capsule +API. The separate controller adapter targets +[EdgeBench PR #23](https://github.com/ByteDance-Seed/EdgeBench/pull/23), pinned at +`57dce5dd1fded9d5d5cbd44fa59c59d27a068d62`; this is proposed source, not a released +upstream version. Start that pinned native judge with `sforge serve --checkpoints`. +Its private controller configuration file has exactly `url`, `admin_secret` and +`bank_root` (the native log directory's `checkpoints` directory). Keep this file, +the bank and controller process outside every solver's filesystem/network authority. +An owner-only mode is a storage precaution, not proof of solver exclusion. + +```sh +python -m benchmark.edgebench.checkpoint_controller \ + --controller-file /private/source-controller.json capture \ + --token-file /private/source-token --capture-id 0123456789abcdef0123456789abcdef \ + --destination /private/new-native-export +python -m benchmark.edgebench.checkpoint_controller \ + --controller-file /private/fresh-controller.json restore \ + --root /private/new-native-export --expected-sha256 EXPORT_DIGEST \ + --run-id isolated-branch --result-file /private/fresh-registration.json +``` + +Use a stable declared capture identity for one point; retries read its original +native state, including after the source improves. The existing typed artifact +owner seals and verifies the native capsule. The native owner +alone validates evaluator compatibility, ranking, released feedback, consumed +rounds, resources and retained cooldown, then creates a fresh registration. +The adapter never grades, extracts incumbent archives or adds controller +configuration/live registration token records to the export. Opaque incumbent +bytes and released reports still require private handling; privacy is not certified. +Restore writes its new token only to the explicit +private result file; stdout contains its path. Wrong authority, corrupt bytes, +aliases, foreign bank entries, source-run reuse and duplicate native admission +fail without a fresh evaluation. Redirects cannot forward the admin secret. +After an uncertain restore response, repeat the same controller `restore` command +with `--recover`, preserving the same capsule digest, run identity, result-file +and optional phase-deadline-file. Recovery reads the original native activation +through `/api/v1/checkpoint/restore-result`; it does not allocate a registration, +replay grading or reset cooldown. A successful result may be read back again. +An absent original activation or lost live authority refuses recovery. Do not +switch identities or perform ordinary fresh registration as a retry. + +For an explicitly owned Docker worker, `capture-worker` can additionally freeze +its writers while copying declared files and capturing the native capsule. Supply +`--worker-request` with a physical JSON file containing the full `container_id`, +`components: [{"id": "code", "source_path": "/task/code"}]`, and the separate +`phase_deadline_path` inside that container. Use the same controller, token-file, +capture-id and destination arguments as `capture`; no phase file is read through +an executing container process. The controller uses the Docker daemon's +pause/archive/unpause path, then seals the copied files and nested native capsule +with the existing typed artifact owner. Component IDs are preserved when they +collide with the automatic native capsule name. Neither Docker access nor judge +administration is passed into the solver. Omit this command to retain ordinary +capture behavior; there is no automatic activation or retention policy. + +The container must initially be running and unpaused. Cancellation, failed file +reads, rejected native capture and a lost pause response attempt to release only +this operation's barrier. Failure to resume the worker is reported and publishes +no outer artifact. An initially observed pause or a definite Docker conflict +rejects without unpausing. Controlled captures on the same host/user share a +daemon-ID/container-ID lock under `~/.cache/loopx/checkpoint-workers`, across +controller banks and Docker endpoint aliases. External actors do not participate +in this lock: an uncertain transport outcome retains the original attempt and +cleanup obligation, not proof of exclusive ownership against arbitrary actors. +The first attempt owns the capture identity; a failure cannot silently pair +later files with an older native point. Missing paths, archive links, traversal and special members reject. +Declare only the intended private code/state stores: a whole home can contain +credentials, and neither archive filtering nor a freeze is privacy certification. + +This explicit command produces an **inert joint file snapshot**, still +`research_point_qualified=false`. It does not snapshot process memory, freeze +external writers or activate canonical research state. Ordinary native restore +rejects the outer unit; callers must use the declared native capsule and existing +fresh-authority admission paths. + +For automatic capture, the heartbeat declaration must include `"trigger": +"fresh_replan"`, `"native_session": true`, `"loopx_state": true`, +`"evaluation_opportunity": true`, and `"controller_handoff": true`, in addition +to its authorized code components, destination and metadata. Pass both +`--checkpoint-request REQUEST.json` and `--checkpoint-controller-file CONTROLLER.json` +to `benchmark.edgebench.run`, with native feedback. The controller file remains +host-private; only the declaration enters the worker. Missing or mismatched opt-ins +reject before trial creation. The backend binds the supervisor to its actual +container and native registration, rather than accepting a solver-supplied token. + +The worker prepares native history, the original decision, canonical research +context and its file witness while holding the model boundary. The controller +pauses that worker, rereads the exact live handoff and copies declared sources +through Docker archive. A differing source witness or expired/removed request +rejects **before** native evaluator capture. Ordinary permission bits survive safe +archive extraction so private authority directories do not falsely appear changed. +The controller seals the frozen files with incumbent, used opportunities, cooldown +and remaining Host phase. It releases the barrier before publishing a private reply; +the source continues from the original packet and deadline. Optional capture +failure records a missed point, never a later replacement or a new model/eval budget. +A barrier that cannot be released remains an explicit controller recovery problem. + +A real native CLI/worker plus isolated Docker/native-HTTP qualification covers the +mid-wake path, complete canonical context, conserved evaluator opportunity and +original source continuation. It uses a deterministic local model and synthetic +grading. A fresh isolated Linux receiver can bind that interrupted native prefix, +adopt captured research context on its currently leased Todo, restore native +evaluation/Host allowances and run a new admitted Turn. Qualification covers a +resumed tool consuming retained code/context, producing a synthetic candidate, +receiving its next native grade and settling the destination Turn exactly once. +Lost binding replies and native activation readback retain their original +identities; wrong-source/digest and advanced-history admission reject. No old +lease, quota, timer or token is activated. Scientific restart still requires the +chosen installed source/profile and experiment inputs to be admitted; this +mechanical fixture proves neither model judgment nor policy benefit. +The completed-source qualification also runs six actual settled native work wakes +without fabricated history, with stride enabled and Explore planning separately +off/on. The resulting periodic review is captured before policy selection; a +recorded grow decision is followed by a larger synthetic work slice and normal +settlement. This proves mechanical composition, not model-selected strategy +quality. Retained research remains inert destination context: fresh graph, +vision/policy review adoption and same-point scientific comparison remain open. +Omit the controller file and `controller_handoff` on a new run to disable automatic +controller capture; removing the checkpoint request disables capture entirely. +Existing artifacts remain inert, and the source trial is not restarted by disabling it. + +An explicit **SForge phase-clock participant** preserves a declared Host file's +remaining duration alongside native opportunities. Add +`--phase-deadline-file /private/source-phase-deadline` to capture, and +`--phase-deadline-file /private/new-host-control/phase-deadline` to restore. +Supply the original physical integer deadline from the admitted source Host, +through an authorized controller read route. The destination file must be new, +outside the artifact and native bank, and installed at the Host's existing +phase-file route before starting its ordinary entry command. + +The sealed deadline and native capture timestamp determine the remainder; the +receiver recomputes it. A point with two hours remaining keeps that duration +after an eight-hour offline gap. It does not reuse the old absolute epoch or +initialize another configured twelve-hour trial. Native admission still owns +consumed rounds and remaining cooldown. Host time begins immediately before +native activation, including its transport time; the integer phase format can +round the retained allowance down by less than one second. The existing worker's +startup/settlement reserve remains in force. + +A native-only point or visited native/Host capture attempt cannot acquire a later Host clock. A capsule +with this participant requires the matching explicit restore option; omitting it +cannot discard its allowance. Existing/aliased files, source movement, changed +sealed bytes and exhausted work allowance refuse activation. An uncertain native +response leaves the reserved phase file empty and unusable by the ordinary Host +entry, preventing a fresh-budget fallback. Before native activation, the adapter +syncs a private sibling `.restore.json` containing the exact attempt +binding and destination deadline, without tokens. Explicit `--recover` uses that +original deadline: recovery delay consumes the remaining allowance, and expiry +refuses continuation. Result/phase locks prevent concurrent controller recovery; +partial records, aliases, changed identities and missing records fail closed. +Do not edit, copy independently or replace the attempt files to retry. The record +is local provider state, not part of the portable capsule or a new authorization +source. This recovery is explicit, with no automatic replay or new run. Omit the +option on **new captures** to retain native-only transport. Full Host source attribution, source/native clock alignment and the +joint session/research/effect barrier remain qualification requirements. + +This is an independently usable **native participant transport**, still partial +relative to full research continuation. It does not join the worker's replan +barrier or restore Codex/LoopX authority. Its optional declared phase clock does +not qualify full research-state adoption or a same-point scientific comparison. +Never put its controller configuration in `--checkpoint-request` or a worker +payload. Omit both these controller operations and the native `--checkpoints` +flag to disable this transport on a fresh judge; preserved capsules stay inert. +Native HTTP tests cover lost replies, exact readback, unavailable authority and +invalid local recovery state. The explicitly enabled Docker qualification also +loses a reply after real native commit, recovers the original deadline, runs the +ordinary SForge phase-entry shell in Linux, and produces and grades a synthetic +candidate without refilling rounds/cooldown. It excludes model inference and +does not establish full research continuation. Run these checks with: + +```sh +python -m pytest -q benchmark/tests/test_checkpoint_controller.py +LOOPX_EDGEBENCH_DOCKER_SMOKE=1 python -m pytest -q \ + benchmark/tests/test_checkpoint_controller.py::test_real_docker_controller_export_import_and_next_grade +``` + +新的 controller transport 使用上述 PR #23 固定源及 native `--checkpoints`， +不复用旧只读接口来伪装接续。admin 配置和 bank 必须由 solver 权限范围外的 +controller 保管，不能放进 worker；0600 不能证明隔离。原生 owner 保留历史、 +已用轮次和剩余冷却，生成新注册；完整研究接续仍需联合会话、LoopX 新权限、 +原点剩余总预算及 replan 前决策边界。此切片不启动研究、不证明收益，也不改历史源。 + +这是研究 checkpoint 的开发前置切片。EdgeBench CLI 的显式 +`--checkpoint-request` 会把 host JSON 原文安全放入 heartbeat worker，并在 +runtime receipt 绑定摘要；programmatic adapter 也可指定已放入 guest 的请求 +路径。省略选项即关闭，Official/native-goal 不接受。产物 +验证和隔离物化不授予执行权，不代表研究点已资格化。请求中的 metadata 只是 +证据，不能代替会话、效果、状态采用及评测机会的联合验收。 +已放入 guest 的声明文件丢失、权限失败或损坏只记录捕获失败，原工作仍按 +完整的原始 quota guard 继续；准入 owner 不可用仍停止。首次 replan 捕获 +失败后，修复声明不得把后来的状态补记成同一原点。评测参与者使用上文明确的 +公开 SForge fork/pin 和 Python 3.13；普通 upstream 安装缺少该 seam 时明确 +拒绝。Codex 原生会话的真实分页/compaction 与隔离 fork 须单独验证。 +评测机会及 incumbent 参与者也分配未占用的 ID，并在封存的 +`point_metadata.evaluation_opportunity` 中记录实际映射；无 incumbent 时对应值为 null。 +文件 witness 校验按该映射定位，不会把同名调用者文件误当原生候选。 +自动原生存储参与者分配未占用的 artifact ID，保留调用者原 ID；封存的 +`native_session.store_members` 绑定这些 ID 与既有允许的原生文件名。恢复仍 +支持旧固定 ID，拒绝重复或未声明的映射，并保留源历史一致性校验。 + +显式 `"loopx_state": true` 复用现有 state-backup inventory，捕获单 Goal 试验的 +注册状态、项目研究状态、runtime journal 和 registry；File/SQLite authority +物化后仍是惰性历史。缺少来源、共享注册表、路径别名或重叠声明会拒绝，捕获 +不授予新的租约、配额、效果或执行权，不能替代完整接续验收。 + +Host 预检已提交 fresh replan，但尚待 Agent 选择工作时，仍可在模型决策前捕获。 +类型化 owner 核对原 Goal、已注册 Agent、Turn 和语义义务；不会伪造尚未形成的 +settlement identity，也不授予选择前的交付权限。源边界仍须来自上轮已结算的成功工作； +已选 guard 的原结算身份检查保持。 + +运行中出现的 fresh replan 可通过显式交接捕获。完整捕获需同时声明 +`controller_handoff`、`native_session`、`loopx_state`、`evaluation_opportunity` +为 true，使用 `fresh_replan`，并在 host CLI 提供 `--checkpoint-controller-file`。 +controller 配置及 admin 权限不进入 solver；backend 绑定真实 container 与原生注册。 +worker 保留原 guard，controller 暂停全部容器写入者，核对仍有效的请求和准备时 +文件 witness，再封存原生评测机会、incumbent 及剩余预算。源变化、过期或失去 +交接会拒绝；可恢复的捕获失败不取消原研究，也不得改用稍后的状态。解除暂停后， +源会话继续使用原 packet 和 deadline。 +真实 native worker、Docker 与 native HTTP 的合成验证覆盖了联合捕获和源接续； +产物仍是 `research_point_qualified=false`。独立目标环境的新权限准入、有效工作、 +联合丢回复恢复及后续评测尚待一体验证，不能据此重启研究实验。新运行省略 +controller 文件及 handoff 开关即可关闭；移除 checkpoint 请求关闭全部捕获， +已有产物保持惰性。 diff --git a/benchmark/edgebench/checkpoint_controller.py b/benchmark/edgebench/checkpoint_controller.py new file mode 100644 index 0000000000..35bbce0195 --- /dev/null +++ b/benchmark/edgebench/checkpoint_controller.py @@ -0,0 +1,414 @@ +"""Controller-only transport for the native SForge checkpoint API. + +Native admission/ranking retain their native owner. The existing typed artifact +owner seals and verifies the bank bytes. Nothing here belongs in a solver payload. +""" +from __future__ import annotations + +import argparse +import fcntl +import hashlib +import json +import math +import os +import re +import shutil +import stat +import tempfile +import time +from pathlib import Path +from urllib.error import HTTPError +from urllib.parse import urlsplit +from urllib.request import HTTPRedirectHandler, Request, build_opener + +from loopx.control_plane.effect_runtime import effect_runtime_result +from benchmark.runtime.worker import PHASE_WORK_RESERVE_SECONDS + + +class _NoRedirect(HTTPRedirectHandler): + def redirect_request(self, req, fp, code, msg, headers, newurl): + return None # Never forward controller credentials to another origin. + + +def _physical(path: Path) -> Path: + path = Path(path) + if not path.is_absolute() or path.resolve(strict=True) != path: + raise ValueError("Controller paths must be normalized, physical and absolute") + return path + + +def _controller(path: Path) -> dict: + path = _physical(path) + if path.stat().st_mode & 0o077: + raise ValueError("Controller configuration must be owner-only") + value = json.loads(path.read_bytes()) + if not isinstance(value, dict) or set(value) != {"url", "admin_secret", "bank_root"}: + raise ValueError("Controller configuration requires url, admin_secret and bank_root") + if not isinstance(value["admin_secret"], str) or not value["admin_secret"]: + raise ValueError("Fresh controller authority is unavailable") + url = urlsplit(value["url"]) + if (url.scheme not in {"http", "https"} or not url.hostname or url.username or + url.password or url.query or url.fragment or url.path not in {"", "/"}): + raise ValueError("Controller URL must identify one explicit HTTP origin") + value["bank_root"] = _physical(Path(value["bank_root"])) + return value + + +def _post(controller: dict, route: str, body: dict, timeout: float) -> dict: + if not math.isfinite(timeout) or timeout <= 0: + raise ValueError("Controller timeout must be finite and positive") + request = Request(controller["url"].rstrip("/") + route, + data=json.dumps({**body, "admin_secret": controller["admin_secret"]}).encode(), + headers={"Content-Type": "application/json"}, method="POST") + try: + with build_opener(_NoRedirect).open(request, timeout=timeout) as response: + result = json.load(response) + except HTTPError as error: + # A remote body can echo submitted credentials. Keep only the status. + raise ValueError(f"Native checkpoint operation rejected: HTTP {error.code}") from None + if not isinstance(result, dict): + raise ValueError("Native checkpoint operation returned an invalid object") + return result + + +def _reference(value: dict) -> dict: + if (set(value) != {"checkpoint_id", "manifest_sha256", "task_id", "source_run_id", "opportunity", "incumbent"} or + not re.fullmatch(r"[0-9a-f]{32}", value.get("checkpoint_id", "")) or + not re.fullmatch(r"[0-9a-f]{64}", value.get("manifest_sha256", ""))): + raise ValueError("Native checkpoint reference is malformed") + return value + + +def _bound_bank(folder: Path, reference: dict) -> dict: + """Byte/source binding only; native restore owns eligibility and validity.""" + raw = _physical(folder / "manifest.json").read_bytes() + digest = hashlib.sha256(raw).hexdigest() + witnessed = _physical(folder / "manifest.sha256").read_text() + packet = json.loads(raw) + if (digest != reference["manifest_sha256"] or witnessed != digest or + packet.get("schema_version") != "sforge_judge_checkpoint_v1" or + any(packet.get(key) != reference[key] for key in ( + "task_id", "source_run_id", "opportunity", "incumbent"))): + raise ValueError("Native bank bytes differ from the captured reference") + expected = {"manifest.json", "manifest.sha256"} + incumbent = reference["incumbent"] + if incumbent is not None: + expected.add("incumbent.tar.gz") + archive = _physical(folder / "incumbent.tar.gz").read_bytes() + if (len(archive) != incumbent["archive_bytes"] or + hashlib.sha256(archive).hexdigest() != incumbent["archive_sha256"]): + raise ValueError("Native selected archive differs from the captured reference") + if {p.name for p in folder.iterdir()} != expected: + raise ValueError("Native bank contains undeclared members") + return packet + + +def _phase_budget(raw: bytes, native: dict) -> dict: + """Adapt the existing SForge integer phase clock, not quota authority. + + Native capture owns this time. The declared physical Host file owns its + deadline. Neither fact qualifies complete Host/research-state capture. + """ + if len(raw) > 64 or not re.fullmatch(rb"[0-9]+\n?", raw): + raise ValueError("Host phase deadline must be an integer epoch file") + epoch = int(raw) + captured = native.get("captured_at_epoch") + if (not 0 < epoch < 2 ** 53 or type(captured) not in (int, float) or + not math.isfinite(captured) or captured <= 0): + raise ValueError("Host/native phase clocks cannot be represented exactly") + return {"component_id": "host_phase_deadline", "source_deadline_epoch": epoch, + "captured_at_epoch": captured, "remaining_seconds": max(0, epoch - captured)} + + +def _phase_destination(path: Path, *, root: Path, bank: Path, recover: bool = False) -> Path: + path = Path(path) + _physical(path.parent) + if (not path.is_absolute() or path.resolve() != path or (path.exists() and not recover) or path.is_symlink() + or path.is_relative_to(root) or path.is_relative_to(bank)): + raise ValueError("Host phase restoration requires a fresh physical file outside artifact/bank roots") + return path + + +def _private_file(path: Path) -> None: + info = _physical(path).stat() + if not stat.S_ISREG(info.st_mode) or info.st_mode & 0o077 or info.st_nlink != 1: + raise ValueError("Recovery requires an owner-only physical regular file") + + +def _sync_parent(path: Path) -> None: + descriptor = os.open(path.parent, os.O_RDONLY | os.O_DIRECTORY) + try: + os.fsync(descriptor) + finally: + os.close(descriptor) + + +def _activation_epoch(path: Path, binding: dict, remaining: float, recover: bool) -> int: + """Provider-local attempt binding; no token or historical authority is stored.""" + record = path.with_name(path.name + ".restore.json") + if recover: + _private_file(record) + value = json.loads(record.read_bytes()) + if (not isinstance(value, dict) or set(value) != {"binding", "deadline_epoch"} + or value["binding"] != binding): + raise ValueError("Host recovery does not match the original activation") + epoch = value["deadline_epoch"] + else: + epoch = math.floor(time.time() + remaining) + # Commit this before contacting native admission. A partial record + # cannot be recovered, and retry never computes another deadline. + with os.fdopen(os.open(record, os.O_WRONLY | os.O_CREAT | os.O_EXCL, 0o600), "w") as target: + json.dump({"binding": binding, "deadline_epoch": epoch}, target) + target.flush() + os.fsync(target.fileno()) + _sync_parent(record) + if type(epoch) is not int or not 0 < epoch < 2 ** 53: + raise ValueError("Destination Host phase clock is incompatible") + if epoch - time.time() <= PHASE_WORK_RESERVE_SECONDS: + raise ValueError("Original Host activation budget is exhausted") + return epoch + + +def _activation_result(result: dict, body: dict) -> None: + if (set(result) != {*body, "token"} or any(result.get(k) != v for k, v in body.items()) + or not isinstance(result.get("token"), str) or not result["token"]): + raise ValueError("Native activation result differs from the original request") + + +def capture(controller_file: Path, *, token: str, capture_id: str, destination: Path, + timeout: float = 30, phase_deadline_file: Path | None = None) -> dict: + """Capture one native-owned point without exporting controller configuration.""" + if not re.fullmatch(r"[0-9a-f]{32}", capture_id): + raise ValueError("Capture identity must be the declared 32-hex native identity") + destination = Path(destination) + _physical(destination.parent) + if not destination.is_absolute() or destination != destination.resolve() or destination.exists(): + raise ValueError("Capture export requires a fresh physical destination") + controller = _controller(controller_file) + phase_raw = None + if phase_deadline_file is not None: + phase_deadline_file = _physical(phase_deadline_file) + phase_raw = phase_deadline_file.read_bytes() + _phase_budget(phase_raw, {"captured_at_epoch": time.time()}) + # An earlier native-only point cannot acquire a later Host clock. + if (controller["bank_root"] / capture_id).exists(): + raise ValueError("Existing native point has no new Host phase capture boundary") + attempts = controller["bank_root"] / ".loopx-phase-attempts" + attempts.mkdir(mode=0o700, exist_ok=True) + _physical(attempts) + # First attempt wins, including lost responses or failed exports. + # A later source cannot replace the Host clock at that native point. + (attempts / capture_id).mkdir(mode=0o700) + reference = _reference(_post(controller, "/api/v1/checkpoint/capture", + {"token": token, "capture_id": capture_id}, timeout)) + if reference["checkpoint_id"] != capture_id: + raise ValueError("Native capture returned a different identity") + folder = _physical(controller["bank_root"] / capture_id) + native = _bound_bank(folder, reference) + components = [{"id": "native_judge", "source_path": str(folder)}] + metadata = {"native_reference": reference} + if phase_raw is not None: + if phase_deadline_file.read_bytes() != phase_raw: + raise ValueError("Host phase deadline moved across native capture") + components.append({"id": "host_phase_deadline", "source_path": str(phase_deadline_file)}) + metadata["host_phase_budget"] = _phase_budget(phase_raw, native) + probe = effect_runtime_result("runtime.checkpoint_artifacts", {"action": "probe", "components": components}) + result = effect_runtime_result("runtime.checkpoint_artifacts", { + "action": "capture", "components": components, "destination": str(destination), + "expected_source_sha256": probe["source_sha256"], + "point_metadata": metadata, + }, retry_safe=False) + # Bind the actually copied bytes, not merely a prior source observation. + _bound_bank(destination / "components/native_judge", reference) + if phase_raw is not None and (destination / "components/host_phase_deadline").read_bytes() != phase_raw: + raise ValueError("Captured Host phase bytes differ from the native boundary") + return {**result, "native_reference": reference} + + +def restore(controller_file: Path, *, root: Path, expected_sha256: str, run_id: str, + timeout: float = 30, phase_deadline_file: Path | None = None, recover: bool = False) -> dict: + """Import verified inert bytes, then ask fresh native authority to admit them. + + An explicitly captured Host clock preserves its remaining phase duration. + Does not restore a model, LoopX Goal or effect authority. + An uncertain response is not replayed under a different run or capture id. + """ + if not re.fullmatch(r"[A-Za-z0-9][A-Za-z0-9_.-]{0,127}", run_id): + raise ValueError("Continuation requires one explicit native run component") + controller = _controller(controller_file) + root = _physical(root) + effect_runtime_result("runtime.checkpoint_artifacts", { + "action": "verify", "root": str(root), "expected_sha256": expected_sha256}) + manifest = json.loads((root / "checkpoint.json").read_bytes()) + expected_components = ["native_judge"] + if phase_deadline_file is not None: + expected_components.append("host_phase_deadline") + if [c["id"] for c in manifest["components"]] != expected_components: + raise ValueError("Native import requires its declared capsule and explicit matching Host clock") + reference = _reference(manifest["point_metadata"]["native_reference"]) + if run_id == reference["source_run_id"]: + raise ValueError("Continuation requires a fresh run identity") + folder = root / "components/native_judge" + native = _bound_bank(folder, reference) + bank = controller["bank_root"] + phase_path = None + phase = None + if phase_deadline_file is not None: + phase = _phase_budget((root / "components/host_phase_deadline").read_bytes(), native) + if manifest["point_metadata"].get("host_phase_budget") != phase: + raise ValueError("Host phase metadata differs from sealed deadline/native clock") + if phase["remaining_seconds"] <= PHASE_WORK_RESERVE_SECONDS: + raise ValueError("Captured Host phase has no work allowance beyond its existing reserve") + phase_path = _phase_destination(phase_deadline_file, root=root, bank=bank, recover=recover) + target = bank / reference["checkpoint_id"] + stage = Path(tempfile.mkdtemp(prefix=".loopx-native-import-", dir=bank)) + phase_fd = None + phase_published = False + phase_locked = False + native_attempted = False + try: + if phase_path is not None: + # Empty is unusable by the existing SForge phase-entry shell. + # An uncertain native response must not fall back to a new budget. + if recover: + _private_file(phase_path) + phase_fd = os.open(phase_path, os.O_RDWR | os.O_NOFOLLOW | (0 if recover else os.O_CREAT | os.O_EXCL), 0o600) + fcntl.flock(phase_fd, fcntl.LOCK_EX | fcntl.LOCK_NB) + phase_locked = True + original_phase = phase_path.read_bytes() + phase_published = bool(original_phase) + if recover: + _bound_bank(_physical(target), reference) + else: + effect_runtime_result("runtime.checkpoint_artifacts", { + "action": "materialize", "root": str(root), "expected_sha256": expected_sha256, + "destination": str(stage / "unit"), + }, retry_safe=False) + imported = stage / "unit/components/native_judge" + _bound_bank(imported, reference) + if target.exists(): + _bound_bank(_physical(target), reference) # No overwrite, even on retry. + else: + # POSIX rename can replace an empty destination. Reserve its name + # exclusively; interruption leaves an unusable point, never grants. + target.mkdir(mode=0o700) + try: + imported.rename(target) + except BaseException: + target.rmdir() + raise + descriptor = os.open(bank, os.O_RDONLY | os.O_DIRECTORY) + try: + os.fsync(descriptor) + finally: + os.close(descriptor) + body = {"checkpoint_id": reference["checkpoint_id"], + "manifest_sha256": reference["manifest_sha256"], "run_id": run_id} + phase_epoch = None + if phase_path is not None: + phase_epoch = _activation_epoch(phase_path, { + **body, "artifact_sha256": expected_sha256, "controller_url": controller["url"], + "bank_root": str(bank), "phase_path": str(phase_path), + }, phase["remaining_seconds"], recover) + if original_phase not in (b"", f"{phase_epoch}\n".encode()): + raise ValueError("Host phase bytes differ from original activation") + native_attempted = True + route = "/api/v1/checkpoint/restore-result" if recover else "/api/v1/checkpoint/restore" + result = _post(controller, route, body, timeout) + _activation_result(result, body) + if phase_fd is not None: + if phase_epoch - time.time() <= PHASE_WORK_RESERVE_SECONDS: + raise ValueError("Original Host activation budget is exhausted") + # Charge activation/transport, not offline time or a fresh trial. + # Native alone owns accepted counters and remaining cooldown. + encoded = f"{phase_epoch}\n".encode() + os.lseek(phase_fd, 0, os.SEEK_SET) + if os.write(phase_fd, encoded) != len(encoded): + raise OSError("Destination Host phase write was incomplete") + os.fsync(phase_fd) + _sync_parent(phase_path) + if phase_path.read_bytes() != f"{phase_epoch}\n".encode(): + raise ValueError("Destination Host phase readback differs from activation") + phase_published = True + return result + finally: + if phase_fd is not None: + try: + if phase_locked and native_attempted and not phase_published: + os.ftruncate(phase_fd, 0) + os.fsync(phase_fd) + finally: + os.close(phase_fd) + if phase_locked and not native_attempted and not recover: + phase_path.unlink() + shutil.rmtree(stage) + + +def main(argv=None) -> None: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--controller-file", required=True, type=Path) + commands = parser.add_subparsers(dest="command", required=True) + capture_parser = commands.add_parser("capture") + capture_parser.add_argument("--token-file", required=True, type=Path) + capture_parser.add_argument("--capture-id", required=True) + capture_parser.add_argument("--destination", required=True, type=Path) + capture_parser.add_argument("--phase-deadline-file", type=Path, + help="Explicit physical SForge Host phase file; omitted keeps native-only capture") + worker_parser = commands.add_parser("capture-worker", + help="Controller-owned Docker pause/archive plus native capsule; inert files, not process restoration") + worker_parser.add_argument("--token-file", required=True, type=Path) + worker_parser.add_argument("--capture-id", required=True) + worker_parser.add_argument("--destination", required=True, type=Path) + worker_parser.add_argument("--worker-request", required=True, type=Path) + restore_parser = commands.add_parser("restore") + restore_parser.add_argument("--root", required=True, type=Path) + restore_parser.add_argument("--expected-sha256", required=True) + restore_parser.add_argument("--run-id", required=True) + restore_parser.add_argument("--result-file", required=True, type=Path) + restore_parser.add_argument("--phase-deadline-file", type=Path, + help="Fresh physical Host phase file, required for a capsule containing that participant") + restore_parser.add_argument("--recover", action="store_true", + help="Read the exact original native result and retain its original Host deadline") + args = parser.parse_args(argv) + if args.command in {"capture", "capture-worker"}: + source_token = _physical(args.token_file).read_text().strip() + if args.command == "capture-worker": + from benchmark.edgebench.checkpoint_worker import capture_worker + + result = capture_worker(args.controller_file, token=source_token, + capture_id=args.capture_id, destination=args.destination, + request_file=args.worker_request) + else: + result = capture(args.controller_file, token=source_token, + capture_id=args.capture_id, destination=args.destination, + phase_deadline_file=args.phase_deadline_file) + print(json.dumps(result)) + else: + # Reserve before admission. A lost output is explicit recovery, never a + # fresh registration that silently refills the captured opportunity. + _physical(args.result_file.parent) + if args.result_file != args.result_file.resolve(): + raise ValueError("Continuation result must be a normalized physical path") + if args.recover: + _private_file(args.result_file) + with args.result_file.open("r+" if args.recover else "x") as result_file: + args.result_file.chmod(0o600) + fcntl.flock(result_file.fileno(), fcntl.LOCK_EX | fcntl.LOCK_NB) + prior_raw = result_file.read() if args.recover else "" + prior_result = json.loads(prior_raw) if prior_raw else None + result = restore(args.controller_file, root=args.root, + expected_sha256=args.expected_sha256, run_id=args.run_id, + phase_deadline_file=args.phase_deadline_file, recover=args.recover) + if prior_result is not None and prior_result != result: + raise ValueError("Existing private result differs from original native activation") + result_file.seek(0) + json.dump(result, result_file) + result_file.truncate() + result_file.flush() + os.fsync(result_file.fileno()) + _sync_parent(args.result_file) + print(json.dumps({"result_file": str(args.result_file), "run_id": args.run_id})) + + +if __name__ == "__main__": + main() diff --git a/benchmark/edgebench/checkpoint_judge.py b/benchmark/edgebench/checkpoint_judge.py new file mode 100644 index 0000000000..77117b7e80 --- /dev/null +++ b/benchmark/edgebench/checkpoint_judge.py @@ -0,0 +1,183 @@ +"""Opt-in read-only checkpoint participant over the native SForge judge. + +Native admission, grading, history and ranking retain their owners. This +adapter exports a quiescent registration's public history and opportunity +clock; it cannot register, submit, restore or extend a trial. +""" +from __future__ import annotations + +import argparse +import hashlib +import json +import os +import stat +import time +import uuid +from dataclasses import replace +from pathlib import Path +from benchmark.runtime.sforge_checkpoint import ARCHIVE_ROUTE, ROUTE, SCHEMA + + +def _read_archive(root, run, task, round_id): + """Read an opaque native archive, never extract or inspect its members.""" + root = root.resolve(strict=True) + path = root / "runs" / run / task / "submissions" / round_id / "submission.tar.gz" + if not path.is_relative_to(root) or path.resolve(strict=True) != path: + raise ValueError("Native incumbent archive must be a physical file below the log owner") + with os.fdopen(os.open(path, os.O_RDONLY | os.O_NOFOLLOW), "rb") as source: + before = os.fstat(source.fileno()) + if not stat.S_ISREG(before.st_mode): + raise ValueError("Native incumbent archive must be a regular file") + raw = source.read() + after = os.fstat(source.fileno()) + observed = path.stat(follow_symlinks=False) + def signature(value): + return value.st_dev, value.st_ino, value.st_size, value.st_mtime_ns, value.st_ctime_ns + if (signature(before) != signature(after) or signature(after) != signature(observed) or + path.resolve(strict=True) != path or len(raw) != before.st_size): + raise ValueError("Native incumbent archive moved during capture") + return raw + + +def create_app(config, *, checkpoint_capture=False): + if type(checkpoint_capture) is not bool: + raise ValueError("checkpoint_capture must be an explicit boolean") + from sforge.harness.judge_server import SubmissionStatus, create_app as native_app + app = native_app(config) + if not checkpoint_capture: + return app # Preserve the native route/admission surface when disabled. + from fastapi import HTTPException + from fastapi.responses import Response + from pydantic import BaseModel, ConfigDict + import sforge + + state = getattr(app.state, "judge", None) + if state is None or not all(hasattr(state, key) for key in ( + "_admission_lock", "_outstanding_by_run", "_tokens_lock", "_history_lock")): + raise ValueError("Checkpoint capture requires a pinned SForge with its native admission owner exposed") + epoch = uuid.uuid4().hex + source = hashlib.sha256() + source_root = Path(sforge.__file__).parent + for path in sorted(source_root.rglob("*.py")): + source.update(path.relative_to(source_root).as_posix().encode() + b"\0") + source.update(hashlib.sha256(path.read_bytes()).digest()) + tasks = {key: { + "native_source_sha256": source.hexdigest(), + "task_spec_sha256": hashlib.sha256((config.tasks_dir / f"{key}.json").read_bytes()).hexdigest(), + "judge_image_key": task.judge_image_key, + "selection": task.judge.selection, "score_direction": task.judge.score_direction, + } for key, task in state.tasks.items()} + + class ReadRequest(BaseModel): + model_config = ConfigDict(extra="forbid") + token: str + + class ArchiveRequest(ReadRequest): + capture_binding_sha256: str + + # FastAPI resolves annotations in module globals; keep this factory's + # private request type concrete rather than a forward reference. + def snapshot_locked(token): + from sforge.harness.selection import select_best + + # Reuse the native lock order and outstanding admission counter. A + # pending grade or unexplained consumed round is never a complete point. + # Called only under the native admission -> token -> history locks. + info = state.tokens.get(token) + if info is None: + raise HTTPException(403, "Unknown registration") + run, task = info["run_id"], info["task_id"] + if state._closed or state.tasks[task].game_mode: + raise HTTPException(409, "Checkpoint requires an open non-game registration") + if state._outstanding_by_run.get((run, task), 0): + raise HTTPException(409, "Accepted evaluation is still outstanding") + if info["next_auto"] != 1: + raise HTTPException(409, "This checkpoint profile requires no automatic evaluations") + history = state.run_history.get(state._run_key(run, task), []) + rounds = [entry.get("round") for entry in history] + expected = {f"agent-{index}" for index in range(1, info["next_agent"])} + if (len(rounds) != len(expected) or set(rounds) != expected or + any(entry.get("status") not in {SubmissionStatus.COMPLETED, SubmissionStatus.ERROR} for entry in history)): + raise HTTPException(409, "Consumed rounds and released history are not reconciled") + results = {} + for entry in history: + identifier = entry.get("submission_id") + native = state.submissions.get(identifier) + if (native is None or native.get("run_id") != run or native.get("task_id") != task or + native.get("round") != entry["round"] or native.get("status") != entry["status"]): + raise HTTPException(409, "Released result details are unavailable or differ") + results[identifier] = {key: native.get(key) for key in ("status", "report", "error")} + # Reuse native ranking, including validity, direction and tie rules. + # The working candidate can differ from this older selected archive. + best = select_best(history, state.tasks[task].judge.score_direction, + state.tasks[task].judge.selection) + incumbent = None + raw = None + if best["best_round"]: + selected = next(entry for entry in history if entry["round"] == best["best_round"]) + try: + raw = _read_archive(config.log_dir, run, task, selected["round"]) + except (OSError, ValueError) as error: + raise HTTPException(409, "Native selected incumbent archive is unavailable or moved") from error + incumbent = {"round": selected["round"], "submission_id": selected["submission_id"], + "archive_sha256": hashlib.sha256(raw).hexdigest(), "archive_bytes": len(raw)} + observed = time.time() + last = info["last_agent_submit_at"] + cooldown = info["submission_cooldown"] + remaining = max(0, last + (cooldown or 0) - observed) if last else 0 + packet = { + "schema_version": SCHEMA, "judge_epoch": epoch, + "run_id": run, "task_id": task, "evaluator": tasks[task], + "observed_at_epoch": observed, "last_agent_submit_at_epoch": last, + "submission_cooldown_seconds": cooldown, + "remaining_cooldown_seconds": remaining, + "max_agent_submissions": info["max_agent_submissions"], + "next_agent_round": info["next_agent"], "next_auto_round": info["next_auto"], + "released_history": history, "released_results": results, "outstanding_evaluations": 0, + "native_incumbent": incumbent, + "execution_authority_granted": False, + } + # Bind both reads to the same released state, not a wall clock. A new + # accepted round, source epoch or changed archive invalidates download. + binding = {key: value for key, value in packet.items() + if key not in {"observed_at_epoch", "remaining_cooldown_seconds"}} + packet["capture_binding_sha256"] = hashlib.sha256( + json.dumps(binding, sort_keys=True, separators=(",", ":")).encode()).hexdigest() + return json.loads(json.dumps(packet)), raw # No live references or tokens leave the lock. + + def capture(request): + with state._admission_lock, state._tokens_lock, state._history_lock: + return snapshot_locked(request.token)[0] + + def archive(request): + with state._admission_lock, state._tokens_lock, state._history_lock: + packet, raw = snapshot_locked(request.token) + incumbent = packet["native_incumbent"] + if packet["capture_binding_sha256"] != request.capture_binding_sha256 or incumbent is None: + raise HTTPException(409, "Native checkpoint basis moved or has no selected incumbent") + return Response(raw, media_type="application/octet-stream") + capture.__annotations__["request"] = ReadRequest + archive.__annotations__["request"] = ArchiveRequest + app.post(ROUTE)(capture) + app.post(ARCHIVE_ROUTE)(archive) + return app + + +def main(argv=None): + import uvicorn + from sforge.harness.config import load_config + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--checkpoint-capture", action="store_true") + parser.add_argument("--host", default="127.0.0.1") + parser.add_argument("--port", type=int, default=8080) + parser.add_argument("--tasks-dir", type=Path) + parser.add_argument("--log-dir", type=Path) + args = parser.parse_args(argv) + config = load_config() + overrides = {key: getattr(args, key) for key in ("tasks_dir", "log_dir") if getattr(args, key) is not None} + uvicorn.run(create_app(replace(config, **overrides), checkpoint_capture=args.checkpoint_capture), + host=args.host, port=args.port, access_log=False) + + +if __name__ == "__main__": + main() diff --git a/benchmark/edgebench/checkpoint_supervisor.py b/benchmark/edgebench/checkpoint_supervisor.py new file mode 100644 index 0000000000..fc7a6778d2 --- /dev/null +++ b/benchmark/edgebench/checkpoint_supervisor.py @@ -0,0 +1,118 @@ +"""Host-only adapter for the worker's explicitly enabled checkpoint mailbox.""" +from __future__ import annotations + +import hashlib +import io +import json +import re +import tarfile +import tempfile +import threading +import time +from pathlib import Path + +from .checkpoint_controller import _controller +from .checkpoint_worker import _archive, capture_worker + + +class CheckpointSupervisor: + def __init__(self, controller_file: Path, destination: Path): + _controller(controller_file) # Fail before native registration. + self.controller_file, self.destination = controller_file, destination + self.token = None + self.stop = threading.Event() + self.thread = None + self.failure = None + + def bind(self, environment): + if self.token is not None or not (environment or {}).get("SFORGE_TOKEN"): + raise ValueError("Checkpoint controller requires one native worker registration") + self.token = environment["SFORGE_TOKEN"] + + def start(self, container): + if self.thread is not None or self.token is None: + raise ValueError("Checkpoint supervisor requires one bound worker execution") + self.destination.mkdir(mode=0o700, parents=True, exist_ok=False) + self.container = container + self.thread = threading.Thread(target=self._run, args=(container,), daemon=True) + self.thread.start() + + def close(self): + self.stop.set() + if self.thread is not None: + self.thread.join(60) + if self.thread.is_alive(): + raise RuntimeError("Checkpoint writer barrier is still active; preserve the worker for recovery") + if self.failure is not None: + # A missed optional point does not cancel otherwise useful source + # work. An uncertain/unreleased writer barrier still needs recovery. + self.container.reload() + if self.container.attrs["State"]["Paused"]: + raise RuntimeError("Checkpoint writer barrier requires controller recovery") from self.failure + + def _reply(self, container, directory, nonce, response): + container.reload() + if container.attrs["State"]["Paused"] or not container.attrs["State"]["Running"]: + raise RuntimeError("Checkpoint reply requires the released, running source worker") + raw = json.dumps(response).encode() + pending, final = nonce + ".pending", nonce + ".json" + stream = io.BytesIO() + with tarfile.open(fileobj=stream, mode="w") as archive: + entry = tarfile.TarInfo(pending) + entry.size, entry.mode = len(raw), 0o444 + archive.addfile(entry, io.BytesIO(raw)) + if not container.put_archive(directory, stream.getvalue()): + raise RuntimeError("Checkpoint reply transport was not accepted") + result = container.exec_run(["/bin/mv", directory + "/" + pending, directory + "/" + final]) + if result.exit_code: + raise RuntimeError("Checkpoint reply could not be published atomically") + + def _run(self, container): + from docker.errors import NotFound + + mailbox = "/logs/agent/checkpoint-controller" + remote = mailbox + "/ready.json" + seen = set() + try: + while not self.stop.is_set(): + with tempfile.TemporaryDirectory(prefix=".controller-poll-", dir=self.destination) as temp: + stage = Path(temp) + try: + raw = _archive(container, remote, stage / "ready").read_bytes() + except NotFound: + self.stop.wait(5) + continue + digest = hashlib.sha256(raw).hexdigest() + if digest in seen: + self.stop.wait(5) + continue + seen.add(digest) # Never retry a later state under an earlier point. + value = json.loads(raw) + nonce = value.get("capture_id") + if not isinstance(nonce, str) or not re.fullmatch(r"[0-9a-f]{32}", nonce): + raise ValueError("Malformed checkpoint request identity") + response = {"request_sha256": digest, "ok": False} + try: + request = stage / "request.json" + request.write_text(json.dumps({"container_id": container.id, + "components": value["components"], + "phase_deadline_path": "/opt/loopx-benchmark/control/phase-deadline", + "handoff": {"source_path": remote, "sha256": digest}})) + remaining = value["deadline_epoch"] - time.time() + if remaining <= 0: + raise TimeoutError("Checkpoint handoff deadline expired") + captured = capture_worker(self.controller_file, token=self.token, capture_id=nonce, + destination=self.destination / nonce, request_file=request, timeout=min(30, remaining)) + response.update(ok=True, capture=captured) + except Exception as error: + # Optional capture failure resumes the original source; + # credentials and raw provider errors never enter it. + response["error_kind"] = type(error).__name__ + diagnostic = self.destination / (nonce + ".error.txt") + diagnostic.write_text(str(error).replace(self.token, "[redacted]")[:2000]) + diagnostic.chmod(0o600) + (self.destination / (nonce + ".json")).write_text(json.dumps(response)) + self._reply(container, mailbox, nonce, response) + except Exception as error: + self.failure = error + (self.destination / "supervisor-failure.json").write_text(json.dumps({"error_kind": type(error).__name__})) diff --git a/benchmark/edgebench/checkpoint_worker.py b/benchmark/edgebench/checkpoint_worker.py new file mode 100644 index 0000000000..8d299c917d --- /dev/null +++ b/benchmark/edgebench/checkpoint_worker.py @@ -0,0 +1,266 @@ +"""Explicit controller-owned Docker writer barrier for inert checkpoint export. + +This provider adapts Docker pause/archive; the existing native and typed artifact +owners still own evaluator state and sealing. No code runs inside a frozen worker. +It does not capture process memory or qualify a native conversation cursor. +""" +from __future__ import annotations + +import hashlib +import json +import math +import os +import re +import tarfile +import tempfile +import time +from pathlib import Path, PurePosixPath + +from loopx.control_plane.effect_runtime import effect_runtime_result +from loopx.file_lock import exclusive_file_lock +from benchmark.runtime.checkpoint import _participant_id + +from .checkpoint_controller import _controller, _physical, capture + + +def _request(path: Path) -> dict: + value = json.loads(_physical(path).read_bytes()) + if (not isinstance(value, dict) or + set(value) not in ({"container_id", "components", "phase_deadline_path"}, + {"container_id", "components", "phase_deadline_path", "handoff"}) or + not re.fullmatch(r"[0-9a-f]{64}", value.get("container_id", "")) or + not isinstance(value["components"], list) or not value["components"]): + raise ValueError("Worker request requires a full container ID, explicit components and phase_deadline_path") + ids = set() + paths = [] + for entry in value["components"]: + if (not isinstance(entry, dict) or set(entry) != {"id", "source_path"} or + not isinstance(entry["id"], str) or + not re.fullmatch(r"[A-Za-z][A-Za-z0-9_.-]{0,63}", entry["id"]) or entry["id"] in ids): + raise ValueError("Worker components require unique IDs") + ids.add(entry["id"]) + paths.append(_guest_path(entry["source_path"])) + paths.append(_guest_path(value["phase_deadline_path"])) + if "handoff" in value: + handoff = value["handoff"] + if (not isinstance(handoff, dict) or set(handoff) != {"source_path", "sha256"} or + not re.fullmatch(r"[0-9a-f]{64}", handoff.get("sha256", ""))): + raise ValueError("Worker handoff requires the exact private request digest") + paths.append(_guest_path(handoff["source_path"])) + if any(a == b or a in b.parents or b in a.parents + for index, a in enumerate(paths) for b in paths[index + 1:]): + raise ValueError("Worker components and phase clock must not overlap") + return value + + +def _guest_path(value: str) -> PurePosixPath: + if not isinstance(value, str): + raise ValueError("Worker paths must be normalized absolute paths below root") + path = PurePosixPath(value) + if not path.is_absolute() or str(path) != value or value == "/" or ".." in path.parts: + raise ValueError("Worker paths must be normalized absolute paths below root") + return path + + +def _archive(container, remote: str, destination: Path) -> Path: + """Read through the daemon, rejecting links/devices and archive traversal. + + Docker's archive reader works while paused, unlike exec-based file readers. + Keep only the declared object, not parent filesystem or process credentials. + """ + stream, stat = container.get_archive(remote) + if stat.get("linkTarget"): + raise ValueError("Worker component must not be a symlink") + destination.mkdir(mode=0o700) + with tempfile.TemporaryFile() as raw: + for chunk in stream: + raw.write(chunk) + raw.seek(0) + with tarfile.open(fileobj=raw) as archive: + members = archive.getmembers() + root = PurePosixPath(remote).name + names = set() + for member in members: + path = PurePosixPath(member.name) + if (path.is_absolute() or ".." in path.parts or not path.parts or + path.parts[0] != root or str(path) in names or + not (member.isfile() or member.isdir())): + raise ValueError("Worker archive contains undeclared, linked or special members") + names.add(str(path)) + archive.extractall(destination, members=members, filter="data") + # The data filter intentionally drops directory modes. This is a + # private raw transport witness, so restore only ordinary permission + # bits after safe extraction; never restore owners or special bits. + # Otherwise a private 0700 authority directory becomes 0755 and a + # matching frozen source is falsely rejected as having changed. + for member in sorted(members, key=lambda entry: len(PurePosixPath(entry.name).parts), reverse=True): + (destination / member.name).chmod(member.mode & 0o777) + return _physical(destination / root) + + +def _archive_component(container, remote: str, destination: Path) -> Path: + """Transport a standalone SQLite participant's committed storage too. + + A paused process can still have committed data only in its WAL. Reading + the main file alone silently exports an older, internally valid database. + The typed artifact owner's SQLite backup still decides the committed + snapshot; this Docker adapter only transports its physical input while + the same writer barrier is held. A directory archive already includes it. + """ + from docker.errors import NotFound + + copied = _archive(container, remote, destination) + if not copied.is_file(): + return copied + with copied.open("rb") as source: + if source.read(16) != b"SQLite format 3\0": + return copied + # SHM is a reader index, rebuilt locally. WAL and rollback journals can + # affect committed state and must not be dropped or treated as optional + # when transport, permissions or link validation fails. + for suffix in ("-wal", "-journal"): + with tempfile.TemporaryDirectory(prefix=".sqlite-transport-", dir=destination) as stage: + try: + companion = _archive(container, remote + suffix, Path(stage) / "member") + except NotFound: + continue + if not companion.is_file(): + raise ValueError("SQLite companion must be a physical regular file") + companion.rename(copied.with_name(copied.name + suffix)) + return copied + + +def _worker_lock(client, container_id: str) -> Path: + # Controller banks and endpoint aliases may differ for the same daemon. + # Coordinate captures by this user on this host, not by artifact location. + daemon_id = client.info().get("ID") + if not isinstance(daemon_id, str) or not daemon_id: + raise ValueError("Docker daemon identity is required for worker capture") + root = Path.home() / ".cache" / "loopx" / "checkpoint-workers" + root.mkdir(parents=True, exist_ok=True, mode=0o700) + _physical(root) + info = root.stat() + if info.st_uid != os.getuid() or info.st_mode & 0o077: + raise ValueError("Worker capture lock directory must be private and owned by the current user") + daemon_key = hashlib.sha256(daemon_id.encode()).hexdigest() + return root / f"{daemon_key}-{container_id}" + + +def capture_worker(controller_file: Path, *, token: str, capture_id: str, + destination: Path, request_file: Path, timeout: float = 30) -> dict: + """Freeze one explicitly owned worker and seal its files with native state. + + A fresh identity is required. Failed or uncertain attempts never recapture a + later worker into an earlier evaluator point. The original worker is resumed + before publishing the sealed result; an unsuccessful unpause is an error. + """ + import docker + + if not math.isfinite(timeout) or timeout <= 0: + raise ValueError("Worker capture timeout must be finite and positive") + request = _request(request_file) + controller = _controller(controller_file) + destination = Path(destination) + _physical(destination.parent) + if (destination.resolve() != destination or destination.exists() or destination.is_symlink() or + not re.fullmatch(r"[0-9a-f]{32}", capture_id)): + raise ValueError("Worker capture requires a fresh physical destination and 32-hex identity") + if (controller["bank_root"] / capture_id).exists(): + raise ValueError("Existing native point cannot acquire later worker files") + attempts = controller["bank_root"] / ".loopx-worker-attempts" + attempts.mkdir(exist_ok=True, mode=0o700) + _physical(attempts) + (attempts / capture_id).mkdir(mode=0o700) # First attempt, including failures, wins. + client = docker.from_env(timeout=timeout) + try: + with exclusive_file_lock(_worker_lock(client, request["container_id"]), timeout_seconds=timeout, + operation="checkpoint_worker_capture"): + container = client.containers.get(request["container_id"]) + container.reload() + state = container.attrs["State"] + if container.id != request["container_id"] or not state["Running"] or state["Paused"]: + raise ValueError("Worker must be the exact running, unpaused container owned by this controller") + with tempfile.TemporaryDirectory(prefix=".worker-capture-", dir=destination.parent) as directory: + stage = Path(directory) + (stage / "workers").mkdir(mode=0o700) + components = [] + prepared = None + pause_may_have_committed = True + try: + try: + container.pause() + except docker.errors.APIError as exc: + # Docker's conflict response rejects this pause. The + # observed pause belongs to another actor; leave it alone. + if exc.response is not None and exc.response.status_code == 409: + pause_may_have_committed = False + raise + container.reload() + if not container.attrs["State"]["Paused"]: + raise RuntimeError("Worker writer barrier was not established") + if "handoff" in request: + held = _archive(container, request["handoff"]["source_path"], stage / "held") + raw = held.read_bytes() + if hashlib.sha256(raw).hexdigest() != request["handoff"]["sha256"]: + raise ValueError("Worker handoff changed before its writer barrier") + prepared = json.loads(raw) + if (set(prepared) != {"capture_id", "components", "point_metadata", "source_sha256", "source_witness", "deadline_epoch"} or + prepared["capture_id"] != capture_id or prepared["components"] != request["components"] or + not isinstance(prepared["point_metadata"], dict) or not isinstance(prepared["source_witness"], dict) or + type(prepared["deadline_epoch"]) not in (int, float) or + not math.isfinite(prepared["deadline_epoch"]) or time.time() >= prepared["deadline_epoch"]): + raise ValueError("Worker handoff is expired or differs from the original capture") + for entry in request["components"]: + copied = _archive_component(container, entry["source_path"], stage / "workers" / entry["id"]) + components.append({"id": entry["id"], "source_path": str(copied)}) + if prepared is not None: + observed = effect_runtime_result("runtime.checkpoint_artifacts", { + "action": "probe", "components": components, + }, retry_safe=False) + if observed["source_sha256"] != prepared["source_sha256"]: + changed = [] + for key, rows in observed["source_witness"].items(): + before = {row["path"]: row for row in prepared["source_witness"].get(key, [])} + after = {row["path"]: row for row in rows} + changed.extend(key + ":" + path for path in sorted(before.keys() | after.keys()) + if before.get(path) != after.get(path)) + raise ValueError("Worker source changed after native/research preparation: " + ", ".join(changed)) + if time.time() >= prepared["deadline_epoch"]: + raise TimeoutError("Worker handoff expired before evaluator capture") + phase = _archive(container, request["phase_deadline_path"], stage / "phase") + native = capture(controller_file, token=token, capture_id=capture_id, + destination=stage / "native", timeout=timeout, phase_deadline_file=phase) + container.reload() + if not container.attrs["State"]["Running"] or not container.attrs["State"]["Paused"]: + raise RuntimeError("Worker writer barrier changed during capture") + finally: + # An uncertain/lost reply keeps this original attempt's + # cleanup obligation. Its identity cannot be retried later. + if pause_may_have_committed: + container.reload() + if container.attrs["State"]["Paused"]: + container.unpause() + container.reload() + if container.attrs["State"]["Paused"] or not container.attrs["State"]["Running"]: + raise RuntimeError("Worker was not restored to running; inspect the original container before retry") + native_component = _participant_id(components, "native_capsule") + components.append({"id": native_component, "source_path": str(stage / "native")}) + metadata = {**(prepared["point_metadata"] if prepared is not None else {}), "worker_file_snapshot": { + "container_id": container.id, "image_id": container.attrs["Image"], + "components": request["components"], "phase_deadline_path": request["phase_deadline_path"], + "native_capsule_sha256": native["sha256"], + "native_capsule_component_id": native_component, + "writer_barrier": "docker_pause", "process_memory_captured": False, + "native_cursor_qualified": False, + **({"handoff_sha256": request["handoff"]["sha256"], + "prepared_source_verified": True} if prepared is not None else {}), + }} + if prepared is not None: + metadata["writer_barrier"] = "docker_pause" + probe = effect_runtime_result("runtime.checkpoint_artifacts", {"action": "probe", "components": components}) + return effect_runtime_result("runtime.checkpoint_artifacts", { + "action": "capture", "components": components, "destination": str(destination), + "expected_source_sha256": probe["source_sha256"], "point_metadata": metadata, + }, retry_safe=False) + finally: + client.close() diff --git a/benchmark/edgebench/run.py b/benchmark/edgebench/run.py index 05b2bf6c3b..7de47b8ec4 100644 --- a/benchmark/edgebench/run.py +++ b/benchmark/edgebench/run.py @@ -119,6 +119,10 @@ def main(argv=None): help="Explicit completed-Todo cadence ablation for heartbeat profiles") parser.add_argument("--stride-strategy", choices=stride_strategy_options(), help="Explicit heartbeat Goal-review strategy; omitted keeps the ordinary off treatment") + parser.add_argument("--checkpoint-controller-file", type=Path, + help="Host-private native checkpoint controller; requires full fresh-replan handoff") + parser.add_argument("--checkpoint-request", type=Path, + help="Explicit host JSON declaration staged into a heartbeat worker; omitted disables capture") parser.add_argument("--eval-interval", type=int, help="Native default: 0 (no periodic evaluation); other modes use task defaults; explicit 0 disables") parser.add_argument("--submission-cooldown", type=int, @@ -142,6 +146,15 @@ def main(argv=None): parser.error("Replan cadence requires a heartbeat worker") if args.stride_strategy is not None and not args.worker.startswith("heartbeat-"): parser.error("--stride-strategy requires a heartbeat worker") + checkpoint_request = None + if args.checkpoint_request is not None: + if not args.worker.startswith("heartbeat-"): + parser.error("--checkpoint-request requires a heartbeat worker") + from benchmark.runtime.checkpoint import read_checkpoint_request + declaration, checkpoint_request = read_checkpoint_request(str(args.checkpoint_request)) + handoff = checkpoint_request is not None and declaration.get("controller_handoff", False) + if bool(args.checkpoint_controller_file) != handoff or (handoff and args.feedback != "native"): + parser.error("controller file and full controller_handoff must be enabled together with native feedback") # One directory is one attempt: never reuse native registration or overwrite # source/profile evidence after an ambiguous launch. trial = args.log_dir / "runs" / args.run_id / args.task @@ -196,7 +209,9 @@ def main(argv=None): replan_after_turns=args.replan_after_turns, replan_after_todos=args.replan_after_todos, **({"stride_strategy": args.stride_strategy} - if args.stride_strategy is not None else {})) + if args.stride_strategy is not None else {}), + **({"checkpoint_request": checkpoint_request} + if checkpoint_request is not None else {})) if args.api_proxy_url: agent.default_api_base_url = args.api_proxy_url logger = logging.getLogger("edgebench-runtime") @@ -214,9 +229,14 @@ def main(argv=None): judge_url=args.judge_url.replace("host.docker.internal", "127.0.0.1"), admin_secret=get_admin_secret(args.log_dir), logger=logger, sampler=sampler, ) if args.feedback == "best-only" else None) + checkpoint = None + if args.checkpoint_controller_file is not None: + from benchmark.edgebench.checkpoint_supervisor import CheckpointSupervisor + checkpoint = CheckpointSupervisor(args.checkpoint_controller_file, trial / "checkpoints") backend = RecordingDockerBackend(log_dir=trial / "collected", logger=logger, oauth_proxy=bool(args.api_proxy_url), - blind_api_endpoint=blind_endpoint, feedback=feedback) + blind_api_endpoint=blind_endpoint, feedback=feedback, + **({"checkpoint": checkpoint} if checkpoint is not None else {})) for image in (task.work_image_key, task.judge_image_key): if not backend.image_exists(image): raise RuntimeError(f"Missing native image: {image}") @@ -242,6 +262,8 @@ def main(argv=None): if agent.replan_after_todos is not None else {}), **({"stride_strategy": args.stride_strategy} if args.stride_strategy is not None else {}), + **({"checkpoint_request_sha256": hashlib.sha256(checkpoint_request).hexdigest()} + if checkpoint_request is not None else {}), } receipt_path = trial / "runtime-receipt.json" receipt_path.write_text(json.dumps(receipt, indent=2)) diff --git a/benchmark/runtime/checkpoint.py b/benchmark/runtime/checkpoint.py new file mode 100644 index 0000000000..8cb5d6b494 --- /dev/null +++ b/benchmark/runtime/checkpoint.py @@ -0,0 +1,592 @@ +"""Opt-in inert artifact capture after the existing benchmark wake has settled. + +The TS backup/artifact owner verifies source stability and materialized bytes. +Python adapts the owned wake, source admission and explicit operator file list; +it never restores claims, starts a model or certifies a complete research point. +""" +from __future__ import annotations + +import hashlib +import json +import math +import os +import subprocess +import time +from contextlib import contextmanager +from pathlib import Path, PurePosixPath +from typing import Any + +from loopx.control_plane.effect_runtime import effect_runtime_result +from loopx.control_plane.quota.settlement import read_heartbeat_settlement + + +def normalize_checkpoint_request_path(value: str) -> str: + """An explicit guest path; no ambient host variable enables capture.""" + if (not isinstance(value, str) or not value or + not PurePosixPath(value).is_absolute() or str(PurePosixPath(value)) != value or + ".." in PurePosixPath(value).parts): + raise ValueError("checkpoint_request_path must be a normalized absolute guest file path") + return value + + +def read_checkpoint_request(path: str) -> tuple[dict[str, Any], bytes]: + raw = Path(path).read_bytes() + return parse_checkpoint_request(raw), raw + + +def parse_checkpoint_request(raw: bytes) -> dict[str, Any]: + """Validate the same explicit declaration before staging and before capture.""" + request = json.loads(raw) + if (not isinstance(request, dict) or + set(request) - {"destination", "components", "point_metadata", "native_session", "evaluation_opportunity", "loopx_state", "trigger", "controller_handoff"} or + not {"destination", "components", "point_metadata"}.issubset(request)): + raise ValueError("checkpoint request requires destination, components and point_metadata") + if "native_session" in request and type(request["native_session"]) is not bool: + raise ValueError("checkpoint native_session must be an explicit boolean") + if "evaluation_opportunity" in request and type(request["evaluation_opportunity"]) is not bool: + raise ValueError("checkpoint evaluation_opportunity must be an explicit boolean") + if "loopx_state" in request and type(request["loopx_state"]) is not bool: + raise ValueError("checkpoint loopx_state must be an explicit boolean") + if "controller_handoff" in request and type(request["controller_handoff"]) is not bool: + raise ValueError("checkpoint controller_handoff must be an explicit boolean") + if request.get("controller_handoff") and ( + request.get("trigger") != "fresh_replan" or not request.get("native_session") or + not request.get("loopx_state") or not request.get("evaluation_opportunity")): + raise ValueError("controller handoff requires fresh_replan and all three state participants") + if "trigger" in request and request["trigger"] != "fresh_replan": + raise ValueError("checkpoint trigger must be fresh_replan or omitted") + if not isinstance(request["components"], list) or not request["components"]: + raise ValueError("checkpoint requires explicit components") + if not isinstance(request["point_metadata"], dict): + raise ValueError("checkpoint point_metadata must be an object") + return request + + +def _participant_id(components, preferred): + """Automatic participants cannot reserve an otherwise legal caller ID.""" + occupied = {entry["id"] for entry in components} + candidate, suffix = preferred, 0 + while candidate in occupied: + suffix += 1 + candidate = f"{preferred}_{suffix}" + return candidate + + +def _loopx_state_components(env, components, output_dir): + """Adapt the existing backup owner's inventory for one isolated trial. + + No second state-discovery rule, archive execution or authority adoption. + The runtime owns its original evidence, quota and effect journals; this + adapter declares those private files to the inert artifact owner only. + """ + from loopx.history import load_registry + from loopx.state_backup import build_state_backup_plan + + project, runtime, registry = (Path(env[key]) for key in ( + "LOOPX_PROJECT", "LOOPX_RUNTIME_ROOT", "LOOPX_REGISTRY")) + for path in (project, runtime, registry): + if not path.is_absolute() or path.resolve(strict=True) != path: + raise ValueError("LoopX checkpoint routes must be normalized physical paths") + raw_registry = registry.read_bytes() + source = load_registry(registry) + goals = source.get("goals", []) + if (len(goals) != 1 or goals[0].get("id") != env["LOOPX_GOAL_ID"] or + env["LOOPX_AGENT_ID"] not in goals[0].get("coordination", {}).get("registered_agents", [])): + raise ValueError("LoopX state capture requires this worker's single-Goal trial registry") + # Reject lexical route aliases before backup discovery resolves them. + goal = goals[0] + if Path(goal["repo"]) != project or Path(source.get("common_runtime_root", "")) != runtime: + raise ValueError("LoopX checkpoint routes differ from the registered trial") + state = Path(goal["state_file"]) + state = state if state.is_absolute() else Path(goal["repo"]) / state + if not state.is_file(): + raise ValueError("LoopX checkpoint requires its registered active state source") + routes = [state, *(project / path for path in ( + ".loopx", ".codex/goals", ".claude/goals", ".local/goals"))] + if any(path.exists() and path.resolve(strict=True) != path for path in routes): + raise ValueError("LoopX checkpoint state routes must not contain symlinks") + plan = build_state_backup_plan(project=project, runtime_root=runtime, + registry_path=registry, output_dir=output_dir, + include_registry_projects=False, include_automations=False, include_skills=False) + if plan["warnings"]: + raise ValueError("LoopX backup inventory is incomplete; reconcile its source routes before capture") + # Select physical parent roots once. A caller-declared parent can already + # cover a discovered file; do not duplicate the same runtime/registry/state. + entries = list(components) + targets = sorted(plan["included"], key=lambda entry: len(Path(entry["source_path"]).parts)) + for target in targets: + path = Path(target["source_path"]) + if any(path.is_relative_to(Path(entry["source_path"])) for entry in entries): + continue + if any(Path(entry["source_path"]).is_relative_to(path) for entry in entries): + raise ValueError("LoopX state inventory overlaps a narrower declared component; declare its parent once") + entries.append({"id": _participant_id(entries, f"loopx_state_{len(entries)}"), "source_path": str(path)}) + coverage = [{"key": target["key"], "source_path": target["source_path"], + "component_id": next(entry["id"] for entry in entries + if Path(target["source_path"]).is_relative_to(Path(entry["source_path"]))) } + for target in plan["included"]] + if registry.read_bytes() != raw_registry: + raise ValueError("LoopX registry moved during participant inventory") + registry_component = next(entry for entry in entries if registry.is_relative_to(Path(entry["source_path"]))) + return entries, {"goal_id": env["LOOPX_GOAL_ID"], "agent_id": env["LOOPX_AGENT_ID"], + "source_inventory": coverage, "registry_sha256": hashlib.sha256(raw_registry).hexdigest(), + "registry_component_id": registry_component["id"], + "registry_member_path": (registry.relative_to(Path(registry_component["source_path"])).as_posix() + if registry != Path(registry_component["source_path"]) else ""), + "execution_authority_granted": False} + + +def native_binding_participant(env, components, metadata): + from .codex import Execution + from .sessions import BenchmarkSessionWake + source_wake = BenchmarkSessionWake(env, Execution(context="resume", + sandbox=env.get("LOOPX_CODEX_SANDBOX", "danger-full-access")), {}) + if source_wake.session_id != metadata["session"].get("session_id"): + raise ValueError("native checkpoint differs from the current source session binding") + # This sideband record belongs to the existing session owner. It + # must participate even when loopx_state was not requested: native + # store bytes alone cannot witness a binding/profile replacement. + from loopx.control_plane.turn_driver.codex_sessions import _session_path + binding_path = _session_path(source_wake.root, source_wake.lineage, "agent") + if not any(binding_path.is_relative_to(Path(entry["source_path"])) for entry in components): + components = [*components, {"id": _participant_id(components, "source_session_binding"), + "source_path": str(binding_path)}] + metadata["session"] = {**metadata["session"], "session_profile_digest": source_wake.digest} + return components, source_wake + + +def research_participants(env, components, metadata, destination, evidence_path, deadline): + """Use the same inventory and historical context at either owned boundary.""" + research = None + components, metadata["loopx_state"] = _loopx_state_components(env, components, destination) + from loopx.control_plane.coordination.local_authority import local_authority_is_promoted + if local_authority_is_promoted(runtime_root=Path(env["LOOPX_RUNTIME_ROOT"]), goal_id=env["LOOPX_GOAL_ID"]): + from .research_checkpoint import research_context, _encoded + research = research_context(env, deadline=deadline) + evidence = evidence_path + research_raw = _encoded(research) + with os.fdopen(os.open(evidence, os.O_WRONLY | os.O_CREAT | os.O_EXCL, 0o600), "wb") as file: + file.write(research_raw) + parent = next((entry for entry in components if evidence.is_relative_to(Path(entry["source_path"]))), None) + if parent is None: + parent = {"id": _participant_id(components, "research_context"), "source_path": str(evidence)} + components = [*components, parent] + metadata["research_context"] = {"component_id": parent["id"], + "member_path": evidence.relative_to(Path(parent["source_path"])).as_posix() if evidence != Path(parent["source_path"]) else "", + "sha256": hashlib.sha256(research_raw).hexdigest(), + "provider_revision": research["snapshot"]["provider_revision"], "store_identity": research["snapshot"]["store_identity"]} + else: + metadata["canonical_context_unavailable"] = "Source coordination is unpromoted; its retained inventory remains inert." + return components, research + + +def verify_research_participants(env, components, metadata, research, witness, deadline): + from .research_checkpoint import research_context, verify_context_sources + if research is not None: + if research_context(env, deadline=deadline) != research: + raise ValueError("canonical research context moved before sealing") + verify_context_sources(research, components, witness) + retained = metadata["research_context"] + member = next((entry for entry in witness[retained["component_id"]] + if entry["path"] == retained["member_path"]), None) + if member is None or member.get("sha256") != retained["sha256"]: + raise ValueError("canonical research context bytes moved before sealing") + declared = metadata["loopx_state"] + witness = witness[declared["registry_component_id"]] + member = next((entry for entry in witness if entry["path"] == declared["registry_member_path"]), None) + if member is None or member.get("sha256") != declared["registry_sha256"]: + raise ValueError("LoopX registry differs from the scoped participant inventory") + + +def _mark_replan_attempt(root: Path, obligation_id: str, turn_id: str, request_sha256: str | None) -> bool: + """The first boundary wins even when its optional declaration was lost.""" + if not root.is_absolute() or root != root.resolve(): + raise ValueError("replan checkpoint root must be a normalized physical path") + attempts = root / ".replan-attempts" + attempts.mkdir(parents=True, exist_ok=True, mode=0o700) + if attempts != attempts.resolve(): + raise ValueError("replan attempt directory must not contain symlinks") + attempt_path = attempts / (obligation_id + ".json") + attempt = {"obligation_id": obligation_id, "turn_id": turn_id, + "request_sha256": request_sha256} + try: + with attempt_path.open("x", encoding="utf-8") as marker: + marker.write(json.dumps(attempt)) + attempt_path.chmod(0o600) + return True + except FileExistsError: + if attempt_path.is_symlink(): + raise ValueError("replan attempt must not be a symlink") + prior_attempt = json.loads(attempt_path.read_text()) + # None means the first declaration was unavailable, not permission to + # replace that lost point with a repaired file and later model state. + if (prior_attempt["obligation_id"] != obligation_id or + prior_attempt["request_sha256"] not in (None, request_sha256)): + raise ValueError("existing replan attempt has a different declaration") + return False + + +def _prior_replan_guard_seen(env: dict[str, str], current: Path, obligation_id: str) -> bool: + """An absent optional index does not erase the original Host preflight. + + Reuse saved guard packets and the typed replan selector. They are already + written before optional capture and before the model can decide or work. + No separate journal, model-text classifier or new replan owner is needed. + """ + for path in sorted(Path(env["LOOPX_WAKE_LOG_DIR"]).glob("*/checkpoint-start-guard.json")): + if path == current: + continue + if path.resolve(strict=True) != path: + raise ValueError("prior checkpoint guard must be a physical Host file") + guard = json.loads(path.read_text()) + prior = effect_runtime_result("runtime.checkpoint_artifacts", { + "action": "select_replan", "quota": guard, "goal_id": env["LOOPX_GOAL_ID"], + "agent_id": env["LOOPX_AGENT_ID"], "turn_instance_id": path.parent.name, + }) + if prior["capture"] and prior["obligation_id"] == obligation_id: + return True + return False + + +def capture_before_replan(env: dict[str, str], receipt: dict[str, Any], body: str) -> str: + """Preload the original guard once, then capture before invoking the model. + + The typed owner selects only an admitted replan identity. The prior wake's + existing settlement owner still qualifies the source boundary. A missed + point is explicit and cannot be replaced by an after-decision capture. + """ + path = env.get("LOOPX_CHECKPOINT_REQUEST") + if not path: + return body + declaration_error = None + try: + request, raw = read_checkpoint_request(path) + except (OSError, ValueError) as error: + declaration_error = error + request, raw = {}, None + if declaration_error is None and request.get("trigger") != "fresh_replan": + return body + if receipt.get("mode") != "heartbeat" or receipt.get("stage") != "execute": + raise ValueError("fresh_replan checkpoint trigger requires a heartbeat execute wake") + started = time.monotonic() + guard_path = Path(env["LOOPX_WAKE_LOG_DIR"]) / receipt["turn_id"] / "checkpoint-start-guard.json" + command = [env["LOOPX_CLI"], "--format", "json", "--registry", env["LOOPX_REGISTRY"], + "--runtime-root", env["LOOPX_RUNTIME_ROOT"], "quota", "should-run", + "--goal-id", env["LOOPX_GOAL_ID"], "--agent-id", env["LOOPX_AGENT_ID"], + "--runtime-profile", "generic_cli", "--turn-instance-id", receipt["turn_id"], + "--available-capability", "shell", "--available-capability", "filesystem_write"] + guard_run = subprocess.run(command, cwd=env["LOOPX_PROJECT"], env=env, capture_output=True, + text=True, timeout=120, check=True) + guard = json.loads(guard_run.stdout) + if not isinstance(guard, dict) or guard.get("ok") is not True: + raise ValueError("checkpoint requires a successful original quota guard") + # A previous preflight on this wake may already have allowed model work, + # even if its optional attempt marker (or final receipt) could not be saved. + # Such reentry can verify an existing point, but cannot capture a later one. + preflight_seen = guard_path.exists() + guard_path.write_text(guard_run.stdout, encoding="utf-8") + guard_path.chmod(0o600) + preloaded_body = (f"Host checkpoint preflight has already executed this wake's original quota guard once. " + f"Read the complete packet at {guard_path} before work, then follow its interaction_contract. " + "Do not repeat the initial guard below; use its selection or recovery command when required.\n\n" + body) + receipt["checkpoint"] = {"captured": False, "research_point_qualified": False, "phase": "before_replan"} + try: + intent = effect_runtime_result("runtime.checkpoint_artifacts", { + "action": "select_replan", "quota": guard, "goal_id": env["LOOPX_GOAL_ID"], + "agent_id": env["LOOPX_AGENT_ID"], "turn_instance_id": receipt["turn_id"], + }) + root = Path(env["LOOPX_WAKE_LOG_DIR"]).parent / "checkpoints" + if declaration_error is not None: + if intent["capture"]: + _mark_replan_attempt(root, intent["obligation_id"], receipt["turn_id"], None) + raise declaration_error + if not intent["capture"]: + receipt["checkpoint"]["reason"] = intent["reason"] + else: + destination = root / intent["obligation_id"] + if destination.parent != root or destination != destination.resolve(): + raise ValueError("replan checkpoint identity must select a direct child") + if not destination.exists() and (preflight_seen or + _prior_replan_guard_seen(env, guard_path, intent["obligation_id"]) or + not _mark_replan_attempt(root, intent["obligation_id"], receipt["turn_id"], + hashlib.sha256(raw).hexdigest())): + receipt["checkpoint"]["reason"] = "pre_decision_capture_already_attempted" + return preloaded_body + requested_root = Path(request["destination"]) + if not requested_root.is_absolute() or requested_root != requested_root.resolve() or requested_root != root: + raise ValueError("fresh_replan destination must be this trial's checkpoints directory") + if destination.exists(): + # Readback of the first point, never overwrite it with later work. + manifest = json.loads((destination / "checkpoint.json").read_text()) + metadata = manifest["point_metadata"] + if (metadata["request_sha256"] != hashlib.sha256(raw).hexdigest() or + metadata["operator_metadata"]["replan"]["obligation_id"] != intent["obligation_id"]): + raise ValueError("existing replan point has a different declaration") + verified = effect_runtime_result("runtime.checkpoint_artifacts", { + "action": "verify", "root": str(destination), "expected_sha256": manifest["sha256"], + }) + receipt["checkpoint"].update(reason="replan_already_captured", destination=str(destination), **verified) + else: + # A failed first capture is a missed point. Once its model is + # allowed to decide/work, a retry must not relabel later state + # as the original pre-decision checkpoint. + prior_paths = sorted(Path(env["LOOPX_WAKE_LOG_DIR"]).glob("wake-*/receipt.json")) + prior_paths = [p for p in prior_paths if p.parent.name != receipt["turn_id"]] + if not prior_paths: + raise ValueError("replan capture has no completed prior Host wake") + prior = json.loads(prior_paths[-1].read_text()) + if prior.get("stage") != "execute" or prior.get("mode") != "heartbeat" or not prior.get("ok"): + raise ValueError("replan capture requires the latest successful work wake") + effective_request = {**request, "destination": str(destination), + "point_metadata": {**request["point_metadata"], "replan": intent, + "capture_order": "before_model_replan_decision"}} + effective_request.pop("trigger") + capture_after_wake(env, prior, supplied_request=(effective_request, raw)) + receipt["checkpoint"].update(prior["checkpoint"], phase="before_replan") + receipt["checkpoint"]["source_turn_id"] = prior["turn_id"] + except Exception as error: + receipt["checkpoint"].update(reason="capture_failed", error_kind=type(error).__name__, error=str(error)[:500]) + finally: + receipt["checkpoint"]["elapsed_seconds"] = time.monotonic() - started + # The model consumes exactly the owner's original decision and keeps its + # selection, settlement, required-read and recovery instructions intact. + return preloaded_body + + +def capture_after_wake(env: dict[str, str], receipt: dict[str, Any], *, + supplied_request: tuple[dict[str, Any], bytes] | None = None) -> None: + request_path = env.get("LOOPX_CHECKPOINT_REQUEST") + if not request_path: + return # Feature-off path performs no additional read or write. + if supplied_request is None and (receipt.get("checkpoint") or {}).get("phase") == "before_replan": + return # The after-wake hook cannot replace the pre-decision point. + receipt["checkpoint"] = {"captured": False, "research_point_qualified": False} + if (not receipt.get("ok") or receipt.get("stage") != "execute" or + receipt.get("mode") not in {"heartbeat", "turn"} or receipt.get("budget_exhausted")): + receipt["checkpoint"]["reason"] = "wake_not_successful_work" + return + started = time.monotonic() + try: + phase_deadline = env.get("LOOPX_PHASE_DEADLINE_EPOCH") + deadline = time.time() + 30 + if phase_deadline is not None: + original_deadline = float(phase_deadline) + if not math.isfinite(original_deadline): + raise ValueError("checkpoint requires a finite original phase deadline") + deadline = min(deadline, original_deadline) + if deadline <= time.time(): + receipt["checkpoint"]["reason"] = "capture_window_closed" + return + if supplied_request is None: + request, raw = read_checkpoint_request(request_path) + else: + request, raw = supplied_request + if request.get("trigger") == "fresh_replan": + receipt["checkpoint"]["reason"] = "pre_replan_hook_required" + return + # This is a private evidence artifact below the runner's log owner. + destination = Path(request["destination"]) + root = Path(env["LOOPX_WAKE_LOG_DIR"]).parent / "checkpoints" + if not destination.is_absolute() or destination != destination.resolve() or destination.parent != root: + raise ValueError("checkpoint destination must be a normalized direct child of this trial's checkpoints directory") + if destination.exists(): + receipt["checkpoint"]["reason"] = "destination_already_present_no_replay" + return + observed = read_heartbeat_settlement( + Path(env["LOOPX_RUNTIME_ROOT"]), goal_id=env["LOOPX_GOAL_ID"], + agent_id=env["LOOPX_AGENT_ID"], todo_id=None, + turn_instance_id=receipt["turn_id"], resolve_original_binding=True, + registry_path=Path(env["LOOPX_REGISTRY"]), + ) + if observed is None or (observed.progress or {}).get("state") != "settled": + receipt["checkpoint"]["reason"] = "owned_turn_not_settled" + return + identity = observed.identity.value + if identity is None: + raise ValueError("checkpoint settlement identity is unavailable") + # Only this returned typed owner decision admits the capture boundary. + # Point metadata remains evidence, never authority or scientific validity. + metadata = { + "operator_metadata": request["point_metadata"], + "wake_turn_id": receipt["turn_id"], + "settlement_effect_id": identity.effect_id, + "session": receipt.get("session"), + "request_sha256": hashlib.sha256(raw).hexdigest(), + } + components = request["components"] + if request.get("loopx_state"): + evidence = Path(env["LOOPX_WAKE_LOG_DIR"]) / receipt["turn_id"] / "checkpoint-research-context.json" + components, research = research_participants(env, components, metadata, destination, evidence, deadline) + opportunity = None + if request.get("evaluation_opportunity") and not request.get("controller_handoff"): + from .sforge_checkpoint import capture_opportunity + + evidence = Path(env["LOOPX_WAKE_LOG_DIR"]) / receipt["turn_id"] / "checkpoint-opportunity.json" + incumbent = evidence.with_name("checkpoint-incumbent.tar.gz") + opportunity = capture_opportunity(env, timeout=max(.1, deadline - time.time()), incumbent_path=incumbent) + evidence.write_text(json.dumps(opportunity), encoding="utf-8") + evidence.chmod(0o600) + opportunity_id = _participant_id(components, "evaluation_opportunity") + components = [*components, {"id": opportunity_id, "source_path": str(evidence)}] + incumbent_id = None + if opportunity["native_incumbent"] is not None: + incumbent_id = _participant_id(components, "evaluation_incumbent") + components = [*components, {"id": incumbent_id, "source_path": str(incumbent)}] + # Local provider evidence mapping, not evaluation admission. Keep + # caller IDs intact and bind the witness to the allocated archive. + metadata["evaluation_opportunity"] = {"component_id": opportunity_id, + "incumbent_component_id": incumbent_id} + if request.get("native_session"): + from .native_checkpoint import native_session_checkpoint + from .codex import Execution + from .sessions import BenchmarkSessionWake + + session = receipt.get("session") or {} + components, source_wake = native_binding_participant(env, components, metadata) + native = native_session_checkpoint(env, session.get("session_id", ""), + timeout=max(.1, deadline - time.time())) + # Native names identify provider files, not reserved artifact IDs. + # Allocate in the same namespace as caller and other participants; + # retain the provider observation unchanged for the stability reread. + native_ids = {} + for entry in native["components"]: + component_id = _participant_id(components, entry["id"]) + native_ids[entry["id"]] = component_id + components = [*components, {**entry, "id": component_id}] + metadata["native_session"] = {**native["metadata"], "store_members": [ + {**member, "id": native_ids[member["id"]]} + for member in native["metadata"]["store_members"]]} + probe = effect_runtime_result("runtime.checkpoint_artifacts", { + "action": "probe", "components": components, + "deadline_epoch_ms": deadline * 1000, + }, timeout=max(.1, deadline - time.time()), retry_safe=False) + if request.get("native_session"): + # Native readback precedes the byte witness. A completed later Turn + # in that gap must not be sealed under the earlier cursor. Re-read + # after the probe; capture's existing witness comparison then guards + # source writes during this recheck and through atomic publication. + confirmed = native_session_checkpoint(env, session.get("session_id", ""), + timeout=max(.1, deadline - time.time())) + if confirmed != native: + raise ValueError("native checkpoint cursor or participant basis moved before sealing") + confirmed_wake = BenchmarkSessionWake(env, Execution(context="resume", + sandbox=env.get("LOOPX_CODEX_SANDBOX", "danger-full-access")), {}) + if ((confirmed_wake.session_id, confirmed_wake.goal_ref, confirmed_wake.digest) != + (source_wake.session_id, source_wake.goal_ref, source_wake.digest)): + raise ValueError("native checkpoint source session binding moved before sealing") + if request.get("loopx_state"): + verify_research_participants(env, components, metadata, research, probe["source_witness"], deadline) + if opportunity is not None and opportunity["native_incumbent"] is not None: + selected = opportunity["native_incumbent"] + witness = probe["source_witness"].get(metadata["evaluation_opportunity"]["incumbent_component_id"], []) + if (len(witness) != 1 or witness[0].get("path") != "" or witness[0].get("kind") != "file" or + witness[0].get("sha256") != selected["archive_sha256"] or + witness[0].get("size") != selected["archive_bytes"]): + raise ValueError("native incumbent participant differs from its captured result") + destination.parent.mkdir(parents=True, exist_ok=True) + if request.get("controller_handoff"): + if research is None: + raise ValueError("controller checkpoint requires complete canonical research context") + from .checkpoint_handoff import await_controller_capture + captured = await_controller_capture(root, components, metadata, probe, deadline) + receipt["checkpoint"] = {"captured": True, **captured} + return + captured = effect_runtime_result("runtime.checkpoint_artifacts", { + "action": "capture", "destination": str(destination), + "components": components, "point_metadata": metadata, + "expected_source_sha256": probe["source_sha256"], + "deadline_epoch_ms": deadline * 1000, + }, timeout=max(.1, deadline - time.time()), retry_safe=False) + receipt["checkpoint"] = {"captured": True, "destination": str(destination), **captured} + except Exception as error: + # Capture failure is explicit but cannot rewrite an already settled work + # outcome or replay its model/evaluation. Keep the source trial running. + receipt["checkpoint"].update(reason="capture_failed", error_kind=type(error).__name__, error=str(error)[:500]) + finally: + receipt["checkpoint"]["elapsed_seconds"] = time.monotonic() - started + + +@contextmanager +def held_quota_decision(listener, capture_root: Path, *, goal_id: str, agent_id: str, + turn_instance_id: str | None, native_process=None, interrupt_timeout: float = 10, + interrupt_when=None, deadline: float | None = None): + """Host-side counterpart of the explicit quota pre-output transport. + + The existing typed owner decides whether this exact packet admits a fresh + replan. The caller must capture while inside the context. Normal exit only + releases the original tool response; it certifies no research checkpoint. + Failure/disconnection never retries the guard or substitutes a new point. + The listener must already be bound by the Host in a private directory. + + Passing the live child_process-owned native CLI explicitly interrupts and + reaps it before selection/capture. Its pending tool is aborted; normal exit + returns a source-only continuation body containing the original packet, + rather than acknowledging a dead socket. This is not a complete workspace + barrier, destination admission, or a qualified interrupted native cursor. + interrupt_when selects capture candidates through the typed intent before + interruption; false releases ordinary/duplicate guards without a native + restart. Selection is not a pre-decision proof: interrupted cursor + qualification must still reject any native work beyond the pending guard. + Without native_process this is output ordering only; a slow capture can + outlive the native tool's yield interval and does not stop inference. + """ + if not capture_root.is_absolute() or capture_root.resolve(strict=True) != capture_root: + raise ValueError("decision capture root must be a physical absolute directory") + def remaining(): + limit = 120 if deadline is None else min(120, deadline - time.monotonic()) + if limit <= 0: + raise TimeoutError("original wake budget expired during decision handoff") + return limit + + channel, _ = listener.accept() + with channel: + channel.settimeout(remaining()) + with channel.makefile("rb") as stream: + line = stream.readline(4097) + message = json.loads(line) + if (not isinstance(message, dict) or set(message) != {"path", "sha256"} or + not isinstance(message["path"], str) or not isinstance(message["sha256"], str)): + raise ValueError("invalid decision handoff request") + path = Path(message["path"]) + if (path.name != "decision.json" or path.parent.parent != capture_root or + path.resolve(strict=True) != path): + raise ValueError("decision handoff must reference this Host's capture root") + raw = path.read_bytes() + if hashlib.sha256(raw).hexdigest() != message["sha256"]: + raise ValueError("decision handoff digest mismatch") + quota = json.loads(raw) + interrupted = None + if native_process is not None and interrupt_when is None: + from .worker import interrupt_checkpoint_process + + # Interrupt before the typed selector: even this read can outlive a + # native tool yield. Keep original bytes for same-identity recovery. + interrupted = interrupt_checkpoint_process(native_process, timeout=min(interrupt_timeout, remaining())) + intent = effect_runtime_result("runtime.checkpoint_artifacts", { + "action": "select_replan", "quota": quota, "goal_id": goal_id, + "agent_id": agent_id, "turn_instance_id": (turn_instance_id if turn_instance_id is not None + else (quota.get("heartbeat_receipt") or {}).get("turn_instance_id")), + }, timeout=remaining()) + if native_process is not None and interrupt_when is not None and interrupt_when(intent): + from .worker import interrupt_checkpoint_process + + interrupted = interrupt_checkpoint_process(native_process, timeout=min(interrupt_timeout, remaining())) + held = {"path": path, "sha256": message["sha256"], "quota": quota, "intent": intent} + if interrupted is not None: + held["native_process_return_code"] = interrupted + yield held + if path.read_bytes() != raw: + raise ValueError("original decision changed while held") + if interrupted is None: + channel.sendall(json.dumps({"released": True, "sha256": message["sha256"]}).encode() + b"\n") + else: + # Expose only after capture/readback succeeds. A failing consumer + # cannot mistake the interruption for ordinary continuation. + held["continuation_body"] = ( + "The Host interrupted this source session while its quota tool response was held. " + "The pending tool was aborted; its guard already committed. Consume the original " + "packet below in this same source session and preserve its Turn identity, required " + "reads, selection and settlement instructions. Do not rerun that guard or infer " + "a successful checkpoint from this message. This grants no destination authority.\n\n" + + raw.decode("utf-8") + ) diff --git a/benchmark/runtime/checkpoint_handoff.py b/benchmark/runtime/checkpoint_handoff.py new file mode 100644 index 0000000000..753ab6fc95 --- /dev/null +++ b/benchmark/runtime/checkpoint_handoff.py @@ -0,0 +1,56 @@ +"""Private file rendezvous while the owned native process remains interrupted. + +Only the external controller can freeze all worker writers and capture native +evaluator authority. This transport grants neither capability to the worker. +""" +from __future__ import annotations + +import hashlib +import json +import os +import time +import uuid + + +def await_controller_capture(root, components, metadata, probe, deadline): + directory = root.parent / "checkpoint-controller" + directory.mkdir(mode=0o700, exist_ok=True) + if directory.resolve() != directory: + raise ValueError("checkpoint controller mailbox must be physical") + nonce = uuid.uuid4().hex + ready = directory / "ready.json" + reply = directory / (nonce + ".json") + value = {"capture_id": nonce, "components": components, + "point_metadata": metadata, "source_sha256": probe["source_sha256"], + "source_witness": probe["source_witness"], + "deadline_epoch": deadline} + raw = json.dumps(value, sort_keys=True, separators=(",", ":")).encode() + # A surviving older request is an explicit recovery problem, not authority + # to replace a missed point with this later one. + pending = directory / (nonce + ".request") + try: + with pending.open("xb") as file: + os.chmod(pending, 0o600) + file.write(raw) + file.flush() + os.fsync(file.fileno()) + os.link(pending, ready) # Atomic publication without replacing an old request. + finally: + pending.unlink() + try: + while time.time() < deadline: + if reply.exists(): + if reply.is_symlink(): + raise ValueError("checkpoint controller reply must be physical") + result = json.loads(reply.read_bytes()) + if result.get("request_sha256") != hashlib.sha256(raw).hexdigest(): + raise ValueError("checkpoint controller reply belongs to another boundary") + if result.get("ok") is not True: + raise ValueError("controller checkpoint failed: " + str(result.get("error_kind"))) + return result["capture"] + time.sleep(.05) + raise TimeoutError("controller did not capture before the original checkpoint deadline") + finally: + # The controller must reread this exact file under its pause barrier. + # Once absent, a late controller cannot capture a resumed source. + ready.unlink() diff --git a/benchmark/runtime/checkpoint_wake.py b/benchmark/runtime/checkpoint_wake.py new file mode 100644 index 0000000000..3d3a68c164 --- /dev/null +++ b/benchmark/runtime/checkpoint_wake.py @@ -0,0 +1,208 @@ +"""Opt-in native CLI rendezvous inside an existing heartbeat wake. + +This adapter owns process/transport ordering, not replan admission or complete +research-point qualification. Typed quota/artifact owners retain those rules. +""" +from __future__ import annotations + +import hashlib +import select +import socket +import subprocess +import sys +import tempfile +import time +from contextlib import ExitStack +from dataclasses import replace +from pathlib import Path + +from loopx.control_plane.effect_runtime import effect_runtime_result +from .checkpoint import _mark_replan_attempt, _participant_id, held_quota_decision, read_checkpoint_request + + +class CheckpointWake: + """One private listener survives all source-session resumptions in a wake.""" + + def __init__(self, env, receipt): + self.env, self.receipt = env, receipt + self.stack = ExitStack() + self.listener = None + self.socket_path = None + self.request = None + self.raw = None + self.root = Path(env["LOOPX_WAKE_LOG_DIR"]).parent / "checkpoints" + self.capture_root = Path(env["LOOPX_WAKE_LOG_DIR"]) / receipt["turn_id"] / "decisions" + + def __enter__(self): + path = self.env.get("LOOPX_CHECKPOINT_REQUEST") + if not path or self.receipt["mode"] != "heartbeat" or self.receipt["stage"] != "execute": + return self + try: + self.request, self.raw = read_checkpoint_request(path) + except (OSError, ValueError): + return self # The existing preflight records declaration failure. + if self.request.get("trigger") != "fresh_replan": + return self + if Path(self.request["destination"]) != self.root or self.root.resolve() != self.root: + raise ValueError("fresh_replan destination must be this trial's checkpoints directory") + try: + # AF_UNIX paths are short; trial roots need not be. No ambient socket + # or socket authority is inherited from the caller's environment. + directory = self.stack.enter_context(tempfile.TemporaryDirectory(prefix="loopx-ckpt-", dir="/tmp")) + self.socket_path = str(Path(directory).resolve() / "host.sock") + self.listener = self.stack.enter_context(socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)) + self.listener.bind(self.socket_path) + self.listener.listen(1) + return self + except BaseException: + self.stack.close() + raise + + def __exit__(self, *args): + return self.stack.__exit__(*args) + + def _first_replan(self, intent): + if not intent["capture"] or not _mark_replan_attempt( + self.root, intent["obligation_id"], self.receipt["turn_id"], + hashlib.sha256(self.raw).hexdigest()): + return False + if self.request.get("native_session") and sys.platform == "darwin": + # The current Darwin CLI can persist only the user item after an + # interrupted tool, losing the cursor needed for a verified fork. + # Refuse before signalling it; neither a delay nor a relaxed history + # predicate can recover that point. Completed-wake capture is separate. + self.receipt.setdefault("checkpoint_handoffs", []).append({ + "obligation_id": intent["obligation_id"], "captured": False, + "research_point_qualified": False, "error_kind": "ValueError", + "error": "Interrupted native checkpoint capture is not qualified on Darwin; original tool continues.", + }) + return False + return True + + def _capture(self, held, deadline): + """Capture declared files/native history; never downgrade research/effects.""" + if (any(self.request.get(key) for key in ("loopx_state", "evaluation_opportunity")) + and not self.request.get("controller_handoff")): + raise ValueError("Interrupted full-state capture requires the controller writer barrier") + destination = self.root / held["intent"]["obligation_id"] + metadata = {"operator_metadata": self.request["point_metadata"], + "request_sha256": hashlib.sha256(self.raw).hexdigest(), + "session": self.receipt["session"], "replan": held["intent"], + "decision_sha256": held["sha256"], + "capture_order": "after_native_interruption", + "writer_barrier": "owned_native_process_group_only"} + window = min(time.time() + 30, time.time() + deadline - time.monotonic()) + if window <= time.time(): + raise TimeoutError("original wake budget expired before capture") + components = list(self.request["components"]) + native = None + research = None + if self.request.get("loopx_state"): + from .checkpoint import research_participants + evidence = held["path"].with_name(held["path"].stem + "-research.json") + components, research = research_participants(self.env, components, metadata, destination, evidence, window) + if research is None: + raise ValueError("controller checkpoint requires complete canonical research context") + if self.request.get("native_session"): + from .native_checkpoint import native_session_checkpoint + from .checkpoint import native_binding_participant + components, source_wake = native_binding_participant(self.env, components, metadata) + guard = {"cli": self.env["LOOPX_CLI"], "workspace": self.env["LOOPX_PROJECT"], + "registry": self.env["LOOPX_REGISTRY"], "runtime_root": self.env["LOOPX_RUNTIME_ROOT"], + "goal_id": self.env["LOOPX_GOAL_ID"], "agent_id": self.env["LOOPX_AGENT_ID"], + "turn_instance_id": held["quota"]["heartbeat_receipt"]["turn_instance_id"], + "capture_root": str(self.capture_root), "socket_path": self.socket_path, + "decision_sha256": held["sha256"]} + native = native_session_checkpoint(self.env, self.receipt["session"]["session_id"], + timeout=max(.1, window - time.time()), interrupted_guard=guard) + allocated = {} + for entry in native["components"]: + component_id = _participant_id(components, entry["id"]) + allocated[entry["id"]] = component_id + components.append({**entry, "id": component_id}) + decision_id = _participant_id(components, "held_decision") + components.append({"id": decision_id, "source_path": str(held["path"])}) + metadata["native_session"] = {**native["metadata"], "store_members": [ + {**member, "id": allocated[member["id"]]} for member in native["metadata"]["store_members"]]} + metadata["held_decision_component_id"] = decision_id + probe = effect_runtime_result("runtime.checkpoint_artifacts", { + "action": "probe", "components": components, "deadline_epoch_ms": window * 1000, + }, timeout=max(.1, window - time.time()), retry_safe=False) + if native is not None: + confirmed = native_session_checkpoint(self.env, self.receipt["session"]["session_id"], + timeout=max(.1, window - time.time()), interrupted_guard=guard) + from .sessions import BenchmarkSessionWake + from .codex import Execution + confirmed_wake = BenchmarkSessionWake(self.env, Execution(context="resume", + sandbox=self.env.get("LOOPX_CODEX_SANDBOX", "danger-full-access")), {}) + if ((confirmed_wake.session_id, confirmed_wake.goal_ref, confirmed_wake.digest) != + (source_wake.session_id, source_wake.goal_ref, source_wake.digest)): + raise ValueError("interrupted source session binding moved before sealing") + if confirmed != native: + raise ValueError("interrupted native cursor moved before sealing") + if self.request.get("loopx_state"): + from .checkpoint import verify_research_participants + verify_research_participants(self.env, components, metadata, research, probe["source_witness"], window) + if self.request.get("controller_handoff"): + from .checkpoint_handoff import await_controller_capture + return await_controller_capture(self.root, components, metadata, probe, window) + self.root.mkdir(mode=0o700, parents=True, exist_ok=True) + return effect_runtime_result("runtime.checkpoint_artifacts", { + "action": "capture", "components": components, "destination": str(destination), + "expected_source_sha256": probe["source_sha256"], "point_metadata": metadata, + "deadline_epoch_ms": window * 1000, + }, timeout=max(.1, window - time.time()), retry_safe=False) + + def run(self, execution, body, wake, session_wake, stdout, stderr, timeout): + from .worker import child_process, native_command + from .sessions import BenchmarkSessionWake + + deadline = time.monotonic() + timeout if timeout is not None else float("inf") + while True: + remaining = deadline - time.monotonic() + if remaining <= 0: + raise subprocess.TimeoutExpired("checkpoint source wake", timeout) + command = native_command(self.env, execution, "execute", wake, session_wake) + continuation = None + with child_process(command, env=self.env, stdout=stdout, stderr=stderr) as process: + process.stdin.write(body) + process.stdin.close() + while process.poll() is None: + remaining = deadline - time.monotonic() + if remaining <= 0: + raise subprocess.TimeoutExpired(command, timeout) + if not select.select([self.listener], [], [], min(.1, remaining))[0]: + continue + with held_quota_decision(self.listener, self.capture_root, + goal_id=self.env["LOOPX_GOAL_ID"], agent_id=self.env["LOOPX_AGENT_ID"], + turn_instance_id=None, native_process=process, + interrupt_timeout=min(10, remaining), interrupt_when=self._first_replan, + deadline=deadline) as held: + if "native_process_return_code" not in held: + continue + stdout.flush() + session_wake.observe(wake / "stdout.jsonl") + event = {"obligation_id": held["intent"]["obligation_id"], + "decision_path": str(held["path"]), "decision_sha256": held["sha256"], + "captured": False, "research_point_qualified": False} + self.receipt.setdefault("checkpoint_handoffs", []).append(event) + try: + event.update(self._capture(held, deadline), captured=True) + except Exception as error: + # A missed optional point cannot discard the original + # committed guard or turn a later retry into its capture. + event.update(error_kind=type(error).__name__, error=str(error)[:500]) + continuation = held.get("continuation_body") + if continuation is not None: + break + return_code = process.wait(timeout=max(.1, deadline - time.monotonic()) if timeout is not None else None) + if continuation is None: + return return_code + # A fresh-context profile still resumes THIS source session after an + # interruption. Only the next ordinary Host wake uses its own policy. + source_session_id = self.receipt["session"]["session_id"] + execution = replace(execution, context="resume") + session_wake = BenchmarkSessionWake(self.env, execution, self.receipt) + if not source_session_id or session_wake.session_id != source_session_id: + raise ValueError("checkpoint source continuation lost its native session") + body = continuation diff --git a/benchmark/runtime/harbor.py b/benchmark/runtime/harbor.py index ee5e8fe435..9b73ee047c 100644 --- a/benchmark/runtime/harbor.py +++ b/benchmark/runtime/harbor.py @@ -59,6 +59,7 @@ def __init__( replan_after_todos=None, replan_after_turns=None, stride_strategy=None, + checkpoint_request_path=None, task_entry=None, planning_timeout_sec=300, turn_envelope=False, @@ -102,6 +103,12 @@ def __init__( raise ValueError("stride_strategy requires a LoopX execution mode") self.stride_strategy = (normalize_stride_strategy(stride_strategy) if stride_strategy is not None else None) + self.checkpoint_request_path = None + if checkpoint_request_path is not None: + if self.execution.mode not in {"heartbeat", "turn"}: + raise ValueError("checkpoint_request_path requires a settled-work worker") + from .checkpoint import normalize_checkpoint_request_path + self.checkpoint_request_path = normalize_checkpoint_request_path(checkpoint_request_path) self._task_path = "/usr/local/bin:/usr/bin:/bin" self._phase_number = 0 self._seeded_todo_id: str | None = None @@ -525,6 +532,8 @@ def _worker_env(self, *, cwd: str) -> dict[str, str]: "LOOPX_SHARED_SKILLS": _SHARED_SKILLS, "LOOPX_EXECUTION_MODE": self.execution.mode, "LOOPX_TASK_ENTRY": self.execution.task_entry, + **({"LOOPX_CHECKPOINT_REQUEST": self.checkpoint_request_path} + if self.checkpoint_request_path is not None else {}), **({"LOOPX_TURN_ENVELOPE": "1"} if self.execution.turn_envelope else {}), "LOOPX_ITERATION_CONTEXT": self.execution.context, "LOOPX_CODEX_SANDBOX": self.execution.sandbox, diff --git a/benchmark/runtime/native_checkpoint.py b/benchmark/runtime/native_checkpoint.py new file mode 100644 index 0000000000..841f9a09fb --- /dev/null +++ b/benchmark/runtime/native_checkpoint.py @@ -0,0 +1,492 @@ +"""Codex-specific checkpoint inventory and native history readback. + +This transport adapter does not decide research validity or adopt authority. +The artifact owner seals its declared native-store members after the Host exits. +""" +from __future__ import annotations + +import argparse +import hashlib +import io +import json +import math +import os +import re +import shutil +import shlex +import sqlite3 +import subprocess +import time +import tomllib +from pathlib import Path +from contextlib import redirect_stdout, redirect_stderr +from typing import Any + +from loopx.capabilities.benchmark_toolkit.native_codex_goal import StdioNativeGoalTransport +from .codex import Execution + + +def _complete_history(request, session_id: str, *, interrupted_guard: dict | None = None) -> list[dict[str, Any]]: + turns = [] + cursor = None + seen = set() + while True: + page = request("thread/turns/list", { + "threadId": session_id, "sortDirection": "asc", "itemsView": "full", "limit": 100, + **({"cursor": cursor} if cursor else {}), + }) + if not isinstance(page.get("data"), list): + raise ValueError("native checkpoint history page is unavailable") + turns.extend(page["data"]) + cursor = page.get("nextCursor") + if not cursor: + break + if cursor in seen: + raise ValueError("native checkpoint repeated history cursor") + seen.add(cursor) + if not turns or any(t.get("status") not in {"completed", "failed", "interrupted"} or not t.get("id") for t in turns): + raise ValueError("native checkpoint requires complete retained turns") + if interrupted_guard is None: + if turns[-1].get("status") != "completed": + raise ValueError("native checkpoint requires a completed final cursor") + else: + _verify_interrupted_guard(turns, interrupted_guard) + ids = [t["id"] for t in turns] + if len(set(ids)) != len(ids): + raise ValueError("native checkpoint history contains duplicate turns") + return turns + + +def _guard_cli_arguments(command: str, guard: dict) -> list[str]: + """Parse the provider shell wrapper plus the renderer's optional Turn assignment.""" + outer = shlex.split(command) + if len(outer) != 3 or Path(outer[0]).name not in {"bash", "zsh", "sh"} or outer[1] != "-lc": + raise ValueError("native pending guard is not a supported single shell command") + lines = outer[2].strip().splitlines() + if len(lines) == 2 and shlex.split(lines[0]) == ["LOOPX_TURN=" + guard["turn_instance_id"]]: + script = lines[1].replace('"${LOOPX_TURN:?}"', shlex.quote(guard["turn_instance_id"])) + elif len(lines) == 1: + script = lines[0] + else: + raise ValueError("native pending guard contains an unsupported shell program") + # Conservative provider syntax profile, not Goal/state classification. + # Only the exact renderer assignment above is expanded; never execute shell. + if not re.fullmatch(r"[A-Za-z0-9_./:=+@ '\"-]+", script): + raise ValueError("native pending guard contains unsupported shell syntax") + argv = shlex.split(script) + if not argv or argv[0] != guard["cli"]: + raise ValueError("native pending guard executable differs") + return argv[1:] + + +def _verify_interrupted_guard(turns: list[dict], guard: dict) -> None: + """Provider-specific stopped-tool cursor, never a relaxed completed cursor. + + Only the single simple command emitted by the heartbeat renderer is eligible. + Other than the exact renderer Turn assignment, shell programs and expansions + are unsupported, even if they eventually call quota. CLI parsing owns option + semantics; this adapter binds the native command item to the Host's held packet and rejects any later native item. + """ + from loopx.cli import build_parser + + tail = turns[-1] + items = tail.get("items") or [] + if tail.get("status") != "interrupted" or not items: + raise ValueError("native checkpoint requires an interrupted pending guard") + item = items[-1] + if (item.get("type") != "commandExecution" or item.get("status") != "failed" or + item.get("aggregatedOutput") is not None or not item.get("id") or + item.get("cwd") != guard["workspace"]): + raise ValueError("native checkpoint has output or work beyond its pending guard") + parser = build_parser() + def arguments(command): + argv = _guard_cli_arguments(command, guard) + try: + with redirect_stdout(io.StringIO()), redirect_stderr(io.StringIO()): + return vars(parser.parse_args(argv)) + except (SystemExit, ValueError) as error: + raise ValueError("native pending guard arguments are invalid") from error + args = arguments(item.get("command", "")) + expected = {"command": "quota", "quota_command": "should-run", "format": "json", + "registry": guard["registry"], "runtime_root": guard["runtime_root"], + "goal_id": guard["goal_id"], "agent_id": guard["agent_id"], + "turn_instance_id": guard["turn_instance_id"], "turn_envelope": True, + "decision_output_root": guard["capture_root"], + "decision_output_dir": None, "decision_handoff_socket": guard["socket_path"]} + if any(args.get(key) != value for key, value in expected.items()): + raise ValueError("native pending guard differs from the held packet routes or identity") + # A second matching guard after a tool yield cannot stand in for the first + # held call. Compare parsed routes, not only command spelling or item count. + for prior_turn in turns: + for previous in prior_turn.get("items", []): + if previous is item or previous.get("type") != "commandExecution": + continue + try: + prior_args = arguments(previous.get("command", "")) + except ValueError: + continue + if all(prior_args.get(key) == value for key, value in expected.items()): + raise ValueError("native pending guard is ambiguous after an earlier matching call") + if guard.get("native_call_id") is not None and item["id"] != guard["native_call_id"]: + raise ValueError("native pending guard call identity differs") + + +def native_session_checkpoint(env: dict[str, str], session_id: str, *, timeout: float, + interrupted_guard: dict | None = None) -> dict[str, Any]: + home = Path(env["CODEX_HOME"]) + if not session_id or timeout <= 0: + raise ValueError("native checkpoint requires a session and an open capture window") + deadline = time.monotonic() + timeout + version = subprocess.check_output([env["CODEX_BIN"], "--version"], env=env, text=True, + timeout=timeout).strip() + # Native storage is opaque. Retain both rollout and paginated/index stores; + # auth, logs, memories and the symlinked skill installation are not members. + with StdioNativeGoalTransport.spawn( + [env["CODEX_BIN"], "-c", "features.goals=false", "app-server", "--listen", "stdio://"], + cwd=env["LOOPX_PROJECT"], env=env, stderr=subprocess.DEVNULL, + response_timeout_sec=timeout, + ) as transport: + def request(method, params): + remaining = deadline - time.monotonic() + if remaining <= 0: + raise TimeoutError("native checkpoint window closed") + transport.response_timeout_sec = remaining + return transport.request(method, params) + request("initialize", {"clientInfo": {"name": "loopx_checkpoint", "version": "0.1.0"}, + "capabilities": {"experimentalApi": True}}) + transport.notify("initialized", {}) + readback = request("thread/read", {"threadId": session_id, "includeTurns": False}) + if readback.get("thread", {}).get("id") != session_id: + raise ValueError("native checkpoint session identity mismatch") + workspace = Path(env["LOOPX_PROJECT"]) + if (not workspace.is_absolute() or workspace.resolve(strict=True) != workspace or + readback["thread"].get("cwd") != str(workspace)): + raise ValueError("native checkpoint session workspace differs from the declared code basis") + turns = _complete_history(request, session_id, interrupted_guard=interrupted_guard) + ids = [t["id"] for t in turns] + members = [] + for entry in sorted(home.iterdir()): + if entry.name in {"sessions", "archived_sessions", "session_index.jsonl"} or re.fullmatch( + r"(?:state|thread_history)_[0-9]+\.sqlite", entry.name + ): + members.append({"id": "native_" + entry.name.replace(".", "_"), "source_path": str(entry)}) + if not any(m["id"].startswith("native_state_") for m in members): + raise ValueError("native checkpoint session index is unavailable") + encoded = json.dumps(turns, sort_keys=True, separators=(",", ":")).encode() + return {"components": members, "metadata": { + "session_id": session_id, "turn_ids": ids, + **({"completed_turn_id": ids[-1]} if interrupted_guard is None else { + "interrupted_turn_id": ids[-1], "interrupted_guard": { + **interrupted_guard, "native_call_id": turns[-1]["items"][-1]["id"]}}), + "visible_history_sha256": hashlib.sha256(encoded).hexdigest(), "host_version": version, + "guest_home": str(home), "guest_workspace": env["LOOPX_PROJECT"], + "path_profile": "preserve_guest_paths_in_isolated_environment", + "config_sha256": hashlib.sha256((home / "config.toml").read_bytes()).hexdigest(), + "store_members": [{"id": c["id"], "name": Path(c["source_path"]).name} for c in members], + "model_calls": 0, "execution_authority_granted": False, + }} + + +def _native_thread_ids(home: Path) -> set[str]: + """Read opaque identities omitted by the native interactive thread list. + + At the qualified Host version, thread/list hides a fresh fork until its + first user event. The native index locates candidates only: thread/read + and complete native history still prove provenance and the captured cursor. + Never edit this provider-owned store or infer authority from an index row. + """ + indexes = [path for path in home.iterdir() if re.fullmatch(r"state_[0-9]+\.sqlite", path.name)] + if len(indexes) != 1 or indexes[0].resolve(strict=True) != indexes[0] or not indexes[0].is_file(): + raise ValueError("native fork recovery requires one qualified physical session index") + with sqlite3.connect(indexes[0].as_uri() + "?mode=ro", uri=True) as connection: + rows = connection.execute("SELECT id FROM threads").fetchall() + ids = [row[0] for row in rows] + if any(not isinstance(value, str) or not value for value in ids) or len(ids) != len(set(ids)): + raise ValueError("native fork recovery index has invalid or duplicate identities") + return set(ids) + + +def fork_native_checkpoint(env: dict[str, str], unit: Path, expected_sha256: str, + config_path: Path, *, timeout: float) -> dict[str, Any]: + # The existing process lock serializes reconstruction in this isolated home. + # The private attempt below describes one opaque native RPC, not authority + # to admit a Goal, a work Turn, or a second fork after an uncertain response. + from loopx.file_lock import exclusive_file_lock + + if not math.isfinite(timeout) or timeout <= 0: + raise ValueError("native checkpoint reconstruction timeout must be finite and positive") + deadline = time.monotonic() + timeout + home = Path(env["CODEX_HOME"]) + if not home.is_absolute() or home != home.resolve(): + raise ValueError("native checkpoint destination must be a physical absolute home") + with exclusive_file_lock(home, timeout_seconds=timeout, operation="checkpoint_native_fork"): + remaining = deadline - time.monotonic() + if remaining <= 0: + raise TimeoutError("native checkpoint reconstruction window closed") + return _fork_native_checkpoint(env, unit, expected_sha256, config_path, timeout=remaining) + + +def _fork_native_checkpoint(env: dict[str, str], unit: Path, expected_sha256: str, + config_path: Path, *, timeout: float) -> dict[str, Any]: + """Reconstruct one native branch in a new, fixed-path isolated environment. + + No model starts and no LoopX binding is registered. Existing session/Goal + owners must separately admit this new session before useful continuation. + """ + from loopx.control_plane.effect_runtime import effect_runtime_result + + deadline = time.monotonic() + timeout + effect_runtime_result("runtime.checkpoint_artifacts", { + "action": "verify", "root": str(unit), "expected_sha256": expected_sha256, + }, timeout=timeout) + manifest = json.loads((unit / "checkpoint.json").read_text()) + native = manifest["point_metadata"]["native_session"] + if native.get("interrupted_guard") is not None: + component_id = manifest["point_metadata"].get("held_decision_component_id") + if component_id not in {entry["id"] for entry in manifest["components"]}: + raise ValueError("interrupted checkpoint lacks its held decision participant") + raw = (unit / "components" / component_id).read_bytes() + if hashlib.sha256(raw).hexdigest() != native["interrupted_guard"]["decision_sha256"]: + raise ValueError("interrupted checkpoint held decision digest differs") + guard = native["interrupted_guard"] + intent = effect_runtime_result("runtime.checkpoint_artifacts", { + "action": "select_replan", "quota": json.loads(raw), "goal_id": guard["goal_id"], + "agent_id": guard["agent_id"], "turn_instance_id": guard["turn_instance_id"], + }, timeout=max(.1, deadline - time.monotonic())) + if not intent["capture"] or intent != manifest["point_metadata"].get("replan"): + raise ValueError("interrupted checkpoint original replan binding differs") + home = Path(env["CODEX_HOME"]) + if str(home) != native["guest_home"] or env["LOOPX_PROJECT"] != native["guest_workspace"]: + raise ValueError("native checkpoint requires its qualified fixed guest paths") + attempt_path = home / ".loopx-checkpoint-fork.json" + recovering = home.exists() + if recovering and (not attempt_path.is_file() or attempt_path.resolve() != attempt_path): + raise ValueError("native checkpoint destination home must be new or retain its original fork attempt") + config = config_path.read_bytes() + if hashlib.sha256(config).hexdigest() != native["config_sha256"]: + raise ValueError("native checkpoint effective configuration differs") + version = subprocess.check_output([env["CODEX_BIN"], "--version"], env=env, text=True, + timeout=max(.1, deadline - time.monotonic())).strip() + if version != native["host_version"]: + raise ValueError("native checkpoint Host version differs") + names, component_ids = set(), set() + declared_ids = {entry["id"] for entry in manifest["components"]} + for member in native["store_members"]: + if not isinstance(member, dict) or set(member) != {"id", "name"}: + raise ValueError("native checkpoint member inventory is invalid") + name, component_id = member["name"], member["id"] + # The sealed manifest binds allocated IDs to bytes; provider names + # still select only the existing allowlisted native-store destinations. + # Historical fixed IDs remain valid without reserving caller names. + if (not isinstance(name, str) or not isinstance(component_id, str) or + name in names or component_id in component_ids or component_id not in declared_ids or + not (name in {"sessions", "archived_sessions", "session_index.jsonl"} or + re.fullmatch(r"(?:state|thread_history)_[0-9]+\.sqlite", name))): + raise ValueError("native checkpoint member inventory is invalid") + names.add(name) + component_ids.add(component_id) + interrupted_guard = native.get("interrupted_guard") + if interrupted_guard is not None: + if ("completed_turn_id" in native or not native.get("interrupted_turn_id") or + not native.get("turn_ids") or native["turn_ids"][-1] != native["interrupted_turn_id"]): + raise ValueError("native checkpoint mixes completed and interrupted cursors") + cursor_basis = {"interrupted_turn_id": native["interrupted_turn_id"], + "interrupted_guard": interrupted_guard} + else: + cursor_basis = {"completed_turn_id": native["completed_turn_id"]} + basis = {"checkpoint_sha256": expected_sha256, "source_session_id": native["session_id"], + **cursor_basis, "config_sha256": native["config_sha256"], + "destination_goal_id": env.get("LOOPX_GOAL_ID"), "destination_agent_id": env.get("LOOPX_AGENT_ID"), + "guest_workspace": env["LOOPX_PROJECT"], "guest_home": str(home)} + attempt = None + if recovering: + attempt = json.loads(attempt_path.read_bytes()) + if (not isinstance(attempt, dict) or set(attempt) != {"basis", "source_thread_ids"} or + attempt["basis"] != basis or not isinstance(attempt["source_thread_ids"], list) or + any(not isinstance(v, str) or not v for v in attempt["source_thread_ids"]) or + len(set(attempt["source_thread_ids"])) != len(attempt["source_thread_ids"]) or + native["session_id"] not in attempt["source_thread_ids"]): + raise ValueError("native fork attempt differs from the requested checkpoint or destination") + if (home / "config.toml").read_bytes() != config: + raise ValueError("native fork recovery effective configuration differs") + else: + home.mkdir(parents=True, mode=0o700) + for member in native["store_members"]: + source, destination = unit / "components" / member["id"], home / member["name"] + if source.is_dir(): + shutil.copytree(source, destination) + else: + shutil.copy2(source, destination) + (home / "config.toml").write_bytes(config) + with StdioNativeGoalTransport.spawn( + [env["CODEX_BIN"], "-c", "features.goals=false", "app-server", "--listen", "stdio://"], + cwd=env["LOOPX_PROJECT"], env=env, stderr=subprocess.DEVNULL, + response_timeout_sec=max(.1, deadline - time.monotonic()), + ) as transport: + def request(method, params): + remaining = deadline - time.monotonic() + if remaining <= 0: + raise TimeoutError("native checkpoint reconstruction window closed") + transport.response_timeout_sec = remaining + return transport.request(method, params) + request("initialize", {"clientInfo": {"name": "loopx_checkpoint", "version": "0.1.0"}, + "capabilities": {"experimentalApi": True}}) + transport.notify("initialized", {}) + inventory = _native_thread_ids(home) + if recovering: + added = set(inventory) - set(attempt["source_thread_ids"]) + if len(added) != 1 or not set(attempt["source_thread_ids"]).issubset(inventory): + raise ValueError("native fork outcome remains absent or ambiguous; a second fork is prohibited") + session = added.pop() + fork = request("thread/read", {"threadId": session, "includeTurns": False}) + else: + if native["session_id"] not in inventory: + raise ValueError("native checkpoint source is missing from the reconstructed store") + attempt = {"basis": basis, "source_thread_ids": sorted(inventory)} + # Persist before dispatch. Partial preparation is not a license to + # fork again; a missing outcome stays explicitly unresolved. + with os.fdopen(os.open(attempt_path, os.O_WRONLY | os.O_CREAT | os.O_EXCL, 0o600), "w") as file: + json.dump(attempt, file, sort_keys=True) + file.flush() + os.fsync(file.fileno()) + directory = os.open(home, os.O_RDONLY) + try: + os.fsync(directory) + finally: + os.close(directory) + fork = request("thread/fork", {"threadId": native["session_id"], + **({"lastTurnId": native["completed_turn_id"]} if interrupted_guard is None else {}), + "cwd": env["LOOPX_PROJECT"], + "sandbox": "read-only", "approvalPolicy": "never", "deferGoalContinuation": True, + "excludeTurns": True}) + session = fork.get("thread", {}).get("id") + if not session or session == native["session_id"]: + raise ValueError("native checkpoint did not create a distinct session") + if (fork.get("thread", {}).get("forkedFromId") != native["session_id"] or + fork["thread"].get("cwd") != env["LOOPX_PROJECT"]): + raise ValueError("native fork provenance differs from the original attempt") + turns = _complete_history(request, session, interrupted_guard=interrupted_guard) + encoded = json.dumps(turns, sort_keys=True, separators=(",", ":")).encode() + if ([t["id"] for t in turns] != native["turn_ids"] or + hashlib.sha256(encoded).hexdigest() != native["visible_history_sha256"]): + raise ValueError("native checkpoint reconstructed history differs") + return {"session_id": session, **cursor_basis, + "visible_history_sha256": native["visible_history_sha256"], "history_verified": True, + "model_calls": 0, "execution_authority_granted": False, "continuation_qualified": False} + + +def admit_native_checkpoint_session(env: dict[str, str], unit: Path, expected_sha256: str, + config_path: Path, *, execution: Execution, timeout: float) -> dict[str, Any]: + """Explicit controller admission of a verified fork into a destination Goal. + + Existing native fork, Goal lifetime and session owners retain their rules. + Historical research/effects and evaluator/whole-trial budget are not adopted + by this participant. Admission alone never launches work or a model. + An interrupted cursor is retained as history, not as an executable source + guard. The destination must obtain its own current work/replan admission. + """ + from loopx.control_plane.effect_runtime import effect_runtime_result + from loopx.agent_registry import load_goal_from_registry, require_registered_agent_id + from loopx.control_plane.projects.registry_codec import ( + project_registry_transaction, + ) + from loopx.control_plane.turn_driver.codex_sessions import select_codex_cli_session + from .sessions import BenchmarkSessionWake + + if execution.context != "resume" or execution.mode != "heartbeat": + raise ValueError("checkpoint session admission requires explicit heartbeat resume") + if not math.isfinite(timeout) or timeout <= 0: + raise ValueError("checkpoint session admission timeout must be finite and positive") + deadline = time.monotonic() + timeout + effect_runtime_result("runtime.checkpoint_artifacts", { + "action": "verify", "root": str(unit), "expected_sha256": expected_sha256, + }, timeout=timeout) + metadata = json.loads((unit / "checkpoint.json").read_bytes())["point_metadata"] + source_profile = (metadata.get("session") or {}).get("session_profile_digest") + if not isinstance(source_profile, str) or not re.fullmatch(r"[0-9a-f]{64}", source_profile): + raise ValueError("checkpoint session admission requires its captured source session profile") + if tomllib.loads(config_path.read_text()).get("features", {}).get("goals") is not False: + raise ValueError("checkpoint session admission requires native automatic Goals explicitly disabled") + if hashlib.sha256(config_path.read_bytes()).hexdigest() != metadata["native_session"]["config_sha256"]: + raise ValueError("native checkpoint effective configuration differs") + source = metadata.get("loopx_state", {}) + lineage = {"goal_id": env["LOOPX_GOAL_ID"], "agent_id": env["LOOPX_AGENT_ID"]} + if (not source.get("goal_id") or not source.get("agent_id") or + any(source[key] == lineage[key] for key in lineage)): + raise ValueError("checkpoint session admission requires distinct destination Goal and Agent identities") + registry = Path(env["LOOPX_REGISTRY"]) + # Use the current governed benchmark path. The lifecycle-only source-session + # profile cannot run governed Turns and is rejected by the registry owner. + goal = load_goal_from_registry(registry, lineage["goal_id"]) + require_registered_agent_id(registry_path=registry, goal_id=lineage["goal_id"], agent_id=lineage["agent_id"]) + if select_codex_cli_session(Path(env["LOOPX_RUNTIME_ROOT"]), lineage=lineage, + session_scope="agent") is not None: + # The original session binding is also the admission receipt. Only a + # source-bound committed admission can recover a lost controller reply; + # an unrelated binding or a partial native fork is never replaced. + with project_registry_transaction(registry, operation="checkpoint_session_binding", agent_id=lineage["agent_id"]): + current = load_goal_from_registry(registry, lineage["goal_id"]) + require_registered_agent_id(registry_path=registry, goal_id=lineage["goal_id"], agent_id=lineage["agent_id"]) + if current != goal: + raise ValueError("destination Goal configuration changed during native checkpoint readback") + wake = BenchmarkSessionWake(env, execution, {}) + binding = select_codex_cli_session(wake.root, lineage=lineage, session_scope="agent", + goal_admission=wake.admission) + if not binding or binding.get("checkpoint_artifact_sha256") != expected_sha256: + raise ValueError("destination session binding already exists without this checkpoint admission") + if wake.digest != source_profile: + raise ValueError("checkpoint source session profile changed") + native = metadata["native_session"] + observed = native_session_checkpoint(env, binding["session_id"], + timeout=max(.1, deadline - time.monotonic()), + interrupted_guard=native.get("interrupted_guard"))["metadata"] + cursor_key = "interrupted_turn_id" if native.get("interrupted_guard") is not None else "completed_turn_id" + keys = (cursor_key, "turn_ids", "visible_history_sha256", "host_version", + "guest_home", "guest_workspace", "config_sha256") + if binding["session_id"] == native["session_id"] or any(observed[key] != native[key] for key in keys): + raise ValueError("checkpoint admitted native session advanced or differs; use its existing continuation") + return {"session_id": binding["session_id"], cursor_key: observed[cursor_key], + "visible_history_sha256": observed["visible_history_sha256"], "history_verified": True, + "model_calls": 0, "execution_authority_granted": False, "continuation_qualified": False, + "destination_goal_ref": wake.goal_ref, "destination_agent_id": lineage["agent_id"], + "session_binding_admitted": True, "session_profile_digest": wake.digest} + result = fork_native_checkpoint(env, unit, expected_sha256, config_path, + timeout=max(.1, deadline - time.monotonic())) + # Hold the existing registry owner's fence through the session write. This + # does not mint a lease, task or work admission from historical state. + with project_registry_transaction(registry, operation="checkpoint_session_binding", agent_id=lineage["agent_id"]): + current = load_goal_from_registry(registry, lineage["goal_id"]) + require_registered_agent_id(registry_path=registry, goal_id=lineage["goal_id"], agent_id=lineage["agent_id"]) + if current != goal: + raise ValueError("destination Goal configuration changed during native checkpoint reconstruction") + receipt: dict[str, Any] = {} + wake = BenchmarkSessionWake(env, execution, receipt) + wake._bind_verified_fork(result["session_id"], source_profile, expected_sha256) + return {**result, "destination_goal_ref": wake.goal_ref, "destination_agent_id": lineage["agent_id"], + "session_binding_admitted": True, "session_profile_digest": wake.digest} + + +def main(argv=None) -> None: + """Controller entrypoint inside an explicitly prepared isolated guest.""" + import os + from .codex import SANDBOXES + + parser = argparse.ArgumentParser(description="Admit a checkpoint's native session participant; no model starts.") + parser.add_argument("--root", type=Path, required=True) + parser.add_argument("--expected-sha256", required=True) + parser.add_argument("--config-path", type=Path, required=True) + parser.add_argument("--sandbox", choices=SANDBOXES, required=True) + parser.add_argument("--timeout-seconds", type=float, default=30) + args = parser.parse_args(argv) + required = ("CODEX_BIN", "CODEX_HOME", "LOOPX_PROJECT", "LOOPX_REGISTRY", "LOOPX_RUNTIME_ROOT", + "LOOPX_GOAL_ID", "LOOPX_AGENT_ID", "MODEL_NAME", "REASONING_EFFORT") + missing = [key for key in required if not os.environ.get(key)] + if missing: + parser.error("Explicit destination controller environment is missing: " + ", ".join(missing)) + result = admit_native_checkpoint_session(dict(os.environ), args.root, args.expected_sha256, + args.config_path, execution=Execution(context="resume", sandbox=args.sandbox), timeout=args.timeout_seconds) + print(json.dumps(result)) + + +if __name__ == "__main__": + main() diff --git a/benchmark/runtime/research_checkpoint.py b/benchmark/runtime/research_checkpoint.py new file mode 100644 index 0000000000..44bc934284 --- /dev/null +++ b/benchmark/runtime/research_checkpoint.py @@ -0,0 +1,303 @@ +"""Private checkpoint context transport through existing canonical Todo owners. + +Captured records are historical input, never a replacement authority store. +Only the existing explicit continuation owner can adopt a note on an already +claimed destination Todo. Graph/effect/vision reactivation is outside this seam. +""" +from __future__ import annotations + +import argparse +import hashlib +import json +import os +import time +from pathlib import Path, PurePosixPath + +from loopx.control_plane.effect_runtime import effect_runtime_result +from loopx.control_plane.coordination.canonical_snapshot import read_canonical_snapshot + + +def canonical_context(env: dict[str, str], *, deadline: float) -> dict: + """Retain the complete revision-bound read, without the execution leases.""" + def rpc(method, payload, **kwargs): + remaining = deadline - time.time() + if remaining <= 0: + raise TimeoutError("research checkpoint capture window closed") + return effect_runtime_result(method, payload, timeout=remaining) + result = read_canonical_snapshot(rpc=rpc, runtime_root=env["LOOPX_RUNTIME_ROOT"], + goal_id=env["LOOPX_GOAL_ID"], include_leases=False, projection_readback=None, + timeout=max(.1, deadline - time.time())) + if result.get("status") != "loaded": + raise ValueError("research checkpoint requires complete canonical Todo context") + return {"goal_id": env["LOOPX_GOAL_ID"], "snapshot": result} + + +def research_context(env: dict[str, str], *, deadline: float) -> dict: + """Retain decision evidence as history, without importing operational state. + + The existing Explore codec owns event validity and topology; run-index + records own vision/stride history. This transport preserves their values + instead of inferring a new frontier or reviving a historical decision. + """ + from loopx.capabilities.explore.result_log import ( + explore_result_log_path, validate_explore_result_event, + ) + from loopx.history import load_index_snapshot + from loopx.runtime import validate_goal_id_path_segment + + root = Path(env["LOOPX_RUNTIME_ROOT"]) + goal = validate_goal_id_path_segment(env["LOOPX_GOAL_ID"]) + paths = [explore_result_log_path(root, goal), root / "goals" / goal / "runs/index.jsonl"] + before = {} + for path in paths: + if not path.is_absolute() or path.resolve() != path: + raise ValueError("research evidence source must be a physical absolute path") + raw = path.read_bytes() if path.exists() else None + # An incomplete append is not a complete research boundary. The source + # owners validate event fields; the transport checks complete framing. + if raw and not raw.endswith(b"\n"): + raise ValueError("research evidence source has an incomplete tail") + before[path] = raw + events = [] + # Decode exactly the witnessed bytes through the existing event validator; + # rereading the file for decoding would leave an avoidable ABA window. + for line in (before[paths[0]] or b"").splitlines(): + if line.strip(): + try: + row = json.loads(line) + if not isinstance(row, dict): + raise ValueError("event must be an object") + events.append(validate_explore_result_event(row, expected_goal_id=goal)) + except ValueError as error: + raise ValueError("invalid Explore result in research context") from error + raw_index = before[paths[1]] + if raw_index: + for line in raw_index.splitlines(): + if line.strip(): + row = json.loads(line) + if not isinstance(row, dict) or row.get("goal_id", goal) != goal: + raise ValueError("research history contains a malformed or foreign record") + index = load_index_snapshot(paths[1], include_artifact_status=False) + if index.digest != ("sha256:" + hashlib.sha256(raw_index).hexdigest() if raw_index is not None else None): + raise ValueError("research history source moved during decoding") + runs = index.records + # Preserve every indexed vision, including retired decisions and their + # explicit provenance; a latest-only display could erase negative evidence. + visions = [{key: row[key] for key in ("generated_at", "goal_id", "agent_id", "goal_ref", + "turn_instance_id", "agent_vision", "vision_checkpoint") if key in row} + for row in runs if row.get("agent_vision") or row.get("vision_checkpoint")] + result = canonical_context(env, deadline=deadline) + for path, raw in before.items(): + if (path.read_bytes() if path.exists() else None) != raw: + raise ValueError("research evidence source moved during the context read") + return {**result, "research_evidence": {"explore_events": events, "vision_history": visions}, + "source_files": [{"path": str(path), "sha256": hashlib.sha256(raw).hexdigest() if raw is not None else None} + for path, raw in before.items()]} + + +def verify_context_sources(context: dict, components: list[dict], witness: dict) -> None: + """Bind exported history to the artifact owner's actual source witness. + + A second read alone ends too early: a source append immediately before the + probe must not be captured next to an older exported context. Absence is + witnessed by the containing component too, so a first append also rejects. + """ + for source in context.get("source_files", []): + path = Path(source["path"]) + parent = next((entry for entry in components if path.is_relative_to(Path(entry["source_path"]))), None) + if parent is None: + raise ValueError("research evidence source is outside the captured inventory") + relative = path.relative_to(Path(parent["source_path"])) + member = "" if str(relative) == "." else relative.as_posix() + observed = next((item for item in witness[parent["id"]] if item["path"] == member), None) + if source["sha256"] is None: + if observed is not None: + raise ValueError("research evidence source appeared before sealing") + elif observed is None or observed.get("kind") != "file" or observed.get("sha256") != source["sha256"]: + raise ValueError("research evidence source differs from the captured context") + + +def _encoded(value): + return json.dumps(value, sort_keys=True, separators=(",", ":"), ensure_ascii=False).encode() + + +def _handoff(env, payload): + from loopx.agent_registry import registered_agent_ids_from_registry + request = {"runtime_root": env["LOOPX_RUNTIME_ROOT"], "goal_id": env["LOOPX_GOAL_ID"], + "agent_id": env["LOOPX_AGENT_ID"], "registered_agents": registered_agent_ids_from_registry( + Path(env["LOOPX_REGISTRY"]), env["LOOPX_GOAL_ID"]), **payload} + result = effect_runtime_result("coordination.local_authority.todo_continuation", request) + if result.get("ok") is not True: + raise ValueError("checkpoint context continuation rejected: " + str(result.get("reason_code", result.get("status")))) + return result + + + +def _context_session(env, metadata): + """Read the admitted native branch before publishing captured Todo context.""" + from .codex import Execution + from .sessions import BenchmarkSessionWake + from .native_checkpoint import native_session_checkpoint + + wake = BenchmarkSessionWake(env, Execution(context="resume", sandbox=env["LOOPX_CODEX_SANDBOX"]), {}) + native = metadata["native_session"] + if (not wake.session_id or wake.session_id == native["session_id"] or + wake.digest != metadata.get("session", {}).get("session_profile_digest")): + raise ValueError("context adoption requires the admitted destination session profile") + observed = native_session_checkpoint(env, wake.session_id, timeout=30, + interrupted_guard=native.get("interrupted_guard"))["metadata"] + cursor_key = "interrupted_turn_id" if native.get("interrupted_guard") is not None else "completed_turn_id" + for key in ("visible_history_sha256", "turn_ids", cursor_key, "host_version", "config_sha256"): + if observed[key] != native[key]: + raise ValueError("destination native history or configuration moved beyond the captured context") + return wake.session_id + + +def adopt_todo_context(env: dict[str, str], unit: Path, digest: str, *, source_todo_id: str, + todo_id: str, operation_id: str, expected_revision: str, + lease_key: str, lease_version: int) -> dict: + """Adopt retained context using the destination's current claim and lease. + + No Todo is created, completed or transferred, and no historical authority, + effect, strategy or graph is installed. Prepare/adopt each use stable original + operation identities. A failed/uncertain call must be read back through that + same owner; this adapter never guesses a new identity or starts model work. + """ + from loopx.control_plane.turn_driver.codex_sessions import select_codex_cli_session + from loopx.control_plane.todos.provider_projection import ( + settle_canonical_todo_projection, projection_delivery_requires_ack, + ) + + effect_runtime_result("runtime.checkpoint_artifacts", { + "action": "verify", "root": str(unit), "expected_sha256": digest}) + metadata = json.loads((unit / "checkpoint.json").read_bytes())["point_metadata"] + if not all(isinstance(metadata.get(key), dict) for key in ("loopx_state", "native_session", "research_context")): + raise ValueError("checkpoint lacks its canonical Todo or native session context participant") + source = metadata["loopx_state"] + if (source["goal_id"] == env["LOOPX_GOAL_ID"] or source["agent_id"] == env["LOOPX_AGENT_ID"]): + raise ValueError("context adoption requires distinct destination Goal and Agent") + declared = metadata["research_context"] + component, member = declared["component_id"], declared["member_path"] + if (not isinstance(component, str) or Path(component).name != component or component in {".", ".."} or + not isinstance(member, str) or PurePosixPath(member).is_absolute() or + ".." in PurePosixPath(member).parts or (member and str(PurePosixPath(member)) != member)): + raise ValueError("retained context member must stay within its artifact component") + context_file = unit / "components" / component + if member: + context_file /= member + raw = context_file.read_bytes() + if hashlib.sha256(raw).hexdigest() != declared["sha256"]: + raise ValueError("retained canonical context digest differs") + carrier = json.loads(raw) + context = carrier["snapshot"] + if (carrier.get("goal_id") != source["goal_id"] or + context.get("provider_revision") != declared["provider_revision"] or + context.get("store_identity") != declared["store_identity"]): + raise ValueError("retained canonical context revision differs") + retained = next((row for row in context["todos"] if row["todo_id"] == source_todo_id), None) + if retained is None: + raise ValueError("source Todo is absent from the captured context") + session_id = _context_session(env, metadata) + bound = select_codex_cli_session(Path(env["LOOPX_RUNTIME_ROOT"]), lineage={ + "goal_id": env["LOOPX_GOAL_ID"], "agent_id": env["LOOPX_AGENT_ID"]}, session_scope="agent") + if bound is None or bound["session_id"] != session_id: + raise ValueError("context adoption requires the current distinct destination session binding") + workspace = Path(env["LOOPX_PROJECT"]) + if not workspace.is_absolute() or workspace.resolve(strict=True) != workspace: + raise ValueError("destination context workspace must be a physical absolute path") + proof = {"idempotency_key": lease_key, "expected_version": lease_version} + common = {"todo_id": todo_id, "session_id": bound["session_id"], "workspace": str(workspace), + "artifacts": [], "lease_proof": proof} + # The owner's inspect performs the real current claim/lease/dependency and + # acceptance checks. Source records only contribute context; they are never + # passed as operational fields to prepare or adopt. + inspect = _handoff(env, {**common, "action": "inspect"}) + if (inspect["claimed_by"] != env["LOOPX_AGENT_ID"] or + inspect.get("execution_authority", {}).get("allowed") is not True): + raise ValueError("destination context revision, owner or execution authority changed") + destination = canonical_context(env, deadline=time.time() + 30)["snapshot"] + current = next((row for row in destination["todos"] if row["todo_id"] == todo_id), None) + if destination["provider_revision"] != inspect["provider_revision"] or current is None: + raise ValueError("destination canonical context moved before preparation") + if current.get("note") and destination["provider_revision"] == expected_revision: + raise ValueError("destination Todo already has context; replay its original operation and revision") + if not current.get("note") and destination["provider_revision"] != expected_revision: + raise ValueError("destination context revision, owner or execution authority changed") + # With an existing note, the caller's stale ORIGINAL revision can only + # replay prepare through its typed receipt owner; it cannot overwrite the + # current note. Changed operation/context/proof must still fail there. + # Keep the complete source read in a workspace-relative private artifact. + # Never overwrite a moved artifact on an operation retry. + relative = Path(".local/checkpoint-context") / (digest + ".json") + target = workspace / relative + for directory in (workspace / ".local", target.parent): + if directory.is_symlink() or (directory.exists() and directory.resolve(strict=True) != directory): + raise ValueError("destination context artifact must not use a symlink") + directory.mkdir(exist_ok=True, mode=0o700) + if target.is_symlink(): + raise ValueError("destination context artifact must not use a symlink") + try: + with os.fdopen(os.open(target, os.O_WRONLY | os.O_CREAT | os.O_EXCL, 0o600), "wb") as file: + if file.write(raw) != len(raw): + raise OSError("destination context artifact write is incomplete") + file.flush() + os.fsync(file.fileno()) + except FileExistsError: + if target.read_bytes() != raw: + raise ValueError("destination context artifact already differs") + if target.read_bytes() != raw: + raise ValueError("destination context artifact readback differs") + effect_runtime_result("runtime.checkpoint_artifacts", { + "action": "verify", "root": str(unit), "expected_sha256": digest}) + # Keep old context-only request bytes stable for original receipt recovery + # across an adapter upgrade. New history is disclosed only when captured. + history_guidance = (" Read its research_evidence for prior branches, negative results and vision/stride decisions." + if "research_evidence" in carrier else "") + note = {"work_summary": "Retained canonical Todo context from the verified checkpoint is available in " + relative.as_posix() + + ". Historical owners, statuses and operational references are evidence only; current Goal/Todo/lease rules govern work." + history_guidance, + "source_refs": ["checkpoint:" + digest, "source-todo:" + source_todo_id, + "source-revision:" + declared["provider_revision"]], + "files_touched": [{"path": relative.as_posix(), "action": "read", + "summary": "Complete captured canonical Todo population, including retained notes and rejected approaches."}]} + prepared = _handoff(env, {**common, "action": "prepare", "session_id": metadata["native_session"]["session_id"], + "operation_id": operation_id + "-prepare", "expected_provider_revision": expected_revision, "context": note}) + if select_codex_cli_session(Path(env["LOOPX_RUNTIME_ROOT"]), lineage={ + "goal_id": env["LOOPX_GOAL_ID"], "agent_id": env["LOOPX_AGENT_ID"]}, session_scope="agent") != bound: + raise ValueError("destination session changed; inspect the original prepared context") + if _context_session(env, metadata) != session_id: + raise ValueError("destination native branch changed; inspect the original prepared context") + # Preserve the original prepare commit revision. The outer readback may + # already include adoption if its response was lost; using that newer head + # would change the adopt request hash and defeat original-receipt recovery. + adopted = _handoff(env, {**common, "action": "adopt", "artifacts": [relative.as_posix()], + "operation_id": operation_id + "-adopt", "expected_provider_revision": prepared["result"]["provider_revision"]}) + effect_runtime_result("runtime.checkpoint_artifacts", { + "action": "verify", "root": str(unit), "expected_sha256": digest}) + if target.read_bytes() != raw: + raise ValueError("adopted context artifact changed; inspect the original continuation receipt") + result = settle_canonical_todo_projection(adopted, registry_path=Path(env["LOOPX_REGISTRY"]), + runtime_root=Path(env["LOOPX_RUNTIME_ROOT"]), goal_id=env["LOOPX_GOAL_ID"]) + if result.get("ok") is not True or not projection_delivery_requires_ack(result["projection_delivery"]): + raise ValueError("continuation committed; reconcile its original projection receipt") + return {"context_adopted": True, "session_id": bound["session_id"], "todo_id": todo_id, + "artifact": str(target), "context_sha256": declared["sha256"], + "continuation": result, "model_calls": 0, "research_point_qualified": False} + + +def main(argv=None): + parser = argparse.ArgumentParser(description="Adopt checkpoint Todo context on an already owned destination Todo.") + for name in ("root", "expected-sha256", "source-todo-id", "todo-id", "operation-id", "expected-revision", "lease-key"): + parser.add_argument("--" + name, required=True) + parser.add_argument("--lease-version", type=int, required=True) + args = parser.parse_args(argv) + required = ("LOOPX_PROJECT", "LOOPX_REGISTRY", "LOOPX_RUNTIME_ROOT", "LOOPX_GOAL_ID", "LOOPX_AGENT_ID", + "CODEX_BIN", "CODEX_HOME", "MODEL_NAME", "REASONING_EFFORT", "LOOPX_CODEX_SANDBOX") + if any(not os.environ.get(key) for key in required): + parser.error("Explicit destination registry, runtime, workspace, Goal and Agent are required") + print(json.dumps(adopt_todo_context(dict(os.environ), Path(args.root), args.expected_sha256, + source_todo_id=args.source_todo_id, todo_id=args.todo_id, operation_id=args.operation_id, + expected_revision=args.expected_revision, lease_key=args.lease_key, lease_version=args.lease_version))) + + +if __name__ == "__main__": + main() diff --git a/benchmark/runtime/sessions.py b/benchmark/runtime/sessions.py index 87eb73a774..131c60f149 100644 --- a/benchmark/runtime/sessions.py +++ b/benchmark/runtime/sessions.py @@ -115,3 +115,29 @@ def observe(self, path: Path) -> None: ) ) self.receipt["session"]["session_id"] = observed + + def _bind_verified_fork(self, session_id: str, source_profile_digest: str, + checkpoint_artifact_sha256: str) -> None: + """Bind a controller-verified native fork through the existing session owner. + + This admits the conversation binding only. It does not claim work, + restore research authority or start a model. Unknown result delivery + requires readback; it cannot replace an existing binding. + """ + if self.session_id is not None: + raise ValueError("checkpoint session adoption requires an absent destination binding") + require_codex_session_profile({"session_profile_digest": source_profile_digest}, self.digest) + self.admission.accept_result( + lambda: _store_codex_cli_session( + self.root, + lineage=self.lineage, + session_scope="agent", + session_id=session_id, + goal_ref=self.goal_ref, + session_profile_digest=self.digest, + require_absent=True, + checkpoint_artifact_sha256=checkpoint_artifact_sha256, + ) + ) + self.session_id = session_id + self.receipt["session"].update(action="resume", session_id=session_id) diff --git a/benchmark/runtime/sforge.py b/benchmark/runtime/sforge.py index b2bfff5e0f..08df45236d 100644 --- a/benchmark/runtime/sforge.py +++ b/benchmark/runtime/sforge.py @@ -21,9 +21,10 @@ from .codex import DEFAULT_REPLAN_AFTER_TURNS, Execution, prepare_codex_home from .codex_offline import CodexOffline, stage_portable_python from .harbor import ( - BenchmarkCodex, _GOAL_ID, _PYTHON, _SCHEDULER_STATE, _SRC, + BenchmarkCodex, _CONTROL, _GOAL_ID, _PYTHON, _SCHEDULER_STATE, _SRC, ) from .scheduler import worker_command +from .worker import PHASE_WORK_RESERVE_SECONDS # Experiment profiles, local to the research runner; not new LoopX modes. @@ -91,7 +92,9 @@ def __init__(self, config, *, profile: str, cwd: str, turn_envelope: bool = False, replan_after_turns: int | None = None, replan_after_todos: int | None = None, - stride_strategy: str | None = None): + stride_strategy: str | None = None, + checkpoint_request_path: str | None = None, + checkpoint_request: bytes | None = None): super().__init__(config) if profile not in PROFILES: raise ValueError("Unknown benchmark worker profile") @@ -100,7 +103,7 @@ def __init__(self, config, *, profile: str, cwd: str, ): raise ValueError("turn_envelope requires a heartbeat worker") self.turn_envelope = turn_envelope - if timeout_seconds <= 160: + if timeout_seconds <= PHASE_WORK_RESERVE_SECONDS: raise ValueError("Worker budget must exceed the 160s startup/settlement reserve") if not config.agent_model or not config.agent_effort: raise ValueError("Explicit model and reasoning effort are required") @@ -136,6 +139,19 @@ def __init__(self, config, *, profile: str, cwd: str, from loopx.control_plane.work_items.stride_policy_codec import normalize_stride_strategy stride_strategy = normalize_stride_strategy(stride_strategy) self.stride_strategy = stride_strategy + self.checkpoint_request_path = None + self.checkpoint_request = checkpoint_request + if checkpoint_request is not None: + if checkpoint_request_path is not None or not profile.startswith("heartbeat-"): + raise ValueError("checkpoint_request requires a heartbeat profile and no guest-path override") + from .checkpoint import parse_checkpoint_request + parse_checkpoint_request(checkpoint_request) + checkpoint_request_path = f"{_CONTROL}/checkpoint-request.json" + if checkpoint_request_path is not None: + if not profile.startswith("heartbeat-"): + raise ValueError("checkpoint_request_path requires a heartbeat profile") + from .checkpoint import normalize_checkpoint_request_path + self.checkpoint_request_path = normalize_checkpoint_request_path(checkpoint_request_path) self.profile, self.cwd = profile, cwd if feedback not in FEEDBACK_MODES: raise ValueError("Unknown feedback mode") @@ -160,7 +176,7 @@ def __init__(self, config, *, profile: str, cwd: str, self.timeout_seconds = timeout_seconds # Let one call use the trial budget. The shared worker recomputes the # remaining deadline on every continuation, retaining cleanup time. - self.turn_timeout = timeout_seconds - 160 + self.turn_timeout = timeout_seconds - PHASE_WORK_RESERVE_SECONDS self.runtime = None self.prepared = False # Only the official profile delegates continuation to SForge. Heartbeat @@ -222,9 +238,20 @@ def _install_worker(self, backend, handle, log_dir, logger): replan_after_turns=self.replan_after_turns, replan_after_todos=self.replan_after_todos, stride_strategy=self.stride_strategy, + checkpoint_request_path=self.checkpoint_request_path, task_objective=self.task_objective, ) asyncio.run(self.runtime.install(self.environment)) + if self.checkpoint_request is not None: + with tempfile.TemporaryDirectory(prefix="benchmark-checkpoint-") as directory: + declaration = Path(directory) / "request.json" + declaration.write_bytes(self.checkpoint_request) + asyncio.run(self.environment.upload_file(declaration, self.checkpoint_request_path)) + result = backend.exec_run(handle, ["/bin/bash", "-c", + f"chown agent:agent {shlex.quote(self.checkpoint_request_path)} && " + f"chmod 0600 {shlex.quote(self.checkpoint_request_path)}"], user="root") + if result.exit_code: + raise RuntimeError("Could not protect the explicit checkpoint declaration") # Prompt replacement describes the policy; the feedback mode owns access. if self.feedback != "native": result = backend.exec_run(handle, ["rm", "-f", "/usr/local/bin/sforge-submit"], user="root") @@ -260,6 +287,8 @@ def _install_worker(self, backend, handle, log_dir, logger): "task_entry": self.task_entry, "reasoning_effort": effort, "timeout_seconds": self.timeout_seconds, "loopx_usage_ping_enabled": False, + **({"checkpoint_capture_enabled": True} + if self.checkpoint_request_path is not None else {}), **({"planning_timeout_seconds": None} if self.task_entry == "loopx-planned" else {}), "stop_hook": self.profile == "official", diff --git a/benchmark/runtime/sforge_backend.py b/benchmark/runtime/sforge_backend.py index 71c095a0d3..aa301db308 100644 --- a/benchmark/runtime/sforge_backend.py +++ b/benchmark/runtime/sforge_backend.py @@ -15,11 +15,12 @@ class RecordingDockerBackend(DockerBackend): def __init__(self, *, log_dir: Path, logger, oauth_proxy: bool = False, - blind_api_endpoint: tuple[str, int] | None = None, feedback=None, client=None): + blind_api_endpoint: tuple[str, int] | None = None, feedback=None, checkpoint=None, client=None): super().__init__(client) self.log_dir, self.logger = log_dir, logger self.blind_api_endpoint = blind_api_endpoint self.feedback = feedback + self.checkpoint = checkpoint self.execution_command = None self.execution_start_marker = None self.execution_receipt = None @@ -34,6 +35,8 @@ def create_container(self, image, name, **kwargs): kwargs["extra_hosts"] = hosts if self.feedback is not None: self.feedback.bind(kwargs.get("environment")) + if self.checkpoint is not None: + self.checkpoint.bind(kwargs.get("environment")) kwargs["environment"] = self._agent_environment(kwargs.get("environment")) return super().create_container(image, name, **kwargs) @@ -46,6 +49,8 @@ def _agent_environment(self, environment): def exec_run_with_timeout(self, handle, cmd, timeout=60, **kwargs): kwargs["environment"] = self._agent_environment(kwargs.get("environment")) is_solver = cmd == ["/bin/bash", "-c", getattr(self, "execution_command", None)] + if is_solver and self.checkpoint is not None: + self.checkpoint.start(handle.raw) try: result = super().exec_run_with_timeout(handle, cmd, timeout, **kwargs) if is_solver: @@ -64,6 +69,8 @@ def exec_run_with_timeout(self, handle, cmd, timeout=60, **kwargs): pending.replace(target) return result finally: + if is_solver and self.checkpoint is not None: + self.checkpoint.close() if is_solver and self.feedback is not None: self.feedback.pause() @@ -92,6 +99,8 @@ def start_feedback(self, handle): self.feedback.start(self, handle) def cleanup_container(self, handle, logger=None): + if self.checkpoint is not None: + self.checkpoint.close() if self.feedback is not None: self.feedback.close() if handle is not None: diff --git a/benchmark/runtime/sforge_checkpoint.py b/benchmark/runtime/sforge_checkpoint.py new file mode 100644 index 0000000000..2f19c7da96 --- /dev/null +++ b/benchmark/runtime/sforge_checkpoint.py @@ -0,0 +1,74 @@ +"""SForge checkpoint read transport, included in the existing worker payload.""" +from __future__ import annotations + +import json +import hashlib +import math +import time +from urllib.request import Request, urlopen +from pathlib import Path + +SCHEMA = "sforge_checkpoint_opportunity_v0" # Local provider artifact, not a LoopX authority contract. +ROUTE = "/api/v1/checkpoint/opportunity" +ARCHIVE_ROUTE = "/api/v1/checkpoint/incumbent" + + +def capture_opportunity(env, *, timeout, incumbent_path: Path): + """Read only this worker's registration; retain its original deadline.""" + deadline = float(env["LOOPX_PHASE_DEADLINE_EPOCH"]) + if not math.isfinite(deadline) or deadline <= time.time() or not math.isfinite(timeout) or timeout <= 0: + raise ValueError("checkpoint requires an open original trial deadline") + stop = min(deadline, time.time() + timeout) + request = Request(env["SFORGE_JUDGE_URL"].rstrip("/") + ROUTE, + data=json.dumps({"token": env["SFORGE_TOKEN"]}).encode(), + headers={"Content-Type": "application/json"}, method="POST") + with urlopen(request, timeout=max(.1, stop - time.time())) as response: + packet = json.load(response) + if (packet.get("schema_version") != SCHEMA or packet.get("execution_authority_granted") is not False or + packet.get("outstanding_evaluations") != 0): + raise ValueError("native opportunity checkpoint readback is unavailable") + if "native_incumbent" not in packet or not packet.get("capture_binding_sha256"): + raise ValueError("native opportunity checkpoint omits incumbent coverage") + incumbent = packet["native_incumbent"] + if incumbent is not None: + if (not isinstance(incumbent, dict) or type(incumbent.get("archive_bytes")) is not int or + incumbent["archive_bytes"] < 0 or not isinstance(incumbent.get("archive_sha256"), str)): + raise ValueError("native incumbent descriptor is invalid") + incumbent_path = Path(incumbent_path) + if not incumbent_path.is_absolute() or incumbent_path != incumbent_path.resolve(): + raise ValueError("incumbent destination must be a normalized physical private file") + if time.time() >= stop: + raise ValueError("checkpoint capture window closed before incumbent transport") + request = Request(env["SFORGE_JUDGE_URL"].rstrip("/") + ARCHIVE_ROUTE, + data=json.dumps({"token": env["SFORGE_TOKEN"], + "capture_binding_sha256": packet["capture_binding_sha256"]}).encode(), + headers={"Content-Type": "application/json"}, method="POST") + created = False + try: + with urlopen(request, timeout=max(.1, stop - time.time())) as response: + # Never overwrite a prior participant or follow its symlink. + with incumbent_path.open("xb") as target: + created = True + incumbent_path.chmod(0o600) + digest, size = hashlib.sha256(), 0 + while chunk := response.read(1024 * 1024): + size += len(chunk) + if size > incumbent["archive_bytes"] or time.time() >= stop: + raise ValueError("native incumbent transport exceeded its bound") + digest.update(chunk) + target.write(chunk) + if size != incumbent["archive_bytes"] or digest.hexdigest() != incumbent["archive_sha256"]: + raise ValueError("native incumbent bytes differ from the captured result") + except Exception: + if created: + incumbent_path.unlink() + raise + observed = time.time() + if observed >= stop: + if incumbent is not None: + incumbent_path.unlink() + raise ValueError("checkpoint capture window closed during opportunity transport") + return {**packet, "source_trial_deadline_epoch": deadline, + "client_observed_at_epoch": observed, + "remaining_trial_seconds": max(0, deadline - observed), + "continuation_qualified": False} diff --git a/benchmark/runtime/sforge_entry.py b/benchmark/runtime/sforge_entry.py index 01a0f76da9..e3cf9886bb 100644 --- a/benchmark/runtime/sforge_entry.py +++ b/benchmark/runtime/sforge_entry.py @@ -11,7 +11,7 @@ import time from .planning import task_plan_packet, validate_plan_readback -from .worker import loopx_command, run_once +from .worker import PHASE_WORK_RESERVE_SECONDS, loopx_command, run_once def prepare_entry(env: dict[str, str]) -> bool: @@ -21,7 +21,7 @@ def prepare_entry(env: dict[str, str]) -> bool: remaining = float(env["LOOPX_PHASE_DEADLINE_EPOCH"]) - time.time() if not math.isfinite(remaining): raise ValueError("Invalid planning budget") - if remaining <= 160: + if remaining <= PHASE_WORK_RESERVE_SECONDS: return False path = Path(env["LOOPX_PLANNING_RESULT"]) if path.exists(): @@ -70,7 +70,7 @@ def cancelled(signum, frame): # Native process completion must not qualify it as a finished trial. return 1 # A successful plan may itself consume the remaining execution allowance. - if float(env["LOOPX_PHASE_DEADLINE_EPOCH"]) - time.time() <= 160: + if float(env["LOOPX_PHASE_DEADLINE_EPOCH"]) - time.time() <= PHASE_WORK_RESERVE_SECONDS: return 1 # Native SForge owns environment filtering and the outer phase deadline; # retain its proxy/isolation inputs rather than launching from setup hooks. diff --git a/benchmark/runtime/worker.py b/benchmark/runtime/worker.py index a1f2b7b05b..6ec291afaf 100644 --- a/benchmark/runtime/worker.py +++ b/benchmark/runtime/worker.py @@ -10,12 +10,14 @@ import subprocess import time import uuid -from contextlib import contextmanager +from contextlib import ExitStack, contextmanager from dataclasses import replace from pathlib import Path from benchmark.runtime.codex import Execution, prepare_codex_home, process_environment +PHASE_WORK_RESERVE_SECONDS = 160 + def read_json(text: str) -> dict: payload = json.loads(text) @@ -36,7 +38,8 @@ def loopx_command(env: dict[str, str]) -> list[str]: ] -def heartbeat_body(env: dict[str, str], turn_id: str, *, native_goal: bool) -> str: +def heartbeat_body(env: dict[str, str], turn_id: str, *, native_goal: bool, + decision_handoff_socket: str | None = None) -> str: command = loopx_command(env) + [ "heartbeat-prompt", "--thin", @@ -57,13 +60,15 @@ def heartbeat_body(env: dict[str, str], turn_id: str, *, native_goal: bool) -> s command += ["--turn-instance-id", turn_id] if env.get("LOOPX_TURN_ENVELOPE", "0") not in {"0", "1"}: raise ValueError("LOOPX_TURN_ENVELOPE must be 0 or 1") - if env.get("LOOPX_TURN_ENVELOPE") == "1": + if env.get("LOOPX_TURN_ENVELOPE") == "1" or decision_handoff_socket is not None: if native_goal: raise ValueError("TurnEnvelope capture requires a host-owned heartbeat Turn") # Wake identity comes from this worker, never from model-authored text. root = Path(env["LOOPX_WAKE_LOG_DIR"]) / turn_id / "decisions" root.mkdir(mode=0o700, exist_ok=False) command += ["--decision-output-root", str(root)] + if decision_handoff_socket is not None: + command += ["--decision-handoff-socket", decision_handoff_socket] result = subprocess.run( command, cwd=env["LOOPX_PROJECT"], @@ -98,22 +103,47 @@ def child_process(command: list[str], *, env: dict[str, str], stdout, stderr): try: yield process finally: - # Reap descendants even when their leader exited or Harbor cancelled us. - try: - # Python's standard SIGINT handler lets the Turn host's finally - # path reap its separately grouped Codex child before forced kill. - os.killpg(process.pid, signal.SIGINT) - except ProcessLookupError: - pass - try: - process.wait(timeout=5) - except subprocess.TimeoutExpired: - pass - try: - os.killpg(process.pid, signal.SIGKILL) - except ProcessLookupError: - pass - process.wait(timeout=10) + _reap_child_process(process) + + +def _reap_child_process(process): + """Reap the process group created by child_process, including descendants.""" + try: + os.killpg(process.pid, signal.SIGINT) + except ProcessLookupError: + pass + try: + process.wait(timeout=5) + except subprocess.TimeoutExpired: + pass + try: + os.killpg(process.pid, signal.SIGKILL) + except ProcessLookupError: + pass + process.wait(timeout=10) + + +def interrupt_checkpoint_process(process, *, timeout: float = 10) -> int: + """Close an owned native CLI before inspecting a mid-wake checkpoint. + + A held tool response is not a native execution barrier: the native tool can + yield and schedule more inference. Require graceful leader exit so native + history can flush, then reap its group. An unacknowledged interruption is a + missed capture, not a completed cursor or permission to continue elsewhere. + This does not freeze independent workspace writers. + """ + import math + + if not math.isfinite(timeout) or timeout <= 0: + raise ValueError("checkpoint interruption requires a finite positive timeout") + if (not isinstance(process, subprocess.Popen) or process.poll() is not None or + os.getpgid(process.pid) != process.pid or os.getsid(process.pid) != process.pid): + raise ValueError("checkpoint interruption requires a live owned child process group") + try: + os.killpg(process.pid, signal.SIGINT) + return process.wait(timeout=timeout) + finally: + _reap_child_process(process) def turn_command( @@ -274,6 +304,7 @@ def run_once(env: dict[str, str]) -> dict: "timed_out": False, } session_wake = None + checkpoint_context = ExitStack() pending_path = ( Path(env.get("LOOPX_RUNTIME_ROOT", str(home))) / "benchmark-pending-turn.json" ) @@ -282,12 +313,12 @@ def run_once(env: dict[str, str]) -> dict: remaining = float(env["LOOPX_PHASE_DEADLINE_EPOCH"]) - time.time() # Reserve startup and settlement on every wake, then allow the # remaining time for work instead of reusing the initial timeout. - if remaining <= 160: + if remaining <= PHASE_WORK_RESERVE_SECONDS: receipt.update(ok=True, budget_exhausted=True, host_invoked=False) return receipt execution = replace( - execution, timeout_seconds=(remaining - 160 if execution.timeout_seconds is None - else min(execution.timeout_seconds, remaining - 160)) + execution, timeout_seconds=(remaining - PHASE_WORK_RESERVE_SECONDS if execution.timeout_seconds is None + else min(execution.timeout_seconds, remaining - PHASE_WORK_RESERVE_SECONDS)) ) prepare_codex_home( home, @@ -300,6 +331,9 @@ def run_once(env: dict[str, str]) -> dict: wire_api=env.get("CODEX_WIRE_API", "responses"), skills=Path(env["LOOPX_SHARED_SKILLS"]) if execution.uses_loopx else None, ) + from benchmark.runtime.checkpoint_wake import CheckpointWake + + checkpoint_wake = checkpoint_context.enter_context(CheckpointWake(env, receipt)) body = "Finish the task." if stage == "plan": from benchmark.runtime.planning import task_plan_packet @@ -309,7 +343,8 @@ def run_once(env: dict[str, str]) -> dict: (wake / "planning-input.json").write_text(body, encoding="utf-8") (wake / "planning-schema.json").write_text(json.dumps(planning_before["result_schema"])) elif execution.mode in {"heartbeat", "loopx-goal"}: - body = heartbeat_body(env, turn_id, native_goal=execution.native_goal) + body = heartbeat_body(env, turn_id, native_goal=execution.native_goal, + decision_handoff_socket=checkpoint_wake.socket_path) elif execution.mode == "plain": body = Path(env["LOOPX_TASK_DOC"]).read_text(encoding="utf-8") session_wake = None @@ -317,6 +352,10 @@ def run_once(env: dict[str, str]) -> dict: from benchmark.runtime.sessions import BenchmarkSessionWake session_wake = BenchmarkSessionWake(env, execution, receipt) + if stage == "execute" and execution.mode == "heartbeat" and env.get("LOOPX_CHECKPOINT_REQUEST"): + from benchmark.runtime.checkpoint import capture_before_replan + + body = capture_before_replan(env, receipt, body) with (wake / "stderr.log").open("w") as stderr: if execution.native_goal and stage == "execute": run_native_goal(env, execution, body, receipt, stderr) @@ -340,24 +379,25 @@ def run_once(env: dict[str, str]) -> dict: else native_command(env, execution, stage, wake, session_wake) ) with (wake / "stdout.jsonl").open("w") as stdout: - with child_process( - command, env=env, stdout=stdout, stderr=stderr - ) as process: - allowance = 150 if execution.mode == "turn" and stage == "execute" else 0 - if stage == "plan": - limit = env.get("LOOPX_PLANNING_TIMEOUT_SEC") - # SForge planning has no stage cap: the same absolute - # trial deadline bounds this subprocess and its handoff. - timeout = (float(limit) if limit is not None else - max(0, float(env["LOOPX_PHASE_DEADLINE_EPOCH"]) - time.time())) - else: - timeout = (None if execution.timeout_seconds is None - else execution.timeout_seconds + allowance) - process.communicate( - input=body, timeout=timeout - ) - receipt["return_code"] = process.returncode - receipt["ok"] = process.returncode == 0 + allowance = 150 if execution.mode == "turn" and stage == "execute" else 0 + if stage == "plan": + limit = env.get("LOOPX_PLANNING_TIMEOUT_SEC") + # SForge planning has no stage cap: the same absolute + # trial deadline bounds this subprocess and its handoff. + timeout = (float(limit) if limit is not None else + max(0, float(env["LOOPX_PHASE_DEADLINE_EPOCH"]) - time.time())) + else: + timeout = (None if execution.timeout_seconds is None + else execution.timeout_seconds + allowance) + if checkpoint_wake.listener is not None: + receipt["return_code"] = checkpoint_wake.run( + execution, body, wake, session_wake, stdout, stderr, timeout) + receipt["ok"] = receipt["return_code"] == 0 + else: + with child_process(command, env=env, stdout=stdout, stderr=stderr) as process: + process.communicate(input=body, timeout=timeout) + receipt["return_code"] = process.returncode + receipt["ok"] = process.returncode == 0 if stage == "plan" and receipt["ok"]: from benchmark.runtime.planning import task_plan_packet, validate_plan_readback @@ -390,6 +430,7 @@ def run_once(env: dict[str, str]) -> dict: receipt["error_kind"] = type(exc).__name__ raise finally: + checkpoint_context.close() try: if session_wake is not None: session_wake.observe(wake / "stdout.jsonl") @@ -397,6 +438,10 @@ def run_once(env: dict[str, str]) -> dict: receipt.update(ok=False, error_kind=type(exc).__name__) raise finally: + if env.get("LOOPX_CHECKPOINT_REQUEST"): + from benchmark.runtime.checkpoint import capture_after_wake + + capture_after_wake(env, receipt) if (home / "sessions").is_dir(): # One authoritative copy; resumed prefixes are never double counted. shutil.copytree( diff --git a/benchmark/tests/test_checkpoint_controller.py b/benchmark/tests/test_checkpoint_controller.py new file mode 100644 index 0000000000..6a2a149692 --- /dev/null +++ b/benchmark/tests/test_checkpoint_controller.py @@ -0,0 +1,652 @@ +"""Actual native HTTP and typed artifact transport; grading is synthetic here.""" +from __future__ import annotations + +import copy +import io +import json +import os +import subprocess +import socket +import tarfile +import threading +import time +from contextlib import contextmanager +from dataclasses import replace +from types import SimpleNamespace +from unittest.mock import Mock + +import pytest + +from benchmark.edgebench import checkpoint_controller as transport + + +@contextmanager +def server(app): + import uvicorn + with socket.socket() as probe: + probe.bind(("127.0.0.1", 0)) + port = probe.getsockname()[1] + host = uvicorn.Server(uvicorn.Config(app, host="127.0.0.1", port=port, + log_level="error", access_log=False)) + thread = threading.Thread(target=host.run) + thread.start() + try: + wait(lambda: host.started) + yield f"http://127.0.0.1:{port}" + finally: + host.should_exit = True + thread.join(timeout=10) + assert not thread.is_alive() + + +def wait(predicate): + until = time.monotonic() + 30 + while time.monotonic() < until: + value = predicate() + if value: + return value + time.sleep(.01) + raise AssertionError("Native synthetic HTTP journey did not settle") + + +@pytest.fixture +def controllers(tmp_path, monkeypatch): + pytest.importorskip("sforge.harness.checkpoint", reason="Requires native checkpoint API source pin") + from sforge.harness import judge_server as native + from sforge.harness.config import SForgeConfig + tasks = tmp_path / "tasks" + tasks.mkdir() + (tasks / "BENCHMARK.yaml").write_text("name: transport-test\nbase_images: {python: {}}\n") + (tasks / "fixture.json").write_text(json.dumps({ + "task_id": "fixture", "name": "Synthetic transport", "base_image": "python", + "platform": "linux/arm64", "cwd": "/tmp", "internet": False, + "submit_paths": ["candidate.json"], "submit_exclude": [], + "work": {"image_tag": "fixture", "agent_query": "Synthetic fixture"}, + "judge": {"image_tag": "fixture", "eval_timeout": 15, "parser": "structured_json", + "selection": "score_first", "score_direction": "maximize", "eval_cmd": "true"}, + })) + backend = Mock(backend_name="docker") + backend.client.images.get.return_value.id = "sha256:synthetic-transport-image" + monkeypatch.setattr(native, "create_backend_from_config", lambda config: backend) + grades = [] + def grade(**kwargs): + grades.append(kwargs["archive"]) + kwargs["log_dir"].mkdir(parents=True, exist_ok=True) + (kwargs["log_dir"] / "submission.tar.gz").write_bytes(kwargs["archive"]) + return SimpleNamespace(to_dict=lambda: {"score": float(kwargs["archive"]), + "valid": True, "pass_rate": 1.0, "passed": 1, "failed": 0, "total_tests": 1}) + monkeypatch.setattr(native, "judge_submission", grade) + config = SForgeConfig(tasks_dir=tasks, log_dir=tmp_path / "source") + source = native.create_app(config, checkpoints=True) + destination = native.create_app(replace(config, log_dir=tmp_path / "branch"), checkpoints=True) + try: + with server(source) as first, server(destination) as second: + files = [] + for name, app, url in (("source", source, first), ("branch", destination, second)): + bank = app.state.judge.config.log_dir / "checkpoints" + bank.mkdir(parents=True, mode=0o700) + path = tmp_path / (name + "-controller.json") + path.write_text(json.dumps({"url": url, "admin_secret": app.state.judge.admin_secret, + "bank_root": str(bank)})) + path.chmod(0o600) + files.append(path) + yield source, destination, first, second, files, grades + finally: + for app in (source, destination): + app.state.judge._reaper_stop.set() + app.state.judge._reaper_thread.join(timeout=1) + + +def register(app, url, **options): + import requests + response = requests.post(url + "/api/v1/register", json={"task_id": "fixture", "run_id": "source", + "admin_secret": app.state.judge.admin_secret, **options}, timeout=5) + response.raise_for_status() + return response.json()["token"] + + +def submit(url, token, score): + import requests + response = requests.post(url + "/api/v1/submit", data={"token": token}, + files={"archive": ("candidate.tar.gz", str(score).encode())}, timeout=5) + if response.status_code == 200: + identifier = response.json()["submission_id"] + wait(lambda: requests.get(url + "/api/v1/result/" + identifier, timeout=5).json()["status"] == "completed") + return response + + +def test_native_controller_transport_preserves_original_point_and_fresh_authority(controllers, tmp_path, capsys): + import requests + source, destination, first, second, files, grades = controllers + token = register(source, first, max_agent_submissions=3, submission_cooldown=0) + assert submit(first, token, 7).status_code == 200 + assert submit(first, token, 3).status_code == 200 + # Retain a future cooldown without changing the original accepted clock. + source.state.judge.tokens[token]["submission_cooldown"] = 3600 + before = copy.deepcopy((source.state.judge.tokens, source.state.judge.submissions, source.state.judge.run_history)) + token_file = tmp_path / "token" + token_file.write_text(token) + exported = tmp_path / "exported" + transport.main(["--controller-file", str(files[0]), "capture", "--token-file", str(token_file), + "--capture-id", "a" * 32, "--destination", str(exported)]) + receipt = json.loads(capsys.readouterr().out) + assert receipt["research_point_qualified"] is False + assert receipt["native_reference"]["incumbent"]["round"] == "agent-1" + assert (exported / "components/native_judge/incumbent.tar.gz").read_bytes() == b"7" + manifest = json.loads((exported / "components/native_judge/manifest.json").read_text()) + assert set(json.loads((exported / "checkpoint.json").read_bytes())["point_metadata"]) == {"native_reference"} + assert not (source.state.judge.config.log_dir / "checkpoints/.loopx-phase-attempts").exists() + assert len(manifest["history"]) == 2 and manifest["opportunity"]["next_agent"] == 3 + assert 0 < manifest["opportunity"]["remaining_cooldown_seconds"] <= 3600 + for secret in (token, source.state.judge.admin_secret, destination.state.judge.admin_secret): + assert secret not in json.dumps(receipt) + assert all(secret.encode() not in p.read_bytes() for p in exported.rglob("*") if p.is_file()) + assert (source.state.judge.tokens, source.state.judge.submissions, source.state.judge.run_history) == before + # Later source state cannot replace this first native capture identity. + source.state.judge.tokens[token]["submission_cooldown"] = 0 + assert submit(first, token, 99).status_code == 200 + repeated = transport.capture(files[0], token=token, capture_id="a" * 32, destination=tmp_path / "same-point") + assert repeated["native_reference"] == receipt["native_reference"] + private_result = tmp_path / "fresh-result.json" + transport.main(["--controller-file", str(files[1]), "restore", "--root", str(exported), + "--expected-sha256", receipt["sha256"], "--run-id", "branch", "--result-file", str(private_result)]) + output = capsys.readouterr().out + fresh = json.loads(private_result.read_text())["token"] + assert fresh != token and fresh not in output + assert private_result.stat().st_mode & 0o077 == 0 and len(grades) == 3 + history = requests.get(second + "/api/v1/history", params={"token": fresh}, timeout=5).json() + assert history["best_score"] == 7 and history["best_round"] == "agent-1" and history["agent_submissions"] == 2 + assert submit(second, fresh, 1).status_code == 429 + assert source.state.judge.get_run_history("source", "fixture")["best_score"] == 99 + with pytest.raises(ValueError, match="HTTP 409"): + transport.restore(files[1], root=exported, expected_sha256=receipt["sha256"], run_id="branch") + assert len(grades) == 3 + + +def host_phase_command(tmp_path, monkeypatch): + """Real command builder; only installation/inference are outside this oracle.""" + pytest.importorskip("harbor") + from sforge.harness.config import SForgeConfig + from benchmark.runtime.sforge import BenchmarkCodex, SForgeWorker + + async def installed(self, environment): + pass + monkeypatch.setattr(BenchmarkCodex, "install", installed) + monkeypatch.setenv("CODEX_AUTH_JSON_PATH", "/synthetic-not-read") + logs = tmp_path / "phase-worker" + logs.mkdir() + worker = SForgeWorker(SForgeConfig(agent_model="fixture", agent_effort="xhigh"), + profile="heartbeat-resume", cwd="/task", timeout_seconds=43200, task_entry="seeded-todo") + worker.install_stop_hook(None, None, logs, None) + worker.prepared = True + # The provider's own phase-file/admission shell executes. Its following + # model command is excluded explicitly; candidate writing below is synthetic. + return worker.format_run_cmd("/task.md").split( + "set +e; timeout --signal=TERM --kill-after=30 ${remaining}s env", 1)[0] + + +def test_host_phase_and_native_opportunities_continue_without_offline_refill( + controllers, tmp_path, monkeypatch, capsys): + import requests + source, destination, first, second, files, grades = controllers + token = register(source, first, max_agent_submissions=3, submission_cooldown=2) + assert submit(first, token, 7).status_code == 200 + before = copy.deepcopy((source.state.judge.tokens, source.state.judge.submissions, source.state.judge.run_history)) + # An already admitted Host has 2h left from its original 12h configured + # trial. Fixture seeding represents that prior elapsed research, not a new run. + source_phase = tmp_path / "source-phase" + source_phase.write_text(f"{int(time.time()) + 7200}\n") + source_bytes = source_phase.read_bytes() + unit = tmp_path / "unit" + point = transport.capture(files[0], token=token, capture_id="c" * 32, + destination=unit, phase_deadline_file=source_phase) + body = json.loads((unit / "checkpoint.json").read_bytes()) + retained = body["point_metadata"]["host_phase_budget"]["remaining_seconds"] + assert 7190 < retained <= 7200 + prefix = host_phase_command(tmp_path, monkeypatch) + control = tmp_path / "host-control" + control.mkdir() + phase = control / "phase-deadline" + # A destination Host clock 8h later must not expire a copied source epoch + # or mint the configured 12h allowance. Native uses its real server clock. + clock = time.time() + 28800 + monkeypatch.setattr(transport, "time", SimpleNamespace(time=lambda: clock)) + private_result = tmp_path / "branch-result" + transport.main(["--controller-file", str(files[1]), "restore", "--root", str(unit), + "--expected-sha256", point["sha256"], "--run-id", "phase-branch", "--result-file", str(private_result), + "--phase-deadline-file", str(phase)]) + fresh = json.loads(private_result.read_bytes())["token"] + assert fresh not in capsys.readouterr().out and phase.stat().st_mode & 0o077 == 0 + assert 0 <= clock + retained - int(phase.read_bytes()) < 1 + bin = tmp_path / "bin" + bin.mkdir() + date = bin / "date" + date.write_text(f"#!/bin/sh\nprintf '%s\\n' {int(clock)}\n") + date.chmod(0o700) + candidate = tmp_path / "continued-candidate" + shell = prefix.replace("/opt/loopx-benchmark/control", str(control)) + result = subprocess.run(["/bin/bash", "-c", shell + + f"printf '8' > '{candidate}'; printf '%s' \"$remaining\""], + env={"PATH": str(bin) + ":/usr/bin:/bin"}, capture_output=True, text=True, check=True) + assert 7180 < int(result.stdout) <= 7200 and candidate.read_bytes() == b"8" + history = requests.get(second + "/api/v1/history", params={"token": fresh}, timeout=5).json() + assert history["agent_submissions"] == 1 and history["best_score"] == 7 + assert submit(second, fresh, 8).status_code == 429 + time.sleep(body["point_metadata"]["native_reference"]["opportunity"]["remaining_cooldown_seconds"] + .05) + assert submit(second, fresh, float(candidate.read_bytes())).status_code == 200 + history = requests.get(second + "/api/v1/history", params={"token": fresh}, timeout=5).json() + assert history["agent_submissions"] == 2 and history["best_round"] == "agent-2" and history["best_score"] == 8 + assert source_phase.read_bytes() == source_bytes + assert (source.state.judge.tokens, source.state.judge.submissions, source.state.judge.run_history) == before + assert len(grades) == 2 # No implicit grading of the retained incumbent. + with pytest.raises(ValueError, match="fresh physical"): + transport.restore(files[1], root=unit, expected_sha256=point["sha256"], + run_id="second-branch", phase_deadline_file=phase) + assert len(destination.state.judge.tokens) == 1 and len(grades) == 2 + + +@pytest.mark.parametrize("fault", ["missing_clock", "exhausted", "existing", "alias", "modified"]) +def test_host_phase_faults_refuse_before_native_admission(controllers, tmp_path, fault): + source, destination, first, second, files, grades = controllers + token = register(source, first) + assert submit(first, token, 7).status_code == 200 + phase_source = tmp_path / "source-phase" + phase_source.write_text(f"{int(time.time()) + (10 if fault == 'exhausted' else 7200)}\n") + unit = tmp_path / "unit" + point = transport.capture(files[0], token=token, capture_id="d" * 32, + destination=unit, phase_deadline_file=phase_source) + target = tmp_path / "phase" + if fault == "existing": + target.write_text("keep") + elif fault == "alias": + target.symlink_to(phase_source) + elif fault == "modified": + (unit / "components/host_phase_deadline").write_text(f"{int(time.time()) + 43200}\n") + with pytest.raises((ValueError, RuntimeError)): + transport.restore(files[1], root=unit, expected_sha256=point["sha256"], run_id="phase-branch", + phase_deadline_file=None if fault == "missing_clock" else target) + assert destination.state.judge.tokens == {} and len(grades) == 1 + if fault == "existing": + assert target.read_text() == "keep" + + +@pytest.mark.parametrize("fault", ["source_moves", "response_lost", "empty_response"]) +def test_phase_ambiguity_never_replaces_point_or_refills_budget(controllers, tmp_path, monkeypatch, fault): + source, destination, first, second, files, grades = controllers + token = register(source, first) + phase_source = tmp_path / "source-phase" + phase_source.write_text(f"{int(time.time()) + 7200}\n") + unit = tmp_path / "unit" + native_post = transport._post + def post(controller, route, body, timeout): + result = native_post(controller, route, body, timeout) + if route.endswith("/capture") and fault == "source_moves": + phase_source.write_text(f"{int(time.time()) + 43200}\n") + if route.endswith("/restore"): + if fault == "response_lost": + raise TimeoutError("Synthetic transport lost a committed native result") + if fault == "empty_response": + return {} + return result + monkeypatch.setattr(transport, "_post", post) + if fault == "source_moves": + with pytest.raises(ValueError, match="moved"): + transport.capture(files[0], token=token, capture_id="e" * 32, + destination=unit, phase_deadline_file=phase_source) + assert not unit.exists() + with pytest.raises(ValueError, match="Existing native point"): + transport.capture(files[0], token=token, capture_id="e" * 32, + destination=unit, phase_deadline_file=phase_source) + assert destination.state.judge.tokens == {} + else: + point = transport.capture(files[0], token=token, capture_id="e" * 32, + destination=unit, phase_deadline_file=phase_source) + control = tmp_path / "host-control" + control.mkdir() + target = control / "phase-deadline" + with pytest.raises((TimeoutError, ValueError)): + transport.restore(files[1], root=unit, expected_sha256=point["sha256"], + run_id="phase-branch", phase_deadline_file=target) + assert target.read_bytes() == b"" and len(destination.state.judge.tokens) == 1 + # Existing SForge entry treats empty as expired, never initializes a + # fresh configured allowance. Same identity requires owner readback. + shell = host_phase_command(tmp_path, monkeypatch).replace("/opt/loopx-benchmark/control", str(control)) + observed = subprocess.run(["/bin/bash", "-c", shell + "printf unexpected-work"], + capture_output=True, text=True) + assert observed.returncode != 0 and observed.stdout == "" + with pytest.raises(ValueError, match="fresh physical"): + transport.restore(files[1], root=unit, expected_sha256=point["sha256"], + run_id="different-branch", phase_deadline_file=target) + assert grades == [] + + +@pytest.mark.parametrize("fault", ["wrong_run", "expired", "partial_record", "phase_alias", "result_corrupt", "lost_authority", "busy"]) +def test_recovery_refuses_mismatched_or_unavailable_original_activation(controllers, tmp_path, monkeypatch, fault): + import fcntl + source, destination, first, second, files, grades = controllers + token = register(source, first) + source_phase = tmp_path / "source-phase" + source_phase.write_text(f"{int(time.time()) + 7200}\n") + unit = tmp_path / "unit" + point = transport.capture(files[0], token=token, capture_id="9" * 32, + destination=unit, phase_deadline_file=source_phase) + phase, result = tmp_path / "phase", tmp_path / "result" + args = ["--controller-file", str(files[1]), "restore", "--root", str(unit), + "--expected-sha256", point["sha256"], "--run-id", "branch", + "--result-file", str(result), "--phase-deadline-file", str(phase)] + post = transport._post + calls = [] + def lost(controller, route, body, timeout): + calls.append(route) + response = post(controller, route, body, timeout) + if route.endswith("/restore"): + raise TimeoutError("Lost committed result") + return response + monkeypatch.setattr(transport, "_post", lost) + with pytest.raises(TimeoutError): + transport.main(args) + record = phase.with_name("phase.restore.json") + original_epoch = json.loads(record.read_bytes())["deadline_epoch"] + lock = None + if fault == "wrong_run": + args[args.index("--run-id") + 1] = "another-branch" + elif fault == "expired": + monkeypatch.setattr(transport, "time", SimpleNamespace(time=lambda: original_epoch + 1)) + elif fault == "partial_record": + record.write_bytes(b"{") + elif fault == "phase_alias": + phase.unlink() + phase.symlink_to(source_phase) + elif fault == "result_corrupt": + result.write_bytes(b"{") + elif fault == "lost_authority": + destination.state.judge.tokens.clear() + elif fault == "busy": + lock = phase.open("r+") + fcntl.flock(lock.fileno(), fcntl.LOCK_EX | fcntl.LOCK_NB) + before = copy.deepcopy((destination.state.judge.tokens, destination.state.judge.run_history)) + old_bytes = (phase.read_bytes(), result.read_bytes(), record.read_bytes()) + try: + with pytest.raises((ValueError, OSError)): + transport.main(args + ["--recover"]) + finally: + if lock: + lock.close() + assert (phase.read_bytes(), result.read_bytes(), record.read_bytes()) == old_bytes + assert (destination.state.judge.tokens, destination.state.judge.run_history) == before + assert calls.count("/api/v1/checkpoint/restore") == 1 and grades == [] + if fault != "lost_authority": + assert calls == ["/api/v1/checkpoint/restore"] + + +@pytest.mark.parametrize("raw", [b"", b"nan\n", b"1.5\n", b"9007199254740992\n"]) +def test_invalid_host_epoch_cannot_create_native_point(controllers, tmp_path, raw): + source, destination, first, second, files, grades = controllers + token = register(source, first) + phase = tmp_path / "phase" + phase.write_bytes(raw) + with pytest.raises(ValueError, match="phase|clocks"): + transport.capture(files[0], token=token, capture_id="f" * 32, + destination=tmp_path / "unit", phase_deadline_file=phase) + bank = source.state.judge.config.log_dir / "checkpoints" + assert not (bank / ("f" * 32)).exists() and not (bank / ".loopx-phase-attempts").exists() + assert destination.state.judge.tokens == {} and grades == [] + + +def test_native_only_point_cannot_acquire_a_later_host_budget(controllers, tmp_path): + source, destination, first, second, files, grades = controllers + token = register(source, first) + transport.capture(files[0], token=token, capture_id="f" * 32, destination=tmp_path / "first") + phase = tmp_path / "phase" + phase.write_text(f"{int(time.time()) + 43200}\n") + with pytest.raises(ValueError, match="Existing native point"): + transport.capture(files[0], token=token, capture_id="f" * 32, + destination=tmp_path / "later", phase_deadline_file=phase) + assert not (tmp_path / "later").exists() and destination.state.judge.tokens == {} and grades == [] + + +def test_partial_phase_write_keeps_committed_native_result_inert(controllers, tmp_path, monkeypatch): + source, destination, first, second, files, grades = controllers + token = register(source, first) + phase_source = tmp_path / "source-phase" + phase_source.write_text(f"{int(time.time()) + 7200}\n") + unit = tmp_path / "unit" + point = transport.capture(files[0], token=token, capture_id="f" * 32, + destination=unit, phase_deadline_file=phase_source) + class PartialWrite: + def __getattr__(self, name): + return getattr(os, name) + def write(self, descriptor, data): + return os.write(descriptor, data[:1]) + monkeypatch.setattr(transport, "os", PartialWrite()) + target = tmp_path / "phase" + with pytest.raises(OSError, match="incomplete"): + transport.restore(files[1], root=unit, expected_sha256=point["sha256"], + run_id="phase-branch", phase_deadline_file=target) + assert target.read_bytes() == b"" and len(destination.state.judge.tokens) == 1 and grades == [] + original = copy.deepcopy(destination.state.judge.tokens) + epoch = json.loads(target.with_name("phase.restore.json").read_bytes())["deadline_epoch"] + monkeypatch.setattr(transport, "os", os) + fresh = transport.restore(files[1], root=unit, expected_sha256=point["sha256"], + run_id="phase-branch", phase_deadline_file=target, recover=True) + assert fresh["token"] in original and destination.state.judge.tokens == original + assert int(target.read_bytes()) == epoch and grades == [] + + +@pytest.mark.parametrize("fault", ["wrong_admin", "corrupt", "wrong_digest", "aliased", "existing_bank"]) +def test_invalid_controller_import_never_registers_or_regrades(controllers, tmp_path, fault): + source, destination, first, second, files, grades = controllers + token = register(source, first) + assert submit(first, token, 7).status_code == 200 + exported = tmp_path / "exported" + point = transport.capture(files[0], token=token, capture_id="b" * 32, destination=exported) + root, digest = exported, point["sha256"] + if fault == "wrong_admin": + config = json.loads(files[1].read_text()) + config["admin_secret"] = "not-current-admin" + files[1].write_text(json.dumps(config)) + elif fault == "corrupt": + (exported / "components/native_judge/incumbent.tar.gz").write_bytes(b"99") + elif fault == "wrong_digest": + digest = "0" * 64 + elif fault == "aliased": + root = tmp_path / "alias" + root.symlink_to(exported, target_is_directory=True) + else: + bank = destination.state.judge.config.log_dir / "checkpoints" / ("b" * 32) + bank.mkdir() + (bank / "foreign").write_text("keep") + with pytest.raises((ValueError, RuntimeError, FileNotFoundError)): + transport.restore(files[1], root=root, expected_sha256=digest, run_id="branch") + assert destination.state.judge.tokens == {} and len(grades) == 1 + if fault == "existing_bank": + assert (bank / "foreign").read_text() == "keep" + + +def test_redirect_never_forwards_controller_authority(tmp_path): + pytest.importorskip("fastapi") + from fastapi import FastAPI + from fastapi.responses import RedirectResponse + first, second = FastAPI(), FastAPI() + received = [] + @second.post("/steal") + def steal(): + received.append(True) + return {} + with server(second) as target: + @first.post("/api/v1/checkpoint/capture") + def redirect(): + return RedirectResponse(target + "/steal", status_code=307) + with server(first) as origin: + config = {"url": origin, "admin_secret": "synthetic-credential"} + with pytest.raises(ValueError, match="HTTP 307"): + transport._post(config, "/api/v1/checkpoint/capture", {"token": "source"}, 5) + assert received == [] + + +@pytest.mark.skipif(os.environ.get("LOOPX_EDGEBENCH_DOCKER_SMOKE") != "1", + reason="Requires explicit isolated native Docker qualification") +@pytest.mark.parametrize("phase_budget,lose_response", [(False, False), (True, False), (True, True)]) +def test_real_docker_controller_export_import_and_next_grade(tmp_path, monkeypatch, phase_budget, lose_response): + import uuid + import docker + import requests + from sforge.harness import checkpoint, judge_server as native + from sforge.harness.config import SForgeConfig + + name = "loopx-controller-" + uuid.uuid4().hex[:10] + client = docker.from_env() + image = client.images.get("python:3.12-slim") + image.tag(name + ".judge.fixture", tag="fixture") + tasks = tmp_path / "tasks" + tasks.mkdir() + (tasks / "BENCHMARK.yaml").write_text(f"name: {name}\nbase_images: {{python: {{}}}}\n") + (tasks / "fixture.json").write_text(json.dumps({ + "task_id": "fixture", "name": "Synthetic real grade", "base_image": "python", + "platform": "linux/arm64", "cwd": "/tmp", "internet": False, + "submit_paths": ["candidate.json"], "submit_exclude": [], + "work": {"image_tag": "fixture", "agent_query": "Synthetic fixture"}, + "judge": {"image_tag": "fixture", "eval_timeout": 20, "parser": "structured_json", + "selection": "score_first", "score_direction": "maximize", + "eval_cmd": "python3 -c 'import json; print(json.dumps(dict(score=json.load(open(\"candidate.json\"))[\"score\"], valid=True)))'"}, + })) + config = SForgeConfig(tasks_dir=tasks, log_dir=tmp_path / "source", judge_cpu_limit=1, judge_mem_limit="256m") + source = native.create_app(config, checkpoints=True) + destination = native.create_app(replace(config, log_dir=tmp_path / "branch"), checkpoints=True) + grades = [] + original = native.judge_submission + def actual_grade(**kwargs): + grades.append(kwargs["archive"]) + return original(**kwargs) + monkeypatch.setattr(native, "judge_submission", actual_grade) + def grade(url, token, score): + raw = io.BytesIO() + with tarfile.open(fileobj=raw, mode="w:gz") as archive: + body = json.dumps({"score": score}).encode() + member = tarfile.TarInfo("candidate.json") + member.size = len(body) + archive.addfile(member, io.BytesIO(body)) + response = requests.post(url + "/api/v1/submit", data={"token": token}, + files={"archive": ("candidate.tar.gz", raw.getvalue())}, timeout=10) + if response.status_code == 200: + sid = response.json()["submission_id"] + report = wait(lambda: (packet if (packet := requests.get(url + "/api/v1/result/" + sid, + timeout=10).json())["status"] in {"completed", "error"} else None)) + assert report["status"] == "completed" and report["report"]["score"] == score + return response + try: + with server(source) as first, server(destination) as second: + files = [] + for label, app, url in (("source", source, first), ("branch", destination, second)): + bank = app.state.judge.config.log_dir / "checkpoints" + bank.mkdir(parents=True, mode=0o700) + path = tmp_path / (label + "-admin.json") + path.write_text(json.dumps({"url": url, "admin_secret": app.state.judge.admin_secret, + "bank_root": str(bank)})) + path.chmod(0o600) + files.append(path) + token = register(source, first, max_agent_submissions=2, submission_cooldown=3600) + assert grade(first, token, 7).status_code == 200 + before = copy.deepcopy((source.state.judge.tokens, source.state.judge.submissions, source.state.judge.run_history)) + exported = tmp_path / "capsule" + phase_source = tmp_path / "source-phase" + if phase_budget: + phase_source.write_text(f"{int(time.time()) + 7200}\n") + point = transport.capture(files[0], token=token, capture_id="c" * 32, destination=exported, + phase_deadline_file=phase_source if phase_budget else None) + # Admission clock only: HTTP, transport/fsync, Docker and native + # grading remain real. Offline/import IO must not spend cooldown. + now = [time.time() + 7200] + clock = SimpleNamespace(time=lambda: now[0]) + monkeypatch.setattr(native, "time", clock) + monkeypatch.setattr(checkpoint, "time", clock) + monkeypatch.setattr(transport, "time", clock) + control = tmp_path / "host-control" + control.mkdir() + phase = control / "phase-deadline" + args = ["--controller-file", str(files[1]), "restore", "--root", str(exported), + "--expected-sha256", point["sha256"], "--run-id", "branch", + "--result-file", str(control / "result.json")] + if phase_budget: + args += ["--phase-deadline-file", str(phase)] + post = transport._post + routes = [] + def observed_post(controller, route, body, timeout): + routes.append(route) + result = post(controller, route, body, timeout) + if lose_response and route.endswith("/restore"): + raise TimeoutError("Lost reply after real native commit") + return result + monkeypatch.setattr(transport, "_post", observed_post) + if lose_response: + with pytest.raises(TimeoutError): + transport.main(args) + assert phase.read_bytes() == (control / "result.json").read_bytes() == b"" + retained_epoch = json.loads((control / "phase-deadline.restore.json").read_bytes())["deadline_epoch"] + native_state = copy.deepcopy((destination.state.judge.tokens, destination.state.judge.run_history)) + now[0] += 60 + transport.main(args + ["--recover"]) + transport.main(args + ["--recover"]) + assert int(phase.read_bytes()) == retained_epoch + assert (destination.state.judge.tokens, destination.state.judge.run_history) == native_state + assert routes == ["/api/v1/checkpoint/restore", "/api/v1/checkpoint/restore-result", + "/api/v1/checkpoint/restore-result"] + else: + transport.main(args) + fresh = json.loads((control / "result.json").read_bytes())["token"] + assert fresh != token and len(grades) == 1 + assert grade(second, fresh, 3).status_code == 429 + now[0] += 3601 + continued_score = 3 + if phase_budget: + # The actual SForge phase shell consumes the restored duration + # inside a disposable Linux container, then writes a new candidate. + prefix = host_phase_command(tmp_path, monkeypatch) + guest_control = "/opt/loopx-benchmark/control" + command = prefix + f"printf '%s' \"$remaining\"; printf '{{\"score\":3}}' > '{guest_control}/candidate.json'" + guest = client.containers.create(image.id, ["/bin/bash", "-c", command], + environment={"PATH": guest_control + "/bin:/usr/bin:/bin"}, + network_disabled=True, mem_limit="128m", nano_cpus=1_000_000_000) + try: + # Upload explicitly: a remote Docker daemon cannot read + # the caller's temporary directory through a host mount. + upload = io.BytesIO() + with tarfile.open(fileobj=upload, mode="w") as archive: + for relative in ("opt", "opt/loopx-benchmark", "opt/loopx-benchmark/control", + "opt/loopx-benchmark/control/bin"): + member = tarfile.TarInfo(relative) + member.type, member.mode = tarfile.DIRTYPE, 0o700 + archive.addfile(member) + for relative, data, mode in ( + ("phase-deadline", phase.read_bytes(), 0o600), + ("bin/date", f"#!/bin/sh\nprintf '%s\\n' {int(now[0])}\n".encode(), 0o700)): + member = tarfile.TarInfo("opt/loopx-benchmark/control/" + relative) + member.size, member.mode = len(data), mode + archive.addfile(member, io.BytesIO(data)) + assert guest.put_archive("/", upload.getvalue()) + guest.start() + assert guest.wait(timeout=30)["StatusCode"] == 0 + assert 3580 - (60 if lose_response else 0) < int(guest.logs()) <= 3600 - (60 if lose_response else 0) + stream, _ = guest.get_archive(guest_control + "/candidate.json") + try: + with tarfile.open(fileobj=io.BytesIO(b"".join(stream))) as archive: + continued_score = json.loads(archive.extractfile("candidate.json").read())["score"] + assert continued_score == 3 + finally: + stream.close() + finally: + guest.remove(force=True) + assert grade(second, fresh, continued_score).json()["round_id"] == "agent-2" + history = requests.get(second + "/api/v1/history", params={"token": fresh}, timeout=5).json() + assert history["best_score"] == 7 and history["best_round"] == "agent-1" + assert grade(second, fresh, 99).status_code == 429 and len(grades) == 2 + assert (source.state.judge.tokens, source.state.judge.submissions, source.state.judge.run_history) == before + with pytest.raises(ValueError, match="fresh physical" if phase_budget else "HTTP 409"): + transport.restore(files[1], root=exported, expected_sha256=point["sha256"], run_id="branch", + phase_deadline_file=phase if phase_budget else None) + finally: + for app in (source, destination): + app.state.judge._reaper_stop.set() + app.state.judge._reaper_thread.join(timeout=1) + client.images.remove(name + ".judge.fixture:fixture") + client.close() diff --git a/benchmark/tests/test_checkpoint_handoff.py b/benchmark/tests/test_checkpoint_handoff.py new file mode 100644 index 0000000000..0e2dacc8fa --- /dev/null +++ b/benchmark/tests/test_checkpoint_handoff.py @@ -0,0 +1,57 @@ +"""Private source rendezvous never substitutes a late or foreign point.""" +import hashlib +import json +import threading +import time + +import pytest + +from benchmark.runtime.checkpoint_handoff import await_controller_capture + + +@pytest.mark.parametrize("response", ["captured", "failed", "foreign", "timeout"]) +def test_original_request_only_and_cleanup_after_missed_point(tmp_path, response): + mailbox = tmp_path / "checkpoint-controller" + ready = mailbox / "ready.json" + observed = [] + def controller(): + until = time.time() + 2 + while not ready.exists() and time.time() < until: + time.sleep(.005) + raw = ready.read_bytes() + request = json.loads(raw) # Publication exposes a complete request. + observed.append(request) + if response == "timeout": + return + result = {"request_sha256": hashlib.sha256(raw).hexdigest(), + "ok": response == "captured", "error_kind": "SyntheticFailure", + "capture": {"research_point_qualified": False, "sha256": "retained"}} + if response == "foreign": + result["request_sha256"] = "another-point" + pending = mailbox / "reply.tmp" + pending.write_text(json.dumps(result)) + pending.rename(mailbox / (request["capture_id"] + ".json")) + thread = threading.Thread(target=controller) + thread.start() + try: + args = (tmp_path / "checkpoints", [{"id": "code", "source_path": "/task/code"}], {}, + {"source_sha256": "prepared", "source_witness": {}}, time.time() + .5) + if response == "captured": + assert await_controller_capture(*args)["sha256"] == "retained" + else: + with pytest.raises(TimeoutError if response == "timeout" else ValueError): + await_controller_capture(*args) + finally: + thread.join(3) + assert not thread.is_alive() and not ready.exists() and len(observed) == 1 + + +def test_surviving_request_is_not_overwritten(tmp_path): + mailbox = tmp_path / "checkpoint-controller" + mailbox.mkdir() + ready = mailbox / "ready.json" + ready.write_bytes(b"original uncertain request") + with pytest.raises(FileExistsError): + await_controller_capture(tmp_path / "checkpoints", [], {}, + {"source_sha256": "later", "source_witness": {}}, time.time() + 1) + assert ready.read_bytes() == b"original uncertain request" diff --git a/benchmark/tests/test_checkpoint_joint_receiver.py b/benchmark/tests/test_checkpoint_joint_receiver.py new file mode 100644 index 0000000000..7f6c5926bc --- /dev/null +++ b/benchmark/tests/test_checkpoint_joint_receiver.py @@ -0,0 +1,470 @@ +"""Qualification of an actual controller-captured point in a fresh Linux guest. + +The external fixture supplies a sealed point and its matching synthetic native +configuration. No source authority directories are mounted into this receiver. +""" + +import hashlib +import json +import os +from pathlib import Path +import shlex +import subprocess +import sys +import threading +import tomllib +from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer +from urllib.parse import urlsplit + +import pytest + +from benchmark.tests.test_checkpoint_controller import controllers # noqa: F401 + + +@pytest.mark.skipif( + not os.environ.get("LOOPX_JOINT_RECEIVER_POINT"), + reason="Requires a real controller point in a fresh isolated guest", +) +def test_captured_point_continues_with_fresh_authority( + tmp_path, monkeypatch, capsys, request +): + from benchmark.edgebench.checkpoint_controller import restore + from benchmark.runtime.codex import Execution + from benchmark.runtime.native_checkpoint import ( + admit_native_checkpoint_session, + native_session_checkpoint, + ) + from benchmark.runtime.research_checkpoint import adopt_todo_context + from benchmark.runtime.sessions import BenchmarkSessionWake + from benchmark.runtime.worker import native_command + from loopx.cli import main + from loopx.control_plane.effect_runtime import ( + effect_runtime_result, + EffectRuntimeRemoteError, + ) + from loopx.control_plane.quota.settlement import read_heartbeat_settlement + + unit = Path(os.environ["LOOPX_JOINT_RECEIVER_POINT"]) + manifest = json.loads((unit / "checkpoint.json").read_bytes()) + digest, metadata = manifest["sha256"], manifest["point_metadata"] + native, snapshot = metadata["native_session"], metadata["worker_file_snapshot"] + assert snapshot["prepared_source_verified"] + assert ("interrupted_guard" in native) != ("completed_turn_id" in native) + if "completed_turn_id" in native: + assert native["completed_turn_id"] == native["turn_ids"][-1] + else: + assert native["interrupted_guard"]["decision_sha256"] == metadata["decision_sha256"] + config = Path(os.environ["LOOPX_JOINT_RECEIVER_CONFIG"]) + workspace, home = Path(native["guest_workspace"]), Path(native["guest_home"]) + assert not workspace.exists() and not home.exists(), ( + "receiver must have fresh fixed guest paths" + ) + workspace.mkdir(parents=True) + monkeypatch.chdir(workspace) + candidate = workspace / "candidate.txt" + candidate.write_bytes((unit / "components/code").read_bytes()) + captured_candidate = candidate.read_text() + registry, runtime = tmp_path / "registry.json", tmp_path / "runtime" + goal, agent, turn = "joint-receiver", "joint-researcher", "joint-work" + prefix = [ + "--format", + "json", + "--registry", + str(registry), + "--runtime-root", + str(runtime), + ] + + def cli(*args): + code = main([*prefix, *args]) + output = capsys.readouterr().out + assert code == 0, output + return json.loads(output) + + cli( + "bootstrap", + "--project", + str(workspace), + "--goal-id", + goal, + "--objective", + "Continue the captured synthetic experiment.", + "--adapter-kind", + "read_only_project_map_v0", + "--adapter-status", + "connected-read-only", + "--write-scope", + "**", + "--no-global-sync", + ) + cli( + "configure-goal", + "--goal-id", + goal, + "--registered-agent", + agent, + "--agent-work-mode", + agent + "=active", + "--execute", + ) + todo = cli( + "todo", + "add", + "--goal-id", + goal, + "--role", + "agent", + "--claimed-by", + agent, + "--required-capability", + "shell", + "--task-class", + "advancement_task", + "--text", + "Use retained code and evidence for one synthetic candidate, then verify its native result.", + )["todo_id"] + execution = Execution(context="resume") + env = { + **os.environ, + "CODEX_BIN": os.environ["LOOPX_NATIVE_CHECKPOINT_BIN"], + "CODEX_HOME": str(home), + "LOOPX_PROJECT": str(workspace), + "LOOPX_REGISTRY": str(registry), + "LOOPX_RUNTIME_ROOT": str(runtime), + "LOOPX_GOAL_ID": goal, + "LOOPX_AGENT_ID": agent, + "MODEL_NAME": "fixture", + "REASONING_EFFORT": "high", + "LOOPX_CODEX_SANDBOX": execution.sandbox, + "SYNTHETIC_PROVIDER_KEY": "synthetic-local-only", + "OPENAI_API_KEY": "synthetic-local-only", + "LOOPX_USAGE_PING": "0", + } + with pytest.raises(ValueError, match="distinct destination"): + admit_native_checkpoint_session( + {**env, "LOOPX_GOAL_ID": metadata["loopx_state"]["goal_id"]}, + unit, + digest, + config, + execution=execution, + timeout=30, + ) + with pytest.raises(EffectRuntimeRemoteError): + admit_native_checkpoint_session( + env, unit, "f" * 64, config, execution=execution, timeout=30 + ) + assert not home.exists() + original_bind = BenchmarkSessionWake._bind_verified_fork + + def lose_binding_reply(self, *args): + original_bind(self, *args) + raise TimeoutError("synthetic loss after committed native binding") + + with monkeypatch.context() as lost: + lost.setattr(BenchmarkSessionWake, "_bind_verified_fork", lose_binding_reply) + with pytest.raises(TimeoutError, match="committed native binding"): + admit_native_checkpoint_session( + env, unit, digest, config, execution=execution, timeout=30 + ) + admitted = admit_native_checkpoint_session( + env, unit, digest, config, execution=execution, timeout=30 + ) + assert ( + admitted["session_id"] != native["session_id"] and admitted["model_calls"] == 0 + ) + replay = admit_native_checkpoint_session( + env, unit, digest, config, execution=execution, timeout=30 + ) + assert replay["session_id"] == admitted["session_id"] and replay["history_verified"] + lease = cli( + "task-lease", + "acquire", + "--goal-id", + goal, + "--todo-id", + todo, + "--owner", + agent, + "--idempotency-key", + "receiver-work", + ) + revision = cli("todo", "list", "--goal-id", goal, "--todo-id", todo)[ + "authority_read" + ]["provider_revision"] + carrier = json.loads((unit / "components/research_context").read_bytes()) + context_args = dict( + source_todo_id=carrier["snapshot"]["todos"][0]["todo_id"], + todo_id=todo, + operation_id="receiver-context", + expected_revision=revision, + lease_key="receiver-work", + lease_version=lease["lease"]["version"], + ) + adopted = adopt_todo_context(env, unit, digest, **context_args) + assert ( + adopt_todo_context(env, unit, digest, **context_args)["continuation"][ + "adoption" + ]["status"] + == "replayed" + ) + binding = [ + "--goal-id", + goal, + "--agent-id", + agent, + "--todo-id", + todo, + "--turn-instance-id", + turn, + ] + guard = cli("quota", "should-run", "--runtime-profile", "generic_cli", *binding) + assert guard["interaction_contract"]["agent_channel"]["delivery_allowed"] + _, branch, _, url, controller_files, grades = request.getfixturevalue("controllers") + capsule = unit / "components" / snapshot["native_capsule_component_id"] + opportunity = json.loads((capsule / "checkpoint.json").read_bytes())["point_metadata"][ + "native_reference" + ]["opportunity"] + remaining_cooldown = opportunity["remaining_cooldown_seconds"] + # Explicit fixture boundaries: either already eligible at capture, or a + # cooldown longer than this entire 60-second native work window. A short + # partial cooldown would make a timing-dependent expectation ambiguous. + assert remaining_cooldown == 0 or remaining_cooldown > 60 + cooldown_pending = remaining_cooldown > 60 + phase = tmp_path / "phase-deadline" + fresh = restore( + controller_files[1], + root=capsule, + expected_sha256=snapshot["native_capsule_sha256"], + run_id="isolated-receiver", + phase_deadline_file=phase, + ) + original_phase = phase.read_bytes() + replay_activation = restore( + controller_files[1], + root=capsule, + expected_sha256=snapshot["native_capsule_sha256"], + run_id="isolated-receiver", + phase_deadline_file=phase, + recover=True, + ) + assert ( + replay_activation == fresh + and not grades + and phase.read_bytes() == original_phase + ) + from benchmark.tests.test_checkpoint_controller import host_phase_command + + phase_shell = host_phase_command(tmp_path, monkeypatch).replace( + "/opt/loopx-benchmark/control", str(tmp_path) + ) + (workspace / "judge.json").write_text( + json.dumps({"url": url, "token": fresh["token"]}) + ) + (workspace / "judge.json").chmod(0o600) + script = workspace / "continue.py" + script.write_text("""import json,time,requests +from pathlib import Path +b=json.loads(Path('judge.json').read_text()); url,submission_token=b['url'],b['token'] +h=requests.get(url+'/api/v1/history',params={'token':submission_token},timeout=5).json() +assert h['best_score']==7 and h['agent_submissions']==1 +assert Path('candidate.txt').read_text()==CAPTURED_CANDIDATE +Path('candidate.txt').write_text('8') +r=requests.post(url+'/api/v1/submit',data={'token':submission_token},files={'archive':('candidate.tar.gz',b'8')},timeout=5) +if EXPECTED_COOLDOWN_PENDING: + assert r.status_code==429 + h=requests.get(url+'/api/v1/history',params={'token':submission_token},timeout=5).json() + assert h['agent_submissions']==1 and h['best_score']==7 + print('Candidate 8 validated locally; captured cooldown still blocks submission.') +else: + r.raise_for_status(); sid=r.json()['submission_id'] + for _ in range(100): + result=requests.get(url+'/api/v1/result/'+sid,timeout=5).json() + if result['status']=='completed':break + time.sleep(.01) + assert result['status']=='completed' + print('Retained candidate and public best 7 used; candidate 8 graded.') +""".replace("EXPECTED_COOLDOWN_PENDING", repr(cooldown_pending)) + .replace("CAPTURED_CANDIDATE", repr(captured_candidate))) + requests = [] + + class Provider(BaseHTTPRequestHandler): + def log_message(self, *args): + pass + + def do_POST(self): + requests.append( + json.loads(self.rfile.read(int(self.headers["Content-Length"]))) + ) + if len(requests) == 1: + message = { + "type": "function_call", + "id": "fc_receiver", + "call_id": "call_receiver", + "name": "exec_command", + "arguments": json.dumps( + { + "cmd": "cat " + + shlex.quote(adopted["artifact"]) + + " && " + + shlex.quote(sys.executable) + + " continue.py", + "max_output_tokens": 6000, + } + ), + } + else: + message = { + "type": "message", + "id": "msg_receiver", + "role": "assistant", + "content": [ + {"type": "output_text", "text": "Receiver candidate verified."} + ], + } + result = { + "id": "resp_receiver", + "status": "completed", + "output": [message], + "usage": {"input_tokens": 1, "output_tokens": 1, "total_tokens": 2}, + } + events = [ + { + "type": "response.created", + "response": {**result, "status": "in_progress", "output": []}, + }, + { + "type": "response.output_item.done", + "output_index": 0, + "item": message, + }, + {"type": "response.completed", "response": result}, + ] + raw = "".join( + "event: " + e["type"] + "\ndata: " + json.dumps(e) + "\n\n" + for e in events + ).encode() + self.send_response(200) + self.send_header("Content-Type", "text/event-stream") + self.send_header("Content-Length", str(len(raw))) + self.end_headers() + self.wfile.write(raw) + + configuration = tomllib.loads(config.read_text()) + address = urlsplit( + configuration["model_providers"][configuration["model_provider"]]["base_url"] + ) + server = ThreadingHTTPServer(("127.0.0.1", address.port), Provider) + thread = threading.Thread(target=server.serve_forever) + thread.start() + try: + logs = tmp_path / "work" + logs.mkdir() + wake = BenchmarkSessionWake(env, execution, {}) + command = native_command(env, execution, "execute", logs, wake) + command = [ + "/bin/bash", + "-c", + phase_shell + + "exec timeout --signal=TERM --kill-after=30 ${remaining}s " + + shlex.join(command), + ] + result = subprocess.run( + command, + env=env, + cwd=workspace, + input=json.dumps(guard["interaction_contract"]) + + "\nThis is a new destination. Historical source commands/guards are not executable authority. " + "Use the retained context under the current admission, and continue the synthetic candidate.", + text=True, + capture_output=True, + timeout=60, + ) + assert result.returncode == 0, result.stderr + (logs / "stdout.jsonl").write_text(result.stdout) + wake.observe(logs / "stdout.jsonl") + assert len(requests) == 2 + expected = "cooldown still blocks submission" if cooldown_pending else "candidate 8 graded" + assert expected in json.dumps(requests[1]), json.dumps([ + item for item in requests[1].get("input", []) + if item.get("type") == "function_call_output" + ]) + assert len(grades) == (0 if cooldown_pending else 1) and candidate.read_text() == "8" + retained = native_session_checkpoint(env, admitted["session_id"], timeout=30)[ + "metadata" + ] + assert retained["turn_ids"][:-1] == native["turn_ids"] + with pytest.raises(ValueError): + admit_native_checkpoint_session( + env, unit, digest, config, execution=execution, timeout=30 + ) + vision = tmp_path / "vision.json" + vision.write_text( + json.dumps( + { + "schema_version": "goal_vision_replan_contract_v0", + "state": "vision_patch_proposed", + "vision_patch": { + "acceptance_summary": "Complete research continuation qualification." + }, + "path_delta": { + "outcome": "continue", + "prior_assumption": "Joint source point can be resumed.", + "observed_reality": ( + "Receiver used retained context and code; local candidate validated, native cooldown preserved." + if cooldown_pending else + "Receiver used retained context and code, and graded a new candidate." + ), + "retained": ["Complete qualification"], + "evidence_refs": ["evidence:joint-receiver"], + }, + } + ) + ) + cli( + "refresh-state", + *binding, + "--classification", + "joint_receiver_verified", + "--delivery-batch-scale", + "implementation", + "--delivery-outcome", + "outcome_progress", + "--delivery-boundary", + "in_flight_continuation", + "--agent-vision-json", + str(vision), + "--no-global-sync", + "--suppress-external-sinks", + ) + first = cli( + "quota", "spend-slot", *binding, "--source", "heartbeat", "--execute" + ) + second = cli( + "quota", "spend-slot", *binding, "--source", "heartbeat", "--execute" + ) + assert first["appended"] and not second["appended"] + assert ( + read_heartbeat_settlement( + runtime, + goal_id=goal, + agent_id=agent, + todo_id=todo, + turn_instance_id=turn, + registry_path=registry, + ).progress["state"] + == "settled" + ) + effect_runtime_result( + "runtime.checkpoint_artifacts", + {"action": "verify", "root": str(unit), "expected_sha256": digest}, + ) + assert ( + hashlib.sha256( + (unit / "components/research_context").read_bytes() + ).hexdigest() + == metadata["research_context"]["sha256"] + ) + finally: + server.shutdown() + thread.join() + server.server_close() diff --git a/benchmark/tests/test_checkpoint_judge.py b/benchmark/tests/test_checkpoint_judge.py new file mode 100644 index 0000000000..0c6708ccae --- /dev/null +++ b/benchmark/tests/test_checkpoint_judge.py @@ -0,0 +1,170 @@ +"""Native owner semantics; real grading/transport is qualified by the Docker smoke.""" +import json +import hashlib +from types import SimpleNamespace + +import pytest + + +@pytest.fixture +def native_config(tmp_path, monkeypatch): + pytest.importorskip("sforge") + from sforge.harness import judge_server + from sforge.harness.config import SForgeConfig + tasks = tmp_path / "tasks" + tasks.mkdir() + (tasks / "BENCHMARK.yaml").write_text("name: fixture\nbase_images: {python: {}}\n") + (tasks / "fixture.json").write_text(json.dumps({ + "task_id": "fixture", "name": "Synthetic checkpoint", "base_image": "python", + "platform": "linux/arm64", "cwd": "/tmp", "internet": False, + "submit_paths": ["solver.py"], "submit_exclude": [], + "work": {"image_tag": "fixture", "agent_query": "Synthetic fixture"}, + "judge": {"image_tag": "fixture", "eval_timeout": 15, "parser": "structured_json", + "selection": "score_first", "score_direction": "maximize", "eval_cmd": "true"}, + })) + monkeypatch.setattr(judge_server, "create_backend_from_config", lambda config: SimpleNamespace()) + return SForgeConfig(tasks_dir=tasks, log_dir=tmp_path) + + +def test_disabled_extension_preserves_the_native_route_surface(native_config): + from benchmark.edgebench.checkpoint_judge import create_app + from sforge.harness.judge_server import create_app as native_app + first, second = native_app(native_config), create_app(native_config) + try: + assert [(r.path, r.methods) for r in first.routes] == [(r.path, r.methods) for r in second.routes] + finally: + first.state.judge.close() + second.state.judge.close() + + +def test_capture_retains_consumed_round_history_and_cooldown_without_mutation(native_config): + from fastapi.testclient import TestClient + from benchmark.edgebench.checkpoint_judge import create_app, ROUTE + app = create_app(native_config, checkpoint_capture=True) + state = app.state.judge + with TestClient(app) as client: + token = state.register_session("fixture", "source", max_agent_submissions=12, submission_cooldown=3600) + state.consume_round(token) + state._record_submission("source", "accepted-error", "fixture", "agent-1", None, + "synthetic evaluation failure", status="error") + state.submissions["accepted-error"] = {"run_id": "source", "task_id": "fixture", "round": "agent-1", + "status": "error", "report": None, "error": "synthetic evaluation failure"} + before = state.resolve_token(token) + result = client.post(ROUTE, json={"token": token}) + assert result.status_code == 200 + point = result.json() + assert point["next_agent_round"] == 2 and point["next_auto_round"] == 1 + assert point["max_agent_submissions"] == 12 + assert 0 < point["remaining_cooldown_seconds"] <= 3600 + assert point["last_agent_submit_at_epoch"] == before["last_agent_submit_at"] + assert point["released_history"] == state.get_run_history("source", "fixture")["entries"] + assert point["released_history"][0]["status"] == "error" + assert point["released_results"]["accepted-error"]["error"] == "synthetic evaluation failure" + assert point["execution_authority_granted"] is False + assert point["native_incumbent"] is None # Failure does not fabricate a candidate. + assert token not in json.dumps(point) and "admin_secret" not in json.dumps(point) + assert state.resolve_token(token) == before + assert client.post(ROUTE, json={"token": token}).json()["judge_epoch"] == point["judge_epoch"] + + +@pytest.mark.parametrize("gap", ["pending", "unreconciled", "automatic", "closed"]) +def test_incomplete_opportunity_state_cannot_be_a_checkpoint(native_config, gap): + from fastapi.testclient import TestClient + from benchmark.edgebench.checkpoint_judge import create_app, ROUTE + app = create_app(native_config, checkpoint_capture=True) + state = app.state.judge + with TestClient(app) as client: + token = state.register_session("fixture", "source", submission_cooldown=3600) + if gap == "pending": + state._outstanding_by_run[("source", "fixture")] = 1 + elif gap == "unreconciled": + state.consume_round(token) + elif gap == "automatic": + state.consume_round(token, "auto") + else: + state._closed = True + assert client.post(ROUTE, json={"token": token}).status_code == 409 + assert client.post(ROUTE, json={"token": "unregistered"}).status_code == 403 + assert client.post(ROUTE, json={"token": token, "restore": True}).status_code == 422 + + +@pytest.mark.parametrize("policy,direction,expected", [ + ("score_first", "maximize", "agent-1"), ("score_first", "minimize", "agent-2"), + ("pass_rate_first", "maximize", "agent-2"), ("valid_then_score", "maximize", "agent-2"), +]) +def test_native_selected_incumbent_archive_is_bound_to_its_result(native_config, policy, direction, expected): + from fastapi.testclient import TestClient + from benchmark.edgebench.checkpoint_judge import create_app, ROUTE, ARCHIVE_ROUTE + + task = json.loads((native_config.tasks_dir / "fixture.json").read_text()) + task["judge"].update(selection=policy, score_direction=direction) + (native_config.tasks_dir / "fixture.json").write_text(json.dumps(task)) + app = create_app(native_config, checkpoint_capture=True) + state = app.state.judge + with TestClient(app) as client: + token = state.register_session("fixture", "source") + candidates = {"agent-1": b"earlier higher score", "agent-2": b"later candidate"} + for index, (round_id, raw) in enumerate(candidates.items(), 1): + state.consume_round(token) + report = {"score": 9 if index == 1 else 3, "valid": index == 2, + "pass_rate": .5 if index == 1 else 1.0} + identifier = "submission-" + str(index) + state._record_submission("source", identifier, "fixture", round_id, report, None) + state.submissions[identifier] = {"run_id": "source", "task_id": "fixture", "round": round_id, + "status": "completed", "report": report, "error": None} + path = native_config.log_dir / "runs/source/fixture/submissions" / round_id / "submission.tar.gz" + path.parent.mkdir(parents=True) + path.write_bytes(raw) + before = state.resolve_token(token) + point = client.post(ROUTE, json={"token": token}).json() + native = point["native_incumbent"] + assert native["round"] == expected + assert native["archive_sha256"] == hashlib.sha256(candidates[expected]).hexdigest() + assert native["archive_bytes"] == len(candidates[expected]) + request = {"token": token, "capture_binding_sha256": point["capture_binding_sha256"]} + downloaded = client.post(ARCHIVE_ROUTE, json=request) + assert downloaded.status_code == 200 and downloaded.content == candidates[expected] + assert state.resolve_token(token) == before + assert client.post(ARCHIVE_ROUTE, json={**request, "token": "unregistered"}).status_code == 403 + assert client.post(ARCHIVE_ROUTE, json={**request, "capture_binding_sha256": "wrong-epoch"}).status_code == 409 + selected = native_config.log_dir / "runs/source/fixture/submissions" / expected / "submission.tar.gz" + selected.write_bytes(b"changed selected bytes") + assert client.post(ARCHIVE_ROUTE, json=request).status_code == 409 + selected.unlink() + assert client.post(ROUTE, json={"token": token}).status_code == 409 + selected.symlink_to(native_config.tasks_dir / "fixture.json") + assert client.post(ROUTE, json={"token": token}).status_code == 409 + selected.unlink() + selected.write_bytes(candidates[expected]) + state.consume_round(token) # A new accepted, unreconciled round invalidates the old point. + assert client.post(ARCHIVE_ROUTE, json=request).status_code == 409 + + +@pytest.mark.parametrize('payload', [b'wrong bytes', b'truncated']) +def test_incumbent_transport_mismatch_never_publishes_a_participant(tmp_path, monkeypatch, payload): + import io + import time + from benchmark.runtime import sforge_checkpoint + + expected = b'original accepted bytes' + packet = {'schema_version': sforge_checkpoint.SCHEMA, 'execution_authority_granted': False, + 'outstanding_evaluations': 0, 'capture_binding_sha256': 'bound-state', + 'native_incumbent': {'archive_bytes': len(expected), + 'archive_sha256': hashlib.sha256(expected).hexdigest()}} + calls = [] + def read(request, *, timeout): + calls.append(request) + return io.BytesIO(json.dumps(packet).encode() if len(calls) == 1 else payload) + monkeypatch.setattr(sforge_checkpoint, 'urlopen', read) + env = {'SFORGE_JUDGE_URL': 'http://synthetic.invalid', 'SFORGE_TOKEN': 'synthetic', + 'LOOPX_PHASE_DEADLINE_EPOCH': str(time.time() + 30)} + target = tmp_path / 'incumbent.tar.gz' + with pytest.raises(ValueError, match='differ'): + sforge_checkpoint.capture_opportunity(env, timeout=10, incumbent_path=target) + assert len(calls) == 2 and not target.exists() + assert json.loads(calls[1].data)['capture_binding_sha256'] == 'bound-state' + target.write_bytes(b'prior participant') + calls.clear() + with pytest.raises(FileExistsError): + sforge_checkpoint.capture_opportunity(env, timeout=10, incumbent_path=target) + assert target.read_bytes() == b'prior participant' diff --git a/benchmark/tests/test_checkpoint_worker.py b/benchmark/tests/test_checkpoint_worker.py new file mode 100644 index 0000000000..404542a0fd --- /dev/null +++ b/benchmark/tests/test_checkpoint_worker.py @@ -0,0 +1,397 @@ +"""Real Docker writer barrier plus native HTTP capture; no model inference.""" +from __future__ import annotations + +import io +import json +import os +import sqlite3 +import tarfile +import threading +import time +import uuid +from pathlib import Path + +import pytest + +from benchmark.edgebench import checkpoint_controller as transport +from benchmark.edgebench.checkpoint_worker import capture_worker +from benchmark.tests import test_checkpoint_controller as native_fixtures +from benchmark.tests.test_checkpoint_controller import register, submit, wait + + +controllers = native_fixtures.controllers + + +pytestmark = pytest.mark.skipif(os.environ.get("LOOPX_EDGEBENCH_DOCKER_SMOKE") != "1", + reason="Requires explicit isolated Docker writer-barrier qualification") + + +@pytest.fixture +def writer(tmp_path): + import docker + + client = docker.from_env(timeout=15) + image = os.environ.get("LOOPX_CHECKPOINT_WRITER_IMAGE", "python:3.12-slim") + client.images.get(image) # Never pull implicitly for this test. + script = """from pathlib import Path +import time, sqlite3 +Path('/data').mkdir() +Path('/phase').write_text(str(int(time.time())+7200)) +Path('/link').symlink_to('/data', target_is_directory=True) +db=sqlite3.connect('/data/research.sqlite') +db.execute('PRAGMA journal_mode=WAL') +db.execute('PRAGMA wal_autocheckpoint=0') +db.execute('CREATE TABLE evidence (value INTEGER)') +db.commit() +db.execute('PRAGMA wal_checkpoint(TRUNCATE)') +db.execute('INSERT INTO evidence VALUES (42)') +db.commit() +db.execute('INSERT INTO evidence VALUES (99)') +count=0 +while True: + count+=1 + Path('/data/next').write_text(str(count)) + Path('/data/next').replace('/data/counter') + time.sleep(.03) +""" + container = client.containers.run(image, ["python", "-c", script], detach=True, + name="loopx-checkpoint-writer-" + uuid.uuid4().hex[:12], + network_disabled=True, mem_limit="64m", nano_cpus=250_000_000) + request = tmp_path / "worker.json" + request.write_text(json.dumps({"container_id": container.id, + "components": [{"id": "native_capsule", "source_path": "/data"}], + "phase_deadline_path": "/phase"})) + try: + wait(lambda: container.exec_run(["test", "-f", "/data/counter"]).exit_code == 0) + yield container, request + finally: + container.remove(force=True) + assert not client.containers.list(all=True, filters={"id": container.id}) + client.close() + + +def counter(container): + stream, _ = container.get_archive("/data/counter") + with tarfile.open(fileobj=io.BytesIO(b"".join(stream))) as archive: + return int(archive.extractfile("counter").read()) + + +def test_archive_keeps_private_directory_modes_without_special_bits(writer, tmp_path): + from benchmark.edgebench.checkpoint_worker import _archive + container, _ = writer + assert container.exec_run(["python", "-c", "from pathlib import Path; " + "p=Path('/private-state'); p.mkdir(mode=0o700); " + "(p/'child').mkdir(mode=0o700); (p/'child').chmod(0o2700)"]).exit_code == 0 + copied = _archive(container, "/private-state", tmp_path / "copy") + assert copied.stat().st_mode & 0o7777 == 0o700 + assert (copied / "child").stat().st_mode & 0o7777 == 0o700 + + +@pytest.mark.parametrize("fault", [None, "source_moved", "expired"]) +def test_supervisor_captures_only_the_prepared_worker_boundary(controllers, writer, tmp_path, fault): + from loopx.control_plane.effect_runtime import effect_runtime_result + from benchmark.edgebench.checkpoint_worker import _archive + from benchmark.edgebench.checkpoint_supervisor import CheckpointSupervisor + + source, _, first, _, files, grades = controllers + container, _ = writer + token = register(source, first, submission_cooldown=3600) + assert submit(first, token, 7).status_code == 200 + assert container.exec_run(["python", "-c", "from pathlib import Path; " + "Path('/candidate').write_text('retained'); " + "Path('/logs/agent/checkpoint-controller').mkdir(parents=True); " + "Path('/opt/loopx-benchmark/control').mkdir(parents=True); " + "Path('/opt/loopx-benchmark/control/phase-deadline').write_bytes(Path('/phase').read_bytes())"]).exit_code == 0 + prepared = _archive(container, "/candidate", tmp_path / "prepared") + probe = effect_runtime_result("runtime.checkpoint_artifacts", {"action": "probe", + "components": [{"id": "code", "source_path": str(prepared)}]}) + nonce = "e" * 32 + request = {"capture_id": nonce, "components": [{"id": "code", "source_path": "/candidate"}], + "point_metadata": {"synthetic_boundary": True}, "source_sha256": probe["source_sha256"], + "source_witness": probe["source_witness"], + "deadline_epoch": time.time() + (-1 if fault == "expired" else 30)} + if fault == "source_moved": + assert container.exec_run(["python", "-c", "from pathlib import Path; Path('/candidate').write_text('later')"]).exit_code == 0 + assert container.exec_run(["python", "-c", "from pathlib import Path; " + "Path('/logs/agent/checkpoint-controller/ready.json').write_text(" + repr(json.dumps(request)) + ")"]).exit_code == 0 + supervisor = CheckpointSupervisor(files[0], tmp_path / "points") + supervisor.bind({"SFORGE_TOKEN": token}) + supervisor.start(container) + try: + wait(lambda: (tmp_path / "points" / (nonce + ".json")).exists()) + wait(lambda: container.exec_run(["test", "-f", "/logs/agent/checkpoint-controller/" + nonce + ".json"]).exit_code == 0) + finally: + supervisor.close() + response = json.loads((tmp_path / "points" / (nonce + ".json")).read_bytes()) + assert response["ok"] is (fault is None) + unit = tmp_path / "points" / nonce + if fault is None: + manifest = json.loads((unit / "checkpoint.json").read_bytes()) + metadata = manifest["point_metadata"] + assert metadata["synthetic_boundary"] is True + assert metadata["worker_file_snapshot"]["prepared_source_verified"] is True + assert (unit / "components/code").read_text() == "retained" + assert response["capture"]["research_point_qualified"] is False + else: + assert not unit.exists() + assert not (Path(json.loads(files[0].read_bytes())["bank_root"]) / nonce).exists() + container.reload() + assert not container.attrs["State"]["Paused"] and len(grades) == 1 + + +@pytest.mark.parametrize("directory", [False, True]) +def test_paused_sqlite_capture_retains_committed_wal_not_pending_transaction( + controllers, writer, tmp_path, directory): + """A valid main DB can omit all recent research while integrity_check passes.""" + source, _, first, _, files, _ = controllers + container, request = writer + token = register(source, first) + payload = json.loads(request.read_text()) + payload["components"] = [{"id": "research", "source_path": + "/data" if directory else "/data/research.sqlite"}] + request.write_text(json.dumps(payload)) + unit = tmp_path / "wal-point" + capture_worker(files[0], token=token, capture_id="b" * 32, + destination=unit, request_file=request) + captured = unit / "components/research" + database = captured / "research.sqlite" if directory else captured + with sqlite3.connect(database) as readback: + assert readback.execute("PRAGMA integrity_check").fetchone() == ("ok",) + assert readback.execute("SELECT value FROM evidence").fetchall() == [(42,)] + assert not database.with_name(database.name + "-wal").exists() + container.reload() + assert container.attrs["State"]["Running"] and not container.attrs["State"]["Paused"] + + +@pytest.mark.parametrize("fault", ["symlink", "directory", "transport"]) +def test_sqlite_companion_failure_cannot_publish_an_older_valid_database( + controllers, writer, tmp_path, monkeypatch, fault): + from docker.errors import APIError + from benchmark.edgebench import checkpoint_worker + + source, _, first, _, files, grades = controllers + container, request = writer + token = register(source, first) + payload = json.loads(request.read_text()) + payload["components"] = [{"id": "research", "source_path": "/data/research.sqlite"}] + request.write_text(json.dumps(payload)) + if fault == "transport": + original = checkpoint_worker._archive + def unavailable(worker, remote, destination): + if remote.endswith("-wal"): + raise APIError("Synthetic Docker archive transport failure") + return original(worker, remote, destination) + monkeypatch.setattr(checkpoint_worker, "_archive", unavailable) + else: + replacement = "p.symlink_to('/phase')" if fault == "symlink" else "p.mkdir()" + assert container.exec_run(["python", "-c", "from pathlib import Path; " + "p=Path('/data/research.sqlite-wal'); p.unlink(); " + replacement]).exit_code == 0 + unit = tmp_path / "invalid-point" + with pytest.raises((ValueError, APIError)): + capture_worker(files[0], token=token, capture_id="c" * 32, + destination=unit, request_file=request) + assert not unit.exists() and not grades + assert not (Path(json.loads(files[0].read_text())["bank_root"]) / ("c" * 32)).exists() + container.reload() + assert container.attrs["State"]["Running"] and not container.attrs["State"]["Paused"] + + +def test_non_sqlite_file_does_not_capture_similarly_named_neighbors(controllers, writer, tmp_path): + source, _, first, _, files, _ = controllers + container, request = writer + token = register(source, first) + assert container.exec_run(["python", "-c", + "from pathlib import Path; Path('/data/counter-wal').write_text('PRIVATE_NEIGHBOR')"]).exit_code == 0 + payload = json.loads(request.read_text()) + payload["components"] = [{"id": "counter", "source_path": "/data/counter"}] + request.write_text(json.dumps(payload)) + unit = tmp_path / "ordinary-point" + capture_worker(files[0], token=token, capture_id="d" * 32, + destination=unit, request_file=request) + assert all(b"PRIVATE_NEIGHBOR" not in member.read_bytes() + for member in unit.rglob("*") if member.is_file()) + + +def test_controller_cli_captures_frozen_worker_with_original_native_point( + controllers, writer, tmp_path, monkeypatch, capsys): + source, destination, first, second, files, grades = controllers + container, request = writer + token = register(source, first, max_agent_submissions=3, submission_cooldown=3600) + assert submit(first, token, 7).status_code == 200 + observed = [] + original = transport._post + def while_frozen(*args, **kwargs): + container.reload() + assert container.attrs["State"]["Paused"] + before = counter(container) + result = original(*args, **kwargs) + time.sleep(.15) + assert counter(container) == before + observed.append(before) + return result + monkeypatch.setattr(transport, "_post", while_frozen) + token_file = tmp_path / "token" + token_file.write_text(token) + unit = tmp_path / "unit" + transport.main(["--controller-file", str(files[0]), "capture-worker", + "--token-file", str(token_file), "--capture-id", "7" * 32, + "--destination", str(unit), "--worker-request", str(request)]) + receipt = json.loads(capsys.readouterr().out) + assert receipt["research_point_qualified"] is False + manifest = json.loads((unit / "checkpoint.json").read_bytes()) + metadata = manifest["point_metadata"]["worker_file_snapshot"] + assert metadata["native_cursor_qualified"] is False and metadata["process_memory_captured"] is False + assert metadata["native_capsule_component_id"] == "native_capsule_1" + assert int((unit / "components/native_capsule/counter").read_text()) == observed[0] + nested = unit / "components/native_capsule_1" + capsule = json.loads((nested / "checkpoint.json").read_bytes()) + native = capsule["point_metadata"]["native_reference"] + assert native["opportunity"]["next_agent"] == 2 and native["incumbent"]["round"] == "agent-1" + assert 3590 < native["opportunity"]["remaining_cooldown_seconds"] <= 3600 + assert 7190 < capsule["point_metadata"]["host_phase_budget"]["remaining_seconds"] <= 7200 + container.reload() + assert not container.attrs["State"]["Paused"] + wait(lambda: counter(container) > observed[0]) + assert int((unit / "components/native_capsule/counter").read_text()) == observed[0] + for secret in (token, source.state.judge.admin_secret): + assert all(secret.encode() not in p.read_bytes() for p in unit.rglob("*") if p.is_file()) + # This is an inert joint file export, not a shortcut to activation. + with pytest.raises(ValueError, match="declared capsule"): + transport.restore(files[1], root=unit, expected_sha256=receipt["sha256"], run_id="branch") + with pytest.raises(ValueError, match="later worker"): + capture_worker(files[0], token=token, capture_id="7" * 32, + destination=tmp_path / "later", request_file=request) + assert not (tmp_path / "later").exists() and destination.state.judge.tokens == {} and len(grades) == 1 + + +@pytest.mark.parametrize("fault", ["missing", "symlink", "judge", "pause_reply", "prepaused", "cancel", "unpause"]) +def test_worker_capture_failure_releases_only_its_barrier( + controllers, writer, tmp_path, monkeypatch, fault): + source, _, first, _, files, _ = controllers + container, request = writer + token = register(source, first) + from docker.errors import NotFound + + expected_error = {"missing": NotFound, "symlink": ValueError, "judge": ValueError, + "pause_reply": OSError, "prepaused": ValueError, "cancel": KeyboardInterrupt, + "unpause": OSError}[fault] + if fault in {"missing", "symlink"}: + body = json.loads(request.read_bytes()) + body["components"][0]["source_path"] = "/missing" if fault == "missing" else "/link" + request.write_text(json.dumps(body)) + elif fault == "judge": + body = json.loads(files[0].read_bytes()) + body["admin_secret"] = "wrong-synthetic-controller" + files[0].write_text(json.dumps(body)) + elif fault == "pause_reply": + original = type(container).pause + def lost_reply(self): + original(self) + raise OSError("injected lost pause reply") + monkeypatch.setattr(type(container), "pause", lost_reply) + elif fault == "prepaused": + container.pause() + elif fault == "cancel": + def cancel(*args, **kwargs): + raise KeyboardInterrupt("injected controller cancellation") + monkeypatch.setattr(transport, "_post", cancel) + else: + def unavailable(self): + raise OSError("injected daemon unpause failure") + monkeypatch.setattr(type(container), "unpause", unavailable) + before = counter(container) + with pytest.raises(expected_error): + capture_worker(files[0], token=token, capture_id="8" * 32, + destination=tmp_path / "unit", request_file=request) + assert not (tmp_path / "unit").exists() + container.reload() + assert container.attrs["State"]["Paused"] == (fault in {"prepaused", "unpause"}) + if fault not in {"prepaused", "unpause"}: + wait(lambda: counter(container) > before) + # No failed attempt is silently replaced by a later research state. + with pytest.raises((FileExistsError, ValueError)): + capture_worker(files[0], token=token, capture_id="8" * 32, + destination=tmp_path / "later", request_file=request) + assert not (tmp_path / "later").exists() + + +@pytest.mark.parametrize("separate_bank", [False, True]) +def test_concurrent_capture_does_not_release_another_operation_barrier( + controllers, writer, tmp_path, monkeypatch, separate_bank): + from concurrent.futures import ThreadPoolExecutor + from loopx.file_lock import LockAcquireTimeoutError + + source, _, first, _, files, _ = controllers + container, request = writer + token = register(source, first) + other_controller = files[0] + if separate_bank: + bank = tmp_path / "other-bank" + bank.mkdir() + config = json.loads(files[0].read_bytes()) + config["bank_root"] = str(bank) + other_controller = tmp_path / "other-controller.json" + other_controller.write_text(json.dumps(config)) + other_controller.chmod(0o600) + entered, release = threading.Event(), threading.Event() + original = transport._post + def held(*args, **kwargs): + entered.set() + assert release.wait(15) + return original(*args, **kwargs) + monkeypatch.setattr(transport, "_post", held) + with ThreadPoolExecutor(max_workers=1) as pool: + pending = pool.submit(capture_worker, files[0], token=token, capture_id="1" * 32, + destination=tmp_path / "first", request_file=request) + try: + assert entered.wait(15) + frozen = counter(container) + with pytest.raises(LockAcquireTimeoutError): + capture_worker(other_controller, token=token, capture_id="2" * 32, + destination=tmp_path / "second", request_file=request, timeout=.5) + container.reload() + assert container.attrs["State"]["Paused"] and counter(container) == frozen + assert not (tmp_path / "second").exists() + finally: + release.set() + assert pending.result(timeout=20)["research_point_qualified"] is False + container.reload() + assert not container.attrs["State"]["Paused"] + + +@pytest.mark.parametrize("entrypoint", ["provider", "cli"]) +def test_rejected_pause_preserves_foreign_barrier( + controllers, writer, tmp_path, monkeypatch, entrypoint): + from docker.errors import APIError + + source, _, first, _, files, _ = controllers + container, request = writer + token = register(source, first) + original = type(container).pause + def raced_pause(self): + # Another actor pauses after our initial state read; the second RPC + # reaches the real daemon and is determinately rejected. + original(container) + original(self) + monkeypatch.setattr(type(container), "pause", raced_pause) + def unexpected_native_call(*args, **kwargs): + pytest.fail("Rejected writer barrier must not capture evaluator state") + monkeypatch.setattr(transport, "_post", unexpected_native_call) + unit = tmp_path / "rejected" + with pytest.raises(APIError) as error: + if entrypoint == "provider": + capture_worker(files[0], token=token, capture_id="3" * 32, + destination=unit, request_file=request) + else: + token_file = tmp_path / "token" + token_file.write_text(token) + transport.main(["--controller-file", str(files[0]), "capture-worker", + "--token-file", str(token_file), "--capture-id", "3" * 32, + "--destination", str(unit), "--worker-request", str(request)]) + assert error.value.response.status_code == 409 + container.reload() + assert container.attrs["State"]["Paused"] + assert not unit.exists() + with pytest.raises(FileExistsError): + capture_worker(files[0], token=token, capture_id="3" * 32, + destination=tmp_path / "retry", request_file=request) diff --git a/benchmark/tests/test_edgebench_feedback_docker.py b/benchmark/tests/test_edgebench_feedback_docker.py index 921c067cc2..6148984671 100644 --- a/benchmark/tests/test_edgebench_feedback_docker.py +++ b/benchmark/tests/test_edgebench_feedback_docker.py @@ -93,6 +93,175 @@ def test_native_execution_disposition_uses_real_docker_exit(tmp_path, agent_imag backend.client.close() +def test_sforge_checkpoint_declaration_is_a_private_guest_artifact(tmp_path, agent_image, monkeypatch): + from sforge.harness.config import SForgeConfig + from benchmark.runtime.sforge import SForgeWorker, BenchmarkCodex + from benchmark.runtime.sforge_backend import RecordingDockerBackend + + # Existing installation prepares this directory. Substitute installation, + # while retaining the production upload, permissions and worker environment. + async def installed(self, environment): + result = await environment.exec( + "mkdir -p /opt/loopx-benchmark/control", user="root") + assert result.return_code == 0 + monkeypatch.setattr(BenchmarkCodex, "install", installed) + monkeypatch.setenv("CODEX_AUTH_JSON_PATH", "/synthetic-unused-credential") + declaration = b'{"destination":"/logs/agent/checkpoints","trigger":"fresh_replan","native_session":true,"components":[{"id":"code","source_path":"/tmp/solver.py"}],"point_metadata":{}}' + worker = SForgeWorker(SForgeConfig(agent_model="fixture", agent_effort="xhigh"), + profile="heartbeat-resume", cwd="/tmp", checkpoint_request=declaration) + backend = RecordingDockerBackend(log_dir=tmp_path / "collected", + logger=logging.getLogger("checkpoint-staging-smoke"), oauth_proxy=True) + handle = backend.create_container(agent_image.id, "checkpoint-stage-" + uuid.uuid4().hex[:10], + cpu_limit=1, mem_limit="256m") + backend.start_container(handle) + try: + worker.install_stop_hook(backend, handle, tmp_path, None) + path = worker.runtime._worker_env(cwd="/tmp")["LOOPX_CHECKPOINT_REQUEST"] + readback = backend.exec_run(handle, ["cat", path], user="agent") + assert readback.exit_code == 0 and readback.output.encode() == declaration + mode = backend.exec_run(handle, ["stat", "-c", "%a:%U", path], user="root") + assert mode.exit_code == 0 and mode.output.strip() == "600:agent" + finally: + backend.cleanup_container(handle) + backend.client.close() + + +def test_checkpoint_opportunity_uses_native_admission_and_real_grade(tmp_path, monkeypatch): + import docker + import requests + import uvicorn + from sforge.harness import judge_server + from sforge.harness.config import SForgeConfig + from benchmark.edgebench.checkpoint_judge import create_app, ROUTE + from benchmark.runtime.sforge_checkpoint import capture_opportunity + + name = "checkpoint-judge-" + uuid.uuid4().hex[:10] + tag = f"{name}.judge.fixture:fixture" + client = docker.from_env() + client.images.get("python:3.12-slim").tag(f"{name}.judge.fixture", tag="fixture") + tasks = tmp_path / "tasks" + tasks.mkdir() + (tasks / "BENCHMARK.yaml").write_text(f"name: {name}\nbase_images: {{python: {{}}}}\n") + (tasks / "fixture.json").write_text(json.dumps({ + "task_id": "fixture", "name": "Synthetic checkpoint", "base_image": "python", + "platform": "linux/arm64", "cwd": "/tmp", "internet": False, + "submit_paths": ["candidate.json"], "submit_exclude": [], + "work": {"image_tag": "fixture", "agent_query": "Synthetic fixture"}, + "judge": {"image_tag": "fixture", "eval_timeout": 15, "parser": "structured_json", + "selection": "score_first", "score_direction": "maximize", + "eval_cmd": "python3 -c 'import json; print(json.dumps(dict(score=json.load(open(\"candidate.json\"))[\"score\"], valid=True)))'"}, + })) + config = SForgeConfig(tasks_dir=tasks, log_dir=tmp_path, judge_cpu_limit=1, judge_mem_limit="256m", + judge_max_outstanding_per_run=1) + finish_grade = threading.Event() + original_grade = judge_server.judge_submission + def held_grade(**kwargs): + assert finish_grade.wait(60), "Synthetic native grade was not released" + return original_grade(**kwargs) + monkeypatch.setattr(judge_server, "judge_submission", held_grade) + app = create_app(config, checkpoint_capture=True) + with socket.socket() as port_probe: + port_probe.bind(("127.0.0.1", 0)) + port = port_probe.getsockname()[1] + url = f"http://127.0.0.1:{port}" + server = uvicorn.Server(uvicorn.Config(app, host="127.0.0.1", port=port, + log_level="error", access_log=False)) + thread = threading.Thread(target=server.run) + thread.start() + try: + wait_for(lambda: server.started, timeout=20) + registered = requests.post(url + "/api/v1/register", json={"task_id": "fixture", + "run_id": "synthetic-source", "admin_secret": app.state.judge.admin_secret, + "submission_cooldown": 3600, "max_agent_submissions": 12}, timeout=10) + registered.raise_for_status() + token = registered.json()["token"] + def candidate(score): + payload = io.BytesIO() + with tarfile.open(fileobj=payload, mode="w:gz") as archive: + member = tarfile.TarInfo("candidate.json") + data = json.dumps({"score": score}).encode() + member.size = len(data) + archive.addfile(member, io.BytesIO(data)) + return payload.getvalue() + first_archive = candidate(7) + accepted = requests.post(url + "/api/v1/submit", data={"token": token}, + files={"archive": ("candidate.tar.gz", first_archive)}, timeout=15) + accepted.raise_for_status() + assert accepted.json()["round_id"] == "agent-1" + assert requests.post(url + ROUTE, json={"token": token}, timeout=10).status_code == 409 + finish_grade.set() + wait_for(lambda: not app.state.judge._outstanding_by_run, timeout=60) + released = requests.post(url + ROUTE, json={"token": token}, timeout=10) + assert released.status_code == 200, released.json() + env = {"SFORGE_JUDGE_URL": url, "SFORGE_TOKEN": token, + "LOOPX_PHASE_DEADLINE_EPOCH": str(time.time() + 100)} + point = capture_opportunity(env, timeout=10, incumbent_path=tmp_path / "incumbent.tar.gz") + assert (tmp_path / "incumbent.tar.gz").read_bytes() == first_archive + assert point["released_history"][0]["score"] == 7 + assert point["next_agent_round"] == 2 and 0 < point["remaining_cooldown_seconds"] <= 3600 + assert 0 < point["remaining_trial_seconds"] <= 100 + assert point["continuation_qualified"] is False + before = app.state.judge.resolve_token(token) + assert capture_opportunity(env, timeout=10, incumbent_path=tmp_path / "second-read.tar.gz")["judge_epoch"] == point["judge_epoch"] + assert app.state.judge.resolve_token(token) == before + cooled = requests.post(url + "/api/v1/submit", data={"token": token}, + files={"archive": ("candidate.tar.gz", first_archive)}, timeout=15) + assert cooled.status_code == 429 # Reading the checkpoint cannot refill admission. + assert app.state.judge.resolve_token(token) == before + # A fresh identity is a valid new trial, but not a same-point branch. + # Even with the same remaining deadline and declared limits, native + # registration resets the counter and makes its first submission eligible. + cold = requests.post(url + "/api/v1/register", json={"task_id": "fixture", + "run_id": "synthetic-new-registration", "admin_secret": app.state.judge.admin_secret, + "submission_cooldown": 3600, "max_agent_submissions": 12}, timeout=10) + cold.raise_for_status() + cold_token = cold.json()["token"] + cold_point = capture_opportunity({**env, "SFORGE_TOKEN": cold_token}, timeout=10, + incumbent_path=tmp_path / "cold-incumbent.tar.gz") + assert cold_point["next_agent_round"] == 1 and cold_point["remaining_cooldown_seconds"] == 0 + assert cold_point["released_history"] == [] and cold_point["native_incumbent"] is None + assert cold_point["max_agent_submissions"] - (cold_point["next_agent_round"] - 1) == 12 + assert point["max_agent_submissions"] - (point["next_agent_round"] - 1) == 11 + cold_accepted = requests.post(url + "/api/v1/submit", data={"token": cold_token}, + files={"archive": ("candidate.tar.gz", first_archive)}, timeout=15) + cold_accepted.raise_for_status() + assert cold_accepted.json()["round_id"] == "agent-1" + wait_for(lambda: not app.state.judge._outstanding_by_run, timeout=60) + # Neither copying the incumbent bytes nor replaying its grade restored + # consumed source rounds/history or its currently active cooldown. + cold_after = capture_opportunity({**env, "SFORGE_TOKEN": cold_token}, timeout=10, + incumbent_path=tmp_path / "cold-after-incumbent.tar.gz") + assert cold_after["next_agent_round"] == 2 and len(cold_after["released_history"]) == 1 + assert cold_after["released_history"][0]["submission_id"] != point["released_history"][0]["submission_id"] + assert app.state.judge.resolve_token(token) == before + with pytest.raises(ValueError, match="original trial deadline"): + capture_opportunity({**env, "LOOPX_PHASE_DEADLINE_EPOCH": "0"}, timeout=10, + incumbent_path=tmp_path / "closed.tar.gz") + # Separate native registration with no cooldown: two actual grades must + # retain the older winner, not substitute the later working candidate. + registered = requests.post(url + "/api/v1/register", json={"task_id": "fixture", + "run_id": "synthetic-selection", "admin_secret": app.state.judge.admin_secret}, timeout=10) + registered.raise_for_status() + selection_token = registered.json()["token"] + for raw in (first_archive, candidate(3)): + response = requests.post(url + "/api/v1/submit", data={"token": selection_token}, + files={"archive": ("candidate.tar.gz", raw)}, timeout=15) + response.raise_for_status() + wait_for(lambda: not app.state.judge._outstanding_by_run, timeout=60) + selected = capture_opportunity({**env, "SFORGE_TOKEN": selection_token}, timeout=10, + incumbent_path=tmp_path / "selected.tar.gz") + assert selected["next_agent_round"] == 3 and selected["native_incumbent"]["round"] == "agent-1" + assert [entry["score"] for entry in selected["released_history"]] == [7, 3] + assert (tmp_path / "selected.tar.gz").read_bytes() == first_archive + finally: + finish_grade.set() + server.should_exit = True + thread.join(timeout=60) + assert not thread.is_alive(), "Synthetic checkpoint service did not drain" + client.images.remove(tag) + client.close() + + def test_sforge_trial_deadline_requires_execution_entry(tmp_path, agent_image, monkeypatch): """Exercise the actual formatted shell, including its inner GNU timeout.""" from sforge.harness.config import SForgeConfig diff --git a/benchmark/tests/test_native_checkpoint.py b/benchmark/tests/test_native_checkpoint.py new file mode 100644 index 0000000000..a0e5264fcc --- /dev/null +++ b/benchmark/tests/test_native_checkpoint.py @@ -0,0 +1,847 @@ +"""Native checkpoint history must be complete and cannot become execution authority.""" +import hashlib +import json +import os +import threading +import time +import uuid +from contextlib import contextmanager +from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer + +import pytest + +from benchmark.runtime.native_checkpoint import native_session_checkpoint +from benchmark.tests.test_checkpoint_controller import controllers # noqa: F401 -- shared real HTTP fixture + + +def fixture(tmp_path, monkeypatch, pages): + home = tmp_path / "native" + home.mkdir() + for name in ("state_5.sqlite", "thread_history_1.sqlite", "config.toml", "auth.json", "logs_1.sqlite"): + (home / name).write_text("synthetic private fixture") + (home / "sessions").mkdir() + (home / "skills").symlink_to(home / "sessions", target_is_directory=True) + calls = [] + + class Native: + response_timeout_sec = 1 + + def __enter__(self): + return self + + def __exit__(self, *args): + pass + + def notify(self, method, params): + calls.append(method) + + def request(self, method, params): + calls.append(method) + if method == "initialize": + return {} + if method == "thread/read": + return {"thread": {"id": "fixture-session", "cwd": str(tmp_path)}} + assert method == "thread/turns/list" # No model or fork is requested. + return pages.pop(0) + + monkeypatch.setattr("benchmark.runtime.native_checkpoint.StdioNativeGoalTransport.spawn", lambda *a, **k: Native()) + monkeypatch.setattr("benchmark.runtime.native_checkpoint.subprocess.check_output", lambda *a, **k: "codex-cli fixture") + return {"CODEX_HOME": str(home), "CODEX_BIN": "/fixture/codex", "LOOPX_PROJECT": str(tmp_path)}, calls + + +def test_all_native_pages_and_stores_are_retained_without_credentials(tmp_path, monkeypatch): + turns = [{"id": "before-compaction", "status": "completed", "items": [{"type": "compaction"}]}, + {"id": "after-compaction", "status": "completed", "items": []}] + env, calls = fixture(tmp_path, monkeypatch, [{"data": turns[:1], "nextCursor": "next"}, + {"data": turns[1:], "nextCursor": None}]) + result = native_session_checkpoint(env, "fixture-session", timeout=10) + metadata = result["metadata"] + assert metadata["turn_ids"] == [t["id"] for t in turns] + assert metadata["completed_turn_id"] == "after-compaction" + assert metadata["visible_history_sha256"] == hashlib.sha256( + json.dumps(turns, sort_keys=True, separators=(",", ":")).encode()).hexdigest() + assert metadata["execution_authority_granted"] is False + assert {c["id"] for c in result["components"]} == { + "native_sessions", "native_state_5_sqlite", "native_thread_history_1_sqlite"} + assert calls == ["initialize", "initialized", "thread/read", "thread/turns/list", "thread/turns/list"] + + +@pytest.mark.parametrize("pages,reason", [ + ([{"data": [], "nextCursor": None}], "complete retained"), + ([{"data": [{"id": "active", "status": "inProgress"}], "nextCursor": None}], "complete retained"), + ([{"data": [{"id": "failed", "status": "failed"}], "nextCursor": None}], "completed final"), + ([{"data": [], "nextCursor": "same"}, {"data": [], "nextCursor": "same"}], "repeated history"), + ([{"data": [{"id": "same", "status": "completed"}] * 2, "nextCursor": None}], "duplicate turns"), +]) +def test_incomplete_or_ambiguous_native_history_is_not_a_checkpoint(tmp_path, monkeypatch, pages, reason): + env, _ = fixture(tmp_path, monkeypatch, pages) + with pytest.raises(ValueError, match=reason): + native_session_checkpoint(env, "fixture-session", timeout=10) + + +def test_foreign_native_session_is_rejected_before_history(tmp_path, monkeypatch): + env, calls = fixture(tmp_path, monkeypatch, []) + with pytest.raises(ValueError, match="identity mismatch"): + native_session_checkpoint(env, "foreign-session", timeout=10) + assert "thread/turns/list" not in calls + + +@contextmanager +def real_native_source(root, binary, workspace, *, close_source=False, response_builder=None, sandbox="read-only"): + """Real native writer/readback; a local static stream replaces only the model. + + No paid model, tool invocation, user credential or benchmark cell. Fixtures + become completed Turns through the native protocol, not copied fake receipts. + """ + from loopx.capabilities.benchmark_toolkit.native_codex_goal import ( + NativeGoalProtocolError, StdioNativeGoalTransport, + ) + home = root / "home" + home.mkdir(parents=True) + calls = [] + class Provider(BaseHTTPRequestHandler): + def log_message(self, *args): + pass + + def do_POST(self): + request = json.loads(self.rfile.read(int(self.headers.get("Content-Length", 0)))) + calls.append(self.path) + message = {"type": "message", "id": "msg_" + uuid.uuid4().hex, "role": "assistant", + "content": [{"type": "output_text", "text": "Synthetic source result."}]} + if response_builder is not None: + message = response_builder(request, len(calls), message) + result = {"id": "resp_" + uuid.uuid4().hex, "status": "completed", "output": [message], + "usage": {"input_tokens": 1, "output_tokens": 1, "total_tokens": 2}} + events = [{"type": "response.created", "response": {**result, "status": "in_progress", "output": []}}, + {"type": "response.output_item.done", "output_index": 0, "item": message}, + {"type": "response.completed", "response": result}] + raw = "".join("event: " + event["type"] + "\ndata: " + json.dumps(event) + "\n\n" + for event in events).encode() + self.send_response(200) + self.send_header("Content-Type", "text/event-stream") + self.send_header("Content-Length", str(len(raw))) + self.end_headers() + self.wfile.write(raw) + server = ThreadingHTTPServer(("127.0.0.1", 0), Provider) + thread = threading.Thread(target=server.serve_forever) + thread.start() + (home / "config.toml").write_text( + 'model="fixture"\nmodel_reasoning_effort="high"\nmodel_provider="fixture"\napproval_policy="never"\n' + f'sandbox_mode={json.dumps(sandbox)}\n' + + ('[sandbox_workspace_write]\nnetwork_access=true\n' if sandbox == "workspace-write" else "") + + '[features]\ngoals=false\nresponses_websockets_v2=false\nplugins=false\nremote_plugin=false\nplugin_sharing=false\n' + '[model_providers.fixture]\nname="Synthetic local"\n' + f'base_url="http://127.0.0.1:{server.server_port}/v1"\nwire_api="responses"\n' + 'env_key="SYNTHETIC_PROVIDER_KEY"\n') + # No ambient provider credentials or app integrations enter this fixture. + env = {key: value for key, value in os.environ.items() + if key in {"PATH", "HOME", "USER", "LOGNAME", "SHELL", "TMPDIR", "LANG", "LC_ALL", + "SYSTEMROOT", "PYTHONPATH", "VIRTUAL_ENV"}} + env.update(CODEX_HOME=str(home), CODEX_BIN=binary, LOOPX_PROJECT=str(workspace), + SYNTHETIC_PROVIDER_KEY="synthetic-local-only", LOOPX_USAGE_PING="0") + try: + with (root / "stderr.txt").open("w") as stderr, StdioNativeGoalTransport.spawn( + [binary, "-c", "features.goals=false", "app-server", "--listen", "stdio://"], + cwd=str(workspace), env=env, stderr=stderr, response_timeout_sec=20, + ) as native: + native.request("initialize", {"clientInfo": {"name": "checkpoint_qualification", "version": "0.1.0"}, + "capabilities": {"experimentalApi": True}}) + native.notify("initialized", {}) + source = native.request("thread/start", {"cwd": str(workspace), "sandbox": sandbox, + "approvalPolicy": "never", "persistExtendedHistory": True})["thread"]["id"] + def complete_turn(): + expected_turn = native.request("turn/start", {"threadId": source, + "input": [{"type": "text", "text": "Synthetic storage fixture. Return its source result."}]})["turn"]["id"] + stop = time.monotonic() + 30 + while time.monotonic() < stop: + try: + page = native.request("thread/turns/list", {"threadId": source, + "sortDirection": "asc", "itemsView": "full", "limit": 100}) + except NativeGoalProtocolError: + # A first live Turn may not have a persisted page yet. + time.sleep(.05) + continue + if (page.get("data") and page["data"][-1].get("id") == expected_turn and + page["data"][-1].get("status") == "completed"): + return page["data"][-1]["id"] + time.sleep(.05) + raise AssertionError("Real native fixture did not complete its source Turn") + complete_turn() + if close_source: + native.close() + yield env, source, complete_turn, calls + finally: + server.shutdown() + server.server_close() + thread.join() + + +@pytest.mark.skipif(not os.environ.get("LOOPX_NATIVE_CHECKPOINT_BIN"), + reason="Requires an explicit real native Host binary; no model service used") +def test_real_native_interrupt_retains_branch_but_does_not_qualify_checkpoint(tmp_path): + """A resumable native interruption is not a completed research boundary. + + This characterizes the provider seam needed by midwake replan handoff. The + existing checkpoint must keep rejecting it until the Host proves the joint + boundary; a successful native fork alone must not loosen that contract. + """ + import subprocess + from loopx.capabilities.benchmark_toolkit.native_codex_goal import StdioNativeGoalTransport + + workspace = tmp_path / "work" + workspace.mkdir() + (workspace / "candidate.txt").write_text("retained-candidate\n") + requests = [] + + def response(request, index, message): + requests.append(request) + if index == 1: + message["content"][0]["text"] = "Retained research marker: retained-candidate." + elif index in {2, 3}: + command = ( + "python3 -c \"from pathlib import Path; import time; " + "Path('boundary-ready').write_text('ready'); time.sleep(30)\"" + if index == 2 else + "cat candidate.txt && printf 'continued-candidate\\n' > continued.txt" + ) + return {"type": "function_call", "id": f"fc_{index}", "call_id": f"call_{index}", + "name": "exec_command", "arguments": json.dumps({"cmd": command, + "yield_time_ms": 1000, "max_output_tokens": 200})} + else: + message["content"][0]["text"] = "Verified retained inputs and produced the next candidate." + return message + + @contextmanager + def native_host(env): + with StdioNativeGoalTransport.spawn( + [env["CODEX_BIN"], "-c", "features.goals=false", "app-server", "--listen", "stdio://"], + cwd=str(workspace), env=env, stderr=subprocess.DEVNULL, response_timeout_sec=15, + ) as native: + native.request("initialize", {"clientInfo": {"name": "handoff_fixture", "version": "0.1.0"}, + "capabilities": {"experimentalApi": True}}) + native.notify("initialized", {}) + yield native + + def history(native, session): + return native.request("thread/turns/list", {"threadId": session, + "sortDirection": "asc", "itemsView": "full", "limit": 100})["data"] + + def wait_terminal(native, session, turn, status): + deadline = time.monotonic() + 15 + while time.monotonic() < deadline: + rows = history(native, session) + if rows and rows[-1]["id"] == turn and rows[-1]["status"] == status: + return rows + time.sleep(.02) + raise AssertionError("Native handoff did not reach its expected terminal state") + + with real_native_source(tmp_path / "native", os.environ["LOOPX_NATIVE_CHECKPOINT_BIN"], + workspace, close_source=True, response_builder=response, + sandbox="workspace-write") as (env, source, _, _calls): + with native_host(env) as native: + native.request("thread/resume", {"threadId": source, "cwd": str(workspace), + "sandbox": "workspace-write", "approvalPolicy": "never", "persistExtendedHistory": True}) + turn = native.request("turn/start", {"threadId": source, "input": [ + {"type": "text", "text": "Reach the synthetic Host handoff boundary."}]})["turn"]["id"] + deadline = time.monotonic() + 10 + while not (workspace / "boundary-ready").exists() and time.monotonic() < deadline: + time.sleep(.01) + assert (workspace / "boundary-ready").exists() + native.request("turn/interrupt", {"threadId": source, "turnId": turn}) + retained = wait_terminal(native, source, turn, "interrupted") + assert len(requests) == 2 # No later model decision consumed the boundary. + with pytest.raises(ValueError, match="completed final cursor"): + native_session_checkpoint(env, source, timeout=15) + with native_host(env) as native: + fork = native.request("thread/fork", {"threadId": source, "cwd": str(workspace), + "lastTurnId": turn, "persistExtendedHistory": True})["thread"]["id"] + assert fork != source and history(native, fork) == retained + next_turn = native.request("turn/start", {"threadId": fork, "input": [ + {"type": "text", "text": "Verify the retained inputs and produce continued.txt."}]})["turn"]["id"] + wait_terminal(native, fork, next_turn, "completed") + assert (workspace / "continued.txt").read_text() == "continued-candidate\n" + assert "retained-candidate" in json.dumps(requests[2]) + assert history(native, source) == retained + + +@pytest.mark.skipif(not os.environ.get("LOOPX_NATIVE_CHECKPOINT_BIN"), + reason="Requires an explicit real native Host binary; no model service used") +def test_real_native_workspace_readback_rejects_foreign_code_basis(tmp_path): + from pathlib import Path + workspace = tmp_path / "source-workspace" + workspace.mkdir() + with real_native_source(tmp_path / "native", os.environ["LOOPX_NATIVE_CHECKPOINT_BIN"], workspace) as (env, source, _, calls): + valid = native_session_checkpoint(env, source, timeout=20) + assert valid["metadata"]["guest_workspace"] == str(workspace) + assert len(valid["metadata"]["turn_ids"]) == 1 and calls == ["/v1/responses"] + foreign = Path(env["LOOPX_PROJECT"]).with_name("foreign-workspace") + foreign.mkdir() + with pytest.raises(ValueError, match="workspace"): + native_session_checkpoint({**env, "LOOPX_PROJECT": str(foreign)}, source, timeout=20) + + +@pytest.mark.skipif(not os.environ.get("LOOPX_NATIVE_CHECKPOINT_BIN"), + reason="Requires an explicit real native Host binary; only a local static provider is used") +@pytest.mark.parametrize("case", ["resume", "scope_change", "profile_change", "joint", "binding_reply_loss", + "fork_reply_loss", "fork_missing", "fork_ambiguous"]) +def test_real_fork_admission_and_native_resume_use_retained_state(tmp_path, capsys, monkeypatch, case, request): + """Real Host/CLI/Goal/session/artifact owners; deterministic synthetic work. + + This jointly qualifies native conversation and canonical Todo evidence + adoption. The joint case adds actual native HTTP and Host phase entry, with + synthetic grading. Full cross-owner barriers and research utility remain + outside this fixture. No benchmark cell or paid model is launched. + """ + import shutil + import subprocess + import sys + from pathlib import Path + from benchmark.runtime.codex import Execution + from benchmark.runtime.native_checkpoint import admit_native_checkpoint_session + from benchmark.runtime.sessions import BenchmarkSessionWake + from benchmark.runtime.worker import native_command + from loopx.cli import main + from loopx.control_plane.effect_runtime import effect_runtime_result + from loopx.control_plane.turn_driver.codex_sessions import load_codex_cli_session, _store_codex_cli_session + from loopx.control_plane.quota.settlement import read_heartbeat_settlement + from loopx.control_plane.projects.registry_codec import ProjectRegistryProtocolError + + workspace = tmp_path / "workspace" + workspace.mkdir() + monkeypatch.chdir(workspace) # Real settlement verifies the producing workspace. + candidate = workspace / "candidate.txt" + candidate.write_text("retained candidate: 7\n") + research = workspace / "research.txt" + research.write_text("candidate 3 was rejected by the original synthetic check\n") + source_goal, source_agent = "source-research", "source-researcher" + destination_goal, destination_agent = "branch-research", "branch-researcher" + def register(name, agent): + registry = tmp_path / name / "registry.json" + runtime = tmp_path / name / "runtime" + prefix = ["--format", "json", "--registry", str(registry), "--runtime-root", str(runtime)] + assert main([*prefix, "bootstrap", "--project", str(workspace), "--goal-id", name, + "--objective", "Continue the synthetic candidate investigation.", "--adapter-kind", "read_only_project_map_v0", + "--adapter-status", "connected-read-only", "--write-scope", "**", "--no-global-sync"]) == 0 + capsys.readouterr() + assert main([*prefix, "configure-goal", "--goal-id", name, "--registered-agent", agent, + "--agent-work-mode", agent + "=active", "--execute"]) == 0 + capsys.readouterr() + assert main([*prefix, "todo", "add", "--goal-id", name, "--role", "agent", "--claimed-by", agent, + "--task-class", "advancement_task", "--required-capability", "shell", + "--text", "Verify the retained candidate and preserve applicable negative evidence."]) == 0 + added = json.loads(capsys.readouterr().out) + return registry, runtime, added["todo_id"] + source_registry, source_runtime, source_todo = register(source_goal, source_agent) + registry, runtime, destination_todo = register(destination_goal, destination_agent) + prefix = ["--format", "json", "--registry", str(registry), "--runtime-root", str(runtime)] + turn_id = "checkpoint-continuation-fixture" + binding = ["--goal-id", destination_goal, "--agent-id", destination_agent, + "--todo-id", destination_todo, "--turn-instance-id", turn_id] + assert main([*prefix, "quota", "should-run", "--runtime-profile", "generic_cli", *binding]) == 0 + guard = json.loads(capsys.readouterr().out) + assert guard["interaction_contract"]["agent_channel"]["delivery_allowed"] is True + from benchmark.runtime.research_checkpoint import canonical_context, research_context, adopt_todo_context, _encoded + source_prefix = ["--format", "json", "--registry", str(source_registry), "--runtime-root", str(source_runtime)] + def invoke(arguments): + code = main(arguments) + output = capsys.readouterr().out + assert code == 0, output + return json.loads(output) + source_binding = ["--goal-id", source_goal, "--todo-id", source_todo] + invoke([*source_prefix, "task-lease", "acquire", *source_binding, + "--owner", source_agent, "--idempotency-key", "source-context"]) + source_revision = invoke([*source_prefix, "todo", "list", *source_binding])["authority_read"]["provider_revision"] + invoke([*source_prefix, "todo", "update", *source_binding, "--agent-id", source_agent, + "--note", "Candidate 3 failed the synthetic check; preserve candidate 7.", + "--update-operation-id", "source-negative-evidence", "--update-expected-provider-revision", source_revision, + "--task-lease-idempotency-key", "source-context", "--task-lease-expected-version", "1"]) + requests = [] + retained_context_path = None + joint_command = "" + if case == "joint": + import copy + import shlex + import requests as http + from benchmark.edgebench import checkpoint_controller as transport + from benchmark.tests.test_checkpoint_controller import register as register_judge, submit, host_phase_command + source_judge, branch_judge, source_url, branch_url, controllers_files, grades = request.getfixturevalue("controllers") + source_token = register_judge(source_judge, source_url, max_agent_submissions=3, submission_cooldown=0) + assert submit(source_url, source_token, 7).status_code == 200 + assert submit(source_url, source_token, 3).status_code == 200 + source_judge.state.judge.tokens[source_token]["submission_cooldown"] = 3 + source_phase = tmp_path / "source-phase" + source_phase.write_text(f"{int(time.time()) + 7200}\n") + evaluator_unit = tmp_path / "evaluator-point" + evaluator_point = transport.capture(controllers_files[0], token=source_token, + capture_id="e" * 32, destination=evaluator_unit, phase_deadline_file=source_phase) + evaluator_body = json.loads((evaluator_unit / "checkpoint.json").read_bytes()) + # Real participants are quiesced by this fixture. It is not a production + # cross-owner capture barrier or a proof of autonomous research utility. + + def response(request, index, message): + requests.append(request) + if index == 2: + return {"type": "function_call", "id": "fc_retained", "call_id": "call_retained", + "name": "exec_command", "arguments": json.dumps({ + "cmd": "cat candidate.txt research.txt " + retained_context_path + joint_command, "max_output_tokens": 6000})} + if index == 3: + message["content"][0]["text"] = "Retained candidate 7 verified; candidate 3 remains rejected." + return message + + binary = os.environ["LOOPX_NATIVE_CHECKPOINT_BIN"] + execution = Execution(context="resume", sandbox="workspace-write" if case == "joint" else "read-only") + with real_native_source(tmp_path / "native", binary, workspace, close_source=True, + response_builder=response, sandbox=execution.sandbox) as (env, source_session, _, calls): + env.update(LOOPX_REGISTRY=str(source_registry), LOOPX_RUNTIME_ROOT=str(source_runtime), + LOOPX_GOAL_ID=source_goal, LOOPX_AGENT_ID=source_agent, + MODEL_NAME="fixture", REASONING_EFFORT="high", LOOPX_CODEX_SANDBOX=execution.sandbox) + source_wake = BenchmarkSessionWake(env, execution, {}) + source_wake.admission.accept_result(lambda: _store_codex_cli_session(source_runtime, + lineage={"goal_id": source_goal, "agent_id": source_agent}, session_scope="agent", + session_id=source_session, goal_ref=source_wake.goal_ref, session_profile_digest=source_wake.digest)) + native = native_session_checkpoint(env, source_session, timeout=20) + config = tmp_path / "config.toml" + config.write_bytes((Path(env["CODEX_HOME"]) / "config.toml").read_bytes()) + from loopx.capabilities.explore.result_log import ( + append_explore_result_event, build_explore_node_event, explore_result_log_path, + ) + graph_path = explore_result_log_path(source_runtime, source_goal) + append_explore_result_event(graph_path, build_explore_node_event( + goal_id=source_goal, node_id="rejected-route", title="Rejected route", status="dead_end", + summary="Route rejected because fresh samples reversed its pilot gain.")) + source_vision = tmp_path / "source-vision.json" + source_vision.write_text(json.dumps({"schema_version": "goal_vision_replan_contract_v0", + "state": "vision_patch_proposed", "vision_patch": {"acceptance_summary": + "Complete a fresh-sample experiment before changing the retained route."}})) + invoke([*source_prefix, "quota", "should-run", *source_binding, "--agent-id", source_agent, + "--turn-instance-id", "source-research-evidence", "--runtime-profile", "generic_cli"]) + invoke([*source_prefix, "refresh-state", *source_binding, "--agent-id", source_agent, + "--turn-instance-id", "source-research-evidence", "--classification", "synthetic_research_evidence", + "--delivery-batch-scale", "multi_surface", "--delivery-outcome", "outcome_progress", + "--agent-vision-json", str(source_vision)]) + source_context = research_context(env, deadline=time.time() + 30) + context_raw = _encoded(source_context) + context_export = tmp_path / "source-context.json" + context_export.write_bytes(context_raw) + components = [*native["components"], {"id": "code", "source_path": str(candidate)}, + {"id": "research", "source_path": str(research)}, + {"id": "research_context", "source_path": str(context_export)}] + if case == "joint": + components.append({"id": "evaluation", "source_path": str(evaluator_unit)}) + probe = effect_runtime_result("runtime.checkpoint_artifacts", {"action": "probe", "components": components}) + unit = tmp_path / "point" + captured = effect_runtime_result("runtime.checkpoint_artifacts", {"action": "capture", + "destination": str(unit), "components": components, "expected_source_sha256": probe["source_sha256"], + "point_metadata": {"native_session": native["metadata"], + "session": {"session_id": source_session, "session_profile_digest": source_wake.digest}, + "loopx_state": {"goal_id": source_goal, "agent_id": source_agent}, + "research_context": {"component_id": "research_context", "member_path": "", + "sha256": hashlib.sha256(context_raw).hexdigest(), + "provider_revision": source_context["snapshot"]["provider_revision"], + "store_identity": source_context["snapshot"]["store_identity"]}}}, retry_safe=False) + digest = captured["sha256"] + if case == "joint": + source_judge.state.judge.tokens[source_token]["submission_cooldown"] = 0 + assert submit(source_url, source_token, 99).status_code == 200 + judge_before = copy.deepcopy((source_judge.state.judge.tokens, + source_judge.state.judge.submissions, source_judge.state.judge.run_history)) + invoke([*source_prefix, "todo", "update", *source_binding, "--agent-id", source_agent, + "--note", "Candidate 999 is later evidence outside the captured point.", + "--update-operation-id", "later-negative-evidence", "--update-expected-provider-revision", + source_context["snapshot"]["provider_revision"], "--task-lease-idempotency-key", "source-context", + "--task-lease-expected-version", "1"]) + append_explore_result_event(graph_path, build_explore_node_event( + goal_id=source_goal, node_id="future-route", title="Future route outside this checkpoint", + status="exploring")) + later_context = canonical_context(env, deadline=time.time() + 30) + # Qualify use of captured inputs, not the source's later working files. + # A quiesced same-machine fixture swaps fixed paths; this is not evidence + # of complete guest/environment isolation or a scientific matched trial. + candidate.write_text("later candidate: 999\n") + research.write_text("later evidence unavailable at the captured point\n") + late_source = tmp_path / "retained-later-source" + late_source.mkdir() + shutil.move(candidate, late_source / candidate.name) + shutil.move(research, late_source / research.name) + materialized = tmp_path / "materialized" + effect_runtime_result("runtime.checkpoint_artifacts", {"action": "materialize", "root": str(unit), + "expected_sha256": digest, "destination": str(materialized)}, retry_safe=False) + shutil.copy2(materialized / "components/code", candidate) + shutil.copy2(materialized / "components/research", research) + original_home = tmp_path / "original-home" + shutil.move(env["CODEX_HOME"], original_home) + retained_names = {m["name"] for m in native["metadata"]["store_members"]} | {"config.toml"} + def retained_source(): + return {p.relative_to(original_home).as_posix(): p.read_bytes() + for p in original_home.rglob("*") if p.is_file() and p.relative_to(original_home).parts[0] in retained_names} + before = retained_source() + env.update(LOOPX_REGISTRY=str(registry), LOOPX_RUNTIME_ROOT=str(runtime), + LOOPX_GOAL_ID=destination_goal, LOOPX_AGENT_ID=destination_agent, + MODEL_NAME="fixture", REASONING_EFFORT="high") + lifecycle_registry = tmp_path / "lifecycle-only" / "registry.json" + assert main(["--format", "json", "--registry", str(lifecycle_registry), + "--runtime-root", str(tmp_path / "lifecycle-only/runtime"), "project", "register", + "--goal-instance-profile", "source_session_v1", "--operation-id", "lifecycle-only", + "--project-id", "lifecycle-only", "--project-kind", "work", "--knowledge-root", str(workspace), + "--goal-id", "lifecycle-only", "--objective", "Synthetic lifecycle fixture.", + "--acceptance", "Lifecycle only.", "--next-effect", "Read.", "--stop-condition", "Stop."]) == 0 + capsys.readouterr() + with pytest.raises(ProjectRegistryProtocolError, match="lifecycle-only"): + admit_native_checkpoint_session({**env, "LOOPX_REGISTRY": str(lifecycle_registry), + "LOOPX_GOAL_ID": "lifecycle-only"}, unit, digest, config, execution=execution, timeout=30) + assert not Path(env["CODEX_HOME"]).exists() and len(calls) == 1 + unsupported_config = tmp_path / "automatic-goal-config.toml" + unsupported_config.write_text(config.read_text().replace("goals=false", "goals=true")) + with monkeypatch.context() as automatic: + def forbidden_automatic_fork(*args, **kwargs): + raise AssertionError("an unsupported automatic Goal profile must fail before native reconstruction") + automatic.setattr("benchmark.runtime.native_checkpoint.fork_native_checkpoint", forbidden_automatic_fork) + with pytest.raises(ValueError, match="automatic Goals explicitly disabled"): + admit_native_checkpoint_session(env, unit, digest, unsupported_config, execution=execution, timeout=30) + with pytest.raises(ValueError, match="distinct destination"): + admit_native_checkpoint_session({**env, "LOOPX_GOAL_ID": source_goal}, unit, digest, + config, execution=execution, timeout=30) + assert not Path(env["CODEX_HOME"]).exists() and len(calls) == 1 + if case == "scope_change": + from benchmark.runtime import native_checkpoint + original_fork = native_checkpoint.fork_native_checkpoint + def change_scope_after_real_fork(*args, **kwargs): + result = original_fork(*args, **kwargs) + assert main([*prefix, "configure-goal", "--goal-id", destination_goal, + "--agent-work-mode", destination_agent + "=monitor_only", "--execute"]) == 0 + capsys.readouterr() + return result + monkeypatch.setattr(native_checkpoint, "fork_native_checkpoint", change_scope_after_real_fork) + with pytest.raises(ValueError, match="Goal configuration changed"): + admit_native_checkpoint_session(env, unit, digest, config, execution=execution, timeout=30) + assert load_codex_cli_session(runtime, lineage={"goal_id": destination_goal, + "agent_id": destination_agent}, session_scope="agent") is None + assert len(calls) == 1 # A native fork occurred; no binding or model did. + assert retained_source() == before + return + if case == "profile_change": + with pytest.raises(ValueError, match="profile changed"): + admit_native_checkpoint_session({**env, "REASONING_EFFORT": "low"}, unit, digest, + config, execution=execution, timeout=30) + assert load_codex_cli_session(runtime, lineage={"goal_id": destination_goal, + "agent_id": destination_agent}, session_scope="agent") is None + assert len(calls) == 1 + assert retained_source() == before + return + recovered_fork = None + if case in {"fork_reply_loss", "fork_missing", "fork_ambiguous"}: + from benchmark.runtime.native_checkpoint import StdioNativeGoalTransport + original_request = StdioNativeGoalTransport.request + forks = [] + def lose_fork_reply(self, method, params): + if method != "thread/fork": + return original_request(self, method, params) + if case != "fork_missing": + forks.append(original_request(self, method, params)["thread"]["id"]) + if case == "fork_ambiguous": + forks.append(original_request(self, method, params)["thread"]["id"]) + raise TimeoutError("synthetic loss before fork reply") + with monkeypatch.context() as lost: + lost.setattr(StdioNativeGoalTransport, "request", lose_fork_reply) + with pytest.raises(TimeoutError, match="loss before fork reply"): + admit_native_checkpoint_session(env, unit, digest, config, execution=execution, timeout=30) + assert load_codex_cli_session(runtime, lineage={"goal_id": destination_goal, + "agent_id": destination_agent}, session_scope="agent") is None + assert len(calls) == 1 and retained_source() == before + if case != "fork_reply_loss": + def no_second_fork(self, method, params): + assert method != "thread/fork", "uncertainty must not dispatch a second fork" + return original_request(self, method, params) + with monkeypatch.context() as retry: + retry.setattr(StdioNativeGoalTransport, "request", no_second_fork) + with pytest.raises(ValueError, match="absent or ambiguous"): + admit_native_checkpoint_session(env, unit, digest, config, execution=execution, timeout=30) + assert load_codex_cli_session(runtime, lineage={"goal_id": destination_goal, + "agent_id": destination_agent}, session_scope="agent") is None + assert len(calls) == 1 and retained_source() == before + return + recovered_fork = forks[0] + # The isolated attempt is bound to this destination, not merely an + # equivalent visible history or another Goal's transport directory. + from benchmark.runtime.native_checkpoint import fork_native_checkpoint + with pytest.raises(ValueError, match="attempt differs"): + fork_native_checkpoint({**env, "LOOPX_AGENT_ID": "unrelated-destination"}, + unit, digest, config, timeout=30) + if case == "binding_reply_loss": + original_bind = BenchmarkSessionWake._bind_verified_fork + def lose_binding_reply(self, *args): + original_bind(self, *args) + raise TimeoutError("synthetic loss after atomic binding") + with monkeypatch.context() as lost: + lost.setattr(BenchmarkSessionWake, "_bind_verified_fork", lose_binding_reply) + with pytest.raises(TimeoutError, match="loss after atomic binding"): + admit_native_checkpoint_session(env, unit, digest, config, execution=execution, timeout=30) + assert load_codex_cli_session(runtime, lineage={"goal_id": destination_goal, + "agent_id": destination_agent}, session_scope="agent")["checkpoint_artifact_sha256"] == digest + assert len(calls) == 1 + admitted_cli = subprocess.run([sys.executable, "-m", "benchmark.runtime.native_checkpoint", + "--root", str(unit), "--expected-sha256", digest, "--config-path", str(config), "--sandbox", execution.sandbox], + env=env, text=True, capture_output=True, timeout=40) + assert admitted_cli.returncode == 0, admitted_cli.stderr + admitted = json.loads(admitted_cli.stdout) + assert admitted["session_binding_admitted"] and admitted["history_verified"] + assert admitted["model_calls"] == 0 and not admitted["continuation_qualified"] + assert admitted["session_id"] != source_session and len(calls) == 1 + if recovered_fork: + assert admitted["session_id"] == recovered_fork + bound = load_codex_cli_session(runtime, lineage={"goal_id": destination_goal, "agent_id": destination_agent}, + session_scope="agent") + assert bound["session_id"] == admitted["session_id"] + assert bound.get("goal_ref") == admitted["destination_goal_ref"] + with monkeypatch.context() as replay: + def forbidden_fork(*args, **kwargs): + raise AssertionError("a repeated admission must not create a second native fork") + replay.setattr("benchmark.runtime.native_checkpoint.fork_native_checkpoint", forbidden_fork) + recovered = admit_native_checkpoint_session(env, unit, digest, config, execution=execution, timeout=30) + assert recovered == admitted + assert bound["checkpoint_artifact_sha256"] == digest + # Transport reply loss leaves the atomic source-bound binding as the + # original receipt. An exact CLI retry must also recover it. + retry_cli = subprocess.run(admitted_cli.args, env=env, text=True, capture_output=True, timeout=40) + assert retry_cli.returncode == 0, retry_cli.stderr + assert json.loads(retry_cli.stdout) == admitted + with pytest.raises(ValueError, match="profile changed"): + admit_native_checkpoint_session({**env, "REASONING_EFFORT": "low"}, unit, digest, + config, execution=execution, timeout=30) + assert len(calls) == 1 and load_codex_cli_session(runtime, lineage={ + "goal_id": destination_goal, "agent_id": destination_agent}, session_scope="agent") == bound + if case == "resume": + wrong_config = tmp_path / "changed-source-config.toml" + wrong_config.write_text(config.read_text() + "\n# different source configuration\n") + with pytest.raises(ValueError, match="effective configuration differs"): + admit_native_checkpoint_session(env, unit, digest, wrong_config, execution=execution, timeout=30) + from loopx.control_plane.turn_driver.codex_sessions import _session_path + binding_path = _session_path(runtime, {"goal_id": destination_goal, "agent_id": destination_agent}, "agent") + original_binding = binding_path.read_bytes() + # Isolated fixture corruption: neither an unrelated binding nor a + # binding for another checkpoint may be relabeled as this admission. + for wrong_source in (None, "f" * 64): + changed = dict(bound) + if wrong_source is None: + changed.pop("checkpoint_artifact_sha256") + else: + changed["checkpoint_artifact_sha256"] = wrong_source + binding_path.write_text(json.dumps(changed)) + with pytest.raises(ValueError, match="without this checkpoint admission"): + admit_native_checkpoint_session(env, unit, digest, config, execution=execution, timeout=30) + assert json.loads(binding_path.read_text()) == changed and len(calls) == 1 + binding_path.write_bytes(original_binding) + + # Both participants meet at the real destination continuation owner. + # No source claim/lease is replayed; adoption alone still starts no work. + lease = invoke([*prefix, "task-lease", "acquire", "--goal-id", destination_goal, + "--todo-id", destination_todo, "--owner", destination_agent, "--idempotency-key", "branch-context"]) + current = invoke([*prefix, "todo", "list", "--goal-id", destination_goal, "--todo-id", destination_todo]) + context_command = [sys.executable, "-m", "benchmark.runtime.research_checkpoint", + "--root", str(unit), "--expected-sha256", digest, "--source-todo-id", source_todo, + "--todo-id", destination_todo, "--operation-id", "branch-context", + "--expected-revision", current["authority_read"]["provider_revision"], + "--lease-key", "branch-context", "--lease-version", str(lease["lease"]["version"])] + adopted_cli = subprocess.run(context_command, env=env, text=True, capture_output=True, timeout=60) + assert adopted_cli.returncode == 0, adopted_cli.stderr + context_result = json.loads(adopted_cli.stdout) + # Retrying the exact CLI after a lost result reuses both original + # receipts; it must not fork, charge, edit the Todo or run a model. + replay_cli = subprocess.run(context_command, env=env, text=True, capture_output=True, timeout=60) + assert replay_cli.returncode == 0, replay_cli.stderr + replay_result = json.loads(replay_cli.stdout) + assert replay_result["continuation"]["adoption"]["status"] == "replayed" + assert replay_result["continuation"]["provider_revision"] == context_result["continuation"]["provider_revision"] + assert len(calls) == 1 + retained_context_path = str(Path(context_result["artifact"]).relative_to(workspace)) + assert Path(context_result["artifact"]).read_bytes() == context_raw and len(calls) == 1 + adopted_todo = invoke([*prefix, "todo", "list", "--goal-id", destination_goal, "--todo-id", destination_todo]) + assert adopted_todo["todo"]["claimed_by"] == destination_agent + assert json.loads(adopted_todo["todo"]["note"])["source_session"] == source_session + guard = invoke([*prefix, "quota", "should-run", "--runtime-profile", "generic_cli", *binding]) + assert guard["interaction_contract"]["agent_channel"]["delivery_allowed"] is True + if case == "joint": + control = tmp_path / "host-control" + control.mkdir() + branch_phase = control / "phase-deadline" + # Restore only the nested capsule from the same verified artifact, + # never the later source service or an unrelated caller file. + fresh = transport.restore(controllers_files[1], root=materialized / "components/evaluation", + expected_sha256=evaluator_point["sha256"], run_id="joint-branch", phase_deadline_file=branch_phase) + assert submit(branch_url, fresh["token"], 8).status_code == 429 + assert len(grades) == 3 # No implicit regrade on restore or cooldown rejection. + shell = host_phase_command(tmp_path, monkeypatch).replace( + "/opt/loopx-benchmark/control", str(control)) + budget = subprocess.run(["/bin/bash", "-c", shell + 'printf "%s" "$remaining"'], + text=True, capture_output=True, check=True) + assert 7100 < int(budget.stdout) <= 7200 + assert int(branch_phase.read_bytes()) <= time.time() + evaluator_body["point_metadata"]["host_phase_budget"]["remaining_seconds"] + # Explicit fresh token only, never controller/admin credentials. + # The native resumed tool reads public history, uses captured code, + # submits one new candidate, and observes the preserved round cap. + solver_input = workspace / "branch-judge.json" + solver_input.write_text(json.dumps({"url": branch_url, "token": fresh["token"]})) + solver_input.chmod(0o600) + solver = workspace / "continue_synthetic.py" + solver.write_text("""import json, time +from pathlib import Path +import requests +binding = json.loads(Path('branch-judge.json').read_text()) +url, token = binding['url'], binding['token'] +history = requests.get(url + '/api/v1/history', params={'token': token}, timeout=5).json() +assert history['best_score'] == 7 and history['agent_submissions'] == 2 +value = float(Path('candidate.txt').read_text().split(':')[1]) + 1 +response = requests.post(url + '/api/v1/submit', data={'token': token}, + files={'archive': ('candidate.tar.gz', str(value).encode())}, timeout=5) +response.raise_for_status() +identifier = response.json()['submission_id'] +for _ in range(100): + result = requests.get(url + '/api/v1/result/' + identifier, timeout=5).json() + if result['status'] == 'completed': break + time.sleep(.01) +assert result['status'] == 'completed' +exhausted = requests.post(url + '/api/v1/submit', data={'token': token}, + files={'archive': ('candidate.tar.gz', b'999')}, timeout=5) +assert exhausted.status_code == 429 +print('Captured public best 7; continued candidate 8; consumed round agent-3; no extra opportunity.') +""") + # Wait only the inherited short synthetic cooldown, never create an + # opportunity by changing source/destination counters or timestamps. + time.sleep(evaluator_body["point_metadata"]["native_reference"]["opportunity"]["remaining_cooldown_seconds"] + .05) + joint_command = " && " + shlex.quote(sys.executable) + " continue_synthetic.py" + receipt = {} + wake = BenchmarkSessionWake(env, execution, receipt) + assert wake.session_id == admitted["session_id"] and receipt["session"]["action"] == "resume" + logs = tmp_path / "continuation" + logs.mkdir() + command = native_command(env, execution, "execute", logs, wake) + if case == "joint": + command = ["/bin/bash", "-c", shell + "exec " + shlex.join(command)] + current_body = json.dumps(guard["interaction_contract"], ensure_ascii=False) + result = subprocess.run(command, env=env, cwd=workspace, input=current_body + "\nContinue: verify the retained files.", + text=True, capture_output=True, timeout=40) + assert result.returncode == 0, result.stderr + (logs / "stdout.jsonl").write_text(result.stdout) + wake.observe(logs / "stdout.jsonl") + assert wake.session_id == admitted["session_id"] and len(calls) == 3 + assert "Synthetic source result." in json.dumps(requests[1]) + assert destination_goal in json.dumps(requests[1]) and destination_todo in json.dumps(requests[1]) + assert "retained candidate: 7" in json.dumps(requests[2]) + assert "candidate 3 was rejected" in json.dumps(requests[2]) + assert "Candidate 3 failed the synthetic check" in json.dumps(requests[2]) + assert "fresh samples reversed its pilot gain" in json.dumps(requests[2]) + assert "Complete a fresh-sample experiment" in json.dumps(requests[2]) + assert source_context["research_evidence"]["explore_events"][0]["status"] == "dead_end" + assert not explore_result_log_path(runtime, destination_goal).exists() + assert "Candidate 999" not in json.dumps(requests[2]) + assert "Future route outside this checkpoint" not in json.dumps(requests[2]) + assert canonical_context({**env, "LOOPX_GOAL_ID": source_goal, + "LOOPX_RUNTIME_ROOT": str(source_runtime)}, deadline=time.time() + 30) == later_context + if case == "joint": + assert "Captured public best 7; continued candidate 8" in json.dumps(requests[2]) + history = http.get(branch_url + "/api/v1/history", params={"token": fresh["token"]}, timeout=5).json() + assert history["best_score"] == 8 and history["best_round"] == "agent-3" + assert history["agent_submissions"] == 3 and len(grades) == 4 + assert (source_judge.state.judge.tokens, source_judge.state.judge.submissions, + source_judge.state.judge.run_history) == judge_before + for secret in (source_token, fresh["token"], source_judge.state.judge.admin_secret, + branch_judge.state.judge.admin_secret): + assert secret not in json.dumps(requests) + assert "Retained candidate 7 verified" in result.stdout + retained = native_session_checkpoint(env, admitted["session_id"], timeout=20) + assert retained["metadata"]["turn_ids"][:-1] == native["metadata"]["turn_ids"] + assert retained["metadata"]["completed_turn_id"] != native["metadata"]["completed_turn_id"] + after_work = load_codex_cli_session(runtime, lineage={"goal_id": destination_goal, + "agent_id": destination_agent}, session_scope="agent") + assert after_work["checkpoint_artifact_sha256"] == digest + with pytest.raises(ValueError, match="advanced or differs"): + admit_native_checkpoint_session(env, unit, digest, config, execution=execution, timeout=30) + assert load_codex_cli_session(runtime, lineage={"goal_id": destination_goal, + "agent_id": destination_agent}, session_scope="agent") == after_work + with pytest.raises(ValueError, match="native history or configuration moved"): + adopt_todo_context(env, unit, digest, source_todo_id=source_todo, + todo_id=destination_todo, operation_id="must-not-adopt-later-session", + expected_revision=adopted_todo["authority_read"]["provider_revision"], + lease_key="branch-context", lease_version=lease["lease"]["version"]) + assert len(calls) == 3 + assert invoke([*prefix, "todo", "list", "--goal-id", destination_goal, + "--todo-id", destination_todo])["todo"] == adopted_todo["todo"] + assert candidate.read_text() == "retained candidate: 7\n" + assert (late_source / candidate.name).read_text() == "later candidate: 999\n" + assert (late_source / research.name).read_text() == "later evidence unavailable at the captured point\n" + assert retained_source() == before + vision = tmp_path / "continuation-vision.json" + vision.write_text(json.dumps({"schema_version": "goal_vision_replan_contract_v0", "state": "vision_patch_proposed", + "vision_patch": {"acceptance_summary": "Synthetic conversation continuation verified; full research remains open."}, + "path_delta": {"outcome": "continue", "prior_assumption": "Retained candidate and evidence are available.", + "observed_reality": "The resumed native tool read verifies candidate 7 and the rejection of candidate 3.", + "retained": ["Complete checkpoint qualification"], "evidence_refs": ["evidence:native-continuation-fixture"]}})) + assert main([*prefix, "refresh-state", *binding, "--classification", "checkpoint_continuation_verified", + "--delivery-batch-scale", "multi_surface", "--delivery-outcome", "outcome_progress", + "--agent-vision-json", str(vision)]) == 0 + assert json.loads(capsys.readouterr().out)["ok"] is True + assert main([*prefix, "quota", "spend-slot", *binding, "--source", "heartbeat", "--execute"]) == 0 + assert json.loads(capsys.readouterr().out)["ok"] is True + settled = read_heartbeat_settlement(runtime, goal_id=destination_goal, agent_id=destination_agent, + todo_id=destination_todo, turn_instance_id=turn_id, registry_path=registry) + assert settled.progress["state"] == "settled" + + +def test_interrupted_cursor_requires_exact_unreleased_final_guard(tmp_path, monkeypatch): + """A stopped command is eligible only at the declared pre-output boundary.""" + import copy + import shlex + from benchmark.runtime.native_checkpoint import _verify_interrupted_guard + + guard = {"cli": "/fixture/loopx", "workspace": str(tmp_path), "registry": "/fixture/registry.json", + "runtime_root": "/fixture/runtime", "goal_id": "research", "agent_id": "worker", + "turn_instance_id": "turn-1", "capture_root": "/fixture/decisions", "socket_path": "/tmp/fixture/socket", + "decision_sha256": "a" * 64} + argv = [guard["cli"], "--registry", guard["registry"], "--runtime-root", guard["runtime_root"], + "--format", "json", "quota", "should-run", "--goal-id", "research", "--agent-id", "worker", + "--turn-instance-id", "turn-1", "--turn-envelope", "--decision-output-root", guard["capture_root"], + "--decision-handoff-socket", guard["socket_path"]] + command = shlex.join(["/bin/bash", "-lc", shlex.join(argv)]) + item = {"type": "commandExecution", "id": "guard-call", "command": command, "cwd": str(tmp_path), + "status": "failed", "exitCode": 137, "aggregatedOutput": None} + turns = [{"id": "retained-compaction", "status": "completed", "items": [{"type": "contextCompaction"}]}, + {"id": "pending", "status": "interrupted", "items": [item]}] + env, calls = fixture(tmp_path, monkeypatch, [{"data": turns[:1], "nextCursor": "next"}, + {"data": turns[1:], "nextCursor": None}]) + captured = native_session_checkpoint(env, "fixture-session", timeout=10, interrupted_guard=guard) + assert captured["metadata"]["turn_ids"] == ["retained-compaction", "pending"] + assert "completed_turn_id" not in captured["metadata"] + assert captured["metadata"]["interrupted_guard"]["native_call_id"] == "guard-call" + assert "thread/fork" not in calls + + rendered = copy.deepcopy(turns) + rendered[-1]["items"][-1]["command"] = shlex.join(["/bin/bash", "-lc", + "LOOPX_TURN=turn-1\n" + shlex.join(argv).replace("--turn-instance-id turn-1", '--turn-instance-id "${LOOPX_TURN:?}"')]) + _verify_interrupted_guard(rendered, guard) + variants = [] + for field, value in [("aggregatedOutput", "score already exposed"), ("cwd", "/elsewhere"), + ("status", "completed"), ("command", shlex.join(["/bin/bash", "-lc", shlex.join(argv) + "; touch later"]))]: + changed = copy.deepcopy(turns) + changed[-1]["items"][-1][field] = value + variants.append(changed) + for option, value in [("--turn-instance-id", "wrong-turn"), ("--decision-handoff-socket", "/tmp/other"), + ("--registry", "/other.json"), ("--goal-id", "other")]: + altered = list(argv) + altered[altered.index(option) + 1] = value + changed = copy.deepcopy(turns) + changed[-1]["items"][-1]["command"] = shlex.join(["/bin/bash", "-lc", shlex.join(altered)]) + variants.append(changed) + changed = copy.deepcopy(turns) + changed[-1]["items"].append({"type": "reasoning", "text": "later decision"}) + variants.append(changed) + changed = copy.deepcopy(turns) + changed[-1]["status"] = "completed" + variants.append(changed) + changed = copy.deepcopy(turns) + changed[-1]["items"].insert(0, copy.deepcopy(item)) + variants.append(changed) + for changed in variants: + with pytest.raises(ValueError): + _verify_interrupted_guard(changed, guard) + with pytest.raises(ValueError, match="call identity"): + _verify_interrupted_guard(turns, {**guard, "native_call_id": "different"}) diff --git a/benchmark/tests/test_native_checkpoint_handoff.py b/benchmark/tests/test_native_checkpoint_handoff.py new file mode 100644 index 0000000000..529417f404 --- /dev/null +++ b/benchmark/tests/test_native_checkpoint_handoff.py @@ -0,0 +1,749 @@ +"""Characterize live native forks before choosing a midwake capture transport. + +A local deterministic provider replaces inference; the actual Host owns fork, +history and tool execution. This does not qualify a joint research checkpoint. +""" +import json +import os +import shlex +import subprocess +import sys +import time + +import pytest + +from benchmark.runtime.native_checkpoint import native_session_checkpoint +from benchmark.tests.test_native_checkpoint import real_native_source +from loopx.capabilities.benchmark_toolkit.native_codex_goal import ( + NativeGoalProtocolError, StdioNativeGoalTransport, +) + + +@pytest.mark.skipif(not os.environ.get("LOOPX_NATIVE_CHECKPOINT_BIN"), + reason="Requires an explicit real native Host; no model service used") +def test_live_fork_preserves_context_without_quiescing_source(tmp_path): + source_work, branch_work = tmp_path / "source", tmp_path / "branch" + source_work.mkdir() + branch_work.mkdir() + requests = [] + # A bounded tool withholds its output until the test Host releases it. + # This models transport ordering only, not a LoopX guard or writer barrier. + code = """from pathlib import Path +import time +Path('waiting').write_text('ready') +deadline = time.monotonic() + 20 +while not Path('release').exists() and time.monotonic() < deadline: + time.sleep(.01) +assert Path('release').exists(), 'fixture release missing' +Path('source-result').write_text('one original execution') +print('WITHHELD_RESULT') +""" + + def response(request, index, message): + requests.append(request) + if index == 2: + return {"type": "function_call", "id": "fc_boundary", "call_id": "call_boundary", + "name": "exec_command", "arguments": json.dumps({ + "cmd": "python3 -c " + shlex.quote(code), + "yield_time_ms": 10000, "max_output_tokens": 100})} + message["content"][0]["text"] = "RETAINED_RESEARCH" if index == 1 else "Continued." + return message + + def history(host, session): + return host.request("thread/turns/list", {"threadId": session, + "sortDirection": "asc", "itemsView": "full", "limit": 100})["data"] + + def terminal(host, session, turn): + deadline = time.monotonic() + 10 + while time.monotonic() < deadline: + rows = history(host, session) + if rows and rows[-1]["id"] == turn and rows[-1]["status"] == "completed": + return rows + time.sleep(.02) + raise AssertionError("Native fixture did not complete") + + binary = os.environ["LOOPX_NATIVE_CHECKPOINT_BIN"] + with real_native_source(tmp_path / "native", binary, source_work, close_source=True, + response_builder=response, sandbox="workspace-write") as (env, source, _, calls): + with StdioNativeGoalTransport.spawn([binary, "app-server", "--listen", "stdio://"], + cwd=str(source_work), env=env, stderr=subprocess.DEVNULL, response_timeout_sec=15) as host: + host.request("initialize", {"clientInfo": {"name": "handoff_fixture", "version": "0.1"}, + "capabilities": {"experimentalApi": True}}) + host.notify("initialized", {}) + host.request("thread/resume", {"threadId": source, "cwd": str(source_work), + "sandbox": "workspace-write", "approvalPolicy": "never", "persistExtendedHistory": True}) + turn = host.request("turn/start", {"threadId": source, "input": [ + {"type": "text", "text": "Reach the synthetic withheld-output boundary."}]})["turn"]["id"] + try: + deadline = time.monotonic() + 10 + while not (source_work / "waiting").exists() and time.monotonic() < deadline: + time.sleep(.01) + assert (source_work / "waiting").exists() and len(calls) == 2 + source_history = history(host, source) + assert source_history[-1]["status"] == "inProgress" + with pytest.raises(NativeGoalProtocolError, match="thread/fork"): + host.request("thread/fork", {"threadId": source, "cwd": str(branch_work), + "lastTurnId": turn, "persistExtendedHistory": True}) + fork = host.request("thread/fork", {"threadId": source, "cwd": str(branch_work), + "persistExtendedHistory": True})["thread"]["id"] + fork_history = history(host, fork) + assert fork != source and len(calls) == 2 + assert fork_history[:-1] == source_history[:-1] + assert fork_history[-1]["id"] == turn and fork_history[-1]["status"] == "interrupted" + assert history(host, source)[-1]["status"] == "inProgress" + # Native fork success grants neither a complete cursor nor a + # writer barrier. Keep the production capture rejection intact. + with pytest.raises(ValueError, match="completed final cursor"): + native_session_checkpoint({**env, "LOOPX_PROJECT": str(branch_work)}, fork, timeout=15) + branch_turn = host.request("turn/start", {"threadId": fork, "input": [ + {"type": "text", "text": "Fresh fixture admission: retain context; do not replay the source tool."}]})["turn"]["id"] + terminal(host, fork, branch_turn) + assert len(calls) == 3 + items = requests[2]["input"] + assert "RETAINED_RESEARCH" in json.dumps(items) + assert any(item.get("type") == "function_call_output" and + item.get("call_id") == "call_boundary" and item.get("output") == "aborted" + for item in items) + assert history(host, source)[-1]["status"] == "inProgress" + assert not (source_work / "source-result").exists() + assert not (branch_work / "waiting").exists() # No automatic tool replay. + finally: + (source_work / "release").write_text("release original tool") + terminal(host, source, turn) + assert (source_work / "source-result").read_text() == "one original execution" + assert not (branch_work / "source-result").exists() + assert len(calls) == 4 and "WITHHELD_RESULT" in json.dumps(requests[3]) + + +@pytest.mark.skipif(not os.environ.get("LOOPX_NATIVE_CHECKPOINT_BIN") or os.name != "posix", + reason="Requires a real native Host and local Unix transport") +def test_real_quota_output_is_held_across_native_fork(tmp_path): + """The real guard executes once; neither branch observes it before release.""" + import hashlib + import socket + import sys + import tempfile + from pathlib import Path + + from benchmark.runtime.checkpoint import held_quota_decision + from tests.control_plane.test_quota_settlement_cli import ( + AGENT_ID, GOAL_ID, TURN_ID, _write_fixture, _configure_autonomous_replan_fixture, + _heartbeat_receipt_count, + ) + + project, runtime, registry = _write_fixture(tmp_path / "authority") + _configure_autonomous_replan_fixture(project, runtime, registry) + capture_root = project / "captures" + capture_root.mkdir(mode=0o700) + requests = [] + with tempfile.TemporaryDirectory(prefix="handoff-", dir="/tmp") as short: + socket_path = Path(short).resolve() / "host.sock" + with socket.socket(socket.AF_UNIX, socket.SOCK_STREAM) as listener: + listener.bind(str(socket_path)) + listener.listen(1) + listener.settimeout(30) + argv = [sys.executable, "-m", "loopx.cli", "--format", "json", "--registry", str(registry), + "--runtime-root", str(runtime), "quota", "should-run", "--goal-id", GOAL_ID, + "--agent-id", AGENT_ID, "--turn-instance-id", TURN_ID, + "--runtime-profile", "generic_cli", "--available-capability", "shell", + "--available-capability", "filesystem_read", "--decision-output-root", str(capture_root), + "--decision-handoff-socket", str(socket_path)] + command = "PYTHONPATH=" + shlex.quote(str(Path(__file__).resolve().parents[2])) + " " + shlex.join(argv) + + def response(request, index, message): + requests.append(request) + if index == 2: + return {"type": "function_call", "id": "fc_guard", "call_id": "call_guard", + "name": "exec_command", "arguments": json.dumps({ + "cmd": command, "yield_time_ms": 10000, "max_output_tokens": 200})} + message["content"][0]["text"] = "RESEARCH_BEFORE_GUARD" if index == 1 else "Continued." + return message + + binary = os.environ["LOOPX_NATIVE_CHECKPOINT_BIN"] + with real_native_source(tmp_path / "native", binary, project, close_source=True, + response_builder=response, sandbox="danger-full-access") as (env, source, _, calls): + with StdioNativeGoalTransport.spawn([binary, "app-server", "--listen", "stdio://"], + cwd=str(project), env=env, stderr=subprocess.DEVNULL, response_timeout_sec=30) as host: + host.request("initialize", {"clientInfo": {"name": "quota_handoff_fixture", "version": "0.1"}, + "capabilities": {"experimentalApi": True}}) + host.notify("initialized", {}) + host.request("thread/resume", {"threadId": source, "cwd": str(project), + "sandbox": "danger-full-access", "approvalPolicy": "never", "persistExtendedHistory": True}) + turn = host.request("turn/start", {"threadId": source, "input": [ + {"type": "text", "text": "Execute the original synthetic guard."}]})["turn"]["id"] + with held_quota_decision(listener, capture_root, goal_id=GOAL_ID, + agent_id=AGENT_ID, turn_instance_id=TURN_ID) as held: + assert held["quota"]["ok"] is True and len(calls) == 2 + assert held["intent"]["capture"] is True + assert held["sha256"] == hashlib.sha256(held["path"].read_bytes()).hexdigest() + receipts = _heartbeat_receipt_count(runtime, TURN_ID) + assert receipts >= 1 + # Native events, rather than the incomplete history projection, + # witness the actual in-flight tool and its exact command. + pending = None + deadline = time.monotonic() + 5 + while time.monotonic() < deadline: + event = host.next_event(timeout_sec=.1) + if event and event.get("method") == "item/started": + params = event["params"] + item = params.get("item", {}) + if item.get("type") == "commandExecution": + pending = params + break + assert pending and pending["threadId"] == source and pending["turnId"] == turn + assert shlex.split(pending["item"]["command"])[-1] == command + fork = host.request("thread/fork", {"threadId": source, + "cwd": str(project), "persistExtendedHistory": True})["thread"]["id"] + rows = host.request("thread/turns/list", {"threadId": fork, + "sortDirection": "asc", "itemsView": "full", "limit": 100})["data"] + assert rows[-1]["status"] == "interrupted" and rows[-1]["id"] == turn + assert len(calls) == 2 # No post-guard inference while held. + assert _heartbeat_receipt_count(runtime, TURN_ID) == receipts + deadline = time.monotonic() + 10 + while len(calls) < 3 and time.monotonic() < deadline: + time.sleep(.02) + assert len(calls) == 3 and "heartbeat_receipt" in json.dumps(requests[2]) + assert _heartbeat_receipt_count(runtime, TURN_ID) == receipts + assert held["path"].read_bytes() # Original receipt stays recoverable. + + +@pytest.mark.skipif(not os.environ.get("LOOPX_NATIVE_CHECKPOINT_BIN") or os.name != "posix", + reason="Requires a real native Host and local Unix transport") +@pytest.mark.parametrize("outcome", ["resume", "capture_failure", "changed_packet"]) +def test_native_cli_interruption_retains_guard_beyond_tool_yield(tmp_path, outcome): + """A real native CLI stops, then explicitly resumes the one committed guard.""" + import socket + import sys + import tempfile + from pathlib import Path + + from benchmark.runtime.checkpoint import held_quota_decision + from tests.control_plane.test_quota_settlement_cli import ( + AGENT_ID, GOAL_ID, TURN_ID, _write_fixture, _configure_autonomous_replan_fixture, + _heartbeat_receipt_count, + ) + + project, runtime, registry = _write_fixture(tmp_path / "authority") + _configure_autonomous_replan_fixture(project, runtime, registry) + capture_root = project / "captures" + capture_root.mkdir(mode=0o700) + requests = [] + with tempfile.TemporaryDirectory(prefix="handoff-", dir="/tmp") as short: + socket_path = Path(short).resolve() / "host.sock" + with socket.socket(socket.AF_UNIX, socket.SOCK_STREAM) as listener: + listener.bind(str(socket_path)) + listener.listen(1) + listener.settimeout(30) + argv = [sys.executable, "-m", "loopx.cli", "--format", "json", "--registry", str(registry), + "--runtime-root", str(runtime), "quota", "should-run", "--goal-id", GOAL_ID, + "--agent-id", AGENT_ID, "--turn-instance-id", TURN_ID, + "--runtime-profile", "generic_cli", "--available-capability", "shell", + "--available-capability", "filesystem_read", "--decision-output-root", str(capture_root), + "--decision-handoff-socket", str(socket_path)] + command = "PYTHONPATH=" + shlex.quote(str(Path(__file__).resolve().parents[2])) + " " + shlex.join(argv) + + def response(request, index, message): + requests.append(request) + if index == 2: + return {"type": "function_call", "id": "fc_guard", "call_id": "call_guard", + "name": "exec_command", "arguments": json.dumps({ + "cmd": command, "yield_time_ms": 10000, "max_output_tokens": 200})} + message["content"][0]["text"] = "RESEARCH_BEFORE_GUARD" if index == 1 else "Continued." + return message + + binary = os.environ["LOOPX_NATIVE_CHECKPOINT_BIN"] + with real_native_source(tmp_path / "native", binary, project, close_source=True, + response_builder=response, sandbox="danger-full-access") as (env, source, _, calls): + from types import SimpleNamespace + from benchmark.runtime.worker import child_process, native_command + from benchmark.runtime.codex import Execution + env.update(MODEL_NAME="fixture", REASONING_EFFORT="high") + wake = tmp_path / "cli-wake" + wake.mkdir() + cli_command = native_command(env, Execution(context="resume", sandbox="danger-full-access"), + "execute", wake, SimpleNamespace(session_id=source)) + with (wake / "stdout.jsonl").open("w") as output, (wake / "stderr.log").open("w") as error: + with child_process(cli_command, env=env, stdout=output, stderr=error) as process: + process.stdin.write("Execute the original synthetic guard.\n") + process.stdin.flush() + process.stdin.close() + from contextlib import nullcontext + expected = (nullcontext() if outcome == "resume" else + pytest.raises(ValueError, match="capture failed|decision changed")) + with expected: + with held_quota_decision(listener, capture_root, goal_id=GOAL_ID, + agent_id=AGENT_ID, turn_instance_id=TURN_ID, + native_process=process) as held: + assert held["intent"]["capture"] and len(calls) == 2 + assert process.poll() is not None + assert "continuation_body" not in held + receipts = _heartbeat_receipt_count(runtime, TURN_ID) + if outcome == "capture_failure": + raise ValueError("capture failed") + if outcome == "changed_packet": + held["path"].write_text("{}") + else: + time.sleep(12) # Exceeds the real tool's 10-second yield. + assert len(calls) == 2 + assert _heartbeat_receipt_count(runtime, TURN_ID) == receipts + if outcome != "resume": + assert "continuation_body" not in held + assert process.poll() is not None and len(calls) == 2 + assert _heartbeat_receipt_count(runtime, TURN_ID) == receipts + return + original = held["path"].read_text() + assert held["continuation_body"].endswith(original) + result = subprocess.run(cli_command, env=env, cwd=project, + input=held["continuation_body"], + text=True,capture_output=True, timeout=30) + assert result.returncode == 0, result.stderr + assert len(calls) == 3 and "heartbeat_receipt" in json.dumps(requests[2]) + assert _heartbeat_receipt_count(runtime, TURN_ID) == receipts + assert "RESEARCH_BEFORE_GUARD" in json.dumps(requests[2]) + assert any(item.get("type") == "function_call_output" and + item.get("call_id") == "call_guard" and + "aborted" in item.get("output", "") + for item in requests[2]["input"]) + + +@pytest.mark.skipif(os.name != "posix", reason="Requires owned Unix process groups") +def test_checkpoint_interrupt_rejects_shared_group_and_invalid_timeout(tmp_path): + import sys + from benchmark.runtime.worker import interrupt_checkpoint_process + + with subprocess.Popen([sys.executable, "-c", "import time; time.sleep(20)"]) as process: + try: + with pytest.raises(ValueError, match="positive timeout"): + interrupt_checkpoint_process(process, timeout=float("nan")) + with pytest.raises(ValueError, match="owned child process group"): + interrupt_checkpoint_process(process) + assert process.poll() is None # No signal to the caller's own group. + finally: + process.terminate() + process.wait(timeout=5) + + +@pytest.mark.skipif(os.name != "posix", reason="Requires owned Unix process groups") +def test_checkpoint_interrupt_timeout_reaps_without_qualifying_capture(tmp_path): + import sys + from benchmark.runtime.worker import child_process, interrupt_checkpoint_process + + ready = tmp_path / "ready" + code = ("import signal,time; from pathlib import Path; " + "signal.signal(signal.SIGINT, signal.SIG_IGN); " + f"Path({str(ready)!r}).touch(); time.sleep(30)") + with child_process([sys.executable, "-c", code], + env={**os.environ, "LOOPX_PROJECT": str(tmp_path)}, + stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL) as process: + deadline = time.monotonic() + 5 + while not ready.exists() and time.monotonic() < deadline: + time.sleep(.01) + assert ready.exists() + with pytest.raises(subprocess.TimeoutExpired): + interrupt_checkpoint_process(process, timeout=.1) + assert process.poll() is not None + + +@pytest.mark.parametrize("platform,native,interrupt", [ + ("darwin", True, False), ("linux", True, True), ("darwin", False, True), +]) +def test_midwake_native_profile_refusal_precedes_interruption(tmp_path, monkeypatch, platform, native, interrupt): + from benchmark.runtime.checkpoint_wake import CheckpointWake + + wake = CheckpointWake({"LOOPX_WAKE_LOG_DIR": str(tmp_path / "wakes")}, {"turn_id": "fixture-turn"}) + wake.request, wake.raw = {"native_session": native}, b"fixture declaration" + monkeypatch.setattr(sys, "platform", platform) + intent = {"capture": True, "obligation_id": "replan-0123456789abcdef"} + assert wake._first_replan(intent) is interrupt + assert wake._first_replan(intent) is False # Same point never becomes a later attempt. + events = wake.receipt.get("checkpoint_handoffs", []) + assert len(events) == (0 if interrupt else 1) + if events: + assert not events[0]["captured"] and not events[0]["research_point_qualified"] + + +@pytest.mark.skipif(not os.environ.get("LOOPX_NATIVE_CHECKPOINT_BIN") or + os.environ.get("LOOPX_CHECKPOINT_EXTERNAL_CONTROLLER") != "1", + reason="Requires the real native CLI and explicitly attached checkpoint supervisor") +@pytest.mark.parametrize("explore_mode", ["off", "planning"]) +def test_actual_worker_captures_pending_replan_before_native_model(tmp_path, explore_mode): + """Six real settled work wakes -> pre-decision capture -> stride -> work. + + Inference is deterministic; native history/tools, canonical SQLite, quota, + periodic cadence and the Host writer barrier use their production paths. + No synthetic run-history rows are inserted to manufacture a review. + """ + import sys + import tomllib + from pathlib import Path + from benchmark.runtime.worker import run_once + from tests.control_plane import test_quota_settlement_cli as cli + from tests.control_plane.canonical_authority_fixture import initialize_canonical_authority + from loopx.control_plane.coordination.runtime_shadow import build_todo_runtime_shadow_projection + + project, runtime, registry = cli._write_fixture(tmp_path / "authority") + document = json.loads(registry.read_text()) + document["goals"][0]["quota"]["allowed_slots"] = 18 + registry.write_text(json.dumps(document)) + state_path = project / document["goals"][0]["state_file"] + logs = Path("/logs/agent/wakes") + candidate = project / "candidate.txt" + candidate.write_text("0") + executable = tmp_path / "loopx-cli" + executable.write_text("#!/bin/sh\nexec " + shlex.quote(sys.executable) + ' -m loopx.cli "$@"\n') + executable.chmod(0o700) + skills = tmp_path / "skills" + skills.mkdir() + requests, guards, work_turns = [], [], [] + phase = {"kind": "initial", "values": [], "tool_sent": False} + + def run(*args): + code, result = cli._run_cli(registry, runtime, *args, cwd=project) + assert code == 0, json.dumps(result, indent=2) + return result + + listed = run("todo", "list", "--goal-id", cli.GOAL_ID) + initialize_canonical_authority(runtime, cli.GOAL_ID, + build_todo_runtime_shadow_projection(goal_id=cli.GOAL_ID, todos=listed["todos"], + handoff_mode="hard_lease", leases=[]), + state_path=state_path, provider="sqlite") + run("task-lease", "acquire", "--goal-id", cli.GOAL_ID, "--todo-id", cli.TODO_ID, + "--owner", cli.AGENT_ID, "--idempotency-key", "source-work") + configured = run("configure-goal", "--goal-id", cli.GOAL_ID, + "--execution-replan-after-turns", "6", "--execution-stride-strategy", "evidence_conditioned_v1", + "--explore-mode", explore_mode, "--execute") + assert configured["after"]["execution_profile"]["replan_after_effective_turns"] == 6 + + def response(payload, index, message): + requests.append(payload) + if phase["kind"] == "work" and not phase["tool_sent"]: + phase["tool_sent"] = True + prefix = [sys.executable, "-m", "loopx.entrypoint", "--registry", str(registry), + "--runtime-root", str(runtime), "--format", "json", "quota", "should-run", + "--goal-id", cli.GOAL_ID, "--agent-id", cli.AGENT_ID, "--todo-id", cli.TODO_ID, + "--runtime-profile", "generic_cli", "--available-capability", "shell"] + # The native tool selects its current Turn before changing code. + # Native work and validation precede the real CLI settlement below. + code = ("import os,json,subprocess; from pathlib import Path; " + f"p=subprocess.run({prefix!r}+['--turn-instance-id',os.environ['LOOPX_TURN']]," + "capture_output=True,text=True,check=True); g=json.loads(p.stdout); " + "assert not g.get('autonomous_replan_obligation'); " + "assert g['interaction_contract']['agent_channel']['delivery_allowed']; " + "Path('work-guard.json').write_text(p.stdout); " + f"values={phase['values']!r}; " + "[(Path('candidate.txt').write_text(str(v))," + "Path('evidence-'+str(v)+'.json').write_text(json.dumps({'value':v,'synthetic':True}))) for v in values]; " + "assert Path('candidate.txt').read_text()==str(values[-1]); print('CANDIDATE_VERIFIED')") + return {"type": "function_call", "id": f"fc_work_{index}", "call_id": f"call_work_{index}", + "name": "exec_command", "arguments": json.dumps({ + "cmd": shlex.quote(sys.executable) + " -c " + shlex.quote(code), + "yield_time_ms": 10000, "max_output_tokens": 100})} + if phase["kind"] == "review" and not phase["tool_sent"]: + phase["tool_sent"] = True + replies = list((logs.parent / "checkpoint-controller").glob("*.json")) + reply = next(json.loads(path.read_text()) for path in replies if path.name != "ready.json") + assert reply["ok"] and len(reply["capture"]["sha256"]) == 64 + paths = list(logs.glob("*/checkpoint-start-guard.json")) + assert len(paths) == 1 + guard = json.loads(paths[0].read_text()) + guards.append(guard) + trigger = next(t for t in guard["autonomous_replan_obligation"]["triggers"] + if t["kind"] == "periodic_review_due") + assert trigger["run_count"] == trigger["threshold"] == 6 + control = guard["replan_action_packet"]["stride_control"] + assert set(control["actions"]) == {"grow", "switch", "verify"} + basis = control["explore_basis"] + assert bool(basis and basis["harness_enabled"]) == (explore_mode == "planning") + assert guard["heartbeat_receipt"]["settlement_binding_owed"] + assert "settlement_identity" not in guard["heartbeat_receipt"] + assert not guard["interaction_contract"]["agent_channel"]["delivery_allowed"] + # The sealed point precedes the policy decision and seventh work. + assert candidate.read_text() == "6" + return {"type": "function_call", "id": "fc_packet", "call_id": "call_packet", + "name": "exec_command", "arguments": json.dumps({ + "cmd": "cat " + shlex.quote(str(paths[0])), "max_output_tokens": 30000})} + message["content"][0]["text"] = "Original packet read." if phase["kind"] == "review" else "Synthetic work verified." + return message + + def settle_work(result, expected): + assert result["ok"], result + assert candidate.read_text() == str(expected) + guard = json.loads((project / "work-guard.json").read_text()) + assert guard["heartbeat_receipt"]["turn_instance_id"] == result["turn_id"] + binding = ("--goal-id", cli.GOAL_ID, "--agent-id", cli.AGENT_ID, + "--todo-id", cli.TODO_ID, "--turn-instance-id", result["turn_id"]) + run("refresh-state", *binding, "--classification", "synthetic_candidate_verified", + "--delivery-batch-scale", "implementation", "--delivery-outcome", "outcome_progress", + "--delivery-boundary", "in_flight_continuation", "--progress-result-class", "advanced", + "--progress-surface-id", "fixture:candidate", "--progress-evidence-id", f"evidence:candidate-{expected}", + "--no-global-sync", "--suppress-external-sinks") + spent = run("quota", "spend-slot", *binding, "--source", "heartbeat", "--execute") + assert spent["settlement_progress"]["state"] == "settled" + work_turns.append(result["turn_id"]) + if expected == 3: + replay = run("quota", "spend-slot", *binding, "--source", "heartbeat", "--execute") + assert replay["idempotent_replay"] and not replay["appended"] + + with real_native_source(tmp_path / "provider", os.environ["LOOPX_NATIVE_CHECKPOINT_BIN"], + project, close_source=True, response_builder=response, + sandbox="danger-full-access") as (env, _, _, calls): + base = tomllib.loads((Path(env["CODEX_HOME"]) / "config.toml").read_text()) + env.update(LOOPX_CLI=str(executable), LOOPX_REGISTRY=str(registry), + LOOPX_RUNTIME_ROOT=str(runtime), LOOPX_GOAL_ID=cli.GOAL_ID, LOOPX_AGENT_ID=cli.AGENT_ID, + LOOPX_WAKE_LOG_DIR=str(logs), LOOPX_CODEX_HOME=str(tmp_path / "worker-home"), + LOOPX_ITERATION_CONTEXT="resume", LOOPX_SHARED_SKILLS=str(skills), MODEL_NAME="fixture", + REASONING_EFFORT="high", OPENAI_BASE_URL=base["model_providers"]["fixture"]["base_url"], + OPENAI_API_KEY="synthetic-local-only", LOOPX_CODEX_TURN_TIMEOUT_SEC="60") + for value in range(1, 7): + phase.update(kind="work", values=[value], tool_sent=False) + prior = run_once(env) + settle_work(prior, value) + assert len(set(work_turns)) == 6 and len(calls) == 13 + request = tmp_path / "request.json" + request.write_text(json.dumps({"destination": str(logs.parent / "checkpoints"), + "components": [{"id": "code", "source_path": str(candidate)}], "point_metadata": {}, + "trigger": "fresh_replan", "native_session": True, "loopx_state": True, + "evaluation_opportunity": True, "controller_handoff": True})) + phase.update(kind="review", tool_sent=False) + result = run_once({**env, "LOOPX_CHECKPOINT_REQUEST": str(request)}) + assert result["ok"] and result["checkpoint"]["captured"], result + assert result["checkpoint"]["phase"] == "before_replan" + assert result["checkpoint"]["source_turn_id"] == prior["turn_id"] + assert result["session"]["session_id"] == prior["session"]["session_id"] + assert not result.get("checkpoint_handoffs") and len(calls) == 15 + assert cli._heartbeat_receipt_count(runtime, result["turn_id"]) == 1 + assert "settlement_binding_owed" in json.dumps(requests[14]) and len(guards) == 1 + + # The original source now admits its review; capture has granted nothing. + selected = run("quota", "should-run", "--goal-id", cli.GOAL_ID, "--agent-id", cli.AGENT_ID, + "--todo-id", cli.TODO_ID, "--turn-instance-id", result["turn_id"], "--runtime-profile", "generic_cli") + control = selected["replan_action_packet"]["stride_control"] + decision = json.loads(json.dumps(control["minimal_example"])) + decision.update(basis="supported_progress", action="grow", axes={ + "work_slice": {"plan": "Complete candidates 7 and 8 with local validation in one work wake.", + "reason": "Six settled candidates establish the same complete synthetic work path."}, + "route_distance": {"plan": "Retain the counter increment method.", + "reason": "This checks continuation, not a scientific route improvement."}, + "validation_scope": {"plan": "Verify both candidate evidence files and final value 8.", + "reason": "A larger complete slice must preserve the tested postcondition."}}, + next_probe={"scope": "One two-candidate synthetic work slice.", "prediction": "Final candidate is 8.", + "adoption_criterion": "Both evidence files and final value agree.", + "cost_bound": "One native wake plus its validation and settlement.", + "minimum_evidence": "Native tool output, candidate bytes, evidence and accountable settlement."}) + if "explore_review" in decision: + decision["explore_review"].update(disposition="no_applicable_evidence", considered_refs=[], + reason="The delivered graph contains no findings; retain the tested synthetic method.") + vision = tmp_path / "vision.json" + vision.write_text(json.dumps({"schema_version": "goal_vision_replan_contract_v0", + "state": "vision_patch_proposed", "vision_patch": {"acceptance_summary": "Continuation qualification remains."}, + "path_delta": {"outcome": "continue", "prior_assumption": "One candidate per work wake.", + "observed_reality": "Six native work wakes and their once-only settlements succeeded.", + "changed": ["Complete two synthetic candidates in the next work wake."], + "evidence_refs": ["evidence:candidate-6"]}, "stride_decision": decision})) + binding = ("--goal-id", cli.GOAL_ID, "--agent-id", cli.AGENT_ID, + "--turn-instance-id", result["turn_id"]) + identity = selected["heartbeat_receipt"]["settlement_identity"] + # Selection may preserve a legal Todo binding. Use its actual causal + # identity instead of assuming every periodic review is semantic-only. + if identity.get("todo_id"): + binding += ("--todo-id", identity["todo_id"]) + else: + binding += ("--replan-obligation-id", identity["replan_obligation_id"]) + run("refresh-state", *binding, "--classification", "synthetic_stride_review", + "--autonomous-replan-recorded", "--delivery-outcome", "outcome_progress", + "--agent-vision-json", str(vision), + "--no-global-sync", "--suppress-external-sinks") + run("quota", "spend-slot", *binding, "--source", "heartbeat", "--execute") + phase.update(kind="work", values=[7, 8], tool_sent=False) + adopted = run_once(env) + settle_work(adopted, 8) + assert len(calls) == 17 and len(work_turns) == 7 + assert all(json.loads((project / f"evidence-{v}.json").read_text())["value"] == v for v in (7, 8)) + (tmp_path / "controller-result.json").write_text(json.dumps({**result["checkpoint"], + "natural_work_turns": work_turns[:6], "stride_action": "grow", "explore_mode": explore_mode, + "source_followthrough_verified": True, "adopted_turn_id": adopted["turn_id"]})) + + +@pytest.mark.skipif(not os.environ.get("LOOPX_NATIVE_CHECKPOINT_BIN") or os.name != "posix", + reason="Requires the real native CLI and private Unix transport") +@pytest.mark.parametrize("capture", ["files", "native", "full", "off", "controller"]) +def test_actual_worker_handles_midwake_replan_and_duplicate_without_guard_replay(tmp_path, capture): + """Use run_once, its renderer/listener/session owner and the actual quota CLI.""" + import sys + import tomllib + from pathlib import Path + from benchmark.runtime.worker import run_once + from tests.control_plane.test_quota_settlement_cli import ( + AGENT_ID, GOAL_ID, TURN_ID, _write_fixture, _configure_autonomous_replan_fixture, + _heartbeat_receipt_count, _run_cli, + ) + + if capture == "controller" and os.environ.get("LOOPX_CHECKPOINT_EXTERNAL_CONTROLLER") != "1": + pytest.skip("Requires an explicitly attached host checkpoint supervisor") + project, runtime, registry = _write_fixture(tmp_path / "authority") + logs = Path("/logs/agent/wakes") if capture == "controller" else tmp_path / "trial" / "wakes" + candidate = project / "candidate.txt" + candidate.write_text("original candidate") + request = tmp_path / "request.json" + request.write_text(json.dumps({"destination": str(logs.parent / "checkpoints"), + "components": [{"id": "code", "source_path": str(candidate)}], + "point_metadata": {}, "trigger": "fresh_replan", "native_session": capture in {"native", "full", "controller"}, + "loopx_state": capture in {"full", "controller"}, + "evaluation_opportunity": capture == "controller", "controller_handoff": capture == "controller"})) + skills = tmp_path / "skills" + skills.mkdir() + cli = tmp_path / "loopx-cli" + cli.write_text("#!/bin/sh\nexec " + shlex.quote(sys.executable) + ' -m loopx.cli "$@"\n') + cli.chmod(0o700) + requests, guards = [], [] + + def texts(value): + if isinstance(value, str): + yield value + elif isinstance(value, list): + for child in value: + yield from texts(child) + elif isinstance(value, dict): + for child in value.values(): + yield from texts(child) + + def response(payload, index, message): + requests.append(payload) + if index == 2: + # The Host preflight was ordinary. A new admitted replan arises + # later in this same native wake; only the fixture state is synthetic. + preflights = list(logs.glob("*/checkpoint-start-guard.json")) + if preflights: + packet = json.loads(preflights[0].read_text()) + identity = packet["heartbeat_receipt"]["settlement_identity"] + rc, refreshed = _run_cli(registry, runtime, "refresh-state", "--goal-id", GOAL_ID, + "--agent-id", AGENT_ID, "--todo-id", identity["todo_id"], + "--turn-instance-id", identity["turn_instance_id"], + "--classification", "validated_progress", "--delivery-batch-scale", "implementation", + "--delivery-outcome", "outcome_progress", "--delivery-boundary", "in_flight_continuation", + "--no-global-sync", "--suppress-external-sinks", cwd=project) + assert rc == 0, json.dumps(refreshed) + rc, settled = _run_cli(registry, runtime, *shlex.split(refreshed["settlement_owed"]["command"])[1:], cwd=project) + assert rc == 0, settled + index_path = runtime / "goals" / GOAL_ID / "runs" / "index.jsonl" + prior_runs = index_path.read_text() if index_path.exists() else "" + _configure_autonomous_replan_fixture(project, runtime, registry) + # Preserve the actual preflight settlement evidence. The reusable + # fixture replaces the index; its two later synthetic observations + # must not erase the already settled original Host Turn. + from datetime import datetime, timezone, timedelta + later = [] + for offset, line in enumerate(index_path.read_text().splitlines(), 1): + row = json.loads(line) + row["generated_at"] = (datetime.now(timezone.utc) + timedelta(seconds=offset)).isoformat() + Path(row["json_path"]).write_text(json.dumps(row)) + later.append(json.dumps(row) + "\n") + index_path.write_text(prior_runs + "".join(later)) + if capture == "controller": + from tests.control_plane.canonical_authority_fixture import initialize_canonical_authority + from loopx.control_plane.coordination.runtime_shadow import build_todo_runtime_shadow_projection + rc, listed = _run_cli(registry, runtime, "todo", "list", "--goal-id", GOAL_ID) + assert rc == 0, listed + initialize_canonical_authority(runtime, GOAL_ID, + build_todo_runtime_shadow_projection(goal_id=GOAL_ID, todos=listed["todos"], + handoff_mode="hard_lease", leases=[]), + state_path=project / f".codex/goals/{GOAL_ID}/ACTIVE_GOAL_STATE.md", provider="sqlite") + body = next(t for t in texts(payload) if "quota should-run" in t and "```sh" in t) + guard = next(line for line in body.splitlines() if "quota should-run" in line) + guard = guard.replace('${LOOPX_TURN:?}', TURN_ID) + guards.append(guard) + rendered_guard = ("LOOPX_TURN=" + TURN_ID + "\n" + guard.replace(TURN_ID, "${LOOPX_TURN:?}")) if capture == "native" else guard + return {"type": "function_call", "id": "fc_midwake", "call_id": "call_midwake", + "name": "exec_command", "arguments": json.dumps({"cmd": rendered_guard, + "yield_time_ms": 10000, "max_output_tokens": 100})} + if index == 3 and capture != "off": + # The resumed source deliberately rereads the same obligation. It + # must be released without another capture or native interruption. + return {"type": "function_call", "id": "fc_duplicate", "call_id": "call_duplicate", + "name": "exec_command", "arguments": json.dumps({"cmd": guards[0] + + "\nprintf continued > continued.txt", "yield_time_ms": 10000, + "max_output_tokens": 100})} + message["content"][0]["text"] = "Source research." if index == 1 else "Continued research." + return message + + with real_native_source(tmp_path / "provider", os.environ["LOOPX_NATIVE_CHECKPOINT_BIN"], project, + close_source=True, response_builder=response, sandbox="danger-full-access") as (env, _, _, calls): + base_url = tomllib.loads((Path(env["CODEX_HOME"]) / "config.toml").read_text())["model_providers"]["fixture"]["base_url"] + env.update(LOOPX_CLI=str(cli), LOOPX_REGISTRY=str(registry), LOOPX_RUNTIME_ROOT=str(runtime), + LOOPX_GOAL_ID=GOAL_ID, LOOPX_AGENT_ID=AGENT_ID, LOOPX_WAKE_LOG_DIR=str(logs), + LOOPX_CODEX_HOME=str(tmp_path / "worker-home"), LOOPX_SHARED_SKILLS=str(skills), + MODEL_NAME="fixture", REASONING_EFFORT="high", OPENAI_BASE_URL=base_url, + OPENAI_API_KEY="synthetic-local-only", LOOPX_CODEX_TURN_TIMEOUT_SEC="60") + if capture != "off": + env["LOOPX_CHECKPOINT_REQUEST"] = str(request) + result = run_once(env) + assert result["ok"] and not result["timed_out"], result + assert _heartbeat_receipt_count(runtime, TURN_ID) == 1 + if capture == "off": + assert "checkpoint_handoffs" not in result + assert "--decision-handoff-socket" not in guards[0] + assert not list(logs.glob("*/decisions")) + return + assert len(result["checkpoint_handoffs"]) == 1 + event = result["checkpoint_handoffs"][0] + if json.loads(request.read_text())["native_session"] and sys.platform == "darwin": + # Unsupported native interruption is refused before any signal; + # the original tool result is delivered and no abort is fabricated. + assert not event["captured"] and "not qualified on Darwin" in event["error"] + assert len(calls) == 4 and (project / "continued.txt").read_text() == "continued" + assert any(item.get("type") == "function_call_output" and + item.get("call_id") == "call_midwake" for item in requests[2]["input"]) + assert not any(item.get("type") == "function_call_output" and + "aborted" in item.get("output", "") for item in requests[2]["input"]) + assert not list((logs.parent / "checkpoints").glob("*/checkpoint.json")) + return + assert event["captured"] is (capture in {"files", "native", "controller"}), json.dumps(event) + assert event["research_point_qualified"] is False + assert len(calls) == 4 and (project / "continued.txt").read_text() == "continued" + assert "heartbeat_receipt" in json.dumps(requests[2]) + assert any(item.get("type") == "function_call_output" and item.get("call_id") == "call_midwake" + and "aborted" in item.get("output", "") for item in requests[2]["input"]) + if capture == "controller": + (tmp_path / "controller-result.json").write_text(json.dumps(event)) + return # The external controller verifies the combined artifact. + if capture in {"files", "native"}: + point = logs.parent / "checkpoints" / event["obligation_id"] + assert (point / "components" / "code").read_text() == "original candidate" + if capture == "native": + manifest = json.loads((point / "checkpoint.json").read_text()) + native = manifest["point_metadata"]["native_session"] + assert "completed_turn_id" not in native + assert native["interrupted_guard"]["native_call_id"] == "call_midwake" + assert native["interrupted_guard"]["decision_sha256"] == event["decision_sha256"] + # Reconstruct the earlier captured store after the source has + # already continued. Fixed guest paths require an isolated home; + # retain and restore the source bytes around this synthetic test. + from benchmark.runtime.native_checkpoint import fork_native_checkpoint, admit_native_checkpoint_session + from benchmark.runtime.codex import Execution + home = Path(native["guest_home"]) + saved_home = home.with_name(home.name + "-retained") + config = tmp_path / "captured-config.toml" + config.write_bytes((home / "config.toml").read_bytes()) + with pytest.raises(ValueError, match="distinct destination"): + admit_native_checkpoint_session({**env, "CODEX_HOME": str(home)}, + point, event["sha256"], config, execution=Execution(context="resume"), timeout=20) + home.rename(saved_home) + try: + forked = fork_native_checkpoint({**env, "CODEX_HOME": str(home)}, + point, event["sha256"], config, timeout=20) + assert forked["history_verified"] and forked["model_calls"] == 0 + assert forked["interrupted_turn_id"] == native["interrupted_turn_id"] + assert forked["session_id"] != native["session_id"] + recovered = fork_native_checkpoint({**env, "CODEX_HOME": str(home)}, + point, event["sha256"], config, timeout=20) + assert recovered == forked # No second fork after a lost reply. + finally: + if home.exists(): + home.rename(home.with_name(home.name + "-receiver")) + saved_home.rename(home) + else: + assert "full-state capture" in event["error"] + assert not list((logs.parent / "checkpoints").glob("*/checkpoint.json")) + # The private endpoint belongs to this wake and is gone after readback. + argv = shlex.split(guards[0]) + assert not Path(argv[argv.index("--decision-handoff-socket") + 1]).exists() diff --git a/benchmark/tests/test_resume_sessions.py b/benchmark/tests/test_resume_sessions.py index 5d98f734a0..5c36ff37ea 100644 --- a/benchmark/tests/test_resume_sessions.py +++ b/benchmark/tests/test_resume_sessions.py @@ -15,6 +15,64 @@ from loopx.control_plane.turn_driver.codex_sessions import load_codex_cli_session +@pytest.mark.parametrize("existing", ["valid", "malformed", "dangling"]) +def test_absent_only_session_admission_preserves_every_existing_binding(tmp_path, existing): + """The no-replacement condition is checked under the session owner's lock.""" + from loopx.control_plane.turn_driver.codex_sessions import _session_path, _store_codex_cli_session + + lineage = {"goal_id": "continuation-fixture", "agent_id": "continuation-agent"} + path = _session_path(tmp_path, lineage, "agent") + path.parent.mkdir(parents=True) + if existing == "valid": + _store_codex_cli_session(tmp_path, lineage=lineage, session_scope="agent", session_id="original-session") + elif existing == "malformed": + path.write_bytes(b"unconfirmed original binding") + else: + path.symlink_to(path.with_name("unavailable-original")) + before = path.readlink() if path.is_symlink() else path.read_bytes() + with pytest.raises(ValueError, match="already exists"): + _store_codex_cli_session(tmp_path, lineage=lineage, session_scope="agent", session_id="replacement-session", + require_absent=True) + assert (path.readlink() if path.is_symlink() else path.read_bytes()) == before + if existing == "valid": + # Ordinary native observation retains its existing default behavior. + _store_codex_cli_session(tmp_path, lineage=lineage, session_scope="agent", session_id="observed-session") + assert load_codex_cli_session(tmp_path, lineage=lineage, session_scope="agent")["session_id"] == "observed-session" + + +@pytest.mark.parametrize("replacement", [None, "session", "profile", "lifetime"]) +def test_checkpoint_provenance_belongs_only_to_original_binding(tmp_path, replacement): + from loopx.control_plane.turn_driver.codex_sessions import _session_path, _store_codex_cli_session + + args = dict(lineage={"goal_id": "branch", "agent_id": "actor"}, session_scope="agent", + session_id="original", session_profile_digest="profile", goal_ref={"goal_id": "branch"}) + _store_codex_cli_session(tmp_path, **args, checkpoint_artifact_sha256="a" * 64, require_absent=True) + path = _session_path(tmp_path, args["lineage"], "agent") + original = path.read_bytes() + with pytest.raises(ValueError, match="cannot change"): + _store_codex_cli_session(tmp_path, **args, checkpoint_artifact_sha256="b" * 64) + assert path.read_bytes() == original + if replacement == "session": + args["session_id"] = "replacement" + elif replacement == "profile": + args["session_profile_digest"] = "changed" + elif replacement == "lifetime": + args["goal_ref"] = {"goal_id": "branch", "goal_instance_id": "replacement"} + _store_codex_cli_session(tmp_path, **args) + value = load_codex_cli_session(tmp_path, lineage=args["lineage"], session_scope="agent") + if replacement is None: + assert path.read_bytes() == original + assert value["checkpoint_artifact_sha256"] == "a" * 64 + else: + assert "checkpoint_artifact_sha256" not in value + # A never-enabled ordinary record retains exactly the original field shape. + plain = dict(args, session_id="ordinary") + _store_codex_cli_session(tmp_path, **plain) + assert set(load_codex_cli_session(tmp_path, lineage=args["lineage"], session_scope="agent")) == { + "schema_version", "host", "goal_id", "agent_id", "session_scope", "session_id", + "session_profile_digest", "goal_ref"} + + def resume_env(tmp_path, monkeypatch): env = worker_env(tmp_path) skills = tmp_path / "skills" diff --git a/benchmark/tests/test_sforge_runtime.py b/benchmark/tests/test_sforge_runtime.py index 4651474de4..597da27ac9 100644 --- a/benchmark/tests/test_sforge_runtime.py +++ b/benchmark/tests/test_sforge_runtime.py @@ -153,6 +153,28 @@ def command(handle, cmd, **kwargs): assert len(staged) == int(profile in {"official", "single"}) +@pytest.mark.parametrize("capture_path", [None, "/control/checkpoint-request.json"]) +def test_checkpoint_request_reaches_sforge_worker_without_changing_default_profile(tmp_path, monkeypatch, capture_path): + pytest.importorskip("sforge") + pytest.importorskip("harbor") + from sforge.harness.config import SForgeConfig + from benchmark.runtime.sforge import SForgeWorker, BenchmarkCodex + monkeypatch.setenv("CODEX_AUTH_JSON_PATH", "/synthetic-credential") + async def installed(self, environment): + pass # Exercise the real adapter handoff without starting a trial. + monkeypatch.setattr(BenchmarkCodex, "install", installed) + config = SForgeConfig(agent_model="fixture", agent_effort="xhigh") + worker = SForgeWorker(config, profile="heartbeat-explore", cwd="/task", checkpoint_request_path=capture_path) + worker.install_stop_hook(None, None, tmp_path, None) + env = worker.runtime._worker_env(cwd="/task") + profile = json.loads((tmp_path / "worker-profile.json").read_text()) + assert env.get("LOOPX_CHECKPOINT_REQUEST") == capture_path + assert ("checkpoint_capture_enabled" in profile) is (capture_path is not None) + assert profile.get("checkpoint_capture_enabled") is (True if capture_path else None) + with pytest.raises(ValueError, match="checkpoint_request_path"): + SForgeWorker(config, profile="official", cwd="/task", checkpoint_request_path="/control/request.json") + + @pytest.mark.parametrize("profile,total,expected", [ ("native-goal", 64800, 64640), ("native-goal", 1800, 1640), ("heartbeat-resume", 64800, 64640), ("heartbeat-explore", 64800, 64640), @@ -202,6 +224,7 @@ def test_artifact_collection_preserves_both_session_homes_without_auth(tmp_path, pytest.importorskip("harbor") from benchmark.runtime.sforge_backend import RecordingDockerBackend, DockerBackend backend = object.__new__(RecordingDockerBackend) + backend.checkpoint = None backend.log_dir = tmp_path / "artifacts" backend.feedback = None copied, cleaned = [], [] @@ -255,6 +278,7 @@ def test_blind_policy_removes_judge_route_and_credentials_native_is_unchanged(mo pytest.importorskip("harbor") from benchmark.runtime.sforge_backend import RecordingDockerBackend, DockerBackend, AllowedEndpoint backend = object.__new__(RecordingDockerBackend) + backend.checkpoint = None backend.auth_ips = [] backend.blind_api_endpoint = None judge = AllowedEndpoint(ip="192.0.2.1", port=8080, hostname="judge") @@ -457,6 +481,8 @@ def test_envelope_rejects_incompatible_sforge_worker(profile): ("official", ["--stride-strategy", "off"]), ("native-goal", ["--stride-strategy", "evidence_conditioned_v1"]), ("heartbeat-resume", ["--stride-strategy", "future_policy"]), + ("official", ["--checkpoint-request", "missing.json"]), + ("native-goal", ["--checkpoint-request", "missing.json"]), ]) def test_edgebench_rejects_invalid_profile_settings_before_creating_trial(tmp_path, profile, options): pytest.importorskip('sforge') @@ -470,6 +496,104 @@ def test_edgebench_rejects_invalid_profile_settings_before_creating_trial(tmp_pa assert not (tmp_path / 'runs').exists() +@pytest.mark.parametrize("missing", ["controller", "declaration", "native_feedback"]) +def test_controller_handoff_cannot_be_partially_enabled(tmp_path, missing): + pytest.importorskip("sforge") + pytest.importorskip("harbor") + from benchmark.edgebench.run import main + request = tmp_path / "checkpoint.json" + request.write_text(json.dumps({"destination": "/logs/agent/checkpoints", "point_metadata": {}, + "components": [{"id": "code", "source_path": "/task/code"}], "trigger": "fresh_replan", + "native_session": True, "loopx_state": True, "evaluation_opportunity": True, + "controller_handoff": True})) + options = [] if missing == "controller" else ["--checkpoint-controller-file", str(tmp_path / "private.json")] + if missing != "declaration": + options.extend(["--checkpoint-request", str(request)]) + if missing == "native_feedback": + options.extend(["--feedback", "best-only"]) + with pytest.raises(SystemExit) as error: + main(["--task", "fixture", "--tasks-dir", str(tmp_path), "--log-dir", str(tmp_path), + "--run-id", "invalid", "--worker", "heartbeat-resume", "--model", "fixture", + "--effort", "xhigh", "--judge-url", "http://127.0.0.1:9999", *options]) + assert error.value.code == 2 and not (tmp_path / "runs").exists() + + +def test_checkpoint_supervisor_is_bound_to_container_and_only_solver_execution(tmp_path, monkeypatch): + pytest.importorskip("sforge") + pytest.importorskip("harbor") + from types import SimpleNamespace + from benchmark.runtime.sforge_backend import RecordingDockerBackend, DockerBackend + calls = [] + controller = SimpleNamespace(bind=lambda env: calls.append(("bind", env["SFORGE_TOKEN"])), + start=lambda raw: calls.append(("start", raw)), close=lambda: calls.append(("close",))) + backend = RecordingDockerBackend(log_dir=tmp_path / "collected", logger=None, + oauth_proxy=True, checkpoint=controller) + handle = SimpleNamespace(raw="exact-container") + monkeypatch.setattr(DockerBackend, "create_container", lambda *a, **kw: handle) + backend.create_container("image", "name", environment={"SFORGE_TOKEN": "synthetic"}) + backend.execution_command = "solver" + monkeypatch.setattr(DockerBackend, "exec_run_with_timeout", lambda *a, **kw: + SimpleNamespace(exit_code=0, timed_out=False, elapsed_seconds=1)) + backend.exec_run_with_timeout(handle, ["setup"]) + assert calls == [("bind", "synthetic")] + backend.exec_run_with_timeout(handle, ["/bin/bash", "-c", "solver"]) + assert calls == [("bind", "synthetic"), ("start", "exact-container"), ("close",)] + + +@pytest.mark.parametrize("protect_fails", [False, True]) +def test_host_checkpoint_declaration_is_staged_verbatim_and_protected(tmp_path, monkeypatch, protect_fails): + pytest.importorskip("sforge") + pytest.importorskip("harbor") + from types import SimpleNamespace + from sforge.harness.config import SForgeConfig + from benchmark.runtime.sforge import SForgeWorker, BenchmarkCodex + monkeypatch.setenv("CODEX_AUTH_JSON_PATH", "/synthetic-credential") + async def installed(self, environment): + pass # Exercise the installation transport; no model or evaluator. + monkeypatch.setattr(BenchmarkCodex, "install", installed) + staged, commands = [], [] + def copy(handle, source, target): + staged.append((str(target), source.read_bytes())) + def command(handle, cmd, **kwargs): + commands.append((cmd, kwargs)) + return SimpleNamespace(exit_code=int(protect_fails)) + backend = SimpleNamespace(copy_to_container=copy, exec_run=command) + declaration = b'{"destination":"/logs/agent/checkpoints","trigger":"fresh_replan", "native_session":true,"components":[{"id":"code","source_path":"/task"}],"point_metadata":{}}' + config = SForgeConfig(agent_model="fixture", agent_effort="xhigh") + worker = SForgeWorker(config, profile="heartbeat-resume", cwd="/task", checkpoint_request=declaration) + if protect_fails: + with pytest.raises(RuntimeError, match="protect the explicit checkpoint declaration"): + worker.install_stop_hook(backend, None, tmp_path, None) + assert not (tmp_path / "worker-profile.json").exists() + else: + worker.install_stop_hook(backend, None, tmp_path, None) + assert json.loads((tmp_path / "worker-profile.json").read_text())["checkpoint_capture_enabled"] + guest_path = "/opt/loopx-benchmark/control/checkpoint-request.json" + assert staged == [(guest_path, declaration)] + assert worker.runtime._worker_env(cwd="/task")["LOOPX_CHECKPOINT_REQUEST"] == guest_path + assert len(commands) == 1 and commands[0][1]["user"] == "root" + assert "chmod 0600" in commands[0][0][-1] + with pytest.raises(ValueError, match="no guest-path override"): + SForgeWorker(config, profile="heartbeat-resume", cwd="/task", checkpoint_request=declaration, + checkpoint_request_path="/other") + with pytest.raises(ValueError, match="requires destination"): + SForgeWorker(config, profile="heartbeat-resume", cwd="/task", checkpoint_request=b'{}') + + +def test_invalid_checkpoint_declaration_rejects_before_trial_registration(tmp_path): + pytest.importorskip("sforge") + pytest.importorskip("harbor") + from benchmark.edgebench.run import main + declaration = tmp_path / "request.json" + declaration.write_text('{"components":[]}') + with pytest.raises(ValueError, match="requires destination"): + main(["--task", "fixture", "--tasks-dir", str(tmp_path), "--log-dir", str(tmp_path), + "--run-id", "invalid", "--worker", "heartbeat-resume", "--model", "fixture", + "--effort", "xhigh", "--judge-url", "http://127.0.0.1:9999", + "--checkpoint-request", str(declaration)]) + assert not (tmp_path / "runs").exists() + + @pytest.mark.parametrize("cadence_args,field,count", [ ([], "replan_after_effective_turns", 6), (["--replan-after-turns", "2"], "replan_after_effective_turns", 2), @@ -651,6 +775,7 @@ def test_recorded_solver_exit_does_not_take_status_from_output(tmp_path, monkeyp from types import SimpleNamespace from benchmark.runtime.sforge_backend import RecordingDockerBackend, DockerBackend backend = object.__new__(RecordingDockerBackend) + backend.checkpoint = None backend.log_dir = tmp_path / "collected" backend.blind_api_endpoint = None backend.feedback = None diff --git a/benchmark/tests/test_shared_codex_runtime.py b/benchmark/tests/test_shared_codex_runtime.py index e9e3e2787d..791d41fd74 100644 --- a/benchmark/tests/test_shared_codex_runtime.py +++ b/benchmark/tests/test_shared_codex_runtime.py @@ -362,6 +362,21 @@ def test_baseline_and_treatment_use_same_harbor_entry(tmp_path): assert agent.replan_after_turns == (6 if agent.execution.uses_loopx else None) +def test_checkpoint_request_is_explicit_and_reaches_the_existing_worker_env(tmp_path, monkeypatch): + pytest.importorskip("harbor") + from benchmark.runtime.harbor import BenchmarkCodex + monkeypatch.setenv("LOOPX_CHECKPOINT_REQUEST", "/ambient-request.json") + off = BenchmarkCodex(logs_dir=tmp_path / "off", model_name="openai/fixture") + assert "LOOPX_CHECKPOINT_REQUEST" not in off._worker_env(cwd="/workspace") + on = BenchmarkCodex(logs_dir=tmp_path / "on", model_name="openai/fixture", + checkpoint_request_path="/control/checkpoint-request.json") + assert on._worker_env(cwd="/workspace")["LOOPX_CHECKPOINT_REQUEST"] == "/control/checkpoint-request.json" + for settings in ({"execution_mode": "plain", "checkpoint_request_path": "/control/request.json"}, + {"checkpoint_request_path": "relative"}, {"checkpoint_request_path": "/control/../request.json"}): + with pytest.raises(ValueError, match="checkpoint_request_path"): + BenchmarkCodex(logs_dir=tmp_path / "invalid", model_name="openai/fixture", **settings) + + @pytest.mark.parametrize("existing", [False, True]) @pytest.mark.parametrize("settings,field,value", [ ({}, "replan_after_effective_turns", 6), diff --git a/docs/architecture/rfcs/hierarchical-agent-stride-control-v0.md b/docs/architecture/rfcs/hierarchical-agent-stride-control-v0.md index b9ece5f271..bd2816734c 100644 --- a/docs/architecture/rfcs/hierarchical-agent-stride-control-v0.md +++ b/docs/architecture/rfcs/hierarchical-agent-stride-control-v0.md @@ -679,10 +679,31 @@ consistency and comparison requirements, not a second restore authority. Before capture, declare the point by elapsed research budget or a semantic event; do not choose it retrospectively for a favorable score. A later point such as ten hours is an experiment setting, not a product default or automatic pivot. + +The proposed automatic capture policy is **off by default; when explicitly +enabled, capture at each fresh replan node before its strategy decision**. Reuse +the canonical replan obligation identity: repeated context reads or retries of +that node do not create another point. Ordinary work Turns do not trigger a full +capture. A successful point records the replan identity, the completed Host +cursor, source revisions and current strategy version; these are evidence +bindings, not authority. Branches can then compare different replan decisions +from the same retained research state, including a decision to keep the route. + +The trigger still waits for the settled boundary below and must precede the +replan decision and its new work. A replan can occur inside one Host wake, so an +after-wake hook alone cannot implement this ordering. The replan and Host owners +must provide that handoff without an extra model call or work settlement. If the +boundary cannot be qualified, record the missed point and continue under the +existing execution contract; do not substitute a later state for that node. +Measure capture time and storage per node before choosing compression, shared +immutable members or retention limits. Retention must preserve declared study +points and their branches rather than silently selecting favorable outcomes. + Record the actual boundary and any skew: -- native conversation cursor and a Host-qualified export/fork path; copied log - text does not establish a resumable session; +- native conversation cursor and a Host-qualified export/fork path, including + compaction records, paginated history and the session index needed for native + readback; copied log text does not establish a resumable session; - settled LoopX research evidence, original goal/priorities, prior decisions, scoped negative results and unresolved questions; - exact working code, necessary authorized untracked files, incumbent artifact @@ -691,6 +712,259 @@ Record the actual boundary and any skew: - remaining total budget, accepted submission history and next feedback opportunity under the original protocol. +Capture belongs at the existing **settled work-Turn boundary**, before the next +work admission. A time-based request waits for that boundary and records its +actual elapsed budget; it cannot interrupt an arbitrary tool operation and +label the resulting files consistent. The runner/Host owns this barrier, while +each existing participant owns its snapshot: + +1. Finish owed Turn settlement and reconcile outstanding tool, child-work and + evaluator effects by their original identities. Unresolved effects make the + point unqualified; copying their journals does not resolve them. +2. Capture the native completed-turn cursor, canonical research-state revision, + working-artifact manifest and evaluator opportunity state at that barrier. + Use the existing SQLite snapshot owner where applicable, not a live database + file copy. Keep code and incumbent/result bindings independently verifiable. +3. Recheck each participant's revision/cursor and artifact digests before + publishing one immutable verification unit. Any movement, missing member or + interrupted capture leaves an incomplete unit, never a usable point. This + establishes the declared barrier's consistency, not a cross-owner transaction. +4. Restore into an isolated destination, verify its contents, then use the + existing owners to admit fresh sessions and research work. Restore alone + cannot start a model, claim work, register a timer or submit a candidate. + +The Host adapter must qualify native history reconstruction at its pinned +version, including compaction and paginated storage. For example, Codex's +[native fork protocol](https://learn.chatgpt.com/docs/app-server) +can bound a branch by a completed turn. An exported file still needs the native +store's import/readback path; an arbitrary file path or successful log parse is +insufficient. Compare the reconstructed prefixes and excluded later history, +not just returned session identifiers. Keep provider-specific migration and +storage handling outside the generic consistency decision. + +Live native forks need a different qualification from completed-cursor forks. +The real Codex `0.162.0-alpha.2` characterization in +`benchmark/tests/test_native_checkpoint_handoff.py` rejects an explicit +`lastTurnId` while that turn is active. Omitting it can retain the current prefix +as an interrupted branch while the source tool keeps running. Resuming that +branch exposes an aborted tool result; it does not replay the tool automatically. +This avoids requiring a source interruption merely to investigate a fork, but +neither the returned branch nor its history proves source quiescence. The next +joint handoff must bind the withheld original guard and pending tool identity, +reconcile any admitted effects, and verify frozen participant bytes before +qualifying that cursor. Keep the completed-final-cursor production check until +this separate path is implemented and qualified; do not silently drop the active +turn or classify every interrupted session as resumable research state. + +The quota transport now has an explicit local pre-output rendezvous. Its real +native/CLI qualification binds a pending command event and the saved original +guard digest, forks without another model call, and releases that same response +without another guard receipt. Lost acknowledgement and altered bytes fail +without manufacturing another point. This closes the response-ordering gap only: +work-subprocess yielding can still return control to the model, so source writer +quiescence, effect reconciliation, automatic worker/controller connection and +isolated useful continuation remain required. The completed-cursor acceptance +check remains unchanged. + +The existing CLI execution path now has an explicit interruption/recovery +adapter at the rendezvous: it closes the owned native process group before +selection and capture, requires graceful leader exit, reaps descendants, and +returns the retained original guard only after successful context exit. A real +CLI test exceeds the tool-yield interval and resumes the same source session +without another quota receipt. The aborted tool remains visible; this is a new +native turn, not transparent live-stack recovery. The adapter does not yet prove +that no model action preceded its interruption. The actual heartbeat worker now +installs the rendezvous for an explicit fresh-replan request, deduplicates the +original obligation, and resumes the same source session with the original guard +and remaining wake budget. Ordinary/duplicate guards do not restart the native +CLI. Declared file capture and failed optional capture use this real entrypoint; +full-state mid-wake requests remain explicitly unqualified. The native participant +now binds an explicit interrupted final guard, its parsed routes/Turn identity and +original packet digest, rejects released output/later visible items or duplicate +matching calls, and preserves the full prefix in a native fork. Completed-cursor +checks stay strict. The integrated controller/receiver qualification below now +covers the joined source and isolated destination path; participant reconstruction +by itself still grants no work or destination authority. + +The Host profile must also declare path portability. A native index can retain +absolute guest paths: qualification must either preserve those paths inside a +fresh isolated environment or use the Host's supported import/rebinding API. +Editing native database rows by hand does not qualify import. Bind the effective +Host configuration and installed tool/skill versions separately from credentials; +reauthentication remains with the existing owner. Full research state means the +declared participants needed to resume useful work, not a copy of every file or +the model's unobservable internal state. + +Historical conversation truncation cannot reconstruct earlier code, research +state or feedback consumption from their terminal versions. Without a qualified +contemporaneous capture of those participants, such a probe establishes Host +feasibility only and must not enter a same-state intervention comparison. + +The initial implementation slice is [inert artifact capture](../../../benchmark/README.md#inert-checkpoint-capture). +The benchmark adapter requests it after the existing Turn settlement; the typed +artifact owner seals explicit files/directories and committed SQLite snapshots, +rechecks source witnesses and verifies an isolated materialization. Its manifest +always declares the research point unqualified. The heartbeat adapter also accepts +an explicit fresh-replan trigger: it preloads the original quota guard and captures +an admitted replan projected at wake start before invoking the model, deduplicated +by the existing obligation. It does not cover replans first created inside a running +Host wake. The optional native participant reads complete history through Codex, +checks the session's actual workspace and current source binding/profile, and rechecks them after the byte +probe. A later source Turn in the read-to-probe gap is rejected; the artifact +owner's witness checks cover subsequent writes. It seals rollout/index/paged +stores and verifies native forks at fixed guest paths. These source-basis checks +are qualified separately from research utility. An explicit source-checkout +controller now prepares a verified fork through the existing registry/session +owners, with distinct registered Goal/Agent identities and a matching captured +session profile. The registry fence guards configuration through the binding write; +existing lifetime semantics remain. The lifecycle-only source-session profile +cannot run governed work and is rejected rather than used as a benchmark surrogate. +Replacement, changed configuration/profile and native automatic Goal activation +are refused. Real native CLI/resume qualification uses a local static provider +to carry retained conversation into a new Turn, verify materialized code/negative +evidence and settle the original LoopX guard/writeback/spend. The quiesced fixture +swaps fixed guest paths; the newer joint receiver fixture below uses a fresh Linux +guest instead. This earlier test proves +the conversation participant's adoption and synthetic continuation, not full research/effect adoption, remaining +whole-trial budget, installed frontend recovery or scientific benefit. +An explicit checkpoint-aware native judge also exports released public result +details, accepted counters/clock, remaining cooldown and original trial deadline, +plus the opaque native-selected incumbent archive bound to its released result. +The incumbent can precede the working candidate; native ranking remains the owner. +Missing bytes, changed capture bindings or mismatched file witnesses reject before +publication, without grading or refilling admission; pending or unreconciled grades reject. +The opt-in LoopX-state participant additionally seals a complete paged canonical +Todo read at one promoted provider revision, with byte/revision rechecks. It +retains complete Explore events and indexed vision/checkpoint history, including +stride decisions, as historical context. The source owner validates Explore rows; +framing, source-file digests and absence are checked against the sealed inventory. +Malformed or moved history rejects the optional capture. These retained records +do not activate a destination graph, old vision or historical effects. +Unpromoted coordination retains its previous inert inventory only, explicitly +without canonical context adoption. A controller can +attach this retained context to an already owned destination Todo through the +existing leased continuation prepare/adopt owner and canonical projection +readback. The source population, notes and original negative evidence are +available as private context; current requirements, status, claims and leases +remain current authority. Native profile and complete-history readback precede +both preparation and adoption. A real native fork/resume plus CLI and canonical +owner fixture consumes the captured candidate, negative Todo/Explore evidence and vision, excludes +later source evidence, and settles governed work. Its local deterministic provider +qualifies transport/adoption and mechanical use, not research judgment or benefit; +context-command retries now reuse the original prepare/adopt identities and +revision after lost responses or projection failure, while changed authority, +notes, artifacts and advanced native history still refuse recovery. Canonical +graph/vision/effect activation, full native/Host/evaluator consistency, joint +participant recovery, packaged delivery and useful continued research remain open. Participant/effect reconciliation and complete fresh research-state +adoption cannot be certified by operator metadata or native history equality. + +The explicit Docker controller now offers a bounded writer barrier: pause the +owned running worker, archive declared files and its phase clock through the +daemon, capture the native evaluator capsule, then unpause before publishing an +inert joint unit. A live-writer/native-HTTP qualification proves file stability +and original opportunity retention. Standalone SQLite participants also transport +their WAL and rollback journal under that same pause, then use the existing +SQLite snapshot owner: a valid main database alone can omit committed research. +Committed WAL rows survive, uncommitted rows stay excluded, and unavailable or +linked companions reject capture before native publication. Cancellation, lost +pause responses and failed capture exercise cleanup. An existing pause is not +owned or released, and failed +resume prevents publication. This closes the manual provider's file-writer +barrier gap only. No process memory, native cursor eligibility, external writers, +automatic pre-decision handoff or isolated continuation are qualified by it. + +A native fresh registration is not restored admission. In the controlled native +Docker qualification, one of twelve source submissions was consumed and its +cooldown remained active. A same-limit fresh registration admitted `agent-1` +immediately with twelve unused rounds. Regrading the retained archive created a +new result, without restoring the source history or opportunity state. This is +valid new-trial behavior; it cannot substitute for a same-point continuation. +The evaluator owner must separately qualify destination initialization from the +bound history/counters and remaining cooldown duration, together with the Host's +remaining total budget. Copying captured token rows is not admission proof. + +The [controller-only native capsule transport](../../../benchmark/README.md#controller-only-native-capsule-transport) +now connects the proposed native initializer to the existing typed artifact owner: +explicit controller authority captures/imports a sealed native bank and requests +fresh native registration. Controller configuration stays out of solver payloads +and exported participants; opaque data remains private and uncertified. This slice preserves native history, selected +archive, counters and cooldown; it does not yet qualify the joint replan barrier, +model/LoopX state adoption or useful research. Its explicit SForge phase-file +participant also seals the declared Host deadline against the native capture +timestamp. Restore recomputes the remaining duration and anchors a new Host phase +file at activation, charging transport rather than offline time or a fresh trial. +The existing work reserve and native opportunity owner remain unchanged. An +uncertain response leaves an unusable phase reservation, not a fresh-budget +fallback. Explicit controller recovery now reads the original native activation +and retains the destination deadline durably bound before the first admission. +Lost replies and partial phase writes recover without a new registration, +round, cooldown or budget reset; exhausted or mismatched attempts refuse. +An isolated real native HTTP/Docker journey verifies recovery, the ordinary +Linux phase-entry shell, candidate production and the next accepted grade. +Model inference is excluded. Full Host provenance/clock alignment and joint +session/research/effect recovery still need their profile's qualification; +this participant transport cannot certify them. + +The automatic source-capture boundary composes the held fresh-replan decision, +native cursor and canonical research inventory under the controller Docker writer +barrier. It verifies the exact pending request and prepared source witness before +sealing incumbent, opportunity and remaining phase budget. A changed or expired +boundary is missed, never recaptured after source continuation. + +The joint receiver qualification now consumes an actual controller-captured +interrupted point in a fresh Linux guest with no mounted source authority. +Explicit native binding and leased Todo-context adoption retain the pending call +as history; a fresh destination guard owns subsequent work. The same nested +native capsule supplies opportunities and the Host phase deadline. A resumed +native tool reads retained context/code, creates a synthetic candidate and receives +its next grade under that deadline, followed by exactly-once destination settlement. +Lost binding acknowledgment and activation readback reuse their committed identities; +wrong source/digest and advanced history reject. This closes the joined mechanical +continuation gap for the tested profile, not installed campaign admission, packaged +recovery UX, autonomous research quality or a stride-effect claim. + +The completed-cursor path is also exercised through the real worker, native Host, +SQLite inventory and Docker controller before the model sees a fresh replan. +Its committed guard may still owe work selection: inert capture preserves that +prohibition instead of fabricating a causal settlement binding. An isolated +receiver resumes the retained code/context under fresh authority. When the sealed +cooldown outlasts its work window, local candidate work succeeds while native +submission returns 429 without consuming a round or invoking a grader. The +separate eligible-cursor fixture receives its next grade. These two lawful states +must remain distinct; restoring a point does not make its next submission eligible. + +The completed-source fixture now reaches that review through six actual native +work wakes and exactly-once CLI settlements rather than inserted history rows. +Stride is enabled with Explore planning separately off/on. Capture precedes the +decision; the source records grow with independent axis plans and completes its +larger synthetic next slice. Fresh isolated receivers consume the sealed code and +context while preserving pending cooldown. This qualifies mechanical composition, +not model judgment, automatic destination graph/policy adoption or utility. +The tested Darwin native Host can omit the pending command after interruption; +delayed reads and leader-only signalling do not repair it. That profile refuses +mid-wake native capture before signalling and releases the original tool response. +Linux interrupted and completed-wake capture retain their separate qualification; +strict history, ambiguity, source and deadline predicates remain unchanged. + +Committed native session admission now retains the captured artifact digest in +the original atomic session binding. Exact controller retries after a lost binding +reply verify that receipt, current Goal/Agent/profile and unchanged complete +native history, then return the same session without a fork or model call. +Ordinary subsequent observations preserve provenance only for that conversation; +replacement identity drops it. Unmarked/foreign bindings and advanced native +history reject old-point admission. Before native fork dispatch, a private +transport attempt retains the source artifact/destination basis and native +identity inventory under the existing process lock. A lost pre-binding reply can +recover the unique new native child, after native parent/workspace and complete +history verification, without dispatching another fork. Missing or ambiguous +outcomes stay unresolved. The pinned native index supplies only opaque lookup +IDs because the interactive listing hides unused forks; native APIs verify their +meaning, and no native rows are rewritten. Tests inject pre- and post-binding +reply loss, recover through the CLI and continue real native governed work with +a deterministic local provider; absent/multiple outcomes and foreign destination +attempts reject. The controller participant recovery above does not yet qualify +automatic joint session/research/evaluator/Host activation recovery. + An inert verification unit grants no execution authority. The destination uses fresh experiment/Host identities and the existing Goal/session/state admission paths. Historical leases, quota, timers, pending effects and submission tokens @@ -709,6 +983,10 @@ continuation and evidence-conditioned scope review, with equal continuation budgets and resources. Change one control factor; preserve feedback opportunity counts and cooldown timing through the evaluator owner. Label session restart, summary replacement, source upgrades or policy bundles as additional factors. +The evaluator carries the remaining cooldown and consumed opportunities into +each newly authorized branch; a fresh identity must not create an extra immediate +submission. Wall-clock waiting between capture and admission does not replenish +the checkpoint's opportunity allowance or extend the continuation budget. Keep all branches and invalid/negative results; compare adopted native outcomes, information gained and full continuation cost. The result qualifies the scoped intervention, not all LoopX behavior or an optimal universal stride. diff --git a/docs/architecture/rfcs/hierarchical-agent-stride-control-v0.zh-CN.md b/docs/architecture/rfcs/hierarchical-agent-stride-control-v0.zh-CN.md index 97a51adc7c..d880f9ec99 100644 --- a/docs/architecture/rfcs/hierarchical-agent-stride-control-v0.zh-CN.md +++ b/docs/architecture/rfcs/hierarchical-agent-stride-control-v0.zh-CN.md @@ -576,16 +576,207 @@ Host 会话 owner 和 artifact provider；本 RFC 定义研究实验的一致性 不新建 restore 权威。 捕获前按已分配研究预算或语义事件声明位置，不事后挑分数有利的点。十小时等 -中后期位置属于实验设置，不是产品默认值或自动转向条件。记录实际边界与 skew： +中后期位置属于实验设置，不是产品默认值或自动转向条件。 -- 原生会话 cursor 和已资格化的 Host export/fork 路径；复制日志不证明能恢复 - session； +自动捕获策略提案是**默认关闭；显式开启后，在每个新的 replan 节点、策略决策 +之前捕获**。复用 canonical replan obligation 身份：重复读取 context 或重试 +同一节点不会生成另一研究点。普通工作 Turn 不触发完整捕获。成功研究点绑定 +replan 身份、已完成 Host cursor、来源 revision 及当前策略版本；这些是证据 +绑定，不是权限。不同分支可从同一份积累比较不同 replan 决策，包括保留原路线。 + +触发仍须等待下述已结算边界，并先于 replan 决策及其新工作。一次 Host wake +内部也可能发生 replan，因此单独的 after-wake hook 不能实现这个顺序。Replan +与 Host owner 须提供该衔接，不增加模型调用或工作结算。边界无法资格化时记录 +缺失研究点，按既有执行契约继续；不能用更晚状态代替该节点。先记录每个节点 +的捕获时间和存储量，再决定压缩、共享不可变成员或保留上限。保留策略须保护 +已声明研究点及其分支，不能静默挑选有利结果。 + +记录实际边界与 skew： + +- 原生会话 cursor 和已资格化的 Host export/fork 路径，包括 compaction 记录、 + 分页历史及原生读回所需的 session 索引；复制日志不证明能恢复 session； - 已结算的 LoopX 研究证据、原目标/优先级、既有决定、有范围负结果及未解问题； - 精确工作源码、必要且获授权的未跟踪文件、incumbent 与其 qualifying result； 尚未测量的候选与 incumbent 分开保留； - source/model/tool/evaluator pins 与已消费的 solver-visible feedback； - 剩余总预算、已受理提交历史及原协议下的下一次反馈机会。 +捕获接在既有**有效工作 Turn 已结算、下一次工作尚未准入**的边界。按时间请求 +捕获时，等待该边界并记录实际已用预算；不能任意打断工具操作，再把所得文件 +称为一致状态。Runner/Host 持有这道 barrier，各既有参与 owner 分别负责快照： + +1. 完成欠账的 Turn 结算，并按原身份 reconcile 未决工具、子任务和 evaluator + 效果。未决效果使研究点尚未资格化；复制 journal 不等于解决效果。 +2. 在 barrier 记录原生已完成 turn cursor、canonical 研究状态 revision、工作 + 产物 manifest 和 evaluator 机会状态。适用时复用既有 SQLite snapshot owner， + 不复制活数据库文件；代码与 incumbent/result 绑定分别可验证。 +3. 发布一份不可变 verification unit 前，复查各参与者 revision/cursor 和产物 + digest。来源移动、缺失成员或中断捕获都留下 incomplete unit，不能成为可用 + 研究点；这只证明所声明 barrier 的一致性，不宣称跨 owner 事务。 +4. 恢复到隔离目标、验证内容，再由既有 owner 准入新 session 和研究工作。 + Restore 本身不能启动模型、claim 工作、注册 timer 或提交候选。 + +Host adapter 须在固定版本验证原生历史重建，包括 compaction 和分页存储。 +例如 Codex 的[原生 fork 协议](https://learn.chatgpt.com/docs/app-server) +可按已完成 turn 限定分支；导出文件仍须通过原生 store 的 import/readback 路径， +任意文件路径或日志解析成功并不充分。比较重建的历史前缀及被排除的后续历史， +不只检查返回了 session id。Provider 特有迁移与存储处理不进入通用一致性决策。 + +运行中会话的原生 fork 须与已完成 cursor 分别资格化。 +`benchmark/tests/test_native_checkpoint_handoff.py` 在真实 Codex +`0.162.0-alpha.2` 上确认：显式 `lastTurnId` 不能指向 active turn；省略后可保留 +当前前缀为 interrupted 分支，而来源工具仍运行。接续分支会看到 aborted 工具 +结果，不会自动重放工具。因此研究 fork 路径不必先中断来源，但返回分支或历史 +都不能证明来源已静止。下一步联合 handoff 须绑定尚未返回的原 guard 与 pending +工具身份、核回已受理效果，并验证冻结的参与者字节，再资格化该 cursor。 +在该独立路径实现并验证前保留生产代码的 completed-final-cursor 检查；不能 +静默丢弃 active turn，也不能把所有 interrupted 会话视为可接续研究状态。 + +quota transport 已增加显式本地 pre-output rendezvous。真实原生 Host/CLI 验证 +绑定 pending command event 与保存的原 guard digest，在没有额外模型调用时 +fork，并释放同一响应，不新增 guard receipt。丢失确认或字节变化会失败，不另造 +捕获点。此处只补齐响应顺序：工具子进程 yield 仍可能让模型继续；来源 writer +静止、effect 核回、worker/controller 自动衔接和隔离有效接续仍须完成。 +completed cursor 的验收检查保持不变。 + +既有 CLI 路径现在提供显式中断/恢复适配：在 rendezvous 收到原包后，先关闭 +Host 拥有的原生进程组，要求主进程正常响应中断退出并清理后代，再进行选择和 +捕获；只有捕获上下文成功退出才返回原 guard 的接续正文。真实 CLI 测试等待 +超过工具 yield 周期后，仍能在同一来源 session 显式接续且不新增 quota receipt。 +被中断的工具保留 aborted 记录；接续是新的原生 turn，不是透明恢复执行栈。 +它尚未证明中断之前没有模型动作。实际 heartbeat worker 现已在显式 fresh-replan +请求下安装 rendezvous，按原义务去重，并用原 guard 和剩余 wake 预算接续同一 +来源 session；普通或重复 guard 不重启原生 CLI。声明文件的捕获与可选捕获失败 +均经过该实际入口；中途的完整状态请求仍明确不具备资格。原生参与者已显式绑定 +interrupted 末尾 guard、解析后的路由/Turn 身份及原包摘要，拒绝已释放输出、 +后续可见条目或重复匹配调用，并通过原生 fork 保留完整前缀。completed cursor +检查不放宽。下文的集成 controller/receiver 验证已覆盖联合源捕获和隔离目标路径； +参与者重建不授予新的工作或目标执行权限。 + +Host profile 还须声明路径可移植性。原生索引可能保留绝对 guest 路径：资格化 +须在新隔离环境保留这些路径，或使用 Host 支持的 import/rebinding API；手改 +原生数据库行不能资格化导入。有效 Host 配置及已安装工具/skill 版本须与凭证 +分开绑定，重新认证仍由既有 owner 负责。完整研究状态是继续有效工作所需的 +已声明参与者，不是复制所有文件或模型不可观测的内部状态。 + +截短历史会话不能从终局版本重建当时的代码、研究状态或反馈消费。没有其他参与者 +同时捕获且已资格化的状态时，这类探针只证明 Host 可行性，不能进入同状态干预比较。 + +第一项实现切片是 [inert 产物捕获](../../../benchmark/README.md#inert-checkpoint-capture)。 +Benchmark adapter 在既有 Turn 结算后请求捕获；typed 产物 owner 封存显式文件/ +目录及已提交 SQLite 快照，复查来源 witness 并验证隔离物化。Manifest 始终声明 +研究点尚未资格化。Heartbeat adapter 还支持显式 fresh-replan 触发：预加载原 +quota guard，在模型调用前捕获 wake 开始已投影的准入 replan，并按既有义务 +去重；尚不覆盖运行中 Host wake 首次产生的 replan。可选 native 参与者通过 +Codex 读取完整历史、核对会话真实工作区及当前 source binding/profile，并在字节 probe 后复查。读取 +到 probe 之间出现的后续 source Turn 会拒绝；之后的写入由产物 witness 校验 +拦截。它封存 rollout/索引/分页库并验证固定 guest 路径下的原生分叉；来源基准 +校验不等于研究收益。显式 source-checkout controller 现可通过既有 registry/ +session owner 准备已验证 fork，要求不同且已注册的 Goal/Agent 身份与匹配的 +捕获 session profile。Registry fence 保护配置复查到绑定写入，保持原有 profile +的 lifetime 语义；lifecycle-only source-session profile 不能运行 governed 工作， +会被拒绝而非当作 benchmark 替代路径。已有绑定、配置/profile 变化和原生自动 +Goal 激活均会拒绝。真实 native CLI/resume 资格化通过本地静态 provider，将保留 +会话传入新 Turn，实际验证物化代码与负证据，并按原身份结算 LoopX guard/ +writeback/spend。合成源静止后交换固定 guest 路径；完整环境隔离仍未资格化。 +这证明会话参与者采用与合成接续，不等于完整研究/ +effect 采用、剩余总预算、已安装前端恢复或科学收益。显式 checkpoint-aware native judge 还可导出已公开结果明细、受理计数/ +时钟、剩余冷却及原试验期限，并封存与已公开结果绑定的原生选优 incumbent +原始 archive。它可能早于工作区候选；排序仍由原生规则决定。缺失源码、捕获 +绑定变化或文件 witness 不符均在发布前拒绝；不评测、不补发机会，未完成或 +未对账评测会拒绝。 +可选 LoopX-state 参与者还封存 promoted provider 同一 revision 的完整分页 +canonical Todo 读取，并复查字节和 revision。同时保留完整 Explore 事件、索引中的 +vision/checkpoint 历史及原 stride 决策，供后续读取原分支、负证据与路线判断。 +Explore 行仍由原 owner 校验；不完整追加、坏行、源文件 digest/缺失状态与封存 +inventory 不一致均拒绝可选捕获。历史记录不激活目标 graph、旧 vision 或 effect。 +未 promoted 的来源仍只保存原有 +惰性 inventory，明确不支持 canonical context 采用。控制器通过既有 leased +continuation prepare/adopt owner，将保留上下文挂到已被目标 Agent 持有的 Todo， +读回 canonical note 与 Markdown 投影。原始 Todo 集合、note 和负证据仅作为私有 +上下文；目标 requirements、status、claim 和 lease 仍由当前 authority 决定。 +prepare/adopt 前均复查 native profile 与完整历史。真实 native fork/resume、CLI +和 canonical owner 的合成资格测试实际读取保留候选、Todo/Explore 负证据与 vision，排除来源后续证据， +并完成当前 governed 工作结算。本地确定性 provider 只证明传输、采用及机械使用， +不证明研究判断或收益。context 命令在响应丢失或投影失败后，复用原 prepare/adopt +身份与 revision 恢复；authority、note、产物变化或 native 历史已推进仍拒绝。 +canonical graph/vision/effect 激活、完整 native/Host/evaluator 一致性、参与者联合 +恢复、packaged 交付和实际有效研究仍待验收。operator metadata 或历史一致不能 +认证参与者/effect 对账与完整新研究状态采用。 + +已提交的 native session 准入现在将捕获产物 digest 与原 session binding 原子保存。 +binding 已提交但控制器响应丢失时，重复原命令会核对原凭据、当前 Goal/Agent/profile +及未推进的完整 native 历史，返回同一 session，不另起 fork 或模型调用。同一会话的 +后续正常观察保留来源，替换 session、profile 或 Goal lifetime 不继承来源。无来源 +标记、外来 binding 或已推进历史均拒绝旧点准入。发送 native fork 前，控制器在 +既有进程锁内持久保存来源产物、目标身份与原 native identity 清单。binding 前 +响应丢失时，只恢复唯一新增且经原生 parent/workspace/完整历史核验的子会话， +不再次发送 fork。结果缺失、多个结果或目标变化继续保持未决。固定版本的原生 +交互列表会隐藏尚无新用户事件的 fork，因此只读 native SQLite index 仅用于定位 +不透明 ID，再由 native API 验证语义；不改写数据库、不从 index 授予权限。 +真实 binding 前后注入响应丢失，再经 CLI 恢复并继续 native governed 工作的 +确定性测试覆盖这一边界，并验证缺失、多结果和外来目标拒绝；evaluator/Host +联合激活恢复仍未资格化。 + +显式 Docker controller 现提供有界的文件写入屏障：暂停自己拥有的运行中 worker， +经 daemon archive 复制声明文件及 phase 时钟，捕获原生 evaluator capsule，恢复 +worker 后再发布惰性联合产物。真实持续写入容器与 native HTTP 验证了文件稳定、 +原机会保留，以及取消、暂停响应丢失和捕获失败后的清理。已有暂停不会被接管或 +释放；恢复失败不发布产物。此项仅关闭手动 provider 的文件写入屏障缺口，未 +资格化进程内存、native cursor、外部 writer、自动策略决策前交接或隔离接续。 + +原生新注册不等于恢复准入。真实原生 Docker 资格测试中，source 已消耗十二次 +中的一次，且冷却仍有效；同额度的新注册却立即受理 `agent-1`，保留十二次未用 +机会。重评已保存 archive 产生新的结果，没有恢复 source 的历史或机会状态。 +这是合法的新试验行为，不能代替同一研究点接续。Evaluator owner 须另行资格化 +从已绑定历史/计数及剩余冷却时长初始化目标状态，同时保留 Host 剩余总预算； +复制旧 token 行不是准入证明。 + +[Controller-only native capsule transport](../../../benchmark/README.md#controller-only-native-capsule-transport) +已将提议中的 native initializer 接入既有 typed 产物 owner：显式 controller +权限捕获/导入封存 native bank，再请求新的原生注册。controller 配置不进入 solver +payload 或导出参与者；opaque 内容仍私有且未获隐私认证。此切片保留原生历史、选优 archive、计数及冷却，仍不 +资格化联合 replan barrier、模型/LoopX 状态采用或实际有效研究。其显式 SForge +phase 文件参与者还将声明的 Host deadline 与 native 捕获时间绑定；恢复时重算 +剩余时长，在激活处锚定新的 Host phase 文件，计入传输时间，不计离线等待， +也不重置为新试验的完整预算。既有工作预留和 native 评测机会 owner 不变。 +不确定响应留下不可执行的 phase 预留，不能回退到新预算。显式 controller +恢复现可读回原 native 激活，并沿用首次准入前持久绑定的目标 deadline。 +响应丢失或 phase 部分写入可恢复，不新建注册、不补轮次、不重置冷却或预算； +耗尽与不匹配的尝试拒绝。隔离的真实 native HTTP/Docker 链路已验证恢复、 +普通 Linux phase 入口、合成候选产出及下一次受理评分；不包含模型推理。 +完整 Host 来源/时钟对齐及联合会话/研究/effect 恢复仍须按 profile 资格化， +参与者传输本身不能证明这些结果。 + +自动源捕获已将真实 worker 的 fresh-replan 原始决策、原生 cursor、canonical +研究 inventory 与 controller 的 Docker pause/archive 屏障组合起来。controller +先验证仍有效的请求及准备时的 source witness，再封存 incumbent、评测机会和 +剩余 phase 预算。变化或过期的边界记为未捕获，不能在源恢复后补作原点。 +真实 native CLI/Docker/HTTP 的合成验证已从联合捕获接到全新 Linux guest： +目标绑定原生历史，在当前已租用 Todo 上采用保留上下文，恢复同一 capsule 的 +评测机会与 Host phase 预算。源 pending call 只作历史；目标重新取得 guard 后， +原生工具读取保留代码/上下文、产生合成候选、取得下一次评分，并对目标原 Turn +单次结算。binding 回复丢失、激活 readback 复用原身份，错误源/digest 和已推进 +历史拒绝。此项关闭测试 profile 的机械联合接续缺口；安装版本与实验输入准入、 +packaged 恢复体验、模型判断及 stride 收益仍需各自验证。 + +completed cursor 路径也经真实 worker、native Host、SQLite inventory 与 Docker +controller 验证，捕获发生在模型看见 fresh replan 之前。已提交 guard 尚待选择 +工作时，惰性捕获保留选择前交付禁令，不伪造因果 settlement binding。隔离目标 +在新权限下接续保留代码/上下文；若封存冷却长于本次工作窗口，本地候选工作可 +推进，原生提交仍返回 429，不消耗轮次也不调用 grader。另一已 eligible 的 cursor +用例取得下一次评分。这两种合法状态须保持区分；恢复研究点不让下一次提交立即 eligible。 + +completed 源用例现在通过六个真实 native 工作 wake 及单次 CLI 结算自然触发 +复盘，不插入历史行。stride 开启，Explore planning 分别关闭/开启；先捕获， +再记录三个轴独立计划的 grow 决策，随后完成较大的合成工作切片。新隔离目标 +使用封存代码/上下文并保留未到期冷却。此项验证机械组合，不证明模型判断、 +目标 graph/policy 自动采用或科研收益。当前验证的 Darwin Host 在中断后会丢失 +pending command；延迟重读及仅向主进程发信号均未恢复。因此该 profile 在发信号 +前明确拒绝中途 native 捕获，继续原工具响应。Linux interrupted 与 completed-wake +路径分别保留验收，不放宽历史、歧义、源或期限检查。 + + inert verification unit 不授予执行权。目标侧使用新的实验/Host 身份，通过既有 Goal/session/state admission 进入。历史租约、quota、timer、pending effect 和 submission token 不能变成目标侧权限；未知效果先按原身份 reconcile，再决定 @@ -599,7 +790,9 @@ submission token 不能变成目标侧权限；未知效果先按原身份 recon 研究消融从**同一份**已资格化研究点分成原策略接续与证据驱动范围复盘，使用 等额接续预算与资源，只改变一个控制因素；通过 evaluator owner 保留反馈机会数 与冷却时机。Session 重启、摘要替换、版本升级或策略组合属于额外变量。保留 -所有分支及无效/负结果，比较实际采用的原生成绩、新信息与完整接续成本。结果 +所有分支及无效/负结果。Evaluator 向每个新获授权分支传递剩余冷却和已消费机会， +不能因新身份多得一次立即提交；捕获到准入之间的墙钟等待不会补充 checkpoint +的机会额度，也不会延长接续预算。比较实际采用的原生成绩、新信息与完整接续成本。结果 只资格化这项有范围干预，不证明整个 LoopX 或一个通用最优 stride。 ## 8. 测量模型 diff --git a/loopx/cli_commands/quota.py b/loopx/cli_commands/quota.py index 94a2efaab9..4542035aef 100644 --- a/loopx/cli_commands/quota.py +++ b/loopx/cli_commands/quota.py @@ -5,7 +5,7 @@ from collections.abc import Callable, Mapping from pathlib import Path -from .quota_capture import bind_capture_selection_transport, capture_decision, prepare_decision_capture +from .quota_capture import bind_capture_selection_transport, capture_decision, handoff_decision, prepare_decision_capture from ..usage_goal import observe_quota_result from ..capabilities.explore.composition_frontier import ( @@ -458,6 +458,8 @@ def _emit_quota_result( ) bind_capture_selection_transport(payload, args) capture_decision(capture_directory, payload) + if payload.get("ok") is True: + handoff_decision(capture_directory, getattr(args, "decision_handoff_socket", None)) payload = _project_quota_cli_payload( payload, args, detail_sections, context.scheduler_context if context is not None else None, @@ -502,6 +504,7 @@ def handle_quota_command( if capture_directory is None and ( getattr(args, "decision_output_dir", None) is not None or getattr(args, "decision_output_root", None) is not None + or getattr(args, "decision_handoff_socket", None) is not None ): validate_quota_command_context_request(args) capture_directory = prepare_decision_capture(args) diff --git a/loopx/cli_commands/quota_capture.py b/loopx/cli_commands/quota_capture.py index 5a8832603e..29da4a23fe 100644 --- a/loopx/cli_commands/quota_capture.py +++ b/loopx/cli_commands/quota_capture.py @@ -2,17 +2,26 @@ from __future__ import annotations import argparse +import hashlib +import json import os import shlex +import socket import tempfile from pathlib import Path from ..control_plane.quota.error_codes import QuotaCommandValidationError +from ..control_plane.turn_driver.decision_capture import validate_handoff_socket def prepare_decision_capture(args: argparse.Namespace) -> Path | None: requested = getattr(args, "decision_output_dir", None) root = getattr(args, "decision_output_root", None) + handoff = getattr(args, "decision_handoff_socket", None) + if handoff is not None: + validate_handoff_socket(handoff) + if requested is None and root is None: + raise QuotaCommandValidationError("decision handoff requires an explicit decision capture destination") if requested is None and root is None: return None if requested is not None and root is not None: @@ -76,6 +85,41 @@ def bind_capture_selection_transport(payload: dict, args: argparse.Namespace) -> if not isinstance(template, str): return suffix = " --decision-output-root " + shlex.quote(str(Path(root).expanduser().absolute())) + if getattr(args, "decision_handoff_socket", None): + suffix += " --decision-handoff-socket " + shlex.quote(args.decision_handoff_socket) if getattr(args, "turn_envelope", False): suffix += " --turn-envelope" selection["command_args_template"] = template + suffix + + +def handoff_decision(directory: Path | None, socket_path: str | None) -> None: + """Withhold the original output until the Host releases these exact bytes. + + No second guard, admission, settlement or retry is performed here. A lost + acknowledgement leaves the saved decision available for original-identity + recovery; it must never be silently replaced by a later observation. + """ + if socket_path is None: + return + if directory is None: + raise ValueError("decision handoff requires a captured decision") + path = directory / "decision.json" + raw = path.read_bytes() + digest = hashlib.sha256(raw).hexdigest() + try: + with socket.socket(socket.AF_UNIX, socket.SOCK_STREAM) as channel: + channel.settimeout(120) + channel.connect(str(validate_handoff_socket(socket_path))) + channel.sendall(json.dumps({"path": str(path), "sha256": digest}).encode() + b"\n") + with channel.makefile("rb") as stream: + line = stream.readline(1025) + response = json.loads(line) + if response != {"released": True, "sha256": digest}: + raise ValueError("Host did not release the original decision digest") + if path.read_bytes() != raw: + raise ValueError("captured decision changed during handoff") + except (OSError, ValueError) as exc: + raise RuntimeError( + "Decision handoff failed after guard evaluation; inspect the original " + f"Turn receipt and {path}. Do not blindly rerun the guard." + ) from exc diff --git a/loopx/cli_commands/quota_registration.py b/loopx/cli_commands/quota_registration.py index 5a8027bcb5..ac456517c0 100644 --- a/loopx/cli_commands/quota_registration.py +++ b/loopx/cli_commands/quota_registration.py @@ -191,6 +191,12 @@ def register_quota_command( "--decision-output-root", help="Create a unique private decision directory below this existing root on every guard invocation.", ) + quota_parser.add_argument( + "--decision-handoff-socket", + help=("Opt-in local Host rendezvous before printing a captured should-run decision. " + "Requires a capture destination and a same-user Unix socket in a private directory. " + "Waits at most 120s for a digest-bound release; failure retains the original receipt."), + ) quota_parser.add_argument( "--turn-instance-id", help=( diff --git a/loopx/cli_commands/support_control.py b/loopx/cli_commands/support_control.py index df8ea7a8c6..cdf0135e69 100644 --- a/loopx/cli_commands/support_control.py +++ b/loopx/cli_commands/support_control.py @@ -391,6 +391,7 @@ def handle_support_control_command( turn_granularity=turn_granularity, turn_instance_id=args.turn_instance_id, decision_output_root=args.decision_output_root, + decision_handoff_socket=args.decision_handoff_socket, reward_memory_enabled=reward_memory_enabled, ) if args.bootstrap and payload.get("ok"): diff --git a/loopx/cli_commands/support_control_heartbeat_registration.py b/loopx/cli_commands/support_control_heartbeat_registration.py index 679eaf3ffe..d313de0235 100644 --- a/loopx/cli_commands/support_control_heartbeat_registration.py +++ b/loopx/cli_commands/support_control_heartbeat_registration.py @@ -162,6 +162,11 @@ def register_heartbeat_control_commands( help="Opt into the short TurnEnvelope with full local captures; requires thin mode and an explicit host-owned Turn ID.", ) + heartbeat_prompt_parser.add_argument( + "--decision-handoff-socket", + help="Retain the explicit private Host handoff on generated guards; requires --decision-output-root.", + ) + heartbeat_prequota_parser = subparsers.add_parser( "heartbeat-prequota", help=( diff --git a/loopx/control_plane/configuration_backup.ts b/loopx/control_plane/configuration_backup.ts index 40d8366784..0ee9e2b670 100644 --- a/loopx/control_plane/configuration_backup.ts +++ b/loopx/control_plane/configuration_backup.ts @@ -1,10 +1,10 @@ /** Configuration checkpoint IO. Configuration owners retain validation and activation. */ -import {lstat, mkdir, mkdtemp, readFile, rename, rm, rmdir} from "node:fs/promises"; +import {lstat, mkdir, mkdtemp, readFile, rm} from "node:fs/promises"; import {dirname, isAbsolute, join, resolve} from "node:path"; import type {JsonObject} from "./effect_program.ts"; import {requireJsonObject, requireNonEmptyString} from "./runtime_decode.ts"; import {canonicalAuthorityJson, canonicalAuthoritySha256, hasExactAuthorityKeys} from "./coordination/authority_store_codec.ts"; -import {durableWriteJson} from "./effect_runtime_io.ts"; +import {durableWriteJson, publishNewDirectory} from "./effect_runtime_io.ts"; import {EffectRuntimeConflictError, EffectRuntimeRequestError} from "./effect_runtime_errors.ts"; const SCHEMA = "loopx_configuration_backup_v0"; @@ -88,16 +88,7 @@ export async function restoreConfigurationBackup(value: unknown): Promise name)}; await durableWriteJson(join(staging, "restore-receipt.json"), receipt); - if (process.platform === "win32") { - // Windows refuses a directory rename into an existing destination, - // so the absent-target rename publishes the complete checkpoint. - await rename(staging, destination); - } else { - // POSIX can replace an empty directory; reserve the name exclusively. - await mkdir(destination, {mode: 0o700}); - try { await rename(staging, destination); } - catch (error) { await rmdir(destination); throw error; } - } + await publishNewDirectory(staging, destination); return receipt; } finally { await rm(staging, {recursive: true, force: true}); } } diff --git a/loopx/control_plane/effect_runtime_handlers.ts b/loopx/control_plane/effect_runtime_handlers.ts index d07e0c1b2f..4ec9eb707f 100644 --- a/loopx/control_plane/effect_runtime_handlers.ts +++ b/loopx/control_plane/effect_runtime_handlers.ts @@ -411,6 +411,7 @@ export function createEffectRuntimeHandlers( ["coordination.local_authority.promote", lazyHandler(() => import("./coordination/local_authority_runtime.ts"), ({promoteLocalCoordinationAuthority}) => promoteLocalCoordinationAuthority)], ["coordination.authority_archive.manage", lazyHandler(() => import("./coordination/local_authority_archive.ts"), ({manageLocalAuthorityArchive}) => manageLocalAuthorityArchive)], ["configuration.backup", lazyHandler(() => import("./configuration_backup.ts"), ({configurationBackupOperation}) => configurationBackupOperation)], + ["runtime.checkpoint_artifacts", lazyHandler(() => import("./runtime/checkpoint_artifacts.ts"), ({checkpointArtifacts}) => checkpointArtifacts)], ["coordination.sqlite_backup.snapshot", lazyHandler(() => import("./coordination/sqlite_backup.ts"), ({snapshotSqliteBackup}) => snapshotSqliteBackup)], ["coordination.local_authority.new_goal_storage", lazyHandler(() => import("./coordination/local_authority_defaults.ts"), ({manageNewGoalStorage}) => manageNewGoalStorage)], ["coordination.local_authority.promotion_review", lazyHandler(() => Promise.all([import("./coordination/local_authority_runtime.ts"), import("./coordination/source_transfer.ts")]), ([{reviewLocalCoordinationAuthorityPromotion}, {withCoordinationSourceTransfer}]) => withCoordinationSourceTransfer("coordination.local_authority.promotion_review", reviewLocalCoordinationAuthorityPromotion))], diff --git a/loopx/control_plane/effect_runtime_io.ts b/loopx/control_plane/effect_runtime_io.ts index bf44e1ebcd..a8f4b14dc4 100644 --- a/loopx/control_plane/effect_runtime_io.ts +++ b/loopx/control_plane/effect_runtime_io.ts @@ -8,7 +8,7 @@ import { readSync, writeFileSync, } from "node:fs"; -import { mkdir, open, readFile, rename, rm, stat } from "node:fs/promises"; +import { mkdir, open, readFile, rename, rm, rmdir, stat } from "node:fs/promises"; import { dirname } from "node:path"; import type { JsonObject } from "./effect_program.ts"; @@ -484,6 +484,19 @@ export async function atomicWriteText( await atomicWriteTextFile(path, content); } +/** Publish a complete staged directory without replacing an existing target. + * Callers own source validation, staging cleanup and content durability. + */ +export async function publishNewDirectory(staging: string, destination: string): Promise { + if (process.platform === "win32") await rename(staging, destination); + else { + // POSIX rename can replace an empty directory; reserve the name first. + await mkdir(destination, {mode: 0o700}); + try {await rename(staging, destination);} + catch (error) {await rmdir(destination); throw error;} + } +} + export async function atomicWriteJson( path: string, payload: JsonObject, diff --git a/loopx/control_plane/heartbeat/builder.py b/loopx/control_plane/heartbeat/builder.py index d4bd8da9a1..9e8f35fdba 100644 --- a/loopx/control_plane/heartbeat/builder.py +++ b/loopx/control_plane/heartbeat/builder.py @@ -175,6 +175,7 @@ def _heartbeat_prompt_commands( capability_args: str, turn_identity_arg: str, decision_output_root: Path | None, + decision_handoff_socket: str | None, ) -> dict[str, str | None]: quota_guard_command = render_quota_guard_command( goal_id, @@ -276,6 +277,8 @@ def _heartbeat_prompt_commands( } if decision_output_root is not None: capture_arg = " --decision-output-root " + shlex.quote(str(decision_output_root)) + if decision_handoff_socket is not None: + capture_arg += " --decision-handoff-socket " + shlex.quote(decision_handoff_socket) for key in ("quota_guard_command", "task_body_quota_guard_command"): commands[key] = str(commands[key]) + " --turn-envelope" + capture_arg commands["thin_prompt_command"] = str(commands["thin_prompt_command"]) + capture_arg @@ -309,6 +312,7 @@ def build_heartbeat_prompt( turn_instance_id: str | None = None, reward_memory_enabled: bool = True, decision_output_root: str | Path | None = None, + decision_handoff_socket: str | None = None, ) -> dict[str, Any]: if not (full or compact or brief or thin): thin = True @@ -371,6 +375,7 @@ def build_heartbeat_prompt( thin=thin, full=full, compact=compact, brief=brief, native_goal_host=native_goal_host, turn_instance_id=normalized_turn_instance_id, + decision_handoff_socket=decision_handoff_socket, ) explicit_agent_scopes = normalize_agent_scopes(agent_scopes) if explicit_agent_scopes: @@ -458,6 +463,7 @@ def build_heartbeat_prompt( capability_args=capability_args, turn_identity_arg=turn_identity_arg, decision_output_root=decision_output_root, + decision_handoff_socket=decision_handoff_socket, ) cli_preflight = render_cli_preflight(cli_bin=cli_bin) task_body_renderer = _select_task_body_renderer( diff --git a/loopx/control_plane/heartbeat/host.py b/loopx/control_plane/heartbeat/host.py index 0645b9a22e..afcf51cda3 100644 --- a/loopx/control_plane/heartbeat/host.py +++ b/loopx/control_plane/heartbeat/host.py @@ -29,15 +29,22 @@ def resolve_heartbeat_capture_root( brief: bool, native_goal_host: bool, turn_instance_id: str | None, + decision_handoff_socket: str | None = None, ) -> Path | None: """Validate capture transport for an already resolved host-owned Turn.""" if root is None: + if decision_handoff_socket is not None: + raise ValueError("decision handoff requires an explicit decision capture root") return None if not thin or full or compact or brief or native_goal_host or not turn_instance_id: raise ValueError("--decision-output-root requires thin mode and an explicit host-owned Turn ID") directory = Path(root).expanduser().absolute() if directory.is_symlink() or not directory.is_dir(): raise ValueError("--decision-output-root must be an existing directory, not a symlink") + if decision_handoff_socket is not None: + from ..turn_driver.decision_capture import validate_handoff_socket + + validate_handoff_socket(decision_handoff_socket) return directory diff --git a/loopx/control_plane/heartbeat/task_body.py b/loopx/control_plane/heartbeat/task_body.py index a569f70094..2467fb9225 100644 --- a/loopx/control_plane/heartbeat/task_body.py +++ b/loopx/control_plane/heartbeat/task_body.py @@ -678,8 +678,9 @@ def render_thin_heartbeat_task_body( Read `detail_ref.full_decision` for selection, replan, capability context or missing/truncated commands: same-invocation observation, not fresh admission. Selection uses the saved `interaction_contract.cli_channel.selection_command`. -Keep the guard's --turn-envelope and --decision-output-root on every fresh -selection/repair/replan guard; each call saves its own immutable observation. +Keep all supplied guard transport flags (--turn-envelope, --decision-output-root, +and --decision-handoff-socket when present) on every fresh selection/repair/replan +guard; each call saves its own immutable observation. After a workspace or state change, re-enter the guard before continuing work. ```sh diff --git a/loopx/control_plane/runtime/checkpoint_artifacts.ts b/loopx/control_plane/runtime/checkpoint_artifacts.ts new file mode 100644 index 0000000000..fc799a856e --- /dev/null +++ b/loopx/control_plane/runtime/checkpoint_artifacts.ts @@ -0,0 +1,279 @@ +/** Inert artifact capture for an already-owned source barrier. + * Source stability and byte integrity are not research-point or execution authority. + * Native sessions, canonical state adoption and evaluator opportunities retain owners. + */ +import {createHash} from "node:crypto"; +import {createReadStream} from "node:fs"; +import {copyFile, lstat, mkdir, mkdtemp, open, readFile, readdir, rm} from "node:fs/promises"; +import {dirname, isAbsolute, join, resolve, sep} from "node:path"; +import type {JsonObject} from "../effect_program.ts"; +import {requireJsonObject, requireNonEmptyString} from "../runtime_decode.ts"; +import {canonicalAuthoritySha256, hasExactAuthorityKeys} from "../coordination/authority_store_codec.ts"; +import {snapshotSqliteDatabase} from "../coordination/sqlite_backup.ts"; +import {durableWriteJson, publishNewDirectory} from "../effect_runtime_io.ts"; +import {settlementIdentity} from "../effect_program.ts"; +import {normalizeHeartbeatReplanObligationId, normalizeHeartbeatTodoId} from "../quota/heartbeat_receipt_identity.ts"; +import {EffectRuntimeConflictError} from "../effect_runtime_errors.ts"; +import {EffectiveAction} from "../quota/effective_action.generated.ts"; + +const SCHEMA = "loopx_checkpoint_artifact_set_v0"; +type Component = {id: string; source_path: string}; +type Member = {path: string; kind: "file" | "directory"; mode: number; size?: number; sha256?: string}; + +function absolute(value: unknown, label: string): string { + const path = requireNonEmptyString(value, label); + if (!isAbsolute(path) || resolve(path) !== path) throw new Error(`${label} must be normalized and absolute`); + return path; +} +async function physical(path: string): Promise { + for (let current = path;; current = dirname(current)) { + const entry = await lstat(current); + if (entry.isSymbolicLink()) throw new Error("checkpoint path must not contain symlinks"); + if (current !== path && !entry.isDirectory()) throw new Error("checkpoint ancestor must be a directory"); + if (dirname(current) === current) break; + } +} +function components(value: unknown): Component[] { + if (!Array.isArray(value) || value.length === 0) throw new Error("checkpoint requires explicit components"); + const ids = new Set(); + const result = value.map(raw => { + const entry = requireJsonObject(raw, "checkpoint component"); + if (!hasExactAuthorityKeys(entry, ["id", "source_path"])) throw new Error("invalid checkpoint component fields"); + const id = requireNonEmptyString(entry.id, "checkpoint component id"); + if (!/^[a-z][a-z0-9_-]{0,63}$/.test(id) || ids.has(id)) throw new Error("checkpoint component identity is invalid or duplicated"); + ids.add(id); + return {id, source_path: absolute(entry.source_path, "checkpoint component source")}; + }); + for (const a of result) for (const b of result) { + if (a !== b && (a.source_path === b.source_path || a.source_path.startsWith(b.source_path + sep))) { + throw new Error("checkpoint component sources overlap"); + } + } + return result; +} +async function digest(path: string): Promise { + const hash = createHash("sha256"); + for await (const block of createReadStream(path)) hash.update(block); + return hash.digest("hex"); +} +async function members(root: string, sourceObservation: boolean): Promise { + await physical(root); + const output: Member[] = []; + async function visit(path: string, name: string): Promise { + const stat = await lstat(path); + if (stat.isSymbolicLink()) throw new Error("checkpoint source contains a symlink"); + // SQLite's reader index and an empty WAL contain no committed data. A + // read-only backup can create either on a closed WAL database. Exclude + // only these recognized sidecars, not a nonempty WAL or an explicitly + // requested standalone file. Artifact verification excludes nothing. + if (sourceObservation && name !== "" && stat.isFile() && + (await sqliteSidecar(path, ["-shm"]) || + (stat.size === 0 && await sqliteSidecar(path, ["-wal"])))) return; + if (stat.isDirectory()) { + output.push({path: name, kind: "directory", mode: stat.mode & 0o777}); + for (const child of (await readdir(path)).sort()) await visit(join(path, child), name ? `${name}/${child}` : child); + } else if (stat.isFile()) { + output.push({path: name, kind: "file", mode: stat.mode & 0o777, size: stat.size, sha256: await digest(path)}); + } else throw new Error("checkpoint source contains an unsupported file type"); + } + await visit(root, ""); + if ((await lstat(root)).isFile() && await sqliteFile(root)) { + for (const suffix of sourceObservation ? ["-wal", "-journal"] : ["-wal", "-shm", "-journal"]) { + try {await visit(root + suffix, suffix);} + catch (error) {if ((error as NodeJS.ErrnoException).code !== "ENOENT") throw error;} + } + } + return output; +} +async function witness(entries: Component[], sourceObservation = true): Promise { + const result: JsonObject = {}; + for (const entry of entries) result[entry.id] = await members(entry.source_path, sourceObservation); + return result; +} +function unchanged(a: unknown, b: unknown, message = "checkpoint source moved; retry at a new owned barrier"): void { + if (canonicalAuthoritySha256(a) !== canonicalAuthoritySha256(b)) throw new EffectRuntimeConflictError(message); +} +async function absentDestination(path: string): Promise { + await physical(dirname(path)); + try {await lstat(path); throw new Error("checkpoint destination already exists");} + catch (error) {if ((error as NodeJS.ErrnoException).code !== "ENOENT") throw error;} +} +async function sqliteFile(path: string): Promise { + const handle = await open(path, "r"); + try { + const bytes = Buffer.alloc(16); + const {bytesRead} = await handle.read(bytes, 0, 16, 0); + return bytesRead === 16 && bytes.equals(Buffer.from("SQLite format 3\0")); + } finally {await handle.close();} +} +async function sqliteSidecar(path: string, suffixes: string[]): Promise { + const suffix = suffixes.find(s => path.endsWith(s)); + if (!suffix) return false; + const database = path.slice(0, -suffix.length); + try {return (await lstat(database)).isFile() && await sqliteFile(database);} + catch (error) {if ((error as NodeJS.ErrnoException).code === "ENOENT") return false; throw error;} +} +async function copyComponent(source: string, destination: string, snapshotDatabases = true): Promise { + const stat = await lstat(source); + if (stat.isSymbolicLink()) throw new Error("checkpoint source became a symlink"); + if (stat.isDirectory()) { + await mkdir(destination, {mode: 0o700}); + const children = (await readdir(source)).sort(); + const databases = new Set(); + for (const child of snapshotDatabases ? children : []) { + const path = join(source, child); + if ((await lstat(path)).isFile()) { + if (await sqliteFile(path)) databases.add(child); + } + } + for (const child of children) { + if (["-wal", "-shm", "-journal"].some(suffix => child.endsWith(suffix) && databases.has(child.slice(0, -suffix.length)))) continue; + await copyComponent(join(source, child), join(destination, child), snapshotDatabases); + } + } else if (stat.isFile()) { + // SQLite's existing physical snapshot owner seals committed WAL contents. + if (snapshotDatabases && await sqliteFile(source)) await snapshotSqliteDatabase(source, destination); + else await copyFile(source, destination); + const handle = await open(destination, "r+"); + try {await handle.sync();} finally {await handle.close();} + } else throw new Error("checkpoint source contains an unsupported file type"); +} +function manifestBody(raw: unknown): JsonObject { + const value = requireJsonObject(raw, "checkpoint manifest"); + if (!hasExactAuthorityKeys(value, ["schema_version", "point_metadata", "components", "source_witness", "artifact_witness", "research_point_qualified", "execution_authority_granted", "privacy_certified", "sha256"]) || + value.schema_version !== SCHEMA || value.research_point_qualified !== false || + value.execution_authority_granted !== false || value.privacy_certified !== false) throw new Error("checkpoint manifest fields are invalid"); + const {sha256, ...body} = value; + if (sha256 !== canonicalAuthoritySha256(body)) throw new Error("checkpoint manifest digest mismatch"); + components(value.components); + requireJsonObject(value.point_metadata, "checkpoint point metadata"); + requireJsonObject(value.source_witness, "checkpoint source witness"); + requireJsonObject(value.artifact_witness, "checkpoint artifact witness"); + return value; +} +async function verify(root: string, expected: unknown): Promise { + await physical(root); + await physical(join(root, "checkpoint.json")); + const value = manifestBody(JSON.parse(await readFile(join(root, "checkpoint.json"), "utf8"))); + if (value.sha256 !== expected) throw new Error("checkpoint reviewed digest mismatch"); + const entries = components(value.components); + const actual = await witness(entries.map(c => ({id: c.id, source_path: join(root, "components", c.id)})), false); + unchanged(value.artifact_witness, actual, "checkpoint artifact members differ from the reviewed capture"); + if ((await readdir(root)).sort().join(",") !== "checkpoint.json,components" || + (await readdir(join(root, "components"))).sort().join(",") !== entries.map(c => c.id).sort().join(",")) throw new Error("checkpoint has unexpected members"); + return {ok: true, sha256: value.sha256, integrity_verified: true, research_point_qualified: false, + execution_authority_granted: false, privacy_certified: false}; +} +export async function checkpointArtifacts(value: unknown): Promise { + const request = requireJsonObject(value, "checkpoint artifacts request"); + if (request.action === "select_replan") { + if (!hasExactAuthorityKeys(request, ["action", "quota", "goal_id", "agent_id", "turn_instance_id"])) { + throw new Error("checkpoint replan selection fields are invalid"); + } + const quota = requireJsonObject(request.quota, "checkpoint quota readback"); + const goal = requireNonEmptyString(request.goal_id, "checkpoint goal"); + const agent = requireNonEmptyString(request.agent_id, "checkpoint agent"); + const turn = requireNonEmptyString(request.turn_instance_id, "checkpoint Turn"); + if (quota.ok !== true || quota.goal_id !== goal) throw new Error("checkpoint requires a matching current quota readback"); + if (quota.effective_action !== EffectiveAction.AUTONOMOUS_REPLAN_REQUIRED) { + return {capture: false, reason: "no_fresh_replan", execution_authority_granted: false}; + } + const packet = requireJsonObject(quota.replan_action_packet, "checkpoint replan packet"); + const id = normalizeHeartbeatReplanObligationId(packet.obligation_id); + const receipt = requireJsonObject(quota.heartbeat_receipt, "checkpoint heartbeat receipt"); + if (packet.schema_version !== "replan_action_packet_v0" || packet.decision !== "replan_required" || + !id || (receipt.status !== "committed" && receipt.status !== "replayed") || receipt.turn_instance_id !== turn) { + throw new Error("checkpoint replan does not match the current admitted identity"); + } + const semanticReplan = normalizeHeartbeatReplanObligationId(receipt.semantic_replan_obligation_id); + if (receipt.settlement_identity === undefined && receipt.settlement_binding_owed === true) { + // The original guard owns the semantic node before the Agent chooses + // work. An inert pre-decision capture does not invent that later work's + // causal settlement identity or grant permission to execute it. + const actor = requireJsonObject(quota.agent_identity, "checkpoint source Agent"); + requireNonEmptyString(receipt.event_id, "checkpoint committed guard event"); + if (actor.agent_id !== agent || actor.registered !== true || semanticReplan !== id) { + throw new Error("checkpoint replan does not match the current admitted identity"); + } + return {capture: true, obligation_id: id, execution_authority_granted: false}; + } + const identity = requireJsonObject(receipt.settlement_identity, "checkpoint settlement identity"); + const todo = normalizeHeartbeatTodoId(identity.todo_id); + const boundReplan = normalizeHeartbeatReplanObligationId(identity.replan_obligation_id); + if (identity.goal_id !== goal || identity.agent_id !== agent || identity.turn_instance_id !== turn || + (todo ? boundReplan !== null || semanticReplan !== id : boundReplan !== id) || + identity.effect_id !== settlementIdentity({goal_id: goal, agent_id: agent, todo_id: todo, + replan_obligation_id: boundReplan, turn_instance_id: turn}).effect_id) { + throw new Error("checkpoint replan does not match the current admitted identity"); + } + return {capture: true, obligation_id: id, settlement_effect_id: identity.effect_id, + execution_authority_granted: false}; + } + const fields = request.action === "probe" ? ["action", "components"] : + request.action === "verify" ? ["action", "root", "expected_sha256"] : + request.action === "capture" ? ["action", "components", "destination", "point_metadata", "expected_source_sha256"] : + request.action === "materialize" ? ["action", "root", "expected_sha256", "destination"] : null; + if (fields === null) throw new Error("unsupported checkpoint artifact operation"); + if (!hasExactAuthorityKeys(request, [...fields, ...(request.deadline_epoch_ms === undefined ? [] : ["deadline_epoch_ms"])])) { + throw new Error("checkpoint request fields are invalid"); + } + const deadline = request.deadline_epoch_ms; + if (deadline !== undefined && (typeof deadline !== "number" || !Number.isFinite(deadline))) throw new Error("checkpoint deadline must be finite"); + function withinDeadline(): void { + if (typeof deadline === "number" && Date.now() >= deadline) throw new Error("checkpoint capture window has closed"); + } + withinDeadline(); + if (request.action === "probe") { + const sources = components(request.components); + const observed = await witness(sources); + return {source_witness: observed, source_sha256: canonicalAuthoritySha256(observed)}; + } + if (request.action === "verify") return verify(absolute(request.root, "checkpoint root"), request.expected_sha256); + if (request.action !== "capture" && request.action !== "materialize") throw new Error("unsupported checkpoint artifact operation"); + const destination = absolute(request.destination, "checkpoint destination"); + await absentDestination(destination); + const staging = await mkdtemp(join(dirname(destination), ".loopx-checkpoint-")); + try { + if (request.action === "materialize") { + const root = absolute(request.root, "checkpoint root"); + const verification = await verify(root, request.expected_sha256); + await copyComponent(root, join(staging, "unit"), false); + await verify(join(staging, "unit"), request.expected_sha256); + withinDeadline(); + await publishNewDirectory(join(staging, "unit"), destination); + return {...verification, materialized: true, activation_performed: false}; + } + const sources = components(request.components); + if (sources.some(c => destination === c.source_path || destination.startsWith(c.source_path + sep) || c.source_path.startsWith(destination + sep))) throw new Error("checkpoint output overlaps its source"); + const before = await witness(sources); + if (request.expected_source_sha256 !== canonicalAuthoritySha256(before)) throw new EffectRuntimeConflictError("checkpoint source moved before capture"); + await mkdir(join(staging, "components"), {mode: 0o700}); + for (const c of sources) await copyComponent(c.source_path, join(staging, "components", c.id)); + unchanged(before, await witness(sources)); + const artifacts = await witness(sources.map(c => ({id: c.id, source_path: join(staging, "components", c.id)})), false); + // Byte-copy members must match the reviewed source, including ABA movement + // that could otherwise escape a before/after comparison. SQLite is sealed + // by its existing snapshot owner and has a separate physical witness. + for (const c of sources) { + for (const member of before[c.id] as Member[]) { + if (member.kind !== "file") continue; + const source = (await lstat(c.source_path)).isDirectory() ? join(c.source_path, member.path) : c.source_path + member.path; + if (await sqliteFile(source) || (member.path !== "" && await sqliteSidecar(source, ["-wal", "-shm", "-journal"]))) continue; + const copied = (artifacts[c.id] as Member[]).find(m => m.path === member.path && m.kind === "file"); + if (!copied || copied.sha256 !== member.sha256 || copied.size !== member.size || copied.mode !== member.mode) { + throw new Error("checkpoint copied bytes do not match the reviewed source"); + } + } + } + const body: JsonObject = {schema_version: SCHEMA, components: sources, point_metadata: requireJsonObject(request.point_metadata, "checkpoint point metadata"), + source_witness: before, artifact_witness: artifacts, research_point_qualified: false, + execution_authority_granted: false, privacy_certified: false}; + const manifest = {...body, sha256: canonicalAuthoritySha256(body)}; + await durableWriteJson(join(staging, "checkpoint.json"), manifest); + await verify(staging, manifest.sha256); + unchanged(before, await witness(sources)); + withinDeadline(); + await publishNewDirectory(staging, destination); + return {...await verify(destination, manifest.sha256), source_stability_verified: true}; + } finally {await rm(staging, {recursive: true, force: true});} +} diff --git a/loopx/control_plane/turn_driver/codex_sessions.py b/loopx/control_plane/turn_driver/codex_sessions.py index 7b486ac48d..7381765d09 100644 --- a/loopx/control_plane/turn_driver/codex_sessions.py +++ b/loopx/control_plane/turn_driver/codex_sessions.py @@ -5,6 +5,7 @@ import hashlib import json import os +import re import tempfile from collections.abc import Mapping from pathlib import Path @@ -174,8 +175,28 @@ def _store_codex_cli_session( operation_profile_digest: str | None = None, operation_model: str | None = None, operation_reasoning_effort: str | None = None, + require_absent: bool = False, + checkpoint_artifact_sha256: str | None = None, ) -> None: - with exclusive_file_lock(_session_path(runtime_root, lineage, session_scope)): + if checkpoint_artifact_sha256 is not None and not re.fullmatch(r"[0-9a-f]{64}", checkpoint_artifact_sha256): + raise ValueError("checkpoint session source digest is invalid") + path = _session_path(runtime_root, lineage, session_scope) + with exclusive_file_lock(path): + if require_absent and (path.exists() or path.is_symlink()): + raise ValueError("session binding already exists; read back the original admission") + # Provenance survives ordinary observations of this exact conversation, + # but cannot migrate to a replacement session, profile or Goal lifetime. + prior = load_codex_cli_session(runtime_root, lineage=lineage, session_scope=session_scope) + retained_source = None + if prior and (prior.get("session_id"), prior.get("goal_ref"), prior.get("session_profile_digest")) == ( + session_id, dict(goal_ref) if goal_ref is not None else None, session_profile_digest + ): + retained_source = prior.get("checkpoint_artifact_sha256") + if retained_source is not None: + if not isinstance(retained_source, str) or not re.fullmatch(r"[0-9a-f]{64}", retained_source): + raise ValueError("stored checkpoint session source digest is invalid") + if checkpoint_artifact_sha256 not in (None, retained_source): + raise ValueError("checkpoint session source digest cannot change") _write_codex_cli_session( runtime_root, lineage=lineage, @@ -186,6 +207,7 @@ def _store_codex_cli_session( operation_profile_digest=operation_profile_digest, operation_model=operation_model, operation_reasoning_effort=operation_reasoning_effort, + checkpoint_artifact_sha256=checkpoint_artifact_sha256 or retained_source, ) @@ -200,6 +222,7 @@ def _write_codex_cli_session( operation_profile_digest: str | None = None, operation_model: str | None = None, operation_reasoning_effort: str | None = None, + checkpoint_artifact_sha256: str | None = None, ) -> None: normalized_session_id = _valid_session_id(session_id) if not normalized_session_id: @@ -228,6 +251,8 @@ def _write_codex_cli_session( payload["session_profile_digest"] = session_profile_digest if goal_ref is not None: payload["goal_ref"] = dict(goal_ref) + if checkpoint_artifact_sha256 is not None: + payload["checkpoint_artifact_sha256"] = checkpoint_artifact_sha256 if operation_profile_digest is not None: payload["operation_transport"] = "app-server-operation-tools-v0" payload["operation_profile_digest"] = operation_profile_digest @@ -241,6 +266,9 @@ def _write_codex_cli_session( sort_keys=True, ) handle.write("\n") + if checkpoint_artifact_sha256 is not None: + handle.flush() + os.fsync(handle.fileno()) os.replace(temporary, path) path.chmod(0o600) finally: diff --git a/loopx/control_plane/turn_driver/decision_capture.py b/loopx/control_plane/turn_driver/decision_capture.py index 44f1e7119a..72adf33ff4 100644 --- a/loopx/control_plane/turn_driver/decision_capture.py +++ b/loopx/control_plane/turn_driver/decision_capture.py @@ -3,12 +3,15 @@ import json import os +import socket +import stat import tempfile from collections.abc import Mapping from pathlib import Path from typing import Any from ..quota.turn_envelope import bind_turn_decision_capture +from ..quota.error_codes import QuotaCommandValidationError def write_decision_capture( @@ -64,3 +67,18 @@ def prepare_turn_decision_capture( path.parent.mkdir(parents=True, exist_ok=True, mode=0o700) write_decision_capture(path, observation) return {**plan, "turn_envelope": bound} + + +def validate_handoff_socket(value: str) -> Path: + """A caller-owned local transport, not an authority or admission endpoint.""" + path = Path(value) + try: + if (os.name != "posix" or not hasattr(socket, "AF_UNIX") or not path.is_absolute() or + path.resolve(strict=True) != path or + not stat.S_ISSOCK(path.stat().st_mode) or + path.parent.stat().st_mode & 0o077 or + path.stat().st_uid != os.getuid()): + raise OSError("expected a same-user socket in a private physical directory") + except OSError as exc: + raise QuotaCommandValidationError(f"invalid decision handoff socket: {exc}") from exc + return path diff --git a/tests/control_plane/test_heartbeat_envelope_transport.py b/tests/control_plane/test_heartbeat_envelope_transport.py index be2a6864f5..03d7a4ef8c 100644 --- a/tests/control_plane/test_heartbeat_envelope_transport.py +++ b/tests/control_plane/test_heartbeat_envelope_transport.py @@ -130,3 +130,26 @@ def test_native_worker_and_stable_bootstrap_cannot_use_capture(tmp_path): assert rc != 0 and rejected['ok'] is False assert _heartbeat_receipt_count(runtime, TURN_ID) == 0 assert not list(tmp_path.glob('decision-*')) + + +@pytest.mark.skipif(os.name != 'posix', reason='Requires a private Unix socket') +def test_renderer_retains_owned_handoff_without_enabling_it_by_default(tmp_path): + import socket + import tempfile + args = dict(goal_id=GOAL_ID, agent_id=AGENT_ID, runtime_profile='generic_cli', + turn_instance_id=TURN_ID, thin=True) + assert build_heartbeat_prompt(**args) == build_heartbeat_prompt(**args, decision_handoff_socket=None) + with tempfile.TemporaryDirectory(prefix='hb-handoff-', dir='/tmp') as directory: + endpoint = str(Path(directory).resolve() / 'host.sock') + with socket.socket(socket.AF_UNIX, socket.SOCK_STREAM) as listener: + listener.bind(endpoint) + packet = build_heartbeat_prompt(**args, decision_output_root=tmp_path, + decision_handoff_socket=endpoint) + for field in ('quota_guard_command', 'thin_prompt_command'): + argv = shlex.split(packet[field]) + assert argv[argv.index('--decision-handoff-socket') + 1] == endpoint + assert '--decision-handoff-socket when present' in packet['task_body'] + with pytest.raises(ValueError, match='capture root'): + build_heartbeat_prompt(**args, decision_handoff_socket=endpoint) + with pytest.raises(ValueError, match='invalid decision handoff socket'): + build_heartbeat_prompt(**args, decision_output_root=tmp_path, decision_handoff_socket=endpoint) diff --git a/tests/control_plane/test_quota_decision_capture.py b/tests/control_plane/test_quota_decision_capture.py index 46bbafe613..a3e2ac61cd 100644 --- a/tests/control_plane/test_quota_decision_capture.py +++ b/tests/control_plane/test_quota_decision_capture.py @@ -199,6 +199,108 @@ def test_capture_root_allocates_private_distinct_children(tmp_path): prepare_decision_capture(request(None, decision_output_root=tmp_path, turn_instance_id=None)) +@pytest.mark.skipif(os.name != 'posix', reason='Unix local Host transport') +@pytest.mark.parametrize('release', ['exact', 'wrong_digest', 'disconnect', 'changed_capture']) +def test_real_cli_handoff_preserves_original_receipt(tmp_path, release): + import hashlib + import socket + import sys + import tempfile + from pathlib import Path + + project, runtime, registry = _write_fixture(tmp_path) + _configure_read_only_todo(project) + capture = tmp_path / 'handoff-capture' + with tempfile.TemporaryDirectory(prefix='quota-host-', dir='/tmp') as short: + endpoint = Path(short).resolve() / 'host.sock' + with socket.socket(socket.AF_UNIX, socket.SOCK_STREAM) as listener: + listener.bind(str(endpoint)) + listener.listen(1) + listener.settimeout(20) + command = [sys.executable, '-m', 'loopx.cli', '--format', 'json', + '--registry', str(registry), '--runtime-root', str(runtime), + 'quota', 'should-run', '--goal-id', GOAL_ID, '--agent-id', AGENT_ID, + '--todo-id', TODO_ID, '--turn-instance-id', TURN_ID, + '--runtime-profile', 'generic_cli', '--available-capability', 'shell', + '--available-capability', 'filesystem_read', + '--decision-output-dir', str(capture), '--decision-handoff-socket', str(endpoint)] + with subprocess.Popen(command, stdout=subprocess.PIPE, stderr=subprocess.PIPE, + text=True, env={**os.environ, 'PYTHONPATH': str(Path(__file__).resolve().parents[2])}) as process: + try: + channel, _ = listener.accept() + with channel: + channel.settimeout(5) + with channel.makefile('rb') as stream: + message = json.loads(stream.readline()) + original = (capture / 'decision.json').read_bytes() + digest = hashlib.sha256(original).hexdigest() + assert message == {'path': str(capture / 'decision.json'), 'sha256': digest} + receipts = _heartbeat_receipt_count(runtime, TURN_ID) + assert receipts >= 1 and process.poll() is None + # The actual CLI has persisted admission but cannot expose + # successful output while the Host is taking its snapshot. + import select + assert not select.select([process.stdout], [], [], 0)[0] + if release == 'changed_capture': + (capture / 'decision.json').write_bytes(original + b'\n') + if release != 'disconnect': + channel.sendall(json.dumps({'released': True, 'sha256': ( + '0' * 64 if release == 'wrong_digest' else digest)}).encode() + b'\n') + stdout, stderr = process.communicate(timeout=15) + if release == 'exact': + assert process.returncode == 0 and json.loads(stdout)['ok'] is True + assert not stderr + else: + assert process.returncode != 0 + assert 'Do not blindly rerun' in stderr + stdout + assert _heartbeat_receipt_count(runtime, TURN_ID) == receipts + assert json.loads(original)['heartbeat_receipt']['turn_instance_id'] == TURN_ID + finally: + if process.poll() is None: + process.kill() + process.wait() + + +def test_handoff_requires_explicit_capture_before_guard(tmp_path): + with pytest.raises(QuotaCommandValidationError, match='invalid decision handoff'): + prepare_decision_capture(request(None, decision_handoff_socket=str(tmp_path / 'missing'))) + assert list(tmp_path.iterdir()) == [] + + +@pytest.mark.skipif(os.name != 'posix', reason='Unix local Host transport') +def test_handoff_rejects_shared_or_symlink_endpoint_and_preserves_selection(tmp_path): + import socket + import tempfile + from pathlib import Path + from loopx.cli_commands.quota_capture import validate_handoff_socket, bind_capture_selection_transport + + with tempfile.TemporaryDirectory(prefix='quota-host-', dir='/tmp') as short: + private = Path(short).resolve() + endpoint = private / 'host.sock' + with socket.socket(socket.AF_UNIX, socket.SOCK_STREAM) as listener: + listener.bind(str(endpoint)) + assert validate_handoff_socket(str(endpoint)) == endpoint + alias = private / 'alias.sock' + alias.symlink_to(endpoint) + with pytest.raises(QuotaCommandValidationError): + validate_handoff_socket(str(alias)) + private.chmod(0o755) + try: + with pytest.raises(QuotaCommandValidationError): + validate_handoff_socket(str(endpoint)) + finally: + private.chmod(0o700) + with pytest.raises(QuotaCommandValidationError, match='explicit decision capture'): + prepare_decision_capture(request(None, decision_handoff_socket=str(endpoint))) + payload = {'interaction_contract': {'cli_channel': {'selection_command': { + 'command_args_template': 'quota should-run --todo-id TODO'}}}} + bind_capture_selection_transport(payload, request(None, decision_output_root=tmp_path, + decision_handoff_socket=str(endpoint))) + words = shlex.split(payload['interaction_contract']['cli_channel']['selection_command']['command_args_template']) + assert words[-4:] == ['--decision-output-root', str(tmp_path), '--decision-handoff-socket', str(endpoint)] + assert '--turn-envelope' not in words # Independent from display mode. + + def test_capture_root_accepts_guided_turn_before_identity_is_minted(tmp_path): target = prepare_decision_capture(request( None, diff --git a/tests/control_plane/test_research_checkpoint_capture.py b/tests/control_plane/test_research_checkpoint_capture.py new file mode 100644 index 0000000000..e26b85a551 --- /dev/null +++ b/tests/control_plane/test_research_checkpoint_capture.py @@ -0,0 +1,807 @@ +"""Inert checkpoint capture reuses the real settled-Turn source owner.""" +from pathlib import Path +import json +import hashlib +import os +import pytest +import time + +import test_quota_settlement_cli as cli +from canonical_authority_fixture import isolate_sqlite_runtime + +from benchmark.runtime.checkpoint import capture_after_wake, capture_before_replan +from loopx.control_plane.effect_runtime import effect_runtime_result + + +def test_feature_off_and_failed_wakes_do_not_read_request_or_settlement(monkeypatch): + def forbidden(*args, **kwargs): + raise AssertionError("feature-off/wake-failure must perform no IO") + monkeypatch.setattr("benchmark.runtime.checkpoint.read_heartbeat_settlement", forbidden) + monkeypatch.setattr(Path, "read_bytes", forbidden) + receipt = {"ok": True, "stage": "execute", "mode": "heartbeat"} + capture_after_wake({}, receipt) + assert receipt == {"ok": True, "stage": "execute", "mode": "heartbeat"} + assert capture_before_replan({}, receipt, "original body") == "original body" + failed = {"ok": False, "stage": "execute", "mode": "heartbeat"} + capture_after_wake({"LOOPX_CHECKPOINT_REQUEST": "not-readable"}, failed) + assert failed["ok"] is False + assert failed["checkpoint"]["captured"] is False + + +def test_closed_budget_never_reads_the_request_or_replays_work(monkeypatch): + def forbidden(*args, **kwargs): + raise AssertionError("closed budget must not read sources") + monkeypatch.setattr(Path, "read_bytes", forbidden) + receipt = {"ok": True, "stage": "execute", "mode": "heartbeat"} + capture_after_wake({"LOOPX_CHECKPOINT_REQUEST": "not-readable", + "LOOPX_PHASE_DEADLINE_EPOCH": str(time.time() - 1)}, receipt) + assert receipt["ok"] is True + assert receipt["checkpoint"]["reason"] == "capture_window_closed" + capture_after_wake({"LOOPX_CHECKPOINT_REQUEST": "not-readable", + "LOOPX_PHASE_DEADLINE_EPOCH": "nan"}, receipt) + assert receipt["ok"] is True + assert receipt["checkpoint"]["reason"] == "capture_failed" + + +@pytest.mark.parametrize("trigger", [None, "fresh_replan"]) +@pytest.mark.parametrize("failure", ["missing", "denied", "malformed"]) +def test_worker_keeps_work_running_when_staged_capture_declaration_fails( + tmp_path, monkeypatch, trigger, failure, +): + """Real worker/session/guard; synthetic executable replaces only the model.""" + import os + import sys + from benchmark.runtime.worker import run_once + + isolate_sqlite_runtime(tmp_path, monkeypatch) + project, runtime, registry = cli._write_fixture(tmp_path) + if trigger: + cli._configure_autonomous_replan_fixture(project, runtime, registry) + skills = tmp_path / "skills" + skills.mkdir() + binary = tmp_path / "codex" + binary.write_text(f"#!{sys.executable}\n" + + "import json,pathlib,sys\n" + "body=sys.stdin.read()\n" + "pathlib.Path('host-input.txt').write_text(body)\n" + "pathlib.Path('solution.py').write_text('after_decision = 11\\n')\n" + "print(json.dumps({'type':'thread.started','thread_id':'synthetic-capture-session'}))\n") + binary.chmod(0o755) + logs = tmp_path / "logs" + request = tmp_path / "request.json" + declaration = json.dumps({"destination": str(logs / "checkpoints" / "point"), + "components": [{"id": "code", "source_path": str(project / "solution.py")}], + "point_metadata": {"purpose": "synthetic worker observer failure"}, + **({"trigger": trigger, "destination": str(logs / "checkpoints")} if trigger else {})}) + request.write_text(declaration) + env = {**os.environ, "LOOPX_PROJECT": str(project), "LOOPX_CLI": str(Path(sys.executable).parent / "loopx"), + "LOOPX_CODEX_HOME": str(tmp_path / "home"), "LOOPX_WAKE_LOG_DIR": str(logs / "wakes"), + "LOOPX_SHARED_SKILLS": str(skills), "CODEX_BIN": str(binary), + "MODEL_NAME": "fixture", "REASONING_EFFORT": "high", "OPENAI_BASE_URL": "http://localhost:8123/v1", + "OPENAI_API_KEY": "fixture-key", "LOOPX_RUNTIME_ROOT": str(runtime), + "LOOPX_REGISTRY": str(registry), "LOOPX_GOAL_ID": cli.GOAL_ID, "LOOPX_AGENT_ID": cli.AGENT_ID} + monkeypatch.setattr("benchmark.runtime.worker.heartbeat_body", lambda *a, **kw: "original work contract") + off = run_once(env) + assert off["ok"] and "checkpoint" not in off + (project / "host-input.txt").unlink() + if failure == "missing": + request.unlink() + elif failure == "malformed": + request.write_text("{") + read_bytes = Path.read_bytes + with monkeypatch.context() as faults: + if failure == "denied": + def denied_read(path): + if path == request: + raise PermissionError("synthetic staged-file permission loss") + return read_bytes(path) + faults.setattr(Path, "read_bytes", denied_read) + failed = run_once({**env, "LOOPX_CHECKPOINT_REQUEST": str(request)}) + assert failed["ok"] and failed["session"]["session_id"] == "synthetic-capture-session" + assert (project / "host-input.txt").read_text().endswith("original work contract") + assert failed["checkpoint"]["reason"] == "capture_failed" + assert not failed["checkpoint"]["captured"] + saved = json.loads((logs / "wakes" / failed["turn_id"] / "receipt.json").read_text()) + assert saved == failed + request.write_text(declaration) + if trigger: + # A repaired declaration cannot label the model's later code as the + # original missed pre-decision point, even on exact guard replay. + replay = {"turn_id": failed["turn_id"], "mode": "heartbeat", "stage": "execute", "ok": False} + body = capture_before_replan({**env, "LOOPX_CHECKPOINT_REQUEST": str(request)}, replay, + "original work contract") + assert replay["checkpoint"]["reason"] == "pre_decision_capture_already_attempted", replay + assert body.endswith("original work contract") + assert not list((logs / "checkpoints").glob("replan-*/checkpoint.json")) + recovered = run_once({**env, "LOOPX_CHECKPOINT_REQUEST": str(request)}) + assert recovered["ok"] + if trigger: + # An unavailable declaration is optional; an unavailable original + # admission owner is not. This boundary must still stop before work. + import subprocess + (project / "host-input.txt").unlink() + request.unlink() + registry.rename(registry.with_suffix(".unavailable")) + with pytest.raises(subprocess.CalledProcessError): + run_once({**env, "LOOPX_CHECKPOINT_REQUEST": str(request)}) + assert not (project / "host-input.txt").exists() + + +@pytest.mark.parametrize("opportunity,corrupt,collisions", [ + (False, False, False), (True, False, False), (True, True, False), + (True, False, True), (True, True, True), ("empty", False, True), +]) +def test_real_cli_settlement_capture_verification_and_inert_materialization( + tmp_path, monkeypatch, opportunity, corrupt, collisions): + isolate_sqlite_runtime(tmp_path, monkeypatch) + project, runtime, registry = cli._write_fixture(tmp_path) + binding = ("--agent-id", cli.AGENT_ID, "--todo-id", cli.TODO_ID, "--turn-instance-id", cli.TURN_ID) + def run(*args): + code, result = cli._run_cli(registry, runtime, *args) + assert code == 0, result + return result + run("quota", "should-run", "--goal-id", cli.GOAL_ID, "--codex-app", *binding) + logs = tmp_path / "logs" + wake = logs / "wakes" + wake.mkdir(parents=True) + (wake / cli.TURN_ID).mkdir() + observed = [] + incumbent_bytes = b"opaque older adopted candidate" + opportunity_packet = {"meaning": "synthetic opportunity participant; no admission", "native_incumbent": { + "round": "agent-1", "archive_sha256": hashlib.sha256(incumbent_bytes).hexdigest(), "archive_bytes": len(incumbent_bytes)}} + if opportunity == "empty": + opportunity_packet["native_incumbent"] = None + def export(env, *, timeout, incumbent_path): + assert 0 < timeout <= 30 + observed.append(True) + if opportunity_packet["native_incumbent"] is not None: + incumbent_path.write_bytes(incumbent_bytes) + return opportunity_packet + monkeypatch.setattr("benchmark.runtime.sforge_checkpoint.capture_opportunity", export) + if corrupt: + def altered_probe(method, params, **kwargs): + if params["action"] == "probe": + next(wake.glob("*/checkpoint-incumbent.tar.gz")).write_bytes(b"altered after transport") + return effect_runtime_result(method, params, **kwargs) + monkeypatch.setattr("benchmark.runtime.checkpoint.effect_runtime_result", altered_probe) + (project / "solution.py").write_text("retained_candidate = 7\n") + request = tmp_path / "request.json" + target = logs / "checkpoints" / "point" + caller_components = [] + if collisions: + for name in ("evaluation_opportunity", "evaluation_opportunity_1", "evaluation_incumbent", "evaluation_incumbent_1"): + member = project / name + member.write_text("caller-owned " + name) + caller_components.append({"id": name, "source_path": str(member)}) + request.write_text(json.dumps({"destination": str(target), "components": [ + {"id": "code", "source_path": str(project / "solution.py")}, + {"id": "research", "source_path": str(project / f".codex/goals/{cli.GOAL_ID}/ACTIVE_GOAL_STATE.md")}, + *caller_components, + ], "point_metadata": {"meaning": "synthetic capture; no model/session or evaluator qualification"}, + **({"evaluation_opportunity": True} if opportunity else {})})) + env = {"LOOPX_CHECKPOINT_REQUEST": str(request), "LOOPX_WAKE_LOG_DIR": str(wake), + "LOOPX_RUNTIME_ROOT": str(runtime), "LOOPX_GOAL_ID": cli.GOAL_ID, + "LOOPX_AGENT_ID": cli.AGENT_ID, "LOOPX_REGISTRY": str(registry)} + receipt = {"ok": True, "stage": "execute", "mode": "heartbeat", "turn_id": cli.TURN_ID} + capture_after_wake(env, receipt) + assert receipt["checkpoint"]["reason"] == "owned_turn_not_settled" + assert not target.exists() + assert observed == [] + vision = tmp_path / "vision.json" + vision.write_text(json.dumps({"schema_version": "goal_vision_replan_contract_v0", "state": "vision_patch_proposed", + "vision_patch": {"acceptance_summary": "Synthetic source fixture checked; checkpoint work remains."}, + "path_delta": {"outcome": "continue", "prior_assumption": "Fixture source is available.", + "observed_reality": "The isolated source fixture was verified.", "retained": ["Inert checkpoint qualification"], + "evidence_refs": ["evidence:synthetic-capture"]}})) + run("refresh-state", "--goal-id", cli.GOAL_ID, *binding, "--classification", "checkpoint_fixture_verified", + "--delivery-batch-scale", "multi_surface", "--delivery-outcome", "outcome_progress", "--agent-vision-json", str(vision)) + run("quota", "spend-slot", "--goal-id", cli.GOAL_ID, *binding, "--source", "heartbeat", "--execute") + goal_state = project / f".codex/goals/{cli.GOAL_ID}/ACTIVE_GOAL_STATE.md" + retained_goal = goal_state.read_bytes() + capture_after_wake(env, receipt) + assert receipt["ok"] is True + if corrupt: + assert receipt["checkpoint"]["captured"] is False and not target.exists() + assert "differs" in receipt["checkpoint"]["error"] + assert goal_state.read_bytes() == retained_goal + return + assert receipt["checkpoint"].get("captured") is True, receipt + assert receipt["checkpoint"]["research_point_qualified"] is False + assert goal_state.read_bytes() == retained_goal + if opportunity: + mapping = json.loads((target / "checkpoint.json").read_text())["point_metadata"]["evaluation_opportunity"] + assert json.loads((target / "components" / mapping["component_id"]).read_text()) == opportunity_packet + if opportunity == "empty": + assert mapping["incumbent_component_id"] is None + assert not (wake / cli.TURN_ID / "checkpoint-incumbent.tar.gz").exists() + else: + assert (target / "components" / mapping["incumbent_component_id"]).read_bytes() == incumbent_bytes + for member in caller_components: + assert (target / "components" / member["id"]).read_text() == "caller-owned " + member["id"] + + digest = receipt["checkpoint"]["sha256"] + restored = effect_runtime_result("runtime.checkpoint_artifacts", {"action": "materialize", "root": str(target), + "expected_sha256": digest, "destination": str(tmp_path / "inert")}) + assert restored["activation_performed"] is False + assert (tmp_path / "inert/components/code").read_text() == "retained_candidate = 7\n" + if opportunity: + if opportunity != "empty": + assert (tmp_path / "inert/components" / mapping["incumbent_component_id"]).read_bytes() == incumbent_bytes + for member in caller_components: + assert (tmp_path / "inert/components" / member["id"]).read_text() == "caller-owned " + member["id"] + before = (target / "checkpoint.json").read_bytes() + capture_after_wake(env, receipt) + assert receipt["checkpoint"]["reason"] == "destination_already_present_no_replay" + assert (target / "checkpoint.json").read_bytes() == before + assert goal_state.read_bytes() == retained_goal + assert len(observed) == int(bool(opportunity)) + + +@pytest.mark.skipif(not os.environ.get("LOOPX_NATIVE_CHECKPOINT_BIN"), + reason="Requires an explicit real native Host binary; static local provider only") +@pytest.mark.parametrize("source_moves", [False, True, "binding"]) +@pytest.mark.parametrize("mutation_stage", ["probe", "capture"]) +def test_real_native_cursor_must_match_sealed_store_basis(tmp_path, monkeypatch, source_moves, mutation_stage, + caller_id="code"): + """Actual CLI settlement, native completed Turns and TS SQLite artifacts.""" + from benchmark.tests.test_native_checkpoint import real_native_source + from benchmark.runtime.codex import Execution + from benchmark.runtime.sessions import BenchmarkSessionWake + from loopx.control_plane.turn_driver.codex_sessions import _store_codex_cli_session + + isolate_sqlite_runtime(tmp_path, monkeypatch) + project, runtime, registry = cli._write_fixture(tmp_path) + binding = ("--agent-id", cli.AGENT_ID, "--todo-id", cli.TODO_ID, "--turn-instance-id", cli.TURN_ID) + def run(*args): + code, result = cli._run_cli(registry, runtime, *args) + assert code == 0, result + run("quota", "should-run", "--goal-id", cli.GOAL_ID, "--codex-app", *binding) + vision = tmp_path / "vision.json" + vision.write_text(json.dumps({"schema_version": "goal_vision_replan_contract_v0", "state": "vision_patch_proposed", + "vision_patch": {"acceptance_summary": "Synthetic source boundary; full continuation remains open."}, + "path_delta": {"outcome": "continue", "prior_assumption": "Declared source is available.", + "observed_reality": "The isolated fixture source is checked.", "retained": ["Original capture acceptance"], + "evidence_refs": ["evidence:native-source-fixture"]}})) + run("refresh-state", "--goal-id", cli.GOAL_ID, *binding, "--classification", "native_fixture_verified", + "--delivery-batch-scale", "multi_surface", "--delivery-outcome", "outcome_progress", "--agent-vision-json", str(vision)) + run("quota", "spend-slot", "--goal-id", cli.GOAL_ID, *binding, "--source", "heartbeat", "--execute") + logs = tmp_path / "logs" + wake = logs / "wakes" / cli.TURN_ID + wake.mkdir(parents=True) + destination = logs / "checkpoints/point" + destination.parent.mkdir() + candidate = project / "solution.py" + candidate.write_text("source_candidate = 7\n") + request = tmp_path / "request.json" + request.write_text(json.dumps({"destination": str(destination), "components": [ + {"id": caller_id, "source_path": str(candidate)}], "native_session": True, + "point_metadata": {"meaning": "Native/CLI/TS fixture, not research or model utility"}})) + with real_native_source(tmp_path / "native", os.environ["LOOPX_NATIVE_CHECKPOINT_BIN"], project) as (env, source, complete, calls): + env.update(LOOPX_CHECKPOINT_REQUEST=str(request), LOOPX_WAKE_LOG_DIR=str(logs / "wakes"), + LOOPX_RUNTIME_ROOT=str(runtime), LOOPX_REGISTRY=str(registry), LOOPX_GOAL_ID=cli.GOAL_ID, + LOOPX_AGENT_ID=cli.AGENT_ID, MODEL_NAME="fixture", REASONING_EFFORT="high", + LOOPX_CODEX_SANDBOX="read-only") + source_wake = BenchmarkSessionWake(env, Execution(context="resume", sandbox="read-only"), {}) + source_wake.admission.accept_result(lambda: _store_codex_cli_session(runtime, + lineage={"goal_id": cli.GOAL_ID, "agent_id": cli.AGENT_ID}, session_scope="agent", session_id=source, + goal_ref=source_wake.goal_ref, session_profile_digest=source_wake.digest)) + receipt = {"ok": True, "stage": "execute", "mode": "heartbeat", "turn_id": cli.TURN_ID, + "session": {"session_id": source}} + injected = [] + def between_read_and_probe(method, params, **kwargs): + if source_moves and params["action"] == mutation_stage and not injected: + if source_moves == "binding": + injected.append(True) + _store_codex_cli_session(runtime, lineage={"goal_id": cli.GOAL_ID, "agent_id": cli.AGENT_ID}, + session_scope="agent", session_id=source, goal_ref=source_wake.goal_ref, + session_profile_digest="b" * 64) + else: + injected.append(complete()) # Native writer commits later history before the byte witness. + return effect_runtime_result(method, params, **kwargs) + monkeypatch.setattr("benchmark.runtime.checkpoint.effect_runtime_result", between_read_and_probe) + capture_after_wake(env, receipt) + assert receipt["ok"] and candidate.read_text() == "source_candidate = 7\n" + assert len(calls) == (2 if source_moves is True else 1) # Static local streams only; no implicit new work. + if source_moves: + assert not receipt["checkpoint"]["captured"], receipt + assert not destination.exists() + expected_error = ("checkpoint source moved" if mutation_stage == "capture" else + "profile changed" if source_moves == "binding" else "native") + assert expected_error in receipt["checkpoint"]["error"] + else: + assert receipt["checkpoint"]["captured"], receipt + manifest = json.loads((destination / "checkpoint.json").read_text()) + assert len(manifest["point_metadata"]["native_session"]["turn_ids"]) == 1 + assert (destination / "components" / caller_id).read_bytes() == candidate.read_bytes() + assert receipt["checkpoint"]["research_point_qualified"] is False + + if not source_moves and caller_id.startswith("native_"): + # The source app-server is now closed. Reconstruct through the actual + # provider fork entry, retaining the original home for byte comparison. + # Fixed paths are swapped in this fixture; this is not guest isolation. + import shutil + from benchmark.runtime.native_checkpoint import fork_native_checkpoint + + native = manifest["point_metadata"]["native_session"] + assert all(member["id"] != caller_id for member in native["store_members"]) + config = tmp_path / "native-config.toml" + home = Path(env["CODEX_HOME"]) + config.write_bytes((home / "config.toml").read_bytes()) + retained_home = tmp_path / "retained-native-home" + shutil.move(home, retained_home) + def source_bytes(): + return {p.relative_to(retained_home).as_posix(): p.read_bytes() + for p in retained_home.rglob("*") if p.is_file()} + before = source_bytes() + if caller_id == "native_sessions": + import copy + # Even a correctly sealed artifact cannot select undeclared bytes, + # alias a participant, duplicate a destination or import credentials. + for fault in ("undeclared", "duplicate_id", "duplicate_name", "credential"): + bad_metadata = copy.deepcopy(manifest["point_metadata"]) + members = bad_metadata["native_session"]["store_members"] + if fault == "undeclared": + members[0]["id"] = "not_declared" + elif fault == "duplicate_id": + members[1]["id"] = members[0]["id"] + elif fault == "duplicate_name": + members[1]["name"] = members[0]["name"] + else: + members[0]["name"] = "auth.json" + bad_unit = tmp_path / fault + entries = [{"id": item["id"], "source_path": str(destination / "components" / item["id"])} + for item in manifest["components"]] + witness = effect_runtime_result("runtime.checkpoint_artifacts", {"action": "probe", "components": entries}) + bad = effect_runtime_result("runtime.checkpoint_artifacts", { + "action": "capture", "destination": str(bad_unit), "components": entries, + "point_metadata": bad_metadata, "expected_source_sha256": witness["source_sha256"]}, retry_safe=False) + with pytest.raises(ValueError, match="member inventory is invalid"): + fork_native_checkpoint(env, bad_unit, bad["sha256"], config, timeout=30) + assert not home.exists() and len(calls) == 1 + fork = fork_native_checkpoint(env, destination, receipt["checkpoint"]["sha256"], config, timeout=30) + assert fork["history_verified"] and fork["session_id"] != source + assert fork["visible_history_sha256"] == native["visible_history_sha256"] + assert source_bytes() == before and len(calls) == 1 + assert (destination / "components" / caller_id).read_bytes() == candidate.read_bytes() + + +@pytest.mark.skipif(not os.environ.get("LOOPX_NATIVE_CHECKPOINT_BIN"), reason="Requires a real native Host") +@pytest.mark.parametrize("caller_id", ["native_state_5_sqlite", "native_sessions"]) +@pytest.mark.parametrize("source_moves", [False, True]) +def test_native_store_participants_preserve_caller_ids_and_source_witness( + tmp_path, monkeypatch, caller_id, source_moves): + test_real_native_cursor_must_match_sealed_store_basis( + tmp_path, monkeypatch, source_moves, "probe", caller_id=caller_id) + + +@pytest.mark.skipif(not os.environ.get("LOOPX_NATIVE_CHECKPOINT_BIN"), reason="Requires a real native Host") +def test_native_binding_participant_preserves_legal_caller_id(tmp_path, monkeypatch): + test_real_native_cursor_must_match_sealed_store_basis(tmp_path, monkeypatch, False, "probe", + caller_id="source_session_binding") + + +@pytest.mark.parametrize("capture_fails", [False, True]) +@pytest.mark.parametrize("selection_pending", [False, True]) +def test_real_pre_replan_capture_is_before_decision_and_same_node_is_not_recaptured( + tmp_path, monkeypatch, capture_fails, selection_pending): + import os + import sys + + isolate_sqlite_runtime(tmp_path, monkeypatch) + project, runtime, registry = cli._write_fixture(tmp_path) + binding = ("--agent-id", cli.AGENT_ID, "--todo-id", cli.TODO_ID, "--turn-instance-id", cli.TURN_ID) + + def run(*args): + code, result = cli._run_cli(registry, runtime, *args) + assert code == 0, result + return result + + run("quota", "should-run", "--goal-id", cli.GOAL_ID, "--codex-app", *binding) + vision = tmp_path / "vision.json" + vision.write_text(json.dumps({"schema_version": "goal_vision_replan_contract_v0", "state": "vision_patch_proposed", + "vision_patch": {"acceptance_summary": "Synthetic source fixture checked; capture work remains."}, + "path_delta": {"outcome": "continue", "prior_assumption": "Source fixture is available.", + "observed_reality": "Isolated source fixture verified.", "retained": ["Capture qualification"], + "evidence_refs": ["evidence:synthetic-pre-replan"]}})) + run("refresh-state", "--goal-id", cli.GOAL_ID, *binding, "--classification", "pre_replan_fixture_verified", + "--delivery-batch-scale", "multi_surface", "--delivery-outcome", "outcome_progress", "--agent-vision-json", str(vision)) + run("quota", "spend-slot", "--goal-id", cli.GOAL_ID, *binding, "--source", "heartbeat", "--execute") + index = runtime / "goals" / cli.GOAL_ID / "runs/index.jsonl" + original_settlement = index.read_bytes() + cli._configure_autonomous_replan_fixture(project, runtime, registry) + if selection_pending: + # Keep executable work, as a continuing research worker does. The + # original guard commits the semantic replan before choosing its Todo; + # no settlement identity exists for the new Turn yet. + state_path = project / f".codex/goals/{cli.GOAL_ID}/ACTIVE_GOAL_STATE.md" + state_path.write_text(state_path.read_text().replace("## Agent Todo\n\n", + "## Agent Todo\n\n- [ ] [P1] Continue the retained experiment.\n" + f" \n")) + # This helper supplies synthetic repeated observations; retain the real + # source settlement history instead of replacing its authoritative readback. + index.write_bytes(original_settlement + index.read_bytes()) + logs = tmp_path / "logs" + wakes = logs / "wakes" + prior = wakes / "wake-100-prior" + prior.mkdir(parents=True) + prior_receipt = {"turn_id": cli.TURN_ID, "mode": "heartbeat", "stage": "execute", "ok": True} + (prior / "receipt.json").write_text(json.dumps(prior_receipt)) + solution = project / "solution.py" + solution.write_text("retained_candidate = 7\n") + request = tmp_path / "request.json" + request.write_text(json.dumps({"trigger": "fresh_replan", "loopx_state": True, "destination": str(logs / "checkpoints"), + "components": [{"id": "code", "source_path": str(solution)}], + "point_metadata": {"purpose": "synthetic original settlement and replan ordering"}})) + env = {**os.environ, "LOOPX_CLI": str(Path(sys.executable).parent / "loopx"), + "LOOPX_PROJECT": str(project), "LOOPX_CHECKPOINT_REQUEST": str(request), + "LOOPX_WAKE_LOG_DIR": str(wakes), "LOOPX_RUNTIME_ROOT": str(runtime), + "LOOPX_GOAL_ID": cli.GOAL_ID, "LOOPX_AGENT_ID": cli.AGENT_ID, "LOOPX_REGISTRY": str(registry)} + receipt = {"turn_id": "synthetic-replan-point", "mode": "heartbeat", "stage": "execute", "ok": False} + (wakes / receipt["turn_id"]).mkdir() + attempted = [] + if capture_fails: + def fail_probe(method, params, **kwargs): + if params["action"] == "probe": + attempted.append(True) + raise RuntimeError("synthetic capture failure") + return effect_runtime_result(method, params, **kwargs) + monkeypatch.setattr("benchmark.runtime.checkpoint.effect_runtime_result", fail_probe) + body = capture_before_replan(env, receipt, "original instructions") + guard = json.loads((wakes / receipt["turn_id"] / "checkpoint-start-guard.json").read_text()) + if selection_pending: + assert guard["heartbeat_receipt"]["settlement_binding_owed"] is True + assert "settlement_identity" not in guard["heartbeat_receipt"] + assert guard["interaction_contract"]["agent_channel"]["delivery_allowed"] is False + if capture_fails: + assert receipt["checkpoint"]["reason"] == "capture_failed", receipt + solution.write_text("later_decision = 11\n") + again = {"turn_id": receipt["turn_id"], "mode": "heartbeat", "stage": "execute", "ok": False} + capture_before_replan(env, again, "original instructions") + assert again["checkpoint"]["reason"] == "pre_decision_capture_already_attempted", again + assert attempted == [True] and not list((logs / "checkpoints").glob("replan-*/checkpoint.json")) + assert body.endswith("original instructions") and receipt["ok"] is False + return + assert receipt["checkpoint"]["captured"] is True, json.dumps(receipt) + target = Path(receipt["checkpoint"]["destination"]) + captured = (target / "checkpoint.json").read_bytes() + assert (target / "components/code").read_text() == "retained_candidate = 7\n" + assert body.endswith("original instructions") and "original quota guard once" in body + assert receipt["ok"] is False # Capturing cannot declare the new work completed. + assert (prior / "receipt.json").read_text() == json.dumps(prior_receipt) + # The model's subsequent decision/work must not replace the pre-decision point. + solution.write_text("new_route = 11\n") + capture_after_wake(env, receipt) + assert (target / "checkpoint.json").read_bytes() == captured + # Reentry on the exact original guard reads back one point, even if code moved. + again = {"turn_id": receipt["turn_id"], "mode": "heartbeat", "stage": "execute", "ok": False} + capture_before_replan(env, again, "original instructions") + assert again["checkpoint"]["reason"] == "replan_already_captured", json.dumps(again) + assert (target / "checkpoint.json").read_bytes() == captured + + +@pytest.mark.parametrize('provider', ['file', 'sqlite']) +@pytest.mark.parametrize(("drift", "mutation_stage"), [(None, "probe"), + ("canonical_head", "probe"), ("exported_bytes", "probe"), + ("canonical_head", "capture"), ("exported_bytes", "capture"), + ("research_log", "probe"), ("research_log", "capture"), + ("vision_index", "probe"), ("vision_index", "capture")]) +def test_controller_participant_keeps_canonical_state_and_journals_inert(tmp_path, monkeypatch, provider, drift, + mutation_stage, caller_ids=False): + from canonical_authority_fixture import initialize_canonical_authority + from loopx.control_plane.coordination.runtime_shadow import build_todo_runtime_shadow_projection + from loopx.control_plane.goals.first_party_host_admission import capture_first_party_host_goal_ref + from loopx.control_plane.quota.settlement import read_heartbeat_settlement + import subprocess + + def authority_head(root, advance=False): + module = Path(__file__).resolve().parents[2]/'loopx/control_plane/coordination/local_authority_provider.ts' + script = ('import {openLocalAuthorityStore} from '+json.dumps(module.as_uri())+';' + "let input='';for await(const chunk of process.stdin)input+=chunk;" + 'const x=JSON.parse(input);const store=await openLocalAuthorityStore(x.root,x.goal,{}, {existingOnly:true});' + "if(x.advance){const head=await store.loadAuthority();const result=await store.commitAuthority({" + "expected_provider_revision:head.provider_revision,operation_id:'synthetic-source-observation'," + "events:[],receipts:[],next_projection:head.head});if(result.status!=='applied')throw new Error(JSON.stringify(result));}" + 'process.stdout.write(JSON.stringify(await store.loadAuthority()));') + result = subprocess.run(['node','--no-warnings','--experimental-strip-types','--input-type=module','-e',script], + input=json.dumps({'root':str(root),'goal':cli.GOAL_ID,'advance':advance}),capture_output=True,text=True,timeout=45) + assert result.returncode == 0, result.stderr + return json.loads(result.stdout) + + isolate_sqlite_runtime(tmp_path, monkeypatch) + project, runtime, registry = cli._write_fixture(tmp_path) + source_ref = capture_first_party_host_goal_ref(registry_path=registry, goal_id=cli.GOAL_ID) + state = project / f'.codex/goals/{cli.GOAL_ID}/ACTIVE_GOAL_STATE.md' + code, listed = cli._run_cli(registry, runtime, 'todo', 'list', '--goal-id', cli.GOAL_ID) + assert code == 0, listed + initialize_canonical_authority(runtime, cli.GOAL_ID, + build_todo_runtime_shadow_projection(goal_id=cli.GOAL_ID, todos=listed['todos'], + handoff_mode='hard_lease', leases=[]), + state_path=state, provider=provider) + def run(*args): + code, result = cli._run_cli(registry, runtime, *args) + assert code == 0 and result['ok'] is True, result + return result + lease = run('task-lease', 'acquire', '--goal-id', cli.GOAL_ID, '--todo-id', cli.TODO_ID, + '--owner', cli.AGENT_ID, '--idempotency-key', 'source-lease') + binding = ('--agent-id', cli.AGENT_ID, '--todo-id', cli.TODO_ID, '--turn-instance-id', cli.TURN_ID) + run('quota', 'should-run', '--goal-id', cli.GOAL_ID, '--codex-app', *binding) + vision = tmp_path / 'vision.json' + vision.write_text(json.dumps({'schema_version':'goal_vision_replan_contract_v0', 'state':'vision_patch_proposed', + 'vision_patch':{'acceptance_summary':'Controlled participant probe passed; research continuation remains.'}, + 'path_delta':{'outcome':'continue','prior_assumption':'Capture requires settled original work.', + 'observed_reality':'Isolated source work was verified.', 'retained':['Participant capture'], + 'evidence_refs':['evidence:controller-fixture']}})) + run('refresh-state', '--goal-id', cli.GOAL_ID, *binding, '--classification','controller_fixture_verified', + '--delivery-batch-scale','multi_surface','--delivery-outcome','outcome_progress','--agent-vision-json',str(vision)) + run('quota','spend-slot','--goal-id',cli.GOAL_ID,*binding,'--source','heartbeat','--execute') + before = run('todo','list','--goal-id',cli.GOAL_ID,'--todo-id',cli.TODO_ID) + from loopx.capabilities.explore.result_log import ( + explore_result_log_path, append_explore_result_event, build_explore_node_event, + ) + graph_path = explore_result_log_path(runtime, cli.GOAL_ID) + def append_source_evidence(title): + append_explore_result_event(graph_path, build_explore_node_event(goal_id=cli.GOAL_ID, + title=title, status="dead_end", summary="Fresh validation rejected this route.")) + append_source_evidence("Original rejected route") + original_head = authority_head(runtime) + assert original_head['status'] == 'loaded' + source_settlement = read_heartbeat_settlement(runtime, goal_id=cli.GOAL_ID,agent_id=cli.AGENT_ID, + todo_id=cli.TODO_ID,turn_instance_id=cli.TURN_ID,registry_path=registry) + assert source_settlement.progress['state'] == 'settled' + logs = tmp_path / 'logs' + wakes = logs / 'wakes' + (wakes / cli.TURN_ID).mkdir(parents=True) + (project / 'solver.py').write_text('candidate = 7\n') + target = logs / 'checkpoints/controller' + request = tmp_path / 'request.json' + components = [{'id':'research_context' if caller_ids else 'code','source_path':str(project/'solver.py')}] + if caller_ids: + (project/'notes.txt').write_text('Retained caller-owned context.') + components.append({'id':'loopx_state_2','source_path':str(project/'notes.txt')}) + request.write_text(json.dumps({'destination':str(target), 'loopx_state':True, + 'components':components, 'point_metadata':{}})) + env = {'LOOPX_CHECKPOINT_REQUEST':str(request),'LOOPX_WAKE_LOG_DIR':str(wakes), + 'LOOPX_PROJECT':str(project),'LOOPX_RUNTIME_ROOT':str(runtime),'LOOPX_REGISTRY':str(registry), + 'LOOPX_GOAL_ID':cli.GOAL_ID,'LOOPX_AGENT_ID':cli.AGENT_ID} + receipt = {'ok':True,'stage':'execute','mode':'heartbeat','turn_id':cli.TURN_ID} + if drift: + def changed_context(method, params, **kwargs): + if method == "runtime.checkpoint_artifacts" and params.get("action") == mutation_stage: + if drift == "canonical_head": + # Receipt-only provider commits also move the consistency + # cursor; no new knowledge is needed to invalidate the point. + authority_head(runtime, advance=True) + elif drift == "research_log": + append_source_evidence("Later evidence outside the point") + elif drift == "vision_index": + with (runtime / "goals" / cli.GOAL_ID / "runs/index.jsonl").open("a") as stream: + stream.write(json.dumps({"goal_id": cli.GOAL_ID, "agent_id": cli.AGENT_ID, + "generated_at": "2099-01-01T00:00:00Z", "agent_vision": { + "vision_patch": {"acceptance_summary": "Later source vision"}}}) + "\n") + else: + (wakes/cli.TURN_ID/'checkpoint-research-context.json').write_text('{"later":true}') + return effect_runtime_result(method, params, **kwargs) + monkeypatch.setattr('benchmark.runtime.checkpoint.effect_runtime_result', changed_context) + capture_after_wake(env, receipt) + if drift: + assert not receipt['checkpoint']['captured'] and not target.exists(), receipt + assert ('context' if mutation_stage == 'probe' else 'source') in receipt['checkpoint']['error'] + assert receipt['ok'] is True + return + assert receipt['checkpoint'].get('captured') is True, json.dumps(receipt['checkpoint']) + manifest = json.loads((target/'checkpoint.json').read_text()) + for component in components: + assert (target/'components'/component['id']).read_bytes() == Path(component['source_path']).read_bytes() + declared_context = manifest['point_metadata']['research_context'] + context_root = target/'components'/declared_context['component_id'] + exported_context = context_root/declared_context['member_path'] if declared_context['member_path'] else context_root + assert hashlib.sha256(exported_context.read_bytes()).hexdigest() == declared_context['sha256'] + context = json.loads(exported_context.read_bytes()) + assert context['snapshot']['provider_revision'] == before['authority_read']['provider_revision'] + assert context['goal_id'] == cli.GOAL_ID and 'leases' not in context['snapshot'] + assert context['research_evidence']['explore_events'][0]['title'] == 'Original rejected route' + assert any('Controlled participant probe passed' in json.dumps(row) + for row in context['research_evidence']['vision_history']) + coverage = manifest['point_metadata']['loopx_state']['source_inventory'] + covered = {entry['source_path']:entry['component_id'] for entry in coverage} + assert str(runtime) in covered and str(registry) in covered and str(state) in covered + materialized = tmp_path / 'materialized' + effect_runtime_result('runtime.checkpoint_artifacts', {'action':'materialize','root':str(target), + 'expected_sha256':receipt['checkpoint']['sha256'],'destination':str(materialized)}) + copied_runtime = materialized/'components'/covered[str(runtime)] + assert authority_head(copied_runtime) == original_head + copied_code, copied_todos = cli._run_cli(registry, copied_runtime, 'todo', 'list', + '--goal-id', cli.GOAL_ID, '--todo-id', cli.TODO_ID) + assert copied_code == 0, copied_todos + assert copied_todos['todo'] == before['todo'] + assert copied_todos['authority_read']['source_authority'] == provider + '_v0' + copied = read_heartbeat_settlement(copied_runtime,goal_id=cli.GOAL_ID,agent_id=cli.AGENT_ID, + todo_id=cli.TODO_ID,turn_instance_id=cli.TURN_ID,registry_path=registry) + assert copied.progress['state'] == 'settled' and copied.identity.value == source_settlement.identity.value + registry_copy = materialized/'components'/covered[str(registry)]/manifest['point_metadata']['loopx_state']['registry_member_path'] + assert registry_copy.read_bytes() == registry.read_bytes() + assert manifest['research_point_qualified'] is False and manifest['execution_authority_granted'] is False + assert source_ref is None # Preserve this legacy fixture; capture does not create an exact GoalRef. + assert lease['lease']['status'] == 'active' + after = run('todo','list','--goal-id',cli.GOAL_ID,'--todo-id',cli.TODO_ID) + assert after['todo'] == before['todo'] + assert after['authority_read']['provider_revision'] == before['authority_read']['provider_revision'] + + +@pytest.mark.parametrize('provider', ['file', 'sqlite']) +def test_research_participants_preserve_legal_caller_ids(tmp_path, monkeypatch, provider): + test_controller_participant_keeps_canonical_state_and_journals_inert( + tmp_path, monkeypatch, provider, None, 'probe', caller_ids=True) + + +@pytest.mark.parametrize('gap', ['shared_registry','wrong_actor','wrong_runtime','missing_state','symlink_state','overlap']) +def test_loopx_participant_rejects_wrong_or_ambiguous_routes(tmp_path, gap): + from benchmark.runtime.checkpoint import _loopx_state_components + project, runtime, registry = cli._write_fixture(tmp_path) + runtime.mkdir() + (project/'solver.py').write_text('candidate = 7\n') + env = {'LOOPX_PROJECT':str(project),'LOOPX_RUNTIME_ROOT':str(runtime),'LOOPX_REGISTRY':str(registry), + 'LOOPX_GOAL_ID':cli.GOAL_ID,'LOOPX_AGENT_ID':cli.AGENT_ID} + state = project/f'.codex/goals/{cli.GOAL_ID}/ACTIVE_GOAL_STATE.md' + components = [{'id':'code','source_path':str(project/'solver.py')}] + source = json.loads(registry.read_text()) + if gap == 'shared_registry': + source['goals'].append({**source['goals'][0],'id':'other-goal'}) + registry.write_text(json.dumps(source)) + elif gap == 'wrong_actor': + env['LOOPX_AGENT_ID'] = 'other-agent' + elif gap == 'wrong_runtime': + other = tmp_path/'other' + other.mkdir() + env['LOOPX_RUNTIME_ROOT'] = str(other) + elif gap == 'missing_state': + state.unlink() + elif gap == 'symlink_state': + state.unlink() + state.symlink_to(project/'solver.py') + else: + components.append({'id':'narrow','source_path':str(registry)}) + with pytest.raises(ValueError): + _loopx_state_components(env, components, tmp_path/'logs/checkpoints/point') + assert not (tmp_path/'logs').exists() + + +def test_loopx_participant_preserves_legacy_and_current_owned_goal_directories(tmp_path): + from benchmark.runtime.checkpoint import _loopx_state_components, parse_checkpoint_request + project, runtime, registry = cli._write_fixture(tmp_path) + # These are legitimate controller partitions, not two Goal registrations. + # Inventory follows the backup owner, not a basename-based Goal classifier. + for name in (cli.GOAL_ID, cli.GOAL_ID+'-partition'): + (runtime/'goals'/name).mkdir(parents=True) + components, metadata = _loopx_state_components({'LOOPX_PROJECT':str(project), + 'LOOPX_RUNTIME_ROOT':str(runtime),'LOOPX_REGISTRY':str(registry), + 'LOOPX_GOAL_ID':cli.GOAL_ID,'LOOPX_AGENT_ID':cli.AGENT_ID}, [], tmp_path/'logs/checkpoints/point') + assert sum(entry['source_path'] == str(runtime) for entry in components) == 1 + assert metadata['execution_authority_granted'] is False + with pytest.raises(ValueError, match='explicit boolean'): + parse_checkpoint_request(json.dumps({'destination':'/logs/agent/checkpoints/point', + 'components':[{'id':'code','source_path':'/tmp/solver.py'}], 'point_metadata':{}, + 'loopx_state':'true'}).encode()) + + +def test_failed_replan_index_cannot_relabel_later_work_after_permissions_recover(tmp_path, monkeypatch): + """Real CLI/worker/permissions; only the model executable is synthetic.""" + import os + import sys + from benchmark.runtime.worker import run_once + + if os.geteuid() == 0: + pytest.skip("Real POSIX permission denial requires an unprivileged test user") + isolate_sqlite_runtime(tmp_path, monkeypatch) + project, runtime, registry = cli._write_fixture(tmp_path) + skills = tmp_path / "skills" + skills.mkdir() + binary = tmp_path / "codex" + binary.write_text(f"#!{sys.executable}\n" + + "import json,pathlib,sys\n" + "pathlib.Path('host-input.txt').write_text(sys.stdin.read())\n" + "pathlib.Path('solution.py').write_text('after_strategy_decision = 999\\n')\n" + "print(json.dumps({'type':'thread.started','thread_id':'synthetic-index-session'}))\n") + binary.chmod(0o755) + logs = tmp_path / "logs" + env = {**os.environ, "LOOPX_PROJECT": str(project), "LOOPX_CLI": str(Path(sys.executable).parent / "loopx"), + "LOOPX_CODEX_HOME": str(tmp_path / "home"), "LOOPX_WAKE_LOG_DIR": str(logs / "wakes"), + "LOOPX_SHARED_SKILLS": str(skills), "CODEX_BIN": str(binary), "MODEL_NAME": "fixture", + "REASONING_EFFORT": "high", "OPENAI_BASE_URL": "http://localhost:8123/v1", + "OPENAI_API_KEY": "fixture-key", "LOOPX_RUNTIME_ROOT": str(runtime), + "LOOPX_REGISTRY": str(registry), "LOOPX_GOAL_ID": cli.GOAL_ID, "LOOPX_AGENT_ID": cli.AGENT_ID} + monkeypatch.setattr("benchmark.runtime.worker.heartbeat_body", lambda *a, **kw: "original work contract") + prior = run_once(env) + assert prior["ok"] + + def run(*args): + code, packet = cli._run_cli(registry, runtime, *args) + assert code == 0, packet + return packet + + vision = tmp_path / "vision.json" + vision.write_text(json.dumps({"schema_version": "goal_vision_replan_contract_v0", "state": "vision_patch_proposed", + "vision_patch": {"acceptance_summary": "Synthetic work completed; original checkpoint qualification remains."}, + "path_delta": {"outcome": "continue", "prior_assumption": "The original worker remains available.", + "observed_reality": "Real worker ran the explicitly synthetic host successfully.", + "retained": ["Checkpoint source boundary"], "evidence_refs": ["evidence:index-permission-fixture"]}})) + binding = ("--agent-id", cli.AGENT_ID, "--todo-id", cli.TODO_ID, "--turn-instance-id", prior["turn_id"]) + run("quota", "should-run", "--goal-id", cli.GOAL_ID, "--codex-app", *binding) + run("refresh-state", "--goal-id", cli.GOAL_ID, *binding, "--classification", "worker_fixture_completed", + "--delivery-batch-scale", "multi_surface", "--delivery-outcome", "outcome_progress", "--agent-vision-json", str(vision)) + run("quota", "spend-slot", "--goal-id", cli.GOAL_ID, *binding, "--source", "heartbeat", "--execute") + index = runtime / "goals" / cli.GOAL_ID / "runs/index.jsonl" + original = index.read_bytes() + cli._configure_autonomous_replan_fixture(project, runtime, registry) + index.write_bytes(original + index.read_bytes()) + solution = project / "solution.py" + solution.write_text("retained_candidate = 7\n") + request = tmp_path / "request.json" + request.write_text(json.dumps({"trigger": "fresh_replan", "destination": str(logs / "checkpoints"), + "components": [{"id": "code", "source_path": str(solution)}], + "point_metadata": {"purpose": "Original research boundary, not post-decision code"}})) + env["LOOPX_CHECKPOINT_REQUEST"] = str(request) + attempts = logs / "checkpoints/.replan-attempts" + attempts.mkdir(parents=True) + attempts.chmod(0o500) + try: + failed = run_once(env) + finally: + attempts.chmod(0o700) + assert failed["ok"] + assert failed["checkpoint"]["reason"] == "capture_failed" + assert failed["checkpoint"]["error_kind"] == "PermissionError" + assert not list(attempts.iterdir()) + saved_path = logs / "wakes" / failed["turn_id"] / "receipt.json" + saved = json.loads(saved_path.read_text()) + assert saved == failed + assert solution.read_text() == "after_strategy_decision = 999\n" + replay = {"turn_id": failed["turn_id"], "mode": "heartbeat", "stage": "execute", "ok": False} + body = capture_before_replan(env, replay, "original work contract") + assert body.endswith("original work contract") + assert not replay["checkpoint"]["captured"], replay + assert replay["checkpoint"]["reason"] == "pre_decision_capture_already_attempted", replay + assert not list((logs / "checkpoints").glob("replan-*/checkpoint.json")) + assert json.loads(saved_path.read_text()) == saved + # A new wake on the same outstanding obligation is also a retry, not a + # new original point. The durable guard remains evidence without its index. + again = run_once(env) + assert again["ok"] and again["turn_id"] != failed["turn_id"] + assert not again["checkpoint"]["captured"] + assert again["checkpoint"]["reason"] == "pre_decision_capture_already_attempted" + assert not list((logs / "checkpoints").glob("replan-*/checkpoint.json")) + # Settle that actual admitted review, then supply a separately declared + # later pair of synthetic observations. The real owner must issue a new + # obligation; the lost point must not suppress this distinct boundary. + guard = json.loads((logs / "wakes" / again["turn_id"] / "checkpoint-start-guard.json").read_text()) + old_obligation = guard["replan_action_packet"]["obligation_id"] + replan_binding = ("--agent-id", cli.AGENT_ID, "--replan-obligation-id", old_obligation, + "--turn-instance-id", again["turn_id"]) + data = json.loads(vision.read_text()) + data["vision_patch"]["acceptance_summary"] = "The missed original point stays missed; later research boundaries still qualify separately." + data["path_delta"]["observed_reality"] = "Real permission recovery did not recapture later code as the original research point." + data["path_delta"].update(schema_version="goal_path_delta_v0", outcome="replan") + vision.write_text(json.dumps(data)) + run("refresh-state", "--goal-id", cli.GOAL_ID, *replan_binding, + "--classification", "missed_original_point_preserved", "--delivery-batch-scale", "multi_surface", + "--delivery-outcome", "outcome_progress", "--agent-vision-json", str(vision), + "--autonomous-replan-recorded", "--repair-delta-kind", "goal_vision_patch", + "--progress-result-class", "advanced", "--progress-surface-id", "surface:checkpoint-boundary", + "--progress-hypothesis-id", "hypothesis:missed-index", "--progress-probe-kind", "probe:real-permission", + "--progress-evidence-id", "evidence:index-recovery") + run("quota", "spend-slot", "--goal-id", cli.GOAL_ID, *replan_binding, "--source", "heartbeat", "--execute") + from datetime import datetime, timedelta, timezone + source_history = index.read_bytes() + cli._configure_autonomous_replan_fixture(project, runtime, registry) + observations = [json.loads(line) for line in index.read_text().splitlines()] + observed_at = datetime.now(timezone.utc) + for number, row in enumerate(observations): + row["generated_at"] = (observed_at + timedelta(microseconds=number)).isoformat() + row["progress_observation"].update(surface_id="surface-later", hypothesis_id="hypothesis-later") + Path(row["json_path"]).write_text(json.dumps(row)) + index.write_bytes(source_history + b"".join((json.dumps(row) + "\n").encode() for row in observations)) + fresh = run_once(env) + assert fresh["ok"] and fresh["checkpoint"]["captured"], fresh + point = Path(fresh["checkpoint"]["destination"]) + assert point.name != old_obligation + assert (point / "components/code").read_text() == "after_strategy_decision = 999\n" diff --git a/tests/control_plane/test_research_context_adoption.py b/tests/control_plane/test_research_context_adoption.py new file mode 100644 index 0000000000..da9ef62e3f --- /dev/null +++ b/tests/control_plane/test_research_context_adoption.py @@ -0,0 +1,243 @@ +"""Actual bootstrap/SQLite, artifact, session and leased-continuation owners. + +This stage adopts Todo context, not a native/whole research-state continuation. +No model call or historical lease/effect is activated by these fixtures. +""" +import hashlib +import json +import time +from pathlib import Path + +import pytest + +from loopx.cli import main +from loopx.control_plane.effect_runtime import effect_runtime_result +from benchmark.runtime.research_checkpoint import canonical_context, research_context, verify_context_sources, adopt_todo_context, _encoded +from canonical_authority_fixture import isolate_sqlite_runtime +from loopx.control_plane.turn_driver.codex_sessions import _store_codex_cli_session + + +@pytest.mark.parametrize("lost_response", [None, "prepare", "adopt", "projection"]) +def test_retained_context_is_adopted_only_with_fresh_current_ownership(tmp_path, monkeypatch, capsys, lost_response): + isolate_sqlite_runtime(tmp_path, monkeypatch) + workspace = tmp_path / 'branch' + workspace.mkdir() + def setup(name, agent): + registry, root = tmp_path/name/'registry.json', tmp_path/name/'runtime' + prefix = ['--format','json','--registry',str(registry),'--runtime-root',str(root)] + def run(*args): + code = main([*prefix,*args]) + output = capsys.readouterr().out + assert code == 0, output + return json.loads(output) + run('bootstrap','--project',str(workspace),'--goal-id',name,'--objective','Investigate synthetic candidates.', + '--adapter-kind','read_only_project_map_v0','--adapter-status','connected-read-only','--write-scope','**','--no-global-sync') + run('configure-goal','--goal-id',name,'--registered-agent',agent,'--agent-work-mode',agent+'=active','--execute') + todo = run('todo','add','--goal-id',name,'--role','agent','--claimed-by',agent, + '--task-class','advancement_task','--required-capability','shell','--text','Investigate the retained synthetic candidates.')['todo_id'] + return registry,root,todo,run + source_registry,source_root,source_todo,source_run = setup('source','source-actor') + destination_registry,destination_root,destination_todo,run = setup('branch','branch-actor') + source_env={'LOOPX_RUNTIME_ROOT':str(source_root),'LOOPX_GOAL_ID':'source'} + source_revision=source_run('todo','list','--goal-id','source','--todo-id',source_todo)['authority_read']['provider_revision'] + source_run('task-lease','acquire','--goal-id','source','--todo-id',source_todo,'--owner','source-actor','--idempotency-key','source-lease') + # The evidence includes a rejected approach, with its original status/owner. + source_current=source_run('todo','list','--goal-id','source','--todo-id',source_todo)['authority_read']['provider_revision'] + source_run('todo','update','--goal-id','source','--todo-id',source_todo,'--agent-id','source-actor', + '--note','Candidate 3 failed the synthetic check; preserve candidate 7.', + '--update-operation-id','source-evidence','--update-expected-provider-revision',source_current, + '--task-lease-idempotency-key','source-lease','--task-lease-expected-version','1') + source_context=canonical_context(source_env,deadline=time.time()+30) + assert source_context['snapshot']['provider_revision'] != source_revision + assert 'leases' not in source_context['snapshot'] + raw=_encoded(source_context) + export=tmp_path/'context.json' + export.write_bytes(raw) + unit=tmp_path/'point' + components=[{'id':'research_context','source_path':str(export)}] + probe=effect_runtime_result('runtime.checkpoint_artifacts',{'action':'probe','components':components}) + captured=effect_runtime_result('runtime.checkpoint_artifacts',{'action':'capture','destination':str(unit), + 'components':components,'expected_source_sha256':probe['source_sha256'], + 'point_metadata':{'native_session':{'session_id':'source-session'}, + 'loopx_state':{'goal_id':'source','agent_id':'source-actor'}, + 'research_context':{'component_id':'research_context','member_path':'','sha256':hashlib.sha256(raw).hexdigest(), + 'provider_revision':source_context['snapshot']['provider_revision'],'store_identity':source_context['snapshot']['store_identity']}}}) + digest=captured['sha256'] + env={'LOOPX_REGISTRY':str(destination_registry),'LOOPX_RUNTIME_ROOT':str(destination_root), + 'LOOPX_PROJECT':str(workspace),'LOOPX_GOAL_ID':'branch','LOOPX_AGENT_ID':'branch-actor'} + _store_codex_cli_session(destination_root,lineage={'goal_id':'branch','agent_id':'branch-actor'}, + session_scope='agent',session_id='branch-session',session_profile_digest='synthetic-profile') + # This owner-focused fixture substitutes only native history verification. + # The real Host + CLI + canonical context journey lives in test_native_checkpoint. + monkeypatch.setattr('benchmark.runtime.research_checkpoint._context_session', lambda *_: 'branch-session') + before=run('todo','list','--goal-id','branch','--todo-id',destination_todo) + args=dict(source_todo_id=source_todo,todo_id=destination_todo,operation_id='context-adoption', + expected_revision=before['authority_read']['provider_revision'],lease_key='branch-lease',lease_version=1) + with pytest.raises(ValueError,match='execution authority changed'): + adopt_todo_context(env,unit,digest,**args) + assert not (workspace/'.local/checkpoint-context').exists() + lease=run('task-lease','acquire','--goal-id','branch','--todo-id',destination_todo, + '--owner','branch-actor','--idempotency-key','branch-lease') + assert lease['lease']['owner']=='branch-actor' and lease['lease']['version']==1 + current=run('todo','list','--goal-id','branch','--todo-id',destination_todo) + args['expected_revision']=current['authority_read']['provider_revision'] + with pytest.raises(ValueError,match='absent'): + adopt_todo_context(env,unit,digest,**{**args,'source_todo_id':'missing'}) + with pytest.raises(ValueError,match='execution authority changed'): + adopt_todo_context(env,unit,digest,**{**args,'expected_revision':'stale-revision'}) + outside=tmp_path/'outside' + outside.mkdir() + (workspace/'.local').symlink_to(outside,target_is_directory=True) + with pytest.raises(ValueError,match='symlink'): + adopt_todo_context(env,unit,digest,**args) + assert list(outside.iterdir())==[] + (workspace/'.local').unlink() + source_run('todo','update','--goal-id','source','--todo-id',source_todo,'--agent-id','source-actor', + '--note','Candidate 999 is later evidence excluded from the captured point.', + '--update-operation-id','later-source-evidence','--update-expected-provider-revision',source_context['snapshot']['provider_revision'], + '--task-lease-idempotency-key','source-lease','--task-lease-expected-version','1') + source_before=source_run('todo','list','--goal-id','source','--todo-id',source_todo) + if lost_response: + with monkeypatch.context() as interrupted: + if lost_response == "projection": + def lost_projection(*args, **kwargs): + raise OSError("synthetic projection response lost") + interrupted.setattr("loopx.control_plane.todos.provider_projection.project_current_canonical_todos", lost_projection) + else: + from benchmark.runtime import research_checkpoint + original_handoff = research_checkpoint._handoff + def lost_handoff(env, payload): + result = original_handoff(env, payload) + if payload["action"] == lost_response: + raise OSError("synthetic committed response lost") + return result + interrupted.setattr(research_checkpoint, "_handoff", lost_handoff) + expected_error = ValueError if lost_response == "projection" else OSError + expected_message = "original projection receipt" if lost_response == "projection" else "response lost" + with pytest.raises(expected_error, match=expected_message): + adopt_todo_context(env,unit,digest,**args) + interrupted_todo=run('todo','list','--goal-id','branch','--todo-id',destination_todo) + assert json.loads(interrupted_todo['todo']['note'])['source_session']=='source-session' + # A replacement identity cannot recover the original accepted operation. + with pytest.raises(ValueError, match='continuation rejected'): + adopt_todo_context(env,unit,digest,**{**args,'operation_id':'replacement-operation'}) + assert run('todo','list','--goal-id','branch','--todo-id',destination_todo)['todo']==interrupted_todo['todo'] + result=adopt_todo_context(env,unit,digest,**args) + assert result['context_adopted'] and result['model_calls']==0 and not result['research_point_qualified'] + assert Path(result['artifact']).read_bytes()==raw + after=run('todo','list','--goal-id','branch','--todo-id',destination_todo) + assert after['todo']['claimed_by']=='branch-actor' and after['todo']['status']=='open' + for key,value in current['todo'].items(): + if key not in {'note','updated_at','last_actor_agent_id'}: + assert after['todo'].get(key)==value + note=json.loads(after['todo']['note']) + assert note['source_session']=='source-session' and 'checkpoint:'+digest in note['source_refs'] + inspected=run('handoff','inspect','--goal-id','branch','--todo-id',destination_todo,'--agent-id','branch-actor', + '--session-id','branch-session','--workspace',str(workspace),'--artifact',str(Path(result['artifact']).relative_to(workspace)), + '--task-lease-idempotency-key','branch-lease','--task-lease-expected-version','1') + assert inspected['can_adopt'] and inspected['availability']['ready'] + assert 'source-revision:'+source_context['snapshot']['provider_revision'] in inspected['evidence_refs'] + assert 'Candidate 3 failed' in Path(result['artifact']).read_text() + assert 'Candidate 999' not in Path(result['artifact']).read_text() + assert source_run('todo','list','--goal-id','source','--todo-id',source_todo)==source_before + assert run('task-lease','inspect','--goal-id','branch','--todo-id',destination_todo)['lease']==lease['lease'] + # The adapter does not mint another receipt/session when retried after commit. + replay=adopt_todo_context(env,unit,digest,**args) + assert replay['continuation']['adoption']['status']=='replayed' + assert replay['continuation']['provider_revision']==result['continuation']['provider_revision'] + assert run('todo','list','--goal-id','branch','--todo-id',destination_todo)==after + with pytest.raises(ValueError,match='execution authority changed'): + adopt_todo_context(env,unit,digest,**{**args,'lease_version':999}) + with pytest.raises(ValueError,match='already has context'): + adopt_todo_context(env,unit,digest,**{**args,'operation_id':'different-operation', + 'expected_revision':after['authority_read']['provider_revision']}) + assert run('todo','list','--goal-id','branch','--todo-id',destination_todo)['todo']==after['todo'] + + target=Path(result['artifact']) + target.write_bytes(b'changed retained context') + with pytest.raises(ValueError,match='artifact already differs'): + adopt_todo_context(env,unit,digest,**args) + target.write_bytes(raw) + run('todo','update','--goal-id','branch','--todo-id',destination_todo,'--agent-id','branch-actor', + '--note','New destination evidence must not be replaced by a historical retry.', + '--update-operation-id','new-destination-evidence','--update-expected-provider-revision',after['authority_read']['provider_revision'], + '--task-lease-idempotency-key','branch-lease','--task-lease-expected-version','1') + moved=run('todo','list','--goal-id','branch','--todo-id',destination_todo) + with pytest.raises(ValueError,match='continuation rejected'): + adopt_todo_context(env,unit,digest,**args) + assert run('todo','list','--goal-id','branch','--todo-id',destination_todo)==moved + + +def test_research_history_is_complete_and_bound_to_original_source(tmp_path, monkeypatch, capsys): + from loopx.capabilities.explore.result_log import ( + append_explore_result_events, build_explore_node_event, build_explore_edge_event, + build_explore_finding_event, explore_result_log_path, + ) + isolate_sqlite_runtime(tmp_path, monkeypatch) + registry, root = tmp_path / "registry.json", tmp_path / "runtime" + prefix = ["--format", "json", "--registry", str(registry), "--runtime-root", str(root)] + assert main([*prefix, "bootstrap", "--project", str(tmp_path), "--goal-id", "source", + "--objective", "Investigate synthetic candidates.", "--adapter-kind", "read_only_project_map_v0", + "--adapter-status", "connected-read-only", "--write-scope", "**", "--no-global-sync"]) == 0 + capsys.readouterr() + env = {"LOOPX_RUNTIME_ROOT": str(root), "LOOPX_GOAL_ID": "source"} + log = explore_result_log_path(root, "source") + events = [build_explore_node_event(goal_id="source", node_id="candidate-3", title="Rejected candidate", + status="dead_end", summary="Fresh samples refuted candidate 3."), + build_explore_node_event(goal_id="source", node_id="candidate-7", title="Retained candidate", + status="exploring"), + build_explore_edge_event(goal_id="source", from_node="candidate-7", to_node="candidate-3", + edge_type="refutes")] + # More than the display finding limit: a checkpoint must retain the earliest + # counterexample as well as recent observations, with original topology. + events += [build_explore_finding_event(goal_id="source", node_id="candidate-3", finding_id=f"finding-{i}", + title=f"Negative observation {i}", status="confirmed") for i in range(45)] + append_explore_result_events(log, events, expected_goal_id="source") + index = root / "goals/source/runs/index.jsonl" + # Synthetic persisted source records test transport, not vision judgment. + rows = [{"goal_id": "source", "agent_id": "source-actor", "generated_at": f"2026-01-01T00:00:{i:02d}Z", + "agent_vision": {"vision_patch": {"current_path": f"Complete experiment {i}"}, + "stride_decision": {"original": i}}} for i in range(2)] + index.parent.mkdir(parents=True, exist_ok=True) + index.write_text("".join(json.dumps(row) + "\n" for row in rows)) + captured = research_context(env, deadline=time.time() + 30) + assert captured["research_evidence"]["explore_events"] == events + assert captured["research_evidence"]["vision_history"] == rows + assert "leases" not in captured["snapshot"] + components = [{"id": "runtime", "source_path": str(root)}] + probe = effect_runtime_result("runtime.checkpoint_artifacts", {"action": "probe", "components": components}) + verify_context_sources(captured, components, probe["source_witness"]) + raw = index.read_bytes() + index.write_bytes(raw + json.dumps({**rows[-1], "generated_at": "2026-01-01T01:00:00Z"}).encode() + b"\n") + later = effect_runtime_result("runtime.checkpoint_artifacts", {"action": "probe", "components": components}) + with pytest.raises(ValueError, match="differs from the captured context"): + verify_context_sources(captured, components, later["source_witness"]) + with pytest.raises(ValueError, match="outside the captured inventory"): + verify_context_sources(captured, [], {}) + for bad in (b'{', b'not-json\n', b'[]\n', b'{"goal_id":"foreign"}\n'): + index.write_bytes(raw + bad) + with pytest.raises(ValueError): + research_context(env, deadline=time.time() + 30) + index.write_bytes(raw) + original = log.read_bytes() + log.write_bytes(original + b'not-json\n') + with pytest.raises(ValueError, match="invalid Explore result"): + research_context(env, deadline=time.time() + 30) + log.write_bytes(original) + log.unlink() + absent = research_context(env, deadline=time.time() + 30) + assert absent["research_evidence"]["explore_events"] == [] + log.write_bytes(original) + appeared = effect_runtime_result("runtime.checkpoint_artifacts", {"action": "probe", "components": components}) + with pytest.raises(ValueError, match="appeared before sealing"): + verify_context_sources(absent, components, appeared["source_witness"]) + # An append during the canonical read is also rejected, not silently mixed. + from benchmark.runtime import research_checkpoint + original_read = research_checkpoint.canonical_context + def racing_read(*args, **kwargs): + result = original_read(*args, **kwargs) + index.write_bytes(raw + b'{}\n') + return result + monkeypatch.setattr(research_checkpoint, "canonical_context", racing_read) + with pytest.raises(ValueError, match="moved during the context read"): + research_context(env, deadline=time.time() + 30) diff --git a/tests/control_plane_ts/checkpoint_artifacts.test.ts b/tests/control_plane_ts/checkpoint_artifacts.test.ts new file mode 100644 index 0000000000..89330e678f --- /dev/null +++ b/tests/control_plane_ts/checkpoint_artifacts.test.ts @@ -0,0 +1,234 @@ +import assert from "node:assert/strict"; +import test from "node:test"; +import {chmod, mkdir, mkdtemp, readFile, readdir, realpath, rm, symlink, writeFile} from "node:fs/promises"; +import {tmpdir} from "node:os"; +import {join} from "node:path"; +import {checkpointArtifacts} from "../../loopx/control_plane/runtime/checkpoint_artifacts.ts"; +import {sqliteAuthorityRuntime} from "../../loopx/control_plane/coordination/sqlite_runtime.ts"; + +async function fixture(t: test.TestContext) { + const root = await realpath(await mkdtemp(join(tmpdir(), "checkpoint-artifacts-"))); + t.after(() => rm(root, {recursive: true, force: true})); + await mkdir(join(root, "code")); + await mkdir(join(root, "code/empty")); + await writeFile(join(root, "code/solution.sh"), "#!/bin/sh\nexit 0\n"); + await chmod(join(root, "code/solution.sh"), 0o755); + await writeFile(join(root, "conversation"), "native cursor fixture\n"); + const components = [{id: "code", source_path: join(root, "code")}, {id: "conversation", source_path: join(root, "conversation")}]; + const probe = await checkpointArtifacts({action: "probe", components}); + const request = {action: "capture", components, expected_source_sha256: probe.source_sha256, + destination: join(root, "captured"), point_metadata: {cursor: "completed-native-turn", effects: "historical only"}}; + return {root, request}; +} + +test("capture, verify and inert materialization preserve executable code and empty directories", async t => { + const {root, request} = await fixture(t); + const captured = await checkpointArtifacts(request); + assert.equal(captured.integrity_verified, true); + assert.equal(captured.source_stability_verified, true); + assert.equal(captured.research_point_qualified, false); + assert.equal(captured.execution_authority_granted, false); + assert.equal(captured.privacy_certified, false); + const restored = await checkpointArtifacts({action: "materialize", root: request.destination, + expected_sha256: captured.sha256, destination: join(root, "restored")}); + assert.equal(restored.activation_performed, false); + assert.equal(await readFile(join(root, "restored/components/code/solution.sh"), "utf8"), "#!/bin/sh\nexit 0\n"); + assert.deepEqual(await readdir(join(root, "restored/components/code/empty")), []); + assert.deepEqual(await readFile(join(root, "restored/checkpoint.json")), await readFile(join(root, "captured/checkpoint.json"))); + await assert.rejects(checkpointArtifacts({action: "materialize", root: request.destination, + expected_sha256: captured.sha256, destination: join(root, "restored")}), /already exists/); +}); + +test("changed source, duplicate identities and overlapping sources publish no unit", async t => { + const {root, request} = await fixture(t); + await writeFile(join(root, "conversation"), "new native cursor"); + await assert.rejects(checkpointArtifacts(request), /source moved/); + await assert.rejects(checkpointArtifacts({...request, components: [...request.components, request.components[0]]}), /duplicated/); + await assert.rejects(checkpointArtifacts({action: "probe", components: [request.components[0], {id: "nested", source_path: join(root, "code/empty")}]}), /overlap/); + assert.deepEqual((await readdir(root)).sort(), ["code", "conversation"]); +}); + +test("missing, changed and extra artifact members or foreign expected digest reject", async t => { + const {root, request} = await fixture(t), captured = await checkpointArtifacts(request); + const verify = {action: "verify", root: request.destination, expected_sha256: captured.sha256}; + await assert.rejects(checkpointArtifacts({...verify, expected_sha256: "foreign"}), /digest/); + await writeFile(join(root, "captured/components/conversation"), "tampered"); + await assert.rejects(checkpointArtifacts(verify), /artifact members/); + await writeFile(join(root, "captured/components/conversation"), "native cursor fixture\n"); + await writeFile(join(root, "captured/foreign"), "unreviewed member"); + await assert.rejects(checkpointArtifacts(verify), /unexpected members/); + await rm(join(root, "captured/foreign")); + await rm(join(root, "captured/components/conversation")); + await assert.rejects(checkpointArtifacts(verify), /ENOENT/); + await assert.rejects(checkpointArtifacts({action: "materialize", root: request.destination, + expected_sha256: captured.sha256, destination: join(root, "must-not-exist")})); + assert.ok(!(await readdir(root)).includes("must-not-exist")); +}); + +test("expired or invalid capture windows publish no unit and leave the source intact", async t => { + const {root, request} = await fixture(t); + const bytes = await readFile(join(root, "conversation")); + await assert.rejects(checkpointArtifacts({...request, deadline_epoch_ms: Date.now() - 1}), /window has closed/); + await assert.rejects(checkpointArtifacts({...request, deadline_epoch_ms: Number.NaN}), /finite/); + await assert.rejects(checkpointArtifacts({...request, execution_authority_granted: true}), /fields are invalid/); + assert.deepEqual(await readFile(join(root, "conversation")), bytes); + assert.deepEqual((await readdir(root)).sort(), ["code", "conversation"]); +}); + +test("concurrent capture reserves one destination and cannot replace an existing unit", async t => { + const {root, request} = await fixture(t); + const results = await Promise.allSettled([checkpointArtifacts(request), checkpointArtifacts(request)]); + assert.equal(results.filter(r => r.status === "fulfilled").length, 1); + assert.equal(results.filter(r => r.status === "rejected").length, 1); + assert.deepEqual((await readdir(root)).sort(), ["captured", "code", "conversation"]); + const bytes = await readFile(join(root, "captured/checkpoint.json")); + await assert.rejects(checkpointArtifacts(request), /already exists/); + assert.deepEqual(await readFile(join(root, "captured/checkpoint.json")), bytes); +}); + +test("symlink sources, nested links and redirected destinations reject without publication", async t => { + const {root, request} = await fixture(t); + await symlink(join(root, "code"), join(root, "alias"), process.platform === "win32" ? "junction" : "dir"); + await assert.rejects(checkpointArtifacts({action: "probe", components: [{id: "code", source_path: join(root, "alias")}]}), /symlink/); + await symlink(join(root, "conversation"), join(root, "code/link")); + await assert.rejects(checkpointArtifacts(request), /symlink/); + await assert.rejects(checkpointArtifacts({...request, destination: join(root, "alias/capture")}), /symlink/); +}); + +test("SQLite uses the existing committed WAL snapshot owner and materializes verbatim", async t => { + const {root, request} = await fixture(t); + await mkdir(join(root, "state")); + const {driver} = sqliteAuthorityRuntime(); + const writer = new driver.DatabaseSync(join(root, "state/authority.sqlite")); + try { + writer.exec("PRAGMA journal_mode=WAL; PRAGMA wal_autocheckpoint=0; CREATE TABLE evidence(value TEXT); INSERT INTO evidence VALUES('retained negative result')"); + await writeFile(join(root, "state/notes-wal"), "ordinary retained note"); + const components = [...request.components, {id: "state", source_path: join(root, "state")}]; + const probe = await checkpointArtifacts({action: "probe", components}); + const captured = await checkpointArtifacts({...request, components, expected_source_sha256: probe.source_sha256}); + assert.deepEqual((await readdir(join(root, "captured/components/state"))).sort(), ["authority.sqlite", "notes-wal"]); + await writeFile(join(root, "captured/components/state/authority.sqlite-shm"), "foreign index"); + await assert.rejects(checkpointArtifacts({action: "verify", root: request.destination, + expected_sha256: captured.sha256}), /artifact members/); + await rm(join(root, "captured/components/state/authority.sqlite-shm")); + await checkpointArtifacts({action: "materialize", root: request.destination, + expected_sha256: captured.sha256, destination: join(root, "restored")}); + assert.deepEqual(await readFile(join(root, "restored/components/state/authority.sqlite")), await readFile(join(root, "captured/components/state/authority.sqlite"))); + const copy = new driver.DatabaseSync(join(root, "restored/components/state/authority.sqlite"), {readOnly: true}); + try {assert.equal(copy.prepare("SELECT value FROM evidence").get()?.value, "retained negative result");} + finally {copy.close();} + } finally {writer.close();} +}); + +test("a committed WAL change after the probe invalidates capture, including a standalone database component", async t => { + const {root, request} = await fixture(t); + const path = join(root, "authority.sqlite"), {driver} = sqliteAuthorityRuntime(); + const writer = new driver.DatabaseSync(path); + try { + writer.exec("PRAGMA journal_mode=WAL; PRAGMA wal_autocheckpoint=0; CREATE TABLE evidence(value TEXT)"); + const components = [{id: "state", source_path: path}]; + const probe = await checkpointArtifacts({action: "probe", components}); + writer.exec("INSERT INTO evidence VALUES('new committed evidence')"); + await assert.rejects(checkpointArtifacts({...request, components, expected_source_sha256: probe.source_sha256}), /source moved/); + assert.ok(!(await readdir(root)).includes("captured")); + const stable = await checkpointArtifacts({action: "probe", components}); + await checkpointArtifacts({...request, components, expected_source_sha256: stable.source_sha256}); + const copy = new driver.DatabaseSync(join(root, "captured/components/state"), {readOnly: true}); + try {assert.equal(copy.prepare("SELECT value FROM evidence").get()?.value, "new committed evidence");} + finally {copy.close();} + } finally {writer.close();} +}); + +test("read-only snapshot housekeeping on a closed WAL database is not source movement", async t => { + const {root, request} = await fixture(t), {driver} = sqliteAuthorityRuntime(); + await mkdir(join(root, "state")); + const path = join(root, "state/authority.sqlite"), writer = new driver.DatabaseSync(path); + writer.exec("PRAGMA journal_mode=WAL; CREATE TABLE evidence(value TEXT); INSERT INTO evidence VALUES('retained')"); + writer.close(); + assert.deepEqual(await readdir(join(root, "state")), ["authority.sqlite"]); + const components = [{id: "state", source_path: join(root, "state")}]; + const probe = await checkpointArtifacts({action: "probe", components}); + const captured = await checkpointArtifacts({...request, components, expected_source_sha256: probe.source_sha256}); + assert.equal(captured.source_stability_verified, true); + const after = await checkpointArtifacts({action: "probe", components}); + assert.equal(after.source_sha256, probe.source_sha256); + assert.deepEqual(await readdir(join(root, "captured/components/state")), ["authority.sqlite"]); + const changed = new driver.DatabaseSync(path); + try { + changed.exec("INSERT INTO evidence VALUES('new committed data')"); + await assert.rejects(checkpointArtifacts({...request, components, destination: join(root, "must-not-exist"), + expected_source_sha256: probe.source_sha256}), /source moved/); + } finally {changed.close();} + const copy = new driver.DatabaseSync(join(root, "captured/components/state/authority.sqlite"), {readOnly: true}); + try {assert.equal(copy.prepare("SELECT COUNT(*) AS n FROM evidence").get()?.n, 1);} + finally {copy.close();} +}); + +test("an explicitly selected sidecar remains a byte artifact rather than disappearing from its witness", async t => { + const {root, request} = await fixture(t), {driver} = sqliteAuthorityRuntime(); + const path = join(root, "authority.sqlite"), writer = new driver.DatabaseSync(path); + try { + writer.exec("PRAGMA journal_mode=WAL; CREATE TABLE evidence(value TEXT)"); + const components = [{id: "explicit_index", source_path: path + "-shm"}]; + const probe = await checkpointArtifacts({action: "probe", components}); + assert.equal((probe.source_witness as Record).explicit_index.length, 1); + const captured = await checkpointArtifacts({...request, components, expected_source_sha256: probe.source_sha256}); + assert.equal(captured.integrity_verified, true); + assert.deepEqual(await readFile(join(root, "captured/components/explicit_index")), await readFile(path + "-shm")); + } finally {writer.close();} +}); + +test("replan capture selection requires the original admitted identity and grants no execution", async () => { + const id = "replan-0123456789abcdef", turn = "fixture-turn"; + const identity = {goal_id: "fixture-goal", agent_id: "fixture-agent", turn_instance_id: turn, + replan_obligation_id: id, binding_kind: "autonomous_replan", binding_id: id, + effect_id: `fixture-goal:fixture-agent:autonomous_replan:${id}:${turn}`}; + const quota = {ok: true, goal_id: "fixture-goal", effective_action: "autonomous_replan_required", + replan_action_packet: {schema_version: "replan_action_packet_v0", decision: "replan_required", obligation_id: id}, + heartbeat_receipt: {status: "committed", turn_instance_id: turn, settlement_identity: identity}}; + const request = {action: "select_replan", quota, goal_id: "fixture-goal", agent_id: "fixture-agent", turn_instance_id: turn}; + const selected = await checkpointArtifacts(request); + assert.equal(selected.capture, true); + assert.equal(selected.obligation_id, id); + assert.equal(selected.execution_authority_granted, false); + const todoBound = await checkpointArtifacts({...request, quota: {...quota, heartbeat_receipt: { + ...quota.heartbeat_receipt, semantic_replan_obligation_id: id, settlement_identity: { + goal_id: "fixture-goal", agent_id: "fixture-agent", turn_instance_id: turn, todo_id: "todo_existing", + effect_id: `fixture-goal:fixture-agent:todo_existing:${turn}`}}}}); + assert.equal(todoBound.capture, true); + assert.equal(todoBound.settlement_effect_id, `fixture-goal:fixture-agent:todo_existing:${turn}`); + const ordinary = await checkpointArtifacts({...request, quota: {...quota, effective_action: "normal_run"}}); + assert.equal(ordinary.capture, false); + await assert.rejects(checkpointArtifacts({...request, turn_instance_id: "another-turn"}), /admitted identity/); + await assert.rejects(checkpointArtifacts({...request, agent_id: "another-agent"}), /admitted identity/); + await assert.rejects(checkpointArtifacts({...request, quota: {...quota, replan_action_packet: + {...quota.replan_action_packet, obligation_id: "replan-fedcba9876543210"}}}), /admitted identity/); + await assert.rejects(checkpointArtifacts({...request, quota: {...quota, heartbeat_receipt: + {...quota.heartbeat_receipt, settlement_identity: {...identity, todo_id: "todo_conflict"}}}}), /admitted identity/); + await assert.rejects(checkpointArtifacts({...request, quota: {...quota, ok: false}}), /matching current/); +}); + +test("committed semantic replan can capture before work selection without inventing settlement authority", async () => { + const id = "replan-0123456789abcdef", turn = "fixture-turn"; + const quota = {ok: true, goal_id: "fixture-goal", effective_action: "autonomous_replan_required", + agent_identity: {agent_id: "fixture-agent", registered: true}, + replan_action_packet: {schema_version: "replan_action_packet_v0", decision: "replan_required", obligation_id: id}, + heartbeat_receipt: {status: "committed", turn_instance_id: turn, event_id: "fixture-event", + semantic_replan_obligation_id: id, settlement_binding_owed: true}}; + const request = {action: "select_replan", quota, goal_id: "fixture-goal", agent_id: "fixture-agent", turn_instance_id: turn}; + const before = structuredClone(quota); + assert.deepEqual(await checkpointArtifacts(request), { + capture: true, obligation_id: id, execution_authority_granted: false, + }); + assert.deepEqual(quota, before); + for (const receipt of [ + {...quota.heartbeat_receipt, status: "pending"}, + {...quota.heartbeat_receipt, settlement_binding_owed: false}, + {...quota.heartbeat_receipt, semantic_replan_obligation_id: "replan-fedcba9876543210"}, + {...quota.heartbeat_receipt, turn_instance_id: "another-turn"}, + {...quota.heartbeat_receipt, settlement_identity: null}, + ]) await assert.rejects(checkpointArtifacts({...request, quota: {...quota, heartbeat_receipt: receipt}})); + await assert.rejects(checkpointArtifacts({...request, agent_id: "another-agent"})); + await assert.rejects(checkpointArtifacts({...request, quota: {...quota, + agent_identity: {...quota.agent_identity, registered: false}}})); +});