Compare ABC-VLA and adapted pi0.5 in repeatable bimanual episodes

.github/workflows/ci.yml

@@ -32,7 +32,7 @@ jobs:
           sudo apt-get install -y ffmpeg
       - name: Static checks
         run: |
-          uvx ruff==0.16.7 check src tests scripts
+          uvx ruff==0.16.7 check src tests scripts examples/abc-pi05
           uv run --frozen python -m compileall -q src
           uv run --frozen bandit -q -r src -c pyproject.toml
           node --check frontend/strategy-table.js
@@ -41,6 +41,7 @@ jobs:
           node --check frontend/tooltips.js
           node --check frontend/sample-cases.js
           node --check frontend/benchmarks.js
+          node --check frontend/bimanual.js
           node --check frontend/history.js
           node --check frontend/history-traces.js
           node --check frontend/datasets.js

CHANGELOG.md

@@ -8,6 +8,10 @@ Format: each entry includes the date, a short title, and a description of what c
 
 ## Unreleased
 
+### Compare ABC-VLA and adapted pi0.5 on bimanual episodes
+
+Run compatible policies through repeated ABC simulator episodes using existing Trials, Campaigns, evidence, traces and reports. A guided comparison and matching CLI preserve scene resets, checkpoint identities, task outcomes and policy-call measurements; a separate ABC/XDOF conversion and pi0.5 training recipe prepares adapted checkpoints. Real data conversion and CPU simulation are validated; GPU training and learned-policy comparison remain pending. *(PR #35)*
+
 ### Explore ABC and XDOF in Jupyter
 
 An executable notebook follows a prepared robot demonstration through camera/state inspection, optional ABC-VLA inference, complete simulator episodes and a small fine-tuning exercise. It uses a separate ABC environment and records actual evidence without adding training to the ROVE application. *(PR #33)*

CLAUDE.md

@@ -16,7 +16,7 @@ in [concepts](docs/product/concepts.md) and
 
 ROVE evaluates robotics agent pipelines by running real inference through VLM+VLA+LLM+Agent combinations. It orchestrates a standardized 4-step pipeline (perceive → plan → act → verify) across multiple model combinations in parallel, producing ranked comparisons of success rate, latency, and cost for each agent configuration.
 
-ROVE is inference-only. It does not train, fine-tune, or modify models. It runs them in realistic robotics pipelines and measures the outcomes.
+ROVE's evaluation engine runs models and measures outcomes. Explicit standalone preparation examples may fine-tune a model before evaluation; training never starts implicitly inside a trial or campaign.
 
 - **PyPI package**: `rove-eval`
 - **Python import**: `import rove`
@@ -97,8 +97,10 @@ lifecycle reusable. See [reusable core](docs/product/reusable-evaluation-core.md
 `rove.evaluations` integrations accept structured inputs, run separate candidate
 and grader calls, and share SQLite history, evidence, deadlines and reporting.
 Generic case authoring is currently manifest/CLI based; do not claim the robotics
-case builder or strategy catalog already supports arbitrary domains. ABC rollout
-hosting and Azure/Fabric integrations are outside this pivot.
+case builder or strategy catalog already supports arbitrary domains. The built-in
+`abc-bimanual` executor now hosts ABC simulation episodes through this lifecycle;
+see [ADR-038](docs/architecture/ADR-038-abc-bimanual-episodes.md). Azure/Fabric
+deployment remains future work.
 
 ## Changelog
 
@@ -149,7 +151,7 @@ When creating a PR, always update `CHANGELOG.md`:
 
 ## What NOT to Build
 
-- No model training or fine-tuning (ROVE is inference-only — it consumes models, not produces them)
+- No training inside evaluation workers. Standalone preparation recipes may produce checkpoints which Trials and Campaigns then consume explicitly.
 - No real robot control (simulation only — output is data, not robot motion)
 - No model serving infrastructure (ROVE calls models, does not host them)
 - Keep robotics behavior compatible while reusing Trials/Campaigns for other evaluation domains. Domain-specific inputs and grading belong behind execution adapters; do not build a second campaign engine.

README.md

@@ -29,6 +29,15 @@ For a separate hands-on learning exercise, open the
 run a released VLA in ABC's simulator, and optionally explore fine-tuning.
 Its GPU steps use an independent environment and are not ROVE application features.
 
+The experimental [ABC-VLA versus pi0.5 comparison](docs/product/abc-pi05-comparison.md)
+connects complete bimanual simulator episodes to ROVE's Trials and Campaigns.
+Register compatible checkpoints with `rove bimanual setup`, preview fixed scene
+cases and repeated policy seeds, then run comparisons with preserved measurements
+and evidence. The pi0.5 checkpoint must be adapted to ABC's robot. Local checks
+cover the integration contracts, a real ABC CPU simulation with rendering, and
+LeRobot dataset conversion/readback. GPU inference, fine-tuning and model
+performance remain unverified on the development host.
+
 ![ROVE dashboard](docs/images/rove-welcome.png)
 
 ## Try it locally
@@ -123,7 +132,9 @@ ROVE has a validated timing, inertia and actuator-mapping contract.
 A VLM's success judgment is not measured task success. Stage attribution is a
 heuristic debugging hypothesis, not a demonstrated causal root cause. Perception
 and planning diagnostics do not control VLA actions in the parallel pipeline.
-There is no real closed-loop simulator integration yet; `mock-sim` is a stub.
+The experimental ABC integration adds a separate closed-loop episode path;
+its [validation scope](docs/product/abc-pi05-comparison.md) remains explicit.
+The original image-based pipeline's `mock-sim` remains a stub.
 
 ## Connect your models
 

docs/README.md

@@ -6,6 +6,7 @@
 
 | Read | Purpose |
 | --- | --- |
+| [ABC-VLA versus pi0.5](product/abc-pi05-comparison.md) | Experimental bimanual episode comparison, repeated scene cases, measured outcomes and checkpoint adaptation |
 | [ABC + XDOF Jupyter tutorial](product/abc-jupyter-tutorial.md) | Standalone demonstration-to-policy notebook with optional GPU evaluation and fine-tuning |
 | [Product specification](PRODUCT_SPEC.md) | Core vision, founding personas, campaign model and delivery status |
 | [Concepts](product/concepts.md) | Cases, trials, campaigns, strategies, datasets and provenance |

docs/architecture/ADR-038-abc-bimanual-episodes.md

@@ -0,0 +1,138 @@
+# ADR-038: Run ABC bimanual episodes through the shared evaluation lifecycle
+
+## Status
+
+Accepted. Experimental integration; GPU policy inference, simulator execution and
+pi0.5 adaptation require validation on a suitable host before reporting model
+performance. Local tests validate contracts, failure handling and durable records.
+
+## Context
+
+The user wants to compare ABC-VLA and pi0.5 on the same bimanual robot task and
+measure whether adapting a checkpoint improves the results. The image-based
+ROVE pipeline produces a single action chunk. It cannot establish completion of
+a task requiring repeated observations and actions.
+
+ABC already supplies the simulator and task evaluator. The reusable evaluation
+core from [ADR-036](ADR-036-reusable-evaluation-core.md) already supplies isolated
+trial workers, timeouts, campaign attempts, SQLite records, evidence and reports.
+The standalone notebook in [ADR-037](ADR-037-standalone-abc-notebook.md) remains a
+separate learning exercise; it does not provide the application lifecycle.
+
+## Decision
+
+Add the built-in `abc-bimanual` evaluation adapter. It uses the existing
+`EvaluationAdapter.predict` and `grade` boundary and the existing campaign runner.
+The candidate phase executes a complete episode; the grader interprets recorded
+simulator evidence under the frozen success criterion.
+
+ABC's environment and native VLA run in their pinned runtime. A pi0.5 strategy
+uses a separate LeRobot runtime and compatible checkpoint. Interprocess messages
+carry current observations and native action chunks. They do not turn ROVE's
+application environment into a combined ML dependency installation.
+
+```mermaid
+flowchart TB
+    UI[ROVE browser and CLI] --> P[Host-registered comparison profile]
+    P --> C[Existing trial and campaign services]
+    C --> W[Isolated ROVE evaluation worker]
+    subgraph PG[One owned process group per episode]
+        W --> A[ABC environment and optional ABC-VLA runtime]
+        A <--> L[Optional pi0.5 LeRobot runtime]
+    end
+    W --> S[Existing SQLite trial and campaign journals]
+    W --> E[Existing content-addressed evidence assets]
+    S --> R[Reports, traces and history]
+    E --> R
+    W -. Deadline or cancellation terminates descendants .-> PG
+```
+
+## Preserve case and attempt semantics
+
+A case fixes the task and scene reset. Campaign attempt seeds control policy
+randomness separately. Each policy receives its own post-action observations,
+starting from the same requested scene conditions. A fresh environment per trial
+prevents prior robot state from leaking between policies or repeated episodes.
+
+The environment revision, reset identity, task prompt, action interpretation,
+budget and success mode are part of the comparison conditions. Checkpoint and
+native processor contents identify the strategy. A different checkpoint can be
+compared while keeping those conditions fixed; a changed evaluator or task is
+reported as changed conditions.
+
+ABC's `ever` and `final` success signals are separate measurements. The configured
+signal supplies the binary outcome. Inference, simulated duration and complete
+worker duration remain separate measures. No collision, intervention or physical
+safety measure is inferred merely because the simulator executed.
+
+## Optional evaluation context
+
+Existing installed adapters keep their current `predict` and `grade` signatures.
+An adapter may implement `bind_context(context)` to receive:
+
+- A copied declaration of the frozen environment.
+- A durable, clocked progress emitter.
+- Bounded JSON, binary and local-file artifact recording through managed assets.
+- The identities of currently recorded, integrity-checked evidence.
+
+Core evidence IDs are reserved. An assessment may reference only evidence that
+actually exists and remains available. Private case references are passed to the
+grader separately, not included in the context environment or candidate input.
+This contract separates responsibilities; trusted Python adapters still execute
+on the host and are not a security sandbox.
+
+The resolver fingerprints optional binding source as well as candidate/grader
+source. Runtime fingerprints retain the surrounding ROVE implementation identity.
+The worker records exception type and failed stage without persisting provider
+exception strings that may contain credentials or customer inputs.
+
+## Configuration and process ownership
+
+Operator setup registers local paths and fingerprints outside the browser. Browser
+requests select known strategies, scene seeds, policy seeds and bounded budgets.
+Previews show readiness and planned trial count before execution. No arbitrary
+Python module or executable path is loaded from a browser-supplied manifest.
+
+Descendants inherit the ROVE worker's process group. They must not detach into a
+new session, because the existing timeout/cancellation cleanup kills the owned
+group. Evidence and progress are written before continuing so interruption does
+not remove already-recorded observations. Unfinished trials retain unknown
+outcomes and a distinct execution status.
+
+## Alternatives and consequences
+
+**Extend the image-only VLA adapter to emit a longer chunk.** This would still
+omit observations after actions and would not establish completion. It would also
+misrepresent a single model prediction as an entire episode.
+
+**Add a second ABC-specific campaign runner.** This would duplicate identity,
+resume, cancellation, persistence and statistical reporting. Reusing the generic
+execution boundary keeps Trials and Campaigns consistent.
+
+**Install both ML stacks in ROVE.** This creates dependency conflicts and couples
+the UI/server environment to CUDA and model-specific versions. Separate runtimes
+add process overhead but make ownership and validation explicit.
+
+Fresh model processes are slower than a persistent serving pool. This is an
+acceptable initial tradeoff for reproducibility and bounded resource use.
+Checkpoint loading is therefore visible in worker duration and excluded from
+reported policy-call latency. Throughput optimization requires its own design
+and must preserve episode reset and mutable policy-state isolation.
+
+## Validation and delivery
+
+Local tests cover environment binding, reference separation, managed artifact
+limits, evidence integrity, inherited binding provenance, error/cancellation
+records, report measurement coverage and existing generic executor compatibility.
+The real ABC CPU environment completed a 30-control-step smoke test with rendered
+observations. Conversion and LeRobot readback also passed for two genuine ABC
+training episodes containing 1,870 frames. Neither check ran a learned policy.
+Live GPU validation must demonstrate both policies controlling the same supported
+ABC robot, compatible native preprocessing, matched resets, finite actions,
+complete episode evidence and correct success grading.
+
+GPU inference and training remain unrun. No published-model comparison is
+established until those episodes actually run.
+Fine-tuning remains an explicit preparation activity, not a hidden action triggered
+by evaluation. See the [product workflow](../product/abc-pi05-comparison.md) for
+personas, outcome meanings and the customer's checkpoint-improvement journey.

docs/architecture/README.md

@@ -5,6 +5,9 @@
 [ADR-037: Standalone ABC notebook](ADR-037-standalone-abc-notebook.md) keeps the
 XDOF/ABC learning exercise and its optional training environment separate from ROVE.
 
+[ADR-038: ABC bimanual episodes](ADR-038-abc-bimanual-episodes.md) adds a closed-loop
+executor using the same Trials, Campaigns, evidence and reporting lifecycle.
+
 Use [current implementation](current-implementation.md) to explain the shipped code,
 the [target architecture](target-architecture.md) and [system diagram](system-diagram.md)
 to explain the accepted direction, and [the product specification](../PRODUCT_SPEC.md)