diff --git a/CHANGELOG.md b/CHANGELOG.md index bd25d59..b7f655d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -8,6 +8,10 @@ Format: each entry includes the date, a short title, and a description of what c ## Unreleased +### Choose and sign in to an assistant provider + +Assistant settings and CLI commands now support a dedicated GitHub Copilot sign-in, model discovery, an existing configured endpoint, or a Foundry model deployment using Azure CLI authentication. Saved choices and explicit disablement survive restart; connection tests and workflow confirmations remain separate. Documentation reflects the current trial, campaign and ABC episode boundaries. *(PR #37)* + ### Learn with the GR00T, MuJoCo and MCAP notebook The supplied GR00T N1.7 lab is available alongside the ABC/XDOF notebook in a shared notebook index, with dedicated kernel setup and local artifact guidance. LIBERO task selection follows the chosen suite, and before/after comparisons require distinct checkpoints saved during training. ABC video conversion also supports current FFmpeg; GPU execution remains unvalidated. *(PR #36)* diff --git a/CLAUDE.md b/CLAUDE.md index 5f221ac..3880ea6 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -12,82 +12,72 @@ status and source/test evidence. The product vocabulary and current roadmap are in [concepts](docs/product/concepts.md) and [workflows](docs/product/evaluation-workflows.md). -## Project Overview +## Current project overview -ROVE evaluates robotics agent pipelines by running real inference through VLM+VLA+LLM+Agent combinations. It orchestrates a standardized 4-step pipeline (perceive → plan → act → verify) across multiple model combinations in parallel, producing ranked comparisons of success rate, latency, and cost for each agent configuration. +**ROVE evaluates robotics agent pipelines on your task.** Compare the same task, +observation and relevant robot input across configured strategies. Quick trials +provide immediate outputs; campaigns add repeated attempts, success contracts, +reviews, baselines and versioned improvement history. Execution, outcome acceptance +and physical completion are separate claims. ROVE's evaluation engine runs models and measures outcomes. Explicit standalone preparation examples may fine-tune a model before evaluation; training never starts implicitly inside a trial or campaign. -- **PyPI package**: `rove-eval` -- **Python import**: `import rove` -- **CLI command**: `rove` - -## Architecture Principles - -1. **The API is the product.** CLI, Python library, Jupyter, and dashboard all use the same engine. If you delete `frontend/`, the evaluation system still works. -2. **Adapter pattern via Python Protocols.** `VLMAdapter`, `PolicyAdapter`, `AgentAdapter`, `SimAdapter` are `@runtime_checkable` Protocols. Adding a new model = implement the Protocol + add a YAML config entry. No other code changes. -3. **Configuration over code.** `rove.yaml` is the single source of truth for all model configurations and evaluation definitions. Both backend and frontend read from it. -4. **Deterministic orchestrator.** The 5-step pipeline is fixed. The orchestrator does NOT use an LLM to decide steps. LLMs are only used by adapters. -5. **Azure AI Foundry alignment.** Evaluation data uses Foundry-compatible field names (`query`, `response`, `context`). JSONL export works with `azure-ai-evaluation` SDK. - -## Key Design Decisions - -- **3 MCP servers** (VLM:8081, VLA:8082, Grounding+Sim:8083) — thin wrappers routing to adapters via `model_id`. FastAPI backend calls adapters directly for low latency. MCP servers expose tools externally for Claude, Copilot, and agent frameworks. -- **Hybrid deployment** — Cloud VLMs (Azure OpenAI, NIM, HF), local VLAs (LeRobot, MPS), local grounding (PyTorch/CoreML), local sim (MuJoCo). -- **Mock-first development** — Phase 1 runs entirely on mock adapters. All real model adapters come in Phase 2+. - -## Technology Stack - -### Backend (Python) - -- **FastAPI** — async REST + WebSocket + OpenAPI spec generation -- **Click** — CLI (`rove evaluate`, `rove models`, `rove export`) -- **PyYAML** — model registry configuration -- **SQLite** — evaluation persistence (stdlib) -- **FastMCP v2.x** — MCP server framework -- **Pydantic** — request/response models - -### Frontend (Plain HTML+JS) - -- **Static HTML + vanilla JS** — no React, no npm, no build step -- Served by FastAPI as static files -- Dark theme (`#09090b` background) -- **SSE (Server-Sent Events)** — real-time streaming via `EventSource` - -### ML Libraries (Phase 2+) - -- `torch` (MPS backend), `transformers`, `mlx`, `lerobot`, `mujoco`, `coremltools` - -## Project Structure +A local VLA action chunk is not a closed-loop robot episode; see +[action and episode scope](docs/product/action-and-episode-evaluation.md). + +- **PyPI package:** `rove-eval`; **Python import:** `rove`; **CLI:** `rove`. +- **Shared services:** API and CLI use the same configuration, execution, assessment + and storage services. Keep product behavior out of frontend-only implementations. +- **Adapters:** `VLMAdapter`, `VLAAdapter`, `AgentAdapter`, `SimAdapter` and + `VerifierAdapter` protocols define supported contracts. Compatibility still needs + real tests and cannot be inferred from a provider/model name. +- **Configuration:** `rove.yaml` supplies endpoint and strategy definitions; saved + strategy revisions and campaign snapshots preserve actual versioned selections. +- **Orchestration:** configured optional perceive/plan/act/verify stages support + sequential and parallel paths. ROVE schedules them; an LLM does not secretly + choose the campaign's execution graph. +- **Runtime:** Copilot hosts optional assistant and configured agent roles. Direct + customer models/agents do not need Copilot planning around them. Local LeRobot + inference can use the separately locked experimental process runtime. +- **Persistence:** local SQLite stores plus content-addressed assets retain trials, + cases, reviews, configuration, evidence and lineage. Telemetry is supplementary. +- **Future integrations:** general physical environment lifecycle, hosted monitoring, + Fabric/Databricks/Delta and MCP exposure are not automatically implemented because + an older design document names them. + +## Technology and structure + +Python uses FastAPI, Pydantic, argparse CLI dispatch, YAML configuration and SQLite. +The frontend is static HTML, vanilla JavaScript and shared CSS, with no React build. +Quick execution uses SSE; campaign progress reads durable state. Optional provider, +Copilot, kinematics and local policy dependencies have separate setup/validation +requirements. Python 3.12 is the tested local setup recommendation; package metadata +allows Python 3.11+ where dependencies support it. ```text -src/ -└── rove/ # Python package (import rove) — src layout - ├── adapters/ - │ ├── protocols.py # THE contracts — VLMAdapter, PolicyAdapter, AgentAdapter, SimAdapter - │ ├── registry.py # YAML → adapter resolution + health checks - │ ├── vlm/ # mock, azure_openai, azure_foundry, local_mlx - │ ├── policy/ # mock, local_lerobot, local_hf - │ ├── agent/ # mock, azure_foundry (Foundry agents) - │ └── sim/ # mock, local_mujoco - ├── orchestrator/ - │ ├── pipeline.py # EvaluationPipeline (4-step pipeline) - │ └── run_manager.py # Parallel multi-strategy execution - ├── api/ - │ └── app.py # FastAPI app + SSE streaming - ├── models.py # Dataclass shared models - ├── config.py # Load rove.yaml, env vars, strategies - └── cli.py # CLI entry point -tests/ # pytest test suite -├── conftest.py # Shared fixtures -├── test_models.py -├── test_config.py -├── test_pipeline.py -├── test_run_manager.py -├── test_mock_adapters.py -└── test_api.py +src/rove/ + adapters/ # Protocols, registry, model/agent/verifier adapters and workers + api/ # HTTP services and local application + benchmarks/ # Campaign planning, execution, assessment and reports + datasets/ # Cases, reviews, success contracts, ABC imports and frozen sets + evaluators/ # Configured task/constraint/diagnostic evaluation + models/ # Shared typed configuration and result models + orchestrator/ # Stage execution and recording bridge + runtime/ # Copilot roles, assistant selection and observability + strategies/ # Immutable strategy catalog + trials/ # SQLite records, assets, evidence and exchange + ui/ # Server and CLI entry points; cli.py owns the rove command +frontend/ # Shared navigation, strategy table and evaluation workspaces +tests/ # Python and Node regression suites ``` +Current navigation is **Trials** and **Campaigns**, with **New trial** and +**New campaign** creation actions and separate Settings. Follow the +[user journey](docs/product/user-journey.md) and source rather than older navigation +or React/MCP sketches. Keep task/image/URDF comparison, recorded pipeline progress, +CLI parity, private reference boundaries and baseline immutability intact during UI +changes. + ## Reusable evaluation core The 2026-09-14 pivot preserves the robotics workbench and makes its trial/campaign @@ -125,12 +115,12 @@ When creating a PR, always update `CHANGELOG.md`: - No global state — pass dependencies explicitly (registry, store) - Protocol pattern: adapters implement `@runtime_checkable` Protocols, never inherit from ABC -### TypeScript/React +### Frontend -- Strict mode TypeScript -- Types auto-generated from FastAPI OpenAPI spec -- Functional components with hooks -- No business logic in frontend — render what API tells it, send user actions to API +- Use the existing HTML/JavaScript/CSS architecture and shared components. +- Render authoritative API records; keep evaluation and scoring in shared services. +- Preserve keyboard focus, accessible dialogs, saved theme and draft/saved-state boundaries. +- Run applicable Node behavior tests and inspect meaningful interaction changes in the browser. ### Testing @@ -146,7 +136,7 @@ When creating a PR, always update `CHANGELOG.md`: | `src/rove/adapters/protocols.py` | Adapter contracts — change these carefully | | `src/rove/orchestrator/pipeline.py` | 4-step pipeline — deterministic, no LLM decisions | | `src/rove/orchestrator/run_manager.py` | Concurrent multi-strategy execution | -| `src/rove/models.py` | Shared data models (dataclasses) | +| `src/rove/models/` | Shared data and configuration models | | `docs/PRODUCT_SPEC.md` | Full product specification and reference | ## What NOT to Build @@ -160,14 +150,21 @@ When creating a PR, always update `CHANGELOG.md`: ## Terminology -- **adapter** (not connector/driver) — model-specific implementation of a Protocol -- **orchestrator** (not runner/agent) — the 4-step pipeline executor -- **evaluation** (not run/experiment) — a top-level assessment of agent pipeline configurations on a task -- **combination** (not pair/config) — a specific agent pipeline configuration (VLM+VLA+LLM+Agent pairing) -- Pipeline steps: **perceive**, **plan**, **act**, **verify** (always lowercase in code/prose) -- **task_planning** (not grasp_planning) — the capability for the plan stage +Use the [current concepts](docs/product/concepts.md): a task is an instruction, +a case supplies concrete inputs/conditions, a trial is one strategy attempt, and a +campaign organizes selected cases, strategies and repetitions. Baseline is a saved +reference to a completed strategy's assessment; Improve prepares a linked editable +copy. Evaluation and run describe activities rather than extra persisted containers. + +An adapter implements a supported integration contract. The orchestrator executes +the configured stage graph. Stage names remain perceive, plan, act and verify; +`task_planning` is the plan capability. Descriptions and AI suggestions are not +executable graders or SME judgments. + +## Historical phased build order -## Phased Build Order +This sequence records the original roadmap, not current delivery status. Use +[current implementation](docs/architecture/current-implementation.md) for what exists. 1. **Phase 1**: Mock adapters + orchestrator + CLI + API + dashboard (no external deps) 2. **Phase 2**: Local models (SmolVLA, GroundingDINO, CoreML SAM2, MuJoCo/LIBERO) diff --git a/README.md b/README.md index 015cdfa..c3391e7 100644 --- a/README.md +++ b/README.md @@ -4,12 +4,16 @@ **ROVE evaluates robotics agent pipelines on your task.** -Bring your task and data, configure the models and agents in your pipeline, -and compare their outputs. ROVE is an open-source workbench for inspecting the evidence -behind each assessment, and the checks or observations that are still missing. -A strategy configures the system being evaluated: vision-language reasoning, -tool-using agents, action policies such as VLAs, or a combination. Configure the -stages relevant to the task; these capabilities can coexist in one pipeline. +Compare the same **task + observation + robot configuration** across your chosen +strategies. Inspect each strategy's outputs and traces, then use campaigns to repeat +that comparison, preserve a baseline and track changes to cases, models and success +criteria over time. + +A strategy can use a VLM, a tool-using agent, a VLA or a configured combination of +pipeline stages. ROVE evaluates the connected system through supported adapters; +training is an explicit standalone preparation step, separate from trials and +campaigns. This is an experimental OSS +workbench for robotics evaluation engineering. It connects supported model and agent adapters, optional MuJoCo diagnostics, and a local browser dashboard. This is an experimental OSS project for @@ -43,8 +47,8 @@ performance remain unverified on the development host. ## Try it locally Python 3.12 and [uv](https://docs.astral.sh/uv/) are recommended. Run from the -repository root; the dashboard currently needs the source checkout's assets. -No cloud account or model download is needed for the mock demo. +repository root because the dashboard uses the source checkout's assets. The mock +workflow needs no cloud account or model download. ```bash git clone https://github.com/niksacdev/rove.git @@ -54,64 +58,98 @@ uv run --no-sync rove serve # Open http://127.0.0.1:5001 ``` -Choose a sample case, select **Mock (Test)**, open **Review & run**, then click -**Run trials**. Inspect -stage outputs, evidence confidence, missing checks, and comparison results. -Mock outputs demonstrate the workflow; they do not measure a robot's ability. - -Open **Cases** to evaluate your own PNG/JPEG images and task instructions: - -1. Choose from the 49 bundled **Sample cases**, import your own case, or load the [synthetic customer cases](examples/customer-cases/README.md). -2. Define success criteria, choose strategies and preview the planned trials and measures. -3. Run a baseline, inspect its trace, and rate individual outputs when SME judgment is required. -4. Freeze exact cases/reviews, then compare a candidate configuration under the same conditions. - -The existing gallery is imported into versioned Cases automatically. Sample case links -open the exact imported revision; source/license metadata, robot state and reference -annotations are retained. Re-importing preserves user edits and creates no extra trials. -Reference annotations are unreviewed source material, kept separate from candidate -inputs and observed episode evidence. Historical runs remain in History. - -Use **Cases → Import cases → XDOF ABC episode** to turn an upstream ABC export -into a case with a selected camera observation and current robot state. The same -operation is available as `rove data case import-abc` for automated dataset -preparation. ROVE builds on the ABC project's converter and public samples; -demonstrated actions remain reference material. Follow the -[ABC dataset guide](examples/abc/README.md) for preparation, source versions and -the supported export formats. - -Name and pin a baseline, inspect clock-aware trace lanes beside a candidate trial, -and open preserved stage outputs or exact recording selections. Explicit frozen -annotation bindings can grade later candidates without exposing labels to them. -Campaign targets report met, not met or unknown alongside robotics evidence coverage. -Try the [action-dependent synthetic robotics example](examples/robotics/README.md), -or read [evidence and exchange](docs/product/evidence-and-exchange.md) for portable -relational exports. The [guided assistant](docs/product/named-baselines-and-assistant.md) -prepares imports, revisions, freezes and comparisons with host-confirmed mutations. - -Missing or disputed assessments stay unknown. An accepted plan is not observed -robot completion. **Trial history** can promote a finished quick run into a reusable -case while retaining the original trial as an exploratory reference; new campaigns -plan fresh attempts. No cloud service is needed for this workflow with mock adapters. - -For a shareable, seeded example without opening the dashboard: +Choose **New trial**, load a sample observation or provide an image and task, and +add a URDF when your evaluation needs robot geometry. In **Configure**, select one +or several strategies, including **Mock (Test)** for an offline demonstration. +**Review & run** shows the exact inputs; **Run trials** starts execution. Results +retains an **All strategies** overview, individual outputs and **Inspect trial & +traces**. Mock output verifies the workflow, not robot competence. + +**Trials** and **Campaigns** browse saved work. **Settings** maintains reusable +configuration and the optional assistant. Creation, inspection and execution are +separate actions: opening history never reruns a model. + +## From one comparison to an improvement history + +```mermaid +flowchart LR + T[New trial: same task, image and robot input] --> S[Compare strategies] + S --> E[Inspect outputs and traces] + E --> C[Add to campaign] + C --> M[Add cases and review success criteria] + M --> R[Run repeated trials] + R --> B[Save a baseline] + B --> I[Improve: review a proposed change] + I --> R + R --> H[Timeline: changes, outcomes and evidence] +``` + +Start a **New campaign** directly when you already have a case collection. Its stages +are **Cases → Configure → Success metrics → Run → Review & improve**. The header +keeps the campaign name, attempts per case and planned total visible: +**cases × strategies × attempts = trials**. Saved campaigns show their recorded setup; +**Improve** creates a linked editable copy. + +- **Bring cases.** Select bundled samples, add customer images/tasks, or import + JSONL with separate images. Case revisions retain source identity, optional robot + input and private reference material. [Synthetic customer cases](examples/customer-cases/README.md) + exercise onboarding without claiming measured outcomes. +- **Define success.** Each criterion has a pass condition and a human reviewer or + configured verifier. AI can suggest a draft; writing a condition does not implement + a check. Reports calculate eligible success/reliability measures and timing from + actual assessments and measurements. Missing judgments remain unknown. +- **Run and inspect.** Watch each strategy's pipeline progress, compare outputs and + inspect individual trials. **Set as baseline** freezes one completed strategy's + assessment reference; review coverage is visible even when outcomes are unknown. +- **Iterate deliberately.** Save a changed strategy revision, inspect the proposed + change and run new trials. The timeline records changes to cases, strategies and + criteria. Changed assessment conditions remain useful history but do not establish + a controlled improvement. + +**Add to campaign** retains the exploratory trial and carries its case/strategy +context forward. A campaign schedules fresh attempts. **Save case collection** or +**Add to an existing collection** creates a dataset revision with exact membership +and review choices; previous revisions remain available. + +ROVE stores durable trials, case/configuration revisions, reviews and lineage in +local SQLite databases, with large media in managed asset files. Inspect clock-aware +trace lanes, stage outputs and exact evidence references; telemetry supplements these +records rather than becoming the source of scores. The [evidence and exchange guide](docs/product/evidence-and-exchange.md) +explains portable studies. Fabric/Databricks/Delta integration remains a future +consumer of these records, not a replacement for transactional storage. + +## Build on ABC data + +Use **Cases → Import cases → XDOF ABC episode**, or `rove data case import-abc`, +to turn a selected observation from an upstream converted episode into a ROVE case. +ROVE reuses the ABC team's converter and public samples, preserving current state, +source hashes and private demonstration artifacts. The [ABC guide](examples/abc/README.md) +explains supported formats, bounded imports and source attribution. This imports +observations for your own strategy comparisons; it does not run ABC's simulator or +turn a demonstration into candidate success. + +## Automate evaluations + +Compare your own observation across two configured strategies: ```bash -uv run --frozen python scripts/demo.py --output data/output/demo.json +uv run --no-sync rove trial run --task "Pick up the bracket" --image observation.png \ + --strategy mock-consistent --strategy mock-variable --config examples/benchmarks/rove.yaml ``` -The demo uses fresh seeded mock adapters and produces a JSON evidence report. -Latency and timestamps vary; mock actions and judgments repeat for the same seed. - -For repeated trials, open **Campaigns** in the dashboard or run: +Add `--urdf robot.urdf` when applicable. Each strategy produces its own saved trial. +For repeated offline trials and an HTML/JSON/CSV report: ```bash -uv run --frozen rove benchmark run examples/benchmarks/mock.json --config examples/benchmarks/rove.yaml +uv run --no-sync rove benchmark run examples/benchmarks/mock.json --config examples/benchmarks/rove.yaml ``` -This produces offline HTML, JSON and CSV reports with pass@k, pass^k, task -outcomes and comparable history. See [benchmark configuration and scoring](docs/BENCHMARKS.md) -for seed support, unknown outcomes, cancellation and resume behavior. +The [CLI guide](docs/product/cli-ui-parity.md) covers cases, reviews, campaigns, +strategy revisions, baselines, reports, timeline and assistant controls. The +[GitHub Actions example](examples/ci/README.md) restores explicit study state, records +a source revision, writes reports and applies declared quality gates. It is an +example to adapt, not a preconfigured hosted service. See [report metrics](docs/BENCHMARKS.md) +for pass@k, pass^k, eligibility, unknown outcomes and seed limitations. ## What you can inspect @@ -129,6 +167,13 @@ IK produce estimated evidence. Missing collision geometry produces **unknown**, not “no collision.” Torque and payload feasibility remain **not computed** until ROVE has a validated timing, inertia and actuator-mapping contract. +**Action output is not a complete episode.** The local VLA path generates actions +from the supplied image/state; it does not execute a chunk, observe the changed +scene and repeat. A URDF supports robot description and applicable checks, not +universal checkpoint compatibility. [Action-level vs episode-level evaluation](docs/product/action-and-episode-evaluation.md) +explains the current value, missing lifecycle and why multiple chunks still belong +to one episode trial rather than separate reliability samples. + A VLM's success judgment is not measured task success. Stage attribution is a heuristic debugging hypothesis, not a demonstrated causal root cause. Perception and planning diagnostics do not control VLA actions in the parallel pipeline. @@ -157,33 +202,30 @@ credentials and compatible checkpoints. Known dependency advisories remain; review the runtime limits before use. Pi0.5 also requires authorized access to its tokenizer. -For AI campaign explanations, open **Settings → Assistant**, choose an existing -Copilot assistant endpoint, **Save**, then **Test connection**. The selection persists -locally across restarts. `ROVE_ASSISTANT_ENDPOINT` remains an explicit server override. -The configured local assistant uses the existing model server; a successful test proves -connectivity, not evaluation quality. The same controls are available through -`rove assistant settings`, `rove assistant configure --endpoint local-assistant`, -`rove assistant test` and `rove assistant disable` (add `--url http://127.0.0.1:5010` -when the app runs on that port). +For AI drafts and campaign explanations, open **Settings → Assistant**. New setups +default to **GitHub Copilot**, with explicit ROVE sign-in; saved local and Disabled +choices are retained. You can instead select an existing endpoint or a **Microsoft +Foundry** model resource and deployment using Azure CLI authentication on the ROVE +host. Save then **Test connection**. Provider choice does not change the strategies +under evaluation or authorize proposed workflow mutations. -Use `/docs` for the implemented HTTP API. From this source checkout, compare your -observation against two configured strategies without opening the dashboard: +The [assistant setup guide](docs/product/assistant-setup.md) covers dedicated-profile +GitHub sign-in, model discovery, local endpoints, Foundry and equivalent CLI commands. +These provider paths are under verification in this change; authenticated GitHub/Azure +inference is not implied by configuration or a successful local Qwen test. -```bash -uv run --no-sync rove trial run --task "Pick up the bracket" --image observation.png \ - --strategy mock-consistent --strategy mock-variable --config examples/benchmarks/rove.yaml -``` +Use `/docs` for the implemented HTTP API. UI and CLI call the same underlying +services; an older published package may not include all commands in this checkout. -Supply your own `observation.png`; add `--urdf robot.urdf` to retain a robot description. -Each strategy produces its own saved trial. Campaigns retain cases, criteria and -repetitions; **Improve** creates a linked iteration whose timeline shows what changed -and whether outcomes are comparable. Unknown assessments remain unknown. +## The remaining industrial evaluation gap -See the [CLI capability guide](docs/product/cli-ui-parity.md), -[iteration architecture](docs/architecture/ADR-027-trial-to-campaign-improvement.md) -and [manual GitHub Actions example](examples/ci/README.md) for scripted data/review -management, baselines, report gates and restored study history. These commands are -verified in this checkout; an older published package may not contain them. +Image + goal + URDF and forward-kinematics checks alone cannot identify the most +reliable strategy for completing an industrial task. Today's offline comparison is +preliminary screening with inspectable evidence. The next recommended milestone is +one resettable industrial pick-and-place scenario, measurable object-level success, +fresh observations after execution and repeated trials of two executable strategies. +This is a proposed next step, not a shipped loop or approved implementation commitment. +More geometric checks or report polish do not supply the missing outcome evidence. ## What a useful comparison needs @@ -206,14 +248,16 @@ Use the [documentation guide](docs/README.md) to explain ROVE's [current implementation](docs/architecture/current-implementation.md). The [target architecture](docs/architecture/target-architecture.md), [system diagram](docs/architecture/system-diagram.md) and -[implementation plan](docs/product/implementation-plan.md) show how ROVE will use +[implementation plan](docs/product/implementation-plan.md) show how ROVE uses Copilot SDK for ROVE-owned agents while preserving direct evaluation of customer agents, VLMs and VLAs. The linked [architecture decisions](docs/architecture/README.md) include diagrams, trade-offs and implementation status. The local customer workflow is implemented; general video ingestion and production cloud exporters remain future work. Bounded recording imports, action-dependent synthetic trials and aggregate targets are implemented -with local validation; live providers and physical robot execution remain unvalidated. +with local validation. Local Qwen and native SmolVLA inference were exercised under +the limits in [ADR-033](docs/architecture/ADR-033-isolated-vla-runtime-and-trial-inspection.md); +physical robot execution and production cloud integrations remain unvalidated. ## Development @@ -244,11 +288,3 @@ this README and executable tests describe the supported release behavior. Use [configured verification](docs/VERIFICATION.md) to provide a task evaluator, required constraints and optional FK diagnostics through the existing verify stage. Results carry measurements, evidence quality and evaluator versions into campaign reports. - -### Your evaluation journey - -The workspace uses the same navigation on every page: **Start → Evaluate → Results**, -with strategies, endpoints and settings under **Configure**. Prepare cases, define -success and run repeated trials, then inspect outcomes and compare a changed -strategy with a saved baseline. Quick trials remain available from Start; saved -attempts live under Results → Trials. See the [user journey](docs/product/user-journey.md). diff --git a/docs/PRODUCT_SPEC.md b/docs/PRODUCT_SPEC.md index 62d0ef2..c9eb20f 100644 --- a/docs/PRODUCT_SPEC.md +++ b/docs/PRODUCT_SPEC.md @@ -1,6 +1,6 @@ # ROVE Product Specification -**Version:** 0.9 +**Version:** 1.0 **Updated:** 2026-09-14 **Status:** Current product vision and requirements. Capability status is explicit below; proposed features are not shipped functionality. @@ -15,7 +15,9 @@ The [reusable evaluation core](product/reusable-evaluation-core.md) is implement for structured-input CLI trials/campaigns and saved-result inspection. Robotics remains the default integration and current case-authoring UI. This is an OSS foundation for future evaluation uses, not a claim of a complete general-purpose -product or an implemented ABC/cloud service. The original robotics vision and +product or a deployed cloud service. The separate +[ABC bimanual integration](product/abc-pi05-comparison.md) implements simulator +episodes; GPU inference and model performance remain unverified. The original robotics vision and personas below remain the specification for the preserved robotics integration. ## 1. Core vision @@ -28,7 +30,7 @@ Compare the same task, image and robot description across a chosen set of strate The unit being assessed is a configured robotics agent pipeline: its models, prompts, stage settings, tools and applicable action components. A strategy can combine vision-language reasoning, tool-using agents and action policies such as VLAs through supported adapters. An agent is a running system, while VLM and VLA describe model capabilities; these are not mutually exclusive choices. Perception or planning assessments do not require every action or simulation stage. The evaluation stays anchored to the customer's robotics task and business outcome. -ROVE is an inference and evaluation workbench. It does not train models or choose a deployment on the customer's behalf. It records outputs, grades and evidence so a person can make that decision. Human review can assess scene understanding and plan quality; claims about physical task completion require associated episode outcome evidence. +ROVE is an inference and evaluation workbench. Explicit standalone preparation examples can fine-tune models before evaluation; trials and campaigns never start training implicitly. ROVE does not choose a deployment on the customer's behalf. It records outputs, grades and evidence so a person can make that decision. Human review can assess scene understanding and plan quality; claims about physical task completion require associated episode outcome evidence. ## 2. Target users @@ -118,11 +120,10 @@ This local workflow is designed to deliver a useful first report before requirin The first report shows accepted/completed, failed and unknown cases; representative failures link to evidence. It shows what is unreviewed or unmeasured. A configurable pilot checks data and grading before the full campaign, with attempt count and available budget information shown before launch. -The primary UI starts a campaign, including the smallest useful evaluation of one -case, one strategy and one attempt. Existing independent runs retain durable inputs, -configuration, interruption state and saved trial evidence. Their compatibility routes -remain usable. Promotion preserves an existing trial as an exploratory reference and -creates a reusable case; later campaigns schedule fresh attempts separately. +Home offers **New trial** for an immediate comparison and **New campaign** for a +prepared case collection. Both create durable trials through the same underlying +services. Saved work is browsed through peer **Trials** and **Campaigns** destinations; +Settings contains reusable configuration and assistant setup. ### Trial-first entry and guided improvement @@ -181,61 +182,46 @@ do not establish live-provider, GitHub-hosted workflow or hardware performance. ### Navigation and workspace design -Primary navigation is **Start · Evaluate · Results**, with **Settings** separately -on the right. Settings maintains reusable strategies, models/connections and preferences. -Evaluate is a single campaign workspace: **Cases → Configure → Success metrics → Run → Results**. - -Cases offers **Add new**, **Select existing** and **Import cases**. Add new opens a -single-case dialog. The existing library groups Cases and Datasets in one bounded -picker; a dataset loads exact case versions into Cases for inspection before choosing -strategies. JSONL metadata import pairs records with explicitly selected PNG/JPEG -images. Selected cases appear as compact editable cards with expandable expectations. Expected outcomes and expert-review material may be prefilled from -available metadata or an explicit assistant, but remain labelled drafts. Suggested -content cannot create a human judgment or turn an initial image into completion evidence. - -Configure selects one or several available strategies. **Success metrics** is a -dedicated stage for editable criteria, expected report measures and evidence needs. -Its optional assistant drafts criteria from task and annotation context for review; -it does not claim to have inspected image bytes. Advanced contract JSON stays collapsed. -Existing strategies can be compared within one campaign on the same case set. A -component ablation saves a new revision of a baseline strategy and compares that -candidate under the same recorded assessment conditions. The optional assistant's -chat helps prepare the same evaluation. Progressive assistance offers assessment -drafts in Success metrics and an explanation of retained outcomes in Results. The -manual path works without it; both use validated previews and explicit launch. - -Run confirms case count, strategy selection, repetitions, criteria and execution limits. -It explains the total: **cases × strategies × repetitions = planned trials**. During -execution, each selected strategy has visible progress and nested trials showing -case, strategy and attempt identity. Campaign -progress and active trial stage updates poll while visible; expanding pipeline details or pressing -Refresh fetches recorded stage events and output. This does not imply token streaming. -Results leads with per-strategy outcome and reliability charts, followed by an optional -AI outcome summary and **Inspect trials** details for outputs, traces and expert -assessments. Unknown outcomes and unavailable measures remain explicit; an explanation -cannot alter scores or claim physical completion without evidence. **Set as -baseline** names and saves one completed campaign strategy as the reference; Results -shows a **Baseline** badge only for persisted references. Versioning and pinning stay -secondary. **Compare a strategy change** prepares a matching candidate; only **Start -comparison campaign** runs new trials. Neither review nor setting a baseline reruns -the original campaign. - -Use **Save case collection** and **Add to an existing collection** for dataset actions. -Adding creates a new immutable revision retaining old members; old revisions remain -unchanged. Reviewed collections retain exact annotations and review identities. -Opaque freeze/reuse terminology should not be a prerequisite for the main workflow. - -Results returns to the exact campaign or trial through saved identities. Local stage -changes retain draft selections and never execute work. Legacy URLs remain usable. -Use warm neutral/slate surfaces with restrained teal accents, readable controls, -keyboard focus and a connection status that never obscures the task input. This -earlier workspace was verified with revised behavior tests, a desktop three-trial mock -journey and a 390px gallery in both themes. The separate metrics stage, progressive AI -assistance, strategy progress overview and graph-first Results were then checked with -a separate six-trial mock campaign and a 390px Results layout. Unknown assessments -remained visible after execution completed. Assistant behavior has fake-runtime -regression coverage; live AI generation and hardware performance were not validated. See the [campaign workspace](product/user-journey.md) and -[ADR-025](architecture/ADR-025-workflow-navigation.md) for requirements and validation. +The current journey is [specified in one place](product/user-journey.md): +**Trials** and **Campaigns** are record browsers; **New trial** and **New campaign** +create drafts. Settings is a separate right-hand gear. A quick comparison follows +**Case → Configure → Review & run → Results**, retaining task conversation, image, +optional URDF, shared strategy selection and explicit execution. Results preserves +All strategies and exact trial/trace inspection. + +Campaigns extend the same setup through **Cases → Configure → Success metrics → +Run → Review & improve**. A persistent header shows name, attempts per case and +case × strategy × attempt total. Draft setup is editable; saved setup comes from +its recorded specification, including exact seed identities. Improve creates a +linked editable copy. The shared strategy table presents configured stages and +endpoints; readiness feedback is inline and does not imply live model quality. + +Cases combines Add new, Select existing and Import cases with a bounded image/task +library. Datasets are saved case collections inside this workflow, not an additional +execution concept. Source labels remain separate from candidate input; populated +expectations and assistant suggestions never become SME judgments automatically. + +Success metrics distinguishes each trial's **Pass condition** and **Checked by** +from automatically calculated report statistics. Manual checks need a recorded review; +configured checks need a verifier and suitable evidence. A textual description is not +executable scoring logic. Technical settings retains contracts, targets and annotation +bindings. Generated criteria and model-authored summaries have explicit provenance. + +Run shows each strategy's execution and stage progress. Results begins with outcomes, +strategy charts and optional evidence-grounded interpretation, followed by inspection +and scoring/baseline details. A fresh draft shows No results yet and Go to Run rather +than another campaign picker. Completion, acceptance, failure and missing assessment +remain distinct. Setting a baseline, opening history or preparing an improvement never +starts another trial. Unsaved-work navigation and locked execution inputs preserve the +reviewed observation; [ADR-032](architecture/ADR-032-trial-review-and-navigation-state.md) +records the execution boundary. + +Saved collections preserve membership and selected reviews in new revisions. A new +candidate strategy preserves its source and shows an exact change preview before +use. Case or rubric changes remain visible in the timeline and can invalidate a +controlled comparison. See [ADR-031](architecture/ADR-031-shared-evaluation-workspace.md) +and [ADR-033](architecture/ADR-033-isolated-vla-runtime-and-trial-inspection.md) for +workspace and trace-inspection validation limits. ## 5. Create datasets and validate cases @@ -249,7 +235,9 @@ recordings and demonstrated actions remain reference material; they never establ that a newly evaluated strategy completed the physical task. This provides an onboarding path into existing strategy comparisons, SME review and campaign history. See [ADR-028](architecture/ADR-028-abc-episode-import.md) for scope and validation; -integration of upstream simulation or physical evaluation results is a future step. +the separate [ABC bimanual integration](product/abc-pi05-comparison.md) now runs +simulator episodes with fresh candidate actions and retained measurements. Physical +robot evaluation remains outside that implementation. ### Review customer cases and outputs @@ -305,6 +293,19 @@ An ablation copies cases, repetitions and grading conditions and exposes both th Robotics episode evidence must identify the producing configuration and attempt. Reusing policy A's recorded outcome cannot demonstrate policy B's performance. Matching seed labels alone do not establish matching physical conditions or statistical independence. Curated development data remains distinct from an independent held-out assessment. +### Action comparison and complete episodes + +The current local VLA path collects action output from one prepared observation/state; +it does not execute, observe the changed environment and replan until completion. +This supports integration and action-level diagnostics. The separate +[ABC bimanual integration](product/abc-pi05-comparison.md) owns simulator reset, fresh +observations, action execution and terminal evidence, +with multiple chunks retained inside one trial. Match initial conditions for strategy +comparisons; subsequent observations naturally diverge. A URDF does not adapt a +checkpoint to arbitrary cameras, state, normalization or control conventions. See +[action and episode evaluation](product/action-and-episode-evaluation.md) for the +adapter-specific boundary and requirements for other episode integrations. + ## 8. Storage and integration direction Use SQLite for the local product. The shared journal stores trials, snapshots, events, cases, reviews, frozen datasets, named baseline revisions, evidence references and operation identities with schema migrations and integrity checks. Observations, stage outputs and bounded auxiliary recordings are separate content-addressed assets. A complete backup preserves the journal, assets and separate campaign database together. The versioned exchange exports a validated relational manifest and assets and restores only into a fresh destination. It is a portable local format; PostgreSQL and cloud analytical connectors remain demand-driven. @@ -326,28 +327,45 @@ baselines and campaign/ablation operations through validated services. Each writ requires its exact host confirmation; the assistant cannot manufacture SME reviews or labels. Local telemetry copies can use a bounded loopback OTLP exporter, disabled by default. Controlled real-SDK/CLI tests establish local infrastructure behavior, -not live-provider quality or a deployed cloud monitoring profile. Live provider, -Azure and hosted analytics validation remain specification-only in this delivery. +not live-provider quality or a deployed cloud monitoring profile. Local Qwen and native SmolVLA inference were exercised in the later +[ADR-033 validation](architecture/ADR-033-isolated-vla-runtime-and-trial-inspection.md). +That does not establish physical success, cloud-provider coverage or deployed +Azure/Grafana analytics. See [runtime lifecycle](architecture/runtime-lifecycle.md), [local observability](architecture/runtime-observability.md) and the [live-provider validation specification](product/live-provider-validation.md). +### Recommended next industrial validation milestone + +Beyond the implemented ABC simulator episode path, an industrial task needs its own +validated environment and outcome measurement. Image/task/URDF comparisons and FK diagnostics support preliminary screening; +they cannot establish the most reliable industrial task solution. The next recommended +slice is one resettable pick-and-place environment, one declared robot, measurable +object-level success and repeated complete episodes of two executable strategies. +Fresh observations and execution acknowledgements must connect actions to outcomes. +Agent-with-robot-tools and VLA/controller systems are compared as full strategies. +This is a proposed milestone requiring scope approval, not an implemented feature or +a commitment to add generic closed-loop control in the current update. + ## 9. Capability and delivery status -| Capability | Status in the customer workflow implementation slice, 2026-09-12 | +| Capability | Current implementation and validation boundary, 2026-09-14 | | --- | --- | | Configured optional stages, supported models/agents and local dashboard | Implemented; adapter availability depends on installed dependencies and endpoints | | Repeated campaigns, frozen selected configuration, SQLite trial records and HTML/JSON/CSV reports | Implemented in PR #13 | | Local task evaluators, required constraints, optional FK diagnostics and versioned evidence | Implemented in PR #14 | | Durable shared quick/campaign trial recording and initial observation assets | Implemented; frozen snapshots, interrupted-work recovery and idempotent legacy quick-history import | | Durable activity, evidence ranges and paired trace inspector | Implemented for supplied events and managed assets; separate source clocks, indexed JSON ranges and explicit missing coverage; no arbitrary remote fetch or video decoding | -| Copilot runtime for hosted perception, planning and verification | Optional pinned runtime with fresh role scopes, lifecycle contracts and controlled real-CLI validation; live-provider validation is specification-only | +| Copilot runtime for hosted perception, planning and verification | Optional pinned runtime with fresh role scopes, lifecycle contracts and local provider validation; cloud/provider-specific acceptance remains separate | | ROVE/native spans and optional monitoring | Local stage/tool correlation, native capture and bounded loopback OTLP export implemented; exporter disabled by default; no validated Azure/Grafana deployment | -| Evaluation assistant | Typed reads and host-confirmed case, contract, freeze, baseline and campaign/ablation operations implemented; no SME judgment or label-writing tools | +| Evaluation assistant | Typed reads and host-confirmed operations; saved selection and explicit connection tests; no SME judgment or label-writing tools | | Robot action integrations | Direct customer action adapters and the synthetic test world are supported; no Copilot hardware-action tools or hardware drivers | | Customer intake, case revisions and expected-metric preview | Managed images and bounded auxiliary evidence, typed episode validation, deterministic action-dependent synthetic rollout and aggregate targets implemented | | SME review, reusable annotations and frozen datasets | Optimistic drafts, immutable final reviews, atomic freezing and explicit frozen annotation-to-local-verifier bindings implemented | | Baseline/ablation references, component diffs and quick promotion | Named/pinned immutable baseline revisions, captured assessments, paired traces and exploratory quick promotion implemented | +| Shared trial/campaign workspace and versioned iteration history | Implemented with retained strategy comparison, exact saved configuration, explicit launch, trace inspection and linked Improve copies | +| ABC dataset support | Converted local real/sim observation import through CLI/API/UI; source assets private, no automatic candidate-success labels | +| Isolated native VLA inference | Local SmolVLA path exercised; probes are execution-ineligible; Pi0.5 and physical deployment are not validated by that probe | | Relational exchange; PostgreSQL or Delta Lake integration | Validated local exchange/restore implemented; PostgreSQL and cloud connectors remain demand-driven | | Real closed-loop robot/simulator integration, universal adapter compatibility or safety certification | Not provided by the current release | diff --git a/docs/README.md b/docs/README.md index 6f46087..50d5d92 100644 --- a/docs/README.md +++ b/docs/README.md @@ -11,13 +11,16 @@ | [Product specification](PRODUCT_SPEC.md) | Core vision, founding personas, campaign model and delivery status | | [Concepts](product/concepts.md) | Cases, trials, campaigns, strategies, datasets and provenance | | [Product workflows](product/evaluation-workflows.md) | Baselines, SME review, frozen datasets, ablations and acceptance criteria | -| [User journey and navigation](product/user-journey.md) | Start, Evaluate, Results and Configure; staged workspace, routes and accessibility acceptance | -| [Navigation decision](architecture/ADR-025-workflow-navigation.md) | Why navigation follows the evaluation journey while retaining existing routes | +| [User journey and navigation](product/user-journey.md) | New trial and New campaign; shared configuration, explicit execution, saved state and improvement | +| [Shared workspace decision](architecture/ADR-031-shared-evaluation-workspace.md) | Peer browsers, shared strategy selection, persistent campaign identity and scoring clarity | | [Success and performance measures](product/metrics-and-success.md) | Define success before a campaign; understand the report and missing evidence | | [Traces and measurements](product/traces-and-measurements.md) | Inspect trial timelines, supporting evidence and baseline differences | | [Robotics evidence and targets](product/robotics-evidence.md) | Action-dependent fixtures, approved annotation bindings and supplied episode measures | | [Named baselines and assistant](product/named-baselines-and-assistant.md) | Frozen baseline assessments and confirmed guided workflows | | [Preserved evidence and exchange](product/evidence-and-exchange.md) | Recording selections, clock-aware traces and portable relational tables | +| [Trial review and navigation](architecture/ADR-032-trial-review-and-navigation-state.md) | Reviewed inputs, explicit launch, locked execution and unsaved-work protection | +| [Assistant setup](product/assistant-setup.md) | Copilot default for new setups, preserved local/disabled choices, explicit authentication, Foundry and CLI | +| [Actions and complete episodes](product/action-and-episode-evaluation.md) | What one VLA output establishes, why chunks are not trials, and the future execute/observe lifecycle | | [Experimental local VLA runtime](../examples/runtime/lerobot/README.md) | Trusted local checkpoints, native preprocessing, preparation and evidence limits | | [Live provider validation spec](product/live-provider-validation.md) | Provider/Azure acceptance plan; local validation scope is recorded in ADR-033 | | [Model assessment](product/model-landscape-2026-09.md) | September 2026 evidence on frontier agents, VLAs and evaluation scope | @@ -39,7 +42,7 @@ from external deployment and provider validation. Product specifications include proposed follow-up work and label it explicitly; an accepted direction alone does not establish deployed integration support. -## Explain ROVE in three diagrams +## Explain ROVE with diagrams 1. Use the concepts diagram to explain the units being evaluated. 2. Use the current implementation diagram to explain how ROVE runs and records them today. diff --git a/docs/TRIAL_HISTORY.md b/docs/TRIAL_HISTORY.md index 534fb4a..2d95e89 100644 --- a/docs/TRIAL_HISTORY.md +++ b/docs/TRIAL_HISTORY.md @@ -1,4 +1,4 @@ -# Trial History and Optional Copilot Stages +# Saved trials, trace inspection and optional Copilot stages ROVE saves one trial for each strategy attempt before execution starts. A quick evaluation and a campaign attempt use the same history model. You can close the @@ -10,11 +10,12 @@ From the source checkout, install the normal local dependencies and start ROVE: ```sh uv sync --frozen --extra dev --extra data --extra kinematics -uv run rove serve +uv run --no-sync rove serve ``` -Choose an image, task and strategy in **Evaluate**, then open **Trial history** in -the navigation. Copilot is optional: existing direct models, agents and synthetic +Choose **New trial**, supply an image/task and optional robot input, select strategies +and explicitly run from **Review & run**. Open **Trials** to browse saved attempts, +or use **Inspect trial & traces** from a strategy result without losing the comparison. Copilot is optional: existing direct models, agents and synthetic strategies do not require a Copilot installation or account. The history view supports these workflows: @@ -55,7 +56,9 @@ Expand the assessment, stage or event details to inspect the recorded payload. Measurements preserve their units, source quality and references. The stage filter helps follow one part of the pipeline. Arbitrary evidence strings are preserved; they are not automatically fetched as remote files or resolved to video ranges. -Parallel trace graphs and aligned baseline comparisons are later work. +Producer-clock lanes and paired baseline/candidate traces are available. Different +clock domains remain unaligned without an explicit mapping; see +[traces and measurements](product/traces-and-measurements.md). Usage is shown only when reported, and the summary covers **loaded event pages**. Missing token counts are unknown, not zero. SDK content capture defaults off, so a diff --git a/docs/architecture/ADR-019-relational-storage-and-assets.md b/docs/architecture/ADR-019-relational-storage-and-assets.md index 30cdfa3..6d82deb 100644 --- a/docs/architecture/ADR-019-relational-storage-and-assets.md +++ b/docs/architecture/ADR-019-relational-storage-and-assets.md @@ -1,5 +1,7 @@ # ADR-019: Keep local records relational and large assets separate +> **Current implementation:** Local SQLite records, managed assets and validated relational exchange are implemented. See [current storage and process boundaries](system-diagram.md) and [evidence/exchange](../product/evidence-and-exchange.md). Historical alternatives below remain the decision record. + **Status:** Accepted direction **Implementation status:** Shared trial/case/review SQLite schema, managed images and versioned dataset membership implemented; campaign journal remains separate **Date:** 2026-09-12 diff --git a/docs/architecture/ADR-021-campaign-success-and-reporting.md b/docs/architecture/ADR-021-campaign-success-and-reporting.md index 9665f8d..917dad3 100644 --- a/docs/architecture/ADR-021-campaign-success-and-reporting.md +++ b/docs/architecture/ADR-021-campaign-success-and-reporting.md @@ -1,5 +1,7 @@ # ADR-021: Campaign success definitions and report explanations +> **Current implementation:** Success contracts, manual/configured assessment and aggregate targets are implemented. [Current measures](../product/metrics-and-success.md) and [ADR-031](ADR-031-shared-evaluation-workspace.md) describe the current Success metrics UI; earlier screen designs below are historical. + **Status:** Accepted; local contracts, explicit frozen-label bindings, synthetic episode measures and aggregate target evaluation implemented. Real provider/robot validation remains separate. ## Context at proposal diff --git a/docs/architecture/ADR-022-trial-telemetry.md b/docs/architecture/ADR-022-trial-telemetry.md index 199f014..28b0ed0 100644 --- a/docs/architecture/ADR-022-trial-telemetry.md +++ b/docs/architecture/ADR-022-trial-telemetry.md @@ -1,5 +1,7 @@ # ADR-022: Retain Trial Events and Link Measurements to Evidence +> **Current implementation:** Durable events, source-clock lanes, paired traces and bounded local OTLP export are implemented. Hosted collectors remain separately validated integrations. See [traces and measurements](../product/traces-and-measurements.md) and [local observability](runtime-observability.md); the proposal history below is retained. + **Status:** Proposed **Implementation status:** Durable event recording and linked trial inspection are pending **Date:** 2026-09-12 diff --git a/docs/architecture/ADR-025-workflow-navigation.md b/docs/architecture/ADR-025-workflow-navigation.md index 3a9c18c..f934fac 100644 --- a/docs/architecture/ADR-025-workflow-navigation.md +++ b/docs/architecture/ADR-025-workflow-navigation.md @@ -1,5 +1,7 @@ # ADR-025: Organize navigation around the evaluation journey +> **Superseded navigation:** [ADR-031](ADR-031-shared-evaluation-workspace.md), [ADR-032](ADR-032-trial-review-and-navigation-state.md) and the [current journey](../product/user-journey.md) replace the earlier Start/Evaluate/Results arrangement. Trials and Campaigns are peer browsers, with separate creation and execution actions. + **Status:** Accepted and implemented; five-stage mock campaign and narrow Results layout verified locally. Live AI and hardware validation excluded. **Date:** 2026-09-12 **Product contract:** [Campaign workspace](../product/user-journey.md). diff --git a/docs/architecture/ADR-026-versioned-strategy-catalog.md b/docs/architecture/ADR-026-versioned-strategy-catalog.md index 2e84a08..a9e6bf7 100644 --- a/docs/architecture/ADR-026-versioned-strategy-catalog.md +++ b/docs/architecture/ADR-026-versioned-strategy-catalog.md @@ -1,5 +1,7 @@ # ADR-026: Keep UI-created strategy revisions in an immutable local catalog +> **Current interaction:** Saved strategy revisions are selectable in the shared Configure table, and Improve exposes reviewed stage changes with exact previews. See [ADR-029](ADR-029-workspace-interaction-consistency.md) and [ADR-031](ADR-031-shared-evaluation-workspace.md). Original endpoint-resolution and snapshot boundaries below still apply. + **Status:** Accepted and implemented; local mock browser integration verified. **Date:** 2026-09-12 **Product contract:** [Campaign workspace](../product/user-journey.md). diff --git a/docs/architecture/ADR-030-foundry-visual-language.md b/docs/architecture/ADR-030-foundry-visual-language.md index 26b5dca..4b95219 100644 --- a/docs/architecture/ADR-030-foundry-visual-language.md +++ b/docs/architecture/ADR-030-foundry-visual-language.md @@ -1,5 +1,7 @@ # ADR-030: Foundry-inspired visual language +> **Navigation/setup refinement:** [ADR-031](ADR-031-shared-evaluation-workspace.md) and [ADR-032](ADR-032-trial-review-and-navigation-state.md) supersede conflicting navigation and setup behavior below. Shared tokens and theme history remain relevant; the [current journey](../product/user-journey.md) is the reader-facing workflow. + - Status: Accepted, implemented - Date: 2026-09-13 - Extends: [ADR-029](ADR-029-workspace-interaction-consistency.md) diff --git a/docs/architecture/ADR-033-isolated-vla-runtime-and-trial-inspection.md b/docs/architecture/ADR-033-isolated-vla-runtime-and-trial-inspection.md index 214916e..f225ffe 100644 --- a/docs/architecture/ADR-033-isolated-vla-runtime-and-trial-inspection.md +++ b/docs/architecture/ADR-033-isolated-vla-runtime-and-trial-inspection.md @@ -51,6 +51,16 @@ flowchart TD E --> T[Trace-backed repair recommendation] ``` +### Clarification: inference horizon versus an episode + +The implemented worker/adapter repeatedly selects actions from the same prepared +batch while collecting output. It does not execute a chunk and feed a new measured +observation back into the policy. A returned horizon is therefore action-level +inference evidence, not a complete task rollout. Multiple chunks from a future +closed-loop integration belong inside one episode trial; they must not be counted +as independent reliability attempts. See [the product boundary and lifecycle](../product/action-and-episode-evaluation.md). +This clarification does not add a robot control loop to the accepted runtime change. + ## Inspection and improvement Run and Results retain an **All strategies** overview. Selecting a strategy shows its output without losing the other executions; **Inspect trial & traces** opens the exact durable trial in a separate inspector. Campaign progress has an explicit return to all strategy lanes. Inspecting evidence does not launch another inference. diff --git a/docs/architecture/ADR-035-assistant-provider-selection.md b/docs/architecture/ADR-035-assistant-provider-selection.md new file mode 100644 index 0000000..b9b4a9f --- /dev/null +++ b/docs/architecture/ADR-035-assistant-provider-selection.md @@ -0,0 +1,134 @@ +# ADR-035: Explicit assistant providers with isolated authentication + +**Status:** Implemented; verification in progress. No authenticated GitHub or Azure +model inference claim is made by this implementation status. +**Date:** 2026-09-14. +**Extends:** [ADR-034](ADR-034-persistent-assistant-settings.md) and +[ADR-024](ADR-024-copilot-runtime-and-observability.md). +**Product guide:** [Assistant setup](../product/assistant-setup.md). + +## Context + +An existing-endpoint-only selector makes users configure a local model even when they +intend to use GitHub Copilot or an Azure model deployment. A new default must not +overwrite an existing local selection or re-enable a user-disabled assistant. Model +provider choice also must not blur the boundary between assistant configuration, +strategy configuration and authorization to change an evaluation. + +## Decision + +Persist a typed selection with four modes: `copilot`, `endpoint`, `foundry` and +`disabled`. A missing settings row defaults to Copilot; a saved null/disabled row +continues to mean Disabled. Existing endpoint selections migrate without changing +their effective model. Explicit host and environment endpoint overrides retain their +read-only precedence. + +```mermaid +flowchart TD + U[Settings or assistant CLI] --> S[Typed provider selection] + S --> DB[(Local assistant settings SQLite)] + DB --> RES[Resolve host override, environment, saved choice or new default] + RES --> COP[GitHub Copilot: model optional] + RES --> EP[Existing copilot_agent endpoint] + RES --> AZ[Foundry: resource endpoint and deployment] + RES --> OFF[Disabled] + COP --> AUTH[Dedicated ROVE profile and explicit device login] + AUTH --> SDK[Copilot SDK runtime] + EP --> SDK + AZ --> TOKEN[Azure CLI credential: ephemeral token] + TOKEN --> SDK + SDK --> PROOF[Explicit synthetic connection test] + SDK --> HELP[Drafts, evidence summaries and prepared operations] + HELP --> CONF[Separate exact host confirmation for mutations] +``` + +The settings schema adds `settings_json` while retaining migration of old endpoint +rows. It stores provider/model/endpoint/deployment choices, not provider secrets. +Authentication state belongs to the dedicated runtime profile or the Azure CLI +credential source, not the browser form or campaign spec. + +### Copilot uses an explicit ROVE identity + +The default Copilot session supplies neither a custom provider dictionary nor a model +ID unless the user selected one. It does not silently route through a local fallback. +A dedicated ROVE Copilot profile isolates the explicit login from ambient GitHub CLI +configuration/tokens. Device-flow start, status and cancellation are visible operations; +cancellation affects the pending ROVE login, not global GitHub identity. + +SDK authentication status and model discovery are explicit. They are distinct from +runtime installation checks and from a successful inference. This extends the prior +custom-provider-only restriction without broadly enabling ambient credentials for all +agent or grader roles. + +### Foundry uses an Azure model resource + +Accept an HTTPS public Azure model host at `*.services.ai.azure.com` or +`*.openai.azure.com`, with no credential/query/fragment fields and only the root or +`/openai/v1` API path. Normalize to `/openai/v1/`; retain an explicit deployment name. +Reject a project/agent endpoint or arbitrary host before obtaining a credential. + +At execution, `AzureCliCredential` obtains a token for +`https://ai.azure.com/.default`. The transient bearer token is supplied to the +configured model session, excluded from persisted settings and ordinary logs, and +not returned to the browser. The user must already have Azure CLI authentication and +deployment permission on the ROVE host. This does not provision Azure or validate all +sovereign/private endpoint variants. An existing Foundry Agent strategy remains a +different adapter path. + +### Saving, testing and proposing remain distinct + +Read/Save settings do not run inference. Test uses a fixed synthetic JSON exchange +without customer cases or tools. Timeout, malformed output and provider failure are +not successful tests. A selection change invalidates a late test result in the UI. +Provider changes do not modify historical campaigns; summary cache identity includes +the effective assistant configuration. + +Assistant-generated workflow changes still pass through the operation preview and +exact host confirmation from ADR-034. Authentication or successful connectivity never +grants permission to alter grades, fabricate SME reviews or rerun a campaign. + +## Interfaces + +| Interface | Purpose | +| --- | --- | +| `GET/PUT /api/assistant/settings` | Read or save the typed selection; old endpoint requests remain compatible | +| `POST /api/assistant/copilot/status` | Inspect dedicated-profile authentication; optional `include_models` | +| `POST /api/assistant/copilot/login` | Start the explicit device flow | +| `GET/DELETE /api/assistant/copilot/login/{operation_id}` | Read/cancel that login operation | +| `POST /api/assistant/test` | Run the fixed synthetic connection test | +| `rove assistant configure` | Choose Copilot, existing endpoint or Foundry through the owning local server | +| `rove assistant login`, `login-status`, `cancel-login`, `auth-status`, `models` | Equivalent authentication/discovery controls | + +See the [CLI setup examples](../product/assistant-setup.md) for exact flags. Assistant +CLI operations use a literal loopback server URL; ordinary campaign execution remains +available headlessly through shared services. + +## Alternatives and consequences + +**Overwrite local choices with a new default.** This would change working deployments +and undo explicit disablement. Apply the default only when no saved row exists. + +**Reuse every ambient GitHub identity.** Convenient but difficult to explain and +isolate. A dedicated profile makes the user's sign-in and runtime identity explicit. + +**Accept arbitrary Azure token destinations.** This would expose credentials to +untrusted hosts. Bound this first implementation to validated public model resources; +additional endpoints require a reviewed contract. + +**Use a Foundry project/agent identifier as a model.** These are different API and +runtime contracts. Keep model-provider assistant setup separate from agent strategies. + +## Validation boundary + +Required checks cover migration/default/disabled preservation, override precedence, +typed option validation, dedicated-profile authentication, device-flow lifecycle, +explicit model discovery, synthetic test failures, token redaction and permitted +Azure destinations, UI stale-response handling and CLI parity. A preserved local +endpoint can be tested independently of GitHub/Azure credentials. Live authentication +and inference require actual user access and must be reported separately by provider. + +Source: [selection](../../src/rove/runtime/assistant.py), +[runtime](../../src/rove/runtime/copilot.py), +[login lifecycle](../../src/rove/runtime/assistant_login.py), +[API](../../src/rove/api/assistant.py), and +[CLI](../../src/rove/ui/assistant_cli.py). diff --git a/docs/architecture/README.md b/docs/architecture/README.md index ad0b31c..6cf42ad 100644 --- a/docs/architecture/README.md +++ b/docs/architecture/README.md @@ -20,20 +20,22 @@ Each new decision includes a diagram, alternatives, consequences and validation | [018: Trial lineage and snapshots](ADR-018-trial-lineage-and-snapshots.md) | Accepted | Shared trial recording, case revisions, baseline references and exploratory promotion implemented | | [019: Relational storage and assets](ADR-019-relational-storage-and-assets.md) | Accepted direction | Shared trial/case/review schema and managed images implemented; campaign journal remains separate | | [020: SME-reviewed datasets](ADR-020-sme-reviewed-datasets.md) | Accepted | Local review drafts, immutable corrections and previewed dataset freezing implemented | -| [021: Campaign success and reporting](ADR-021-campaign-success-and-reporting.md) | Accepted | Success contracts, preview and assessment comparison implemented; aggregate targets pending | -| [022: Trial telemetry](ADR-022-trial-telemetry.md) | Proposed | Durable event and inspector foundation implemented; aligned traces and complete external export pending | +| [021: Campaign success and reporting](ADR-021-campaign-success-and-reporting.md) | Accepted | Success contracts, aggregate targets, preview and assessed comparison implemented | +| [022: Trial telemetry](ADR-022-trial-telemetry.md) | Accepted direction; local implementation delivered | Durable events, clock-aware lanes, native/stage correlation and bounded loopback export; hosted collectors remain separate | | [023: Evaluation semantics and execution services](ADR-023-evaluation-and-execution-harnesses.md) | Accepted direction, amended | Existing pipeline and shared recording retained; complete robot lifecycle contracts pending | -| [024: Copilot runtime and observability](ADR-024-copilot-runtime-and-observability.md) | Accepted direction | Optional pinned runtime and controlled transport proof implemented; live provider/collector validation pending | +| [024: Copilot runtime and observability](ADR-024-copilot-runtime-and-observability.md) | Accepted direction | Optional pinned runtime, controlled transport and local-provider checks; hosted provider/collector validation remains separate | | [025: Workflow navigation](ADR-025-workflow-navigation.md) | Accepted | Campaign workspace, local mock journey and responsive gallery verified | -| [026: Versioned strategy catalog](ADR-026-versioned-strategy-catalog.md) | Accepted | Immutable local strategy catalog and multi-strategy execution tested; editor browser integration pending | +| [026: Versioned strategy catalog](ADR-026-versioned-strategy-catalog.md) | Accepted | Immutable local catalog, shared selection and reviewed strategy editor; campaign snapshots retain resolved endpoints | | [027: Versioned evaluation iterations](ADR-027-trial-to-campaign-improvement.md) | Accepted | Shared CLI, managed trial handoff and validated iteration timeline implemented; full mock browser loop and narrow timeline verified | -| [028: ABC episode import](ADR-028-abc-episode-import.md) | Accepted | Shared import design reuses upstream conversion, existing Case records and managed source assets; see ADR for validation status | +| [028: ABC episode import](ADR-028-abc-episode-import.md) | Accepted | Shared CLI/API/UI importer reuses upstream conversion and managed source assets; real/sim sample validation recorded in ADR | | [029: Workspace interaction consistency](ADR-029-workspace-interaction-consistency.md) | Accepted | Compact campaign list, reviewed strategy changes and unclipped accessible overlays | | [030: Foundry-inspired visual language](ADR-030-foundry-visual-language.md) | Accepted; navigation/setup refined by ADR-031 | Shared Fluent tokens, evaluation list, controls and theme contrast checks | | [031: Shared evaluation workspace](ADR-031-shared-evaluation-workspace.md) | Implemented; verification limits recorded | Peer Trials/Campaigns browsers, shared strategy selection, persistent campaign identity and explicit scoring semantics | - | [032: Trial review and navigation](ADR-032-trial-review-and-navigation-state.md) | Accepted | Gated setup, explicit review-to-execution boundary and unsaved-work protection | +| [033: Isolated VLA runtime and trial inspection](ADR-033-isolated-vla-runtime-and-trial-inspection.md) | Implemented locally; scope limited | Native SmolVLA inference, execution-ineligible probes and trace-backed repair; no physical-success claim | +| [034: Persistent assistant settings](ADR-034-persistent-assistant-settings.md) | Implemented locally | Saved selection, explicit provider tests and validated summary references | + ## How the decisions connect ```mermaid @@ -66,10 +68,6 @@ flowchart TD This diagram shows decision dependencies, not a statement that every component is implemented. -- [033: Isolated VLA runtime and trial inspection](ADR-033-isolated-vla-runtime-and-trial-inspection.md) — native inference, probe boundaries and evidence-backed improvement. - -- [034: Persistent assistant settings](ADR-034-persistent-assistant-settings.md) — saved selection, explicit provider testing and useful summary recovery. - ## Historical decisions ADRs 001–016 record earlier choices. They are retained to explain context, and may contain planned behavior that differs from current code. The [older architecture overview](high-level-architecture.md) is also historical. In particular, [ADR-012](ADR-012-evaluation-provenance-and-failure-attribution.md) has a current implementation note about quick-history provenance and reproducibility limits. diff --git a/docs/architecture/current-implementation.md b/docs/architecture/current-implementation.md index e63e2ad..50bb99e 100644 --- a/docs/architecture/current-implementation.md +++ b/docs/architecture/current-implementation.md @@ -15,7 +15,7 @@ The trial interface uses shared neutral tokens and viewport-bounded tooltips. ROVE runs configurable robotics evaluation pipelines from a local dashboard and a repeated-trial campaign runner. Both now create durable trial identities and frozen configuration records before execution, retain available intermediate events, and -expose saved evidence through Trial history. The optional Copilot adapter hosts +expose saved evidence through the Trials browser and inspector. The optional Copilot adapter hosts perception, planning and verification stages; customer adapters remain directly callable. Versioned cases, success contracts, SME review, dataset freezing and baseline comparisons extend that foundation. The linked [product concepts](../product/concepts.md) and @@ -31,6 +31,15 @@ provider connectivity. Campaign summary links now reach this section, with a ret path to the campaign. A real local Copilot/Qwen summary passed evidence-reference validation; unavailable and invalid replies remain explicit rather than invented AI. +The current provider extension is implemented and under verification in +[ADR-035](ADR-035-assistant-provider-selection.md): Copilot is the default only with no +saved row; existing endpoint/Disabled settings survive migration. Explicit ROVE-profile +GitHub device authentication and model discovery are separate from Test connection. +Foundry uses a validated public Azure model resource plus deployment, with an ephemeral +Azure CLI bearer token. This is distinct from the Foundry Agent strategy adapter. +No authenticated GitHub/Azure model test is claimed here; see the +[setup guide](../product/assistant-setup.md) for the exact CLI/UI contract. + ## Trial-to-campaign handoff and guided improvement The implemented refinement connects chat trial results to **Add to campaign**, then @@ -57,84 +66,74 @@ mock behavior and no-change comparison, not live AI, model improvement or hardwa [ADR-027](ADR-027-trial-to-campaign-improvement.md) links implementation and tests. Earlier campaign-only browser evidence below has its original narrower scope. -## Workflow navigation - -The existing shared navigation and saved campaign/trial deep links were delivered in -PR #21. User feedback found that route consistency left the workflow unclear: case -selection, contracts, chat, datasets and execution still competed for attention. - -The accepted amendment is **Start, Evaluate, Results**, with **Settings** separately -on the right. The latest accepted refinement is **Cases → Configure → Success metrics -→ Run → Results**, implemented with local regression and mock browser verification. It -uses an image/task gallery dialog and selected-case cards, existing strategies from -configuration, plain-language success criteria with draft suggestions, and an optional -assistant with the campaign context. Success metrics separates assessment agreement -from strategy selection. Run exposes the planned trial count and each recorded trial's -case, strategy and attempt identity. Dataset actions use case-collection wording; -adding to a collection requires a new revision retaining its earlier membership. - -The five-stage amendment extends the existing DOM behavior regression coverage with -metrics drafting/confirmation, strategy progress and graph-first Results checks. Run -confirms the selected strategy and exact scoring criteria. Case expectations are keyed -by immutable case revision in the saved contract; scoring edits and late save responses -cannot silently activate an outdated preview. Saved collection cards load their actual -members, and adding cases preserves the old revision and its review references. - -Active campaign progress polls every two seconds while the Run page is visible, and -active trial events update each strategy overview. Inline -pipeline details fetch recorded stages/output on expansion and manual refresh, showing -up to 200 events with a full-inspector link. They do not stream tokens. Tests cover these -paths alongside draft preservation, search/selection, deep links and no execution from -navigation. Browser verification completed the desktop dark-theme journey with three -isolated mock trials, plus narrow-shell checks in both themes. A stale Review summary -found during that run now refreshes on entry and has a regression test. Saved Run -links show recorded configuration, and campaign refresh retains expanded trial details. -The 390px case gallery was verified in both themes; mock execution does not -establish model quality or physical robot performance. -The [campaign workspace](../product/user-journey.md) and -[ADR-025](ADR-025-workflow-navigation.md) define the requirements and evidence checklist. -Existing API, recording and scoring contracts remain authoritative below. - -The five-stage refinement adds an individual progress overview for each chosen -strategy. Results leads with assessment-derived charts, then a compact optional AI -explanation and expandable **Inspect trials** evidence. Passed/failed/unknown counts, -execution status and assessment coverage remain separate. Unavailable metrics are -not converted to zero; unresolved pass@k/pass^k bounds are not confidence intervals. - -The implemented insight endpoints reuse the configured assistant runtime in read-only -mode. `/api/campaigns/metrics-draft` produces a reviewable success-contract draft from -tasks and annotations, with source and evidence fingerprints. It does not perform -image inspection or write SME judgments. `/api/campaigns/{id}/outcome-summary` explains -retained assessment data without changing it. The `campaign_insights` cache table in -existing `campaigns.sqlite3` keys explanations to the full assessed campaign/trials, -assistant configuration, SDK/CLI versions and prompt version. Only valid completed- -campaign AI interpretations are persisted; templates remain retryable. Evidence changes -during generation cause a regeneration error. Concurrent requests coalesce in process. -The model receives bounded saved aggregates, stage/model/status/timing evidence, -allowlisted check measurements and trial anchors. Raw prompts, outputs and trace journals -are not included. Runtime failures produce repair recommendations rather than invented -model-quality diagnoses. -Credentials and inline media are sanitized; reference annotations remain task context. -Missing/failing providers and incomplete campaigns have distinct template explanations. -The source is [`insights.py`](../../src/rove/benchmarks/insights.py), its -[API router](../../src/rove/api/insights.py) and the shared -[assistant configuration](../../src/rove/runtime/assistant.py). Fake-runtime tests in -[`test_campaign_insights.py`](../../tests/test_campaign_insights.py) verify draft and -summary boundaries without live inference. - -The isolated five-stage browser campaign `0dcb416a21a755b2be325b61b061df0c` ran two mock -strategy revisions on one case with three repetitions. Six persisted executions had -six unassessed outcomes in the graphs, with recorded strategy stages and collapsed -trial inspection. The 390px Results view had no horizontal overflow. This evidence -establishes local workflow behavior, not live assistant or hardware performance. - -Creation uses **Create a campaign**, and **Run campaign** is an execution action on that -record. **Review results** loads recorded evidence. Baseline creation names a completed -campaign strategy and preserves its assessment snapshot; it is not another execution. -The primary **Set as baseline** action and Results **Baseline** badge expose that role, -while pinning and revision history remain secondary. The badge must derive from saved -baseline references. Comparison preparation is read-only until **Start comparison -campaign** explicitly dispatches fresh candidate trials. +## Shared workflow, scoring and results + +**Trials** and **Campaigns** are peer saved-record browsers. Home and browsers use +**New trial** and **New campaign** consistently; Settings is separate on the right. +Quick work follows Case, Configure, Review & run and Results. Campaigns extend it +through Cases, Configure, Success metrics, Run and Review & improve. Both render +[`strategy-table.js`](../../frontend/strategy-table.js) for checkbox selection and +configured stage endpoints. Configuration checks show inline pending/ready/blocked +feedback; server preview remains authoritative. + +Quick review shows the observation, full task, optional robot file and selected +strategies before an explicit Run trials action. Inputs lock during execution. Step +navigation, inspection and Enter in the task field do not run a model. Unsaved-work +navigation has a stay/leave boundary. Results retains All strategies and an exact +Inspect trial & traces link; opening one result does not discard the comparison. +[ADR-031](ADR-031-shared-evaluation-workspace.md), +[ADR-032](ADR-032-trial-review-and-navigation-state.md) and +[ADR-033](ADR-033-isolated-vla-runtime-and-trial-inspection.md) record the changes and +validation scope. + +The campaign header keeps name, attempts per case and the trial total visible across +stages. Saved campaign setup is read-only from its recorded spec; the attempt count is +its seed-list length, including nonconsecutive seed values. Improve creates a linked +editable copy rather than mutating history. Case selection uses a bounded image/task +library; saved datasets supply exact case membership and review choices. + +Success metrics separates trial-level pass conditions and manual/configured checks +from report statistics. A description is not executable logic. Human criteria stay +unassessed until reviews exist; configured criteria require their actual verifier and +evidence. Contracts, targets and annotation bindings remain under Technical settings. +A new draft's Results stage has a focused no-results state. Saved results display +outcomes, per-strategy charts, optional interpretation and inspectable trials, with +scoring/baseline details secondary. Foreground completion opens results automatically. + +The insight endpoints are read-only uses of the configured assistant runtime. +`/api/campaigns/metrics-draft` prepares reviewable criteria from tasks and annotations, +not image inspection or SME judgments. `/api/campaigns/{id}/outcome-summary` interprets +bounded saved stage/model/status/timing/check evidence and allowlisted trial anchors; +it cannot change scores. It distinguishes a missing provider, failed generation and +incomplete campaign. Valid AI summaries are cached in `campaign_insights` against the +assessed evidence, assistant/runtime configuration and prompt version. Changed evidence +invalidates generation/cache identity; templates remain retryable. Raw prompts, action +arrays and private trace journals are not automatically sent to the assistant. + +Sources: [`insights.py`](../../src/rove/benchmarks/insights.py), +[API](../../src/rove/api/insights.py), [assistant selection](../../src/rove/runtime/assistant.py) +and [insight tests](../../tests/test_campaign_insights.py). A real local Qwen summary +passed the evidence-reference validator in the ADR-034 check; that is not proof of +recommendation quality or general provider support. + +Progress uses recorded trial slots and stage events. It does not claim token streaming. +Execution status and outcome acceptance remain independent; unknown metrics are not +zero and unresolved pass@k/pass^k bounds are not confidence intervals. Set as baseline +captures one completed strategy's assessment snapshot. A preview or review operation +never dispatches new trials; a comparison requires explicit execution. + +## Action inference is not full episode execution + +The local LeRobot adapter/worker gathers action outputs from the same prepared batch. +There is no generic execute/observe/replan loop that receives a fresh robot observation +after each chunk. A complete integration could put many chunks inside one trial; they +are not independent reliability samples. Strict inputs, explicit probe limits and +URDF-based diagnostics do not fill this missing episode/outcome layer. See +[action and episode scope](../product/action-and-episode-evaluation.md) for the current +boundary and the proposed industrial pick-and-place milestone. The separate +[ABC bimanual executor](../product/abc-pi05-comparison.md) now supplies a simulator +episode loop through the reusable evaluation core. Its integration and CPU simulator +checks do not validate GPU model performance or arbitrary physical robot control. ## Strategy selection and saved revisions @@ -153,7 +152,8 @@ from the current selected strategy, so a changed source is not silently treated archived baseline. [ADR-026](ADR-026-versioned-strategy-catalog.md) records this boundary; the implemented shared loader, API and local regression evidence are linked there. The revision editor adds new saved candidates without rewriting YAML or starting trials; -its browser integration remains pending. +the browser exposes reviewed stage alternatives and exact change previews through +Improve; see [ADR-029](ADR-029-workspace-interaction-consistency.md). ## Two Entry Paths, One Pipeline @@ -527,7 +527,7 @@ The campaign routes are implemented in current HTTP schema. There is no supported `rove evaluate` batch command or `rove.evaluate()` convenience function. -Open the existing **Trial history** page to reopen a saved attempt without +Open the **Trials** browser to reopen a saved attempt without executing it; the navigation consolidation places it under **Results**. The shared store adds regression tests for [storage](../../tests/test_trial_store.py), [quick/campaign recording](../../tests/test_trial_integration.py), diff --git a/docs/architecture/system-diagram.md b/docs/architecture/system-diagram.md index 3985aef..9322a65 100644 --- a/docs/architecture/system-diagram.md +++ b/docs/architecture/system-diagram.md @@ -1,66 +1,60 @@ -# ROVE System Diagram: Processes, Data and External Integrations +# ROVE system: execution, evidence and integrations -**Status:** Target system design; new runtime, recording and integration components are pending. -**Updated:** 2026-09-12 +**Status:** Current local process boundaries, with future integrations explicitly +labelled. Broader robot lifecycle requirements below remain a target contract. +**Updated:** 2026-09-14. -The first deployment remains a local application. These boundaries explain how -the [logical architecture](target-architecture.md) maps to processes and storage; -they do not require microservices, Kubernetes or a cloud account. +The local application serves both interactive strategy comparison and automated +campaigns. It does not require microservices or a cloud account for mock operation. +See [current implementation](current-implementation.md) for source/test evidence and +[logical architecture](target-architecture.md) for responsibilities. -## System Diagram - -Solid arrows show the intended local execution and recording paths. Dashed arrows -show optional external integrations. This entire diagram is a target design; see -[current implementation](current-implementation.md) for today's separate history -paths and worker behavior. +## Current local system ```mermaid flowchart LR - B["Browser dashboard"] --> API - CLI["CLI / API client"] --> API - DATA["Customer cases and assets"] --> API - subgraph HOST["ROVE host: local workstation initially"] - API["Python application
UI API and evaluation services"] - WORK["Bounded trial execution workers"] - SDK["Copilot SDK host integration"] - RUNTIME["Copilot child runtime
Isolated role/trial sessions"] - DIRECT["Customer system adapters"] - RECORD["Durable recording boundary"] - DB[("SQLite
Trials, revisions, events and assessments")] - ASSET[("Asset files
Images, video and signal ranges")] - API --> WORK - API -->|"Assistant sessions"| SDK - WORK -->|"Configured agent or grader"| SDK - SDK --> RUNTIME - WORK --> DIRECT - API --> RECORD - WORK --> RECORD - SDK -->|"Session events and usage"| RECORD - RECORD --> DB - RECORD --> ASSET - DB --> API - ASSET --> API - end - RUNTIME -.-> MODEL["Azure / Foundry or other configured model provider"] - DIRECT -.-> CUSTOMER["Customer agent or model endpoint / local process"] - WORK -.-> ENV["Configured simulator or robot integration
Future execution contract"] - ENV -.->|"Observations and action acknowledgements"| RECORD - RUNTIME -.->|"Native OTLP"| COL["Optional OpenTelemetry Collector"] - WORK -.->|"ROVE spans and metrics"| COL - API -.->|"ROVE spans and metrics"| COL - COL -.-> AZ["Azure Monitor
Application Insights / Log Analytics"] - COL -.-> GF["Grafana Cloud or Tempo / Loki / Mimir"] - DB -.-> EXPORT["Versioned export job: future"] - ASSET -.-> EXPORT - EXPORT -.-> LAKE["Fabric / Databricks / Delta destination"] + B[Browser: Trials and Campaigns] --> API[FastAPI and static UI] + API --> S[Shared ROVE services] + CLI[CLI: cases, trials, campaigns and reports] --> S + S --> EX[Pipeline and bounded campaign execution] + EX --> DIRECT[Configured customer/model adapters] + EX --> SDK[Optional Copilot role-scoped runtime] + API --> ASSIST[Optional evaluation assistant] + ASSIST --> SDK + ASSIST --> CONF[Validated read or confirmed mutation] + CONF --> S + DIRECT --> LP[Optional isolated local VLA worker] + DIRECT --> EP[Configured local or remote endpoint] + SDK --> PROVIDER[Selected model provider] + EX --> REC[Durable trial and evidence recording] + SDK --> REC + S --> REC + REC --> DB[(Local SQLite stores)] + REC --> ASSET[(Managed content-addressed assets)] + DB --> REPORT[Assessments, reports, baselines and timeline] + ASSET --> REPORT + REPORT --> S + REC -. Optional bounded local OTLP .-> COL[Loopback collector] + COL -. Future hosted validation .-> MON[App Insights, Log Analytics or Grafana] + DB --> X[Validated relational exchange] + ASSET --> X + X -. Future destination integration .-> LAKE[Fabric, Databricks or Delta] + EX -. Future physical lifecycle adapter .-> ROBOT[Robot or general closed-loop simulator] ``` -The worker owns dispatch into the selected environment; an SDK robotics tool calls -that same configured integration. The SDK is not a physical robot driver. Local -adapters need not call external endpoints. The initial implementation must decide -worker/SDK process reuse from isolation tests; no fixed process-per-session promise -is made here. CLI commands can invoke application services directly without an -HTTP round trip. +Ordinary CLI evaluation and data commands call application services directly. Assistant +settings/test/proposal commands contact the owning local server. Copilot hosts optional +assistant or configured agent behavior; it does not schedule every direct-model call +or act as a robot driver. The isolated VLA worker uses its own dependency environment, +not a security sandbox. Its successful inference does not establish task completion. + +SQLite stores retain trials/cases/reviews/assets metadata, campaigns and their insight +cache, strategy revisions, and local assistant selection in their respective databases. +A complete study backup preserves all relevant databases and managed assets together. +SDK session state and optional telemetry are not substitutes for those records. +ABC and JSONL/image imports enter existing Case services; original source artifacts +and private labels remain separate from candidate input. See +[storage and exchange](../product/evidence-and-exchange.md). ## Trial Lifecycle @@ -92,7 +86,7 @@ sequenceDiagram E-->>U: Outcome, timeline and evidence ``` -This target sequence preserves the distinction between execution state and task +This broader target sequence preserves the distinction between execution state and task outcome. A failed task can have a complete trace; an interrupted process can leave an unknown outcome. Reopening history never dispatches actions. An interrupted physical trial is not automatically replayed; continuing it requires the declared diff --git a/docs/architecture/target-architecture.md b/docs/architecture/target-architecture.md index c9572c1..634725f 100644 --- a/docs/architecture/target-architecture.md +++ b/docs/architecture/target-architecture.md @@ -1,7 +1,9 @@ # ROVE Architecture: Evaluation Services and Shared Agent Runtime -**Status:** Accepted direction; implementation is pending except for the existing foundations identified below. -**Updated:** 2026-09-12 +**Status:** Accepted logical architecture. The core services, optional Copilot +runtime, durable records, assistant and local inspection are implemented; general +physical lifecycle tools and hosted analytics integrations remain future work. +**Updated:** 2026-09-14 **Decision:** [ADR-024](ADR-024-copilot-runtime-and-observability.md) **ROVE evaluates robotics agent pipelines on your task.** Copilot SDK supplies the @@ -11,9 +13,10 @@ their own runtime and decision-making behavior. ## Architecture Diagram -This is the target logical architecture, not a deployment inventory or a claim -that the proposed assistant, recorder and integrations already exist. Arrows show -requests or data delivery; they do not imply that each component is a new service. +This is a logical architecture rather than a deployment inventory. Existing local +services and optional runtime roles are shown together with future environment tools. +Arrows show requests or data delivery, not separate microservices. Use the +[system diagram](system-diagram.md) for current process and storage boundaries. ```mermaid flowchart TB @@ -24,9 +27,9 @@ flowchart TB CORE --> DISPATCH["Configured stage and trial execution"] DISPATCH --> HOSTED["ROVE-hosted strategy agent
Copilot SDK"] DISPATCH --> DIRECT["Direct customer agent / VLM / VLA adapters"] - HOSTED --> TOOLS["Configured robotics tools and integrations"] + HOSTED --> TOOLS["Adapter actions; future SDK robotics tools"] DIRECT --> TOOLS - TOOLS --> ENV["Recorded inputs or fresh environment execution
according to the evaluation contract"] + TOOLS --> ENV["Recorded inputs and synthetic test execution
General physical lifecycle: future"] ENV --> EVID["Outputs and available observations"] HOSTED --> EVID DIRECT --> EVID @@ -50,10 +53,10 @@ tools. Candidate inputs exclude grader-only labels and SME reference material. ## Responsibilities and Contracts -| Boundary | Responsibility | Contract to implement | +| Boundary | Responsibility | Contract and remaining extension | | --- | --- | --- | | UI/API/CLI → evaluation services | One behavior across quick runs and campaigns | Validate configuration, assign stable IDs and freeze revisions before dispatch | -| Assistant → evaluation services | Prepare, launch and investigate evaluations | Typed tools call the same services as the UI; tool retries cannot duplicate campaigns | +| Assistant → evaluation services | Prepare and investigate evaluations; confirm selected mutations | Typed tools use the same services; exact host confirmations guard writes and operation IDs prevent duplicates | | Trial executor → configured system | Run the selected system with declared conditions | Start, progress, cancel, terminal reason, memory/reset scope and telemetry coverage | | System → environment | Observe and act when supported | Input/output types, action units/frames, reset acknowledgement and execution evidence | | Evidence → grader | Apply a declared evaluation contract | Versioned rubric, required constraints and independent assessment identity | diff --git a/docs/product/action-and-episode-evaluation.md b/docs/product/action-and-episode-evaluation.md new file mode 100644 index 0000000..630d429 --- /dev/null +++ b/docs/product/action-and-episode-evaluation.md @@ -0,0 +1,115 @@ +# Action comparison and episode evaluation + +**Status:** Local LeRobot inference boundary and episode integration scope, 2026-09-15. +ROVE's local LeRobot adapter generates actions from a supplied observation. It does +not execute those actions, obtain new observations and continue until the task finishes. +The separate [ABC bimanual integration](abc-pi05-comparison.md) now implements +complete simulator episodes through the reusable evaluation core. Its CPU simulation +and integration checks do not establish GPU policy performance or physical success. + +A robotics team needs both levels, but they answer different questions: + +| Level | Question | Evidence required | +| --- | --- | --- | +| Action-level comparison | Can this configured model process this observation/state, and what actions does it produce? | Exact input, checkpoint/processors, returned actions, timing and disclosed assumptions | +| Episode-level evaluation | Does this complete strategy accomplish the task, how reliably and at what cost? | Controlled initial conditions, executed actions, fresh observations, terminal outcome and applicable constraints | + +## An action chunk is not a whole task + +A chunk is a sequence of actions covering part of an execution horizon. Robot systems +typically need further inference conditioned on updated observations. Chunk length, +control rate, how much of each chunk executes and inference delay all affect behavior. +Physical Intelligence's [real-time chunking work](https://www.pi.website/research/real_time_chunking) +describes coordinating successive chunks while the robot moves. LeRobot's +[asynchronous inference design](https://huggingface.co/docs/lerobot/async) separates +policy inference from robot-side observation and action execution. Neither capability +is implied merely by loading a VLA inside ROVE. + +The current in-process LeRobot adapter and isolated worker call `select_action` using +the same prepared input batch while collecting the requested action output. The policy +may consume its internal action queue or perform another inference, but it does not +receive an updated physical observation from ROVE during that loop. More returned +actions therefore do not establish a completed closed-loop episode. + +```mermaid +flowchart LR + O[One supplied image and state] --> P[Current VLA inference] + P --> A[Generated actions] + A --> R[Recorded trial outputs, timing and assumptions] + A --> D[Applicable configured diagnostics] + R --> I[Inspect and compare strategies] + D --> I +``` + +This remains useful for integration checks, incompatible inputs, inference latency, +action-space analysis and model diagnostics. For an observation-only probe, state or +views may be missing; its output remains explicitly execution-ineligible and cannot +be fed to dynamics or simulation as though it were a validated robot command. + +## What a complete episode integration supplies + +```mermaid +flowchart TD + C[Case, strategy and declared initial conditions] --> RESET[Reset and acknowledge environment] + RESET --> O[Fresh observation and measured state] + O --> P[Strategy infers action chunk] + P --> EXEC[Integration executes allowed actions] + EXEC --> RECORD[Record action acknowledgements, observations and timing] + RECORD --> DONE{Terminal outcome or budget reached?} + DONE -->|No| O + DONE -->|Yes| G[Assess task outcome and constraints] + G --> T[One completed episode trial] +``` + +The [ABC executor](../../src/rove/bimanual/adapter.py) and +[episode loop](../../src/rove/bimanual/episode.py) implement this lifecycle for the +supported simulator. Other environments need their own reset/initial-state acknowledgements, camera +and state alignment, action/control conventions, budgets, cancellation behavior, +measurement sources and an independent terminal evaluator. A hardware or simulation +integration should own those mechanics; ROVE should retain the trial, configuration, +trace and assessment. A copied recording cannot stand in for fresh candidate actions. + +One trial may contain many observations and action chunks when its integration runs a +complete episode. Those chunks are correlated steps inside one attempt, not independent +trials for pass@k or pass^k. Repeated episode trials start from the declared initial +conditions and reset policy. Fair strategy comparisons match the initial task and +environment conditions; later observations naturally diverge as strategies act. +Forcing every candidate to consume the same later recorded frames instead evaluates +an offline replay, which answers a different question. + +## Further industrial validation: one measurable task + +To help a robotics team choose a reliable task solution, the next recommended slice +is one industrial pick-and-place task on one declared robot/environment, with a +resettable initial scene and an independently measurable object goal. Connect two +executable strategies, run repeated complete episodes, record the fresh observations, +action acknowledgements and terminal outcome, then compare success, completion time +and applicable intervention/constraint evidence. + +An agent calling robot tools and a VLA producing action chunks can be compared when +each is integrated into a complete executable strategy for those same initial +conditions. Raw models with incompatible observation/action interfaces are not +interchangeable candidates. Their adapters, controllers and recovery behavior belong +to the strategy identity. + +This is a **recommended next milestone, not an approved implementation commitment**. +Current offline image/task/URDF screening and FK checks cannot determine which system +most reliably completes that industrial task. Additional geometric diagnostics or +more polished reports do not close the execution-and-outcome gap. + +## Robot compatibility is more than a URDF + +A URDF describes robot structure and may support applicable geometric/kinematic +checks. Supplying one does not make an arbitrary checkpoint understand or control +that robot. The model still needs compatible camera inputs, measured state ordering, +embodiment, action representation, normalization and controller interpretation. +Unverified mappings remain limitations rather than silently inferred compatibility. + +In the current release, the local SmolVLA probe establishes inference and recording +under its declared assumptions. It does not establish task completion, general +multi-camera support, Pi0.5 availability or transfer to a user-provided robot. + +Sources: [LeRobot adapter](../../src/rove/adapters/local_lerobot.py), +[isolated worker](../../src/rove/adapters/lerobot_worker.py), +[ADR-033](../architecture/ADR-033-isolated-vla-runtime-and-trial-inspection.md), and +[runtime preparation](../../examples/runtime/lerobot/README.md). diff --git a/docs/product/assistant-setup.md b/docs/product/assistant-setup.md new file mode 100644 index 0000000..0804d7c --- /dev/null +++ b/docs/product/assistant-setup.md @@ -0,0 +1,114 @@ +# Choose an evaluation assistant + +**Status:** Provider selection and authentication support implemented with controlled +regression coverage. Authenticated GitHub Copilot and Azure model inference +are not claimed by this documentation update. +**Updated:** 2026-09-15. [Decision: ADR-035](../architecture/ADR-035-assistant-provider-selection.md). + +The assistant drafts success criteria, explains recorded outcomes and prepares +reviewable workflow operations. It does not replace the strategies under evaluation, +rewrite scores or manufacture expert judgments. Models chosen for the assistant are +independent of the models selected in a campaign's strategy table. + +Open **Settings → Assistant** from the gear or a campaign's Configure assistant link. +Choose a provider, save it, explicitly test the connection, then return to the campaign. + +| Choice | What to provide | Authentication | +| --- | --- | --- | +| GitHub Copilot | Use the default model or select an available model | Sign in through ROVE's explicit GitHub device flow | +| Existing endpoint | A configured `copilot_agent` endpoint, such as the local assistant | Existing server-side provider configuration | +| Microsoft Foundry | Azure model resource URL and deployment name | Azure CLI login on the machine running ROVE | +| Disabled | No provider | No assistant inference; manual workflow and saved results remain available | + +A **new setup with no saved selection** defaults to GitHub Copilot. Existing saved +local selections and explicitly disabled settings are preserved. A default choice is +not proof of authentication or a working model connection. An explicit host endpoint +or `ROVE_ASSISTANT_ENDPOINT` overrides saved settings and appears read-only. + +## GitHub Copilot + +Use **Sign in** to start a device flow, follow the returned verification address and +code, and let ROVE report its result. Authentication uses a dedicated ROVE Copilot +profile; signing into GitHub elsewhere, including `gh`, is not automatically reused. +Cancel stops the pending ROVE login operation. It does not log other applications out. + +Checking sign-in and discovering available models are explicit operations. Leaving +model selection at default lets Copilot choose its default rather than silently +substituting a local endpoint or an invented model ID. Save then Test connection +before requesting an explanation. SDK/CLI presence, authentication, model discovery +and successful test inference are separate checks. + +With ROVE running locally: + +```sh +rove assistant configure --provider copilot +rove assistant login +rove assistant login-status LOGIN_OPERATION_ID +rove assistant auth-status +rove assistant models +rove assistant test +``` + +Use `rove assistant configure --provider copilot --model MODEL_ID` for an explicitly +selected available model. `rove assistant cancel-login LOGIN_OPERATION_ID` cancels a +pending sign-in. Add `--url http://127.0.0.1:5010` to each command when that is the +running app's address; the default is port 5001. + +## Existing local or custom endpoint + +Select an existing Copilot adapter endpoint from `rove.yaml`. This retains your +configured provider URL/model and server-side credentials; it does not create a new +model server. The CLI equivalent is: + +```sh +rove assistant configure --endpoint local-assistant +rove assistant test +``` + +The chosen endpoint must exist in the running server's configuration and use the +`copilot_agent` adapter. A local OpenAI-compatible server needs to be running and +serving the configured model. A previous successful local Qwen test does not prove +that every custom endpoint or model supports the required structured responses. + +## Microsoft Foundry model deployment + +Supply the **model resource** endpoint and **deployment name**, not a project URL or +Foundry Agent identifier. Supported public Azure host patterns are +`https://RESOURCE.services.ai.azure.com` and `https://RESOURCE.openai.azure.com`; +ROVE normalizes the model API path to `/openai/v1/`. URLs with credentials, query +strings, unrelated hosts or project/agent paths are rejected. + +Sign into Azure CLI on the ROVE host using `az login` with an account allowed to call +the deployment. ROVE uses `AzureCliCredential` for a short-lived bearer token at +execution time. The browser does not accept an Azure password or access token, and +assistant settings store the endpoint/deployment rather than the token. Installing +dependencies or signing into Azure does not provision a resource or grant permission. + +```sh +rove assistant configure --provider foundry \ + --foundry-endpoint https://YOUR_RESOURCE.services.ai.azure.com \ + --deployment YOUR_DEPLOYMENT +rove assistant test +``` + +This is a model-provider path through the Copilot runtime. It is distinct from a +campaign strategy that invokes an existing Foundry Agent using its project and agent +configuration. Cloud permissions and actual model inference still require validation +against the customer's resource. + +## What saving and testing do + +Save changes only the local provider selection, keyed by configuration path. It does +not start inference or alter old campaigns. **Test connection** sends a fixed +synthetic JSON request without case images, tools or private trial data. It succeeds +only after a valid response; a later outage or unsupported workload can still fail. + +`rove assistant settings` reads the saved/effective selection. `rove assistant +disable` persists Disabled across restarts. Provider/authentication selection does not +authorize assistant-proposed case edits or campaign execution: those still require +the [exact operation confirmation](named-baselines-and-assistant.md). + +For outcome summaries, the assistant receives bounded saved evidence and permitted +references. Raw trial prompts, action arrays and private journals are not automatically +forwarded. A generated explanation is interpretation, not a new measurement or a +physical-success judgment. See [traces and measurements](traces-and-measurements.md). diff --git a/docs/product/cli-ui-parity.md b/docs/product/cli-ui-parity.md index 697300c..959f3ec 100644 --- a/docs/product/cli-ui-parity.md +++ b/docs/product/cli-ui-parity.md @@ -39,6 +39,8 @@ command for request options. Read commands emit JSON; progress and errors use st | Track iterations | `rove campaign timeline ID` | Connected, validated source/baseline lineage; case/criteria/strategy changes are visible, unrelated studies stay separate | | Strategy revisions | `rove strategy list`, `show ID`, `revisions`, `preview FILE`, `save FILE` | Uses `--config PATH`; preview and parent fingerprints protect immutable variant creation | | Configuration | `rove config show`, `models`, `validate` | Uses `--config PATH`; sanitized configuration discovery and validation, not live endpoint health | +| Assistant provider setup | `rove assistant settings`, `configure --provider copilot`, `configure --endpoint ID`, `configure --provider foundry --foundry-endpoint URL --deployment NAME`, `test`, `disable` | Uses the owning local server; [setup and verification limits](assistant-setup.md). New default never overwrites saved local/disabled selection | +| Copilot sign-in and discovery | `rove assistant login`, `login-status ID`, `cancel-login ID`, `auth-status`, `models` | Explicit dedicated ROVE profile; no automatic reuse of GitHub CLI credentials; discovery is separate from inference | | Optional assistant | `rove assistant status`, `ask FILE --proposal-output FILE`, `confirm FILE --confirmed` | Uses the existing local server via `--url`; proposal files are private, and execution requires explicit confirmation | | Recent dashboard history | `rove history list`, `show ID`, `remove ID` | Uses the local server via `--url`; removal affects recent dashboard history, not durable trial evidence | | Portable study state | `rove exchange export BUNDLE.zip --store STORE`, `import BUNDLE.zip --store NEW_STORE` | Relational records and managed assets, validated on import | diff --git a/docs/product/concepts.md b/docs/product/concepts.md index 32b742d..15584e6 100644 --- a/docs/product/concepts.md +++ b/docs/product/concepts.md @@ -14,8 +14,9 @@ for example, whether a planning agent produces an acceptable placement plan. It does not require a VLA or a complete robot recording to be useful. This page defines the product vocabulary and distinguishes it from the current -implementation as of the customer workflow slice on 12 September 2026. Versioned +implementation as of 14 September 2026. Versioned cases, explicit reviews, dataset freezing and baseline references are implemented; +bounded ABC observation imports and explicit recording references are supported; complete robot reset contracts and general recording ingestion remain future work. ## One Attempt Has One Identity @@ -27,7 +28,7 @@ not an additional persisted container. A **trial** is one strategy attempting that case once. A **campaign** organizes trials across selected cases, strategies and repetitions. Selecting three strategies for one case produces three trials, even if the user selects **Run campaign** -once. **Create a campaign** prepares this work; **Run campaign** executes its trials; +once. **New campaign** prepares this work; **Run campaign** executes its trials; **Review results** reads their saved evidence. Evaluation and run describe activities, not additional persisted containers. @@ -45,9 +46,17 @@ This hierarchy now has local implementation support. A quick trial needs no camp name before execution. Promotion preserves the same trial as an exploratory reference and creates a reusable input case without manufacturing another attempt. New planned repetitions receive their own identities. +An action chunk is part of an attempt, not a separate trial. The current local VLA +path returns actions from one prepared observation; it does not run an entire +execute/observe/replan episode. A complete customer/environment integration could +supply many chunks within one trial and independently observed termination. Compare +strategies from matching initial conditions; their later observations can diverge. +See [action and episode evaluation](action-and-episode-evaluation.md). + The vocabulary follows the distinction between a test case, an individual attempt and its outcome in [Anthropic's evaluation guidance](https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents). -ROVE uses **Case** in navigation and retains **task instruction** as a case field. +ROVE uses **Cases** as a campaign stage and retains **task instruction** as a case field. +**Trials** and **Campaigns** browse saved work; neither is another name for a case. An Experiment or Session container is not needed alongside Campaign. | Concept | Meaning | Robotics example | diff --git a/docs/product/evaluation-workflows.md b/docs/product/evaluation-workflows.md index 32f7c9a..be5c7f0 100644 --- a/docs/product/evaluation-workflows.md +++ b/docs/product/evaluation-workflows.md @@ -8,6 +8,16 @@ The [product specification](../PRODUCT_SPEC.md) retains the founding vision and the three personas below. Configurable agents, human-reviewed datasets and business-outcome reports extend that vision. A supported agent or VLM can occupy the relevant stages; a VLA is not mandatory for every robotics evaluation. +## Start small and retain the comparison + +**New trial** compares one task, image and relevant robot configuration across chosen +strategies. Case → Configure → Review & run → Results preserves the original pipeline, +All strategies view and exact trace inspection. **Add to campaign** retains that trial +as an exploratory reference and carries its input/strategy into a repeated evaluation. +**New campaign** starts directly from a prepared collection. Trials and Campaigns are +peer saved-record browsers; Settings holds reusable configuration and the assistant. +See the [current journey](user-journey.md) for interaction and validation details. + ## Personas and user stories | Founding persona | Job to be done | Journey and evidence of value | @@ -26,7 +36,7 @@ An SME is a reviewer collaborating with these personas, not a replacement for th A developer may have images and robot tasks but no labeled evaluation dataset. They need an initial baseline, a way for an expert to validate cases and assess outputs, and a stable reference for comparing a changed model or strategy. Quick evaluations and campaigns now share durable trial identities and saved evidence; named baseline revisions preserve the exact configuration and assessment used for a later comparison. -ROVE should support both starting points: evaluating a prepared dataset, and creating a reviewed dataset from an initial campaign. The outcome is an explainable comparison tied to fixed inputs, criteria, configuration and evidence. Physical completion claims require outcome evidence from the relevant robot episode; expert assessment of perception and planning is useful before such episodes exist. +ROVE supports both starting points: evaluating a prepared dataset, and creating a reviewed dataset from an initial campaign. The outcome is an explainable comparison tied to fixed inputs, criteria, configuration and evidence. Physical completion claims require outcome evidence from the relevant robot episode; expert assessment of perception and planning is useful before such episodes exist. ## First value for a robotics company diff --git a/docs/product/implementation-plan.md b/docs/product/implementation-plan.md index e18252d..4108a54 100644 --- a/docs/product/implementation-plan.md +++ b/docs/product/implementation-plan.md @@ -1,7 +1,9 @@ # ROVE Delivery Plan: From Trial Evidence to Customer Evaluation Workflows -**Status:** Local completion implemented and verified; PR checks govern merge readiness. Live provider/Azure validation is specification-only at user request; hosted integrations remain explicitly unvalidated. -**Updated:** 2026-09-12 +**Status:** Original T01–T12 local delivery recorded below, with subsequent shipped +workflow/runtime refinements summarized here. Hosted integration acceptance remains +separate; local provider checks do not establish physical outcomes. +**Updated:** 2026-09-14 **Design:** [Architecture diagram](../architecture/target-architecture.md), [system diagram](../architecture/system-diagram.md), [ADR-024](../architecture/ADR-024-copilot-runtime-and-observability.md) The first useful milestone is a saved robotics trial whose configuration, agent @@ -10,7 +12,35 @@ workflow: import cases → baseline → SME review → frozen dataset → candid comparison. Copilot powers ROVE-owned agents while direct customer systems share the same evaluation records and reporting rules. -## Already Available +## Subsequent delivery beyond the original plan + +| Capability | Current evidence | +| --- | --- | +| Trial-first and shared campaign setup | [ADR-031](../architecture/ADR-031-shared-evaluation-workspace.md) and [ADR-032](../architecture/ADR-032-trial-review-and-navigation-state.md): shared strategy table, recorded setup, explicit reviewed execution and navigation guards | +| Versioned improvement and CLI workflows | [ADR-027](../architecture/ADR-027-trial-to-campaign-improvement.md): source-linked iterations, baseline snapshots, timeline and shared commands | +| ABC observations | [ADR-028](../architecture/ADR-028-abc-episode-import.md): bounded real/sim converted exports, source identities and private demonstration evidence | +| Local model/runtime checks | [ADR-033](../architecture/ADR-033-isolated-vla-runtime-and-trial-inspection.md): local Qwen and native SmolVLA; inference/probe scope only, not complete robot episodes | +| Reusable evaluation core and ABC episodes | [ADR-036](../architecture/ADR-036-reusable-evaluation-core.md) and [ADR-038](../architecture/ADR-038-abc-bimanual-episodes.md): structured-input execution and a simulator episode integration; GPU model performance remains unverified | +| Persistent assistant | [ADR-034](../architecture/ADR-034-persistent-assistant-settings.md): local selection, explicit synthetic connection tests and validated summary anchors | + +The original decision to leave live-provider validation as a specification applies to +that earlier T01–T12 delivery. Later local Qwen/SmolVLA checks are recorded separately; +Foundry/Pi0.5 and hosted monitoring coverage must be reported by actual provider and +scenario. The current [action-versus-episode boundary](action-and-episode-evaluation.md) +remains explicit: generic closed-loop execution is not implemented by these refinements. + +## Recommended next robotics milestone + +One industrial pick-and-place task, one declared resettable robot/environment and an +independently measured object goal would extend validation beyond the implemented +[ABC bimanual episode path](abc-pi05-comparison.md). Integrate two executable strategies, capture fresh observations and action +acknowledgements, repeat episodes, and report actual success and timing with evidence. +An agent using robot tools and a VLA/controller can then be compared as complete +strategies, rather than treating raw model outputs as interchangeable. This is a +recommendation awaiting scope approval, not work delivered by the current UI or +assistant update. See [action and episode evaluation](action-and-episode-evaluation.md). + +## Original foundations PR #13 supplies repeated campaigns, pass@k/pass^k, cross-strategy coverage and portable reports. PR #14 supplies configured verification, required checks, diff --git a/docs/product/metrics-and-success.md b/docs/product/metrics-and-success.md index c62906f..d60035e 100644 --- a/docs/product/metrics-and-success.md +++ b/docs/product/metrics-and-success.md @@ -21,6 +21,11 @@ flowchart TD G --> H[Report with expandable details] ``` +A repeated action chunk is not another trial. Reliability for a complete robotics +task requires independently assessed episodes, not repeated inference on an unchanged +image or counting actions inside one execution. Current VLA inference and future +closed-loop execution have [different evidence scopes](action-and-episode-evaluation.md). + ## Two levels of success | Level | Definition | Example | @@ -45,7 +50,7 @@ estimates, not an additional target type. ## Setup in configuration and the UI -The implemented Cases workflow saves a contract with assessment scope, evidence +The campaign Success metrics stage saves a contract with assessment scope, evidence mode, required criteria and selected metrics. Automated criteria bind to configured verification; human criteria require per-output ratings. Preview marks measures available, conditional, awaiting review or unavailable, explains k values exceeding repetitions, @@ -59,9 +64,8 @@ configured episode verifier supplies completion, constraint, intervention and recovery measurements with synthetic or supplied observed provenance. Explicit contract bindings can supply selected frozen SME labels to local graders while keeping candidate inputs separate. See [robotics evidence](robotics-evidence.md) -for exact schemas and supported bindings. The API/UI share these contracts; the -example runner demonstrates programmatic setup, without adding a dedicated -contract-file CLI. +for exact schemas and supported bindings. The API, UI and `rove data contract create --file FILE` share these contracts; +see the [CLI guide](cli-ui-parity.md) for scripted setup and report gates. The API and UI use the same validated, versioned contract: diff --git a/docs/product/named-baselines-and-assistant.md b/docs/product/named-baselines-and-assistant.md index b47f282..0a4e93d 100644 --- a/docs/product/named-baselines-and-assistant.md +++ b/docs/product/named-baselines-and-assistant.md @@ -7,7 +7,8 @@ to use the same services and permissions as the visible workflow. ## A baseline is a reference, not another run -The Cases workspace can name and pin a completed campaign strategy. Its baseline +Campaign Results can set a completed strategy as a named baseline; pinning and +revision details remain secondary. Its baseline revision records the campaign/configuration hash, case and dataset identities, contract, assessed trial outcomes, verifier measurements and assessment revision set. Later SME ratings do not rewrite these saved outcomes. Pinning does not certify the @@ -41,6 +42,14 @@ in the unsaved case until the user saves it. Whole-file is the default; no clock physical outcome or evidence quality is guessed. Sample/frame/time ranges require an indexed JSON recording, and typed trajectory/state metadata must match its envelope. +## Select the assistant independently of the strategy + +[Settings → Assistant](assistant-setup.md) selects Copilot, an existing endpoint, +Foundry or Disabled. New setups default to Copilot while saved choices are preserved. +Authentication, model discovery and connection testing are explicit; the assistant +provider does not replace the strategy being evaluated. The provider extension is +under verification and does not establish authenticated GitHub/Azure inference. + ## Assistant prepares; the host confirms The optional assistant can inspect cases, contracts, final reviews, datasets, baselines, @@ -93,10 +102,10 @@ write resumes the original operation rather than creating a duplicate. Prepared authorizations expire after ten minutes or a server restart. A changed head, review preview, configuration or baseline capture requires another preview. Committed -records remain available through their normal history APIs. There is no live-provider -validation claim: acceptance tests use real local services and a controlled fake runtime; -the UI journey uses a simulated API. Provider/Azure setup remains a separate documented -validation task, with no credentials or network calls required for these tests. +records remain available through their normal history APIs. The original acceptance tests used real local services and a controlled fake runtime. +Later local Qwen summary validation is recorded in ADR-034; authenticated GitHub/Azure +and hosted integrations require separate evidence. These checks do not establish +recommendation quality or physical robot success. ## Interfaces diff --git a/docs/research/vla_local_inference_guide.md b/docs/research/vla_local_inference_guide.md index 6af1ffc..151465e 100644 --- a/docs/research/vla_local_inference_guide.md +++ b/docs/research/vla_local_inference_guide.md @@ -6,38 +6,26 @@ --- -## Current ROVE checkout: optional local policy runtime - -ROVE retains the SmolVLA and pi0.5 loading path in `LeRobotVLAAdapter`. The default -development installation omits their ML dependencies. The experimental extra -can be installed in a Python 3.12+ checkout while retaining other installed extras: - -```bash -uv sync --frozen --extra smolvla --inexact -``` - -Restart ROVE afterward: the adapter checks dependency imports when the server -starts. The `smolvla` extra name covers both SmolVLA and pi0.5. Installing it -does not download model weights or validate a checkpoint, tokenizer, robot -embodiment or device. The adapter loads the configured checkpoint when a trial -requests inference. See [the optional-runtime security limitations](../../SECURITY.md#optional-lerobot-environment) -before using it; this remains an experimental environment outside the supported -default dependency audit. - -**Offline import audit, 2026-09-13:** installing the frozen extra alone does not -produce a usable policy runtime. LeRobot 0.6.1 imports its dataset package through -the policy factory and fails on missing `av`. Importing a specific policy directly -also executes the same parent package initializer. Its full dataset extra requires -`datasets<5` and `pandas<3`, while its pi/SmolVLA extras require -`transformers<5.6`; these conflict with ROVE's current dependency choices -(`datasets>=5.0.1`, `transformers>=5.10.0`). Do not work around this by downgrading -the main environment or bypassing upstream package initialization. A validated -compatible runtime or a separately integrated policy endpoint is required before -claiming local inference works. No model weights or live inference were exercised -in this audit. - -The model research below is historical reference, not validation of the current -locked runtime or every checkpoint listed. +## Current ROVE checkout: use the isolated runtime guide + +**Current update: 14 September 2026.** The supported experimental preparation path +is the [isolated LeRobot runtime guide](../../examples/runtime/lerobot/README.md), +with [ADR-033](../architecture/ADR-033-isolated-vla-runtime-and-trial-inspection.md) +recording validation and limits. Native SmolVLA inference was exercised locally; +Pi0.5 still requires authorized tokenizer access and separate validation. Do not +resolve historical dependency conflicts by downgrading the main ROVE environment. + +The worker is a separate dependency environment, not an OS security sandbox. It +loads trusted local artifacts, verifies declared content/runtime identities and +retains upstream dependency limitations. Strict input checks and explicit image-only +probes are different modes. Probes remain execution-ineligible and cannot establish +robot success. Neither mode supplies a generic execute/observe/replan episode loop. +See [action versus episode evaluation](../product/action-and-episode-evaluation.md). + +Everything below is the **February 2026 research reference** identified by the +original date. Its package commands, performance estimates and checkpoint/platform +matrix are not a current installation recipe or verified support inventory. Use the +isolated runtime guide and current adapter tests for this checkout. ## Quick Reference: Hardware Compatibility Matrix diff --git a/frontend/app.js b/frontend/app.js index 976f286..a4e34ef 100644 --- a/frontend/app.js +++ b/frontend/app.js @@ -1341,66 +1341,89 @@ async function loadExample(ex) { } function renderAssistantSettings(wrapper) { - const node=(tag,text)=>{const item=document.createElement(tag);if(text!=null)item.textContent=text;return item;}; + const node=(tag,text)=>{const el=document.createElement(tag);if(text!=null)el.textContent=text;return el;}; const section=node("section");section.id="assistant";section.className="assistant-settings";section.setAttribute("aria-labelledby","assistantSettingsHeading"); const heading=node("h2","Assistant");heading.id="assistantSettingsHeading";heading.tabIndex=-1; section.append(heading,node("p","Choose the assistant that explains campaign results and suggests evaluation criteria. This does not change the strategies being evaluated.")); - const label=node("label","Assistant endpoint");label.htmlFor="assistantEndpoint"; - const select=node("select");select.id="assistantEndpoint";select.disabled=true; - const description=node("p");description.id="assistantEndpointDescription";select.setAttribute("aria-describedby",description.id); + function field(parent,id,label,tag="select"){const caption=node("label",label);caption.htmlFor=id;const input=node(tag);input.id=id;parent.append(caption,input);return input;} + function option(select,value,label){const item=node("option",label);item.value=value;select.append(item);} + function action(parent,id,label){const button=node("button",label);button.id=id;button.type="button";parent.append(button);return button;} + const provider=field(section,"assistantProvider","Provider"); + for(const [id,name] of [["copilot","GitHub Copilot"],["endpoint","Local or own endpoint"],["foundry","Microsoft Foundry"],["disabled","Disabled"]])option(provider,id,name); + const copilot=node("div");copilot.id="assistantCopilotFields";section.append(copilot); + const model=field(copilot,"assistantCopilotModel","Model");option(model,"","Copilot default"); + const authStatus=node("p","Sign-in status has not been checked.");authStatus.id="assistantCopilotStatus";authStatus.setAttribute("role","status");copilot.append(authStatus); + const authActions=node("div");authActions.className="assistant-settings-actions";copilot.append(authActions); + const authCheck=action(authActions,"checkCopilotSignIn","Check sign-in & load models"),signIn=action(authActions,"signInCopilot","Sign in to GitHub"),cancelLogin=action(authActions,"cancelCopilotSignIn","Cancel sign-in");cancelLogin.hidden=true; + const device=node("div");device.id="assistantDeviceLogin";device.hidden=true;copilot.append(device); + const local=node("div");local.id="assistantEndpointFields";section.append(local); + const select=field(local,"assistantEndpoint","Assistant endpoint"),description=node("p");description.id="assistantEndpointDescription";select.setAttribute("aria-describedby",description.id);local.append(description); + const foundry=node("div");foundry.id="assistantFoundryFields";section.append(foundry); + const reuse=field(foundry,"assistantFoundryPreset","Use an existing model connection");option(reuse,"","Enter endpoint and deployment"); + const endpoint=field(foundry,"assistantFoundryEndpoint","Azure model endpoint","input");endpoint.type="url";endpoint.placeholder="https://your-resource.openai.azure.com";endpoint.autocomplete="off"; + const deployment=field(foundry,"assistantFoundryDeployment","Model deployment name","input");deployment.type="text";deployment.autocomplete="off"; + foundry.append(node("p","Authentication uses Azure CLI on the computer running ROVE. Run az login there, then save and test this connection. Use an Azure model deployment, not a Foundry agent endpoint.")); const control=node("p");control.id="assistantSettingsControl"; const status=node("p","Loading assistant settings…");status.id="assistantConnectionStatus";status.setAttribute("role","status"); const actions=node("div");actions.className="assistant-settings-actions"; - const save=node("button","Save");save.id="saveAssistantSettings";save.type="button";save.disabled=true; - const test=node("button","Test connection");test.id="testAssistantConnection";test.type="button";test.disabled=true; - actions.append(save,test);section.append(label,select,description,control,status,actions);wrapper.append(section); + const save=action(actions,"saveAssistantSettings","Save"),test=action(actions,"testAssistantConnection","Test connection");section.append(control,status,actions);wrapper.append(section); const returnValue=new URLSearchParams(location.search).get("return"); - if(returnValue){ - try{const target=new URL(returnValue,location.origin); - if(target.origin===location.origin && target.pathname==="/static/datasets.html" && target.searchParams.get("campaign")){ - const back=node("a","Return to campaign results");back.id="assistantReturnToCampaign";back.href=target.pathname+target.search+target.hash;section.append(back); - } - }catch{/* Invalid return destinations are not navigation targets. */} - } - let settings=null,saved="",locked=false,epoch=0,busy=false,testing=false; + if(returnValue){try{const target=new URL(returnValue,location.origin);if(target.origin===location.origin && target.pathname==="/static/datasets.html" && target.searchParams.get("campaign")){const back=node("a","Return to campaign results");back.id="assistantReturnToCampaign";back.href=target.pathname+target.search+target.hash;section.append(back);}}catch{/* Ignore unsafe return destinations. */}} + let settings=null,saved=null,locked=false,epoch=0,busy=false,testing=false,authEpoch=0,authBusy=false,loginId=null,loginTimer=null,loginStarted=0; const current=version=>version===epoch && section.isConnected; - async function request(url,options){const response=await fetch(API_BASE+url,options);let value;try{value=await response.json();}catch{throw Error("The server returned an unreadable response.");}if(!response.ok)throw Error(typeof value.detail==="string"?value.detail:"The assistant settings request failed.");return value;} + const authCurrent=version=>version===authEpoch && section.isConnected && currentView==="settings" && provider.value==="copilot"; + async function request(url,options){const response=await fetch(API_BASE+url,options);let value;try{value=await response.json();}catch{throw Error("The server returned an unreadable response.");}if(!response.ok)throw Error(typeof value.detail==="string"?value.detail:"The assistant request failed.");return value;} + const post=(url,body={})=>request(url,{method:"POST",headers:{"Content-Type":"application/json"},body:JSON.stringify(body)}); + function selection(){switch(provider.value){case "copilot":return {provider:"copilot",model:model.value};case "endpoint":return {provider:"endpoint",endpoint_id:select.value};case "foundry":return {provider:"foundry",endpoint:endpoint.value.trim(),deployment:deployment.value.trim()};default:return {provider:"disabled"};}} + const same=(a,b)=>JSON.stringify(a,Object.keys(a || {}).sort())===JSON.stringify(b,Object.keys(b || {}).sort()); function update(){ - const endpoint=settings?.endpoints?.find(item=>item.id===select.value); - description.textContent=endpoint?`Model: ${endpoint.model || "Not specified"} · Provider: ${endpoint.provider_base_url || "Configured provider"}`:"Disabled. Campaign execution and recorded results remain available."; - save.disabled=busy || locked || !settings || select.value===saved; - test.disabled=busy || !settings || !saved || select.value!==saved; - select.disabled=locked || !settings || (busy && !testing); + copilot.hidden=provider.value!=="copilot";local.hidden=provider.value!=="endpoint";foundry.hidden=provider.value!=="foundry"; + const selected=settings?.endpoints?.find(item=>item.id===select.value); + description.textContent=selected?`Model: ${selected.model || "Not specified"} · Provider: ${selected.provider_base_url || "Configured provider"}`:"Choose an existing Copilot-compatible endpoint. Configure its provider credentials on the ROVE server."; + const draft=selection(),valid=draft.provider==="endpoint"?Boolean(draft.endpoint_id):draft.provider!=="foundry" || (endpoint.validity.valid && /^https:\/\//.test(draft.endpoint) && Boolean(draft.deployment)); + save.disabled=busy || locked || !settings || !valid || same(draft,saved); + test.disabled=busy || !settings || saved?.provider==="disabled" || !same(draft,saved); + for(const input of [provider,model,select,reuse,endpoint,deployment])input.disabled=locked || !settings || (busy && !testing); + authCheck.disabled=authBusy || !settings;signIn.disabled=authBusy || !settings;cancelLogin.hidden=!loginId && !loginStarted; } + function setModels(models){const selected=model.value;model.replaceChildren();option(model,"","Copilot default");for(const item of models || [])if(item.id)option(model,item.id,item.name || item.id);if(selected && !Array.from(model.options).some(item=>item.value===selected))option(model,selected,selected+" (saved model)");model.value=selected;} function apply(value,runtime){ - settings=value;saved=value.effective_endpoint_id || "";locked=Boolean(value.read_only); - select.replaceChildren();const disabled=node("option","Disabled");disabled.value="";select.append(disabled); - for(const endpoint of value.endpoints || []){const option=node("option",endpoint.display_name || endpoint.id);option.value=endpoint.id;select.append(option);} - if(saved && !Array.from(select.options).some(option=>option.value===saved)){const missing=node("option",`${saved} (unavailable)`);missing.value=saved;select.append(missing);} - select.value=saved; - control.textContent=locked?(value.override_reason || "This assistant is controlled by the server environment. Change it on the server to use a different endpoint."):"Save your choice, then test the connection. The test sends a fixed connectivity prompt, without your campaign data."; - status.textContent=!saved?"Assistant disabled. Choose an endpoint to enable AI explanations.":value.configured===false?(value.status_reason || "The selected endpoint needs server configuration before it can be used."):(runtime?.runtime_available ?? runtime?.available)?"Assistant configured. Runtime available; model connection has not been tested.":runtime?.reason || value.status_reason || "Assistant configured. Test the connection to check the model."; - if(!value.endpoints?.length && !saved)status.textContent="No Copilot assistant endpoints are configured. Add a copilot_agent endpoint to rove.yaml, then reload Settings."; + settings=value;saved=value.selection || (value.effective_endpoint_id?{provider:"endpoint",endpoint_id:value.effective_endpoint_id}:{provider:"disabled"});locked=Boolean(value.read_only); + provider.value=saved.provider;select.replaceChildren();option(select,"","Choose an endpoint");for(const item of value.endpoints || [])option(select,item.id,item.display_name || item.id); + if(saved.provider==="endpoint" && !Array.from(select.options).some(item=>item.value===saved.endpoint_id))option(select,saved.endpoint_id,saved.endpoint_id+" (unavailable)");select.value=saved.endpoint_id || ""; + model.value="";if(saved.model){if(!Array.from(model.options).some(item=>item.value===saved.model))option(model,saved.model,saved.model);model.value=saved.model;} + reuse.replaceChildren();option(reuse,"","Enter endpoint and deployment");for(const item of value.foundry_endpoints || [])option(reuse,item.id,item.display_name || item.id); + endpoint.value=saved.endpoint || "";deployment.value=saved.deployment || ""; + control.textContent=locked?(value.override_reason || "The server environment controls this assistant selection."):"Save your choice, then test the connection. The test sends a fixed connectivity prompt, without your campaign data."; + status.textContent=saved.provider==="disabled"?"Assistant disabled. Choose a provider to enable AI explanations.":value.configured===false?(value.status_reason || "Complete authentication and configuration, then test the assistant."):(runtime?.runtime_available ?? runtime?.available)?"Assistant configured. Runtime available; model connection has not been tested.":runtime?.reason || "Assistant configured. Test the model connection."; + if(!value.selection && !value.endpoints?.length && !value.effective_endpoint_id)status.textContent="No Copilot assistant endpoints are configured. Add a copilot_agent endpoint to rove.yaml, then reload Settings."; update(); } - async function load(){const version=++epoch;busy=true;update();status.textContent="Loading assistant settings…"; - try{const [value,runtime]=await Promise.all([request("/api/assistant/settings"),request("/api/assistant/status").catch(()=>null)]);if(current(version))apply(value,runtime);} - catch(error){if(current(version)){status.textContent="Could not load assistant settings. "+error.message;const retry=node("button","Reload settings");retry.type="button";retry.addEventListener("click",()=>{retry.remove();load();});actions.append(retry);}} - finally{if(current(version)){busy=false;update();}} + async function load(){const version=++epoch;busy=true;update();status.textContent="Loading assistant settings…";try{const [value,runtime]=await Promise.all([request("/api/assistant/settings"),request("/api/assistant/status").catch(()=>null)]);if(current(version))apply(value,runtime);}catch(error){if(current(version)){status.textContent="Could not load assistant settings. "+error.message;const retry=action(actions,"reloadAssistantSettings","Reload settings");retry.addEventListener("click",()=>{retry.remove();load();});}}finally{if(current(version)){busy=false;update();}}} + function stopLogin(cancel=true){authEpoch++;clearTimeout(loginTimer);loginTimer=null;const old=loginId;loginId=null;loginStarted=0;authBusy=false;device.hidden=true;device.replaceChildren();if(old && cancel)request("/api/assistant/copilot/login/"+encodeURIComponent(old),{method:"DELETE"}).catch(()=>{});update();} + function changed(){epoch++;busy=false;testing=false;status.textContent="Unsaved assistant selection. Save before testing or generating explanations.";update();} + provider.addEventListener("change",()=>{stopLogin();changed();}); + for(const input of [model,select,endpoint,deployment])input.addEventListener(input.tagName==="INPUT"?"input":"change",changed); + reuse.addEventListener("change",()=>{const item=settings?.foundry_endpoints?.find(item=>item.id===reuse.value);if(item){endpoint.value=item.endpoint;deployment.value=item.deployment;}changed();}); + save.addEventListener("click",async()=>{if(save.disabled)return;const version=++epoch,draft=selection();busy=true;testing=false;update();status.textContent="Saving assistant selection…";try{const payload=settings.selection?draft:{endpoint_id:draft.endpoint_id || null};const value=await request("/api/assistant/settings",{method:"PUT",headers:{"Content-Type":"application/json"},body:JSON.stringify(payload)});if(current(version)){apply(value,null);status.textContent=draft.provider==="disabled"?"Assistant disabled. Recorded results remain available.":"Assistant selection saved. Test the connection before returning to your campaign.";}}catch(error){if(current(version))status.textContent="Could not save assistant settings. "+error.message;}finally{if(current(version)){busy=false;update();}}}); + test.addEventListener("click",async()=>{if(test.disabled)return;const version=++epoch,fingerprint=settings.fingerprint,selected=saved;busy=true;testing=true;update();status.textContent="Testing the saved assistant connection…";try{const result=await post("/api/assistant/test");if(current(version)){const matches=fingerprint?result.fingerprint===fingerprint:result.endpoint_id===selected.endpoint_id;status.textContent=result.available && result.provider_tested && matches?"Connection successful. The assistant model responded to the test prompt.":"Connection failed. "+(result.reason || "The saved connection could not be verified. Save and test your current selection.");}}catch(error){if(current(version))status.textContent="Connection failed. "+error.message;}finally{if(current(version)){busy=false;update();}}}); + async function checkAuth(){const version=++authEpoch;authBusy=true;update();authStatus.textContent="Checking GitHub sign-in and available models…";try{const result=await post("/api/assistant/copilot/status",{include_models:true});if(authCurrent(version)){authStatus.textContent=result.authenticated?`Signed in${result.login?" as "+result.login:""}. Choose a model, then save and test your assistant.`:result.reason || "Not signed in to GitHub Copilot.";if(result.authenticated)setModels(result.models);}}catch(error){if(authCurrent(version))authStatus.textContent="Could not check sign-in. "+error.message;}finally{if(authCurrent(version)){authBusy=false;update();}}} + authCheck.addEventListener("click",checkAuth); + async function showLogin(result,version){ + if(!authCurrent(version)){if(result.operation_id)request("/api/assistant/copilot/login/"+encodeURIComponent(result.operation_id),{method:"DELETE"}).catch(()=>{});return;} + loginId=result.operation_id || loginId; + if(result.status==="complete"){loginId=null;loginStarted=0;device.hidden=true;authBusy=false;authStatus.textContent="GitHub sign-in completed. Check sign-in to load available models.";update();return;} + if(["failed","expired","cancelled"].includes(result.status)){loginId=null;loginStarted=0;device.hidden=true;authBusy=false;authStatus.textContent=result.reason || "Sign-in did not complete. Try again.";update();return;} + authStatus.textContent=result.status==="starting"?"Starting GitHub sign-in…":"Waiting for you to complete GitHub sign-in.";device.replaceChildren(); + if(result.verification_uri==="https://github.com/login/device" && result.user_code){const code=node("code",result.user_code);code.id="assistantDeviceCode";const link=node("a","Open GitHub and enter this code ↗");link.href=result.verification_uri;link.target="_blank";link.rel="noopener";device.append(node("p","Use this one-time code on GitHub:"),code,link);device.hidden=false;}else device.hidden=true; + update(); + if(!loginId || Date.now()-loginStarted>300000){stopLogin();authStatus.textContent="Sign-in timed out. Start a new sign-in attempt.";return;} + loginTimer=setTimeout(async()=>{if(!authCurrent(version)){stopLogin();return;}try{await showLogin(await request("/api/assistant/copilot/login/"+encodeURIComponent(loginId)),version);}catch(error){if(authCurrent(version)){stopLogin();authStatus.textContent="Could not check sign-in. "+error.message;}}},1500); } - select.addEventListener("change",()=>{epoch++;busy=false;testing=false;status.textContent="Unsaved assistant selection. Save before testing or generating explanations.";update();}); - save.addEventListener("click",async()=>{if(save.disabled)return;const version=++epoch,selected=select.value;busy=true;update();status.textContent="Saving assistant selection…"; - try{const value=await request("/api/assistant/settings",{method:"PUT",headers:{"Content-Type":"application/json"},body:JSON.stringify({endpoint_id:selected || null})});if(current(version)){apply(value,null);status.textContent=selected?"Assistant selection saved. Test the connection before returning to your campaign.":"Assistant disabled. Recorded results and evaluation execution remain available.";}} - catch(error){if(current(version))status.textContent="Could not save assistant settings. "+error.message;} - finally{if(current(version)){busy=false;update();}} - }); - test.addEventListener("click",async()=>{if(test.disabled)return;const version=++epoch;busy=true;testing=true;update();status.textContent="Testing the saved assistant connection…"; - try{const result=await request("/api/assistant/test",{method:"POST",headers:{"Content-Type":"application/json"},body:"{}"});if(current(version))status.textContent=result.available && result.provider_tested && result.endpoint_id===saved?"Connection successful. The assistant model responded to the test prompt.":"Connection failed. "+(result.reason || "The model did not respond successfully.");} - catch(error){if(current(version))status.textContent="Connection failed. "+error.message;} - finally{if(current(version)){busy=false;update();}} - }); - load(); - if(location.hash==="#assistant")requestAnimationFrame(()=>{if(section.isConnected){section.scrollIntoView({block:"start"});heading.focus({preventScroll:true});}}); + signIn.addEventListener("click",async()=>{if(signIn.disabled)return;stopLogin();const version=authEpoch;authBusy=true;loginStarted=Date.now();authStatus.textContent="Starting GitHub sign-in…";update();try{await showLogin(await post("/api/assistant/copilot/login"),version);}catch(error){if(authCurrent(version)){authBusy=false;authStatus.textContent="Could not start sign-in. "+error.message;update();}}}); + cancelLogin.addEventListener("click",()=>{stopLogin();authStatus.textContent="Sign-in cancelled.";}); + const observer=new MutationObserver(()=>{if(!section.isConnected || currentView!=="settings"){stopLogin();observer.disconnect();}});observer.observe(settingsView,{attributes:true,childList:true}); + load();if(location.hash==="#assistant")requestAnimationFrame(()=>{if(section.isConnected){section.scrollIntoView({block:"start"});heading.focus({preventScroll:true});}}); } function renderSettingsView() { diff --git a/frontend/styles.css b/frontend/styles.css index 8461e9a..d0c1560 100644 --- a/frontend/styles.css +++ b/frontend/styles.css @@ -357,3 +357,9 @@ body[data-quick-step] .root-main { overflow-y:auto; } #assistant .assistant-settings-actions {display:flex;gap:12px;flex-wrap:wrap;margin:20px 0;} #assistant button,#assistantReturnToCampaign {display:inline-flex;align-items:center;min-height:40px;padding:8px 14px;font-size:14px;color:var(--f-text);background:var(--f-surface);border:1px solid var(--f-border-strong);border-radius:var(--f-radius-control,4px);} #assistant #saveAssistantSettings {background:var(--f-accent);border-color:var(--f-accent);color:white;}#assistant button:disabled {opacity:.5;cursor:not-allowed;} +#assistant [hidden] {display:none;} +#assistant input {width:100%;min-height:40px;padding:8px 12px;background:var(--f-bg);color:var(--f-text);border:1px solid var(--f-border-strong);border-radius:var(--f-radius-control,4px);font-size:16px;} +#assistantCopilotFields,#assistantEndpointFields,#assistantFoundryFields {margin-top:20px;} +#assistantDeviceLogin {padding:16px;border:1px solid var(--f-border);border-radius:var(--f-radius-panel,6px);background:var(--f-bg);} +#assistantDeviceCode {display:block;margin:12px 0;font-size:24px;letter-spacing:3px;color:var(--f-text);} +#assistantDeviceLogin a {font-size:16px;color:var(--f-accent-text);text-decoration:underline;} diff --git a/src/rove/api/assistant.py b/src/rove/api/assistant.py index 68bbfcf..149effe 100644 --- a/src/rove/api/assistant.py +++ b/src/rove/api/assistant.py @@ -1,7 +1,6 @@ """Bounded workflow assistance; only the host can confirm a prepared mutation. -Enable with ROVE_ASSISTANT_ENDPOINT naming an explicitly configured copilot_agent -endpoint. Proposals expire after ten minutes and after server restart: prepare again +Configure an assistant provider in Settings or through the CLI. Proposals expire after ten minutes and after server restart: prepare again to confirm. No confirmation token, write tool or ambient resource enters the model session. This local API inherits the application's loopback/security boundary. """ @@ -12,6 +11,7 @@ import hmac import json import time +from contextlib import asynccontextmanager from dataclasses import dataclass, replace from pathlib import Path from typing import Annotated, Literal @@ -40,9 +40,11 @@ from rove.runtime.assistant import ( assistant_settings, assistant_settings_view, + github_auth_directory, save_assistant_selection, ) -from rove.runtime.copilot import CopilotRuntime, RuntimeTool +from rove.runtime.assistant_login import CopilotLogin +from rove.runtime.copilot import CopilotRuntime, CopilotRuntimeConfig, RuntimeTool from rove.trials.snapshots import canonical_json, content_hash, sanitize from rove.trials.store import TrialStore, now @@ -71,7 +73,15 @@ class AskRequest(Arguments): class AssistantSelection(Arguments): - endpoint_id: Identifier | None + provider: Literal["copilot", "endpoint", "foundry", "disabled"] | None = None + endpoint_id: Identifier | None = None + model: str = Field(default="", max_length=128) + endpoint: str | None = Field(default=None, max_length=2048) + deployment: str | None = Field(default=None, max_length=128) + + +class IdentityRequest(Arguments): + include_models: bool = False class ConnectionProof(Arguments): @@ -137,7 +147,14 @@ def create_assistant_router( clock=time.monotonic, ) -> APIRouter: """Mount alongside the customer workflow router using its launch callback.""" - router = APIRouter() + login = CopilotLogin(github_auth_directory(root)) + + @asynccontextmanager + async def lifespan(app): + yield + await login.close() + + router = APIRouter(lifespan=lifespan) proposals: dict[str, Proposal] = {} def settings(): @@ -156,14 +173,59 @@ async def get_settings(): async def put_settings(request: AssistantSelection): try: save_assistant_selection( - request.endpoint_id, root=root, config_loader=config_loader, endpoint_id=endpoint_id + request.model_dump(exclude_unset=True), + root=root, + config_loader=config_loader, + endpoint_id=endpoint_id, ) except PermissionError as error: raise HTTPException(409, str(error)) from None - except ValueError: - raise HTTPException(422, "Choose an existing copilot_agent endpoint") from None + except ValueError as error: + raise HTTPException(422, str(error)) from None return settings_view() + @router.post("/api/assistant/copilot/status") + async def copilot_identity(request: IdentityRequest): + try: + config = CopilotRuntimeConfig( + model="", + provider={}, + role="assistant", + auth_mode="github", + github_auth_directory=github_auth_directory(root), + ) + return await runtime_factory(config).github_identity( + include_models=request.include_models + ) + except Exception: + return { + "authenticated": False, + "login": None, + "models": [], + "reason": "Could not check Copilot sign-in. Check the runtime on the ROVE host.", + } + + @router.post("/api/assistant/copilot/login") + async def start_login(): + try: + return await login.start() + except ValueError as error: + raise HTTPException(503, str(error)) from None + + @router.get("/api/assistant/copilot/login/{operation_id}") + async def login_status(operation_id: Identifier): + try: + return login.get(operation_id) + except KeyError: + raise HTTPException(404, "Sign-in request not found. Start again.") from None + + @router.delete("/api/assistant/copilot/login/{operation_id}") + async def cancel_login(operation_id: Identifier): + try: + return await login.cancel(operation_id) + except KeyError: + raise HTTPException(404, "Sign-in request not found.") from None + @router.post("/api/assistant/test") async def test_connection(): view = settings_view() @@ -171,6 +233,7 @@ async def test_connection(): "available": False, "provider_tested": False, "endpoint_id": view["effective_endpoint_id"], + "fingerprint": view["fingerprint"], "reason": view["status_reason"], } if not view["configured"]: @@ -178,7 +241,7 @@ async def test_connection(): try: # Pin this request to the displayed selection if another client saves concurrently. configured_settings = assistant_settings( - root=root, config_loader=config_loader, endpoint_id=view["effective_endpoint_id"] + root=root, config_loader=config_loader, selection=view["selection"] ) runtime_settings = replace( configured_settings, @@ -211,7 +274,7 @@ async def status(): except Exception: available, reason = ( False, - "Configure an explicit Copilot assistant endpoint and runtime", + "Choose an assistant provider and check its runtime", ) return { "available": available, diff --git a/src/rove/benchmarks/insights.py b/src/rove/benchmarks/insights.py index af0791f..8fb05fb 100644 --- a/src/rove/benchmarks/insights.py +++ b/src/rove/benchmarks/insights.py @@ -127,7 +127,7 @@ def provenance(source, settings): return { "method": "copilot_runtime" if source == "ai" else "deterministic_template", "prompt_version": PROMPT_VERSION, - "model": settings.model if source == "ai" else None, + "model": (settings.model or "copilot-default") if source == "ai" else None, } diff --git a/src/rove/runtime/assistant.py b/src/rove/runtime/assistant.py index feba2cd..9593a26 100644 --- a/src/rove/runtime/assistant.py +++ b/src/rove/runtime/assistant.py @@ -1,13 +1,15 @@ -"""Shared explicitly configured assistant runtime; no candidate endpoint fallback.""" +"""Explicit assistant providers with persistent, credential-free local selection.""" +import json import os +import re import sqlite3 from pathlib import Path from urllib.parse import urlsplit, urlunsplit from rove.models.config import active_config_path, load_config from rove.runtime.copilot import CopilotRuntimeConfig -from rove.trials.snapshots import sanitize +from rove.trials.snapshots import content_hash, sanitize def _location(root=None): @@ -16,38 +18,129 @@ def _location(root=None): return directory / "assistant-settings.sqlite3", str(config_path) +def github_auth_directory(root=None): + return str((_location(root)[0].parent / "assistant-auth" / "copilot").resolve()) + + +def foundry_endpoint(value): + """Only public Azure model resources can receive an Azure CLI bearer token.""" + parsed = urlsplit(value) + host = parsed.hostname or "" + if ( + parsed.scheme != "https" + or parsed.username + or parsed.password + or parsed.query + or parsed.fragment + or parsed.port not in (None, 443) + or not re.fullmatch(r"[a-zA-Z0-9-]+\.(?:openai\.azure\.com|services\.ai\.azure\.com)", host) + or parsed.path.rstrip("/") not in ("", "/openai/v1") + ): + raise ValueError("Use an Azure model resource URL, not a project or agent endpoint") + return f"https://{host}/openai/v1/" + + +def normalize_selection(value, config_loader=load_config): + if not isinstance(value, dict): + raise ValueError("Choose an assistant provider") + if "provider" not in value and set(value) == {"endpoint_id"}: + value = ( + {"provider": "endpoint", "endpoint_id": value["endpoint_id"]} + if value["endpoint_id"] + else {"provider": "disabled"} + ) + provider = value.get("provider") + allowed = { + "copilot": {"provider", "model"}, + "endpoint": {"provider", "endpoint_id"}, + "foundry": {"provider", "endpoint", "deployment"}, + "disabled": {"provider"}, + } + if provider not in allowed or set(value) - allowed[provider]: + raise ValueError("Choose valid settings for this assistant provider") + if provider == "copilot": + model = value.get("model", "") + if ( + not isinstance(model, str) + or len(model) > 128 + or (model and not re.fullmatch(r"[\w./:-]+", model)) + ): + raise ValueError("Choose a valid Copilot model") + return {"provider": provider, "model": model} + if provider == "endpoint": + selected = value.get("endpoint_id") + endpoint = config_loader().endpoints.get(selected) if isinstance(selected, str) else None + if endpoint is None or endpoint.adapter != "copilot_agent": + raise ValueError("Choose an existing copilot_agent endpoint") + return {"provider": provider, "endpoint_id": selected} + if provider == "foundry": + deployment = value.get("deployment", "") + if not isinstance(deployment, str) or not re.fullmatch(r"[\w.-]{1,128}", deployment): + raise ValueError("Enter the Foundry model deployment name") + endpoint = value.get("endpoint") + if not isinstance(endpoint, str) or len(endpoint) > 2048: + raise ValueError("Enter an Azure model resource URL") + return { + "provider": provider, + "endpoint": foundry_endpoint(endpoint), + "deployment": deployment, + } + return {"provider": "disabled"} + + def assistant_selection(*, root=None, endpoint_id=None): - """Resolve selection without creating storage or contacting a provider.""" + """No saved row defaults to Copilot; an explicitly disabled row stays disabled.""" path, config_key = _location(root) - selected = None + row = None if path.exists(): with sqlite3.connect(f"{path.resolve().as_uri()}?mode=ro", uri=True) as connection: - exists = connection.execute( - "SELECT name FROM sqlite_master WHERE type='table' AND name='assistant_settings'" - ).fetchone() - row = ( - connection.execute( - "SELECT endpoint_id FROM assistant_settings WHERE config_path = ?", - (config_key,), - ).fetchone() - if exists - else None - ) - selected = row[0] if row else None + columns = { + item[1] for item in connection.execute("PRAGMA table_info(assistant_settings)") + } + if columns: + query = ( + "SELECT endpoint_id, settings_json FROM assistant_settings WHERE config_path = ?" + if "settings_json" in columns + else "SELECT endpoint_id, NULL FROM assistant_settings WHERE config_path = ?" + ) + row = connection.execute(query, (config_key,)).fetchone() + saved = ( + json.loads(row[1]) + if row and row[1] + else {"provider": "endpoint", "endpoint_id": row[0]} + if row and row[0] + else {"provider": "disabled"} + if row + else {"provider": "copilot", "model": ""} + ) override = endpoint_id or os.environ.get("ROVE_ASSISTANT_ENDPOINT") or None + selection = {"provider": "endpoint", "endpoint_id": override} if override else saved source = ( "explicit" if endpoint_id else "environment" if override - else "saved" - if selected + else "default" + if row is None else "disabled" + if saved["provider"] == "disabled" + else "saved" + ) + effective = ( + selection.get("endpoint_id") + if selection["provider"] == "endpoint" + else "github-copilot" + if selection["provider"] == "copilot" + else "microsoft-foundry" + if selection["provider"] == "foundry" + else None ) return { - "selected_endpoint_id": selected, - "effective_endpoint_id": override or selected, + "selected_endpoint_id": saved.get("endpoint_id"), + "effective_endpoint_id": effective, + "selection": selection, "selection_source": source, + "fingerprint": content_hash(selection), "read_only": bool(override), "override_reason": ( "An explicit host endpoint overrides the saved selection." @@ -62,10 +155,9 @@ def assistant_selection(*, root=None, endpoint_id=None): def save_assistant_selection(selected, *, root=None, config_loader=load_config, endpoint_id=None): if assistant_selection(root=root, endpoint_id=endpoint_id)["read_only"]: raise PermissionError("Remove the assistant endpoint override before changing this setting") - if selected is not None: - endpoint = config_loader().endpoints.get(selected) - if endpoint is None or endpoint.adapter != "copilot_agent": - raise ValueError("Choose an existing copilot_agent endpoint") + selection = normalize_selection( + selected if isinstance(selected, dict) else {"endpoint_id": selected}, config_loader + ) path, config_key = _location(root) path.parent.mkdir(parents=True, exist_ok=True, mode=0o700) try: @@ -75,26 +167,30 @@ def save_assistant_selection(selected, *, root=None, config_loader=load_config, else: os.close(descriptor) with sqlite3.connect(path) as connection: + connection.execute("BEGIN IMMEDIATE") connection.execute( - "CREATE TABLE IF NOT EXISTS assistant_settings " - "(config_path TEXT PRIMARY KEY, endpoint_id TEXT)" + "CREATE TABLE IF NOT EXISTS assistant_settings (config_path TEXT PRIMARY KEY, endpoint_id TEXT)" ) + if "settings_json" not in { + item[1] for item in connection.execute("PRAGMA table_info(assistant_settings)") + }: + connection.execute("ALTER TABLE assistant_settings ADD COLUMN settings_json TEXT") connection.execute( - "INSERT INTO assistant_settings VALUES (?, ?) " - "ON CONFLICT(config_path) DO UPDATE SET endpoint_id=excluded.endpoint_id", - (config_key, selected), + "INSERT INTO assistant_settings (config_path, endpoint_id, settings_json) VALUES (?, ?, ?) " + "ON CONFLICT(config_path) DO UPDATE SET endpoint_id=excluded.endpoint_id, settings_json=excluded.settings_json", + (config_key, selection.get("endpoint_id"), json.dumps(selection)), ) def _provider_origin(value): - """Display only the origin: paths, credentials and query parameters may contain secrets.""" try: parsed = urlsplit(value) if parsed.scheme not in {"http", "https"} or not parsed.hostname: return None host = f"[{parsed.hostname}]" if ":" in parsed.hostname else parsed.hostname - netloc = f"{host}:{parsed.port}" if parsed.port else host - return urlunsplit((parsed.scheme, netloc, "", "", "")) + return urlunsplit( + (parsed.scheme, f"{host}:{parsed.port}" if parsed.port else host, "", "", "") + ) except (TypeError, ValueError): return None @@ -116,40 +212,88 @@ def assistant_settings_view(*, root=None, config_loader=load_config, endpoint_id for key, value in config.endpoints.items() if value.adapter == "copilot_agent" ] + foundry_endpoints = [] + for key, value in config.endpoints.items(): + if value.adapter != "azure_openai": + continue + values = value.config + try: + selection = normalize_selection( + { + "provider": "foundry", + "endpoint": value.endpoint or values.get("endpoint"), + "deployment": values.get("deployment_name"), + } + ) + except (ValueError, TypeError): + continue + if "your-resource" in selection["endpoint"]: + continue + foundry_endpoints.append( + { + "id": key, + "display_name": value.display_name or key, + "endpoint": selection["endpoint"], + "deployment": selection["deployment"], + } + ) try: assistant_settings(root=root, config_loader=lambda: config, endpoint_id=endpoint_id) configured, reason = True, None except (ValueError, TypeError, KeyError, AttributeError): configured = False reason = ( - "Choose a Copilot assistant endpoint to enable AI assistance." - if not state["effective_endpoint_id"] + "Assistant disabled. Choose a provider to enable AI assistance." + if state["selection"]["provider"] == "disabled" else "The selected assistant endpoint or its provider configuration is unavailable." ) return sanitize( - {**state, "endpoints": endpoints, "configured": configured, "status_reason": reason} + { + **state, + "endpoints": endpoints, + "foundry_endpoints": foundry_endpoints, + "configured": configured, + "status_reason": reason, + } ) -def assistant_settings(*, root=None, config_loader=load_config, endpoint_id=None): - selected = assistant_selection(root=root, endpoint_id=endpoint_id)["effective_endpoint_id"] - if not selected: - raise ValueError("Configure ROVE_ASSISTANT_ENDPOINT to enable workflow assistance") - config = config_loader() - endpoint = config.endpoints.get(selected) - if endpoint is None or endpoint.adapter != "copilot_agent": - raise ValueError("Assistant endpoint must use the copilot_agent adapter") - values = endpoint.config - provider = dict(values.get("provider", {})) - if key_env := provider.pop("api_key_env", None): - key = os.environ.get(key_env) - if not key: - raise ValueError("Configured assistant provider credential is unavailable") - provider["api_key"] = key +def assistant_settings(*, root=None, config_loader=load_config, endpoint_id=None, selection=None): + selection = ( + selection + if selection is not None + else assistant_selection(root=root, endpoint_id=endpoint_id)["selection"] + ) + mode = selection["provider"] + extra = {} + values = {} + if mode == "disabled": + raise ValueError("Assistant is disabled") + if mode == "copilot": + model, provider = selection.get("model", ""), {} + extra = {"auth_mode": "github", "github_auth_directory": github_auth_directory(root)} + elif mode == "foundry": + selection = normalize_selection(selection) + model = selection["deployment"] + provider = {"type": "openai", "base_url": selection["endpoint"], "wire_api": "completions"} + extra = {"auth_mode": "azure_cli"} + else: + endpoint = config_loader().endpoints.get(selection["endpoint_id"]) + if endpoint is None or endpoint.adapter != "copilot_agent": + raise ValueError("Assistant endpoint must use the copilot_agent adapter") + values = endpoint.config + model = values.get("model", selection["endpoint_id"]) + provider = dict(values.get("provider", {})) + if key_env := provider.pop("api_key_env", None): + key = os.environ.get(key_env) + if not key: + raise ValueError("Configured assistant provider credential is unavailable") + provider["api_key"] = key return CopilotRuntimeConfig( - model=values.get("model", selected), + model=model, provider=provider, role="assistant", + **extra, timeout_seconds=min(float(values.get("timeout_seconds", 60)), 120), cli_path=values.get("cli_path", "copilot"), capture_content=False, diff --git a/src/rove/runtime/assistant_login.py b/src/rove/runtime/assistant_login.py new file mode 100644 index 0000000..ff55ed4 --- /dev/null +++ b/src/rove/runtime/assistant_login.py @@ -0,0 +1,123 @@ +"""User-initiated Copilot device login with a bounded, private CLI process.""" + +import asyncio +import re +import shutil +import time +from contextlib import suppress +from dataclasses import dataclass, field +from pathlib import Path +from uuid import uuid4 + +from rove.runtime.copilot import github_login_environment + + +@dataclass +class LoginOperation: + operation_id: str = field(default_factory=lambda: uuid4().hex) + status: str = "starting" + user_code: str | None = None + reason: str | None = None + started: float = field(default_factory=time.monotonic) + task: asyncio.Task | None = field(default=None, repr=False) + + def public(self): + return { + "operation_id": self.operation_id, + "status": self.status, + "verification_uri": "https://github.com/login/device" if self.user_code else None, + "user_code": self.user_code if self.status == "pending" else None, + "reason": self.reason, + } + + +class CopilotLogin: + def __init__(self, directory, *, spawn=asyncio.create_subprocess_exec, timeout=600): + self.directory = directory + self.spawn = spawn + self.timeout = timeout + self.operation = None + self.lock = asyncio.Lock() + + async def start(self): + async with self.lock: + if self.operation and self.operation.task and not self.operation.task.done(): + return self.operation.public() + cli = shutil.which("copilot") + if not cli: + raise ValueError("Install the Copilot CLI on the ROVE host before signing in") + Path(self.directory).mkdir(parents=True, exist_ok=True, mode=0o700) + self.operation = LoginOperation() + self.operation.task = asyncio.create_task(self._run(self.operation, cli)) + return self.operation.public() + + def get(self, operation_id): + if not self.operation or self.operation.operation_id != operation_id: + raise KeyError(operation_id) + return self.operation.public() + + async def cancel(self, operation_id): + self.get(operation_id) + task = self.operation.task + if task and not task.done(): + task.cancel() + with suppress(asyncio.CancelledError): + await task + self.operation.status, self.operation.reason = "cancelled", "Sign-in cancelled." + return self.operation.public() + + async def close(self): + if self.operation and self.operation.task and not self.operation.task.done(): + await self.cancel(self.operation.operation_id) + + async def _run(self, operation, cli): + process = None + try: + process = await self.spawn( + cli, + "--no-auto-update", + "login", + "--device-code", + env=github_login_environment(self.directory), + cwd=self.directory, + stdin=asyncio.subprocess.DEVNULL, + stdout=asyncio.subprocess.PIPE, + stderr=asyncio.subprocess.STDOUT, + ) + async with asyncio.timeout(self.timeout): + text = "" + while chunk := await process.stdout.read(1024): + text = (text + chunk.decode("utf-8", errors="replace"))[-8192:] + # Only the fixed GitHub device code is exposed; never echo CLI output. + clean = re.sub(r"\x1b\[[0-?]*[ -/]*[@-~]", "", text) + code = re.search(r"\b([A-Z0-9]{4}-[A-Z0-9]{4})\b", clean) + if code and "github.com/login/device" in clean: + operation.user_code, operation.status = code.group(1), "pending" + result = await process.wait() + operation.status = "complete" if result == 0 else "failed" + operation.reason = ( + None if result == 0 else "GitHub sign-in did not complete. Try again." + ) + except TimeoutError: + operation.status, operation.reason = ( + "expired", + "The sign-in request expired. Start again.", + ) + except asyncio.CancelledError: + operation.status, operation.reason = "cancelled", "Sign-in cancelled." + raise + except Exception: + operation.status, operation.reason = ( + "failed", + "Could not start Copilot sign-in on the ROVE host.", + ) + finally: + if process and process.returncode is None: + try: + process.terminate() + await asyncio.wait_for(process.wait(), timeout=3) + except (TimeoutError, ProcessLookupError): + if process.returncode is None: + with suppress(ProcessLookupError): + process.kill() + await process.wait() diff --git a/src/rove/runtime/copilot.py b/src/rove/runtime/copilot.py index 39670ee..2e4bf48 100644 --- a/src/rove/runtime/copilot.py +++ b/src/rove/runtime/copilot.py @@ -24,11 +24,13 @@ from importlib.metadata import PackageNotFoundError, version from pathlib import Path from typing import Any, Literal +from urllib.parse import urlsplit from uuid import UUID, uuid4 SDK_VERSION = "1.0.13" CLI_VERSION = "1.0.81-9" Role = Literal["candidate", "grader", "assistant"] +AuthMode = Literal["byok", "github", "azure_cli"] EventSink = Callable[[dict[str, Any]], None] @@ -54,7 +56,7 @@ class RuntimeTool: @dataclass(frozen=True) class CopilotRuntimeConfig: model: str - provider: dict[str, Any] = field(repr=False) + provider: dict[str, Any] = field(default_factory=dict, repr=False) role: Role = "candidate" timeout_seconds: float = 60.0 cleanup_timeout_seconds: float = 5.0 @@ -66,12 +68,51 @@ class CopilotRuntimeConfig: system_message: str = "Return a JSON object describing the requested robotics pipeline stage." telemetry: dict[str, Any] | None = None native_traces: bool = True + auth_mode: AuthMode = "byok" + github_auth_directory: str | None = field(default=None, repr=False) def __post_init__(self) -> None: if self.role not in {"candidate", "grader", "assistant"}: raise ValueError("Invalid runtime actor role") - if not self.model or not self.provider.get("base_url"): + if self.auth_mode not in {"byok", "github", "azure_cli"}: + raise ValueError("Invalid runtime authentication mode") + if self.auth_mode == "github": + if self.provider: + raise ValueError("GitHub authentication cannot include a BYOK provider") + if not self.github_auth_directory or not Path(self.github_auth_directory).is_absolute(): + raise ValueError("GitHub authentication requires an explicit absolute ROVE profile") + elif not self.model or not self.provider.get("base_url"): raise ValueError("An explicit model and provider base_url are required") + if self.auth_mode == "azure_cli": + parsed = urlsplit(self.provider["base_url"]) + if ( + parsed.scheme != "https" + or not parsed.hostname + or not parsed.hostname.endswith((".openai.azure.com", ".services.ai.azure.com")) + or parsed.port not in {None, 443} + or parsed.path.rstrip("/") != "/openai/v1" + or parsed.username + or parsed.password + or parsed.query + or parsed.fragment + or self.provider.get("type", "openai") != "openai" + ): + raise ValueError( + "Azure CLI authentication requires a public Foundry HTTPS /openai/v1 endpoint" + ) + if any( + key in self.provider + for key in ( + "api_key", + "api_key_env", + "bearer_token", + "bearer_token_provider", + "headers", + ) + ): + raise ValueError( + "Azure CLI authentication cannot include alternative provider credentials" + ) for value in (self.timeout_seconds, self.cleanup_timeout_seconds): if not math.isfinite(value) or value <= 0: raise ValueError("Runtime deadlines must be positive finite seconds") @@ -81,6 +122,69 @@ def __post_init__(self) -> None: raise ValueError("native_traces must be a boolean") +def github_login_environment(directory: str | Path) -> dict[str, str]: + """Use one explicit app profile, without ambient token or GitHub CLI configuration.""" + root = Path(directory) + if not root.is_absolute(): + raise ValueError("GitHub authentication requires an absolute profile directory") + root.mkdir(parents=True, exist_ok=True, mode=0o700) + paths = {name: root / name for name in ("isolated-home", "isolated-gh", "isolated-config")} + for path in paths.values(): + path.mkdir(exist_ok=True, mode=0o700) + environment = { + key: value + for key, value in os.environ.items() + if key + in { + "PATH", + "SYSTEMROOT", + "WINDIR", + "LANG", + "LC_ALL", + "TMPDIR", + "TEMP", + "TMP", + } + } + environment.update( + { + "COPILOT_HOME": str(root), + "HOME": str(paths["isolated-home"]), + "USERPROFILE": str(paths["isolated-home"]), + "GH_CONFIG_DIR": str(paths["isolated-gh"]), + "XDG_CONFIG_HOME": str(paths["isolated-config"]), + "GH_PROMPT_DISABLED": "1", + } + ) + return environment + + +async def _azure_cli_token() -> str: + """Acquire per execution off the event loop; drain bounded CLI work on cancellation.""" + + def acquire(): + from azure.identity import AzureCliCredential + + with AzureCliCredential(process_timeout=10) as credential: + token = credential.get_token("https://ai.azure.com/.default") + if not token.token or token.expires_on <= time.time() + 30: + raise ValueError("Azure CLI returned no usable access token") + return token.token + + operation = asyncio.create_task(asyncio.to_thread(acquire)) + try: + return await asyncio.shield(operation) + except asyncio.CancelledError: + # AzureCliCredential's synchronous subprocess timeout kills and reaps the CLI. + # Never abandon a background credential acquisition on stage cancellation. + await asyncio.gather(operation, return_exceptions=True) + raise + except Exception: + raise RuntimeError( + "Azure CLI authentication failed; sign in with az login and verify resource access" + ) from None + + def _plain(value: Any) -> Any: if dataclasses.is_dataclass(value) and not isinstance(value, type): return { @@ -325,8 +429,8 @@ def __init__( self.tools = tools self.event_sink = event_sink self._factory = client_factory - self.model_id = config.model - self.display_name = f"Copilot: {config.model}" + self.model_id = config.model or "copilot-default" + self.display_name = f"Copilot: {config.model or 'default model'}" if len({tool.name for tool in tools}) != len(tools): raise ValueError("Runtime tool names must be unique") if any(config.role not in tool.roles for tool in tools): @@ -355,6 +459,15 @@ async def _client(self, workspace: str) -> Any: } }, } + if self.config.auth_mode == "github": + options.update( + { + "base_directory": self.config.github_auth_directory, + "use_logged_in_user": True, + "mode": "copilot-cli", + "env": github_login_environment(self.config.github_auth_directory), + } + ) if self.config.telemetry is not None: options["telemetry"] = { **self.config.telemetry, @@ -420,6 +533,46 @@ async def health_check(self) -> bool: except PackageNotFoundError: return False + async def github_identity(self, *, include_models: bool = False) -> dict[str, Any]: + """Inspect the explicitly selected ROVE sign-in profile; never run inference.""" + if self.config.auth_mode != "github": + raise ValueError("GitHub identity is available only in explicit GitHub mode") + result = {"authenticated": False, "login": None, "models": [], "reason": None} + client = None + with tempfile.TemporaryDirectory(prefix="rove-auth-status-") as workspace: + try: + async with asyncio.timeout(min(self.config.timeout_seconds, 20)): + client = await self._client(workspace) + await client.start() + status = await client.get_auth_status() + result["authenticated"] = ( + status.isAuthenticated is True and status.authType == "user" + ) + if result["authenticated"]: + login = status.login + if isinstance(login, str) and len(login) <= 128: + result["login"] = login + if include_models: + result["models"] = [ + {"id": str(model.id)[:128], "name": str(model.name)[:256]} + for model in (await client.list_models())[:200] + ] + else: + result["reason"] = "Sign in to GitHub Copilot for this ROVE profile." + except Exception: + result["reason"] = "GitHub Copilot identity or model discovery is unavailable." + finally: + if client is not None: + try: + await asyncio.wait_for(client.stop(), self.config.cleanup_timeout_seconds) + except Exception: + result["reason"] = "GitHub Copilot runtime cleanup failed." + if hasattr(client, "force_stop"): + await asyncio.wait_for( + client.force_stop(), self.config.cleanup_timeout_seconds + ) + return result + def _sdk_tools(self, recorder: _Recorder) -> list[Any]: from rove.orchestrator.recording import capture_recording_context, event_sink @@ -613,9 +766,23 @@ async def run_stage( async with asyncio.timeout(self.config.timeout_seconds): client = await self._client(workspace) await client.start() - session = await client.create_session( - model=self.config.model, - provider=dict(self.config.provider), + if self.config.auth_mode == "github": + identity = await client.get_auth_status() + if identity.isAuthenticated is not True or identity.authType != "user": + raise RuntimeError( + "Sign in to the dedicated ROVE GitHub Copilot profile before inference" + ) + routing = {} + if self.config.model: + routing["model"] = self.config.model + if self.config.auth_mode != "github": + provider = dict(self.config.provider) + if self.config.auth_mode == "azure_cli": + provider["type"] = "openai" + provider["bearer_token"] = await _azure_cli_token() + routing["provider"] = provider + session = await client.create_session( # nosec B106 - in-memory is a storage mode + **routing, session_id=session_id, available_tools=[tool.name for tool in self.tools], tools=self._sdk_tools(recorder), @@ -628,7 +795,31 @@ async def run_stage( enable_host_git_operations=False, enable_skills=False, enable_session_store=False, + enable_session_telemetry=False, + enable_on_demand_instruction_discovery=False, + skip_embedding_retrieval=True, + embedding_cache_storage="in-memory", + mcp_oauth_token_storage="in-memory", + memory={"enabled": False}, + enable_experimental_mode=False, + custom_agents_local_only=True, + coauthor_enabled=False, + manage_schedule_enabled=False, + included_builtin_skills=[], + custom_agents=[], + mcp_servers={}, + plugin_directories=[], + instruction_directories=[], + skill_directories=[], ) + if self.config.auth_mode == "github": + # copilot-cli mode enables keychain authentication; explicitly restore + # empty mode's installed-plugin exclusion before any model request. + from copilot.generated.rpc import SessionUpdateOptionsParams + + await session.rpc.options.update( + SessionUpdateOptionsParams(installed_plugins=[]) + ) recorder.check() ready_at = time.monotonic() operation = asyncio.create_task( @@ -695,11 +886,22 @@ async def run_stage( raise RuntimeError("Copilot runtime cleanup failed: " + ", ".join(cleanup_errors)) if result is None: raise RuntimeError("Copilot stage terminated without an output") + observed_models = sorted( + { + item["model"] + for item in recorder.usage + if isinstance(item.get("model"), str) and item["model"] + } + ) result["_runtime"] = { "provider": "copilot", "sdk_version": SDK_VERSION, "cli_version": self.config.expected_cli_version, - "model": self.config.model, + "model": self.config.model + or (observed_models[0] if len(observed_models) == 1 else "copilot-default"), + "requested_model": self.config.model or "copilot-default", + "observed_models": observed_models, + "auth_mode": self.config.auth_mode, "role": self.config.role, "session_id": session_id, "memory": "fresh_per_stage", diff --git a/src/rove/ui/assistant_cli.py b/src/rove/ui/assistant_cli.py index 4ff84bd..582a589 100644 --- a/src/rove/ui/assistant_cli.py +++ b/src/rove/ui/assistant_cli.py @@ -4,6 +4,8 @@ import ipaddress import json import os +import re +import time from pathlib import Path from urllib.parse import urlsplit @@ -37,11 +39,29 @@ def local_url(value): def main(argv): parser = argparse.ArgumentParser(description=__doc__) parser.add_argument( - "action", choices=["status", "settings", "configure", "disable", "test", "ask", "confirm"] + "action", + choices=[ + "status", + "settings", + "configure", + "disable", + "test", + "ask", + "confirm", + "login", + "login-status", + "cancel-login", + "auth-status", + "models", + ], ) parser.add_argument("target", nargs="?", help="AskRequest JSON/YAML or saved proposal file") parser.add_argument("--url", default="http://127.0.0.1:5001") parser.add_argument("--endpoint", help="Existing copilot_agent endpoint ID for configure") + parser.add_argument("--provider", choices=["copilot", "endpoint", "foundry"]) + parser.add_argument("--model", help="Copilot model ID; omitted uses its default") + parser.add_argument("--foundry-endpoint", help="Azure model resource URL") + parser.add_argument("--deployment", help="Foundry model deployment name") parser.add_argument( "--proposal-output", type=Path, @@ -56,11 +76,20 @@ def main(argv): args = parser.parse_args(argv) if args.action in {"ask", "confirm"} and not args.target: parser.error("target is required") - if args.action == "configure" and not args.endpoint: - parser.error("configure requires --endpoint") + if args.action == "configure" and not (args.endpoint or args.provider): + parser.error("configure requires --provider or --endpoint") + if ( + any([args.provider, args.model, args.foundry_endpoint, args.deployment]) + and args.action != "configure" + ): + parser.error("provider options are only supported for configure") + if args.action in {"login-status", "cancel-login"} and ( + not args.target or not re.fullmatch(r"[\w.-]{1,128}", args.target) + ): + parser.error("a valid login operation ID is required") if args.endpoint and args.action != "configure": parser.error("--endpoint is only supported for configure") - if args.target and args.action not in {"ask", "confirm"}: + if args.target and args.action not in {"ask", "confirm", "login-status", "cancel-login"}: parser.error("this action does not accept a target") if args.action == "confirm" and not args.confirmed: parser.error("Review the saved proposal, then supply --confirmed to execute it") @@ -72,8 +101,26 @@ def execute(): if args.proposal_output and args.proposal_output.exists(): raise ValueError("Proposal output already exists; choose a new file") payload = None - if args.action in {"configure", "disable"}: - payload = {"endpoint_id": args.endpoint if args.action == "configure" else None} + if args.action == "disable": + payload = {"provider": "disabled"} + elif args.action == "configure": + provider = args.provider or "endpoint" + if provider == "copilot": + if args.endpoint or args.foundry_endpoint or args.deployment: + raise ValueError("Copilot configuration accepts only --model") + payload = {"provider": provider, "model": args.model or ""} + elif provider == "foundry": + if not args.foundry_endpoint or not args.deployment or args.endpoint or args.model: + raise ValueError("Foundry requires --foundry-endpoint and --deployment") + payload = { + "provider": provider, + "endpoint": args.foundry_endpoint, + "deployment": args.deployment, + } + else: + if not args.endpoint or args.model or args.foundry_endpoint or args.deployment: + raise ValueError("Local or own provider requires --endpoint") + payload = {"endpoint_id": args.endpoint} elif args.action == "ask": payload = AskRequest.model_validate(read_document(args.target)).model_dump() elif args.action == "confirm": @@ -106,7 +153,32 @@ def execute(): with httpx.Client(timeout=130, follow_redirects=False, trust_env=False) as client: action = "settings" if args.action in {"configure", "disable"} else args.action endpoint = url + "/api/assistant/" + action - if args.action in {"configure", "disable"}: + if args.action in {"auth-status", "models"}: + response = client.post( + url + "/api/assistant/copilot/status", + json={"include_models": args.action == "models"}, + ) + elif args.action == "login": + response = client.post(url + "/api/assistant/copilot/login") + response.raise_for_status() + operation = response.json() + deadline = time.monotonic() + 20 + while operation.get("status") == "starting" and time.monotonic() < deadline: + operation_id = operation.get("operation_id", "") + if not re.fullmatch(r"[\w.-]{1,128}", operation_id): + raise ValueError("Invalid sign-in operation returned by server") + time.sleep(0.5) + response = client.get(url + "/api/assistant/copilot/login/" + operation_id) + response.raise_for_status() + operation = response.json() + elif args.action in {"login-status", "cancel-login"}: + target = url + "/api/assistant/copilot/login/" + args.target + response = ( + client.delete(target) + if args.action == "cancel-login" + else client.get(target) + ) + elif args.action in {"configure", "disable"}: response = client.put(endpoint, json=payload) elif args.action == "test": response = client.post(endpoint) diff --git a/tests/test_assistant.py b/tests/test_assistant.py index 42b7480..fac4689 100644 --- a/tests/test_assistant.py +++ b/tests/test_assistant.py @@ -12,6 +12,7 @@ from rove.benchmarks.store import CampaignStore from rove.datasets.service import DatasetService from rove.models.config import RoveConfig +from rove.runtime.assistant import save_assistant_selection class ControlledRuntime: @@ -130,7 +131,8 @@ def confirmation(proposal): } -def test_unconfigured_is_optional_and_never_starts_runtime(tmp_path, monkeypatch): +def test_disabled_is_optional_and_never_starts_runtime(tmp_path, monkeypatch): + save_assistant_selection(None, root=tmp_path) monkeypatch.delenv("ROVE_ASSISTANT_ENDPOINT", raising=False) app = FastAPI() app.include_router(create_assistant_router(tmp_path, lambda _: pytest.fail("launch"))) diff --git a/tests/test_assistant_cli.py b/tests/test_assistant_cli.py index 00d2d04..d22b8f5 100644 --- a/tests/test_assistant_cli.py +++ b/tests/test_assistant_cli.py @@ -123,7 +123,7 @@ def test_invalid_confirmation_does_not_echo_rejected_token(tmp_path, capsys): "/api/assistant/settings", {"endpoint_id": "local-agent"}, ), - (["disable"], "PUT", "/api/assistant/settings", {"endpoint_id": None}), + (["disable"], "PUT", "/api/assistant/settings", {"provider": "disabled"}), (["test"], "POST", "/api/assistant/test", None), ], ) diff --git a/tests/test_assistant_providers.py b/tests/test_assistant_providers.py new file mode 100644 index 0000000..d6a1d1c --- /dev/null +++ b/tests/test_assistant_providers.py @@ -0,0 +1,228 @@ +"""Provider migration, credential boundaries and explicit sign-in lifecycle.""" + +import asyncio +import json +import sqlite3 + +import pytest + +from rove.models.config import RoveConfig +from rove.runtime.assistant import ( + assistant_selection, + assistant_settings, + assistant_settings_view, + normalize_selection, + save_assistant_selection, +) +from rove.runtime.assistant_login import CopilotLogin + + +@pytest.fixture +def workspace(tmp_path, monkeypatch): + monkeypatch.delenv("ROVE_ASSISTANT_ENDPOINT", raising=False) + monkeypatch.setattr("rove.runtime.assistant.active_config_path", lambda: tmp_path / "rove.yaml") + return tmp_path + + +def test_new_default_and_explicit_disabled_are_different_after_restart(workspace): + assert assistant_selection(root=workspace)["selection_source"] == "default" + default = assistant_settings(root=workspace) + assert default.auth_mode == "github" and default.model == "" and default.provider == {} + assert default.github_auth_directory.startswith(str(workspace)) + save_assistant_selection({"provider": "disabled"}, root=workspace) + assert assistant_selection(root=workspace)["selection"] == {"provider": "disabled"} + with pytest.raises(ValueError, match="disabled"): + assistant_settings(root=workspace) + + +@pytest.mark.parametrize("old_endpoint", [None, "local"]) +def test_legacy_rows_preserve_local_or_disabled_and_migrate(workspace, old_endpoint): + config = RoveConfig.model_validate( + { + "endpoints": { + "local": { + "type": "agent", + "adapter": "copilot_agent", + "config": { + "model": "local-model", + "provider": {"base_url": "http://127.0.0.1:1234/v1"}, + }, + } + } + } + ) + with sqlite3.connect(workspace / "assistant-settings.sqlite3") as db: + db.execute( + "CREATE TABLE assistant_settings (config_path TEXT PRIMARY KEY, endpoint_id TEXT)" + ) + db.execute( + "INSERT INTO assistant_settings VALUES (?, ?)", + (str(workspace / "rove.yaml"), old_endpoint), + ) + selected = assistant_selection(root=workspace)["selection"] + assert selected["provider"] == ("endpoint" if old_endpoint else "disabled") + if old_endpoint: + assert assistant_settings(root=workspace, config_loader=lambda: config).auth_mode == "byok" + save_assistant_selection({"provider": "copilot", "model": "selected-model"}, root=workspace) + assert assistant_settings(root=workspace).model == "selected-model" + with sqlite3.connect(workspace / "assistant-settings.sqlite3") as db: + saved = json.loads(db.execute("SELECT settings_json FROM assistant_settings").fetchone()[0]) + assert saved == {"provider": "copilot", "model": "selected-model"} + + +@pytest.mark.parametrize( + "url", + [ + "http://resource.openai.azure.com", + "https://resource.openai.azure.com.evil.test", + "https://evil.test/openai/v1", + "https://user:password@resource.openai.azure.com", # pragma: allowlist secret + "https://resource.openai.azure.com/?token=x", + "https://resource.openai.azure.com/#fragment", + "https://resource.services.ai.azure.com/api/projects/project", + "https://resource.inference.ai.azure.com", + "https://resource.openai.azure.com:1234", + "https://openai.azure.com", + "https://resource.openai.azure.com/openai/deployments/model", + ], +) +def test_foundry_cannot_send_azure_identity_to_arbitrary_destinations(url): + with pytest.raises(ValueError): + normalize_selection({"provider": "foundry", "endpoint": url, "deployment": "model"}) + + +def test_foundry_persists_deployment_and_url_without_credentials(workspace): + save_assistant_selection( + { + "provider": "foundry", + "endpoint": "https://factory.services.ai.azure.com", + "deployment": "robot-planner", + }, + root=workspace, + ) + settings = assistant_settings(root=workspace) + assert settings.auth_mode == "azure_cli" and settings.model == "robot-planner" + assert settings.provider == { + "type": "openai", + "base_url": "https://factory.services.ai.azure.com/openai/v1/", + "wire_api": "completions", + } + original = assistant_selection(root=workspace)["fingerprint"] + save_assistant_selection( + { + "provider": "foundry", + "endpoint": "https://factory.services.ai.azure.com", + "deployment": "robot-planner-v2", + }, + root=workspace, + ) + assert assistant_selection(root=workspace)["fingerprint"] != original + with pytest.raises(ValueError): + save_assistant_selection( + { + "provider": "foundry", + "endpoint": "https://factory.services.ai.azure.com", + "deployment": "robot-planner", + "api_key": "fixture", # pragma: allowlist secret + }, + root=workspace, + ) + + +def test_reuse_only_model_endpoints_not_foundry_agents(workspace): + config = RoveConfig.model_validate( + { + "endpoints": { + "model": { + "type": "vlm", + "adapter": "azure_openai", + "endpoint": "https://factory.openai.azure.com", + "config": {"deployment_name": "planner"}, + }, + "agent": { + "type": "agent", + "adapter": "azure_foundry_agent", + "config": { + "project_endpoint": "https://factory.services.ai.azure.com/api/projects/p", + "agent_name": "agent", + }, + }, + "placeholder": { + "type": "vlm", + "adapter": "azure_openai", + "endpoint": "https://your-resource.openai.azure.com", + "config": {"deployment_name": "model"}, + }, + } + } + ) + view = assistant_settings_view(root=workspace, config_loader=lambda: config) + assert [item["id"] for item in view["foundry_endpoints"]] == ["model"] + assert view["foundry_endpoints"][0]["deployment"] == "planner" + + +class Process: + def __init__(self, output=b"", complete=False): + self.stdout = asyncio.StreamReader() + self.stdout.feed_data(output) + self.returncode = 0 if complete else None + self.done = asyncio.Event() + self.terminated = False + if complete: + self.stdout.feed_eof() + self.done.set() + + async def wait(self): + await self.done.wait() + return self.returncode + + def terminate(self): + self.terminated = True + self.returncode = -15 + self.stdout.feed_eof() + self.done.set() + + def kill(self): + self.terminate() + + +@pytest.mark.asyncio +async def test_login_exposes_only_device_code_and_reaps_cancelled_process(workspace, monkeypatch): + monkeypatch.setattr("rove.runtime.assistant_login.shutil.which", lambda _: "/verified/copilot") + process = Process( + b"private provider diagnostics\nTo authenticate, visit https://github.com/login/device and enter code ABCD-1234.\nWaiting for authorization...\n" + ) + calls = [] + + async def spawn(*args, **kwargs): + calls.append((args, kwargs)) + return process + + manager = CopilotLogin(str(workspace / "auth"), spawn=spawn) + first = await manager.start() + await asyncio.sleep(0) + pending = manager.get(first["operation_id"]) + assert pending["status"] == "pending" and pending["user_code"] == "ABCD-1234" + assert "private" not in json.dumps(pending) + assert (await manager.start())["operation_id"] == first["operation_id"] + assert len(calls) == 1 and calls[0][0][-2:] == ("login", "--device-code") + assert calls[0][1]["env"]["COPILOT_HOME"] == str(workspace / "auth") + assert not set(calls[0][1]["env"]) & {"GH_TOKEN", "GITHUB_TOKEN", "COPILOT_GITHUB_TOKEN"} + await manager.cancel(first["operation_id"]) + assert process.terminated and manager.get(first["operation_id"])["user_code"] is None + + +@pytest.mark.asyncio +async def test_login_timeout_reaps_process_and_removes_device_code(workspace, monkeypatch): + monkeypatch.setattr("rove.runtime.assistant_login.shutil.which", lambda _: "/verified/copilot") + process = Process() + + async def spawn(*args, **kwargs): + return process + + manager = CopilotLogin(str(workspace / "auth"), spawn=spawn, timeout=0.01) + operation = await manager.start() + await manager.operation.task + assert manager.get(operation["operation_id"])["status"] == "expired" + assert process.terminated + await manager.close() diff --git a/tests/test_assistant_settings.py b/tests/test_assistant_settings.py index d53f83d..74aecae 100644 --- a/tests/test_assistant_settings.py +++ b/tests/test_assistant_settings.py @@ -86,7 +86,8 @@ def client(override=None): def test_saved_selection_survives_new_router_and_changes_insight_model(setup): root, config, _, calls, client, private = setup initial = client().get("/api/assistant/settings").json() - assert not initial["configured"] and initial["selection_source"] == "disabled" + assert initial["configured"] and initial["selection_source"] == "default" + assert initial["selection"] == {"provider": "copilot", "model": ""} assert not (root / "assistant-settings.sqlite3").exists() assert {e["id"] for e in initial["endpoints"]} == {"first", "second"} assert initial["endpoints"][0]["provider_base_url"] == "https://provider.invalid" @@ -151,7 +152,7 @@ def test_selections_are_scoped_by_active_config_path(setup, monkeypatch): root, _, _, _, client, _ = setup client().put("/api/assistant/settings", json={"endpoint_id": "first"}) monkeypatch.setattr("rove.runtime.assistant.active_config_path", lambda: root / "other.yaml") - assert client().get("/api/assistant/settings").json()["effective_endpoint_id"] is None + assert client().get("/api/assistant/settings").json()["selection_source"] == "default" client().put("/api/assistant/settings", json={"endpoint_id": "second"}) monkeypatch.setattr("rove.runtime.assistant.active_config_path", lambda: root / "rove.yaml") assert client().get("/api/assistant/settings").json()["effective_endpoint_id"] == "first" @@ -160,6 +161,7 @@ def test_selections_are_scoped_by_active_config_path(setup, monkeypatch): def test_status_is_not_provider_proof_but_explicit_test_is(setup): _, _, _, calls, client, _ = setup api = client() + api.put("/api/assistant/settings", json={"provider": "disabled"}) assert not api.post("/api/assistant/test").json()["provider_tested"] api.put("/api/assistant/settings", json={"endpoint_id": "second"}) status = api.get("/api/assistant/status").json() @@ -169,6 +171,7 @@ def test_status_is_not_provider_proof_but_explicit_test_is(setup): "available": True, "provider_tested": True, "endpoint_id": "second", + "fingerprint": api.get("/api/assistant/settings").json()["fingerprint"], "reason": None, } settings, tools, stage, image, prompt, context = calls[0] diff --git a/tests/test_campaign_insights.py b/tests/test_campaign_insights.py index e537b1b..17f05e9 100644 --- a/tests/test_campaign_insights.py +++ b/tests/test_campaign_insights.py @@ -16,6 +16,7 @@ from rove.benchmarks.store import CampaignStore from rove.datasets.service import DatasetService from rove.models.config import RoveConfig +from rove.runtime.assistant import save_assistant_selection class Runtime: @@ -193,7 +194,8 @@ def summary_output(h, cid): @pytest.mark.asyncio -async def test_unconfigured_template_is_honest_and_never_calls_provider_or_writes(h): +async def test_disabled_template_is_honest_and_never_calls_provider_or_writes(h): + save_assistant_selection(None, root=h["root"]) service = CampaignInsights( h["root"], config_loader=lambda: h["config"], runtime_factory=h["runtime"] ) @@ -594,7 +596,8 @@ async def change_evidence(*args): @pytest.mark.asyncio -async def test_unconfigured_summary_distinguishes_missing_setup_from_generation_failure(h): +async def test_disabled_summary_distinguishes_missing_setup_from_generation_failure(h): + save_assistant_selection(None, root=h["root"]) cid = make_campaign(h) insights = CampaignInsights( h["root"], config_loader=lambda: h["config"], runtime_factory=h["runtime"] diff --git a/tests/test_runtime_auth.py b/tests/test_runtime_auth.py new file mode 100644 index 0000000..d206e21 --- /dev/null +++ b/tests/test_runtime_auth.py @@ -0,0 +1,297 @@ +"""Controlled authentication contracts; no live sign-in or paid inference.""" + +import asyncio +import json +import os +import sys +import threading +import time +from dataclasses import asdict +from pathlib import Path +from types import SimpleNamespace + +import pytest + +from rove.runtime import copilot +from rove.runtime.copilot import CopilotRuntime, CopilotRuntimeConfig, github_login_environment + + +class Client: + def __init__(self, auth_type="user", authenticated=True): + self.auth_type = auth_type + self.authenticated = authenticated + self.calls = [] + self.sessions = [] + + async def start(self): + self.calls.append("start") + + async def get_auth_status(self): + self.calls.append("auth") + return SimpleNamespace( + isAuthenticated=self.authenticated, authType=self.auth_type, login="test-user" + ) + + async def list_models(self): + self.calls.append("models") + return [SimpleNamespace(id="model-a", name="Model A")] + + async def create_session(self, **kwargs): + self.sessions.append(kwargs) + return Session(self, kwargs) + + async def stop(self): + self.calls.append("stop") + + +class Session: + def __init__(self, client, config): + self.client = client + self.config = config + self.rpc = SimpleNamespace(options=SimpleNamespace(update=self.update)) + + async def update(self, params): + assert params.installed_plugins == [] + self.client.calls.append("isolate") + + async def send_and_wait(self, *args, **kwargs): + self.client.calls.append("send") + self.config["on_event"]( + { + "id": "usage", + "type": "assistant.usage", + "data": {"model": "resolved-model", "input_tokens": 1}, + } + ) + return {"data": {"content": '{"connection":"ok"}'}} + + async def abort(self): + self.client.calls.append("abort") + + async def disconnect(self): + self.client.calls.append("disconnect") + + +def github(tmp_path, client=None, model=""): + client = client or Client() + options = [] + + def factory(**kwargs): + options.append(kwargs) + return client + + config = CopilotRuntimeConfig( + model=model, + auth_mode="github", + github_auth_directory=str(tmp_path / "auth"), + native_traces=False, + ) + return CopilotRuntime(config, client_factory=factory), client, options + + +async def test_github_default_has_no_byok_route_and_keeps_session_isolation(tmp_path, monkeypatch): + monkeypatch.setenv("GH_TOKEN", "private-ambient-value") + monkeypatch.setenv("COPILOT_PROVIDER_API_KEY", "private-provider-value") + monkeypatch.setenv("GH_CONFIG_DIR", "/private/ambient/config") + runtime, client, options = github(tmp_path) + output = await runtime.run_stage("assist", "", "Synthetic test") + config = client.sessions[0] + assert "model" not in config and "provider" not in config + assert config["available_tools"] == [] and config["memory"] == {"enabled": False} + assert config["enable_session_store"] is False and config["enable_config_discovery"] is False + assert config["enable_skills"] is False and config["enable_host_git_operations"] is False + assert config["enable_on_demand_instruction_discovery"] is False + assert config["plugin_directories"] == [] and config["included_builtin_skills"] == [] + assert client.calls == ["start", "auth", "isolate", "send", "abort", "disconnect", "stop"] + assert options[0]["use_logged_in_user"] and options[0]["mode"] == "copilot-cli" + env = options[0]["env"] + assert "GH_TOKEN" not in env and "COPILOT_PROVIDER_API_KEY" not in env + assert env["GH_CONFIG_DIR"] == str(tmp_path / "auth" / "isolated-gh") + assert options[0]["base_directory"] == str(tmp_path / "auth") + assert not Path(options[0]["working_directory"]).exists() + assert output["_runtime"]["requested_model"] == "copilot-default" + assert output["_runtime"]["observed_models"] == ["resolved-model"] + assert output["_runtime"]["model"] == "resolved-model" + assert "private-ambient-value" not in json.dumps(output) + + +@pytest.mark.parametrize("auth_type", ["gh-cli", "env", "token", None]) +async def test_github_rejects_ambient_auth_sources_before_inference(tmp_path, auth_type): + runtime, client, _ = github(tmp_path, Client(auth_type=auth_type)) + with pytest.raises(RuntimeError, match="dedicated ROVE"): + await runtime.run_stage("assist", "", "Synthetic") + assert not client.sessions and client.calls[-1] == "stop" + status = await runtime.github_identity(include_models=True) + assert not status["authenticated"] and status["models"] == [] + assert "models" not in client.calls + + +async def test_github_identity_discovers_models_without_inference(tmp_path): + runtime, client, _ = github(tmp_path) + assert await runtime.github_identity(include_models=True) == { + "authenticated": True, + "login": "test-user", + "models": [{"id": "model-a", "name": "Model A"}], + "reason": None, + } + assert not client.sessions and client.calls[-1] == "stop" + + +async def test_github_explicit_model_preserves_requested_identity(tmp_path): + runtime, client, _ = github(tmp_path, model="selected-model") + output = await runtime.run_stage("assist", "", "Synthetic") + assert client.sessions[0]["model"] == "selected-model" + assert output["_runtime"]["requested_model"] == "selected-model" + + +@pytest.mark.parametrize( + "url", + [ + "http://resource.openai.azure.com/openai/v1", + "https://resource.openai.azure.com.evil.invalid/openai/v1", + "https://resource.openai.azure.com/inference", + "https://resource.services.ai.azure.com/api/projects/project/openai/v1", + "https://user@resource.openai.azure.com/openai/v1", + "https://resource.openai.azure.com/openai/v1?token=x", + ], +) +def test_azure_cli_bearer_requires_explicit_supported_foundry_endpoint(url): + with pytest.raises(ValueError, match="Foundry"): + CopilotRuntimeConfig("deployment", {"base_url": url}, auth_mode="azure_cli") + + +@pytest.mark.parametrize( + "auth_field", ["api_key", "api_key_env", "bearer_token", "bearer_token_provider", "headers"] +) +def test_azure_cli_never_uses_alternative_credentials(auth_field): + with pytest.raises(ValueError, match="alternative"): + CopilotRuntimeConfig( + "deployment", + {"base_url": "https://resource.openai.azure.com/openai/v1", auth_field: "unused"}, + auth_mode="azure_cli", + ) + + +async def test_azure_token_fresh_per_execution_not_serialized_or_fallback(monkeypatch): + calls = [] + + async def token(): + calls.append("acquire") + return f"private-token-{len(calls)}" + + monkeypatch.setattr(copilot, "_azure_cli_token", token) + config = CopilotRuntimeConfig( + "deployment", + {"type": "openai", "base_url": "https://resource.services.ai.azure.com/openai/v1"}, + auth_mode="azure_cli", + native_traces=False, + ) + client = Client() + options = [] + + def factory(**kwargs): + options.append(kwargs) + return client + + runtime = CopilotRuntime(config, client_factory=factory) + first = await runtime.run_stage("assist", "", "Synthetic") + second = await runtime.run_stage("assist", "", "Synthetic") + assert len(calls) == 2 + assert ( + client.sessions[0]["provider"]["bearer_token"] + != client.sessions[1]["provider"]["bearer_token"] + ) + assert all(not option["use_logged_in_user"] and option["mode"] == "empty" for option in options) + assert "auth" not in client.calls + assert "private-token" not in json.dumps([asdict(config), first, second, options]) + + +async def test_azure_credential_acquisition_is_offloop_bounded_and_drained_on_cancel(monkeypatch): + started, release = threading.Event(), threading.Event() + closed = [] + + class Credential: + def __init__(self, **kwargs): + assert kwargs == {"process_timeout": 10} + + def __enter__(self): + return self + + def __exit__(self, *args): + closed.append(True) + + def get_token(self, scope): + assert scope == "https://ai.azure.com/.default" + assert threading.current_thread() is not threading.main_thread() + started.set() + assert release.wait(2) + return SimpleNamespace(token="private-token", expires_on=time.time() + 120) + + monkeypatch.setitem( + sys.modules, "azure.identity", SimpleNamespace(AzureCliCredential=Credential) + ) + task = asyncio.create_task(copilot._azure_cli_token()) + assert await asyncio.to_thread(started.wait, 1) + task.cancel() + await asyncio.sleep(0) + assert not task.done() + release.set() + with pytest.raises(asyncio.CancelledError): + await task + assert closed == [True] + + +def test_github_requires_explicit_profile_and_cannot_mix_providers(): + with pytest.raises(ValueError, match="absolute ROVE profile"): + CopilotRuntimeConfig("", auth_mode="github") + with pytest.raises(ValueError, match="BYOK"): + CopilotRuntimeConfig("", {"base_url": "http://127.0.0.1"}, auth_mode="github") + with pytest.raises(ValueError, match="absolute profile"): + github_login_environment("relative") + + +@pytest.mark.skipif(sys.platform == "win32", reason="POSIX controlled CLI executable") +async def test_installed_azure_identity_runs_controlled_cli_and_reaps_timeout( + tmp_path, monkeypatch +): + identity = pytest.importorskip("azure.identity") + native_credential = identity.AzureCliCredential + executable = tmp_path / "az" + pid_file = tmp_path / "pid" + executable.write_text( + f"#!{sys.executable}\nimport os,time\n" + f"open({str(pid_file)!r},'w').write(str(os.getpid()))\n" + "time.sleep(30)\n" + ) + executable.chmod(0o700) + monkeypatch.setenv("PATH", str(tmp_path)) + + def bounded_native(**kwargs): + assert kwargs == {"process_timeout": 10} + # Keep the test fast while exercising the installed SDK's actual subprocess timeout. + return native_credential(process_timeout=2) + + monkeypatch.setattr(identity, "AzureCliCredential", bounded_native) + started = time.monotonic() + with pytest.raises(RuntimeError, match="Azure CLI authentication failed"): + await copilot._azure_cli_token() + assert time.monotonic() - started < 6 + with pytest.raises(ProcessLookupError): + os.kill(int(pid_file.read_text()), 0) + + +async def test_azure_authentication_failure_stops_before_any_model_request(monkeypatch): + async def fail(): + raise RuntimeError("Azure CLI authentication failed") + + monkeypatch.setattr(copilot, "_azure_cli_token", fail) + config = CopilotRuntimeConfig( + "deployment", + {"base_url": "https://resource.openai.azure.com/openai/v1"}, + auth_mode="azure_cli", + ) + client = Client() + runtime = CopilotRuntime(config, client_factory=lambda **_: client) + with pytest.raises(RuntimeError, match="authentication failed"): + await runtime.run_stage("assist", "", "Synthetic") + assert client.sessions == [] and client.calls == ["start", "stop"] diff --git a/tests/test_trial_composer_ui.cjs b/tests/test_trial_composer_ui.cjs index e441afb..a7ebd7c 100644 --- a/tests/test_trial_composer_ui.cjs +++ b/tests/test_trial_composer_ui.cjs @@ -13,17 +13,24 @@ async function composer(options = {}) { w.EventSource = class { constructor(){this.listeners = {}; streams.push(this);} addEventListener(type, listener){this.listeners[type] = listener;} close(){} emit(type, data){this.listeners[type]?.({data: JSON.stringify(data)});} }; const item = {id: "case-r1", case_id: "case", name: "Block on tray", task: "Place the block in the tray", revision: 1, image_asset: {sha256: "a".repeat(64)}, candidate_context: {robot: "panda", proprioception: [0, 1], constraints: ["Avoid cup"]}, reference_data: {eval_qa: {answer: "private answer"}, expected_subtasks: ["private plan"]}}; let assistantSettings=options.assistantSettings || {selected_endpoint_id:null,effective_endpoint_id:null,selection_source:"disabled",read_only:false,endpoints:[],configured:false}; + let loginPolls=0; + if(options.fastLoginPoll){const original=w.setTimeout.bind(w);w.setTimeout=(callback,delay,...args)=>original(callback,delay===1500?1:delay,...args);} w.fetch = async (url, request = {}) => { const route = new URL(url, w.location.href).pathname; calls.push({route, method: request.method || "GET", body: request.body}); if(route==="/api/assistant/settings"){ if(request.method==="PUT"){ if(options.assistantSaveFail)return {ok:false,json:async()=>({detail:"Endpoint no longer exists"})}; - const endpoint=JSON.parse(request.body).endpoint_id;assistantSettings={...assistantSettings,selected_endpoint_id:endpoint,effective_endpoint_id:endpoint,configured:Boolean(endpoint),selection_source:endpoint?"saved":"disabled"}; + const payload=JSON.parse(request.body),endpoint=payload.endpoint_id || null; + if(payload.provider)assistantSettings={...assistantSettings,selection:payload,fingerprint:JSON.stringify(payload),effective_endpoint_id:payload.endpoint_id || null,configured:payload.provider!=="disabled",selection_source:payload.provider==="disabled"?"disabled":"saved"}; + else assistantSettings={...assistantSettings,selected_endpoint_id:endpoint,effective_endpoint_id:endpoint,configured:Boolean(endpoint),selection_source:endpoint?"saved":"disabled"}; } return {ok:true,json:async()=>assistantSettings}; } if(route==="/api/assistant/status")return {ok:true,json:async()=>options.assistantStatus || {configured:assistantSettings.configured,runtime_available:true,available:true,provider_tested:false}}; - if(route==="/api/assistant/test"){await options.assistantTestWait;return {ok:true,json:async()=>options.assistantTestResult || {available:true,provider_tested:true,endpoint_id:assistantSettings.effective_endpoint_id}};} + if(route==="/api/assistant/test"){await options.assistantTestWait;return {ok:true,json:async()=>options.assistantTestResult || {available:true,provider_tested:true,endpoint_id:assistantSettings.effective_endpoint_id,fingerprint:assistantSettings.fingerprint}};} + if(route==="/api/assistant/copilot/status")return {ok:true,json:async()=>options.copilotStatus || {authenticated:true,login:"robot-builder",models:[{id:"model-1",name:"Model One"}]}}; + if(route==="/api/assistant/copilot/login"){await options.assistantLoginWait;return {ok:true,json:async()=>options.loginStart || {operation_id:"login-1",status:"pending",verification_uri:"https://github.com/login/device",user_code:"ABCD-1234"}};} + if(route.startsWith("/api/assistant/copilot/login/"))return {ok:true,json:async()=>request.method==="DELETE"?{status:"cancelled"}:options.loginPoll?.[loginPolls++] || {operation_id:"login-1",status:"complete"}}; if (route === "/api/history") return {ok:true,json:async()=>options.serverHistory || []}; if (route === "/api/cases/case-r1") return {ok: !options.missing, json: async () => item}; if (route.startsWith("/api/trial-assets/")) return {ok: true, blob: async () => new w.Blob(["image"], {type: "image/png"})}; @@ -422,14 +429,14 @@ test("Assistant settings saves an explicit endpoint and tests the provider separ try{ assert.equal(el("assistantSettingsHeading").textContent,"Assistant");assert.equal(el("assistantEndpoint").value,"");assert.match(el("assistantConnectionStatus").textContent,/disabled/); assert.equal(el("testAssistantConnection").disabled,true); - el("assistantEndpoint").value="assistant-a";el("assistantEndpoint").dispatchEvent(new w.Event("change")); + el("assistantProvider").value="endpoint";el("assistantProvider").dispatchEvent(new w.Event("change"));el("assistantEndpoint").value="assistant-a";el("assistantEndpoint").dispatchEvent(new w.Event("change")); assert.match(el("assistantEndpointDescription").textContent,/model-a.*127.0.0.1/);assert.equal(el("testAssistantConnection").disabled,true); el("saveAssistantSettings").click();await pause(); const save=calls.find(call=>call.route==="/api/assistant/settings"&&call.method==="PUT");assert.deepEqual(JSON.parse(save.body),{endpoint_id:"assistant-a"}); assert.equal(calls.some(call=>call.route==="/api/assistant/test"),false);assert.match(el("assistantConnectionStatus").textContent,/saved.*Test the connection/); el("testAssistantConnection").click();await pause();assert.match(el("assistantConnectionStatus").textContent,/Connection successful/); assert.equal(el("assistantReturnToCampaign").getAttribute("href"),"/static/datasets.html?step=review&campaign=campaign"); - el("assistantEndpoint").value="";el("assistantEndpoint").dispatchEvent(new w.Event("change"));el("saveAssistantSettings").click();await pause(); + el("assistantProvider").value="disabled";el("assistantProvider").dispatchEvent(new w.Event("change"));el("saveAssistantSettings").click();await pause(); assert.equal(JSON.parse(calls.filter(call=>call.method==="PUT").at(-1).body).endpoint_id,null);assert.equal(el("testAssistantConnection").disabled,true); }finally{w.close();} }); @@ -438,7 +445,7 @@ test("Assistant settings distinguish environment control, failed saves and real const locked=await composer({url:"http://localhost/?view=settings#assistant",assistantSettings:{...assistantFixture,effective_endpoint_id:"assistant-a",configured:true,read_only:true,selection_source:"environment",override_reason:"Set by ROVE_ASSISTANT_ENDPOINT"},assistantTestResult:{available:false,provider_tested:false,endpoint_id:"assistant-a",reason:"Provider unreachable"}}); try{assert.equal(locked.el("assistantEndpoint").disabled,true);assert.equal(locked.el("saveAssistantSettings").disabled,true);assert.match(locked.el("assistantSettingsControl").textContent,/ROVE_ASSISTANT_ENDPOINT/);assert.match(locked.el("assistantConnectionStatus").textContent,/connection has not been tested/);locked.el("testAssistantConnection").click();await locked.pause();assert.match(locked.el("assistantConnectionStatus").textContent,/Provider unreachable/);}finally{locked.w.close();} const failed=await composer({url:"http://localhost/?view=settings&return=https%3A%2F%2Fevil.test%2Fstatic%2Fdatasets.html%3Fcampaign%3Dx#assistant",assistantSettings:assistantFixture,assistantSaveFail:true}); - try{failed.el("assistantEndpoint").value="assistant-a";failed.el("assistantEndpoint").dispatchEvent(new failed.w.Event("change"));failed.el("saveAssistantSettings").click();await failed.pause();assert.match(failed.el("assistantConnectionStatus").textContent,/Could not save.*no longer exists/);assert.equal(failed.el("testAssistantConnection").disabled,true);assert.equal(failed.el("assistantReturnToCampaign"),null);}finally{failed.w.close();} + try{failed.el("assistantProvider").value="endpoint";failed.el("assistantProvider").dispatchEvent(new failed.w.Event("change"));failed.el("assistantEndpoint").value="assistant-a";failed.el("assistantEndpoint").dispatchEvent(new failed.w.Event("change"));failed.el("saveAssistantSettings").click();await failed.pause();assert.match(failed.el("assistantConnectionStatus").textContent,/Could not save.*no longer exists/);assert.equal(failed.el("testAssistantConnection").disabled,true);assert.equal(failed.el("assistantReturnToCampaign"),null);}finally{failed.w.close();} }); test("a late connection probe cannot mark a different unsaved assistant selection connected",async()=>{ @@ -459,3 +466,76 @@ test("Assistant settings with no eligible endpoint explain configuration instead assert.equal(calls.some(call=>call.route==="/api/assistant/test"),false); }finally{w.close();} }); + +test("new Copilot defaults preserve explicit local/disabled choices and model loading is an explicit action",async()=>{ + for(const selection of [{model:"",provider:"copilot"},{provider:"endpoint",endpoint_id:"assistant-a"},{provider:"disabled"}]){ + const {w,el,calls,pause}=await composer({url:"http://localhost/?view=settings#assistant",assistantSettings:{...assistantFixture,selection,fingerprint:"initial",configured:selection.provider!=="disabled",selection_source:selection.provider==="copilot"?"default":"saved"}}); + try{ + assert.equal(el("assistantProvider").value,selection.provider);assert.equal(el("saveAssistantSettings").disabled,true,"JSON key ordering never creates a false unsaved change"); + assert.equal(calls.some(call=>call.route.includes("/copilot/")),false,"opening settings never starts sign-in or model enumeration"); + assert.equal(el("assistantEndpointFields").hidden,selection.provider!=="endpoint"); + if(selection.provider==="copilot"){ + assert.equal(el("assistantCopilotModel").options[0].textContent,"Copilot default");el("checkCopilotSignIn").click();await pause(); + assert.deepEqual(JSON.parse(calls.find(call=>call.route==="/api/assistant/copilot/status").body),{include_models:true}); + assert.match(el("assistantCopilotStatus").textContent,/Signed in as robot-builder/); + el("assistantCopilotModel").value="model-1";el("assistantCopilotModel").dispatchEvent(new w.Event("change"));assert.equal(el("testAssistantConnection").disabled,true); + el("saveAssistantSettings").click();await pause();assert.deepEqual(JSON.parse(calls.find(call=>call.method==="PUT").body),{provider:"copilot",model:"model-1"}); + el("testAssistantConnection").click();await pause();assert.match(el("assistantConnectionStatus").textContent,/Connection successful/); + } + }finally{w.close();} + } +}); + +test("Foundry uses a model endpoint and deployment, validates inputs and reuses only supplied model connections",async()=>{ + const {w,el,calls,pause}=await composer({url:"http://localhost/?view=settings#assistant",assistantSettings:{...assistantFixture,selection:{provider:"endpoint",endpoint_id:"assistant-a"},fingerprint:"local",foundry_endpoints:[{id:"azure-model",display_name:"Azure model",endpoint:"https://robot.openai.azure.com",deployment:"robot-deployment"}]}}); + try{ + el("assistantProvider").value="foundry";el("assistantProvider").dispatchEvent(new w.Event("change"));assert.equal(el("assistantFoundryFields").hidden,false);assert.match(el("assistantFoundryFields").textContent,/Azure CLI.*az login.*not a Foundry agent/s); + assert.equal(el("saveAssistantSettings").disabled,true);assert.equal(el("assistant").querySelector('input[type="password"]'),null); + el("assistantFoundryEndpoint").value="http://unsafe.test";el("assistantFoundryDeployment").value="model";el("assistantFoundryEndpoint").dispatchEvent(new w.Event("input"));assert.equal(el("saveAssistantSettings").disabled,true); + el("assistantFoundryPreset").value="azure-model";el("assistantFoundryPreset").dispatchEvent(new w.Event("change"));assert.equal(el("saveAssistantSettings").disabled,false); + el("saveAssistantSettings").click();await pause();assert.deepEqual(JSON.parse(calls.find(call=>call.method==="PUT").body),{provider:"foundry",endpoint:"https://robot.openai.azure.com",deployment:"robot-deployment"}); + }finally{w.close();} +}); + +test("Copilot device sign-in exposes only the GitHub device page, polls completion and never implicitly lists models",async()=>{ + const {w,el,calls,pause}=await composer({url:"http://localhost/?view=settings#assistant",assistantSettings:{...assistantFixture,selection:{provider:"copilot",model:""},fingerprint:"copilot"},fastLoginPoll:true}); + try{ + el("signInCopilot").click();await pause();assert.match(el("assistantCopilotStatus").textContent,/sign-in completed/); + assert.ok(calls.some(call=>call.route==="/api/assistant/copilot/login/login-1"&&call.method==="GET"));assert.equal(el("assistantDeviceLogin").hidden,true); + assert.equal(calls.some(call=>call.route==="/api/assistant/copilot/status"),false); + }finally{w.close();} +}); + +test("provider changes cancel device sign-in and stale starts cannot expose a code or keep polling",async()=>{ + let release;const wait=new Promise(resolve=>{release=resolve;}); + const {w,el,calls,pause}=await composer({url:"http://localhost/?view=settings#assistant",assistantSettings:{...assistantFixture,selection:{provider:"copilot",model:""},fingerprint:"copilot"},assistantLoginWait:wait}); + try{ + el("signInCopilot").click();el("assistantProvider").value="endpoint";el("assistantProvider").dispatchEvent(new w.Event("change"));release();await pause(); + assert.equal(el("assistantDeviceLogin").hidden,true);assert.ok(calls.some(call=>call.route==="/api/assistant/copilot/login/login-1"&&call.method==="DELETE")); + assert.equal(calls.some(call=>call.route==="/api/assistant/copilot/login/login-1"&&call.method==="GET"),false); + }finally{release();w.close();} +}); + +test("device codes use only the allowed GitHub URL and cancel when leaving Settings",async()=>{ + for(const verification_uri of ["https://github.com/login/device","https://evil.test/login/device"]){ + const {w,el,calls,pause}=await composer({url:"http://localhost/?view=settings#assistant",assistantSettings:{...assistantFixture,selection:{provider:"copilot",model:""},fingerprint:"copilot"},loginStart:{operation_id:"login-1",status:"pending",verification_uri,user_code:"ABCD-1234"}}); + try{ + el("signInCopilot").click();await pause(); + if(verification_uri.startsWith("https://github.com/")){assert.equal(el("assistantDeviceCode").textContent,"ABCD-1234");assert.equal(el("assistantDeviceLogin").querySelector("a").href,verification_uri);}else assert.equal(el("assistantDeviceLogin").querySelector("a"),null); + w.document.querySelector('[data-root-view="models"]').click();await pause();assert.ok(calls.some(call=>call.route==="/api/assistant/copilot/login/login-1"&&call.method==="DELETE")); + assert.equal(el("assistantDeviceLogin").hidden,true); + }finally{w.close();} + } +}); + +test("provider test fingerprints must match the saved model configuration",async()=>{ + const {w,el,pause}=await composer({url:"http://localhost/?view=settings#assistant",assistantSettings:{...assistantFixture,selection:{provider:"copilot",model:"model-1"},fingerprint:"new-model",configured:true},assistantTestResult:{available:true,provider_tested:true,fingerprint:"old-model"}}); + try{el("testAssistantConnection").click();await pause();assert.match(el("assistantConnectionStatus").textContent,/could not be verified/);assert.doesNotMatch(el("assistantConnectionStatus").textContent,/Connection successful/);}finally{w.close();} +}); + +test("device sign-in can be cancelled while starting and terminal failures remain explicit",async()=>{ + let release;const wait=new Promise(resolve=>{release=resolve;}); + const pending=await composer({url:"http://localhost/?view=settings#assistant",assistantSettings:{...assistantFixture,selection:{provider:"copilot",model:""},fingerprint:"copilot"},assistantLoginWait:wait}); + try{pending.el("signInCopilot").click();assert.equal(pending.el("cancelCopilotSignIn").hidden,false);pending.el("cancelCopilotSignIn").click();release();await pending.pause();assert.match(pending.el("assistantCopilotStatus").textContent,/cancelled/);assert.equal(pending.el("assistantDeviceLogin").hidden,true);assert.ok(pending.calls.some(call=>call.method==="DELETE"));}finally{release();pending.w.close();} + for(const status of ["failed","expired"]){const ended=await composer({url:"http://localhost/?view=settings#assistant",assistantSettings:{...assistantFixture,selection:{provider:"copilot",model:""},fingerprint:"copilot"},loginStart:{operation_id:"login-1",status,reason:"Start a new sign-in attempt"}});try{ended.el("signInCopilot").click();await ended.pause();assert.match(ended.el("assistantCopilotStatus").textContent,/Start a new/);assert.equal(ended.el("signInCopilot").disabled,false);assert.equal(ended.el("cancelCopilotSignIn").hidden,true);}finally{ended.w.close();}} +}); diff --git a/tests/test_workflow_cli.py b/tests/test_workflow_cli.py index 8ebf339..538a09d 100644 --- a/tests/test_workflow_cli.py +++ b/tests/test_workflow_cli.py @@ -11,6 +11,8 @@ from rove.benchmarks.report import saved_report from rove.benchmarks.store import CampaignStore from rove.datasets.service import DatasetService +from rove.models.config import load_config +from rove.runtime.assistant import save_assistant_selection from rove.trials import cli as trial_cli from rove.trials.store import TrialStore @@ -251,8 +253,10 @@ def test_saved_case_cli_uses_its_robot_without_separate_urdf_and_rejects_changed assert "differs" in error and TrialStore(h["root"]).count() == before -def test_cli_metrics_draft_is_honest_when_unconfigured(h, capsys): +def test_cli_metrics_draft_is_honest_when_disabled(h, capsys): draft, _ = seed_and_request(h, capsys) + load_config(h["config"]) + save_assistant_selection(None, root=h["root"]) output = invoke( workflow_cli, [