control-plane: per-agent model providers, OpenRouter support, failure-injection audit

control-plane/README.md

@@ -188,9 +188,10 @@ instance the worker happened to reach. Fine for one instance (`O10` as currently
 instance would need Postgres `LISTEN`/`NOTIFY` or a shared queue, which is out of scope here.
 
 **Configuration** (`mergit_cp/config.py`). Required: `DATABASE_URL`, `APP_DOMAIN`, `SESSION_SECRET` (32+
-characters), `GATEWAY_URL`, `STORE_URL`, `AGENTS_FILE`, `PLANNER_MODEL`, `EVENTS_TOKEN`, `CHAIN_EVENTS_TOKEN`,
-`WORKER_TOKEN`, plus the model provider key. Defaults: `CHAIN_ID` (84532), `PLANS_PER_HOUR` (10),
-`SUGGESTIONS_PER_HOUR` (60, exploring suggestions only), `PLANNER_MAX_TASKS` (8), `MAX_ATTEMPTS` (3), `REPO_HOSTS` (github.com), `CORS_ORIGINS`, `SSE_POLL_S` (1),
+characters), `GATEWAY_URL`, `STORE_URL`, `AGENTS_FILE`, `PLANNER_MODEL`, `PLANNER_PROVIDER`, `EVENTS_TOKEN`,
+`CHAIN_EVENTS_TOKEN`, `WORKER_TOKEN`, plus whichever model key `PLANNER_PROVIDER` needs. Defaults: `CHAIN_ID`
+(84532), `PLANS_PER_HOUR` (10), `SUGGESTIONS_PER_HOUR` (60, exploring suggestions only), `PLANNER_MAX_TASKS` (8),
+`MAX_ATTEMPTS` (3), `REPO_HOSTS` (github.com), `CORS_ORIGINS`, `SSE_POLL_S` (1),
 `OUTCOME_POLL_S` (1).
 `WORKER_TOKEN` is this deployment's own choice, like `CHAIN_EVENTS_TOKEN`: spec/events-v0.md does not name a
 token for this leg either, since the whole endpoint is new in O7. The planner model must be priced in
@@ -261,7 +262,9 @@ depends on it. `NullProgressReporter` (the default) sends nothing, for a worker
 
 Typed, from the environment (`mergit_worker/config.py`). Required: `DATABASE_URL`, `GATEWAY_URL`, `STORE_URL`,
 `EVENTS_TOKEN`, `STORE_WRITE_TOKEN`, `RECEIPT_REGISTRY`, `AGENTS_FILE`, and `MERGIT_AGENT_KEY_<id>` per agent,
-plus the model provider key (`ANTHROPIC_API_KEY`). Everything else has a default, including `CHAIN_ID`
+plus whichever model key(s) its deployed agents' providers need (`ModelClients.build` only builds a client
+for a provider a *deployed* agent actually names, so e.g. `ANTHROPIC_API_KEY` is not required when every
+deployed agent is `openai_chat`). Everything else has a default, including `CHAIN_ID`
 (84532), `SANDBOX_IMAGE`, `SANDBOX_RUNTIME`, `SANDBOX_MEMORY` (2g), `SANDBOX_NANO_CPUS` (2 CPUs),
 `MAX_STEPS` (60), `MAX_OUTPUT_BYTES` (1 MiB), `COMMAND_TIMEOUT_S`, `JOB_DEADLINE_S`, `MODEL_MAX_TOKENS`
 (16000; Opus 5.5's thinking tokens count against it, so this is also a per-call ceiling the harness clamps
@@ -367,23 +370,53 @@ uv run pytest -m "docker or postgres"          # real sandbox and real Postgres
 ## Not done here
 
 Tracked by other plan tasks and issue #12 bullets: egress allowlist, receipt body encryption (K4), data
-availability checks, additional agents, evals, failure injection, and hosting (O10, in progress). gVisor is
-wired through
+availability checks, additional agents, evals, failure injection: audited, see
+`docs/failure-injection-audit.md`, and hosting (O10, in progress). gVisor is wired through
 `SANDBOX_RUNTIME` but has not been exercised in CI.
 
-## Model provider (Anthropic, with Groq as the fallback)
+## Model provider
 
-The planner and the agents call one model provider, chosen once at start-up from the environment
-(`mergit_worker/providers.py`):
+Each agent and the planner name their own model provider; there is no process-wide default and no
+fallback. `agents.toml` carries both:
 
-| Environment | Provider |
-| --- | --- |
-| `ANTHROPIC_API_KEY`, or `ANTHROPIC_AUTH_TOKEN` (with `ANTHROPIC_BASE_URL`, for a gateway such as Claude Lend) set | Anthropic (the models in `agents.toml` and `PLANNER_MODEL`) |
-| only `GROQ_API_KEY` set | Groq, through its OpenAI-compatible API; every agent and the planner run `GROQ_MODEL` (default `openai/gpt-oss-120b`) |
-| `LLM_PROVIDER=anthropic` or `groq` | forces one |
-
-The Claude model ids are swapped for the Groq model when an agent is loaded, before anything is priced, so a
-receipt names the model that really ran and prices it with the Groq entry in `prices.toml` (list prices from Groq's
-own pages). Before a real run, check the credential: `python -m mergit_cp.llm_check` (inside the api
-container) makes one plain and one tool-calling request and prints the result, never the key. The worker refuses to start with no key; the API starts and a planning request answers `502`.
-There is no per-call fallback: a receipt belongs to one model, so a run never changes model halfway.
+```toml
+[providers.openrouter]
+kind = "openai_chat"
+base_url = "https://openrouter.ai/api/v1"
+api_key_env = "OPENROUTER_API_KEY"
+
+[agents.claude-worker]
+provider = "openrouter"
+model = "anthropic/claude-sonnet-5"
+...
+```
+
+A `[providers.<name>]` table names that provider's implementation: `kind = "anthropic"` talks to the
+Anthropic SDK directly and needs nothing else (it falls back to the SDK's own `ANTHROPIC_API_KEY` /
+`ANTHROPIC_AUTH_TOKEN`); `kind = "openai_chat"` speaks an OpenAI-compatible chat API (Groq, OpenRouter, or
+anything else shaped like one) and needs `base_url` plus `api_key_env`, the environment variable holding
+its key. Each `[agents.<name>]` table then names which provider it uses by `provider`, and the planner
+names its own by `Settings.planner_provider` (`PLANNER_PROVIDER`) — a deployment choice, same as
+`PLANNER_MODEL`, not derived from which keys happen to be set.
+
+The provider key goes to `base_url` as a Bearer token, and `agents.toml` ships with every push to main, so
+`load_providers` refuses a provider that could send a key somewhere new. `base_url` must be https on the
+default port to a host in `_PROVIDER_HOSTS` (`api.groq.com`, `openrouter.ai`), or exactly
+`http://keyrelay:8787` (the compose-internal key rotator, deploy/server/README.md); a URL with credentials
+in it is refused. `api_key_env` must be `<PROVIDER>_API_KEY` in capitals and not start with `MERGIT_`, so it
+cannot name one of Mergit's own secrets (`*_TOKEN`, `*_SECRET`, `*_KEY`, `DATABASE_URL`). An `anthropic`
+provider may not set `base_url`. Adding a provider host is a change to `_PROVIDER_HOSTS`, reviewed as code.
+
+`load_providers` (`mergit_worker/providers.py`) reads every `[providers.*]` table once at start-up; the
+worker builds one client per provider actually used by a *deployed* agent (`ModelClients.build`), so an
+unused provider's key is never required. A run never changes provider or model halfway: the choice is made
+once, at start-up, and a receipt names the model that produced it, priced from the matching entry in
+`prices.toml`.
+
+Before a real run, check the credentials where they live. Each container holds only its own keys, so each
+checks its own: `python -m mergit_cp.llm_check planner` in the api container (the planner's provider and
+model) and `python -m mergit_cp.llm_check agents` in the worker container (every deployed agent's), or
+`all` from a shell that has every key. Each makes one plain and one tool-calling request per distinct
+provider and model, printing the result and never the key. The worker refuses to start if a deployed
+agent's provider has no key; the API starts and a planning request answers `502`. A deploy checks the keys
+are present (not that they work) before it swaps containers: deploy/server/README.md, "Upgrading".

control-plane/agents.toml

@@ -1,25 +1,46 @@
-# Model and system prompt per agent, keyed by the `name` in deployments/agents.json.
+# Which model provider implementation each agent below names by `provider`. `kind = "anthropic"` uses the
+# Anthropic SDK directly; `kind = "openai_chat"` speaks Groq/OpenRouter's OpenAI-compatible chat API, read
+# from `base_url` with the key in the environment variable named by `api_key_env`.
+[providers.anthropic]
+kind = "anthropic"
+
+[providers.groq]
+kind = "openai_chat"
+base_url = "https://api.groq.com/openai/v1"
+api_key_env = "GROQ_API_KEY"
+
+[providers.openrouter]
+kind = "openai_chat"
+base_url = "https://openrouter.ai/api/v1"
+api_key_env = "OPENROUTER_API_KEY"
+
+# Provider, model and system prompt per agent, keyed by the `name` in deployments/agents.json.
 # Models are provider model ids; the worker reads this file, nothing is hardcoded in code.
 
 [agents.claude-worker]
-model = "claude-sonnet-5"
+provider = "openrouter"
+model = "anthropic/claude-sonnet-5"
 system_prompt = """\
 You are a software engineer working in a sandboxed checkout of a repository.
 Use the tools to inspect files, edit them and run commands. Make the smallest change that
 satisfies the task instructions and keep to the paths the task allows.
 Call `finish` when you believe the work is complete. A separate check decides whether it is."""
 
 [agents.second-worker]
-model = "claude-opus-5-5"
+provider = "openrouter"
+model = "deepseek/deepseek-v4-flash"
 system_prompt = """\
 You are a careful software engineer working in a sandboxed checkout of a repository.
 Read the relevant code before changing it, run the tests you can, and keep changes minimal.
 Call `finish` when you believe the work is complete. A separate check decides whether it is."""
 
-# The agent seeded on Base Sepolia by mergit-contracts' SeedAgents (AGENT_NAME_0). With no Anthropic key the model
-# below is replaced by GROQ_MODEL when the agent is loaded (mergit_worker/providers.py).
+# The agent seeded on Base Sepolia by mergit-contracts' SeedAgents (AGENT_NAME_0).
 [agents.mergit-coder-0]
-model = "claude-sonnet-5"
+# On the hosted deployment this agent runs Claude Sonnet 5 through OpenRouter. To run the same model on
+# Anthropic's API directly instead (an ANTHROPIC_API_KEY on the worker), use provider = "anthropic" and
+# model = "claude-sonnet-5"; both ids are priced in prices.toml.
+provider = "openrouter"
+model = "anthropic/claude-sonnet-5"
 system_prompt = """\
 You are a software engineer working in a sandboxed checkout of a repository.
 Use the tools to inspect files, edit them and run commands. Make the smallest change that

control-plane/docs/failure-injection-audit.md

@@ -0,0 +1,26 @@
+# Failure-injection audit
+
+**Source:** HLD decisions doc, "M5 · wk 8-9 · Hardening for production": "Failure-injection suite:
+fabricated success, empty diff, wrong file, budget overrun, step tampering, cache replay. Each must
+be caught." Plus issue #28's "multi-worker load test with no lost or double-run tasks."
+
+This records what the code and tests already guarantee for each named scenario. Evidence is cited by
+file and test name, not line numbers, so it stays true as the files change: `pytest
+'tests/test_harness.py::test_dropped_step_is_rejected'` runs a cited test on its own. Paths without a
+`control-plane/` prefix are relative to the repo root (the Rust crates and the `mergit_receipt` type
+stub live outside this subtree); all other paths are relative to `control-plane/`.
+
+| HLD scenario | Status | Evidence |
+|---|---|---|
+| Fabricated success (agent claims done, nothing actually passed) | **Covered, structurally and by test** | The `finish` tool in `mergit_worker/tools.py` takes only a `summary` string: there is no field an agent could set to claim success, and only Φ decides. `tests/test_harness.py::test_finish_alone_does_not_complete_the_task` drives this through the real pipeline: the agent calls `finish` and `outcome.holds is False`. |
+| Wrong file (agent edits outside the task's paths, the check still exits 0) | **Covered** | `tests/test_phi.py::test_change_outside_allowed_paths_fails_even_when_check_passes` writes the allowed fix plus an out-of-scope file; the command exits 0 and `holds is False`. |
+| Empty diff passing | **Accepted by design, not a defect** | The pull-request gate in `README.md` (plan task O9) does not treat an empty patch, a task that passed without changing a tracked file, as a blocker, and skips it when applying a plan's combined diff because `git apply` refuses an empty patch. A task can pass Φ without a change (a check that was already satisfied), and failing it would make the gate stricter than the predicate it trusts. |
+| Budget overrun | **Covered, including the one gap this audit found** | `tests/test_harness.py::test_budget_exhausted_still_seals_with_phi` shows sealing still happens and partial, useful work still passes Φ. `tests/test_harness.py::test_budget_too_small_for_one_token_stops_before_calling` stops the loop with no model call. Nothing had checked that running out of budget with no completed work ends in a recorded failure rather than a silent pass, so `tests/test_harness.py::test_budget_exhausted_with_no_completed_work_fails` was added: with `budget_usdc_units=1` it asserts no model call, `outcome.notes == ["cost_budget"]` and `outcome.holds is False`. It passed against the unmodified harness, so this is new coverage of an unexercised branch, not a bug fix. |
+| Step tampering (a receipt's hash chain or signature altered after sealing) | **Covered in the Rust crates, and from Python** | The vectors in `spec/vectors/` (`bad_signature.json`, `chain_mismatch.json`, `body_hash_mismatch.json`, `commitment_mismatch.json` and others) are consumed by `verifier/tests/vectors.rs` and `spec/tests/attestation.rs`. `mergit_receipt.verify()` is exposed to Python in `runtime/py/mergit_receipt.pyi`. Three tests in `tests/test_harness.py` mutate a copy of a real sealed bundle the way an attacker would and confirm the control plane's own `verify()` call rejects it: `test_tampered_step_output_is_rejected` (one character of a step's `output_digest`), `test_reordered_steps_are_rejected` (two steps swapped) and `test_dropped_step_is_rejected` (a step removed; CLAUDE.md invariant 1, the complete trajectory). Each asserts the status is not `"unattested_valid"` and `reasons` is non-empty, and `test_fake_agent_fixes_repo_and_phi_holds` first shows the untampered bundle verifies. |
+| Cache replay (a stale cached tool or model response presented as fresh) | **Not buildable yet: the feature it would guard does not exist** | The only cache fields are `cache_read_tokens` and `cache_write_tokens`, for billing (`mergit_worker/meter.py`, `LLMResult` in `mergit_cp/llm.py`): they count tokens a provider served from its own cache, with no notion of freshness or replay. There is no cache-provenance field on a receipt step, no cache store, and nothing a step could replay. A guard needs a receipt-schema decision first (mergit-research), like TEE attestation. |
+| No lost or double-run tasks under concurrent workers (issue #28) | **Covered** | `tests/test_jobs.py::test_concurrent_workers_never_claim_the_same_job` runs 6 worker threads against `SELECT ... FOR UPDATE SKIP LOCKED` on a real Postgres and checks that 30 jobs are each claimed exactly once. |
+
+**Summary:** five of seven items were already guarded and tested. Empty diff is accepted by design, and
+cache replay cannot be built without a schema decision. The one real gap, a budget overrun with no
+completed work, is closed by a new test, and step tampering now has Python-side tests as well as the
+Rust vector suite.

control-plane/mergit_cp/agents.py

@@ -7,7 +7,7 @@
 
 from pydantic import BaseModel, ConfigDict, Field
 
-from mergit_worker.providers import effective_model
+from mergit_worker.providers import load_providers
 
 _ADDRESS = r"^0x[0-9a-fA-F]{40}$"
 
@@ -18,6 +18,7 @@ class Agent:
     name: str
     operator: str
     runtime_key: str
+    provider: str
     model: str
     system_prompt: str
 
@@ -38,6 +39,7 @@ class _Deployment(BaseModel):
 class _ModelConfig(BaseModel):
     model_config = ConfigDict(extra="forbid")
 
+    provider: str = Field(min_length=1)
     model: str = Field(min_length=1)
     system_prompt: str = Field(min_length=1)
 
@@ -46,7 +48,8 @@ def load_agents(path: str, config_path: str = "agents.toml") -> list[Agent]:
     """Join the deployed agents (``agents.json``) with their model config (``[agents.<name>]`` in
     ``agents.toml``, the same file the worker reads).
 
-    Raises ValueError when an agent has no config or an id appears twice, so a bad deploy fails at start.
+    Raises ValueError when an agent has no config, names a provider ``agents.toml`` does not define, or
+    an id appears twice, so a bad deploy fails at start.
     """
     deployed = _Deployment.model_validate(json.loads(Path(path).read_text())).agents
     tables = tomllib.loads(Path(config_path).read_text()).get("agents")
@@ -56,6 +59,7 @@ def load_agents(path: str, config_path: str = "agents.toml") -> list[Agent]:
         str(name): _ModelConfig.model_validate(table)  # pyright: ignore[reportUnknownArgumentType]
         for name, table in tables.items()  # pyright: ignore[reportUnknownVariableType]
     }
+    providers = load_providers(Path(config_path))
     seen: set[str] = set()
     agents: list[Agent] = []
     for chain_agent in deployed:
@@ -65,13 +69,19 @@ def load_agents(path: str, config_path: str = "agents.toml") -> list[Agent]:
         config = configs.get(chain_agent.name)
         if config is None:
             raise ValueError(f"agent {chain_agent.name!r} has no entry in {config_path}")
+        if config.provider not in providers:
+            raise ValueError(
+                f"agent {chain_agent.name!r} names provider {config.provider!r}, which has no "
+                f"[providers.{config.provider}] table in {config_path}"
+            )
         agents.append(
             Agent(
                 agent_id=chain_agent.agent_id,
                 name=chain_agent.name,
                 operator=chain_agent.operator.lower(),
                 runtime_key=chain_agent.runtime_key.lower(),
-                model=effective_model(config.model),
+                provider=config.provider,
+                model=config.model,
                 system_prompt=config.system_prompt,
             )
         )

control-plane/mergit_cp/app.py

@@ -2,15 +2,13 @@
 hosts build it the same way."""
 
 import asyncio
+import logging
 from collections.abc import AsyncGenerator
 from contextlib import asynccontextmanager, suppress
-from pathlib import Path
 
 from fastapi import FastAPI
 from fastapi.middleware.cors import CORSMiddleware
 
-from mergit_worker.meter import PriceTable
-
 from . import (
     auth,
     chain_events,
@@ -25,18 +23,19 @@
     routes_runs,
     wallet_interest,
 )
-from .agents import load_agents
 from .config import Settings
 from .db import migrate, open_pool
 from .events import EventDeliverer
-from .llm import LLM, build_llm
+from .llm import LLM
 from .outcomes import OutcomeProcessor
 from .planner import PgCostSink
 from .pr_store import PgPrStore
+from .preflight import check_config, planner_llm
 from .reputation import GatewayCosts, ReputationUpdater
-from .spec import require_schemas
 from .stream import PlanBroadcaster
 
+log = logging.getLogger(__name__)
+
 
 def create_app(
     settings: Settings | None = None,
@@ -53,19 +52,19 @@ def create_app(
 
     @asynccontextmanager
     async def lifespan(app: FastAPI) -> AsyncGenerator[None]:
-        # Fail at start, not on the first request, when configuration is wrong.
-        app.state.agents = load_agents(settings.agents_file, settings.agents_config)
-        require_schemas()  # the plan checks and events are validated against spec/schemas
-        prices = PriceTable.load(Path(settings.prices_file))
-        for model in settings.model_choices:  # every planner model a user may choose must have a price
-            prices.model_call_cost(model, 0, 0, 0)
+        # Fail at start, not on the first request, when configuration is wrong. The deploy runs the same
+        # checks first (mergit_cp/preflight.py), before it swaps the running container.
+        checked = check_config(settings)
+        app.state.agents, prices = checked.agents, checked.prices
+        if checked.stale:
+            log.warning("no longer read, remove from the env file: %s", ", ".join(checked.stale))
         pool = await open_pool(settings.database_url, settings.db_pool_max)
         async with pool.connection() as conn:
             await migrate(conn)
         app.state.pool = pool
         # The pull-request routes share the worker's queue table; the API holds no GitHub token (see README).
         app.state.pr_store = PgPrStore.connect(settings.database_url, "api")
-        app.state.llm = llm or build_llm()
+        app.state.llm = llm or planner_llm(settings)
         app.state.costs = PgCostSink(pool, prices)
         app.state.broadcaster = PlanBroadcaster()
         app.state.mailer = mailer or (otp.SmtpMailer(settings) if settings.email_enabled else None)