Multi-key model support: deterministic discovery across all capable keys and model-aware key resolution at request time #1458

Problem

Each model scanner discovers models with one key that the token manager picks in rotation:

• apps/model-scanners/copilot uses acquireToken("copilot-models"), then calls GET https://api.githubcopilot.com/models.
• apps/model-scanners/anthropic uses acquireToken("anthropic-api").

Keys with the same capability don't all see the same catalog. Catalogs differ by account, plan, org policy and credential type, and two keys can each see models the other lacks.

Consequences today:

• The discovered catalog is nondeterministic. It depends on which key the rotation hands to the scanner on that run. Models can appear and disappear between scans.
• Models only some keys can use are missed when the scanner gets a different key.
• Runs fail late. Workers also acquire keys in rotation, so a run can pick a model the discovered catalog lists, get a key that can't use it, and fail inside the worker ("Model X is not available…") instead of at submission.

Requirement: discovery must be deterministic

The discovered catalog must depend only on the set of valid keys in the token manager, never on rotation order or which key a scan happens to get. Every model scanner gets all keys that have the capability it needs (copilot-models, anthropic-api, and any future scanner's capability) and scans each one.

Requirement: request processing resolves a key that supports the requested model

At request processing time, every key acquisition that serves a model must pick a key whose last scan included that model. Rotation only applies among those keys. Today all of these choose by capability alone:

Rules:

• acquire({ capability, model }) returns only keys that have capability and list model in the discovered availability. Availability is recorded per key id, so it works across capabilities: a key scanned under copilot-models can be picked under copilot-cli or copilot-sdk.
• A key with the capability but no scan data for the model's provider is not eligible when a model is given. This keeps resolution deterministic. Scanners should cover every key that can serve the provider's models, not just keys tagged with the scanner's capability.
• If no key supports the model, return 404 with a clear error (No valid <capability> key supports model '<model>'). The worker fails fast, and the API catches it earlier at submission (below).
• The local env-var override in TokenManagerClient (e.g. GITHUB_TOKEN) still wins for local development, unchanged.

Proposal

Applies to all model scanners. They share the logic for getting and merging keys (e.g. in packages/model-scanning).

1. Token manager: get all keys for a capability. Add a service-only endpoint, e.g. POST /api/v1/keys/acquire-all { capability }. It returns { keyId, keyType, value } for every enabled, valid, non-deleted key with that capability. GET /api/v1/keys?capability= already lists the metadata, but not the values. It should not bump the rotation or acquireCount.
2. Scanners: scan every key. Each scanner gets all keys for its capability. Call /models once per key, with limited concurrency. If a key fails, keep that key's last-known catalog rather than dropping its models. Merge results by model id.
3. Data model / POST /api/v1/models/sync: store the key ids that can use each model (e.g. availableKeyIds), plus per-key metadata such as model_picker_enabled. A model counts as available if at least one valid key has it. Add a migration if the schema changes.
4. Token manager and callers: model-aware acquire (see the requirement above). Add model to AcquireKeyRequest and TokenManagerClient.acquireToken(capability, { model }), and pass it from every call site in the table.
5. Submission-time validation (supersedes Enforce model availability server-side (API/CLI can still submit unavailable models) #1107). The API rejects a run, profile or profile version when no valid key for the worker's capability can use the requested model, and the error names the model and capability. This covers every path Enforce model availability server-side (API/CLI can still submit unavailable models) #1107 listed: normal submit, profile-variation submit and bulk resubmit in apps/api/src/routes/requests/index.ts; profile create and new version in apps/api/src/routes/profiles.ts / apps/api/src/utils/agent-helpers.ts. In the Portal, canSubmit in SubmitRun.tsx and NewProfileVersion.tsx also blocks a pre-filled value (restored draft, profile default, or a model that lost its keys after selection) and shows an inline message. Enforce model availability server-side (API/CLI can still submit unavailable models) #1107 was unsure whether to hard-reject, because availability was one global flag. With per-key availability, a hard reject is safe.
6. Extensible key types. Each key type defines how it authenticates the /models call. User tokens send Authorization: Bearer. New key types can be added without changing the scanner logic. Validation and capability derivation for those types are pluggable the same way.
7. Portal and CLI parity. Show which keys (or how many) can use each model in the Portal model views and in the CLI (e.g. scope models list).

Acceptance criteria

• With a single key, behavior is unchanged.
• Every model scanner (Copilot and Anthropic) gets all valid keys for its capability and scans each one.
• Discovery is deterministic: two scans with the same set of keys produce the same catalog, whatever the rotation order. A test covers this.
• Each model records which keys can use it. A key failing to scan doesn't remove models other keys can still use.
• acquire with a model only returns keys that have the capability and support that model, whatever the capability.
• Every call site in the table passes its model. A run, judge or report never gets a key that can't use its model.
• A run, profile or profile version whose model no valid key can use is rejected with a clear error, on every submit path (normal, profile variation, bulk resubmit, profile create, new version).
• Portal canSubmit blocks a pre-filled model that no key can use, with an inline message.
• Portal and CLI show per-model key availability.
• Tests cover acquire-all, the per-key merge, model-aware acquire and submission validation.
• Docs updated: docs/architecture/token-manager.md, docs/architecture/app-design.md (model data model), model scanning docs, and ENV_VARIABLES.md if config is added.

Related

• Supersedes Enforce model availability server-side (API/CLI can still submit unavailable models) #1107 (Enforce model availability server-side). This issue implements its option 3, availability scoped to each credential, and takes over its submission and Portal enforcement scope (see Proposal item 5).
• [Archived PR] feat(models): show auth-unavailable models as disabled instead of dropping them #1106 — Adds a single available / unavailableReason flag per model. With per-key availability, that flag becomes derived: a model is available if at least one enabled key with the capability supports it.