diff --git a/docs-site/src/content/docs/fr/reference/configuration/providers.md b/docs-site/src/content/docs/fr/reference/configuration/providers.md index 442c597f0db..aee5bf64f31 100644 --- a/docs-site/src/content/docs/fr/reference/configuration/providers.md +++ b/docs-site/src/content/docs/fr/reference/configuration/providers.md @@ -95,7 +95,9 @@ sauvegarde dont le contenu diffère, puis réécrit en identifiants sans préfix | --- | --- | --- | | `adapter` | `string` | L'un des `openai-chat`, `openai-responses`, `anthropic`, `google`, `kiro`, `cursor`, `ollama-native`, `azure-openai` (ou alias `azure`). | | `baseUrl` | `string` | URL de base de l'API en amont. La plupart des points de terminaison fixes intégrés ignorent une valeur incompatible ; les préréglages de clés protégés contre les collisions préservent une ancienne destination personnalisée portant le même nom. | -| `requestPacing?` | `{ enabled, requestsPerMinute?, minIntervalMs?, maxConcurrentRequests?, models? }` | Cadencement facultatif du démarrage des requêtes sortantes côté client, distinct de l’utilisation, de la facturation et des indicateurs de limitation en amont. `maxConcurrentRequests` est une limite entière positive des requêtes en cours ; une règle fournisseur ou modèle peut utiliser uniquement cette limite. Les limites du fournisseur s’appliquent à tous ses modèles, tandis que les entrées `models` ciblent les identifiants exacts des modèles en amont et peuvent ajouter du délai ou réduire la concurrence. L’attente dans la file ne consomme pas le délai d’expiration des en-têtes de réponse en amont. Les requêtes HTTP et les distributions explicites `fetchResponse`/`runTurn` des adaptateurs sont couvertes. Avec une limite de concurrence, un tour Responses WebSocket canonique utilise HTTP/SSE afin de libérer la capacité à la fin, en cas d’erreur ou d’annulation du corps de réponse. Pour les adaptateurs `runTurn`, dont Cursor, la limite compte les tours actifs plutôt que les envois physiques : RunSSE et BidiAppend peuvent se chevaucher dans un même tour, tandis qu’un autre tour attend. Les envois suivants respectent toujours les intervalles de démarrage. | +| `requestPacing?` | `{ enabled, requestsPerMinute?, minIntervalMs?, maxConcurrentRequests?, models?, keySelection?, keys? }` | Cadencement facultatif du démarrage des requêtes sortantes côté client, distinct de l’utilisation, de la facturation et des indicateurs de limitation en amont. `maxConcurrentRequests` est une limite entière positive des requêtes en cours ; une règle fournisseur ou modèle peut utiliser uniquement cette limite. Les limites du fournisseur s’appliquent à tous ses modèles, tandis que les entrées `models` ciblent les identifiants exacts des modèles en amont et peuvent ajouter du délai ou réduire la concurrence. L’attente dans la file ne consomme pas le délai d’expiration des en-têtes de réponse en amont. Les requêtes HTTP et les distributions explicites `fetchResponse`/`runTurn` des adaptateurs sont couvertes. Avec une limite de concurrence, un tour Responses WebSocket canonique utilise HTTP/SSE afin de libérer la capacité à la fin, en cas d’erreur ou d’annulation du corps de réponse. Pour les adaptateurs `runTurn`, dont Cursor, la limite compte les tours actifs plutôt que les envois physiques : RunSSE et BidiAppend peuvent se chevaucher dans un même tour, tandis qu’un autre tour attend. Les envois suivants respectent toujours les intervalles de démarrage. | +| `requestPacing.keySelection?` | `"active"`, `"round-robin"` | Sélection des clés pour les requêtes normales : `active` par défaut conserve la clé enregistrée et la rotation sur erreur ; `round-robin` répartit les allocations entre les clés disponibles sans modifier la sélection enregistrée. Cela équilibre les allocations, pas les jetons ni les coûts. | +| `requestPacing.keys?` | `{ requestsPerMinute?, minIntervalMs?, maxConcurrentRequests? }` | Limites uniformes par référence de clé : RPM, intervalle minimal et concurrence, partagées entre modèles et cumulées avec les plafonds fournisseur et modèle. Les références identiques partagent leur capacité ; plusieurs références du même projet ne garantissent pas des quotas indépendants. TPM et quotas quotidiens ne sont pas gérés. | | `responsesPath?` | `string` | Chemin de ressource relatif pour les requêtes d'authentification par clé `openai-responses`. Il doit commencer par `/` et ne contenir aucun schéma, requête ou fragment. | | `chatCompletionsPath?` | `string` | Chemin de ressource relatif pour les requêtes `openai-chat`, miroir de `responsesPath` et soumis aux mêmes règles de forme. Nécessaire lorsqu'un même service en amont sert Chat Completions et Responses sous des préfixes différents : un override wire par modèle change l'adaptateur sans toucher `baseUrl`, donc sans ce réglage une requête Chat activée serait envoyée vers la base Responses. L'exemple fourni est Z.AI. | | `upstreamWebsocket?` | `boolean` | Active le transport Responses WebSocket en amont pour les requêtes `openai-responses` (désactivé par défaut). N'est honoré que pour l'amont first-party `https://api.openai.com/v1` ; les points de terminaison des fournisseurs personnalisés utilisent toujours HTTP/SSE borné, car Bun ne peut pas appliquer de limite de taille aux messages WebSocket entrants avant d'avoir alloué le message complet. Pour le fournisseur canonique ChatGPT `openai`, l'omettre conserve le WebSocket en amont sur les tours éligibles, `false` envoie les tours en streaming via HTTP/SSE, et `true` est refusé ; avec `false`, le pilotage et l'injection natifs en cours de tour sont indisponibles. Ce champ est indépendant du réglage `websockets` côté client et ne change ni le point de terminaison ni les identifiants. Une base HTTP reste en SSE ; les chemins qui ne sont pas Responses et les requêtes `openai-chat` restent en HTTP. | diff --git a/docs-site/src/content/docs/guides/providers.md b/docs-site/src/content/docs/guides/providers.md index 87f7a28b3e6..654eff4041a 100644 --- a/docs-site/src/content/docs/guides/providers.md +++ b/docs-site/src/content/docs/guides/providers.md @@ -93,6 +93,8 @@ credentials must never be replayed on the same token, and local runtimes have no preserve. It is opt-in: when the option is absent the feature is off; object presence enables it unless `enabled: false`. +When round-robin key selection is enabled, queued requests recheck key cooldowns before sending and acquire pacing capacity for any replacement key. This applies to native Chat, Messages, Responses and compact. Waiting for replacement-key capacity does not consume the upstream response-header timeout. Per-key pacing and round-robin use guarded HTTP/SSE even when upstream WebSockets are enabled; WebSocket-only steering and injection require a route without those key policies. This does not change the saved active key. An established upstream turn keeps its credential; later sends are refused if that key cools. Pre-output key rejection can recover to a healthy key within the existing request limit. Direct Images and compact requests record genuine 401/429 cooldowns without replaying that paid POST. + ### Anthropic API-key credit exhaustion Anthropic API-key pools with at least two keys automatically try a sibling key when a complete HTTP `400` or `402` error states that the credit balance is too low to access the Anthropic API. This applies to `invalid_request_error`, `billing_error`, and `permission_error` before any text or tool output. Key rotation and pool strategies skip the failed key for one hour, because an empty balance only clears when credits are added; when every key is out of credit, requests still use the active key and return Anthropic's original error; choosing the key as active in the dashboard, or restarting the proxy, clears that hold after a top-up. Replacement sends share the request's send limit. Ordinary `400` errors stay terminal, and this does not enable same-key retries or change OAuth account routing. A header override that sends a different credential cannot cool or rotate the selected pool key. diff --git a/docs-site/src/content/docs/ja/reference/configuration/providers.md b/docs-site/src/content/docs/ja/reference/configuration/providers.md index 23e2efd7611..e935054b7a5 100644 --- a/docs-site/src/content/docs/ja/reference/configuration/providers.md +++ b/docs-site/src/content/docs/ja/reference/configuration/providers.md @@ -89,7 +89,9 @@ account を削除しても mapping は保持され、同じ id を再追加す | --- | --- | --- | | `adapter` | `string` | `openai-chat`、`openai-responses`、`anthropic`、`google`、`kiro`、`cursor`、`ollama-native`、`azure-openai` (または別名 `azure`) のいずれか。 | | `baseUrl` | `string` |アップストリーム API のベース URL。ほとんどの組み込み固定エンドポイントは不一致を無視します。衝突安全キー プリセットは、古い同じ名前のカスタム宛先を保持します。 | -| `requestPacing?` | `{ enabled, requestsPerMinute?, minIntervalMs?, maxConcurrentRequests?, models? }` | 上流の使用量、請求、レート制限表示とは別の、クライアント側の送信開始間隔調整です。`maxConcurrentRequests` は実行中リクエスト数を制限する正の整数で、プロバイダー規則またはモデル規則だけに設定できます。プロバイダー制限は全モデルに適用され、`models` は上流の正確なモデル ID に一致し、遅延を増やすか同時実行数を減らします。キュー待機は応答ヘッダーのタイムアウトを消費しません。HTTP と明示的なアダプターの `fetchResponse`/`runTurn` 送信を対象にします。同時実行数の上限がある場合、標準の Responses WebSocket ターンは応答本文の完了・エラー・キャンセル時に枠を解放できるよう HTTP/SSE を使います。 Cursor を含む `runTurn` アダプターでは、上限は物理送信数ではなく進行中のターン数を数えます。同じターンの RunSSE と BidiAppend は重なり得ますが、別のターンは待機します。後続送信にも開始間隔は適用されます。 | +| `requestPacing?` | `{ enabled, requestsPerMinute?, minIntervalMs?, maxConcurrentRequests?, models?, keySelection?, keys? }` | 上流の使用量、請求、レート制限表示とは別の、クライアント側の送信開始間隔調整です。`maxConcurrentRequests` は実行中リクエスト数を制限する正の整数で、プロバイダー規則またはモデル規則だけに設定できます。プロバイダー制限は全モデルに適用され、`models` は上流の正確なモデル ID に一致し、遅延を増やすか同時実行数を減らします。キュー待機は応答ヘッダーのタイムアウトを消費しません。HTTP と明示的なアダプターの `fetchResponse`/`runTurn` 送信を対象にします。同時実行数の上限がある場合、標準の Responses WebSocket ターンは応答本文の完了・エラー・キャンセル時に枠を解放できるよう HTTP/SSE を使います。 Cursor を含む `runTurn` アダプターでは、上限は物理送信数ではなく進行中のターン数を数えます。同じターンの RunSSE と BidiAppend は重なり得ますが、別のターンは待機します。後続送信にも開始間隔は適用されます。 | +| `requestPacing.keySelection?` | `"active"`, `"round-robin"` | 通常リクエストのキー選択です。既定の `active` は保存済みキーと既存の障害時切替を維持します。`round-robin` は保存済み選択を変えず、利用可能なプールのキーを順に割り当てます。トークンや料金ではなく割当回数を分散します。 | +| `requestPacing.keys?` | `{ requestsPerMinute?, minIntervalMs?, maxConcurrentRequests? }` | キー参照ごとの共通 RPM・開始間隔・同時実行上限です。全モデルで共有し、プロバイダーとモデルの上限も同時に適用します。同一参照は枠を共有しますが、同じ上流プロジェクトの別参照は独立クォータを意味しません。TPM と日次上限は管理しません。 | | `responsesPath?` | `string` |キー認証 `openai-responses` リクエストの相対リソース パス。 `/` で始まり、スキーム、クエリ、またはフラグメントが含まれていない必要があります。 | | `chatCompletionsPath?` | `string` | `openai-chat` リクエストの相対リソース パス。 `responsesPath` の対となる設定で、同じ形式ルールが適用されます。1つのアップストリームが Chat Completions と Responses を異なるプレフィックスで提供する場合に必要です。モデルごとの wire override はアダプターのみを変更し `baseUrl` は変更しないため、この設定がないと有効化された Chat リクエストが Responses ベースへ送信されます。同梱例は Z.AI です。 | | `upstreamWebsocket?` | `boolean` | `openai-responses` リクエストで使用するアップストリーム Responses WebSocket トランスポート（既定値は無効）。ファーストパーティの `https://api.openai.com/v1` アップストリームでのみ有効です。カスタムプロバイダーのエンドポイントは常に制限付き HTTP/SSE を使用します。Bun はメッセージ全体を確保する前に受信 WebSocket メッセージのサイズ上限を適用できないためです。正規の ChatGPT `openai` プロバイダーでは、省略すると対象となるターンでアップストリーム WebSocket を使用し、`false` はストリーミングのターンを HTTP/SSE で送信し、`true` は拒否されます。`false` の間はネイティブのターン途中のステアリングとインジェクションを利用できません。このフィールドはクライアント側の `websockets` 設定とは独立しており、エンドポイントと認証情報のどちらも変更しません。HTTP のベース URL は SSE のままとなり、Responses 以外のパスと `openai-chat` リクエストは HTTP を使用します。 | diff --git a/docs-site/src/content/docs/ko/reference/configuration/providers.md b/docs-site/src/content/docs/ko/reference/configuration/providers.md index 6ece10e3718..8ac4d029818 100644 --- a/docs-site/src/content/docs/ko/reference/configuration/providers.md +++ b/docs-site/src/content/docs/ko/reference/configuration/providers.md @@ -89,7 +89,9 @@ managed map을 활성화하면 privacy-safe selector를 만들고, 이후 계정 | --- | --- | --- | | `adapter` | `string` | `openai-chat`, `openai-responses`, `anthropic`, `google`, `kiro`, `cursor`, `ollama-native`, `azure-openai` 중 하나이며, `azure`는 별칭입니다. | | `baseUrl` | `string` | 상위 API 기본 URL입니다. 대부분의 내장 고정 엔드포인트는 불일치를 무시합니다. 충돌 안전 키 프리셋은 같은 이름의 이전 사용자 지정 목적지를 보존합니다. | -| `requestPacing?` | `{ enabled, requestsPerMinute?, minIntervalMs?, maxConcurrentRequests?, models? }` | 업스트림 사용량, 과금, rate-limit 지표와 별개인 선택적 클라이언트 측 아웃바운드 요청 시작 속도 조절입니다. `maxConcurrentRequests`는 진행 중 요청 수를 제한하는 양의 정수이며 provider 또는 model 규칙에서 단독으로 사용할 수 있습니다. Provider 제한은 모든 모델에 적용되고 `models` 항목은 정확한 업스트림 모델 ID와 일치하며 지연을 늘리거나 동시성을 줄일 수 있습니다. 큐 대기는 응답 헤더 타임아웃을 소모하지 않습니다. HTTP 및 명시적 어댑터 `fetchResponse`/`runTurn` 전송을 포함합니다. 동시성 제한이 있으면 표준 Responses WebSocket 턴은 응답 본문의 완료·오류·취소 때 점유 슬롯을 해제할 수 있도록 HTTP/SSE를 사용합니다. Cursor를 포함한 `runTurn` 어댑터에서는 물리적 전송 수가 아니라 진행 중인 턴 수를 제한합니다. 같은 턴의 RunSSE와 BidiAppend는 겹칠 수 있지만 다른 턴은 대기합니다. 후속 전송에도 시작 간격은 적용됩니다. | +| `requestPacing?` | `{ enabled, requestsPerMinute?, minIntervalMs?, maxConcurrentRequests?, models?, keySelection?, keys? }` | 업스트림 사용량, 과금, rate-limit 지표와 별개인 선택적 클라이언트 측 아웃바운드 요청 시작 속도 조절입니다. `maxConcurrentRequests`는 진행 중 요청 수를 제한하는 양의 정수이며 provider 또는 model 규칙에서 단독으로 사용할 수 있습니다. Provider 제한은 모든 모델에 적용되고 `models` 항목은 정확한 업스트림 모델 ID와 일치하며 지연을 늘리거나 동시성을 줄일 수 있습니다. 큐 대기는 응답 헤더 타임아웃을 소모하지 않습니다. HTTP 및 명시적 어댑터 `fetchResponse`/`runTurn` 전송을 포함합니다. 동시성 제한이 있으면 표준 Responses WebSocket 턴은 응답 본문의 완료·오류·취소 때 점유 슬롯을 해제할 수 있도록 HTTP/SSE를 사용합니다. Cursor를 포함한 `runTurn` 어댑터에서는 물리적 전송 수가 아니라 진행 중인 턴 수를 제한합니다. 같은 턴의 RunSSE와 BidiAppend는 겹칠 수 있지만 다른 턴은 대기합니다. 후속 전송에도 시작 간격은 적용됩니다. | +| `requestPacing.keySelection?` | `"active"`, `"round-robin"` | 정상 요청의 키 선택 방식입니다. 기본 `active`는 저장된 키와 기존 오류 전환을 유지하며, `round-robin`은 저장된 활성 키를 변경하지 않고 사용 가능한 풀 키를 요청별 순서대로 배정합니다. 토큰이나 비용이 아니라 배정 횟수를 분산합니다. | +| `requestPacing.keys?` | `{ requestsPerMinute?, minIntervalMs?, maxConcurrentRequests? }` | 키 참조별 동일한 RPM·최소 시작 간격·동시 요청 제한입니다. 모든 모델에서 공유하며 프로바이더·정확한 모델 상한과 함께 적용합니다. 중복 참조는 한 슬롯을 공유하지만 같은 업스트림 프로젝트의 다른 참조가 독립 할당량을 뜻하지는 않습니다. TPM·일일 한도는 처리하지 않습니다. | | `responsesPath?` | `string` | 키 인증 `openai-responses` 요청의 상대 리소스 경로입니다. 반드시 `/`로 시작해야 하며 스킴, query, fragment를 포함하면 안 됩니다. | | `chatCompletionsPath?` | `string` | `openai-chat` 요청의 상대 리소스 경로로, `responsesPath`와 동일한 형식 규칙이 적용되는 대응 항목입니다. 하나의 업스트림이 Chat Completions와 Responses를 서로 다른 접두사로 제공할 때 필요합니다. 모델별 wire override는 어댑터만 바꾸고 `baseUrl`은 그대로 두므로, 이 설정이 없으면 옵트인된 Chat 요청이 Responses base로 전송됩니다. Z.AI가 제공되는 예시입니다. | | `upstreamWebsocket?` | `boolean` | `openai-responses` 요청에 대한 업스트림 Responses WebSocket 전송을 선택적으로 활성화합니다(기본값 `false`). 퍼스트파티 `https://api.openai.com/v1` 업스트림에서만 적용되며, 사용자 지정 공급자 엔드포인트는 항상 제한된 HTTP/SSE를 사용합니다. Bun은 전체 메시지를 할당하기 전에는 수신 WebSocket 메시지 크기 제한을 적용할 수 없기 때문입니다. 정식 ChatGPT `openai` 공급자에서는 생략하면 대상 턴에서 업스트림 WebSocket을 사용하고, `false`는 스트리밍 턴을 HTTP/SSE로 전송하며, `true`는 거부됩니다. `false`이면 네이티브 턴 중 스티어링과 주입을 사용할 수 없습니다. 이 필드는 클라이언트 측 `websockets` 설정과 독립적이며 엔드포인트와 자격 증명을 변경하지 않습니다. HTTP 기본 URL은 SSE를 유지하고, Responses가 아닌 경로와 `openai-chat` 요청은 HTTP를 사용합니다. | diff --git a/docs-site/src/content/docs/reference/configuration/providers.md b/docs-site/src/content/docs/reference/configuration/providers.md index 50c3520b0df..a844667f137 100644 --- a/docs-site/src/content/docs/reference/configuration/providers.md +++ b/docs-site/src/content/docs/reference/configuration/providers.md @@ -245,7 +245,9 @@ Providers can expose a built-in shorthand, such as `agy` for `google-antigravity | `baseUrl` | `string` | Upstream API base URL. Most built-in fixed endpoints ignore a mismatch; collision-safe key presets preserve an older same-named custom destination. | | `proxy?` | `string \| null` | Per-provider egress route. Omit it to inherit the global proxy decision; use `"direct"` or `null` to force direct egress; or provide an absolute `http://`, `https://`, `socks5://`, or `socks5h://` proxy URL. An empty string is rejected. | | `noProxy?` | `string \| string[]` | Destinations this provider reaches directly, using `NO_PROXY` host-pattern syntax. A match bypasses both this provider's own proxy and an inherited global proxy. | -| `requestPacing?` | `{ enabled, requestsPerMinute?, minIntervalMs?, maxConcurrentRequests?, models? }` | Optional client-side outbound request-start pacing, separate from upstream usage, billing, and rate-limit indicators. RPM is converted to an even interval; `minIntervalMs` may impose a longer interval. `maxConcurrentRequests` is a positive integer cap on in-flight requests. A provider or model rule may use the concurrency cap alone; provider limits apply across all models, while `models` entries use exact upstream model IDs (for example `nvidia/llama-3.1-nemotron-ultra-253b-v1`) and can only add delay or narrow concurrency. Queue waits do not consume the upstream response-header timeout. HTTP and explicit adapter `fetchResponse`/`runTurn` dispatches are covered. A concurrency-capped canonical Responses WebSocket turn uses HTTP/SSE so its lease can be released when the response body completes, errors, or is cancelled. For `runTurn` adapters, including Cursor, the cap counts active turns rather than physical sends: RunSSE and BidiAppend may overlap within one turn, while another turn waits. Follow-up sends still obey start intervals. | +| `requestPacing?` | `{ enabled, requestsPerMinute?, minIntervalMs?, maxConcurrentRequests?, models?, keySelection?, keys? }` | Optional client-side outbound request-start pacing, separate from upstream usage, billing, and rate-limit indicators. RPM is converted to an even interval; `minIntervalMs` may impose a longer interval. `maxConcurrentRequests` is a positive integer cap on in-flight requests. A provider or model rule may use the concurrency cap alone; provider limits apply across all models, while `models` entries use exact upstream model IDs (for example `nvidia/llama-3.1-nemotron-ultra-253b-v1`) and can only add delay or narrow concurrency. Queue waits do not consume the upstream response-header timeout. HTTP and explicit adapter `fetchResponse`/`runTurn` dispatches are covered. A concurrency-capped canonical Responses WebSocket turn uses HTTP/SSE so its lease can be released when the response body completes, errors, or is cancelled. For `runTurn` adapters, including Cursor, the cap counts active turns rather than physical sends: RunSSE and BidiAppend may overlap within one turn, while another turn waits. Follow-up sends still obey start intervals. | +| `requestPacing.keySelection?` | `"active"`, `"round-robin"` | Normal-request key selection. Default `active` preserves the saved key and existing failure rotation; `round-robin` distributes request-local selections across eligible API-key pool references without changing the saved active key. It balances allocations, not tokens or cost. | +| `requestPacing.keys?` | `{ requestsPerMinute?, minIntervalMs?, maxConcurrentRequests? }` | Uniform per-key-reference RPM, minimum start interval and concurrency limits, shared across models. These compose with provider and exact-model upper bounds. Duplicate references share a bucket; separate references in one upstream project do not imply independent quota. TPM and daily quotas are not enforced. | | `upstreamHttpVersion?` | `"auto" \| "http1.1" \| "h1" \| "http2" \| "h2"` | Pin the HTTP version used for upstream requests to this provider. Defaults to `auto`, which lets Bun negotiate. An explicit pin requires an HTTPS target and fails locally when it cannot be honored. Set `http1.1` when a provider's HTTP/2 SSE stream stalls instead of delivering events — the symptom is a long-running streaming request that produces nothing and eventually times out. For Cursor, `http1.1`/`h1` selects its `RunSSE` + `BidiAppend` compatibility transport for inference and also pins live model discovery. Management `POST`/`PATCH` accept `null` to clear it back to `auto`. | | `tlsProfile?` | `"antigravity-browser"` | **Use at your own risk.** Opt-in browser-like TLS handshake (through the optional `wreq-js` dependency) for the canonical `google-antigravity` OAuth provider. It changes only how the connection looks on the wire; it is not an official Google client, and it does not change what Google's terms allow. Google can still detect, rate-limit, suspend, or ban the account you signed in with, and you alone carry that risk. It is off by default and accepted only with the Google adapter, Cloud Code Assist mode, and Google's canonical HTTPS Antigravity hosts. Redirects stay manual. The provider's own `proxy`/`noProxy` route is carried by the TLS transport; a route it cannot keep fails the request instead of leaving by another path, which includes any direct route while `HTTP_PROXY`, `HTTPS_PROXY`, or `ALL_PROXY` is set. `GET /api/providers` reports the profile state as `pending`, `active`, or `failed`. Omitting the field keeps the normal Bun transport and never loads the dependency. | | `responsesPath?` | `string` | Relative resource path for key-auth `openai-responses` requests. It must start with `/` and contain no scheme, query, or fragment. | @@ -377,6 +379,23 @@ For example, this applies a provider-wide concurrency cap and a stricter cap to The concurrency slot stays occupied until the upstream request finishes, including the final streamed response bytes. Requests above the applicable provider and model limits wait in the pacing queue; when a slot is released, the next eligible request is admitted. Queue waiting does not consume the upstream response-header timeout. +For multiple API-key pool entries, opt into normal-request distribution separately from the existing failure-selection `apiKeyPoolStrategy`: + +```json +{ + "requestPacing": { + "enabled": true, + "keySelection": "round-robin", + "maxConcurrentRequests": 10, + "keys": { "requestsPerMinute": 15, "maxConcurrentRequests": 1 } + } +} +``` + +This admits at most ten active requests across the provider and one per credential reference, with starts on each reference at least four seconds apart. These are local upper bounds, not a promise of upstream capacity. A provider-wide RPM setting, when present, remains an aggregate limit and does not multiply with key count. Key limits span models; model rules remain additional restrictions. Projects sharing quota need an appropriate aggregate limit; TPM and daily quotas are not tracked here. + +Round-robin selects keys per request without saving a new active key. Known 401/429 cooldowns are skipped; when none remains eligible, new requests receive a retryable 429 instead of repeatedly sending. Already queued work normally retains its selected key and may wait behind that key even when another is idle. If that selection becomes cooled or invalid, the request selects an eligible key again and reacquires pacing admission for it; the bounded queue expires rather than waiting forever. Selection and diagnostics are process-local, reset on restart, and do not equal balanced token usage or billing. Starts count pacing admissions, including work cancelled before dispatch. OAuth and forwarded credentials keep their existing selection. The dashboard shows key handles, starts, in-flight work, queued work and known cooldowns, never secret key values. These limits cover the inference pacing executor, not the standalone Images relay's direct fetch. + ### Azure OpenAI model metadata discovery For a base URL whose hostname ends in `.openai.azure.com`, discovery fills missing image/text diff --git a/docs-site/src/content/docs/ru/reference/configuration/providers.md b/docs-site/src/content/docs/ru/reference/configuration/providers.md index 70e46f22113..a5aeba9f4fc 100644 --- a/docs-site/src/content/docs/ru/reference/configuration/providers.md +++ b/docs-site/src/content/docs/ru/reference/configuration/providers.md @@ -102,7 +102,9 @@ cross-route credential fallback не существует. Строки API GPT- | --- | --- | --- | | `adapter` | `string` | Один из `openai-chat`, `openai-responses`, `anthropic`, `google`, `kiro`, `cursor`, `ollama-native`, `azure-openai` (или alias `azure`). | | `baseUrl` | `string` | Базовый URL API upstream'а. Большинство built-in fixed-endpoint'ов игнорируют несовпадение; collision-safe key-preset'ы сохраняют старый custom destination с тем же именем. | -| `requestPacing?` | `{ enabled, requestsPerMinute?, minIntervalMs?, maxConcurrentRequests?, models? }` | Опциональное клиентское выравнивание начала исходящих запросов, отдельное от учёта использования, биллинга и индикаторов rate limit апстрима. `maxConcurrentRequests` — положительное целое, ограничивающее число запросов в работе; правило провайдера или модели может содержать только этот параметр. Лимит провайдера действует на все модели, а `models` сопоставляется с точными ID моделей апстрима и может увеличить задержку или уменьшить параллелизм. Ожидание очереди не расходует таймаут заголовков ответа. Поддерживаются HTTP и явные вызовы адаптеров `fetchResponse`/`runTurn`. При ограничении параллелизма канонический запрос Responses WebSocket использует HTTP/SSE, чтобы освободить слот после завершения, ошибки или отмены тела ответа. Для адаптеров `runTurn`, включая Cursor, лимит считает активные ходы, а не физические отправки: RunSSE и BidiAppend могут пересекаться внутри одного хода, пока следующий ход ждёт. Последующие отправки по-прежнему соблюдают интервалы запуска. | +| `requestPacing?` | `{ enabled, requestsPerMinute?, minIntervalMs?, maxConcurrentRequests?, models?, keySelection?, keys? }` | Опциональное клиентское выравнивание начала исходящих запросов, отдельное от учёта использования, биллинга и индикаторов rate limit апстрима. `maxConcurrentRequests` — положительное целое, ограничивающее число запросов в работе; правило провайдера или модели может содержать только этот параметр. Лимит провайдера действует на все модели, а `models` сопоставляется с точными ID моделей апстрима и может увеличить задержку или уменьшить параллелизм. Ожидание очереди не расходует таймаут заголовков ответа. Поддерживаются HTTP и явные вызовы адаптеров `fetchResponse`/`runTurn`. При ограничении параллелизма канонический запрос Responses WebSocket использует HTTP/SSE, чтобы освободить слот после завершения, ошибки или отмены тела ответа. Для адаптеров `runTurn`, включая Cursor, лимит считает активные ходы, а не физические отправки: RunSSE и BidiAppend могут пересекаться внутри одного хода, пока следующий ход ждёт. Последующие отправки по-прежнему соблюдают интервалы запуска. | +| `requestPacing.keySelection?` | `"active"`, `"round-robin"` | Выбор ключа для обычных запросов: `active` по умолчанию сохраняет выбранный ключ и переключение при ошибке; `round-robin` последовательно распределяет запросы между доступными ключами без изменения сохранённого выбора. Балансируются назначения, не токены или стоимость. | +| `requestPacing.keys?` | `{ requestsPerMinute?, minIntervalMs?, maxConcurrentRequests? }` | Общие для всех моделей ограничения RPM, интервала и параллелизма на ссылку ключа, вместе с лимитами провайдера и модели. Одинаковые ссылки делят ёмкость; разные ссылки одного проекта не означают независимые квоты. TPM и дневные квоты не контролируются. | | `responsesPath?` | `string` | Relative resource path для key-auth запросов `openai-responses`. Должен начинаться с `/` и не может содержать scheme, query или fragment. | | `chatCompletionsPath?` | `string` | Relative resource path для запросов `openai-chat`, зеркало `responsesPath` с теми же правилами формы. Нужен, когда один upstream обслуживает Chat Completions и Responses под разными префиксами: per-model wire override меняет адаптер и не трогает `baseUrl`, поэтому без него включённый Chat-запрос ушёл бы в Responses base. Поставляемый пример — Z.AI. | | `upstreamWebsocket?` | `boolean` | Необязательный upstream Responses WebSocket для запросов `openai-responses` (по умолчанию `false`). Учитывается только для first-party upstream `https://api.openai.com/v1`; конечные точки пользовательских провайдеров всегда используют ограниченный HTTP/SSE, поскольку Bun не может применить ограничение размера входящего сообщения WebSocket до выделения памяти под всё сообщение. У канонического провайдера ChatGPT `openai` пропуск сохраняет upstream WebSocket для подходящих ходов, `false` отправляет потоковые ходы по HTTP/SSE, а `true` отклоняется; при `false` нативное управление и внедрение в середине хода недоступны. Это поле не зависит от клиентской настройки `websockets` и не меняет ни конечную точку, ни учётные данные. Для HTTP остаётся SSE; пути, не относящиеся к Responses, и запросы `openai-chat` остаются на HTTP. | diff --git a/docs-site/src/content/docs/zh-cn/reference/configuration/providers.md b/docs-site/src/content/docs/zh-cn/reference/configuration/providers.md index 4b738b35b26..ba158ffe388 100644 --- a/docs-site/src/content/docs/zh-cn/reference/configuration/providers.md +++ b/docs-site/src/content/docs/zh-cn/reference/configuration/providers.md @@ -89,7 +89,9 @@ selector，而不是分配一个新名称。 | --- | --- | --- | | `adapter` | `string` | `openai-chat`、`openai-responses`、`anthropic`、`google`、`kiro`、`cursor`、`ollama-native`、`azure-openai`（或别名 `azure`）之一。 | | `baseUrl` | `string` | 上游 API 基础 URL。大多数内置固定端点会忽略不匹配的值；具备冲突安全键的预设会保留一个更早、同名的自定义目标。 | -| `requestPacing?` | `{ enabled, requestsPerMinute?, minIntervalMs?, maxConcurrentRequests?, models? }` | 可选的客户端出站请求启动节流，与上游用量、计费和限流指标相互独立。`maxConcurrentRequests` 是限制进行中请求数的正整数，提供商或模型规则都可以只配置此项。提供商限制适用于所有模型，`models` 按上游模型精确 ID 匹配，并可增加延迟或收紧并发限制。排队等待不计入响应头超时。覆盖 HTTP 以及显式适配器 `fetchResponse`/`runTurn` 调用。配置并发上限时，标准 Responses WebSocket 请求改用 HTTP/SSE，以便在响应体完成、出错或取消时释放并发名额。 对包括 Cursor 在内的 `runTurn` 适配器，并发上限计算进行中的回合数，而非物理发送数：同一回合内的 RunSSE 和 BidiAppend 可以重叠，其他回合仍须等待。后续发送仍遵守启动间隔。 | +| `requestPacing?` | `{ enabled, requestsPerMinute?, minIntervalMs?, maxConcurrentRequests?, models?, keySelection?, keys? }` | 可选的客户端出站请求启动节流，与上游用量、计费和限流指标相互独立。`maxConcurrentRequests` 是限制进行中请求数的正整数，提供商或模型规则都可以只配置此项。提供商限制适用于所有模型，`models` 按上游模型精确 ID 匹配，并可增加延迟或收紧并发限制。排队等待不计入响应头超时。覆盖 HTTP 以及显式适配器 `fetchResponse`/`runTurn` 调用。配置并发上限时，标准 Responses WebSocket 请求改用 HTTP/SSE，以便在响应体完成、出错或取消时释放并发名额。 对包括 Cursor 在内的 `runTurn` 适配器，并发上限计算进行中的回合数，而非物理发送数：同一回合内的 RunSSE 和 BidiAppend 可以重叠，其他回合仍须等待。后续发送仍遵守启动间隔。 | +| `requestPacing.keySelection?` | `"active"`, `"round-robin"` | 正常请求的密钥选择：默认 `active` 保留已保存密钥和现有错误切换；`round-robin` 按顺序分配可用池密钥，不修改已保存的选择。均衡的是分配次数，不是令牌或费用。 | +| `requestPacing.keys?` | `{ requestsPerMinute?, minIntervalMs?, maxConcurrentRequests? }` | 按密钥引用设置统一 RPM、最小启动间隔和并发上限，跨模型共享，并同时受提供商和模型上限约束。相同引用共享容量；同一上游项目的不同引用不代表独立配额。不管理 TPM 或每日配额。 | | `responsesPath?` | `string` | 用于 key-auth `openai-responses` 请求的相对资源路径。必须以 `/` 开头，且不能包含 scheme、query 或 fragment。 | | `chatCompletionsPath?` | `string` | 用于 `openai-chat` 请求的相对资源路径，是 `responsesPath` 的对应项，适用相同的路径规则。当同一上游以不同前缀提供 Chat Completions 和 Responses 时需要此配置：按模型的 wire override 只更换适配器而不改动 `baseUrl`，否则已启用的 Chat 请求会被发送到 Responses base。随附示例为 Z.AI。 | | `upstreamWebsocket?` | `boolean` | 为 `openai-responses` 请求选择性启用上游 Responses WebSocket 传输（默认 `false`）。仅对第一方 `https://api.openai.com/v1` 上游生效；自定义提供者端点始终使用有界 HTTP/SSE，因为 Bun 无法在分配完整消息之前对入站 WebSocket 消息实施大小限制。对于规范 ChatGPT `openai` 提供商，省略该字段会在符合条件的轮次使用上游 WebSocket，`false` 通过 HTTP/SSE 发送流式轮次，`true` 会被拒绝；设为 `false` 时，原生轮次中操控与注入不可用。该字段独立于客户端侧的 `websockets` 设置，且不改变端点或凭据。普通 HTTP 仍使用 SSE；非 Responses 路径和 `openai-chat` 请求仍使用 HTTP。 | diff --git a/docs-site/src/content/docs/zh-tw/reference/configuration/providers.md b/docs-site/src/content/docs/zh-tw/reference/configuration/providers.md index 4e805cf5d69..c6c97630311 100644 --- a/docs-site/src/content/docs/zh-tw/reference/configuration/providers.md +++ b/docs-site/src/content/docs/zh-tw/reference/configuration/providers.md @@ -69,7 +69,9 @@ ocx models provider openrouter on | --- | --- | --- | | `adapter` | `string` | `openai-chat`、`openai-responses`、`anthropic`、`google`、`kiro`、`cursor`、`ollama-native`、`azure-openai`（或別名 `azure`）之一。 | | `baseUrl` | `string` | 上游 API base URL。多數內建固定端點忽略不符；碰撞安全的金鑰預設保留較舊的同名自訂目的地。 | -| `requestPacing?` | `{ enabled, requestsPerMinute?, minIntervalMs?, maxConcurrentRequests?, models? }` | 選用的用戶端出站請求啟動節流，與上游用量、計費及限流指標彼此獨立。`maxConcurrentRequests` 是限制進行中請求數的正整數，供應商或模型規則都可單獨設定此項。供應商限制適用於所有模型，`models` 依上游模型精確 ID 比對，並可增加延遲或收緊並發限制。排隊等待不計入回應標頭逾時。涵蓋 HTTP 及明確的適配器 `fetchResponse`/`runTurn` 呼叫。設定並行上限時，標準 Responses WebSocket 請求改用 HTTP/SSE，以便在回應本文完成、出錯或取消時釋放並行名額。 對包含 Cursor 的 `runTurn` 適配器，並行上限計算進行中的回合數，而非實體傳送數：同一回合內的 RunSSE 與 BidiAppend 可重疊，其他回合仍須等待。後續傳送仍遵守啟動間隔。 | +| `requestPacing?` | `{ enabled, requestsPerMinute?, minIntervalMs?, maxConcurrentRequests?, models?, keySelection?, keys? }` | 選用的用戶端出站請求啟動節流，與上游用量、計費及限流指標彼此獨立。`maxConcurrentRequests` 是限制進行中請求數的正整數，供應商或模型規則都可單獨設定此項。供應商限制適用於所有模型，`models` 依上游模型精確 ID 比對，並可增加延遲或收緊並發限制。排隊等待不計入回應標頭逾時。涵蓋 HTTP 及明確的適配器 `fetchResponse`/`runTurn` 呼叫。設定並行上限時，標準 Responses WebSocket 請求改用 HTTP/SSE，以便在回應本文完成、出錯或取消時釋放並行名額。 對包含 Cursor 的 `runTurn` 適配器，並行上限計算進行中的回合數，而非實體傳送數：同一回合內的 RunSSE 與 BidiAppend 可重疊，其他回合仍須等待。後續傳送仍遵守啟動間隔。 | +| `requestPacing.keySelection?` | `"active"`, `"round-robin"` | 正常請求的金鑰選擇：預設 `active` 保留已儲存金鑰及既有錯誤切換；`round-robin` 依序分配可用池金鑰，不修改已儲存選擇。均衡的是分配次數，而非權杖或費用。 | +| `requestPacing.keys?` | `{ requestsPerMinute?, minIntervalMs?, maxConcurrentRequests? }` | 依金鑰參照設定統一 RPM、最小啟動間隔及並行上限，跨模型共享，並同時受供應商及模型上限約束。相同參照共享容量；同一上游專案的不同參照不代表獨立配額。不管理 TPM 或每日配額。 | | `responsesPath?` | `string` | Key-auth `openai-responses` 請求的相對資源路徑。必須以 `/` 開頭且不含 scheme、query 或 fragment。 | | `chatCompletionsPath?` | `string` | `openai-chat` 請求的相對資源路徑，為 `responsesPath` 的對應項，適用相同的路徑規則。當同一上游以不同前綴提供 Chat Completions 與 Responses 時需要此設定：按模型的 wire override 只更換適配器而不改動 `baseUrl`，否則已啟用的 Chat 請求會送往 Responses base。隨附範例為 Z.AI。 | | `upstreamWebsocket?` | `boolean` | 為 `openai-responses` 請求選用上游 Responses WebSocket 傳輸（預設 `false`）。僅對第一方 `https://api.openai.com/v1` 上游生效；自訂供應商端點一律使用有界 HTTP/SSE，因為 Bun 無法在配置完整訊息之前對傳入 WebSocket 訊息套用大小限制。對於規範 ChatGPT `openai` 供應商，省略此欄位會在符合條件的回合使用上游 WebSocket，`false` 會以 HTTP/SSE 傳送串流回合，`true` 會被拒絕；設為 `false` 時，原生回合中操控與注入無法使用。此欄位獨立於用戶端 `websockets` 設定，且不會變更端點或認證資料。一般 HTTP 仍使用 SSE；非 Responses 路徑與 `openai-chat` 請求仍使用 HTTP。 | diff --git a/gui/src/components/provider-workspace/ProviderKeyPacing.tsx b/gui/src/components/provider-workspace/ProviderKeyPacing.tsx new file mode 100644 index 00000000000..7ca4d9708b0 --- /dev/null +++ b/gui/src/components/provider-workspace/ProviderKeyPacing.tsx @@ -0,0 +1,37 @@ +import { useT } from "../../i18n/shared"; +import type { WorkspaceItem } from "./types"; + +export type KeyPacingDraft = Pick, "keySelection" | "keys">; +export type KeyPacingStatus = { id: string; starts: number; inFlight: number; queued: number; nextSlotInMs: number; cooldownUntil?: number | null }; + +export function ProviderKeyPacing({ value, onChange, status = [] }: { + value: KeyPacingDraft; onChange: (value: KeyPacingDraft) => void; status?: KeyPacingStatus[]; +}) { + const t = useT(); + const numberField = (field: "requestsPerMinute" | "minIntervalMs" | "maxConcurrentRequests", raw: string) => { + const keys = { ...value.keys }; + if (!raw.trim()) delete keys[field]; + else keys[field] = Number(raw); + onChange({ ...value, keys: Object.keys(keys).length ? keys : undefined }); + }; + return <> + +

{t("pws.keyLimits")}

+
+ {(["requestsPerMinute", "minIntervalMs", "maxConcurrentRequests"] as const).map((field, index) => )} +
+

{t("pws.keySharedQuota")}

+ {status.length > 0 && + + {status.map(row => + )}
{t("pws.keyHandle")}{t("pws.keyStarts")}{t("pws.keyInFlight")}{t("pws.pacingQueued")}{t("pws.keyWait")}
{row.id}{row.starts}{row.inFlight}{row.queued}{row.cooldownUntil ? t("pws.keyCooldown") : `${row.nextSlotInMs} ms`}
} + ; +} diff --git a/gui/src/components/provider-workspace/ProviderSettings.tsx b/gui/src/components/provider-workspace/ProviderSettings.tsx index 9856e7b90f9..4a811b5fd88 100644 --- a/gui/src/components/provider-workspace/ProviderSettings.tsx +++ b/gui/src/components/provider-workspace/ProviderSettings.tsx @@ -24,6 +24,7 @@ import { providerSupportsLiveModelDiscovery } from "../../provider-workspace/cat import type { CatalogPreset } from "../provider-catalog/provider-presets"; import { authModeLabel } from "./ProviderRail"; import { ProviderProtocolPanel } from "./ProviderProtocolPanel"; +import { ProviderKeyPacing, type KeyPacingDraft, type KeyPacingStatus } from "./ProviderKeyPacing"; import type { WorkspaceItem, ProviderUpdatePatch, ProviderUpdateResult } from "./types"; const ADAPTERS = ["openai-responses", "openai-chat", "anthropic", "google", "azure-openai", "cursor"] as const; @@ -31,7 +32,7 @@ const EMPTY_MODELS: string[] = []; type ChoicesStatus = "idle" | "loading" | "ready" | "error"; type PacingRule = { requestsPerMinute?: number; minIntervalMs?: number; maxConcurrentRequests?: number }; -type PacingStatus = { enabled: boolean; queued: number; nextSlotInMs: number; lastStartedAt?: number; lastModelId?: string }; +type PacingStatus = { enabled: boolean; queued: number; nextSlotInMs: number; lastStartedAt?: number; lastModelId?: string; keys?: KeyPacingStatus[] }; type CursorHttpVersion = "http2" | "http1.1"; function effectiveCursorHttpVersion(value: WorkspaceItem["upstreamHttpVersion"]): CursorHttpVersion { @@ -58,6 +59,8 @@ function pacingSignature(value: WorkspaceItem["requestPacing"] | undefined): str value?.requestsPerMinute ?? null, value?.minIntervalMs ?? null, value?.maxConcurrentRequests ?? null, + value?.keySelection ?? "active", + [value?.keys?.requestsPerMinute ?? null, value?.keys?.minIntervalMs ?? null, value?.keys?.maxConcurrentRequests ?? null], models, ]); } @@ -98,6 +101,7 @@ export default function ProviderSettings({ const [choicesStatus, setChoicesStatus] = useState(apiBase ? "loading" : "idle"); const [endpointChoice, setEndpointChoice] = useState(() => "custom"); const [pacingEnabled, setPacingEnabled] = useState(item.requestPacing?.enabled === true); + const [keyPacing, setKeyPacing] = useState({ keySelection: item.requestPacing?.keySelection, keys: item.requestPacing?.keys }); const [pacingRpm, setPacingRpm] = useState(() => numberDraft(item.requestPacing?.requestsPerMinute)); const [pacingDelay, setPacingDelay] = useState(() => numberDraft(item.requestPacing?.minIntervalMs)); const [pacingConcurrency, setPacingConcurrency] = useState(() => numberDraft(item.requestPacing?.maxConcurrentRequests)); @@ -121,6 +125,7 @@ export default function ProviderSettings({ setLiveModels(savedLiveModels); setCursorHttpVersion(savedCursorHttpVersion); setPacingEnabled(item.requestPacing?.enabled === true); + setKeyPacing({ keySelection: item.requestPacing?.keySelection, keys: item.requestPacing?.keys }); setPacingRpm(numberDraft(item.requestPacing?.requestsPerMinute)); setPacingDelay(numberDraft(item.requestPacing?.minIntervalMs)); setPacingConcurrency(numberDraft(item.requestPacing?.maxConcurrentRequests)); @@ -192,11 +197,12 @@ export default function ProviderSettings({ const pacingDraft = useMemo(() => ({ enabled: pacingEnabled, + ...keyPacing, ...(positiveRpm(pacingRpm) !== undefined ? { requestsPerMinute: positiveRpm(pacingRpm) } : {}), ...(positiveInteger(pacingDelay) !== undefined ? { minIntervalMs: positiveInteger(pacingDelay) } : {}), ...(positiveInteger(pacingConcurrency) !== undefined ? { maxConcurrentRequests: positiveInteger(pacingConcurrency) } : {}), ...(Object.keys(pacingModels).length > 0 ? { models: pacingModels } : {}), - }), [pacingConcurrency, pacingDelay, pacingEnabled, pacingModels, pacingRpm]); + }), [keyPacing, pacingConcurrency, pacingDelay, pacingEnabled, pacingModels, pacingRpm]); const supportsReasoningReplay = !isCatalogProviderId(item.name) && canConfigureResponsesReasoningReplay({ adapter, baseUrl, authMode }); @@ -250,7 +256,14 @@ export default function ProviderSettings({ if (pacingConcurrency.trim() !== "" && positiveInteger(pacingConcurrency) === undefined) { setMsg({ ok: false, text: t("pws.pacingCapInvalid") }); return false; } - if (pacingEnabled && !pacingDraft.requestsPerMinute && !pacingDraft.minIntervalMs && !pacingDraft.maxConcurrentRequests && !pacingDraft.models) { + if (Object.entries(keyPacing.keys ?? {}).some(([field, value]) => + !Number.isFinite(value) || value <= 0 + || (field !== "requestsPerMinute" && !Number.isSafeInteger(value)) + || (field === "requestsPerMinute" && (value < 1 / 60 || value > 60_000)) + || (field === "minIntervalMs" && value > 3_600_000))) { + setMsg({ ok: false, text: t("pws.keyLimitInvalid") }); return false; + } + if (pacingEnabled && !pacingDraft.requestsPerMinute && !pacingDraft.minIntervalMs && !pacingDraft.maxConcurrentRequests && !pacingDraft.models && !pacingDraft.keys && pacingDraft.keySelection !== "round-robin") { setMsg({ ok: false, text: t("pws.pacingRuleRequired") }); return false; } const pacingOnly = pacingDirty && !dirty; @@ -339,6 +352,7 @@ export default function ProviderSettings({ setPreserveReasoningReplay(item.preserveResponsesReasoningContent === true); setCursorHttpVersion(savedCursorHttpVersion); setMsg(null); setPacingEnabled(item.requestPacing?.enabled === true); setPacingRpm(numberDraft(item.requestPacing?.requestsPerMinute)); + setKeyPacing({ keySelection: item.requestPacing?.keySelection, keys: item.requestPacing?.keys }); setPacingDelay(numberDraft(item.requestPacing?.minIntervalMs)); setPacingConcurrency(numberDraft(item.requestPacing?.maxConcurrentRequests)); setPacingModels({ ...(item.requestPacing?.models ?? {}) }); @@ -540,6 +554,7 @@ export default function ProviderSettings({ {pacingStatus?.nextSlotInMs ?? 0} ms {t("pws.pacingNextSlot")} {pacingStatus?.lastModelId ?? t("pws.pacingNone")} {t("pws.pacingLastModel")} + {authMode !== "oauth" && authMode !== "forward" && }

{t("pws.pacingModelOverrides")}

diff --git a/gui/src/i18n/de.ts b/gui/src/i18n/de.ts index 9cec5a6564f..9db753a5b40 100644 --- a/gui/src/i18n/de.ts +++ b/gui/src/i18n/de.ts @@ -2934,6 +2934,17 @@ export const de: Record = { "pws.removeDefaultConfirmBody": "Standardanbieter \"{name}\" entfernen? \"{defaultProvider}\" wird zum Standardanbieter. Dies kann nicht rückgängig gemacht werden.", "pws.removeConfirmTitle": "Anbieter entfernen", "pws.saveSettings": "Speichern", + "pws.keySelection": "API-Schlüsselauswahl", + "pws.keyActive": "Aktiver Schlüssel (bisheriges Verhalten)", + "pws.keyLimitInvalid": "Grenzen pro Schlüssel müssen positiv und im unterstützten Bereich sein; Intervall und Parallelität müssen ganze Zahlen sein.", + "pws.keyRoundRobin": "Round-Robin", + "pws.keyLimits": "Limits pro Schlüssel", + "pws.keySharedQuota": "Provider-Limits gelten weiterhin für alle Schlüssel. Schlüssel desselben Projekts können Kontingente teilen; RPM begrenzt keine Token- oder Tagesbudgets.", + "pws.keyHandle": "Schlüssel-ID", + "pws.keyStarts": "Gestartet", + "pws.keyInFlight": "Aktiv", + "pws.keyWait": "Warten", + "pws.keyCooldown": "Abkühlzeit", "pws.pacingTitle": "Anfragetaktung", "pws.pacingDesc": "Verteilt ausgehende Anfragestarts für diesen Anbieter gleichmäßig. Streaming-Antworten dürfen sich überlappen.", "pws.pacingEnabled": "Aktiviert", diff --git a/gui/src/i18n/en.ts b/gui/src/i18n/en.ts index 7e2ba2804e0..fc6cd0a4958 100644 --- a/gui/src/i18n/en.ts +++ b/gui/src/i18n/en.ts @@ -1826,6 +1826,17 @@ export const en = { "pws.removeDefaultConfirmBody": "Remove default provider \"{name}\"? \"{defaultProvider}\" will become the default provider. This cannot be undone.", "pws.removeConfirmTitle": "Remove provider", "pws.saveSettings": "Save", + "pws.keySelection": "API key selection", + "pws.keyActive": "Active key (existing behavior)", + "pws.keyLimitInvalid": "Per-key limits must be positive and within the supported range; interval and concurrency must be whole numbers.", + "pws.keyRoundRobin": "Round-robin", + "pws.keyLimits": "Per-key limits", + "pws.keySharedQuota": "Provider limits still apply across all keys. Keys in the same project may share quotas; RPM limits do not enforce token or daily budgets.", + "pws.keyHandle": "Key ID", + "pws.keyStarts": "Started", + "pws.keyInFlight": "In flight", + "pws.keyWait": "Wait", + "pws.keyCooldown": "Cooldown", "pws.pacingTitle": "Request pacing", "pws.pacingDesc": "Evenly delay outbound request starts for this provider. Streaming responses may overlap.", "pws.pacingEnabled": "Enabled", diff --git a/gui/src/i18n/fr.ts b/gui/src/i18n/fr.ts index 9fe9862b17e..796ebc270b8 100644 --- a/gui/src/i18n/fr.ts +++ b/gui/src/i18n/fr.ts @@ -1773,6 +1773,17 @@ export const fr: Record = { "pws.cursorTransportHttp1": "HTTP/1.1 (compatibilité proxy)", "pws.cursorTransportDesc": "Utilisez HTTP/1.1 si votre proxy ne transporte pas de façon fiable le flux HTTP/2 de Cursor.", "pws.optionalPlaceholder": "Facultatif", + "pws.keySelection": "Sélection de clé API", + "pws.keyActive": "Clé active (comportement existant)", + "pws.keyLimitInvalid": "Les limites par clé doivent être positives et dans la plage autorisée ; l’intervalle et la simultanéité doivent être des entiers.", + "pws.keyRoundRobin": "Tourniquet", + "pws.keyLimits": "Limites par clé", + "pws.keySharedQuota": "Les limites du fournisseur restent communes à toutes les clés. Un même projet peut partager ses quotas ; les RPM ne limitent ni les jetons ni le budget quotidien.", + "pws.keyHandle": "ID de clé", + "pws.keyStarts": "Démarrées", + "pws.keyInFlight": "En cours", + "pws.keyWait": "Attente", + "pws.keyCooldown": "Temporisation", "pws.pacingTitle": "Cadencement des requêtes", "pws.pacingDesc": "Répartissez uniformément le démarrage des requêtes sortantes pour ce fournisseur. Les réponses diffusées peuvent se chevaucher.", "pws.pacingEnabled": "Activé", diff --git a/gui/src/i18n/ja.ts b/gui/src/i18n/ja.ts index b8af80188eb..42a8ac3d7c5 100644 --- a/gui/src/i18n/ja.ts +++ b/gui/src/i18n/ja.ts @@ -1694,6 +1694,17 @@ export const ja: Record = { "pws.removeDefaultConfirmBody": "既定のプロバイダー \"{name}\" を削除しますか? \"{defaultProvider}\" が既定のプロバイダーになります。この操作は元に戻せません。", "pws.removeConfirmTitle": "プロバイダーを削除", "pws.saveSettings": "保存", + "pws.keySelection": "API キーの選択", + "pws.keyActive": "有効なキー優先（既存の動作）", + "pws.keyLimitInvalid": "キー別の制限は対応範囲内の正の値にし、間隔と同時リクエスト数は整数にしてください。", + "pws.keyRoundRobin": "ラウンドロビン", + "pws.keyLimits": "キーごとの制限", + "pws.keySharedQuota": "プロバイダー全体の制限も適用されます。同じプロジェクトのキーは割当量を共有する場合があります。RPM はトークン数や日次上限を制御しません。", + "pws.keyHandle": "キー ID", + "pws.keyStarts": "開始回数", + "pws.keyInFlight": "実行中", + "pws.keyWait": "待機", + "pws.keyCooldown": "クールダウン", "pws.pacingTitle": "リクエスト間隔調整", "pws.pacingDesc": "このプロバイダーへの送信開始を均等に遅延します。ストリーミング応答は重複できます。", "pws.pacingEnabled": "有効", diff --git a/gui/src/i18n/ko.ts b/gui/src/i18n/ko.ts index d82537229cf..d219d300d1c 100644 --- a/gui/src/i18n/ko.ts +++ b/gui/src/i18n/ko.ts @@ -2981,6 +2981,17 @@ export const ko: Record = { "pws.removeDefaultConfirmBody": "기본 프로바이더 \"{name}\"을(를) 제거하시겠습니까? \"{defaultProvider}\"이(가) 기본 프로바이더가 됩니다. 이 작업은 되돌릴 수 없습니다.", "pws.removeConfirmTitle": "프로바이더 제거", "pws.saveSettings": "저장", + "pws.keySelection": "API 키 선택 방식", + "pws.keyActive": "활성 키 우선 (기존 동작)", + "pws.keyLimitInvalid": "키별 제한은 허용 범위의 양수여야 하며, 간격과 동시 요청 수는 정수여야 합니다.", + "pws.keyRoundRobin": "라운드로빈", + "pws.keyLimits": "키별 제한", + "pws.keySharedQuota": "프로바이더 제한은 모든 키에 합산 적용됩니다. 같은 프로젝트의 키는 할당량을 공유할 수 있으며, RPM 제한은 토큰·일일 한도를 관리하지 않습니다.", + "pws.keyHandle": "키 ID", + "pws.keyStarts": "시작 횟수", + "pws.keyInFlight": "처리 중", + "pws.keyWait": "대기", + "pws.keyCooldown": "쿨다운", "pws.pacingTitle": "요청 속도 조절", "pws.pacingDesc": "이 프로바이더로 나가는 요청 시작을 일정 간격으로 지연합니다. 스트리밍 응답은 서로 겹칠 수 있습니다.", "pws.pacingEnabled": "사용", diff --git a/gui/src/i18n/pt.ts b/gui/src/i18n/pt.ts index 89ef6471c86..8cae48d2f90 100644 --- a/gui/src/i18n/pt.ts +++ b/gui/src/i18n/pt.ts @@ -1810,6 +1810,17 @@ export const pt: Record = { "pws.removeDefaultConfirmBody": "Remover o provedor padrão \"{name}\"? \"{defaultProvider}\" passará a ser o provedor padrão. Esta ação não pode ser desfeita.", "pws.removeConfirmTitle": "Remover provedor", "pws.saveSettings": "Salvar", + "pws.keySelection": "Seleção de chave API", + "pws.keyActive": "Priorizar a chave ativa (comportamento atual)", + "pws.keyLimitInvalid": "Os limites por chave devem ser positivos e estar no intervalo permitido; intervalo e simultaneidade devem ser números inteiros.", + "pws.keyRoundRobin": "Round-robin", + "pws.keyLimits": "Limites por chave", + "pws.keySharedQuota": "Os limites do provedor continuam valendo para todas as chaves. Chaves do mesmo projeto podem compartilhar cotas; RPM não limita tokens nem o orçamento diário.", + "pws.keyHandle": "ID da chave", + "pws.keyStarts": "Iniciadas", + "pws.keyInFlight": "Em andamento", + "pws.keyWait": "Espera", + "pws.keyCooldown": "Intervalo de recuperação", "pws.pacingTitle": "Ritmo de requisições", "pws.pacingDesc": "Distribui uniformemente o início das requisições de saída deste provedor. Respostas em streaming podem se sobrepor.", "pws.pacingEnabled": "Ativado", diff --git a/gui/src/i18n/ru.ts b/gui/src/i18n/ru.ts index 5baa1942d14..987eb444281 100644 --- a/gui/src/i18n/ru.ts +++ b/gui/src/i18n/ru.ts @@ -1785,6 +1785,17 @@ export const ru: Record = { "pws.removeDefaultConfirmBody": "Удалить провайдера по умолчанию \"{name}\"? \"{defaultProvider}\" станет провайдером по умолчанию. Это действие нельзя отменить.", "pws.removeConfirmTitle": "Удалить провайдера", "pws.saveSettings": "Сохранить", + "pws.keySelection": "Выбор API-ключа", + "pws.keyActive": "Сначала активный ключ (прежнее поведение)", + "pws.keyLimitInvalid": "Лимиты для ключа должны быть положительными и в допустимом диапазоне; интервал и число одновременных запросов должны быть целыми.", + "pws.keyRoundRobin": "По кругу", + "pws.keyLimits": "Лимиты на ключ", + "pws.keySharedQuota": "Лимиты провайдера действуют для всех ключей. Ключи одного проекта могут делить квоты; RPM не ограничивает токены и суточный бюджет.", + "pws.keyHandle": "ID ключа", + "pws.keyStarts": "Запущено", + "pws.keyInFlight": "В работе", + "pws.keyWait": "Ожидание", + "pws.keyCooldown": "Пауза", "pws.pacingTitle": "Интервал запросов", "pws.pacingDesc": "Равномерно задерживает начало исходящих запросов к провайдеру. Потоковые ответы могут пересекаться.", "pws.pacingEnabled": "Включено", diff --git a/gui/src/i18n/tr.ts b/gui/src/i18n/tr.ts index a5242dfd27f..c6c1405fff8 100644 --- a/gui/src/i18n/tr.ts +++ b/gui/src/i18n/tr.ts @@ -1812,6 +1812,17 @@ export const tr: Record = { "pws.removeDefaultConfirmBody": "Varsayılan sağlayıcı \"{name}\" kaldırılsın mı? \"{defaultProvider}\" varsayılan sağlayıcı olacaktır. Bu işlem geri alınamaz.", "pws.removeConfirmTitle": "Sağlayıcıyı kaldır", "pws.saveSettings": "Kaydet", + "pws.keySelection": "API anahtarı seçimi", + "pws.keyActive": "Etkin anahtar (mevcut davranış)", + "pws.keyLimitInvalid": "Anahtar sınırları pozitif ve desteklenen aralıkta olmalıdır; aralık ve eşzamanlılık tam sayı olmalıdır.", + "pws.keyRoundRobin": "Sıralı dağıtım", + "pws.keyLimits": "Anahtar başına sınırlar", + "pws.keySharedQuota": "Sağlayıcı sınırları tüm anahtarlara uygulanır. Aynı projedeki anahtarlar kota paylaşabilir; RPM, token veya günlük bütçeyi sınırlamaz.", + "pws.keyHandle": "Anahtar ID", + "pws.keyStarts": "Başlatılan", + "pws.keyInFlight": "Çalışan", + "pws.keyWait": "Bekleme", + "pws.keyCooldown": "Soğuma", "pws.pacingTitle": "İstek aralığı", "pws.pacingDesc": "Bu sağlayıcıya giden istek başlangıçlarını eşit aralıklarla geciktirir. Akış yanıtları çakışabilir.", "pws.pacingEnabled": "Etkin", diff --git a/gui/src/i18n/vi.ts b/gui/src/i18n/vi.ts index 0bdcea43641..b86f6ec6127 100644 --- a/gui/src/i18n/vi.ts +++ b/gui/src/i18n/vi.ts @@ -1796,6 +1796,17 @@ export const vi: Record = { "pws.removeDefaultConfirmBody": "Xoá provider mặc định \\\"{name}\\\"? \\\"{defaultProvider}\\\" sẽ trở thành provider mặc định. Hành động này không thể hoàn tác.", "pws.removeConfirmTitle": "Xóa Provider", "pws.saveSettings": "Lưu", + "pws.keySelection": "Chọn khóa API", + "pws.keyActive": "Khóa đang hoạt động (hành vi hiện tại)", + "pws.keyLimitInvalid": "Giới hạn mỗi khóa phải dương và trong phạm vi hỗ trợ; khoảng thời gian và số yêu cầu đồng thời phải là số nguyên.", + "pws.keyRoundRobin": "Luân phiên", + "pws.keyLimits": "Giới hạn mỗi khóa", + "pws.keySharedQuota": "Giới hạn nhà cung cấp vẫn áp dụng cho mọi khóa. Khóa cùng dự án có thể dùng chung hạn mức; RPM không giới hạn token hay ngân sách ngày.", + "pws.keyHandle": "ID khóa", + "pws.keyStarts": "Đã bắt đầu", + "pws.keyInFlight": "Đang chạy", + "pws.keyWait": "Chờ", + "pws.keyCooldown": "Tạm nghỉ", "pws.pacingTitle": "Pacing của yêu cầu", "pws.pacingDesc": "Đồng đều trì hoãn thời điểm bắt đầu yêu cầu ra ngoài cho provider này. Các phản hồi stream có thể chồng chéo.", "pws.pacingEnabled": "Đã bật", diff --git a/gui/src/i18n/zh-TW.ts b/gui/src/i18n/zh-TW.ts index ebd784e3c9c..ba096880cf1 100644 --- a/gui/src/i18n/zh-TW.ts +++ b/gui/src/i18n/zh-TW.ts @@ -1490,6 +1490,17 @@ export const zhTW: Record = { "pws.removeDefaultConfirmBody": "移除預設供應商「{name}」？「{defaultProvider}」將成為預設供應商。此操作無法撤消。", "pws.removeConfirmTitle": "移除供應商", "pws.saveSettings": "儲存", + "pws.keySelection": "API 金鑰選擇", + "pws.keyActive": "優先使用作用中金鑰（現有行為）", + "pws.keyLimitInvalid": "每個金鑰的限制必須為支援範圍內的正數；間隔和並行數必須為整數。", + "pws.keyRoundRobin": "輪詢", + "pws.keyLimits": "每個金鑰的限制", + "pws.keySharedQuota": "供應商限制仍對所有金鑰合計生效。同一專案的金鑰可能共享配額；RPM 不限制權杖或每日預算。", + "pws.keyHandle": "金鑰 ID", + "pws.keyStarts": "已開始", + "pws.keyInFlight": "處理中", + "pws.keyWait": "等待", + "pws.keyCooldown": "冷卻中", "pws.pacingTitle": "請求節流", "pws.pacingDesc": "均勻延遲送往此供應商的請求啟動。串流回應可以重疊。", "pws.pacingEnabled": "啟用", diff --git a/gui/src/i18n/zh.ts b/gui/src/i18n/zh.ts index ee57413daf0..cbbefbfffd6 100644 --- a/gui/src/i18n/zh.ts +++ b/gui/src/i18n/zh.ts @@ -2954,6 +2954,17 @@ export const zh: Record = { "pws.removeDefaultConfirmBody": "移除默认提供方「{name}」？「{defaultProvider}」将成为默认提供方。此操作无法撤销。", "pws.removeConfirmTitle": "移除提供商", "pws.saveSettings": "保存", + "pws.keySelection": "API 密钥选择", + "pws.keyActive": "活动密钥优先（现有行为）", + "pws.keyLimitInvalid": "每个密钥的限制必须为支持范围内的正数；间隔和并发数必须为整数。", + "pws.keyRoundRobin": "轮询", + "pws.keyLimits": "每个密钥的限制", + "pws.keySharedQuota": "提供商限制仍对所有密钥合并生效。同一项目的密钥可能共享配额；RPM 不限制令牌或每日预算。", + "pws.keyHandle": "密钥 ID", + "pws.keyStarts": "已开始", + "pws.keyInFlight": "处理中", + "pws.keyWait": "等待", + "pws.keyCooldown": "冷却中", "pws.pacingTitle": "请求节流", "pws.pacingDesc": "均匀延迟发往此提供商的请求启动。流式响应可以重叠。", "pws.pacingEnabled": "启用", diff --git a/gui/src/provider-workspace/catalog.ts b/gui/src/provider-workspace/catalog.ts index 566a195f7b2..21ffc01b168 100644 --- a/gui/src/provider-workspace/catalog.ts +++ b/gui/src/provider-workspace/catalog.ts @@ -58,6 +58,8 @@ export interface WorkspaceProvider { minIntervalMs?: number; maxConcurrentRequests?: number; models?: Record; + keySelection?: "active" | "round-robin"; + keys?: { requestsPerMinute?: number; minIntervalMs?: number; maxConcurrentRequests?: number }; }; /** Codex account routing mode for the canonical `openai` forward provider. */ codexAccountMode?: "direct" | "pool"; diff --git a/gui/tests/provider-settings-request-pacing.test.tsx b/gui/tests/provider-settings-request-pacing.test.tsx index 588671f975b..9d1070cd23d 100644 --- a/gui/tests/provider-settings-request-pacing.test.tsx +++ b/gui/tests/provider-settings-request-pacing.test.tsx @@ -90,6 +90,21 @@ test("settings saves provider pacing and a slower model override", async () => { await act(async () => { container.unmount(); }); }); +test("key allocation and per-key limits save through the existing provider settings action", async () => { + const patches: ProviderUpdatePatch[] = []; + const container = await renderSettings({ name: "fixture", adapter: "openai-chat", baseUrl: "https://example.test", authMode: "key" }, + async (_name, patch) => { patches.push(patch); return { ok: true }; }); + await act(async () => { container.querySelector(".pwi-pacing-toggle input")!.click(); }); + const select = [...container.querySelectorAll("select")].find(el => el.querySelector('option[value="round-robin"]'))!; + await act(async () => { select.value = "round-robin"; select.dispatchEvent(new testWindow.Event("change", { bubbles: true })); }); + const numbers = container.querySelectorAll(".pwi-key-limits input"); + await setInput(numbers[0]!, "15"); await setInput(numbers[2]!, "1"); + await act(async () => { container.querySelector(".pwi-settings-sticky-bar .btn-primary")!.click(); }); + expect(patches[0]?.requestPacing).toMatchObject({ enabled: true, keySelection: "round-robin", keys: { requestsPerMinute: 15, maxConcurrentRequests: 1 } }); + expect(container.textContent).toContain("Keys in the same project may share quotas"); + container.unmount(); +}); + test("a concurrency-only pacing rule saves without the rule-required error", async () => { const item: WorkspaceItem = { name: "nvidia", @@ -135,6 +150,21 @@ test("discard reverts an unsaved concurrency draft to the stored cap", async () await act(async () => { container.unmount(); }); }); +test("invalid key limits block save and discard restores the stored distribution policy", async () => { + const patches: ProviderUpdatePatch[] = []; + const container = await renderSettings({ name: "fixture", adapter: "openai-chat", baseUrl: "https://example.test", authMode: "key", + requestPacing: { enabled: true, keySelection: "round-robin", keys: { requestsPerMinute: 15, maxConcurrentRequests: 1 } } }, + async (_name, patch) => { patches.push(patch); return { ok: true }; }); + const numbers = container.querySelectorAll(".pwi-key-limits input"); + await setInput(numbers[2]!, "0"); + await act(async () => { container.querySelector(".pwi-settings-sticky-bar .btn-primary")!.click(); }); + expect(patches).toEqual([]); + expect(container.textContent).toContain("Per-key limits must be positive"); + await act(async () => { container.querySelector(".pwi-settings-sticky-bar .btn-ghost")!.click(); }); + expect(numbers[2]!.value).toBe("1"); expect(numbers[0]!.value).toBe("15"); + await act(async () => { container.unmount(); }); +}); + test("pacing grids keep every field and the add button on one row", async () => { const css = await Bun.file(new URL("../src/styles/provider-workspace-settings.css", import.meta.url)).text(); // The provider grid owns three inputs (rpm, interval, concurrency) and the model diff --git a/scripts/test-layout/layout.json b/scripts/test-layout/layout.json index 0c289a8fdfc..38f2bc7fb8a 100644 --- a/scripts/test-layout/layout.json +++ b/scripts/test-layout/layout.json @@ -2,8 +2,8 @@ "version": 1, "root": "tests", "explicitShards": ["layout-clients.json"], - "explicit": { "service-log.test.ts": "service", - "codex-history-pending-wal.test.ts": "codex-integration", "responses-anthropic-billing-send-budget.test.ts": "responses", "anthropic-billing-ownership.test.ts": "adapters/anthropic", "server-anthropic-billing.test.ts": "server", "server-anthropic-billing-boundaries.test.ts": "server", "router-anthropic-billing-failover.test.ts": "routing", "anthropic-billing-refusal.test.ts": "adapters/anthropic", "messages-request-id-headers.test.ts": "claude-integration", "bounded-file-read.test.ts": "lib", "responses-undeclared-tool-guard-inbound.test.ts": "responses", + "explicit": { "api-key-selection-cooldown.test.ts": "providers", "server-chat-native-key-selection.test.ts": "server", "compact-key-selection.test.ts": "responses", "messages-native-key-selection.test.ts": "claude-integration", "service-log.test.ts": "service", + "codex-history-pending-wal.test.ts": "codex-integration", "responses-anthropic-billing-send-budget.test.ts": "responses", "anthropic-billing-ownership.test.ts": "adapters/anthropic", "server-anthropic-billing.test.ts": "server", "server-anthropic-billing-boundaries.test.ts": "server", "router-anthropic-billing-failover.test.ts": "routing", "anthropic-billing-refusal.test.ts": "adapters/anthropic", "messages-request-id-headers.test.ts": "claude-integration", "bounded-file-read.test.ts": "lib", "responses-undeclared-tool-guard-inbound.test.ts": "responses", "messages-request-id-endpoint.test.ts": "claude-integration", "anthropic-revoked-token.test.ts": "adapters/anthropic", "messages-revoked-token.test.ts": "claude-integration", @@ -1428,7 +1428,7 @@ "request-log-generation-window.test.ts": "usage", "request-log.test.ts": "usage", "upstream-failure-diagnostics.test.ts": "usage", "request-outcome-agreement.test.ts": "usage", - "request-pacing.test.ts": "usage", + "request-pacing.test.ts": "usage", "request-pacing-provenance.test.ts": "usage", "reserve-auth-context.test.ts": "codex-integration", "reserve-availability.test.ts": "codex-integration", "reserve-catalog-lifecycle.test.ts": "codex-integration", @@ -1470,7 +1470,7 @@ "responses-custom-tool-repair.test.ts": "responses", "responses-custom-tool-stream-consistency.test.ts": "responses", "responses-default-namespace-emit-normalize.test.ts": "responses", - "responses-fetch-helpers-boundary.test.ts": "responses", + "responses-fetch-helpers-boundary.test.ts": "responses", "responses-run-turn-key-pacing.test.ts": "responses", "responses-field-backfill.test.ts": "responses", "responses-forward-dangling-call.test.ts": "responses", "responses-forward-incomplete-quota.test.ts": "responses", @@ -1583,7 +1583,7 @@ "server-images-bodyless-content-length.test.ts": "server", "server-images-pool-admission.test.ts": "server", "server-images.test.ts": "server", - "server-key-failover-e2e.test.ts": "server", + "server-key-failover-e2e.test.ts": "server", "distributed-key-401.test.ts": "server", "server-jev-combo-e2e.test.ts": "server", "combo-requested-effort.test.ts": "server", "server-kiro-completion-e2e.test.ts": "server", "server-kiro-device-login.test.ts": "server", "server-kiro-oauth-401-replay.test.ts": "server", diff --git a/src/config/schema/leaf-validators.ts b/src/config/schema/leaf-validators.ts index 75b39a29ba3..7302663df1d 100644 --- a/src/config/schema/leaf-validators.ts +++ b/src/config/schema/leaf-validators.ts @@ -133,12 +133,15 @@ const requestPacingSchema = z.object({ minIntervalMs: z.number().int().min(1).max(3_600_000).optional(), maxConcurrentRequests: z.number().int().min(1).optional(), models: z.record(z.string().trim().min(1), requestPacingRuleSchema).optional(), + keySelection: z.enum(["active", "round-robin"]).optional(), + keys: requestPacingRuleSchema.optional(), }).strict().refine(value => value.enabled === false || value.requestsPerMinute !== undefined || value.minIntervalMs !== undefined || value.maxConcurrentRequests !== undefined + || value.keys !== undefined || value.keySelection === "round-robin" || (value.models !== undefined && Object.keys(value.models).length > 0), { - message: "enabled request pacing needs a provider rule or model override", + message: "enabled request pacing needs a provider, model, or key rule, or round-robin selection", }); export function requestPacingConfigError(value: unknown): string | null { diff --git a/src/providers/api-key-cooldowns.ts b/src/providers/api-key-cooldowns.ts new file mode 100644 index 00000000000..42b00424ee3 --- /dev/null +++ b/src/providers/api-key-cooldowns.ts @@ -0,0 +1,49 @@ +/** Process-local cooldown state shared by selection validation and key failover. */ +const keyCooldowns = new Map(); + +function cooldownKey(providerName: string, keyId: string): string { + return `${providerName}\0${keyId}`; +} + +export function setKeyCooldown(providerName: string, keyId: string, cooldownUntil: number): void { + const key = cooldownKey(providerName, keyId); + keyCooldowns.set(key, { cooldownUntil: Math.max(keyCooldowns.get(key)?.cooldownUntil ?? 0, cooldownUntil) }); +} + +export function isKeyInCooldown(providerName: string, keyId: string, now = Date.now()): boolean { + const entry = keyCooldowns.get(cooldownKey(providerName, keyId)); + if (!entry) return false; + if (entry.cooldownUntil <= now) { + keyCooldowns.delete(cooldownKey(providerName, keyId)); + return false; + } + return true; +} + +export function getKeyCooldownUntil(providerName: string, keyId: string, now = Date.now()): number | null { + const entry = keyCooldowns.get(cooldownKey(providerName, keyId)); + if (!entry) return null; + return entry.cooldownUntil > now ? entry.cooldownUntil : null; +} + +export function sweepExpiredApiKeyCooldowns(now = Date.now()): number { + let removed = 0; + for (const [key, cooldown] of keyCooldowns) { + if (cooldown.cooldownUntil > now) continue; + keyCooldowns.delete(key); + removed += 1; + } + return removed; +} + +/** Manual key management clears this provider's cooldowns. */ +export function clearKeyCooldowns(providerName?: string): void { + if (!providerName) { + keyCooldowns.clear(); + return; + } + const prefix = `${providerName}\0`; + for (const key of keyCooldowns.keys()) { + if (key.startsWith(prefix)) keyCooldowns.delete(key); + } +} diff --git a/src/providers/api-key-selection.ts b/src/providers/api-key-selection.ts index 40eff2d49a9..74ec7c314b0 100644 --- a/src/providers/api-key-selection.ts +++ b/src/providers/api-key-selection.ts @@ -1,3 +1,4 @@ +import { isKeyInCooldown } from "./api-key-cooldowns"; import { randomUUID } from "node:crypto"; import { mutatePersistedConfig } from "../config"; import { publishAccountSelection } from "../lib/account-selection-events"; @@ -17,10 +18,17 @@ function matchesSelection(provider: OcxProviderConfig, expected: ProviderApiKeyS && current.revision === expected.revision; } -function currentKeyProvider(config: OcxConfig, name: string): OcxProviderConfig | null { +function currentKeyProvider(config: OcxConfig, name: string, selection?: ProviderApiKeySelection): OcxProviderConfig | null { const configured = config.providers[name]; if (!configured || configured.disabled) return null; - const current = routedProviderConfig(name, { ...configured, _apiKeyAttempt: undefined }); + let row = configured; + if (selection?.distributed) { + const entry = configured.apiKeyPool?.find(key => key.id === selection.entryId && key.key === selection.reference); + if (!entry || isKeyInCooldown(name, entry.id) || configured.apiKeySelectionRevision !== selection.revision + || !configured.requestPacing?.enabled || configured.requestPacing.keySelection !== "round-robin") return null; + row = { ...configured, apiKey: entry.key, _apiKeyAttempt: selection }; + } + const current = routedProviderConfig(name, { ...row, _apiKeyAttempt: selection?.distributed ? selection : undefined }); if (current.authMode === "oauth" || current.authMode === "forward") return null; if (current.authMode === "key" && !current.keyOptional && !current.apiKey?.trim()) return null; return current; @@ -32,10 +40,10 @@ export function providerApiKeySelectionIsCurrent( name: string, routedProvider: OcxProviderConfig, ): boolean { - const current = currentKeyProvider(config, name); const expected = routedProvider._apiKeyAttempt; + const current = currentKeyProvider(config, name, expected); return current !== null && expected !== undefined - && matchesSelection(config.providers[name]!, expected) + && (expected.distributed === true || matchesSelection(config.providers[name]!, expected)) && current.apiKey === routedProvider.apiKey && current.authMode === routedProvider.authMode && current.baseUrl === routedProvider.baseUrl; @@ -47,7 +55,11 @@ export function resolveCurrentProviderApiKeyTransport( name: string, routedProvider: OcxProviderConfig, ): OcxProviderConfig | null { - const current = currentKeyProvider(config, name); + // A permit acquired for one key cannot authorize a different key after a manual edit. + if (routedProvider.requestPacing?.enabled && routedProvider.requestPacing.keys + && routedProvider._apiKeyAttempt && config.providers[name] + && !routedProvider._apiKeyAttempt.distributed && !matchesSelection(config.providers[name]!, routedProvider._apiKeyAttempt)) return null; + const current = currentKeyProvider(config, name, routedProvider._apiKeyAttempt); if (!current) return null; const runtime = routedProvider as OcxProviderTransport; const headers = { ...current.headers }; diff --git a/src/providers/key-failover.ts b/src/providers/key-failover.ts index 564f03e19e8..f0eb227ff1d 100644 --- a/src/providers/key-failover.ts +++ b/src/providers/key-failover.ts @@ -8,7 +8,10 @@ * * Modelled after src/codex/routing.ts cooldown logic but scoped to plain API-key pools. */ +import { isKeyInCooldown, getKeyCooldownUntil, setKeyCooldown } from "./api-key-cooldowns"; +export { clearKeyCooldowns, getKeyCooldownUntil, sweepExpiredApiKeyCooldowns } from "./api-key-cooldowns"; import { commitProviderApiKeySelection, providerApiKeySelectionIsCurrent } from "./api-key-selection"; +import { RequestPacingQueueOverloadError } from "./request-pacing"; import type { ProviderApiKeySelection } from "../types/provider"; import { routedProviderConfig } from "../router"; import { getProviderRegistryEntry } from "./registry"; @@ -24,10 +27,6 @@ import { cachedApiKeyQuota } from "./quota-key-accounts"; // ---- cooldown state (in-memory, same as codex/routing.ts) ---- -interface KeyCooldown { - cooldownUntil: number; -} - const DEFAULT_COOLDOWN_MS = 60_000; const MAX_COOLDOWN_MS = 10 * 60_000; // cap at 10 min for api-key rotation /** @@ -347,11 +346,6 @@ const DEFAULT_RESET_REPLAY = { } as const satisfies Required; /** Map<`${providerName}\0${keyId}`, KeyCooldown> */ -const keyCooldowns = new Map(); - -function cooldownKey(providerName: string, keyId: string): string { - return `${providerName}\0${keyId}`; -} /** * Parse an upstream `Retry-After` header: numeric seconds (including `0`) or an HTTP-date. @@ -379,16 +373,6 @@ function parseRetryAfterMs(value: string | null | undefined, now = Date.now()): * True while the given key is inside its 429 cooldown window (lazily evicting the entry once the * window expires). Used to skip keys that the upstream just rate-limited during failover. */ -function isKeyInCooldown(providerName: string, keyId: string, now = Date.now()): boolean { - const entry = keyCooldowns.get(cooldownKey(providerName, keyId)); - if (!entry) return false; - if (entry.cooldownUntil <= now) { - keyCooldowns.delete(cooldownKey(providerName, keyId)); - return false; - } - return true; -} - // ---- public API ---- /** @@ -418,9 +402,11 @@ const keyRotationCursor = new Map(); export function forgetApiKeyRotationCursor(providerName?: string): void { if (!providerName) { keyRotationCursor.clear(); + dispatchKeyCursor.clear(); return; } keyRotationCursor.delete(providerName); + dispatchKeyCursor.delete(providerName); } /** The pool entry shape is inline on OcxProviderConfig; name it once rather than re-spelling it. */ @@ -566,11 +552,33 @@ export function selectProactiveApiKeyTransport( promptCacheKey?: string, now = Date.now(), ): OcxProviderTransport | null { + const configured = config.providers[providerName]; + if (configured?.requestPacing?.enabled && configured.requestPacing.keySelection === "round-robin" + && hasKeyPoolFailover(configured)) { + return applyRotatedTransport(providerName, routedProvider, allocateDistributedKey(providerName, configured, now), promptCacheKey); + } const committed = selectProactiveApiKey(config, providerName, now); if (!committed) return null; return applyRotatedTransport(providerName, routedProvider, committed, promptCacheKey); } +const dispatchKeyCursor = new Map(); +function allocateDistributedKey(providerName: string, provider: OcxProviderConfig, now: number): OcxProviderConfig { + const pool = (provider.apiKeyPool ?? []).filter((entry, index, all) => all.findIndex(other => other.key === entry.key) === index); + const previous = dispatchKeyCursor.get(providerName); + const start = pool.findIndex(entry => entry.id === previous); + for (let offset = 1; offset <= pool.length; offset++) { + const entry = pool[(start + offset) % pool.length]!; + if (isKeyInCooldown(providerName, entry.id, now)) continue; + dispatchKeyCursor.set(providerName, entry.id); + return { ...provider, apiKey: entry.key, _apiKeyAttempt: { + entryId: entry.id, reference: entry.key, revision: provider.apiKeySelectionRevision, distributed: true, + } }; + } + const next = Math.min(...pool.map(entry => getKeyCooldownUntil(providerName, entry.id, now) ?? now + 1000)); + throw new RequestPacingQueueOverloadError(providerName, "keys_exhausted", Math.max(1, Math.ceil((next - now) / 1000))); +} + /** * Normalize a provider's `retryOn429` policy. An explicit object always wins (including * `enabled: false` to opt out). When the knob is absent, the OpenCode Go destination @@ -695,6 +703,24 @@ export function rateLimitRetryDelayMs( * rebuilds from this committed row, reapplies registry metadata, and retains only explicit * runtime transport state (`fetch` and generated OpenCode session affinity). */ +/** Record a request-local key failure without selecting or sending a replacement. */ +export function recordDistributedKeyFailure( + config: OcxConfig, providerName: string, selection: ProviderApiKeySelection | undefined, + status: 401 | 429 | "billing", now = Date.now(), retryAfter?: string | null, quotaResetAt?: number, +): boolean { + const provider = config.providers[providerName]; + if (!selection?.distributed || !provider || provider.authMode === "oauth" || provider.authMode === "forward") return false; + const failed = provider.apiKeyPool?.find(entry => entry.id === selection.entryId && entry.key === selection.reference); + if (!failed || provider.apiKeySelectionRevision !== selection.revision) return false; + const delay = status === "billing" ? ANTHROPIC_BILLING_COOLDOWN_MS : status === 401 ? MAX_COOLDOWN_MS : quotaResetAt !== undefined + ? Math.max(1, quotaResetAt - now) : parseRetryAfterMs(retryAfter, now) ?? DEFAULT_COOLDOWN_MS; + for (const entry of provider.apiKeyPool ?? []) if (entry.key === failed.key) { + setKeyCooldown(providerName, entry.id, now + delay); + } + sweepExpiredOnWrite(now); + return true; +} + function rotateKeyAfterFailure( config: OcxConfig, providerName: string, @@ -710,6 +736,13 @@ function rotateKeyAfterFailure( if (!provider) return null; if (provider.authMode === "oauth" || provider.authMode === "forward") return null; + if (attemptedSelection?.distributed) { + if (acceptsFreshProvider && !acceptsFreshProvider(provider)) return null; + if (!recordDistributedKeyFailure(config, providerName, attemptedSelection, failureStatus, now, retryAfterHeader, quotaResetAt)) return null; + if (!(provider.apiKeyPool ?? []).some(entry => !isKeyInCooldown(providerName, entry.id, now))) return null; + return allocateDistributedKey(providerName, provider, now); + } + const failedKey = attemptedSelection?.reference ?? attemptedKey ?? provider.apiKey; type Rotation = | { failedId?: string; candidateId?: string } @@ -778,7 +811,7 @@ function rotateKeyAfterFailure( : quotaResetAt !== undefined ? Math.max(quotaResetAt - now, 1) : parseRetryAfterMs(retryAfterHeader, now) ?? DEFAULT_COOLDOWN_MS; - keyCooldowns.set(cooldownKey(providerName, outcome.value.failedId), { cooldownUntil: now + cooldownMs }); + setKeyCooldown(providerName, outcome.value.failedId, now + cooldownMs); sweepExpiredOnWrite(now); } if ("exhaustedCount" in outcome.value) { @@ -831,16 +864,6 @@ export function rotateKeyOn401( return rotateKeyAfterFailure(config, providerName, 401, null, now, attemptedKey, attemptedSelection); } -export function sweepExpiredApiKeyCooldowns(now = Date.now()): number { - let removed = 0; - for (const [key, cooldown] of keyCooldowns) { - if (cooldown.cooldownUntil > now) continue; - keyCooldowns.delete(key); - removed += 1; - } - return removed; -} - interface RotateProviderTransportOptions { retryAfter?: string | null; /** Epoch ms from `parseQuotaResetAt`, when the upstream dated the reset in its body. */ @@ -934,22 +957,3 @@ function applyRotatedTransport( }; return resolveProviderTransport(providerName, retryProvider, promptCacheKey); } - -/** Clear cooldown state for a provider (e.g. after manual key management). */ -export function clearKeyCooldowns(providerName?: string): void { - if (!providerName) { - keyCooldowns.clear(); - return; - } - const prefix = `${providerName}\0`; - for (const key of keyCooldowns.keys()) { - if (key.startsWith(prefix)) keyCooldowns.delete(key); - } -} - -/** Visible-for-testing: get the cooldown-until timestamp for a key. */ -export function getKeyCooldownUntil(providerName: string, keyId: string, now = Date.now()): number | null { - const entry = keyCooldowns.get(cooldownKey(providerName, keyId)); - if (!entry) return null; - return entry.cooldownUntil > now ? entry.cooldownUntil : null; -} diff --git a/src/providers/request-pacing.ts b/src/providers/request-pacing.ts index 576c49cfc2e..4d27f25f2a6 100644 --- a/src/providers/request-pacing.ts +++ b/src/providers/request-pacing.ts @@ -1,4 +1,5 @@ import type { OcxProviderConfig, RequestPacingRule } from "../types"; +import { createHash } from "node:crypto"; import type { GenerationContext } from "../lib/state-store-sweeper"; import { carryReplayRefusal, isNonReplayableResponse, markResponseNonReplayable } from "../lib/upstream-retry"; import { redactSecretString } from "../lib/redact"; @@ -39,7 +40,7 @@ export function registerProviderResponseProvenance(markers: WeakMap; + keyInFlight: Map; + keyStarts: Map; queue: Waiter[]; providerNextStartAt: number; modelNextStartAt: Map; @@ -138,6 +146,7 @@ export interface ProviderRequestPacingStatus { inFlight?: number; lastStartedAt?: number; lastModelId?: string; + keys?: Array<{ id: string; starts: number; inFlight: number; queued: number; nextSlotInMs: number }>; } const pacers = new Map(); @@ -165,6 +174,13 @@ function normalizedInterval(rule: RequestPacingRule | undefined): number { return Math.max(rpmInterval, fixedInterval); } +function keyBucket(provider: OcxProviderConfig): string | undefined { + if (provider.authMode === "oauth" || provider.authMode === "forward" || !provider.apiKey) return; + if (!provider.requestPacing?.enabled || (!provider.requestPacing.keys && provider.requestPacing.keySelection !== "round-robin")) return; + // Internal only: duplicate references share one bucket; never serialize a key or digest. + return createHash("sha256").update(provider._apiKeyAttempt?.reference ?? provider.apiKey).digest("hex"); +} + function normalizedMaxConcurrent(rule: RequestPacingRule | undefined): number { return typeof rule?.maxConcurrentRequests === "number" && rule.maxConcurrentRequests > 0 // Floor so runtime-injected configs that bypass the integer schema cannot admit @@ -178,10 +194,14 @@ export function requestPacingIntervalMs(provider: OcxProviderConfig, modelId?: s const policy = provider.requestPacing; if (!policy?.enabled) return 0; const override = modelId ? policy.models?.[modelId] : undefined; - return Math.max(normalizedInterval(policy), normalizedInterval(override)); + return Math.max(normalizedInterval(policy), normalizedInterval(override), keyBucket(provider) ? normalizedInterval(policy.keys) : 0); } function requestPacingIntervals(provider: OcxProviderConfig, modelId?: string): { + keyLease: boolean; + keyId?: string; + keyIntervalMs: number; + keyMaxConcurrent: number; providerIntervalMs: number; modelIntervalMs: number; providerMaxConcurrent: number; @@ -189,9 +209,14 @@ function requestPacingIntervals(provider: OcxProviderConfig, modelId?: string): } { const policy = provider.requestPacing; if (!policy?.enabled) { - return { providerIntervalMs: 0, modelIntervalMs: 0, providerMaxConcurrent: 0, modelMaxConcurrent: 0 }; + return { keyLease: false, keyIntervalMs: 0, keyMaxConcurrent: 0, providerIntervalMs: 0, modelIntervalMs: 0, providerMaxConcurrent: 0, modelMaxConcurrent: 0 }; } + const keyId = keyBucket(provider); return { + keyLease: true, + keyId, + keyIntervalMs: keyId ? normalizedInterval(policy.keys) : 0, + keyMaxConcurrent: keyId ? normalizedMaxConcurrent(policy.keys) : 0, providerIntervalMs: normalizedInterval(policy), modelIntervalMs: modelId ? normalizedInterval(policy.models?.[modelId]) : 0, providerMaxConcurrent: normalizedMaxConcurrent(policy), @@ -209,13 +234,14 @@ function requestPacingIntervals(provider: OcxProviderConfig, modelId?: string): */ export function requestPacingMaxConcurrentRequests(provider: OcxProviderConfig, modelId?: string): number { const limits = requestPacingIntervals(provider, modelId); - return Math.max(limits.providerMaxConcurrent, limits.modelMaxConcurrent); + return Math.max(limits.providerMaxConcurrent, limits.modelMaxConcurrent, limits.keyMaxConcurrent); } -function waiterReadyAt(state: ProviderPacer, modelId: string | undefined): number { +function waiterReadyAt(state: ProviderPacer, modelId: string | undefined, keyId?: string): number { return Math.max( state.providerNextStartAt, modelId ? (state.modelNextStartAt.get(modelId) ?? 0) : 0, + keyId ? (state.keyNextStartAt.get(keyId) ?? 0) : 0, ); } @@ -224,8 +250,9 @@ function pacingRetryAfterSeconds( modelId: string | undefined, now: number, concurrencyCapped = false, + keyId?: string, ): number { - const intervalSeconds = Math.max(1, Math.ceil(Math.max(0, waiterReadyAt(state, modelId) - now) / 1000)); + const intervalSeconds = Math.max(1, Math.ceil(Math.max(0, waiterReadyAt(state, modelId, keyId) - now) / 1000)); // A waiter blocked by an in-flight lease has no interval to report: when the wait ends // in refusal, the honest answer is that the cap is saturated, not "retry every second". return concurrencyCapped @@ -240,7 +267,7 @@ function makeProviderRequestSlot( ): BodyTrackableProviderRequestSlot { let released = false; let bodyTracked = false; - const leased = waiter.providerMaxConcurrent > 0 || waiter.modelMaxConcurrent > 0; + const leased = (!!waiter.keyId && waiter.keyLease) || waiter.providerMaxConcurrent > 0 || waiter.modelMaxConcurrent > 0; const slot: BodyTrackableProviderRequestSlot = { providerName, leased, @@ -258,6 +285,7 @@ function makeProviderRequestSlot( released = true; waiter.signal?.removeEventListener("abort", releaseOnAbort); if (waiter.providerMaxConcurrent > 0 && state.providerInFlight > 0) state.providerInFlight -= 1; + if (waiter.keyId && waiter.keyLease) state.keyInFlight.set(waiter.keyId, Math.max(0, (state.keyInFlight.get(waiter.keyId) ?? 0) - 1)); if (waiter.modelId && waiter.modelMaxConcurrent > 0) { const current = state.modelInFlight.get(waiter.modelId) ?? 0; if (current <= 1) state.modelInFlight.delete(waiter.modelId); @@ -293,7 +321,8 @@ function rejectExpiredWaiters(providerName: string, state: ProviderPacer, now: n state, waiter.modelId, now, - waiter.providerMaxConcurrent > 0 || waiter.modelMaxConcurrent > 0, + waiter.providerMaxConcurrent > 0 || waiter.modelMaxConcurrent > 0 || waiter.keyMaxConcurrent > 0, + waiter.keyId, ), )); } @@ -327,17 +356,18 @@ function runQueue(providerName: string, state: ProviderPacer): void { const providerReadyAt = Math.max(now, state.providerNextStartAt); const waiterIndex = state.queue.findIndex(waiter => { + if (waiter.keyId && waiter.keyMaxConcurrent > 0 && (state.keyInFlight.get(waiter.keyId) ?? 0) >= waiter.keyMaxConcurrent) return false; if (waiter.providerMaxConcurrent > 0 && state.providerInFlight >= waiter.providerMaxConcurrent) return false; if (waiter.modelId && waiter.modelMaxConcurrent > 0 && (state.modelInFlight.get(waiter.modelId) ?? 0) >= waiter.modelMaxConcurrent) return false; const modelReadyAt = waiter.modelId ? (state.modelNextStartAt.get(waiter.modelId) ?? 0) : 0; - return Math.max(providerReadyAt, modelReadyAt) <= now; + return Math.max(providerReadyAt, modelReadyAt, waiter.keyId ? state.keyNextStartAt.get(waiter.keyId) ?? 0 : 0) <= now; }); if (waiterIndex < 0) { let earliestAt = Number.POSITIVE_INFINITY; for (const waiter of state.queue) { const modelReadyAt = waiter.modelId ? (state.modelNextStartAt.get(waiter.modelId) ?? 0) : 0; - const readyAt = Math.max(providerReadyAt, modelReadyAt); + const readyAt = Math.max(providerReadyAt, modelReadyAt, waiter.keyId ? state.keyNextStartAt.get(waiter.keyId) ?? 0 : 0); const expiresAt = waiter.queuedAt + maxQueueAgeMs; // A waiter whose start time already passed is blocked on an in-flight lease; its release // re-runs this queue through a microtask, so the timer only needs its expiry backstop. @@ -362,6 +392,11 @@ function runQueue(providerName: string, state: ProviderPacer): void { state.lastStartedAt = startedAt; state.lastModelId = waiter.modelId; state.providerNextStartAt = startedAt + waiter.providerIntervalMs; + if (waiter.keyId) { + state.keyNextStartAt.set(waiter.keyId, startedAt + waiter.keyIntervalMs); + if (waiter.keyLease) state.keyInFlight.set(waiter.keyId, (state.keyInFlight.get(waiter.keyId) ?? 0) + 1); + state.keyStarts.set(waiter.keyId, (state.keyStarts.get(waiter.keyId) ?? 0) + 1); + } if (waiter.modelId && waiter.modelIntervalMs > 0) { state.modelNextStartAt.set(waiter.modelId, startedAt + waiter.modelIntervalMs); } @@ -386,22 +421,34 @@ export async function waitForProviderRequestSlot( // by interval only, or a follow-up would queue behind the lease its own turn holds. const concurrency = options?.concurrency !== false; const waiterIntervals = concurrency ? intervals : { + keyId: intervals.keyId, keyLease: false, keyIntervalMs: intervals.keyIntervalMs, keyMaxConcurrent: 0, providerIntervalMs: intervals.providerIntervalMs, modelIntervalMs: intervals.modelIntervalMs, providerMaxConcurrent: 0, modelMaxConcurrent: 0, }; const paced = Math.max(intervals.providerIntervalMs, intervals.modelIntervalMs) > 0 + || intervals.keyId !== undefined || waiterIntervals.providerMaxConcurrent > 0 || waiterIntervals.modelMaxConcurrent > 0; if (!paced) return inertProviderRequestSlot; if (signal?.aborted) throw abortReason(signal); const state = pacers.get(providerName) ?? { + keyNextStartAt: new Map(), keyInFlight: new Map(), keyStarts: new Map(), queue: [], providerNextStartAt: 0, modelNextStartAt: new Map(), providerInFlight: 0, modelInFlight: new Map(), }; pacers.set(providerName, state); + // Retire removed credentials only after their queued work, leases and spacing debt end. + const currentKeys = new Set((provider.apiKeyPool ?? []).map(entry => + keyBucket({ ...provider, apiKey: entry.key, _apiKeyAttempt: undefined }))); + currentKeys.add(intervals.keyId); + for (const key of state.keyStarts.keys()) { + if (currentKeys.has(key) || (state.keyInFlight.get(key) ?? 0) > 0 + || (state.keyNextStartAt.get(key) ?? 0) > runtime.now() || state.queue.some(waiter => waiter.keyId === key)) continue; + state.keyStarts.delete(key); state.keyInFlight.delete(key); state.keyNextStartAt.delete(key); + } // Give already-eligible or expired waiters a chance to leave before applying the // admission bound to the newest request. This preserves FIFO-ish fairness while @@ -419,7 +466,8 @@ export async function waitForProviderRequestSlot( state, modelId, runtime.now(), - waiterIntervals.providerMaxConcurrent > 0 || waiterIntervals.modelMaxConcurrent > 0, + waiterIntervals.providerMaxConcurrent > 0 || waiterIntervals.modelMaxConcurrent > 0 || waiterIntervals.keyMaxConcurrent > 0, + waiterIntervals.keyId, ), ); } @@ -464,14 +512,13 @@ export async function waitForProviderRequestSlot( * means the exchange is already finished, so the lease returns immediately. A body that * is neither read nor cancelled for REQUEST_PACING_UNCONSUMED_BODY_MS releases its lease * and cancels the body, so a dropped Response cannot hold a slot forever. The returned - * Response preserves status, statusText, headers, and the identity-based replay markers. + * Response preserves status, statusText, headers, replay markers, and registered provenance. */ export function trackProviderRequestSlotBody( slot: ProviderRequestSlot | undefined, response: Response, ): Response { - // An unleased slot has nothing to return on body close, and rewrapping the Response - // would break identity-based markers (the eager WS relay registry is a WeakSet). + // An unleased slot has nothing to return on body close; preserve the response identity. if (!slot?.leased) return response; // Slots built outside waitForProviderRequestSlot (test doubles) satisfy only the public // ProviderRequestSlot shape; their bodies still own the release through the wrapper below. @@ -657,24 +704,35 @@ export function providerRequestPacingStatus( if (state && state.queue.length > 0) { let earliestQueuedSlotAt = Number.POSITIVE_INFINITY; for (const waiter of state.queue) { - earliestQueuedSlotAt = Math.min(earliestQueuedSlotAt, waiterReadyAt(state, waiter.modelId)); + earliestQueuedSlotAt = Math.min(earliestQueuedSlotAt, waiterReadyAt(state, waiter.modelId, waiter.keyId)); } if (Number.isFinite(earliestQueuedSlotAt)) nextSlotAt = earliestQueuedSlotAt; } - const providerConcurrencyCap = requestPacingMaxConcurrentRequests(provider); + const providerConcurrencyCap = normalizedMaxConcurrent(provider.requestPacing); + const hasKeyAccounting = provider.requestPacing?.enabled === true && keyBucket(provider) !== undefined; // Gate on enabled so the status surface agrees with enforcement: a disabled requestPacing // block must not report in-flight leases alongside enabled: false. - const anyConcurrencyCap = provider.requestPacing?.enabled === true && (providerConcurrencyCap > 0 + const anyConcurrencyCap = provider.requestPacing?.enabled === true && (hasKeyAccounting || providerConcurrencyCap > 0 || Object.values(provider.requestPacing?.models ?? {}).some(rule => normalizedMaxConcurrent(rule) > 0)); return { provider: providerName, enabled: provider.requestPacing?.enabled === true, queued: state?.queue.length ?? 0, nextSlotInMs: Math.max(0, Math.ceil(nextSlotAt - now)), + ...(provider.requestPacing?.enabled && (provider.requestPacing.keys || provider.requestPacing.keySelection === "round-robin") ? { + keys: (provider.apiKeyPool ?? []).map(entry => { + const bucket = keyBucket({ ...provider, apiKey: entry.key, _apiKeyAttempt: undefined })!; + return { id: entry.id, starts: state?.keyStarts.get(bucket) ?? 0, inFlight: state?.keyInFlight.get(bucket) ?? 0, + queued: state?.queue.filter(waiter => waiter.keyId === bucket).length ?? 0, + nextSlotInMs: Math.max(0, Math.ceil((state?.keyNextStartAt.get(bucket) ?? 0) - now)) }; + }), + } : {}), // A provider cap counts every paced send in providerInFlight; model-only caps keep // providerInFlight at zero, so report the per-model sum for those providers instead. ...(anyConcurrencyCap ? { - inFlight: providerConcurrencyCap > 0 + inFlight: hasKeyAccounting + ? [...(state?.keyInFlight.values() ?? [])].reduce((sum, count) => sum + count, 0) + : providerConcurrencyCap > 0 ? state?.providerInFlight ?? 0 : [...(state?.modelInFlight.values() ?? [])].reduce((sum, count) => sum + count, 0), } : {}), diff --git a/src/server/chat-native.ts b/src/server/chat-native.ts index cebac32464c..595f3716117 100644 --- a/src/server/chat-native.ts +++ b/src/server/chat-native.ts @@ -54,9 +54,11 @@ import { rateLimitRetryDelayMs, rateLimitRetryPolicyFor, rotateProviderTransportOn429, + rotateProviderTransportOn401, transientRetryPolicyFor, } from "../providers/key-failover"; import { fastPolicyForModel } from "../providers/service-tier"; +import { RequestPacingQueueOverloadError } from "../providers/request-pacing"; import { stampApiKeyAccountLabel } from "../providers/label"; import { providerApiKeySelectionIsCurrent, resolveCurrentProviderApiKeyTransport } from "../providers/api-key-selection"; import { enrichOpenCodeZenFreeTierMessage } from "../providers/opencode-zen-rate-limit"; @@ -276,8 +278,10 @@ export async function runNativeChatAttempt( const { req, config, logCtx, logIds, route, requestedModel, requestedStream, translatorBudget, finishLog } = execution; const { sendBudget } = execution; const { attempt } = attemptHandle; - const onFirstOutput = execution.onFirstOutput + let outputDelivered = false; + const notifyFirstOutput = execution.onFirstOutput ?? (logIds ? () => recordFirstOutput(logCtx, logIds.start) : undefined); + const onFirstOutput = () => { outputDelivered = true; notifyFirstOutput?.(); }; const fail = (status: number, message: string, type?: string, code?: string | null): Response => { const safeMessage = redactSecretString(message); finishLog(status, safeMessage); @@ -313,8 +317,16 @@ export async function runNativeChatAttempt( // assigned before the transport pin in core.ts. // Transport variant: the bare picker answers with the persisted row, which for a built-in // provider carries no adapter id or base URL until routedProviderConfig backfills it. - const proactiveKeyProvider = selectProactiveApiKeyTransport(config, route.providerName, route.provider); - if (proactiveKeyProvider) route.provider = proactiveKeyProvider; + try { + const proactiveKeyProvider = selectProactiveApiKeyTransport(config, route.providerName, route.provider); + if (proactiveKeyProvider) route.provider = proactiveKeyProvider; + } catch (error) { + cleanupAbort(); upstream.abort(); + if (!(error instanceof RequestPacingQueueOverloadError)) throw error; + const refusal = fail(429, error.message, "rate_limit_error"); + refusal.headers.set("Retry-After", String(error.retryAfterSeconds)); + return refusal; + } let activeProvider: OcxProviderConfig = route.provider; stampApiKeyAccountLabel(logCtx, route.providerName, activeProvider); const spendTracker = sendBudget ? undefined : attachRequestSpendTracker(req, logCtx); @@ -421,9 +433,10 @@ export async function runNativeChatAttempt( } else if (!spendTracker?.charge()) throw new NativeChatSpendRefusal(); }; + let responseOwnsDistributedKey = true; const send = async ( request: AdapterRequest, - recovery?: "rate-limit-429" | "key-429" | UpstreamSendRecovery, + recovery?: "rate-limit-429" | "key-429" | "key-401" | UpstreamSendRecovery, singleShot = false, ): Promise => { try { @@ -453,9 +466,16 @@ export async function runNativeChatAttempt( onPhysicalDispatch: comboReceipt ? () => { if (!physicalBudget.spendEnforced) settleNonEnforcedSend(request.url); } : undefined, - dispatchOverride: async (_input, init, execute) => { - if (!providerApiKeySelectionIsCurrent(config, route.providerName, activeProvider)) { - const current = resolveCurrentProviderApiKeyTransport(config, route.providerName, activeProvider); + dispatchOverride: async (_input, init, execute, reacquirePacing) => { + for (let attempt = 0; !providerApiKeySelectionIsCurrent(config, route.providerName, activeProvider); attempt++) { + if (outputDelivered) throw new Error("Provider key selection is unavailable during the native Chat turn"); + if (attempt >= 3) throw new Error("Provider key selection changed repeatedly before native Chat dispatch"); + let current = resolveCurrentProviderApiKeyTransport(config, route.providerName, activeProvider); + if (!current && activeProvider.requestPacing?.enabled + && (activeProvider.requestPacing.keys || activeProvider.requestPacing.keySelection === "round-robin")) { + const fresh = resolveCurrentProviderApiKeyTransport(config, route.providerName, { ...activeProvider, _apiKeyAttempt: undefined }); + if (fresh) current = selectProactiveApiKeyTransport(config, route.providerName, fresh) ?? fresh; + } if (!current || !isNativeChatRouteEligible({ ...route, provider: current }, execution.chatBody, config)) { throw new Error("Provider key selection is no longer available for native Chat"); } @@ -467,6 +487,11 @@ export async function runNativeChatAttempt( activeRequest = buildActiveRequest(); try { retainRequest(activeRequest); } catch (error) { activeRequest.releaseBodyObservation?.(); throw error; } + if (current.requestPacing?.enabled + && (current.requestPacing.keys || current.requestPacing.keySelection === "round-robin")) { + if (!reacquirePacing) throw new Error("Refreshed native Chat dispatch requires pacing readmission"); + await reacquirePacing(current, route.modelId); + } } // The retry closure may still hold a pre-pacing request. Replace its entire // wire shape, not just Authorization, and retain transport recovery flags. @@ -478,6 +503,8 @@ export async function runNativeChatAttempt( const spendReport = createPhysicalSendReporter(physicalBudget, () => ({ poolId: logCtx.spendPoolId ?? route.providerName, identityId: logCtx.accountLogLabel, }), execution.comboDispatchPermit); + const ownsDistributedKey = !activeProvider._apiKeyAttempt?.distributed + || headers.get("authorization") === `Bearer ${activeProvider.apiKey}`; let dispatched: Response; try { // Capture this request's spend policy at its first physical boundary, including @@ -510,6 +537,7 @@ export async function runNativeChatAttempt( try { if (physicalBudget.spendEnforced) spendReport(1); } finally { spendReport.close?.(); } } + responseOwnsDistributedKey = ownsDistributedKey; if (!dispatched.ok) await recordKeyAttemptFailure(logCtx, dispatched, init.signal ?? upstream.signal); return dispatched; }, @@ -566,8 +594,11 @@ export async function runNativeChatAttempt( } // Same reason as above, plus a second one: rotating here would write a cooldown against // a key that rate-limited nothing, and that false signal outlives the request. - while (response.status === 429 && !isNonReplayableResponse(response) && hasKeyPoolFailover(activeProvider)) { - const rotated = rotateProviderTransportOn429(config, route.providerName, activeProvider, { + while ((response.status === 429 || (response.status === 401 && activeProvider._apiKeyAttempt?.distributed)) + && responseOwnsDistributedKey && !isNonReplayableResponse(response) && hasKeyPoolFailover(activeProvider)) { + const failureStatus = response.status; + const rotate = failureStatus === 401 ? rotateProviderTransportOn401 : rotateProviderTransportOn429; + const rotated = rotate(config, route.providerName, activeProvider, { retryAfter: response.headers.get("retry-after"), now: Date.now(), attemptedKey: activeProvider.apiKey, @@ -586,7 +617,7 @@ export async function runNativeChatAttempt( releaseRetainedRequest(); activeRequest = buildActiveRequest(); retainRequest(activeRequest); - response = await send(activeRequest, "key-429"); + response = await send(activeRequest, failureStatus === 401 ? "key-401" : "key-429"); } } catch (error) { releaseRetainedRequest(); @@ -594,6 +625,11 @@ export async function runNativeChatAttempt( upstream.abort(); if (req.signal.aborted) return fail(499, "Client cancelled request", "client_cancelled"); const sendError = error instanceof UpstreamRetryEvidenceError ? error.cause : error; + if (sendError instanceof RequestPacingQueueOverloadError) { + const refusal = fail(429, sendError.message, "rate_limit_error"); + refusal.headers.set("Retry-After", String(sendError.retryAfterSeconds)); + return refusal; + } if (sendError instanceof SendBudgetExhaustedError) { // A decision this process made, answered as the Responses path answers it: 429, not 502. return fail(429, sendError.message, SEND_BUDGET_EXHAUSTED_CODE, SEND_BUDGET_EXHAUSTED_CODE); diff --git a/src/server/images.ts b/src/server/images.ts index d971538555c..302e49a494c 100644 --- a/src/server/images.ts +++ b/src/server/images.ts @@ -27,9 +27,11 @@ import { formatCodexProviderForLog } from "../codex/routing"; import { signalWithTimeout } from "../lib/abort"; import { readBoundedResponseBytes, type BoundedBytesResult } from "../lib/bounded-body"; import { sidecarEnter } from "../lib/sidecar-tracker"; +import { isNonReplayableResponse } from "../lib/upstream-retry"; import type { OcxConfig } from "../types"; import { resolveFirstUsableOpenAiSidecar, selectImagesProvider } from "../providers/openai-sidecar"; -import { selectProactiveApiKeyTransport } from "../providers/key-failover"; +import { recordDistributedKeyFailure, selectProactiveApiKeyTransport } from "../providers/key-failover"; +import { requestPacingOverloadResponse } from "./responses/pacing-overload"; import { getProviderRegistryEntry } from "../providers/registry"; import { readJsonRequestBody, resolveInboundBodyLimitBytes } from "./request-decompress"; import { ForwardAdmissionCredentialError, validateForwardAdmissionCredential } from "./auth-cors"; @@ -756,6 +758,7 @@ export async function handleImages( catch { console.error("[images] Failed to release probe lease"); } }; let url: string; + let recordKeyFailure = (_status: 401 | 429, _retryAfter: string | null) => {}; // Each branch is re-checked as it is entered: the forward check is defence in // depth for the pre-filter above, and the keyed check runs before a key // rotation is committed. @@ -805,8 +808,15 @@ export async function handleImages( // // Transport variant: this branch reads `provider.baseUrl` and `provider.headers` to build // the URL and the request, and it comes back with the credential already resolved. - const warmKeyProvider = selectProactiveApiKeyTransport(config, providerName, candidates.keyed.provider); + let warmKeyProvider; + try { warmKeyProvider = selectProactiveApiKeyTransport(config, providerName, candidates.keyed.provider); } + catch (error) { + const refusal = requestPacingOverloadResponse(error); + if (refusal) return refusal; + throw error; + } const provider = warmKeyProvider ?? candidates.keyed.provider; + recordKeyFailure = (status, retryAfter) => { recordDistributedKeyFailure(config, providerName, provider._apiKeyAttempt, status, Date.now(), retryAfter); }; // No fall back to the earlier snapshot once a pick has happened. The picker COMMITS its // choice before returning, so if the chosen reference will not resolve -- revoked keychain // entry, unset env var -- sending the previous key would authenticate a non-idempotent @@ -865,6 +875,9 @@ export async function handleImages( // compact already set this; the credential-bearing sidecars did not. redirect: "manual", }); + if ((upstreamResponse.status === 401 || upstreamResponse.status === 429) && !isNonReplayableResponse(upstreamResponse)) { + recordKeyFailure(upstreamResponse.status, upstreamResponse.headers.get("retry-after")); + } const observed = await readImageResponseBytes(upstreamResponse, { maxBytes: IMAGES_RESPONSE_MAX_BYTES, signal: linkedSignal.signal, diff --git a/src/server/management/provider-routes.ts b/src/server/management/provider-routes.ts index fd0c1e68df0..befc4876425 100644 --- a/src/server/management/provider-routes.ts +++ b/src/server/management/provider-routes.ts @@ -77,7 +77,7 @@ import { PROVIDER_QUOTA_MAX_AGE_MS, type ProviderRoutingQuota } from "../../prov import { cachedProviderQuotaIsExhausted } from "../../combos/resolve"; import { probeJevDecisionProvider } from "../../combos/jev"; import { comboDependsOnProviderRoute } from "./decision-model-validation"; -import { clearKeyCooldowns, forgetApiKeyRotationCursor } from "../../providers/key-failover"; +import { clearKeyCooldowns, forgetApiKeyRotationCursor, getKeyCooldownUntil } from "../../providers/key-failover"; import { providerRequestPacingStatus } from "../../providers/request-pacing"; import { CODEX_FORWARD_BASE_URL, isCanonicalOpenAiForwardProvider } from "../../providers/openai-tiers"; import { codexAccountNamespaceProviderCollisionError } from "../../codex/account-namespace-match"; @@ -896,17 +896,22 @@ export async function handleProviderRoutes(ctx: ManagementContext): Promise { + const status = providerRequestPacingStatus(name, provider); + if (status.keys) status.keys = status.keys.map(row => ({ ...row, cooldownUntil: getKeyCooldownUntil(name, row.id) })); + return status; + }; const name = url.searchParams.get("name")?.trim(); if (name) { if (!isValidProviderName(name) || !hasOwnProvider(config.providers, name)) { return jsonResponse({ error: "unknown provider" }, 404); } - return jsonResponse(providerRequestPacingStatus(name, config.providers[name]!)); + return jsonResponse(statusFor(name, config.providers[name]!)); } return jsonResponse(Object.fromEntries( Object.entries(config.providers).map(([providerName, provider]) => [ providerName, - providerRequestPacingStatus(providerName, provider), + statusFor(providerName, provider), ]), )); } diff --git a/src/server/messages-native.ts b/src/server/messages-native.ts index dab3432d8ba..ce0f09f3179 100644 --- a/src/server/messages-native.ts +++ b/src/server/messages-native.ts @@ -62,10 +62,12 @@ import { UPSTREAM_RESET_REPLAY_REFUSED_CODE, } from "../lib/upstream-retry"; import { providerApiKeySelectionIsCurrent, resolveCurrentProviderApiKeyTransport } from "../providers/api-key-selection"; +import { RequestPacingQueueOverloadError } from "../providers/request-pacing"; import { hasKeyPoolFailover, rateLimitRetryDelayMs, rateLimitRetryPolicyFor, + recordDistributedKeyFailure, rotateProviderTransportOn401, rotateProviderTransportOn429, selectProactiveApiKeyTransport, @@ -436,8 +438,16 @@ export async function handleNativeMessages(options: HandleNativeMessagesOptions) } } else { // Same pre-dispatch key preference every direct send path applies (see chat-native.ts). - const proactiveKeyProvider = selectProactiveApiKeyTransport(config, route.providerName, route.provider); - if (proactiveKeyProvider) route.provider = proactiveKeyProvider; + try { + const proactiveKeyProvider = selectProactiveApiKeyTransport(config, route.providerName, route.provider); + if (proactiveKeyProvider) route.provider = proactiveKeyProvider; + } catch (error) { + cleanupAbort(); upstream.abort(); + if (!(error instanceof RequestPacingQueueOverloadError)) throw error; + const refusal = fail(429, error.message, "rate_limit_error"); + refusal.headers.set("Retry-After", String(error.retryAfterSeconds)); + return refusal; + } } let activeProvider: OcxProviderConfig = oauthBinding ? oauthProvider(oauthBinding) : route.provider; stampApiKeyAccountLabel(logCtx, route.providerName, activeProvider); @@ -521,6 +531,15 @@ export async function handleNativeMessages(options: HandleNativeMessagesOptions) && nativeMessagesDeclineReason({ ...route, provider: routed }, body, config, selector) === undefined; }; + // Keep the actual failed wire identity if a later recovery is refused before sending. + // A 429 remains eligible for explicit same-key retries until that ladder is finished. + let pendingDistributedFailure: (() => void) | undefined; + let distributedResponseOwnsKey = true; + const settleDistributedFailure = () => { + const record = pendingDistributedFailure; + pendingDistributedFailure = undefined; + record?.(); + }; const send = async (recovery?: "rate-limit-429" | "oauth-401" | "oauth-account-403" | "key-429" | "key-401"): Promise => { const remaining = remainingTransientSends(); if (requestTransientPolicy && remaining <= 0) { @@ -538,7 +557,7 @@ export async function handleNativeMessages(options: HandleNativeMessagesOptions) providerFetch(activeProvider, undefined, { providerName: route.providerName, modelId: route.modelId, - dispatchOverride: async (_input, init, execute) => { + dispatchOverride: async (_input, init, execute, reacquirePacing) => { if (oauthBinding) { if (!nativeOAuthRouteIsCurrent(oauthBinding)) throw new NativeOAuthSelectionChangedError(); // The OAuth twin of the key check below: re-resolve through the same selection @@ -555,12 +574,25 @@ export async function handleNativeMessages(options: HandleNativeMessagesOptions) rebuildFor(oauthProvider(oauthBinding)); logCtx.provider = formatAnthropicProviderForLog(route.providerName, oauthBinding.snapshot.accountId, config); } - } else if (!providerApiKeySelectionIsCurrent(config, route.providerName, activeProvider)) { - const current = resolveCurrentProviderApiKeyTransport(config, route.providerName, activeProvider); - if (!current || nativeMessagesDeclineReason({ ...route, provider: current }, body, config, selector) !== undefined) { - throw new Error("Provider key selection is no longer available for native Messages"); + } else { + for (let attempt = 0; !providerApiKeySelectionIsCurrent(config, route.providerName, activeProvider); attempt++) { + if (attempt >= 3) throw new Error("Provider key selection changed repeatedly before native Messages dispatch"); + let current = resolveCurrentProviderApiKeyTransport(config, route.providerName, activeProvider); + if (!current && activeProvider.requestPacing?.enabled + && (activeProvider.requestPacing.keys || activeProvider.requestPacing.keySelection === "round-robin")) { + const fresh = resolveCurrentProviderApiKeyTransport(config, route.providerName, { ...activeProvider, _apiKeyAttempt: undefined }); + if (fresh) current = selectProactiveApiKeyTransport(config, route.providerName, fresh) ?? fresh; + } + if (!current || nativeMessagesDeclineReason({ ...route, provider: current }, body, config, selector) !== undefined) { + throw new Error("Provider key selection is no longer available for native Messages"); + } + rebuildFor(current); + if (current.requestPacing?.enabled + && (current.requestPacing.keys || current.requestPacing.keySelection === "round-robin")) { + if (!reacquirePacing) throw new Error("Refreshed native Messages dispatch requires pacing readmission"); + await reacquirePacing(current, route.modelId); + } } - rebuildFor(current); } // The retry closure may hold a pre-reselection request: send the current one whole. const wire = activeRequest; @@ -579,6 +611,11 @@ export async function handleNativeMessages(options: HandleNativeMessagesOptions) } const releaseFamily = snapshot && oauthBinding ? anthropicModelQuotaFor(oauthBinding.instance).claimAnthropicFamilyRevalidation(snapshot.accountId, route.modelId) : () => {}; if (!releaseFamily) throw new AnthropicAccountCooldownError(1); + const selection = activeProvider._apiKeyAttempt; + const ownsDistributedKey = !selection?.distributed + || (activeProvider.apiKeyTransport === "bearer" + ? headers.get("authorization") === `Bearer ${activeProvider.apiKey}` && !headers.has("x-api-key") + : headers.get("x-api-key") === activeProvider.apiKey && !headers.has("authorization")); let dispatched: Response; const spendReport = createPhysicalSendReporter(physicalBudget, () => ({ poolId: logCtx.spendPoolId ?? route.providerName, identityId: logCtx.accountLogLabel, @@ -623,6 +660,13 @@ export async function handleNativeMessages(options: HandleNativeMessagesOptions) } } catch { /* Passive observation must not fail the response. */ } } + distributedResponseOwnsKey = ownsDistributedKey; + const failureStatus = dispatched.status; + pendingDistributedFailure = selection?.distributed && ownsDistributedKey && (failureStatus === 401 || failureStatus === 429) + && !isNonReplayableResponse(dispatched) + ? () => { recordDistributedKeyFailure(config, route.providerName, selection, failureStatus, + Date.now(), dispatched.headers.get("retry-after")); } + : undefined; if (!dispatched.ok) await recordKeyAttemptFailure(logCtx, dispatched, init.signal ?? upstream.signal); return dispatched; }, @@ -647,25 +691,28 @@ export async function handleNativeMessages(options: HandleNativeMessagesOptions) let response: Response; try { response = await send(); - const retryPolicy = oauthBinding ? undefined : rateLimitRetryPolicyFor(activeProvider); + let retryPolicy: ReturnType | undefined; let retries = 0; let billingRecoveryActive = false; const keySendAvailable = () => transientSendAvailable() && (!billingRecoveryActive || Math.max(physicalSendsUsed, physicalBudget.used) < physicalBudget.policy.maxTotalModelSends); - for (;;) { + const recoveryPasses = activeProvider._apiKeyAttempt?.distributed ? activeProvider.apiKeyPool?.length ?? 1 : 1; + for (let pass = 0; ; pass++) { const priorResponse = response; // A credential-scoped 401 on a static key pool says nothing about the sibling keys. - while (response.status === 401 && hasKeyPoolFailover(activeProvider) && keySendAvailable()) { + while (response.status === 401 && distributedResponseOwnsKey && hasKeyPoolFailover(activeProvider) && keySendAvailable()) { const rotated = rotateProviderTransportOn401(config, route.providerName, activeProvider, { now: Date.now(), attemptedKey: activeProvider.apiKey, + attemptedSelection: activeProvider._apiKeyAttempt, }); + pendingDistributedFailure = undefined; // Rotation already records this physical failure. if (!rotated) break; discard(response); rebuildFor(rotated); response = await send("key-401"); } - while ((response.status === 400 || response.status === 402) && hasKeyPoolFailover(activeProvider)) { + while ((response.status === 400 || response.status === 402) && distributedResponseOwnsKey && hasKeyPoolFailover(activeProvider)) { const rotated = await rotateAnthropicBillingKeyOnResponse(response, config, route.providerName, activeProvider, { signal: upstream.signal }); if (!rotated) break; billingRecoveryActive = true; @@ -674,9 +721,11 @@ export async function handleNativeMessages(options: HandleNativeMessagesOptions) rebuildFor(rotated); response = await send("key-401"); } + if (pass === 0) retryPolicy = oauthBinding ? undefined : rateLimitRetryPolicyFor(activeProvider); // A refusal this proxy synthesized for a reset replay is not a provider rate limit. while ( response.status === 429 + && distributedResponseOwnsKey && !isNonReplayableResponse(response) && retryPolicy && retries < retryPolicy.attempts @@ -691,13 +740,14 @@ export async function handleNativeMessages(options: HandleNativeMessagesOptions) if (upstream.signal.aborted) throw upstream.signal.reason; response = await send("rate-limit-429"); } - while (response.status === 429 && !isNonReplayableResponse(response) && hasKeyPoolFailover(activeProvider)) { + while (response.status === 429 && distributedResponseOwnsKey && !isNonReplayableResponse(response) && hasKeyPoolFailover(activeProvider)) { const rotated = rotateProviderTransportOn429(config, route.providerName, activeProvider, { retryAfter: response.headers.get("retry-after"), now: Date.now(), attemptedKey: activeProvider.apiKey, attemptedSelection: activeProvider._apiKeyAttempt, }); + pendingDistributedFailure = undefined; // Rotation already records this physical failure. if (!rotated) break; // Rotation already recorded the cooldown; keep the terminal 429 when no send remains. if (!keySendAvailable()) break; @@ -705,13 +755,17 @@ export async function handleNativeMessages(options: HandleNativeMessagesOptions) rebuildFor(rotated); response = await send("key-429"); } - // A sibling's next refusal may change class; the same counters survive that transition. - if (response !== priorResponse && ( + if (response === priorResponse || isNonReplayableResponse(response) || !keySendAvailable()) break; + // Billing class transitions share the physical-send ceiling; distributed-only + // transitions additionally retain their original bounded pool-pass allowance. + if (distributedResponseOwnsKey && ( (billingRecoveryActive && [401, 429].includes(response.status)) || (activeProvider.adapter === "anthropic" && hasKeyPoolFailover(activeProvider) && [400, 402].includes(response.status)) )) continue; - break; + if (!activeProvider._apiKeyAttempt?.distributed || !distributedResponseOwnsKey + || (response.status !== 401 && response.status !== 429) || pass + 1 >= recoveryPasses) break; } + settleDistributedFailure(); const oauthRetryKey = {}; const triedAccountIds = new Set(); let oauthFailovers = 0; @@ -749,7 +803,13 @@ export async function handleNativeMessages(options: HandleNativeMessagesOptions) cleanupAbort(); upstream.abort(); if (req.signal.aborted) return fail(499, "Client cancelled request", "api_error"); + settleDistributedFailure(); const sendError = error instanceof UpstreamRetryEvidenceError ? error.cause : error; + if (sendError instanceof RequestPacingQueueOverloadError) { + const refusal = fail(429, sendError.message, "rate_limit_error"); + refusal.headers.set("Retry-After", String(sendError.retryAfterSeconds)); + return refusal; + } if (sendError instanceof SendBudgetExhaustedError) return fail(429, sendError.message, "send_budget_exhausted", "send_budget_exhausted"); if (sendError instanceof NativeMessagesSpendRefusal) { const refusal = workflowRefusalResponse("workflow-spend-exhausted", logCtx); diff --git a/src/server/responses/codex-ws-wire.ts b/src/server/responses/codex-ws-wire.ts index 59fe27f2699..f87ba0e881a 100644 --- a/src/server/responses/codex-ws-wire.ts +++ b/src/server/responses/codex-ws-wire.ts @@ -8,6 +8,7 @@ import { import { isRateLimitOrQuotaFailureMessage, recognizedUpstreamError } from "../../lib/errors"; import type { RequestFailureCause, RequestFailureStage } from "../../lib/request-failure-model"; import { packageVersion } from "../../lib/package-version"; +import { registerProviderResponseProvenance } from "../../providers/request-pacing"; // If the 101 never arrives (network black hole), give SSE a chance well before // the caller's connect timeout (default 200s) would fire. export const UPGRADE_DEADLINE_MS = 10_000; @@ -49,9 +50,12 @@ export const CODEX_WS_CREATE_FRAME_LIMIT_BYTES = /** Close code the backend uses for an oversized message (RFC 6455 "message too big"). */ const WS_CLOSE_MESSAGE_TOO_BIG = 1009; -const codexWsUpstreamResponses = new WeakSet(); -const quotaObservedResponses = new WeakSet(); -const codexWsPreludeProjections = new WeakSet(); +const codexWsUpstreamResponses = new WeakMap(); +const quotaObservedResponses = new WeakMap(); +const codexWsPreludeProjections = new WeakMap(); +registerProviderResponseProvenance(codexWsUpstreamResponses); +registerProviderResponseProvenance(quotaObservedResponses); +registerProviderResponseProvenance(codexWsPreludeProjections); /** Quota arrived directly at its captured account; do not replay old HTTP prelude headers. */ export function isCodexWsQuotaObservedResponse(response: Response): boolean { @@ -65,7 +69,7 @@ export function isCodexWsUpstreamResponse(response: Response): boolean { /** Pre-response projection; its headers retain the exchange's prelude snapshot. */ export function markCodexWsPreludeProjection(response: Response): void { - codexWsPreludeProjections.add(response); + codexWsPreludeProjections.set(response, true); } export function isCodexWsPreludeProjection(response: Response): boolean { @@ -73,8 +77,8 @@ export function isCodexWsPreludeProjection(response: Response): boolean { } export function markCodexWsResponse(response: Response, observed: boolean): void { - codexWsUpstreamResponses.add(response); - if (observed) quotaObservedResponses.add(response); + codexWsUpstreamResponses.set(response, true); + if (observed) quotaObservedResponses.set(response, true); } /** @@ -313,6 +317,7 @@ export function restoreCodexWsPreOutputRefusal(response: Response): Response { } const codexWsStageByResponse = new WeakMap(); +registerProviderResponseProvenance(codexWsStageByResponse); export function markCodexWsStage(response: Response, record: CodexWsStageRecord): void { const current = codexWsStageByResponse.get(response); @@ -457,6 +462,7 @@ export function projectCodexWsFailure( } const socketDeathStages = new WeakMap(); +registerProviderResponseProvenance(socketDeathStages); /** * Record that a pre-response settle came from the socket closing or failing under the send (#4191), diff --git a/src/server/responses/compact.ts b/src/server/responses/compact.ts index c3c74102035..c3c869ebef2 100644 --- a/src/server/responses/compact.ts +++ b/src/server/responses/compact.ts @@ -1,5 +1,5 @@ import { spendLedgerStorageErrorResponse } from "./spend-storage-error"; -import { createPhysicalSendReporter } from "../../lib/request-execution-budget"; +import { createPhysicalSendReporter, rebindPhysicalSend } from "../../lib/request-execution-budget"; import { createInferenceSendBudget } from "../inference/context"; import { unboundPoolSpendRefusalResponse } from "../workflow-refusal"; import { capturePoolQuotaWriter } from "../../codex/account-store"; @@ -151,7 +151,9 @@ import { UnsupportedContentEncodingError, } from "../request-decompress"; import { resolveAdapter, resolveWireProtocolOverride } from "../adapter-resolve"; -import { hasKeyPoolFailover, rotateProviderTransportOn429, selectProactiveApiKeyTransport } from "../../providers/key-failover"; +import { hasKeyPoolFailover, recordDistributedKeyFailure, rotateProviderTransportOn429, selectProactiveApiKeyTransport } from "../../providers/key-failover"; +import { providerApiKeySelectionIsCurrent, resolveCurrentProviderApiKeyTransport } from "../../providers/api-key-selection"; +import { stampApiKeyAccountLabel } from "../../providers/label"; import { shouldAttemptImageTierRetry } from "../image-retry"; import { resolveProviderTransport } from "../../providers/xai-transport"; import type { WsData } from "../ws-bridge"; @@ -168,6 +170,7 @@ import { catalogModelSupportsServiceTier, finishRequestAttempt, inspectResponseLogJson, + noteProviderAttemptSend, readConfiguredCodexServiceTier, requestLogSpeedLabel, sealRequestAttemptIdentity, @@ -199,7 +202,8 @@ import { upstreamHostCircuitOpenResponse, usesCodexForwardPoolAuth, } from "./core"; -import { fetchWithHeaderTimeout, providerFetch, safeHostLabel, safeOriginLabel } from "./fetch-helpers"; +import { fetchWithHeaderTimeout, providerFetch, safeHostLabel, safeOriginLabel, sendWithConnectionPolicy } from "./fetch-helpers"; +import { requestPacingOverloadResponse } from "./pacing-overload"; import { shouldRetryCodexPoolAccountQuota, shouldRetryCodexPoolAccountTransient } from "./core-codex-account"; import { mapCodexAuthContextErrorToResponse, nativeMainRefreshFailureResponse } from "./codex-auth-error"; import { linkRequestSessionLane, sessionLaneIdFromRequest } from "../request-log-conversation"; @@ -906,9 +910,16 @@ export async function handleResponsesCompact( // codexAccountMode cases, and the picker returns null for them anyway. // Transport variant, for the same reason as the Responses core: the persisted row has no // registry backfills, and compactProvider is read for its transport fields below. - const warmKeyProvider = selectProactiveApiKeyTransport(config, route.providerName, compactProvider); + let warmKeyProvider; + try { warmKeyProvider = selectProactiveApiKeyTransport(config, route.providerName, compactProvider); } + catch (error) { + const refusal = requestPacingOverloadResponse(error); + if (refusal) return refusal; + throw error; + } if (warmKeyProvider?.apiKey) compactProvider = warmKeyProvider; headers.set("authorization", `Bearer ${resolveProviderApiKey(compactProvider.apiKey)}`); + stampApiKeyAccountLabel(logCtx, route.providerName, compactProvider); } const { reasoning: _reasoning, ...compactBodyRaw } = raw as typeof raw & { reasoning?: unknown }; // The regular /v1/responses path applies sanitizeReasoningInputContent via the adapter's @@ -945,9 +956,9 @@ export async function handleResponsesCompact( }); } } - const compactUrl = `${base}/responses/compact`; + let compactUrl = `${base}/responses/compact`; const compactTargetKey = `${route.providerName}|${route.modelId}|compact`; - const actualCompactHostKey = upstreamHostHealthKey( + let actualCompactHostKey = upstreamHostHealthKey( route.providerName, safeOriginLabel(compactUrl), ); @@ -1038,7 +1049,7 @@ export async function handleResponsesCompact( try { if (report.beforeSend?.() === false) throw new SendBudgetExhaustedError(safeHostLabel(compactUrl)); started = true; - return await doFetch(); + return await (report.execute ? report.execute(doFetch) : doFetch()); } finally { try { if (started) report(1); } finally { report.close?.(); } @@ -1049,12 +1060,14 @@ export async function handleResponsesCompact( targetKey: compactTargetKey, }); if (!decision.allowed) return Promise.reject(new SendBudgetExhaustedError(safeHostLabel(compactUrl))); - if (!decision.permit.use()) { - // Nothing was sent, so the reservation goes back rather than holding a send against the request. - decision.permit.release(); - return Promise.reject(new SendBudgetExhaustedError(safeHostLabel(compactUrl))); - } - return doFetch(); + const dispatch = () => { + if (!decision.permit.use()) { + decision.permit.release(); + throw new SendBudgetExhaustedError(safeHostLabel(compactUrl)); + } + return doFetch(); + }; + return decision.permit.execute ? decision.permit.execute(dispatch) : dispatch(); }; const sendCompactAttempt = ( sendProvider: OcxProviderConfig, @@ -1078,6 +1091,41 @@ export async function handleResponsesCompact( modelId: route.modelId, beforeDispatch: isCanonicalOpenAiForwardProvider(sendProvider) ? createCodexAuthDispatchGuard(sendAuthCtx, config, selectedModelId, admission) : undefined, + dispatchOverride: sendProvider.authMode !== "forward" && sendProvider.apiKey + ? async (_input, init, execute, reacquirePacing) => { + for (let attempt = 0; !providerApiKeySelectionIsCurrent(config, route.providerName, sendProvider); attempt++) { + if (attempt >= 3) throw new Error("Provider key selection changed repeatedly before compact dispatch"); + let current = resolveCurrentProviderApiKeyTransport(config, route.providerName, sendProvider); + if (!current && sendProvider.requestPacing?.enabled + && (sendProvider.requestPacing.keys || sendProvider.requestPacing.keySelection === "round-robin")) { + const fresh = resolveCurrentProviderApiKeyTransport(config, route.providerName, { ...sendProvider, _apiKeyAttempt: undefined }); + if (fresh) current = selectProactiveApiKeyTransport(config, route.providerName, fresh) ?? fresh; + } + if (!current?.apiKey || !supportsNativeResponsesCompactEndpoint(route.providerName, current)) { + throw new Error("Provider key selection is no longer available for compact"); + } + const rebuilt = new Headers(sendHeaders); + for (const name of Object.keys(sendProvider.headers ?? {})) rebuilt.delete(name); + for (const [name, value] of Object.entries(current.headers ?? {})) rebuilt.set(name, value); + rebuilt.set("authorization", `Bearer ${resolveProviderApiKey(current.apiKey)}`); + sendHeaders = headers = rebuilt; + sendProvider = compactProvider = current; + compactUrl = `${(current.baseUrl ?? "").replace(/\/+$/, "")}/responses/compact`; + actualCompactHostKey = upstreamHostHealthKey(route.providerName, safeOriginLabel(compactUrl)); + if (current.requestPacing?.enabled + && (current.requestPacing.keys || current.requestPacing.keySelection === "round-robin")) { + if (!reacquirePacing) throw new Error("Refreshed compact dispatch requires pacing readmission"); + await reacquirePacing(current, route.modelId); + } + } + if (req.signal.aborted) throw req.signal.reason; + stampApiKeyAccountLabel(logCtx, route.providerName, sendProvider); + if (!rebindPhysicalSend(sendBudget, { poolId: logCtx.spendPoolId ?? route.providerName, + identityId: logCtx.accountLogLabel })) throw new SendBudgetExhaustedError(safeHostLabel(compactUrl)); + noteProviderAttemptSend(logCtx, route.providerName, sendProvider, logCtx.usageLogInputTokens, upstreamRecovery); + return sendWithConnectionPolicy((sendProvider as { fetch?: typeof fetch }).fetch ?? execute, compactUrl, + { ...init, headers: sendHeaders }, { providerName: route.providerName, provider: sendProvider }); + } : undefined, ...(onPhysicalDispatch ? { onPhysicalDispatch } : {}), }), // Every credential-bearing forward send gets manual redirects, not only @@ -1107,7 +1155,7 @@ export async function handleResponsesCompact( let outcomeCtx = authCtx; const localDispatchRefusal = (error: unknown): Response | undefined => { const cause = unwrapUpstreamRetryEvidenceError(error); - const response = spendLedgerStorageErrorResponse(error, logCtx) ?? (cause instanceof SendBudgetExhaustedError + const response = requestPacingOverloadResponse(cause) ?? spendLedgerStorageErrorResponse(error, logCtx) ?? (cause instanceof SendBudgetExhaustedError ? unboundPoolSpendRefusalResponse(logCtx) ?? formatErrorResponse(429, "request_send_budget_exhausted", cause.message) : mapCodexAuthContextErrorToResponse(cause, { now: Date.now(), accountSelector: route.codexAccountNamespace, @@ -1152,6 +1200,9 @@ export async function handleResponsesCompact( return formatErrorResponse(502, "upstream_error", "Failed to connect to compact upstream"); } + if ((upstream.status === 401 || upstream.status === 429) && !isNonReplayableResponse(upstream)) { + recordDistributedKeyFailure(config, route.providerName, compactProvider._apiKeyAttempt, upstream.status, Date.now(), upstream.headers.get("retry-after")); + } if ( upstream.status === 401 && (authCtx.kind === "main-pool" || authCtx.kind === "pool") diff --git a/src/server/responses/fetch-helpers.ts b/src/server/responses/fetch-helpers.ts index 1781ea19694..3dd4568dba9 100644 --- a/src/server/responses/fetch-helpers.ts +++ b/src/server/responses/fetch-helpers.ts @@ -12,6 +12,7 @@ import type { WsData } from "../ws-bridge"; import { requestPacingMaxConcurrentRequests, sendTrackingRequestSlot, + trackProviderRequestSlotBody, waitForProviderRequestSlot, type ProviderRequestSlot, } from "../../providers/request-pacing"; @@ -49,6 +50,9 @@ const UPSTREAM_REWRITTEN = Symbol.for("opencodex.plugins.upstream-rewritten"); const PHYSICAL_DISPATCH = Symbol.for("opencodex.physical-dispatch"); // One WS attempt and its HTTP fallback are one logical physical-send receipt. const WS_FALLBACK_RECEIPT = Symbol.for("opencodex.ws-fallback-receipt"); +// Replacement admission waits on the client signal, outside the response-header deadline. +const HEADER_DEADLINE = Symbol("header-deadline"); +type HeaderDeadline = { clientSignal: AbortSignal; pause(): void; resume(): void }; /** * Announce once, per provider, that an explicit egress route moved this provider off the @@ -139,6 +143,7 @@ export function wantsFreshConnection( export interface PaceAwareFetch { waitForPacing?: (signal?: AbortSignal) => Promise; unpacedFetch?: typeof globalThis.fetch; + fetchWithPacingSlot?: (slot: ProviderRequestSlot | undefined, input: Parameters[0], init?: RequestInit) => Promise; } export type ProviderFetch = typeof globalThis.fetch & PaceAwareFetch; @@ -252,7 +257,8 @@ export interface ProviderFetchOptions { */ preOutputRefusalHold?: CodexWsPreOutputRefusalHold; /** Revalidate/rebuild a queued request at its physical send boundary, after pacing. */ - dispatchOverride?: (input: Parameters[0], init: RequestInit, execute: typeof globalThis.fetch) => Promise; + dispatchOverride?: (input: Parameters[0], init: RequestInit, execute: typeof globalThis.fetch, + reacquirePacing?: (provider: OcxProviderConfig, modelId?: string) => Promise) => Promise; } /** Compose provider pacing and transport admission; receipts run only at executor entry. */ @@ -301,7 +307,8 @@ export function providerFetch( { preconnect }, ) as typeof globalThis.fetch); const httpFetch = Object.assign( - async (input: Parameters[0], init?: RequestInit) => { + async (input: Parameters[0], init?: RequestInit, + reacquirePacing?: (provider: OcxProviderConfig, modelId?: string) => Promise) => { // Refuse before any dispatch side effect where that is sound. `beforeDispatch` commits // attempt accounting and consumes admission state, so a refusal firing after it would // charge an attempt for a send that never happens, and a throwing hook would mask the @@ -329,30 +336,33 @@ export function providerFetch( } } : {}), }; return options.dispatchOverride - ? options.dispatchOverride(input, dispatchInit, dispatch) + ? options.dispatchOverride(input, dispatchInit, dispatch, reacquirePacing) : dispatch(input, dispatchInit); }, { preconnect }, - ) as typeof globalThis.fetch; + ); // ChatGPT Codex backend: streaming turns ride the responses_websockets // transport (measured ~3s faster TTFT than the SSE POST queue); everything // else keeps the provider's HTTP fetch. See ws-upstream.ts for the details. - const unpaced = async (input: Parameters[0], init?: RequestInit) => { + const unpaced = async (input: Parameters[0], init?: RequestInit, + reacquirePacing?: (provider: OcxProviderConfig, modelId?: string) => Promise) => { const upstreamWebsocket = provider.upstreamWebsocket; if (!options.httpOnly && typeof input === "string" && init && shouldUseCodexWsUpstream(input, init, runtime, upstreamWebsocket)) { - if (options.providerName && requestPacingMaxConcurrentRequests(provider, options.modelId) > 0) { + const keyedPacing = provider.requestPacing?.enabled + && (provider.requestPacing.keys || provider.requestPacing.keySelection === "round-robin"); + if (keyedPacing || (options.providerName && requestPacingMaxConcurrentRequests(provider, options.modelId) > 0)) { if (!pacingWebsocketDowngradeWarned.has(providerName) && pacingWebsocketDowngradeWarned.size < EGRESS_DOWNGRADE_NOTICE_LIMIT) { pacingWebsocketDowngradeWarned.add(providerName); - console.warn(`[opencodex] provider ${JSON.stringify(redactSecretString(providerName))} has a request concurrency cap; serving over HTTP/SSE.`); + console.warn(`[opencodex] provider ${JSON.stringify(redactSecretString(providerName))} requires guarded request pacing; serving over HTTP/SSE.`); } - return httpFetch(input, init); + return httpFetch(input, init, reacquirePacing); } const egress = egressFor(input); if (providerEgressIsExplicit(egress)) { warnEgressWebsocketDowngradeOnce(providerName, describeProviderEgressForLog(egress)); - return httpFetch(input, init); + return httpFetch(input, init, reacquirePacing); } // The fallback has to be the same HTTP fetch the non-WS branch would have // used, protocol pin included: a WS turn that falls back is serving the @@ -365,11 +375,11 @@ export function providerFetch( options.onPhysicalDispatch?.(); }; const fallback = ((url: Parameters[0], fallbackInit?: RequestInit) => - httpFetch(url, { ...fallbackInit, [WS_FALLBACK_RECEIPT]: notifyOnce } as RequestInit)) as typeof globalThis.fetch; + httpFetch(url, { ...fallbackInit, [WS_FALLBACK_RECEIPT]: notifyOnce } as RequestInit, reacquirePacing)) as typeof globalThis.fetch; return codexWsUpstreamFetch(input, init, fallback, runtime, options.onCodexWsQuota, options.beforeDispatch, options.nativeControl, async () => { (await waitForPacing(init.signal ?? undefined))?.release(); }, notifyOnce, options.preOutputRefusalHold); } - return httpFetch(input, init); + return httpFetch(input, init, reacquirePacing); }; let pacingSlotAcquired = options.pacingSlotAcquired === true; const waitForPacing = async (signal?: AbortSignal): Promise => { @@ -392,15 +402,35 @@ export function providerFetch( throw new Error("A provider name is required for request concurrency pacing"); } }; - const wrapped = async (input: Parameters[0], init?: RequestInit) => { - return sendTrackingRequestSlot(await waitForPacing(init?.signal ?? undefined), () => unpaced(input, init)); + const fetchWithPacingSlot = async (slot: ProviderRequestSlot | undefined, input: Parameters[0], init?: RequestInit) => { + const reacquirePacing = async (refreshed: OcxProviderConfig, modelId?: string) => { + if (options.turnScopedPacing) throw new Error("Cannot replace a turn-scoped pacing lease during dispatch"); + if (!options.providerName) throw new Error("Refreshed dispatch requires a named pacing owner"); + slot?.release(); + slot = undefined; + const deadline = (init as Record | undefined)?.[HEADER_DEADLINE]; + deadline?.pause(); + try { + slot = await waitForProviderRequestSlot(providerName, refreshed, modelId, deadline?.clientSignal ?? init?.signal ?? undefined); + } finally { deadline?.resume(); } + }; + try { + const response = await unpaced(input, init, reacquirePacing); + return trackProviderRequestSlotBody(slot, response); + } catch (error) { + slot?.release(); + throw error; + } }; + const wrapped = async (input: Parameters[0], init?: RequestInit) => + fetchWithPacingSlot(await waitForPacing(init?.signal ?? undefined), input, init); // The returned wrapper forwards its init down to `dispatch`, which applies the route at the // physical send. Adapters that hand this executor back as `provider.fetch` (Cursor does) // therefore still carry a per-provider route instead of being refused as opaque. const paceAware = Object.assign(wrapped, { preconnect, waitForPacing, + fetchWithPacingSlot, unpacedFetch: Object.assign(unpaced, { preconnect }), }); markEgressTransparentExecutor(paceAware as unknown as typeof globalThis.fetch); @@ -432,6 +462,9 @@ export function storedPoolReplayDispatchNotifier( notified = true; onDispatch(); }; + const notifyWithSlot = (slot: ProviderRequestSlot | undefined): void => { + try { notifyOnce(); } catch (error) { slot?.release(); throw error; } + }; const unpacedSource = executor.unpacedFetch ?? executor; const unpaced = Object.assign( (input: Parameters[0], init?: RequestInit) => { @@ -441,12 +474,21 @@ export function storedPoolReplayDispatchNotifier( { preconnect: unpacedSource.preconnect }, ) as ProviderFetch["unpacedFetch"]; const wrapped = async (input: Parameters[0], init?: RequestInit) => { + if (executor.fetchWithPacingSlot) { + const slot = await executor.waitForPacing?.(init?.signal ?? undefined); + notifyWithSlot(slot); + return executor.fetchWithPacingSlot(slot, input, init); + } return sendTrackingRequestSlot(await executor.waitForPacing?.(init?.signal ?? undefined), () => unpaced!(input, init)); }; return Object.assign(wrapped, { preconnect: executor.preconnect, waitForPacing: executor.waitForPacing, unpacedFetch: unpaced, + ...(executor.fetchWithPacingSlot ? { fetchWithPacingSlot: (slot: ProviderRequestSlot | undefined, input: Parameters[0], init?: RequestInit) => { + notifyWithSlot(slot); + return executor.fetchWithPacingSlot!(slot, input, init); + } } : {}), }) as ProviderFetch; } @@ -470,24 +512,36 @@ export async function fetchWithHeaderTimeout( let timer: ReturnType | undefined; try { const timeout = new AbortController(); - timer = setTimeout(() => { - if (!timeout.signal.aborted) timeout.abort(new DOMException("Timeout elapsed", "TimeoutError")); - }, timeoutMs); + const deadline: HeaderDeadline = { + clientSignal: abortSignal, + pause() { if (timer) clearTimeout(timer); timer = undefined; }, + resume() { + this.pause(); + timer = setTimeout(() => { + if (!timeout.signal.aborted) timeout.abort(new DOMException("Timeout elapsed", "TimeoutError")); + }, timeoutMs); + }, + }; + deadline.resume(); const headers = new Headers(init.headers); // Compressed SSE can be held until the decompressor has a complete block. Streaming calls // default to identity for low-latency frame delivery, while an explicit caller choice wins. if (preferIdentityEncoding && !headers.has("accept-encoding")) { headers.set("accept-encoding", "identity"); } - return await sendTrackingRequestSlot(slot, () => fetchExecutor(url, { + const dispatchInit = { ...init, headers, // Never replay provider credentials or request bodies to a redirect destination. // Preserve the 3xx for the owner's existing response/health policy (#914, #1471). - redirect: "manual", + redirect: "manual" as const, signal: AbortSignal.any([abortSignal, timeout.signal]), + [HEADER_DEADLINE]: deadline, timeout: 0, - })); + }; + return pacing.fetchWithPacingSlot + ? await pacing.fetchWithPacingSlot(slot, url, dispatchInit) + : await sendTrackingRequestSlot(slot, () => fetchExecutor(url, dispatchInit)); } finally { if (timer) clearTimeout(timer); if (!slot?.bodyTracked) slot?.release(); diff --git a/src/server/responses/native-response-control.ts b/src/server/responses/native-response-control.ts index 13c4b4b48f9..c990c41dcd1 100644 --- a/src/server/responses/native-response-control.ts +++ b/src/server/responses/native-response-control.ts @@ -4,6 +4,7 @@ import type { NativeSteeringReplayObserver } from "./native-steering-replay"; import type { ProviderExecutedCallType } from "../responses-undeclared-tool-guard"; import { isInjectionRequest } from "./native-injection-protocol"; +import { registerProviderResponseProvenance } from "../../providers/request-pacing"; /** Shared transport ownership, not a shared steer/inject protocol state machine. */ export interface NativeResponseControl { @@ -25,9 +26,10 @@ export interface NativeResponseControl { export const OPENAI_API_RESPONSES_URL = "https://api.openai.com/v1/responses"; -const nativeControlResponses = new WeakSet(); +const nativeControlResponses = new WeakMap(); +registerProviderResponseProvenance(nativeControlResponses); /** Mark the exact response for multi-response delivery without serializing a wire field. */ -export function markNativeControlResponse(response: Response): Response { nativeControlResponses.add(response); return response; } +export function markNativeControlResponse(response: Response): Response { nativeControlResponses.set(response, true); return response; } /** Recognize a marked native response by identity, not by caller-controlled content. */ export function isNativeControlResponse(response: Response): boolean { return nativeControlResponses.has(response); } diff --git a/src/server/responses/request-transport.ts b/src/server/responses/request-transport.ts index 1b6cb6f4c44..29b08f0ead5 100644 --- a/src/server/responses/request-transport.ts +++ b/src/server/responses/request-transport.ts @@ -597,7 +597,12 @@ export async function prepareResponsesTransport( throw new Error("OAuth account selection changed before dispatch"); } } else { - const current = resolveCurrentProviderApiKeyTransport(config, route.providerName, route.provider); + let current = resolveCurrentProviderApiKeyTransport(config, route.providerName, route.provider); + if (!current && route.provider.requestPacing?.enabled + && (route.provider.requestPacing.keys || route.provider.requestPacing.keySelection === "round-robin")) { + const fresh = resolveCurrentProviderApiKeyTransport(config, route.providerName, { ...route.provider, _apiKeyAttempt: undefined }); + if (fresh) current = selectProactiveApiKeyTransport(config, route.providerName, fresh) ?? fresh; + } if (!current) throw new Error("API key selection is unavailable before dispatch"); route.provider = current; await resolveCopilotSelection(requestParsed, true); @@ -625,7 +630,8 @@ export async function prepareResponsesTransport( ? (options.sendBudget as import("../../lib/request-execution-budget").RequestExecutionBudget).beginSpendProducer?.() : undefined; try { for (let attempt = 0; attempt < 3; attempt++) { - if (!selectionIsCurrent(adapterBindings.get(selectedAdapter))) selectedAdapter = await refreshRunTurnAdapter(requestParsed); + const selectionRefreshed = !selectionIsCurrent(adapterBindings.get(selectedAdapter)); + if (selectionRefreshed) selectedAdapter = await refreshRunTurnAdapter(requestParsed); const binding = adapterBindings.get(selectedAdapter); const run = rawRunTurns.get(selectedAdapter); if (!run) throw new Error("Selected provider no longer supports this turn transport"); @@ -635,15 +641,21 @@ export async function prepareResponsesTransport( // Both main and image-loop callers already acquired the initial pacing slot. // Subsequent physical messages retain this adapter/credential and are paced normally. try { - attemptSlot = attempt === 0 - ? incoming.pacingSlot ?? await waitForProviderRequestSlot(route.providerName, route.provider, route.modelId, incoming.abortSignal) + // A pre-acquired lease belongs to the original selection, never its replacement. + if (attempt === 0 && selectionRefreshed) releaseProviderRequestSlot(incoming.pacingSlot); + attemptSlot = attempt === 0 && !selectionRefreshed && incoming.pacingSlot + ? incoming.pacingSlot : await waitForProviderRequestSlot(route.providerName, route.provider, route.modelId, incoming.abortSignal); const fetch = providerFetch(route.provider, options.codexWsRuntimeIdentity, { providerName: route.providerName, modelId: route.modelId, pacingSlotAcquired: true, pacingSlot: attemptSlot, turnScopedPacing: true, beforeDispatch: () => { - if (sent) return; + if (sent) { + if (binding?.kind === "api-key" && binding.provider._apiKeyAttempt?.distributed + && !selectionIsCurrent(binding)) throw new Error("API key became unavailable during the turn"); + return; + } if (!selectionIsCurrent(binding)) { refused = true; throw new Error("Account selection changed before the first turn dispatch"); @@ -652,7 +664,7 @@ export async function prepareResponsesTransport( sent = true; }, }); - await run(requestParsed, { ...incoming, providerFetch: fetch }, event => { if (!refused) emit(event); }); + await run(requestParsed, { ...incoming, pacingSlot: attemptSlot, providerFetch: fetch }, event => { if (!refused) emit(event); }); } catch (error) { if (!refused) throw error; } finally { @@ -669,10 +681,16 @@ export async function prepareResponsesTransport( /** Fence the physical send, with bounded credential/session rebuilds and safe negotiated refusals. */ const oauthDispatch = (wireRequest: AdapterRequest, requestParsed = parsed, owner: "adapter" | "passthrough" = "adapter"): ProviderFetchOptions["dispatchOverride"] => { if (route.provider.authMode === "forward") return undefined; - return /** Dispatch with the current binding and session, rebuilding at most three times before returning a safe refusal. */ async (input, init, execute) => { + return /** Dispatch with the current binding and session, rebuilding at most three times before returning a safe refusal. */ async (input, init, execute, reacquirePacing) => { let destination = input; let dispatchInit = init; + let pacingChanged = false; for (let attempt = 0; attempt < 3; attempt++) { + if (pacingChanged) { + if (!reacquirePacing) throw new Error("Refreshed dispatch requires pacing readmission"); + await reacquirePacing(route.provider, route.modelId); + pacingChanged = false; + } if (selectionIsCurrent(requestBindings.get(wireRequest)) && copilotSessionExpiresAt > Date.now() + 1000) { const fetchImpl = (route.provider as OcxProviderConfig & { fetch?: typeof globalThis.fetch }).fetch ?? execute; const binding = requestBindings.get(wireRequest); @@ -775,7 +793,9 @@ export async function prepareResponsesTransport( sameTargetToken = transportToken; destination = rebuilt.url; dispatchInit = { ...dispatchInit, method: rebuilt.method, headers, body: rebuilt.body }; - // The next iteration validates synchronously and calls fetch in that same turn. + pacingChanged = route.provider.requestPacing?.enabled === true + && (!!route.provider.requestPacing.keys || route.provider.requestPacing.keySelection === "round-robin"); + // A refreshed key must reacquire admission before the next synchronous send guard. } throw new Error("OAuth account selection changed repeatedly before dispatch"); }; diff --git a/src/server/responses/run-turn-execution.ts b/src/server/responses/run-turn-execution.ts index 689730cfae0..5fcd9a78258 100644 --- a/src/server/responses/run-turn-execution.ts +++ b/src/server/responses/run-turn-execution.ts @@ -197,12 +197,13 @@ export async function executeResponsesRunTurn( const queue = createAdapterEventQueue({ onBacklogExceeded: () => runTurnAbort.abort(), }); - const refreshRunTurnSelection = async (): Promise => { - if (selectionIsCurrent(adapterBindings.get(transportState.runTurnAdapter))) return; + const refreshRunTurnSelection = async (): Promise => { + if (selectionIsCurrent(adapterBindings.get(transportState.runTurnAdapter))) return false; await refreshRunTurnAdapter(parsed); bindRouteReasoningReplayScope({ parsed, providerName: route.providerName, provider: route.provider, adapterName: transportState.runTurnAdapter.name, oauthCredentialSnapshot: transportState.replayOAuthCredentialSnapshot }); sealRequestAttemptIdentity(logCtx.activeAttempt, logCtx.provider, transportState.runTurnAdapter.name, logCtx.accountLogLabel); + return true; }; // Initial admission must settle before the streaming Response commits HTTP 200. // Let the outer Responses facade preserve the local retryable-429 contract. @@ -256,13 +257,18 @@ export async function executeResponsesRunTurn( if (!pacingSlotAcquired) { pacingSlot = await waitForProviderRequestSlot(route.providerName, route.provider, route.modelId, runTurnAbort.signal); } - await refreshRunTurnSelection(); + const selectionChanged = await refreshRunTurnSelection(); const selectedBinding = adapterBindings.get(transportState.runTurnAdapter); if (lastKeyFailure && selectedBinding?.kind === "api-key" && failedKeys.has(selectedBinding.provider.apiKey ?? "")) { emit(lastKeyFailure); return; } + if (selectionChanged) { + releaseProviderRequestSlot(pacingSlot); + pacingSlot = undefined; + pacingSlot = await waitForProviderRequestSlot(route.providerName, route.provider, route.modelId, runTurnAbort.signal); + } // LOCAL PATCH (runturn-websearch): refreshRunTurnSelection binds route // state onto `parsed`; mirror it onto the iteration-local copy the // adapter actually receives. diff --git a/src/types/provider.ts b/src/types/provider.ts index a06e907a420..49b2fde56bb 100644 --- a/src/types/provider.ts +++ b/src/types/provider.ts @@ -14,6 +14,8 @@ export interface ProviderApiKeySelection { entryId?: string; reference?: string; revision?: string; + /** Request-local allocation; does not replace the saved active key. */ + distributed?: boolean; } export interface OpenRouterProviderRouting { @@ -190,6 +192,10 @@ export interface ProviderRequestPacingConfig extends RequestPacingRule { enabled: boolean; /** Exact upstream model-id overrides; other models inherit the provider rule. */ models?: Record; + /** Normal request allocation, separate from cooldown recovery ordering. */ + keySelection?: "active" | "round-robin"; + /** Additional uniform limits per pool credential, across all models. */ + keys?: RequestPacingRule; } export interface FastWire { diff --git a/structure/config.md b/structure/config.md index feda166f9c5..d0721f8c9da 100644 --- a/structure/config.md +++ b/structure/config.md @@ -97,7 +97,7 @@ A schema-invalid top-level JSON value is repairable only when it is a non-array | --- | --- | --- | | Listener | `port`, `hostname` | The listener owns the port; `runtime-port.json` reports where it actually landed. | | Routing | `defaultProvider`, `providers`, per-provider `selectedModels`, `combos` | Explicit `provider/model` wins over `defaultProvider`; combo dispatch uses the selected target's existing capability ladder and does not create a second catalog authority. For Kiro OAuth, the management API validates `providers.kiro.oauthAccountFailover.strategy` (`least-loaded`) and `maxConcurrentPerAccount` (1–100) only for Kiro; the cap persists, while active lease counts remain process-local. | -| Request pacing | `providers..requestPacing`, `requestPacing.models.` | Optional client-side request-start pacing supports interval limits and positive-integer `maxConcurrentRequests` caps. A provider or model rule may be concurrency-only; model entries target exact upstream IDs and can only add delay or narrow concurrency. | +| Request pacing | `providers..requestPacing`, `requestPacing.models.` | Optional client-side request-start pacing supports intervals and positive-integer concurrency caps. Provider, exact-model, and uniform per-credential-reference `keys` rules compose atomically. `keySelection` defaults to `active`; opt-in `round-robin` assigns healthy pool keys per request without persisting the active choice. Project quota, TPM and daily limits are not inferred. | | Compaction and memory routing | `compactionRouting.model`, optional `compactionRouting.reasoningEffort`, optional `compactionRouting.triggers`, optional `compactionRouting.sourceModels`; `memoryModels.extract`, `memoryModels.consolidation` | Explicit Codex compaction metadata whose `compaction.trigger` is one the block names activates a request-local override; `triggers` defaults to `["manual"]`. Optional source selectors restrict the override to exact incoming models or `provider/*`. See [Responses compaction](transports/responses-failover.md#compaction-routing-overrides). Invalid hand edits disable the block with a load warning without discarding providers; candidate writes reject invalid blocks. Each optional memory phase names a nonblank model and optional declared `reasoningEffort`; absence preserves the current route, candidate writes reject malformed values, and load degrades only the invalid phase with a warning. See [memory phase routing](transports/responses-failover.md#memory-phase-routing). | | Catalog | `disabledModels`, `customModels`, `modelCacheTtlMs`, `providerContextCaps`, `contextCapValue`, per-provider `modelDisplayNames`, `codexAccountNamespaces`, `codexAccountPickerEnabled` | Catalog state is derived; config only records intent. Exact provider model display names are durable display only overlays. The picker flag is an explicit visibility override, while selector mappings remain the durable exact-routing contract. | | Retained state | `appOwnedMemoryBudgetMb` | Process-wide eviction target for app-owned logs, caches, blobs, and continuation payloads. Default 256 MiB, valid 64..4096; pinned state may temporarily exceed the target, but every pin-capable store has a finite local cap and their documented aggregate stays below `APP_OWNED_WORST_CASE_PINNED_BYTES` (512 MiB). Neither value caps RSS or native runtime memory. | diff --git a/structure/data-planes/images.md b/structure/data-planes/images.md index b1f578757a9..ca7a53aa704 100644 --- a/structure/data-planes/images.md +++ b/structure/data-planes/images.md @@ -103,7 +103,7 @@ error rather than falling through to a separately billed keyed provider. The out has one selected Authorization value, validated before the non-idempotent upstream POST. The keyed path never enters `handleResponses`, so `src/server/images.ts` repeats `selectProactiveApiKeyTransport` inside the keyed branch and rebuilds Authorization from the -returned clone rather than the earlier snapshot. +returned clone rather than the earlier snapshot. Opt-in request-local round-robin uses that same selection seam and maps exhausted-key allocation to retryable 429. Genuine distributed 401/429 responses record the actual selected key cooldown, retaining Retry-After; synthetic replay refusals do not cool keys, and no paid POST is replayed. The standalone Images relay remains a single direct non-idempotent fetch outside the inference request-pacing executor; provider/key pacing limits do not cover that relay. A configured key's model and provider scope applies to whichever destination the request settles on: the ChatGPT forward account, the keyed provider, the xAI Imagine bridge, or the Antigravity diff --git a/structure/gui-and-management-api.md b/structure/gui-and-management-api.md index 961a55f07ed..1e625447507 100644 --- a/structure/gui-and-management-api.md +++ b/structure/gui-and-management-api.md @@ -369,7 +369,7 @@ to fully quit and reopen Desktop after selecting the picker profile. Provider writes must not round-trip masked API keys as real secrets. Dashboard actions that change model visibility or subagent selection should trigger catalog/cache sync behavior through the server -path that owns it. +path that owns it. Provider request-pacing settings expose independent provider and uniform key limits plus opt-in request-local round-robin selection. `GET /api/provider-request-pacing` returns key handles, starts, in-flight counts, queue counts, next spacing slots and known cooldowns, never credential values or internal bucket hashes. Saving these rules uses the existing requestPacing-only PATCH path. ### Fast selector rows setting diff --git a/structure/providers-and-adapters.md b/structure/providers-and-adapters.md index c7c9665547a..712b198c8ae 100644 --- a/structure/providers-and-adapters.md +++ b/structure/providers-and-adapters.md @@ -288,7 +288,7 @@ Cursor, one lease spans the whole turn: RunSSE and BidiAppend may overlap inside turns wait at the cap. Follow-up sends still obey start intervals. The turn owner returns its lease after `runTurn` settles, so RunSSE body completion cannot admit another turn early. A capped canonical Codex WebSocket turn uses HTTP/SSE because the socket has no response-body lifecycle. -Capacity waits use the same bounded queue and retryable queue-overload errors as interval waits. +Capacity waits use the same bounded queue and retryable queue-overload errors as interval waits. Optional `requestPacing.keys` adds a shared cross-model limit for each credential reference; duplicate references share capacity, but distinct references resolving to one upstream project do not imply separate quota. `keySelection: round-robin` assigns request-local healthy keys without mutating persisted selection, skips known cooldowns, and refuses exhausted pools with retryable 429. Provider, model and key gates reserve atomically; key counters and the allocation cursor are process-local, not token or billing balances. OAuth and forwarded credentials retain their existing selection. Cooldown state is shared through the dependency-light `src/providers/api-key-cooldowns.ts` (retaining the longest outstanding expiry until explicit manual clearing) owner. Distributed selections are checked again at physical dispatch; a cooled queued key releases its old permit, selects a healthy sibling, and reacquires pacing before sending. Native Messages follows the same readmission rule. An established distributed turn refuses later sends after its key cools instead of changing credentials mid-turn. Command Code effort defaults in `src/providers/command-code-efforts.ts` combine public-profile facts with the live API measurements from #5096. Both presets share the exact per-model rows; diff --git a/structure/transports/responses-failover.md b/structure/transports/responses-failover.md index 535ffb4c842..3bd6c185877 100644 --- a/structure/transports/responses-failover.md +++ b/structure/transports/responses-failover.md @@ -156,8 +156,8 @@ The shared Responses path follows the [bounded multipart recovery contract](../s ## Upstream key attempt accounting -Translated Anthropic billing recovery uses the [immutable physical-send ownership contract](../providers/anthropic-key-billing.md). Its initial send, credential hop and later mixed-class refusals share the request budget. A denied hop retains the upstream refusal, and a leased pacing body rewrap carries the existing marker rather than attributing the response to rebuilt headers. Key identity is sealed at the guarded physical dispatch after queued selections are rebuilt. -Raw adapter terminal usage is recorded before continuation, search, or image loops merge it; +Key identity is sealed at guarded physical dispatch after queued selections are rebuilt. Distributed selections retain their request-local entry and revision. A rebuild after selection drift releases the old per-key lease and reacquires admission for the refreshed key; the send guard checks again after waiting. Turn-scoped adapters follow the same rule before invocation. Native Chat, Messages and keyed compact all rebuild stale queued selections before dispatch, including a manual active-key switch under per-key limits; they release the old lease, readmit the new key and honor cancellation while its capacity is occupied. Direct compact and Images record genuine distributed 401/429 cooldowns without replaying paid POSTs; synthetic replay refusals never cool a key. Native Messages revisits mixed distributed 401/429 outcomes with its existing counters. Native Chat and Messages require the physical outgoing credential to match the selected distributed key before pooling a rejection; Messages honors the configured `x-api-key` or Bearer transport while rejecting a conflicting credential header; configured credential overrides cannot cool unused entries. Replacement admission pauses the response-header deadline and retains client cancellation; the header deadline restarts once admission completes. Unpaced dispatch without a readmission owner fails closed. Replay notification errors release acquired slots before propagating, including pre-acquired-slot dispatch. Translated Anthropic billing recovery uses the [immutable physical-send ownership contract](../providers/anthropic-key-billing.md). Its initial send, credential hop and later mixed-class refusals share the request budget. A denied hop retains the upstream refusal, and a leased pacing body rewrap carries the existing marker rather than attributing the response to rebuilt headers. +Before a turn-scoped retry readmits a refreshed selection, it rejects a binding already recorded as failed for this request and returns the last key failure. Eligible replacement keys still release the old lease and reacquire their own pacing capacity before invocation. Raw adapter terminal usage is recorded before continuation, search, or image loops merge it; repeated parsing of one physical response does not count it twice. Key changes preserve the previous attempt while retaining the active attempt object shared by streaming/combo callbacks. Bounded failure-body observation retains reported usage and releases cloned readers on abort. @@ -583,7 +583,7 @@ provider's input. Object-identity metadata is recomputed per attempt. Compaction route identity excludes policy selectors and combos on either side; synthetic fast/effort suffixes are removed before identity checks. A stale selector resolved only through the default -provider cannot establish the original serving identity and remains portable. +provider cannot establish the original serving identity and remains portable. Queued distributed API-key bindings are revalidated after every pacing admission. A replacement releases the old permit before waiting for provider, model and key capacity; enabled round-robin performs this step even without a per-key cap. Once a turn has physically sent, a cooled binding is refused rather than rotated inside that turn. ## Anthropic Fast downgrade recovery diff --git a/structure/transports/responses.md b/structure/transports/responses.md index 0a065aec8ba..17ab5eb2198 100644 --- a/structure/transports/responses.md +++ b/structure/transports/responses.md @@ -51,11 +51,11 @@ performs the physical send, after a dispatch override has selected or rebuilt th matching follows the URL sent on the wire rather than the URL supplied before credential revalidation; prepaid initial-send receipts follow the [spend contract](responses-spend.md#prepaid-initial-sends). At this final HTTP boundary, native ChatGPT Responses and compact JSON strings of at least 1 MiB (UTF-8) become byte buffers to avoid Bun's large-string upload resets. Content, headers, abort signals and retry policy are preserved; WebSocket selection still receives the original string. Other destinations, small strings and existing byte/stream bodies retain their representation. -The wrapped executor alone is not that boundary. An override that revalidates credentials re-reads +Opt-in per-key pacing and round-robin use HTTP/SSE so their final dispatch override cannot be bypassed by a WebSocket upgrade. Transport-owned response markers and stage metadata survive owned pacing body rewraps. Replacement-key pacing admission retains client cancellation but pauses the header deadline; the full header interval restarts after admission. The wrapped executor alone is not that boundary. An override that revalidates credentials re-reads `route.provider.fetch` at send time, because reselection can install a different provider transport after the wrapper was built, and then calls that implementation instead of the executor. Both -production overrides do this -- `oauthDispatch` in `request-transport.ts` and the native Chat -key-revalidation override in `chat-native.ts` -- so both wrap the selected implementation rather +production overrides do this -- `oauthDispatch` in `request-transport.ts`, native Chat in +`chat-native.ts`, native Messages and keyed compact -- so they wrap the selected implementation rather than choosing between policy and provider transport. Reporting the executor as the boundary while the code let a provider-scoped transport past it is what #4992 recorded, and it is why a regression for this policy has to enter through `handleResponses` rather than through a @@ -427,7 +427,7 @@ is composed from the following owners in `src/server/responses/`; none is a gene | `passthrough-delivery.ts`, `terminal-error-redaction.ts`, `non-replayable-error.ts` | Native HTTP/SSE/JSON delivery and terminal accounting. Marked real errors use bounded reads to retain only allowlisted error type and code, keep status and non-replayability plus `x-should-retry: false` when present, and withhold upstream body text and all other upstream headers; the client receives a fixed generic message and synthetic refusals stay distinct. SSE diagnostics and xAI tool-envelope filtering precede client delivery/continuation storage. | | `policy-refusal.ts` | Rewrites an allowlisted non-combo HTTP 403 model refusal (`isUpstreamPolicyRefusal` in `src/lib/errors.ts`) from an xAI destination only (`isXaiResponsesDestination`: api.x.ai or the Grok CLI proxy, on either wire) to an HTTP 200 Responses `incomplete` / `content_filter` payload, JSON or SSE, for both `adapter-dispatch.ts` and `passthrough-delivery.ts`. A streamed rewrite takes the turn admission lease and releases it when the body finishes, so the refusal stays inside active-turn accounting. Combo attempts keep the original 403 so failover classifies it as a hop. | | `sidecar-execution.ts`, `sidecar-send-budget.ts` | Image/video versus web-search execution, shared rotation, and adapter-aware prepaid inference/producer ownership under the [spend contract](responses-spend.md#prepaid-initial-sends). | -| `completion-policy.ts`, `run-turn-execution.ts` | Empty-completion eligibility and adapter-owned event turns. | +| `completion-policy.ts`, `run-turn-execution.ts` | Empty-completion eligibility and adapter-owned event turns; refreshed retries reject previously failed key bindings before [replacement-key readmission](responses-failover.md#upstream-key-attempt-accounting). | | `adapter-dispatch.ts` | Translated initial dispatch, bounded recovery and the shared continuation retry counter. Initial and rebuilt adapter budget refusals retain the HTTP 413 [byte-accounting contract](byte-accounting.md#request-copy-accounting); ordinary build errors remain 400. | | `adapter-continuation.ts`, `adapter-delivery.ts` | Continuation event sources and final streaming/buffered bridging; a streamed turn with a `clientEncoder` option is handed to `src/server/inference/client-encoder-delivery.ts` instead of the bridge. | diff --git a/tests/adapters/key-failover.test.ts b/tests/adapters/key-failover.test.ts index e6ccbff9fdf..f09564c3d7f 100644 --- a/tests/adapters/key-failover.test.ts +++ b/tests/adapters/key-failover.test.ts @@ -3,6 +3,7 @@ import { mkdtempSync, unlinkSync, writeFileSync } from "node:fs"; import { tmpdir } from "node:os"; import { join } from "node:path"; import { createOpenAIChatAdapter } from "../../src/adapters/openai-chat"; +import { providerFetch } from "../../src/server/responses/fetch-helpers"; import { getConfigPath, loadConfig, @@ -34,6 +35,7 @@ import { subscribeAccountSelections } from "../../src/lib/account-selection-even import { providerManagementConfigError, safeConfigDTO } from "../../src/server/auth-cors"; import type { OcxConfig, OcxParsedRequest, OcxProviderConfig } from "../../src/types"; import { removeTreeWithRetry } from "../helpers/remove-tree"; +import { providerApiKeySelectionIsCurrent, resolveCurrentProviderApiKeyTransport } from "../../src/providers/api-key-selection"; let home: string; const previousHome = process.env.OPENCODEX_HOME; @@ -463,6 +465,64 @@ describe("rotateKeyOn401", () => { describe("proactive key selection", () => { const now = 2_000_000; + test("request-local round-robin keeps parallel keys stable without changing saved selection", async () => { + clearKeyCooldowns("distributed"); + const config = makeConfig({ apiKey: "key-alpha-000111222333", apiKeyPool: pool3(), + requestPacing: { enabled: true, keySelection: "round-robin" } }); + const row = config.providers.p!; + config.providers.distributed = row; + const selected = Array.from({ length: 6 }, () => selectProactiveApiKeyTransport(config, "distributed", routedProviderConfig("distributed", row), undefined, now)!); + expect(selected.map(provider => provider._apiKeyAttempt?.entryId)).toEqual(["k1", "k2", "k3", "k1", "k2", "k3"]); + expect(config.providers.distributed!.apiKey).toBe(row.apiKey); + expect(loadConfig().providers.p!.apiKey).toBe(row.apiKey); + for (const provider of selected) { + expect(providerApiKeySelectionIsCurrent(config, "distributed", provider)).toBe(true); + expect(resolveCurrentProviderApiKeyTransport(config, "distributed", provider)?.apiKey).toBe(provider.apiKey); + } + const sent: string[] = []; + await Promise.all(selected.map(async provider => { + const transport = resolveCurrentProviderApiKeyTransport(config, "distributed", provider)!; + transport.fetch = Object.assign(async (_input: unknown, init?: RequestInit) => { + sent.push(new Headers(init?.headers).get("authorization")!); + return new Response("ok"); + }, { preconnect() {} }) as typeof fetch; + const parsed: OcxParsedRequest = { modelId: "fake-model", context: { messages: [{ role: "user", content: "hi", timestamp: 0 }] }, stream: false, options: {} }; + const request = createOpenAIChatAdapter(transport).buildRequest(parsed); + const response = await providerFetch(transport, undefined, { providerName: "distributed" })(request.url, { + method: "POST", headers: request.headers, body: request.body, + }); + await response.text(); + })); + expect(sent).toEqual(selected.map(provider => `Bearer ${provider.apiKey}`)); + row.apiKeySelectionRevision = "manual-edit"; + expect(providerApiKeySelectionIsCurrent(config, "distributed", selected[0]!)).toBe(false); + expect(resolveCurrentProviderApiKeyTransport(config, "distributed", selected[0]!)).toBeNull(); + }); + + test("distributed 429 and 401 exclude only attempted keys and exhaustion refuses allocation", () => { + const config = makeConfig({ apiKey: "key-alpha-000111222333", apiKeyPool: pool3(), + requestPacing: { enabled: true, keySelection: "round-robin" } }); + const row = config.providers.p!; + config.providers.distributedFailure = row; + clearKeyCooldowns("distributedFailure"); + let transport = selectProactiveApiKeyTransport(config, "distributedFailure", routedProviderConfig("distributedFailure", row), undefined, now)!; + transport = rotateProviderTransportOn429(config, "distributedFailure", transport, { retryAfter: "60", now })!; + expect(getKeyCooldownUntil("distributedFailure", "k1", now)).toBe(now + 60_000); + transport = rotateProviderTransportOn401(config, "distributedFailure", transport, { now })!; + expect(rotateProviderTransportOn429(config, "distributedFailure", transport, { now })).toBeNull(); + expect(() => selectProactiveApiKeyTransport(config, "distributedFailure", routedProviderConfig("distributedFailure", row), undefined, now)).toThrow("No eligible API key"); + expect(row.apiKey).toBe("key-alpha-000111222333"); + }); + + test("an active-key permit cannot dispatch a manually replaced key under per-key limits", () => { + const config = makeConfig({ apiKey: "key-alpha-000111222333", apiKeyPool: pool3(), + requestPacing: { enabled: true, keys: { maxConcurrentRequests: 1 } } }); + const transport = routedProviderConfig("p", config.providers.p!); + expect(selectProactiveApiKeyTransport(config, "p", transport, undefined, now)).toBeNull(); + config.providers.p!.apiKey = "key-beta-444555666777"; + expect(resolveCurrentProviderApiKeyTransport(config, "p", transport)).toBeNull(); + }); + test("does nothing without a configured strategy", () => { const config = makeConfig({ apiKey: "key-alpha-000111222333", apiKeyPool: pool3() }); forgetApiKeyRotationCursor("p"); diff --git a/tests/claude-integration/messages-native-key-selection.test.ts b/tests/claude-integration/messages-native-key-selection.test.ts new file mode 100644 index 00000000000..6881ce0ac04 --- /dev/null +++ b/tests/claude-integration/messages-native-key-selection.test.ts @@ -0,0 +1,357 @@ +import { afterEach, beforeEach, describe, expect, test } from "bun:test"; +import { mkdtempSync, readFileSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { saveConfig } from "../../src/config"; +import { apiKeyAccountLogLabel } from "../../src/codex/account-label"; +import { configureSharedSpendLedger, spendPolicyFromConfig } from "../../src/lib/spend-reservation-ledger"; +import { captureProviderApiKeySelection } from "../../src/providers/api-key-selection"; +import { clearKeyCooldowns, forgetApiKeyRotationCursor, getKeyCooldownUntil, recordDistributedKeyFailure } from "../../src/providers/key-failover"; +import { providerRequestPacingStatus, resetProviderRequestPacingForTest, waitForProviderRequestSlot } from "../../src/providers/request-pacing"; +import { handleClaudeMessages } from "../../src/server/claude-messages"; +import { getRequestLogEntries } from "../../src/server/request-log"; +import type { OcxConfig, OcxProviderConfig } from "../../src/types"; +import { acquireOwnedSpendHome } from "../helpers/owned-spend-home"; +import { removeTreeWithRetry } from "../helpers/remove-tree"; + +const NAME = "native-key-selection"; +const KEYS = ["fixture-key-alpha", "fixture-key-beta", "fixture-key-gamma"]; +const MESSAGE = { id: "msg_key_selection", type: "message", role: "assistant", model: "claude-x", + content: [{ type: "text", text: "fixture reply" }], stop_reason: "end_turn", stop_sequence: null, + usage: { input_tokens: 11, output_tokens: 3 } }; +let home: string; +let previousHome: string | undefined; +let releaseOwner: () => void; +let upstream: ReturnType | undefined; +let seen: { key: string | null; authorization?: string | null; path: string; body: unknown }[]; + +beforeEach(() => { + previousHome = process.env.OPENCODEX_HOME; + home = mkdtempSync(join(tmpdir(), "ocx-native-key-selection-")); + process.env.OPENCODEX_HOME = home; + releaseOwner = acquireOwnedSpendHome(); + clearKeyCooldowns(); forgetApiKeyRotationCursor(); resetProviderRequestPacingForTest(); + seen = []; +}); + +afterEach(async () => { + await upstream?.stop(true); upstream = undefined; + releaseOwner(); + clearKeyCooldowns(); forgetApiKeyRotationCursor(); resetProviderRequestPacingForTest(); + if (previousHome === undefined) delete process.env.OPENCODEX_HOME; + else process.env.OPENCODEX_HOME = previousHome; + removeTreeWithRetry(home); +}); + +function fixture(count = 2, rejectFirst = false, keyCap = true, statuses: number[] = []): OcxConfig { + upstream = Bun.serve({ hostname: "127.0.0.1", port: 0, async fetch(req) { + seen.push({ key: req.headers.get("x-api-key") ?? req.headers.get("authorization")?.replace(/^Bearer /, "") ?? null, authorization: req.headers.get("authorization"), path: new URL(req.url).pathname, body: await req.json() }); + const status = statuses[seen.length - 1] ?? (rejectFirst && seen.length === 1 ? 401 : 200); + if (status !== 200) return Response.json({ type: "error", + error: { type: status === 401 ? "authentication_error" : "rate_limit_error", message: "rejected fixture key" } }, + { status, headers: status === 429 ? { "retry-after": "30" } : undefined }); + return Response.json(MESSAGE); + } }); + const config = { port: 0, defaultProvider: NAME, protocols: { rollout: { managedMessagesNative: true } }, + providers: { [NAME]: { adapter: "anthropic", baseUrl: `http://127.0.0.1:${upstream.port}`, + authMode: "key", allowPrivateNetwork: true, apiKey: KEYS[0], apiKeySelectionRevision: "saved-revision", + models: ["claude-x"], apiKeyPool: KEYS.slice(0, count).map((key, index) => ({ id: `k${index + 1}`, key, addedAt: index + 1 })), + requestPacing: { enabled: true, keySelection: "round-robin", + ...(keyCap ? { keys: { maxConcurrentRequests: 1 } } : { maxConcurrentRequests: 1 }) }, + } } } as OcxConfig; + saveConfig(config); + return config; +} + +function provider(config: OcxConfig): OcxProviderConfig { return config.providers[NAME]!; } +function status(config: OcxConfig) { return providerRequestPacingStatus(NAME, provider(config)); } +function keyStatus(config: OcxConfig, id: string) { return status(config).keys!.find(key => key.id === id)!; } +function cool(config: OcxConfig, index: number): void { + const selection = { ...captureProviderApiKeySelection({ ...provider(config), apiKey: KEYS[index] }), distributed: true }; + expect(recordDistributedKeyFailure(config, NAME, selection, 429, Date.now(), "30")).toBe(true); +} +async function waitUntil(predicate: () => boolean, message: string): Promise { + const deadline = Date.now() + 2_000; + while (!predicate()) { + if (Date.now() >= deadline) throw new Error(message); + await Bun.sleep(5); + } +} +function queue(config: OcxConfig, id: string) { + return waitUntil(() => keyStatus(config, id).queued === 1, `key ${id} never queued: ${JSON.stringify(status(config))}`); +} +function start(config: OcxConfig, abort: AbortController) { + const requestId = `native-key-selection-${crypto.randomUUID()}`; + const signal = AbortSignal.any([abort.signal, AbortSignal.timeout(5_000)]); + const pending = handleClaudeMessages(new Request("http://localhost/v1/messages", { method: "POST", + headers: { "content-type": "application/json" }, signal, + body: JSON.stringify({ model: `${NAME}/claude-x`, max_tokens: 64, + messages: [{ role: "user", content: "fixture question" }], stream: false }), + }), config, { model: "", provider: "" }, { requestId, start: Date.now() }) + .then(async response => ({ response, text: await response.text(), requestId })); + void pending.catch(() => {}); + return pending; +} +function unchanged(config: OcxConfig) { + expect(provider(config).apiKey).toBe(KEYS[0]); + expect(provider(config).apiKeySelectionRevision).toBe("saved-revision"); + const persisted = JSON.parse(readFileSync(join(home, "config.json"), "utf8")) as OcxConfig; + expect(persisted.providers[NAME]!.apiKey).toBe(KEYS[0]); + expect(persisted.providers[NAME]!.apiKeySelectionRevision).toBe("saved-revision"); + expect(persisted.providers[NAME]!._apiKeyAttempt).toBeUndefined(); +} +function noLeaks(config: OcxConfig) { + expect(status(config)).toMatchObject({ queued: 0, inFlight: 0 }); + for (const key of status(config).keys!) expect(key).toMatchObject({ queued: 0, inFlight: 0 }); +} +function nativeSuccess(result: Awaited>) { + expect(result.response.status).toBe(200); + expect(JSON.parse(result.text).content).toEqual(MESSAGE.content); + expect(getRequestLogEntries().find(row => row.requestId === result.requestId)?.protocolTrace?.mode).toBe("native"); + expect(seen.every(entry => entry.path === "/v1/messages")).toBe(true); +} + +describe("native Messages distributed key selection", () => { + for (const status of [401, 429]) test(`an overridden wire key C${status} never cools or replays selected A/B`, async () => { + const config = fixture(2, false, true, [status]); + provider(config).headers = { "x-api-key": KEYS[2]! }; + // A credential mismatch also refuses an explicitly requested same-key 429 retry. + provider(config).retryOn429 = { attempts: 1, intervalMs: 1, respectRetryAfter: false }; + saveConfig(config); + const abort = new AbortController(); + const pending = start(config, abort); + try { + const result = await pending; + expect(result.response.status).toBe(status); + expect(seen.map(entry => entry.key)).toEqual([KEYS[2]]); + expect(seen[0]!.path).toBe("/v1/messages"); + expect(getKeyCooldownUntil(NAME, "k1")).toBeNull(); + expect(getKeyCooldownUntil(NAME, "k2")).toBeNull(); + unchanged(config); noLeaks(config); + } finally { abort.abort(); await pending.catch(() => undefined); } + }); + + test("mixed A429 B401 C200 recovers across the ladder without changing saved selection", async () => { + const config = fixture(3, false, true, [429, 401, 200]); + const abort = new AbortController(); + const pending = start(config, abort); + try { + const result = await pending; + nativeSuccess(result); + expect(seen.map(entry => entry.key)).toEqual(KEYS); + expect(seen[1]!.body).toEqual(seen[0]!.body); + expect(seen[2]!.body).toEqual(seen[0]!.body); + const row = getRequestLogEntries().find(entry => entry.requestId === result.requestId)!; + expect(row.attempts!.map(attempt => attempt.sendCount)).toEqual([1, 1, 1]); + expect(row.attempts!.map(attempt => attempt.accountLogLabel)).toEqual(KEYS.map((reference, index) => + apiKeyAccountLogLabel(NAME, { entryId: `k${index + 1}`, reference }))); + expect(row.attempts!.map(attempt => attempt.status)).toEqual([429, 401, 200]); + expect(getKeyCooldownUntil(NAME, "k1")).toBeGreaterThan(Date.now() + 25_000); + expect(getKeyCooldownUntil(NAME, "k2")).toBeGreaterThan(Date.now()); + expect(getKeyCooldownUntil(NAME, "k3")).toBeNull(); + unchanged(config); noLeaks(config); + } finally { abort.abort(); await pending.catch(() => undefined); } + }); + + for (const terminal of [401, 429]) test(`mixed failures record terminal B${terminal} at the finite send limit`, async () => { + const config = fixture(2, false, true, [429, terminal]); + provider(config).transientRetryOn5xx = { attempts: 2 }; + saveConfig(config); + const abort = new AbortController(); + const pending = start(config, abort); + try { + const result = await pending; + expect(result.response.status).toBe(terminal); + expect(seen.map(entry => entry.key)).toEqual(KEYS.slice(0, 2)); + expect(getKeyCooldownUntil(NAME, "k1")).toBeGreaterThan(Date.now() + 25_000); + expect(getKeyCooldownUntil(NAME, "k2")).toBeGreaterThan(Date.now() + 25_000); + const exhausted = await start(config, abort); + expect(exhausted.response.status).toBe(429); + expect(Number(exhausted.response.headers.get("retry-after"))).toBeGreaterThan(0); + expect(seen).toHaveLength(2); + unchanged(config); noLeaks(config); + } finally { abort.abort(); await pending.catch(() => undefined); } + }); + + test("same-key retries precede cooldown and share their counter across mixed recovery", async () => { + const config = fixture(3, false, true, [429, 401, 429, 200]); + provider(config).retryOn429 = { attempts: 1, intervalMs: 0, respectRetryAfter: false }; + provider(config).transientRetryOn5xx = { attempts: 4 }; + saveConfig(config); + const abort = new AbortController(); + const pending = start(config, abort); + try { + nativeSuccess(await pending); + expect(seen.map(entry => entry.key)).toEqual([KEYS[0], KEYS[0], KEYS[1], KEYS[2]]); + expect(getKeyCooldownUntil(NAME, "k1")).not.toBeNull(); + expect(getKeyCooldownUntil(NAME, "k2")).not.toBeNull(); + expect(getKeyCooldownUntil(NAME, "k3")).toBeNull(); + unchanged(config); noLeaks(config); + } finally { abort.abort(); await pending.catch(() => undefined); } + }); + + test("a shared physical-send refusal records the last genuine 429 without cooling an unsent key", async () => { + const config = fixture(2, false, true, [429, 429, 429, 429, 200]); + provider(config).retryOn429 = { attempts: 6, intervalMs: 1, maxIntervalMs: 1, respectRetryAfter: false }; + saveConfig(config); + configureSharedSpendLedger(spendPolicyFromConfig({ pool: { maxTokens: 100_000 } }, undefined, [NAME])); + const abort = new AbortController(); + const pending = start(config, abort); + try { + const result = await pending; + expect(result.response.status).toBe(429); + expect(JSON.parse(result.text).error.type).toBe("send_budget_exhausted"); + expect(seen.map(entry => entry.key)).toEqual(Array(4).fill(KEYS[0])); + expect(getKeyCooldownUntil(NAME, "k1")).toBeGreaterThan(Date.now() + 25_000); + expect(getKeyCooldownUntil(NAME, "k2")).toBeNull(); + const next = await start(config, abort); + nativeSuccess(next); + expect(seen.map(entry => entry.key)).toEqual([...Array(4).fill(KEYS[0]), KEYS[1]]); + unchanged(config); noLeaks(config); + } finally { abort.abort(); await pending.catch(() => undefined); } + }); + + test("401 recovers A to B without changing the saved key or revision", async () => { + const config = fixture(2, true); + const abort = new AbortController(); + const pending = start(config, abort); + try { + nativeSuccess(await pending); + expect(seen.map(entry => entry.key)).toEqual(KEYS.slice(0, 2)); + expect(seen[1]!.body).toEqual(seen[0]!.body); + unchanged(config); noLeaks(config); + } finally { abort.abort(); await pending.catch(() => undefined); } + }); + + for (const cancel of [false, true]) test(`cooled queued A releases its permit and readmits B (abort=${cancel})`, async () => { + const config = fixture(); + const first = await waitForProviderRequestSlot(NAME, provider(config)); + const second = await waitForProviderRequestSlot(NAME, { ...provider(config), apiKey: KEYS[1] }); + const abort = new AbortController(); + const pending = start(config, abort); + try { + await queue(config, "k1"); + expect(seen).toEqual([]); + cool(config, 0); first.release(); + await queue(config, "k2"); + expect(first.released).toBe(true); + expect(keyStatus(config, "k1")).toMatchObject({ inFlight: 0, queued: 0, starts: 2 }); + expect(keyStatus(config, "k2")).toMatchObject({ inFlight: 1, queued: 1, starts: 1 }); + expect(status(config).inFlight).toBe(1); + expect(seen).toEqual([]); + if (cancel) { + abort.abort(); + await pending.catch(() => undefined); + expect(seen).toEqual([]); + expect(status(config).queued).toBe(0); + second.release(); + } else { + second.release(); nativeSuccess(await pending); + expect(seen.map(entry => entry.key)).toEqual([KEYS[1]]); + } + unchanged(config); noLeaks(config); + } finally { + abort.abort(); first.release(); second.release(); await pending.catch(() => undefined); + } + }, 10_000); + + test("round-robin without a keys rule reselects behind the provider concurrency cap", async () => { + const config = fixture(2, false, false); + expect(provider(config).requestPacing!.keys).toBeUndefined(); + const held = await waitForProviderRequestSlot(NAME, provider(config)); + const abort = new AbortController(); + const pending = start(config, abort); + try { + await queue(config, "k1"); cool(config, 0); + expect(seen).toEqual([]); + held.release(); nativeSuccess(await pending); + expect(seen.map(entry => entry.key)).toEqual([KEYS[1]]); + expect(keyStatus(config, "k1").starts).toBe(2); + expect(keyStatus(config, "k2").starts).toBe(1); + unchanged(config); noLeaks(config); + } finally { abort.abort(); held.release(); await pending.catch(() => undefined); } + }); + + for (const exhausted of [false, true]) test(`revalidates B after readmission when B cools while queued (exhausted=${exhausted})`, async () => { + const config = fixture(3); + const first = await waitForProviderRequestSlot(NAME, provider(config)); + const second = await waitForProviderRequestSlot(NAME, { ...provider(config), apiKey: KEYS[1] }); + const abort = new AbortController(); + const pending = start(config, abort); + try { + await queue(config, "k1"); cool(config, 0); first.release(); + await queue(config, "k2"); + expect(keyStatus(config, "k1").inFlight).toBe(0); + expect(seen).toEqual([]); + cool(config, 1); + if (exhausted) cool(config, 2); + second.release(); + const result = await pending; + if (exhausted) { + expect(result.response.status).toBe(429); + expect(Number(result.response.headers.get("retry-after"))).toBeGreaterThan(0); + expect(seen).toEqual([]); + } else { + nativeSuccess(result); + expect(seen.map(entry => entry.key)).toEqual([KEYS[2]]); + } + unchanged(config); noLeaks(config); + } finally { abort.abort(); first.release(); second.release(); await pending.catch(() => undefined); } + }, 10_000); +}); + +for (const statusCode of [401, 429]) for (const overridden of [false, true]) test(`Messages bearer ${statusCode} owns the configured key transport (override=${overridden})`, async () => { + const config = fixture(2, false, true, [statusCode, 200]); + provider(config).apiKeyTransport = "bearer"; + saveConfig(config); + if (overridden) provider(config).headers = { Authorization: "Bearer fixture-bearer-override" }; + const result = await start(config, new AbortController()); + if (overridden) { + expect(result.response.status).toBe(statusCode); + expect(seen.map(entry => entry.key)).toEqual(["fixture-bearer-override"]); + expect(getKeyCooldownUntil(NAME, "k1")).toBeNull(); + expect(getKeyCooldownUntil(NAME, "k2")).toBeNull(); + } else { + nativeSuccess(result); + expect(seen.map(entry => entry.key)).toEqual(KEYS.slice(0, 2)); + expect(seen.map(entry => entry.authorization)).toEqual(KEYS.slice(0, 2).map(key => `Bearer ${key}`)); + expect(getKeyCooldownUntil(NAME, "k1")).not.toBeNull(); + expect(getKeyCooldownUntil(NAME, "k2")).toBeNull(); + } + unchanged(config); noLeaks(config); +}); + +for (const cancel of [false, true]) test(`active manual key switch readmits the selected key (cancel=${cancel})`, async () => { + const config = fixture(); + config.providers[NAME]!.requestPacing = { enabled: true, keys: { maxConcurrentRequests: 1 } }; + saveConfig(config); + const heldA = await waitForProviderRequestSlot(NAME, config.providers[NAME]!); + const heldB = await waitForProviderRequestSlot(NAME, { ...config.providers[NAME]!, apiKey: KEYS[1] }); + const abort = new AbortController(); + const pending = start(config, abort); + try { + await queue(config, "k1"); + config.providers[NAME]!.apiKey = KEYS[1]; + config.providers[NAME]!.apiKeySelectionRevision = "manual-key-switch"; + saveConfig(config); + heldA.release(); + await queue(config, "k2"); + expect(keyStatus(config, "k1")).toMatchObject({ queued: 0, inFlight: 0 }); + expect(seen).toEqual([]); + if (cancel) abort.abort(); + else heldB.release(); + const result = await pending; + if (cancel) { + expect(result.response.status).toBe(499); + expect(seen).toEqual([]); + heldB.release(); + } else { nativeSuccess(result); + expect(seen.map(entry => entry.key)).toEqual([KEYS[1]]); } + const saved = JSON.parse(readFileSync(join(home, "config.json"), "utf8")) as OcxConfig; + expect(config.providers[NAME]!.apiKey).toBe(KEYS[1]); + expect(saved.providers[NAME]!.apiKey).toBe(KEYS[1]); + expect(saved.providers[NAME]!.apiKeySelectionRevision).toBe("manual-key-switch"); + expect(saved.providers[NAME]!._apiKeyAttempt).toBeUndefined(); + noLeaks(config); + } finally { abort.abort(); heldA.release(); heldB.release(); await pending.catch(() => undefined); } +}, 10_000); diff --git a/tests/claude-integration/messages-native.test.ts b/tests/claude-integration/messages-native.test.ts index c6b46ea7bd7..583f21667e7 100644 --- a/tests/claude-integration/messages-native.test.ts +++ b/tests/claude-integration/messages-native.test.ts @@ -10,7 +10,7 @@ import { mkdtempSync, readFileSync } from "node:fs"; import { tmpdir } from "node:os"; import { join } from "node:path"; import { saveConfig } from "../../src/config"; -import { clearKeyCooldowns } from "../../src/providers/key-failover"; +import { clearKeyCooldowns, forgetApiKeyRotationCursor } from "../../src/providers/key-failover"; import { estimateClaudeRequestTokens, handleClaudeCountTokens, handleClaudeMessages } from "../../src/server/claude-messages"; import { getRequestLogEntries } from "../../src/server/request-log"; import type { OcxConfig } from "../../src/types"; @@ -363,6 +363,30 @@ describe("managed native Messages", () => { expect(seen[1]!.body).toEqual(seen[0]!.body); }); + test("round-robin native sends preserve per-request keys and the saved active key", async () => { + const config = fixtureConfig(startUpstream(), { pool: true }); + config.providers.anth!.requestPacing = { enabled: true, keySelection: "round-robin", keys: { maxConcurrentRequests: 1 } }; + forgetApiKeyRotationCursor("anth"); saveConfig(config); + const results = await Promise.all(Array.from({ length: 4 }, () => send(config, { ...SOURCE_BODY, stream: false }))); + expect(results.map(result => result.response.status)).toEqual([200, 200, 200, 200]); + expect(seen.map(entry => entry.headers.get("x-api-key")).sort()).toEqual([ + "fixture-key-alpha", "fixture-key-alpha", "fixture-key-beta", "fixture-key-beta", + ]); + expect(config.providers.anth!.apiKey).toBe("fixture-key-alpha"); + }); + + test("an exhausted distributed pool refuses the next native request without an upstream send", async () => { + const config = fixtureConfig(startUpstream(() => Response.json({ type: "error", error: { type: "rate_limit_error", message: "fixture cooldown" } }, + { status: 429, headers: { "retry-after": "30" } })), { pool: true }); + config.providers.anth!.requestPacing = { enabled: true, keySelection: "round-robin" }; + forgetApiKeyRotationCursor("anth"); saveConfig(config); + expect((await send(config, { ...SOURCE_BODY, stream: false })).response.status).toBe(429); + expect(seen).toHaveLength(2); + const next = await send(config, { ...SOURCE_BODY, stream: false }); + expect(next.response.status).toBe(429); expect(seen).toHaveLength(2); + expect(Number(next.response.headers.get("retry-after"))).toBeGreaterThan(0); + }); + test("a 429 on the first pooled key fails over to the next key", async () => { const config = fixtureConfig(startUpstream((entry, index) => index === 0 ? Response.json({ type: "error", error: { type: "rate_limit_error", message: "slow down" } }, diff --git a/tests/fixtures/test-layout-expected.json b/tests/fixtures/test-layout-expected.json index 67e3147d48d..08d569d2738 100644 --- a/tests/fixtures/test-layout-expected.json +++ b/tests/fixtures/test-layout-expected.json @@ -1,4 +1,6 @@ { + "api-key-selection-cooldown.test.ts": "providers", "server-chat-native-key-selection.test.ts": "server", "compact-key-selection.test.ts": "responses", + "messages-native-key-selection.test.ts": "claude-integration", "codex-history-pending-wal.test.ts": "codex-integration", "service-log.test.ts": "service", "service-health-sentinel-policy.test.ts": "service", @@ -959,7 +961,7 @@ "request-log-served-model.test.ts": "usage", "request-log-model-filter.test.ts": "usage", "request-log-generation-window.test.ts": "usage", "request-log.test.ts": "usage", "request-outcome-agreement.test.ts": "usage", "upstream-failure-diagnostics.test.ts": "usage", - "request-pacing.test.ts": "usage", "reserve-auth-context.test.ts": "codex-integration", + "request-pacing.test.ts": "usage", "request-pacing-provenance.test.ts": "usage", "reserve-auth-context.test.ts": "codex-integration", "reserve-availability.test.ts": "codex-integration", "reserve-catalog-lifecycle.test.ts": "codex-integration", "reserve-catalog.test.ts": "codex-integration", "reserve-claude-policy.test.ts": "server", "reserve-dispatch-ws.test.ts": "responses", "reserve-dispatch.test.ts": "codex-integration", @@ -984,6 +986,7 @@ "responses-custom-tool-stream-consistency.test.ts": "responses", "responses-default-namespace-emit-normalize.test.ts": "responses", "responses-fetch-helpers-boundary.test.ts": "responses", "responses-field-backfill.test.ts": "responses", + "responses-run-turn-key-pacing.test.ts": "responses", "responses-forward-dangling-call.test.ts": "responses", "responses-forward-incomplete-quota.test.ts": "responses", "responses-forward-posit-continuation.test.ts": "responses", "responses-forward-prompt-envelope.test.ts": "responses", "responses-freeform-wrapper-keys.test.ts": "responses", "responses-function-tool-repair.test.ts": "responses", @@ -1038,7 +1041,7 @@ "server-combo-zero-output-failover.test.ts": "server", "server-google-antigravity-oauth-401-replay.test.ts": "server", "server-antigravity-web-search-validation.test.ts": "server", "server-google-antigravity-oauth-429-budget.test.ts": "server", "server-gui-bundle-freshness.test.ts": "server", "server-images-bodyless-content-length.test.ts": "server", "server-images-pool-admission.test.ts": "server", - "server-images.test.ts": "server", "server-key-failover-e2e.test.ts": "server", + "server-images.test.ts": "server", "server-key-failover-e2e.test.ts": "server", "distributed-key-401.test.ts": "server", "server-jev-combo-e2e.test.ts": "server", "server-kiro-completion-e2e.test.ts": "server", "combo-requested-effort.test.ts": "server", "server-kiro-device-login.test.ts": "server", "server-kiro-oauth-401-replay.test.ts": "server", diff --git a/tests/providers/api-key-selection-cooldown.test.ts b/tests/providers/api-key-selection-cooldown.test.ts new file mode 100644 index 00000000000..d1f04eaf55f --- /dev/null +++ b/tests/providers/api-key-selection-cooldown.test.ts @@ -0,0 +1,158 @@ +import { afterEach, beforeEach, describe, expect, spyOn, test } from "bun:test"; +import { mkdtempSync, readFileSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { saveConfig } from "../../src/config"; +import { captureProviderApiKeySelection, providerApiKeySelectionIsCurrent, resolveCurrentProviderApiKeyTransport } from "../../src/providers/api-key-selection"; +import { clearKeyCooldowns, forgetApiKeyRotationCursor, getKeyCooldownUntil, recordDistributedKeyFailure, + rotateKeyOn401, selectProactiveApiKeyTransport } from "../../src/providers/key-failover"; +import { resetProviderRequestPacingForTest } from "../../src/providers/request-pacing"; +import { routedProviderConfig } from "../../src/router"; +import type { OcxConfig, OcxProviderConfig } from "../../src/types"; +import { acquireOwnedSpendHome } from "../helpers/owned-spend-home"; +import { removeTreeWithRetry } from "../helpers/remove-tree"; + +const NAME = "selection-cooldown"; +const A = "fixture-selection-alpha"; +const B = "fixture-selection-beta"; +let home: string; +let previousHome: string | undefined; +let releaseOwner: () => void; + +beforeEach(() => { + previousHome = process.env.OPENCODEX_HOME; + home = mkdtempSync(join(tmpdir(), "ocx-selection-cooldown-")); + process.env.OPENCODEX_HOME = home; + releaseOwner = acquireOwnedSpendHome(); + clearKeyCooldowns(); forgetApiKeyRotationCursor(); resetProviderRequestPacingForTest(); +}); +afterEach(() => { + releaseOwner(); clearKeyCooldowns(); forgetApiKeyRotationCursor(); resetProviderRequestPacingForTest(); + if (previousHome === undefined) delete process.env.OPENCODEX_HOME; + else process.env.OPENCODEX_HOME = previousHome; + removeTreeWithRetry(home); +}); + +function fixture(): OcxConfig { + return { port: 0, defaultProvider: NAME, providers: { [NAME]: { adapter: "anthropic", authMode: "key", + baseUrl: "https://example.test", apiKey: A, apiKeySelectionRevision: "revision-fixture", + apiKeyPool: [{ id: "a", key: A }, { id: "b", key: B }], + requestPacing: { enabled: true, keySelection: "round-robin", keys: { maxConcurrentRequests: 1 } }, + } } } as OcxConfig; +} +function binding(config: OcxConfig, index = 0): OcxProviderConfig { + const provider = config.providers[NAME]!; + const entry = provider.apiKeyPool![index]!; + return routedProviderConfig(NAME, { ...provider, apiKey: entry.key, + _apiKeyAttempt: { entryId: entry.id, reference: entry.key, revision: provider.apiKeySelectionRevision, distributed: true } }); +} +function cool(config: OcxConfig, route: OcxProviderConfig, now = Date.now(), retryAfter = "30") { + expect(recordDistributedKeyFailure(config, NAME, route._apiKeyAttempt, 429, now, retryAfter)).toBe(true); +} +function accepted(config: OcxConfig, route: OcxProviderConfig) { + expect(providerApiKeySelectionIsCurrent(config, NAME, route)).toBe(true); + expect(resolveCurrentProviderApiKeyTransport(config, NAME, route)?.apiKey).toBe(route.apiKey); +} +function rejected(config: OcxConfig, route: OcxProviderConfig) { + expect(providerApiKeySelectionIsCurrent(config, NAME, route)).toBe(false); + expect(resolveCurrentProviderApiKeyTransport(config, NAME, route)).toBeNull(); +} + +describe("distributed selection cooldown validation", () => { + test("rejects a cooled exact binding and resolver while its healthy sibling remains current", () => { + const config = fixture(); + const first = binding(config); const second = binding(config, 1); + accepted(config, first); accepted(config, second); + cool(config, first); + rejected(config, first); accepted(config, second); + expect(config.providers[NAME]!.apiKey).toBe(A); + expect(config.providers[NAME]!.apiKeySelectionRevision).toBe("revision-fixture"); + }); + + for (const field of ["entry", "reference", "revision"] as const) test(`rejects a stale distributed ${field} independently of cooldown`, () => { + const config = fixture(); const route = binding(config); + if (field === "entry") config.providers[NAME]!.apiKeyPool![0]!.id = "replacement-entry"; + if (field === "reference") config.providers[NAME]!.apiKeyPool![0]!.key = "fixture-replacement-reference"; + if (field === "revision") config.providers[NAME]!.apiKeySelectionRevision = "replacement-revision"; + expect(recordDistributedKeyFailure(config, NAME, route._apiKeyAttempt, 429)).toBe(false); + rejected(config, route); + expect(getKeyCooldownUntil(NAME, "a")).toBeNull(); + }); + + test("expiry makes the same distributed binding usable at the exact cooldown boundary", () => { + const config = fixture(); const route = binding(config); + const now = Date.now(); + const clock = spyOn(Date, "now").mockReturnValue(now); + try { + cool(config, route, now, "1"); + clock.mockReturnValue(now + 999); rejected(config, route); + expect(getKeyCooldownUntil(NAME, "a")).toBe(now + 1_000); + clock.mockReturnValue(now + 1_000); accepted(config, route); + expect(getKeyCooldownUntil(NAME, "a")).toBeNull(); + } finally { clock.mockRestore(); } + }); + + test("duplicate references cool together without cooling a healthy distinct reference", () => { + const config = fixture(); + config.providers[NAME]!.apiKeyPool!.splice(1, 0, { id: "a-duplicate", key: A }); + const first = binding(config); const duplicate = binding(config, 1); const sibling = binding(config, 2); + cool(config, duplicate); + expect(getKeyCooldownUntil(NAME, "a")).toBe(getKeyCooldownUntil(NAME, "a-duplicate")); + rejected(config, first); rejected(config, duplicate); accepted(config, sibling); + const replacement = selectProactiveApiKeyTransport(config, NAME, first)!; + expect(replacement.apiKey).toBe(B); + expect(replacement._apiKeyAttempt?.entryId).toBe("b"); + }); + + test("a shorter concurrent 429 cannot undo a 401 cooldown, including duplicate references", () => { + const config = fixture(); + config.providers[NAME]!.apiKeyPool!.splice(1, 0, { id: "a-duplicate", key: A }); + const first = binding(config); const duplicate = binding(config, 1); const sibling = binding(config, 2); + const now = Date.now(); + const clock = spyOn(Date, "now").mockReturnValue(now); + try { + expect(recordDistributedKeyFailure(config, NAME, first._apiKeyAttempt, 401, now)).toBe(true); + clock.mockReturnValue(now + 100); + expect(recordDistributedKeyFailure(config, NAME, duplicate._apiKeyAttempt, 429, now + 100, "1")).toBe(true); + expect(getKeyCooldownUntil(NAME, "a")).toBe(now + 600_000); + expect(getKeyCooldownUntil(NAME, "a-duplicate")).toBe(now + 600_000); + clock.mockReturnValue(now + 2_000); + rejected(config, first); rejected(config, duplicate); accepted(config, sibling); + expect(config.providers[NAME]!.apiKey).toBe(A); + expect(config.providers[NAME]!.apiKeySelectionRevision).toBe("revision-fixture"); + clearKeyCooldowns(NAME); + accepted(config, first); accepted(config, duplicate); + expect(getKeyCooldownUntil(NAME, "a")).toBeNull(); + expect(getKeyCooldownUntil(NAME, "a-duplicate")).toBeNull(); + } finally { clock.mockRestore(); } + }); + + test("a persistent binding keeps its existing validation semantics even when that entry is cooled", () => { + const config = fixture(); const distributed = binding(config); + cool(config, distributed); + delete config.providers[NAME]!.requestPacing; + const configured = config.providers[NAME]!; + const persistent = routedProviderConfig(NAME, { ...configured, _apiKeyAttempt: captureProviderApiKeySelection(configured) }); + accepted(config, persistent); + expect(recordDistributedKeyFailure(config, NAME, persistent._apiKeyAttempt, 429)).toBe(false); + }); + + test("default-off persistent mode retains committed 401 rotation and revision updates", () => { + const config = fixture(); delete config.providers[NAME]!.requestPacing; + saveConfig(config); + const route = routedProviderConfig(NAME, config.providers[NAME]!); + expect(selectProactiveApiKeyTransport(config, NAME, route)).toBeNull(); + expect(route._apiKeyAttempt?.distributed).toBeUndefined(); + expect(recordDistributedKeyFailure(config, NAME, route._apiKeyAttempt, 401)).toBe(false); + const replacement = rotateKeyOn401(config, NAME, Date.now(), A, route._apiKeyAttempt)!; + expect(replacement.apiKey).toBe(B); + expect(config.providers[NAME]!.apiKey).toBe(B); + expect(config.providers[NAME]!.apiKeySelectionRevision).not.toBe("revision-fixture"); + const persisted = JSON.parse(readFileSync(join(home, "config.json"), "utf8")) as OcxConfig; + expect(persisted.providers[NAME]!.apiKey).toBe(B); + expect(persisted.providers[NAME]!.apiKeySelectionRevision).toBe(config.providers[NAME]!.apiKeySelectionRevision); + expect(getKeyCooldownUntil(NAME, "a")).not.toBeNull(); + accepted(config, routedProviderConfig(NAME, config.providers[NAME]!)); + expect(providerApiKeySelectionIsCurrent(config, NAME, route)).toBe(false); + }); +}); diff --git a/tests/responses/chat-native-combo.test.ts b/tests/responses/chat-native-combo.test.ts index 1c17d98191b..b53cbe282d7 100644 --- a/tests/responses/chat-native-combo.test.ts +++ b/tests/responses/chat-native-combo.test.ts @@ -74,7 +74,7 @@ afterEach(async () => { }); /** A loopback upstream that records every body it receives. */ -function upstream(answer: (body: Rec, hit: number) => Response | Promise) { +function upstream(answer: (body: Rec, hit: number, headers: Headers) => Response | Promise) { const bodies: Rec[] = []; const server = Bun.serve({ hostname: "127.0.0.1", @@ -82,7 +82,7 @@ function upstream(answer: (body: Rec, hit: number) => Response | Promise { + test("distributed 401 recovery stays bounded and avoids the failed key on subsequent requests", async () => { + const keys: string[] = []; + const a = upstream((_body, hit, headers) => { + keys.push(headers.get("authorization")!); + return hit === 1 ? Response.json({ error: { message: "invalid key" } }, { status: 401 }) : twoChoiceCompletion(); + }); + const row = provider("openai-chat", a.baseUrl, { apiKey: "synthetic-first", + apiKeyPool: [{ id: "first", key: "synthetic-first" }, { id: "second", key: "synthetic-second" }], + requestPacing: { enabled: true, keySelection: "round-robin" } }); + const config = comboConfig({ a: row }, [{ provider: "a", model: "m1" }], NATIVE_ON); + const first = await send(config, { stream: false }); + expect(first.response.status).toBe(200); + expect(a.bodies).toHaveLength(2); + const second = await send(config, { stream: false }); + expect(second.response.status).toBe(200); + expect(a.bodies).toHaveLength(3); + expect(keys).toEqual(["Bearer synthetic-first", "Bearer synthetic-second", "Bearer synthetic-second"]); + expect(config.providers.a!.apiKey).toBe("synthetic-first"); + }); test("a native candidate receives the caller's own Chat body, n and logprobs included", async () => { const a = upstream(() => twoChoiceCompletion()); const b = upstream(() => responsesStream("bridge")); diff --git a/tests/responses/compact-key-selection.test.ts b/tests/responses/compact-key-selection.test.ts new file mode 100644 index 00000000000..f232b0e32e9 --- /dev/null +++ b/tests/responses/compact-key-selection.test.ts @@ -0,0 +1,282 @@ +import { afterEach, beforeEach, expect, test } from "bun:test"; +import { mkdtempSync, readFileSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { markResponseNonReplayable } from "../../src/lib/upstream-retry"; +import { saveConfig } from "../../src/config"; +import { apiKeyAccountLogLabel } from "../../src/codex/account-label"; +import { captureProviderApiKeySelection } from "../../src/providers/api-key-selection"; +import { clearKeyCooldowns, forgetApiKeyRotationCursor, getKeyCooldownUntil, recordDistributedKeyFailure } from "../../src/providers/key-failover"; +import { providerRequestPacingStatus, resetProviderRequestPacingForTest, waitForProviderRequestSlot } from "../../src/providers/request-pacing"; +import { handleResponsesCompact } from "../../src/server/responses"; +import type { RequestLogContext } from "../../src/server/request-log"; +import type { OcxConfig } from "../../src/types"; +import { acquireOwnedSpendHome } from "../helpers/owned-spend-home"; +import { removeCompactionFixture } from "../helpers/compaction-routing-fixtures"; + +const NAME = "openai-apikey"; +const KEYS = ["fixture-key-alpha", "fixture-key-beta", "fixture-key-gamma"]; +const realFetch = globalThis.fetch; +let home: string; +let previousHome: string | undefined; +let releaseOwner: () => void; +let seen: { key: string | null; url: string; body: unknown; headers: Headers }[]; +beforeEach(() => { + previousHome = process.env.OPENCODEX_HOME; + home = mkdtempSync(join(tmpdir(), "ocx-compact-key-pacing-")); + process.env.OPENCODEX_HOME = home; + releaseOwner = acquireOwnedSpendHome(); + seen = []; + clearKeyCooldowns(); forgetApiKeyRotationCursor(); resetProviderRequestPacingForTest(); + globalThis.fetch = (async (input, init) => { + seen.push({ key: new Headers(init?.headers).get("authorization"), url: String(input), body: JSON.parse(String(init?.body)), headers: new Headers(init?.headers) }); + return Response.json({ output: [{ type: "message", role: "assistant", content: [{ type: "output_text", text: "fixture reply" }] }] }); + }) as typeof fetch; +}); +afterEach(async () => { + globalThis.fetch = realFetch; + releaseOwner(); + clearKeyCooldowns(); forgetApiKeyRotationCursor(); resetProviderRequestPacingForTest(); + await removeCompactionFixture(home); + if (previousHome === undefined) delete process.env.OPENCODEX_HOME; + else process.env.OPENCODEX_HOME = previousHome; +}); +function fixture(keyCap = true): OcxConfig { + const config = { port: 0, defaultProvider: NAME, providers: { [NAME]: { + adapter: "openai-responses", baseUrl: "https://api.openai.com/v1", authMode: "key", + apiKey: KEYS[0], apiKeySelectionRevision: "saved-revision", + apiKeyPool: KEYS.map((key, index) => ({ id: `k${index + 1}`, key, addedAt: index + 1 })), + requestPacing: { enabled: true, keySelection: "round-robin", + ...(keyCap ? { keys: { maxConcurrentRequests: 1 } } : { maxConcurrentRequests: 1 }) }, + } } } as OcxConfig; + saveConfig(config); + return config; +} +function status(config: OcxConfig) { return providerRequestPacingStatus(NAME, config.providers[NAME]!); } +function keyStatus(config: OcxConfig, id: string) { return status(config).keys!.find(key => key.id === id)!; } +function cool(config: OcxConfig, index: number) { + const selection = { ...captureProviderApiKeySelection({ ...config.providers[NAME]!, apiKey: KEYS[index] }), distributed: true }; + expect(recordDistributedKeyFailure(config, NAME, selection, 429, Date.now(), "30")).toBe(true); +} +async function queue(config: OcxConfig, id: string) { + const deadline = Date.now() + 2_000; + while (keyStatus(config, id).queued !== 1) { + if (Date.now() >= deadline) throw new Error(`key ${id} never queued: ${JSON.stringify(status(config))}`); + await Bun.sleep(5); + } +} +function start(config: OcxConfig, abort: AbortController) { + const logCtx: RequestLogContext = { model: "", provider: "" }; + const req = new Request("http://localhost/v1/responses/compact", { method: "POST", headers: { "content-type": "application/json" }, + signal: AbortSignal.any([abort.signal, AbortSignal.timeout(5_000)]), body: JSON.stringify({ model: `${NAME}/test`, input: [{ role: "user", content: "fixture question" }], stream: false }) }); + const pending = handleResponsesCompact(req, config, logCtx).then(async response => ({ response, text: await response.text(), logCtx })); + void pending.catch(() => {}); + return pending; +} +function unchanged(config: OcxConfig) { + expect(config.providers[NAME]!.apiKey).toBe(KEYS[0]); + expect(config.providers[NAME]!.apiKeySelectionRevision).toBe("saved-revision"); + const saved = JSON.parse(readFileSync(join(home, "config.json"), "utf8")) as OcxConfig; + expect(saved.providers[NAME]!.apiKey).toBe(KEYS[0]); + expect(saved.providers[NAME]!.apiKeySelectionRevision).toBe("saved-revision"); + expect(saved.providers[NAME]!._apiKeyAttempt).toBeUndefined(); +} +function noLeaks(config: OcxConfig) { + expect(status(config)).toMatchObject({ queued: 0, inFlight: 0 }); + for (const key of status(config).keys!) expect(key).toMatchObject({ queued: 0, inFlight: 0 }); +} +function success(result: Awaited>, index: number) { + expect(result.response.status).toBe(200); + expect(result.text).toContain("fixture reply"); + expect(seen.map(entry => entry.key)).toEqual([`Bearer ${KEYS[index]}`]); + expect(seen[0]!.url).toBe("https://api.openai.com/v1/responses/compact"); + expect(result.logCtx.accountLogLabel).toBe(apiKeyAccountLogLabel(NAME, { entryId: `k${index + 1}`, reference: KEYS[index] })); +} +for (const cancel of [false, true]) test(`compact cooled queued A readmits B (abort=${cancel})`, async () => { + const config = fixture(); + const first = await waitForProviderRequestSlot(NAME, config.providers[NAME]!); + const second = await waitForProviderRequestSlot(NAME, { ...config.providers[NAME]!, apiKey: KEYS[1] }); + const abort = new AbortController(); + const pending = start(config, abort); + try { + await queue(config, "k1"); cool(config, 0); first.release(); + await queue(config, "k2"); + expect(keyStatus(config, "k1")).toMatchObject({ inFlight: 0, queued: 0, starts: 2 }); + expect(keyStatus(config, "k2")).toMatchObject({ inFlight: 1, queued: 1, starts: 1 }); + expect(seen).toEqual([]); + if (cancel) { + abort.abort(); + const result = await pending; + expect(result.response.status).toBe(499); + expect(seen).toEqual([]); + expect(status(config).queued).toBe(0); + second.release(); + } else { second.release(); success(await pending, 1); } + unchanged(config); noLeaks(config); + } finally { abort.abort(); first.release(); second.release(); await pending.catch(() => undefined); } +}, 10_000); +for (const exhausted of [false, true]) test(`compact revalidates B cooling during readmission (exhausted=${exhausted})`, async () => { + const config = fixture(); + const first = await waitForProviderRequestSlot(NAME, config.providers[NAME]!); + const second = await waitForProviderRequestSlot(NAME, { ...config.providers[NAME]!, apiKey: KEYS[1] }); + const abort = new AbortController(); + const pending = start(config, abort); + try { + await queue(config, "k1"); cool(config, 0); first.release(); + await queue(config, "k2"); cool(config, 1); + if (exhausted) cool(config, 2); + second.release(); + const result = await pending; + if (exhausted) { + expect(result.response.status).toBe(429); + expect(Number(result.response.headers.get("retry-after"))).toBeGreaterThan(0); + expect(seen).toEqual([]); + } else success(result, 2); + unchanged(config); noLeaks(config); + } finally { abort.abort(); first.release(); second.release(); await pending.catch(() => undefined); } +}, 10_000); +test("compact reselects behind the provider cap without a keys rule", async () => { + const config = fixture(false); + const held = await waitForProviderRequestSlot(NAME, config.providers[NAME]!); + const abort = new AbortController(); + const pending = start(config, abort); + try { + await queue(config, "k1"); cool(config, 0); held.release(); + success(await pending, 1); + expect(keyStatus(config, "k1").starts).toBe(2); + expect(keyStatus(config, "k2").starts).toBe(1); + unchanged(config); noLeaks(config); + } finally { abort.abort(); held.release(); await pending.catch(() => undefined); } +}); +test("compact refreshes a queued stale revision and sends rebuilt Authorization and static headers", async () => { + const config = fixture(); + const held = await waitForProviderRequestSlot(NAME, config.providers[NAME]!); + const abort = new AbortController(); + const pending = start(config, abort); + try { + await queue(config, "k1"); + config.providers[NAME]!.apiKeySelectionRevision = "updated-revision"; + config.providers[NAME]!.headers = { "x-current-static": "replacement" }; + held.release(); + const result = await pending; + expect(result.response.status).toBe(200); + expect(seen).toHaveLength(1); + expect(seen[0]!.url).toBe("https://api.openai.com/v1/responses/compact"); + expect(seen[0]!.key).toBe(`Bearer ${KEYS[1]}`); + expect(seen[0]!.headers.get("x-current-static")).toBe("replacement"); + expect(result.logCtx.accountLogLabel).toBe(apiKeyAccountLogLabel(NAME, { entryId: "k2", reference: KEYS[1] })); + noLeaks(config); + } finally { abort.abort(); held.release(); await pending.catch(() => undefined); } +}); + +test("compact refuses after three queued rebuilds without spending a physical send", async () => { + const config = fixture(); + const extra = ["fixture-key-delta", "fixture-key-epsilon"]; + config.providers[NAME]!.apiKeyPool!.push(...extra.map((key, index) => ({ id: `k${index + 4}`, key }))); + const keys = [...KEYS, ...extra]; + const held = await Promise.all(keys.map(apiKey => waitForProviderRequestSlot(NAME, { ...config.providers[NAME]!, apiKey }))); + const abort = new AbortController(); + const pending = start(config, abort); + try { + for (let index = 0; index < 4; index++) { + await queue(config, `k${index + 1}`); + const selection = { ...captureProviderApiKeySelection({ ...config.providers[NAME]!, apiKey: keys[index] }), distributed: true }; + expect(recordDistributedKeyFailure(config, NAME, selection, 429, Date.now(), "30")).toBe(true); + held[index]!.release(); + } + const result = await pending; + expect(result.response.status).toBe(502); + expect(seen).toEqual([]); + expect(keyStatus(config, "k5").starts).toBe(1); + for (const slot of held) slot.release(); + unchanged(config); noLeaks(config); + } finally { abort.abort(); for (const slot of held) slot.release(); await pending.catch(() => undefined); } +}, 10_000); + +test("compact 401 after queued reselection cools the physical key B without replaying", async () => { + const config = fixture(); + const first = await waitForProviderRequestSlot(NAME, config.providers[NAME]!); + const abort = new AbortController(); + const wireFetch = globalThis.fetch; + globalThis.fetch = (async (input, init) => { + await wireFetch(input, init); + return Response.json({ error: { message: "invalid fixture key" } }, { status: 401 }); + }) as typeof fetch; + const pending = start(config, abort); + try { + await queue(config, "k1"); cool(config, 0); + const firstCooldown = getKeyCooldownUntil(NAME, "k1"); + first.release(); + const result = await pending; + expect(result.response.status).toBe(401); + expect(seen.map(entry => entry.key)).toEqual([`Bearer ${KEYS[1]}`]); + expect(result.logCtx.accountLogLabel).toBe(apiKeyAccountLogLabel(NAME, { entryId: "k2", reference: KEYS[1] })); + expect(getKeyCooldownUntil(NAME, "k1")).toBe(firstCooldown); + expect(getKeyCooldownUntil(NAME, "k2")).not.toBeNull(); + expect(getKeyCooldownUntil(NAME, "k3")).toBeNull(); + unchanged(config); noLeaks(config); + } finally { abort.abort(); first.release(); await pending.catch(() => undefined); } +}); + +test("direct compact distributed 429 cools each physical key without same-request replay", async () => { + const config = fixture(); + globalThis.fetch = (async (input, init) => { + seen.push({ key: new Headers(init?.headers).get("authorization"), url: String(input), body: JSON.parse(String(init?.body)), headers: new Headers(init?.headers) }); + return Response.json({ error: { message: "fixture capacity" } }, { status: 429, headers: { "retry-after": "60" } }); + }) as typeof fetch; + const began = Date.now(); + for (let index = 0; index < 4; index++) { + const result = await start(config, new AbortController()); + expect(result.response.status).toBe(429); + expect(seen).toHaveLength(Math.min(index + 1, 3)); + } + expect(seen.map(row => row.key)).toEqual(KEYS.map(key => `Bearer ${key}`)); + for (let index = 0; index < 3; index++) expect(getKeyCooldownUntil(NAME, `k${index + 1}`)).toBeGreaterThanOrEqual(began + 60_000); + unchanged(config); noLeaks(config); +}); + +test("a synthetic non-replayable compact 429 never cools a distributed key", async () => { + const config = fixture(); + globalThis.fetch = (async () => { + const response = Response.json({ error: { message: "fixture local refusal" } }, { status: 429, headers: { "retry-after": "60" } }); + markResponseNonReplayable(response); return response; + }) as typeof fetch; + const result = await start(config, new AbortController()); + expect(result.response.status).toBe(429); + for (let index = 0; index < 3; index++) expect(getKeyCooldownUntil(NAME, `k${index + 1}`)).toBeNull(); + unchanged(config); noLeaks(config); +}); + +for (const cancel of [false, true]) test(`active manual key switch readmits the selected key (cancel=${cancel})`, async () => { + const config = fixture(); + config.providers[NAME]!.requestPacing = { enabled: true, keys: { maxConcurrentRequests: 1 } }; + saveConfig(config); + const heldA = await waitForProviderRequestSlot(NAME, config.providers[NAME]!); + const heldB = await waitForProviderRequestSlot(NAME, { ...config.providers[NAME]!, apiKey: KEYS[1] }); + const abort = new AbortController(); + const pending = start(config, abort); + try { + await queue(config, "k1"); + config.providers[NAME]!.apiKey = KEYS[1]; + config.providers[NAME]!.apiKeySelectionRevision = "manual-key-switch"; + saveConfig(config); + heldA.release(); + await queue(config, "k2"); + expect(keyStatus(config, "k1")).toMatchObject({ queued: 0, inFlight: 0 }); + expect(seen).toEqual([]); + if (cancel) abort.abort(); + else heldB.release(); + const result = await pending; + if (cancel) { + expect(result.response.status).toBe(499); + expect(seen).toEqual([]); + heldB.release(); + } else { success(result, 1); } + const saved = JSON.parse(readFileSync(join(home, "config.json"), "utf8")) as OcxConfig; + expect(config.providers[NAME]!.apiKey).toBe(KEYS[1]); + expect(saved.providers[NAME]!.apiKey).toBe(KEYS[1]); + expect(saved.providers[NAME]!.apiKeySelectionRevision).toBe("manual-key-switch"); + expect(saved.providers[NAME]!._apiKeyAttempt).toBeUndefined(); + noLeaks(config); + } finally { abort.abort(); heldA.release(); heldB.release(); await pending.catch(() => undefined); } +}, 10_000); diff --git a/tests/responses/responses-fetch-helpers-boundary.test.ts b/tests/responses/responses-fetch-helpers-boundary.test.ts index e6963d05749..6e10829342f 100644 --- a/tests/responses/responses-fetch-helpers-boundary.test.ts +++ b/tests/responses/responses-fetch-helpers-boundary.test.ts @@ -310,4 +310,19 @@ describe("storedPoolReplayDispatchNotifier", () => { const executor = pacedExecutor({ pacing: async () => {} }); expect(storedPoolReplayDispatchNotifier(executor.wrapped, undefined)).toBe(executor.wrapped); }); + test.each([false, true])("notification failure releases the slot (pre-acquired=%s)", async preAcquired => { + let releases = 0, sends = 0; + const slot = { leased: true, bodyTracked: false, released: false, release() { releases += 1; } }; + const execute = Object.assign(async () => new Response("unexpected"), { + preconnect() {}, + waitForPacing: async () => slot, + fetchWithPacingSlot: async () => { sends += 1; return new Response("unexpected"); }, + }); + const notifier = storedPoolReplayDispatchNotifier(execute, () => { throw new Error("notification failed"); }); + await expect((async () => preAcquired + ? notifier.fetchWithPacingSlot!(slot, "https://example.test") + : notifier("https://example.test"))()).rejects.toThrow("notification failed"); + expect(releases).toBe(1); + expect(sends).toBe(0); + }); }); diff --git a/tests/responses/responses-run-turn-key-pacing.test.ts b/tests/responses/responses-run-turn-key-pacing.test.ts new file mode 100644 index 00000000000..ef5bc160159 --- /dev/null +++ b/tests/responses/responses-run-turn-key-pacing.test.ts @@ -0,0 +1,122 @@ +import { expect, spyOn, test } from "bun:test"; +import { mkdtempSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { ADAPTER_REGISTRY } from "../../src/adapters/registry"; +import { handleResponses } from "../../src/server/responses"; +import { providerRequestPacingStatus, resetProviderRequestPacingForTest, waitForProviderRequestSlot } from "../../src/providers/request-pacing"; +import type { OcxConfig } from "../../src/types"; +import { acquireOwnedSpendHome } from "../helpers/owned-spend-home"; +import { removeTreeWithRetry } from "../helpers/remove-tree"; + +test.each([false, true])("runTurn refresh releases the old key and readmits the replacement (abort=%s)", async abortWhileWaiting => { + const previousHome = process.env.OPENCODEX_HOME; + const home = mkdtempSync(join(tmpdir(), "ocx-runturn-pacing-")); + process.env.OPENCODEX_HOME = home; + resetProviderRequestPacingForTest(); + const releaseOwner = acquireOwnedSpendHome(); + const seen: string[] = []; + const create = ADAPTER_REGISTRY["openai-chat"].create; + const override = spyOn(ADAPTER_REGISTRY["openai-chat"], "create").mockImplementation((...args) => ({ + ...create(...args), + async runTurn(_parsed, _incoming, emit) { + seen.push(args[0].apiKey!); + emit({ type: "text_delta", text: "readmitted turn" }); emit({ type: "done" }); + }, + })); + const config = { port: 0, defaultProvider: "paced", providers: { paced: { + adapter: "openai-chat", baseUrl: "https://example.test/v1", authMode: "key", apiKey: "synthetic-first", + apiKeyPool: [{ id: "first", key: "synthetic-first" }, { id: "second", key: "synthetic-second" }], + requestPacing: { enabled: true, keys: { maxConcurrentRequests: 1 } }, + } } } as OcxConfig; + const first = await waitForProviderRequestSlot("paced", config.providers.paced); + const second = await waitForProviderRequestSlot("paced", { ...config.providers.paced, apiKey: "synthetic-second" }); + const abort = new AbortController(); + let pending: Promise | undefined; + const waitForKeyQueue = async (id: string) => { + for (let i = 0; i < 200; i++) { + if (providerRequestPacingStatus("paced", config.providers.paced).keys?.find(key => key.id === id)?.queued === 1) return; + await Bun.sleep(5); + } + throw new Error(`key ${id} never queued`); + }; + try { + pending = handleResponses(new Request("http://localhost/v1/responses", { + method: "POST", headers: { "content-type": "application/json" }, signal: abort.signal, + body: JSON.stringify({ model: "paced/test", input: "hello", stream: false }), + }), config, { provider: "", model: "" }, { abortSignal: abort.signal }); + // Observe any rejected handler immediately while the test controls its queue. + void pending.catch(() => {}); + await waitForKeyQueue("first"); + config.providers.paced.apiKey = "synthetic-second"; + config.providers.paced.apiKeySelectionRevision = "replacement"; + first.release(); + await waitForKeyQueue("second"); + expect(seen).toEqual([]); + expect(providerRequestPacingStatus("paced", config.providers.paced).keys?.find(key => key.id === "first")?.inFlight).toBe(0); + if (abortWhileWaiting) { + abort.abort(); + await pending.catch(() => undefined); + expect(seen).toEqual([]); + expect(providerRequestPacingStatus("paced", config.providers.paced).queued).toBe(0); + } else { + second.release(); + const response = await pending; + expect(response.status).toBe(200); + expect(await response.text()).toContain("readmitted turn"); + expect(seen).toEqual(["synthetic-second"]); + } + } finally { + abort.abort(); first.release(); second.release(); + await pending?.catch(() => undefined); + override.mockRestore(); resetProviderRequestPacingForTest(); releaseOwner(); removeTreeWithRetry(home); + if (previousHome === undefined) delete process.env.OPENCODEX_HOME; + else process.env.OPENCODEX_HOME = previousHome; + } +}); + +test("a distributed runTurn does not send again after its bound key cools following output", async () => { + const { recordDistributedKeyFailure, clearKeyCooldowns, forgetApiKeyRotationCursor } = await import("../../src/providers/key-failover"); + const previousHome = process.env.OPENCODEX_HOME; + const home = mkdtempSync(join(tmpdir(), "ocx-runturn-cooled-")); + process.env.OPENCODEX_HOME = home; + resetProviderRequestPacingForTest(); clearKeyCooldowns(); forgetApiKeyRotationCursor("paced"); + const releaseOwner = acquireOwnedSpendHome(); + const config = { port: 0, defaultProvider: "paced", providers: { paced: { + adapter: "openai-chat", baseUrl: "https://example.test/v1", authMode: "key", apiKey: "synthetic-first", + apiKeySelectionRevision: "stable", + apiKeyPool: [{ id: "first", key: "synthetic-first" }, { id: "second", key: "synthetic-second" }], + requestPacing: { enabled: true, keySelection: "round-robin", keys: { maxConcurrentRequests: 1 } }, + } } } as OcxConfig; + let sends = 0; + const outgoing = spyOn(globalThis, "fetch").mockImplementation(async () => { + sends++; return new Response("physical response"); + }); + const create = ADAPTER_REGISTRY["openai-chat"].create; + const override = spyOn(ADAPTER_REGISTRY["openai-chat"], "create").mockImplementation((...args) => ({ + ...create(...args), + async runTurn(_parsed, incoming, emit) { + const first = await incoming.providerFetch!("https://example.test/v1/messages", { method: "POST" }); + await first.text(); + emit({ type: "text_delta", text: "established turn" }); + expect(recordDistributedKeyFailure(config, "paced", args[0]._apiKeyAttempt, 401)).toBe(true); + await expect(incoming.providerFetch!("https://example.test/v1/messages", { method: "POST" })).rejects.toThrow("unavailable during the turn"); + emit({ type: "done" }); + }, + })); + try { + const response = await handleResponses(new Request("http://localhost/v1/responses", { + method: "POST", headers: { "content-type": "application/json" }, + body: JSON.stringify({ model: "paced/test", input: "hello", stream: false }), + }), config, { provider: "", model: "" }); + expect(response.status).toBe(200); + expect(await response.text()).toContain("established turn"); + expect(sends).toBe(1); + expect(config.providers.paced.apiKey).toBe("synthetic-first"); + expect(providerRequestPacingStatus("paced", config.providers.paced).inFlight).toBe(0); + } finally { + override.mockRestore(); outgoing.mockRestore(); clearKeyCooldowns(); forgetApiKeyRotationCursor("paced"); + resetProviderRequestPacingForTest(); releaseOwner(); removeTreeWithRetry(home); + if (previousHome === undefined) delete process.env.OPENCODEX_HOME; else process.env.OPENCODEX_HOME = previousHome; + } +}); diff --git a/tests/server/distributed-key-401.test.ts b/tests/server/distributed-key-401.test.ts new file mode 100644 index 00000000000..bea402e972f --- /dev/null +++ b/tests/server/distributed-key-401.test.ts @@ -0,0 +1,134 @@ +import { afterEach, beforeEach, expect, test } from "bun:test"; +import { mkdtempSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { saveConfig } from "../../src/config"; +import { clearKeyCooldowns, forgetApiKeyRotationCursor, getKeyCooldownUntil } from "../../src/providers/key-failover"; +import { startServer } from "../../src/server"; +import { handleResponsesCompact } from "../../src/server/responses"; +import type { OcxConfig } from "../../src/types"; +import { installIsolatedCodexHome, type IsolatedCodexHome } from "../helpers/isolated-codex-home"; +import { acquireOwnedSpendHome } from "../helpers/owned-spend-home"; +import { baseCompactionBody, compactionRequest, completedPayload, jsonResponse, removeCompactionFixture } from "../helpers/compaction-routing-fixtures"; +import { fakeChatGptJwt } from "../helpers/fake-chatgpt-jwt"; + +const originalFetch = globalThis.fetch; +const DIRECT_CHATGPT_TOKEN = fakeChatGptJwt({ chatgpt_account_id: "fixture" }); +const disabledOpenAiProvider = { adapter: "openai-responses", baseUrl: "https://chatgpt.com/backend-api/codex", authMode: "forward", disabled: true }; +function keyedProvider(_baseUrl: string) { return { adapter: "openai-responses", baseUrl: "https://api.openai.com/v1", apiKey: "sk-platform-key" }; } +let home: string; +let previousHome: string | undefined; +let previousToken: string | undefined; +let codexHome: IsolatedCodexHome; +let releaseSpendHome: (() => void) | undefined; +const takeSpendHome = () => { releaseSpendHome ??= acquireOwnedSpendHome(); }; +beforeEach(() => { + previousHome = process.env.OPENCODEX_HOME; + previousToken = process.env.OPENCODEX_API_AUTH_TOKEN; + home = mkdtempSync(join(tmpdir(), "ocx-distributed-401-")); + process.env.OPENCODEX_HOME = home; + delete process.env.OPENCODEX_API_AUTH_TOKEN; + codexHome = installIsolatedCodexHome("ocx-distributed-401-codex-"); + clearKeyCooldowns(); forgetApiKeyRotationCursor(); +}); +afterEach(async () => { + globalThis.fetch = originalFetch; + releaseSpendHome?.(); releaseSpendHome = undefined; + await removeCompactionFixture(home); + codexHome.restore(); + if (previousHome === undefined) delete process.env.OPENCODEX_HOME; + else process.env.OPENCODEX_HOME = previousHome; + if (previousToken === undefined) delete process.env.OPENCODEX_API_AUTH_TOKEN; + else process.env.OPENCODEX_API_AUTH_TOKEN = previousToken; + clearKeyCooldowns(); forgetApiKeyRotationCursor(); +}); + +test("distributed image 401 cools the failed key without replaying the paid POST", async () => { + const seen: string[] = []; + const upstream = Bun.serve({ hostname: "127.0.0.1", port: 0, fetch(req) { + seen.push(req.headers.get("authorization")!); + return Response.json(seen.length === 1 ? { error: "invalid key" } : { data: [] }, { status: seen.length === 1 ? 401 : 200 }); + } }); + globalThis.fetch = ((input: RequestInfo | URL, init?: RequestInit) => { + const url = new URL(typeof input === "string" ? input : input instanceof URL ? input.href : input.url); + return originalFetch(url.hostname === "api.openai.com" ? new URL(url.pathname, upstream.url) : input, init); + }) as typeof fetch; + clearKeyCooldowns(); + saveConfig({ port: 0, defaultProvider: "openai-apikey", openaiProviderTierVersion: 2, + providers: { openai: disabledOpenAiProvider, "openai-apikey": { + ...keyedProvider(upstream.url.toString().replace(/\/$/, "")), + apiKeyPool: [{ id: "first", key: "sk-platform-key" }, { id: "second", key: "sk-warm-key" }], + requestPacing: { enabled: true, keySelection: "round-robin" }, + } }, + } as unknown as OcxConfig); + const server = startServer(0); + try { + for (const [index, status] of [401, 200, 200].entries()) { + const response = await fetch(new URL("/v1/images/generations", server.url), { method: "POST", + headers: { "content-type": "application/json", authorization: `Bearer ${DIRECT_CHATGPT_TOKEN}` }, + body: JSON.stringify({ prompt: "fixture", model: "gpt-image-2" }), + }); + expect(response.status).toBe(status); + await response.text(); + expect(seen.length).toBe(index + 1); + } + expect(seen).toEqual(["Bearer sk-platform-key", "Bearer sk-warm-key", "Bearer sk-warm-key"]); + } finally { await server.stop(true); await upstream.stop(true); clearKeyCooldowns(); } +}); + +test("native compact distributed 401 cools the static key without replaying or refreshing an account", async () => { + takeSpendHome(); + clearKeyCooldowns("openai-apikey"); + const config = { defaultProvider: "openai-apikey", openaiProviderTierVersion: 2, providers: { + "openai-apikey": { adapter: "openai-responses", baseUrl: "https://api.openai.com/v1", authMode: "key", + apiKey: "synthetic-first", apiKeyPool: [{ id: "first", key: "synthetic-first" }, { id: "second", key: "synthetic-second" }], + requestPacing: { enabled: true, keySelection: "round-robin" }, + }, + } } as OcxConfig; + const keys: string[] = []; + globalThis.fetch = (async (input, init) => { + expect(String(input)).toBe("https://api.openai.com/v1/responses/compact"); + keys.push(new Headers(init?.headers).get("authorization")!); + return keys.length === 1 ? Response.json({ error: { message: "invalid key" } }, { status: 401 }) : jsonResponse(completedPayload("compact fixture")); + }) as typeof fetch; + try { + for (const [index, status] of [401, 200, 200].entries()) { + const response = await handleResponsesCompact(compactionRequest(baseCompactionBody({ model: "openai-apikey/gpt-5.5" })), config, { model: "", provider: "" }); + expect(response.status).toBe(status); + await response.text(); + expect(keys).toHaveLength(index + 1); + } + expect(keys).toEqual(["Bearer synthetic-first", "Bearer synthetic-second", "Bearer synthetic-second"]); + } finally { clearKeyCooldowns("openai-apikey"); } +}); + +test("direct distributed Images 429 cools both keys without replaying the paid POST", async () => { + const seen: string[] = []; + const upstream = Bun.serve({ hostname: "127.0.0.1", port: 0, fetch(req) { + seen.push(req.headers.get("authorization")!); + return Response.json({ error: { message: "fixture capacity" } }, { status: 429, headers: { "retry-after": "60" } }); + } }); + globalThis.fetch = ((input: RequestInfo | URL, init?: RequestInit) => { + const url = new URL(typeof input === "string" ? input : input instanceof URL ? input.href : input.url); + return originalFetch(url.hostname === "api.openai.com" ? new URL(url.pathname, upstream.url) : input, init); + }) as typeof fetch; + saveConfig({ port: 0, defaultProvider: "openai-apikey", openaiProviderTierVersion: 2, + providers: { openai: disabledOpenAiProvider, "openai-apikey": { + ...keyedProvider(upstream.url.toString()), apiKeyPool: [{ id: "first", key: "sk-platform-key" }, { id: "second", key: "sk-warm-key" }], + requestPacing: { enabled: true, keySelection: "round-robin" }, + } }, + } as unknown as OcxConfig); + const server = startServer(0); const began = Date.now(); + try { + for (let index = 0; index < 3; index++) { + const response = await fetch(new URL("/v1/images/generations", server.url), { method: "POST", + headers: { "content-type": "application/json", authorization: `Bearer ${DIRECT_CHATGPT_TOKEN}` }, + body: JSON.stringify({ prompt: "fixture", model: "gpt-image-2" }), + }); + expect(response.status).toBe(429); await response.text(); + expect(seen).toHaveLength(Math.min(index + 1, 2)); + } + expect(seen).toEqual(["Bearer sk-platform-key", "Bearer sk-warm-key"]); + for (const id of ["first", "second"]) expect(getKeyCooldownUntil("openai-apikey", id)).toBeGreaterThanOrEqual(began + 60_000); + } finally { await server.stop(true); await upstream.stop(true); clearKeyCooldowns(); } +}); diff --git a/tests/server/fetch-header-timeout.test.ts b/tests/server/fetch-header-timeout.test.ts index 1ce097def08..d4f436ed40a 100644 --- a/tests/server/fetch-header-timeout.test.ts +++ b/tests/server/fetch-header-timeout.test.ts @@ -189,3 +189,110 @@ describe("#2567 the upstream fetch disables Bun's per-request idle timeout", () expect(init.signal).toBeInstanceOf(AbortSignal); }); }); + +describe("replacement key admission excludes the response-header deadline", () => { + async function replacementFixture(physicalFetch: typeof fetch) { + const { providerFetch } = await import("../../src/server/responses/fetch-helpers"); + const { waitForProviderRequestSlot, resetProviderRequestPacingForTest } = await import("../../src/providers/request-pacing"); + resetProviderRequestPacingForTest(); + const provider = { adapter: "openai-chat", baseUrl: "https://fixture.test/v1", authMode: "key", apiKey: "fixture-A", + apiKeyPool: [{ id: "a", key: "fixture-A", addedAt: 1 }, { id: "b", key: "fixture-B", addedAt: 2 }], + requestPacing: { enabled: true, keys: { maxConcurrentRequests: 1 } } } as import("../../src/types").OcxProviderConfig; + const replacement = { ...provider, apiKey: "fixture-B" }; + const held = await waitForProviderRequestSlot("header-replacement", replacement); + let notify!: () => void; + const queued = new Promise(resolve => { notify = resolve; }); + const executor = providerFetch({ ...provider, fetch: physicalFetch } as typeof provider, undefined, { providerName: "header-replacement", + dispatchOverride: async (input, init, execute, reacquire) => { + notify(); + await reacquire!(replacement); + return execute(input, init); + } }); + return { executor, held, queued, resetProviderRequestPacingForTest }; + } + + test("a blocked replacement may wait beyond the header timeout and then send", async () => { + let sends = 0; + const fixture = await replacementFixture((async () => { sends++; return new Response("ok"); }) as typeof fetch); + const pending = fetchWithHeaderTimeout("https://fixture.test/v1/chat/completions", {}, new AbortController().signal, 25, false, fixture.executor); + void pending.catch(() => {}); + try { + await fixture.queued; + await Bun.sleep(80); + expect(sends).toBe(0); + fixture.held.release(); + expect(await (await pending).text()).toBe("ok"); + expect(sends).toBe(1); + } finally { fixture.held.release(); fixture.resetProviderRequestPacingForTest(); } + }); + + test("the fresh header deadline still aborts a stalled physical send", async () => { + let sends = 0; + const fixture = await replacementFixture((async (_input, init) => { + sends++; + return new Promise((_resolve, reject) => { + if (init?.signal?.aborted) reject(init.signal.reason); + else init?.signal?.addEventListener("abort", () => reject(init.signal!.reason), { once: true }); + }); + }) as typeof fetch); + const pending = fetchWithHeaderTimeout("https://fixture.test/v1/chat/completions", {}, new AbortController().signal, 25, false, fixture.executor); + void pending.catch(() => {}); + try { + await fixture.queued; + await Bun.sleep(80); + fixture.held.release(); + await expect(pending).rejects.toMatchObject({ name: "TimeoutError" }); + expect(sends).toBe(1); + } finally { fixture.held.release(); fixture.resetProviderRequestPacingForTest(); } + }); + + test("client abort remains active while replacement admission waits", async () => { + let sends = 0; + const fixture = await replacementFixture((async () => { sends++; return new Response("ok"); }) as typeof fetch); + const controller = new AbortController(); + const pending = fetchWithHeaderTimeout("https://fixture.test/v1/chat/completions", {}, controller.signal, 25, false, fixture.executor); + void pending.catch(() => {}); + try { + await fixture.queued; + controller.abort(new DOMException("client cancelled", "AbortError")); + await expect(pending).rejects.toMatchObject({ name: "AbortError" }); + expect(sends).toBe(0); + } finally { fixture.held.release(); fixture.resetProviderRequestPacingForTest(); } + }); +}); + +for (const policy of [ + { enabled: true, keySelection: "round-robin", minIntervalMs: 1 }, + { enabled: true, keys: { minIntervalMs: 1 } }, +] as const) test(`key pacing keeps WebSocket requests on the guarded HTTP path: ${JSON.stringify(policy)}`, async () => { + const { providerFetch } = await import("../../src/server/responses/fetch-helpers"); + const { resetProviderRequestPacingForTest } = await import("../../src/providers/request-pacing"); + const { BOUNDED_WS_RUNTIME } = await import("../helpers/ws-upstream-fixtures"); + const previousSocket = globalThis.WebSocket; + let sockets = 0; let overrides = 0; const sentKeys: (string | null)[] = []; + globalThis.WebSocket = class { constructor() { sockets++; throw new Error("unexpected WebSocket dispatch"); } } as unknown as typeof WebSocket; + const physical = (async (_input, init) => { + sentKeys.push(new Headers(init?.headers).get("authorization")); + return new Response("fixture reply"); + }) as typeof fetch; + const provider = { adapter: "openai-responses", authMode: "key", baseUrl: "https://api.openai.com/v1", + upstreamWebsocket: true, apiKey: "fixture-A", requestPacing: policy, fetch: physical, + } as import("../../src/types").OcxProviderConfig; + try { + const executor = providerFetch(provider, BOUNDED_WS_RUNTIME, { providerName: "ws-key-pacing", + dispatchOverride: async (input, init, execute, reacquire) => { + overrides++; + await reacquire!({ ...provider, apiKey: "fixture-B" }); + return execute(input, { ...init, headers: { authorization: "Bearer fixture-B" } }); + }, + }); + const response = await executor("https://api.openai.com/v1/responses", { method: "POST", + headers: { authorization: "Bearer fixture-A", "content-type": "application/json" }, + body: JSON.stringify({ model: "fixture-model", stream: true, input: [] }), + }); + expect(await response.text()).toBe("fixture reply"); + expect(overrides).toBe(1); + expect(sockets).toBe(0); + expect(sentKeys).toEqual(["Bearer fixture-B"]); + } finally { globalThis.WebSocket = previousSocket; resetProviderRequestPacingForTest(); } +}); diff --git a/tests/server/management-provider-request-pacing.test.ts b/tests/server/management-provider-request-pacing.test.ts index f42b0e3a722..1a99bed4d78 100644 --- a/tests/server/management-provider-request-pacing.test.ts +++ b/tests/server/management-provider-request-pacing.test.ts @@ -36,6 +36,7 @@ describe("provider management request pacing", () => { adapter: "openai-chat", baseUrl: "https://integrate.api.nvidia.com/v1", apiKey: "sk-nvidia", + apiKeyPool: [{ id: "fixture-1", key: "sk-nvidia" }, { id: "fixture-2", key: "fixture-secondary" }], }, }, }; @@ -52,6 +53,8 @@ describe("provider management request pacing", () => { requestsPerMinute: 38, minIntervalMs: 1_600, maxConcurrentRequests: 4, + keySelection: "round-robin", + keys: { requestsPerMinute: 15, maxConcurrentRequests: 1 }, models: { "deepseek-ai/deepseek-v4-flash-0731": { requestsPerMinute: 10 } }, }; @@ -68,7 +71,11 @@ describe("provider management request pacing", () => { const providers = await request("/api/providers"); expect((await providers?.json()).find((row: { name: string }) => row.name === "nvidia").requestPacing).toEqual(policy); const status = await request("/api/provider-request-pacing?name=nvidia"); - expect(await status?.json()).toMatchObject({ provider: "nvidia", enabled: true, queued: 0, nextSlotInMs: 0 }); + const statusBody = await status?.json(); + expect(statusBody).toMatchObject({ provider: "nvidia", enabled: true, queued: 0, nextSlotInMs: 0 }); + expect(statusBody.keys.map((row: { id: string }) => row.id)).toEqual(["fixture-1", "fixture-2"]); + expect(JSON.stringify(statusBody)).not.toContain("sk-nvidia"); + expect(JSON.stringify(statusBody)).not.toContain("fixture-secondary"); const invalid = await request("/api/providers?name=nvidia", { method: "PATCH", @@ -78,6 +85,16 @@ describe("provider management request pacing", () => { expect(invalid?.status).toBe(400); expect(liveConfig.providers.nvidia?.requestPacing).toEqual(policy); + for (const rule of [{ keySelection: "random" }, { keys: {} }, { keys: { requestsPerMinute: 0 } }, + { keys: { maxConcurrentRequests: 1.5 } }]) { + const rejected = await request("/api/providers?name=nvidia", { + method: "PATCH", headers: { "content-type": "application/json" }, + body: JSON.stringify({ requestPacing: { enabled: true, ...rule } }), + }); + expect(rejected?.status).toBe(400); + expect(liveConfig.providers.nvidia?.requestPacing).toEqual(policy); + } + const timerOverflow = await request("/api/providers?name=nvidia", { method: "PATCH", headers: { "content-type": "application/json" }, diff --git a/tests/server/server-anthropic-billing.test.ts b/tests/server/server-anthropic-billing.test.ts index ec1329278d3..cf0abb9f243 100644 --- a/tests/server/server-anthropic-billing.test.ts +++ b/tests/server/server-anthropic-billing.test.ts @@ -419,6 +419,28 @@ for (const lane of ["native", "translated"] as const) describe(`${lane} Anthropi expect(seen).toHaveLength(statuses.length + 1); }); + for (const statuses of [[401, 400], [400, 429], [429, 402]]) test(`distributed ${statuses.join("→")} preserves billing and key recovery`, async () => { + const { forgetApiKeyRotationCursor, ANTHROPIC_BILLING_COOLDOWN_MS } = await import("../../src/providers/key-failover"); + forgetApiKeyRotationCursor("anth"); + const config = fixture((index, stream) => index < statuses.length + ? refusal(statuses[index]!, [400, 402].includes(statuses[index]!) ? creditMessage : "fixture refusal", + statuses[index] === 401 ? "authentication_error" : statuses[index] === 429 ? "rate_limit_error" : "invalid_request_error") + : success(stream), 3); + config.providers.anth!.requestPacing = { enabled: true, keySelection: "round-robin", keys: { maxConcurrentRequests: 1 } }; + saveConfig(config); + const startedAt = Date.now(); + try { + const response = await send(lane, config); + expect(response.status).toBe(200); + expect(await response.text()).toContain("billing recovered"); + expect(seen.map(entry => entry.key)).toEqual(["fixture-key-0", "fixture-key-1", "fixture-key-2"]); + expect(loadConfig().providers.anth!.apiKey).toBe("fixture-key-0"); + const billingKey = `k${statuses.findIndex(status => status === 400 || status === 402)}`; + expect(getKeyCooldownUntil("anth", billingKey)).toBeGreaterThanOrEqual(startedAt + ANTHROPIC_BILLING_COOLDOWN_MS); + expect(providerRequestPacingStatus("anth", config.providers.anth!).inFlight).toBe(0); + } finally { forgetApiKeyRotationCursor("anth"); } + }); + if (lane === "translated") for (const status of [401, 429]) test(`mixed billing→${status} keeps B's terminal body at an exact two-send budget`, async () => { const terminalMessage = `fixture terminal ${status} from B`; const config = fixture((index, stream) => index === 0 ? refusal() diff --git a/tests/server/server-chat-native-key-selection.test.ts b/tests/server/server-chat-native-key-selection.test.ts new file mode 100644 index 00000000000..4f36ea91044 --- /dev/null +++ b/tests/server/server-chat-native-key-selection.test.ts @@ -0,0 +1,263 @@ +import { afterEach, beforeEach, expect, test } from "bun:test"; +import { mkdtempSync, readFileSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { saveConfig } from "../../src/config"; +import { apiKeyAccountLogLabel } from "../../src/codex/account-label"; +import { captureProviderApiKeySelection } from "../../src/providers/api-key-selection"; +import { clearKeyCooldowns, forgetApiKeyRotationCursor, getKeyCooldownUntil, recordDistributedKeyFailure } from "../../src/providers/key-failover"; +import { providerRequestPacingStatus, resetProviderRequestPacingForTest, waitForProviderRequestSlot } from "../../src/providers/request-pacing"; +import { handleChatCompletions } from "../../src/server/chat-completions"; +import type { RequestLogContext } from "../../src/server/request-log"; +import type { OcxConfig } from "../../src/types"; +import { acquireOwnedSpendHome } from "../helpers/owned-spend-home"; +import { removeCompactionFixture } from "../helpers/compaction-routing-fixtures"; + +const NAME = "native-chat-pacing"; +const KEYS = ["fixture-key-alpha", "fixture-key-beta", "fixture-key-gamma"]; +const realFetch = globalThis.fetch; +let home: string; +let previousHome: string | undefined; +let releaseOwner: () => void; +let seen: { key: string | null; url: string; body: unknown }[]; +beforeEach(() => { + previousHome = process.env.OPENCODEX_HOME; + home = mkdtempSync(join(tmpdir(), "ocx-chat-key-pacing-")); + process.env.OPENCODEX_HOME = home; + releaseOwner = acquireOwnedSpendHome(); + seen = []; + clearKeyCooldowns(); forgetApiKeyRotationCursor(); resetProviderRequestPacingForTest(); + globalThis.fetch = (async (input, init) => { + seen.push({ key: new Headers(init?.headers).get("authorization"), url: String(input), body: JSON.parse(String(init?.body)) }); + return Response.json({ id: "chatcmpl-key-selection", object: "chat.completion", choices: [{ index: 0, message: { role: "assistant", content: "fixture reply" }, finish_reason: "stop" }] }); + }) as typeof fetch; +}); +afterEach(async () => { + globalThis.fetch = realFetch; + releaseOwner(); + clearKeyCooldowns(); forgetApiKeyRotationCursor(); resetProviderRequestPacingForTest(); + await removeCompactionFixture(home); + if (previousHome === undefined) delete process.env.OPENCODEX_HOME; + else process.env.OPENCODEX_HOME = previousHome; +}); +function fixture(keyCap = true): OcxConfig { + const config = { port: 0, defaultProvider: NAME, providers: { [NAME]: { + adapter: "openai-chat", baseUrl: "https://gateway.example/v1", authMode: "key", + apiKey: KEYS[0], apiKeySelectionRevision: "saved-revision", + apiKeyPool: KEYS.map((key, index) => ({ id: `k${index + 1}`, key, addedAt: index + 1 })), + requestPacing: { enabled: true, keySelection: "round-robin", + ...(keyCap ? { keys: { maxConcurrentRequests: 1 } } : { maxConcurrentRequests: 1 }) }, + } } } as OcxConfig; + saveConfig(config); + return config; +} +function status(config: OcxConfig) { return providerRequestPacingStatus(NAME, config.providers[NAME]!); } +function keyStatus(config: OcxConfig, id: string) { return status(config).keys!.find(key => key.id === id)!; } +function cool(config: OcxConfig, index: number) { + const selection = { ...captureProviderApiKeySelection({ ...config.providers[NAME]!, apiKey: KEYS[index] }), distributed: true }; + expect(recordDistributedKeyFailure(config, NAME, selection, 429, Date.now(), "30")).toBe(true); +} +async function queue(config: OcxConfig, id: string) { + const deadline = Date.now() + 2_000; + while (keyStatus(config, id).queued !== 1) { + if (Date.now() >= deadline) throw new Error(`key ${id} never queued: ${JSON.stringify(status(config))}`); + await Bun.sleep(5); + } +} +function start(config: OcxConfig, abort: AbortController) { + const logCtx: RequestLogContext = { model: "", provider: "" }; + const req = new Request("http://localhost/v1/chat/completions", { method: "POST", headers: { "content-type": "application/json" }, + signal: AbortSignal.any([abort.signal, AbortSignal.timeout(5_000)]), body: JSON.stringify({ model: `${NAME}/test`, messages: [{ role: "user", content: "fixture question" }], stream: false }) }); + const pending = handleChatCompletions(req, config, logCtx).then(async response => ({ response, text: await response.text(), logCtx })); + void pending.catch(() => {}); + return pending; +} +function unchanged(config: OcxConfig) { + expect(config.providers[NAME]!.apiKey).toBe(KEYS[0]); + expect(config.providers[NAME]!.apiKeySelectionRevision).toBe("saved-revision"); + const saved = JSON.parse(readFileSync(join(home, "config.json"), "utf8")) as OcxConfig; + expect(saved.providers[NAME]!.apiKey).toBe(KEYS[0]); + expect(saved.providers[NAME]!.apiKeySelectionRevision).toBe("saved-revision"); + expect(saved.providers[NAME]!._apiKeyAttempt).toBeUndefined(); +} +function noLeaks(config: OcxConfig) { + expect(status(config)).toMatchObject({ queued: 0, inFlight: 0 }); + for (const key of status(config).keys!) expect(key).toMatchObject({ queued: 0, inFlight: 0 }); +} +function success(result: Awaited>, index: number) { + expect(result.response.status).toBe(200); + expect(result.text).toContain("fixture reply"); + expect(seen.map(entry => entry.key)).toEqual([`Bearer ${KEYS[index]}`]); + expect(seen[0]!.url).toBe("https://gateway.example/v1/chat/completions"); + expect(result.logCtx.accountLogLabel).toBe(apiKeyAccountLogLabel(NAME, { entryId: `k${index + 1}`, reference: KEYS[index] })); + expect(result.logCtx.attempts).toHaveLength(1); + expect(result.logCtx.attempts![0]!.sendCount).toBe(1); + expect(result.logCtx.attempts![0]!.accountLogLabel).toBe(result.logCtx.accountLogLabel); +} +for (const cancel of [false, true]) test(`chat cooled queued A readmits B (abort=${cancel})`, async () => { + const config = fixture(); + const first = await waitForProviderRequestSlot(NAME, config.providers[NAME]!); + const second = await waitForProviderRequestSlot(NAME, { ...config.providers[NAME]!, apiKey: KEYS[1] }); + const abort = new AbortController(); + const pending = start(config, abort); + try { + await queue(config, "k1"); cool(config, 0); first.release(); + await queue(config, "k2"); + expect(keyStatus(config, "k1")).toMatchObject({ inFlight: 0, queued: 0, starts: 2 }); + expect(keyStatus(config, "k2")).toMatchObject({ inFlight: 1, queued: 1, starts: 1 }); + expect(seen).toEqual([]); + if (cancel) { + abort.abort(); + const result = await pending; + expect(result.response.status).toBe(499); + expect(seen).toEqual([]); + expect(status(config).queued).toBe(0); + second.release(); + } else { second.release(); success(await pending, 1); } + unchanged(config); noLeaks(config); + } finally { abort.abort(); first.release(); second.release(); await pending.catch(() => undefined); } +}, 10_000); +for (const exhausted of [false, true]) test(`chat revalidates B cooling during readmission (exhausted=${exhausted})`, async () => { + const config = fixture(); + const first = await waitForProviderRequestSlot(NAME, config.providers[NAME]!); + const second = await waitForProviderRequestSlot(NAME, { ...config.providers[NAME]!, apiKey: KEYS[1] }); + const abort = new AbortController(); + const pending = start(config, abort); + try { + await queue(config, "k1"); cool(config, 0); first.release(); + await queue(config, "k2"); cool(config, 1); + if (exhausted) cool(config, 2); + second.release(); + const result = await pending; + if (exhausted) { + expect(result.response.status).toBe(429); + expect(Number(result.response.headers.get("retry-after"))).toBeGreaterThan(0); + expect(seen).toEqual([]); + } else success(result, 2); + unchanged(config); noLeaks(config); + } finally { abort.abort(); first.release(); second.release(); await pending.catch(() => undefined); } +}, 10_000); +test("chat reselects behind the provider cap without a keys rule", async () => { + const config = fixture(false); + const held = await waitForProviderRequestSlot(NAME, config.providers[NAME]!); + const abort = new AbortController(); + const pending = start(config, abort); + try { + await queue(config, "k1"); cool(config, 0); held.release(); + success(await pending, 1); + expect(keyStatus(config, "k1").starts).toBe(2); + expect(keyStatus(config, "k2").starts).toBe(1); + unchanged(config); noLeaks(config); + } finally { abort.abort(); held.release(); await pending.catch(() => undefined); } +}); +test("chat refreshes a queued stale revision and sends the rebuilt destination", async () => { + const config = fixture(); + const held = await waitForProviderRequestSlot(NAME, config.providers[NAME]!); + const abort = new AbortController(); + const pending = start(config, abort); + try { + await queue(config, "k1"); + config.providers[NAME]!.apiKeySelectionRevision = "updated-revision"; + config.providers[NAME]!.baseUrl = "https://replacement.example/v1"; + held.release(); + const result = await pending; + expect(result.response.status).toBe(200); + expect(seen).toHaveLength(1); + expect(seen[0]!.url).toBe("https://replacement.example/v1/chat/completions"); + expect(seen[0]!.key).toBe(`Bearer ${KEYS[1]}`); + expect(result.logCtx.accountLogLabel).toBe(apiKeyAccountLogLabel(NAME, { entryId: "k2", reference: KEYS[1] })); + noLeaks(config); + } finally { abort.abort(); held.release(); await pending.catch(() => undefined); } +}); + +test("chat refuses after three queued rebuilds without spending a physical send", async () => { + const config = fixture(); + const extra = ["fixture-key-delta", "fixture-key-epsilon"]; + config.providers[NAME]!.apiKeyPool!.push(...extra.map((key, index) => ({ id: `k${index + 4}`, key }))); + const keys = [...KEYS, ...extra]; + const held = await Promise.all(keys.map(apiKey => waitForProviderRequestSlot(NAME, { ...config.providers[NAME]!, apiKey }))); + const abort = new AbortController(); + const pending = start(config, abort); + try { + for (let index = 0; index < 4; index++) { + await queue(config, `k${index + 1}`); + const selection = { ...captureProviderApiKeySelection({ ...config.providers[NAME]!, apiKey: keys[index] }), distributed: true }; + expect(recordDistributedKeyFailure(config, NAME, selection, 429, Date.now(), "30")).toBe(true); + held[index]!.release(); + } + const result = await pending; + expect(result.response.status).toBe(502); + expect(seen).toEqual([]); + expect(keyStatus(config, "k5").starts).toBe(1); + for (const slot of held) slot.release(); + unchanged(config); noLeaks(config); + } finally { abort.abort(); for (const slot of held) slot.release(); await pending.catch(() => undefined); } +}, 10_000); + +for (const statusCode of [401, 429]) test(`chat wire override ${statusCode} does not reject unused distributed pool keys`, async () => { + const config = fixture(); + config.providers[NAME]!.headers = { Authorization: "Bearer fixture-key-override" }; + globalThis.fetch = (async (input, init) => { + seen.push({ key: new Headers(init?.headers).get("authorization"), url: String(input), body: JSON.parse(String(init?.body)) }); + return Response.json({ error: { type: statusCode === 401 ? "authentication_error" : "rate_limit_error", message: "fixture overridden credential refusal" } }, { status: statusCode }); + }) as typeof fetch; + const result = await start(config, new AbortController()); + expect(result.response.status).toBe(statusCode); + expect(result.text).toContain("fixture overridden credential refusal"); + expect(seen.map(entry => entry.key)).toEqual(["Bearer fixture-key-override"]); + for (let index = 0; index < KEYS.length; index++) expect(getKeyCooldownUntil(NAME, `k${index + 1}`)).toBeNull(); + unchanged(config); noLeaks(config); +}); + +test("chat owned 401 rotates the rejected distributed credential before output", async () => { + const config = fixture(); + const reply = globalThis.fetch; + globalThis.fetch = (async (input, init) => { + if (seen.length === 0) { + seen.push({ key: new Headers(init?.headers).get("authorization"), url: String(input), body: JSON.parse(String(init?.body)) }); + return Response.json({ error: { type: "authentication_error", message: "fixture selected credential refusal" } }, { status: 401 }); + } + return reply(input, init); + }) as typeof fetch; + const result = await start(config, new AbortController()); + expect(result.response.status).toBe(200); + expect(result.text).toContain("fixture reply"); + expect(seen.map(entry => entry.key)).toEqual([`Bearer ${KEYS[0]}`, `Bearer ${KEYS[1]}`]); + expect(getKeyCooldownUntil(NAME, "k1")).not.toBeNull(); + expect(getKeyCooldownUntil(NAME, "k2")).toBeNull(); + unchanged(config); noLeaks(config); +}); + +for (const cancel of [false, true]) test(`active manual key switch readmits the selected key (cancel=${cancel})`, async () => { + const config = fixture(); + config.providers[NAME]!.requestPacing = { enabled: true, keys: { maxConcurrentRequests: 1 } }; + saveConfig(config); + const heldA = await waitForProviderRequestSlot(NAME, config.providers[NAME]!); + const heldB = await waitForProviderRequestSlot(NAME, { ...config.providers[NAME]!, apiKey: KEYS[1] }); + const abort = new AbortController(); + const pending = start(config, abort); + try { + await queue(config, "k1"); + config.providers[NAME]!.apiKey = KEYS[1]; + config.providers[NAME]!.apiKeySelectionRevision = "manual-key-switch"; + saveConfig(config); + heldA.release(); + await queue(config, "k2"); + expect(keyStatus(config, "k1")).toMatchObject({ queued: 0, inFlight: 0 }); + expect(seen).toEqual([]); + if (cancel) abort.abort(); + else heldB.release(); + const result = await pending; + if (cancel) { + expect(result.response.status).toBe(499); + expect(seen).toEqual([]); + heldB.release(); + } else { success(result, 1); } + const saved = JSON.parse(readFileSync(join(home, "config.json"), "utf8")) as OcxConfig; + expect(config.providers[NAME]!.apiKey).toBe(KEYS[1]); + expect(saved.providers[NAME]!.apiKey).toBe(KEYS[1]); + expect(saved.providers[NAME]!.apiKeySelectionRevision).toBe("manual-key-switch"); + expect(saved.providers[NAME]!._apiKeyAttempt).toBeUndefined(); + noLeaks(config); + } finally { abort.abort(); heldA.release(); heldB.release(); await pending.catch(() => undefined); } +}, 10_000); diff --git a/tests/server/server-key-failover-e2e.test.ts b/tests/server/server-key-failover-e2e.test.ts index 894638482ae..15af36b5706 100644 --- a/tests/server/server-key-failover-e2e.test.ts +++ b/tests/server/server-key-failover-e2e.test.ts @@ -27,7 +27,7 @@ import { installIsolatedCodexHome, type IsolatedCodexHome } from "../helpers/iso import { removeTreeWithRetry } from "../helpers/remove-tree"; import { managementFetch } from "../helpers/management-auth"; import { resolveContextPrincipal } from "../../src/server/auth-cors"; -import { resetProviderRequestPacingForTest, setProviderRequestPacingRuntimeForTest, waitForProviderRequestSlot } from "../../src/providers/request-pacing"; +import { providerRequestPacingStatus, resetProviderRequestPacingForTest, setProviderRequestPacingRuntimeForTest, waitForProviderRequestSlot } from "../../src/providers/request-pacing"; import { providerApiKeySelectionIsCurrent, resolveCurrentProviderApiKeyTransport, commitProviderApiKeySelection } from "../../src/providers/api-key-selection"; import { routedProviderConfig } from "../../src/router"; import type { OcxProviderTransport } from "../../src/providers/xai-transport"; @@ -130,6 +130,49 @@ describe("server 429 key failover (end-to-end)", () => { expect(providerApiKeySelectionIsCurrent(config, "current", current)).toBe(true); }); + test("Responses readmits a refreshed key and waits for its existing concurrency lease", async () => { + const seen: string[] = []; + upstream = Bun.serve({ hostname: "127.0.0.1", port: 0, fetch(req) { + seen.push(req.headers.get("authorization")!); + return Response.json({ id: "paced", object: "chat.completion", choices: [ + { index: 0, message: { role: "assistant", content: "readmitted" }, finish_reason: "stop" }, + ] }); + } }); + const config = { port: 0, defaultProvider: "paced", providers: { paced: { + adapter: "openai-chat", baseUrl: `http://127.0.0.1:${upstream.port}/v1`, allowPrivateNetwork: true, + authMode: "key", apiKey: "synthetic-first", + apiKeyPool: [{ id: "first", key: "synthetic-first" }, { id: "second", key: "synthetic-second" }], + requestPacing: { enabled: true, keys: { maxConcurrentRequests: 1 } }, + } } } as OcxConfig; + saveConfig(config); + const first = await waitForProviderRequestSlot("paced", config.providers.paced); + const second = await waitForProviderRequestSlot("paced", { ...config.providers.paced, apiKey: "synthetic-second" }); + const server = startTrackedServer(); + const abort = new AbortController(); + try { + const pending = fetch(new URL("/v1/responses", server.url), { method: "POST", signal: abort.signal, + headers: { "content-type": "application/json" }, body: JSON.stringify({ model: "paced/test", input: "hello", stream: false }) }); + for (let i = 0; i < 200 && providerRequestPacingStatus("paced", config.providers.paced).queued === 0; i++) await Bun.sleep(5); + expect(providerRequestPacingStatus("paced", config.providers.paced).queued).toBe(1); + const selected = await managementFetch(new URL("/api/providers/keys/active", server.url), { + method: "PUT", headers: { "content-type": "application/json" }, body: JSON.stringify({ name: "paced", id: "second" }), + }); + expect(selected.status).toBe(200); + await selected.text(); + first.release(); + await Bun.sleep(30); + expect(seen).toEqual([]); + second.release(); + const response = await pending; + expect(response.status).toBe(200); + expect(await response.text()).toContain("readmitted"); + expect(seen).toEqual(["Bearer synthetic-second"]); + } finally { + abort.abort(); first.release(); second.release(); await server.stop(true); + resetProviderRequestPacingForTest(); + } + }); + test.each(["responses", "chat/completions"])("%s logs only the key selected after pacing", async surface => { resetUsageReadCacheForTests(); let now = 0; diff --git a/tests/usage/request-pacing-provenance.test.ts b/tests/usage/request-pacing-provenance.test.ts new file mode 100644 index 00000000000..bc8409f075c --- /dev/null +++ b/tests/usage/request-pacing-provenance.test.ts @@ -0,0 +1,194 @@ +import { afterEach, describe, expect, test } from "bun:test"; +import { + providerRequestPacingStatus, registerProviderResponseProvenance, + resetProviderRequestPacingForTest, sendTrackingRequestSlot, + trackProviderRequestSlotBody, waitForProviderRequestSlot, +} from "../../src/providers/request-pacing"; +import { + isNativeControlResponse, markNativeControlResponse, +} from "../../src/server/responses/native-response-control"; +import { + codexWsSocketDeathStage, isCodexWsPreludeProjection, isCodexWsQuotaObservedResponse, + isCodexWsUpstreamResponse, markCodexWsPreludeProjection, markCodexWsResponse, + markCodexWsSocketDeath, markCodexWsStage, readCodexWsStage, type CodexWsStageRecord, +} from "../../src/server/responses/codex-ws-wire"; +import type { OcxProviderConfig } from "../../src/types"; + +afterEach(() => resetProviderRequestPacingForTest()); + +const provider: OcxProviderConfig = { + adapter: "openai-responses", apiKey: "fixture-key", baseUrl: "https://example.test/v1", + apiKeyPool: [{ id: "fixture", key: "fixture-key" }], + requestPacing: { enabled: true, keySelection: "round-robin" }, +}; + +function stage(): CodexWsStageRecord { + return { + requestBytes: 24, sent: true, upstreamFrames: 2, controlFrames: 1, relayedEvents: 1, + firstFrameMs: 1, firstResponseMs: 2, elapsedMs: 3, pings: 0, pongs: 0, + closeCode: null, reused: false, ocxVersion: "0.0.0", bunVersion: "fixture", + }; +} + +function expectUnmarked(response: Response): void { + expect(isNativeControlResponse(response)).toBe(false); + expect(isCodexWsUpstreamResponse(response)).toBe(false); + expect(isCodexWsQuotaObservedResponse(response)).toBe(false); + expect(isCodexWsPreludeProjection(response)).toBe(false); + expect(readCodexWsStage(response)).toBeUndefined(); + expect(codexWsSocketDeathStage(response)).toBeUndefined(); +} + +function countedSlot() { + let releases = 0; + return { + slot: { leased: true, bodyTracked: false, released: false, release() { releases++; } }, + releases: () => releases, + }; +} + +describe("pacing body transport provenance", () => { + test("round-robin lease preserves native and WS markers and live stage identity", async () => { + const slot = await waitForProviderRequestSlot("provenance", provider); + expect(slot.leased).toBe(true); // Key accounting alone owns a lease, with no concurrency cap. + expect(providerRequestPacingStatus("provenance", provider).keys?.[0]?.inFlight).toBe(1); + const original = markNativeControlResponse(new Response("first response\nsecond response\n")); + markCodexWsResponse(original, true); + const record = stage(); + markCodexWsStage(original, record); + const wrapped = trackProviderRequestSlotBody(slot, original); + expect(wrapped).not.toBe(original); + expect(slot.bodyTracked).toBe(true); + expect(isNativeControlResponse(wrapped)).toBe(true); + expect(isCodexWsUpstreamResponse(wrapped)).toBe(true); + expect(isCodexWsQuotaObservedResponse(wrapped)).toBe(true); + expect(isCodexWsPreludeProjection(wrapped)).toBe(false); + expect(readCodexWsStage(wrapped)).toBe(record); + markCodexWsStage(original, { ...record, upstreamFrames: 5, relayedEvents: 4 }); + expect(readCodexWsStage(wrapped)?.relayedEvents).toBe(4); + expect(slot.released).toBe(false); + expect(await wrapped.text()).toBe("first response\nsecond response\n"); + expect(slot.released).toBe(true); + expect(providerRequestPacingStatus("provenance", provider).keys?.[0]?.inFlight).toBe(0); + }); + + test.each([0, 1])("preserves prelude, stage and socket-death metadata (%i control frames)", async controlFrames => { + const slot = await waitForProviderRequestSlot("prelude", provider); + const original = new Response("refusal", { status: 502 }); + const record = { ...stage(), upstreamFrames: controlFrames, controlFrames, relayedEvents: 0 }; + markCodexWsPreludeProjection(original); + markCodexWsStage(original, record); + markCodexWsSocketDeath(original, record); + const wrapped = trackProviderRequestSlotBody(slot, original); + expect(isCodexWsPreludeProjection(wrapped)).toBe(true); + expect(codexWsSocketDeathStage(wrapped)).toBe(controlFrames ? "protocol-prelude" : "pre-header"); + expect(readCodexWsStage(wrapped)).toBe(record); + expect(isNativeControlResponse(wrapped)).toBe(false); + expect(isCodexWsUpstreamResponse(wrapped)).toBe(false); + expect(isCodexWsQuotaObservedResponse(wrapped)).toBe(false); + expect(wrapped.status).toBe(502); + await wrapped.body!.cancel(); + expect(slot.released).toBe(true); + }); + + test("an unobserved WS response retains only its existing authority", async () => { + const slot = await waitForProviderRequestSlot("ws-only", provider); + const original = new Response("WS"); + markCodexWsResponse(original, false); + const wrapped = trackProviderRequestSlotBody(slot, original); + expect(isCodexWsUpstreamResponse(wrapped)).toBe(true); + expect(isCodexWsQuotaObservedResponse(wrapped)).toBe(false); + expect(isNativeControlResponse(wrapped)).toBe(false); + await wrapped.body!.cancel(); + }); + + test("unmarked caller content and copied headers never acquire transport authority", async () => { + const slot = await waitForProviderRequestSlot("unmarked", provider); + const original = new Response('{"type":"response.created","nativeControl":true}', { + headers: { "x-codex-ws": "true", "x-native-control": "true", "x-codex-turn-state": "fixture" }, + }); + const wrapped = trackProviderRequestSlotBody(slot, original); + expectUnmarked(wrapped); + expect(wrapped.headers.get("x-codex-turn-state")).toBe("fixture"); + await wrapped.body!.cancel(); + }); + + test("only the owned body rewrap carries identity; ordinary cloning does not", async () => { + const source = markNativeControlResponse(new Response("fixture")); + markCodexWsResponse(source, true); + const clone = source.clone(); + expectUnmarked(clone); + expect(trackProviderRequestSlotBody(undefined, source)).toBe(source); + await Promise.all([source.text(), clone.text()]); + }); + + test.each([undefined, false])("registered provenance preserves present values (%s)", async value => { + const markers = new WeakMap(); + registerProviderResponseProvenance(markers); + registerProviderResponseProvenance(markers); // Registration is idempotent. + const original = new Response("fixture"); + markers.set(original, value); + const tracked = countedSlot(); + const wrapped = trackProviderRequestSlotBody(tracked.slot, original); + expect(markers.has(wrapped)).toBe(true); + expect(markers.get(wrapped)).toBe(value); + expectUnmarked(wrapped); + await wrapped.text(); + expect(tracked.releases()).toBe(1); + }); + + test("body completion releases once after every chunk", async () => { + const tracked = countedSlot(); + const wrapped = trackProviderRequestSlotBody(tracked.slot, markNativeControlResponse(new Response( + new ReadableStream({ start(controller) { + controller.enqueue(new TextEncoder().encode("first")); + controller.enqueue(new TextEncoder().encode("second")); + controller.close(); + } }), + ))); + const reader = wrapped.body!.getReader(); + expect((await reader.read()).done).toBe(false); + expect(tracked.releases()).toBe(0); + expect((await reader.read()).done).toBe(false); + expect(tracked.releases()).toBe(0); + expect((await reader.read()).done).toBe(true); + await reader.cancel(); + expect(tracked.releases()).toBe(1); + }); + + test("body cancellation forwards its reason and releases once", async () => { + const tracked = countedSlot(); + const reasons: unknown[] = []; + const original = new Response(new ReadableStream({ cancel(reason) { reasons.push(reason); } })); + markCodexWsResponse(original, true); + const wrapped = trackProviderRequestSlotBody(tracked.slot, original); + await wrapped.body!.cancel("consumer-abort"); + await wrapped.body!.cancel("second-cancel"); + expect(reasons).toEqual(["consumer-abort"]); + expect(tracked.releases()).toBe(1); + }); + + test("body error preserves the failure and releases once", async () => { + const tracked = countedSlot(); + const failure = new Error("fixture stream failure"); + const original = new Response(new ReadableStream({ pull(controller) { controller.error(failure); } })); + markCodexWsResponse(original, false); + const wrapped = trackProviderRequestSlotBody(tracked.slot, original); + await expect(wrapped.text()).rejects.toBe(failure); + expect(tracked.releases()).toBe(1); + }); + + test("abort and send error both return round-robin key leases", async () => { + const abort = new AbortController(); + const failure = new DOMException("fixture abort", "AbortError"); + const slot = await waitForProviderRequestSlot("abort", provider, undefined, abort.signal); + abort.abort(failure); + await expect(sendTrackingRequestSlot(slot, async () => { throw failure; })).rejects.toBe(failure); + expect(slot.released).toBe(true); + expect(providerRequestPacingStatus("abort", provider).keys?.[0]?.inFlight).toBe(0); + const next = await waitForProviderRequestSlot("abort", provider); + await expect(sendTrackingRequestSlot(next, async () => { throw new Error("send failed"); })).rejects.toThrow("send failed"); + expect(next.released).toBe(true); + expect(providerRequestPacingStatus("abort", provider).keys?.[0]?.inFlight).toBe(0); + }); +}); diff --git a/tests/usage/request-pacing.test.ts b/tests/usage/request-pacing.test.ts index 5183100b743..38a7acf1bf4 100644 --- a/tests/usage/request-pacing.test.ts +++ b/tests/usage/request-pacing.test.ts @@ -17,6 +17,53 @@ import type { OcxProviderConfig } from "../../src/types"; afterEach(() => resetProviderRequestPacingForTest()); +test("dispatch readmission observes the refreshed key interval and releases its replacement lease", async () => { + const clock = fakePacingClock(); + setProviderRequestPacingRuntimeForTest(clock.runtime); + const row = { ...provider({ enabled: true, keys: { minIntervalMs: 100, maxConcurrentRequests: 1 } }), apiKey: "first" }; + const refreshed = { ...row, apiKey: "second" }; + (await waitForProviderRequestSlot("readmission", refreshed)).release(); + const sends: number[] = []; + const physical = Object.assign(async () => { sends.push(clock.now()); return new Response("fixture"); }, { preconnect() {} }) as typeof fetch; + const executor = providerFetch({ ...row, fetch: physical }, undefined, { providerName: "readmission", + dispatchOverride: async (input, init, execute, reacquire) => { + await reacquire!(refreshed); + return execute(input, init); + }, + }); + const pending = fetchWithHeaderTimeout("https://example.test/v1/chat/completions", {}, new AbortController().signal, 5000, false, executor); + for (let i = 0; i < 10; i++) await Promise.resolve(); + expect(sends).toEqual([]); + clock.advanceBy(100); + const response = await pending; + expect(sends).toEqual([100]); + await response.text(); + const slot = await waitForProviderRequestSlot("readmission", row); + slot.release(); +}); + +test("abort during refreshed-key admission releases the old lease without sending", async () => { + const row = { ...provider({ enabled: true, keys: { maxConcurrentRequests: 1 } }), apiKey: "first" }; + const refreshed = { ...row, apiKey: "second" }; + const held = await waitForProviderRequestSlot("readmission-abort", refreshed); + const controller = new AbortController(); + let sends = 0; + const executor = providerFetch(row, undefined, { providerName: "readmission-abort", + dispatchOverride: async (_input, _init, _execute, reacquire) => { + await reacquire!(refreshed); + sends++; + return new Response("fixture"); + }, + }); + const pending = executor("https://example.test/v1/chat/completions", { signal: controller.signal }); + for (let i = 0; i < 10; i++) await Promise.resolve(); + controller.abort(); + await expect(pending).rejects.toThrow(); + expect(sends).toBe(0); + const slot = await waitForProviderRequestSlot("readmission-abort", row); + slot.release(); held.release(); +}); + function provider(requestPacing: OcxProviderConfig["requestPacing"]): OcxProviderConfig { return { adapter: "openai-chat", baseUrl: "https://example.test/v1", requestPacing }; } @@ -61,6 +108,70 @@ function fakePacingClock(): { } describe("requestPacingIntervalMs", () => { + test("per-key caps admit siblings while provider capacity remains an upper bound", async () => { + const clock = fakePacingClock(); + setProviderRequestPacingRuntimeForTest(clock.runtime); + const configured = provider({ enabled: true, maxConcurrentRequests: 2, keys: { maxConcurrentRequests: 1, requestsPerMinute: 15 } }); + configured.apiKeyPool = [{ id: "one", key: "fake-one" }, { id: "two", key: "fake-two" }]; + const one = { ...configured, apiKey: "fake-one" }; + const two = { ...configured, apiKey: "fake-two" }; + const first = await waitForProviderRequestSlot("key-cap", one); + let repeatedStarted = false; + const repeated = waitForProviderRequestSlot("key-cap", one).then(slot => { repeatedStarted = true; return slot; }); + const sibling = await waitForProviderRequestSlot("key-cap", two); + expect(repeatedStarted).toBe(false); + expect(providerRequestPacingStatus("key-cap", configured).keys?.map(row => row.inFlight)).toEqual([1, 1]); + first.release(); + clock.advanceBy(3999); + expect(repeatedStarted).toBe(false); + clock.advanceBy(1); + const next = await repeated; + expect(repeatedStarted).toBe(true); + expect(providerRequestPacingStatus("key-cap", configured).keys?.map(row => row.starts)).toEqual([2, 1]); + next.release(); sibling.release(); + expect(providerRequestPacingStatus("key-cap", configured).keys?.map(row => row.inFlight)).toEqual([0, 0]); + expect(JSON.stringify(providerRequestPacingStatus("key-cap", configured))).not.toContain("fake-"); + }); + + test("key-only pacing obeys body cancellation and turn follow-up intervals without a second lease", async () => { + const clock = fakePacingClock(); + setProviderRequestPacingRuntimeForTest(clock.runtime); + const configured = { ...provider({ enabled: true, keys: { maxConcurrentRequests: 1, minIntervalMs: 100 } }), apiKey: "fake", apiKeyPool: [{ id: "id", key: "fake" }] }; + const first = await waitForProviderRequestSlot("key-only", configured); + const followUp = waitForProviderRequestSlot("key-only", configured, undefined, undefined, { concurrency: false }); + clock.advanceBy(100); + const follow = await followUp; + expect(follow.leased).toBe(false); + expect(providerRequestPacingStatus("key-only", configured).keys?.[0]?.inFlight).toBe(1); + const controller = new AbortController(); + const queued = waitForProviderRequestSlot("key-only", configured, undefined, controller.signal); + controller.abort(); + await expect(queued).rejects.toThrow(); + first.release(); + }); + + test("provider cap also blocks an otherwise eligible different key", async () => { + const clock = fakePacingClock(); setProviderRequestPacingRuntimeForTest(clock.runtime); + const configured = provider({ enabled: true, maxConcurrentRequests: 1, keys: { maxConcurrentRequests: 1 } }); + const first = await waitForProviderRequestSlot("global-key-cap", { ...configured, apiKey: "fake-one" }); + let started = false; + const second = waitForProviderRequestSlot("global-key-cap", { ...configured, apiKey: "fake-two" }).then(slot => { started = true; return slot; }); + await Promise.resolve(); expect(started).toBe(false); + first.release(); const slot = await second; expect(started).toBe(true); slot.release(); + }); + + test("duplicate key references share capacity and diagnostics count key-only leases", async () => { + const configured = { ...provider({ enabled: true, keys: { maxConcurrentRequests: 1 } }), apiKey: "fake-one", + apiKeyPool: [{ id: "one", key: "fake-one" }, { id: "duplicate", key: "fake-one" }] }; + const first = await waitForProviderRequestSlot("duplicate-key", configured); + expect(providerRequestPacingStatus("duplicate-key", configured).inFlight).toBe(1); + const abort = new AbortController(); + const second = waitForProviderRequestSlot("duplicate-key", configured, undefined, abort.signal); + expect(providerRequestPacingStatus("duplicate-key", configured).keys?.map(row => row.queued)).toEqual([1, 1]); + abort.abort(); await expect(second).rejects.toThrow(); first.release(); + expect(providerRequestPacingStatus("duplicate-key", configured).inFlight).toBe(0); + }); + test("uses the slower of provider RPM, provider delay, and model override", () => { const configured = provider({ enabled: true,