From 636a7d2e3342d9f07141c8116bf29d3caae5a7d8 Mon Sep 17 00:00:00 2001 From: GSL-R <174104911+GSL-R@users.noreply.github.com> Date: Sun, 11 Oct 2026 21:58:38 +0900 Subject: [PATCH 01/17] feat(providers): add opt-in API key round-robin and per-key pacing --- .../fr/reference/configuration/providers.md | 4 +- .../ja/reference/configuration/providers.md | 4 +- .../ko/reference/configuration/providers.md | 4 +- .../docs/reference/configuration/providers.md | 21 ++++- .../ru/reference/configuration/providers.md | 4 +- .../reference/configuration/providers.md | 4 +- .../reference/configuration/providers.md | 4 +- .../provider-workspace/ProviderKeyPacing.tsx | 37 ++++++++ .../provider-workspace/ProviderSettings.tsx | 21 ++++- gui/src/i18n/de.ts | 10 ++ gui/src/i18n/en.ts | 10 ++ gui/src/i18n/fr.ts | 10 ++ gui/src/i18n/ja.ts | 10 ++ gui/src/i18n/ko.ts | 10 ++ gui/src/i18n/pt.ts | 10 ++ gui/src/i18n/ru.ts | 10 ++ gui/src/i18n/tr.ts | 10 ++ gui/src/i18n/vi.ts | 10 ++ gui/src/i18n/zh-TW.ts | 10 ++ gui/src/i18n/zh.ts | 10 ++ gui/src/provider-workspace/catalog.ts | 2 + .../provider-settings-request-pacing.test.tsx | 29 ++++++ src/config/schema/leaf-validators.ts | 5 +- src/providers/api-key-selection.ts | 21 ++++- src/providers/key-failover.ts | 36 ++++++++ src/providers/request-pacing.ts | 91 +++++++++++++++---- src/server/chat-native.ts | 18 +++- src/server/images.ts | 9 +- src/server/management/provider-routes.ts | 11 ++- src/server/messages-native.ts | 18 +++- src/server/responses/compact.ts | 9 +- src/types/provider.ts | 6 ++ structure/config.md | 2 +- structure/data-planes/images.md | 2 +- structure/gui-and-management-api.md | 2 +- structure/providers-and-adapters.md | 2 +- structure/transports/responses-failover.md | 2 +- tests/adapters/key-failover.test.ts | 60 ++++++++++++ .../messages-native.test.ts | 26 +++++- ...management-provider-request-pacing.test.ts | 19 +++- tests/usage/request-pacing.test.ts | 64 +++++++++++++ 41 files changed, 599 insertions(+), 48 deletions(-) create mode 100644 gui/src/components/provider-workspace/ProviderKeyPacing.tsx diff --git a/docs-site/src/content/docs/fr/reference/configuration/providers.md b/docs-site/src/content/docs/fr/reference/configuration/providers.md index 92910bf7206..b3d115b5b9a 100644 --- a/docs-site/src/content/docs/fr/reference/configuration/providers.md +++ b/docs-site/src/content/docs/fr/reference/configuration/providers.md @@ -89,7 +89,9 @@ sauvegarde dont le contenu diffère, puis réécrit en identifiants sans préfix | --- | --- | --- | | `adapter` | `string` | L'un des `openai-chat`, `openai-responses`, `anthropic`, `google`, `kiro`, `cursor`, `ollama-native`, `azure-openai` (ou alias `azure`). | | `baseUrl` | `string` | URL de base de l'API en amont. La plupart des points de terminaison fixes intégrés ignorent une valeur incompatible ; les préréglages de clés protégés contre les collisions préservent une ancienne destination personnalisée portant le même nom. | -| `requestPacing?` | `{ enabled, requestsPerMinute?, minIntervalMs?, maxConcurrentRequests?, models? }` | Cadencement facultatif du démarrage des requêtes sortantes côté client, distinct de l’utilisation, de la facturation et des indicateurs de limitation en amont. `maxConcurrentRequests` est une limite entière positive des requêtes en cours ; une règle fournisseur ou modèle peut utiliser uniquement cette limite. Les limites du fournisseur s’appliquent à tous ses modèles, tandis que les entrées `models` ciblent les identifiants exacts des modèles en amont et peuvent ajouter du délai ou réduire la concurrence. L’attente dans la file ne consomme pas le délai d’expiration des en-têtes de réponse en amont. Les requêtes HTTP et les distributions explicites `fetchResponse`/`runTurn` des adaptateurs sont couvertes. Avec une limite de concurrence, un tour Responses WebSocket canonique utilise HTTP/SSE afin de libérer la capacité à la fin, en cas d’erreur ou d’annulation du corps de réponse. Pour les adaptateurs `runTurn`, dont Cursor, la limite compte les tours actifs plutôt que les envois physiques : RunSSE et BidiAppend peuvent se chevaucher dans un même tour, tandis qu’un autre tour attend. Les envois suivants respectent toujours les intervalles de démarrage. | +| `requestPacing?` | `{ enabled, requestsPerMinute?, minIntervalMs?, maxConcurrentRequests?, models?, keySelection?, keys? }` | Cadencement facultatif du démarrage des requêtes sortantes côté client, distinct de l’utilisation, de la facturation et des indicateurs de limitation en amont. `maxConcurrentRequests` est une limite entière positive des requêtes en cours ; une règle fournisseur ou modèle peut utiliser uniquement cette limite. Les limites du fournisseur s’appliquent à tous ses modèles, tandis que les entrées `models` ciblent les identifiants exacts des modèles en amont et peuvent ajouter du délai ou réduire la concurrence. L’attente dans la file ne consomme pas le délai d’expiration des en-têtes de réponse en amont. Les requêtes HTTP et les distributions explicites `fetchResponse`/`runTurn` des adaptateurs sont couvertes. Avec une limite de concurrence, un tour Responses WebSocket canonique utilise HTTP/SSE afin de libérer la capacité à la fin, en cas d’erreur ou d’annulation du corps de réponse. Pour les adaptateurs `runTurn`, dont Cursor, la limite compte les tours actifs plutôt que les envois physiques : RunSSE et BidiAppend peuvent se chevaucher dans un même tour, tandis qu’un autre tour attend. Les envois suivants respectent toujours les intervalles de démarrage. | +| `requestPacing.keySelection?` | `"active"`, `"round-robin"` | Sélection des clés pour les requêtes normales : `active` par défaut conserve la clé enregistrée et la rotation sur erreur ; `round-robin` répartit les allocations entre les clés disponibles sans modifier la sélection enregistrée. Cela équilibre les allocations, pas les jetons ni les coûts. | +| `requestPacing.keys?` | `{ requestsPerMinute?, minIntervalMs?, maxConcurrentRequests? }` | Limites uniformes par référence de clé : RPM, intervalle minimal et concurrence, partagées entre modèles et cumulées avec les plafonds fournisseur et modèle. Les références identiques partagent leur capacité ; plusieurs références du même projet ne garantissent pas des quotas indépendants. TPM et quotas quotidiens ne sont pas gérés. | | `responsesPath?` | `string` | Chemin de ressource relatif pour les requêtes d'authentification par clé `openai-responses`. Il doit commencer par `/` et ne contenir aucun schéma, requête ou fragment. | | `chatCompletionsPath?` | `string` | Chemin de ressource relatif pour les requêtes `openai-chat`, miroir de `responsesPath` et soumis aux mêmes règles de forme. Nécessaire lorsqu'un même service en amont sert Chat Completions et Responses sous des préfixes différents : un override wire par modèle change l'adaptateur sans toucher `baseUrl`, donc sans ce réglage une requête Chat activée serait envoyée vers la base Responses. L'exemple fourni est Z.AI. | | `upstreamWebsocket?` | `boolean` | Active le transport Responses WebSocket en amont pour les requêtes `openai-responses` (désactivé par défaut). N'est honoré que pour l'amont first-party `https://api.openai.com/v1` ; les points de terminaison des fournisseurs personnalisés utilisent toujours HTTP/SSE borné, car Bun ne peut pas appliquer de limite de taille aux messages WebSocket entrants avant d'avoir alloué le message complet. Pour le fournisseur canonique ChatGPT `openai`, l'omettre conserve le WebSocket en amont sur les tours éligibles, `false` envoie les tours en streaming via HTTP/SSE, et `true` est refusé ; avec `false`, le pilotage et l'injection natifs en cours de tour sont indisponibles. Ce champ est indépendant du réglage `websockets` côté client et ne change ni le point de terminaison ni les identifiants. Une base HTTP reste en SSE ; les chemins qui ne sont pas Responses et les requêtes `openai-chat` restent en HTTP. | diff --git a/docs-site/src/content/docs/ja/reference/configuration/providers.md b/docs-site/src/content/docs/ja/reference/configuration/providers.md index 13184d191e5..399118518aa 100644 --- a/docs-site/src/content/docs/ja/reference/configuration/providers.md +++ b/docs-site/src/content/docs/ja/reference/configuration/providers.md @@ -83,7 +83,9 @@ account を削除しても mapping は保持され、同じ id を再追加す | --- | --- | --- | | `adapter` | `string` | `openai-chat`、`openai-responses`、`anthropic`、`google`、`kiro`、`cursor`、`ollama-native`、`azure-openai` (または別名 `azure`) のいずれか。 | | `baseUrl` | `string` |アップストリーム API のベース URL。ほとんどの組み込み固定エンドポイントは不一致を無視します。衝突安全キー プリセットは、古い同じ名前のカスタム宛先を保持します。 | -| `requestPacing?` | `{ enabled, requestsPerMinute?, minIntervalMs?, maxConcurrentRequests?, models? }` | 上流の使用量、請求、レート制限表示とは別の、クライアント側の送信開始間隔調整です。`maxConcurrentRequests` は実行中リクエスト数を制限する正の整数で、プロバイダー規則またはモデル規則だけに設定できます。プロバイダー制限は全モデルに適用され、`models` は上流の正確なモデル ID に一致し、遅延を増やすか同時実行数を減らします。キュー待機は応答ヘッダーのタイムアウトを消費しません。HTTP と明示的なアダプターの `fetchResponse`/`runTurn` 送信を対象にします。同時実行数の上限がある場合、標準の Responses WebSocket ターンは応答本文の完了・エラー・キャンセル時に枠を解放できるよう HTTP/SSE を使います。 Cursor を含む `runTurn` アダプターでは、上限は物理送信数ではなく進行中のターン数を数えます。同じターンの RunSSE と BidiAppend は重なり得ますが、別のターンは待機します。後続送信にも開始間隔は適用されます。 | +| `requestPacing?` | `{ enabled, requestsPerMinute?, minIntervalMs?, maxConcurrentRequests?, models?, keySelection?, keys? }` | 上流の使用量、請求、レート制限表示とは別の、クライアント側の送信開始間隔調整です。`maxConcurrentRequests` は実行中リクエスト数を制限する正の整数で、プロバイダー規則またはモデル規則だけに設定できます。プロバイダー制限は全モデルに適用され、`models` は上流の正確なモデル ID に一致し、遅延を増やすか同時実行数を減らします。キュー待機は応答ヘッダーのタイムアウトを消費しません。HTTP と明示的なアダプターの `fetchResponse`/`runTurn` 送信を対象にします。同時実行数の上限がある場合、標準の Responses WebSocket ターンは応答本文の完了・エラー・キャンセル時に枠を解放できるよう HTTP/SSE を使います。 Cursor を含む `runTurn` アダプターでは、上限は物理送信数ではなく進行中のターン数を数えます。同じターンの RunSSE と BidiAppend は重なり得ますが、別のターンは待機します。後続送信にも開始間隔は適用されます。 | +| `requestPacing.keySelection?` | `"active"`, `"round-robin"` | 通常リクエストのキー選択です。既定の `active` は保存済みキーと既存の障害時切替を維持します。`round-robin` は保存済み選択を変えず、利用可能なプールのキーを順に割り当てます。トークンや料金ではなく割当回数を分散します。 | +| `requestPacing.keys?` | `{ requestsPerMinute?, minIntervalMs?, maxConcurrentRequests? }` | キー参照ごとの共通 RPM・開始間隔・同時実行上限です。全モデルで共有し、プロバイダーとモデルの上限も同時に適用します。同一参照は枠を共有しますが、同じ上流プロジェクトの別参照は独立クォータを意味しません。TPM と日次上限は管理しません。 | | `responsesPath?` | `string` |キー認証 `openai-responses` リクエストの相対リソース パス。 `/` で始まり、スキーム、クエリ、またはフラグメントが含まれていない必要があります。 | | `chatCompletionsPath?` | `string` | `openai-chat` リクエストの相対リソース パス。 `responsesPath` の対となる設定で、同じ形式ルールが適用されます。1つのアップストリームが Chat Completions と Responses を異なるプレフィックスで提供する場合に必要です。モデルごとの wire override はアダプターのみを変更し `baseUrl` は変更しないため、この設定がないと有効化された Chat リクエストが Responses ベースへ送信されます。同梱例は Z.AI です。 | | `upstreamWebsocket?` | `boolean` | `openai-responses` リクエストで使用するアップストリーム Responses WebSocket トランスポート（既定値は無効）。ファーストパーティの `https://api.openai.com/v1` アップストリームでのみ有効です。カスタムプロバイダーのエンドポイントは常に制限付き HTTP/SSE を使用します。Bun はメッセージ全体を確保する前に受信 WebSocket メッセージのサイズ上限を適用できないためです。正規の ChatGPT `openai` プロバイダーでは、省略すると対象となるターンでアップストリーム WebSocket を使用し、`false` はストリーミングのターンを HTTP/SSE で送信し、`true` は拒否されます。`false` の間はネイティブのターン途中のステアリングとインジェクションを利用できません。このフィールドはクライアント側の `websockets` 設定とは独立しており、エンドポイントと認証情報のどちらも変更しません。HTTP のベース URL は SSE のままとなり、Responses 以外のパスと `openai-chat` リクエストは HTTP を使用します。 | diff --git a/docs-site/src/content/docs/ko/reference/configuration/providers.md b/docs-site/src/content/docs/ko/reference/configuration/providers.md index 40db0c8ca04..ff8c4aafb05 100644 --- a/docs-site/src/content/docs/ko/reference/configuration/providers.md +++ b/docs-site/src/content/docs/ko/reference/configuration/providers.md @@ -83,7 +83,9 @@ managed map을 활성화하면 privacy-safe selector를 만들고, 이후 계정 | --- | --- | --- | | `adapter` | `string` | `openai-chat`, `openai-responses`, `anthropic`, `google`, `kiro`, `cursor`, `ollama-native`, `azure-openai` 중 하나이며, `azure`는 별칭입니다. | | `baseUrl` | `string` | 상위 API 기본 URL입니다. 대부분의 내장 고정 엔드포인트는 불일치를 무시합니다. 충돌 안전 키 프리셋은 같은 이름의 이전 사용자 지정 목적지를 보존합니다. | -| `requestPacing?` | `{ enabled, requestsPerMinute?, minIntervalMs?, maxConcurrentRequests?, models? }` | 업스트림 사용량, 과금, rate-limit 지표와 별개인 선택적 클라이언트 측 아웃바운드 요청 시작 속도 조절입니다. `maxConcurrentRequests`는 진행 중 요청 수를 제한하는 양의 정수이며 provider 또는 model 규칙에서 단독으로 사용할 수 있습니다. Provider 제한은 모든 모델에 적용되고 `models` 항목은 정확한 업스트림 모델 ID와 일치하며 지연을 늘리거나 동시성을 줄일 수 있습니다. 큐 대기는 응답 헤더 타임아웃을 소모하지 않습니다. HTTP 및 명시적 어댑터 `fetchResponse`/`runTurn` 전송을 포함합니다. 동시성 제한이 있으면 표준 Responses WebSocket 턴은 응답 본문의 완료·오류·취소 때 점유 슬롯을 해제할 수 있도록 HTTP/SSE를 사용합니다. Cursor를 포함한 `runTurn` 어댑터에서는 물리적 전송 수가 아니라 진행 중인 턴 수를 제한합니다. 같은 턴의 RunSSE와 BidiAppend는 겹칠 수 있지만 다른 턴은 대기합니다. 후속 전송에도 시작 간격은 적용됩니다. | +| `requestPacing?` | `{ enabled, requestsPerMinute?, minIntervalMs?, maxConcurrentRequests?, models?, keySelection?, keys? }` | 업스트림 사용량, 과금, rate-limit 지표와 별개인 선택적 클라이언트 측 아웃바운드 요청 시작 속도 조절입니다. `maxConcurrentRequests`는 진행 중 요청 수를 제한하는 양의 정수이며 provider 또는 model 규칙에서 단독으로 사용할 수 있습니다. Provider 제한은 모든 모델에 적용되고 `models` 항목은 정확한 업스트림 모델 ID와 일치하며 지연을 늘리거나 동시성을 줄일 수 있습니다. 큐 대기는 응답 헤더 타임아웃을 소모하지 않습니다. HTTP 및 명시적 어댑터 `fetchResponse`/`runTurn` 전송을 포함합니다. 동시성 제한이 있으면 표준 Responses WebSocket 턴은 응답 본문의 완료·오류·취소 때 점유 슬롯을 해제할 수 있도록 HTTP/SSE를 사용합니다. Cursor를 포함한 `runTurn` 어댑터에서는 물리적 전송 수가 아니라 진행 중인 턴 수를 제한합니다. 같은 턴의 RunSSE와 BidiAppend는 겹칠 수 있지만 다른 턴은 대기합니다. 후속 전송에도 시작 간격은 적용됩니다. | +| `requestPacing.keySelection?` | `"active"`, `"round-robin"` | 정상 요청의 키 선택 방식입니다. 기본 `active`는 저장된 키와 기존 오류 전환을 유지하며, `round-robin`은 저장된 활성 키를 변경하지 않고 사용 가능한 풀 키를 요청별 순서대로 배정합니다. 토큰이나 비용이 아니라 배정 횟수를 분산합니다. | +| `requestPacing.keys?` | `{ requestsPerMinute?, minIntervalMs?, maxConcurrentRequests? }` | 키 참조별 동일한 RPM·최소 시작 간격·동시 요청 제한입니다. 모든 모델에서 공유하며 프로바이더·정확한 모델 상한과 함께 적용합니다. 중복 참조는 한 슬롯을 공유하지만 같은 업스트림 프로젝트의 다른 참조가 독립 할당량을 뜻하지는 않습니다. TPM·일일 한도는 처리하지 않습니다. | | `responsesPath?` | `string` | 키 인증 `openai-responses` 요청의 상대 리소스 경로입니다. 반드시 `/`로 시작해야 하며 스킴, query, fragment를 포함하면 안 됩니다. | | `chatCompletionsPath?` | `string` | `openai-chat` 요청의 상대 리소스 경로로, `responsesPath`와 동일한 형식 규칙이 적용되는 대응 항목입니다. 하나의 업스트림이 Chat Completions와 Responses를 서로 다른 접두사로 제공할 때 필요합니다. 모델별 wire override는 어댑터만 바꾸고 `baseUrl`은 그대로 두므로, 이 설정이 없으면 옵트인된 Chat 요청이 Responses base로 전송됩니다. Z.AI가 제공되는 예시입니다. | | `upstreamWebsocket?` | `boolean` | `openai-responses` 요청에 대한 업스트림 Responses WebSocket 전송을 선택적으로 활성화합니다(기본값 `false`). 퍼스트파티 `https://api.openai.com/v1` 업스트림에서만 적용되며, 사용자 지정 공급자 엔드포인트는 항상 제한된 HTTP/SSE를 사용합니다. Bun은 전체 메시지를 할당하기 전에는 수신 WebSocket 메시지 크기 제한을 적용할 수 없기 때문입니다. 정식 ChatGPT `openai` 공급자에서는 생략하면 대상 턴에서 업스트림 WebSocket을 사용하고, `false`는 스트리밍 턴을 HTTP/SSE로 전송하며, `true`는 거부됩니다. `false`이면 네이티브 턴 중 스티어링과 주입을 사용할 수 없습니다. 이 필드는 클라이언트 측 `websockets` 설정과 독립적이며 엔드포인트와 자격 증명을 변경하지 않습니다. HTTP 기본 URL은 SSE를 유지하고, Responses가 아닌 경로와 `openai-chat` 요청은 HTTP를 사용합니다. | diff --git a/docs-site/src/content/docs/reference/configuration/providers.md b/docs-site/src/content/docs/reference/configuration/providers.md index b07aac6f57a..1e9df7881ce 100644 --- a/docs-site/src/content/docs/reference/configuration/providers.md +++ b/docs-site/src/content/docs/reference/configuration/providers.md @@ -214,7 +214,9 @@ Providers can expose a built-in shorthand, such as `agy` for `google-antigravity | `baseUrl` | `string` | Upstream API base URL. Most built-in fixed endpoints ignore a mismatch; collision-safe key presets preserve an older same-named custom destination. | | `proxy?` | `string \| null` | Per-provider egress route. Omit it to inherit the global proxy decision; use `"direct"` or `null` to force direct egress; or provide an absolute `http://`, `https://`, `socks5://`, or `socks5h://` proxy URL. An empty string is rejected. | | `noProxy?` | `string \| string[]` | Destinations this provider reaches directly, using `NO_PROXY` host-pattern syntax. A match bypasses both this provider's own proxy and an inherited global proxy. | -| `requestPacing?` | `{ enabled, requestsPerMinute?, minIntervalMs?, maxConcurrentRequests?, models? }` | Optional client-side outbound request-start pacing, separate from upstream usage, billing, and rate-limit indicators. RPM is converted to an even interval; `minIntervalMs` may impose a longer interval. `maxConcurrentRequests` is a positive integer cap on in-flight requests. A provider or model rule may use the concurrency cap alone; provider limits apply across all models, while `models` entries use exact upstream model IDs (for example `nvidia/llama-3.1-nemotron-ultra-253b-v1`) and can only add delay or narrow concurrency. Queue waits do not consume the upstream response-header timeout. HTTP and explicit adapter `fetchResponse`/`runTurn` dispatches are covered. A concurrency-capped canonical Responses WebSocket turn uses HTTP/SSE so its lease can be released when the response body completes, errors, or is cancelled. For `runTurn` adapters, including Cursor, the cap counts active turns rather than physical sends: RunSSE and BidiAppend may overlap within one turn, while another turn waits. Follow-up sends still obey start intervals. | +| `requestPacing?` | `{ enabled, requestsPerMinute?, minIntervalMs?, maxConcurrentRequests?, models?, keySelection?, keys? }` | Optional client-side outbound request-start pacing, separate from upstream usage, billing, and rate-limit indicators. RPM is converted to an even interval; `minIntervalMs` may impose a longer interval. `maxConcurrentRequests` is a positive integer cap on in-flight requests. A provider or model rule may use the concurrency cap alone; provider limits apply across all models, while `models` entries use exact upstream model IDs (for example `nvidia/llama-3.1-nemotron-ultra-253b-v1`) and can only add delay or narrow concurrency. Queue waits do not consume the upstream response-header timeout. HTTP and explicit adapter `fetchResponse`/`runTurn` dispatches are covered. A concurrency-capped canonical Responses WebSocket turn uses HTTP/SSE so its lease can be released when the response body completes, errors, or is cancelled. For `runTurn` adapters, including Cursor, the cap counts active turns rather than physical sends: RunSSE and BidiAppend may overlap within one turn, while another turn waits. Follow-up sends still obey start intervals. | +| `requestPacing.keySelection?` | `"active"`, `"round-robin"` | Normal-request key selection. Default `active` preserves the saved key and existing failure rotation; `round-robin` distributes request-local selections across eligible API-key pool references without changing the saved active key. It balances allocations, not tokens or cost. | +| `requestPacing.keys?` | `{ requestsPerMinute?, minIntervalMs?, maxConcurrentRequests? }` | Uniform per-key-reference RPM, minimum start interval and concurrency limits, shared across models. These compose with provider and exact-model upper bounds. Duplicate references share a bucket; separate references in one upstream project do not imply independent quota. TPM and daily quotas are not enforced. | | `upstreamHttpVersion?` | `"auto" \| "http1.1" \| "h1" \| "http2" \| "h2"` | Pin the HTTP version used for upstream requests to this provider. Defaults to `auto`, which lets Bun negotiate. An explicit pin requires an HTTPS target and fails locally when it cannot be honored. Set `http1.1` when a provider's HTTP/2 SSE stream stalls instead of delivering events — the symptom is a long-running streaming request that produces nothing and eventually times out. For Cursor, `http1.1`/`h1` selects its `RunSSE` + `BidiAppend` compatibility transport for inference and also pins live model discovery. Management `POST`/`PATCH` accept `null` to clear it back to `auto`. | | `tlsProfile?` | `"antigravity-browser"` | **Use at your own risk.** Opt-in browser-like TLS handshake (through the optional `wreq-js` dependency) for the canonical `google-antigravity` OAuth provider. It changes only how the connection looks on the wire; it is not an official Google client, and it does not change what Google's terms allow. Google can still detect, rate-limit, suspend, or ban the account you signed in with, and you alone carry that risk. It is off by default and accepted only with the Google adapter, Cloud Code Assist mode, and Google's canonical HTTPS Antigravity hosts. Redirects stay manual. The provider's own `proxy`/`noProxy` route is carried by the TLS transport; a route it cannot keep fails the request instead of leaving by another path, which includes any direct route while `HTTP_PROXY`, `HTTPS_PROXY`, or `ALL_PROXY` is set. `GET /api/providers` reports the profile state as `pending`, `active`, or `failed`. Omitting the field keeps the normal Bun transport and never loads the dependency. | | `responsesPath?` | `string` | Relative resource path for key-auth `openai-responses` requests. It must start with `/` and contain no scheme, query, or fragment. | @@ -344,6 +346,23 @@ For example, this applies a provider-wide concurrency cap and a stricter cap to The concurrency slot stays occupied until the upstream request finishes, including the final streamed response bytes. Requests above the applicable provider and model limits wait in the pacing queue; when a slot is released, the next eligible request is admitted. Queue waiting does not consume the upstream response-header timeout. +For multiple API-key pool entries, opt into normal-request distribution separately from the existing failure-selection `apiKeyPoolStrategy`: + +```json +{ + "requestPacing": { + "enabled": true, + "keySelection": "round-robin", + "maxConcurrentRequests": 10, + "keys": { "requestsPerMinute": 15, "maxConcurrentRequests": 1 } + } +} +``` + +This admits at most ten active requests across the provider and one per credential reference, with starts on each reference at least four seconds apart. These are local upper bounds, not a promise of upstream capacity. A provider-wide RPM setting, when present, remains an aggregate limit and does not multiply with key count. Key limits span models; model rules remain additional restrictions. Projects sharing quota need an appropriate aggregate limit; TPM and daily quotas are not tracked here. + +Round-robin selects keys per request without saving a new active key. Known 401/429 cooldowns are skipped; when none remains eligible, new requests receive a retryable 429 instead of repeatedly sending. Already queued work retains its selected key and may wait behind that key even when another is idle; the bounded queue expires rather than waiting forever. Selection and diagnostics are process-local, reset on restart, and do not equal balanced token usage or billing. Starts count pacing admissions, including work cancelled before dispatch. OAuth and forwarded credentials keep their existing selection. The dashboard shows key handles, starts, in-flight work, queued work and known cooldowns, never secret key values. These limits cover the inference pacing executor, not the standalone Images relay's direct fetch. + ### Azure OpenAI model metadata discovery For a base URL whose hostname ends in `.openai.azure.com`, discovery fills missing image/text diff --git a/docs-site/src/content/docs/ru/reference/configuration/providers.md b/docs-site/src/content/docs/ru/reference/configuration/providers.md index 424cd5af5aa..a503e44881c 100644 --- a/docs-site/src/content/docs/ru/reference/configuration/providers.md +++ b/docs-site/src/content/docs/ru/reference/configuration/providers.md @@ -96,7 +96,9 @@ cross-route credential fallback не существует. Строки API GPT- | --- | --- | --- | | `adapter` | `string` | Один из `openai-chat`, `openai-responses`, `anthropic`, `google`, `kiro`, `cursor`, `ollama-native`, `azure-openai` (или alias `azure`). | | `baseUrl` | `string` | Базовый URL API upstream'а. Большинство built-in fixed-endpoint'ов игнорируют несовпадение; collision-safe key-preset'ы сохраняют старый custom destination с тем же именем. | -| `requestPacing?` | `{ enabled, requestsPerMinute?, minIntervalMs?, maxConcurrentRequests?, models? }` | Опциональное клиентское выравнивание начала исходящих запросов, отдельное от учёта использования, биллинга и индикаторов rate limit апстрима. `maxConcurrentRequests` — положительное целое, ограничивающее число запросов в работе; правило провайдера или модели может содержать только этот параметр. Лимит провайдера действует на все модели, а `models` сопоставляется с точными ID моделей апстрима и может увеличить задержку или уменьшить параллелизм. Ожидание очереди не расходует таймаут заголовков ответа. Поддерживаются HTTP и явные вызовы адаптеров `fetchResponse`/`runTurn`. При ограничении параллелизма канонический запрос Responses WebSocket использует HTTP/SSE, чтобы освободить слот после завершения, ошибки или отмены тела ответа. Для адаптеров `runTurn`, включая Cursor, лимит считает активные ходы, а не физические отправки: RunSSE и BidiAppend могут пересекаться внутри одного хода, пока следующий ход ждёт. Последующие отправки по-прежнему соблюдают интервалы запуска. | +| `requestPacing?` | `{ enabled, requestsPerMinute?, minIntervalMs?, maxConcurrentRequests?, models?, keySelection?, keys? }` | Опциональное клиентское выравнивание начала исходящих запросов, отдельное от учёта использования, биллинга и индикаторов rate limit апстрима. `maxConcurrentRequests` — положительное целое, ограничивающее число запросов в работе; правило провайдера или модели может содержать только этот параметр. Лимит провайдера действует на все модели, а `models` сопоставляется с точными ID моделей апстрима и может увеличить задержку или уменьшить параллелизм. Ожидание очереди не расходует таймаут заголовков ответа. Поддерживаются HTTP и явные вызовы адаптеров `fetchResponse`/`runTurn`. При ограничении параллелизма канонический запрос Responses WebSocket использует HTTP/SSE, чтобы освободить слот после завершения, ошибки или отмены тела ответа. Для адаптеров `runTurn`, включая Cursor, лимит считает активные ходы, а не физические отправки: RunSSE и BidiAppend могут пересекаться внутри одного хода, пока следующий ход ждёт. Последующие отправки по-прежнему соблюдают интервалы запуска. | +| `requestPacing.keySelection?` | `"active"`, `"round-robin"` | Выбор ключа для обычных запросов: `active` по умолчанию сохраняет выбранный ключ и переключение при ошибке; `round-robin` последовательно распределяет запросы между доступными ключами без изменения сохранённого выбора. Балансируются назначения, не токены или стоимость. | +| `requestPacing.keys?` | `{ requestsPerMinute?, minIntervalMs?, maxConcurrentRequests? }` | Общие для всех моделей ограничения RPM, интервала и параллелизма на ссылку ключа, вместе с лимитами провайдера и модели. Одинаковые ссылки делят ёмкость; разные ссылки одного проекта не означают независимые квоты. TPM и дневные квоты не контролируются. | | `responsesPath?` | `string` | Relative resource path для key-auth запросов `openai-responses`. Должен начинаться с `/` и не может содержать scheme, query или fragment. | | `chatCompletionsPath?` | `string` | Relative resource path для запросов `openai-chat`, зеркало `responsesPath` с теми же правилами формы. Нужен, когда один upstream обслуживает Chat Completions и Responses под разными префиксами: per-model wire override меняет адаптер и не трогает `baseUrl`, поэтому без него включённый Chat-запрос ушёл бы в Responses base. Поставляемый пример — Z.AI. | | `upstreamWebsocket?` | `boolean` | Необязательный upstream Responses WebSocket для запросов `openai-responses` (по умолчанию `false`). Учитывается только для first-party upstream `https://api.openai.com/v1`; конечные точки пользовательских провайдеров всегда используют ограниченный HTTP/SSE, поскольку Bun не может применить ограничение размера входящего сообщения WebSocket до выделения памяти под всё сообщение. У канонического провайдера ChatGPT `openai` пропуск сохраняет upstream WebSocket для подходящих ходов, `false` отправляет потоковые ходы по HTTP/SSE, а `true` отклоняется; при `false` нативное управление и внедрение в середине хода недоступны. Это поле не зависит от клиентской настройки `websockets` и не меняет ни конечную точку, ни учётные данные. Для HTTP остаётся SSE; пути, не относящиеся к Responses, и запросы `openai-chat` остаются на HTTP. | diff --git a/docs-site/src/content/docs/zh-cn/reference/configuration/providers.md b/docs-site/src/content/docs/zh-cn/reference/configuration/providers.md index a02f4d243b4..1c5120a3b04 100644 --- a/docs-site/src/content/docs/zh-cn/reference/configuration/providers.md +++ b/docs-site/src/content/docs/zh-cn/reference/configuration/providers.md @@ -83,7 +83,9 @@ selector，而不是分配一个新名称。 | --- | --- | --- | | `adapter` | `string` | `openai-chat`、`openai-responses`、`anthropic`、`google`、`kiro`、`cursor`、`ollama-native`、`azure-openai`（或别名 `azure`）之一。 | | `baseUrl` | `string` | 上游 API 基础 URL。大多数内置固定端点会忽略不匹配的值；具备冲突安全键的预设会保留一个更早、同名的自定义目标。 | -| `requestPacing?` | `{ enabled, requestsPerMinute?, minIntervalMs?, maxConcurrentRequests?, models? }` | 可选的客户端出站请求启动节流，与上游用量、计费和限流指标相互独立。`maxConcurrentRequests` 是限制进行中请求数的正整数，提供商或模型规则都可以只配置此项。提供商限制适用于所有模型，`models` 按上游模型精确 ID 匹配，并可增加延迟或收紧并发限制。排队等待不计入响应头超时。覆盖 HTTP 以及显式适配器 `fetchResponse`/`runTurn` 调用。配置并发上限时，标准 Responses WebSocket 请求改用 HTTP/SSE，以便在响应体完成、出错或取消时释放并发名额。 对包括 Cursor 在内的 `runTurn` 适配器，并发上限计算进行中的回合数，而非物理发送数：同一回合内的 RunSSE 和 BidiAppend 可以重叠，其他回合仍须等待。后续发送仍遵守启动间隔。 | +| `requestPacing?` | `{ enabled, requestsPerMinute?, minIntervalMs?, maxConcurrentRequests?, models?, keySelection?, keys? }` | 可选的客户端出站请求启动节流，与上游用量、计费和限流指标相互独立。`maxConcurrentRequests` 是限制进行中请求数的正整数，提供商或模型规则都可以只配置此项。提供商限制适用于所有模型，`models` 按上游模型精确 ID 匹配，并可增加延迟或收紧并发限制。排队等待不计入响应头超时。覆盖 HTTP 以及显式适配器 `fetchResponse`/`runTurn` 调用。配置并发上限时，标准 Responses WebSocket 请求改用 HTTP/SSE，以便在响应体完成、出错或取消时释放并发名额。 对包括 Cursor 在内的 `runTurn` 适配器，并发上限计算进行中的回合数，而非物理发送数：同一回合内的 RunSSE 和 BidiAppend 可以重叠，其他回合仍须等待。后续发送仍遵守启动间隔。 | +| `requestPacing.keySelection?` | `"active"`, `"round-robin"` | 正常请求的密钥选择：默认 `active` 保留已保存密钥和现有错误切换；`round-robin` 按顺序分配可用池密钥，不修改已保存的选择。均衡的是分配次数，不是令牌或费用。 | +| `requestPacing.keys?` | `{ requestsPerMinute?, minIntervalMs?, maxConcurrentRequests? }` | 按密钥引用设置统一 RPM、最小启动间隔和并发上限，跨模型共享，并同时受提供商和模型上限约束。相同引用共享容量；同一上游项目的不同引用不代表独立配额。不管理 TPM 或每日配额。 | | `responsesPath?` | `string` | 用于 key-auth `openai-responses` 请求的相对资源路径。必须以 `/` 开头，且不能包含 scheme、query 或 fragment。 | | `chatCompletionsPath?` | `string` | 用于 `openai-chat` 请求的相对资源路径，是 `responsesPath` 的对应项，适用相同的路径规则。当同一上游以不同前缀提供 Chat Completions 和 Responses 时需要此配置：按模型的 wire override 只更换适配器而不改动 `baseUrl`，否则已启用的 Chat 请求会被发送到 Responses base。随附示例为 Z.AI。 | | `upstreamWebsocket?` | `boolean` | 为 `openai-responses` 请求选择性启用上游 Responses WebSocket 传输（默认 `false`）。仅对第一方 `https://api.openai.com/v1` 上游生效；自定义提供者端点始终使用有界 HTTP/SSE，因为 Bun 无法在分配完整消息之前对入站 WebSocket 消息实施大小限制。对于规范 ChatGPT `openai` 提供商，省略该字段会在符合条件的轮次使用上游 WebSocket，`false` 通过 HTTP/SSE 发送流式轮次，`true` 会被拒绝；设为 `false` 时，原生轮次中操控与注入不可用。该字段独立于客户端侧的 `websockets` 设置，且不改变端点或凭据。普通 HTTP 仍使用 SSE；非 Responses 路径和 `openai-chat` 请求仍使用 HTTP。 | diff --git a/docs-site/src/content/docs/zh-tw/reference/configuration/providers.md b/docs-site/src/content/docs/zh-tw/reference/configuration/providers.md index 9689311fb77..830785f8ad2 100644 --- a/docs-site/src/content/docs/zh-tw/reference/configuration/providers.md +++ b/docs-site/src/content/docs/zh-tw/reference/configuration/providers.md @@ -63,7 +63,9 @@ ocx models provider openrouter on | --- | --- | --- | | `adapter` | `string` | `openai-chat`、`openai-responses`、`anthropic`、`google`、`kiro`、`cursor`、`ollama-native`、`azure-openai`（或別名 `azure`）之一。 | | `baseUrl` | `string` | 上游 API base URL。多數內建固定端點忽略不符；碰撞安全的金鑰預設保留較舊的同名自訂目的地。 | -| `requestPacing?` | `{ enabled, requestsPerMinute?, minIntervalMs?, maxConcurrentRequests?, models? }` | 選用的用戶端出站請求啟動節流，與上游用量、計費及限流指標彼此獨立。`maxConcurrentRequests` 是限制進行中請求數的正整數，供應商或模型規則都可單獨設定此項。供應商限制適用於所有模型，`models` 依上游模型精確 ID 比對，並可增加延遲或收緊並發限制。排隊等待不計入回應標頭逾時。涵蓋 HTTP 及明確的適配器 `fetchResponse`/`runTurn` 呼叫。設定並行上限時，標準 Responses WebSocket 請求改用 HTTP/SSE，以便在回應本文完成、出錯或取消時釋放並行名額。 對包含 Cursor 的 `runTurn` 適配器，並行上限計算進行中的回合數，而非實體傳送數：同一回合內的 RunSSE 與 BidiAppend 可重疊，其他回合仍須等待。後續傳送仍遵守啟動間隔。 | +| `requestPacing?` | `{ enabled, requestsPerMinute?, minIntervalMs?, maxConcurrentRequests?, models?, keySelection?, keys? }` | 選用的用戶端出站請求啟動節流，與上游用量、計費及限流指標彼此獨立。`maxConcurrentRequests` 是限制進行中請求數的正整數，供應商或模型規則都可單獨設定此項。供應商限制適用於所有模型，`models` 依上游模型精確 ID 比對，並可增加延遲或收緊並發限制。排隊等待不計入回應標頭逾時。涵蓋 HTTP 及明確的適配器 `fetchResponse`/`runTurn` 呼叫。設定並行上限時，標準 Responses WebSocket 請求改用 HTTP/SSE，以便在回應本文完成、出錯或取消時釋放並行名額。 對包含 Cursor 的 `runTurn` 適配器，並行上限計算進行中的回合數，而非實體傳送數：同一回合內的 RunSSE 與 BidiAppend 可重疊，其他回合仍須等待。後續傳送仍遵守啟動間隔。 | +| `requestPacing.keySelection?` | `"active"`, `"round-robin"` | 正常請求的金鑰選擇：預設 `active` 保留已儲存金鑰及既有錯誤切換；`round-robin` 依序分配可用池金鑰，不修改已儲存選擇。均衡的是分配次數，而非權杖或費用。 | +| `requestPacing.keys?` | `{ requestsPerMinute?, minIntervalMs?, maxConcurrentRequests? }` | 依金鑰參照設定統一 RPM、最小啟動間隔及並行上限，跨模型共享，並同時受供應商及模型上限約束。相同參照共享容量；同一上游專案的不同參照不代表獨立配額。不管理 TPM 或每日配額。 | | `responsesPath?` | `string` | Key-auth `openai-responses` 請求的相對資源路徑。必須以 `/` 開頭且不含 scheme、query 或 fragment。 | | `chatCompletionsPath?` | `string` | `openai-chat` 請求的相對資源路徑，為 `responsesPath` 的對應項，適用相同的路徑規則。當同一上游以不同前綴提供 Chat Completions 與 Responses 時需要此設定：按模型的 wire override 只更換適配器而不改動 `baseUrl`，否則已啟用的 Chat 請求會送往 Responses base。隨附範例為 Z.AI。 | | `upstreamWebsocket?` | `boolean` | 為 `openai-responses` 請求選用上游 Responses WebSocket 傳輸（預設 `false`）。僅對第一方 `https://api.openai.com/v1` 上游生效；自訂供應商端點一律使用有界 HTTP/SSE，因為 Bun 無法在配置完整訊息之前對傳入 WebSocket 訊息套用大小限制。對於規範 ChatGPT `openai` 供應商，省略此欄位會在符合條件的回合使用上游 WebSocket，`false` 會以 HTTP/SSE 傳送串流回合，`true` 會被拒絕；設為 `false` 時，原生回合中操控與注入無法使用。此欄位獨立於用戶端 `websockets` 設定，且不會變更端點或認證資料。一般 HTTP 仍使用 SSE；非 Responses 路徑與 `openai-chat` 請求仍使用 HTTP。 | diff --git a/gui/src/components/provider-workspace/ProviderKeyPacing.tsx b/gui/src/components/provider-workspace/ProviderKeyPacing.tsx new file mode 100644 index 00000000000..7ca4d9708b0 --- /dev/null +++ b/gui/src/components/provider-workspace/ProviderKeyPacing.tsx @@ -0,0 +1,37 @@ +import { useT } from "../../i18n/shared"; +import type { WorkspaceItem } from "./types"; + +export type KeyPacingDraft = Pick, "keySelection" | "keys">; +export type KeyPacingStatus = { id: string; starts: number; inFlight: number; queued: number; nextSlotInMs: number; cooldownUntil?: number | null }; + +export function ProviderKeyPacing({ value, onChange, status = [] }: { + value: KeyPacingDraft; onChange: (value: KeyPacingDraft) => void; status?: KeyPacingStatus[]; +}) { + const t = useT(); + const numberField = (field: "requestsPerMinute" | "minIntervalMs" | "maxConcurrentRequests", raw: string) => { + const keys = { ...value.keys }; + if (!raw.trim()) delete keys[field]; + else keys[field] = Number(raw); + onChange({ ...value, keys: Object.keys(keys).length ? keys : undefined }); + }; + return <> + +

{t("pws.keyLimits")}

+
+ {(["requestsPerMinute", "minIntervalMs", "maxConcurrentRequests"] as const).map((field, index) => )} +
+

{t("pws.keySharedQuota")}

+ {status.length > 0 && + + {status.map(row => + )}
{t("pws.keyHandle")}{t("pws.keyStarts")}{t("pws.keyInFlight")}{t("pws.pacingQueued")}{t("pws.keyWait")}
{row.id}{row.starts}{row.inFlight}{row.queued}{row.cooldownUntil ? t("pws.keyCooldown") : `${row.nextSlotInMs} ms`}
} + ; +} diff --git a/gui/src/components/provider-workspace/ProviderSettings.tsx b/gui/src/components/provider-workspace/ProviderSettings.tsx index 436dc3a97d5..774757e5b9a 100644 --- a/gui/src/components/provider-workspace/ProviderSettings.tsx +++ b/gui/src/components/provider-workspace/ProviderSettings.tsx @@ -22,6 +22,7 @@ import { providerSupportsLiveModelDiscovery } from "../../provider-workspace/cat import type { CatalogPreset } from "../provider-catalog/provider-presets"; import { authModeLabel } from "./ProviderRail"; import { ProviderProtocolPanel } from "./ProviderProtocolPanel"; +import { ProviderKeyPacing, type KeyPacingDraft, type KeyPacingStatus } from "./ProviderKeyPacing"; import type { WorkspaceItem, ProviderUpdatePatch, ProviderUpdateResult } from "./types"; const ADAPTERS = ["openai-responses", "openai-chat", "anthropic", "google", "azure-openai", "cursor"] as const; @@ -29,7 +30,7 @@ const EMPTY_MODELS: string[] = []; type ChoicesStatus = "idle" | "loading" | "ready" | "error"; type PacingRule = { requestsPerMinute?: number; minIntervalMs?: number; maxConcurrentRequests?: number }; -type PacingStatus = { enabled: boolean; queued: number; nextSlotInMs: number; lastStartedAt?: number; lastModelId?: string }; +type PacingStatus = { enabled: boolean; queued: number; nextSlotInMs: number; lastStartedAt?: number; lastModelId?: string; keys?: KeyPacingStatus[] }; type CursorHttpVersion = "http2" | "http1.1"; function effectiveCursorHttpVersion(value: WorkspaceItem["upstreamHttpVersion"]): CursorHttpVersion { @@ -56,6 +57,8 @@ function pacingSignature(value: WorkspaceItem["requestPacing"] | undefined): str value?.requestsPerMinute ?? null, value?.minIntervalMs ?? null, value?.maxConcurrentRequests ?? null, + value?.keySelection ?? "active", + [value?.keys?.requestsPerMinute ?? null, value?.keys?.minIntervalMs ?? null, value?.keys?.maxConcurrentRequests ?? null], models, ]); } @@ -95,6 +98,7 @@ export default function ProviderSettings({ const [choicesStatus, setChoicesStatus] = useState(apiBase ? "loading" : "idle"); const [endpointChoice, setEndpointChoice] = useState(() => "custom"); const [pacingEnabled, setPacingEnabled] = useState(item.requestPacing?.enabled === true); + const [keyPacing, setKeyPacing] = useState({ keySelection: item.requestPacing?.keySelection, keys: item.requestPacing?.keys }); const [pacingRpm, setPacingRpm] = useState(() => numberDraft(item.requestPacing?.requestsPerMinute)); const [pacingDelay, setPacingDelay] = useState(() => numberDraft(item.requestPacing?.minIntervalMs)); const [pacingConcurrency, setPacingConcurrency] = useState(() => numberDraft(item.requestPacing?.maxConcurrentRequests)); @@ -117,6 +121,7 @@ export default function ProviderSettings({ setLiveModels(savedLiveModels); setCursorHttpVersion(savedCursorHttpVersion); setPacingEnabled(item.requestPacing?.enabled === true); + setKeyPacing({ keySelection: item.requestPacing?.keySelection, keys: item.requestPacing?.keys }); setPacingRpm(numberDraft(item.requestPacing?.requestsPerMinute)); setPacingDelay(numberDraft(item.requestPacing?.minIntervalMs)); setPacingConcurrency(numberDraft(item.requestPacing?.maxConcurrentRequests)); @@ -188,11 +193,12 @@ export default function ProviderSettings({ const pacingDraft = useMemo(() => ({ enabled: pacingEnabled, + ...keyPacing, ...(positiveRpm(pacingRpm) !== undefined ? { requestsPerMinute: positiveRpm(pacingRpm) } : {}), ...(positiveInteger(pacingDelay) !== undefined ? { minIntervalMs: positiveInteger(pacingDelay) } : {}), ...(positiveInteger(pacingConcurrency) !== undefined ? { maxConcurrentRequests: positiveInteger(pacingConcurrency) } : {}), ...(Object.keys(pacingModels).length > 0 ? { models: pacingModels } : {}), - }), [pacingConcurrency, pacingDelay, pacingEnabled, pacingModels, pacingRpm]); + }), [keyPacing, pacingConcurrency, pacingDelay, pacingEnabled, pacingModels, pacingRpm]); const dirty = adapter.trim() !== item.adapter || baseUrl.trim() !== item.baseUrl @@ -243,7 +249,14 @@ export default function ProviderSettings({ if (pacingConcurrency.trim() !== "" && positiveInteger(pacingConcurrency) === undefined) { setMsg({ ok: false, text: t("pws.pacingCapInvalid") }); return false; } - if (pacingEnabled && !pacingDraft.requestsPerMinute && !pacingDraft.minIntervalMs && !pacingDraft.maxConcurrentRequests && !pacingDraft.models) { + if (Object.entries(keyPacing.keys ?? {}).some(([field, value]) => + !Number.isFinite(value) || value <= 0 + || (field !== "requestsPerMinute" && !Number.isSafeInteger(value)) + || (field === "requestsPerMinute" && (value < 1 / 60 || value > 60_000)) + || (field === "minIntervalMs" && value > 3_600_000))) { + setMsg({ ok: false, text: t("pws.pacingRuleRequired") }); return false; + } + if (pacingEnabled && !pacingDraft.requestsPerMinute && !pacingDraft.minIntervalMs && !pacingDraft.maxConcurrentRequests && !pacingDraft.models && !pacingDraft.keys && pacingDraft.keySelection !== "round-robin") { setMsg({ ok: false, text: t("pws.pacingRuleRequired") }); return false; } const pacingOnly = pacingDirty && !dirty; @@ -330,6 +343,7 @@ export default function ProviderSettings({ setNote(item.note ?? ""); setAllowPrivateNetwork(item.allowPrivateNetwork ?? false); setLiveModels(savedLiveModels); setCursorHttpVersion(savedCursorHttpVersion); setMsg(null); setPacingEnabled(item.requestPacing?.enabled === true); setPacingRpm(numberDraft(item.requestPacing?.requestsPerMinute)); + setKeyPacing({ keySelection: item.requestPacing?.keySelection, keys: item.requestPacing?.keys }); setPacingDelay(numberDraft(item.requestPacing?.minIntervalMs)); setPacingConcurrency(numberDraft(item.requestPacing?.maxConcurrentRequests)); setPacingModels({ ...(item.requestPacing?.models ?? {}) }); @@ -530,6 +544,7 @@ export default function ProviderSettings({ {pacingStatus?.nextSlotInMs ?? 0} ms {t("pws.pacingNextSlot")} {pacingStatus?.lastModelId ?? t("pws.pacingNone")} {t("pws.pacingLastModel")} + {authMode !== "oauth" && authMode !== "forward" && }

{t("pws.pacingModelOverrides")}

diff --git a/gui/src/i18n/de.ts b/gui/src/i18n/de.ts index d1484ec0dc3..d492ab37bb8 100644 --- a/gui/src/i18n/de.ts +++ b/gui/src/i18n/de.ts @@ -2888,6 +2888,16 @@ export const de: Record = { "pws.removeDefaultConfirmBody": "Standardanbieter \"{name}\" entfernen? \"{defaultProvider}\" wird zum Standardanbieter. Dies kann nicht rückgängig gemacht werden.", "pws.removeConfirmTitle": "Anbieter entfernen", "pws.saveSettings": "Speichern", + "pws.keySelection": "API-Schlüsselauswahl", + "pws.keyActive": "Aktiver Schlüssel (bisheriges Verhalten)", + "pws.keyRoundRobin": "Round-Robin", + "pws.keyLimits": "Limits pro Schlüssel", + "pws.keySharedQuota": "Provider-Limits gelten weiterhin für alle Schlüssel. Schlüssel desselben Projekts können Kontingente teilen; RPM begrenzt keine Token- oder Tagesbudgets.", + "pws.keyHandle": "Schlüssel-ID", + "pws.keyStarts": "Gestartet", + "pws.keyInFlight": "Aktiv", + "pws.keyWait": "Warten", + "pws.keyCooldown": "Abkühlzeit", "pws.pacingTitle": "Anfragetaktung", "pws.pacingDesc": "Verteilt ausgehende Anfragestarts für diesen Anbieter gleichmäßig. Streaming-Antworten dürfen sich überlappen.", "pws.pacingEnabled": "Aktiviert", diff --git a/gui/src/i18n/en.ts b/gui/src/i18n/en.ts index a5bd623dfa3..3d1f741d63f 100644 --- a/gui/src/i18n/en.ts +++ b/gui/src/i18n/en.ts @@ -1793,6 +1793,16 @@ export const en = { "pws.removeDefaultConfirmBody": "Remove default provider \"{name}\"? \"{defaultProvider}\" will become the default provider. This cannot be undone.", "pws.removeConfirmTitle": "Remove provider", "pws.saveSettings": "Save", + "pws.keySelection": "API key selection", + "pws.keyActive": "Active key (existing behavior)", + "pws.keyRoundRobin": "Round-robin", + "pws.keyLimits": "Per-key limits", + "pws.keySharedQuota": "Provider limits still apply across all keys. Keys in the same project may share quotas; RPM limits do not enforce token or daily budgets.", + "pws.keyHandle": "Key ID", + "pws.keyStarts": "Started", + "pws.keyInFlight": "In flight", + "pws.keyWait": "Wait", + "pws.keyCooldown": "Cooldown", "pws.pacingTitle": "Request pacing", "pws.pacingDesc": "Evenly delay outbound request starts for this provider. Streaming responses may overlap.", "pws.pacingEnabled": "Enabled", diff --git a/gui/src/i18n/fr.ts b/gui/src/i18n/fr.ts index 325dffd175c..505fa31e9de 100644 --- a/gui/src/i18n/fr.ts +++ b/gui/src/i18n/fr.ts @@ -1740,6 +1740,16 @@ export const fr: Record = { "pws.cursorTransportHttp1": "HTTP/1.1 (compatibilité proxy)", "pws.cursorTransportDesc": "Utilisez HTTP/1.1 si votre proxy ne transporte pas de façon fiable le flux HTTP/2 de Cursor.", "pws.optionalPlaceholder": "Facultatif", + "pws.keySelection": "Sélection de clé API", + "pws.keyActive": "Clé active (comportement existant)", + "pws.keyRoundRobin": "Tourniquet", + "pws.keyLimits": "Limites par clé", + "pws.keySharedQuota": "Les limites du fournisseur restent communes à toutes les clés. Un même projet peut partager ses quotas ; les RPM ne limitent ni les jetons ni le budget quotidien.", + "pws.keyHandle": "ID de clé", + "pws.keyStarts": "Démarrées", + "pws.keyInFlight": "En cours", + "pws.keyWait": "Attente", + "pws.keyCooldown": "Temporisation", "pws.pacingTitle": "Cadencement des requêtes", "pws.pacingDesc": "Répartissez uniformément le démarrage des requêtes sortantes pour ce fournisseur. Les réponses diffusées peuvent se chevaucher.", "pws.pacingEnabled": "Activé", diff --git a/gui/src/i18n/ja.ts b/gui/src/i18n/ja.ts index 12dc523cd69..aa09906632e 100644 --- a/gui/src/i18n/ja.ts +++ b/gui/src/i18n/ja.ts @@ -1661,6 +1661,16 @@ export const ja: Record = { "pws.removeDefaultConfirmBody": "既定のプロバイダー \"{name}\" を削除しますか? \"{defaultProvider}\" が既定のプロバイダーになります。この操作は元に戻せません。", "pws.removeConfirmTitle": "プロバイダーを削除", "pws.saveSettings": "保存", + "pws.keySelection": "API キーの選択", + "pws.keyActive": "有効なキー優先（既存の動作）", + "pws.keyRoundRobin": "ラウンドロビン", + "pws.keyLimits": "キーごとの制限", + "pws.keySharedQuota": "プロバイダー全体の制限も適用されます。同じプロジェクトのキーは割当量を共有する場合があります。RPM はトークン数や日次上限を制御しません。", + "pws.keyHandle": "キー ID", + "pws.keyStarts": "開始回数", + "pws.keyInFlight": "実行中", + "pws.keyWait": "待機", + "pws.keyCooldown": "クールダウン", "pws.pacingTitle": "リクエスト間隔調整", "pws.pacingDesc": "このプロバイダーへの送信開始を均等に遅延します。ストリーミング応答は重複できます。", "pws.pacingEnabled": "有効", diff --git a/gui/src/i18n/ko.ts b/gui/src/i18n/ko.ts index 499b65bcf64..928e0df576c 100644 --- a/gui/src/i18n/ko.ts +++ b/gui/src/i18n/ko.ts @@ -2935,6 +2935,16 @@ export const ko: Record = { "pws.removeDefaultConfirmBody": "기본 프로바이더 \"{name}\"을(를) 제거하시겠습니까? \"{defaultProvider}\"이(가) 기본 프로바이더가 됩니다. 이 작업은 되돌릴 수 없습니다.", "pws.removeConfirmTitle": "프로바이더 제거", "pws.saveSettings": "저장", + "pws.keySelection": "API 키 선택 방식", + "pws.keyActive": "활성 키 우선 (기존 동작)", + "pws.keyRoundRobin": "라운드로빈", + "pws.keyLimits": "키별 제한", + "pws.keySharedQuota": "프로바이더 제한은 모든 키에 합산 적용됩니다. 같은 프로젝트의 키는 할당량을 공유할 수 있으며, RPM 제한은 토큰·일일 한도를 관리하지 않습니다.", + "pws.keyHandle": "키 ID", + "pws.keyStarts": "시작 횟수", + "pws.keyInFlight": "처리 중", + "pws.keyWait": "대기", + "pws.keyCooldown": "쿨다운", "pws.pacingTitle": "요청 속도 조절", "pws.pacingDesc": "이 프로바이더로 나가는 요청 시작을 일정 간격으로 지연합니다. 스트리밍 응답은 서로 겹칠 수 있습니다.", "pws.pacingEnabled": "사용", diff --git a/gui/src/i18n/pt.ts b/gui/src/i18n/pt.ts index 098b8ec9d3b..7bb50fdd37c 100644 --- a/gui/src/i18n/pt.ts +++ b/gui/src/i18n/pt.ts @@ -1777,6 +1777,16 @@ export const pt: Record = { "pws.removeDefaultConfirmBody": "Remover o provedor padrão \"{name}\"? \"{defaultProvider}\" passará a ser o provedor padrão. Esta ação não pode ser desfeita.", "pws.removeConfirmTitle": "Remover provedor", "pws.saveSettings": "Salvar", + "pws.keySelection": "Seleção de chave API", + "pws.keyActive": "Chave ativa (comportamento atual)", + "pws.keyRoundRobin": "Round-robin", + "pws.keyLimits": "Limites por chave", + "pws.keySharedQuota": "Os limites do provedor continuam valendo para todas as chaves. Chaves do mesmo projeto podem compartilhar cotas; RPM não limita tokens nem o orçamento diário.", + "pws.keyHandle": "ID da chave", + "pws.keyStarts": "Iniciadas", + "pws.keyInFlight": "Em andamento", + "pws.keyWait": "Espera", + "pws.keyCooldown": "Intervalo de recuperação", "pws.pacingTitle": "Ritmo de requisições", "pws.pacingDesc": "Distribui uniformemente o início das requisições de saída deste provedor. Respostas em streaming podem se sobrepor.", "pws.pacingEnabled": "Ativado", diff --git a/gui/src/i18n/ru.ts b/gui/src/i18n/ru.ts index c1b349abe7c..9e3d7afd51e 100644 --- a/gui/src/i18n/ru.ts +++ b/gui/src/i18n/ru.ts @@ -1752,6 +1752,16 @@ export const ru: Record = { "pws.removeDefaultConfirmBody": "Удалить провайдера по умолчанию \"{name}\"? \"{defaultProvider}\" станет провайдером по умолчанию. Это действие нельзя отменить.", "pws.removeConfirmTitle": "Удалить провайдера", "pws.saveSettings": "Сохранить", + "pws.keySelection": "Выбор API-ключа", + "pws.keyActive": "Активный ключ (прежнее поведение)", + "pws.keyRoundRobin": "По кругу", + "pws.keyLimits": "Лимиты на ключ", + "pws.keySharedQuota": "Лимиты провайдера действуют для всех ключей. Ключи одного проекта могут делить квоты; RPM не ограничивает токены и суточный бюджет.", + "pws.keyHandle": "ID ключа", + "pws.keyStarts": "Запущено", + "pws.keyInFlight": "В работе", + "pws.keyWait": "Ожидание", + "pws.keyCooldown": "Пауза", "pws.pacingTitle": "Интервал запросов", "pws.pacingDesc": "Равномерно задерживает начало исходящих запросов к провайдеру. Потоковые ответы могут пересекаться.", "pws.pacingEnabled": "Включено", diff --git a/gui/src/i18n/tr.ts b/gui/src/i18n/tr.ts index 002b64024bf..40ce45e2bc4 100644 --- a/gui/src/i18n/tr.ts +++ b/gui/src/i18n/tr.ts @@ -1779,6 +1779,16 @@ export const tr: Record = { "pws.removeDefaultConfirmBody": "Varsayılan sağlayıcı \"{name}\" kaldırılsın mı? \"{defaultProvider}\" varsayılan sağlayıcı olacaktır. Bu işlem geri alınamaz.", "pws.removeConfirmTitle": "Sağlayıcıyı kaldır", "pws.saveSettings": "Kaydet", + "pws.keySelection": "API anahtarı seçimi", + "pws.keyActive": "Etkin anahtar (mevcut davranış)", + "pws.keyRoundRobin": "Sıralı dağıtım", + "pws.keyLimits": "Anahtar başına sınırlar", + "pws.keySharedQuota": "Sağlayıcı sınırları tüm anahtarlara uygulanır. Aynı projedeki anahtarlar kota paylaşabilir; RPM, token veya günlük bütçeyi sınırlamaz.", + "pws.keyHandle": "Anahtar ID", + "pws.keyStarts": "Başlatılan", + "pws.keyInFlight": "Çalışan", + "pws.keyWait": "Bekleme", + "pws.keyCooldown": "Soğuma", "pws.pacingTitle": "İstek aralığı", "pws.pacingDesc": "Bu sağlayıcıya giden istek başlangıçlarını eşit aralıklarla geciktirir. Akış yanıtları çakışabilir.", "pws.pacingEnabled": "Etkin", diff --git a/gui/src/i18n/vi.ts b/gui/src/i18n/vi.ts index 5c66b31cc03..2876d2a00db 100644 --- a/gui/src/i18n/vi.ts +++ b/gui/src/i18n/vi.ts @@ -1763,6 +1763,16 @@ export const vi: Record = { "pws.removeDefaultConfirmBody": "Xoá provider mặc định \\\"{name}\\\"? \\\"{defaultProvider}\\\" sẽ trở thành provider mặc định. Hành động này không thể hoàn tác.", "pws.removeConfirmTitle": "Xóa Provider", "pws.saveSettings": "Lưu", + "pws.keySelection": "Chọn khóa API", + "pws.keyActive": "Khóa đang hoạt động (hành vi hiện tại)", + "pws.keyRoundRobin": "Luân phiên", + "pws.keyLimits": "Giới hạn mỗi khóa", + "pws.keySharedQuota": "Giới hạn nhà cung cấp vẫn áp dụng cho mọi khóa. Khóa cùng dự án có thể dùng chung hạn mức; RPM không giới hạn token hay ngân sách ngày.", + "pws.keyHandle": "ID khóa", + "pws.keyStarts": "Đã bắt đầu", + "pws.keyInFlight": "Đang chạy", + "pws.keyWait": "Chờ", + "pws.keyCooldown": "Tạm nghỉ", "pws.pacingTitle": "Pacing của yêu cầu", "pws.pacingDesc": "Đồng đều trì hoãn thời điểm bắt đầu yêu cầu ra ngoài cho provider này. Các phản hồi stream có thể chồng chéo.", "pws.pacingEnabled": "Đã bật", diff --git a/gui/src/i18n/zh-TW.ts b/gui/src/i18n/zh-TW.ts index baf13ebb18a..d3aaf035eb2 100644 --- a/gui/src/i18n/zh-TW.ts +++ b/gui/src/i18n/zh-TW.ts @@ -1457,6 +1457,16 @@ export const zhTW: Record = { "pws.removeDefaultConfirmBody": "移除預設供應商「{name}」？「{defaultProvider}」將成為預設供應商。此操作無法撤消。", "pws.removeConfirmTitle": "移除供應商", "pws.saveSettings": "儲存", + "pws.keySelection": "API 金鑰選擇", + "pws.keyActive": "優先使用作用中金鑰（現有行為）", + "pws.keyRoundRobin": "輪詢", + "pws.keyLimits": "每個金鑰的限制", + "pws.keySharedQuota": "供應商限制仍對所有金鑰合計生效。同一專案的金鑰可能共享配額；RPM 不限制權杖或每日預算。", + "pws.keyHandle": "金鑰 ID", + "pws.keyStarts": "已開始", + "pws.keyInFlight": "處理中", + "pws.keyWait": "等待", + "pws.keyCooldown": "冷卻中", "pws.pacingTitle": "請求節流", "pws.pacingDesc": "均勻延遲送往此供應商的請求啟動。串流回應可以重疊。", "pws.pacingEnabled": "啟用", diff --git a/gui/src/i18n/zh.ts b/gui/src/i18n/zh.ts index e86031fa907..9f05fa50328 100644 --- a/gui/src/i18n/zh.ts +++ b/gui/src/i18n/zh.ts @@ -2908,6 +2908,16 @@ export const zh: Record = { "pws.removeDefaultConfirmBody": "移除默认提供方「{name}」？「{defaultProvider}」将成为默认提供方。此操作无法撤销。", "pws.removeConfirmTitle": "移除提供商", "pws.saveSettings": "保存", + "pws.keySelection": "API 密钥选择", + "pws.keyActive": "活动密钥优先（现有行为）", + "pws.keyRoundRobin": "轮询", + "pws.keyLimits": "每个密钥的限制", + "pws.keySharedQuota": "提供商限制仍对所有密钥合并生效。同一项目的密钥可能共享配额；RPM 不限制令牌或每日预算。", + "pws.keyHandle": "密钥 ID", + "pws.keyStarts": "已开始", + "pws.keyInFlight": "处理中", + "pws.keyWait": "等待", + "pws.keyCooldown": "冷却中", "pws.pacingTitle": "请求节流", "pws.pacingDesc": "均匀延迟发往此提供商的请求启动。流式响应可以重叠。", "pws.pacingEnabled": "启用", diff --git a/gui/src/provider-workspace/catalog.ts b/gui/src/provider-workspace/catalog.ts index 5647481f718..d7269eb33d6 100644 --- a/gui/src/provider-workspace/catalog.ts +++ b/gui/src/provider-workspace/catalog.ts @@ -57,6 +57,8 @@ export interface WorkspaceProvider { minIntervalMs?: number; maxConcurrentRequests?: number; models?: Record; + keySelection?: "active" | "round-robin"; + keys?: { requestsPerMinute?: number; minIntervalMs?: number; maxConcurrentRequests?: number }; }; /** Codex account routing mode for the canonical `openai` forward provider. */ codexAccountMode?: "direct" | "pool"; diff --git a/gui/tests/provider-settings-request-pacing.test.tsx b/gui/tests/provider-settings-request-pacing.test.tsx index 588671f975b..78b5292fbbf 100644 --- a/gui/tests/provider-settings-request-pacing.test.tsx +++ b/gui/tests/provider-settings-request-pacing.test.tsx @@ -90,6 +90,21 @@ test("settings saves provider pacing and a slower model override", async () => { await act(async () => { container.unmount(); }); }); +test("key allocation and per-key limits save through the existing provider settings action", async () => { + const patches: ProviderUpdatePatch[] = []; + const container = await renderSettings({ name: "fixture", adapter: "openai-chat", baseUrl: "https://example.test", authMode: "key" }, + async (_name, patch) => { patches.push(patch); return { ok: true }; }); + await act(async () => { container.querySelector(".pwi-pacing-toggle input")!.click(); }); + const select = [...container.querySelectorAll("select")].find(el => el.querySelector('option[value="round-robin"]'))!; + await act(async () => { select.value = "round-robin"; select.dispatchEvent(new testWindow.Event("change", { bubbles: true })); }); + const numbers = container.querySelectorAll(".pwi-key-limits input"); + await setInput(numbers[0]!, "15"); await setInput(numbers[2]!, "1"); + await act(async () => { container.querySelector(".pwi-settings-sticky-bar .btn-primary")!.click(); }); + expect(patches[0]?.requestPacing).toMatchObject({ enabled: true, keySelection: "round-robin", keys: { requestsPerMinute: 15, maxConcurrentRequests: 1 } }); + expect(container.textContent).toContain("Keys in the same project may share quotas"); + container.unmount(); +}); + test("a concurrency-only pacing rule saves without the rule-required error", async () => { const item: WorkspaceItem = { name: "nvidia", @@ -135,6 +150,20 @@ test("discard reverts an unsaved concurrency draft to the stored cap", async () await act(async () => { container.unmount(); }); }); +test("invalid key limits block save and discard restores the stored distribution policy", async () => { + const patches: ProviderUpdatePatch[] = []; + const container = await renderSettings({ name: "fixture", adapter: "openai-chat", baseUrl: "https://example.test", authMode: "key", + requestPacing: { enabled: true, keySelection: "round-robin", keys: { requestsPerMinute: 15, maxConcurrentRequests: 1 } } }, + async (_name, patch) => { patches.push(patch); return { ok: true }; }); + const numbers = container.querySelectorAll(".pwi-key-limits input"); + await setInput(numbers[2]!, "0"); + await act(async () => { container.querySelector(".pwi-settings-sticky-bar .btn-primary")!.click(); }); + expect(patches).toEqual([]); + await act(async () => { container.querySelector(".pwi-settings-sticky-bar .btn-ghost")!.click(); }); + expect(numbers[2]!.value).toBe("1"); expect(numbers[0]!.value).toBe("15"); + await act(async () => { container.unmount(); }); +}); + test("pacing grids keep every field and the add button on one row", async () => { const css = await Bun.file(new URL("../src/styles/provider-workspace-settings.css", import.meta.url)).text(); // The provider grid owns three inputs (rpm, interval, concurrency) and the model diff --git a/src/config/schema/leaf-validators.ts b/src/config/schema/leaf-validators.ts index d9e1d07226e..ecedf0dc6f4 100644 --- a/src/config/schema/leaf-validators.ts +++ b/src/config/schema/leaf-validators.ts @@ -133,12 +133,15 @@ const requestPacingSchema = z.object({ minIntervalMs: z.number().int().min(1).max(3_600_000).optional(), maxConcurrentRequests: z.number().int().min(1).optional(), models: z.record(z.string().trim().min(1), requestPacingRuleSchema).optional(), + keySelection: z.enum(["active", "round-robin"]).optional(), + keys: requestPacingRuleSchema.optional(), }).strict().refine(value => value.enabled === false || value.requestsPerMinute !== undefined || value.minIntervalMs !== undefined || value.maxConcurrentRequests !== undefined + || value.keys !== undefined || value.keySelection === "round-robin" || (value.models !== undefined && Object.keys(value.models).length > 0), { - message: "enabled request pacing needs a provider rule or model override", + message: "enabled request pacing needs a provider, model, or key rule, or round-robin selection", }); export function requestPacingConfigError(value: unknown): string | null { diff --git a/src/providers/api-key-selection.ts b/src/providers/api-key-selection.ts index 40eff2d49a9..8e961dbacde 100644 --- a/src/providers/api-key-selection.ts +++ b/src/providers/api-key-selection.ts @@ -17,10 +17,17 @@ function matchesSelection(provider: OcxProviderConfig, expected: ProviderApiKeyS && current.revision === expected.revision; } -function currentKeyProvider(config: OcxConfig, name: string): OcxProviderConfig | null { +function currentKeyProvider(config: OcxConfig, name: string, selection?: ProviderApiKeySelection): OcxProviderConfig | null { const configured = config.providers[name]; if (!configured || configured.disabled) return null; - const current = routedProviderConfig(name, { ...configured, _apiKeyAttempt: undefined }); + let row = configured; + if (selection?.distributed) { + const entry = configured.apiKeyPool?.find(key => key.id === selection.entryId && key.key === selection.reference); + if (!entry || configured.apiKeySelectionRevision !== selection.revision + || !configured.requestPacing?.enabled || configured.requestPacing.keySelection !== "round-robin") return null; + row = { ...configured, apiKey: entry.key, _apiKeyAttempt: selection }; + } + const current = routedProviderConfig(name, { ...row, _apiKeyAttempt: selection?.distributed ? selection : undefined }); if (current.authMode === "oauth" || current.authMode === "forward") return null; if (current.authMode === "key" && !current.keyOptional && !current.apiKey?.trim()) return null; return current; @@ -32,10 +39,10 @@ export function providerApiKeySelectionIsCurrent( name: string, routedProvider: OcxProviderConfig, ): boolean { - const current = currentKeyProvider(config, name); const expected = routedProvider._apiKeyAttempt; + const current = currentKeyProvider(config, name, expected); return current !== null && expected !== undefined - && matchesSelection(config.providers[name]!, expected) + && (expected.distributed === true || matchesSelection(config.providers[name]!, expected)) && current.apiKey === routedProvider.apiKey && current.authMode === routedProvider.authMode && current.baseUrl === routedProvider.baseUrl; @@ -47,7 +54,11 @@ export function resolveCurrentProviderApiKeyTransport( name: string, routedProvider: OcxProviderConfig, ): OcxProviderConfig | null { - const current = currentKeyProvider(config, name); + // A permit acquired for one key cannot authorize a different key after a manual edit. + if (routedProvider.requestPacing?.enabled && routedProvider.requestPacing.keys + && routedProvider._apiKeyAttempt && config.providers[name] + && !routedProvider._apiKeyAttempt.distributed && !matchesSelection(config.providers[name]!, routedProvider._apiKeyAttempt)) return null; + const current = currentKeyProvider(config, name, routedProvider._apiKeyAttempt); if (!current) return null; const runtime = routedProvider as OcxProviderTransport; const headers = { ...current.headers }; diff --git a/src/providers/key-failover.ts b/src/providers/key-failover.ts index cbaca42739c..bfa89938b16 100644 --- a/src/providers/key-failover.ts +++ b/src/providers/key-failover.ts @@ -9,6 +9,7 @@ * Modelled after src/codex/routing.ts cooldown logic but scoped to plain API-key pools. */ import { commitProviderApiKeySelection } from "./api-key-selection"; +import { RequestPacingQueueOverloadError } from "./request-pacing"; import type { ProviderApiKeySelection } from "../types/provider"; import { routedProviderConfig } from "../router"; import { getProviderRegistryEntry } from "./registry"; @@ -408,9 +409,11 @@ const keyRotationCursor = new Map(); export function forgetApiKeyRotationCursor(providerName?: string): void { if (!providerName) { keyRotationCursor.clear(); + dispatchKeyCursor.clear(); return; } keyRotationCursor.delete(providerName); + dispatchKeyCursor.delete(providerName); } /** The pool entry shape is inline on OcxProviderConfig; name it once rather than re-spelling it. */ @@ -556,11 +559,33 @@ export function selectProactiveApiKeyTransport( promptCacheKey?: string, now = Date.now(), ): OcxProviderTransport | null { + const configured = config.providers[providerName]; + if (configured?.requestPacing?.enabled && configured.requestPacing.keySelection === "round-robin" + && hasKeyPoolFailover(configured)) { + return applyRotatedTransport(providerName, routedProvider, allocateDistributedKey(providerName, configured, now), promptCacheKey); + } const committed = selectProactiveApiKey(config, providerName, now); if (!committed) return null; return applyRotatedTransport(providerName, routedProvider, committed, promptCacheKey); } +const dispatchKeyCursor = new Map(); +function allocateDistributedKey(providerName: string, provider: OcxProviderConfig, now: number): OcxProviderConfig { + const pool = (provider.apiKeyPool ?? []).filter((entry, index, all) => all.findIndex(other => other.key === entry.key) === index); + const previous = dispatchKeyCursor.get(providerName); + const start = pool.findIndex(entry => entry.id === previous); + for (let offset = 1; offset <= pool.length; offset++) { + const entry = pool[(start + offset) % pool.length]!; + if (isKeyInCooldown(providerName, entry.id, now)) continue; + dispatchKeyCursor.set(providerName, entry.id); + return { ...provider, apiKey: entry.key, _apiKeyAttempt: { + entryId: entry.id, reference: entry.key, revision: provider.apiKeySelectionRevision, distributed: true, + } }; + } + const next = Math.min(...pool.map(entry => getKeyCooldownUntil(providerName, entry.id, now) ?? now + 1000)); + throw new RequestPacingQueueOverloadError(providerName, "keys_exhausted", Math.max(1, Math.ceil((next - now) / 1000))); +} + /** * Normalize a provider's `retryOn429` policy. An explicit object always wins (including * `enabled: false` to opt out). When the knob is absent, the OpenCode Go destination @@ -699,6 +724,17 @@ function rotateKeyAfterFailure( if (!provider) return null; if (provider.authMode === "oauth" || provider.authMode === "forward") return null; + if (attemptedSelection?.distributed) { + const failed = provider.apiKeyPool?.find(entry => entry.id === attemptedSelection.entryId && entry.key === attemptedSelection.reference); + if (!failed || provider.apiKeySelectionRevision !== attemptedSelection.revision) return null; + const delay = failureStatus === 401 ? MAX_COOLDOWN_MS : quotaResetAt !== undefined + ? Math.max(1, quotaResetAt - now) : parseRetryAfterMs(retryAfterHeader, now) ?? DEFAULT_COOLDOWN_MS; + for (const entry of provider.apiKeyPool ?? []) if (entry.key === failed.key) keyCooldowns.set(cooldownKey(providerName, entry.id), { cooldownUntil: now + delay }); + sweepExpiredOnWrite(now); + if (!(provider.apiKeyPool ?? []).some(entry => !isKeyInCooldown(providerName, entry.id, now))) return null; + return allocateDistributedKey(providerName, provider, now); + } + const failedKey = attemptedSelection?.reference ?? attemptedKey ?? provider.apiKey; type Rotation = | { failedId?: string; candidateId?: string } diff --git a/src/providers/request-pacing.ts b/src/providers/request-pacing.ts index 1320d4cb35a..4a29bd175e0 100644 --- a/src/providers/request-pacing.ts +++ b/src/providers/request-pacing.ts @@ -1,4 +1,5 @@ import type { OcxProviderConfig, RequestPacingRule } from "../types"; +import { createHash } from "node:crypto"; import type { GenerationContext } from "../lib/state-store-sweeper"; import { carryReplayRefusal, isNonReplayableResponse, markResponseNonReplayable } from "../lib/upstream-retry"; import { redactSecretString } from "../lib/redact"; @@ -33,7 +34,7 @@ export const REQUEST_PACING_CONCURRENCY_RETRY_AFTER_SECONDS = 5; let maxQueueDepth = REQUEST_PACING_MAX_QUEUE_DEPTH; let maxQueueAgeMs = REQUEST_PACING_MAX_QUEUE_AGE_MS; -export type RequestPacingQueueOverloadReason = "queue_full" | "queue_expired"; +export type RequestPacingQueueOverloadReason = "queue_full" | "queue_expired" | "keys_exhausted"; export class RequestPacingQueueOverloadError extends Error { constructor( @@ -41,7 +42,7 @@ export class RequestPacingQueueOverloadError extends Error { public readonly reason: RequestPacingQueueOverloadReason, public readonly retryAfterSeconds: number, ) { - super(reason === "queue_full" + super(reason === "keys_exhausted" ? "No eligible API key remains; wait for the provider cooldown before retrying" : reason === "queue_full" ? `request pacing queue for provider '${providerName}' is full` : `request pacing queue for provider '${providerName}' exceeded the maximum queued age`); this.name = "RequestPacingQueueOverloadError"; @@ -56,6 +57,10 @@ export class RequestPacingProviderRemovedError extends Error { } interface Waiter { + keyLease: boolean; + keyId?: string; + keyIntervalMs: number; + keyMaxConcurrent: number; modelId?: string; providerIntervalMs: number; modelIntervalMs: number; @@ -69,6 +74,9 @@ interface Waiter { } interface ProviderPacer { + keyNextStartAt: Map; + keyInFlight: Map; + keyStarts: Map; queue: Waiter[]; providerNextStartAt: number; modelNextStartAt: Map; @@ -132,6 +140,7 @@ export interface ProviderRequestPacingStatus { inFlight?: number; lastStartedAt?: number; lastModelId?: string; + keys?: Array<{ id: string; starts: number; inFlight: number; queued: number; nextSlotInMs: number }>; } const pacers = new Map(); @@ -159,6 +168,13 @@ function normalizedInterval(rule: RequestPacingRule | undefined): number { return Math.max(rpmInterval, fixedInterval); } +function keyBucket(provider: OcxProviderConfig): string | undefined { + if (provider.authMode === "oauth" || provider.authMode === "forward" || !provider.apiKey) return; + if (!provider.requestPacing?.enabled || (!provider.requestPacing.keys && provider.requestPacing.keySelection !== "round-robin")) return; + // Internal only: duplicate references share one bucket; never serialize a key or digest. + return createHash("sha256").update(provider._apiKeyAttempt?.reference ?? provider.apiKey).digest("hex"); +} + function normalizedMaxConcurrent(rule: RequestPacingRule | undefined): number { return typeof rule?.maxConcurrentRequests === "number" && rule.maxConcurrentRequests > 0 // Floor so runtime-injected configs that bypass the integer schema cannot admit @@ -172,10 +188,14 @@ export function requestPacingIntervalMs(provider: OcxProviderConfig, modelId?: s const policy = provider.requestPacing; if (!policy?.enabled) return 0; const override = modelId ? policy.models?.[modelId] : undefined; - return Math.max(normalizedInterval(policy), normalizedInterval(override)); + return Math.max(normalizedInterval(policy), normalizedInterval(override), keyBucket(provider) ? normalizedInterval(policy.keys) : 0); } function requestPacingIntervals(provider: OcxProviderConfig, modelId?: string): { + keyLease: boolean; + keyId?: string; + keyIntervalMs: number; + keyMaxConcurrent: number; providerIntervalMs: number; modelIntervalMs: number; providerMaxConcurrent: number; @@ -183,9 +203,14 @@ function requestPacingIntervals(provider: OcxProviderConfig, modelId?: string): } { const policy = provider.requestPacing; if (!policy?.enabled) { - return { providerIntervalMs: 0, modelIntervalMs: 0, providerMaxConcurrent: 0, modelMaxConcurrent: 0 }; + return { keyLease: false, keyIntervalMs: 0, keyMaxConcurrent: 0, providerIntervalMs: 0, modelIntervalMs: 0, providerMaxConcurrent: 0, modelMaxConcurrent: 0 }; } + const keyId = keyBucket(provider); return { + keyLease: true, + keyId, + keyIntervalMs: keyId ? normalizedInterval(policy.keys) : 0, + keyMaxConcurrent: keyId ? normalizedMaxConcurrent(policy.keys) : 0, providerIntervalMs: normalizedInterval(policy), modelIntervalMs: modelId ? normalizedInterval(policy.models?.[modelId]) : 0, providerMaxConcurrent: normalizedMaxConcurrent(policy), @@ -203,13 +228,14 @@ function requestPacingIntervals(provider: OcxProviderConfig, modelId?: string): */ export function requestPacingMaxConcurrentRequests(provider: OcxProviderConfig, modelId?: string): number { const limits = requestPacingIntervals(provider, modelId); - return Math.max(limits.providerMaxConcurrent, limits.modelMaxConcurrent); + return Math.max(limits.providerMaxConcurrent, limits.modelMaxConcurrent, limits.keyMaxConcurrent); } -function waiterReadyAt(state: ProviderPacer, modelId: string | undefined): number { +function waiterReadyAt(state: ProviderPacer, modelId: string | undefined, keyId?: string): number { return Math.max( state.providerNextStartAt, modelId ? (state.modelNextStartAt.get(modelId) ?? 0) : 0, + keyId ? (state.keyNextStartAt.get(keyId) ?? 0) : 0, ); } @@ -218,8 +244,9 @@ function pacingRetryAfterSeconds( modelId: string | undefined, now: number, concurrencyCapped = false, + keyId?: string, ): number { - const intervalSeconds = Math.max(1, Math.ceil(Math.max(0, waiterReadyAt(state, modelId) - now) / 1000)); + const intervalSeconds = Math.max(1, Math.ceil(Math.max(0, waiterReadyAt(state, modelId, keyId) - now) / 1000)); // A waiter blocked by an in-flight lease has no interval to report: when the wait ends // in refusal, the honest answer is that the cap is saturated, not "retry every second". return concurrencyCapped @@ -234,7 +261,7 @@ function makeProviderRequestSlot( ): BodyTrackableProviderRequestSlot { let released = false; let bodyTracked = false; - const leased = waiter.providerMaxConcurrent > 0 || waiter.modelMaxConcurrent > 0; + const leased = (!!waiter.keyId && waiter.keyLease) || waiter.providerMaxConcurrent > 0 || waiter.modelMaxConcurrent > 0; const slot: BodyTrackableProviderRequestSlot = { providerName, leased, @@ -252,6 +279,7 @@ function makeProviderRequestSlot( released = true; waiter.signal?.removeEventListener("abort", releaseOnAbort); if (waiter.providerMaxConcurrent > 0 && state.providerInFlight > 0) state.providerInFlight -= 1; + if (waiter.keyId && waiter.keyLease) state.keyInFlight.set(waiter.keyId, Math.max(0, (state.keyInFlight.get(waiter.keyId) ?? 0) - 1)); if (waiter.modelId && waiter.modelMaxConcurrent > 0) { const current = state.modelInFlight.get(waiter.modelId) ?? 0; if (current <= 1) state.modelInFlight.delete(waiter.modelId); @@ -287,7 +315,8 @@ function rejectExpiredWaiters(providerName: string, state: ProviderPacer, now: n state, waiter.modelId, now, - waiter.providerMaxConcurrent > 0 || waiter.modelMaxConcurrent > 0, + waiter.providerMaxConcurrent > 0 || waiter.modelMaxConcurrent > 0 || waiter.keyMaxConcurrent > 0, + waiter.keyId, ), )); } @@ -321,17 +350,18 @@ function runQueue(providerName: string, state: ProviderPacer): void { const providerReadyAt = Math.max(now, state.providerNextStartAt); const waiterIndex = state.queue.findIndex(waiter => { + if (waiter.keyId && waiter.keyMaxConcurrent > 0 && (state.keyInFlight.get(waiter.keyId) ?? 0) >= waiter.keyMaxConcurrent) return false; if (waiter.providerMaxConcurrent > 0 && state.providerInFlight >= waiter.providerMaxConcurrent) return false; if (waiter.modelId && waiter.modelMaxConcurrent > 0 && (state.modelInFlight.get(waiter.modelId) ?? 0) >= waiter.modelMaxConcurrent) return false; const modelReadyAt = waiter.modelId ? (state.modelNextStartAt.get(waiter.modelId) ?? 0) : 0; - return Math.max(providerReadyAt, modelReadyAt) <= now; + return Math.max(providerReadyAt, modelReadyAt, waiter.keyId ? state.keyNextStartAt.get(waiter.keyId) ?? 0 : 0) <= now; }); if (waiterIndex < 0) { let earliestAt = Number.POSITIVE_INFINITY; for (const waiter of state.queue) { const modelReadyAt = waiter.modelId ? (state.modelNextStartAt.get(waiter.modelId) ?? 0) : 0; - const readyAt = Math.max(providerReadyAt, modelReadyAt); + const readyAt = Math.max(providerReadyAt, modelReadyAt, waiter.keyId ? state.keyNextStartAt.get(waiter.keyId) ?? 0 : 0); const expiresAt = waiter.queuedAt + maxQueueAgeMs; // A waiter whose start time already passed is blocked on an in-flight lease; its release // re-runs this queue through a microtask, so the timer only needs its expiry backstop. @@ -356,6 +386,11 @@ function runQueue(providerName: string, state: ProviderPacer): void { state.lastStartedAt = startedAt; state.lastModelId = waiter.modelId; state.providerNextStartAt = startedAt + waiter.providerIntervalMs; + if (waiter.keyId) { + state.keyNextStartAt.set(waiter.keyId, startedAt + waiter.keyIntervalMs); + if (waiter.keyLease) state.keyInFlight.set(waiter.keyId, (state.keyInFlight.get(waiter.keyId) ?? 0) + 1); + state.keyStarts.set(waiter.keyId, (state.keyStarts.get(waiter.keyId) ?? 0) + 1); + } if (waiter.modelId && waiter.modelIntervalMs > 0) { state.modelNextStartAt.set(waiter.modelId, startedAt + waiter.modelIntervalMs); } @@ -380,22 +415,34 @@ export async function waitForProviderRequestSlot( // by interval only, or a follow-up would queue behind the lease its own turn holds. const concurrency = options?.concurrency !== false; const waiterIntervals = concurrency ? intervals : { + keyId: intervals.keyId, keyLease: false, keyIntervalMs: intervals.keyIntervalMs, keyMaxConcurrent: 0, providerIntervalMs: intervals.providerIntervalMs, modelIntervalMs: intervals.modelIntervalMs, providerMaxConcurrent: 0, modelMaxConcurrent: 0, }; const paced = Math.max(intervals.providerIntervalMs, intervals.modelIntervalMs) > 0 + || intervals.keyId !== undefined || waiterIntervals.providerMaxConcurrent > 0 || waiterIntervals.modelMaxConcurrent > 0; if (!paced) return inertProviderRequestSlot; if (signal?.aborted) throw abortReason(signal); const state = pacers.get(providerName) ?? { + keyNextStartAt: new Map(), keyInFlight: new Map(), keyStarts: new Map(), queue: [], providerNextStartAt: 0, modelNextStartAt: new Map(), providerInFlight: 0, modelInFlight: new Map(), }; pacers.set(providerName, state); + // Retire removed credentials only after their queued work, leases and spacing debt end. + const currentKeys = new Set((provider.apiKeyPool ?? []).map(entry => + keyBucket({ ...provider, apiKey: entry.key, _apiKeyAttempt: undefined }))); + currentKeys.add(intervals.keyId); + for (const key of state.keyStarts.keys()) { + if (currentKeys.has(key) || (state.keyInFlight.get(key) ?? 0) > 0 + || (state.keyNextStartAt.get(key) ?? 0) > runtime.now() || state.queue.some(waiter => waiter.keyId === key)) continue; + state.keyStarts.delete(key); state.keyInFlight.delete(key); state.keyNextStartAt.delete(key); + } // Give already-eligible or expired waiters a chance to leave before applying the // admission bound to the newest request. This preserves FIFO-ish fairness while @@ -413,7 +460,8 @@ export async function waitForProviderRequestSlot( state, modelId, runtime.now(), - waiterIntervals.providerMaxConcurrent > 0 || waiterIntervals.modelMaxConcurrent > 0, + waiterIntervals.providerMaxConcurrent > 0 || waiterIntervals.modelMaxConcurrent > 0 || waiterIntervals.keyMaxConcurrent > 0, + waiterIntervals.keyId, ), ); } @@ -648,24 +696,35 @@ export function providerRequestPacingStatus( if (state && state.queue.length > 0) { let earliestQueuedSlotAt = Number.POSITIVE_INFINITY; for (const waiter of state.queue) { - earliestQueuedSlotAt = Math.min(earliestQueuedSlotAt, waiterReadyAt(state, waiter.modelId)); + earliestQueuedSlotAt = Math.min(earliestQueuedSlotAt, waiterReadyAt(state, waiter.modelId, waiter.keyId)); } if (Number.isFinite(earliestQueuedSlotAt)) nextSlotAt = earliestQueuedSlotAt; } - const providerConcurrencyCap = requestPacingMaxConcurrentRequests(provider); + const providerConcurrencyCap = normalizedMaxConcurrent(provider.requestPacing); + const hasKeyAccounting = provider.requestPacing?.enabled === true && keyBucket(provider) !== undefined; // Gate on enabled so the status surface agrees with enforcement: a disabled requestPacing // block must not report in-flight leases alongside enabled: false. - const anyConcurrencyCap = provider.requestPacing?.enabled === true && (providerConcurrencyCap > 0 + const anyConcurrencyCap = provider.requestPacing?.enabled === true && (hasKeyAccounting || providerConcurrencyCap > 0 || Object.values(provider.requestPacing?.models ?? {}).some(rule => normalizedMaxConcurrent(rule) > 0)); return { provider: providerName, enabled: provider.requestPacing?.enabled === true, queued: state?.queue.length ?? 0, nextSlotInMs: Math.max(0, Math.ceil(nextSlotAt - now)), + ...(provider.requestPacing?.enabled && (provider.requestPacing.keys || provider.requestPacing.keySelection === "round-robin") ? { + keys: (provider.apiKeyPool ?? []).map(entry => { + const bucket = keyBucket({ ...provider, apiKey: entry.key, _apiKeyAttempt: undefined })!; + return { id: entry.id, starts: state?.keyStarts.get(bucket) ?? 0, inFlight: state?.keyInFlight.get(bucket) ?? 0, + queued: state?.queue.filter(waiter => waiter.keyId === bucket).length ?? 0, + nextSlotInMs: Math.max(0, Math.ceil((state?.keyNextStartAt.get(bucket) ?? 0) - now)) }; + }), + } : {}), // A provider cap counts every paced send in providerInFlight; model-only caps keep // providerInFlight at zero, so report the per-model sum for those providers instead. ...(anyConcurrencyCap ? { - inFlight: providerConcurrencyCap > 0 + inFlight: hasKeyAccounting + ? [...(state?.keyInFlight.values() ?? [])].reduce((sum, count) => sum + count, 0) + : providerConcurrencyCap > 0 ? state?.providerInFlight ?? 0 : [...(state?.modelInFlight.values() ?? [])].reduce((sum, count) => sum + count, 0), } : {}), diff --git a/src/server/chat-native.ts b/src/server/chat-native.ts index cebac32464c..75a9edfc3d9 100644 --- a/src/server/chat-native.ts +++ b/src/server/chat-native.ts @@ -57,6 +57,7 @@ import { transientRetryPolicyFor, } from "../providers/key-failover"; import { fastPolicyForModel } from "../providers/service-tier"; +import { RequestPacingQueueOverloadError } from "../providers/request-pacing"; import { stampApiKeyAccountLabel } from "../providers/label"; import { providerApiKeySelectionIsCurrent, resolveCurrentProviderApiKeyTransport } from "../providers/api-key-selection"; import { enrichOpenCodeZenFreeTierMessage } from "../providers/opencode-zen-rate-limit"; @@ -313,8 +314,16 @@ export async function runNativeChatAttempt( // assigned before the transport pin in core.ts. // Transport variant: the bare picker answers with the persisted row, which for a built-in // provider carries no adapter id or base URL until routedProviderConfig backfills it. - const proactiveKeyProvider = selectProactiveApiKeyTransport(config, route.providerName, route.provider); - if (proactiveKeyProvider) route.provider = proactiveKeyProvider; + try { + const proactiveKeyProvider = selectProactiveApiKeyTransport(config, route.providerName, route.provider); + if (proactiveKeyProvider) route.provider = proactiveKeyProvider; + } catch (error) { + cleanupAbort(); upstream.abort(); + if (!(error instanceof RequestPacingQueueOverloadError)) throw error; + const refusal = fail(429, error.message, "rate_limit_error"); + refusal.headers.set("Retry-After", String(error.retryAfterSeconds)); + return refusal; + } let activeProvider: OcxProviderConfig = route.provider; stampApiKeyAccountLabel(logCtx, route.providerName, activeProvider); const spendTracker = sendBudget ? undefined : attachRequestSpendTracker(req, logCtx); @@ -594,6 +603,11 @@ export async function runNativeChatAttempt( upstream.abort(); if (req.signal.aborted) return fail(499, "Client cancelled request", "client_cancelled"); const sendError = error instanceof UpstreamRetryEvidenceError ? error.cause : error; + if (sendError instanceof RequestPacingQueueOverloadError) { + const refusal = fail(429, sendError.message, "rate_limit_error"); + refusal.headers.set("Retry-After", String(sendError.retryAfterSeconds)); + return refusal; + } if (sendError instanceof SendBudgetExhaustedError) { // A decision this process made, answered as the Responses path answers it: 429, not 502. return fail(429, sendError.message, SEND_BUDGET_EXHAUSTED_CODE, SEND_BUDGET_EXHAUSTED_CODE); diff --git a/src/server/images.ts b/src/server/images.ts index d971538555c..e1b2723656a 100644 --- a/src/server/images.ts +++ b/src/server/images.ts @@ -30,6 +30,7 @@ import { sidecarEnter } from "../lib/sidecar-tracker"; import type { OcxConfig } from "../types"; import { resolveFirstUsableOpenAiSidecar, selectImagesProvider } from "../providers/openai-sidecar"; import { selectProactiveApiKeyTransport } from "../providers/key-failover"; +import { requestPacingOverloadResponse } from "./responses/pacing-overload"; import { getProviderRegistryEntry } from "../providers/registry"; import { readJsonRequestBody, resolveInboundBodyLimitBytes } from "./request-decompress"; import { ForwardAdmissionCredentialError, validateForwardAdmissionCredential } from "./auth-cors"; @@ -805,7 +806,13 @@ export async function handleImages( // // Transport variant: this branch reads `provider.baseUrl` and `provider.headers` to build // the URL and the request, and it comes back with the credential already resolved. - const warmKeyProvider = selectProactiveApiKeyTransport(config, providerName, candidates.keyed.provider); + let warmKeyProvider; + try { warmKeyProvider = selectProactiveApiKeyTransport(config, providerName, candidates.keyed.provider); } + catch (error) { + const refusal = requestPacingOverloadResponse(error); + if (refusal) return refusal; + throw error; + } const provider = warmKeyProvider ?? candidates.keyed.provider; // No fall back to the earlier snapshot once a pick has happened. The picker COMMITS its // choice before returning, so if the chosen reference will not resolve -- revoked keychain diff --git a/src/server/management/provider-routes.ts b/src/server/management/provider-routes.ts index ee1733d2b2b..bd936e62539 100644 --- a/src/server/management/provider-routes.ts +++ b/src/server/management/provider-routes.ts @@ -77,7 +77,7 @@ import { PROVIDER_QUOTA_MAX_AGE_MS, type ProviderRoutingQuota } from "../../prov import { cachedProviderQuotaIsExhausted } from "../../combos/resolve"; import { probeJevDecisionProvider } from "../../combos/jev"; import { comboDependsOnProviderRoute } from "./decision-model-validation"; -import { clearKeyCooldowns, forgetApiKeyRotationCursor } from "../../providers/key-failover"; +import { clearKeyCooldowns, forgetApiKeyRotationCursor, getKeyCooldownUntil } from "../../providers/key-failover"; import { providerRequestPacingStatus } from "../../providers/request-pacing"; import { CODEX_FORWARD_BASE_URL, isCanonicalOpenAiForwardProvider } from "../../providers/openai-tiers"; import { codexAccountNamespaceProviderCollisionError } from "../../codex/account-namespace-match"; @@ -896,17 +896,22 @@ export async function handleProviderRoutes(ctx: ManagementContext): Promise { + const status = providerRequestPacingStatus(name, provider); + if (status.keys) status.keys = status.keys.map(row => ({ ...row, cooldownUntil: getKeyCooldownUntil(name, row.id) })); + return status; + }; const name = url.searchParams.get("name")?.trim(); if (name) { if (!isValidProviderName(name) || !hasOwnProvider(config.providers, name)) { return jsonResponse({ error: "unknown provider" }, 404); } - return jsonResponse(providerRequestPacingStatus(name, config.providers[name]!)); + return jsonResponse(statusFor(name, config.providers[name]!)); } return jsonResponse(Object.fromEntries( Object.entries(config.providers).map(([providerName, provider]) => [ providerName, - providerRequestPacingStatus(providerName, provider), + statusFor(providerName, provider), ]), )); } diff --git a/src/server/messages-native.ts b/src/server/messages-native.ts index 58277a0a707..63d342098ba 100644 --- a/src/server/messages-native.ts +++ b/src/server/messages-native.ts @@ -62,6 +62,7 @@ import { UPSTREAM_RESET_REPLAY_REFUSED_CODE, } from "../lib/upstream-retry"; import { providerApiKeySelectionIsCurrent, resolveCurrentProviderApiKeyTransport } from "../providers/api-key-selection"; +import { RequestPacingQueueOverloadError } from "../providers/request-pacing"; import { hasKeyPoolFailover, rateLimitRetryDelayMs, @@ -433,8 +434,16 @@ export async function handleNativeMessages(options: HandleNativeMessagesOptions) } } else { // Same pre-dispatch key preference every direct send path applies (see chat-native.ts). - const proactiveKeyProvider = selectProactiveApiKeyTransport(config, route.providerName, route.provider); - if (proactiveKeyProvider) route.provider = proactiveKeyProvider; + try { + const proactiveKeyProvider = selectProactiveApiKeyTransport(config, route.providerName, route.provider); + if (proactiveKeyProvider) route.provider = proactiveKeyProvider; + } catch (error) { + cleanupAbort(); upstream.abort(); + if (!(error instanceof RequestPacingQueueOverloadError)) throw error; + const refusal = fail(429, error.message, "rate_limit_error"); + refusal.headers.set("Retry-After", String(error.retryAfterSeconds)); + return refusal; + } } let activeProvider: OcxProviderConfig = oauthBinding ? oauthProvider(oauthBinding) : route.provider; stampApiKeyAccountLabel(logCtx, route.providerName, activeProvider); @@ -720,6 +729,11 @@ export async function handleNativeMessages(options: HandleNativeMessagesOptions) upstream.abort(); if (req.signal.aborted) return fail(499, "Client cancelled request", "api_error"); const sendError = error instanceof UpstreamRetryEvidenceError ? error.cause : error; + if (sendError instanceof RequestPacingQueueOverloadError) { + const refusal = fail(429, sendError.message, "rate_limit_error"); + refusal.headers.set("Retry-After", String(sendError.retryAfterSeconds)); + return refusal; + } if (sendError instanceof SendBudgetExhaustedError) return fail(429, sendError.message, "send_budget_exhausted", "send_budget_exhausted"); if (sendError instanceof NativeMessagesSpendRefusal) { const refusal = workflowRefusalResponse("workflow-spend-exhausted", logCtx); diff --git a/src/server/responses/compact.ts b/src/server/responses/compact.ts index 5215b5a2796..a2aad9c026b 100644 --- a/src/server/responses/compact.ts +++ b/src/server/responses/compact.ts @@ -200,6 +200,7 @@ import { usesCodexForwardPoolAuth, } from "./core"; import { fetchWithHeaderTimeout, providerFetch, safeHostLabel, safeOriginLabel } from "./fetch-helpers"; +import { requestPacingOverloadResponse } from "./pacing-overload"; import { mapCodexAuthContextErrorToResponse, nativeMainRefreshFailureResponse } from "./codex-auth-error"; import { linkRequestSessionLane, sessionLaneIdFromRequest } from "../request-log-conversation"; import { recallComboForLane } from "./combo-session-recall"; @@ -905,7 +906,13 @@ export async function handleResponsesCompact( // codexAccountMode cases, and the picker returns null for them anyway. // Transport variant, for the same reason as the Responses core: the persisted row has no // registry backfills, and compactProvider is read for its transport fields below. - const warmKeyProvider = selectProactiveApiKeyTransport(config, route.providerName, compactProvider); + let warmKeyProvider; + try { warmKeyProvider = selectProactiveApiKeyTransport(config, route.providerName, compactProvider); } + catch (error) { + const refusal = requestPacingOverloadResponse(error); + if (refusal) return refusal; + throw error; + } if (warmKeyProvider?.apiKey) compactProvider = warmKeyProvider; headers.set("authorization", `Bearer ${resolveProviderApiKey(compactProvider.apiKey)}`); } diff --git a/src/types/provider.ts b/src/types/provider.ts index 7c310535b92..09016e8a4f9 100644 --- a/src/types/provider.ts +++ b/src/types/provider.ts @@ -14,6 +14,8 @@ export interface ProviderApiKeySelection { entryId?: string; reference?: string; revision?: string; + /** Request-local allocation; does not replace the saved active key. */ + distributed?: boolean; } export interface OpenRouterProviderRouting { @@ -190,6 +192,10 @@ export interface ProviderRequestPacingConfig extends RequestPacingRule { enabled: boolean; /** Exact upstream model-id overrides; other models inherit the provider rule. */ models?: Record; + /** Normal request allocation, separate from cooldown recovery ordering. */ + keySelection?: "active" | "round-robin"; + /** Additional uniform limits per pool credential, across all models. */ + keys?: RequestPacingRule; } export interface FastWire { diff --git a/structure/config.md b/structure/config.md index 86a27b4f835..771fb42ee96 100644 --- a/structure/config.md +++ b/structure/config.md @@ -97,7 +97,7 @@ A schema-invalid top-level JSON value is repairable only when it is a non-array | --- | --- | --- | | Listener | `port`, `hostname` | The listener owns the port; `runtime-port.json` reports where it actually landed. | | Routing | `defaultProvider`, `providers`, per-provider `selectedModels`, `combos` | Explicit `provider/model` wins over `defaultProvider`; combo dispatch uses the selected target's existing capability ladder and does not create a second catalog authority. For Kiro OAuth, the management API validates `providers.kiro.oauthAccountFailover.strategy` (`least-loaded`) and `maxConcurrentPerAccount` (1–100) only for Kiro; the cap persists, while active lease counts remain process-local. | -| Request pacing | `providers..requestPacing`, `requestPacing.models.` | Optional client-side request-start pacing supports interval limits and positive-integer `maxConcurrentRequests` caps. A provider or model rule may be concurrency-only; model entries target exact upstream IDs and can only add delay or narrow concurrency. | +| Request pacing | `providers..requestPacing`, `requestPacing.models.` | Optional client-side request-start pacing supports intervals and positive-integer concurrency caps. Provider, exact-model, and uniform per-credential-reference `keys` rules compose atomically. `keySelection` defaults to `active`; opt-in `round-robin` assigns healthy pool keys per request without persisting the active choice. Project quota, TPM and daily limits are not inferred. | | Compaction and memory routing | `compactionRouting.model`, optional `compactionRouting.reasoningEffort`, optional `compactionRouting.triggers`, optional `compactionRouting.sourceModels`; `memoryModels.extract`, `memoryModels.consolidation` | Explicit Codex compaction metadata whose `compaction.trigger` is one the block names activates a request-local override; `triggers` defaults to `["manual"]`. Optional source selectors restrict the override to exact incoming models or `provider/*`. See [Responses compaction](transports/responses-failover.md#compaction-routing-overrides). Invalid hand edits disable the block with a load warning without discarding providers; candidate writes reject invalid blocks. Each optional memory phase names a nonblank model and optional declared `reasoningEffort`; absence preserves the current route, candidate writes reject malformed values, and load degrades only the invalid phase with a warning. See [memory phase routing](transports/responses-failover.md#memory-phase-routing). | | Catalog | `disabledModels`, `customModels`, `modelCacheTtlMs`, `providerContextCaps`, `contextCapValue`, per-provider `modelDisplayNames`, `codexAccountNamespaces`, `codexAccountPickerEnabled` | Catalog state is derived; config only records intent. Exact provider model display names are durable display only overlays. The picker flag is an explicit visibility override, while selector mappings remain the durable exact-routing contract. | | Retained state | `appOwnedMemoryBudgetMb` | Process-wide eviction target for app-owned logs, caches, blobs, and continuation payloads. Default 256 MiB, valid 64..4096; pinned state may temporarily exceed the target, but every pin-capable store has a finite local cap and their documented aggregate stays below `APP_OWNED_WORST_CASE_PINNED_BYTES` (512 MiB). Neither value caps RSS or native runtime memory. | diff --git a/structure/data-planes/images.md b/structure/data-planes/images.md index b1f578757a9..eb69eeadf31 100644 --- a/structure/data-planes/images.md +++ b/structure/data-planes/images.md @@ -103,7 +103,7 @@ error rather than falling through to a separately billed keyed provider. The out has one selected Authorization value, validated before the non-idempotent upstream POST. The keyed path never enters `handleResponses`, so `src/server/images.ts` repeats `selectProactiveApiKeyTransport` inside the keyed branch and rebuilds Authorization from the -returned clone rather than the earlier snapshot. +returned clone rather than the earlier snapshot. Opt-in request-local round-robin uses that same selection seam and maps exhausted-key allocation to retryable 429. The standalone Images relay remains a single direct non-idempotent fetch outside the inference request-pacing executor; provider/key pacing limits do not cover that relay. A configured key's model and provider scope applies to whichever destination the request settles on: the ChatGPT forward account, the keyed provider, the xAI Imagine bridge, or the Antigravity diff --git a/structure/gui-and-management-api.md b/structure/gui-and-management-api.md index 46d4bae711a..a6a0432ea34 100644 --- a/structure/gui-and-management-api.md +++ b/structure/gui-and-management-api.md @@ -369,7 +369,7 @@ to fully quit and reopen Desktop after selecting the picker profile. Provider writes must not round-trip masked API keys as real secrets. Dashboard actions that change model visibility or subagent selection should trigger catalog/cache sync behavior through the server -path that owns it. +path that owns it. Provider request-pacing settings expose independent provider and uniform key limits plus opt-in request-local round-robin selection. `GET /api/provider-request-pacing` returns key handles, starts, in-flight counts, queue counts, next spacing slots and known cooldowns, never credential values or internal bucket hashes. Saving these rules uses the existing requestPacing-only PATCH path. ### Fast selector rows setting diff --git a/structure/providers-and-adapters.md b/structure/providers-and-adapters.md index 90b13c2640f..5e45abef6d3 100644 --- a/structure/providers-and-adapters.md +++ b/structure/providers-and-adapters.md @@ -288,7 +288,7 @@ Cursor, one lease spans the whole turn: RunSSE and BidiAppend may overlap inside turns wait at the cap. Follow-up sends still obey start intervals. The turn owner returns its lease after `runTurn` settles, so RunSSE body completion cannot admit another turn early. A capped canonical Codex WebSocket turn uses HTTP/SSE because the socket has no response-body lifecycle. -Capacity waits use the same bounded queue and retryable queue-overload errors as interval waits. +Capacity waits use the same bounded queue and retryable queue-overload errors as interval waits. Optional `requestPacing.keys` adds a shared cross-model limit for each credential reference; duplicate references share capacity, but distinct references resolving to one upstream project do not imply separate quota. `keySelection: round-robin` assigns request-local healthy keys without mutating persisted selection, skips known cooldowns, and refuses exhausted pools with retryable 429. Provider, model and key gates reserve atomically; key counters and the allocation cursor are process-local, not token or billing balances. OAuth and forwarded credentials retain their existing selection. Command Code effort defaults in `src/providers/command-code-efforts.ts` combine public-profile facts with the live API measurements from #5096. Both presets share the exact per-model rows; diff --git a/structure/transports/responses-failover.md b/structure/transports/responses-failover.md index 09d6ae67677..30bcf7233d6 100644 --- a/structure/transports/responses-failover.md +++ b/structure/transports/responses-failover.md @@ -156,7 +156,7 @@ The shared Responses path follows the [bounded multipart recovery contract](../s ## Upstream key attempt accounting -Key identity is sealed at the guarded physical dispatch after queued selections are rebuilt. +Key identity is sealed at the guarded physical dispatch after queued selections are rebuilt. Distributed pool selections retain their request-local entry and revision rather than adopting another request's active key; a queued per-key permit cannot authorize a different credential after manual selection drift. Raw adapter terminal usage is recorded before continuation, search, or image loops merge it; repeated parsing of one physical response does not count it twice. Key changes preserve the previous attempt while retaining the active attempt object shared by streaming/combo callbacks. diff --git a/tests/adapters/key-failover.test.ts b/tests/adapters/key-failover.test.ts index e6ccbff9fdf..f09564c3d7f 100644 --- a/tests/adapters/key-failover.test.ts +++ b/tests/adapters/key-failover.test.ts @@ -3,6 +3,7 @@ import { mkdtempSync, unlinkSync, writeFileSync } from "node:fs"; import { tmpdir } from "node:os"; import { join } from "node:path"; import { createOpenAIChatAdapter } from "../../src/adapters/openai-chat"; +import { providerFetch } from "../../src/server/responses/fetch-helpers"; import { getConfigPath, loadConfig, @@ -34,6 +35,7 @@ import { subscribeAccountSelections } from "../../src/lib/account-selection-even import { providerManagementConfigError, safeConfigDTO } from "../../src/server/auth-cors"; import type { OcxConfig, OcxParsedRequest, OcxProviderConfig } from "../../src/types"; import { removeTreeWithRetry } from "../helpers/remove-tree"; +import { providerApiKeySelectionIsCurrent, resolveCurrentProviderApiKeyTransport } from "../../src/providers/api-key-selection"; let home: string; const previousHome = process.env.OPENCODEX_HOME; @@ -463,6 +465,64 @@ describe("rotateKeyOn401", () => { describe("proactive key selection", () => { const now = 2_000_000; + test("request-local round-robin keeps parallel keys stable without changing saved selection", async () => { + clearKeyCooldowns("distributed"); + const config = makeConfig({ apiKey: "key-alpha-000111222333", apiKeyPool: pool3(), + requestPacing: { enabled: true, keySelection: "round-robin" } }); + const row = config.providers.p!; + config.providers.distributed = row; + const selected = Array.from({ length: 6 }, () => selectProactiveApiKeyTransport(config, "distributed", routedProviderConfig("distributed", row), undefined, now)!); + expect(selected.map(provider => provider._apiKeyAttempt?.entryId)).toEqual(["k1", "k2", "k3", "k1", "k2", "k3"]); + expect(config.providers.distributed!.apiKey).toBe(row.apiKey); + expect(loadConfig().providers.p!.apiKey).toBe(row.apiKey); + for (const provider of selected) { + expect(providerApiKeySelectionIsCurrent(config, "distributed", provider)).toBe(true); + expect(resolveCurrentProviderApiKeyTransport(config, "distributed", provider)?.apiKey).toBe(provider.apiKey); + } + const sent: string[] = []; + await Promise.all(selected.map(async provider => { + const transport = resolveCurrentProviderApiKeyTransport(config, "distributed", provider)!; + transport.fetch = Object.assign(async (_input: unknown, init?: RequestInit) => { + sent.push(new Headers(init?.headers).get("authorization")!); + return new Response("ok"); + }, { preconnect() {} }) as typeof fetch; + const parsed: OcxParsedRequest = { modelId: "fake-model", context: { messages: [{ role: "user", content: "hi", timestamp: 0 }] }, stream: false, options: {} }; + const request = createOpenAIChatAdapter(transport).buildRequest(parsed); + const response = await providerFetch(transport, undefined, { providerName: "distributed" })(request.url, { + method: "POST", headers: request.headers, body: request.body, + }); + await response.text(); + })); + expect(sent).toEqual(selected.map(provider => `Bearer ${provider.apiKey}`)); + row.apiKeySelectionRevision = "manual-edit"; + expect(providerApiKeySelectionIsCurrent(config, "distributed", selected[0]!)).toBe(false); + expect(resolveCurrentProviderApiKeyTransport(config, "distributed", selected[0]!)).toBeNull(); + }); + + test("distributed 429 and 401 exclude only attempted keys and exhaustion refuses allocation", () => { + const config = makeConfig({ apiKey: "key-alpha-000111222333", apiKeyPool: pool3(), + requestPacing: { enabled: true, keySelection: "round-robin" } }); + const row = config.providers.p!; + config.providers.distributedFailure = row; + clearKeyCooldowns("distributedFailure"); + let transport = selectProactiveApiKeyTransport(config, "distributedFailure", routedProviderConfig("distributedFailure", row), undefined, now)!; + transport = rotateProviderTransportOn429(config, "distributedFailure", transport, { retryAfter: "60", now })!; + expect(getKeyCooldownUntil("distributedFailure", "k1", now)).toBe(now + 60_000); + transport = rotateProviderTransportOn401(config, "distributedFailure", transport, { now })!; + expect(rotateProviderTransportOn429(config, "distributedFailure", transport, { now })).toBeNull(); + expect(() => selectProactiveApiKeyTransport(config, "distributedFailure", routedProviderConfig("distributedFailure", row), undefined, now)).toThrow("No eligible API key"); + expect(row.apiKey).toBe("key-alpha-000111222333"); + }); + + test("an active-key permit cannot dispatch a manually replaced key under per-key limits", () => { + const config = makeConfig({ apiKey: "key-alpha-000111222333", apiKeyPool: pool3(), + requestPacing: { enabled: true, keys: { maxConcurrentRequests: 1 } } }); + const transport = routedProviderConfig("p", config.providers.p!); + expect(selectProactiveApiKeyTransport(config, "p", transport, undefined, now)).toBeNull(); + config.providers.p!.apiKey = "key-beta-444555666777"; + expect(resolveCurrentProviderApiKeyTransport(config, "p", transport)).toBeNull(); + }); + test("does nothing without a configured strategy", () => { const config = makeConfig({ apiKey: "key-alpha-000111222333", apiKeyPool: pool3() }); forgetApiKeyRotationCursor("p"); diff --git a/tests/claude-integration/messages-native.test.ts b/tests/claude-integration/messages-native.test.ts index c6b46ea7bd7..583f21667e7 100644 --- a/tests/claude-integration/messages-native.test.ts +++ b/tests/claude-integration/messages-native.test.ts @@ -10,7 +10,7 @@ import { mkdtempSync, readFileSync } from "node:fs"; import { tmpdir } from "node:os"; import { join } from "node:path"; import { saveConfig } from "../../src/config"; -import { clearKeyCooldowns } from "../../src/providers/key-failover"; +import { clearKeyCooldowns, forgetApiKeyRotationCursor } from "../../src/providers/key-failover"; import { estimateClaudeRequestTokens, handleClaudeCountTokens, handleClaudeMessages } from "../../src/server/claude-messages"; import { getRequestLogEntries } from "../../src/server/request-log"; import type { OcxConfig } from "../../src/types"; @@ -363,6 +363,30 @@ describe("managed native Messages", () => { expect(seen[1]!.body).toEqual(seen[0]!.body); }); + test("round-robin native sends preserve per-request keys and the saved active key", async () => { + const config = fixtureConfig(startUpstream(), { pool: true }); + config.providers.anth!.requestPacing = { enabled: true, keySelection: "round-robin", keys: { maxConcurrentRequests: 1 } }; + forgetApiKeyRotationCursor("anth"); saveConfig(config); + const results = await Promise.all(Array.from({ length: 4 }, () => send(config, { ...SOURCE_BODY, stream: false }))); + expect(results.map(result => result.response.status)).toEqual([200, 200, 200, 200]); + expect(seen.map(entry => entry.headers.get("x-api-key")).sort()).toEqual([ + "fixture-key-alpha", "fixture-key-alpha", "fixture-key-beta", "fixture-key-beta", + ]); + expect(config.providers.anth!.apiKey).toBe("fixture-key-alpha"); + }); + + test("an exhausted distributed pool refuses the next native request without an upstream send", async () => { + const config = fixtureConfig(startUpstream(() => Response.json({ type: "error", error: { type: "rate_limit_error", message: "fixture cooldown" } }, + { status: 429, headers: { "retry-after": "30" } })), { pool: true }); + config.providers.anth!.requestPacing = { enabled: true, keySelection: "round-robin" }; + forgetApiKeyRotationCursor("anth"); saveConfig(config); + expect((await send(config, { ...SOURCE_BODY, stream: false })).response.status).toBe(429); + expect(seen).toHaveLength(2); + const next = await send(config, { ...SOURCE_BODY, stream: false }); + expect(next.response.status).toBe(429); expect(seen).toHaveLength(2); + expect(Number(next.response.headers.get("retry-after"))).toBeGreaterThan(0); + }); + test("a 429 on the first pooled key fails over to the next key", async () => { const config = fixtureConfig(startUpstream((entry, index) => index === 0 ? Response.json({ type: "error", error: { type: "rate_limit_error", message: "slow down" } }, diff --git a/tests/server/management-provider-request-pacing.test.ts b/tests/server/management-provider-request-pacing.test.ts index f42b0e3a722..1a99bed4d78 100644 --- a/tests/server/management-provider-request-pacing.test.ts +++ b/tests/server/management-provider-request-pacing.test.ts @@ -36,6 +36,7 @@ describe("provider management request pacing", () => { adapter: "openai-chat", baseUrl: "https://integrate.api.nvidia.com/v1", apiKey: "sk-nvidia", + apiKeyPool: [{ id: "fixture-1", key: "sk-nvidia" }, { id: "fixture-2", key: "fixture-secondary" }], }, }, }; @@ -52,6 +53,8 @@ describe("provider management request pacing", () => { requestsPerMinute: 38, minIntervalMs: 1_600, maxConcurrentRequests: 4, + keySelection: "round-robin", + keys: { requestsPerMinute: 15, maxConcurrentRequests: 1 }, models: { "deepseek-ai/deepseek-v4-flash-0731": { requestsPerMinute: 10 } }, }; @@ -68,7 +71,11 @@ describe("provider management request pacing", () => { const providers = await request("/api/providers"); expect((await providers?.json()).find((row: { name: string }) => row.name === "nvidia").requestPacing).toEqual(policy); const status = await request("/api/provider-request-pacing?name=nvidia"); - expect(await status?.json()).toMatchObject({ provider: "nvidia", enabled: true, queued: 0, nextSlotInMs: 0 }); + const statusBody = await status?.json(); + expect(statusBody).toMatchObject({ provider: "nvidia", enabled: true, queued: 0, nextSlotInMs: 0 }); + expect(statusBody.keys.map((row: { id: string }) => row.id)).toEqual(["fixture-1", "fixture-2"]); + expect(JSON.stringify(statusBody)).not.toContain("sk-nvidia"); + expect(JSON.stringify(statusBody)).not.toContain("fixture-secondary"); const invalid = await request("/api/providers?name=nvidia", { method: "PATCH", @@ -78,6 +85,16 @@ describe("provider management request pacing", () => { expect(invalid?.status).toBe(400); expect(liveConfig.providers.nvidia?.requestPacing).toEqual(policy); + for (const rule of [{ keySelection: "random" }, { keys: {} }, { keys: { requestsPerMinute: 0 } }, + { keys: { maxConcurrentRequests: 1.5 } }]) { + const rejected = await request("/api/providers?name=nvidia", { + method: "PATCH", headers: { "content-type": "application/json" }, + body: JSON.stringify({ requestPacing: { enabled: true, ...rule } }), + }); + expect(rejected?.status).toBe(400); + expect(liveConfig.providers.nvidia?.requestPacing).toEqual(policy); + } + const timerOverflow = await request("/api/providers?name=nvidia", { method: "PATCH", headers: { "content-type": "application/json" }, diff --git a/tests/usage/request-pacing.test.ts b/tests/usage/request-pacing.test.ts index 5183100b743..b6b180e6f51 100644 --- a/tests/usage/request-pacing.test.ts +++ b/tests/usage/request-pacing.test.ts @@ -61,6 +61,70 @@ function fakePacingClock(): { } describe("requestPacingIntervalMs", () => { + test("per-key caps admit siblings while provider capacity remains an upper bound", async () => { + const clock = fakePacingClock(); + setProviderRequestPacingRuntimeForTest(clock.runtime); + const configured = provider({ enabled: true, maxConcurrentRequests: 2, keys: { maxConcurrentRequests: 1, requestsPerMinute: 15 } }); + configured.apiKeyPool = [{ id: "one", key: "fake-one" }, { id: "two", key: "fake-two" }]; + const one = { ...configured, apiKey: "fake-one" }; + const two = { ...configured, apiKey: "fake-two" }; + const first = await waitForProviderRequestSlot("key-cap", one); + let repeatedStarted = false; + const repeated = waitForProviderRequestSlot("key-cap", one).then(slot => { repeatedStarted = true; return slot; }); + const sibling = await waitForProviderRequestSlot("key-cap", two); + expect(repeatedStarted).toBe(false); + expect(providerRequestPacingStatus("key-cap", configured).keys?.map(row => row.inFlight)).toEqual([1, 1]); + first.release(); + clock.advanceBy(3999); + expect(repeatedStarted).toBe(false); + clock.advanceBy(1); + const next = await repeated; + expect(repeatedStarted).toBe(true); + expect(providerRequestPacingStatus("key-cap", configured).keys?.map(row => row.starts)).toEqual([2, 1]); + next.release(); sibling.release(); + expect(providerRequestPacingStatus("key-cap", configured).keys?.map(row => row.inFlight)).toEqual([0, 0]); + expect(JSON.stringify(providerRequestPacingStatus("key-cap", configured))).not.toContain("fake-"); + }); + + test("key-only pacing obeys body cancellation and turn follow-up intervals without a second lease", async () => { + const clock = fakePacingClock(); + setProviderRequestPacingRuntimeForTest(clock.runtime); + const configured = { ...provider({ enabled: true, keys: { maxConcurrentRequests: 1, minIntervalMs: 100 } }), apiKey: "fake", apiKeyPool: [{ id: "id", key: "fake" }] }; + const first = await waitForProviderRequestSlot("key-only", configured); + const followUp = waitForProviderRequestSlot("key-only", configured, undefined, undefined, { concurrency: false }); + clock.advanceBy(100); + const follow = await followUp; + expect(follow.leased).toBe(false); + expect(providerRequestPacingStatus("key-only", configured).keys?.[0]?.inFlight).toBe(1); + const controller = new AbortController(); + const queued = waitForProviderRequestSlot("key-only", configured, undefined, controller.signal); + controller.abort(); + await expect(queued).rejects.toThrow(); + first.release(); + }); + + test("provider cap also blocks an otherwise eligible different key", async () => { + const clock = fakePacingClock(); setProviderRequestPacingRuntimeForTest(clock.runtime); + const configured = provider({ enabled: true, maxConcurrentRequests: 1, keys: { maxConcurrentRequests: 1 } }); + const first = await waitForProviderRequestSlot("global-key-cap", { ...configured, apiKey: "fake-one" }); + let started = false; + const second = waitForProviderRequestSlot("global-key-cap", { ...configured, apiKey: "fake-two" }).then(slot => { started = true; return slot; }); + await Promise.resolve(); expect(started).toBe(false); + first.release(); const slot = await second; expect(started).toBe(true); slot.release(); + }); + + test("duplicate key references share capacity and diagnostics count key-only leases", async () => { + const configured = { ...provider({ enabled: true, keys: { maxConcurrentRequests: 1 } }), apiKey: "fake-one", + apiKeyPool: [{ id: "one", key: "fake-one" }, { id: "duplicate", key: "fake-one" }] }; + const first = await waitForProviderRequestSlot("duplicate-key", configured); + expect(providerRequestPacingStatus("duplicate-key", configured).inFlight).toBe(1); + const abort = new AbortController(); + const second = waitForProviderRequestSlot("duplicate-key", configured, undefined, abort.signal); + expect(providerRequestPacingStatus("duplicate-key", configured).keys?.map(row => row.queued)).toEqual([1, 1]); + abort.abort(); await expect(second).rejects.toThrow(); first.release(); + expect(providerRequestPacingStatus("duplicate-key", configured).inFlight).toBe(0); + }); + test("uses the slower of provider RPM, provider delay, and model override", () => { const configured = provider({ enabled: true, From fd28e0afcaf80f94c8a6352f5fdd65e111a1f438 Mon Sep 17 00:00:00 2001 From: GSL-R <174104911+GSL-R@users.noreply.github.com> Date: Mon, 12 Oct 2026 02:50:41 +0900 Subject: [PATCH 02/17] fix(providers): cool rejected distributed keys and readmit refreshed dispatches --- .../provider-workspace/ProviderSettings.tsx | 2 +- gui/src/i18n/de.ts | 1 + gui/src/i18n/en.ts | 1 + gui/src/i18n/fr.ts | 1 + gui/src/i18n/ja.ts | 1 + gui/src/i18n/ko.ts | 1 + gui/src/i18n/pt.ts | 3 +- gui/src/i18n/ru.ts | 3 +- gui/src/i18n/tr.ts | 1 + gui/src/i18n/vi.ts | 1 + gui/src/i18n/zh-TW.ts | 1 + gui/src/i18n/zh.ts | 1 + .../provider-settings-request-pacing.test.tsx | 1 + src/providers/key-failover.ts | 25 ++++- src/server/chat-native.ts | 12 +- src/server/images.ts | 5 +- src/server/responses/compact.ts | 3 +- src/server/responses/fetch-helpers.ts | 61 ++++++++--- src/server/responses/request-transport.ts | 17 ++- structure/transports/responses-failover.md | 2 +- tests/responses/chat-native-combo.test.ts | 23 +++- tests/server/distributed-key-401.test.ts | 103 ++++++++++++++++++ tests/server/server-key-failover-e2e.test.ts | 42 +++++++ tests/usage/request-pacing.test.ts | 47 ++++++++ 24 files changed, 323 insertions(+), 35 deletions(-) create mode 100644 tests/server/distributed-key-401.test.ts diff --git a/gui/src/components/provider-workspace/ProviderSettings.tsx b/gui/src/components/provider-workspace/ProviderSettings.tsx index 774757e5b9a..cdffab33f4a 100644 --- a/gui/src/components/provider-workspace/ProviderSettings.tsx +++ b/gui/src/components/provider-workspace/ProviderSettings.tsx @@ -254,7 +254,7 @@ export default function ProviderSettings({ || (field !== "requestsPerMinute" && !Number.isSafeInteger(value)) || (field === "requestsPerMinute" && (value < 1 / 60 || value > 60_000)) || (field === "minIntervalMs" && value > 3_600_000))) { - setMsg({ ok: false, text: t("pws.pacingRuleRequired") }); return false; + setMsg({ ok: false, text: t("pws.keyLimitInvalid") }); return false; } if (pacingEnabled && !pacingDraft.requestsPerMinute && !pacingDraft.minIntervalMs && !pacingDraft.maxConcurrentRequests && !pacingDraft.models && !pacingDraft.keys && pacingDraft.keySelection !== "round-robin") { setMsg({ ok: false, text: t("pws.pacingRuleRequired") }); return false; diff --git a/gui/src/i18n/de.ts b/gui/src/i18n/de.ts index d492ab37bb8..2e6842c84e3 100644 --- a/gui/src/i18n/de.ts +++ b/gui/src/i18n/de.ts @@ -2890,6 +2890,7 @@ export const de: Record = { "pws.saveSettings": "Speichern", "pws.keySelection": "API-Schlüsselauswahl", "pws.keyActive": "Aktiver Schlüssel (bisheriges Verhalten)", + "pws.keyLimitInvalid": "Grenzen pro Schlüssel müssen positiv und im unterstützten Bereich sein; Intervall und Parallelität müssen ganze Zahlen sein.", "pws.keyRoundRobin": "Round-Robin", "pws.keyLimits": "Limits pro Schlüssel", "pws.keySharedQuota": "Provider-Limits gelten weiterhin für alle Schlüssel. Schlüssel desselben Projekts können Kontingente teilen; RPM begrenzt keine Token- oder Tagesbudgets.", diff --git a/gui/src/i18n/en.ts b/gui/src/i18n/en.ts index 3d1f741d63f..446aeaea286 100644 --- a/gui/src/i18n/en.ts +++ b/gui/src/i18n/en.ts @@ -1795,6 +1795,7 @@ export const en = { "pws.saveSettings": "Save", "pws.keySelection": "API key selection", "pws.keyActive": "Active key (existing behavior)", + "pws.keyLimitInvalid": "Per-key limits must be positive and within the supported range; interval and concurrency must be whole numbers.", "pws.keyRoundRobin": "Round-robin", "pws.keyLimits": "Per-key limits", "pws.keySharedQuota": "Provider limits still apply across all keys. Keys in the same project may share quotas; RPM limits do not enforce token or daily budgets.", diff --git a/gui/src/i18n/fr.ts b/gui/src/i18n/fr.ts index 505fa31e9de..ae6f766b071 100644 --- a/gui/src/i18n/fr.ts +++ b/gui/src/i18n/fr.ts @@ -1742,6 +1742,7 @@ export const fr: Record = { "pws.optionalPlaceholder": "Facultatif", "pws.keySelection": "Sélection de clé API", "pws.keyActive": "Clé active (comportement existant)", + "pws.keyLimitInvalid": "Les limites par clé doivent être positives et dans la plage autorisée ; l’intervalle et la simultanéité doivent être des entiers.", "pws.keyRoundRobin": "Tourniquet", "pws.keyLimits": "Limites par clé", "pws.keySharedQuota": "Les limites du fournisseur restent communes à toutes les clés. Un même projet peut partager ses quotas ; les RPM ne limitent ni les jetons ni le budget quotidien.", diff --git a/gui/src/i18n/ja.ts b/gui/src/i18n/ja.ts index aa09906632e..272d8d915c3 100644 --- a/gui/src/i18n/ja.ts +++ b/gui/src/i18n/ja.ts @@ -1663,6 +1663,7 @@ export const ja: Record = { "pws.saveSettings": "保存", "pws.keySelection": "API キーの選択", "pws.keyActive": "有効なキー優先（既存の動作）", + "pws.keyLimitInvalid": "キー別の制限は対応範囲内の正の値にし、間隔と同時リクエスト数は整数にしてください。", "pws.keyRoundRobin": "ラウンドロビン", "pws.keyLimits": "キーごとの制限", "pws.keySharedQuota": "プロバイダー全体の制限も適用されます。同じプロジェクトのキーは割当量を共有する場合があります。RPM はトークン数や日次上限を制御しません。", diff --git a/gui/src/i18n/ko.ts b/gui/src/i18n/ko.ts index 928e0df576c..a65cc4e1b75 100644 --- a/gui/src/i18n/ko.ts +++ b/gui/src/i18n/ko.ts @@ -2937,6 +2937,7 @@ export const ko: Record = { "pws.saveSettings": "저장", "pws.keySelection": "API 키 선택 방식", "pws.keyActive": "활성 키 우선 (기존 동작)", + "pws.keyLimitInvalid": "키별 제한은 허용 범위의 양수여야 하며, 간격과 동시 요청 수는 정수여야 합니다.", "pws.keyRoundRobin": "라운드로빈", "pws.keyLimits": "키별 제한", "pws.keySharedQuota": "프로바이더 제한은 모든 키에 합산 적용됩니다. 같은 프로젝트의 키는 할당량을 공유할 수 있으며, RPM 제한은 토큰·일일 한도를 관리하지 않습니다.", diff --git a/gui/src/i18n/pt.ts b/gui/src/i18n/pt.ts index 7bb50fdd37c..8da2dbec990 100644 --- a/gui/src/i18n/pt.ts +++ b/gui/src/i18n/pt.ts @@ -1778,7 +1778,8 @@ export const pt: Record = { "pws.removeConfirmTitle": "Remover provedor", "pws.saveSettings": "Salvar", "pws.keySelection": "Seleção de chave API", - "pws.keyActive": "Chave ativa (comportamento atual)", + "pws.keyActive": "Priorizar a chave ativa (comportamento atual)", + "pws.keyLimitInvalid": "Os limites por chave devem ser positivos e estar no intervalo permitido; intervalo e simultaneidade devem ser números inteiros.", "pws.keyRoundRobin": "Round-robin", "pws.keyLimits": "Limites por chave", "pws.keySharedQuota": "Os limites do provedor continuam valendo para todas as chaves. Chaves do mesmo projeto podem compartilhar cotas; RPM não limita tokens nem o orçamento diário.", diff --git a/gui/src/i18n/ru.ts b/gui/src/i18n/ru.ts index 9e3d7afd51e..eafe49eb637 100644 --- a/gui/src/i18n/ru.ts +++ b/gui/src/i18n/ru.ts @@ -1753,7 +1753,8 @@ export const ru: Record = { "pws.removeConfirmTitle": "Удалить провайдера", "pws.saveSettings": "Сохранить", "pws.keySelection": "Выбор API-ключа", - "pws.keyActive": "Активный ключ (прежнее поведение)", + "pws.keyActive": "Сначала активный ключ (прежнее поведение)", + "pws.keyLimitInvalid": "Лимиты для ключа должны быть положительными и в допустимом диапазоне; интервал и число одновременных запросов должны быть целыми.", "pws.keyRoundRobin": "По кругу", "pws.keyLimits": "Лимиты на ключ", "pws.keySharedQuota": "Лимиты провайдера действуют для всех ключей. Ключи одного проекта могут делить квоты; RPM не ограничивает токены и суточный бюджет.", diff --git a/gui/src/i18n/tr.ts b/gui/src/i18n/tr.ts index 40ce45e2bc4..5e8199af493 100644 --- a/gui/src/i18n/tr.ts +++ b/gui/src/i18n/tr.ts @@ -1781,6 +1781,7 @@ export const tr: Record = { "pws.saveSettings": "Kaydet", "pws.keySelection": "API anahtarı seçimi", "pws.keyActive": "Etkin anahtar (mevcut davranış)", + "pws.keyLimitInvalid": "Anahtar sınırları pozitif ve desteklenen aralıkta olmalıdır; aralık ve eşzamanlılık tam sayı olmalıdır.", "pws.keyRoundRobin": "Sıralı dağıtım", "pws.keyLimits": "Anahtar başına sınırlar", "pws.keySharedQuota": "Sağlayıcı sınırları tüm anahtarlara uygulanır. Aynı projedeki anahtarlar kota paylaşabilir; RPM, token veya günlük bütçeyi sınırlamaz.", diff --git a/gui/src/i18n/vi.ts b/gui/src/i18n/vi.ts index 2876d2a00db..7494b8159fe 100644 --- a/gui/src/i18n/vi.ts +++ b/gui/src/i18n/vi.ts @@ -1765,6 +1765,7 @@ export const vi: Record = { "pws.saveSettings": "Lưu", "pws.keySelection": "Chọn khóa API", "pws.keyActive": "Khóa đang hoạt động (hành vi hiện tại)", + "pws.keyLimitInvalid": "Giới hạn mỗi khóa phải dương và trong phạm vi hỗ trợ; khoảng thời gian và số yêu cầu đồng thời phải là số nguyên.", "pws.keyRoundRobin": "Luân phiên", "pws.keyLimits": "Giới hạn mỗi khóa", "pws.keySharedQuota": "Giới hạn nhà cung cấp vẫn áp dụng cho mọi khóa. Khóa cùng dự án có thể dùng chung hạn mức; RPM không giới hạn token hay ngân sách ngày.", diff --git a/gui/src/i18n/zh-TW.ts b/gui/src/i18n/zh-TW.ts index d3aaf035eb2..14a986771a3 100644 --- a/gui/src/i18n/zh-TW.ts +++ b/gui/src/i18n/zh-TW.ts @@ -1459,6 +1459,7 @@ export const zhTW: Record = { "pws.saveSettings": "儲存", "pws.keySelection": "API 金鑰選擇", "pws.keyActive": "優先使用作用中金鑰（現有行為）", + "pws.keyLimitInvalid": "每個金鑰的限制必須為支援範圍內的正數；間隔和並行數必須為整數。", "pws.keyRoundRobin": "輪詢", "pws.keyLimits": "每個金鑰的限制", "pws.keySharedQuota": "供應商限制仍對所有金鑰合計生效。同一專案的金鑰可能共享配額；RPM 不限制權杖或每日預算。", diff --git a/gui/src/i18n/zh.ts b/gui/src/i18n/zh.ts index 9f05fa50328..1cb850278aa 100644 --- a/gui/src/i18n/zh.ts +++ b/gui/src/i18n/zh.ts @@ -2910,6 +2910,7 @@ export const zh: Record = { "pws.saveSettings": "保存", "pws.keySelection": "API 密钥选择", "pws.keyActive": "活动密钥优先（现有行为）", + "pws.keyLimitInvalid": "每个密钥的限制必须为支持范围内的正数；间隔和并发数必须为整数。", "pws.keyRoundRobin": "轮询", "pws.keyLimits": "每个密钥的限制", "pws.keySharedQuota": "提供商限制仍对所有密钥合并生效。同一项目的密钥可能共享配额；RPM 不限制令牌或每日预算。", diff --git a/gui/tests/provider-settings-request-pacing.test.tsx b/gui/tests/provider-settings-request-pacing.test.tsx index 78b5292fbbf..9d1070cd23d 100644 --- a/gui/tests/provider-settings-request-pacing.test.tsx +++ b/gui/tests/provider-settings-request-pacing.test.tsx @@ -159,6 +159,7 @@ test("invalid key limits block save and discard restores the stored distribution await setInput(numbers[2]!, "0"); await act(async () => { container.querySelector(".pwi-settings-sticky-bar .btn-primary")!.click(); }); expect(patches).toEqual([]); + expect(container.textContent).toContain("Per-key limits must be positive"); await act(async () => { container.querySelector(".pwi-settings-sticky-bar .btn-ghost")!.click(); }); expect(numbers[2]!.value).toBe("1"); expect(numbers[0]!.value).toBe("15"); await act(async () => { container.unmount(); }); diff --git a/src/providers/key-failover.ts b/src/providers/key-failover.ts index bfa89938b16..c82994cbd44 100644 --- a/src/providers/key-failover.ts +++ b/src/providers/key-failover.ts @@ -710,6 +710,24 @@ export function rateLimitRetryDelayMs( * rebuilds from this committed row, reapplies registry metadata, and retains only explicit * runtime transport state (`fetch` and generated OpenCode session affinity). */ +/** Record a request-local key failure without selecting or sending a replacement. */ +export function recordDistributedKeyFailure( + config: OcxConfig, providerName: string, selection: ProviderApiKeySelection | undefined, + status: 401 | 429, now = Date.now(), retryAfter?: string | null, quotaResetAt?: number, +): boolean { + const provider = config.providers[providerName]; + if (!selection?.distributed || !provider || provider.authMode === "oauth" || provider.authMode === "forward") return false; + const failed = provider.apiKeyPool?.find(entry => entry.id === selection.entryId && entry.key === selection.reference); + if (!failed || provider.apiKeySelectionRevision !== selection.revision) return false; + const delay = status === 401 ? MAX_COOLDOWN_MS : quotaResetAt !== undefined + ? Math.max(1, quotaResetAt - now) : parseRetryAfterMs(retryAfter, now) ?? DEFAULT_COOLDOWN_MS; + for (const entry of provider.apiKeyPool ?? []) if (entry.key === failed.key) { + keyCooldowns.set(cooldownKey(providerName, entry.id), { cooldownUntil: now + delay }); + } + sweepExpiredOnWrite(now); + return true; +} + function rotateKeyAfterFailure( config: OcxConfig, providerName: string, @@ -725,12 +743,7 @@ function rotateKeyAfterFailure( if (provider.authMode === "oauth" || provider.authMode === "forward") return null; if (attemptedSelection?.distributed) { - const failed = provider.apiKeyPool?.find(entry => entry.id === attemptedSelection.entryId && entry.key === attemptedSelection.reference); - if (!failed || provider.apiKeySelectionRevision !== attemptedSelection.revision) return null; - const delay = failureStatus === 401 ? MAX_COOLDOWN_MS : quotaResetAt !== undefined - ? Math.max(1, quotaResetAt - now) : parseRetryAfterMs(retryAfterHeader, now) ?? DEFAULT_COOLDOWN_MS; - for (const entry of provider.apiKeyPool ?? []) if (entry.key === failed.key) keyCooldowns.set(cooldownKey(providerName, entry.id), { cooldownUntil: now + delay }); - sweepExpiredOnWrite(now); + if (!recordDistributedKeyFailure(config, providerName, attemptedSelection, failureStatus, now, retryAfterHeader, quotaResetAt)) return null; if (!(provider.apiKeyPool ?? []).some(entry => !isKeyInCooldown(providerName, entry.id, now))) return null; return allocateDistributedKey(providerName, provider, now); } diff --git a/src/server/chat-native.ts b/src/server/chat-native.ts index 75a9edfc3d9..2996601352b 100644 --- a/src/server/chat-native.ts +++ b/src/server/chat-native.ts @@ -54,6 +54,7 @@ import { rateLimitRetryDelayMs, rateLimitRetryPolicyFor, rotateProviderTransportOn429, + rotateProviderTransportOn401, transientRetryPolicyFor, } from "../providers/key-failover"; import { fastPolicyForModel } from "../providers/service-tier"; @@ -432,7 +433,7 @@ export async function runNativeChatAttempt( const send = async ( request: AdapterRequest, - recovery?: "rate-limit-429" | "key-429" | UpstreamSendRecovery, + recovery?: "rate-limit-429" | "key-429" | "key-401" | UpstreamSendRecovery, singleShot = false, ): Promise => { try { @@ -575,8 +576,11 @@ export async function runNativeChatAttempt( } // Same reason as above, plus a second one: rotating here would write a cooldown against // a key that rate-limited nothing, and that false signal outlives the request. - while (response.status === 429 && !isNonReplayableResponse(response) && hasKeyPoolFailover(activeProvider)) { - const rotated = rotateProviderTransportOn429(config, route.providerName, activeProvider, { + while ((response.status === 429 || (response.status === 401 && activeProvider._apiKeyAttempt?.distributed)) + && !isNonReplayableResponse(response) && hasKeyPoolFailover(activeProvider)) { + const failureStatus = response.status; + const rotate = failureStatus === 401 ? rotateProviderTransportOn401 : rotateProviderTransportOn429; + const rotated = rotate(config, route.providerName, activeProvider, { retryAfter: response.headers.get("retry-after"), now: Date.now(), attemptedKey: activeProvider.apiKey, @@ -595,7 +599,7 @@ export async function runNativeChatAttempt( releaseRetainedRequest(); activeRequest = buildActiveRequest(); retainRequest(activeRequest); - response = await send(activeRequest, "key-429"); + response = await send(activeRequest, failureStatus === 401 ? "key-401" : "key-429"); } } catch (error) { releaseRetainedRequest(); diff --git a/src/server/images.ts b/src/server/images.ts index e1b2723656a..b07f9b66251 100644 --- a/src/server/images.ts +++ b/src/server/images.ts @@ -29,7 +29,7 @@ import { readBoundedResponseBytes, type BoundedBytesResult } from "../lib/bounde import { sidecarEnter } from "../lib/sidecar-tracker"; import type { OcxConfig } from "../types"; import { resolveFirstUsableOpenAiSidecar, selectImagesProvider } from "../providers/openai-sidecar"; -import { selectProactiveApiKeyTransport } from "../providers/key-failover"; +import { recordDistributedKeyFailure, selectProactiveApiKeyTransport } from "../providers/key-failover"; import { requestPacingOverloadResponse } from "./responses/pacing-overload"; import { getProviderRegistryEntry } from "../providers/registry"; import { readJsonRequestBody, resolveInboundBodyLimitBytes } from "./request-decompress"; @@ -757,6 +757,7 @@ export async function handleImages( catch { console.error("[images] Failed to release probe lease"); } }; let url: string; + let recordKey401 = () => {}; // Each branch is re-checked as it is entered: the forward check is defence in // depth for the pre-filter above, and the keyed check runs before a key // rotation is committed. @@ -814,6 +815,7 @@ export async function handleImages( throw error; } const provider = warmKeyProvider ?? candidates.keyed.provider; + recordKey401 = () => { recordDistributedKeyFailure(config, providerName, provider._apiKeyAttempt, 401); }; // No fall back to the earlier snapshot once a pick has happened. The picker COMMITS its // choice before returning, so if the chosen reference will not resolve -- revoked keychain // entry, unset env var -- sending the previous key would authenticate a non-idempotent @@ -872,6 +874,7 @@ export async function handleImages( // compact already set this; the credential-bearing sidecars did not. redirect: "manual", }); + if (upstreamResponse.status === 401) recordKey401(); const observed = await readImageResponseBytes(upstreamResponse, { maxBytes: IMAGES_RESPONSE_MAX_BYTES, signal: linkedSignal.signal, diff --git a/src/server/responses/compact.ts b/src/server/responses/compact.ts index a2aad9c026b..6188c7c7f9d 100644 --- a/src/server/responses/compact.ts +++ b/src/server/responses/compact.ts @@ -151,7 +151,7 @@ import { UnsupportedContentEncodingError, } from "../request-decompress"; import { resolveAdapter, resolveWireProtocolOverride } from "../adapter-resolve"; -import { hasKeyPoolFailover, rotateProviderTransportOn429, selectProactiveApiKeyTransport } from "../../providers/key-failover"; +import { hasKeyPoolFailover, recordDistributedKeyFailure, rotateProviderTransportOn429, selectProactiveApiKeyTransport } from "../../providers/key-failover"; import { shouldAttemptImageTierRetry } from "../image-retry"; import { resolveProviderTransport } from "../../providers/xai-transport"; import type { WsData } from "../ws-bridge"; @@ -1152,6 +1152,7 @@ export async function handleResponsesCompact( return formatErrorResponse(502, "upstream_error", "Failed to connect to compact upstream"); } + if (upstream.status === 401) recordDistributedKeyFailure(config, route.providerName, compactProvider._apiKeyAttempt, 401); if ( upstream.status === 401 && (authCtx.kind === "main-pool" || authCtx.kind === "pool") diff --git a/src/server/responses/fetch-helpers.ts b/src/server/responses/fetch-helpers.ts index fa1518169a5..8288bb965c2 100644 --- a/src/server/responses/fetch-helpers.ts +++ b/src/server/responses/fetch-helpers.ts @@ -11,6 +11,7 @@ import type { WsData } from "../ws-bridge"; import { requestPacingMaxConcurrentRequests, sendTrackingRequestSlot, + trackProviderRequestSlotBody, waitForProviderRequestSlot, type ProviderRequestSlot, } from "../../providers/request-pacing"; @@ -138,6 +139,7 @@ export function wantsFreshConnection( export interface PaceAwareFetch { waitForPacing?: (signal?: AbortSignal) => Promise; unpacedFetch?: typeof globalThis.fetch; + fetchWithPacingSlot?: (slot: ProviderRequestSlot | undefined, input: Parameters[0], init?: RequestInit) => Promise; } export type ProviderFetch = typeof globalThis.fetch & PaceAwareFetch; @@ -245,7 +247,8 @@ export interface ProviderFetchOptions { /** One-shot receipt after final local HTTP admission, immediately before executor entry. */ onPhysicalDispatch?: () => void; /** Revalidate/rebuild a queued request at its physical send boundary, after pacing. */ - dispatchOverride?: (input: Parameters[0], init: RequestInit, execute: typeof globalThis.fetch) => Promise; + dispatchOverride?: (input: Parameters[0], init: RequestInit, execute: typeof globalThis.fetch, + reacquirePacing?: (provider: OcxProviderConfig, modelId?: string) => Promise) => Promise; } /** Compose provider pacing and transport admission; receipts run only at executor entry. */ @@ -294,7 +297,8 @@ export function providerFetch( { preconnect }, ) as typeof globalThis.fetch); const httpFetch = Object.assign( - async (input: Parameters[0], init?: RequestInit) => { + async (input: Parameters[0], init?: RequestInit, + reacquirePacing?: (provider: OcxProviderConfig, modelId?: string) => Promise) => { // Refuse before any dispatch side effect where that is sound. `beforeDispatch` commits // attempt accounting and consumes admission state, so a refusal firing after it would // charge an attempt for a send that never happens, and a throwing hook would mask the @@ -322,15 +326,16 @@ export function providerFetch( } } : {}), }; return options.dispatchOverride - ? options.dispatchOverride(input, dispatchInit, dispatch) + ? options.dispatchOverride(input, dispatchInit, dispatch, reacquirePacing) : dispatch(input, dispatchInit); }, { preconnect }, - ) as typeof globalThis.fetch; + ); // ChatGPT Codex backend: streaming turns ride the responses_websockets // transport (measured ~3s faster TTFT than the SSE POST queue); everything // else keeps the provider's HTTP fetch. See ws-upstream.ts for the details. - const unpaced = async (input: Parameters[0], init?: RequestInit) => { + const unpaced = async (input: Parameters[0], init?: RequestInit, + reacquirePacing?: (provider: OcxProviderConfig, modelId?: string) => Promise) => { const upstreamWebsocket = provider.upstreamWebsocket; if (!options.httpOnly && typeof input === "string" && init && shouldUseCodexWsUpstream(input, init, runtime, upstreamWebsocket)) { @@ -340,12 +345,12 @@ export function providerFetch( pacingWebsocketDowngradeWarned.add(providerName); console.warn(`[opencodex] provider ${JSON.stringify(redactSecretString(providerName))} has a request concurrency cap; serving over HTTP/SSE.`); } - return httpFetch(input, init); + return httpFetch(input, init, reacquirePacing); } const egress = egressFor(input); if (providerEgressIsExplicit(egress)) { warnEgressWebsocketDowngradeOnce(providerName, describeProviderEgressForLog(egress)); - return httpFetch(input, init); + return httpFetch(input, init, reacquirePacing); } // The fallback has to be the same HTTP fetch the non-WS branch would have // used, protocol pin included: a WS turn that falls back is serving the @@ -358,11 +363,11 @@ export function providerFetch( options.onPhysicalDispatch?.(); }; const fallback = ((url: Parameters[0], fallbackInit?: RequestInit) => - httpFetch(url, { ...fallbackInit, [WS_FALLBACK_RECEIPT]: notifyOnce } as RequestInit)) as typeof globalThis.fetch; + httpFetch(url, { ...fallbackInit, [WS_FALLBACK_RECEIPT]: notifyOnce } as RequestInit, reacquirePacing)) as typeof globalThis.fetch; return codexWsUpstreamFetch(input, init, fallback, runtime, options.onCodexWsQuota, options.beforeDispatch, options.nativeControl, async () => { (await waitForPacing(init.signal ?? undefined))?.release(); }, notifyOnce); } - return httpFetch(input, init); + return httpFetch(input, init, reacquirePacing); }; let pacingSlotAcquired = options.pacingSlotAcquired === true; const waitForPacing = async (signal?: AbortSignal): Promise => { @@ -385,15 +390,31 @@ export function providerFetch( throw new Error("A provider name is required for request concurrency pacing"); } }; - const wrapped = async (input: Parameters[0], init?: RequestInit) => { - return sendTrackingRequestSlot(await waitForPacing(init?.signal ?? undefined), () => unpaced(input, init)); + const fetchWithPacingSlot = async (slot: ProviderRequestSlot | undefined, input: Parameters[0], init?: RequestInit) => { + const reacquirePacing = async (refreshed: OcxProviderConfig, modelId?: string) => { + if (options.turnScopedPacing) throw new Error("Cannot replace a turn-scoped pacing lease during dispatch"); + if (!options.providerName) throw new Error("Refreshed dispatch requires a named pacing owner"); + slot?.release(); + slot = undefined; + slot = await waitForProviderRequestSlot(providerName, refreshed, modelId, init?.signal ?? undefined); + }; + try { + const response = await unpaced(input, init, reacquirePacing); + return trackProviderRequestSlotBody(slot, response); + } catch (error) { + slot?.release(); + throw error; + } }; + const wrapped = async (input: Parameters[0], init?: RequestInit) => + fetchWithPacingSlot(await waitForPacing(init?.signal ?? undefined), input, init); // The returned wrapper forwards its init down to `dispatch`, which applies the route at the // physical send. Adapters that hand this executor back as `provider.fetch` (Cursor does) // therefore still carry a per-provider route instead of being refused as opaque. const paceAware = Object.assign(wrapped, { preconnect, waitForPacing, + fetchWithPacingSlot, unpacedFetch: Object.assign(unpaced, { preconnect }), }); markEgressTransparentExecutor(paceAware as unknown as typeof globalThis.fetch); @@ -434,12 +455,21 @@ export function storedPoolReplayDispatchNotifier( { preconnect: unpacedSource.preconnect }, ) as ProviderFetch["unpacedFetch"]; const wrapped = async (input: Parameters[0], init?: RequestInit) => { + if (executor.fetchWithPacingSlot) { + const slot = await executor.waitForPacing?.(init?.signal ?? undefined); + notifyOnce(); + return executor.fetchWithPacingSlot(slot, input, init); + } return sendTrackingRequestSlot(await executor.waitForPacing?.(init?.signal ?? undefined), () => unpaced!(input, init)); }; return Object.assign(wrapped, { preconnect: executor.preconnect, waitForPacing: executor.waitForPacing, unpacedFetch: unpaced, + ...(executor.fetchWithPacingSlot ? { fetchWithPacingSlot: (slot: ProviderRequestSlot | undefined, input: Parameters[0], init?: RequestInit) => { + notifyOnce(); + return executor.fetchWithPacingSlot!(slot, input, init); + } } : {}), }) as ProviderFetch; } @@ -472,15 +502,18 @@ export async function fetchWithHeaderTimeout( if (preferIdentityEncoding && !headers.has("accept-encoding")) { headers.set("accept-encoding", "identity"); } - return await sendTrackingRequestSlot(slot, () => fetchExecutor(url, { + const dispatchInit = { ...init, headers, // Never replay provider credentials or request bodies to a redirect destination. // Preserve the 3xx for the owner's existing response/health policy (#914, #1471). - redirect: "manual", + redirect: "manual" as const, signal: AbortSignal.any([abortSignal, timeout.signal]), timeout: 0, - })); + }; + return pacing.fetchWithPacingSlot + ? await pacing.fetchWithPacingSlot(slot, url, dispatchInit) + : await sendTrackingRequestSlot(slot, () => fetchExecutor(url, dispatchInit)); } finally { if (timer) clearTimeout(timer); if (!slot?.bodyTracked) slot?.release(); diff --git a/src/server/responses/request-transport.ts b/src/server/responses/request-transport.ts index f91751f14c3..3d2595bd7c8 100644 --- a/src/server/responses/request-transport.ts +++ b/src/server/responses/request-transport.ts @@ -581,7 +581,11 @@ export async function prepareResponsesTransport( throw new Error("OAuth account selection changed before dispatch"); } } else { - const current = resolveCurrentProviderApiKeyTransport(config, route.providerName, route.provider); + let current = resolveCurrentProviderApiKeyTransport(config, route.providerName, route.provider); + if (!current && route.provider.requestPacing?.enabled && route.provider.requestPacing.keys) { + const fresh = resolveCurrentProviderApiKeyTransport(config, route.providerName, { ...route.provider, _apiKeyAttempt: undefined }); + if (fresh) current = selectProactiveApiKeyTransport(config, route.providerName, fresh) ?? fresh; + } if (!current) throw new Error("API key selection is unavailable before dispatch"); route.provider = current; await resolveCopilotSelection(requestParsed, true); @@ -653,10 +657,16 @@ export async function prepareResponsesTransport( /** Fence the physical send, with bounded credential/session rebuilds and safe negotiated refusals. */ const oauthDispatch = (wireRequest: AdapterRequest, requestParsed = parsed, owner: "adapter" | "passthrough" = "adapter"): ProviderFetchOptions["dispatchOverride"] => { if (route.provider.authMode === "forward") return undefined; - return /** Dispatch with the current binding and session, rebuilding at most three times before returning a safe refusal. */ async (input, init, execute) => { + return /** Dispatch with the current binding and session, rebuilding at most three times before returning a safe refusal. */ async (input, init, execute, reacquirePacing) => { let destination = input; let dispatchInit = init; + let pacingChanged = false; for (let attempt = 0; attempt < 3; attempt++) { + if (pacingChanged) { + if (!reacquirePacing) throw new Error("Refreshed dispatch requires pacing readmission"); + await reacquirePacing(route.provider, route.modelId); + pacingChanged = false; + } if (selectionIsCurrent(requestBindings.get(wireRequest)) && copilotSessionExpiresAt > Date.now() + 1000) { const fetchImpl = (route.provider as OcxProviderConfig & { fetch?: typeof globalThis.fetch }).fetch ?? execute; const binding = requestBindings.get(wireRequest); @@ -758,7 +768,8 @@ export async function prepareResponsesTransport( sameTargetToken = transportToken; destination = rebuilt.url; dispatchInit = { ...dispatchInit, method: rebuilt.method, headers, body: rebuilt.body }; - // The next iteration validates synchronously and calls fetch in that same turn. + pacingChanged = route.provider.requestPacing?.enabled === true && !!route.provider.requestPacing.keys; + // A refreshed key must reacquire admission before the next synchronous send guard. } throw new Error("OAuth account selection changed repeatedly before dispatch"); }; diff --git a/structure/transports/responses-failover.md b/structure/transports/responses-failover.md index 30bcf7233d6..0d4d13d2b77 100644 --- a/structure/transports/responses-failover.md +++ b/structure/transports/responses-failover.md @@ -156,7 +156,7 @@ The shared Responses path follows the [bounded multipart recovery contract](../s ## Upstream key attempt accounting -Key identity is sealed at the guarded physical dispatch after queued selections are rebuilt. Distributed pool selections retain their request-local entry and revision rather than adopting another request's active key; a queued per-key permit cannot authorize a different credential after manual selection drift. +Key identity is sealed at the guarded physical dispatch after queued selections are rebuilt. Distributed pool selections retain their request-local entry and revision rather than adopting another request's active key. A rebuild after manual selection drift releases the old per-key lease and reacquires interval/concurrency admission for the refreshed key; the send guard runs again after that wait. Direct unpaced dispatch without a readmission owner fails closed when this refresh requires a new key slot. Raw adapter terminal usage is recorded before continuation, search, or image loops merge it; repeated parsing of one physical response does not count it twice. Key changes preserve the previous attempt while retaining the active attempt object shared by streaming/combo callbacks. diff --git a/tests/responses/chat-native-combo.test.ts b/tests/responses/chat-native-combo.test.ts index 1c17d98191b..b53cbe282d7 100644 --- a/tests/responses/chat-native-combo.test.ts +++ b/tests/responses/chat-native-combo.test.ts @@ -74,7 +74,7 @@ afterEach(async () => { }); /** A loopback upstream that records every body it receives. */ -function upstream(answer: (body: Rec, hit: number) => Response | Promise) { +function upstream(answer: (body: Rec, hit: number, headers: Headers) => Response | Promise) { const bodies: Rec[] = []; const server = Bun.serve({ hostname: "127.0.0.1", @@ -82,7 +82,7 @@ function upstream(answer: (body: Rec, hit: number) => Response | Promise { + test("distributed 401 recovery stays bounded and avoids the failed key on subsequent requests", async () => { + const keys: string[] = []; + const a = upstream((_body, hit, headers) => { + keys.push(headers.get("authorization")!); + return hit === 1 ? Response.json({ error: { message: "invalid key" } }, { status: 401 }) : twoChoiceCompletion(); + }); + const row = provider("openai-chat", a.baseUrl, { apiKey: "synthetic-first", + apiKeyPool: [{ id: "first", key: "synthetic-first" }, { id: "second", key: "synthetic-second" }], + requestPacing: { enabled: true, keySelection: "round-robin" } }); + const config = comboConfig({ a: row }, [{ provider: "a", model: "m1" }], NATIVE_ON); + const first = await send(config, { stream: false }); + expect(first.response.status).toBe(200); + expect(a.bodies).toHaveLength(2); + const second = await send(config, { stream: false }); + expect(second.response.status).toBe(200); + expect(a.bodies).toHaveLength(3); + expect(keys).toEqual(["Bearer synthetic-first", "Bearer synthetic-second", "Bearer synthetic-second"]); + expect(config.providers.a!.apiKey).toBe("synthetic-first"); + }); test("a native candidate receives the caller's own Chat body, n and logprobs included", async () => { const a = upstream(() => twoChoiceCompletion()); const b = upstream(() => responsesStream("bridge")); diff --git a/tests/server/distributed-key-401.test.ts b/tests/server/distributed-key-401.test.ts new file mode 100644 index 00000000000..61618f23b3f --- /dev/null +++ b/tests/server/distributed-key-401.test.ts @@ -0,0 +1,103 @@ +import { afterEach, beforeEach, expect, test } from "bun:test"; +import { mkdtempSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { saveConfig } from "../../src/config"; +import { clearKeyCooldowns } from "../../src/providers/key-failover"; +import { startServer } from "../../src/server"; +import { handleResponsesCompact } from "../../src/server/responses"; +import type { OcxConfig } from "../../src/types"; +import { installIsolatedCodexHome, type IsolatedCodexHome } from "../helpers/isolated-codex-home"; +import { acquireOwnedSpendHome } from "../helpers/owned-spend-home"; +import { baseCompactionBody, compactionRequest, completedPayload, jsonResponse, removeCompactionFixture } from "../helpers/compaction-routing-fixtures"; +import { fakeChatGptJwt } from "../helpers/fake-chatgpt-jwt"; + +const originalFetch = globalThis.fetch; +const DIRECT_CHATGPT_TOKEN = fakeChatGptJwt({ chatgpt_account_id: "fixture" }); +const disabledOpenAiProvider = { adapter: "openai-responses", baseUrl: "https://chatgpt.com/backend-api/codex", authMode: "forward", disabled: true }; +function keyedProvider(_baseUrl: string) { return { adapter: "openai-responses", baseUrl: "https://api.openai.com/v1", apiKey: "sk-platform-key" }; } +let home: string; +let previousHome: string | undefined; +let previousToken: string | undefined; +let codexHome: IsolatedCodexHome; +let releaseSpendHome: (() => void) | undefined; +const takeSpendHome = () => { releaseSpendHome ??= acquireOwnedSpendHome(); }; +beforeEach(() => { + previousHome = process.env.OPENCODEX_HOME; + previousToken = process.env.OPENCODEX_API_AUTH_TOKEN; + home = mkdtempSync(join(tmpdir(), "ocx-distributed-401-")); + process.env.OPENCODEX_HOME = home; + delete process.env.OPENCODEX_API_AUTH_TOKEN; + codexHome = installIsolatedCodexHome("ocx-distributed-401-codex-"); + clearKeyCooldowns(); +}); +afterEach(async () => { + globalThis.fetch = originalFetch; + releaseSpendHome?.(); releaseSpendHome = undefined; + await removeCompactionFixture(home); + codexHome.restore(); + if (previousHome === undefined) delete process.env.OPENCODEX_HOME; + else process.env.OPENCODEX_HOME = previousHome; + if (previousToken === undefined) delete process.env.OPENCODEX_API_AUTH_TOKEN; + else process.env.OPENCODEX_API_AUTH_TOKEN = previousToken; + clearKeyCooldowns(); +}); + +test("distributed image 401 cools the failed key without replaying the paid POST", async () => { + const seen: string[] = []; + const upstream = Bun.serve({ hostname: "127.0.0.1", port: 0, fetch(req) { + seen.push(req.headers.get("authorization")!); + return Response.json(seen.length === 1 ? { error: "invalid key" } : { data: [] }, { status: seen.length === 1 ? 401 : 200 }); + } }); + globalThis.fetch = ((input: RequestInfo | URL, init?: RequestInit) => { + const url = new URL(typeof input === "string" ? input : input instanceof URL ? input.href : input.url); + return originalFetch(url.hostname === "api.openai.com" ? new URL(url.pathname, upstream.url) : input, init); + }) as typeof fetch; + clearKeyCooldowns(); + saveConfig({ port: 0, defaultProvider: "openai-apikey", openaiProviderTierVersion: 2, + providers: { openai: disabledOpenAiProvider, "openai-apikey": { + ...keyedProvider(upstream.url.toString().replace(/\/$/, "")), + apiKeyPool: [{ id: "first", key: "sk-platform-key" }, { id: "second", key: "sk-warm-key" }], + requestPacing: { enabled: true, keySelection: "round-robin" }, + } }, + } as unknown as OcxConfig); + const server = startServer(0); + try { + for (const [index, status] of [401, 200, 200].entries()) { + const response = await fetch(new URL("/v1/images/generations", server.url), { method: "POST", + headers: { "content-type": "application/json", authorization: `Bearer ${DIRECT_CHATGPT_TOKEN}` }, + body: JSON.stringify({ prompt: "fixture", model: "gpt-image-2" }), + }); + expect(response.status).toBe(status); + await response.text(); + expect(seen.length).toBe(index + 1); + } + expect(seen).toEqual(["Bearer sk-platform-key", "Bearer sk-warm-key", "Bearer sk-warm-key"]); + } finally { await server.stop(true); await upstream.stop(true); clearKeyCooldowns(); } +}); + +test("native compact distributed 401 cools the static key without replaying or refreshing an account", async () => { + takeSpendHome(); + clearKeyCooldowns("openai-apikey"); + const config = { defaultProvider: "openai-apikey", openaiProviderTierVersion: 2, providers: { + "openai-apikey": { adapter: "openai-responses", baseUrl: "https://api.openai.com/v1", authMode: "key", + apiKey: "synthetic-first", apiKeyPool: [{ id: "first", key: "synthetic-first" }, { id: "second", key: "synthetic-second" }], + requestPacing: { enabled: true, keySelection: "round-robin" }, + }, + } } as OcxConfig; + const keys: string[] = []; + globalThis.fetch = (async (input, init) => { + expect(String(input)).toBe("https://api.openai.com/v1/responses/compact"); + keys.push(new Headers(init?.headers).get("authorization")!); + return keys.length === 1 ? Response.json({ error: { message: "invalid key" } }, { status: 401 }) : jsonResponse(completedPayload("compact fixture")); + }) as typeof fetch; + try { + for (const [index, status] of [401, 200, 200].entries()) { + const response = await handleResponsesCompact(compactionRequest(baseCompactionBody({ model: "openai-apikey/gpt-5.5" })), config, { model: "", provider: "" }); + expect(response.status).toBe(status); + await response.text(); + expect(keys).toHaveLength(index + 1); + } + expect(keys).toEqual(["Bearer synthetic-first", "Bearer synthetic-second", "Bearer synthetic-second"]); + } finally { clearKeyCooldowns("openai-apikey"); } +}); diff --git a/tests/server/server-key-failover-e2e.test.ts b/tests/server/server-key-failover-e2e.test.ts index ea964c0eb0a..ef6053acbf5 100644 --- a/tests/server/server-key-failover-e2e.test.ts +++ b/tests/server/server-key-failover-e2e.test.ts @@ -130,6 +130,48 @@ describe("server 429 key failover (end-to-end)", () => { expect(providerApiKeySelectionIsCurrent(config, "current", current)).toBe(true); }); + test("Responses readmits a refreshed key and waits for its existing concurrency lease", async () => { + const seen: string[] = []; + upstream = Bun.serve({ hostname: "127.0.0.1", port: 0, fetch(req) { + seen.push(req.headers.get("authorization")!); + return Response.json({ id: "paced", object: "chat.completion", choices: [ + { index: 0, message: { role: "assistant", content: "readmitted" }, finish_reason: "stop" }, + ] }); + } }); + const config = { port: 0, defaultProvider: "paced", providers: { paced: { + adapter: "openai-chat", baseUrl: `http://127.0.0.1:${upstream.port}/v1`, allowPrivateNetwork: true, + authMode: "key", apiKey: "synthetic-first", + apiKeyPool: [{ id: "first", key: "synthetic-first" }, { id: "second", key: "synthetic-second" }], + requestPacing: { enabled: true, keys: { maxConcurrentRequests: 1 } }, + } } } as OcxConfig; + saveConfig(config); + const first = await waitForProviderRequestSlot("paced", config.providers.paced); + const second = await waitForProviderRequestSlot("paced", { ...config.providers.paced, apiKey: "synthetic-second" }); + const server = startTrackedServer(); + const abort = new AbortController(); + try { + const pending = fetch(new URL("/v1/responses", server.url), { method: "POST", signal: abort.signal, + headers: { "content-type": "application/json" }, body: JSON.stringify({ model: "paced/test", input: "hello", stream: false }) }); + await Bun.sleep(30); + const selected = await managementFetch(new URL("/api/providers/keys/active", server.url), { + method: "PUT", headers: { "content-type": "application/json" }, body: JSON.stringify({ name: "paced", id: "second" }), + }); + expect(selected.status).toBe(200); + await selected.text(); + first.release(); + await Bun.sleep(30); + expect(seen).toEqual([]); + second.release(); + const response = await pending; + expect(response.status).toBe(200); + expect(await response.text()).toContain("readmitted"); + expect(seen).toEqual(["Bearer synthetic-second"]); + } finally { + abort.abort(); first.release(); second.release(); await server.stop(true); + resetProviderRequestPacingForTest(); + } + }); + test.each(["responses", "chat/completions"])("%s logs only the key selected after pacing", async surface => { resetUsageReadCacheForTests(); let now = 0; diff --git a/tests/usage/request-pacing.test.ts b/tests/usage/request-pacing.test.ts index b6b180e6f51..38a7acf1bf4 100644 --- a/tests/usage/request-pacing.test.ts +++ b/tests/usage/request-pacing.test.ts @@ -17,6 +17,53 @@ import type { OcxProviderConfig } from "../../src/types"; afterEach(() => resetProviderRequestPacingForTest()); +test("dispatch readmission observes the refreshed key interval and releases its replacement lease", async () => { + const clock = fakePacingClock(); + setProviderRequestPacingRuntimeForTest(clock.runtime); + const row = { ...provider({ enabled: true, keys: { minIntervalMs: 100, maxConcurrentRequests: 1 } }), apiKey: "first" }; + const refreshed = { ...row, apiKey: "second" }; + (await waitForProviderRequestSlot("readmission", refreshed)).release(); + const sends: number[] = []; + const physical = Object.assign(async () => { sends.push(clock.now()); return new Response("fixture"); }, { preconnect() {} }) as typeof fetch; + const executor = providerFetch({ ...row, fetch: physical }, undefined, { providerName: "readmission", + dispatchOverride: async (input, init, execute, reacquire) => { + await reacquire!(refreshed); + return execute(input, init); + }, + }); + const pending = fetchWithHeaderTimeout("https://example.test/v1/chat/completions", {}, new AbortController().signal, 5000, false, executor); + for (let i = 0; i < 10; i++) await Promise.resolve(); + expect(sends).toEqual([]); + clock.advanceBy(100); + const response = await pending; + expect(sends).toEqual([100]); + await response.text(); + const slot = await waitForProviderRequestSlot("readmission", row); + slot.release(); +}); + +test("abort during refreshed-key admission releases the old lease without sending", async () => { + const row = { ...provider({ enabled: true, keys: { maxConcurrentRequests: 1 } }), apiKey: "first" }; + const refreshed = { ...row, apiKey: "second" }; + const held = await waitForProviderRequestSlot("readmission-abort", refreshed); + const controller = new AbortController(); + let sends = 0; + const executor = providerFetch(row, undefined, { providerName: "readmission-abort", + dispatchOverride: async (_input, _init, _execute, reacquire) => { + await reacquire!(refreshed); + sends++; + return new Response("fixture"); + }, + }); + const pending = executor("https://example.test/v1/chat/completions", { signal: controller.signal }); + for (let i = 0; i < 10; i++) await Promise.resolve(); + controller.abort(); + await expect(pending).rejects.toThrow(); + expect(sends).toBe(0); + const slot = await waitForProviderRequestSlot("readmission-abort", row); + slot.release(); held.release(); +}); + function provider(requestPacing: OcxProviderConfig["requestPacing"]): OcxProviderConfig { return { adapter: "openai-chat", baseUrl: "https://example.test/v1", requestPacing }; } From 9856d46cb98f806bcecf8384cf10dfa569b09151 Mon Sep 17 00:00:00 2001 From: GSL-R <174104911+GSL-R@users.noreply.github.com> Date: Mon, 12 Oct 2026 02:52:48 +0900 Subject: [PATCH 03/17] test(providers): observe queued admission before changing a key --- tests/server/server-key-failover-e2e.test.ts | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/tests/server/server-key-failover-e2e.test.ts b/tests/server/server-key-failover-e2e.test.ts index ef6053acbf5..6cbc72eb9c3 100644 --- a/tests/server/server-key-failover-e2e.test.ts +++ b/tests/server/server-key-failover-e2e.test.ts @@ -27,7 +27,7 @@ import { installIsolatedCodexHome, type IsolatedCodexHome } from "../helpers/iso import { removeTreeWithRetry } from "../helpers/remove-tree"; import { managementFetch } from "../helpers/management-auth"; import { resolveContextPrincipal } from "../../src/server/auth-cors"; -import { resetProviderRequestPacingForTest, setProviderRequestPacingRuntimeForTest, waitForProviderRequestSlot } from "../../src/providers/request-pacing"; +import { providerRequestPacingStatus, resetProviderRequestPacingForTest, setProviderRequestPacingRuntimeForTest, waitForProviderRequestSlot } from "../../src/providers/request-pacing"; import { providerApiKeySelectionIsCurrent, resolveCurrentProviderApiKeyTransport } from "../../src/providers/api-key-selection"; import { routedProviderConfig } from "../../src/router"; import type { OcxProviderTransport } from "../../src/providers/xai-transport"; @@ -152,7 +152,8 @@ describe("server 429 key failover (end-to-end)", () => { try { const pending = fetch(new URL("/v1/responses", server.url), { method: "POST", signal: abort.signal, headers: { "content-type": "application/json" }, body: JSON.stringify({ model: "paced/test", input: "hello", stream: false }) }); - await Bun.sleep(30); + for (let i = 0; i < 200 && providerRequestPacingStatus("paced", config.providers.paced).queued === 0; i++) await Bun.sleep(5); + expect(providerRequestPacingStatus("paced", config.providers.paced).queued).toBe(1); const selected = await managementFetch(new URL("/api/providers/keys/active", server.url), { method: "PUT", headers: { "content-type": "application/json" }, body: JSON.stringify({ name: "paced", id: "second" }), }); From b2ddd79fcaf238f7ba75722143d369e31c360096 Mon Sep 17 00:00:00 2001 From: GSL-R <174104911+GSL-R@users.noreply.github.com> Date: Mon, 12 Oct 2026 03:10:36 +0900 Subject: [PATCH 04/17] fix(pacing): release failed notifications and readmit refreshed turns --- scripts/test-layout/layout.json | 4 +- src/server/responses/fetch-helpers.ts | 7 +- src/server/responses/request-transport.ts | 11 ++- src/server/responses/run-turn-execution.ts | 11 ++- structure/transports/responses-failover.md | 2 +- tests/fixtures/test-layout-expected.json | 3 +- .../responses-fetch-helpers-boundary.test.ts | 15 ++++ .../responses-run-turn-key-pacing.test.ts | 76 +++++++++++++++++++ 8 files changed, 116 insertions(+), 13 deletions(-) create mode 100644 tests/responses/responses-run-turn-key-pacing.test.ts diff --git a/scripts/test-layout/layout.json b/scripts/test-layout/layout.json index bebe0dbe1b9..6ca81c23243 100644 --- a/scripts/test-layout/layout.json +++ b/scripts/test-layout/layout.json @@ -1553,7 +1553,7 @@ "responses-custom-tool-repair.test.ts": "responses", "responses-custom-tool-stream-consistency.test.ts": "responses", "responses-default-namespace-emit-normalize.test.ts": "responses", - "responses-fetch-helpers-boundary.test.ts": "responses", + "responses-fetch-helpers-boundary.test.ts": "responses", "responses-run-turn-key-pacing.test.ts": "responses", "responses-field-backfill.test.ts": "responses", "responses-forward-dangling-call.test.ts": "responses", "responses-forward-incomplete-quota.test.ts": "responses", @@ -1666,7 +1666,7 @@ "server-images-bodyless-content-length.test.ts": "server", "server-images-pool-admission.test.ts": "server", "server-images.test.ts": "server", - "server-key-failover-e2e.test.ts": "server", + "server-key-failover-e2e.test.ts": "server", "distributed-key-401.test.ts": "server", "server-jev-combo-e2e.test.ts": "server", "combo-requested-effort.test.ts": "server", "server-kiro-completion-e2e.test.ts": "server", "server-kiro-device-login.test.ts": "server", "server-kiro-oauth-401-replay.test.ts": "server", diff --git a/src/server/responses/fetch-helpers.ts b/src/server/responses/fetch-helpers.ts index 8288bb965c2..dc29e97af5c 100644 --- a/src/server/responses/fetch-helpers.ts +++ b/src/server/responses/fetch-helpers.ts @@ -446,6 +446,9 @@ export function storedPoolReplayDispatchNotifier( notified = true; onDispatch(); }; + const notifyWithSlot = (slot: ProviderRequestSlot | undefined): void => { + try { notifyOnce(); } catch (error) { slot?.release(); throw error; } + }; const unpacedSource = executor.unpacedFetch ?? executor; const unpaced = Object.assign( (input: Parameters[0], init?: RequestInit) => { @@ -457,7 +460,7 @@ export function storedPoolReplayDispatchNotifier( const wrapped = async (input: Parameters[0], init?: RequestInit) => { if (executor.fetchWithPacingSlot) { const slot = await executor.waitForPacing?.(init?.signal ?? undefined); - notifyOnce(); + notifyWithSlot(slot); return executor.fetchWithPacingSlot(slot, input, init); } return sendTrackingRequestSlot(await executor.waitForPacing?.(init?.signal ?? undefined), () => unpaced!(input, init)); @@ -467,7 +470,7 @@ export function storedPoolReplayDispatchNotifier( waitForPacing: executor.waitForPacing, unpacedFetch: unpaced, ...(executor.fetchWithPacingSlot ? { fetchWithPacingSlot: (slot: ProviderRequestSlot | undefined, input: Parameters[0], init?: RequestInit) => { - notifyOnce(); + notifyWithSlot(slot); return executor.fetchWithPacingSlot!(slot, input, init); } } : {}), }) as ProviderFetch; diff --git a/src/server/responses/request-transport.ts b/src/server/responses/request-transport.ts index 3d2595bd7c8..0da1eb1bc58 100644 --- a/src/server/responses/request-transport.ts +++ b/src/server/responses/request-transport.ts @@ -613,7 +613,8 @@ export async function prepareResponsesTransport( ? (options.sendBudget as import("../../lib/request-execution-budget").RequestExecutionBudget).beginSpendProducer?.() : undefined; try { for (let attempt = 0; attempt < 3; attempt++) { - if (!selectionIsCurrent(adapterBindings.get(selectedAdapter))) selectedAdapter = await refreshRunTurnAdapter(requestParsed); + const selectionRefreshed = !selectionIsCurrent(adapterBindings.get(selectedAdapter)); + if (selectionRefreshed) selectedAdapter = await refreshRunTurnAdapter(requestParsed); const binding = adapterBindings.get(selectedAdapter); const run = rawRunTurns.get(selectedAdapter); if (!run) throw new Error("Selected provider no longer supports this turn transport"); @@ -623,8 +624,10 @@ export async function prepareResponsesTransport( // Both main and image-loop callers already acquired the initial pacing slot. // Subsequent physical messages retain this adapter/credential and are paced normally. try { - attemptSlot = attempt === 0 - ? incoming.pacingSlot ?? await waitForProviderRequestSlot(route.providerName, route.provider, route.modelId, incoming.abortSignal) + // A pre-acquired lease belongs to the original selection, never its replacement. + if (attempt === 0 && selectionRefreshed) releaseProviderRequestSlot(incoming.pacingSlot); + attemptSlot = attempt === 0 && !selectionRefreshed && incoming.pacingSlot + ? incoming.pacingSlot : await waitForProviderRequestSlot(route.providerName, route.provider, route.modelId, incoming.abortSignal); const fetch = providerFetch(route.provider, options.codexWsRuntimeIdentity, { providerName: route.providerName, modelId: route.modelId, pacingSlotAcquired: true, @@ -640,7 +643,7 @@ export async function prepareResponsesTransport( sent = true; }, }); - await run(requestParsed, { ...incoming, providerFetch: fetch }, event => { if (!refused) emit(event); }); + await run(requestParsed, { ...incoming, pacingSlot: attemptSlot, providerFetch: fetch }, event => { if (!refused) emit(event); }); } catch (error) { if (!refused) throw error; } finally { diff --git a/src/server/responses/run-turn-execution.ts b/src/server/responses/run-turn-execution.ts index 459a5cf0817..c4dc3dd5405 100644 --- a/src/server/responses/run-turn-execution.ts +++ b/src/server/responses/run-turn-execution.ts @@ -196,12 +196,13 @@ export async function executeResponsesRunTurn( const queue = createAdapterEventQueue({ onBacklogExceeded: () => runTurnAbort.abort(), }); - const refreshRunTurnSelection = async (): Promise => { - if (selectionIsCurrent(adapterBindings.get(transportState.runTurnAdapter))) return; + const refreshRunTurnSelection = async (): Promise => { + if (selectionIsCurrent(adapterBindings.get(transportState.runTurnAdapter))) return false; await refreshRunTurnAdapter(parsed); bindRouteReasoningReplayScope({ parsed, providerName: route.providerName, provider: route.provider, adapterName: transportState.runTurnAdapter.name, oauthCredentialSnapshot: transportState.replayOAuthCredentialSnapshot }); sealRequestAttemptIdentity(logCtx.activeAttempt, logCtx.provider, transportState.runTurnAdapter.name, logCtx.accountLogLabel); + return true; }; // Initial admission must settle before the streaming Response commits HTTP 200. // Let the outer Responses facade preserve the local retryable-429 contract. @@ -248,7 +249,11 @@ export async function executeResponsesRunTurn( if (!pacingSlotAcquired) { pacingSlot = await waitForProviderRequestSlot(route.providerName, route.provider, route.modelId, runTurnAbort.signal); } - await refreshRunTurnSelection(); + if (await refreshRunTurnSelection()) { + releaseProviderRequestSlot(pacingSlot); + pacingSlot = undefined; + pacingSlot = await waitForProviderRequestSlot(route.providerName, route.provider, route.modelId, runTurnAbort.signal); + } // LOCAL PATCH (runturn-websearch): refreshRunTurnSelection binds route // state onto `parsed`; mirror it onto the iteration-local copy the // adapter actually receives. diff --git a/structure/transports/responses-failover.md b/structure/transports/responses-failover.md index 0d4d13d2b77..0eff1d1d5a2 100644 --- a/structure/transports/responses-failover.md +++ b/structure/transports/responses-failover.md @@ -156,7 +156,7 @@ The shared Responses path follows the [bounded multipart recovery contract](../s ## Upstream key attempt accounting -Key identity is sealed at the guarded physical dispatch after queued selections are rebuilt. Distributed pool selections retain their request-local entry and revision rather than adopting another request's active key. A rebuild after manual selection drift releases the old per-key lease and reacquires interval/concurrency admission for the refreshed key; the send guard runs again after that wait. Direct unpaced dispatch without a readmission owner fails closed when this refresh requires a new key slot. +Key identity is sealed at guarded physical dispatch after queued selections are rebuilt. Distributed selections retain their request-local entry and revision. A rebuild after selection drift releases the old per-key lease and reacquires admission for the refreshed key; the send guard checks again after waiting. Turn-scoped adapters follow the same rule before invocation. Unpaced dispatch without a readmission owner fails closed. Replay notification errors release acquired slots before propagating, including pre-acquired-slot dispatch. Raw adapter terminal usage is recorded before continuation, search, or image loops merge it; repeated parsing of one physical response does not count it twice. Key changes preserve the previous attempt while retaining the active attempt object shared by streaming/combo callbacks. diff --git a/tests/fixtures/test-layout-expected.json b/tests/fixtures/test-layout-expected.json index d5fdc2cafb4..8eed423cf17 100644 --- a/tests/fixtures/test-layout-expected.json +++ b/tests/fixtures/test-layout-expected.json @@ -955,6 +955,7 @@ "responses-custom-tool-stream-consistency.test.ts": "responses", "responses-default-namespace-emit-normalize.test.ts": "responses", "responses-fetch-helpers-boundary.test.ts": "responses", "responses-field-backfill.test.ts": "responses", + "responses-run-turn-key-pacing.test.ts": "responses", "responses-forward-dangling-call.test.ts": "responses", "responses-forward-incomplete-quota.test.ts": "responses", "responses-forward-posit-continuation.test.ts": "responses", "responses-forward-prompt-envelope.test.ts": "responses", "responses-freeform-wrapper-keys.test.ts": "responses", "responses-function-tool-repair.test.ts": "responses", @@ -1009,7 +1010,7 @@ "server-combo-zero-output-failover.test.ts": "server", "server-google-antigravity-oauth-401-replay.test.ts": "server", "server-antigravity-web-search-validation.test.ts": "server", "server-google-antigravity-oauth-429-budget.test.ts": "server", "server-gui-bundle-freshness.test.ts": "server", "server-images-bodyless-content-length.test.ts": "server", "server-images-pool-admission.test.ts": "server", - "server-images.test.ts": "server", "server-key-failover-e2e.test.ts": "server", + "server-images.test.ts": "server", "server-key-failover-e2e.test.ts": "server", "distributed-key-401.test.ts": "server", "server-jev-combo-e2e.test.ts": "server", "server-kiro-completion-e2e.test.ts": "server", "combo-requested-effort.test.ts": "server", "server-kiro-device-login.test.ts": "server", "server-kiro-oauth-401-replay.test.ts": "server", diff --git a/tests/responses/responses-fetch-helpers-boundary.test.ts b/tests/responses/responses-fetch-helpers-boundary.test.ts index e6963d05749..6e10829342f 100644 --- a/tests/responses/responses-fetch-helpers-boundary.test.ts +++ b/tests/responses/responses-fetch-helpers-boundary.test.ts @@ -310,4 +310,19 @@ describe("storedPoolReplayDispatchNotifier", () => { const executor = pacedExecutor({ pacing: async () => {} }); expect(storedPoolReplayDispatchNotifier(executor.wrapped, undefined)).toBe(executor.wrapped); }); + test.each([false, true])("notification failure releases the slot (pre-acquired=%s)", async preAcquired => { + let releases = 0, sends = 0; + const slot = { leased: true, bodyTracked: false, released: false, release() { releases += 1; } }; + const execute = Object.assign(async () => new Response("unexpected"), { + preconnect() {}, + waitForPacing: async () => slot, + fetchWithPacingSlot: async () => { sends += 1; return new Response("unexpected"); }, + }); + const notifier = storedPoolReplayDispatchNotifier(execute, () => { throw new Error("notification failed"); }); + await expect((async () => preAcquired + ? notifier.fetchWithPacingSlot!(slot, "https://example.test") + : notifier("https://example.test"))()).rejects.toThrow("notification failed"); + expect(releases).toBe(1); + expect(sends).toBe(0); + }); }); diff --git a/tests/responses/responses-run-turn-key-pacing.test.ts b/tests/responses/responses-run-turn-key-pacing.test.ts new file mode 100644 index 00000000000..b75e8bf15de --- /dev/null +++ b/tests/responses/responses-run-turn-key-pacing.test.ts @@ -0,0 +1,76 @@ +import { expect, spyOn, test } from "bun:test"; +import { mkdtempSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { ADAPTER_REGISTRY } from "../../src/adapters/registry"; +import { handleResponses } from "../../src/server/responses"; +import { providerRequestPacingStatus, resetProviderRequestPacingForTest, waitForProviderRequestSlot } from "../../src/providers/request-pacing"; +import type { OcxConfig } from "../../src/types"; +import { acquireOwnedSpendHome } from "../helpers/owned-spend-home"; +import { removeTreeWithRetry } from "../helpers/remove-tree"; + +test.each([false, true])("runTurn refresh releases the old key and readmits the replacement (abort=%s)", async abortWhileWaiting => { + const previousHome = process.env.OPENCODEX_HOME; + const home = mkdtempSync(join(tmpdir(), "ocx-runturn-pacing-")); + process.env.OPENCODEX_HOME = home; + resetProviderRequestPacingForTest(); + const releaseOwner = acquireOwnedSpendHome(); + const seen: string[] = []; + const create = ADAPTER_REGISTRY["openai-chat"].create; + const override = spyOn(ADAPTER_REGISTRY["openai-chat"], "create").mockImplementation((...args) => ({ + ...create(...args), + async runTurn(_parsed, _incoming, emit) { + seen.push(args[0].apiKey!); + emit({ type: "text_delta", text: "readmitted turn" }); emit({ type: "done" }); + }, + })); + const config = { port: 0, defaultProvider: "paced", providers: { paced: { + adapter: "openai-chat", baseUrl: "https://example.test/v1", authMode: "key", apiKey: "synthetic-first", + apiKeyPool: [{ id: "first", key: "synthetic-first" }, { id: "second", key: "synthetic-second" }], + requestPacing: { enabled: true, keys: { maxConcurrentRequests: 1 } }, + } } } as OcxConfig; + const first = await waitForProviderRequestSlot("paced", config.providers.paced); + const second = await waitForProviderRequestSlot("paced", { ...config.providers.paced, apiKey: "synthetic-second" }); + const abort = new AbortController(); + let pending: Promise | undefined; + const waitForKeyQueue = async (id: string) => { + for (let i = 0; i < 200; i++) { + if (providerRequestPacingStatus("paced", config.providers.paced).keys?.find(key => key.id === id)?.queued === 1) return; + await Bun.sleep(5); + } + throw new Error(`key ${id} never queued`); + }; + try { + pending = handleResponses(new Request("http://localhost/v1/responses", { + method: "POST", headers: { "content-type": "application/json" }, signal: abort.signal, + body: JSON.stringify({ model: "paced/test", input: "hello", stream: false }), + }), config, { provider: "", model: "" }, { abortSignal: abort.signal }); + // Observe any rejected handler immediately while the test controls its queue. + void pending.catch(() => {}); + await waitForKeyQueue("first"); + config.providers.paced.apiKey = "synthetic-second"; + config.providers.paced.apiKeySelectionRevision = "replacement"; + first.release(); + await waitForKeyQueue("second"); + expect(seen).toEqual([]); + expect(providerRequestPacingStatus("paced", config.providers.paced).keys?.find(key => key.id === "first")?.inFlight).toBe(0); + if (abortWhileWaiting) { + abort.abort(); + await pending.catch(() => undefined); + expect(seen).toEqual([]); + expect(providerRequestPacingStatus("paced", config.providers.paced).queued).toBe(0); + } else { + second.release(); + const response = await pending; + expect(response.status).toBe(200); + expect(await response.text()).toContain("readmitted turn"); + expect(seen).toEqual(["synthetic-second"]); + } + } finally { + abort.abort(); first.release(); second.release(); + await pending?.catch(() => undefined); + override.mockRestore(); resetProviderRequestPacingForTest(); releaseOwner(); removeTreeWithRetry(home); + if (previousHome === undefined) delete process.env.OPENCODEX_HOME; + else process.env.OPENCODEX_HOME = previousHome; + } +}); From 4bd1023b26736cb2143d26da487d46790089846c Mon Sep 17 00:00:00 2001 From: JUN Date: Mon, 12 Oct 2026 05:35:35 +0900 Subject: [PATCH 05/17] fix(providers): revalidate distributed key cooldowns before physical sends --- .../src/content/docs/guides/providers.md | 2 + scripts/test-layout/layout.json | 2 +- src/providers/api-key-cooldowns.ts | 48 +++++ src/providers/api-key-selection.ts | 3 +- src/providers/key-failover.ts | 54 +---- src/server/messages-native.ts | 25 ++- src/server/responses/request-transport.ts | 12 +- structure/providers-and-adapters.md | 2 +- structure/transports/responses-failover.md | 2 +- .../messages-native-key-selection.test.ts | 198 ++++++++++++++++++ tests/fixtures/test-layout-expected.json | 2 + .../api-key-selection-cooldown.test.ts | 135 ++++++++++++ .../responses-run-turn-key-pacing.test.ts | 46 ++++ 13 files changed, 468 insertions(+), 63 deletions(-) create mode 100644 src/providers/api-key-cooldowns.ts create mode 100644 tests/claude-integration/messages-native-key-selection.test.ts create mode 100644 tests/providers/api-key-selection-cooldown.test.ts diff --git a/docs-site/src/content/docs/guides/providers.md b/docs-site/src/content/docs/guides/providers.md index 34e18cf485d..4fefc801591 100644 --- a/docs-site/src/content/docs/guides/providers.md +++ b/docs-site/src/content/docs/guides/providers.md @@ -93,6 +93,8 @@ credentials must never be replayed on the same token, and local runtimes have no preserve. It is opt-in: when the option is absent the feature is off; object presence enables it unless `enabled: false`. +When round-robin key selection is enabled, queued requests recheck key cooldowns before sending and acquire pacing capacity for any replacement key. This does not change the saved active key. If a key cools after a turn has started, further sends in that turn are refused rather than switching its credential. + ### Which account a request spends The question people ask before connecting an account is whether opencodex will draw on the diff --git a/scripts/test-layout/layout.json b/scripts/test-layout/layout.json index ffb6c185c4c..4c6900fa657 100644 --- a/scripts/test-layout/layout.json +++ b/scripts/test-layout/layout.json @@ -2,7 +2,7 @@ "version": 1, "root": "tests", "explicit": { - "messages-request-id-headers.test.ts": "claude-integration", + "api-key-selection-cooldown.test.ts": "providers", "messages-native-key-selection.test.ts": "claude-integration", "messages-request-id-headers.test.ts": "claude-integration", "messages-request-id-endpoint.test.ts": "claude-integration", "anthropic-revoked-token.test.ts": "adapters/anthropic", "messages-revoked-token.test.ts": "claude-integration", diff --git a/src/providers/api-key-cooldowns.ts b/src/providers/api-key-cooldowns.ts new file mode 100644 index 00000000000..378b10cb6ac --- /dev/null +++ b/src/providers/api-key-cooldowns.ts @@ -0,0 +1,48 @@ +/** Process-local cooldown state shared by selection validation and key failover. */ +const keyCooldowns = new Map(); + +function cooldownKey(providerName: string, keyId: string): string { + return `${providerName}\0${keyId}`; +} + +export function setKeyCooldown(providerName: string, keyId: string, cooldownUntil: number): void { + keyCooldowns.set(cooldownKey(providerName, keyId), { cooldownUntil }); +} + +export function isKeyInCooldown(providerName: string, keyId: string, now = Date.now()): boolean { + const entry = keyCooldowns.get(cooldownKey(providerName, keyId)); + if (!entry) return false; + if (entry.cooldownUntil <= now) { + keyCooldowns.delete(cooldownKey(providerName, keyId)); + return false; + } + return true; +} + +export function getKeyCooldownUntil(providerName: string, keyId: string, now = Date.now()): number | null { + const entry = keyCooldowns.get(cooldownKey(providerName, keyId)); + if (!entry) return null; + return entry.cooldownUntil > now ? entry.cooldownUntil : null; +} + +export function sweepExpiredApiKeyCooldowns(now = Date.now()): number { + let removed = 0; + for (const [key, cooldown] of keyCooldowns) { + if (cooldown.cooldownUntil > now) continue; + keyCooldowns.delete(key); + removed += 1; + } + return removed; +} + +/** Manual key management clears this provider's cooldowns. */ +export function clearKeyCooldowns(providerName?: string): void { + if (!providerName) { + keyCooldowns.clear(); + return; + } + const prefix = `${providerName}\0`; + for (const key of keyCooldowns.keys()) { + if (key.startsWith(prefix)) keyCooldowns.delete(key); + } +} diff --git a/src/providers/api-key-selection.ts b/src/providers/api-key-selection.ts index 8e961dbacde..74ec7c314b0 100644 --- a/src/providers/api-key-selection.ts +++ b/src/providers/api-key-selection.ts @@ -1,3 +1,4 @@ +import { isKeyInCooldown } from "./api-key-cooldowns"; import { randomUUID } from "node:crypto"; import { mutatePersistedConfig } from "../config"; import { publishAccountSelection } from "../lib/account-selection-events"; @@ -23,7 +24,7 @@ function currentKeyProvider(config: OcxConfig, name: string, selection?: Provide let row = configured; if (selection?.distributed) { const entry = configured.apiKeyPool?.find(key => key.id === selection.entryId && key.key === selection.reference); - if (!entry || configured.apiKeySelectionRevision !== selection.revision + if (!entry || isKeyInCooldown(name, entry.id) || configured.apiKeySelectionRevision !== selection.revision || !configured.requestPacing?.enabled || configured.requestPacing.keySelection !== "round-robin") return null; row = { ...configured, apiKey: entry.key, _apiKeyAttempt: selection }; } diff --git a/src/providers/key-failover.ts b/src/providers/key-failover.ts index c82994cbd44..189e50ec36e 100644 --- a/src/providers/key-failover.ts +++ b/src/providers/key-failover.ts @@ -8,6 +8,8 @@ * * Modelled after src/codex/routing.ts cooldown logic but scoped to plain API-key pools. */ +import { isKeyInCooldown, getKeyCooldownUntil, setKeyCooldown } from "./api-key-cooldowns"; +export { clearKeyCooldowns, getKeyCooldownUntil, sweepExpiredApiKeyCooldowns } from "./api-key-cooldowns"; import { commitProviderApiKeySelection } from "./api-key-selection"; import { RequestPacingQueueOverloadError } from "./request-pacing"; import type { ProviderApiKeySelection } from "../types/provider"; @@ -25,10 +27,6 @@ import { cachedApiKeyQuota } from "./quota-key-accounts"; // ---- cooldown state (in-memory, same as codex/routing.ts) ---- -interface KeyCooldown { - cooldownUntil: number; -} - const DEFAULT_COOLDOWN_MS = 60_000; const MAX_COOLDOWN_MS = 10 * 60_000; // cap at 10 min for api-key rotation @@ -338,11 +336,6 @@ const DEFAULT_RESET_REPLAY = { } as const satisfies Required; /** Map<`${providerName}\0${keyId}`, KeyCooldown> */ -const keyCooldowns = new Map(); - -function cooldownKey(providerName: string, keyId: string): string { - return `${providerName}\0${keyId}`; -} /** * Parse an upstream `Retry-After` header: numeric seconds (including `0`) or an HTTP-date. @@ -370,16 +363,6 @@ function parseRetryAfterMs(value: string | null | undefined, now = Date.now()): * True while the given key is inside its 429 cooldown window (lazily evicting the entry once the * window expires). Used to skip keys that the upstream just rate-limited during failover. */ -function isKeyInCooldown(providerName: string, keyId: string, now = Date.now()): boolean { - const entry = keyCooldowns.get(cooldownKey(providerName, keyId)); - if (!entry) return false; - if (entry.cooldownUntil <= now) { - keyCooldowns.delete(cooldownKey(providerName, keyId)); - return false; - } - return true; -} - // ---- public API ---- /** @@ -722,7 +705,7 @@ export function recordDistributedKeyFailure( const delay = status === 401 ? MAX_COOLDOWN_MS : quotaResetAt !== undefined ? Math.max(1, quotaResetAt - now) : parseRetryAfterMs(retryAfter, now) ?? DEFAULT_COOLDOWN_MS; for (const entry of provider.apiKeyPool ?? []) if (entry.key === failed.key) { - keyCooldowns.set(cooldownKey(providerName, entry.id), { cooldownUntil: now + delay }); + setKeyCooldown(providerName, entry.id, now + delay); } sweepExpiredOnWrite(now); return true; @@ -807,7 +790,7 @@ function rotateKeyAfterFailure( : quotaResetAt !== undefined ? Math.max(quotaResetAt - now, 1) : parseRetryAfterMs(retryAfterHeader, now) ?? DEFAULT_COOLDOWN_MS; - keyCooldowns.set(cooldownKey(providerName, outcome.value.failedId), { cooldownUntil: now + cooldownMs }); + setKeyCooldown(providerName, outcome.value.failedId, now + cooldownMs); sweepExpiredOnWrite(now); } if ("exhaustedCount" in outcome.value) { @@ -856,16 +839,6 @@ export function rotateKeyOn401( return rotateKeyAfterFailure(config, providerName, 401, null, now, attemptedKey, attemptedSelection); } -export function sweepExpiredApiKeyCooldowns(now = Date.now()): number { - let removed = 0; - for (const [key, cooldown] of keyCooldowns) { - if (cooldown.cooldownUntil > now) continue; - keyCooldowns.delete(key); - removed += 1; - } - return removed; -} - interface RotateProviderTransportOptions { retryAfter?: string | null; /** Epoch ms from `parseQuotaResetAt`, when the upstream dated the reset in its body. */ @@ -936,22 +909,3 @@ function applyRotatedTransport( }; return resolveProviderTransport(providerName, retryProvider, promptCacheKey); } - -/** Clear cooldown state for a provider (e.g. after manual key management). */ -export function clearKeyCooldowns(providerName?: string): void { - if (!providerName) { - keyCooldowns.clear(); - return; - } - const prefix = `${providerName}\0`; - for (const key of keyCooldowns.keys()) { - if (key.startsWith(prefix)) keyCooldowns.delete(key); - } -} - -/** Visible-for-testing: get the cooldown-until timestamp for a key. */ -export function getKeyCooldownUntil(providerName: string, keyId: string, now = Date.now()): number | null { - const entry = keyCooldowns.get(cooldownKey(providerName, keyId)); - if (!entry) return null; - return entry.cooldownUntil > now ? entry.cooldownUntil : null; -} diff --git a/src/server/messages-native.ts b/src/server/messages-native.ts index 63d342098ba..6f74bb190c5 100644 --- a/src/server/messages-native.ts +++ b/src/server/messages-native.ts @@ -539,7 +539,7 @@ export async function handleNativeMessages(options: HandleNativeMessagesOptions) providerFetch(activeProvider, undefined, { providerName: route.providerName, modelId: route.modelId, - dispatchOverride: async (_input, init, execute) => { + dispatchOverride: async (_input, init, execute, reacquirePacing) => { if (oauthBinding) { if (!nativeOAuthRouteIsCurrent(oauthBinding)) throw new NativeOAuthSelectionChangedError(); // The OAuth twin of the key check below: re-resolve through the same selection @@ -556,12 +556,24 @@ export async function handleNativeMessages(options: HandleNativeMessagesOptions) rebuildFor(oauthProvider(oauthBinding)); logCtx.provider = formatAnthropicProviderForLog(route.providerName, oauthBinding.snapshot.accountId, config); } - } else if (!providerApiKeySelectionIsCurrent(config, route.providerName, activeProvider)) { - const current = resolveCurrentProviderApiKeyTransport(config, route.providerName, activeProvider); - if (!current || nativeMessagesDeclineReason({ ...route, provider: current }, body, config, selector) !== undefined) { - throw new Error("Provider key selection is no longer available for native Messages"); + } else { + for (let attempt = 0; !providerApiKeySelectionIsCurrent(config, route.providerName, activeProvider); attempt++) { + if (attempt >= 3) throw new Error("Provider key selection changed repeatedly before native Messages dispatch"); + let current = resolveCurrentProviderApiKeyTransport(config, route.providerName, activeProvider); + if (!current && activeProvider._apiKeyAttempt?.distributed) { + const fresh = resolveCurrentProviderApiKeyTransport(config, route.providerName, { ...activeProvider, _apiKeyAttempt: undefined }); + if (fresh) current = selectProactiveApiKeyTransport(config, route.providerName, fresh); + } + if (!current || nativeMessagesDeclineReason({ ...route, provider: current }, body, config, selector) !== undefined) { + throw new Error("Provider key selection is no longer available for native Messages"); + } + rebuildFor(current); + if (current.requestPacing?.enabled + && (current.requestPacing.keys || current.requestPacing.keySelection === "round-robin")) { + if (!reacquirePacing) throw new Error("Refreshed native Messages dispatch requires pacing readmission"); + await reacquirePacing(current, route.modelId); + } } - rebuildFor(current); } // The retry closure may hold a pre-reselection request: send the current one whole. const wire = activeRequest; @@ -652,6 +664,7 @@ export async function handleNativeMessages(options: HandleNativeMessagesOptions) const rotated = rotateProviderTransportOn401(config, route.providerName, activeProvider, { now: Date.now(), attemptedKey: activeProvider.apiKey, + attemptedSelection: activeProvider._apiKeyAttempt, }); if (!rotated) break; discard(response); diff --git a/src/server/responses/request-transport.ts b/src/server/responses/request-transport.ts index 0da1eb1bc58..e7105a531ae 100644 --- a/src/server/responses/request-transport.ts +++ b/src/server/responses/request-transport.ts @@ -582,7 +582,8 @@ export async function prepareResponsesTransport( } } else { let current = resolveCurrentProviderApiKeyTransport(config, route.providerName, route.provider); - if (!current && route.provider.requestPacing?.enabled && route.provider.requestPacing.keys) { + if (!current && route.provider.requestPacing?.enabled + && (route.provider.requestPacing.keys || route.provider.requestPacing.keySelection === "round-robin")) { const fresh = resolveCurrentProviderApiKeyTransport(config, route.providerName, { ...route.provider, _apiKeyAttempt: undefined }); if (fresh) current = selectProactiveApiKeyTransport(config, route.providerName, fresh) ?? fresh; } @@ -634,7 +635,11 @@ export async function prepareResponsesTransport( pacingSlot: attemptSlot, turnScopedPacing: true, beforeDispatch: () => { - if (sent) return; + if (sent) { + if (binding?.kind === "api-key" && binding.provider._apiKeyAttempt?.distributed + && !selectionIsCurrent(binding)) throw new Error("API key became unavailable during the turn"); + return; + } if (!selectionIsCurrent(binding)) { refused = true; throw new Error("Account selection changed before the first turn dispatch"); @@ -771,7 +776,8 @@ export async function prepareResponsesTransport( sameTargetToken = transportToken; destination = rebuilt.url; dispatchInit = { ...dispatchInit, method: rebuilt.method, headers, body: rebuilt.body }; - pacingChanged = route.provider.requestPacing?.enabled === true && !!route.provider.requestPacing.keys; + pacingChanged = route.provider.requestPacing?.enabled === true + && (!!route.provider.requestPacing.keys || route.provider.requestPacing.keySelection === "round-robin"); // A refreshed key must reacquire admission before the next synchronous send guard. } throw new Error("OAuth account selection changed repeatedly before dispatch"); diff --git a/structure/providers-and-adapters.md b/structure/providers-and-adapters.md index 5a7e8226444..e5b72f24273 100644 --- a/structure/providers-and-adapters.md +++ b/structure/providers-and-adapters.md @@ -288,7 +288,7 @@ Cursor, one lease spans the whole turn: RunSSE and BidiAppend may overlap inside turns wait at the cap. Follow-up sends still obey start intervals. The turn owner returns its lease after `runTurn` settles, so RunSSE body completion cannot admit another turn early. A capped canonical Codex WebSocket turn uses HTTP/SSE because the socket has no response-body lifecycle. -Capacity waits use the same bounded queue and retryable queue-overload errors as interval waits. Optional `requestPacing.keys` adds a shared cross-model limit for each credential reference; duplicate references share capacity, but distinct references resolving to one upstream project do not imply separate quota. `keySelection: round-robin` assigns request-local healthy keys without mutating persisted selection, skips known cooldowns, and refuses exhausted pools with retryable 429. Provider, model and key gates reserve atomically; key counters and the allocation cursor are process-local, not token or billing balances. OAuth and forwarded credentials retain their existing selection. +Capacity waits use the same bounded queue and retryable queue-overload errors as interval waits. Optional `requestPacing.keys` adds a shared cross-model limit for each credential reference; duplicate references share capacity, but distinct references resolving to one upstream project do not imply separate quota. `keySelection: round-robin` assigns request-local healthy keys without mutating persisted selection, skips known cooldowns, and refuses exhausted pools with retryable 429. Provider, model and key gates reserve atomically; key counters and the allocation cursor are process-local, not token or billing balances. OAuth and forwarded credentials retain their existing selection. Cooldown state is shared through the dependency-light `src/providers/api-key-cooldowns.ts` owner. Distributed selections are checked again at physical dispatch; a cooled queued key releases its old permit, selects a healthy sibling, and reacquires pacing before sending. Native Messages follows the same readmission rule. An established distributed turn refuses later sends after its key cools instead of changing credentials mid-turn. Command Code effort defaults in `src/providers/command-code-efforts.ts` combine public-profile facts with the live API measurements from #5096. Both presets share the exact per-model rows; diff --git a/structure/transports/responses-failover.md b/structure/transports/responses-failover.md index 0eff1d1d5a2..e7c6d78d35a 100644 --- a/structure/transports/responses-failover.md +++ b/structure/transports/responses-failover.md @@ -592,7 +592,7 @@ provider's input. Object-identity metadata is recomputed per attempt. Compaction route identity excludes policy selectors and combos on either side; synthetic fast/effort suffixes are removed before identity checks. A stale selector resolved only through the default -provider cannot establish the original serving identity and remains portable. +provider cannot establish the original serving identity and remains portable. Queued distributed API-key bindings are revalidated after every pacing admission. A replacement releases the old permit before waiting for provider, model and key capacity; enabled round-robin performs this step even without a per-key cap. Once a turn has physically sent, a cooled binding is refused rather than rotated inside that turn. ## Anthropic Fast downgrade recovery diff --git a/tests/claude-integration/messages-native-key-selection.test.ts b/tests/claude-integration/messages-native-key-selection.test.ts new file mode 100644 index 00000000000..38f98002582 --- /dev/null +++ b/tests/claude-integration/messages-native-key-selection.test.ts @@ -0,0 +1,198 @@ +import { afterEach, beforeEach, describe, expect, test } from "bun:test"; +import { mkdtempSync, readFileSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { saveConfig } from "../../src/config"; +import { captureProviderApiKeySelection } from "../../src/providers/api-key-selection"; +import { clearKeyCooldowns, forgetApiKeyRotationCursor, recordDistributedKeyFailure } from "../../src/providers/key-failover"; +import { providerRequestPacingStatus, resetProviderRequestPacingForTest, waitForProviderRequestSlot } from "../../src/providers/request-pacing"; +import { handleClaudeMessages } from "../../src/server/claude-messages"; +import { getRequestLogEntries } from "../../src/server/request-log"; +import type { OcxConfig, OcxProviderConfig } from "../../src/types"; +import { acquireOwnedSpendHome } from "../helpers/owned-spend-home"; +import { removeTreeWithRetry } from "../helpers/remove-tree"; + +const NAME = "native-key-selection"; +const KEYS = ["fixture-key-alpha", "fixture-key-beta", "fixture-key-gamma"]; +const MESSAGE = { id: "msg_key_selection", type: "message", role: "assistant", model: "claude-x", + content: [{ type: "text", text: "fixture reply" }], stop_reason: "end_turn", stop_sequence: null, + usage: { input_tokens: 11, output_tokens: 3 } }; +let home: string; +let previousHome: string | undefined; +let releaseOwner: () => void; +let upstream: ReturnType | undefined; +let seen: { key: string | null; path: string; body: unknown }[]; + +beforeEach(() => { + previousHome = process.env.OPENCODEX_HOME; + home = mkdtempSync(join(tmpdir(), "ocx-native-key-selection-")); + process.env.OPENCODEX_HOME = home; + releaseOwner = acquireOwnedSpendHome(); + clearKeyCooldowns(); forgetApiKeyRotationCursor(); resetProviderRequestPacingForTest(); + seen = []; +}); + +afterEach(async () => { + await upstream?.stop(true); upstream = undefined; + releaseOwner(); + clearKeyCooldowns(); forgetApiKeyRotationCursor(); resetProviderRequestPacingForTest(); + if (previousHome === undefined) delete process.env.OPENCODEX_HOME; + else process.env.OPENCODEX_HOME = previousHome; + removeTreeWithRetry(home); +}); + +function fixture(count = 2, rejectFirst = false, keyCap = true): OcxConfig { + upstream = Bun.serve({ hostname: "127.0.0.1", port: 0, async fetch(req) { + seen.push({ key: req.headers.get("x-api-key"), path: new URL(req.url).pathname, body: await req.json() }); + if (rejectFirst && seen.length === 1) return Response.json({ type: "error", + error: { type: "authentication_error", message: "invalid fixture key" } }, { status: 401 }); + return Response.json(MESSAGE); + } }); + const config = { port: 0, defaultProvider: NAME, protocols: { rollout: { managedMessagesNative: true } }, + providers: { [NAME]: { adapter: "anthropic", baseUrl: `http://127.0.0.1:${upstream.port}`, + authMode: "key", allowPrivateNetwork: true, apiKey: KEYS[0], apiKeySelectionRevision: "saved-revision", + models: ["claude-x"], apiKeyPool: KEYS.slice(0, count).map((key, index) => ({ id: `k${index + 1}`, key, addedAt: index + 1 })), + requestPacing: { enabled: true, keySelection: "round-robin", + ...(keyCap ? { keys: { maxConcurrentRequests: 1 } } : { maxConcurrentRequests: 1 }) }, + } } } as OcxConfig; + saveConfig(config); + return config; +} + +function provider(config: OcxConfig): OcxProviderConfig { return config.providers[NAME]!; } +function status(config: OcxConfig) { return providerRequestPacingStatus(NAME, provider(config)); } +function keyStatus(config: OcxConfig, id: string) { return status(config).keys!.find(key => key.id === id)!; } +function cool(config: OcxConfig, index: number): void { + const selection = { ...captureProviderApiKeySelection({ ...provider(config), apiKey: KEYS[index] }), distributed: true }; + expect(recordDistributedKeyFailure(config, NAME, selection, 429, Date.now(), "30")).toBe(true); +} +async function waitUntil(predicate: () => boolean, message: string): Promise { + const deadline = Date.now() + 2_000; + while (!predicate()) { + if (Date.now() >= deadline) throw new Error(message); + await Bun.sleep(5); + } +} +function queue(config: OcxConfig, id: string) { + return waitUntil(() => keyStatus(config, id).queued === 1, `key ${id} never queued: ${JSON.stringify(status(config))}`); +} +function start(config: OcxConfig, abort: AbortController) { + const requestId = `native-key-selection-${crypto.randomUUID()}`; + const signal = AbortSignal.any([abort.signal, AbortSignal.timeout(5_000)]); + const pending = handleClaudeMessages(new Request("http://localhost/v1/messages", { method: "POST", + headers: { "content-type": "application/json" }, signal, + body: JSON.stringify({ model: `${NAME}/claude-x`, max_tokens: 64, + messages: [{ role: "user", content: "fixture question" }], stream: false }), + }), config, { model: "", provider: "" }, { requestId, start: Date.now() }) + .then(async response => ({ response, text: await response.text(), requestId })); + void pending.catch(() => {}); + return pending; +} +function unchanged(config: OcxConfig) { + expect(provider(config).apiKey).toBe(KEYS[0]); + expect(provider(config).apiKeySelectionRevision).toBe("saved-revision"); + const persisted = JSON.parse(readFileSync(join(home, "config.json"), "utf8")) as OcxConfig; + expect(persisted.providers[NAME]!.apiKey).toBe(KEYS[0]); + expect(persisted.providers[NAME]!.apiKeySelectionRevision).toBe("saved-revision"); + expect(persisted.providers[NAME]!._apiKeyAttempt).toBeUndefined(); +} +function noLeaks(config: OcxConfig) { + expect(status(config)).toMatchObject({ queued: 0, inFlight: 0 }); + for (const key of status(config).keys!) expect(key).toMatchObject({ queued: 0, inFlight: 0 }); +} +function nativeSuccess(result: Awaited>) { + expect(result.response.status).toBe(200); + expect(JSON.parse(result.text).content).toEqual(MESSAGE.content); + expect(getRequestLogEntries().find(row => row.requestId === result.requestId)?.protocolTrace?.mode).toBe("native"); + expect(seen.every(entry => entry.path === "/v1/messages")).toBe(true); +} + +describe("native Messages distributed key selection", () => { + test("401 recovers A to B without changing the saved key or revision", async () => { + const config = fixture(2, true); + const abort = new AbortController(); + const pending = start(config, abort); + try { + nativeSuccess(await pending); + expect(seen.map(entry => entry.key)).toEqual(KEYS.slice(0, 2)); + expect(seen[1]!.body).toEqual(seen[0]!.body); + unchanged(config); noLeaks(config); + } finally { abort.abort(); await pending.catch(() => undefined); } + }); + + for (const cancel of [false, true]) test(`cooled queued A releases its permit and readmits B (abort=${cancel})`, async () => { + const config = fixture(); + const first = await waitForProviderRequestSlot(NAME, provider(config)); + const second = await waitForProviderRequestSlot(NAME, { ...provider(config), apiKey: KEYS[1] }); + const abort = new AbortController(); + const pending = start(config, abort); + try { + await queue(config, "k1"); + expect(seen).toEqual([]); + cool(config, 0); first.release(); + await queue(config, "k2"); + expect(first.released).toBe(true); + expect(keyStatus(config, "k1")).toMatchObject({ inFlight: 0, queued: 0, starts: 2 }); + expect(keyStatus(config, "k2")).toMatchObject({ inFlight: 1, queued: 1, starts: 1 }); + expect(status(config).inFlight).toBe(1); + expect(seen).toEqual([]); + if (cancel) { + abort.abort(); + await pending.catch(() => undefined); + expect(seen).toEqual([]); + expect(status(config).queued).toBe(0); + second.release(); + } else { + second.release(); nativeSuccess(await pending); + expect(seen.map(entry => entry.key)).toEqual([KEYS[1]]); + } + unchanged(config); noLeaks(config); + } finally { + abort.abort(); first.release(); second.release(); await pending.catch(() => undefined); + } + }, 10_000); + + test("round-robin without a keys rule reselects behind the provider concurrency cap", async () => { + const config = fixture(2, false, false); + expect(provider(config).requestPacing!.keys).toBeUndefined(); + const held = await waitForProviderRequestSlot(NAME, provider(config)); + const abort = new AbortController(); + const pending = start(config, abort); + try { + await queue(config, "k1"); cool(config, 0); + expect(seen).toEqual([]); + held.release(); nativeSuccess(await pending); + expect(seen.map(entry => entry.key)).toEqual([KEYS[1]]); + expect(keyStatus(config, "k1").starts).toBe(2); + expect(keyStatus(config, "k2").starts).toBe(1); + unchanged(config); noLeaks(config); + } finally { abort.abort(); held.release(); await pending.catch(() => undefined); } + }); + + for (const exhausted of [false, true]) test(`revalidates B after readmission when B cools while queued (exhausted=${exhausted})`, async () => { + const config = fixture(3); + const first = await waitForProviderRequestSlot(NAME, provider(config)); + const second = await waitForProviderRequestSlot(NAME, { ...provider(config), apiKey: KEYS[1] }); + const abort = new AbortController(); + const pending = start(config, abort); + try { + await queue(config, "k1"); cool(config, 0); first.release(); + await queue(config, "k2"); + expect(keyStatus(config, "k1").inFlight).toBe(0); + expect(seen).toEqual([]); + cool(config, 1); + if (exhausted) cool(config, 2); + second.release(); + const result = await pending; + if (exhausted) { + expect(result.response.status).toBe(429); + expect(Number(result.response.headers.get("retry-after"))).toBeGreaterThan(0); + expect(seen).toEqual([]); + } else { + nativeSuccess(result); + expect(seen.map(entry => entry.key)).toEqual([KEYS[2]]); + } + unchanged(config); noLeaks(config); + } finally { abort.abort(); first.release(); second.release(); await pending.catch(() => undefined); } + }, 10_000); +}); diff --git a/tests/fixtures/test-layout-expected.json b/tests/fixtures/test-layout-expected.json index f91135dc03d..0fb9baba590 100644 --- a/tests/fixtures/test-layout-expected.json +++ b/tests/fixtures/test-layout-expected.json @@ -1,4 +1,6 @@ { + "api-key-selection-cooldown.test.ts": "providers", + "messages-native-key-selection.test.ts": "claude-integration", "messages-request-id-headers.test.ts": "claude-integration", "messages-request-id-endpoint.test.ts": "claude-integration", "anthropic-revoked-token.test.ts": "adapters/anthropic", diff --git a/tests/providers/api-key-selection-cooldown.test.ts b/tests/providers/api-key-selection-cooldown.test.ts new file mode 100644 index 00000000000..9863b2118df --- /dev/null +++ b/tests/providers/api-key-selection-cooldown.test.ts @@ -0,0 +1,135 @@ +import { afterEach, beforeEach, describe, expect, spyOn, test } from "bun:test"; +import { mkdtempSync, readFileSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { saveConfig } from "../../src/config"; +import { captureProviderApiKeySelection, providerApiKeySelectionIsCurrent, resolveCurrentProviderApiKeyTransport } from "../../src/providers/api-key-selection"; +import { clearKeyCooldowns, forgetApiKeyRotationCursor, getKeyCooldownUntil, recordDistributedKeyFailure, + rotateKeyOn401, selectProactiveApiKeyTransport } from "../../src/providers/key-failover"; +import { resetProviderRequestPacingForTest } from "../../src/providers/request-pacing"; +import { routedProviderConfig } from "../../src/router"; +import type { OcxConfig, OcxProviderConfig } from "../../src/types"; +import { acquireOwnedSpendHome } from "../helpers/owned-spend-home"; +import { removeTreeWithRetry } from "../helpers/remove-tree"; + +const NAME = "selection-cooldown"; +const A = "fixture-selection-alpha"; +const B = "fixture-selection-beta"; +let home: string; +let previousHome: string | undefined; +let releaseOwner: () => void; + +beforeEach(() => { + previousHome = process.env.OPENCODEX_HOME; + home = mkdtempSync(join(tmpdir(), "ocx-selection-cooldown-")); + process.env.OPENCODEX_HOME = home; + releaseOwner = acquireOwnedSpendHome(); + clearKeyCooldowns(); forgetApiKeyRotationCursor(); resetProviderRequestPacingForTest(); +}); +afterEach(() => { + releaseOwner(); clearKeyCooldowns(); forgetApiKeyRotationCursor(); resetProviderRequestPacingForTest(); + if (previousHome === undefined) delete process.env.OPENCODEX_HOME; + else process.env.OPENCODEX_HOME = previousHome; + removeTreeWithRetry(home); +}); + +function fixture(): OcxConfig { + return { port: 0, defaultProvider: NAME, providers: { [NAME]: { adapter: "anthropic", authMode: "key", + baseUrl: "https://example.test", apiKey: A, apiKeySelectionRevision: "revision-fixture", + apiKeyPool: [{ id: "a", key: A }, { id: "b", key: B }], + requestPacing: { enabled: true, keySelection: "round-robin", keys: { maxConcurrentRequests: 1 } }, + } } } as OcxConfig; +} +function binding(config: OcxConfig, index = 0): OcxProviderConfig { + const provider = config.providers[NAME]!; + const entry = provider.apiKeyPool![index]!; + return routedProviderConfig(NAME, { ...provider, apiKey: entry.key, + _apiKeyAttempt: { entryId: entry.id, reference: entry.key, revision: provider.apiKeySelectionRevision, distributed: true } }); +} +function cool(config: OcxConfig, route: OcxProviderConfig, now = Date.now(), retryAfter = "30") { + expect(recordDistributedKeyFailure(config, NAME, route._apiKeyAttempt, 429, now, retryAfter)).toBe(true); +} +function accepted(config: OcxConfig, route: OcxProviderConfig) { + expect(providerApiKeySelectionIsCurrent(config, NAME, route)).toBe(true); + expect(resolveCurrentProviderApiKeyTransport(config, NAME, route)?.apiKey).toBe(route.apiKey); +} +function rejected(config: OcxConfig, route: OcxProviderConfig) { + expect(providerApiKeySelectionIsCurrent(config, NAME, route)).toBe(false); + expect(resolveCurrentProviderApiKeyTransport(config, NAME, route)).toBeNull(); +} + +describe("distributed selection cooldown validation", () => { + test("rejects a cooled exact binding and resolver while its healthy sibling remains current", () => { + const config = fixture(); + const first = binding(config); const second = binding(config, 1); + accepted(config, first); accepted(config, second); + cool(config, first); + rejected(config, first); accepted(config, second); + expect(config.providers[NAME]!.apiKey).toBe(A); + expect(config.providers[NAME]!.apiKeySelectionRevision).toBe("revision-fixture"); + }); + + for (const field of ["entry", "reference", "revision"] as const) test(`rejects a stale distributed ${field} independently of cooldown`, () => { + const config = fixture(); const route = binding(config); + if (field === "entry") config.providers[NAME]!.apiKeyPool![0]!.id = "replacement-entry"; + if (field === "reference") config.providers[NAME]!.apiKeyPool![0]!.key = "fixture-replacement-reference"; + if (field === "revision") config.providers[NAME]!.apiKeySelectionRevision = "replacement-revision"; + expect(recordDistributedKeyFailure(config, NAME, route._apiKeyAttempt, 429)).toBe(false); + rejected(config, route); + expect(getKeyCooldownUntil(NAME, "a")).toBeNull(); + }); + + test("expiry makes the same distributed binding usable at the exact cooldown boundary", () => { + const config = fixture(); const route = binding(config); + const now = Date.now(); + const clock = spyOn(Date, "now").mockReturnValue(now); + try { + cool(config, route, now, "1"); + clock.mockReturnValue(now + 999); rejected(config, route); + expect(getKeyCooldownUntil(NAME, "a")).toBe(now + 1_000); + clock.mockReturnValue(now + 1_000); accepted(config, route); + expect(getKeyCooldownUntil(NAME, "a")).toBeNull(); + } finally { clock.mockRestore(); } + }); + + test("duplicate references cool together without cooling a healthy distinct reference", () => { + const config = fixture(); + config.providers[NAME]!.apiKeyPool!.splice(1, 0, { id: "a-duplicate", key: A }); + const first = binding(config); const duplicate = binding(config, 1); const sibling = binding(config, 2); + cool(config, duplicate); + expect(getKeyCooldownUntil(NAME, "a")).toBe(getKeyCooldownUntil(NAME, "a-duplicate")); + rejected(config, first); rejected(config, duplicate); accepted(config, sibling); + const replacement = selectProactiveApiKeyTransport(config, NAME, first)!; + expect(replacement.apiKey).toBe(B); + expect(replacement._apiKeyAttempt?.entryId).toBe("b"); + }); + + test("a persistent binding keeps its existing validation semantics even when that entry is cooled", () => { + const config = fixture(); const distributed = binding(config); + cool(config, distributed); + delete config.providers[NAME]!.requestPacing; + const configured = config.providers[NAME]!; + const persistent = routedProviderConfig(NAME, { ...configured, _apiKeyAttempt: captureProviderApiKeySelection(configured) }); + accepted(config, persistent); + expect(recordDistributedKeyFailure(config, NAME, persistent._apiKeyAttempt, 429)).toBe(false); + }); + + test("default-off persistent mode retains committed 401 rotation and revision updates", () => { + const config = fixture(); delete config.providers[NAME]!.requestPacing; + saveConfig(config); + const route = routedProviderConfig(NAME, config.providers[NAME]!); + expect(selectProactiveApiKeyTransport(config, NAME, route)).toBeNull(); + expect(route._apiKeyAttempt?.distributed).toBeUndefined(); + expect(recordDistributedKeyFailure(config, NAME, route._apiKeyAttempt, 401)).toBe(false); + const replacement = rotateKeyOn401(config, NAME, Date.now(), A, route._apiKeyAttempt)!; + expect(replacement.apiKey).toBe(B); + expect(config.providers[NAME]!.apiKey).toBe(B); + expect(config.providers[NAME]!.apiKeySelectionRevision).not.toBe("revision-fixture"); + const persisted = JSON.parse(readFileSync(join(home, "config.json"), "utf8")) as OcxConfig; + expect(persisted.providers[NAME]!.apiKey).toBe(B); + expect(persisted.providers[NAME]!.apiKeySelectionRevision).toBe(config.providers[NAME]!.apiKeySelectionRevision); + expect(getKeyCooldownUntil(NAME, "a")).not.toBeNull(); + accepted(config, routedProviderConfig(NAME, config.providers[NAME]!)); + expect(providerApiKeySelectionIsCurrent(config, NAME, route)).toBe(false); + }); +}); diff --git a/tests/responses/responses-run-turn-key-pacing.test.ts b/tests/responses/responses-run-turn-key-pacing.test.ts index b75e8bf15de..ef5bc160159 100644 --- a/tests/responses/responses-run-turn-key-pacing.test.ts +++ b/tests/responses/responses-run-turn-key-pacing.test.ts @@ -74,3 +74,49 @@ test.each([false, true])("runTurn refresh releases the old key and readmits the else process.env.OPENCODEX_HOME = previousHome; } }); + +test("a distributed runTurn does not send again after its bound key cools following output", async () => { + const { recordDistributedKeyFailure, clearKeyCooldowns, forgetApiKeyRotationCursor } = await import("../../src/providers/key-failover"); + const previousHome = process.env.OPENCODEX_HOME; + const home = mkdtempSync(join(tmpdir(), "ocx-runturn-cooled-")); + process.env.OPENCODEX_HOME = home; + resetProviderRequestPacingForTest(); clearKeyCooldowns(); forgetApiKeyRotationCursor("paced"); + const releaseOwner = acquireOwnedSpendHome(); + const config = { port: 0, defaultProvider: "paced", providers: { paced: { + adapter: "openai-chat", baseUrl: "https://example.test/v1", authMode: "key", apiKey: "synthetic-first", + apiKeySelectionRevision: "stable", + apiKeyPool: [{ id: "first", key: "synthetic-first" }, { id: "second", key: "synthetic-second" }], + requestPacing: { enabled: true, keySelection: "round-robin", keys: { maxConcurrentRequests: 1 } }, + } } } as OcxConfig; + let sends = 0; + const outgoing = spyOn(globalThis, "fetch").mockImplementation(async () => { + sends++; return new Response("physical response"); + }); + const create = ADAPTER_REGISTRY["openai-chat"].create; + const override = spyOn(ADAPTER_REGISTRY["openai-chat"], "create").mockImplementation((...args) => ({ + ...create(...args), + async runTurn(_parsed, incoming, emit) { + const first = await incoming.providerFetch!("https://example.test/v1/messages", { method: "POST" }); + await first.text(); + emit({ type: "text_delta", text: "established turn" }); + expect(recordDistributedKeyFailure(config, "paced", args[0]._apiKeyAttempt, 401)).toBe(true); + await expect(incoming.providerFetch!("https://example.test/v1/messages", { method: "POST" })).rejects.toThrow("unavailable during the turn"); + emit({ type: "done" }); + }, + })); + try { + const response = await handleResponses(new Request("http://localhost/v1/responses", { + method: "POST", headers: { "content-type": "application/json" }, + body: JSON.stringify({ model: "paced/test", input: "hello", stream: false }), + }), config, { provider: "", model: "" }); + expect(response.status).toBe(200); + expect(await response.text()).toContain("established turn"); + expect(sends).toBe(1); + expect(config.providers.paced.apiKey).toBe("synthetic-first"); + expect(providerRequestPacingStatus("paced", config.providers.paced).inFlight).toBe(0); + } finally { + override.mockRestore(); outgoing.mockRestore(); clearKeyCooldowns(); forgetApiKeyRotationCursor("paced"); + resetProviderRequestPacingForTest(); releaseOwner(); removeTreeWithRetry(home); + if (previousHome === undefined) delete process.env.OPENCODEX_HOME; else process.env.OPENCODEX_HOME = previousHome; + } +}); From e9e1bcf04eba4fcbdffd897293e008ec2803f562 Mon Sep 17 00:00:00 2001 From: JUN Date: Mon, 12 Oct 2026 05:49:45 +0900 Subject: [PATCH 06/17] fix(pacing): fence queued native keys and exclude readmission from header timeout --- .../src/content/docs/guides/providers.md | 2 +- scripts/test-layout/layout.json | 2 +- src/server/chat-native.ts | 21 +- src/server/responses/compact.ts | 57 ++++- src/server/responses/fetch-helpers.ts | 24 +- structure/transports/responses-failover.md | 2 +- structure/transports/responses.md | 6 +- tests/fixtures/test-layout-expected.json | 2 +- tests/responses/compact-key-selection.test.ts | 218 ++++++++++++++++++ tests/server/fetch-header-timeout.test.ts | 71 ++++++ .../server-chat-native-key-selection.test.ts | 195 ++++++++++++++++ 11 files changed, 577 insertions(+), 23 deletions(-) create mode 100644 tests/responses/compact-key-selection.test.ts create mode 100644 tests/server/server-chat-native-key-selection.test.ts diff --git a/docs-site/src/content/docs/guides/providers.md b/docs-site/src/content/docs/guides/providers.md index 4fefc801591..c8180f60734 100644 --- a/docs-site/src/content/docs/guides/providers.md +++ b/docs-site/src/content/docs/guides/providers.md @@ -93,7 +93,7 @@ credentials must never be replayed on the same token, and local runtimes have no preserve. It is opt-in: when the option is absent the feature is off; object presence enables it unless `enabled: false`. -When round-robin key selection is enabled, queued requests recheck key cooldowns before sending and acquire pacing capacity for any replacement key. This does not change the saved active key. If a key cools after a turn has started, further sends in that turn are refused rather than switching its credential. +When round-robin key selection is enabled, queued requests recheck key cooldowns before sending and acquire pacing capacity for any replacement key. This applies to native Chat, Messages, Responses and compact. Waiting for replacement-key capacity does not consume the upstream response-header timeout. This does not change the saved active key. If a key cools after a turn has started, further sends in that turn are refused rather than switching its credential. ### Which account a request spends diff --git a/scripts/test-layout/layout.json b/scripts/test-layout/layout.json index 4c6900fa657..b24d2b622df 100644 --- a/scripts/test-layout/layout.json +++ b/scripts/test-layout/layout.json @@ -2,7 +2,7 @@ "version": 1, "root": "tests", "explicit": { - "api-key-selection-cooldown.test.ts": "providers", "messages-native-key-selection.test.ts": "claude-integration", "messages-request-id-headers.test.ts": "claude-integration", + "api-key-selection-cooldown.test.ts": "providers", "server-chat-native-key-selection.test.ts": "server", "compact-key-selection.test.ts": "responses", "messages-native-key-selection.test.ts": "claude-integration", "messages-request-id-headers.test.ts": "claude-integration", "messages-request-id-endpoint.test.ts": "claude-integration", "anthropic-revoked-token.test.ts": "adapters/anthropic", "messages-revoked-token.test.ts": "claude-integration", diff --git a/src/server/chat-native.ts b/src/server/chat-native.ts index 2996601352b..fd470e7123f 100644 --- a/src/server/chat-native.ts +++ b/src/server/chat-native.ts @@ -278,8 +278,10 @@ export async function runNativeChatAttempt( const { req, config, logCtx, logIds, route, requestedModel, requestedStream, translatorBudget, finishLog } = execution; const { sendBudget } = execution; const { attempt } = attemptHandle; - const onFirstOutput = execution.onFirstOutput + let outputDelivered = false; + const notifyFirstOutput = execution.onFirstOutput ?? (logIds ? () => recordFirstOutput(logCtx, logIds.start) : undefined); + const onFirstOutput = () => { outputDelivered = true; notifyFirstOutput?.(); }; const fail = (status: number, message: string, type?: string, code?: string | null): Response => { const safeMessage = redactSecretString(message); finishLog(status, safeMessage); @@ -463,9 +465,15 @@ export async function runNativeChatAttempt( onPhysicalDispatch: comboReceipt ? () => { if (!physicalBudget.spendEnforced) settleNonEnforcedSend(request.url); } : undefined, - dispatchOverride: async (_input, init, execute) => { - if (!providerApiKeySelectionIsCurrent(config, route.providerName, activeProvider)) { - const current = resolveCurrentProviderApiKeyTransport(config, route.providerName, activeProvider); + dispatchOverride: async (_input, init, execute, reacquirePacing) => { + for (let attempt = 0; !providerApiKeySelectionIsCurrent(config, route.providerName, activeProvider); attempt++) { + if (outputDelivered) throw new Error("Provider key selection is unavailable during the native Chat turn"); + if (attempt >= 3) throw new Error("Provider key selection changed repeatedly before native Chat dispatch"); + let current = resolveCurrentProviderApiKeyTransport(config, route.providerName, activeProvider); + if (!current && activeProvider._apiKeyAttempt?.distributed) { + const fresh = resolveCurrentProviderApiKeyTransport(config, route.providerName, { ...activeProvider, _apiKeyAttempt: undefined }); + if (fresh) current = selectProactiveApiKeyTransport(config, route.providerName, fresh); + } if (!current || !isNativeChatRouteEligible({ ...route, provider: current }, execution.chatBody, config)) { throw new Error("Provider key selection is no longer available for native Chat"); } @@ -477,6 +485,11 @@ export async function runNativeChatAttempt( activeRequest = buildActiveRequest(); try { retainRequest(activeRequest); } catch (error) { activeRequest.releaseBodyObservation?.(); throw error; } + if (current.requestPacing?.enabled + && (current.requestPacing.keys || current.requestPacing.keySelection === "round-robin")) { + if (!reacquirePacing) throw new Error("Refreshed native Chat dispatch requires pacing readmission"); + await reacquirePacing(current, route.modelId); + } } // The retry closure may still hold a pre-pacing request. Replace its entire // wire shape, not just Authorization, and retain transport recovery flags. diff --git a/src/server/responses/compact.ts b/src/server/responses/compact.ts index 6188c7c7f9d..ed1b4df0a4d 100644 --- a/src/server/responses/compact.ts +++ b/src/server/responses/compact.ts @@ -1,5 +1,5 @@ import { spendLedgerStorageErrorResponse } from "./spend-storage-error"; -import { createPhysicalSendReporter } from "../../lib/request-execution-budget"; +import { createPhysicalSendReporter, rebindPhysicalSend } from "../../lib/request-execution-budget"; import { createInferenceSendBudget } from "../inference/context"; import { unboundPoolSpendRefusalResponse } from "../workflow-refusal"; import { capturePoolQuotaWriter } from "../../codex/account-store"; @@ -152,6 +152,8 @@ import { } from "../request-decompress"; import { resolveAdapter, resolveWireProtocolOverride } from "../adapter-resolve"; import { hasKeyPoolFailover, recordDistributedKeyFailure, rotateProviderTransportOn429, selectProactiveApiKeyTransport } from "../../providers/key-failover"; +import { providerApiKeySelectionIsCurrent, resolveCurrentProviderApiKeyTransport } from "../../providers/api-key-selection"; +import { stampApiKeyAccountLabel } from "../../providers/label"; import { shouldAttemptImageTierRetry } from "../image-retry"; import { resolveProviderTransport } from "../../providers/xai-transport"; import type { WsData } from "../ws-bridge"; @@ -168,6 +170,7 @@ import { catalogModelSupportsServiceTier, finishRequestAttempt, inspectResponseLogJson, + noteProviderAttemptSend, readConfiguredCodexServiceTier, requestLogSpeedLabel, sealRequestAttemptIdentity, @@ -199,7 +202,7 @@ import { upstreamHostCircuitOpenResponse, usesCodexForwardPoolAuth, } from "./core"; -import { fetchWithHeaderTimeout, providerFetch, safeHostLabel, safeOriginLabel } from "./fetch-helpers"; +import { fetchWithHeaderTimeout, providerFetch, safeHostLabel, safeOriginLabel, sendWithConnectionPolicy } from "./fetch-helpers"; import { requestPacingOverloadResponse } from "./pacing-overload"; import { mapCodexAuthContextErrorToResponse, nativeMainRefreshFailureResponse } from "./codex-auth-error"; import { linkRequestSessionLane, sessionLaneIdFromRequest } from "../request-log-conversation"; @@ -915,6 +918,7 @@ export async function handleResponsesCompact( } if (warmKeyProvider?.apiKey) compactProvider = warmKeyProvider; headers.set("authorization", `Bearer ${resolveProviderApiKey(compactProvider.apiKey)}`); + stampApiKeyAccountLabel(logCtx, route.providerName, compactProvider); } const { reasoning: _reasoning, ...compactBodyRaw } = raw as typeof raw & { reasoning?: unknown }; // The regular /v1/responses path applies sanitizeReasoningInputContent via the adapter's @@ -951,9 +955,9 @@ export async function handleResponsesCompact( }); } } - const compactUrl = `${base}/responses/compact`; + let compactUrl = `${base}/responses/compact`; const compactTargetKey = `${route.providerName}|${route.modelId}|compact`; - const actualCompactHostKey = upstreamHostHealthKey( + let actualCompactHostKey = upstreamHostHealthKey( route.providerName, safeOriginLabel(compactUrl), ); @@ -1044,7 +1048,7 @@ export async function handleResponsesCompact( try { if (report.beforeSend?.() === false) throw new SendBudgetExhaustedError(safeHostLabel(compactUrl)); started = true; - return await doFetch(); + return await (report.execute ? report.execute(doFetch) : doFetch()); } finally { try { if (started) report(1); } finally { report.close?.(); } @@ -1055,8 +1059,11 @@ export async function handleResponsesCompact( targetKey: compactTargetKey, }); if (!decision.allowed) return Promise.reject(new SendBudgetExhaustedError(safeHostLabel(compactUrl))); - if (!decision.permit.use()) return Promise.reject(new SendBudgetExhaustedError(safeHostLabel(compactUrl))); - return doFetch(); + const dispatch = () => { + if (!decision.permit.use()) throw new SendBudgetExhaustedError(safeHostLabel(compactUrl)); + return doFetch(); + }; + return decision.permit.execute ? decision.permit.execute(dispatch) : dispatch(); }; const sendCompactAttempt = ( sendProvider: OcxProviderConfig, @@ -1079,6 +1086,40 @@ export async function handleResponsesCompact( modelId: route.modelId, beforeDispatch: isCanonicalOpenAiForwardProvider(sendProvider) ? createCodexAuthDispatchGuard(sendAuthCtx, config, selectedModelId, admission) : undefined, + dispatchOverride: sendProvider.authMode !== "forward" && sendProvider.apiKey + ? async (_input, init, execute, reacquirePacing) => { + for (let attempt = 0; !providerApiKeySelectionIsCurrent(config, route.providerName, sendProvider); attempt++) { + if (attempt >= 3) throw new Error("Provider key selection changed repeatedly before compact dispatch"); + let current = resolveCurrentProviderApiKeyTransport(config, route.providerName, sendProvider); + if (!current && sendProvider._apiKeyAttempt?.distributed) { + const fresh = resolveCurrentProviderApiKeyTransport(config, route.providerName, { ...sendProvider, _apiKeyAttempt: undefined }); + if (fresh) current = selectProactiveApiKeyTransport(config, route.providerName, fresh); + } + if (!current?.apiKey || !supportsNativeResponsesCompactEndpoint(route.providerName, current)) { + throw new Error("Provider key selection is no longer available for compact"); + } + const rebuilt = new Headers(sendHeaders); + for (const name of Object.keys(sendProvider.headers ?? {})) rebuilt.delete(name); + for (const [name, value] of Object.entries(current.headers ?? {})) rebuilt.set(name, value); + rebuilt.set("authorization", `Bearer ${resolveProviderApiKey(current.apiKey)}`); + sendHeaders = headers = rebuilt; + sendProvider = compactProvider = current; + compactUrl = `${(current.baseUrl ?? "").replace(/\/+$/, "")}/responses/compact`; + actualCompactHostKey = upstreamHostHealthKey(route.providerName, safeOriginLabel(compactUrl)); + if (current.requestPacing?.enabled + && (current.requestPacing.keys || current.requestPacing.keySelection === "round-robin")) { + if (!reacquirePacing) throw new Error("Refreshed compact dispatch requires pacing readmission"); + await reacquirePacing(current, route.modelId); + } + } + if (req.signal.aborted) throw req.signal.reason; + stampApiKeyAccountLabel(logCtx, route.providerName, sendProvider); + if (!rebindPhysicalSend(sendBudget, { poolId: logCtx.spendPoolId ?? route.providerName, + identityId: logCtx.accountLogLabel })) throw new SendBudgetExhaustedError(safeHostLabel(compactUrl)); + noteProviderAttemptSend(logCtx, route.providerName, sendProvider, logCtx.usageLogInputTokens, upstreamRecovery); + return sendWithConnectionPolicy((sendProvider as { fetch?: typeof fetch }).fetch ?? execute, compactUrl, + { ...init, headers: sendHeaders }, { providerName: route.providerName, provider: sendProvider }); + } : undefined, }), // Every credential-bearing forward send gets manual redirects, not only // pool sends: direct mode carries the caller's credential too (#914). @@ -1107,7 +1148,7 @@ export async function handleResponsesCompact( let outcomeCtx = authCtx; const localDispatchRefusal = (error: unknown): Response | undefined => { const cause = unwrapUpstreamRetryEvidenceError(error); - const response = spendLedgerStorageErrorResponse(error, logCtx) ?? (cause instanceof SendBudgetExhaustedError + const response = requestPacingOverloadResponse(cause) ?? spendLedgerStorageErrorResponse(error, logCtx) ?? (cause instanceof SendBudgetExhaustedError ? unboundPoolSpendRefusalResponse(logCtx) ?? formatErrorResponse(429, "request_send_budget_exhausted", cause.message) : mapCodexAuthContextErrorToResponse(cause, { now: Date.now(), accountSelector: route.codexAccountNamespace, diff --git a/src/server/responses/fetch-helpers.ts b/src/server/responses/fetch-helpers.ts index dc29e97af5c..bd2b6fb0c60 100644 --- a/src/server/responses/fetch-helpers.ts +++ b/src/server/responses/fetch-helpers.ts @@ -49,6 +49,9 @@ const UPSTREAM_REWRITTEN = Symbol.for("opencodex.plugins.upstream-rewritten"); const PHYSICAL_DISPATCH = Symbol.for("opencodex.physical-dispatch"); // One WS attempt and its HTTP fallback are one logical physical-send receipt. const WS_FALLBACK_RECEIPT = Symbol.for("opencodex.ws-fallback-receipt"); +// Replacement admission waits on the client signal, outside the response-header deadline. +const HEADER_DEADLINE = Symbol("header-deadline"); +type HeaderDeadline = { clientSignal: AbortSignal; pause(): void; resume(): void }; /** * Announce once, per provider, that an explicit egress route moved this provider off the @@ -396,7 +399,11 @@ export function providerFetch( if (!options.providerName) throw new Error("Refreshed dispatch requires a named pacing owner"); slot?.release(); slot = undefined; - slot = await waitForProviderRequestSlot(providerName, refreshed, modelId, init?.signal ?? undefined); + const deadline = (init as Record | undefined)?.[HEADER_DEADLINE]; + deadline?.pause(); + try { + slot = await waitForProviderRequestSlot(providerName, refreshed, modelId, deadline?.clientSignal ?? init?.signal ?? undefined); + } finally { deadline?.resume(); } }; try { const response = await unpaced(input, init, reacquirePacing); @@ -496,9 +503,17 @@ export async function fetchWithHeaderTimeout( let timer: ReturnType | undefined; try { const timeout = new AbortController(); - timer = setTimeout(() => { - if (!timeout.signal.aborted) timeout.abort(new DOMException("Timeout elapsed", "TimeoutError")); - }, timeoutMs); + const deadline: HeaderDeadline = { + clientSignal: abortSignal, + pause() { if (timer) clearTimeout(timer); timer = undefined; }, + resume() { + this.pause(); + timer = setTimeout(() => { + if (!timeout.signal.aborted) timeout.abort(new DOMException("Timeout elapsed", "TimeoutError")); + }, timeoutMs); + }, + }; + deadline.resume(); const headers = new Headers(init.headers); // Compressed SSE can be held until the decompressor has a complete block. Streaming calls // default to identity for low-latency frame delivery, while an explicit caller choice wins. @@ -512,6 +527,7 @@ export async function fetchWithHeaderTimeout( // Preserve the 3xx for the owner's existing response/health policy (#914, #1471). redirect: "manual" as const, signal: AbortSignal.any([abortSignal, timeout.signal]), + [HEADER_DEADLINE]: deadline, timeout: 0, }; return pacing.fetchWithPacingSlot diff --git a/structure/transports/responses-failover.md b/structure/transports/responses-failover.md index e7c6d78d35a..805126202eb 100644 --- a/structure/transports/responses-failover.md +++ b/structure/transports/responses-failover.md @@ -156,7 +156,7 @@ The shared Responses path follows the [bounded multipart recovery contract](../s ## Upstream key attempt accounting -Key identity is sealed at guarded physical dispatch after queued selections are rebuilt. Distributed selections retain their request-local entry and revision. A rebuild after selection drift releases the old per-key lease and reacquires admission for the refreshed key; the send guard checks again after waiting. Turn-scoped adapters follow the same rule before invocation. Unpaced dispatch without a readmission owner fails closed. Replay notification errors release acquired slots before propagating, including pre-acquired-slot dispatch. +Key identity is sealed at guarded physical dispatch after queued selections are rebuilt. Distributed selections retain their request-local entry and revision. A rebuild after selection drift releases the old per-key lease and reacquires admission for the refreshed key; the send guard checks again after waiting. Turn-scoped adapters follow the same rule before invocation. Native Chat, Messages and keyed compact all rebuild stale queued selections before dispatch. Replacement admission pauses the response-header deadline and retains client cancellation; the header deadline restarts once admission completes. Unpaced dispatch without a readmission owner fails closed. Replay notification errors release acquired slots before propagating, including pre-acquired-slot dispatch. Raw adapter terminal usage is recorded before continuation, search, or image loops merge it; repeated parsing of one physical response does not count it twice. Key changes preserve the previous attempt while retaining the active attempt object shared by streaming/combo callbacks. diff --git a/structure/transports/responses.md b/structure/transports/responses.md index 1608a832257..2c86cadb020 100644 --- a/structure/transports/responses.md +++ b/structure/transports/responses.md @@ -51,11 +51,11 @@ performs the physical send, after a dispatch override has selected or rebuilt th matching follows the URL sent on the wire rather than the URL supplied before credential revalidation; prepaid initial-send receipts follow the [spend contract](responses-spend.md#prepaid-initial-sends). At this final HTTP boundary, native ChatGPT Responses and compact JSON strings of at least 1 MiB (UTF-8) become byte buffers to avoid Bun's large-string upload resets. Content, headers, abort signals and retry policy are preserved; WebSocket selection still receives the original string. Other destinations, small strings and existing byte/stream bodies retain their representation. -The wrapped executor alone is not that boundary. An override that revalidates credentials re-reads +Replacement-key pacing admission retains client cancellation but pauses the header deadline; the full header interval restarts after admission. The wrapped executor alone is not that boundary. An override that revalidates credentials re-reads `route.provider.fetch` at send time, because reselection can install a different provider transport after the wrapper was built, and then calls that implementation instead of the executor. Both -production overrides do this -- `oauthDispatch` in `request-transport.ts` and the native Chat -key-revalidation override in `chat-native.ts` -- so both wrap the selected implementation rather +production overrides do this -- `oauthDispatch` in `request-transport.ts`, native Chat in +`chat-native.ts`, native Messages and keyed compact -- so they wrap the selected implementation rather than choosing between policy and provider transport. Reporting the executor as the boundary while the code let a provider-scoped transport past it is what #4992 recorded, and it is why a regression for this policy has to enter through `handleResponses` rather than through a diff --git a/tests/fixtures/test-layout-expected.json b/tests/fixtures/test-layout-expected.json index 0fb9baba590..528af9b8210 100644 --- a/tests/fixtures/test-layout-expected.json +++ b/tests/fixtures/test-layout-expected.json @@ -1,5 +1,5 @@ { - "api-key-selection-cooldown.test.ts": "providers", + "api-key-selection-cooldown.test.ts": "providers", "server-chat-native-key-selection.test.ts": "server", "compact-key-selection.test.ts": "responses", "messages-native-key-selection.test.ts": "claude-integration", "messages-request-id-headers.test.ts": "claude-integration", "messages-request-id-endpoint.test.ts": "claude-integration", diff --git a/tests/responses/compact-key-selection.test.ts b/tests/responses/compact-key-selection.test.ts new file mode 100644 index 00000000000..a1f542c2a7d --- /dev/null +++ b/tests/responses/compact-key-selection.test.ts @@ -0,0 +1,218 @@ +import { afterEach, beforeEach, expect, test } from "bun:test"; +import { mkdtempSync, readFileSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { saveConfig } from "../../src/config"; +import { apiKeyAccountLogLabel } from "../../src/codex/account-label"; +import { captureProviderApiKeySelection } from "../../src/providers/api-key-selection"; +import { clearKeyCooldowns, forgetApiKeyRotationCursor, getKeyCooldownUntil, recordDistributedKeyFailure } from "../../src/providers/key-failover"; +import { providerRequestPacingStatus, resetProviderRequestPacingForTest, waitForProviderRequestSlot } from "../../src/providers/request-pacing"; +import { handleResponsesCompact } from "../../src/server/responses"; +import type { RequestLogContext } from "../../src/server/request-log"; +import type { OcxConfig } from "../../src/types"; +import { acquireOwnedSpendHome } from "../helpers/owned-spend-home"; +import { removeCompactionFixture } from "../helpers/compaction-routing-fixtures"; + +const NAME = "openai-apikey"; +const KEYS = ["fixture-key-alpha", "fixture-key-beta", "fixture-key-gamma"]; +const realFetch = globalThis.fetch; +let home: string; +let previousHome: string | undefined; +let releaseOwner: () => void; +let seen: { key: string | null; url: string; body: unknown; headers: Headers }[]; +beforeEach(() => { + previousHome = process.env.OPENCODEX_HOME; + home = mkdtempSync(join(tmpdir(), "ocx-compact-key-pacing-")); + process.env.OPENCODEX_HOME = home; + releaseOwner = acquireOwnedSpendHome(); + seen = []; + clearKeyCooldowns(); forgetApiKeyRotationCursor(); resetProviderRequestPacingForTest(); + globalThis.fetch = (async (input, init) => { + seen.push({ key: new Headers(init?.headers).get("authorization"), url: String(input), body: JSON.parse(String(init?.body)), headers: new Headers(init?.headers) }); + return Response.json({ output: [{ type: "message", role: "assistant", content: [{ type: "output_text", text: "fixture reply" }] }] }); + }) as typeof fetch; +}); +afterEach(async () => { + globalThis.fetch = realFetch; + releaseOwner(); + clearKeyCooldowns(); forgetApiKeyRotationCursor(); resetProviderRequestPacingForTest(); + await removeCompactionFixture(home); + if (previousHome === undefined) delete process.env.OPENCODEX_HOME; + else process.env.OPENCODEX_HOME = previousHome; +}); +function fixture(keyCap = true): OcxConfig { + const config = { port: 0, defaultProvider: NAME, providers: { [NAME]: { + adapter: "openai-responses", baseUrl: "https://api.openai.com/v1", authMode: "key", + apiKey: KEYS[0], apiKeySelectionRevision: "saved-revision", + apiKeyPool: KEYS.map((key, index) => ({ id: `k${index + 1}`, key, addedAt: index + 1 })), + requestPacing: { enabled: true, keySelection: "round-robin", + ...(keyCap ? { keys: { maxConcurrentRequests: 1 } } : { maxConcurrentRequests: 1 }) }, + } } } as OcxConfig; + saveConfig(config); + return config; +} +function status(config: OcxConfig) { return providerRequestPacingStatus(NAME, config.providers[NAME]!); } +function keyStatus(config: OcxConfig, id: string) { return status(config).keys!.find(key => key.id === id)!; } +function cool(config: OcxConfig, index: number) { + const selection = { ...captureProviderApiKeySelection({ ...config.providers[NAME]!, apiKey: KEYS[index] }), distributed: true }; + expect(recordDistributedKeyFailure(config, NAME, selection, 429, Date.now(), "30")).toBe(true); +} +async function queue(config: OcxConfig, id: string) { + const deadline = Date.now() + 2_000; + while (keyStatus(config, id).queued !== 1) { + if (Date.now() >= deadline) throw new Error(`key ${id} never queued: ${JSON.stringify(status(config))}`); + await Bun.sleep(5); + } +} +function start(config: OcxConfig, abort: AbortController) { + const logCtx: RequestLogContext = { model: "", provider: "" }; + const req = new Request("http://localhost/v1/responses/compact", { method: "POST", headers: { "content-type": "application/json" }, + signal: AbortSignal.any([abort.signal, AbortSignal.timeout(5_000)]), body: JSON.stringify({ model: `${NAME}/test`, input: [{ role: "user", content: "fixture question" }], stream: false }) }); + const pending = handleResponsesCompact(req, config, logCtx).then(async response => ({ response, text: await response.text(), logCtx })); + void pending.catch(() => {}); + return pending; +} +function unchanged(config: OcxConfig) { + expect(config.providers[NAME]!.apiKey).toBe(KEYS[0]); + expect(config.providers[NAME]!.apiKeySelectionRevision).toBe("saved-revision"); + const saved = JSON.parse(readFileSync(join(home, "config.json"), "utf8")) as OcxConfig; + expect(saved.providers[NAME]!.apiKey).toBe(KEYS[0]); + expect(saved.providers[NAME]!.apiKeySelectionRevision).toBe("saved-revision"); + expect(saved.providers[NAME]!._apiKeyAttempt).toBeUndefined(); +} +function noLeaks(config: OcxConfig) { + expect(status(config)).toMatchObject({ queued: 0, inFlight: 0 }); + for (const key of status(config).keys!) expect(key).toMatchObject({ queued: 0, inFlight: 0 }); +} +function success(result: Awaited>, index: number) { + expect(result.response.status).toBe(200); + expect(result.text).toContain("fixture reply"); + expect(seen.map(entry => entry.key)).toEqual([`Bearer ${KEYS[index]}`]); + expect(seen[0]!.url).toBe("https://api.openai.com/v1/responses/compact"); + expect(result.logCtx.accountLogLabel).toBe(apiKeyAccountLogLabel(NAME, { entryId: `k${index + 1}`, reference: KEYS[index] })); +} +for (const cancel of [false, true]) test(`compact cooled queued A readmits B (abort=${cancel})`, async () => { + const config = fixture(); + const first = await waitForProviderRequestSlot(NAME, config.providers[NAME]!); + const second = await waitForProviderRequestSlot(NAME, { ...config.providers[NAME]!, apiKey: KEYS[1] }); + const abort = new AbortController(); + const pending = start(config, abort); + try { + await queue(config, "k1"); cool(config, 0); first.release(); + await queue(config, "k2"); + expect(keyStatus(config, "k1")).toMatchObject({ inFlight: 0, queued: 0, starts: 2 }); + expect(keyStatus(config, "k2")).toMatchObject({ inFlight: 1, queued: 1, starts: 1 }); + expect(seen).toEqual([]); + if (cancel) { + abort.abort(); + const result = await pending; + expect(result.response.status).toBe(499); + expect(seen).toEqual([]); + expect(status(config).queued).toBe(0); + second.release(); + } else { second.release(); success(await pending, 1); } + unchanged(config); noLeaks(config); + } finally { abort.abort(); first.release(); second.release(); await pending.catch(() => undefined); } +}, 10_000); +for (const exhausted of [false, true]) test(`compact revalidates B cooling during readmission (exhausted=${exhausted})`, async () => { + const config = fixture(); + const first = await waitForProviderRequestSlot(NAME, config.providers[NAME]!); + const second = await waitForProviderRequestSlot(NAME, { ...config.providers[NAME]!, apiKey: KEYS[1] }); + const abort = new AbortController(); + const pending = start(config, abort); + try { + await queue(config, "k1"); cool(config, 0); first.release(); + await queue(config, "k2"); cool(config, 1); + if (exhausted) cool(config, 2); + second.release(); + const result = await pending; + if (exhausted) { + expect(result.response.status).toBe(429); + expect(Number(result.response.headers.get("retry-after"))).toBeGreaterThan(0); + expect(seen).toEqual([]); + } else success(result, 2); + unchanged(config); noLeaks(config); + } finally { abort.abort(); first.release(); second.release(); await pending.catch(() => undefined); } +}, 10_000); +test("compact reselects behind the provider cap without a keys rule", async () => { + const config = fixture(false); + const held = await waitForProviderRequestSlot(NAME, config.providers[NAME]!); + const abort = new AbortController(); + const pending = start(config, abort); + try { + await queue(config, "k1"); cool(config, 0); held.release(); + success(await pending, 1); + expect(keyStatus(config, "k1").starts).toBe(2); + expect(keyStatus(config, "k2").starts).toBe(1); + unchanged(config); noLeaks(config); + } finally { abort.abort(); held.release(); await pending.catch(() => undefined); } +}); +test("compact refreshes a queued stale revision and sends rebuilt Authorization and static headers", async () => { + const config = fixture(); + const held = await waitForProviderRequestSlot(NAME, config.providers[NAME]!); + const abort = new AbortController(); + const pending = start(config, abort); + try { + await queue(config, "k1"); + config.providers[NAME]!.apiKeySelectionRevision = "updated-revision"; + config.providers[NAME]!.headers = { "x-current-static": "replacement" }; + held.release(); + const result = await pending; + expect(result.response.status).toBe(200); + expect(seen).toHaveLength(1); + expect(seen[0]!.url).toBe("https://api.openai.com/v1/responses/compact"); + expect(seen[0]!.key).toBe(`Bearer ${KEYS[1]}`); + expect(seen[0]!.headers.get("x-current-static")).toBe("replacement"); + expect(result.logCtx.accountLogLabel).toBe(apiKeyAccountLogLabel(NAME, { entryId: "k2", reference: KEYS[1] })); + noLeaks(config); + } finally { abort.abort(); held.release(); await pending.catch(() => undefined); } +}); + +test("compact refuses after three queued rebuilds without spending a physical send", async () => { + const config = fixture(); + const extra = ["fixture-key-delta", "fixture-key-epsilon"]; + config.providers[NAME]!.apiKeyPool!.push(...extra.map((key, index) => ({ id: `k${index + 4}`, key }))); + const keys = [...KEYS, ...extra]; + const held = await Promise.all(keys.map(apiKey => waitForProviderRequestSlot(NAME, { ...config.providers[NAME]!, apiKey }))); + const abort = new AbortController(); + const pending = start(config, abort); + try { + for (let index = 0; index < 4; index++) { + await queue(config, `k${index + 1}`); + const selection = { ...captureProviderApiKeySelection({ ...config.providers[NAME]!, apiKey: keys[index] }), distributed: true }; + expect(recordDistributedKeyFailure(config, NAME, selection, 429, Date.now(), "30")).toBe(true); + held[index]!.release(); + } + const result = await pending; + expect(result.response.status).toBe(502); + expect(seen).toEqual([]); + expect(keyStatus(config, "k5").starts).toBe(1); + for (const slot of held) slot.release(); + unchanged(config); noLeaks(config); + } finally { abort.abort(); for (const slot of held) slot.release(); await pending.catch(() => undefined); } +}, 10_000); + +test("compact 401 after queued reselection cools the physical key B without replaying", async () => { + const config = fixture(); + const first = await waitForProviderRequestSlot(NAME, config.providers[NAME]!); + const abort = new AbortController(); + const wireFetch = globalThis.fetch; + globalThis.fetch = (async (input, init) => { + await wireFetch(input, init); + return Response.json({ error: { message: "invalid fixture key" } }, { status: 401 }); + }) as typeof fetch; + const pending = start(config, abort); + try { + await queue(config, "k1"); cool(config, 0); + const firstCooldown = getKeyCooldownUntil(NAME, "k1"); + first.release(); + const result = await pending; + expect(result.response.status).toBe(401); + expect(seen.map(entry => entry.key)).toEqual([`Bearer ${KEYS[1]}`]); + expect(result.logCtx.accountLogLabel).toBe(apiKeyAccountLogLabel(NAME, { entryId: "k2", reference: KEYS[1] })); + expect(getKeyCooldownUntil(NAME, "k1")).toBe(firstCooldown); + expect(getKeyCooldownUntil(NAME, "k2")).not.toBeNull(); + expect(getKeyCooldownUntil(NAME, "k3")).toBeNull(); + unchanged(config); noLeaks(config); + } finally { abort.abort(); first.release(); await pending.catch(() => undefined); } +}); diff --git a/tests/server/fetch-header-timeout.test.ts b/tests/server/fetch-header-timeout.test.ts index 1ce097def08..b920c4f3ce8 100644 --- a/tests/server/fetch-header-timeout.test.ts +++ b/tests/server/fetch-header-timeout.test.ts @@ -189,3 +189,74 @@ describe("#2567 the upstream fetch disables Bun's per-request idle timeout", () expect(init.signal).toBeInstanceOf(AbortSignal); }); }); + +describe("replacement key admission excludes the response-header deadline", () => { + async function replacementFixture(physicalFetch: typeof fetch) { + const { providerFetch } = await import("../../src/server/responses/fetch-helpers"); + const { waitForProviderRequestSlot, resetProviderRequestPacingForTest } = await import("../../src/providers/request-pacing"); + resetProviderRequestPacingForTest(); + const provider = { adapter: "openai-chat", baseUrl: "https://fixture.test/v1", authMode: "key", apiKey: "fixture-A", + apiKeyPool: [{ id: "a", key: "fixture-A", addedAt: 1 }, { id: "b", key: "fixture-B", addedAt: 2 }], + requestPacing: { enabled: true, keys: { maxConcurrentRequests: 1 } } } as import("../../src/types").OcxProviderConfig; + const replacement = { ...provider, apiKey: "fixture-B" }; + const held = await waitForProviderRequestSlot("header-replacement", replacement); + let notify!: () => void; + const queued = new Promise(resolve => { notify = resolve; }); + const executor = providerFetch({ ...provider, fetch: physicalFetch } as typeof provider, undefined, { providerName: "header-replacement", + dispatchOverride: async (input, init, execute, reacquire) => { + notify(); + await reacquire!(replacement); + return execute(input, init); + } }); + return { executor, held, queued, resetProviderRequestPacingForTest }; + } + + test("a blocked replacement may wait beyond the header timeout and then send", async () => { + let sends = 0; + const fixture = await replacementFixture((async () => { sends++; return new Response("ok"); }) as typeof fetch); + const pending = fetchWithHeaderTimeout("https://fixture.test/v1/chat/completions", {}, new AbortController().signal, 25, false, fixture.executor); + void pending.catch(() => {}); + try { + await fixture.queued; + await Bun.sleep(80); + expect(sends).toBe(0); + fixture.held.release(); + expect(await (await pending).text()).toBe("ok"); + expect(sends).toBe(1); + } finally { fixture.held.release(); fixture.resetProviderRequestPacingForTest(); } + }); + + test("the fresh header deadline still aborts a stalled physical send", async () => { + let sends = 0; + const fixture = await replacementFixture((async (_input, init) => { + sends++; + return new Promise((_resolve, reject) => { + if (init?.signal?.aborted) reject(init.signal.reason); + else init?.signal?.addEventListener("abort", () => reject(init.signal!.reason), { once: true }); + }); + }) as typeof fetch); + const pending = fetchWithHeaderTimeout("https://fixture.test/v1/chat/completions", {}, new AbortController().signal, 25, false, fixture.executor); + void pending.catch(() => {}); + try { + await fixture.queued; + await Bun.sleep(80); + fixture.held.release(); + await expect(pending).rejects.toMatchObject({ name: "TimeoutError" }); + expect(sends).toBe(1); + } finally { fixture.held.release(); fixture.resetProviderRequestPacingForTest(); } + }); + + test("client abort remains active while replacement admission waits", async () => { + let sends = 0; + const fixture = await replacementFixture((async () => { sends++; return new Response("ok"); }) as typeof fetch); + const controller = new AbortController(); + const pending = fetchWithHeaderTimeout("https://fixture.test/v1/chat/completions", {}, controller.signal, 25, false, fixture.executor); + void pending.catch(() => {}); + try { + await fixture.queued; + controller.abort(new DOMException("client cancelled", "AbortError")); + await expect(pending).rejects.toMatchObject({ name: "AbortError" }); + expect(sends).toBe(0); + } finally { fixture.held.release(); fixture.resetProviderRequestPacingForTest(); } + }); +}); diff --git a/tests/server/server-chat-native-key-selection.test.ts b/tests/server/server-chat-native-key-selection.test.ts new file mode 100644 index 00000000000..25d5c3ec244 --- /dev/null +++ b/tests/server/server-chat-native-key-selection.test.ts @@ -0,0 +1,195 @@ +import { afterEach, beforeEach, expect, test } from "bun:test"; +import { mkdtempSync, readFileSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { saveConfig } from "../../src/config"; +import { apiKeyAccountLogLabel } from "../../src/codex/account-label"; +import { captureProviderApiKeySelection } from "../../src/providers/api-key-selection"; +import { clearKeyCooldowns, forgetApiKeyRotationCursor, recordDistributedKeyFailure } from "../../src/providers/key-failover"; +import { providerRequestPacingStatus, resetProviderRequestPacingForTest, waitForProviderRequestSlot } from "../../src/providers/request-pacing"; +import { handleChatCompletions } from "../../src/server/chat-completions"; +import type { RequestLogContext } from "../../src/server/request-log"; +import type { OcxConfig } from "../../src/types"; +import { acquireOwnedSpendHome } from "../helpers/owned-spend-home"; +import { removeCompactionFixture } from "../helpers/compaction-routing-fixtures"; + +const NAME = "native-chat-pacing"; +const KEYS = ["fixture-key-alpha", "fixture-key-beta", "fixture-key-gamma"]; +const realFetch = globalThis.fetch; +let home: string; +let previousHome: string | undefined; +let releaseOwner: () => void; +let seen: { key: string | null; url: string; body: unknown }[]; +beforeEach(() => { + previousHome = process.env.OPENCODEX_HOME; + home = mkdtempSync(join(tmpdir(), "ocx-chat-key-pacing-")); + process.env.OPENCODEX_HOME = home; + releaseOwner = acquireOwnedSpendHome(); + seen = []; + clearKeyCooldowns(); forgetApiKeyRotationCursor(); resetProviderRequestPacingForTest(); + globalThis.fetch = (async (input, init) => { + seen.push({ key: new Headers(init?.headers).get("authorization"), url: String(input), body: JSON.parse(String(init?.body)) }); + return Response.json({ id: "chatcmpl-key-selection", object: "chat.completion", choices: [{ index: 0, message: { role: "assistant", content: "fixture reply" }, finish_reason: "stop" }] }); + }) as typeof fetch; +}); +afterEach(async () => { + globalThis.fetch = realFetch; + releaseOwner(); + clearKeyCooldowns(); forgetApiKeyRotationCursor(); resetProviderRequestPacingForTest(); + await removeCompactionFixture(home); + if (previousHome === undefined) delete process.env.OPENCODEX_HOME; + else process.env.OPENCODEX_HOME = previousHome; +}); +function fixture(keyCap = true): OcxConfig { + const config = { port: 0, defaultProvider: NAME, providers: { [NAME]: { + adapter: "openai-chat", baseUrl: "https://gateway.example/v1", authMode: "key", + apiKey: KEYS[0], apiKeySelectionRevision: "saved-revision", + apiKeyPool: KEYS.map((key, index) => ({ id: `k${index + 1}`, key, addedAt: index + 1 })), + requestPacing: { enabled: true, keySelection: "round-robin", + ...(keyCap ? { keys: { maxConcurrentRequests: 1 } } : { maxConcurrentRequests: 1 }) }, + } } } as OcxConfig; + saveConfig(config); + return config; +} +function status(config: OcxConfig) { return providerRequestPacingStatus(NAME, config.providers[NAME]!); } +function keyStatus(config: OcxConfig, id: string) { return status(config).keys!.find(key => key.id === id)!; } +function cool(config: OcxConfig, index: number) { + const selection = { ...captureProviderApiKeySelection({ ...config.providers[NAME]!, apiKey: KEYS[index] }), distributed: true }; + expect(recordDistributedKeyFailure(config, NAME, selection, 429, Date.now(), "30")).toBe(true); +} +async function queue(config: OcxConfig, id: string) { + const deadline = Date.now() + 2_000; + while (keyStatus(config, id).queued !== 1) { + if (Date.now() >= deadline) throw new Error(`key ${id} never queued: ${JSON.stringify(status(config))}`); + await Bun.sleep(5); + } +} +function start(config: OcxConfig, abort: AbortController) { + const logCtx: RequestLogContext = { model: "", provider: "" }; + const req = new Request("http://localhost/v1/chat/completions", { method: "POST", headers: { "content-type": "application/json" }, + signal: AbortSignal.any([abort.signal, AbortSignal.timeout(5_000)]), body: JSON.stringify({ model: `${NAME}/test`, messages: [{ role: "user", content: "fixture question" }], stream: false }) }); + const pending = handleChatCompletions(req, config, logCtx).then(async response => ({ response, text: await response.text(), logCtx })); + void pending.catch(() => {}); + return pending; +} +function unchanged(config: OcxConfig) { + expect(config.providers[NAME]!.apiKey).toBe(KEYS[0]); + expect(config.providers[NAME]!.apiKeySelectionRevision).toBe("saved-revision"); + const saved = JSON.parse(readFileSync(join(home, "config.json"), "utf8")) as OcxConfig; + expect(saved.providers[NAME]!.apiKey).toBe(KEYS[0]); + expect(saved.providers[NAME]!.apiKeySelectionRevision).toBe("saved-revision"); + expect(saved.providers[NAME]!._apiKeyAttempt).toBeUndefined(); +} +function noLeaks(config: OcxConfig) { + expect(status(config)).toMatchObject({ queued: 0, inFlight: 0 }); + for (const key of status(config).keys!) expect(key).toMatchObject({ queued: 0, inFlight: 0 }); +} +function success(result: Awaited>, index: number) { + expect(result.response.status).toBe(200); + expect(result.text).toContain("fixture reply"); + expect(seen.map(entry => entry.key)).toEqual([`Bearer ${KEYS[index]}`]); + expect(seen[0]!.url).toBe("https://gateway.example/v1/chat/completions"); + expect(result.logCtx.accountLogLabel).toBe(apiKeyAccountLogLabel(NAME, { entryId: `k${index + 1}`, reference: KEYS[index] })); + expect(result.logCtx.attempts).toHaveLength(1); + expect(result.logCtx.attempts![0]!.sendCount).toBe(1); + expect(result.logCtx.attempts![0]!.accountLogLabel).toBe(result.logCtx.accountLogLabel); +} +for (const cancel of [false, true]) test(`chat cooled queued A readmits B (abort=${cancel})`, async () => { + const config = fixture(); + const first = await waitForProviderRequestSlot(NAME, config.providers[NAME]!); + const second = await waitForProviderRequestSlot(NAME, { ...config.providers[NAME]!, apiKey: KEYS[1] }); + const abort = new AbortController(); + const pending = start(config, abort); + try { + await queue(config, "k1"); cool(config, 0); first.release(); + await queue(config, "k2"); + expect(keyStatus(config, "k1")).toMatchObject({ inFlight: 0, queued: 0, starts: 2 }); + expect(keyStatus(config, "k2")).toMatchObject({ inFlight: 1, queued: 1, starts: 1 }); + expect(seen).toEqual([]); + if (cancel) { + abort.abort(); + const result = await pending; + expect(result.response.status).toBe(499); + expect(seen).toEqual([]); + expect(status(config).queued).toBe(0); + second.release(); + } else { second.release(); success(await pending, 1); } + unchanged(config); noLeaks(config); + } finally { abort.abort(); first.release(); second.release(); await pending.catch(() => undefined); } +}, 10_000); +for (const exhausted of [false, true]) test(`chat revalidates B cooling during readmission (exhausted=${exhausted})`, async () => { + const config = fixture(); + const first = await waitForProviderRequestSlot(NAME, config.providers[NAME]!); + const second = await waitForProviderRequestSlot(NAME, { ...config.providers[NAME]!, apiKey: KEYS[1] }); + const abort = new AbortController(); + const pending = start(config, abort); + try { + await queue(config, "k1"); cool(config, 0); first.release(); + await queue(config, "k2"); cool(config, 1); + if (exhausted) cool(config, 2); + second.release(); + const result = await pending; + if (exhausted) { + expect(result.response.status).toBe(429); + expect(Number(result.response.headers.get("retry-after"))).toBeGreaterThan(0); + expect(seen).toEqual([]); + } else success(result, 2); + unchanged(config); noLeaks(config); + } finally { abort.abort(); first.release(); second.release(); await pending.catch(() => undefined); } +}, 10_000); +test("chat reselects behind the provider cap without a keys rule", async () => { + const config = fixture(false); + const held = await waitForProviderRequestSlot(NAME, config.providers[NAME]!); + const abort = new AbortController(); + const pending = start(config, abort); + try { + await queue(config, "k1"); cool(config, 0); held.release(); + success(await pending, 1); + expect(keyStatus(config, "k1").starts).toBe(2); + expect(keyStatus(config, "k2").starts).toBe(1); + unchanged(config); noLeaks(config); + } finally { abort.abort(); held.release(); await pending.catch(() => undefined); } +}); +test("chat refreshes a queued stale revision and sends the rebuilt destination", async () => { + const config = fixture(); + const held = await waitForProviderRequestSlot(NAME, config.providers[NAME]!); + const abort = new AbortController(); + const pending = start(config, abort); + try { + await queue(config, "k1"); + config.providers[NAME]!.apiKeySelectionRevision = "updated-revision"; + config.providers[NAME]!.baseUrl = "https://replacement.example/v1"; + held.release(); + const result = await pending; + expect(result.response.status).toBe(200); + expect(seen).toHaveLength(1); + expect(seen[0]!.url).toBe("https://replacement.example/v1/chat/completions"); + expect(seen[0]!.key).toBe(`Bearer ${KEYS[1]}`); + expect(result.logCtx.accountLogLabel).toBe(apiKeyAccountLogLabel(NAME, { entryId: "k2", reference: KEYS[1] })); + noLeaks(config); + } finally { abort.abort(); held.release(); await pending.catch(() => undefined); } +}); + +test("chat refuses after three queued rebuilds without spending a physical send", async () => { + const config = fixture(); + const extra = ["fixture-key-delta", "fixture-key-epsilon"]; + config.providers[NAME]!.apiKeyPool!.push(...extra.map((key, index) => ({ id: `k${index + 4}`, key }))); + const keys = [...KEYS, ...extra]; + const held = await Promise.all(keys.map(apiKey => waitForProviderRequestSlot(NAME, { ...config.providers[NAME]!, apiKey }))); + const abort = new AbortController(); + const pending = start(config, abort); + try { + for (let index = 0; index < 4; index++) { + await queue(config, `k${index + 1}`); + const selection = { ...captureProviderApiKeySelection({ ...config.providers[NAME]!, apiKey: keys[index] }), distributed: true }; + expect(recordDistributedKeyFailure(config, NAME, selection, 429, Date.now(), "30")).toBe(true); + held[index]!.release(); + } + const result = await pending; + expect(result.response.status).toBe(502); + expect(seen).toEqual([]); + expect(keyStatus(config, "k5").starts).toBe(1); + for (const slot of held) slot.release(); + unchanged(config); noLeaks(config); + } finally { abort.abort(); for (const slot of held) slot.release(); await pending.catch(() => undefined); } +}, 10_000); From 02c8296b41538a3b6316def0220de32c2996cdbe Mon Sep 17 00:00:00 2001 From: JUN Date: Mon, 12 Oct 2026 05:52:46 +0900 Subject: [PATCH 07/17] fix(keys): preserve the longest concurrent failure cooldown --- src/providers/api-key-cooldowns.ts | 3 ++- structure/providers-and-adapters.md | 2 +- .../api-key-selection-cooldown.test.ts | 23 +++++++++++++++++++ 3 files changed, 26 insertions(+), 2 deletions(-) diff --git a/src/providers/api-key-cooldowns.ts b/src/providers/api-key-cooldowns.ts index 378b10cb6ac..42b00424ee3 100644 --- a/src/providers/api-key-cooldowns.ts +++ b/src/providers/api-key-cooldowns.ts @@ -6,7 +6,8 @@ function cooldownKey(providerName: string, keyId: string): string { } export function setKeyCooldown(providerName: string, keyId: string, cooldownUntil: number): void { - keyCooldowns.set(cooldownKey(providerName, keyId), { cooldownUntil }); + const key = cooldownKey(providerName, keyId); + keyCooldowns.set(key, { cooldownUntil: Math.max(keyCooldowns.get(key)?.cooldownUntil ?? 0, cooldownUntil) }); } export function isKeyInCooldown(providerName: string, keyId: string, now = Date.now()): boolean { diff --git a/structure/providers-and-adapters.md b/structure/providers-and-adapters.md index e5b72f24273..6a0e2d09c97 100644 --- a/structure/providers-and-adapters.md +++ b/structure/providers-and-adapters.md @@ -288,7 +288,7 @@ Cursor, one lease spans the whole turn: RunSSE and BidiAppend may overlap inside turns wait at the cap. Follow-up sends still obey start intervals. The turn owner returns its lease after `runTurn` settles, so RunSSE body completion cannot admit another turn early. A capped canonical Codex WebSocket turn uses HTTP/SSE because the socket has no response-body lifecycle. -Capacity waits use the same bounded queue and retryable queue-overload errors as interval waits. Optional `requestPacing.keys` adds a shared cross-model limit for each credential reference; duplicate references share capacity, but distinct references resolving to one upstream project do not imply separate quota. `keySelection: round-robin` assigns request-local healthy keys without mutating persisted selection, skips known cooldowns, and refuses exhausted pools with retryable 429. Provider, model and key gates reserve atomically; key counters and the allocation cursor are process-local, not token or billing balances. OAuth and forwarded credentials retain their existing selection. Cooldown state is shared through the dependency-light `src/providers/api-key-cooldowns.ts` owner. Distributed selections are checked again at physical dispatch; a cooled queued key releases its old permit, selects a healthy sibling, and reacquires pacing before sending. Native Messages follows the same readmission rule. An established distributed turn refuses later sends after its key cools instead of changing credentials mid-turn. +Capacity waits use the same bounded queue and retryable queue-overload errors as interval waits. Optional `requestPacing.keys` adds a shared cross-model limit for each credential reference; duplicate references share capacity, but distinct references resolving to one upstream project do not imply separate quota. `keySelection: round-robin` assigns request-local healthy keys without mutating persisted selection, skips known cooldowns, and refuses exhausted pools with retryable 429. Provider, model and key gates reserve atomically; key counters and the allocation cursor are process-local, not token or billing balances. OAuth and forwarded credentials retain their existing selection. Cooldown state is shared through the dependency-light `src/providers/api-key-cooldowns.ts` (retaining the longest outstanding expiry until explicit manual clearing) owner. Distributed selections are checked again at physical dispatch; a cooled queued key releases its old permit, selects a healthy sibling, and reacquires pacing before sending. Native Messages follows the same readmission rule. An established distributed turn refuses later sends after its key cools instead of changing credentials mid-turn. Command Code effort defaults in `src/providers/command-code-efforts.ts` combine public-profile facts with the live API measurements from #5096. Both presets share the exact per-model rows; diff --git a/tests/providers/api-key-selection-cooldown.test.ts b/tests/providers/api-key-selection-cooldown.test.ts index 9863b2118df..d1f04eaf55f 100644 --- a/tests/providers/api-key-selection-cooldown.test.ts +++ b/tests/providers/api-key-selection-cooldown.test.ts @@ -104,6 +104,29 @@ describe("distributed selection cooldown validation", () => { expect(replacement._apiKeyAttempt?.entryId).toBe("b"); }); + test("a shorter concurrent 429 cannot undo a 401 cooldown, including duplicate references", () => { + const config = fixture(); + config.providers[NAME]!.apiKeyPool!.splice(1, 0, { id: "a-duplicate", key: A }); + const first = binding(config); const duplicate = binding(config, 1); const sibling = binding(config, 2); + const now = Date.now(); + const clock = spyOn(Date, "now").mockReturnValue(now); + try { + expect(recordDistributedKeyFailure(config, NAME, first._apiKeyAttempt, 401, now)).toBe(true); + clock.mockReturnValue(now + 100); + expect(recordDistributedKeyFailure(config, NAME, duplicate._apiKeyAttempt, 429, now + 100, "1")).toBe(true); + expect(getKeyCooldownUntil(NAME, "a")).toBe(now + 600_000); + expect(getKeyCooldownUntil(NAME, "a-duplicate")).toBe(now + 600_000); + clock.mockReturnValue(now + 2_000); + rejected(config, first); rejected(config, duplicate); accepted(config, sibling); + expect(config.providers[NAME]!.apiKey).toBe(A); + expect(config.providers[NAME]!.apiKeySelectionRevision).toBe("revision-fixture"); + clearKeyCooldowns(NAME); + accepted(config, first); accepted(config, duplicate); + expect(getKeyCooldownUntil(NAME, "a")).toBeNull(); + expect(getKeyCooldownUntil(NAME, "a-duplicate")).toBeNull(); + } finally { clock.mockRestore(); } + }); + test("a persistent binding keeps its existing validation semantics even when that entry is cooled", () => { const config = fixture(); const distributed = binding(config); cool(config, distributed); From 0df721e25a81d322d290e3d64cb5192e8671389c Mon Sep 17 00:00:00 2001 From: JUN Date: Mon, 12 Oct 2026 06:04:02 +0900 Subject: [PATCH 08/17] fix(pacing): guard keyed websocket routes and retain transport provenance --- .../src/content/docs/guides/providers.md | 2 +- scripts/test-layout/layout.json | 2 +- src/providers/request-pacing.ts | 14 +- src/server/responses/codex-ws-wire.ts | 18 +- src/server/responses/fetch-helpers.ts | 6 +- .../responses/native-response-control.ts | 6 +- structure/transports/responses.md | 2 +- tests/fixtures/test-layout-expected.json | 2 +- tests/server/fetch-header-timeout.test.ts | 36 ++++ tests/usage/request-pacing-provenance.test.ts | 194 ++++++++++++++++++ 10 files changed, 265 insertions(+), 17 deletions(-) create mode 100644 tests/usage/request-pacing-provenance.test.ts diff --git a/docs-site/src/content/docs/guides/providers.md b/docs-site/src/content/docs/guides/providers.md index c8180f60734..9e5b130f1f5 100644 --- a/docs-site/src/content/docs/guides/providers.md +++ b/docs-site/src/content/docs/guides/providers.md @@ -93,7 +93,7 @@ credentials must never be replayed on the same token, and local runtimes have no preserve. It is opt-in: when the option is absent the feature is off; object presence enables it unless `enabled: false`. -When round-robin key selection is enabled, queued requests recheck key cooldowns before sending and acquire pacing capacity for any replacement key. This applies to native Chat, Messages, Responses and compact. Waiting for replacement-key capacity does not consume the upstream response-header timeout. This does not change the saved active key. If a key cools after a turn has started, further sends in that turn are refused rather than switching its credential. +When round-robin key selection is enabled, queued requests recheck key cooldowns before sending and acquire pacing capacity for any replacement key. This applies to native Chat, Messages, Responses and compact. Waiting for replacement-key capacity does not consume the upstream response-header timeout. Per-key pacing and round-robin use guarded HTTP/SSE even when upstream WebSockets are enabled; WebSocket-only steering and injection require a route without those key policies. This does not change the saved active key. If a key cools after a turn has started, further sends in that turn are refused rather than switching its credential. ### Which account a request spends diff --git a/scripts/test-layout/layout.json b/scripts/test-layout/layout.json index b24d2b622df..be93f19536d 100644 --- a/scripts/test-layout/layout.json +++ b/scripts/test-layout/layout.json @@ -1511,7 +1511,7 @@ "request-log-generation-window.test.ts": "usage", "request-log.test.ts": "usage", "upstream-failure-diagnostics.test.ts": "usage", "request-outcome-agreement.test.ts": "usage", - "request-pacing.test.ts": "usage", + "request-pacing.test.ts": "usage", "request-pacing-provenance.test.ts": "usage", "reserve-auth-context.test.ts": "codex-integration", "reserve-availability.test.ts": "codex-integration", "reserve-catalog-lifecycle.test.ts": "codex-integration", diff --git a/src/providers/request-pacing.ts b/src/providers/request-pacing.ts index 4a29bd175e0..4d27f25f2a6 100644 --- a/src/providers/request-pacing.ts +++ b/src/providers/request-pacing.ts @@ -34,6 +34,12 @@ export const REQUEST_PACING_CONCURRENCY_RETRY_AFTER_SECONDS = 5; let maxQueueDepth = REQUEST_PACING_MAX_QUEUE_DEPTH; let maxQueueAgeMs = REQUEST_PACING_MAX_QUEUE_AGE_MS; +// Optional owners register internal response markers; only this owned body rewrap carries them. +const responseProvenance = new Set>(); +export function registerProviderResponseProvenance(markers: WeakMap): void { + responseProvenance.add(markers); +} + export type RequestPacingQueueOverloadReason = "queue_full" | "queue_expired" | "keys_exhausted"; export class RequestPacingQueueOverloadError extends Error { @@ -506,14 +512,13 @@ export async function waitForProviderRequestSlot( * means the exchange is already finished, so the lease returns immediately. A body that * is neither read nor cancelled for REQUEST_PACING_UNCONSUMED_BODY_MS releases its lease * and cancels the body, so a dropped Response cannot hold a slot forever. The returned - * Response preserves status, statusText, headers, and the identity-based replay markers. + * Response preserves status, statusText, headers, replay markers, and registered provenance. */ export function trackProviderRequestSlotBody( slot: ProviderRequestSlot | undefined, response: Response, ): Response { - // An unleased slot has nothing to return on body close, and rewrapping the Response - // would break identity-based markers (the eager WS relay registry is a WeakSet). + // An unleased slot has nothing to return on body close; preserve the response identity. if (!slot?.leased) return response; // Slots built outside waitForProviderRequestSlot (test doubles) satisfy only the public // ProviderRequestSlot shape; their bodies still own the release through the wrapper below. @@ -636,6 +641,9 @@ export function trackProviderRequestSlotBody( // Retry helpers mark the response they return from, and recovery decisions key on these // identity markers; the rewrap must not make a non-replayable response look replayable. if (isNonReplayableResponse(response)) markResponseNonReplayable(wrapped); + for (const markers of responseProvenance) { + if (markers.has(response)) markers.set(wrapped, markers.get(response)); + } return carryReplayRefusal(response, wrapped); } diff --git a/src/server/responses/codex-ws-wire.ts b/src/server/responses/codex-ws-wire.ts index 592d2ef32ad..bb8a1d6f54f 100644 --- a/src/server/responses/codex-ws-wire.ts +++ b/src/server/responses/codex-ws-wire.ts @@ -6,6 +6,7 @@ import { } from "../../lib/upstream-retry"; import type { RequestFailureCause, RequestFailureStage } from "../../lib/request-failure-model"; import { packageVersion } from "../../lib/package-version"; +import { registerProviderResponseProvenance } from "../../providers/request-pacing"; // If the 101 never arrives (network black hole), give SSE a chance well before // the caller's connect timeout (default 200s) would fire. export const UPGRADE_DEADLINE_MS = 10_000; @@ -47,9 +48,12 @@ export const CODEX_WS_CREATE_FRAME_LIMIT_BYTES = /** Close code the backend uses for an oversized message (RFC 6455 "message too big"). */ const WS_CLOSE_MESSAGE_TOO_BIG = 1009; -const codexWsUpstreamResponses = new WeakSet(); -const quotaObservedResponses = new WeakSet(); -const codexWsPreludeProjections = new WeakSet(); +const codexWsUpstreamResponses = new WeakMap(); +const quotaObservedResponses = new WeakMap(); +const codexWsPreludeProjections = new WeakMap(); +registerProviderResponseProvenance(codexWsUpstreamResponses); +registerProviderResponseProvenance(quotaObservedResponses); +registerProviderResponseProvenance(codexWsPreludeProjections); /** Quota arrived directly at its captured account; do not replay old HTTP prelude headers. */ export function isCodexWsQuotaObservedResponse(response: Response): boolean { @@ -63,7 +67,7 @@ export function isCodexWsUpstreamResponse(response: Response): boolean { /** Pre-response projection; its headers retain the exchange's prelude snapshot. */ export function markCodexWsPreludeProjection(response: Response): void { - codexWsPreludeProjections.add(response); + codexWsPreludeProjections.set(response, true); } export function isCodexWsPreludeProjection(response: Response): boolean { @@ -71,8 +75,8 @@ export function isCodexWsPreludeProjection(response: Response): boolean { } export function markCodexWsResponse(response: Response, observed: boolean): void { - codexWsUpstreamResponses.add(response); - if (observed) quotaObservedResponses.add(response); + codexWsUpstreamResponses.set(response, true); + if (observed) quotaObservedResponses.set(response, true); } /** @@ -110,6 +114,7 @@ export type CodexWsStageRecord = Omit & { }; const codexWsStageByResponse = new WeakMap(); +registerProviderResponseProvenance(codexWsStageByResponse); export function markCodexWsStage(response: Response, record: CodexWsStageRecord): void { const current = codexWsStageByResponse.get(response); @@ -254,6 +259,7 @@ export function projectCodexWsFailure( } const socketDeathStages = new WeakMap(); +registerProviderResponseProvenance(socketDeathStages); /** * Record that a pre-response settle came from the socket closing or failing under the send (#4191), diff --git a/src/server/responses/fetch-helpers.ts b/src/server/responses/fetch-helpers.ts index bd2b6fb0c60..c0fb63bdd20 100644 --- a/src/server/responses/fetch-helpers.ts +++ b/src/server/responses/fetch-helpers.ts @@ -342,11 +342,13 @@ export function providerFetch( const upstreamWebsocket = provider.upstreamWebsocket; if (!options.httpOnly && typeof input === "string" && init && shouldUseCodexWsUpstream(input, init, runtime, upstreamWebsocket)) { - if (options.providerName && requestPacingMaxConcurrentRequests(provider, options.modelId) > 0) { + const keyedPacing = provider.requestPacing?.enabled + && (provider.requestPacing.keys || provider.requestPacing.keySelection === "round-robin"); + if (keyedPacing || (options.providerName && requestPacingMaxConcurrentRequests(provider, options.modelId) > 0)) { if (!pacingWebsocketDowngradeWarned.has(providerName) && pacingWebsocketDowngradeWarned.size < EGRESS_DOWNGRADE_NOTICE_LIMIT) { pacingWebsocketDowngradeWarned.add(providerName); - console.warn(`[opencodex] provider ${JSON.stringify(redactSecretString(providerName))} has a request concurrency cap; serving over HTTP/SSE.`); + console.warn(`[opencodex] provider ${JSON.stringify(redactSecretString(providerName))} requires guarded request pacing; serving over HTTP/SSE.`); } return httpFetch(input, init, reacquirePacing); } diff --git a/src/server/responses/native-response-control.ts b/src/server/responses/native-response-control.ts index 13c4b4b48f9..c990c41dcd1 100644 --- a/src/server/responses/native-response-control.ts +++ b/src/server/responses/native-response-control.ts @@ -4,6 +4,7 @@ import type { NativeSteeringReplayObserver } from "./native-steering-replay"; import type { ProviderExecutedCallType } from "../responses-undeclared-tool-guard"; import { isInjectionRequest } from "./native-injection-protocol"; +import { registerProviderResponseProvenance } from "../../providers/request-pacing"; /** Shared transport ownership, not a shared steer/inject protocol state machine. */ export interface NativeResponseControl { @@ -25,9 +26,10 @@ export interface NativeResponseControl { export const OPENAI_API_RESPONSES_URL = "https://api.openai.com/v1/responses"; -const nativeControlResponses = new WeakSet(); +const nativeControlResponses = new WeakMap(); +registerProviderResponseProvenance(nativeControlResponses); /** Mark the exact response for multi-response delivery without serializing a wire field. */ -export function markNativeControlResponse(response: Response): Response { nativeControlResponses.add(response); return response; } +export function markNativeControlResponse(response: Response): Response { nativeControlResponses.set(response, true); return response; } /** Recognize a marked native response by identity, not by caller-controlled content. */ export function isNativeControlResponse(response: Response): boolean { return nativeControlResponses.has(response); } diff --git a/structure/transports/responses.md b/structure/transports/responses.md index 2c86cadb020..61bc83fe40b 100644 --- a/structure/transports/responses.md +++ b/structure/transports/responses.md @@ -51,7 +51,7 @@ performs the physical send, after a dispatch override has selected or rebuilt th matching follows the URL sent on the wire rather than the URL supplied before credential revalidation; prepaid initial-send receipts follow the [spend contract](responses-spend.md#prepaid-initial-sends). At this final HTTP boundary, native ChatGPT Responses and compact JSON strings of at least 1 MiB (UTF-8) become byte buffers to avoid Bun's large-string upload resets. Content, headers, abort signals and retry policy are preserved; WebSocket selection still receives the original string. Other destinations, small strings and existing byte/stream bodies retain their representation. -Replacement-key pacing admission retains client cancellation but pauses the header deadline; the full header interval restarts after admission. The wrapped executor alone is not that boundary. An override that revalidates credentials re-reads +Opt-in per-key pacing and round-robin use HTTP/SSE so their final dispatch override cannot be bypassed by a WebSocket upgrade. Transport-owned response markers and stage metadata survive owned pacing body rewraps. Replacement-key pacing admission retains client cancellation but pauses the header deadline; the full header interval restarts after admission. The wrapped executor alone is not that boundary. An override that revalidates credentials re-reads `route.provider.fetch` at send time, because reselection can install a different provider transport after the wrapper was built, and then calls that implementation instead of the executor. Both production overrides do this -- `oauthDispatch` in `request-transport.ts`, native Chat in diff --git a/tests/fixtures/test-layout-expected.json b/tests/fixtures/test-layout-expected.json index 528af9b8210..7c71b7426de 100644 --- a/tests/fixtures/test-layout-expected.json +++ b/tests/fixtures/test-layout-expected.json @@ -932,7 +932,7 @@ "request-log-served-model.test.ts": "usage", "request-log-model-filter.test.ts": "usage", "request-log-generation-window.test.ts": "usage", "request-log.test.ts": "usage", "request-outcome-agreement.test.ts": "usage", "upstream-failure-diagnostics.test.ts": "usage", - "request-pacing.test.ts": "usage", "reserve-auth-context.test.ts": "codex-integration", + "request-pacing.test.ts": "usage", "request-pacing-provenance.test.ts": "usage", "reserve-auth-context.test.ts": "codex-integration", "reserve-availability.test.ts": "codex-integration", "reserve-catalog-lifecycle.test.ts": "codex-integration", "reserve-catalog.test.ts": "codex-integration", "reserve-claude-policy.test.ts": "server", "reserve-dispatch-ws.test.ts": "responses", "reserve-dispatch.test.ts": "codex-integration", diff --git a/tests/server/fetch-header-timeout.test.ts b/tests/server/fetch-header-timeout.test.ts index b920c4f3ce8..d4f436ed40a 100644 --- a/tests/server/fetch-header-timeout.test.ts +++ b/tests/server/fetch-header-timeout.test.ts @@ -260,3 +260,39 @@ describe("replacement key admission excludes the response-header deadline", () = } finally { fixture.held.release(); fixture.resetProviderRequestPacingForTest(); } }); }); + +for (const policy of [ + { enabled: true, keySelection: "round-robin", minIntervalMs: 1 }, + { enabled: true, keys: { minIntervalMs: 1 } }, +] as const) test(`key pacing keeps WebSocket requests on the guarded HTTP path: ${JSON.stringify(policy)}`, async () => { + const { providerFetch } = await import("../../src/server/responses/fetch-helpers"); + const { resetProviderRequestPacingForTest } = await import("../../src/providers/request-pacing"); + const { BOUNDED_WS_RUNTIME } = await import("../helpers/ws-upstream-fixtures"); + const previousSocket = globalThis.WebSocket; + let sockets = 0; let overrides = 0; const sentKeys: (string | null)[] = []; + globalThis.WebSocket = class { constructor() { sockets++; throw new Error("unexpected WebSocket dispatch"); } } as unknown as typeof WebSocket; + const physical = (async (_input, init) => { + sentKeys.push(new Headers(init?.headers).get("authorization")); + return new Response("fixture reply"); + }) as typeof fetch; + const provider = { adapter: "openai-responses", authMode: "key", baseUrl: "https://api.openai.com/v1", + upstreamWebsocket: true, apiKey: "fixture-A", requestPacing: policy, fetch: physical, + } as import("../../src/types").OcxProviderConfig; + try { + const executor = providerFetch(provider, BOUNDED_WS_RUNTIME, { providerName: "ws-key-pacing", + dispatchOverride: async (input, init, execute, reacquire) => { + overrides++; + await reacquire!({ ...provider, apiKey: "fixture-B" }); + return execute(input, { ...init, headers: { authorization: "Bearer fixture-B" } }); + }, + }); + const response = await executor("https://api.openai.com/v1/responses", { method: "POST", + headers: { authorization: "Bearer fixture-A", "content-type": "application/json" }, + body: JSON.stringify({ model: "fixture-model", stream: true, input: [] }), + }); + expect(await response.text()).toBe("fixture reply"); + expect(overrides).toBe(1); + expect(sockets).toBe(0); + expect(sentKeys).toEqual(["Bearer fixture-B"]); + } finally { globalThis.WebSocket = previousSocket; resetProviderRequestPacingForTest(); } +}); diff --git a/tests/usage/request-pacing-provenance.test.ts b/tests/usage/request-pacing-provenance.test.ts new file mode 100644 index 00000000000..bc8409f075c --- /dev/null +++ b/tests/usage/request-pacing-provenance.test.ts @@ -0,0 +1,194 @@ +import { afterEach, describe, expect, test } from "bun:test"; +import { + providerRequestPacingStatus, registerProviderResponseProvenance, + resetProviderRequestPacingForTest, sendTrackingRequestSlot, + trackProviderRequestSlotBody, waitForProviderRequestSlot, +} from "../../src/providers/request-pacing"; +import { + isNativeControlResponse, markNativeControlResponse, +} from "../../src/server/responses/native-response-control"; +import { + codexWsSocketDeathStage, isCodexWsPreludeProjection, isCodexWsQuotaObservedResponse, + isCodexWsUpstreamResponse, markCodexWsPreludeProjection, markCodexWsResponse, + markCodexWsSocketDeath, markCodexWsStage, readCodexWsStage, type CodexWsStageRecord, +} from "../../src/server/responses/codex-ws-wire"; +import type { OcxProviderConfig } from "../../src/types"; + +afterEach(() => resetProviderRequestPacingForTest()); + +const provider: OcxProviderConfig = { + adapter: "openai-responses", apiKey: "fixture-key", baseUrl: "https://example.test/v1", + apiKeyPool: [{ id: "fixture", key: "fixture-key" }], + requestPacing: { enabled: true, keySelection: "round-robin" }, +}; + +function stage(): CodexWsStageRecord { + return { + requestBytes: 24, sent: true, upstreamFrames: 2, controlFrames: 1, relayedEvents: 1, + firstFrameMs: 1, firstResponseMs: 2, elapsedMs: 3, pings: 0, pongs: 0, + closeCode: null, reused: false, ocxVersion: "0.0.0", bunVersion: "fixture", + }; +} + +function expectUnmarked(response: Response): void { + expect(isNativeControlResponse(response)).toBe(false); + expect(isCodexWsUpstreamResponse(response)).toBe(false); + expect(isCodexWsQuotaObservedResponse(response)).toBe(false); + expect(isCodexWsPreludeProjection(response)).toBe(false); + expect(readCodexWsStage(response)).toBeUndefined(); + expect(codexWsSocketDeathStage(response)).toBeUndefined(); +} + +function countedSlot() { + let releases = 0; + return { + slot: { leased: true, bodyTracked: false, released: false, release() { releases++; } }, + releases: () => releases, + }; +} + +describe("pacing body transport provenance", () => { + test("round-robin lease preserves native and WS markers and live stage identity", async () => { + const slot = await waitForProviderRequestSlot("provenance", provider); + expect(slot.leased).toBe(true); // Key accounting alone owns a lease, with no concurrency cap. + expect(providerRequestPacingStatus("provenance", provider).keys?.[0]?.inFlight).toBe(1); + const original = markNativeControlResponse(new Response("first response\nsecond response\n")); + markCodexWsResponse(original, true); + const record = stage(); + markCodexWsStage(original, record); + const wrapped = trackProviderRequestSlotBody(slot, original); + expect(wrapped).not.toBe(original); + expect(slot.bodyTracked).toBe(true); + expect(isNativeControlResponse(wrapped)).toBe(true); + expect(isCodexWsUpstreamResponse(wrapped)).toBe(true); + expect(isCodexWsQuotaObservedResponse(wrapped)).toBe(true); + expect(isCodexWsPreludeProjection(wrapped)).toBe(false); + expect(readCodexWsStage(wrapped)).toBe(record); + markCodexWsStage(original, { ...record, upstreamFrames: 5, relayedEvents: 4 }); + expect(readCodexWsStage(wrapped)?.relayedEvents).toBe(4); + expect(slot.released).toBe(false); + expect(await wrapped.text()).toBe("first response\nsecond response\n"); + expect(slot.released).toBe(true); + expect(providerRequestPacingStatus("provenance", provider).keys?.[0]?.inFlight).toBe(0); + }); + + test.each([0, 1])("preserves prelude, stage and socket-death metadata (%i control frames)", async controlFrames => { + const slot = await waitForProviderRequestSlot("prelude", provider); + const original = new Response("refusal", { status: 502 }); + const record = { ...stage(), upstreamFrames: controlFrames, controlFrames, relayedEvents: 0 }; + markCodexWsPreludeProjection(original); + markCodexWsStage(original, record); + markCodexWsSocketDeath(original, record); + const wrapped = trackProviderRequestSlotBody(slot, original); + expect(isCodexWsPreludeProjection(wrapped)).toBe(true); + expect(codexWsSocketDeathStage(wrapped)).toBe(controlFrames ? "protocol-prelude" : "pre-header"); + expect(readCodexWsStage(wrapped)).toBe(record); + expect(isNativeControlResponse(wrapped)).toBe(false); + expect(isCodexWsUpstreamResponse(wrapped)).toBe(false); + expect(isCodexWsQuotaObservedResponse(wrapped)).toBe(false); + expect(wrapped.status).toBe(502); + await wrapped.body!.cancel(); + expect(slot.released).toBe(true); + }); + + test("an unobserved WS response retains only its existing authority", async () => { + const slot = await waitForProviderRequestSlot("ws-only", provider); + const original = new Response("WS"); + markCodexWsResponse(original, false); + const wrapped = trackProviderRequestSlotBody(slot, original); + expect(isCodexWsUpstreamResponse(wrapped)).toBe(true); + expect(isCodexWsQuotaObservedResponse(wrapped)).toBe(false); + expect(isNativeControlResponse(wrapped)).toBe(false); + await wrapped.body!.cancel(); + }); + + test("unmarked caller content and copied headers never acquire transport authority", async () => { + const slot = await waitForProviderRequestSlot("unmarked", provider); + const original = new Response('{"type":"response.created","nativeControl":true}', { + headers: { "x-codex-ws": "true", "x-native-control": "true", "x-codex-turn-state": "fixture" }, + }); + const wrapped = trackProviderRequestSlotBody(slot, original); + expectUnmarked(wrapped); + expect(wrapped.headers.get("x-codex-turn-state")).toBe("fixture"); + await wrapped.body!.cancel(); + }); + + test("only the owned body rewrap carries identity; ordinary cloning does not", async () => { + const source = markNativeControlResponse(new Response("fixture")); + markCodexWsResponse(source, true); + const clone = source.clone(); + expectUnmarked(clone); + expect(trackProviderRequestSlotBody(undefined, source)).toBe(source); + await Promise.all([source.text(), clone.text()]); + }); + + test.each([undefined, false])("registered provenance preserves present values (%s)", async value => { + const markers = new WeakMap(); + registerProviderResponseProvenance(markers); + registerProviderResponseProvenance(markers); // Registration is idempotent. + const original = new Response("fixture"); + markers.set(original, value); + const tracked = countedSlot(); + const wrapped = trackProviderRequestSlotBody(tracked.slot, original); + expect(markers.has(wrapped)).toBe(true); + expect(markers.get(wrapped)).toBe(value); + expectUnmarked(wrapped); + await wrapped.text(); + expect(tracked.releases()).toBe(1); + }); + + test("body completion releases once after every chunk", async () => { + const tracked = countedSlot(); + const wrapped = trackProviderRequestSlotBody(tracked.slot, markNativeControlResponse(new Response( + new ReadableStream({ start(controller) { + controller.enqueue(new TextEncoder().encode("first")); + controller.enqueue(new TextEncoder().encode("second")); + controller.close(); + } }), + ))); + const reader = wrapped.body!.getReader(); + expect((await reader.read()).done).toBe(false); + expect(tracked.releases()).toBe(0); + expect((await reader.read()).done).toBe(false); + expect(tracked.releases()).toBe(0); + expect((await reader.read()).done).toBe(true); + await reader.cancel(); + expect(tracked.releases()).toBe(1); + }); + + test("body cancellation forwards its reason and releases once", async () => { + const tracked = countedSlot(); + const reasons: unknown[] = []; + const original = new Response(new ReadableStream({ cancel(reason) { reasons.push(reason); } })); + markCodexWsResponse(original, true); + const wrapped = trackProviderRequestSlotBody(tracked.slot, original); + await wrapped.body!.cancel("consumer-abort"); + await wrapped.body!.cancel("second-cancel"); + expect(reasons).toEqual(["consumer-abort"]); + expect(tracked.releases()).toBe(1); + }); + + test("body error preserves the failure and releases once", async () => { + const tracked = countedSlot(); + const failure = new Error("fixture stream failure"); + const original = new Response(new ReadableStream({ pull(controller) { controller.error(failure); } })); + markCodexWsResponse(original, false); + const wrapped = trackProviderRequestSlotBody(tracked.slot, original); + await expect(wrapped.text()).rejects.toBe(failure); + expect(tracked.releases()).toBe(1); + }); + + test("abort and send error both return round-robin key leases", async () => { + const abort = new AbortController(); + const failure = new DOMException("fixture abort", "AbortError"); + const slot = await waitForProviderRequestSlot("abort", provider, undefined, abort.signal); + abort.abort(failure); + await expect(sendTrackingRequestSlot(slot, async () => { throw failure; })).rejects.toBe(failure); + expect(slot.released).toBe(true); + expect(providerRequestPacingStatus("abort", provider).keys?.[0]?.inFlight).toBe(0); + const next = await waitForProviderRequestSlot("abort", provider); + await expect(sendTrackingRequestSlot(next, async () => { throw new Error("send failed"); })).rejects.toThrow("send failed"); + expect(next.released).toBe(true); + expect(providerRequestPacingStatus("abort", provider).keys?.[0]?.inFlight).toBe(0); + }); +}); From 5e95e82f30f202cd7033d3bb0a9240999d7e1d00 Mon Sep 17 00:00:00 2001 From: JUN Date: Mon, 12 Oct 2026 06:19:32 +0900 Subject: [PATCH 09/17] fix(providers): retain terminal distributed key cooldowns --- .../src/content/docs/guides/providers.md | 2 +- src/server/images.ts | 9 +- src/server/messages-native.ts | 121 +++++++++++------- src/server/responses/compact.ts | 4 +- structure/data-planes/images.md | 2 +- structure/transports/responses-failover.md | 2 +- .../messages-native-key-selection.test.ts | 111 +++++++++++++++- tests/responses/compact-key-selection.test.ts | 30 +++++ tests/server/distributed-key-401.test.ts | 37 +++++- 9 files changed, 261 insertions(+), 57 deletions(-) diff --git a/docs-site/src/content/docs/guides/providers.md b/docs-site/src/content/docs/guides/providers.md index 9e5b130f1f5..2717d122357 100644 --- a/docs-site/src/content/docs/guides/providers.md +++ b/docs-site/src/content/docs/guides/providers.md @@ -93,7 +93,7 @@ credentials must never be replayed on the same token, and local runtimes have no preserve. It is opt-in: when the option is absent the feature is off; object presence enables it unless `enabled: false`. -When round-robin key selection is enabled, queued requests recheck key cooldowns before sending and acquire pacing capacity for any replacement key. This applies to native Chat, Messages, Responses and compact. Waiting for replacement-key capacity does not consume the upstream response-header timeout. Per-key pacing and round-robin use guarded HTTP/SSE even when upstream WebSockets are enabled; WebSocket-only steering and injection require a route without those key policies. This does not change the saved active key. If a key cools after a turn has started, further sends in that turn are refused rather than switching its credential. +When round-robin key selection is enabled, queued requests recheck key cooldowns before sending and acquire pacing capacity for any replacement key. This applies to native Chat, Messages, Responses and compact. Waiting for replacement-key capacity does not consume the upstream response-header timeout. Per-key pacing and round-robin use guarded HTTP/SSE even when upstream WebSockets are enabled; WebSocket-only steering and injection require a route without those key policies. This does not change the saved active key. An established upstream turn keeps its credential; later sends are refused if that key cools. Pre-output key rejection can recover to a healthy key within the existing request limit. Direct Images and compact requests record genuine 401/429 cooldowns without replaying that paid POST. ### Which account a request spends diff --git a/src/server/images.ts b/src/server/images.ts index b07f9b66251..302e49a494c 100644 --- a/src/server/images.ts +++ b/src/server/images.ts @@ -27,6 +27,7 @@ import { formatCodexProviderForLog } from "../codex/routing"; import { signalWithTimeout } from "../lib/abort"; import { readBoundedResponseBytes, type BoundedBytesResult } from "../lib/bounded-body"; import { sidecarEnter } from "../lib/sidecar-tracker"; +import { isNonReplayableResponse } from "../lib/upstream-retry"; import type { OcxConfig } from "../types"; import { resolveFirstUsableOpenAiSidecar, selectImagesProvider } from "../providers/openai-sidecar"; import { recordDistributedKeyFailure, selectProactiveApiKeyTransport } from "../providers/key-failover"; @@ -757,7 +758,7 @@ export async function handleImages( catch { console.error("[images] Failed to release probe lease"); } }; let url: string; - let recordKey401 = () => {}; + let recordKeyFailure = (_status: 401 | 429, _retryAfter: string | null) => {}; // Each branch is re-checked as it is entered: the forward check is defence in // depth for the pre-filter above, and the keyed check runs before a key // rotation is committed. @@ -815,7 +816,7 @@ export async function handleImages( throw error; } const provider = warmKeyProvider ?? candidates.keyed.provider; - recordKey401 = () => { recordDistributedKeyFailure(config, providerName, provider._apiKeyAttempt, 401); }; + recordKeyFailure = (status, retryAfter) => { recordDistributedKeyFailure(config, providerName, provider._apiKeyAttempt, status, Date.now(), retryAfter); }; // No fall back to the earlier snapshot once a pick has happened. The picker COMMITS its // choice before returning, so if the chosen reference will not resolve -- revoked keychain // entry, unset env var -- sending the previous key would authenticate a non-idempotent @@ -874,7 +875,9 @@ export async function handleImages( // compact already set this; the credential-bearing sidecars did not. redirect: "manual", }); - if (upstreamResponse.status === 401) recordKey401(); + if ((upstreamResponse.status === 401 || upstreamResponse.status === 429) && !isNonReplayableResponse(upstreamResponse)) { + recordKeyFailure(upstreamResponse.status, upstreamResponse.headers.get("retry-after")); + } const observed = await readImageResponseBytes(upstreamResponse, { maxBytes: IMAGES_RESPONSE_MAX_BYTES, signal: linkedSignal.signal, diff --git a/src/server/messages-native.ts b/src/server/messages-native.ts index 6f74bb190c5..cf9eaa6fd37 100644 --- a/src/server/messages-native.ts +++ b/src/server/messages-native.ts @@ -67,6 +67,7 @@ import { hasKeyPoolFailover, rateLimitRetryDelayMs, rateLimitRetryPolicyFor, + recordDistributedKeyFailure, rotateProviderTransportOn401, rotateProviderTransportOn429, selectProactiveApiKeyTransport, @@ -522,6 +523,15 @@ export async function handleNativeMessages(options: HandleNativeMessagesOptions) && nativeMessagesDeclineReason({ ...route, provider: routed }, body, config, selector) === undefined; }; + // Keep the actual failed wire identity if a later recovery is refused before sending. + // A 429 remains eligible for explicit same-key retries until that ladder is finished. + let pendingDistributedFailure: (() => void) | undefined; + let distributedResponseOwnsKey = true; + const settleDistributedFailure = () => { + const record = pendingDistributedFailure; + pendingDistributedFailure = undefined; + record?.(); + }; const send = async (recovery?: "rate-limit-429" | "oauth-401" | "oauth-account-403" | "key-429" | "key-401"): Promise => { const remaining = remainingTransientSends(); if (requestTransientPolicy && remaining <= 0) { @@ -592,6 +602,9 @@ export async function handleNativeMessages(options: HandleNativeMessagesOptions) } const releaseFamily = snapshot && oauthBinding ? anthropicModelQuotaFor(oauthBinding.instance).claimAnthropicFamilyRevalidation(snapshot.accountId, route.modelId) : () => {}; if (!releaseFamily) throw new AnthropicAccountCooldownError(1); + const selection = activeProvider._apiKeyAttempt; + const ownsDistributedKey = !selection?.distributed + || (headers.get("x-api-key") === activeProvider.apiKey && !headers.has("authorization")); let dispatched: Response; const spendReport = createPhysicalSendReporter(physicalBudget, () => ({ poolId: logCtx.spendPoolId ?? route.providerName, identityId: logCtx.accountLogLabel, @@ -635,6 +648,13 @@ export async function handleNativeMessages(options: HandleNativeMessagesOptions) } } catch { /* Passive observation must not fail the response. */ } } + distributedResponseOwnsKey = ownsDistributedKey; + const failureStatus = dispatched.status; + pendingDistributedFailure = selection?.distributed && ownsDistributedKey && (failureStatus === 401 || failureStatus === 429) + && !isNonReplayableResponse(dispatched) + ? () => { recordDistributedKeyFailure(config, route.providerName, selection, failureStatus, + Date.now(), dispatched.headers.get("retry-after")); } + : undefined; if (!dispatched.ok) await recordKeyAttemptFailure(logCtx, dispatched, init.signal ?? upstream.signal); return dispatched; }, @@ -659,51 +679,65 @@ export async function handleNativeMessages(options: HandleNativeMessagesOptions) let response: Response; try { response = await send(); - // A credential-scoped 401 on a static key pool says nothing about the sibling keys. - while (response.status === 401 && hasKeyPoolFailover(activeProvider) && transientSendAvailable()) { - const rotated = rotateProviderTransportOn401(config, route.providerName, activeProvider, { - now: Date.now(), - attemptedKey: activeProvider.apiKey, - attemptedSelection: activeProvider._apiKeyAttempt, - }); - if (!rotated) break; - discard(response); - rebuildFor(rotated); - response = await send("key-401"); - } - const retryPolicy = oauthBinding ? undefined : rateLimitRetryPolicyFor(activeProvider); + let retryPolicy: ReturnType | undefined; let retries = 0; - // A refusal this proxy synthesized for a reset replay is not a provider rate limit. - while ( - response.status === 429 - && !isNonReplayableResponse(response) - && retryPolicy - && retries < retryPolicy.attempts - && transientSendAvailable() - ) { - retries += 1; - for await (const _ of prepareSameTarget429Wait({ - body: response.body, - signal: upstream.signal, - delayMs: rateLimitRetryDelayMs(retryPolicy, response.headers.get("retry-after"), Date.now()), - })) { /* pre-stream wait */ } - if (upstream.signal.aborted) throw upstream.signal.reason; - response = await send("rate-limit-429"); - } - while (response.status === 429 && !isNonReplayableResponse(response) && hasKeyPoolFailover(activeProvider)) { - const rotated = rotateProviderTransportOn429(config, route.providerName, activeProvider, { - retryAfter: response.headers.get("retry-after"), - now: Date.now(), - attemptedKey: activeProvider.apiKey, - attemptedSelection: activeProvider._apiKeyAttempt, - }); - if (!rotated) break; - // Rotation already recorded the cooldown; keep the terminal 429 when no send remains. - if (!transientSendAvailable()) break; - discard(response); - rebuildFor(rotated); - response = await send("key-429"); + const recoveryPasses = activeProvider._apiKeyAttempt?.distributed ? activeProvider.apiKeyPool?.length ?? 1 : 1; + for (let pass = 0; pass < recoveryPasses; pass++) { + const priorResponse = response; + // A credential-scoped 401 on a static key pool says nothing about the sibling keys. + while (response.status === 401 && distributedResponseOwnsKey && hasKeyPoolFailover(activeProvider) && transientSendAvailable()) { + const rotated = rotateProviderTransportOn401(config, route.providerName, activeProvider, { + now: Date.now(), + attemptedKey: activeProvider.apiKey, + attemptedSelection: activeProvider._apiKeyAttempt, + }); + pendingDistributedFailure = undefined; // Rotation already records this physical failure. + if (!rotated) break; + discard(response); + rebuildFor(rotated); + response = await send("key-401"); + } + if (pass === 0) retryPolicy = oauthBinding ? undefined : rateLimitRetryPolicyFor(activeProvider); + // A refusal this proxy synthesized for a reset replay is not a provider rate limit. + while ( + response.status === 429 + && distributedResponseOwnsKey + && !isNonReplayableResponse(response) + && retryPolicy + && retries < retryPolicy.attempts + && transientSendAvailable() + ) { + retries += 1; + for await (const _ of prepareSameTarget429Wait({ + body: response.body, + signal: upstream.signal, + delayMs: rateLimitRetryDelayMs(retryPolicy, response.headers.get("retry-after"), Date.now()), + })) { /* pre-stream wait */ } + if (upstream.signal.aborted) throw upstream.signal.reason; + response = await send("rate-limit-429"); + } + while (response.status === 429 && distributedResponseOwnsKey && !isNonReplayableResponse(response) && hasKeyPoolFailover(activeProvider)) { + const rotated = rotateProviderTransportOn429(config, route.providerName, activeProvider, { + retryAfter: response.headers.get("retry-after"), + now: Date.now(), + attemptedKey: activeProvider.apiKey, + attemptedSelection: activeProvider._apiKeyAttempt, + }); + pendingDistributedFailure = undefined; // Rotation already records this physical failure. + if (!rotated) break; + // Rotation already recorded the cooldown; keep the terminal 429 when no send remains. + if (!transientSendAvailable()) break; + discard(response); + rebuildFor(rotated); + response = await send("key-429"); + } + // Only distributed recovery revisits the ladder when a send crossed failure classes. + // Retry and transient counters remain shared across passes; an unchanged response stops. + if (!activeProvider._apiKeyAttempt?.distributed || !distributedResponseOwnsKey || response === priorResponse + || (response.status !== 401 && response.status !== 429) || isNonReplayableResponse(response) + || !transientSendAvailable()) break; } + settleDistributedFailure(); const oauthRetryKey = {}; const triedAccountIds = new Set(); let oauthFailovers = 0; @@ -741,6 +775,7 @@ export async function handleNativeMessages(options: HandleNativeMessagesOptions) cleanupAbort(); upstream.abort(); if (req.signal.aborted) return fail(499, "Client cancelled request", "api_error"); + settleDistributedFailure(); const sendError = error instanceof UpstreamRetryEvidenceError ? error.cause : error; if (sendError instanceof RequestPacingQueueOverloadError) { const refusal = fail(429, sendError.message, "rate_limit_error"); diff --git a/src/server/responses/compact.ts b/src/server/responses/compact.ts index ed1b4df0a4d..04b37588875 100644 --- a/src/server/responses/compact.ts +++ b/src/server/responses/compact.ts @@ -1193,7 +1193,9 @@ export async function handleResponsesCompact( return formatErrorResponse(502, "upstream_error", "Failed to connect to compact upstream"); } - if (upstream.status === 401) recordDistributedKeyFailure(config, route.providerName, compactProvider._apiKeyAttempt, 401); + if ((upstream.status === 401 || upstream.status === 429) && !isNonReplayableResponse(upstream)) { + recordDistributedKeyFailure(config, route.providerName, compactProvider._apiKeyAttempt, upstream.status, Date.now(), upstream.headers.get("retry-after")); + } if ( upstream.status === 401 && (authCtx.kind === "main-pool" || authCtx.kind === "pool") diff --git a/structure/data-planes/images.md b/structure/data-planes/images.md index eb69eeadf31..ca7a53aa704 100644 --- a/structure/data-planes/images.md +++ b/structure/data-planes/images.md @@ -103,7 +103,7 @@ error rather than falling through to a separately billed keyed provider. The out has one selected Authorization value, validated before the non-idempotent upstream POST. The keyed path never enters `handleResponses`, so `src/server/images.ts` repeats `selectProactiveApiKeyTransport` inside the keyed branch and rebuilds Authorization from the -returned clone rather than the earlier snapshot. Opt-in request-local round-robin uses that same selection seam and maps exhausted-key allocation to retryable 429. The standalone Images relay remains a single direct non-idempotent fetch outside the inference request-pacing executor; provider/key pacing limits do not cover that relay. +returned clone rather than the earlier snapshot. Opt-in request-local round-robin uses that same selection seam and maps exhausted-key allocation to retryable 429. Genuine distributed 401/429 responses record the actual selected key cooldown, retaining Retry-After; synthetic replay refusals do not cool keys, and no paid POST is replayed. The standalone Images relay remains a single direct non-idempotent fetch outside the inference request-pacing executor; provider/key pacing limits do not cover that relay. A configured key's model and provider scope applies to whichever destination the request settles on: the ChatGPT forward account, the keyed provider, the xAI Imagine bridge, or the Antigravity diff --git a/structure/transports/responses-failover.md b/structure/transports/responses-failover.md index 805126202eb..3fa049111ba 100644 --- a/structure/transports/responses-failover.md +++ b/structure/transports/responses-failover.md @@ -156,7 +156,7 @@ The shared Responses path follows the [bounded multipart recovery contract](../s ## Upstream key attempt accounting -Key identity is sealed at guarded physical dispatch after queued selections are rebuilt. Distributed selections retain their request-local entry and revision. A rebuild after selection drift releases the old per-key lease and reacquires admission for the refreshed key; the send guard checks again after waiting. Turn-scoped adapters follow the same rule before invocation. Native Chat, Messages and keyed compact all rebuild stale queued selections before dispatch. Replacement admission pauses the response-header deadline and retains client cancellation; the header deadline restarts once admission completes. Unpaced dispatch without a readmission owner fails closed. Replay notification errors release acquired slots before propagating, including pre-acquired-slot dispatch. +Key identity is sealed at guarded physical dispatch after queued selections are rebuilt. Distributed selections retain their request-local entry and revision. A rebuild after selection drift releases the old per-key lease and reacquires admission for the refreshed key; the send guard checks again after waiting. Turn-scoped adapters follow the same rule before invocation. Native Chat, Messages and keyed compact all rebuild stale queued selections before dispatch. Direct compact and Images record genuine distributed 401/429 cooldowns without replaying paid POSTs; synthetic replay refusals never cool a key. Native Messages revisits mixed distributed 401/429 outcomes with its existing counters. Replacement admission pauses the response-header deadline and retains client cancellation; the header deadline restarts once admission completes. Unpaced dispatch without a readmission owner fails closed. Replay notification errors release acquired slots before propagating, including pre-acquired-slot dispatch. Raw adapter terminal usage is recorded before continuation, search, or image loops merge it; repeated parsing of one physical response does not count it twice. Key changes preserve the previous attempt while retaining the active attempt object shared by streaming/combo callbacks. diff --git a/tests/claude-integration/messages-native-key-selection.test.ts b/tests/claude-integration/messages-native-key-selection.test.ts index 38f98002582..e4d3fb46f10 100644 --- a/tests/claude-integration/messages-native-key-selection.test.ts +++ b/tests/claude-integration/messages-native-key-selection.test.ts @@ -3,8 +3,10 @@ import { mkdtempSync, readFileSync } from "node:fs"; import { tmpdir } from "node:os"; import { join } from "node:path"; import { saveConfig } from "../../src/config"; +import { apiKeyAccountLogLabel } from "../../src/codex/account-label"; +import { configureSharedSpendLedger, spendPolicyFromConfig } from "../../src/lib/spend-reservation-ledger"; import { captureProviderApiKeySelection } from "../../src/providers/api-key-selection"; -import { clearKeyCooldowns, forgetApiKeyRotationCursor, recordDistributedKeyFailure } from "../../src/providers/key-failover"; +import { clearKeyCooldowns, forgetApiKeyRotationCursor, getKeyCooldownUntil, recordDistributedKeyFailure } from "../../src/providers/key-failover"; import { providerRequestPacingStatus, resetProviderRequestPacingForTest, waitForProviderRequestSlot } from "../../src/providers/request-pacing"; import { handleClaudeMessages } from "../../src/server/claude-messages"; import { getRequestLogEntries } from "../../src/server/request-log"; @@ -41,11 +43,13 @@ afterEach(async () => { removeTreeWithRetry(home); }); -function fixture(count = 2, rejectFirst = false, keyCap = true): OcxConfig { +function fixture(count = 2, rejectFirst = false, keyCap = true, statuses: number[] = []): OcxConfig { upstream = Bun.serve({ hostname: "127.0.0.1", port: 0, async fetch(req) { seen.push({ key: req.headers.get("x-api-key"), path: new URL(req.url).pathname, body: await req.json() }); - if (rejectFirst && seen.length === 1) return Response.json({ type: "error", - error: { type: "authentication_error", message: "invalid fixture key" } }, { status: 401 }); + const status = statuses[seen.length - 1] ?? (rejectFirst && seen.length === 1 ? 401 : 200); + if (status !== 200) return Response.json({ type: "error", + error: { type: status === 401 ? "authentication_error" : "rate_limit_error", message: "rejected fixture key" } }, + { status, headers: status === 429 ? { "retry-after": "30" } : undefined }); return Response.json(MESSAGE); } }); const config = { port: 0, defaultProvider: NAME, protocols: { rollout: { managedMessagesNative: true } }, @@ -108,6 +112,105 @@ function nativeSuccess(result: Awaited>) { } describe("native Messages distributed key selection", () => { + for (const status of [401, 429]) test(`an overridden wire key C${status} never cools or replays selected A/B`, async () => { + const config = fixture(2, false, true, [status]); + provider(config).headers = { "x-api-key": KEYS[2]! }; + // A credential mismatch also refuses an explicitly requested same-key 429 retry. + provider(config).retryOn429 = { attempts: 1, intervalMs: 1, respectRetryAfter: false }; + saveConfig(config); + const abort = new AbortController(); + const pending = start(config, abort); + try { + const result = await pending; + expect(result.response.status).toBe(status); + expect(seen.map(entry => entry.key)).toEqual([KEYS[2]]); + expect(seen[0]!.path).toBe("/v1/messages"); + expect(getKeyCooldownUntil(NAME, "k1")).toBeNull(); + expect(getKeyCooldownUntil(NAME, "k2")).toBeNull(); + unchanged(config); noLeaks(config); + } finally { abort.abort(); await pending.catch(() => undefined); } + }); + + test("mixed A429 B401 C200 recovers across the ladder without changing saved selection", async () => { + const config = fixture(3, false, true, [429, 401, 200]); + const abort = new AbortController(); + const pending = start(config, abort); + try { + const result = await pending; + nativeSuccess(result); + expect(seen.map(entry => entry.key)).toEqual(KEYS); + expect(seen[1]!.body).toEqual(seen[0]!.body); + expect(seen[2]!.body).toEqual(seen[0]!.body); + const row = getRequestLogEntries().find(entry => entry.requestId === result.requestId)!; + expect(row.attempts!.map(attempt => attempt.sendCount)).toEqual([1, 1, 1]); + expect(row.attempts!.map(attempt => attempt.accountLogLabel)).toEqual(KEYS.map((reference, index) => + apiKeyAccountLogLabel(NAME, { entryId: `k${index + 1}`, reference }))); + expect(row.attempts!.map(attempt => attempt.status)).toEqual([429, 401, 200]); + expect(getKeyCooldownUntil(NAME, "k1")).toBeGreaterThan(Date.now() + 25_000); + expect(getKeyCooldownUntil(NAME, "k2")).toBeGreaterThan(Date.now()); + expect(getKeyCooldownUntil(NAME, "k3")).toBeNull(); + unchanged(config); noLeaks(config); + } finally { abort.abort(); await pending.catch(() => undefined); } + }); + + for (const terminal of [401, 429]) test(`mixed failures record terminal B${terminal} at the finite send limit`, async () => { + const config = fixture(2, false, true, [429, terminal]); + provider(config).transientRetryOn5xx = { attempts: 2 }; + saveConfig(config); + const abort = new AbortController(); + const pending = start(config, abort); + try { + const result = await pending; + expect(result.response.status).toBe(terminal); + expect(seen.map(entry => entry.key)).toEqual(KEYS.slice(0, 2)); + expect(getKeyCooldownUntil(NAME, "k1")).toBeGreaterThan(Date.now() + 25_000); + expect(getKeyCooldownUntil(NAME, "k2")).toBeGreaterThan(Date.now() + 25_000); + const exhausted = await start(config, abort); + expect(exhausted.response.status).toBe(429); + expect(Number(exhausted.response.headers.get("retry-after"))).toBeGreaterThan(0); + expect(seen).toHaveLength(2); + unchanged(config); noLeaks(config); + } finally { abort.abort(); await pending.catch(() => undefined); } + }); + + test("same-key retries precede cooldown and share their counter across mixed recovery", async () => { + const config = fixture(3, false, true, [429, 401, 429, 200]); + provider(config).retryOn429 = { attempts: 1, intervalMs: 0, respectRetryAfter: false }; + provider(config).transientRetryOn5xx = { attempts: 4 }; + saveConfig(config); + const abort = new AbortController(); + const pending = start(config, abort); + try { + nativeSuccess(await pending); + expect(seen.map(entry => entry.key)).toEqual([KEYS[0], KEYS[0], KEYS[1], KEYS[2]]); + expect(getKeyCooldownUntil(NAME, "k1")).not.toBeNull(); + expect(getKeyCooldownUntil(NAME, "k2")).not.toBeNull(); + expect(getKeyCooldownUntil(NAME, "k3")).toBeNull(); + unchanged(config); noLeaks(config); + } finally { abort.abort(); await pending.catch(() => undefined); } + }); + + test("a shared physical-send refusal records the last genuine 429 without cooling an unsent key", async () => { + const config = fixture(2, false, true, [429, 429, 429, 429, 200]); + provider(config).retryOn429 = { attempts: 6, intervalMs: 1, maxIntervalMs: 1, respectRetryAfter: false }; + saveConfig(config); + configureSharedSpendLedger(spendPolicyFromConfig({ pool: { maxTokens: 100_000 } }, undefined, [NAME])); + const abort = new AbortController(); + const pending = start(config, abort); + try { + const result = await pending; + expect(result.response.status).toBe(429); + expect(JSON.parse(result.text).error.type).toBe("send_budget_exhausted"); + expect(seen.map(entry => entry.key)).toEqual(Array(4).fill(KEYS[0])); + expect(getKeyCooldownUntil(NAME, "k1")).toBeGreaterThan(Date.now() + 25_000); + expect(getKeyCooldownUntil(NAME, "k2")).toBeNull(); + const next = await start(config, abort); + nativeSuccess(next); + expect(seen.map(entry => entry.key)).toEqual([...Array(4).fill(KEYS[0]), KEYS[1]]); + unchanged(config); noLeaks(config); + } finally { abort.abort(); await pending.catch(() => undefined); } + }); + test("401 recovers A to B without changing the saved key or revision", async () => { const config = fixture(2, true); const abort = new AbortController(); diff --git a/tests/responses/compact-key-selection.test.ts b/tests/responses/compact-key-selection.test.ts index a1f542c2a7d..4084fc1c268 100644 --- a/tests/responses/compact-key-selection.test.ts +++ b/tests/responses/compact-key-selection.test.ts @@ -2,6 +2,7 @@ import { afterEach, beforeEach, expect, test } from "bun:test"; import { mkdtempSync, readFileSync } from "node:fs"; import { tmpdir } from "node:os"; import { join } from "node:path"; +import { markResponseNonReplayable } from "../../src/lib/upstream-retry"; import { saveConfig } from "../../src/config"; import { apiKeyAccountLogLabel } from "../../src/codex/account-label"; import { captureProviderApiKeySelection } from "../../src/providers/api-key-selection"; @@ -216,3 +217,32 @@ test("compact 401 after queued reselection cools the physical key B without repl unchanged(config); noLeaks(config); } finally { abort.abort(); first.release(); await pending.catch(() => undefined); } }); + +test("direct compact distributed 429 cools each physical key without same-request replay", async () => { + const config = fixture(); + globalThis.fetch = (async (input, init) => { + seen.push({ key: new Headers(init?.headers).get("authorization"), url: String(input), body: JSON.parse(String(init?.body)), headers: new Headers(init?.headers) }); + return Response.json({ error: { message: "fixture capacity" } }, { status: 429, headers: { "retry-after": "60" } }); + }) as typeof fetch; + const began = Date.now(); + for (let index = 0; index < 4; index++) { + const result = await start(config, new AbortController()); + expect(result.response.status).toBe(429); + expect(seen).toHaveLength(Math.min(index + 1, 3)); + } + expect(seen.map(row => row.key)).toEqual(KEYS.map(key => `Bearer ${key}`)); + for (let index = 0; index < 3; index++) expect(getKeyCooldownUntil(NAME, `k${index + 1}`)).toBeGreaterThanOrEqual(began + 60_000); + unchanged(config); noLeaks(config); +}); + +test("a synthetic non-replayable compact 429 never cools a distributed key", async () => { + const config = fixture(); + globalThis.fetch = (async () => { + const response = Response.json({ error: { message: "fixture local refusal" } }, { status: 429, headers: { "retry-after": "60" } }); + markResponseNonReplayable(response); return response; + }) as typeof fetch; + const result = await start(config, new AbortController()); + expect(result.response.status).toBe(429); + for (let index = 0; index < 3; index++) expect(getKeyCooldownUntil(NAME, `k${index + 1}`)).toBeNull(); + unchanged(config); noLeaks(config); +}); diff --git a/tests/server/distributed-key-401.test.ts b/tests/server/distributed-key-401.test.ts index 61618f23b3f..bea402e972f 100644 --- a/tests/server/distributed-key-401.test.ts +++ b/tests/server/distributed-key-401.test.ts @@ -3,7 +3,7 @@ import { mkdtempSync } from "node:fs"; import { tmpdir } from "node:os"; import { join } from "node:path"; import { saveConfig } from "../../src/config"; -import { clearKeyCooldowns } from "../../src/providers/key-failover"; +import { clearKeyCooldowns, forgetApiKeyRotationCursor, getKeyCooldownUntil } from "../../src/providers/key-failover"; import { startServer } from "../../src/server"; import { handleResponsesCompact } from "../../src/server/responses"; import type { OcxConfig } from "../../src/types"; @@ -29,7 +29,7 @@ beforeEach(() => { process.env.OPENCODEX_HOME = home; delete process.env.OPENCODEX_API_AUTH_TOKEN; codexHome = installIsolatedCodexHome("ocx-distributed-401-codex-"); - clearKeyCooldowns(); + clearKeyCooldowns(); forgetApiKeyRotationCursor(); }); afterEach(async () => { globalThis.fetch = originalFetch; @@ -40,7 +40,7 @@ afterEach(async () => { else process.env.OPENCODEX_HOME = previousHome; if (previousToken === undefined) delete process.env.OPENCODEX_API_AUTH_TOKEN; else process.env.OPENCODEX_API_AUTH_TOKEN = previousToken; - clearKeyCooldowns(); + clearKeyCooldowns(); forgetApiKeyRotationCursor(); }); test("distributed image 401 cools the failed key without replaying the paid POST", async () => { @@ -101,3 +101,34 @@ test("native compact distributed 401 cools the static key without replaying or r expect(keys).toEqual(["Bearer synthetic-first", "Bearer synthetic-second", "Bearer synthetic-second"]); } finally { clearKeyCooldowns("openai-apikey"); } }); + +test("direct distributed Images 429 cools both keys without replaying the paid POST", async () => { + const seen: string[] = []; + const upstream = Bun.serve({ hostname: "127.0.0.1", port: 0, fetch(req) { + seen.push(req.headers.get("authorization")!); + return Response.json({ error: { message: "fixture capacity" } }, { status: 429, headers: { "retry-after": "60" } }); + } }); + globalThis.fetch = ((input: RequestInfo | URL, init?: RequestInit) => { + const url = new URL(typeof input === "string" ? input : input instanceof URL ? input.href : input.url); + return originalFetch(url.hostname === "api.openai.com" ? new URL(url.pathname, upstream.url) : input, init); + }) as typeof fetch; + saveConfig({ port: 0, defaultProvider: "openai-apikey", openaiProviderTierVersion: 2, + providers: { openai: disabledOpenAiProvider, "openai-apikey": { + ...keyedProvider(upstream.url.toString()), apiKeyPool: [{ id: "first", key: "sk-platform-key" }, { id: "second", key: "sk-warm-key" }], + requestPacing: { enabled: true, keySelection: "round-robin" }, + } }, + } as unknown as OcxConfig); + const server = startServer(0); const began = Date.now(); + try { + for (let index = 0; index < 3; index++) { + const response = await fetch(new URL("/v1/images/generations", server.url), { method: "POST", + headers: { "content-type": "application/json", authorization: `Bearer ${DIRECT_CHATGPT_TOKEN}` }, + body: JSON.stringify({ prompt: "fixture", model: "gpt-image-2" }), + }); + expect(response.status).toBe(429); await response.text(); + expect(seen).toHaveLength(Math.min(index + 1, 2)); + } + expect(seen).toEqual(["Bearer sk-platform-key", "Bearer sk-warm-key"]); + for (const id of ["first", "second"]) expect(getKeyCooldownUntil("openai-apikey", id)).toBeGreaterThanOrEqual(began + 60_000); + } finally { await server.stop(true); await upstream.stop(true); clearKeyCooldowns(); } +}); From dab8273a30ceb5ee8538af96d66761528d260431 Mon Sep 17 00:00:00 2001 From: JUN Date: Mon, 12 Oct 2026 06:27:26 +0900 Subject: [PATCH 10/17] fix(providers): fence native Chat distributed credential recovery --- src/server/chat-native.ts | 6 +++- structure/transports/responses-failover.md | 2 +- .../server-chat-native-key-selection.test.ts | 36 ++++++++++++++++++- 3 files changed, 41 insertions(+), 3 deletions(-) diff --git a/src/server/chat-native.ts b/src/server/chat-native.ts index fd470e7123f..322dda782c5 100644 --- a/src/server/chat-native.ts +++ b/src/server/chat-native.ts @@ -433,6 +433,7 @@ export async function runNativeChatAttempt( } else if (!spendTracker?.charge()) throw new NativeChatSpendRefusal(); }; + let responseOwnsDistributedKey = true; const send = async ( request: AdapterRequest, recovery?: "rate-limit-429" | "key-429" | "key-401" | UpstreamSendRecovery, @@ -501,6 +502,8 @@ export async function runNativeChatAttempt( const spendReport = createPhysicalSendReporter(physicalBudget, () => ({ poolId: logCtx.spendPoolId ?? route.providerName, identityId: logCtx.accountLogLabel, }), execution.comboDispatchPermit); + const ownsDistributedKey = !activeProvider._apiKeyAttempt?.distributed + || headers.get("authorization") === `Bearer ${activeProvider.apiKey}`; let dispatched: Response; try { // Capture this request's spend policy at its first physical boundary, including @@ -533,6 +536,7 @@ export async function runNativeChatAttempt( try { if (physicalBudget.spendEnforced) spendReport(1); } finally { spendReport.close?.(); } } + responseOwnsDistributedKey = ownsDistributedKey; if (!dispatched.ok) await recordKeyAttemptFailure(logCtx, dispatched, init.signal ?? upstream.signal); return dispatched; }, @@ -590,7 +594,7 @@ export async function runNativeChatAttempt( // Same reason as above, plus a second one: rotating here would write a cooldown against // a key that rate-limited nothing, and that false signal outlives the request. while ((response.status === 429 || (response.status === 401 && activeProvider._apiKeyAttempt?.distributed)) - && !isNonReplayableResponse(response) && hasKeyPoolFailover(activeProvider)) { + && responseOwnsDistributedKey && !isNonReplayableResponse(response) && hasKeyPoolFailover(activeProvider)) { const failureStatus = response.status; const rotate = failureStatus === 401 ? rotateProviderTransportOn401 : rotateProviderTransportOn429; const rotated = rotate(config, route.providerName, activeProvider, { diff --git a/structure/transports/responses-failover.md b/structure/transports/responses-failover.md index 3fa049111ba..6f886c8d283 100644 --- a/structure/transports/responses-failover.md +++ b/structure/transports/responses-failover.md @@ -156,7 +156,7 @@ The shared Responses path follows the [bounded multipart recovery contract](../s ## Upstream key attempt accounting -Key identity is sealed at guarded physical dispatch after queued selections are rebuilt. Distributed selections retain their request-local entry and revision. A rebuild after selection drift releases the old per-key lease and reacquires admission for the refreshed key; the send guard checks again after waiting. Turn-scoped adapters follow the same rule before invocation. Native Chat, Messages and keyed compact all rebuild stale queued selections before dispatch. Direct compact and Images record genuine distributed 401/429 cooldowns without replaying paid POSTs; synthetic replay refusals never cool a key. Native Messages revisits mixed distributed 401/429 outcomes with its existing counters. Replacement admission pauses the response-header deadline and retains client cancellation; the header deadline restarts once admission completes. Unpaced dispatch without a readmission owner fails closed. Replay notification errors release acquired slots before propagating, including pre-acquired-slot dispatch. +Key identity is sealed at guarded physical dispatch after queued selections are rebuilt. Distributed selections retain their request-local entry and revision. A rebuild after selection drift releases the old per-key lease and reacquires admission for the refreshed key; the send guard checks again after waiting. Turn-scoped adapters follow the same rule before invocation. Native Chat, Messages and keyed compact all rebuild stale queued selections before dispatch. Direct compact and Images record genuine distributed 401/429 cooldowns without replaying paid POSTs; synthetic replay refusals never cool a key. Native Messages revisits mixed distributed 401/429 outcomes with its existing counters. Native Chat and Messages require the physical outgoing credential to match the selected distributed key before pooling a rejection; configured credential overrides cannot cool unused entries. Replacement admission pauses the response-header deadline and retains client cancellation; the header deadline restarts once admission completes. Unpaced dispatch without a readmission owner fails closed. Replay notification errors release acquired slots before propagating, including pre-acquired-slot dispatch. Raw adapter terminal usage is recorded before continuation, search, or image loops merge it; repeated parsing of one physical response does not count it twice. Key changes preserve the previous attempt while retaining the active attempt object shared by streaming/combo callbacks. diff --git a/tests/server/server-chat-native-key-selection.test.ts b/tests/server/server-chat-native-key-selection.test.ts index 25d5c3ec244..a566a76113f 100644 --- a/tests/server/server-chat-native-key-selection.test.ts +++ b/tests/server/server-chat-native-key-selection.test.ts @@ -5,7 +5,7 @@ import { join } from "node:path"; import { saveConfig } from "../../src/config"; import { apiKeyAccountLogLabel } from "../../src/codex/account-label"; import { captureProviderApiKeySelection } from "../../src/providers/api-key-selection"; -import { clearKeyCooldowns, forgetApiKeyRotationCursor, recordDistributedKeyFailure } from "../../src/providers/key-failover"; +import { clearKeyCooldowns, forgetApiKeyRotationCursor, getKeyCooldownUntil, recordDistributedKeyFailure } from "../../src/providers/key-failover"; import { providerRequestPacingStatus, resetProviderRequestPacingForTest, waitForProviderRequestSlot } from "../../src/providers/request-pacing"; import { handleChatCompletions } from "../../src/server/chat-completions"; import type { RequestLogContext } from "../../src/server/request-log"; @@ -193,3 +193,37 @@ test("chat refuses after three queued rebuilds without spending a physical send" unchanged(config); noLeaks(config); } finally { abort.abort(); for (const slot of held) slot.release(); await pending.catch(() => undefined); } }, 10_000); + +for (const statusCode of [401, 429]) test(`chat wire override ${statusCode} does not reject unused distributed pool keys`, async () => { + const config = fixture(); + config.providers[NAME]!.headers = { Authorization: "Bearer fixture-key-override" }; + globalThis.fetch = (async (input, init) => { + seen.push({ key: new Headers(init?.headers).get("authorization"), url: String(input), body: JSON.parse(String(init?.body)) }); + return Response.json({ error: { type: statusCode === 401 ? "authentication_error" : "rate_limit_error", message: "fixture overridden credential refusal" } }, { status: statusCode }); + }) as typeof fetch; + const result = await start(config, new AbortController()); + expect(result.response.status).toBe(statusCode); + expect(result.text).toContain("fixture overridden credential refusal"); + expect(seen.map(entry => entry.key)).toEqual(["Bearer fixture-key-override"]); + for (let index = 0; index < KEYS.length; index++) expect(getKeyCooldownUntil(NAME, `k${index + 1}`)).toBeNull(); + unchanged(config); noLeaks(config); +}); + +test("chat owned 401 rotates the rejected distributed credential before output", async () => { + const config = fixture(); + const reply = globalThis.fetch; + globalThis.fetch = (async (input, init) => { + if (seen.length === 0) { + seen.push({ key: new Headers(init?.headers).get("authorization"), url: String(input), body: JSON.parse(String(init?.body)) }); + return Response.json({ error: { type: "authentication_error", message: "fixture selected credential refusal" } }, { status: 401 }); + } + return reply(input, init); + }) as typeof fetch; + const result = await start(config, new AbortController()); + expect(result.response.status).toBe(200); + expect(result.text).toContain("fixture reply"); + expect(seen.map(entry => entry.key)).toEqual([`Bearer ${KEYS[0]}`, `Bearer ${KEYS[1]}`]); + expect(getKeyCooldownUntil(NAME, "k1")).not.toBeNull(); + expect(getKeyCooldownUntil(NAME, "k2")).toBeNull(); + unchanged(config); noLeaks(config); +}); From 06c8153687efe4296725a1b7f43cb796d0c7ef51 Mon Sep 17 00:00:00 2001 From: JUN Date: Mon, 12 Oct 2026 06:35:58 +0900 Subject: [PATCH 11/17] fix(providers): honor configured Messages bearer key transport --- src/server/messages-native.ts | 4 ++- structure/transports/responses-failover.md | 2 +- .../messages-native-key-selection.test.ts | 25 +++++++++++++++++-- 3 files changed, 27 insertions(+), 4 deletions(-) diff --git a/src/server/messages-native.ts b/src/server/messages-native.ts index cf9eaa6fd37..fb572b636a0 100644 --- a/src/server/messages-native.ts +++ b/src/server/messages-native.ts @@ -604,7 +604,9 @@ export async function handleNativeMessages(options: HandleNativeMessagesOptions) if (!releaseFamily) throw new AnthropicAccountCooldownError(1); const selection = activeProvider._apiKeyAttempt; const ownsDistributedKey = !selection?.distributed - || (headers.get("x-api-key") === activeProvider.apiKey && !headers.has("authorization")); + || (activeProvider.apiKeyTransport === "bearer" + ? headers.get("authorization") === `Bearer ${activeProvider.apiKey}` && !headers.has("x-api-key") + : headers.get("x-api-key") === activeProvider.apiKey && !headers.has("authorization")); let dispatched: Response; const spendReport = createPhysicalSendReporter(physicalBudget, () => ({ poolId: logCtx.spendPoolId ?? route.providerName, identityId: logCtx.accountLogLabel, diff --git a/structure/transports/responses-failover.md b/structure/transports/responses-failover.md index 6f886c8d283..1031c4799fd 100644 --- a/structure/transports/responses-failover.md +++ b/structure/transports/responses-failover.md @@ -156,7 +156,7 @@ The shared Responses path follows the [bounded multipart recovery contract](../s ## Upstream key attempt accounting -Key identity is sealed at guarded physical dispatch after queued selections are rebuilt. Distributed selections retain their request-local entry and revision. A rebuild after selection drift releases the old per-key lease and reacquires admission for the refreshed key; the send guard checks again after waiting. Turn-scoped adapters follow the same rule before invocation. Native Chat, Messages and keyed compact all rebuild stale queued selections before dispatch. Direct compact and Images record genuine distributed 401/429 cooldowns without replaying paid POSTs; synthetic replay refusals never cool a key. Native Messages revisits mixed distributed 401/429 outcomes with its existing counters. Native Chat and Messages require the physical outgoing credential to match the selected distributed key before pooling a rejection; configured credential overrides cannot cool unused entries. Replacement admission pauses the response-header deadline and retains client cancellation; the header deadline restarts once admission completes. Unpaced dispatch without a readmission owner fails closed. Replay notification errors release acquired slots before propagating, including pre-acquired-slot dispatch. +Key identity is sealed at guarded physical dispatch after queued selections are rebuilt. Distributed selections retain their request-local entry and revision. A rebuild after selection drift releases the old per-key lease and reacquires admission for the refreshed key; the send guard checks again after waiting. Turn-scoped adapters follow the same rule before invocation. Native Chat, Messages and keyed compact all rebuild stale queued selections before dispatch. Direct compact and Images record genuine distributed 401/429 cooldowns without replaying paid POSTs; synthetic replay refusals never cool a key. Native Messages revisits mixed distributed 401/429 outcomes with its existing counters. Native Chat and Messages require the physical outgoing credential to match the selected distributed key before pooling a rejection; Messages honors the configured `x-api-key` or Bearer transport while rejecting a conflicting credential header; configured credential overrides cannot cool unused entries. Replacement admission pauses the response-header deadline and retains client cancellation; the header deadline restarts once admission completes. Unpaced dispatch without a readmission owner fails closed. Replay notification errors release acquired slots before propagating, including pre-acquired-slot dispatch. Raw adapter terminal usage is recorded before continuation, search, or image loops merge it; repeated parsing of one physical response does not count it twice. Key changes preserve the previous attempt while retaining the active attempt object shared by streaming/combo callbacks. diff --git a/tests/claude-integration/messages-native-key-selection.test.ts b/tests/claude-integration/messages-native-key-selection.test.ts index e4d3fb46f10..b9773855395 100644 --- a/tests/claude-integration/messages-native-key-selection.test.ts +++ b/tests/claude-integration/messages-native-key-selection.test.ts @@ -23,7 +23,7 @@ let home: string; let previousHome: string | undefined; let releaseOwner: () => void; let upstream: ReturnType | undefined; -let seen: { key: string | null; path: string; body: unknown }[]; +let seen: { key: string | null; authorization?: string | null; path: string; body: unknown }[]; beforeEach(() => { previousHome = process.env.OPENCODEX_HOME; @@ -45,7 +45,7 @@ afterEach(async () => { function fixture(count = 2, rejectFirst = false, keyCap = true, statuses: number[] = []): OcxConfig { upstream = Bun.serve({ hostname: "127.0.0.1", port: 0, async fetch(req) { - seen.push({ key: req.headers.get("x-api-key"), path: new URL(req.url).pathname, body: await req.json() }); + seen.push({ key: req.headers.get("x-api-key") ?? req.headers.get("authorization")?.replace(/^Bearer /, "") ?? null, authorization: req.headers.get("authorization"), path: new URL(req.url).pathname, body: await req.json() }); const status = statuses[seen.length - 1] ?? (rejectFirst && seen.length === 1 ? 401 : 200); if (status !== 200) return Response.json({ type: "error", error: { type: status === 401 ? "authentication_error" : "rate_limit_error", message: "rejected fixture key" } }, @@ -299,3 +299,24 @@ describe("native Messages distributed key selection", () => { } finally { abort.abort(); first.release(); second.release(); await pending.catch(() => undefined); } }, 10_000); }); + +for (const statusCode of [401, 429]) for (const overridden of [false, true]) test(`Messages bearer ${statusCode} owns the configured key transport (override=${overridden})`, async () => { + const config = fixture(2, false, true, [statusCode, 200]); + provider(config).apiKeyTransport = "bearer"; + saveConfig(config); + if (overridden) provider(config).headers = { Authorization: "Bearer fixture-bearer-override" }; + const result = await start(config, new AbortController()); + if (overridden) { + expect(result.response.status).toBe(statusCode); + expect(seen.map(entry => entry.key)).toEqual(["fixture-bearer-override"]); + expect(getKeyCooldownUntil(NAME, "k1")).toBeNull(); + expect(getKeyCooldownUntil(NAME, "k2")).toBeNull(); + } else { + nativeSuccess(result); + expect(seen.map(entry => entry.key)).toEqual(KEYS.slice(0, 2)); + expect(seen.map(entry => entry.authorization)).toEqual(KEYS.slice(0, 2).map(key => `Bearer ${key}`)); + expect(getKeyCooldownUntil(NAME, "k1")).not.toBeNull(); + expect(getKeyCooldownUntil(NAME, "k2")).toBeNull(); + } + unchanged(config); noLeaks(config); +}); From 8bf1f2907834eb7c350c5d34b9ace7a8da4aa3c9 Mon Sep 17 00:00:00 2001 From: JUN Date: Mon, 12 Oct 2026 06:46:58 +0900 Subject: [PATCH 12/17] fix(providers): readmit manually selected keys under per-key limits --- src/server/chat-native.ts | 5 +-- src/server/messages-native.ts | 5 +-- src/server/responses/compact.ts | 5 +-- structure/transports/responses-failover.md | 2 +- .../messages-native-key-selection.test.ts | 35 +++++++++++++++++++ tests/responses/compact-key-selection.test.ts | 34 ++++++++++++++++++ .../server-chat-native-key-selection.test.ts | 34 ++++++++++++++++++ 7 files changed, 113 insertions(+), 7 deletions(-) diff --git a/src/server/chat-native.ts b/src/server/chat-native.ts index 322dda782c5..595f3716117 100644 --- a/src/server/chat-native.ts +++ b/src/server/chat-native.ts @@ -471,9 +471,10 @@ export async function runNativeChatAttempt( if (outputDelivered) throw new Error("Provider key selection is unavailable during the native Chat turn"); if (attempt >= 3) throw new Error("Provider key selection changed repeatedly before native Chat dispatch"); let current = resolveCurrentProviderApiKeyTransport(config, route.providerName, activeProvider); - if (!current && activeProvider._apiKeyAttempt?.distributed) { + if (!current && activeProvider.requestPacing?.enabled + && (activeProvider.requestPacing.keys || activeProvider.requestPacing.keySelection === "round-robin")) { const fresh = resolveCurrentProviderApiKeyTransport(config, route.providerName, { ...activeProvider, _apiKeyAttempt: undefined }); - if (fresh) current = selectProactiveApiKeyTransport(config, route.providerName, fresh); + if (fresh) current = selectProactiveApiKeyTransport(config, route.providerName, fresh) ?? fresh; } if (!current || !isNativeChatRouteEligible({ ...route, provider: current }, execution.chatBody, config)) { throw new Error("Provider key selection is no longer available for native Chat"); diff --git a/src/server/messages-native.ts b/src/server/messages-native.ts index fb572b636a0..c0e89353df5 100644 --- a/src/server/messages-native.ts +++ b/src/server/messages-native.ts @@ -570,9 +570,10 @@ export async function handleNativeMessages(options: HandleNativeMessagesOptions) for (let attempt = 0; !providerApiKeySelectionIsCurrent(config, route.providerName, activeProvider); attempt++) { if (attempt >= 3) throw new Error("Provider key selection changed repeatedly before native Messages dispatch"); let current = resolveCurrentProviderApiKeyTransport(config, route.providerName, activeProvider); - if (!current && activeProvider._apiKeyAttempt?.distributed) { + if (!current && activeProvider.requestPacing?.enabled + && (activeProvider.requestPacing.keys || activeProvider.requestPacing.keySelection === "round-robin")) { const fresh = resolveCurrentProviderApiKeyTransport(config, route.providerName, { ...activeProvider, _apiKeyAttempt: undefined }); - if (fresh) current = selectProactiveApiKeyTransport(config, route.providerName, fresh); + if (fresh) current = selectProactiveApiKeyTransport(config, route.providerName, fresh) ?? fresh; } if (!current || nativeMessagesDeclineReason({ ...route, provider: current }, body, config, selector) !== undefined) { throw new Error("Provider key selection is no longer available for native Messages"); diff --git a/src/server/responses/compact.ts b/src/server/responses/compact.ts index 04b37588875..3d71b1d191d 100644 --- a/src/server/responses/compact.ts +++ b/src/server/responses/compact.ts @@ -1091,9 +1091,10 @@ export async function handleResponsesCompact( for (let attempt = 0; !providerApiKeySelectionIsCurrent(config, route.providerName, sendProvider); attempt++) { if (attempt >= 3) throw new Error("Provider key selection changed repeatedly before compact dispatch"); let current = resolveCurrentProviderApiKeyTransport(config, route.providerName, sendProvider); - if (!current && sendProvider._apiKeyAttempt?.distributed) { + if (!current && sendProvider.requestPacing?.enabled + && (sendProvider.requestPacing.keys || sendProvider.requestPacing.keySelection === "round-robin")) { const fresh = resolveCurrentProviderApiKeyTransport(config, route.providerName, { ...sendProvider, _apiKeyAttempt: undefined }); - if (fresh) current = selectProactiveApiKeyTransport(config, route.providerName, fresh); + if (fresh) current = selectProactiveApiKeyTransport(config, route.providerName, fresh) ?? fresh; } if (!current?.apiKey || !supportsNativeResponsesCompactEndpoint(route.providerName, current)) { throw new Error("Provider key selection is no longer available for compact"); diff --git a/structure/transports/responses-failover.md b/structure/transports/responses-failover.md index 1031c4799fd..3f9c366476b 100644 --- a/structure/transports/responses-failover.md +++ b/structure/transports/responses-failover.md @@ -156,7 +156,7 @@ The shared Responses path follows the [bounded multipart recovery contract](../s ## Upstream key attempt accounting -Key identity is sealed at guarded physical dispatch after queued selections are rebuilt. Distributed selections retain their request-local entry and revision. A rebuild after selection drift releases the old per-key lease and reacquires admission for the refreshed key; the send guard checks again after waiting. Turn-scoped adapters follow the same rule before invocation. Native Chat, Messages and keyed compact all rebuild stale queued selections before dispatch. Direct compact and Images record genuine distributed 401/429 cooldowns without replaying paid POSTs; synthetic replay refusals never cool a key. Native Messages revisits mixed distributed 401/429 outcomes with its existing counters. Native Chat and Messages require the physical outgoing credential to match the selected distributed key before pooling a rejection; Messages honors the configured `x-api-key` or Bearer transport while rejecting a conflicting credential header; configured credential overrides cannot cool unused entries. Replacement admission pauses the response-header deadline and retains client cancellation; the header deadline restarts once admission completes. Unpaced dispatch without a readmission owner fails closed. Replay notification errors release acquired slots before propagating, including pre-acquired-slot dispatch. +Key identity is sealed at guarded physical dispatch after queued selections are rebuilt. Distributed selections retain their request-local entry and revision. A rebuild after selection drift releases the old per-key lease and reacquires admission for the refreshed key; the send guard checks again after waiting. Turn-scoped adapters follow the same rule before invocation. Native Chat, Messages and keyed compact all rebuild stale queued selections before dispatch, including a manual active-key switch under per-key limits; they release the old lease, readmit the new key and honor cancellation while its capacity is occupied. Direct compact and Images record genuine distributed 401/429 cooldowns without replaying paid POSTs; synthetic replay refusals never cool a key. Native Messages revisits mixed distributed 401/429 outcomes with its existing counters. Native Chat and Messages require the physical outgoing credential to match the selected distributed key before pooling a rejection; Messages honors the configured `x-api-key` or Bearer transport while rejecting a conflicting credential header; configured credential overrides cannot cool unused entries. Replacement admission pauses the response-header deadline and retains client cancellation; the header deadline restarts once admission completes. Unpaced dispatch without a readmission owner fails closed. Replay notification errors release acquired slots before propagating, including pre-acquired-slot dispatch. Raw adapter terminal usage is recorded before continuation, search, or image loops merge it; repeated parsing of one physical response does not count it twice. Key changes preserve the previous attempt while retaining the active attempt object shared by streaming/combo callbacks. diff --git a/tests/claude-integration/messages-native-key-selection.test.ts b/tests/claude-integration/messages-native-key-selection.test.ts index b9773855395..6881ce0ac04 100644 --- a/tests/claude-integration/messages-native-key-selection.test.ts +++ b/tests/claude-integration/messages-native-key-selection.test.ts @@ -320,3 +320,38 @@ for (const statusCode of [401, 429]) for (const overridden of [false, true]) tes } unchanged(config); noLeaks(config); }); + +for (const cancel of [false, true]) test(`active manual key switch readmits the selected key (cancel=${cancel})`, async () => { + const config = fixture(); + config.providers[NAME]!.requestPacing = { enabled: true, keys: { maxConcurrentRequests: 1 } }; + saveConfig(config); + const heldA = await waitForProviderRequestSlot(NAME, config.providers[NAME]!); + const heldB = await waitForProviderRequestSlot(NAME, { ...config.providers[NAME]!, apiKey: KEYS[1] }); + const abort = new AbortController(); + const pending = start(config, abort); + try { + await queue(config, "k1"); + config.providers[NAME]!.apiKey = KEYS[1]; + config.providers[NAME]!.apiKeySelectionRevision = "manual-key-switch"; + saveConfig(config); + heldA.release(); + await queue(config, "k2"); + expect(keyStatus(config, "k1")).toMatchObject({ queued: 0, inFlight: 0 }); + expect(seen).toEqual([]); + if (cancel) abort.abort(); + else heldB.release(); + const result = await pending; + if (cancel) { + expect(result.response.status).toBe(499); + expect(seen).toEqual([]); + heldB.release(); + } else { nativeSuccess(result); + expect(seen.map(entry => entry.key)).toEqual([KEYS[1]]); } + const saved = JSON.parse(readFileSync(join(home, "config.json"), "utf8")) as OcxConfig; + expect(config.providers[NAME]!.apiKey).toBe(KEYS[1]); + expect(saved.providers[NAME]!.apiKey).toBe(KEYS[1]); + expect(saved.providers[NAME]!.apiKeySelectionRevision).toBe("manual-key-switch"); + expect(saved.providers[NAME]!._apiKeyAttempt).toBeUndefined(); + noLeaks(config); + } finally { abort.abort(); heldA.release(); heldB.release(); await pending.catch(() => undefined); } +}, 10_000); diff --git a/tests/responses/compact-key-selection.test.ts b/tests/responses/compact-key-selection.test.ts index 4084fc1c268..f232b0e32e9 100644 --- a/tests/responses/compact-key-selection.test.ts +++ b/tests/responses/compact-key-selection.test.ts @@ -246,3 +246,37 @@ test("a synthetic non-replayable compact 429 never cools a distributed key", asy for (let index = 0; index < 3; index++) expect(getKeyCooldownUntil(NAME, `k${index + 1}`)).toBeNull(); unchanged(config); noLeaks(config); }); + +for (const cancel of [false, true]) test(`active manual key switch readmits the selected key (cancel=${cancel})`, async () => { + const config = fixture(); + config.providers[NAME]!.requestPacing = { enabled: true, keys: { maxConcurrentRequests: 1 } }; + saveConfig(config); + const heldA = await waitForProviderRequestSlot(NAME, config.providers[NAME]!); + const heldB = await waitForProviderRequestSlot(NAME, { ...config.providers[NAME]!, apiKey: KEYS[1] }); + const abort = new AbortController(); + const pending = start(config, abort); + try { + await queue(config, "k1"); + config.providers[NAME]!.apiKey = KEYS[1]; + config.providers[NAME]!.apiKeySelectionRevision = "manual-key-switch"; + saveConfig(config); + heldA.release(); + await queue(config, "k2"); + expect(keyStatus(config, "k1")).toMatchObject({ queued: 0, inFlight: 0 }); + expect(seen).toEqual([]); + if (cancel) abort.abort(); + else heldB.release(); + const result = await pending; + if (cancel) { + expect(result.response.status).toBe(499); + expect(seen).toEqual([]); + heldB.release(); + } else { success(result, 1); } + const saved = JSON.parse(readFileSync(join(home, "config.json"), "utf8")) as OcxConfig; + expect(config.providers[NAME]!.apiKey).toBe(KEYS[1]); + expect(saved.providers[NAME]!.apiKey).toBe(KEYS[1]); + expect(saved.providers[NAME]!.apiKeySelectionRevision).toBe("manual-key-switch"); + expect(saved.providers[NAME]!._apiKeyAttempt).toBeUndefined(); + noLeaks(config); + } finally { abort.abort(); heldA.release(); heldB.release(); await pending.catch(() => undefined); } +}, 10_000); diff --git a/tests/server/server-chat-native-key-selection.test.ts b/tests/server/server-chat-native-key-selection.test.ts index a566a76113f..4f36ea91044 100644 --- a/tests/server/server-chat-native-key-selection.test.ts +++ b/tests/server/server-chat-native-key-selection.test.ts @@ -227,3 +227,37 @@ test("chat owned 401 rotates the rejected distributed credential before output", expect(getKeyCooldownUntil(NAME, "k2")).toBeNull(); unchanged(config); noLeaks(config); }); + +for (const cancel of [false, true]) test(`active manual key switch readmits the selected key (cancel=${cancel})`, async () => { + const config = fixture(); + config.providers[NAME]!.requestPacing = { enabled: true, keys: { maxConcurrentRequests: 1 } }; + saveConfig(config); + const heldA = await waitForProviderRequestSlot(NAME, config.providers[NAME]!); + const heldB = await waitForProviderRequestSlot(NAME, { ...config.providers[NAME]!, apiKey: KEYS[1] }); + const abort = new AbortController(); + const pending = start(config, abort); + try { + await queue(config, "k1"); + config.providers[NAME]!.apiKey = KEYS[1]; + config.providers[NAME]!.apiKeySelectionRevision = "manual-key-switch"; + saveConfig(config); + heldA.release(); + await queue(config, "k2"); + expect(keyStatus(config, "k1")).toMatchObject({ queued: 0, inFlight: 0 }); + expect(seen).toEqual([]); + if (cancel) abort.abort(); + else heldB.release(); + const result = await pending; + if (cancel) { + expect(result.response.status).toBe(499); + expect(seen).toEqual([]); + heldB.release(); + } else { success(result, 1); } + const saved = JSON.parse(readFileSync(join(home, "config.json"), "utf8")) as OcxConfig; + expect(config.providers[NAME]!.apiKey).toBe(KEYS[1]); + expect(saved.providers[NAME]!.apiKey).toBe(KEYS[1]); + expect(saved.providers[NAME]!.apiKeySelectionRevision).toBe("manual-key-switch"); + expect(saved.providers[NAME]!._apiKeyAttempt).toBeUndefined(); + noLeaks(config); + } finally { abort.abort(); heldA.release(); heldB.release(); await pending.catch(() => undefined); } +}, 10_000); From 75cfbd3395a8820baf25616e5273faf79e50638a Mon Sep 17 00:00:00 2001 From: GSL-R <174104911+GSL-R@users.noreply.github.com> Date: Tue, 13 Oct 2026 15:41:51 +0900 Subject: [PATCH 13/17] docs(providers): clarify queued key reselection and readmission --- docs-site/src/content/docs/reference/configuration/providers.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs-site/src/content/docs/reference/configuration/providers.md b/docs-site/src/content/docs/reference/configuration/providers.md index 0da0f68cea8..d97f7225365 100644 --- a/docs-site/src/content/docs/reference/configuration/providers.md +++ b/docs-site/src/content/docs/reference/configuration/providers.md @@ -368,7 +368,7 @@ For multiple API-key pool entries, opt into normal-request distribution separate This admits at most ten active requests across the provider and one per credential reference, with starts on each reference at least four seconds apart. These are local upper bounds, not a promise of upstream capacity. A provider-wide RPM setting, when present, remains an aggregate limit and does not multiply with key count. Key limits span models; model rules remain additional restrictions. Projects sharing quota need an appropriate aggregate limit; TPM and daily quotas are not tracked here. -Round-robin selects keys per request without saving a new active key. Known 401/429 cooldowns are skipped; when none remains eligible, new requests receive a retryable 429 instead of repeatedly sending. Already queued work retains its selected key and may wait behind that key even when another is idle; the bounded queue expires rather than waiting forever. Selection and diagnostics are process-local, reset on restart, and do not equal balanced token usage or billing. Starts count pacing admissions, including work cancelled before dispatch. OAuth and forwarded credentials keep their existing selection. The dashboard shows key handles, starts, in-flight work, queued work and known cooldowns, never secret key values. These limits cover the inference pacing executor, not the standalone Images relay's direct fetch. +Round-robin selects keys per request without saving a new active key. Known 401/429 cooldowns are skipped; when none remains eligible, new requests receive a retryable 429 instead of repeatedly sending. Already queued work normally retains its selected key and may wait behind that key even when another is idle. If that selection becomes cooled or invalid, the request selects an eligible key again and reacquires pacing admission for it; the bounded queue expires rather than waiting forever. Selection and diagnostics are process-local, reset on restart, and do not equal balanced token usage or billing. Starts count pacing admissions, including work cancelled before dispatch. OAuth and forwarded credentials keep their existing selection. The dashboard shows key handles, starts, in-flight work, queued work and known cooldowns, never secret key values. These limits cover the inference pacing executor, not the standalone Images relay's direct fetch. ### Azure OpenAI model metadata discovery From 3e7270678a482313cec547e8fc229daa838108e3 Mon Sep 17 00:00:00 2001 From: JUN Date: Thu, 15 Oct 2026 16:35:26 +0900 Subject: [PATCH 14/17] fix(providers): settle native key failures and fence CLI dispatch Capture native Responses key ownership at the physical HTTP boundary and settle terminal distributed rejections after same-key retries. Revalidate all queued turn admissions and fence CLI spawn after asynchronous setup. Preserve local pacing refusal metadata without cooling synthetic responses. Co-authored-by: GSL-R <174104911+GSL-R@users.noreply.github.com> --- scripts/test-layout/layout.json | 2 +- src/adapters/base.ts | 2 + src/adapters/coding-agent/turn.ts | 7 + src/providers/api-key-response-ownership.ts | 45 ++++ src/server/responses/passthrough-dispatch.ts | 16 +- src/server/responses/request-transport.ts | 50 +++-- structure/adapters/registry.md | 2 + structure/providers-and-adapters.md | 2 +- structure/transports/responses-failover.md | 4 +- structure/transports/responses.md | 2 +- .../adapters/qoder-spawn-readmission.test.ts | 73 +++++++ tests/fixtures/test-layout-expected.json | 1 + .../responses-native-key-cooldown.test.ts | 193 ++++++++++++++++++ ...responses-run-turn-key-readmission.test.ts | 155 ++++++++++++++ 14 files changed, 523 insertions(+), 31 deletions(-) create mode 100644 src/providers/api-key-response-ownership.ts create mode 100644 tests/adapters/qoder-spawn-readmission.test.ts create mode 100644 tests/responses/responses-native-key-cooldown.test.ts create mode 100644 tests/responses/responses-run-turn-key-readmission.test.ts diff --git a/scripts/test-layout/layout.json b/scripts/test-layout/layout.json index 9294f0d0f5a..d579cff4545 100644 --- a/scripts/test-layout/layout.json +++ b/scripts/test-layout/layout.json @@ -2,7 +2,7 @@ "version": 1, "root": "tests", "explicitShards": ["layout-clients.json"], - "explicit": { "api-key-selection-cooldown.test.ts": "providers", "server-chat-native-key-selection.test.ts": "server", "compact-key-selection.test.ts": "responses", "messages-native-key-selection.test.ts": "claude-integration", "service-log.test.ts": "service", + "explicit": { "responses-native-key-cooldown.test.ts": "responses", "responses-run-turn-key-readmission.test.ts": "responses", "qoder-spawn-readmission.test.ts": "adapters", "api-key-selection-cooldown.test.ts": "providers", "server-chat-native-key-selection.test.ts": "server", "compact-key-selection.test.ts": "responses", "messages-native-key-selection.test.ts": "claude-integration", "service-log.test.ts": "service", "codex-history-pending-wal.test.ts": "codex-integration", "responses-anthropic-billing-send-budget.test.ts": "responses", "anthropic-billing-ownership.test.ts": "adapters/anthropic", "server-anthropic-billing.test.ts": "server", "server-anthropic-billing-boundaries.test.ts": "server", "router-anthropic-billing-failover.test.ts": "routing", "anthropic-billing-refusal.test.ts": "adapters/anthropic", "messages-request-id-headers.test.ts": "claude-integration", "bounded-file-read.test.ts": "lib", "responses-undeclared-tool-guard-inbound.test.ts": "responses", "messages-request-id-endpoint.test.ts": "claude-integration", "anthropic-revoked-token.test.ts": "adapters/anthropic", diff --git a/src/adapters/base.ts b/src/adapters/base.ts index 431e3fb1309..b30f1586a78 100644 --- a/src/adapters/base.ts +++ b/src/adapters/base.ts @@ -22,6 +22,8 @@ export interface IncomingMeta { * the same pacing queue and custom provider fetch seam. */ providerFetch?: typeof globalThis.fetch; + /** Synchronous selection fence for adapter-owned sends; call after async setup, before dispatch. */ + beforePhysicalSend?: () => void; /** * Image-normalization ladder bias for upstream-413 tightened retries: every image * starts one tier lower (devlog/260714_image_normalization_pipeline/030). Consumed by diff --git a/src/adapters/coding-agent/turn.ts b/src/adapters/coding-agent/turn.ts index e6c88eea86c..0d5e5e4d458 100644 --- a/src/adapters/coding-agent/turn.ts +++ b/src/adapters/coding-agent/turn.ts @@ -304,12 +304,19 @@ export async function runCodingAgentTurn(input: CodingAgentTurnInput): Promise(); +registerProviderResponseProvenance(owners); + +/** Capture the credential after connection-policy rewrites, immediately before fetch. */ +export function distributedKeyPhysicalFetch( + physicalFetch: typeof globalThis.fetch, providerName: string, provider: OcxProviderConfig, target: string, +): typeof globalThis.fetch { + if (!provider._apiKeyAttempt?.distributed) return physicalFetch; + const wrapped = (async (input, init) => { + const headers = new Headers(init?.headers ?? (input instanceof Request ? input.headers : undefined)); + const destination = input instanceof Request ? input.url : String(input); + const ownsCredential = destination === target && headers.get("authorization") === `Bearer ${provider.apiKey}` + && !headers.has("x-api-key") && !headers.has("api-key"); + const owner: KeyResponseOwner | null = provider._apiKeyAttempt?.distributed && ownsCredential + ? { providerName, provider: { ...provider, _apiKeyAttempt: { ...provider._apiKeyAttempt } } } : null; + const response = await physicalFetch(input, init); + // Reused response identities cannot safely describe two physical sends. + owners.set(response, owners.has(response) ? null : owner); + return response; + }) as typeof globalThis.fetch; + return isEgressTransparentExecutor(physicalFetch) ? markEgressTransparentExecutor(wrapped) : wrapped; +} + +/** Settle only the terminal refusal; configured same-key 429 retries retain their key. */ +export function recordDistributedResponseFailure( + response: Response, config: OcxConfig, providerName: string, signal?: AbortSignal, +): void { + if (signal?.aborted || isNonReplayableResponse(response) || (response.status !== 401 && response.status !== 429)) return; + const owner = owners.get(response); + if (!owner || owner.providerName !== providerName + || !providerApiKeySelectionIsCurrent(config, providerName, owner.provider)) return; + recordDistributedKeyFailure(config, providerName, owner.provider._apiKeyAttempt, + response.status, Date.now(), response.headers.get("retry-after")); +} diff --git a/src/server/responses/passthrough-dispatch.ts b/src/server/responses/passthrough-dispatch.ts index 58a1d8be395..8f1e8ff7f3b 100644 --- a/src/server/responses/passthrough-dispatch.ts +++ b/src/server/responses/passthrough-dispatch.ts @@ -1,4 +1,6 @@ import { spendLedgerStorageErrorResponse } from "./spend-storage-error"; +import { requestPacingOverloadResponse } from "./pacing-overload"; +import { recordDistributedResponseFailure } from "../../providers/api-key-response-ownership"; import { createSteeringSettingsNormalizer } from "./native-steering-policy"; import { nativeResponseControlEligible } from "./native-response-control"; import { NativeInjectionReplay } from "./native-injection-replay"; @@ -742,7 +744,9 @@ export async function preparePassthroughExchange( const stage = readCodexWsStage(response); if (stage && logCtx.activeAttempt) logCtx.activeAttempt.codexWsStage = stage; }; + let upstreamResponse: Response, lastDistributedResponse: Response | undefined; const adoptObservedResponse = (response: T): T => { + lastDistributedResponse = response.status === 401 || response.status === 429 ? response : undefined; settleObservedHostResponse(); adoptCodexWsStage(response); return response; @@ -759,7 +763,6 @@ export async function preparePassthroughExchange( const upstream = new AbortController(); const cleanupUpstreamAbort = linkAbortSignal(upstream, options.abortSignal ?? req.signal); const connectMs = config.connectTimeoutMs ?? 200_000; - let upstreamResponse: Response; /** * This leg's transient-5xx ladder cap, from the provider's own `transientRetryOn5xx`. * @@ -930,6 +933,7 @@ export async function preparePassthroughExchange( releaseCodexAuthContextProbeLease(admissionState.authCtx); return clientCancelledResponse(); } + if (lastDistributedResponse) recordDistributedResponseFailure(lastDistributedResponse, config, route.providerName, options.abortSignal); // A budget refusal is a proxy decision, not an upstream fault. Reporting it as // 502 upstream_error would blame the provider for a limit this process applied, and // would record a fake reachability failure against the account's health. @@ -945,12 +949,12 @@ export async function preparePassthroughExchange( bindRefreshedAdapter(refusal.adapter); return handoffToChat(refusal.adapter); } - const copilotRefusal = copilotRefusalResponse(refusal); - if (copilotRefusal) { + const dispatchRefusal = requestPacingOverloadResponse(refusal) ?? copilotRefusalResponse(refusal); + if (dispatchRefusal) { releaseUpstreamHostAdmission(nativeHostState.lease); nativeHostState.lease = null; releaseCodexAuthContextProbeLease(admissionState.authCtx); - return copilotRefusal; + return dispatchRefusal; } // Pacing may outlive the selected account's admission. No fetch occurred, so do // not turn an operator pause into a 502 or charge it to host/account health. @@ -1526,7 +1530,6 @@ export async function preparePassthroughExchange( )) noteAttemptRecoveryWithheld(logCtx.activeAttempt, "rotation-send-budget"); } } - // Same-target 429 wait-and-retry (opt-in `retryOn429`) for key-auth providers on the // passthrough wire. This branch returns before the recovery loop below, so Responses-shaped // key-auth gateways (e.g. the built-in DeepSeek preset) would otherwise surface 429 @@ -1597,7 +1600,6 @@ export async function preparePassthroughExchange( return transportFailureResponse(err); } } - const captureAffinityResponse = ( response: Response, captureAuthCtx: CodexAuthContext = admissionState.authCtx, @@ -1619,8 +1621,8 @@ export async function preparePassthroughExchange( status: response.status, }); }; + recordDistributedResponseFailure(upstreamResponse, config, route.providerName, options.abortSignal); captureAffinityResponse(upstreamResponse); - if (usesCodexForwardPoolAuth(admissionState.authCtx, route.provider)) { let poolRetryOutcome: number | undefined; // A success is the freshest evidence there is about this pair, and it outranks any earlier diff --git a/src/server/responses/request-transport.ts b/src/server/responses/request-transport.ts index 29b08f0ead5..45d500ac9d6 100644 --- a/src/server/responses/request-transport.ts +++ b/src/server/responses/request-transport.ts @@ -78,6 +78,7 @@ import { bindRouteReasoningReplayScope } from "./core-replay"; import { sessionIdHeaderFromRequest, normalizeLogConversationId } from "../request-log-conversation"; import { redactSecretString } from "../../lib/redact"; import { selectProactiveApiKeyTransport } from "../../providers/key-failover"; +import { distributedKeyPhysicalFetch } from "../../providers/api-key-response-ownership"; import { isCanonicalOpenAiForwardProvider } from "../../providers/openai-tiers"; import { providerConsumesCallerAuthorization } from "../../providers/caller-authorization"; import { releaseCodexAuthContextProbeLease, stripCodexRuntimeProviderFields } from "../../codex/auth-context"; @@ -598,8 +599,8 @@ export async function prepareResponsesTransport( } } else { let current = resolveCurrentProviderApiKeyTransport(config, route.providerName, route.provider); - if (!current && route.provider.requestPacing?.enabled - && (route.provider.requestPacing.keys || route.provider.requestPacing.keySelection === "round-robin")) { + if (!current && (route.provider._apiKeyAttempt?.distributed || (route.provider.requestPacing?.enabled + && (route.provider.requestPacing.keys || route.provider.requestPacing.keySelection === "round-robin")))) { const fresh = resolveCurrentProviderApiKeyTransport(config, route.providerName, { ...route.provider, _apiKeyAttempt: undefined }); if (fresh) current = selectProactiveApiKeyTransport(config, route.providerName, fresh) ?? fresh; } @@ -630,7 +631,9 @@ export async function prepareResponsesTransport( ? (options.sendBudget as import("../../lib/request-execution-budget").RequestExecutionBudget).beginSpendProducer?.() : undefined; try { for (let attempt = 0; attempt < 3; attempt++) { + incoming.abortSignal?.throwIfAborted(); const selectionRefreshed = !selectionIsCurrent(adapterBindings.get(selectedAdapter)); + if (attempt === 0 && selectionRefreshed) releaseProviderRequestSlot(incoming.pacingSlot); if (selectionRefreshed) selectedAdapter = await refreshRunTurnAdapter(requestParsed); const binding = adapterBindings.get(selectedAdapter); const run = rawRunTurns.get(selectedAdapter); @@ -642,29 +645,38 @@ export async function prepareResponsesTransport( // Subsequent physical messages retain this adapter/credential and are paced normally. try { // A pre-acquired lease belongs to the original selection, never its replacement. - if (attempt === 0 && selectionRefreshed) releaseProviderRequestSlot(incoming.pacingSlot); attemptSlot = attempt === 0 && !selectionRefreshed && incoming.pacingSlot ? incoming.pacingSlot : await waitForProviderRequestSlot(route.providerName, route.provider, route.modelId, incoming.abortSignal); + incoming.abortSignal?.throwIfAborted(); + // Every admission can queue, including a replacement's. Never enter an + // adapter on a lease whose credential changed while it waited. + if (!selectionIsCurrent(binding)) { + refused = true; + throw new Error("Account selection changed before turn admission"); + } + const beforePhysicalSend = (): void => { + incoming.abortSignal?.throwIfAborted(); + if (sent) { + if (binding?.kind === "api-key" && binding.provider._apiKeyAttempt?.distributed + && !selectionIsCurrent(binding)) throw new Error("API key became unavailable during the turn"); + return; + } + if (!selectionIsCurrent(binding)) { + refused = true; + throw new Error("Account selection changed before the first turn dispatch"); + } + incoming.beforePhysicalSend?.(); + commitKeyAttemptSend(); + sent = true; + }; const fetch = providerFetch(route.provider, options.codexWsRuntimeIdentity, { providerName: route.providerName, modelId: route.modelId, pacingSlotAcquired: true, pacingSlot: attemptSlot, turnScopedPacing: true, - beforeDispatch: () => { - if (sent) { - if (binding?.kind === "api-key" && binding.provider._apiKeyAttempt?.distributed - && !selectionIsCurrent(binding)) throw new Error("API key became unavailable during the turn"); - return; - } - if (!selectionIsCurrent(binding)) { - refused = true; - throw new Error("Account selection changed before the first turn dispatch"); - } - commitKeyAttemptSend(); - sent = true; - }, + beforeDispatch: beforePhysicalSend, }); - await run(requestParsed, { ...incoming, pacingSlot: attemptSlot, providerFetch: fetch }, event => { if (!refused) emit(event); }); + await run(requestParsed, { ...incoming, pacingSlot: attemptSlot, providerFetch: fetch, beforePhysicalSend }, event => { if (!refused) emit(event); }); } catch (error) { if (!refused) throw error; } finally { @@ -673,7 +685,7 @@ export async function prepareResponsesTransport( if (!refused) return; // The adapter may map the guard's exception to an error event. Neither that // event nor a refused send may escape before retrying the newly selected account. - selectedAdapter = await refreshRunTurnAdapter(requestParsed); + if (attempt < 2) selectedAdapter = await refreshRunTurnAdapter(requestParsed); } throw new Error("Account selection changed repeatedly before turn dispatch"); } finally { producer?.close(); } @@ -731,7 +743,7 @@ export async function prepareResponsesTransport( } commitKeyAttemptSend(); response = await sendWithConnectionPolicy( - anthropicBillingPhysicalFetch(fetchImpl, route.providerName, route.provider), + distributedKeyPhysicalFetch(anthropicBillingPhysicalFetch(fetchImpl, route.providerName, route.provider), route.providerName, route.provider, target), destination, { ...dispatchInit, redirect: "manual" }, { providerName: route.providerName, provider: route.provider }, diff --git a/structure/adapters/registry.md b/structure/adapters/registry.md index fb58c444e97..98bc00bc034 100644 --- a/structure/adapters/registry.md +++ b/structure/adapters/registry.md @@ -44,6 +44,8 @@ Some adapters share another adapter's routed-tool semantics while retaining inde machine for `api.anthropic.com`, and `local` (Ollama, vLLM, LM Studio) is the classification for traffic that never does. `keyOptional` is the existing exemption from key enforcement, and key rows are what `deriveProviderPresets` lists, so the entry needs no `dashboardPreset` flag either. +The shared `src/adapters/coding-agent/turn.ts` calls the synchronous `IncomingMeta.beforePhysicalSend` fence after awaited tool-bridge setup and before spending its dispatch permit or spawning. Cancellation and stale selections produce no child process; cleanup still returns permits and removes temporary tool files. + - `cursor` stays direct because its `runTurn` transport and gated native-file fallback are distinct. - `devin` is direct for a related reason. It streams Cognition's `ApiServerService/GetChatMessage` over Connect-RPC from `runTurn` with hand-written protobuf diff --git a/structure/providers-and-adapters.md b/structure/providers-and-adapters.md index 9a13e79aeb8..d1779e21168 100644 --- a/structure/providers-and-adapters.md +++ b/structure/providers-and-adapters.md @@ -288,7 +288,7 @@ Cursor, one lease spans the whole turn: RunSSE and BidiAppend may overlap inside turns wait at the cap. Follow-up sends still obey start intervals. The turn owner returns its lease after `runTurn` settles, so RunSSE body completion cannot admit another turn early. A capped canonical Codex WebSocket turn uses HTTP/SSE because the socket has no response-body lifecycle. -Capacity waits use the same bounded queue and retryable queue-overload errors as interval waits. Optional `requestPacing.keys` adds a shared cross-model limit for each credential reference; duplicate references share capacity, but distinct references resolving to one upstream project do not imply separate quota. `keySelection: round-robin` assigns request-local healthy keys without mutating persisted selection, skips known cooldowns, and refuses exhausted pools with retryable 429. Provider, model and key gates reserve atomically; key counters and the allocation cursor are process-local, not token or billing balances. OAuth and forwarded credentials retain their existing selection. Cooldown state is shared through the dependency-light `src/providers/api-key-cooldowns.ts` (retaining the longest outstanding expiry until explicit manual clearing) owner. Distributed selections are checked again at physical dispatch; a cooled queued key releases its old permit, selects a healthy sibling, and reacquires pacing before sending. Native Messages follows the same readmission rule. An established distributed turn refuses later sends after its key cools instead of changing credentials mid-turn. +Capacity waits use the same bounded queue and retryable queue-overload errors as interval waits. Optional `requestPacing.keys` adds a shared cross-model limit for each credential reference; duplicate references share capacity, but distinct references resolving to one upstream project do not imply separate quota. `keySelection: round-robin` assigns request-local healthy keys without mutating persisted selection, skips known cooldowns, and refuses exhausted pools with retryable 429. Provider, model and key gates reserve atomically; key counters and the allocation cursor are process-local, not token or billing balances. OAuth and forwarded credentials retain their existing selection. Cooldown state is shared through the dependency-light `src/providers/api-key-cooldowns.ts` (retaining the longest outstanding expiry until explicit manual clearing) owner. Distributed selections are checked again at physical dispatch; a cooled queued key releases its old permit, selects a healthy sibling, and reacquires pacing before sending. Native Messages follows the same readmission rule. Native Responses settles terminal distributed refusals from immutable physical-send metadata; successful same-key retry and unowned refusals do not cool keys. CLI-backed adapters check `IncomingMeta.beforePhysicalSend` after asynchronous setup and before spawn. An established distributed turn refuses later sends after its key cools instead of changing credentials mid-turn. Command Code effort defaults in `src/providers/command-code-efforts.ts` combine public-profile facts with the live API measurements from #5096. Both presets share the exact per-model rows; diff --git a/structure/transports/responses-failover.md b/structure/transports/responses-failover.md index 3bd6c185877..aafe4d5d68b 100644 --- a/structure/transports/responses-failover.md +++ b/structure/transports/responses-failover.md @@ -156,8 +156,8 @@ The shared Responses path follows the [bounded multipart recovery contract](../s ## Upstream key attempt accounting -Key identity is sealed at guarded physical dispatch after queued selections are rebuilt. Distributed selections retain their request-local entry and revision. A rebuild after selection drift releases the old per-key lease and reacquires admission for the refreshed key; the send guard checks again after waiting. Turn-scoped adapters follow the same rule before invocation. Native Chat, Messages and keyed compact all rebuild stale queued selections before dispatch, including a manual active-key switch under per-key limits; they release the old lease, readmit the new key and honor cancellation while its capacity is occupied. Direct compact and Images record genuine distributed 401/429 cooldowns without replaying paid POSTs; synthetic replay refusals never cool a key. Native Messages revisits mixed distributed 401/429 outcomes with its existing counters. Native Chat and Messages require the physical outgoing credential to match the selected distributed key before pooling a rejection; Messages honors the configured `x-api-key` or Bearer transport while rejecting a conflicting credential header; configured credential overrides cannot cool unused entries. Replacement admission pauses the response-header deadline and retains client cancellation; the header deadline restarts once admission completes. Unpaced dispatch without a readmission owner fails closed. Replay notification errors release acquired slots before propagating, including pre-acquired-slot dispatch. Translated Anthropic billing recovery uses the [immutable physical-send ownership contract](../providers/anthropic-key-billing.md). Its initial send, credential hop and later mixed-class refusals share the request budget. A denied hop retains the upstream refusal, and a leased pacing body rewrap carries the existing marker rather than attributing the response to rebuilt headers. -Before a turn-scoped retry readmits a refreshed selection, it rejects a binding already recorded as failed for this request and returns the last key failure. Eligible replacement keys still release the old lease and reacquire their own pacing capacity before invocation. Raw adapter terminal usage is recorded before continuation, search, or image loops merge it; +Key identity is sealed at guarded physical dispatch after queued selections are rebuilt. Distributed selections retain their request-local entry and revision. A rebuild after selection drift releases the old per-key lease and reacquires admission for the refreshed key; the send guard checks again after waiting. Turn-scoped adapters follow the same rule before invocation. Native Chat, Messages and keyed compact all rebuild stale queued selections before dispatch, including a manual active-key switch under per-key limits; they release the old lease, readmit the new key and honor cancellation while its capacity is occupied. Direct compact and Images record genuine distributed 401/429 cooldowns without replaying paid POSTs; synthetic replay refusals never cool a key. Native Messages revisits mixed distributed 401/429 outcomes with its existing counters. Native Chat and Messages require the physical outgoing credential to match the selected distributed key before pooling a rejection; Messages honors the configured `x-api-key` or Bearer transport while rejecting a conflicting credential header; configured credential overrides cannot cool unused entries. Replacement admission pauses the response-header deadline and retains client cancellation; the header deadline restarts once admission completes. Native Responses captures distributed-key ownership after connection-policy rewrites in `src/providers/api-key-response-ownership.ts`; terminal 401/429 responses settle cooldown only after same-key retry policy, and a failed replay admission retains the last genuine refusal. Reused responses, credential/destination overrides, stale revisions and synthetic refusals cannot confer ownership. Local pacing errors retain 429 and Retry-After at the passthrough catch boundary. Unpaced dispatch without a readmission owner fails closed. Replay notification errors release acquired slots before propagating, including pre-acquired-slot dispatch. Translated Anthropic billing recovery uses the [immutable physical-send ownership contract](../providers/anthropic-key-billing.md). Its initial send, credential hop and later mixed-class refusals share the request budget. A denied hop retains the upstream refusal, and a leased pacing body rewrap carries the existing marker rather than attributing the response to rebuilt headers. +Before a turn-scoped retry readmits a refreshed selection, it rejects a binding already recorded as failed for this request and returns the last key failure. Eligible replacement keys still release the old lease and reacquire their own pacing capacity before invocation, rechecking after every admission. CLI-backed turns carry a synchronous selection fence through asynchronous tool setup to the physical spawn boundary. Raw adapter terminal usage is recorded before continuation, search, or image loops merge it; repeated parsing of one physical response does not count it twice. Key changes preserve the previous attempt while retaining the active attempt object shared by streaming/combo callbacks. Bounded failure-body observation retains reported usage and releases cloned readers on abort. diff --git a/structure/transports/responses.md b/structure/transports/responses.md index 17ab5eb2198..90ce9d141ef 100644 --- a/structure/transports/responses.md +++ b/structure/transports/responses.md @@ -427,7 +427,7 @@ is composed from the following owners in `src/server/responses/`; none is a gene | `passthrough-delivery.ts`, `terminal-error-redaction.ts`, `non-replayable-error.ts` | Native HTTP/SSE/JSON delivery and terminal accounting. Marked real errors use bounded reads to retain only allowlisted error type and code, keep status and non-replayability plus `x-should-retry: false` when present, and withhold upstream body text and all other upstream headers; the client receives a fixed generic message and synthetic refusals stay distinct. SSE diagnostics and xAI tool-envelope filtering precede client delivery/continuation storage. | | `policy-refusal.ts` | Rewrites an allowlisted non-combo HTTP 403 model refusal (`isUpstreamPolicyRefusal` in `src/lib/errors.ts`) from an xAI destination only (`isXaiResponsesDestination`: api.x.ai or the Grok CLI proxy, on either wire) to an HTTP 200 Responses `incomplete` / `content_filter` payload, JSON or SSE, for both `adapter-dispatch.ts` and `passthrough-delivery.ts`. A streamed rewrite takes the turn admission lease and releases it when the body finishes, so the refusal stays inside active-turn accounting. Combo attempts keep the original 403 so failover classifies it as a hop. | | `sidecar-execution.ts`, `sidecar-send-budget.ts` | Image/video versus web-search execution, shared rotation, and adapter-aware prepaid inference/producer ownership under the [spend contract](responses-spend.md#prepaid-initial-sends). | -| `completion-policy.ts`, `run-turn-execution.ts` | Empty-completion eligibility and adapter-owned event turns; refreshed retries reject previously failed key bindings before [replacement-key readmission](responses-failover.md#upstream-key-attempt-accounting). | +| `completion-policy.ts`, `run-turn-execution.ts` | Empty-completion eligibility and adapter-owned event turns; refreshed retries reject previously failed key bindings and revalidate after each queued admission; adapter-owned CLI spawns receive the post-setup `IncomingMeta.beforePhysicalSend` fence, with [replacement-key readmission](responses-failover.md#upstream-key-attempt-accounting). | | `adapter-dispatch.ts` | Translated initial dispatch, bounded recovery and the shared continuation retry counter. Initial and rebuilt adapter budget refusals retain the HTTP 413 [byte-accounting contract](byte-accounting.md#request-copy-accounting); ordinary build errors remain 400. | | `adapter-continuation.ts`, `adapter-delivery.ts` | Continuation event sources and final streaming/buffered bridging; a streamed turn with a `clientEncoder` option is handed to `src/server/inference/client-encoder-delivery.ts` instead of the bridge. | diff --git a/tests/adapters/qoder-spawn-readmission.test.ts b/tests/adapters/qoder-spawn-readmission.test.ts new file mode 100644 index 00000000000..1e499db4dee --- /dev/null +++ b/tests/adapters/qoder-spawn-readmission.test.ts @@ -0,0 +1,73 @@ +import { expect, test } from "bun:test"; +import { EventEmitter } from "node:events"; +import { existsSync } from "node:fs"; +import { writeFile } from "node:fs/promises"; +import { dirname } from "node:path"; +import { Readable, Writable } from "node:stream"; +import type { ChildProcess } from "node:child_process"; +import { createQoderAdapter } from "../../src/adapters/qoder/adapter"; +import { clearQoderBinaryCache } from "../../src/adapters/qoder/profiles"; +import type { IncomingMeta } from "../../src/adapters/base"; +import { createRequestExecutionBudget } from "../../src/lib/request-execution-budget"; +import type { AdapterEvent, OcxParsedRequest } from "../../src/types"; +import { createTestTranslatorBudget } from "../helpers/translator-budget"; + +function completedChild(): ChildProcess { + const child = new EventEmitter() as ChildProcess; + child.stdout = Readable.from([ + JSON.stringify({ type: "system", subtype: "init", mcp_servers: [{ name: "opencodex", status: "connected" }] }) + "\n", + JSON.stringify({ type: "result", subtype: "success", is_error: false }) + "\n", + ].map(frame => new TextEncoder().encode(frame))); + child.stderr = Readable.from([]); + child.stdin = new Writable({ write(_chunk, _encoding, callback) { callback(); } }); + child.exitCode = 0; + child.kill = () => true; + return child; +} + +for (const invalidation of ["selection", "abort", "valid"] as const) { + test(`Qoder fences spawn after asynchronous tool setup (${invalidation})`, async () => { + clearQoderBinaryCache(); + const abort = new AbortController(); + const events: AdapterEvent[] = []; + const sendBudget = createRequestExecutionBudget(); + let current = true; + let staged = 0; + let guardCalls = 0; + let spawns = 0; + let physicalSends = 0; + let bridgeDir: string | undefined; + const adapter = createQoderAdapter({ adapter: "qoder", authMode: "key", baseUrl: "https://qoder.com", apiKey: "synthetic-pat" }, { + which: () => "/synthetic/qoder", + async writeToolBridgeFile(...args) { + bridgeDir = dirname(String(args[0])); + await writeFile(...args); + if (++staged === 2) { + current = invalidation !== "selection"; + if (invalidation === "abort") abort.abort(); + } + }, + spawn: () => { spawns++; return completedChild(); }, + }); + const parsed = { modelId: "Qwen3.8-Max", stream: false, options: {}, + context: { tools: [{ name: "probe", parameters: { type: "object", properties: {} } }], messages: [{ role: "user", content: "hello", timestamp: 0 }] }, + } as OcxParsedRequest; + const incoming: IncomingMeta = { + headers: new Headers(), translatorBudget: createTestTranslatorBudget(), abortSignal: abort.signal, sendBudget, + beforePhysicalSend() { guardCalls++; if (!current) throw new Error("selection invalidated during setup"); }, + onPhysicalSend() { physicalSends++; }, + }; + const pending = adapter.runTurn!(parsed, incoming, event => events.push(event)); + if (invalidation === "selection") await expect(pending).rejects.toThrow("selection invalidated during setup"); + else await pending; + expect(staged).toBe(2); + expect(spawns).toBe(invalidation === "valid" ? 1 : 0); + expect(physicalSends).toBe(invalidation === "valid" ? 1 : 0); + expect(sendBudget.remainingBaseSends(3)).toBe(invalidation === "valid" ? 2 : 3); + expect(guardCalls).toBe(invalidation === "abort" ? 0 : 1); + expect(bridgeDir).toBeDefined(); + expect(existsSync(bridgeDir!)).toBe(false); + if (invalidation === "valid") expect(events.at(-1)).toMatchObject({ type: "done" }); + else expect(events.some(event => event.type === "done")).toBe(false); + }); +} diff --git a/tests/fixtures/test-layout-expected.json b/tests/fixtures/test-layout-expected.json index 7e0cc450d04..ff3da01da62 100644 --- a/tests/fixtures/test-layout-expected.json +++ b/tests/fixtures/test-layout-expected.json @@ -1,4 +1,5 @@ { + "responses-native-key-cooldown.test.ts": "responses", "responses-run-turn-key-readmission.test.ts": "responses", "qoder-spawn-readmission.test.ts": "adapters", "api-key-selection-cooldown.test.ts": "providers", "server-chat-native-key-selection.test.ts": "server", "compact-key-selection.test.ts": "responses", "messages-native-key-selection.test.ts": "claude-integration", "codex-history-pending-wal.test.ts": "codex-integration", diff --git a/tests/responses/responses-native-key-cooldown.test.ts b/tests/responses/responses-native-key-cooldown.test.ts new file mode 100644 index 00000000000..c229381a10a --- /dev/null +++ b/tests/responses/responses-native-key-cooldown.test.ts @@ -0,0 +1,193 @@ +import { afterEach, beforeEach, expect, test } from "bun:test"; +import { mkdtempSync, readFileSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { saveConfig } from "../../src/config"; +import { markResponseNonReplayable } from "../../src/lib/upstream-retry"; +import { captureProviderApiKeySelection } from "../../src/providers/api-key-selection"; +import { clearKeyCooldowns, forgetApiKeyRotationCursor, getKeyCooldownUntil, recordDistributedKeyFailure } from "../../src/providers/key-failover"; +import { providerRequestPacingStatus, resetProviderRequestPacingForTest, setProviderRequestPacingLimitsForTest, waitForProviderRequestSlot } from "../../src/providers/request-pacing"; +import { handleResponses } from "../../src/server/responses"; +import type { OcxConfig } from "../../src/types"; +import { acquireOwnedSpendHome } from "../helpers/owned-spend-home"; +import { removeCompactionFixture } from "../helpers/compaction-routing-fixtures"; + +const NAME = "native-key-fixture"; +const KEYS = ["fixture-key-alpha", "fixture-key-beta"]; +const originalFetch = globalThis.fetch; +let home: string; +let previousHome: string | undefined; +let releaseOwner: () => void; +let seen: string[]; +beforeEach(() => { + previousHome = process.env.OPENCODEX_HOME; + home = mkdtempSync(join(tmpdir(), "ocx-native-keys-")); + process.env.OPENCODEX_HOME = home; + releaseOwner = acquireOwnedSpendHome(); + seen = []; + clearKeyCooldowns(); forgetApiKeyRotationCursor(); resetProviderRequestPacingForTest(); +}); +afterEach(async () => { + globalThis.fetch = originalFetch; + releaseOwner(); + clearKeyCooldowns(); forgetApiKeyRotationCursor(); resetProviderRequestPacingForTest(); + await removeCompactionFixture(home); + if (previousHome === undefined) delete process.env.OPENCODEX_HOME; + else process.env.OPENCODEX_HOME = previousHome; +}); +function fixture(): OcxConfig { + const config = { port: 0, defaultProvider: NAME, providers: { [NAME]: { + adapter: "openai-responses", baseUrl: "https://gateway.example/v1", authMode: "key", + apiKey: KEYS[0], apiKeySelectionRevision: "fixture-revision", + apiKeyPool: KEYS.map((key, index) => ({ id: `k${index + 1}`, key, addedAt: index + 1 })), + requestPacing: { enabled: true, keySelection: "round-robin", keys: { maxConcurrentRequests: 1 } }, + } } } as OcxConfig; + saveConfig(config); + return config; +} +function start(config: OcxConfig, signal?: AbortSignal) { + const req = new Request("http://localhost/v1/responses", { method: "POST", signal, + headers: { "content-type": "application/json" }, + body: JSON.stringify({ model: `${NAME}/test`, input: "fixture question", stream: false, store: false }), + }); + return handleResponses(req, config, { model: "", provider: "" }).then(async response => ({ response, text: await response.text() })); +} +function intercept(reply: (headers: Headers) => Response | Promise) { + globalThis.fetch = (async (input, init) => { + expect(String(input)).toBe("https://gateway.example/v1/responses"); + const headers = new Headers(init?.headers); + seen.push(headers.get("authorization")!); + return reply(headers); + }) as typeof fetch; +} +function unchanged(config: OcxConfig) { + for (const provider of [config.providers[NAME]!, JSON.parse(readFileSync(join(home, "config.json"), "utf8")).providers[NAME]]) { + expect(provider.apiKey).toBe(KEYS[0]); + expect(provider.apiKeySelectionRevision).toBe("fixture-revision"); + expect(provider._apiKeyAttempt).toBeUndefined(); + } + const status = providerRequestPacingStatus(NAME, config.providers[NAME]!); + expect(status.queued).toBe(0); expect(status.inFlight ?? 0).toBe(0); +} +const completed = () => Response.json({ id: "resp_fixture", object: "response", status: "completed", output: [] }); +for (const status of [401, 429] as const) test(`native Responses cools genuine distributed ${status} without replay`, async () => { + const config = fixture(); const began = Date.now(); + intercept(headers => headers.get("authorization") === `Bearer ${KEYS[0]}` + ? Response.json({ error: { message: "fixture refusal", type: "upstream_error" } }, { status, headers: { "retry-after": "45" } }) : completed()); + const first = await start(config); + expect(first.response.status).toBe(status); + if (status === 429) expect(first.response.headers.get("retry-after")).toBe("45"); + expect(seen).toHaveLength(1); + expect(getKeyCooldownUntil(NAME, "k1")).toBeGreaterThanOrEqual(began + (status === 401 ? 600_000 : 45_000)); + expect((await start(config)).response.status).toBe(200); + expect((await start(config)).response.status).toBe(200); + expect(seen).toEqual([`Bearer ${KEYS[0]}`, `Bearer ${KEYS[1]}`, `Bearer ${KEYS[1]}`]); + unchanged(config); +}); +test("native same-key 429 retry succeeds without cooling the credential", async () => { + const config = fixture(); config.providers[NAME]!.retryOn429 = { attempts: 1, intervalMs: 1 }; + intercept(() => seen.length === 1 ? Response.json({ error: { message: "fixture capacity" } }, { status: 429, headers: { "retry-after": "0" } }) : completed()); + expect((await start(config)).response.status).toBe(200); + expect(seen).toEqual([`Bearer ${KEYS[0]}`, `Bearer ${KEYS[0]}`]); + expect(getKeyCooldownUntil(NAME, "k1")).toBeNull(); + unchanged(config); +}); +for (const mode of ["override", "synthetic", "revision"] as const) test(`native Responses does not cool an unowned refusal (${mode})`, async () => { + const config = fixture(); + if (mode === "override") config.providers[NAME]!.headers = { Authorization: "Bearer fixture-header-override" }; + intercept(() => { + if (mode === "revision") config.providers[NAME]!.apiKeySelectionRevision = "replacement-revision"; + const response = Response.json({ error: { message: "fixture refusal" } }, { status: 401 }); + if (mode === "synthetic") markResponseNonReplayable(response); + return response; + }); + expect((await start(config)).response.status).toBe(401); + expect(getKeyCooldownUntil(NAME, "k1")).toBeNull(); + expect(getKeyCooldownUntil(NAME, "k2")).toBeNull(); + expect(seen).toHaveLength(1); +}); +async function queued(config: OcxConfig) { + const deadline = Date.now() + 2_000; + while (providerRequestPacingStatus(NAME, config.providers[NAME]!).queued !== 1) { + if (Date.now() >= deadline) throw new Error("native fixture never queued"); + await Bun.sleep(5); + } +} +function cool(config: OcxConfig, index: number) { + expect(recordDistributedKeyFailure(config, NAME, { + ...captureProviderApiKeySelection({ ...config.providers[NAME]!, apiKey: KEYS[index] }), distributed: true, + }, 429, Date.now(), "45")).toBe(true); +} +for (const mode of ["exhausted", "expired", "cancelled"] as const) test(`queued native key refusal maps to local status (${mode})`, async () => { + const config = fixture(); + if (mode === "expired") setProviderRequestPacingLimitsForTest({ maxQueueAgeMs: 50 }); + intercept(completed); + const first = await waitForProviderRequestSlot(NAME, config.providers[NAME]!); + const second = await waitForProviderRequestSlot(NAME, { ...config.providers[NAME]!, apiKey: KEYS[1] }); + const abort = new AbortController(); + const pending = start(config, abort.signal); void pending.catch(() => {}); + try { + await queued(config); cool(config, 0); + if (mode === "exhausted") cool(config, 1); + first.release(); + if (mode !== "exhausted") await queued(config); + if (mode === "cancelled") abort.abort(); + const result = await pending; + expect(result.response.status).toBe(mode === "cancelled" ? 499 : 429); + if (mode !== "cancelled") expect(Number(result.response.headers.get("retry-after"))).toBeGreaterThan(0); + expect(seen).toEqual([]); + second.release(); unchanged(config); + } finally { abort.abort(); first.release(); second.release(); await pending.catch(() => undefined); } +}, 10_000); +for (const mode of ["exhausted", "expired", "cancelled"] as const) test(`native replay admission keeps local refusal semantics (${mode})`, async () => { + const config = fixture(); + config.providers[NAME]!.retryOn429 = { attempts: 1, intervalMs: 1, respectRetryAfter: false }; + let holder: ReturnType | undefined; + intercept(() => { + holder = waitForProviderRequestSlot(NAME, config.providers[NAME]!); + return Response.json({ error: { message: "fixture capacity" } }, { status: 429, headers: { "retry-after": "45" } }); + }); + const abort = new AbortController(); + const pending = start(config, abort.signal); void pending.catch(() => {}); + while (!holder) await Bun.sleep(5); + const slot = await holder; + try { + await queued(config); + if (mode === "expired") { + // Existing waiter uses the new bound on the next queue evaluation. + setProviderRequestPacingLimitsForTest({ maxQueueAgeMs: 30 }); + const wake = new AbortController(); + const waiter = waitForProviderRequestSlot(NAME, config.providers[NAME]!, "other", wake.signal); + void waiter.catch(() => {}); wake.abort(); await waiter.catch(() => {}); + } else if (mode === "exhausted") { + cool(config, 0); cool(config, 1); slot.release(); + } else abort.abort(); + const result = await pending; + expect(result.response.status).toBe(mode === "cancelled" ? 499 : 429); + if (mode !== "cancelled") expect(Number(result.response.headers.get("retry-after"))).toBeGreaterThan(0); + expect(seen).toHaveLength(1); + if (mode === "expired") expect(getKeyCooldownUntil(NAME, "k1")).toBeGreaterThan(Date.now()); + if (mode === "cancelled") expect(getKeyCooldownUntil(NAME, "k1")).toBeNull(); + } finally { abort.abort(); slot.release(); await pending.catch(() => undefined); } +}, 10_000); +for (const mode of ["active", "disabled"] as const) test(`native Responses retains opt-out key behavior (${mode})`, async () => { + const config = fixture(); + if (mode === "active") config.providers[NAME]!.requestPacing!.keySelection = "active"; + else config.providers[NAME]!.requestPacing!.enabled = false; + intercept(() => Response.json({ error: { message: "fixture rejection" } }, { status: 401 })); + for (let i = 0; i < 3; i++) expect((await start(config)).response.status).toBe(401); + expect(seen).toEqual(Array(3).fill(`Bearer ${KEYS[0]}`)); + expect(getKeyCooldownUntil(NAME, "k1")).toBeNull(); + unchanged(config); +}); +test("native Responses cools a terminal 429 after the configured same-key retry", async () => { + const config = fixture(); + config.providers[NAME]!.retryOn429 = { attempts: 1, intervalMs: 1, respectRetryAfter: false }; + intercept(() => Response.json({ error: { message: "fixture capacity" } }, { status: 429, headers: { "retry-after": "45" } })); + const result = await start(config); + expect(result.response.status).toBe(429); + expect(result.response.headers.get("retry-after")).toBe("45"); + expect(seen).toEqual([`Bearer ${KEYS[0]}`, `Bearer ${KEYS[0]}`]); + expect(getKeyCooldownUntil(NAME, "k1")).toBeGreaterThan(Date.now()); + unchanged(config); +}); diff --git a/tests/responses/responses-run-turn-key-readmission.test.ts b/tests/responses/responses-run-turn-key-readmission.test.ts new file mode 100644 index 00000000000..ecf5742d4ab --- /dev/null +++ b/tests/responses/responses-run-turn-key-readmission.test.ts @@ -0,0 +1,155 @@ +import { expect, spyOn, test } from "bun:test"; +import { EventEmitter } from "node:events"; +import { writeFile } from "node:fs/promises"; +import { Readable, Writable } from "node:stream"; +import type { ChildProcess } from "node:child_process"; +import { createQoderAdapter } from "../../src/adapters/qoder/adapter"; +import { clearQoderBinaryCache } from "../../src/adapters/qoder/profiles"; +import { clearKeyCooldowns, forgetApiKeyRotationCursor } from "../../src/providers/key-failover"; +import { mkdtempSync } from "node:fs"; +import { tmpdir } from "node:os"; +import { join } from "node:path"; +import { ADAPTER_REGISTRY } from "../../src/adapters/registry"; +import { handleResponses } from "../../src/server/responses"; +import { providerRequestPacingStatus, resetProviderRequestPacingForTest, waitForProviderRequestSlot } from "../../src/providers/request-pacing"; +import type { OcxConfig } from "../../src/types"; +import { acquireOwnedSpendHome } from "../helpers/owned-spend-home"; +import { removeTreeWithRetry } from "../helpers/remove-tree"; + +for (const exhaust of [false, true]) { + test(`runTurn rechecks every replacement pacing admission (exhaust=${exhaust})`, async () => { + const previousHome = process.env.OPENCODEX_HOME; + const home = mkdtempSync(join(tmpdir(), "ocx-runturn-readmission-")); + process.env.OPENCODEX_HOME = home; + resetProviderRequestPacingForTest(); + const releaseOwner = acquireOwnedSpendHome(); + const seen: string[] = []; + const create = ADAPTER_REGISTRY["openai-chat"].create; + // Keep the real handler, selection and pacing. Only replace the external turn executor. + const override = spyOn(ADAPTER_REGISTRY["openai-chat"], "create").mockImplementation((...args) => ({ + ...create(...args), + async runTurn(_parsed, _incoming, emit) { + seen.push(args[0].apiKey!); + emit({ type: "text_delta", text: "current turn" }); emit({ type: "done" }); + }, + })); + const keys = ["first", "second", "third", "fourth", "fifth", "sixth"]; + const config = { port: 0, defaultProvider: "paced", providers: { paced: { + adapter: "openai-chat", baseUrl: "https://example.test/v1", authMode: "key", apiKey: "synthetic-first", + apiKeyPool: keys.map(id => ({ id, key: `synthetic-${id}` })), + requestPacing: { enabled: true, keys: { maxConcurrentRequests: 1 } }, + } } } as OcxConfig; + const slots = await Promise.all(keys.map(id => waitForProviderRequestSlot("paced", { ...config.providers.paced, apiKey: `synthetic-${id}` }))); + const abort = new AbortController(); + let pending: Promise | undefined; + const waitForQueue = async (id: string) => { + for (let i = 0; i < 200; i++) { + if (providerRequestPacingStatus("paced", config.providers.paced).keys?.find(key => key.id === id)?.queued === 1) return; + if (seen.length) throw new Error(`stale adapter invoked with ${seen[0]}`); + await Bun.sleep(5); + } + throw new Error(`key ${id} never queued`); + }; + try { + pending = handleResponses(new Request("http://localhost/v1/responses", { + method: "POST", headers: { "content-type": "application/json" }, signal: abort.signal, + body: JSON.stringify({ model: "paced/test", input: "hello", stream: false }), + }), config, { provider: "", model: "" }, { abortSignal: abort.signal }); + void pending.catch(() => {}); + await waitForQueue("first"); + const drifts = exhaust ? 5 : 4; + for (let i = 0; i < drifts; i++) { + config.providers.paced.apiKey = `synthetic-${keys[i + 1]}`; + config.providers.paced.apiKeySelectionRevision = `revision-${i + 1}`; + slots[i]!.release(); + if (i < drifts - 1 || !exhaust) await waitForQueue(keys[i + 1]!); + expect(seen).toEqual([]); + expect(providerRequestPacingStatus("paced", config.providers.paced).keys?.find(key => key.id === keys[i])?.inFlight).toBe(0); + } + if (!exhaust) slots[4]!.release(); + const response = await pending; + const body = await response.text(); + expect(seen).toEqual(exhaust ? [] : ["synthetic-fifth"]); + if (exhaust) { + expect(JSON.parse(body).status).toBe("failed"); + expect(body).toContain("selection changed repeatedly"); + } else { + expect(response.status).toBe(200); + expect(body).toContain("current turn"); + } + expect(providerRequestPacingStatus("paced", config.providers.paced).queued).toBe(0); + } finally { + abort.abort(); slots.forEach(slot => slot.release()); + await pending?.catch(() => undefined); + override.mockRestore(); resetProviderRequestPacingForTest(); releaseOwner(); removeTreeWithRetry(home); + if (previousHome === undefined) delete process.env.OPENCODEX_HOME; else process.env.OPENCODEX_HOME = previousHome; + } + }); +} + +for (const change of ["selection", "opt-out", "abort"] as const) { + test(`Responses carries Qoder's physical guard across async setup (${change})`, async () => { + const previousHome = process.env.OPENCODEX_HOME; + const home = mkdtempSync(join(tmpdir(), "ocx-qoder-readmission-")); + process.env.OPENCODEX_HOME = home; + resetProviderRequestPacingForTest(); clearQoderBinaryCache(); clearKeyCooldowns(); forgetApiKeyRotationCursor("paced"); + const releaseOwner = acquireOwnedSpendHome(); + const abort = new AbortController(); + const seen: string[] = []; + let writes = 0; + const config = { port: 0, defaultProvider: "paced", providers: { paced: { + adapter: "qoder", baseUrl: "https://qoder.com", authMode: "key", apiKey: "synthetic-first", + apiKeyPool: [{ id: "first", key: "synthetic-first" }, { id: "second", key: "synthetic-second" }], + requestPacing: { enabled: true, keySelection: "round-robin", keys: { maxConcurrentRequests: 1 } }, + } } } as OcxConfig; + const override = spyOn(ADAPTER_REGISTRY.qoder, "create").mockImplementation(provider => createQoderAdapter(provider, { + which: () => "/synthetic/qoder", + async writeToolBridgeFile(...args) { + await writeFile(...args); + if (++writes === 2) { + if (change === "abort") abort.abort(); + else if (change === "opt-out") config.providers.paced.requestPacing!.enabled = false; + else { + config.providers.paced.apiKey = "synthetic-second"; + config.providers.paced.apiKeySelectionRevision = "after-setup"; + } + } + }, + spawn(_file, _args, options) { + seen.push(options.env!.QODER_PERSONAL_ACCESS_TOKEN!); + const child = new EventEmitter() as ChildProcess; + child.stdout = Readable.from([ + { type: "system", subtype: "init", mcp_servers: [{ name: "opencodex", status: "connected" }] }, + { type: "stream_event", event: { type: "content_block_delta", index: 0, delta: { type: "text_delta", text: "current Qoder turn" } } }, + { type: "result", subtype: "success", is_error: false }, + ].map(frame => new TextEncoder().encode(JSON.stringify(frame) + "\n"))); + child.stderr = Readable.from([]); + child.stdin = new Writable({ write(_chunk, _encoding, callback) { callback(); } }); + child.exitCode = 0; + child.kill = () => true; + return child; + }, + })); + try { + const response = await handleResponses(new Request("http://localhost/v1/responses", { + method: "POST", headers: { "content-type": "application/json" }, signal: abort.signal, + body: JSON.stringify({ model: "paced/Qwen3.8-Max", input: "hello", stream: false, + tools: [{ type: "function", name: "probe", parameters: { type: "object", properties: {} } }], + }), + }), config, { provider: "", model: "" }, { abortSignal: abort.signal }); + const body = await response.text(); + expect(seen).toEqual(change === "abort" ? [] : [change === "selection" ? "synthetic-second" : "synthetic-first"]); + expect(writes).toBe(change === "abort" ? 2 : 4); + if (change !== "abort") { + expect(response.status).toBe(200); + expect(body).toContain("current Qoder turn"); + } + expect(providerRequestPacingStatus("paced", config.providers.paced).queued).toBe(0); + expect(providerRequestPacingStatus("paced", config.providers.paced).inFlight ?? 0).toBe(0); + } finally { + abort.abort(); override.mockRestore(); resetProviderRequestPacingForTest(); clearKeyCooldowns(); forgetApiKeyRotationCursor("paced"); + releaseOwner(); removeTreeWithRetry(home); + if (previousHome === undefined) delete process.env.OPENCODEX_HOME; else process.env.OPENCODEX_HOME = previousHome; + } + }); +} From 70d24733328f6520463e109f84280cf016c1f164 Mon Sep 17 00:00:00 2001 From: JUN Date: Thu, 15 Oct 2026 16:40:14 +0900 Subject: [PATCH 15/17] fix(providers): retain concurrent native key rejection evidence Separate already-sent failure ownership from dispatch eligibility so a late 401 extends an earlier short cooldown without admitting a cooled key. Preserve genuine rejection evidence when a replay returns a synthetic refusal. Cover both orderings with native Responses regressions. Co-authored-by: GSL-R <174104911+GSL-R@users.noreply.github.com> --- src/providers/api-key-response-ownership.ts | 4 +-- src/providers/api-key-selection.ts | 19 +++++++++-- src/server/responses/passthrough-dispatch.ts | 4 +-- structure/providers-and-adapters.md | 2 +- structure/transports/responses-failover.md | 2 +- .../responses-native-key-cooldown.test.ts | 34 +++++++++++++++++++ 6 files changed, 56 insertions(+), 9 deletions(-) diff --git a/src/providers/api-key-response-ownership.ts b/src/providers/api-key-response-ownership.ts index 5590a0c53d1..7011b56a924 100644 --- a/src/providers/api-key-response-ownership.ts +++ b/src/providers/api-key-response-ownership.ts @@ -1,7 +1,7 @@ import type { OcxConfig, OcxProviderConfig } from "../types"; import { isEgressTransparentExecutor, markEgressTransparentExecutor } from "../lib/provider-egress"; import { isNonReplayableResponse } from "../lib/upstream-retry"; -import { providerApiKeySelectionIsCurrent } from "./api-key-selection"; +import { providerApiKeySelectionOwnsFailure } from "./api-key-selection"; import { recordDistributedKeyFailure } from "./key-failover"; import { registerProviderResponseProvenance } from "./request-pacing"; @@ -39,7 +39,7 @@ export function recordDistributedResponseFailure( if (signal?.aborted || isNonReplayableResponse(response) || (response.status !== 401 && response.status !== 429)) return; const owner = owners.get(response); if (!owner || owner.providerName !== providerName - || !providerApiKeySelectionIsCurrent(config, providerName, owner.provider)) return; + || !providerApiKeySelectionOwnsFailure(config, providerName, owner.provider)) return; recordDistributedKeyFailure(config, providerName, owner.provider._apiKeyAttempt, response.status, Date.now(), response.headers.get("retry-after")); } diff --git a/src/providers/api-key-selection.ts b/src/providers/api-key-selection.ts index 74ec7c314b0..689bbebaae8 100644 --- a/src/providers/api-key-selection.ts +++ b/src/providers/api-key-selection.ts @@ -18,13 +18,13 @@ function matchesSelection(provider: OcxProviderConfig, expected: ProviderApiKeyS && current.revision === expected.revision; } -function currentKeyProvider(config: OcxConfig, name: string, selection?: ProviderApiKeySelection): OcxProviderConfig | null { +function currentKeyProvider(config: OcxConfig, name: string, selection?: ProviderApiKeySelection, allowCooling = false): OcxProviderConfig | null { const configured = config.providers[name]; if (!configured || configured.disabled) return null; let row = configured; if (selection?.distributed) { const entry = configured.apiKeyPool?.find(key => key.id === selection.entryId && key.key === selection.reference); - if (!entry || isKeyInCooldown(name, entry.id) || configured.apiKeySelectionRevision !== selection.revision + if (!entry || (!allowCooling && isKeyInCooldown(name, entry.id)) || configured.apiKeySelectionRevision !== selection.revision || !configured.requestPacing?.enabled || configured.requestPacing.keySelection !== "round-robin") return null; row = { ...configured, apiKey: entry.key, _apiKeyAttempt: selection }; } @@ -39,9 +39,22 @@ export function providerApiKeySelectionIsCurrent( config: OcxConfig, name: string, routedProvider: OcxProviderConfig, +): boolean { + return matchesConfiguredSelection(config, name, routedProvider, false); +} + +/** An already sent failure can extend cooldown, but cannot authorize another dispatch. */ +export function providerApiKeySelectionOwnsFailure( + config: OcxConfig, name: string, routedProvider: OcxProviderConfig, +): boolean { + return matchesConfiguredSelection(config, name, routedProvider, true); +} + +function matchesConfiguredSelection( + config: OcxConfig, name: string, routedProvider: OcxProviderConfig, allowCooling: boolean, ): boolean { const expected = routedProvider._apiKeyAttempt; - const current = currentKeyProvider(config, name, expected); + const current = currentKeyProvider(config, name, expected, allowCooling); return current !== null && expected !== undefined && (expected.distributed === true || matchesSelection(config.providers[name]!, expected)) && current.apiKey === routedProvider.apiKey diff --git a/src/server/responses/passthrough-dispatch.ts b/src/server/responses/passthrough-dispatch.ts index 8f1e8ff7f3b..2f893236868 100644 --- a/src/server/responses/passthrough-dispatch.ts +++ b/src/server/responses/passthrough-dispatch.ts @@ -746,7 +746,7 @@ export async function preparePassthroughExchange( }; let upstreamResponse: Response, lastDistributedResponse: Response | undefined; const adoptObservedResponse = (response: T): T => { - lastDistributedResponse = response.status === 401 || response.status === 429 ? response : undefined; + if (!isNonReplayableResponse(response)) lastDistributedResponse = response.status === 401 || response.status === 429 ? response : undefined; settleObservedHostResponse(); adoptCodexWsStage(response); return response; @@ -1621,7 +1621,7 @@ export async function preparePassthroughExchange( status: response.status, }); }; - recordDistributedResponseFailure(upstreamResponse, config, route.providerName, options.abortSignal); + recordDistributedResponseFailure(lastDistributedResponse ?? upstreamResponse, config, route.providerName, options.abortSignal); captureAffinityResponse(upstreamResponse); if (usesCodexForwardPoolAuth(admissionState.authCtx, route.provider)) { let poolRetryOutcome: number | undefined; diff --git a/structure/providers-and-adapters.md b/structure/providers-and-adapters.md index d1779e21168..a9c2883577e 100644 --- a/structure/providers-and-adapters.md +++ b/structure/providers-and-adapters.md @@ -288,7 +288,7 @@ Cursor, one lease spans the whole turn: RunSSE and BidiAppend may overlap inside turns wait at the cap. Follow-up sends still obey start intervals. The turn owner returns its lease after `runTurn` settles, so RunSSE body completion cannot admit another turn early. A capped canonical Codex WebSocket turn uses HTTP/SSE because the socket has no response-body lifecycle. -Capacity waits use the same bounded queue and retryable queue-overload errors as interval waits. Optional `requestPacing.keys` adds a shared cross-model limit for each credential reference; duplicate references share capacity, but distinct references resolving to one upstream project do not imply separate quota. `keySelection: round-robin` assigns request-local healthy keys without mutating persisted selection, skips known cooldowns, and refuses exhausted pools with retryable 429. Provider, model and key gates reserve atomically; key counters and the allocation cursor are process-local, not token or billing balances. OAuth and forwarded credentials retain their existing selection. Cooldown state is shared through the dependency-light `src/providers/api-key-cooldowns.ts` (retaining the longest outstanding expiry until explicit manual clearing) owner. Distributed selections are checked again at physical dispatch; a cooled queued key releases its old permit, selects a healthy sibling, and reacquires pacing before sending. Native Messages follows the same readmission rule. Native Responses settles terminal distributed refusals from immutable physical-send metadata; successful same-key retry and unowned refusals do not cool keys. CLI-backed adapters check `IncomingMeta.beforePhysicalSend` after asynchronous setup and before spawn. An established distributed turn refuses later sends after its key cools instead of changing credentials mid-turn. +Capacity waits use the same bounded queue and retryable queue-overload errors as interval waits. Optional `requestPacing.keys` adds a shared cross-model limit for each credential reference; duplicate references share capacity, but distinct references resolving to one upstream project do not imply separate quota. `keySelection: round-robin` assigns request-local healthy keys without mutating persisted selection, skips known cooldowns, and refuses exhausted pools with retryable 429. Provider, model and key gates reserve atomically; key counters and the allocation cursor are process-local, not token or billing balances. OAuth and forwarded credentials retain their existing selection. Cooldown state is shared through the dependency-light `src/providers/api-key-cooldowns.ts` (retaining the longest outstanding expiry until explicit manual clearing) owner. Distributed selections are checked again at physical dispatch; a cooled queued key releases its old permit, selects a healthy sibling, and reacquires pacing before sending. Native Messages follows the same readmission rule. Native Responses settles terminal distributed refusals from immutable physical-send metadata; successful same-key retry and unowned refusals do not cool keys. Failure ownership remains valid for a still-configured cooling key, so late in-flight rejections retain the longest cooldown without admitting another send. CLI-backed adapters check `IncomingMeta.beforePhysicalSend` after asynchronous setup and before spawn. An established distributed turn refuses later sends after its key cools instead of changing credentials mid-turn. Command Code effort defaults in `src/providers/command-code-efforts.ts` combine public-profile facts with the live API measurements from #5096. Both presets share the exact per-model rows; diff --git a/structure/transports/responses-failover.md b/structure/transports/responses-failover.md index aafe4d5d68b..1845202aaa4 100644 --- a/structure/transports/responses-failover.md +++ b/structure/transports/responses-failover.md @@ -156,7 +156,7 @@ The shared Responses path follows the [bounded multipart recovery contract](../s ## Upstream key attempt accounting -Key identity is sealed at guarded physical dispatch after queued selections are rebuilt. Distributed selections retain their request-local entry and revision. A rebuild after selection drift releases the old per-key lease and reacquires admission for the refreshed key; the send guard checks again after waiting. Turn-scoped adapters follow the same rule before invocation. Native Chat, Messages and keyed compact all rebuild stale queued selections before dispatch, including a manual active-key switch under per-key limits; they release the old lease, readmit the new key and honor cancellation while its capacity is occupied. Direct compact and Images record genuine distributed 401/429 cooldowns without replaying paid POSTs; synthetic replay refusals never cool a key. Native Messages revisits mixed distributed 401/429 outcomes with its existing counters. Native Chat and Messages require the physical outgoing credential to match the selected distributed key before pooling a rejection; Messages honors the configured `x-api-key` or Bearer transport while rejecting a conflicting credential header; configured credential overrides cannot cool unused entries. Replacement admission pauses the response-header deadline and retains client cancellation; the header deadline restarts once admission completes. Native Responses captures distributed-key ownership after connection-policy rewrites in `src/providers/api-key-response-ownership.ts`; terminal 401/429 responses settle cooldown only after same-key retry policy, and a failed replay admission retains the last genuine refusal. Reused responses, credential/destination overrides, stale revisions and synthetic refusals cannot confer ownership. Local pacing errors retain 429 and Retry-After at the passthrough catch boundary. Unpaced dispatch without a readmission owner fails closed. Replay notification errors release acquired slots before propagating, including pre-acquired-slot dispatch. Translated Anthropic billing recovery uses the [immutable physical-send ownership contract](../providers/anthropic-key-billing.md). Its initial send, credential hop and later mixed-class refusals share the request budget. A denied hop retains the upstream refusal, and a leased pacing body rewrap carries the existing marker rather than attributing the response to rebuilt headers. +Key identity is sealed at guarded physical dispatch after queued selections are rebuilt. Distributed selections retain their request-local entry and revision. A rebuild after selection drift releases the old per-key lease and reacquires admission for the refreshed key; the send guard checks again after waiting. Turn-scoped adapters follow the same rule before invocation. Native Chat, Messages and keyed compact all rebuild stale queued selections before dispatch, including a manual active-key switch under per-key limits; they release the old lease, readmit the new key and honor cancellation while its capacity is occupied. Direct compact and Images record genuine distributed 401/429 cooldowns without replaying paid POSTs; synthetic replay refusals never cool a key. Native Messages revisits mixed distributed 401/429 outcomes with its existing counters. Native Chat and Messages require the physical outgoing credential to match the selected distributed key before pooling a rejection; Messages honors the configured `x-api-key` or Bearer transport while rejecting a conflicting credential header; configured credential overrides cannot cool unused entries. Replacement admission pauses the response-header deadline and retains client cancellation; the header deadline restarts once admission completes. Native Responses captures distributed-key ownership after connection-policy rewrites in `src/providers/api-key-response-ownership.ts`; terminal 401/429 responses settle cooldown only after same-key retry policy, and a failed replay admission or synthetic replay refusal retains the last genuine refusal. Failure ownership accepts an already cooling, still-configured credential so concurrent late 401s extend earlier short cooldowns; physical-send eligibility still refuses cooling credentials. Reused responses, credential/destination overrides, stale revisions and synthetic refusals cannot confer ownership. Local pacing errors retain 429 and Retry-After at the passthrough catch boundary. Unpaced dispatch without a readmission owner fails closed. Replay notification errors release acquired slots before propagating, including pre-acquired-slot dispatch. Translated Anthropic billing recovery uses the [immutable physical-send ownership contract](../providers/anthropic-key-billing.md). Its initial send, credential hop and later mixed-class refusals share the request budget. A denied hop retains the upstream refusal, and a leased pacing body rewrap carries the existing marker rather than attributing the response to rebuilt headers. Before a turn-scoped retry readmits a refreshed selection, it rejects a binding already recorded as failed for this request and returns the last key failure. Eligible replacement keys still release the old lease and reacquire their own pacing capacity before invocation, rechecking after every admission. CLI-backed turns carry a synchronous selection fence through asynchronous tool setup to the physical spawn boundary. Raw adapter terminal usage is recorded before continuation, search, or image loops merge it; repeated parsing of one physical response does not count it twice. Key changes preserve the previous attempt while retaining the active attempt object shared by streaming/combo callbacks. diff --git a/tests/responses/responses-native-key-cooldown.test.ts b/tests/responses/responses-native-key-cooldown.test.ts index c229381a10a..4adac4ab92a 100644 --- a/tests/responses/responses-native-key-cooldown.test.ts +++ b/tests/responses/responses-native-key-cooldown.test.ts @@ -191,3 +191,37 @@ test("native Responses cools a terminal 429 after the configured same-key retry" expect(getKeyCooldownUntil(NAME, "k1")).toBeGreaterThan(Date.now()); unchanged(config); }); +test("an in-flight 401 extends the same key's earlier short 429 cooldown", async () => { + const config = fixture(); config.providers[NAME]!.requestPacing!.keys!.maxConcurrentRequests = 2; + let rejectFirst: ((response: Response) => void) | undefined; + let firstKeySends = 0; + intercept(headers => { + if (headers.get("authorization") !== `Bearer ${KEYS[0]}`) return completed(); + if (++firstKeySends === 1) return new Promise(resolve => { rejectFirst = resolve; }); + return Response.json({ error: { message: "fixture capacity" } }, { status: 429, headers: { "retry-after": "1" } }); + }); + const pending = start(config); + while (!rejectFirst) await Bun.sleep(5); + try { + expect((await start(config)).response.status).toBe(200); + expect((await start(config)).response.status).toBe(429); + expect(getKeyCooldownUntil(NAME, "k1")).toBeGreaterThan(Date.now()); + } finally { rejectFirst(Response.json({ error: { message: "fixture rejection" } }, { status: 401 })); } + expect((await pending).response.status).toBe(401); + expect(getKeyCooldownUntil(NAME, "k1")).toBeGreaterThanOrEqual(Date.now() + 599_000); + unchanged(config); +}); +test("a synthetic replay refusal retains the preceding genuine key rejection", async () => { + const config = fixture(); + config.providers[NAME]!.retryOn429 = { attempts: 1, intervalMs: 1, respectRetryAfter: false }; + intercept(() => { + const response = Response.json({ error: { message: "fixture capacity" } }, { + status: seen.length === 1 ? 429 : 502, headers: { "retry-after": "45" }, + }); + if (seen.length > 1) markResponseNonReplayable(response); + return response; + }); + expect((await start(config)).response.status).toBe(502); + expect(getKeyCooldownUntil(NAME, "k1")).toBeGreaterThan(Date.now()); + unchanged(config); +}); From 1d780a3815f9efffb9a112e88a505e483cee8734 Mon Sep 17 00:00:00 2001 From: JUN Date: Thu, 15 Oct 2026 21:56:03 +0900 Subject: [PATCH 16/17] test(responses): register pacing refusal owner in core inventory The passthrough pacing-error mapping reaches pacing-overload.ts from the Responses core import graph. Include it in the source-oracle inventory so coverage, dependency and strict below-2000-line guards cover the new owner. Co-authored-by: GSL-R <174104911+GSL-R@users.noreply.github.com> --- tests/helpers/responses-core-source.ts | 2 ++ 1 file changed, 2 insertions(+) diff --git a/tests/helpers/responses-core-source.ts b/tests/helpers/responses-core-source.ts index 516521ecf48..b5da48bf257 100644 --- a/tests/helpers/responses-core-source.ts +++ b/tests/helpers/responses-core-source.ts @@ -49,6 +49,8 @@ export const RESPONSES_CORE_MODULES = [ "passthrough-dispatch.ts", // Reached from passthrough/adapter dispatch, compact and runTurn: local spend-ledger refusal mapping. "spend-storage-error.ts", + // Reached from passthrough dispatch: local pacing refusals retain 429 and Retry-After. + "pacing-overload.ts", "reset-replay.ts", "passthrough-delivery.ts", "buffered-sse-json.ts", From d0d7eea16d4df8e8887af69c94e702f42e5b83eb Mon Sep 17 00:00:00 2001 From: JUN Date: Fri, 16 Oct 2026 00:18:14 +0900 Subject: [PATCH 17/17] refactor(management): extract provider key pacing diagnostics route Move the request-pacing status handler, including key cooldown diagnostics, to a cohesive management sibling. Preserve the route position, validation and response shape while leaving the provider router below the size limit after integration with dev. Document the new route owner. Co-authored-by: GSL-R <174104911+GSL-R@users.noreply.github.com> --- .../management/provider-key-pacing-routes.ts | 31 +++++++++++++++++++ src/server/management/provider-routes.ts | 26 +++------------- structure/gui-and-management-api.md | 2 +- 3 files changed, 36 insertions(+), 23 deletions(-) create mode 100644 src/server/management/provider-key-pacing-routes.ts diff --git a/src/server/management/provider-key-pacing-routes.ts b/src/server/management/provider-key-pacing-routes.ts new file mode 100644 index 00000000000..1e91f0c5a94 --- /dev/null +++ b/src/server/management/provider-key-pacing-routes.ts @@ -0,0 +1,31 @@ +import { hasOwnProvider, isValidProviderName } from "../../config"; +import { getKeyCooldownUntil } from "../../providers/key-failover"; +import { providerRequestPacingStatus } from "../../providers/request-pacing"; +import type { OcxProviderConfig } from "../../types"; +import { jsonResponse } from "../auth-cors"; +import type { ManagementContext } from "./context"; + +/** Read-only provider/key pacing diagnostics; configuration writes stay in the provider PATCH owner. */ +export function handleProviderKeyPacingRoutes( + { req, url, config }: Pick, +): Response | null { + if (url.pathname !== "/api/provider-request-pacing" || req.method !== "GET") return null; + const statusFor = (name: string, provider: OcxProviderConfig) => { + const status = providerRequestPacingStatus(name, provider); + if (status.keys) status.keys = status.keys.map(row => ({ ...row, cooldownUntil: getKeyCooldownUntil(name, row.id) })); + return status; + }; + const name = url.searchParams.get("name")?.trim(); + if (name) { + if (!isValidProviderName(name) || !hasOwnProvider(config.providers, name)) { + return jsonResponse({ error: "unknown provider" }, 404); + } + return jsonResponse(statusFor(name, config.providers[name]!)); + } + return jsonResponse(Object.fromEntries( + Object.entries(config.providers).map(([providerName, provider]) => [ + providerName, + statusFor(providerName, provider), + ]), + )); +} diff --git a/src/server/management/provider-routes.ts b/src/server/management/provider-routes.ts index 750ef10b5c0..729d5762f60 100644 --- a/src/server/management/provider-routes.ts +++ b/src/server/management/provider-routes.ts @@ -77,8 +77,8 @@ import { PROVIDER_QUOTA_MAX_AGE_MS, type ProviderRoutingQuota } from "../../prov import { cachedProviderQuotaIsExhausted } from "../../combos/resolve"; import { probeJevDecisionProvider } from "../../combos/jev"; import { comboDependsOnProviderRoute } from "./decision-model-validation"; -import { clearKeyCooldowns, forgetApiKeyRotationCursor, getKeyCooldownUntil } from "../../providers/key-failover"; -import { providerRequestPacingStatus } from "../../providers/request-pacing"; +import { clearKeyCooldowns, forgetApiKeyRotationCursor } from "../../providers/key-failover"; +import { handleProviderKeyPacingRoutes } from "./provider-key-pacing-routes"; import { CODEX_FORWARD_BASE_URL, isCanonicalOpenAiForwardProvider } from "../../providers/openai-tiers"; import { codexAccountNamespaceProviderCollisionError } from "../../codex/account-namespace-match"; import { clearThreadAccountMap } from "../../codex/routing"; @@ -895,26 +895,8 @@ export async function handleProviderRoutes(ctx: ManagementContext): Promise { - const status = providerRequestPacingStatus(name, provider); - if (status.keys) status.keys = status.keys.map(row => ({ ...row, cooldownUntil: getKeyCooldownUntil(name, row.id) })); - return status; - }; - const name = url.searchParams.get("name")?.trim(); - if (name) { - if (!isValidProviderName(name) || !hasOwnProvider(config.providers, name)) { - return jsonResponse({ error: "unknown provider" }, 404); - } - return jsonResponse(statusFor(name, config.providers[name]!)); - } - return jsonResponse(Object.fromEntries( - Object.entries(config.providers).map(([providerName, provider]) => [ - providerName, - statusFor(providerName, provider), - ]), - )); - } + const keyPacingResponse = handleProviderKeyPacingRoutes(ctx); + if (keyPacingResponse) return keyPacingResponse; if (url.pathname === "/api/providers" && req.method === "GET") { return jsonResponse(Object.entries(config.providers).map(([name, p]) => ({ diff --git a/structure/gui-and-management-api.md b/structure/gui-and-management-api.md index e5971b4357f..91ecf8224bf 100644 --- a/structure/gui-and-management-api.md +++ b/structure/gui-and-management-api.md @@ -369,7 +369,7 @@ to fully quit and reopen Desktop after selecting the picker profile. Provider writes must not round-trip masked API keys as real secrets. Dashboard actions that change model visibility or subagent selection should trigger catalog/cache sync behavior through the server -path that owns it. Provider request-pacing settings expose independent provider and uniform key limits plus opt-in request-local round-robin selection. `GET /api/provider-request-pacing` returns key handles, starts, in-flight counts, queue counts, next spacing slots and known cooldowns, never credential values or internal bucket hashes. Saving these rules uses the existing requestPacing-only PATCH path. +path that owns it. Provider request-pacing settings expose independent provider and uniform key limits plus opt-in request-local round-robin selection. `GET /api/provider-request-pacing`, owned by `src/server/management/provider-key-pacing-routes.ts` and delegated from `src/server/management/provider-routes.ts`, returns key handles, starts, in-flight counts, queue counts, next spacing slots and known cooldowns, never credential values or internal bucket hashes. Saving these rules uses the existing requestPacing-only PATCH path. ### Fast selector rows setting