Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions docs-site/src/content/docs/guides/remote-hub.md
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,11 @@ listener, not the management port. Local clients use the configured loopback com
Advisor model used by managed native Anthropic Messages. Bare model IDs and the hub's configured
`provider/model` names are accepted. A disallowed or unnamed Advisor destination returns HTTP 403
before provider dispatch; provider-only and unrestricted keys retain their existing scope.
- GitHub Copilot Auto rechecks the concrete model selected during negotiation and
renewal. A model outside the client's allowed scope ends the request with
`model_not_allowed_for_key` (HTTP 403 before streaming, or a terminal error in
an already-started stream). Credential rotation or a fallback target cannot
override that refusal.
- A raw management admin token can perform ordinary administration, but it cannot mint a browser
session or authorize consent-bearing actions such as starring the repository. Those actions
require a server-issued `gui-session`, matching browser origin, and CSRF token.
Expand Down
8 changes: 7 additions & 1 deletion src/lib/translator-budget.ts
Original file line number Diff line number Diff line change
@@ -1,3 +1,5 @@
import { isNonReplayableResponse, markResponseNonReplayable } from "./upstream-retry";

export const TRANSLATOR_MAX_CALL_ARGUMENT_BYTES = 2 * 1024 * 1024;
export const TRANSLATOR_MAX_TURN_BYTES = 32 * 1024 * 1024;
export const TRANSLATOR_MAX_SSE_EVENT_BYTES = 32 * 1024 * 1024;
Expand Down Expand Up @@ -410,5 +412,9 @@ export function finalizeTranslatorBudgetResponse(response: Response, budget: Tra
});
if (signal?.aborted) finalize();
else signal?.addEventListener("abort", finalize, { once: true });
return new Response(body, { status: response.status, statusText: response.statusText, headers: response.headers });
const finalizedResponse = new Response(body, { status: response.status, statusText: response.statusText, headers: response.headers });
// The rebuild above drops identity markers. A refusal that must not be replayed
// stays non-replayable through accounting finalization.
if (isNonReplayableResponse(response)) markResponseNonReplayable(finalizedResponse);
return finalizedResponse;
}
5 changes: 4 additions & 1 deletion src/server/admission-model-scope.ts
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
import type { DataPlaneAdmission } from "./auth-cors";
import type { OcxApiKeyEntry, OcxConfig } from "../types";
import { markResponseNonReplayable } from "../lib/upstream-retry";

/**
* Per-admission-key model and provider scope.
Expand Down Expand Up @@ -155,10 +156,12 @@ export function admissionModelDeniedBody(error: AdmissionModelDeniedError): {
* indistinguishable from a typo in the model name.
*/
export function admissionModelDeniedResponse(error: AdmissionModelDeniedError): Response {
return new Response(JSON.stringify(admissionModelDeniedBody(error)), {
const response = new Response(JSON.stringify(admissionModelDeniedBody(error)), {
status: 403,
headers: { "Content-Type": "application/json" },
});
markResponseNonReplayable(response);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Preserve the scope error envelope through combo handling

When Copilot Auto is a combo target and negotiation denies its concrete model, this marker correctly stops the combo hop, but consumeComboFailure still consumes the response and rebuilds it as error.type = "upstream_error" with the original JSON embedded in the message. Consequently clients cannot branch on the documented stable model_not_allowed_for_key type (the new test only checks for a substring); preserve the local admission response or carry a structured code through combo normalization.

AGENTS.md reference: src/AGENTS.md:L19-L19

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@-

return response;
}

/**
Expand Down
44 changes: 27 additions & 17 deletions src/server/responses/adapter-continuation.ts
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,8 @@ import {
isNonReplayableResponse,
prepareSameTarget429Wait,
} from "../../lib/upstream-retry";
import { unwrapUpstreamRetryEvidenceError } from "../../codex/auth-context";
import { AdmissionModelDeniedError, MODEL_NOT_ALLOWED_FOR_KEY } from "../admission-model-scope";
import { redactSecretString } from "../../lib/redact";
import { resolveWireProtocolOverride } from "../adapter-resolve";
import { bindRouteReasoningReplayScope } from "./core-replay";
Expand Down Expand Up @@ -146,6 +148,22 @@ export function createAdapterContinuations(
sendBudgetExhausted,
} = sendBudgetState;

const continuationFailureEvent = (error: unknown): Extract<AdapterEvent, { type: "error" }> => {
if (options.abortSignal?.aborted || upstream.signal.aborted) {
return { type: "error", message: "client closed request during terminal continuation", status: 499 };
}
const refusal = unwrapUpstreamRetryEvidenceError(error);
if (refusal instanceof AdmissionModelDeniedError) {
return {
type: "error", message: refusal.message, status: 403,
errorType: MODEL_NOT_ALLOWED_FOR_KEY, code: MODEL_NOT_ALLOWED_FOR_KEY, retryable: false,
};
}
return {
type: "error",
message: `Provider continuation failed: ${redactSecretString(error instanceof Error ? error.message : String(error))}`,
};
};

// One bounded internal continuation re-ask for clean end_turn turns that announced an edit
// without emitting a tool call. Anthropic gets this by default; openai-chat providers opt in
Expand Down Expand Up @@ -304,12 +322,7 @@ export function createAdapterContinuations(
kiroRefusalPendingReplay = undefined;
kiroReplaySetupFailed = true;
} else {
if (options.abortSignal?.aborted || upstream.signal.aborted) {
yield { type: "error", message: "client closed request during terminal continuation", status: 499 };
} else {
yield { type: "error", message: `Provider continuation failed: ${redactSecretString(error instanceof Error ? error.message : String(error))}` };
}
return;
yield continuationFailureEvent(error); return;
}
}

Expand Down Expand Up @@ -360,12 +373,7 @@ export function createAdapterContinuations(
try {
response = await fetchContinuation("rate-limit-429");
} catch (error) {
if (options.abortSignal?.aborted || upstream.signal.aborted) {
yield { type: "error", message: "client closed request during terminal continuation", status: 499 };
} else {
yield { type: "error", message: `Provider continuation failed: ${redactSecretString(error instanceof Error ? error.message : String(error))}` };
}
return;
yield continuationFailureEvent(error); return;
}
}

Expand All @@ -381,9 +389,7 @@ export function createAdapterContinuations(
route.provider = rotated;
try { await transportState.resolveCopilotSelection(nextParsed); }
catch (error) {
yield { type: "error", message: `Provider continuation failed: ${redactSecretString(error instanceof Error ? error.message : String(error))}`,
...(options.abortSignal?.aborted || upstream.signal.aborted ? { status: 499 } : {}) };
return;
yield continuationFailureEvent(error); return;
}
invalidateSameTargetRequest();
transportState.activeAdapter = resolveSelectionAdapter(
Expand Down Expand Up @@ -630,13 +636,17 @@ export function createAdapterContinuations(
nextContinuationRecoveryKind = "oauth-account-429";
continue;
}
} catch {
} catch (error) {
// Everything in this try runs before the replay: the send happens on the next
// iteration, after `continue`. A throw here therefore leaves a reservation that
// never dispatched, and holding it would refuse a later recovery in this same
// request for a send that never left the process.
hop.permit?.release();
// fall through to emit continuation error below
if (unwrapUpstreamRetryEvidenceError(error) instanceof AdmissionModelDeniedError) {
yield continuationFailureEvent(error);
return;
}
// Other negotiation failures preserve the original provider refusal.
}
}
}
Expand Down
24 changes: 23 additions & 1 deletion src/server/responses/adapter-dispatch.ts
Original file line number Diff line number Diff line change
Expand Up @@ -112,6 +112,7 @@ import { mapCodexAuthContextErrorToResponse } from "./codex-auth-error";
import { chargeWorkflowSends } from "../../lib/workflow-budget";
import { isAntigravityValidationRefusal } from "./antigravity-validation-refusal";
import { unboundPoolSpendRefusalResponse } from "../workflow-refusal";
import { AdmissionModelDeniedError, admissionModelDeniedResponse } from "../admission-model-scope";

/** One responsibility of the Responses request pipeline; state owners are explicit. */
export async function prepareAdapterExchange(
Expand Down Expand Up @@ -541,6 +542,10 @@ export async function prepareAdapterExchange(
return storageRefusal;
}
const refusal = err instanceof UpstreamRetryEvidenceError ? err.cause : err;
if (refusal instanceof AdmissionModelDeniedError) {
releaseCodexAuthContextProbeLease(admissionState.authCtx);
return admissionModelDeniedResponse(refusal);
}
const codexRefusal = mapCodexAuthContextErrorToResponse(unwrapUpstreamRetryEvidenceError(err), {
now: Date.now(), accountSelector: route.codexAccountNamespace,
});
Expand Down Expand Up @@ -810,6 +815,13 @@ export async function prepareAdapterExchange(
releaseCodexAuthContextProbeLease(admissionState.authCtx);
return { failed: storageRefusal };
}
const scopeRefusal = unwrapUpstreamRetryEvidenceError(err);
if (!options.abortSignal?.aborted && scopeRefusal instanceof AdmissionModelDeniedError) {
cleanupUpstreamAbort();
upstream.abort();
releaseCodexAuthContextProbeLease(admissionState.authCtx);
return { failed: admissionModelDeniedResponse(scopeRefusal) };
}
const codexRefusal = !options.abortSignal?.aborted && mapCodexAuthContextErrorToResponse(unwrapUpstreamRetryEvidenceError(err), {
now: Date.now(), accountSelector: route.codexAccountNamespace,
});
Expand Down Expand Up @@ -1539,7 +1551,17 @@ export async function prepareAdapterExchange(
// The hop's permit is already settled by the dispatch boundary above; continuing
// only skips the remaining arms, it does not abandon a reservation.
if (isNonReplayableResponse(upstreamResponse)) continue recovery;
} catch {
} catch (err) {
// A local admission denial is terminal for the whole request: rotating to another
// account cannot re-scope the key, so the refusal answers in place of the 429.
const scopeRefusal = unwrapUpstreamRetryEvidenceError(err);
if (!options.abortSignal?.aborted && scopeRefusal instanceof AdmissionModelDeniedError) {
hop.permit?.release();
cleanupUpstreamAbort();
upstream.abort();
releaseCodexAuthContextProbeLease(admissionState.authCtx);
return admissionModelDeniedResponse(scopeRefusal);
}
// A throw before the send — snapshot fetch, credential application, adapter
// resolution — must hand the reservation back. Without this the ladder charges the
// request for a send it never made, and a later recovery in the same request is
Expand Down
11 changes: 10 additions & 1 deletion src/server/responses/core-combo-failure.ts
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@ import { resolveClientRetryAfter } from "../../lib/retry-after";
import { formatErrorResponse } from "../../bridge";
import { OLLAMA_LATE_ATTRIBUTION_LIMIT_MESSAGE } from "../../adapters/ollama-native";
import { isNonReplayableResponse, markResponseNonReplayable } from "../../lib/upstream-retry";
import { MODEL_NOT_ALLOWED_FOR_KEY } from "../admission-model-scope";
import { usageFromResponsesPayload } from "../request-log";
import type { ResponsesTerminalStatus } from "../../bridge";

Expand Down Expand Up @@ -49,6 +50,9 @@ export async function consumeComboFailure(
// display-safe body carries a recognized quota message.
let quotaConfirmedByBody = false;
const serverError = response.status >= 500 && response.status < 600;
// A local key-scope refusal is already the client's answer: rebuilding it as an upstream_error
// would hide the stable model_not_allowed_for_key type clients branch on.
let scopeRefusalBody: string | undefined;
try {
const body = await readBoundedResponseBody(response, { signal, reportUtf8Validity: serverError });
// A 5xx body counts as quota or classification evidence only when it decoded as valid
Expand All @@ -71,6 +75,9 @@ export async function consumeComboFailure(
upstreamMessage = normalized.message;
upstreamType = normalized.type;
}
if (nonReplayable && response.status === 403 && upstreamType === MODEL_NOT_ALLOWED_FOR_KEY) {
scopeRefusalBody = body.text;
}
}
} catch (error) {
if (signal?.aborted) throw error;
Expand Down Expand Up @@ -106,7 +113,9 @@ export async function consumeComboFailure(
now,
includeDefault: false,
});
const failureResponse = formatErrorResponse(
const failureResponse = scopeRefusalBody !== undefined
? new Response(scopeRefusalBody, { status: 403, headers: { "Content-Type": "application/json" } })
: formatErrorResponse(
response.status,
cyberFailure ? (upstreamType ?? CYBER_POLICY_ERROR_CODE) : "upstream_error",
message,
Expand Down
2 changes: 2 additions & 0 deletions src/server/responses/core-lifetime.ts
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
import type { TranslatorBudget } from "../../lib/translator-budget";
import { finalizeTranslatorBudgetResponse } from "../../lib/translator-budget";
import { isNonReplayableResponse, markResponseNonReplayable } from "../../lib/upstream-retry";
import {
isNativePassthroughSseResponse,
markNativePassthroughSseResponse,
Expand Down Expand Up @@ -67,6 +68,7 @@ export function finalizeAccountLease(response: Response, release: () => void): R
async cancel(reason) { try { await reader.cancel(reason); } finally { finish(); } },
});
const wrapped = new Response(body, { status: response.status, statusText: response.statusText, headers: response.headers });
if (isNonReplayableResponse(response)) markResponseNonReplayable(wrapped);
if (isNativePassthroughSseResponse(response)) markNativePassthroughSseResponse(wrapped);
if (isEagerRelaySseResponse(response)) markEagerRelaySseResponse(wrapped);
if (isPreinspectedJsonResponse(response)) markPreinspectedJsonResponse(wrapped);
Expand Down
4 changes: 4 additions & 0 deletions src/server/responses/request-prepare.ts
Original file line number Diff line number Diff line change
Expand Up @@ -1452,6 +1452,10 @@ export async function prepareResponsesRequest(
commitSkillsSnapshot?.();
return {
inboundWire,
// Admission facts later route rewrites must not re-derive: the selector the caller sent and
// the key scope as it stood when this request was admitted.
inboundSelector,
admissionScope,
translatorBudget,
parsed,
toolBridgeMaps,
Expand Down
16 changes: 16 additions & 0 deletions src/server/responses/request-transport.ts
Original file line number Diff line number Diff line change
Expand Up @@ -56,6 +56,11 @@ import { stampOAuthAccountLabel, usesApiKeyAccount } from "../../providers/label
import { resolveProviderTransport } from "../../providers/xai-transport";
import { captureRouteStaticPolicy } from "../../router";
import { CopilotAutoHttpError, resolveCopilotAuto } from "../../providers/github-copilot-auto";
import {
AdmissionModelDeniedError,
admissionModelDeniedResponse,
assertRouteAllowedByScope,
} from "../admission-model-scope";
import { renameRoutedIdentityInContext } from "../../adapters/identity";
import { resolveCopilotApiBaseUrl } from "../../oauth/github-copilot";
import {
Expand Down Expand Up @@ -109,6 +114,8 @@ export async function prepareResponsesTransport(
| "inboundWire"
| "selectedForwardHeaders"
| "translatorBudget"
| "inboundSelector"
| "admissionScope"
>,
) {
const { config, logCtx, options, req } = requestContext;
Expand Down Expand Up @@ -297,6 +304,7 @@ export async function prepareResponsesTransport(
const copilotRequestedModel = route.modelId;
/** Project only bounded Copilot refusal metadata; upstream bodies and session tokens never reach the client. */
const copilotRefusalResponse = (error: unknown): Response | undefined => {
if (error instanceof AdmissionModelDeniedError) return admissionModelDeniedResponse(error);
if (!(error instanceof CopilotAutoHttpError) || ![401, 403, 429].includes(error.status)) return;
const response = formatErrorResponse(error.status, error.status === 429 ? "rate_limit_error" : "authentication_error",
error.message);
Expand Down Expand Up @@ -357,6 +365,13 @@ export async function prepareResponsesTransport(
replayOAuthCredentialSnapshot = { accountId: refreshed.accountId, generation: refreshed.generation };
selected = await negotiate();
}
assertRouteAllowedByScope(
// The scope captured at admission: a key edited or removed while negotiation was in
// flight must not widen this request. The refusal names the caller's own selector.
requestState.admissionScope,
requestState.inboundSelector,
{ providerName: route.providerName, modelId: selected.modelId },
Comment thread
coderabbitai[bot] marked this conversation as resolved.
Comment on lines +368 to +373

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Propagate scope denials out of sidecar rotation

When a scoped Copilot Auto request is using the web-search or image sidecar and the routed inference gets a 429, this new assertion can throw during the sidecar's key/OAuth rotation; however, executeResponsesSidecars catches every error from resolveCopilotSelection/applyFailoverSnapshot at lines 286-287 and 321-331 and returns null. The sidecar loop therefore returns the original replayable 429, so a combo or policy fallback can still run instead of terminating with model_not_allowed_for_key; handle AdmissionModelDeniedError explicitly at that sidecar boundary and return the non-replayable 403.

AGENTS.md reference: src/AGENTS.md:L17-L19

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@-

);
copilotSessionExpiresAt = selected.expiresAt ?? Infinity;
route.provider = selected.provider;
route.modelId = selected.modelId;
Expand Down Expand Up @@ -725,6 +740,7 @@ export async function prepareResponsesTransport(
let nextAdapter: ProviderAdapter;
try { nextAdapter = await refreshDispatchAdapter(requestParsed); }
catch (error) {
if (error instanceof AdmissionModelDeniedError) throw error;
const refusal = copilotRefusalResponse(error);
if (refusal) return refusal;
throw error;
Expand Down
Loading
Loading