Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion crates/aisix-obs/src/usage.rs
Original file line number Diff line number Diff line change
Expand Up @@ -164,7 +164,9 @@ pub struct UsageEvent {
/// it can use —
///
/// - non-streaming: the complete response is written;
/// - streaming: the first token is forwarded downstream.
/// - streaming: the first token is forwarded downstream — for a
/// passthrough route, which relays an opaque byte stream and has no
/// token to recognise, the first relayed frame handed to the client.
///
/// Request-scoped (unlike the two `upstream_*` fields above), so it
/// spans request parsing, guardrail scans, every failed attempt,
Expand Down
23 changes: 23 additions & 0 deletions crates/aisix-proxy/AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -58,6 +58,29 @@ writes the legacy `aisix_requests_total` **and** the detailed `aisix_proxy_*` /
directly silently produces a request that exists in one family and not the
others — the bug AISIX-Cloud#1234 fixed across ten endpoints.

## A passthrough route that detects an envelope must observe what the typed endpoint does

`passthrough_route.rs` relays bytes verbatim, but it *detects* the request
envelope and extracts from it. Whatever it extracts is the whole observation —
there is no bridge behind it to fill anything in. So a new observation added to
a typed endpoint has a second home: a new `UsageEvent` token dimension needs
reading in `usage_of`, a new guardrail scan input needs adding to
`message_scan_text` / `request_guardrail_text`, a new attribution field needs
setting on `RouteTelemetry`. Miss it and the route keeps answering 200 while
metering and enforcement silently weaken — the shape of #988, where the cache
and reasoning counters existed everywhere except here.

Two rules bound the extraction:

- An **opaque** (`Raw`) body is never mined for meaning. Buffered opaque
responses are not probed for usage at all, and an opaque stream's flat token
fields count only on a frame the server itself labelled one
(`event: token_usage`) — a caller-shaped body must not be able to mint tokens.
- Usage accumulates **field-wise max** across frames, never last-wins: one
stream reports it in pieces (Anthropic's `message_start` carries the input
and cache counters, its `message_delta` only the output ones), so an
assignment truncates whatever arrived earlier.

## A new proxy route has to be declared in three places

Adding a `.route(…)` in `build_router` is not enough, and nothing fails loudly
Expand Down
14 changes: 13 additions & 1 deletion crates/aisix-proxy/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -380,6 +380,12 @@ fn inbound_protocol_for_endpoint(endpoint: &str) -> &'static str {
"a2a"
} else if endpoint == "/v1/realtime" {
"realtime"
} else if endpoint == "/passthrough_route" {
// Whatever API a route relays, it is not the gateway's own OpenAI
// surface — and the usage event already tags these rows
// `passthrough`, so labelling the metric `openai` put the two
// halves of one request on different protocols.
"passthrough"
} else {
"openai"
}
Expand Down Expand Up @@ -2415,7 +2421,13 @@ mod tests {
"the caller's path must not reach the label: {sample}"
);
assert!(sample.contains(r#"outcome="completed""#), "{sample}");
assert!(sample.contains(r#"inbound_protocol="openai""#), "{sample}");
// A relayed API is not the gateway's own OpenAI surface, and the
// usage event already tags these rows `passthrough` — the two must
// not put one request on different protocols.
assert!(
sample.contains(r#"inbound_protocol="passthrough""#),
"{sample}"
);
}

/// The chunked path reaches the handler, which rejects at its body
Expand Down
Loading