Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion packages/sie_gateway/docs/architecture-guide.md
Original file line number Diff line number Diff line change
Expand Up @@ -57,7 +57,7 @@ Rules enforced on the inference path:
- Ordinary encode, score, and extract use pool-queue dispatch; capped logical batch pools and generation use worker direct-dispatch. Both paths use JetStream, with no direct-HTTP fallback. `src/handlers/proxy.rs` is the JetStream submission handler despite its name.
- If the queue transport is unavailable (no usable NATS client at init), the gateway returns `503`. It does not fall back to direct mode.
- Unknown model ids fast-fail with `404` whenever the in-memory `ModelRegistry` has been populated (either by the filesystem seed or by a successful bootstrap / delta from `sie-config`). In the pre-bootstrap edge case where the registry is still empty — no seed, no export applied yet — the proxy falls back to the caller-supplied bundle (or `"default"`) so an unseeded gateway can still publish work to a cold pool that a caller pinned via `X-SIE-Pool`. Once any model is registered, this fallback is disabled and the 404 contract applies.
- Automatic pool selection only considers healthy workers whose reported `bundle_config_hash` matches the gateway's expected hash. Explicit `X-SIE-Pool` selects a logical pool; the gateway publishes to that pool's backing queue pool (`PoolSpec.queue_pool`, default `default`) and stamps the logical pool into the work item as `admission_pool`. Deliveries whose hash is not the worker-sidecar's current hash, or logical pools that do not assign that worker, are NAKed before backend IPC. Successful non-streaming results echo the execution hash; the gateway emits `X-SIE-Model-Revision` only when every successful result matches the hash and immutable revision captured atomically at routing time. Streaming responses omit this post-execution attestation because headers precede the terminal result.
- Automatic pool selection only considers healthy workers whose reported `bundle_config_hash` matches the gateway's expected hash. Explicit `X-SIE-Pool` selects a logical pool; the gateway publishes to that pool's backing queue pool (`PoolSpec.queue_pool`, default `default`) and stamps the logical pool into the work item as `admission_pool`. Deliveries whose hash is not the worker-sidecar's current hash, or logical pools that do not assign that worker, are NAKed before backend IPC. Successful non-streaming results echo the execution hash; the gateway emits `X-SIE-Model-Revision` only when every successful result matches the hash and immutable revision captured atomically at routing time. The header value is the lowercase 64-hex executed bundle/config SHA-256, not the catalog weights revision (for example, a 40-hex Hugging Face commit). Buffered generation also follows this rule even though it internally collects worker chunks. True SSE responses (`stream: true`) always omit the header because headers precede the terminal result. Successful terminal SSE events may instead carry the worker-origin `execution_identity_sha256` / `execution_binding_sha256` pair, both lowercase 64-hex digests; absence remains valid for older or self-hosted deployments and does not attest execution. These digests are distinct from both the weights revision and the bundle/config hash.
- On scale-from-zero — i.e. no healthy worker registered for the `(bundle, machine_profile)` tuple and the caller did not pin an explicit pool — the gateway records pending demand for KEDA and returns a retryable `503` provisioning response with `Retry-After: 60`, `X-SIE-Error-Code: PROVISIONING`, and gateway version headers. SIE-native surfaces use the SDK retry envelope (`{"error":{"code","message"}}`); OpenAI-compatible surfaces (`/v1/generate`, `/v1/embeddings`, `/v1/chat/completions`, `/v1/completions`, `/v1/responses`) use the OpenAI error envelope, because standard OpenAI clients parse 2xx as successful model output. This applies whether or not the caller set `X-SIE-MACHINE-PROFILE`; default-routing clients get the same contract as profile-pinned clients.
- On no-consumer conditions for the JetStream publish, the gateway treats the miss as the same pre-execution provisioning state and returns the same retryable `503 PROVISIONING` contract (`Retry-After: 60`, `X-SIE-Error-Code: PROVISIONING`, and surface-specific error envelope). On backpressure conditions the gateway returns `503` with `Retry-After: 5`. Backpressure is evaluated twice: pool-wide against the `WORK_POOL_{pool}` stream's pending count (`SIE_GATEWAY_MAX_STREAM_PENDING`), and per lane (`pool`/`machine_profile`/`bundle`) against a gateway-local in-flight work-item count (`SIE_GATEWAY_MAX_LANE_IN_FLIGHT_ITEMS`). The per-lane decision is always computed and recorded on `sie.gateway.queue.lane_admission.decisions`, but it only sheds when `SIE_GATEWAY_LANE_BACKPRESSURE_ENFORCE` is set; with the flag off (the default) the pool-wide check is the sole gate. Both sheds produce the same `503` contract and both record pending demand for the exact physical lane so KEDA scales it.
- On queue result timeouts the gateway returns `504` with `X-SIE-Error-Code: GATEWAY_TIMEOUT` and `Retry-After: 5`. This means the gateway accepted and published the work item, but no worker result reached the gateway before `SIE_GATEWAY_REQUEST_TIMEOUT`; it must not be collapsed into worker-emitted `MODEL_LOADING`, which remains a separate retryable `503 MODEL_LOADING` signal.
Expand Down
Loading
Loading