Document service accounts

docs/ai-gateway/budgets-and-pricing.mdx

@@ -6,10 +6,11 @@ description:
   spend through the Enterprise Manager API.
 ---
 
-A _budget_ caps model spend in USD for a directory user or group. Before the AI
-Gateway calls a provider, it asks the budgets service in the Enterprise Manager
-to admit the request. After the response, it reports the provider's usage so the
-service can charge it.
+A _budget_ caps model spend in USD for a directory user, group, or
+[service account](../platform/enterprise-directory/service-accounts.mdx). Before
+the AI Gateway calls a provider, it asks the budgets service in the Enterprise
+Manager to admit the request. After the response, it reports the provider's
+usage so the service can charge it.
 
 Use [Budgets](manage-budgets.mdx) in the console for routine administration. Use
 the REST API for automation, bulk import, and pricing. To connect the gateway to
@@ -29,7 +30,9 @@ connect a gateway that serves traffic, either
 directory users without an explicit budget, or create budgets individually. Use
 the [bulk import](#import-in-bulk) for an existing user and group inventory.
 
-The organization default applies only to resolved directory users.
+The organization default applies only to resolved directory users. It never
+funds a service account, so give each service account its own budget or add it
+to a group that has one.
 
 ## Create budgets
 
@@ -41,11 +44,12 @@ In the examples on this page, replace `<ENTERPRISE_MANAGER_URL>` with its
 and `<ADMIN_TOKEN>` with a
 [platform administrator's access token](../platform/enterprise-platform/enterprise-manager-api.mdx#authenticate).
 
-The API identifies a budget by scope and subject. `scope` is `user` or `group`.
-`subject_id` is the user's directory UUID or the directory group's UUID. Find a
-user's UUID in the `id` field that `GET /v1/users` returns. That endpoint
-accepts an `email` filter. `limit_usd` is a decimal string and `period` is
-`day`, `month`, or `year`.
+The API identifies a budget by scope and subject. `scope` is `user`, `group`, or
+`service_account`. `subject_id` is the directory UUID of the user, group, or
+service account. Find a user's UUID in the `id` field that `GET /v1/users`
+returns. That endpoint accepts an `email` filter. Find a service account's UUID
+in the `id` field that `GET /v1/service-accounts` returns. `limit_usd` is a
+decimal string and `period` is `day`, `month`, or `year`.
 
 ### Set an organization default
 
@@ -66,7 +70,7 @@ explicit budget preserves their current-period usage.
 
 ### Update one budget
 
-Use the user or group endpoint to create or update one budget:
+Use the user, group, or service account endpoint to create or update one budget:
 
 ```bash
 curl -X PUT <ENTERPRISE_MANAGER_URL>/v1/budgets/users/<USER_UUID> \
@@ -78,6 +82,11 @@ curl -X PUT <ENTERPRISE_MANAGER_URL>/v1/budgets/groups/<GROUP_UUID> \
   -H "Authorization: Bearer <ADMIN_TOKEN>" \
   -H "Content-Type: application/json" \
   -d '{"limit_usd": "500.000000000", "period": "month", "precedence": 1}'
+
+curl -X PUT <ENTERPRISE_MANAGER_URL>/v1/budgets/service-accounts/<SERVICE_ACCOUNT_UUID> \
+  -H "Authorization: Bearer <ADMIN_TOKEN>" \
+  -H "Content-Type: application/json" \
+  -d '{"limit_usd": "20.000000000", "period": "month"}'
 ```
 
 Group budgets use 1-based `precedence`. See
@@ -114,13 +123,15 @@ means re-read and retry.
 
 The gateway charges budgets in this order:
 
-1. The user's explicit budget, or the organization default.
+1. The caller's own budget. For a user without one, the organization default.
 2. The first group budget with remaining capacity, ordered by ascending
    `precedence`.
 
-The gateway denies the request when no applicable budget has capacity. Because a
-group budget takes over when the user's own budget runs out, a user's budget
-caps their spend only when none of their groups has a budget with capacity.
+A service account has no organization default, so it draws only on its own
+budget and its groups' budgets. The gateway denies the request when no
+applicable budget has capacity. Because a group budget takes over when the
+caller's own budget runs out, a user's budget caps their spend only when none of
+their groups has a budget with capacity.
 
 ## Update the pricing catalog
 

docs/ai-gateway/control-model-access.mdx

@@ -32,6 +32,11 @@ Users** and grant it to that group through another set.
 **All Users** starts out granting every model, so turning model access on
 changes nothing until you narrow it.
 
+[Service accounts](../platform/enterprise-directory/service-accounts.mdx) never
+hold the **All Users** set, so a service account uses only the models in sets
+granted to it or to its groups. A service account can't call the model discovery
+endpoints. Grant a set to an account on the set's **Service Accounts** tab.
+
 Access is separate from spend. A caller also needs an applicable budget; see
 [Budgets and pricing](./budgets-and-pricing.mdx). Grant changes apply on the
 caller's next request, including requests that use a virtual API key, without

docs/ai-gateway/endpoints-reference.mdx

@@ -84,8 +84,11 @@ it from the first of these that is present:
 
 A credential shaped like a JWT is verified by the gateway itself, against the
 signing keys, audience, and optional required scope of the matching identity
-provider in `auth.issuers`. Any other credential is treated as a virtual API key
-and validated by the directory.
+provider in `auth.issuers`. A JWT whose issuer is an enabled trusted issuer, and
+none of `auth.issuers`, is verified as a service account's token instead, and
+only from `Authorization: Bearer`. See
+[Call the AI Gateway from Kubernetes workloads](./kubernetes-workloads.mdx). Any
+other credential is treated as a virtual API key and validated by the directory.
 
 When `directory.endpoint` is set, the gateway then looks up every verified
 caller in the directory to resolve their groups. A JWT caller the directory

docs/ai-gateway/forward-audit-logs.mdx

@@ -104,7 +104,15 @@ canonical subject, and `virtual_key_id` identifies the virtual key on
 virtual-key requests. Directory group labels named in `directory.groupLabels`,
 such as `department`, are added alongside them. A rejected authentication event
 carries no subjects, except an `auth_identity_unresolved` refusal, which names
-the verified caller in `user`.
+the verified caller in `user`, or in the service account subjects described
+next.
+
+A [service account](../platform/enterprise-directory/service-accounts.mdx)'s
+events name its workload instead of a user. `service_account` is the verified
+subject, in the form `system:serviceaccount:<NAMESPACE>:<NAME>`,
+`service_account_authority` identifies the trusted issuer that verified it, and
+`service_account_uid` is the ServiceAccount's UID, which tells a recreated
+ServiceAccount apart from the one it replaced.
 
 **Targets** carry `model`, `provider`, and `route`, each only once the gateway
 has resolved it. `provider` is the provider `kind` from the gateway
@@ -117,9 +125,9 @@ address or user agent.
 **Common extras** are `schema_version`, `request_id`, `http_status`, and, when
 tracing is on, `trace_id` and `span_id` for OpenTelemetry correlation. The
 gateway also returns `request_id` to the client in the `X-Request-ID` response
-header. Authentication events add `auth_method`: `jwt`, `virtual_key`, or
-`unknown`, and `reason_code`, which says why authentication failed, such as
-`auth_identity_unresolved`.
+header. Authentication events add `auth_method`: `jwt`, `virtual_key`,
+`workload`, or `unknown`, and `reason_code`, which says why authentication
+failed, such as `auth_identity_unresolved`.
 
 Interaction events add `interaction_id`, `http_method`, `dialect`, `streaming`,
 `duration_ms`, `terminal_stage`, `reason_code`, and `usage_present`, plus

docs/ai-gateway/index.mdx

@@ -29,15 +29,16 @@ The AI Gateway governs model access. The
 
 ## What you can do
 
-- Set cost budgets for users and groups.
-- Limit which models each user or group can call, and show each caller only the
-  models they can use.
+- Set cost budgets for users, groups, and service accounts.
+- Limit which models each user, group, or service account can call, and show
+  users only the models they can use.
 - Screen requests for prompt injection, payment card data, and personally
   identifiable information (PII).
 - Record structured audit events and forward them to a security information and
   event management (SIEM) system.
-- Authenticate callers with your identity provider or with virtual API keys,
-  while provider credentials stay in the gateway.
+- Authenticate callers with your identity provider or with virtual API keys, and
+  Kubernetes workloads with their ServiceAccount tokens, while provider
+  credentials stay in the gateway.
 - Export traces and metrics to an OpenTelemetry collector, attributed to the
   users and groups that sent the traffic.
 
@@ -74,7 +75,9 @@ Users find their deployment's endpoints and per-client setup guides in the
 console. To prepare access and point users there, see
 [Roll out gateway clients](../platform/enterprise-platform/roll-out-gateway-clients.mdx).
 For the endpoints and credential headers clients use, see the
-[endpoints reference](./endpoints-reference.mdx).
+[endpoints reference](./endpoints-reference.mdx). To call the gateway from a
+Kubernetes workload as a service account, see
+[Call the AI Gateway from Kubernetes workloads](./kubernetes-workloads.mdx).
 
 ## How configuration works
 

docs/ai-gateway/kubernetes-workloads.mdx

@@ -0,0 +1,176 @@
+---
+title: Call the AI Gateway from Kubernetes workloads
+sidebar_label: Kubernetes workloads
+description:
+  Authenticate a Kubernetes workload to the AI Gateway as its service account
+  with a projected ServiceAccount token, and handle the gateway's refusals.
+---
+
+A workload running in Kubernetes calls the AI Gateway as its
+[service account](../platform/enterprise-directory/service-accounts.mdx) by
+sending a projected ServiceAccount token. Kubernetes issues the token to the pod
+and replaces it before it expires, so the workload holds no long-lived
+credential. The same token also authenticates the workload to the
+[Connector Gateway](../connector-gateway/kubernetes-workloads.mdx).
+
+## Prerequisites
+
+- **An active service account bound to the ServiceAccount your pod runs as.** An
+  administrator creates and binds it, and grants it models and a budget when the
+  gateway enforces them. See
+  [Service accounts](../platform/enterprise-directory/service-accounts.mdx).
+- **The trusted issuer's audience**, such as `stacklok-enterprise-platform`.
+  Your administrator sets it and can confirm it on the **Trusted issuers** tab
+  in the console.
+- **The AI Gateway host** and the names of the models the account can use. Model
+  discovery isn't available to service accounts, so the workload names its model
+  in each request.
+
+## Mount a token for the gateway
+
+Add a projected token volume to the pod spec and mount it into the container
+that calls the gateway:
+
+```yaml title="deployment.yaml"
+spec:
+  template:
+    spec:
+      serviceAccountName: <SERVICE_ACCOUNT_NAME>
+      containers:
+        - name: <CONTAINER_NAME>
+          image: <IMAGE>
+          volumeMounts:
+            - name: stacklok-token
+              mountPath: /var/run/secrets/stacklok
+              readOnly: true
+      volumes:
+        - name: stacklok-token
+          projected:
+            sources:
+              - serviceAccountToken:
+                  path: token
+                  # highlight-next-line
+                  audience: <AUDIENCE>
+                  # highlight-next-line
+                  expirationSeconds: 1800
+```
+
+Set `audience` to the trusted issuer's audience. The token carries it as its
+only audience, and the gateway refuses a token with any other audience or with
+more than one. Request tokens with this audience only for Stacklok Enterprise.
+Any other service that receives such a token can present it to the gateways
+until it expires, so request a separate token with its own audience for every
+other service the workload calls. The gateway also refuses the default token
+Kubernetes mounts for the API server.
+
+Set `expirationSeconds` to `1800`. The gateway refuses a token with more than 60
+minutes of lifetime left, and Kubernetes requires at least `600`. The kubelet
+writes a fresh token to the file before the current one expires.
+
+## Send requests
+
+Read the token from the file for each request and send it in the
+`Authorization: Bearer` header. The gateway accepts a workload token only in
+that header. Send it only to the gateway's HTTPS endpoint.
+
+```bash
+curl https://<AI_GATEWAY_HOST>/v1/chat/completions \
+  -H "Authorization: Bearer $(cat /var/run/secrets/stacklok/token)" \
+  -H "Content-Type: application/json" \
+  -d '{"model":"<MODEL_NAME>","messages":[{"role":"user","content":"Reply with pong"}]}'
+```
+
+OpenAI SDKs send their API key in this header, so pass the token as the key with
+`https://<AI_GATEWAY_HOST>/v1` as the base URL. Anthropic SDKs use this header
+when you pass the token as the auth token instead of the API key, with
+`https://<AI_GATEWAY_HOST>/anthropic` as the base URL. In both cases, read the
+file again before each request or create the client per request, because the
+kubelet replaces the token and the previous one expires.
+
+Treat the token as a password until it expires. Don't log it, print it, or send
+it anywhere except the gateways.
+
+## Handle errors
+
+The gateway refuses a request before it calls a provider, and it charges no
+budget for a refused request:
+
+| Status | Message contains                                       | Meaning                                                                                                                                                                                         |
+| ------ | ------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `401`  | `unauthenticated`                                      | The token failed verification: wrong audience, more than 60 minutes of lifetime left, expired, or from an issuer that isn't trusted, is disabled, or that the gateway couldn't fetch or install |
+| `402`  | `budget exhausted` or `no applicable budget`           | The account's budgets are spent, or none applies to it                                                                                                                                          |
+| `403`  | `identity not resolved by the directory`               | The token is valid, but no active service account is bound to this ServiceAccount, such as when the account is suspended or deleted                                                             |
+| `403`  | `model not permitted`                                  | The account has no grant for the requested model                                                                                                                                                |
+| `403`  | `model discovery is not available to service accounts` | The workload called a model list endpoint                                                                                                                                                       |
+| `503`  | `service temporarily unavailable`                      | The gateway can't reach the directory. Retry after the `Retry-After` delay                                                                                                                      |
+
+The workload sees only these messages. An administrator finds the details in the
+AI Gateway's [audit events](./forward-audit-logs.mdx), where each refused
+sign-in is an authentication event:
+
+- `auth_method` set to `workload` with `reason_code`
+  `auth_invalid_workload_token` means the token came from a trusted issuer but
+  failed a check, such as its audience or lifetime.
+- `auth_method` set to `jwt` with `reason_code` `auth_invalid_jwt` means the
+  gateway has no installed issuer for the token: the issuer is missing or
+  disabled, or the gateway couldn't fetch or install it. Check the issuer on the
+  **Trusted issuers** tab, then the gateway logs described in
+  [Troubleshooting](#troubleshooting).
+- `reason_code` `auth_identity_unresolved` is a `403` for a valid token. The
+  event names the ServiceAccount in its `service_account` subject, in the form
+  `system:serviceaccount:<NAMESPACE>:<NAME>`, with its UID in
+  `service_account_uid`.
+
+For an unresolved identity, the Enterprise Manager records why the directory
+found no account. It counts each refusal in the
+`stacklok.enterprise_manager.directory.workload.resolutions` metric with a
+`reason` attribute, such as `no_binding`, `account_inactive`, or
+`trust_disabled`, and logs the same reason at debug level with the namespace,
+name, and UID.
+
+## Next steps
+
+- [Manage budgets](./manage-budgets.mdx) to raise a service account's budget.
+- [Control model access](./control-model-access.mdx) to grant the account
+  another model.
+
+## Related information
+
+- [Service accounts](../platform/enterprise-directory/service-accounts.mdx) for
+  bindings, activation, and revocation.
+- [Call the Connector Gateway from Kubernetes workloads](../connector-gateway/kubernetes-workloads.mdx)
+  to call connector tools with the same token.
+- [Client endpoints](../platform/enterprise-platform/configure-ai-gateway.mdx#client-endpoints)
+  for the base URL of each API format.
+
+## Troubleshooting
+
+<details>
+<summary>Requests fail with 401 after the workload runs for a while</summary>
+
+The workload is sending a token it read earlier, which has since expired. Read
+the token file before each request.
+
+</details>
+
+<details>
+<summary>Requests fail with 401 although the issuer is enabled</summary>
+
+The gateway installs a new issuer on its next poll of the directory, every 30
+seconds by default. Retry after the poll. If requests still fail, an
+administrator checks the gateway logs:
+
+- `ai-gateway-v2: workload trust snapshot poll failed` means the gateway
+  couldn't fetch the issuers. Its `state` is `FailedPrecondition` when the
+  gateway connects to the directory's cleartext listener instead of TLS,
+  `PermissionDenied` when its ServiceAccount is missing from
+  `workloadTrustSaSubjectAllowlist`, and `Unavailable` when it can't reach the
+  directory or verify its certificate.
+- `ai-gateway-v2: workload trust dropped` means the gateway fetched an issuer
+  but couldn't install it. Its `reason` says why, such as `key_fetch_failed`
+  when an egress rule is missing.
+
+See
+[Accept service account workloads](../platform/enterprise-platform/configure-ai-gateway.mdx#accept-service-account-workloads).
+
+</details>