diff --git a/docs/ai-gateway/budgets-and-pricing.mdx b/docs/ai-gateway/budgets-and-pricing.mdx index a43b3033..e4658b6c 100644 --- a/docs/ai-gateway/budgets-and-pricing.mdx +++ b/docs/ai-gateway/budgets-and-pricing.mdx @@ -6,10 +6,11 @@ description: spend through the Enterprise Manager API. --- -A _budget_ caps model spend in USD for a directory user or group. Before the AI -Gateway calls a provider, it asks the budgets service in the Enterprise Manager -to admit the request. After the response, it reports the provider's usage so the -service can charge it. +A _budget_ caps model spend in USD for a directory user, group, or +[service account](../platform/enterprise-directory/service-accounts.mdx). Before +the AI Gateway calls a provider, it asks the budgets service in the Enterprise +Manager to admit the request. After the response, it reports the provider's +usage so the service can charge it. Use [Budgets](manage-budgets.mdx) in the console for routine administration. Use the REST API for automation, bulk import, and pricing. To connect the gateway to @@ -29,7 +30,9 @@ connect a gateway that serves traffic, either directory users without an explicit budget, or create budgets individually. Use the [bulk import](#import-in-bulk) for an existing user and group inventory. -The organization default applies only to resolved directory users. +The organization default applies only to resolved directory users. It never +funds a service account, so give each service account its own budget or add it +to a group that has one. ## Create budgets @@ -41,11 +44,12 @@ In the examples on this page, replace `` with its and `` with a [platform administrator's access token](../platform/enterprise-platform/enterprise-manager-api.mdx#authenticate). -The API identifies a budget by scope and subject. `scope` is `user` or `group`. -`subject_id` is the user's directory UUID or the directory group's UUID. Find a -user's UUID in the `id` field that `GET /v1/users` returns. That endpoint -accepts an `email` filter. `limit_usd` is a decimal string and `period` is -`day`, `month`, or `year`. +The API identifies a budget by scope and subject. `scope` is `user`, `group`, or +`service_account`. `subject_id` is the directory UUID of the user, group, or +service account. Find a user's UUID in the `id` field that `GET /v1/users` +returns. That endpoint accepts an `email` filter. Find a service account's UUID +in the `id` field that `GET /v1/service-accounts` returns. `limit_usd` is a +decimal string and `period` is `day`, `month`, or `year`. ### Set an organization default @@ -66,7 +70,7 @@ explicit budget preserves their current-period usage. ### Update one budget -Use the user or group endpoint to create or update one budget: +Use the user, group, or service account endpoint to create or update one budget: ```bash curl -X PUT /v1/budgets/users/ \ @@ -78,6 +82,11 @@ curl -X PUT /v1/budgets/groups/ \ -H "Authorization: Bearer " \ -H "Content-Type: application/json" \ -d '{"limit_usd": "500.000000000", "period": "month", "precedence": 1}' + +curl -X PUT /v1/budgets/service-accounts/ \ + -H "Authorization: Bearer " \ + -H "Content-Type: application/json" \ + -d '{"limit_usd": "20.000000000", "period": "month"}' ``` Group budgets use 1-based `precedence`. See @@ -114,13 +123,15 @@ means re-read and retry. The gateway charges budgets in this order: -1. The user's explicit budget, or the organization default. +1. The caller's own budget. For a user without one, the organization default. 2. The first group budget with remaining capacity, ordered by ascending `precedence`. -The gateway denies the request when no applicable budget has capacity. Because a -group budget takes over when the user's own budget runs out, a user's budget -caps their spend only when none of their groups has a budget with capacity. +A service account has no organization default, so it draws only on its own +budget and its groups' budgets. The gateway denies the request when no +applicable budget has capacity. Because a group budget takes over when the +caller's own budget runs out, a user's budget caps their spend only when none of +their groups has a budget with capacity. ## Update the pricing catalog diff --git a/docs/ai-gateway/control-model-access.mdx b/docs/ai-gateway/control-model-access.mdx index e04d59c2..ca8f54d9 100644 --- a/docs/ai-gateway/control-model-access.mdx +++ b/docs/ai-gateway/control-model-access.mdx @@ -32,6 +32,11 @@ Users** and grant it to that group through another set. **All Users** starts out granting every model, so turning model access on changes nothing until you narrow it. +[Service accounts](../platform/enterprise-directory/service-accounts.mdx) never +hold the **All Users** set, so a service account uses only the models in sets +granted to it or to its groups. A service account can't call the model discovery +endpoints. Grant a set to an account on the set's **Service Accounts** tab. + Access is separate from spend. A caller also needs an applicable budget; see [Budgets and pricing](./budgets-and-pricing.mdx). Grant changes apply on the caller's next request, including requests that use a virtual API key, without diff --git a/docs/ai-gateway/endpoints-reference.mdx b/docs/ai-gateway/endpoints-reference.mdx index b71cb539..26dea861 100644 --- a/docs/ai-gateway/endpoints-reference.mdx +++ b/docs/ai-gateway/endpoints-reference.mdx @@ -84,8 +84,11 @@ it from the first of these that is present: A credential shaped like a JWT is verified by the gateway itself, against the signing keys, audience, and optional required scope of the matching identity -provider in `auth.issuers`. Any other credential is treated as a virtual API key -and validated by the directory. +provider in `auth.issuers`. A JWT whose issuer is an enabled trusted issuer, and +none of `auth.issuers`, is verified as a service account's token instead, and +only from `Authorization: Bearer`. See +[Call the AI Gateway from Kubernetes workloads](./kubernetes-workloads.mdx). Any +other credential is treated as a virtual API key and validated by the directory. When `directory.endpoint` is set, the gateway then looks up every verified caller in the directory to resolve their groups. A JWT caller the directory diff --git a/docs/ai-gateway/forward-audit-logs.mdx b/docs/ai-gateway/forward-audit-logs.mdx index 02f62033..7d66d1b1 100644 --- a/docs/ai-gateway/forward-audit-logs.mdx +++ b/docs/ai-gateway/forward-audit-logs.mdx @@ -104,7 +104,15 @@ canonical subject, and `virtual_key_id` identifies the virtual key on virtual-key requests. Directory group labels named in `directory.groupLabels`, such as `department`, are added alongside them. A rejected authentication event carries no subjects, except an `auth_identity_unresolved` refusal, which names -the verified caller in `user`. +the verified caller in `user`, or in the service account subjects described +next. + +A [service account](../platform/enterprise-directory/service-accounts.mdx)'s +events name its workload instead of a user. `service_account` is the verified +subject, in the form `system:serviceaccount::`, +`service_account_authority` identifies the trusted issuer that verified it, and +`service_account_uid` is the ServiceAccount's UID, which tells a recreated +ServiceAccount apart from the one it replaced. **Targets** carry `model`, `provider`, and `route`, each only once the gateway has resolved it. `provider` is the provider `kind` from the gateway @@ -117,9 +125,9 @@ address or user agent. **Common extras** are `schema_version`, `request_id`, `http_status`, and, when tracing is on, `trace_id` and `span_id` for OpenTelemetry correlation. The gateway also returns `request_id` to the client in the `X-Request-ID` response -header. Authentication events add `auth_method`: `jwt`, `virtual_key`, or -`unknown`, and `reason_code`, which says why authentication failed, such as -`auth_identity_unresolved`. +header. Authentication events add `auth_method`: `jwt`, `virtual_key`, +`workload`, or `unknown`, and `reason_code`, which says why authentication +failed, such as `auth_identity_unresolved`. Interaction events add `interaction_id`, `http_method`, `dialect`, `streaming`, `duration_ms`, `terminal_stage`, `reason_code`, and `usage_present`, plus diff --git a/docs/ai-gateway/index.mdx b/docs/ai-gateway/index.mdx index 5326fd3b..6825cb87 100644 --- a/docs/ai-gateway/index.mdx +++ b/docs/ai-gateway/index.mdx @@ -29,15 +29,16 @@ The AI Gateway governs model access. The ## What you can do -- Set cost budgets for users and groups. -- Limit which models each user or group can call, and show each caller only the - models they can use. +- Set cost budgets for users, groups, and service accounts. +- Limit which models each user, group, or service account can call, and show + users only the models they can use. - Screen requests for prompt injection, payment card data, and personally identifiable information (PII). - Record structured audit events and forward them to a security information and event management (SIEM) system. -- Authenticate callers with your identity provider or with virtual API keys, - while provider credentials stay in the gateway. +- Authenticate callers with your identity provider or with virtual API keys, and + Kubernetes workloads with their ServiceAccount tokens, while provider + credentials stay in the gateway. - Export traces and metrics to an OpenTelemetry collector, attributed to the users and groups that sent the traffic. @@ -74,7 +75,9 @@ Users find their deployment's endpoints and per-client setup guides in the console. To prepare access and point users there, see [Roll out gateway clients](../platform/enterprise-platform/roll-out-gateway-clients.mdx). For the endpoints and credential headers clients use, see the -[endpoints reference](./endpoints-reference.mdx). +[endpoints reference](./endpoints-reference.mdx). To call the gateway from a +Kubernetes workload as a service account, see +[Call the AI Gateway from Kubernetes workloads](./kubernetes-workloads.mdx). ## How configuration works diff --git a/docs/ai-gateway/kubernetes-workloads.mdx b/docs/ai-gateway/kubernetes-workloads.mdx new file mode 100644 index 00000000..137bdd7b --- /dev/null +++ b/docs/ai-gateway/kubernetes-workloads.mdx @@ -0,0 +1,176 @@ +--- +title: Call the AI Gateway from Kubernetes workloads +sidebar_label: Kubernetes workloads +description: + Authenticate a Kubernetes workload to the AI Gateway as its service account + with a projected ServiceAccount token, and handle the gateway's refusals. +--- + +A workload running in Kubernetes calls the AI Gateway as its +[service account](../platform/enterprise-directory/service-accounts.mdx) by +sending a projected ServiceAccount token. Kubernetes issues the token to the pod +and replaces it before it expires, so the workload holds no long-lived +credential. The same token also authenticates the workload to the +[Connector Gateway](../connector-gateway/kubernetes-workloads.mdx). + +## Prerequisites + +- **An active service account bound to the ServiceAccount your pod runs as.** An + administrator creates and binds it, and grants it models and a budget when the + gateway enforces them. See + [Service accounts](../platform/enterprise-directory/service-accounts.mdx). +- **The trusted issuer's audience**, such as `stacklok-enterprise-platform`. + Your administrator sets it and can confirm it on the **Trusted issuers** tab + in the console. +- **The AI Gateway host** and the names of the models the account can use. Model + discovery isn't available to service accounts, so the workload names its model + in each request. + +## Mount a token for the gateway + +Add a projected token volume to the pod spec and mount it into the container +that calls the gateway: + +```yaml title="deployment.yaml" +spec: + template: + spec: + serviceAccountName: + containers: + - name: + image: + volumeMounts: + - name: stacklok-token + mountPath: /var/run/secrets/stacklok + readOnly: true + volumes: + - name: stacklok-token + projected: + sources: + - serviceAccountToken: + path: token + # highlight-next-line + audience: + # highlight-next-line + expirationSeconds: 1800 +``` + +Set `audience` to the trusted issuer's audience. The token carries it as its +only audience, and the gateway refuses a token with any other audience or with +more than one. Request tokens with this audience only for Stacklok Enterprise. +Any other service that receives such a token can present it to the gateways +until it expires, so request a separate token with its own audience for every +other service the workload calls. The gateway also refuses the default token +Kubernetes mounts for the API server. + +Set `expirationSeconds` to `1800`. The gateway refuses a token with more than 60 +minutes of lifetime left, and Kubernetes requires at least `600`. The kubelet +writes a fresh token to the file before the current one expires. + +## Send requests + +Read the token from the file for each request and send it in the +`Authorization: Bearer` header. The gateway accepts a workload token only in +that header. Send it only to the gateway's HTTPS endpoint. + +```bash +curl https:///v1/chat/completions \ + -H "Authorization: Bearer $(cat /var/run/secrets/stacklok/token)" \ + -H "Content-Type: application/json" \ + -d '{"model":"","messages":[{"role":"user","content":"Reply with pong"}]}' +``` + +OpenAI SDKs send their API key in this header, so pass the token as the key with +`https:///v1` as the base URL. Anthropic SDKs use this header +when you pass the token as the auth token instead of the API key, with +`https:///anthropic` as the base URL. In both cases, read the +file again before each request or create the client per request, because the +kubelet replaces the token and the previous one expires. + +Treat the token as a password until it expires. Don't log it, print it, or send +it anywhere except the gateways. + +## Handle errors + +The gateway refuses a request before it calls a provider, and it charges no +budget for a refused request: + +| Status | Message contains | Meaning | +| ------ | ------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `401` | `unauthenticated` | The token failed verification: wrong audience, more than 60 minutes of lifetime left, expired, or from an issuer that isn't trusted, is disabled, or that the gateway couldn't fetch or install | +| `402` | `budget exhausted` or `no applicable budget` | The account's budgets are spent, or none applies to it | +| `403` | `identity not resolved by the directory` | The token is valid, but no active service account is bound to this ServiceAccount, such as when the account is suspended or deleted | +| `403` | `model not permitted` | The account has no grant for the requested model | +| `403` | `model discovery is not available to service accounts` | The workload called a model list endpoint | +| `503` | `service temporarily unavailable` | The gateway can't reach the directory. Retry after the `Retry-After` delay | + +The workload sees only these messages. An administrator finds the details in the +AI Gateway's [audit events](./forward-audit-logs.mdx), where each refused +sign-in is an authentication event: + +- `auth_method` set to `workload` with `reason_code` + `auth_invalid_workload_token` means the token came from a trusted issuer but + failed a check, such as its audience or lifetime. +- `auth_method` set to `jwt` with `reason_code` `auth_invalid_jwt` means the + gateway has no installed issuer for the token: the issuer is missing or + disabled, or the gateway couldn't fetch or install it. Check the issuer on the + **Trusted issuers** tab, then the gateway logs described in + [Troubleshooting](#troubleshooting). +- `reason_code` `auth_identity_unresolved` is a `403` for a valid token. The + event names the ServiceAccount in its `service_account` subject, in the form + `system:serviceaccount::`, with its UID in + `service_account_uid`. + +For an unresolved identity, the Enterprise Manager records why the directory +found no account. It counts each refusal in the +`stacklok.enterprise_manager.directory.workload.resolutions` metric with a +`reason` attribute, such as `no_binding`, `account_inactive`, or +`trust_disabled`, and logs the same reason at debug level with the namespace, +name, and UID. + +## Next steps + +- [Manage budgets](./manage-budgets.mdx) to raise a service account's budget. +- [Control model access](./control-model-access.mdx) to grant the account + another model. + +## Related information + +- [Service accounts](../platform/enterprise-directory/service-accounts.mdx) for + bindings, activation, and revocation. +- [Call the Connector Gateway from Kubernetes workloads](../connector-gateway/kubernetes-workloads.mdx) + to call connector tools with the same token. +- [Client endpoints](../platform/enterprise-platform/configure-ai-gateway.mdx#client-endpoints) + for the base URL of each API format. + +## Troubleshooting + +
+Requests fail with 401 after the workload runs for a while + +The workload is sending a token it read earlier, which has since expired. Read +the token file before each request. + +
+ +
+Requests fail with 401 although the issuer is enabled + +The gateway installs a new issuer on its next poll of the directory, every 30 +seconds by default. Retry after the poll. If requests still fail, an +administrator checks the gateway logs: + +- `ai-gateway-v2: workload trust snapshot poll failed` means the gateway + couldn't fetch the issuers. Its `state` is `FailedPrecondition` when the + gateway connects to the directory's cleartext listener instead of TLS, + `PermissionDenied` when its ServiceAccount is missing from + `workloadTrustSaSubjectAllowlist`, and `Unavailable` when it can't reach the + directory or verify its certificate. +- `ai-gateway-v2: workload trust dropped` means the gateway fetched an issuer + but couldn't install it. Its `reason` says why, such as `key_fetch_failed` + when an egress rule is missing. + +See +[Accept service account workloads](../platform/enterprise-platform/configure-ai-gateway.mdx#accept-service-account-workloads). + +
diff --git a/docs/ai-gateway/manage-budgets.mdx b/docs/ai-gateway/manage-budgets.mdx index c2b98b9a..8d1974c0 100644 --- a/docs/ai-gateway/manage-budgets.mdx +++ b/docs/ai-gateway/manage-budgets.mdx @@ -62,6 +62,21 @@ runs out. Before you set a small custom budget to limit one user, check the budgets of the groups they belong to. See [Which budget gets charged](budgets-and-pricing.mdx#which-budget-gets-charged). +## Service account budgets + +The **Service Accounts** tab lists each +[service account](../platform/enterprise-directory/service-accounts.mdx)'s own +**Period**, **Budget**, **Used**, and **Utilization**. Select **Create budget**, +then choose a service account, period, and limit. Each account has at most one +budget of its own. + +The organization default never funds a service account, which draws on its own +budget and then its groups' pools. See +[Which budget gets charged](budgets-and-pricing.mdx#which-budget-gets-charged). +On a service account's page, opened from **Service Accounts**, the **Budgets** +tab shows its budgets in the order they're charged. A group budget's **Service +accounts** tab shows the accounts that pool funds and what each has spent. + ## Monitor spend The **Organization** screen, under **Cost Management**, is the overview of AI @@ -72,11 +87,12 @@ The **Total spend** card charts daily spend through the current day, covering the usage history the console retains, four weeks by default. It also compares spend over the last 7 days with the 7 days before. -Below the chart, the usage table's **Groups** and **Users** tabs list each -group's or user's spend, budget, and utilization. Search by name, or filter by -utilization: **All**, **Under 80%**, **80–100%**, or **Over 100%**. The -**Users** tab doesn't list spend charged to the organization default budget, so -it's empty when every user is on the default. +Below the chart, the usage table's **Groups**, **Users**, and **Service +Accounts** tabs list each group's, user's, or service account's spend, budget, +and utilization. Search by name, or filter by utilization: **All**, **Under +80%**, **80–100%**, or **Over 100%**. The **Users** tab doesn't list spend +charged to the organization default budget, so it's empty when every user is on +the default. For finance reports and external dashboards, use the [spend API](budgets-and-pricing.mdx#monitor-spend). diff --git a/docs/connector-gateway/connector-policies.mdx b/docs/connector-gateway/connector-policies.mdx index 05b066d6..c3c29107 100644 --- a/docs/connector-gateway/connector-policies.mdx +++ b/docs/connector-gateway/connector-policies.mdx @@ -26,6 +26,14 @@ Directory groups are separate from the OpenID Connect (OIDC) claim groups that cluster authorization policy matches. See [Directory groups and OIDC claim groups](../platform/concepts/two-group-models.mdx). +A group can also hold +[service accounts](../platform/enterprise-directory/service-accounts.mdx), which +reach a connector only when its authentication type can serve them, as described +in +[What an account can reach](../platform/enterprise-directory/service-accounts.mdx#what-an-account-can-reach). +For any other connector, the **Access** tab warns which granted groups hold +service accounts it can't serve. The people in those groups keep their access. + ## Revoke a group's access 1. On the connector's **Access** tab, open the actions menu for the group. @@ -40,6 +48,24 @@ A connector policy can also be in Cedar mode, where a Cedar policy document decides access instead of group grants. For a Cedar-mode connector, the console shows a notice on the **Access** tab and disables group editing. +Cedar sees a service account as the same `Client` principal as a user, with its +groups as parents and no token claims. On a connector whose authentication type +can serve service accounts, a policy with no condition on the principal, such as +`permit (principal, action, resource);`, admits every active service account, +including one in no group. To admit only the members of certain groups, users +and service accounts alike, scope the principal to those groups: + +```text +permit (principal in UserGroup::"", action, resource); +``` + +`` is the `name` that `GET /v1/groups` in the +[Enterprise Manager API](../platform/enterprise-platform/enterprise-manager-api.mdx) +returns for the group. It's set when the group is created and can differ from +the group's `display_name`, which the console shows. To admit only users, +condition the policy on a token claim, which also denies any user whose token +lacks that claim. + Switch modes and write Cedar policy documents through the connector policy routes of the [Enterprise Manager API](../platform/enterprise-platform/enterprise-manager-api.mdx). diff --git a/docs/connector-gateway/connectors.mdx b/docs/connector-gateway/connectors.mdx index d233ebb1..26bf98e7 100644 --- a/docs/connector-gateway/connectors.mdx +++ b/docs/connector-gateway/connectors.mdx @@ -198,6 +198,10 @@ A connector's **Activity** tab lists its recent allowed tool calls, with the - Select a user to show only that user's calls, and **Clear user filter** to return to everyone's. +A [service account](../platform/enterprise-directory/service-accounts.mdx)'s +calls show its name in the **User** column, with a **Service account** badge, +and you can select it to filter the same way. + The **Activity** tab is a best-effort view, not an audit log: calls can be delayed, sampled, or missing, and calls the connector policy refuses don't appear. For a complete per-request record, including refused calls, see diff --git a/docs/connector-gateway/forward-audit-logs.mdx b/docs/connector-gateway/forward-audit-logs.mdx index 5ddd5a96..a0e91535 100644 --- a/docs/connector-gateway/forward-audit-logs.mdx +++ b/docs/connector-gateway/forward-audit-logs.mdx @@ -128,6 +128,11 @@ readability: | `target.name`, `target.type` | The tool, resource, or prompt the request named, and which of those it is | | `metadata.extra` | `backend_name` (the connector), `duration_ms`, `transport`, and `response_size_bytes` | +For a call from a +[service account](../platform/enterprise-directory/service-accounts.mdx), +`subjects.user_id` holds the account's ID and `subjects.user` is `anonymous`, +because the account has no name, preferred username, or email. + ## Separate audit records from application logs Audit events go to standard output with `"level":"AUDIT"` and diff --git a/docs/connector-gateway/index.mdx b/docs/connector-gateway/index.mdx index d0a27e20..e4f5e4d8 100644 --- a/docs/connector-gateway/index.mdx +++ b/docs/connector-gateway/index.mdx @@ -31,6 +31,9 @@ governs the **tools** they can use. - **Roll out and support users.** [Roll out gateway clients](../platform/enterprise-platform/roll-out-gateway-clients.mdx) and [support users' connector access](./support-connector-access.mdx). +- **Connect Kubernetes workloads.** Workloads call the gateway as service + accounts, with no user signed in. See + [Call the Connector Gateway from Kubernetes workloads](./kubernetes-workloads.mdx). - **Operate the gateway.** Review [tool usage](./tool-usage.mdx), and export [telemetry](./telemetry.mdx) and [audit logs](./forward-audit-logs.mdx). @@ -44,7 +47,8 @@ administration, see the A _connector_ is an MCP server registered with the gateway. Every user points their MCP client at the same gateway endpoint and signs in with their corporate identity. The gateway then serves that user only the connectors they're granted -and connected to. +and connected to. A Kubernetes workload calls the same endpoint as its +[service account](../platform/enterprise-directory/service-accounts.mdx). ```mermaid flowchart LR diff --git a/docs/connector-gateway/kubernetes-workloads.mdx b/docs/connector-gateway/kubernetes-workloads.mdx new file mode 100644 index 00000000..55889756 --- /dev/null +++ b/docs/connector-gateway/kubernetes-workloads.mdx @@ -0,0 +1,185 @@ +--- +title: Call the Connector Gateway from Kubernetes workloads +sidebar_label: Kubernetes workloads +description: + Connect a Kubernetes workload to the Connector Gateway as its service account + with a projected ServiceAccount token, and handle the gateway's refusals. +--- + +A workload running in Kubernetes calls the Connector Gateway's MCP endpoint as +its [service account](../platform/enterprise-directory/service-accounts.mdx) by +sending a projected ServiceAccount token. No one signs in: Kubernetes issues the +token to the pod and replaces it before it expires. The same token also +authenticates the workload to the +[AI Gateway](../ai-gateway/kubernetes-workloads.mdx). + +## Prerequisites + +- **An active service account bound to the ServiceAccount your pod runs as**, in + a group that's granted the connectors the workload needs. See + [Service accounts](../platform/enterprise-directory/service-accounts.mdx). +- **The trusted issuer's audience**, such as `stacklok-enterprise-platform`. + Your administrator sets it and can confirm it on the **Trusted issuers** tab + in the console. +- **The gateway's MCP endpoint**, `https:///gw/mcp`, the same + endpoint users connect their clients to. + +## Mount a token for the gateway + +Add a projected token volume to the pod spec and mount it into the container +that calls the gateway: + +```yaml title="deployment.yaml" +spec: + template: + spec: + serviceAccountName: + containers: + - name: + image: + volumeMounts: + - name: stacklok-token + mountPath: /var/run/secrets/stacklok + readOnly: true + volumes: + - name: stacklok-token + projected: + sources: + - serviceAccountToken: + path: token + # highlight-next-line + audience: + # highlight-next-line + expirationSeconds: 1800 +``` + +Set `audience` to the trusted issuer's audience, and `expirationSeconds` to +`1800`. The gateway applies the same token rules as the AI Gateway: exactly one +audience and at most 60 minutes of lifetime left. See +[Mount a token for the gateway](../ai-gateway/kubernetes-workloads.mdx#mount-a-token-for-the-gateway) +for why to request this audience only for Stacklok Enterprise. A workload that +calls both gateways needs only this one volume. + +## Connect to the gateway + +Configure the workload's MCP client with the gateway's MCP endpoint, and send +the token in the `Authorization: Bearer` header of every request. Read the file +again before each request, because the kubelet replaces the token and the +previous one expires. The gateway binds the MCP session to the service account, +so the session survives the token change. + +To check that the gateway accepts the token, send an MCP `initialize` request +from the pod: + +```bash +curl -i https:///gw/mcp \ + -H "Authorization: Bearer $(cat /var/run/secrets/stacklok/token)" \ + -H "Content-Type: application/json" \ + -H "Accept: application/json, text/event-stream" \ + -d '{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2025-06-18","capabilities":{},"clientInfo":{"name":"token-check","version":"1.0.0"}}}' +``` + +A `200` response means the gateway accepted the token. The token works only on +the MCP endpoint; the gateway's REST API refuses it with `401`. + +The response's `Mcp-Session-Id` header identifies the MCP session. A workload +that sends HTTP requests itself, instead of through an MCP client library, +repeats that header and the same `Accept` header on every later request: first +the `notifications/initialized` notification, then `tools/list` and +`tools/call`. Without the session header, a `tools/list` or `tools/call` request +fails, because the gateway treats it as the start of a new session. + +Treat the token as a password until it expires. Don't log it, print it, or send +it anywhere except the gateways. + +## What the workload can reach + +The gateway serves the account every connector granted to one of its groups that +uses **None**, **Bearer token**, or **Header injection** authentication, as +described in +[What an account can reach](../platform/enterprise-directory/service-accounts.mdx#what-an-account-can-reach). +For a Cedar-mode connector, its Cedar policy decides: a policy with no condition +on the principal admits every active service account, and a policy that requires +a token claim admits none. See +[Cedar-mode connectors](./connector-policies.mdx#cedar-mode-connectors). + +A service account has no connections to make or pause. Its `tools/list` response +includes every tool its groups are allowed on each connector it can reach, and a +grant or revocation applies on its next request. Each tool's name combines the +connector's name, an ID, and the backend's own tool name, so call tools by the +names `tools/list` returns. The gateway sends the backend only the connector's +own credential, never the workload's token. + +## Handle errors + +| Status | Body | Meaning | +| ------ | ------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `401` | `invalid token` | The token failed verification: wrong audience, more than 60 minutes of lifetime left, or expired | +| `401` | Varies | The gateway has no installed issuer for the token, so it handled the request as a user's: the issuer is missing or disabled, or the gateway couldn't install it. Check the issuer on the **Trusted issuers** tab, then the gateway logs described in [Troubleshooting](#troubleshooting) | +| `403` | `service account not resolved` | The token is valid, but no active service account is bound to this ServiceAccount, such as when the account is suspended or deleted | +| `503` | `directory unavailable` | The gateway can't reach the directory. Retry after the `Retry-After` delay | + +The gateway counts each request that carries a token from a trusted issuer in +the `stacklok.connector_gateway.workload_identity.resolutions` metric, with an +`outcome` attribute of `success`, `token_rejected`, `identity_unresolved`, or +`directory_unavailable`. For an unresolved identity, the Enterprise Manager +records the reason in the +`stacklok.enterprise_manager.directory.workload.resolutions` metric, as +described in +[Handle errors](../ai-gateway/kubernetes-workloads.mdx#handle-errors) for the AI +Gateway. + +## Next steps + +- [Grant and revoke connector access](./connector-policies.mdx) to give the + account's groups the connectors the workload needs. +- [Review activity](./connectors.mdx#review-activity) to see the account's tool + calls. + +## Related information + +- [Service accounts](../platform/enterprise-directory/service-accounts.mdx) for + bindings, activation, and revocation. +- [Accept service account workloads](../platform/enterprise-platform/configure-connector-gateway.mdx#accept-service-account-workloads) + for the cluster setup the gateway needs. + +## Troubleshooting + +
+Requests fail with 401 after the workload runs for a while + +The workload is sending a token it read earlier, which has since expired. Read +the token file before each request. + +
+ +
+Requests fail with 401 although the issuer is enabled + +The gateway installs a new issuer on its next poll of the directory, every 30 +seconds by default. Retry after the poll. If requests still fail, an +administrator checks the gateway logs: + +- `workload trust snapshot poll failed` means the gateway couldn't fetch the + issuers. Its `state` is `PermissionDenied` when the gateway's ServiceAccount + is missing from `workloadTrustSaSubjectAllowlist`, and `Unavailable` when it + can't reach the directory or verify its certificate. +- `workload trust dropped` means the gateway fetched an issuer but couldn't + install it. Its `reason` says why, such as `key_fetch_failed` when an egress + rule is missing. + +See +[Accept service account workloads](../platform/enterprise-platform/configure-connector-gateway.mdx#accept-service-account-workloads). + +
+ +
+The workload lists no tools from a connector + +Check that the connector is granted to one of the account's groups and uses an +authentication type that can serve a service account, as listed in +[What an account can reach](../platform/enterprise-directory/service-accounts.mdx#what-an-account-can-reach). +The connector's **Access** tab warns when a granted group holds service accounts +it can't serve. + +
diff --git a/docs/platform/enterprise-console/index.mdx b/docs/platform/enterprise-console/index.mdx index f12d921d..24ecf511 100644 --- a/docs/platform/enterprise-console/index.mdx +++ b/docs/platform/enterprise-console/index.mdx @@ -29,6 +29,7 @@ Each administrative area is documented with the component that provides it: [Tool usage](../../connector-gateway/tool-usage.mdx), with the Connector Gateway that serves them. - [Users and groups](../enterprise-directory/users-and-groups.mdx), + [Service accounts](../enterprise-directory/service-accounts.mdx), [Identity providers](../../connector-gateway/identity-providers.mdx), and [Managed secrets](../../connector-gateway/managed-secrets.mdx), with the directory that stores them. diff --git a/docs/platform/enterprise-directory/index.mdx b/docs/platform/enterprise-directory/index.mdx index be0251b7..a93662f5 100644 --- a/docs/platform/enterprise-directory/index.mdx +++ b/docs/platform/enterprise-directory/index.mdx @@ -22,20 +22,23 @@ or API. ## What it holds -| Record | Used for | -| -------------------- | -------------------------------------------------- | -| Users | Resolve tokens and assign user budgets | -| Groups and subgroups | Grant connector access and assign group budgets | -| Identity providers | Broker OAuth from connectors to upstream services | -| Managed secrets | Store connector and identity provider credentials | -| Virtual API keys | Authenticate automated clients as a directory user | +| Record | Used for | +| -------------------- | ------------------------------------------------------- | +| Users | Resolve tokens and assign user budgets | +| Groups and subgroups | Grant connector access and assign group budgets | +| Service accounts | Give Kubernetes workloads their own identity and budget | +| Trusted issuers | Verify the ServiceAccount tokens workloads present | +| Identity providers | Broker OAuth from connectors to upstream services | +| Managed secrets | Store connector and identity provider credentials | +| Virtual API keys | Authenticate automated clients as a directory user | ## How identity is resolved The directory maps the issuer and subject in a caller's token to a user record. -It returns the user's direct and inherited group memberships to the requesting -component. Controls that require a directory identity deny callers without a -matching record. +For a workload's ServiceAccount token, a binding maps the token's namespace and +ServiceAccount name to a service account instead. It returns the caller's direct +and inherited group memberships to the requesting component. Controls that +require a directory identity deny callers without a matching record. :::warning @@ -47,10 +50,11 @@ authorization policy uses OIDC claim groups. See ## Where you administer it -Use **User management** for users, groups, and subgroups. Use **Identity -providers** and **Managed secrets** for connector authentication. Users manage -their virtual keys on the **API Keys** tab under **AI Gateway** in the console's -user experience. +Use **User management** for users, groups, and subgroups, and **Service +Accounts** for service accounts and trusted issuers. Use **Identity providers** +and **Managed secrets** for connector authentication. Users manage their virtual +keys on the **API Keys** tab under **AI Gateway** in the console's user +experience. The platform admin grant covers directory and budget administration. Use the `/v1` API for automation and bulk changes. diff --git a/docs/platform/enterprise-directory/service-accounts.mdx b/docs/platform/enterprise-directory/service-accounts.mdx new file mode 100644 index 00000000..e382b9d2 --- /dev/null +++ b/docs/platform/enterprise-directory/service-accounts.mdx @@ -0,0 +1,279 @@ +--- +title: Service accounts +sidebar_label: Service accounts +description: + Give Kubernetes workloads their own directory identity, model access, and + budget, authenticated by the ServiceAccount token their pods already have. +--- + +A _service account_ is a directory identity for a workload instead of a person. +An agent, job, or service running in Kubernetes calls the AI Gateway and the +Connector Gateway as its service account by presenting the projected +ServiceAccount token Kubernetes gives its pod. There's no key to issue, store, +or rotate. The account has its own groups, model set grants, and budget, and its +spend is attributed to it. + +Use a service account when a Kubernetes workload should act independently of any +user. Use a [virtual API key](./virtual-api-keys.mdx) when a script or CI job +should act as a person: the key carries its owner's access and budget, and the +client stores it. + +## How a workload is recognized + +A workload's token signs it in when three conditions hold: + +- A _trusted issuer_ tells Stacklok Enterprise which cluster's tokens to accept: + the cluster's issuer URL, where its public signing keys are, and the audience + workloads request their token for. +- A _binding_ maps the workload's Kubernetes ServiceAccount, by namespace and + name and optionally its UID, to one service account. +- The service account is **Active**. + +After sign-in, the account's model access and budgets decide which models it can +call, and its groups decide which connectors it reaches, as they do for a user. +See [What an account can reach](#what-an-account-can-reach). + +## Prerequisites + +- **Each gateway the workload calls, set up for workloads.** For the AI Gateway, + a cluster administrator completes + [Accept service account workloads](../enterprise-platform/configure-ai-gateway.mdx#accept-service-account-workloads) + once per installation. The Connector Gateway + [accepts workloads](../enterprise-platform/configure-connector-gateway.mdx#accept-service-account-workloads) + by default when it connects to the Enterprise Manager over TLS. +- **The platform admin grant**, which covers directory administration. +- **The Kubernetes ServiceAccount the workload runs as.** Stacklok Enterprise + doesn't create it. Note its namespace and name. + +In the console, select **Administration** under **Experience** in the account +menu, then open **Service Accounts**. + +## Trust the cluster's issuer + +Each cluster whose workloads call the gateways needs one trusted issuer. You can +also add an issuer from the **Bindings** step when you create an account. + +1. Open the **Trusted issuers** tab and select **Add issuer**. +2. Choose where the issuer is: + - **This cluster** trusts the workloads in the cluster that runs Stacklok + Enterprise. Choose it only when every gateway also runs in this cluster. + Each gateway reads the issuer's keys from the Kubernetes API of its own + cluster, so a gateway in another cluster refuses this cluster's workloads. + When the two clusters share an issuer URL, that gateway also accepts its + own cluster's tokens under this issuer. For **Issuer URL**, enter the + issuer this command prints, which on managed clusters such as Amazon EKS is + a public URL: + + ```bash + kubectl get --raw /.well-known/openid-configuration | jq -r .issuer + ``` + + Only one **This cluster** issuer can exist. Once it does, the option is + marked **Already trusted**. + + - **Another issuer** trusts a cluster by its public **Issuer URL** and its + public keys, given as a **JWKS URL** or with **Upload JWKS**. Use it for + every other cluster, and also for the cluster that runs Stacklok Enterprise + when gateways run in more than one cluster. + + The directory holds one trusted issuer per issuer URL, so each cluster needs + a distinct issuer URL. Amazon EKS and Google Kubernetes Engine clusters + already have one. For a self-managed cluster that keeps a default issuer URL, + add a distinct value as the first `--service-account-issuer` flag of its API + server and keep the old value after it, so tokens that pods already hold stay + valid. + +3. Enter an **Audience** that only Stacklok Enterprise accepts, such as + `stacklok-enterprise-platform`. Workloads request their token for this + audience, so share it with the teams that deploy them. +4. Select **Add issuer**. + +Both gateways pick up a new or changed issuer on their next poll of the +directory, every 30 seconds by default. + +The default **Signing algorithms**, under **Advanced**, accept `RS256`. Change +them only when the cluster signs its tokens with another type of key. + +Disabling an issuer stops its workloads from signing in and keeps their +bindings, so enabling it again lets them back in. You can delete an issuer only +when no binding uses it. + +### Rotate an issuer's signing keys + +For an issuer with a JWKS URL or an uploaded JWKS, rotate its keys in this order +so both gateways keep accepting the workloads' tokens. The cluster's +administrator changes the signing key, so plan these steps with them: + +1. Add the new public key to the document at the JWKS URL, or upload a JWKS that + holds both the new and the current key. +2. Wait at least 15 minutes before signing tokens with the new key. A gateway + that can't refresh an issuer's keys keeps using its cached copy for up to 15 + minutes. +3. Keep the retired key in the JWKS for at least 60 minutes after the last token + signed with it, the longest token lifetime the gateways accept. + +## Create a service account + +On the **Accounts** tab, select **New service account**. The wizard walks +through five steps, and every step after **Details** can be skipped and +completed later from the account's tabs. + +1. **Details**: enter a **Name** and a description that says which workload uses + the account. +2. **Connectors**: add the account to groups, or create one, and choose each + group's connectors. The account gets everything a group confers: its + connectors, budget pool, and model set grants. A change to a group's + connectors applies to every member of the group. The list leaves out groups + managed through SCIM and connectors that can't serve a service account. +3. **Model Sets**: grant model sets directly, and optionally set a **Budget** + amount and period. The **Budget** fields appear only when the AI Gateway is + enabled and its public URL is set. The **All Users** model set and the + organization default budget never apply to a service account. A workload that + calls the AI Gateway therefore needs model set grants and a budget from this + step or from its groups, unless the gateway turns those controls off, as + described in [What an account can reach](#what-an-account-can-reach). A + workload that only calls connectors needs neither. +4. **Bindings**: choose the **Issuer URL**, then enter the **Namespace** and + **ServiceAccount** name of the workload's ServiceAccount. To pin the binding + to that exact ServiceAccount, also enter its **UID**: + + ```bash + kubectl get serviceaccount -n \ + -o jsonpath='{.metadata.uid}' + ``` + +5. **Review**: check the summary and select **Create and activate service + account**. + +Without a binding, the button reads **Create service account** and the account +is created **Suspended**. A service account authenticates only while it's +**Active**, and activating it requires at least one binding to an enabled +issuer. Add a binding on the account's **Bindings** tab, then select +**Activate** in the alert on its page. + +An account's page opens on its **Model Sets** tab, followed by **Bindings**, +**Budgets** when the AI Gateway is enabled and its public URL is set, and +**Connectors**, where you complete any step you skipped. To rename the account +or change its description, select **Edit service account** in its actions menu. + +Once the account is active, the team that deploys the workload configures it as +described in +[Call the AI Gateway from Kubernetes workloads](../../ai-gateway/kubernetes-workloads.mdx) +and +[Call the Connector Gateway from Kubernetes workloads](../../connector-gateway/kubernetes-workloads.mdx). +One token serves both gateways. + +## Choose between a pinned and a name-only binding + +A name-only binding matches any ServiceAccount with its namespace and name. That +includes a ServiceAccount deleted and created again under the same name, and one +created by anyone who can create ServiceAccounts in that namespace, which the +built-in Kubernetes `edit` role allows. Pin the UID for any account whose grants +or budget you wouldn't give to everyone who can create a ServiceAccount with +that name in the namespace. + +A pinned binding still admits every pod that runs as its ServiceAccount, and +anyone who can create pods in the namespace, which the `edit` role also allows, +can run a pod as that ServiceAccount and get its token. For an account with +sensitive grants, also restrict who can create pods in that namespace. + +A pinned binding takes precedence over a name-only binding for its UID. When +another account already reaches the same ServiceAccount by name, the console +asks you to confirm with **Reassign**. The new account takes over the +ServiceAccount with that UID, and the other account keeps its name-only binding +for any other ServiceAccount with that name, such as one created again later. + +To change a binding, remove it from the account's **Bindings** tab and add a new +one. Each change applies on the workload's next request. + +## Revoke a workload's access + +To stop a workload at once, select **Suspend** in its account's actions menu and +confirm. The gateways refuse the workload's next request, and requests already +admitted finish. + +Removing the account's binding also stops the workload, unless another account +has a name-only binding for the same ServiceAccount, as after a **Reassign**. +That account then takes the workload over on its next request, so remove its +binding too, or suspend the account instead. + +Deleting the Kubernetes ServiceAccount doesn't revoke access right away. The +gateways verify tokens by their signature, so tokens already issued stay valid +until they expire, up to 60 minutes. + +An account that loses its last binding, or whose issuers are all disabled, stays +**Active** but its workloads are refused. Its page shows **Active, but its +workloads can't authenticate** with a link to fix it. + +Deleting a service account is permanent, so the console asks you to type the +account's name to confirm. It removes the account's bindings, group memberships, +and model set grants, and deletes its own budget. The account is kept with the +status **Deleted**, so its past spend remains attributed to it. The **Accounts** +tab shows deleted accounts when you filter **Status** by **Deleted**. + +## What an account can reach + +A trusted issuer accepts tokens from every namespace in its cluster, and the +gateways refuse any workload without an active, bound account. + +An account reaches a connector only through a group the connector is granted to, +and only when the connector uses **None**, **Bearer token**, or **Header +injection** authentication. The other +[authentication types](../../connector-gateway/connector-authentication.mdx#choose-an-authentication-type) +act as the calling person, and a service account has no person behind it. An +account has no connections to make, so the gateway serves it every connector it +reaches. The account's **Connectors** tab lists the connectors granted to its +groups and the group each one comes through, marks the ones it can't use as +**Not available to service accounts**, and lets you change its groups and their +connectors. Connectors governed by a Cedar policy aren't listed there, and a +Cedar policy with no condition on the principal admits every active account, +even one in no group. See +[Cedar-mode connectors](../../connector-gateway/connector-policies.mdx#cedar-mode-connectors). +A connector's **Activity** tab shows the account's tool calls under its name, +with a **Service account** badge. + +The AI Gateway's model access and budget settings decide which models the +account can call: + +| Model access | Budgets | An active, bound account can call | +| ------------ | ------- | ------------------------------------------ | +| On | On | The models its grants cover, within budget | +| On | Off | The models its grants cover, unmetered | +| Off | On | Any model, within budget | +| Off | Off | Any model, unmetered | + +Users get the same treatment, so a gateway with neither model access nor budgets +admits every bound workload as freely as every signed-in user. A cluster +administrator sets both in the gateway's chart values. See +[Control model access](../../ai-gateway/control-model-access.mdx) and +[Enable the AI Gateway](../enterprise-platform/configure-ai-gateway.mdx#enable-it). + +## Next steps + +- [Call the AI Gateway from Kubernetes workloads](../../ai-gateway/kubernetes-workloads.mdx) + to configure the workload's token and requests. +- [Call the Connector Gateway from Kubernetes workloads](../../connector-gateway/kubernetes-workloads.mdx) + to connect the workload to its connectors. +- [Manage budgets](../../ai-gateway/manage-budgets.mdx) to give an account its + own budget. + +## Troubleshooting + +
+Activate is unavailable + +The account has no binding, or every issuer its bindings use is disabled. Add a +binding on the account's **Bindings** tab, or enable an issuer on the **Trusted +issuers** tab. + +
+ +
+A workload is refused with 401 or 403 + +See Handle errors for the +[AI Gateway](../../ai-gateway/kubernetes-workloads.mdx#handle-errors) or the +[Connector Gateway](../../connector-gateway/kubernetes-workloads.mdx#handle-errors) +for what each response means and where to find the reason. + +
diff --git a/docs/platform/enterprise-directory/users-and-groups.mdx b/docs/platform/enterprise-directory/users-and-groups.mdx index bebf7469..983f71c0 100644 --- a/docs/platform/enterprise-directory/users-and-groups.mdx +++ b/docs/platform/enterprise-directory/users-and-groups.mdx @@ -41,6 +41,13 @@ organizational groups. To maintain groups from values in SCIM user profiles, see [Create groups from user attributes](./scim-provisioning.mdx#create-groups-from-user-attributes). +A group created in the console or API can also hold +[service accounts](./service-accounts.mdx). Add them from the group's **Service +accounts** tab with **Add service account**. A service account gets everything +the group confers, including through parent groups. Groups your identity +provider manages through SCIM take no service accounts, because SCIM controls +their membership. + ## Subgroups and inherited membership A group can contain other groups, listed on its **Subgroups** tab. Membership is @@ -65,8 +72,9 @@ The directory returns direct and inherited membership to downstream components. | --------------------- | --------------------------------------------------------------------------------- | | Access to a connector | What that group's members can reach through the Connector Gateway | | A budget | What that group's members may spend, once their own budget is absent or exhausted | +| A model set | Which models that group's members can call through the AI Gateway | -Both follow inherited membership. +All three follow inherited membership. :::warning diff --git a/docs/platform/enterprise-directory/virtual-api-keys.mdx b/docs/platform/enterprise-directory/virtual-api-keys.mdx index b88950d0..57acfd90 100644 --- a/docs/platform/enterprise-directory/virtual-api-keys.mdx +++ b/docs/platform/enterprise-directory/virtual-api-keys.mdx @@ -11,6 +11,11 @@ clients that cannot complete an interactive sign-in. Each key is bound to its owner's directory identity, so requests retain the owner's access, budget, and audit attribution. +For an agent or job running in Kubernetes, use a +[service account](./service-accounts.mdx) instead. It gives the workload its own +identity, access, and budget, and authenticates with the pod's ServiceAccount +token, so there's no key to store. + ## Where to find them Users manage their own keys in the console's user experience. Open the account diff --git a/docs/platform/enterprise-platform/configure-ai-gateway.mdx b/docs/platform/enterprise-platform/configure-ai-gateway.mdx index b074d92f..d2a46b39 100644 --- a/docs/platform/enterprise-platform/configure-ai-gateway.mdx +++ b/docs/platform/enterprise-platform/configure-ai-gateway.mdx @@ -143,7 +143,9 @@ Complete the following sequence before sending production traffic: is the `host:port` of the directory's gRPC listener, without a scheme. The gateway connects to the cleartext listener on port `9091`, so set `directory.insecure: true`. It still authenticates every call with its - projected ServiceAccount token. + projected ServiceAccount token. To accept service account workloads, connect + to the TLS listener instead, as described in + [Accept service account workloads](#accept-service-account-workloads). Set `directory.groupsResolutionIssuer` to the issuer the directory knows your users under, usually your corporate identity provider's issuer. Requests that @@ -313,6 +315,114 @@ ai-gateway-v2: A missing egress rule fails closed: governance calls can't complete, so every request is refused with `503` or `504`. +## Accept service account workloads + +Kubernetes workloads call the gateway as +[service accounts](../enterprise-directory/service-accounts.mdx), using trusted +issuers that an administrator adds in the console. Once connected to the +directory, the gateway polls it for those issuers every 30 seconds, set by +`directory.workloadTrust.pollInterval`, and fetches each issuer's public keys. +Complete these steps before the first workload calls the gateway: + +1. **Connect to the directory over TLS.** The directory serves trusted issuers + only on the Enterprise Manager's TLS gRPC port (`9443`). Over the cleartext + listener, the gateway installs no issuer and refuses every workload with + `401`, while users and virtual API keys keep working. Turn on the Enterprise + Manager's TLS listener, as described in + [Encrypting stored credentials](./deployment.mdx#encrypting-stored-credentials), + and point the gateway at it: + + ```yaml title="values.yaml" + enterprise-manager: + grpc: + tls: + enabled: true + existingSecret: 'enterprise-manager-grpc-tls' + + ai-gateway-v2: + directory: + endpoint: '..svc.cluster.local:9443' + insecure: false + caBundle: + secretName: 'enterprise-manager-grpc-ca' + ``` + + Use a host name that the Enterprise Manager's gRPC certificate covers. A + certificate that cert-manager issues through + `enterprise-manager.grpc.tls.issuerRef` covers every form of the Service + name. + + `directory.caBundle.secretName` names a Secret in the gateway's namespace + that holds only the CA certificate, under the key `ca.crt`. If cert-manager + issues the Enterprise Manager's certificate, copy `ca.crt` out of its Secret + rather than referencing that Secret, which also holds the private key: + + ```bash + kubectl create secret generic enterprise-manager-grpc-ca \ + --namespace \ + --from-file=ca.crt=./ca.crt + ``` + + A copied Secret goes stale when cert-manager rotates the CA, so update it + then, or keep it in sync automatically, for example with a trust-manager + `Bundle` that targets a Secret. The gateway reads the Secret again at every + reconnect, so an updated CA needs no restart. + +2. **Keep the gateway on the issuer allowlist.** The platform chart adds the + gateway's ServiceAccount to + `enterprise-manager.grpc.callerAuth.workloadTrustSaSubjectAllowlist`. If you + set that list yourself, include the gateway's subject, by default + `system:serviceaccount::-ai-gateway-v2`, so the + gateway can fetch the issuers. Your list replaces the chart's, so also + include the Connector Gateway's subject when it + [accepts service account workloads](./configure-connector-gateway.mdx#accept-service-account-workloads). + +3. **Confirm the gateway can read this cluster's keys.** For an issuer of the + cluster the gateway runs in, the gateway reads keys from the Kubernetes API. + The chart binds the `system:service-account-issuer-discovery` ClusterRole to + the gateway's ServiceAccount for that read, which most distributions grant to + no one by default. Keep that binding if a policy tool reviews + ClusterRoleBindings. + +4. **Allow egress to the key sources** when `networkPolicy.allowExternalEgress` + is `false`. The cluster's own keys come from the API server's endpoint + addresses, often on port `6443`. List them: + + ```bash + kubectl get endpointslices -n default \ + -l kubernetes.io/service-name=kubernetes + ``` + + Then add a rule for each address, with `/128` in place of `/32` for an IPv6 + address: + + ```yaml title="values.yaml" + ai-gateway-v2: + networkPolicy: + egress: + - to: + - ipBlock: + cidr: /32 + ports: + - protocol: TCP + port: 6443 + ``` + + Admitting the `kubernetes.default` Service address on port `443` isn't + enough, because most network plugins evaluate egress after rewriting that + address to an endpoint. An issuer with a JWKS URL needs its host on port + `443`, which the provider rule already admits. A missing rule doesn't affect + readiness or users' traffic. The gateway logs + `ai-gateway-v2: workload trust dropped` and refuses that issuer's workloads. + +To choose an issuer type, see +[Trust the cluster's issuer](../enterprise-directory/service-accounts.mdx#trust-the-clusters-issuer). +To change an issuer's keys, see +[Rotate an issuer's signing keys](../enterprise-directory/service-accounts.mdx#rotate-an-issuers-signing-keys). + +To configure a workload, see +[Call the AI Gateway from Kubernetes workloads](../../ai-gateway/kubernetes-workloads.mdx). + ## Disconnect the gateway An administrator can stop all model traffic without a redeploy by setting the AI diff --git a/docs/platform/enterprise-platform/configure-connector-gateway.mdx b/docs/platform/enterprise-platform/configure-connector-gateway.mdx index faaabd30..71e6c74f 100644 --- a/docs/platform/enterprise-platform/configure-connector-gateway.mdx +++ b/docs/platform/enterprise-platform/configure-connector-gateway.mdx @@ -531,6 +531,62 @@ connector grants. Each connector's [connector policy](../../connector-gateway/connector-policies.mdx) determines which callers the gateway exposes it to. +## Accept service account workloads + +Kubernetes workloads call the gateway's MCP endpoint as +[service accounts](../enterprise-directory/service-accounts.mdx), using the same +trusted issuers and accounts as the AI Gateway. The chart turns this on whenever +the gateway connects to the Enterprise Manager over TLS, as described in +[Connect the gateway to the Enterprise Manager](#connect-the-gateway-to-the-enterprise-manager). +It then: + +- Binds the `system:service-account-issuer-discovery` ClusterRole to the + gateway's ServiceAccount and mounts an API server token, so the gateway can + read the signing keys of an issuer added with **This cluster**. +- Adds the gateway's ServiceAccount to + `enterprise-manager.grpc.callerAuth.workloadTrustSaSubjectAllowlist`, so the + gateway can fetch the trusted issuers. If you set that list yourself, include + `system:serviceaccount::-connector-gateway` alongside + the AI Gateway's entry. + +The gateway polls the directory for trusted issuers every 30 seconds, set by +`connector-gateway.enterpriseConfig.workloadIdentity.pollInterval`. A workload +authenticates once an administrator adds its cluster's issuer and binds an +active service account to its ServiceAccount. + +Both gateways share trusted issuers and service accounts, so an account bound +for the AI Gateway can also reach the connectors its groups are granted, as +described in +[What an account can reach](../enterprise-directory/service-accounts.mdx#what-an-account-can-reach). +Review those groups' connector grants. To refuse workload tokens on this +gateway, set: + +```yaml title="values.yaml" +global: + stacklok: + connectorGateway: + workloadIdentity: + enabled: false +``` + +When `connector-gateway.networkPolicy.enabled` is `true` and +`connector-gateway.networkPolicy.allowExternalEgress` is `false`, add rules to +`connector-gateway.networkPolicy.egress` for the key sources: the API server's +endpoint addresses for an issuer added with **This cluster**, and the host of +each JWKS URL on port `443`. List the API server's addresses and write the rule +as shown for the AI Gateway in +[Accept service account workloads](./configure-ai-gateway.mdx#accept-service-account-workloads). +Without these rules, the gateway logs `workload trust dropped` and refuses every +workload token with `401`. + +To choose an issuer type, see +[Trust the cluster's issuer](../enterprise-directory/service-accounts.mdx#trust-the-clusters-issuer). +To change an issuer's keys, see +[Rotate an issuer's signing keys](../enterprise-directory/service-accounts.mdx#rotate-an-issuers-signing-keys). + +To configure a workload, see +[Call the Connector Gateway from Kubernetes workloads](../../connector-gateway/kubernetes-workloads.mdx). + ## Turn on tool call recording and observability Tool call recording, which populates the console's **Tool Usage** screen, is off @@ -591,6 +647,8 @@ together. This transport carries credentials in cleartext with no caller authentication, so reserve it for local testing. The Enterprise Manager refuses to start with `grpc.insecure: true` while `directory.kek` is configured, so this transport works only on an install without the key-encryption key. +[Service account workloads](#accept-service-account-workloads) need the TLS +transport. ### Override the gateway's name @@ -604,6 +662,7 @@ these values to match: | ------------------------------------------------------------------------------------------------------------------------------------------------------ | -------------------------------------------------------------------- | | `enterprise-manager.grpc.callerAuth.saSubjectAllowlist`, `gatewaySaSubjectAllowlist`, `credentialSaSubjectAllowlist`, and `meteringSaSubjectAllowlist` | `system:serviceaccount::` | | `enterprise-manager.grpc.callerAuth.httpSaSubjectAllowlist`, only when `global.stacklok.enterpriseManager.url` is set | The same subject, alongside any AI Gateway entry already on the list | +| `enterprise-manager.grpc.callerAuth.workloadTrustSaSubjectAllowlist`, unless workload tokens are turned off | The same subject, alongside the AI Gateway's entry | | `toolhive-cloud-ui.connectorGatewayUrl`, which enables the console's **Connectors** pages | `http://.:` | | `enterprise-manager.grpc.networkPolicy.allowedClientSelector`, only with `nameOverride` | `app.kubernetes.io/name: ` | @@ -715,11 +774,12 @@ the client registered with `openid` and then requested `offline_access`. Set
The render fails or the gateway is refused after a name override -The umbrella chart checks `meteringSaSubjectAllowlist` and -`toolhive-cloud-ui.connectorGatewayUrl` against the gateway's actual names and -fails the render when they differ. The error message prints the value to set. No -check covers `httpSaSubjectAllowlist`, so a stale entry there shows up only as -refused HTTP calls from the gateway to the Enterprise Manager. See +While `meteringSaSubjectAllowlist`, `workloadTrustSaSubjectAllowlist`, and +`toolhive-cloud-ui.connectorGatewayUrl` keep their computed defaults, the +umbrella chart checks them against the gateway's actual names and fails the +render when they differ. The error message prints the value to set. No check +covers `httpSaSubjectAllowlist`, so a stale entry there shows up only as refused +HTTP calls from the gateway to the Enterprise Manager. See [Override the gateway's name](#override-the-gateways-name).
diff --git a/docs/platform/enterprise-platform/roll-out-gateway-clients.mdx b/docs/platform/enterprise-platform/roll-out-gateway-clients.mdx index 4f07611b..455c56b0 100644 --- a/docs/platform/enterprise-platform/roll-out-gateway-clients.mdx +++ b/docs/platform/enterprise-platform/roll-out-gateway-clients.mdx @@ -51,7 +51,12 @@ use. caller through your corporate identity provider. - Use a [virtual API key](../enterprise-directory/virtual-api-keys.mdx) for scripts, scheduled jobs, continuous integration, and other clients without a - browser flow. Requests remain attributed to the key owner. + browser flow that act as a person. Requests remain attributed to the key + owner. +- Use a [service account](../enterprise-directory/service-accounts.mdx) for + agents and jobs running in Kubernetes that act on their own. The workload + authenticates with its pod's ServiceAccount token, and its requests are + attributed to the account. ## Point users to the console diff --git a/sidebars.ts b/sidebars.ts index 0fa8baf3..bf5a05fc 100644 --- a/sidebars.ts +++ b/sidebars.ts @@ -433,6 +433,7 @@ const platformSidebar: SidebarsConfig[string] = [ 'platform/enterprise-directory/users-and-groups', 'platform/enterprise-directory/scim-provisioning', 'platform/enterprise-directory/okta-directory-sync', + 'platform/enterprise-directory/service-accounts', 'platform/enterprise-directory/virtual-api-keys', ], }, @@ -528,6 +529,11 @@ const connectorGatewaySidebar: SidebarsConfig[string] = [ defaultStyle: false, }, 'connector-gateway/support-connector-access', + { + type: 'doc', + id: 'connector-gateway/kubernetes-workloads', + label: 'Kubernetes workloads', + }, { type: 'html', @@ -572,6 +578,11 @@ const aiGatewaySidebar: SidebarsConfig[string] = [ }, { type: 'doc', id: 'ai-gateway/manage-budgets', label: 'Manage budgets' }, 'ai-gateway/budgets-and-pricing', + { + type: 'doc', + id: 'ai-gateway/kubernetes-workloads', + label: 'Kubernetes workloads', + }, { type: 'doc', id: 'ai-gateway/prompt-injection-screening',