docs(ax-audit): native-runtime HTTP 401 named the model, not the guardrail (entry 86)

docs/development/agent-experience-audit.md

@@ -4220,3 +4220,13 @@ Measured with `createParser(catalogue).parse(source)`: a `Text` body of 601 char
 **Repair:** inspect every statement/call before accepting source, then validate the materialized element tree with a full JSON Schema validator. Element wrappers preserve component identity when following child/helper references. `backend/__tests__/unit/services/uiCardValidation.test.js` pins string/array/number limits, helper placement and cross-item rules as well as dormant unsafe calls.
 
 **Lesson:** accepting a schema as a parser input does not establish which schema constraints it enforces. Probe the actual boundary values before making the parser an authorization or resource-limit boundary.
+
+## 86. A native-runtime `HTTP 401` named the model, but the 401 came from a guardrail's upstream call (2026-10-31, sre)
+
+*Origin: the Scout-staging incident. Sam: "is scout not working? I saw it not responding in staging." Button clicks in the onboarding pod enqueued native-runtime turns that all failed; the scripted welcome still posted, so the pod looked half-alive. Fixed across #3077 (litellm was not deployed in `commonly-staging`) and #3079 (the guardrail below).*
+
+After #3077 brought litellm up healthy in staging, Scout turns still failed. The `AgentRun` carried `errorKind="llm_error"`, `errorMessage="Model deepseek-v4-flash: HTTP 401"` — which reads as a bad DeepSeek key or a model-routing problem. It was neither. `nativeRuntimeService.ts` opts every native turn into the guardrail pair `openai-moderation-enforce,injection-guard` (overridable via `NATIVE_RUNTIME_GUARDRAILS`). `openai-moderation-enforce` calls OpenAI's moderation API (`https://api.openai.com/v1/moderations`) with `OPENAI_API_KEY`, as a `during_call` check **before** the model call. Staging's `api-keys.openai-api-key` is an 11-byte placeholder (prod's was the literal `"placeholder"`), so the guardrail 401'd and litellm returned 401 for the whole `/chat/completions`. The decisive evidence was only in **litellm's own pod logs** — `Client error '401 Unauthorized' for url 'https://api.openai.com/v1/moderations'` — not in the AgentRun error, which attributed the 401 to `deepseek-v4-flash`. The DeepSeek key was valid (35 bytes) the whole time.
+
+**Repair:** per env without a real OpenAI moderation key, set `backend.env.nativeRuntimeGuardrails: "injection-guard"` (LiteLLM-native, no external key). Prod (`values-dev.yaml`) already carried this; staging was simply missing it — the same omission class as the missing litellm stanza. Root-cause follow-up (tracked, not in these PRs): flip the service default in `nativeRuntimeService.ts` to `injection-guard` so a new/unconfigured env does not inherit the keyless 401. The sibling `llmService` path is only safe via a Helm-env hardcode (`LLMSERVICE_GUARDRAILS: "injection-guard"`, `backend-deployment.yaml:573`) — `llmService.ts`'s own *code* default is the same unsafe pair, so the thing to copy is the deployment override, not the sibling's code default. Guard test: `backend/__tests__/unit/helm/litellmStagingEnabled.test.js`.
+
+**Lesson:** a native-runtime error names the model it was trying to reach, not the dependency that actually failed. A `"Model <X>: HTTP 401"` can be a pre-call guardrail's upstream 401 (a different vendor, a different key) — read litellm's own pod logs for the failing URL before concluding the model's key is wrong. And "litellm is up" is necessary, not sufficient, for a native agent to reply: the guardrail is an independent failure surface on the same turn.