[lilyshen0722]

Second half of the 2026-10-31 Scout-staging incident

Sam: "is scout not working? I saw it not responding in staging." (D20)

Two causes, two fixes. PR #3077 (merged) fixed the first — litellm was not deployed in commonly-staging, so native-runtime turns 401'd on a missing backend. After it deployed, litellm came up healthy (2/2 on default-pool) but Scout turns still failed:

AgentRun errorKind="llm_error" errorMessage="Model deepseek-v4-flash: HTTP 401"
litellm log: 401 Unauthorized for url 'https://api.openai.com/v1/moderations'

The native runtime's default guardrail pair (nativeRuntimeService.ts) includes openai-moderation-enforce, which calls OpenAI's moderation API with OPENAI_API_KEY. Staging's api-keys.openai-api-key is an 11-byte placeholder, so that guardrail returned 401 on every native turn, before the model call.

Fix — prod parity

Prod (values-dev.yaml:131) already sets backend.env.nativeRuntimeGuardrails: "injection-guard" for exactly this reason (its OpenAI key was the literal "placeholder"; the comment there notes it was "the probable cause of the native fleet's 98% July failure rate"). Staging was simply missing the override — the same omission class as the missing litellm stanza in #3077.

injection-guard is LiteLLM-native (no key, no external service) and keeps the prompt-injection block for untrusted input. Restore the pair only after a real OpenAI moderation key lands in the staging api-keys secret.

Scope / safety

• One values line under backend.env + a guard test. No template change — the env is already values-driven (backend-deployment.yaml:654).
• Prod render unchanged — values-dev.yaml already carries the identical value; this PR does not touch it.
• deepseek-api-key in staging is real (35 bytes), so the model call succeeds once the guardrail is out of the path — verified before shipping.
• Guard test asserts staging runs injection-guard only and stays in lockstep with prod.

🤖 Generated with Claude Code

https://claude.ai/code/session_01N4VWLErW29t4LnhRfUy1NH

[coderabbitai]

No actionable comments were generated in the recent review.

Recent review info

Run configuration

• Source mode: Auto
• Repository: Team-Commonly/commonly
• Branch: sre-staging-guardrails
• File: .coderabbit.yaml
• Commit: fa459dc
• Layers: Repository YAML
• Review profile: QUIET
• Plan: Advanced
• Run ID: fe1eea83-38f3-4d82-847f-4dca0849d4cb

Commits

Reviewing files that changed from the base of the PR and between 628051f and fa459dc.

Files selected for processing: 2

• backend/__tests__/unit/helm/litellmStagingEnabled.test.js
• k8s/helm/commonly/values-staging.yaml

Included review availability: This review used your included allowance. Your plan provides up to 10 included reviews per hour; 5 remain after this review.

Walkthrough

Staging now sets backend.env.nativeRuntimeGuardrails to injection-guard. A regression test checks that value and verifies it matches the dev value.

Changes: Staging Guardrail Configuration

Layer / File(s)	Summary
Configure and verify the staging guardrail k8s/helm/commonly/values-staging.yaml, backend/__tests__/unit/helm/litellmStagingEnabled.test.js	Staging sets nativeRuntimeGuardrails to injection-guard. The unit test checks this value and verifies that it matches dev.

Estimated code review effort: 2 (Simple) | ~8 minutes

Change: Bug fix

Possibly related issues

• Issue 541 addresses scoped prompt-injection guardrails. This change configures staging to use injection-guard.

Possibly related PRs

• Team-Commonly/commonly#3077: This PR changes the staging LiteLLM deployment and guard test, which are also part of the staging native-runtime path.
• Team-Commonly/commonly#2748: This PR changes LiteLLM guardrail-check key selection; this change configures which guardrail runs in staging.

Merge Risk: Minimal · up to fa459

The staging guardrail configuration is consistent with the intended native-runtime behavior and introduces no actionable merge risk.

Pre-merge checks: 4

Check name	Status	Explanation
Title check		The title names the staging change, the injection-guard scope, and the Scout fix, which match the changeset. It is a short, single sentence and does not use vague terms. It is specific enough that a t…
Description check		The description explains the incident, the placeholder OpenAI key that caused 401s from the openai-moderation-enforce guardrail, and the staging-only fix that sets nativeRuntimeGuardrails to injection…
Linked Issues check		Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check		Check skipped because no linked issues were found for this pull request.

Full details: Title check

Explanation

The title names the staging change, the injection-guard scope, and the Scout fix, which match the changeset. It is a short, single sentence and does not use vague terms. It is specific enough that a teammate scanning history understands the primary change.

Full details: Description check

Explanation

The description explains the incident, the placeholder OpenAI key that caused 401s from the openai-moderation-enforce guardrail, and the staging-only fix that sets nativeRuntimeGuardrails to injection-guard. It also covers prod parity and the guard test. It is directly related to the changeset.

Finishing Touches

Generate docstrings

• Commit to this branch
• Create a new PR

Generate unit tests (beta)

• Commit to this branch
• Create a new PR

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

[lilyshen0722]

Writer's-subagent review (D218, medium): done. It verified the mechanism end-to-end — nativeRuntimeGuardrails → NATIVE_RUNTIME_GUARDRAILS (backend-deployment.yaml:654) → parsed at nativeRuntimeService.ts:115, injection-guard exists in litellm-config.yaml, values-dev carries the identical override, YAML parses, assertions hold. The change itself is correct.

Two findings, both latent/pre-existing and deliberately not folded into this targeted P0 fix:

1. Service default still includes openai-moderation-enforce — the real root-cause fix is to flip the code default in nativeRuntimeService.ts:116 to injection-guard so new/unconfigured envs don't inherit the keyless 401 (prod already hardcodes LLMSERVICE_GUARDRAILS: injection-guard for the sibling path). That's a code change affecting all envs and needs its own PR + review — out of scope for a staging values fix.
2. Empty-string off-switch unreachable via Helm — {{- if ...nativeRuntimeGuardrails }} treats "" as falsy, so nativeRuntimeGuardrails: "" is omitted and falls back to the default pair. Pre-existing template behavior this PR doesn't touch; no caller sets "" today.

Both tracked as SRE follow-ups. This PR stays minimal: one values line (prod parity) + a guard test.

GATE: PASS

[lilyshen0722]

Writer's-subagent review (D218, medium): done — mechanism verified end-to-end (nativeRuntimeGuardrails → NATIVE_RUNTIME_GUARDRAILS → nativeRuntimeService.ts:115; injection-guard exists in litellm-config; prod carries the identical override; YAML parses; assertions hold). One values line (prod parity) + a guard test. Two findings deferred as pre-existing/broader-scope (documented in the PR comment and tracked as SRE follow-ups): flip the service default to injection-guard (code, all envs), and the empty-string Helm off-switch guard. Neither is a defect in this change.

GATE: PASS