Repository navigation
Conversation
|
Important Draft PR not reviewedDraft PRs are not automatically reviewed by default.
To automatically review draft PRs, update your CodeRabbit configuration: reviews:
auto_review:
drafts: true
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. |
This was referenced Nov 13, 2026
knowlet
force-pushed
the
fix/llm-provider-configuration
branch
from
November 14, 2026 15:08
bb62224 to
7f03fe3
Compare
Preserve CORE/Ollama defaults while resolving provider-native gateway credentials and reloading changed configuration during upgrades. Bind each tenant endpoint to its credential source across runtime consumers, probes and status, with explicit provider model guidance. Combine the related AI configuration fixes on the current upstream base. Keep demo seeding and investigation-ledger repairs outside this change. Retain real gateway, PostgreSQL, provider-HTTP and Settings browser coverage. Signed-off-by: Albert Hsieh <knowlet@pm.me>
Keep the gateway state helper's timeout path explicitly terminating. Preserve the existing error message and failure semantics. Signed-off-by: Albert Hsieh <knowlet@pm.me>
knowlet
force-pushed
the
fix/llm-provider-configuration
branch
from
November 17, 2026 02:02
7f03fe3 to
8211548
Compare
9 of 17 tasks
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Native Amazon Bedrock deployments could fail despite a correctly configured host
.env: Compose did not forward the AWS settings, every gateway alias explicitly suppliedOPENAI_API_KEY, and an empty endpoint could override the provider's default; upgrades could also leave LiteLLM running with its previous startup configuration. This PR fixes AiSOC's integration with LiteLLM's existing native Bedrock support so operators can use a Bedrock bearer token, AWS environment credentials, an existing host profile through an optional read-only mount, or an EC2 instance role through the bundled gateway, while preserving the default CORE/Ollama install and providing an explicit recreate path for configuration changes. It also resolves tenant BYOK endpoint, model and credential choices together for Explain, automatic triage, connection probes and status, while keeping native AWS authentication in the deployment gateway.Type of change
Related issues
Contribution guidance: beenuar#353. No issue is auto-closed.
Supersedes the fork's gateway draft #7, BYOK draft #8, and gateway validation draft #9. Those drafts are closed; their branches remain available.
Changes
Bedrock support through the deployment gateway
This fixes AiSOC's Compose integration with LiteLLM's existing Bedrock implementation for the authentication paths below. The nine task aliases remain intact: operators select accessible
bedrock/<model-or-inference-profile-id>values withAISOC_LLM_MODEL_FASTandAISOC_LLM_MODEL_DEEP, set the intended AWS region, and choose an authentication path below. No per-alias credential edit or AWS credential inOPENAI_API_KEYis required.AWS_BEARER_TOKEN_BEDROCKto LiteLLM; remove generic alias-levelapi_keybindings that could override native bearer resolution.AWS_ACCESS_KEY_ID,AWS_SECRET_ACCESS_KEYandAWS_SESSION_TOKENfor LiteLLM's native signing path.infra/compose/docker-compose.aws-profile.ymloverlay, mounting the host AWS directory read-only into LiteLLM only; forward profile and credential-file settings.AWS_PROFILEas needed and use the same overlay for startup, upgrade and reload. Docker does not inherit the host's~/.awsautomatically; this avoids copying existing profile secrets into AiSOC's YAML.AWS_EC2_METADATA_DISABLEDmust remain false.Metadata control is independent of bearer/profile authentication.
AWS_EC2_METADATA_DISABLED=truedisables metadata lookup, while environment and profile credentials remain available. The pinned-image fixtures cover a mounted profile with metadata disabled, an instance role obtained through IMDSv2, and a bearer that skips IMDS even when metadata is enabled. To intentionally switch from bearer to IAM, remove the bearer and recreate the gateway.An explicitly blank endpoint now means provider default. Set
AISOC_LLM_API_BASE=when moving to Bedrock's regional endpoint, or supply an explicit Bedrock runtime URL. The startup wrapper unsets that blank value before LiteLLM starts. Leaving the variable entirely unconfigured retains Compose's existing Ollama default, preserving the account-free CORE install. Blank optional AWS/profile/region values are likewise removed; an explicit region must still agree with the intended deployment, and should be cleared when deliberately relying on the profile's region.Configuration changes have an explicit application step.
make llm-reloadrecreates only LiteLLM with its current environment and mounts, andmake upgradenow invokes that same step. The target performs recreation; the documentation and regression tests separately check the actual mounted/etc/litellm/config.yamland loaded model. A changed host file or successful liveness check alone does not establish that the new routing is active.Credential boundaries stay separate.
LITELLM_MASTER_KEYprotects access to the gateway; AWS credentials authenticate LiteLLM to Bedrock. Tenant BYOK is a separate override for the supported consumers listed below. The patch does not add a native Bedrock provider or SigV4 implementation to the settings UI.The profile overlay exposes existing files; it does not implement an in-container AWS sign-in or refresh flow. SSO sessions must be refreshed on the host, and external credential helpers still need to be available in the container. The profile fixture covers shared credentials/config files, not end-to-end SSO renewal.
Authentication selection is also separate from AWS authorization. A usable bearer, profile or role still needs access to the chosen model and any applicable Marketplace subscription. Switching identities can change permissions; a successful bearer request does not establish that the original IAM role is authorized. The fixtures validate credential selection and protocol behavior without contacting AWS; live account/model access remains unverified.
The gateway also preserves OpenAI/Anthropic keyless compatible endpoints by normalizing existing whitespace-only provider keys to an explicit empty string, without creating missing keys. The obsolete
--profile fullguidance is corrected, and the production overlay inherits the same startup wrapper.Tenant BYOK
aisoc-no-authplaceholder.How this relates to Bedrock in the UI: leave BYOK disabled when a tenant should inherit the bundled gateway's Bedrock configuration. A tenant can also select
local-litellmorcustomwith a gateway's OpenAI-compatible URL, served model/alias and gateway key; a raw native Bedrock runtime URL is not that API.For models supporting AWS's Chat Completions API, the existing
customflow can instead be configured with the regional/openai/v1base, an explicit compatible model ID and a Bedrock API key. This is a documented protocol-compatible path requiring live acceptance in the target account, not a new native AWS client or a claim of live validation. AWS's model/API compatibility table does not list Anthropic models as Chat Completions compatible; those native Bedrock deployments use the gateway path. The direct custom flow does not obtain IAM-role credentials or sign SigV4 requests.Scope: gateway runtime changes cover Compose and its production overlay; Helm runtime configuration is unchanged. No native Bedrock UI provider or SigV4 client was added. Other AI workloads retain deployment routing. Demo seeding, case UUID, investigation-ledger and replay repairs were excluded from this consolidation.
Testing
This is the single consolidated validation draft in the contributor fork, intended for
beenuar/AiSOC:main. The rebased branch contains 2 commits, 45 files (+3114/-324): exact three-dot diff.Current head:
82115480b66cacb26eb232e717a8d6da5aa54481Current upstream base:
2a0d550165b9f82108a3fe1c487c44aa96ed1b24Verified source tree:
11ebda832dc7d2e4753ea9447fec714fc4d5de7dConflict resolution and current-head verification — 2026-11-17
Upstream advanced by 59 commits since the previous feature base. Rebase both feature commits onto that main. The only manual conflict was in
services/agents/app/llm/factory.py: retain upstream'sAISOC_LLM_THINKINGoption andextra_bodycomposition together with the existing direct-Anthropic JSON/temperature/token guards. No additional provider behavior was introduced to resolve the conflict.Independent review confirms that all 38 non-overlapping feature files are byte-identical to the previous published head
7f03fe3a; six auto-merged overlapping files exactly match independently recomputed three-way unions. The seventh overlapping file is the factory conflict above. Upstream's Settings API-key scopes (cases:read,alerts:read), Compose fusion/UEBA/Cyble/Prometheus wiring, Helm action pins and new case-resolution OpenAPI fields are preserved. The second commit is patch-identical, and both commits retain their DCO sign-offs. GitHub reports the branch mergeable against this upstream base.Local checks ran on the rebased source tree, then the published commits were fetched and verified to have that exact tree. Commit IDs differ from the local rebase because publication recreated the commit metadata; tested source contents are identical.
scripts/export_openapi.py --checkpasses.extra_body. Uses the actual ChatOpenAI/OpenAI SDK withhttpx.MockTransport; confirms URL/model/key isolation and parameter composition. This is not live-provider acceptance evidence.bash -n. Workflow concurrency, required-check substance (static), merge-marker and whitespace checks pass.Python tests used separate Python 3.11 environments installed exclusively from each service's generated locked requirements with
--no-deps. Inherited proxy variables were removed only for the local test subprocesses; dependencies and fixtures were not changed. Existing deprecation/instrumentation warnings are not hidden. Full Python suites and Docker-backed acceptance were not rerun locally.Fresh CI for
82115480At 2026-11-17 10:04 Taiwan time (UTC+8), an exact-head snapshot shows 19 successful workflow runs, 13 running and 35 queued; no failed checks observed. OpenAPI schema and breaking-change checks have passed. Full CI is not yet complete.
The links above track final results; the passing historical counts below must not be read as results for this new head.
Local Docker is unavailable, so fresh gateway-image HTTP/Compose reload, live PostgreSQL BYOK, full Compose smoke and real-login Settings browser evidence must come from this head's CI. No live AWS account/model acceptance is claimed.
Historical CI at
7f03fe3a— completed, not evidence for the new headAn exact-
head_shaaudit found 68/68 workflow runs completed successfully. Main CI completed on 2026-11-15 00:33 Taiwan time (UTC+8). These results replace the stale “31/32 jobs, Python still running” snapshot; they precede the current upstream rebase.The intentional pytest failures inside successful negative-control steps demonstrate that the restored bugs are detected; they are not failed CI jobs.
Screenshots (if UI changes)
Historical
7f03fe3areal-login Settings browser artifact contains the OpenAI and local-Ollama model-guidance screenshots. It was available and unexpired when checked on 2026-11-17. This browser case covers login and guidance; provider save/probe transport is covered separately. No new-head manual browser inspection or live AWS UI test is claimed.Checklist
ruff,eslint,gofmt)Eval harness (required for substrate / playbook / detection changes)
Ran all suites against base
2a0d550165b9f82108a3fe1c487c44aa96ed1b24and the exact source tree published as82115480b66cacb26eb232e717a8d6da5aa54481, using the same locked agents environment on Python 3.11.18 withPYTHONHASHSEED=0:Both commands exited 0. All 11 axes passed, and every non-timing suite result is identical. The following are the complete suite blocks from each saved JSON report, with dataset and pass status; the per-investigation telemetry section is omitted. Timings naturally differ. These are deterministic evaluations against synthetic substrate fixtures, not live LLM or AWS performance measurements.
The pre-existing failing MITRE templates remain
outlook-auto-forward-ruleandcompromised-ci-runner; they do not represent a new regression. Macro accuracy/completeness/response scores remain 0.9636 / 0.9429 / 1.0000.Before — upstream 2a0d550
{ "generated_at": "2026-11-17T02:00:11.632228+00:00", "dataset": "synthetic_incidents.json (200 cases, deterministic)", "suite_filter": "all", "suites": { "mitre_accuracy": { "metric": "accuracy", "value": 0.97, "target": 0.8, "passed": true, "duration_ms": 6.3, "details": { "incidents": 200, "correct": 194, "precision": 0.7058, "recall": 0.85, "f1": 0.7712 }, "per_template": { "metric": "macro_accuracy", "value": 0.9636, "target": 0.8, "passed": true, "template_count": 55, "failing_templates": [ "outlook-auto-forward-rule", "compromised-ci-runner" ] } }, "alert_reduction": { "metric": "reduction_ratio", "value": 0.753, "target": 0.7, "passed": true, "duration_ms": 13.0, "details": { "alerts_in": 1000, "incidents_out": 247, "reduction_pct": "75.3%", "storm_incidents": 16 } }, "investigation_completeness": { "metric": "mean_keyword_coverage", "value": 0.9425, "target": 0.85, "passed": true, "duration_ms": 4.6, "details": { "incidents": 200, "fully_covered": 134, "fully_covered_pct": 0.67 }, "per_template": { "metric": "macro_completeness", "value": 0.9429, "target": 0.8, "passed": true, "template_count": 55, "failing_templates": [] } }, "response_quality": { "metric": "mean_rubric_score", "value": 1.0, "target": 0.8, "passed": true, "duration_ms": 2.1, "details": { "incidents": 200, "criteria": { "action_aligned": 1.0, "severity_aware": 1.0, "mitre_aligned": 1.0, "evidence_grounded": 1.0, "actionable": 1.0 } }, "per_template": { "metric": "macro_score", "value": 1.0, "target": 0.75, "passed": true, "template_count": 55, "failing_templates": [] } }, "hunt_corpus": { "metric": "positive_scenario_catch_rate", "value": 1.0, "target": 1.0, "passed": true, "duration_ms": 129.3, "details": { "hunts": 68, "positives_expected": 68, "positives_caught": 68, "negatives_expected": 68, "false_positives": 0, "false_positive_rate": 0.0, "negative_ceiling": 0.0, "orphan_incident_ids": [], "misses": [], "false_positive_details": [], "positive_pass": true, "negative_pass": true, "no_orphans": true } }, "adversary_eval": { "metric": "graceful_degradation_catch_rate", "value": 0.475, "target": 0.4, "passed": true, "duration_ms": 8.2, "details": { "incidents": 200, "correct": 95, "lost_all_tactics": 105, "buckets": { "heavy": { "incidents": 93, "correct": 4, "accuracy": 0.043 }, "medium": { "incidents": 69, "correct": 55, "accuracy": 0.7971 }, "light": { "incidents": 38, "correct": 36, "accuracy": 0.9474 } }, "light_floor": 0.85, "heavy_ceiling": 0.5, "light_pass": true, "heavy_pass": true } }, "confidence_calibration": { "metric": "investigation_brier_score", "value": 0.0647, "target": 0.18, "passed": true, "duration_ms": 101.4, "details": { "lower_is_better": true, "investigation_brier": 0.0647, "investigation_brier_max": 0.18, "investigation_ece": 0.1712, "investigation_ece_max": 0.18, "triage_brier": 0.0706, "triage_brier_max": 0.2, "triage_ece": 0.1675, "triage_ece_max": 0.2, "investigation_separation": 0.6313, "triage_separation": 0.6122, "mean_confidence_positive_investigation": 0.6922, "mean_confidence_benign_investigation": 0.0609, "investigation_brier_pass": true, "investigation_ece_pass": true, "triage_brier_pass": true, "triage_ece_pass": true } }, "memory_recall": { "metric": "memory_recall_accuracy", "value": 1.0, "target": 1.0, "passed": true, "duration_ms": 1.2, "details": { "total_cases": 14, "passed_cases": 14, "failed_cases": [] } }, "override_accuracy": { "metric": "override_accuracy", "value": 1.0, "target": 1.0, "passed": true, "duration_ms": 1.4, "details": { "total_cases": 6, "passed_cases": 6, "failed_cases": [] } }, "playbook_completion_rate": { "metric": "completion_rate", "value": 0.735, "target": 0.5, "passed": true, "duration_ms": 28.4, "details": { "incidents": 200, "covered": 147, "aligned": 132, "overall_completion_rate": 0.735, "overall_completion_pass": true, "high_critical_completion_rate_mapped": 1.0, "high_critical_completion_rate_raw": 0.8122, "high_critical_covered_mapped": 147, "high_critical_total_mapped": 147, "high_critical_floor": 0.95, "high_critical_pass": true, "action_alignment_rate": 0.898, "action_alignment_floor": 0.85, "action_alignment_pass": true, "per_severity": { "high": { "incidents": 93, "covered": 63, "aligned": 55 }, "critical": { "incidents": 88, "covered": 84, "aligned": 77 }, "medium": { "incidents": 19, "covered": 0, "aligned": 0 } }, "per_category": { "account-takeover": { "incidents": 27, "covered": 98, "aligned": 98 }, "ai-agent-kill-switch": { "incidents": 0, "covered": 0, "aligned": 0 }, "anomalous-data": { "incidents": 12, "covered": 12, "aligned": 8 }, "bec": { "incidents": 14, "covered": 45, "aligned": 23 }, "brute-force": { "incidents": 7, "covered": 7, "aligned": 7 }, "cloud-account-takeover": { "incidents": 10, "covered": 10, "aligned": 7 }, "cloud-misconfig": { "incidents": 9, "covered": 66, "aligned": 6 }, "container-escape": { "incidents": 8, "covered": 8, "aligned": 8 }, "data-exfil": { "incidents": 22, "covered": 47, "aligned": 12 }, "ddos": { "incidents": 3, "covered": 12, "aligned": 6 }, "endpoint-isolation": { "incidents": 12, "covered": 12, "aligned": 12 }, "ids-critical": { "incidents": 6, "covered": 6, "aligned": 6 }, "insider-risk": { "incidents": 14, "covered": 34, "aligned": 14 }, "lateral-movement": { "incidents": 24, "covered": 72, "aligned": 36 }, "malware": { "incidents": 8, "covered": 8, "aligned": 8 }, "network-containment": { "incidents": 3, "covered": 3, "aligned": 3 }, "phishing": { "incidents": 12, "covered": 12, "aligned": 12 }, "privilege-escalation": { "incidents": 12, "covered": 12, "aligned": 0 }, "ransomware": { "incidents": 3, "covered": 15, "aligned": 15 }, "supply-chain": { "incidents": 13, "covered": 45, "aligned": 3 }, "suspicious-signin": { "incidents": 9, "covered": 15, "aligned": 15 }, "web-compromise": { "incidents": 8, "covered": 8, "aligned": 8 } }, "orphan_templates": [], "orphan_playbooks": [], "no_orphan_playbooks": true, "no_orphan_templates": true } }, "detection_fp_rate": { "metric": "worst_per_rule_fp_rate", "value": 0.0031, "target": 0.05, "passed": true, "duration_ms": 5117.1, "details": { "lower_is_better": true, "rules_evaluated": 1293, "rules_passed": 1293, "rules_failed": 0, "mean_fpr": 0.0, "worst_fpr": 0.0031, "ceiling": 0.05, "failing_rules_sample": [] } } }, "all_passed": true }After — 8211548
{ "generated_at": "2026-11-17T02:00:11.616253+00:00", "dataset": "synthetic_incidents.json (200 cases, deterministic)", "suite_filter": "all", "suites": { "mitre_accuracy": { "metric": "accuracy", "value": 0.97, "target": 0.8, "passed": true, "duration_ms": 6.8, "details": { "incidents": 200, "correct": 194, "precision": 0.7058, "recall": 0.85, "f1": 0.7712 }, "per_template": { "metric": "macro_accuracy", "value": 0.9636, "target": 0.8, "passed": true, "template_count": 55, "failing_templates": [ "outlook-auto-forward-rule", "compromised-ci-runner" ] } }, "alert_reduction": { "metric": "reduction_ratio", "value": 0.753, "target": 0.7, "passed": true, "duration_ms": 12.5, "details": { "alerts_in": 1000, "incidents_out": 247, "reduction_pct": "75.3%", "storm_incidents": 16 } }, "investigation_completeness": { "metric": "mean_keyword_coverage", "value": 0.9425, "target": 0.85, "passed": true, "duration_ms": 4.9, "details": { "incidents": 200, "fully_covered": 134, "fully_covered_pct": 0.67 }, "per_template": { "metric": "macro_completeness", "value": 0.9429, "target": 0.8, "passed": true, "template_count": 55, "failing_templates": [] } }, "response_quality": { "metric": "mean_rubric_score", "value": 1.0, "target": 0.8, "passed": true, "duration_ms": 2.4, "details": { "incidents": 200, "criteria": { "action_aligned": 1.0, "severity_aware": 1.0, "mitre_aligned": 1.0, "evidence_grounded": 1.0, "actionable": 1.0 } }, "per_template": { "metric": "macro_score", "value": 1.0, "target": 0.75, "passed": true, "template_count": 55, "failing_templates": [] } }, "hunt_corpus": { "metric": "positive_scenario_catch_rate", "value": 1.0, "target": 1.0, "passed": true, "duration_ms": 158.0, "details": { "hunts": 68, "positives_expected": 68, "positives_caught": 68, "negatives_expected": 68, "false_positives": 0, "false_positive_rate": 0.0, "negative_ceiling": 0.0, "orphan_incident_ids": [], "misses": [], "false_positive_details": [], "positive_pass": true, "negative_pass": true, "no_orphans": true } }, "adversary_eval": { "metric": "graceful_degradation_catch_rate", "value": 0.475, "target": 0.4, "passed": true, "duration_ms": 8.4, "details": { "incidents": 200, "correct": 95, "lost_all_tactics": 105, "buckets": { "heavy": { "incidents": 93, "correct": 4, "accuracy": 0.043 }, "medium": { "incidents": 69, "correct": 55, "accuracy": 0.7971 }, "light": { "incidents": 38, "correct": 36, "accuracy": 0.9474 } }, "light_floor": 0.85, "heavy_ceiling": 0.5, "light_pass": true, "heavy_pass": true } }, "confidence_calibration": { "metric": "investigation_brier_score", "value": 0.0647, "target": 0.18, "passed": true, "duration_ms": 98.5, "details": { "lower_is_better": true, "investigation_brier": 0.0647, "investigation_brier_max": 0.18, "investigation_ece": 0.1712, "investigation_ece_max": 0.18, "triage_brier": 0.0706, "triage_brier_max": 0.2, "triage_ece": 0.1675, "triage_ece_max": 0.2, "investigation_separation": 0.6313, "triage_separation": 0.6122, "mean_confidence_positive_investigation": 0.6922, "mean_confidence_benign_investigation": 0.0609, "investigation_brier_pass": true, "investigation_ece_pass": true, "triage_brier_pass": true, "triage_ece_pass": true } }, "memory_recall": { "metric": "memory_recall_accuracy", "value": 1.0, "target": 1.0, "passed": true, "duration_ms": 1.2, "details": { "total_cases": 14, "passed_cases": 14, "failed_cases": [] } }, "override_accuracy": { "metric": "override_accuracy", "value": 1.0, "target": 1.0, "passed": true, "duration_ms": 1.5, "details": { "total_cases": 6, "passed_cases": 6, "failed_cases": [] } }, "playbook_completion_rate": { "metric": "completion_rate", "value": 0.735, "target": 0.5, "passed": true, "duration_ms": 28.8, "details": { "incidents": 200, "covered": 147, "aligned": 132, "overall_completion_rate": 0.735, "overall_completion_pass": true, "high_critical_completion_rate_mapped": 1.0, "high_critical_completion_rate_raw": 0.8122, "high_critical_covered_mapped": 147, "high_critical_total_mapped": 147, "high_critical_floor": 0.95, "high_critical_pass": true, "action_alignment_rate": 0.898, "action_alignment_floor": 0.85, "action_alignment_pass": true, "per_severity": { "high": { "incidents": 93, "covered": 63, "aligned": 55 }, "critical": { "incidents": 88, "covered": 84, "aligned": 77 }, "medium": { "incidents": 19, "covered": 0, "aligned": 0 } }, "per_category": { "account-takeover": { "incidents": 27, "covered": 98, "aligned": 98 }, "ai-agent-kill-switch": { "incidents": 0, "covered": 0, "aligned": 0 }, "anomalous-data": { "incidents": 12, "covered": 12, "aligned": 8 }, "bec": { "incidents": 14, "covered": 45, "aligned": 23 }, "brute-force": { "incidents": 7, "covered": 7, "aligned": 7 }, "cloud-account-takeover": { "incidents": 10, "covered": 10, "aligned": 7 }, "cloud-misconfig": { "incidents": 9, "covered": 66, "aligned": 6 }, "container-escape": { "incidents": 8, "covered": 8, "aligned": 8 }, "data-exfil": { "incidents": 22, "covered": 47, "aligned": 12 }, "ddos": { "incidents": 3, "covered": 12, "aligned": 6 }, "endpoint-isolation": { "incidents": 12, "covered": 12, "aligned": 12 }, "ids-critical": { "incidents": 6, "covered": 6, "aligned": 6 }, "insider-risk": { "incidents": 14, "covered": 34, "aligned": 14 }, "lateral-movement": { "incidents": 24, "covered": 72, "aligned": 36 }, "malware": { "incidents": 8, "covered": 8, "aligned": 8 }, "network-containment": { "incidents": 3, "covered": 3, "aligned": 3 }, "phishing": { "incidents": 12, "covered": 12, "aligned": 12 }, "privilege-escalation": { "incidents": 12, "covered": 12, "aligned": 0 }, "ransomware": { "incidents": 3, "covered": 15, "aligned": 15 }, "supply-chain": { "incidents": 13, "covered": 45, "aligned": 3 }, "suspicious-signin": { "incidents": 9, "covered": 15, "aligned": 15 }, "web-compromise": { "incidents": 8, "covered": 8, "aligned": 8 } }, "orphan_templates": [], "orphan_playbooks": [], "no_orphan_playbooks": true, "no_orphan_templates": true } }, "detection_fp_rate": { "metric": "worst_per_rule_fp_rate", "value": 0.0031, "target": 0.05, "passed": true, "duration_ms": 4901.0, "details": { "lower_is_better": true, "rules_evaluated": 1293, "rules_passed": 1293, "rules_failed": 0, "mean_fpr": 0.0, "worst_fpr": 0.0031, "ceiling": 0.05, "failing_rules_sample": [] } } }, "all_passed": true }Risk and rollback
Gateway recreation briefly interrupts in-flight requests. Reverting the change and recreating LiteLLM restores the previous behavior and defects. Image upgrades must rerun the gateway tests to verify the pinned entrypoint path. Real AWS permissions remain account/model-specific.