Skip to content

fix(llm): align gateway authentication and tenant BYOK routing [validation only] - #10

Draft
knowlet wants to merge 2 commits into
mainfrom
fix/llm-provider-configuration
Draft

knowlet wants to merge 2 commits into
mainfrom
fix/llm-provider-configuration

Conversation

@knowlet

@knowlet knowlet commented Nov 13, 2026 •

Copy link
Copy Markdown
Owner

Summary

Native Amazon Bedrock deployments could fail despite a correctly configured host .env: Compose did not forward the AWS settings, every gateway alias explicitly supplied OPENAI_API_KEY, and an empty endpoint could override the provider's default; upgrades could also leave LiteLLM running with its previous startup configuration. This PR fixes AiSOC's integration with LiteLLM's existing native Bedrock support so operators can use a Bedrock bearer token, AWS environment credentials, an existing host profile through an optional read-only mount, or an EC2 instance role through the bundled gateway, while preserving the default CORE/Ollama install and providing an explicit recreate path for configuration changes. It also resolves tenant BYOK endpoint, model and credential choices together for Explain, automatic triage, connection probes and status, while keeping native AWS authentication in the deployment gateway.

Type of change

  • Bug fix (non-breaking change that fixes an issue)
  • New feature (non-breaking change that adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Documentation update
  • Refactor / tech debt
  • CI / infra change

Related issues

Contribution guidance: beenuar#353. No issue is auto-closed.

Supersedes the fork's gateway draft #7, BYOK draft #8, and gateway validation draft #9. Those drafts are closed; their branches remain available.

Changes

Bedrock support through the deployment gateway

This fixes AiSOC's Compose integration with LiteLLM's existing Bedrock implementation for the authentication paths below. The nine task aliases remain intact: operators select accessible bedrock/<model-or-inference-profile-id> values with AISOC_LLM_MODEL_FAST and AISOC_LLM_MODEL_DEEP, set the intended AWS region, and choose an authentication path below. No per-alias credential edit or AWS credential in OPENAI_API_KEY is required.

Authentication path What this PR fixes or wires into Compose Requirement for the operator
Bedrock API key / bearer token Forward AWS_BEARER_TOKEN_BEDROCK to LiteLLM; remove generic alias-level api_key bindings that could override native bearer resolution. Use a token authorized for the selected models. A rejected bearer remains an error and does not silently fall back to another IAM identity.
AWS access/session credentials Forward AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY and AWS_SESSION_TOKEN for LiteLLM's native signing path. Supply the session token when using temporary credentials; no separate OpenAI provider key is needed for Bedrock.
Existing host AWS profile Add the optional infra/compose/docker-compose.aws-profile.yml overlay, mounting the host AWS directory read-only into LiteLLM only; forward profile and credential-file settings. Select AWS_PROFILE as needed and use the same overlay for startup, upgrade and reload. Docker does not inherit the host's ~/.aws automatically; this avoids copying existing profile secrets into AiSOC's YAML.
EC2 instance role Remove blank optional AWS values so they do not mask the SDK credential chain; leave metadata lookup enabled by default. No static AWS keys or profile mount are needed. IMDSv2 must be reachable from the gateway container, and AWS_EC2_METADATA_DISABLED must remain false.

Metadata control is independent of bearer/profile authentication. AWS_EC2_METADATA_DISABLED=true disables metadata lookup, while environment and profile credentials remain available. The pinned-image fixtures cover a mounted profile with metadata disabled, an instance role obtained through IMDSv2, and a bearer that skips IMDS even when metadata is enabled. To intentionally switch from bearer to IAM, remove the bearer and recreate the gateway.

An explicitly blank endpoint now means provider default. Set AISOC_LLM_API_BASE= when moving to Bedrock's regional endpoint, or supply an explicit Bedrock runtime URL. The startup wrapper unsets that blank value before LiteLLM starts. Leaving the variable entirely unconfigured retains Compose's existing Ollama default, preserving the account-free CORE install. Blank optional AWS/profile/region values are likewise removed; an explicit region must still agree with the intended deployment, and should be cleared when deliberately relying on the profile's region.

Configuration changes have an explicit application step. make llm-reload recreates only LiteLLM with its current environment and mounts, and make upgrade now invokes that same step. The target performs recreation; the documentation and regression tests separately check the actual mounted /etc/litellm/config.yaml and loaded model. A changed host file or successful liveness check alone does not establish that the new routing is active.

Credential boundaries stay separate. LITELLM_MASTER_KEY protects access to the gateway; AWS credentials authenticate LiteLLM to Bedrock. Tenant BYOK is a separate override for the supported consumers listed below. The patch does not add a native Bedrock provider or SigV4 implementation to the settings UI.

The profile overlay exposes existing files; it does not implement an in-container AWS sign-in or refresh flow. SSO sessions must be refreshed on the host, and external credential helpers still need to be available in the container. The profile fixture covers shared credentials/config files, not end-to-end SSO renewal.

Authentication selection is also separate from AWS authorization. A usable bearer, profile or role still needs access to the chosen model and any applicable Marketplace subscription. Switching identities can change permissions; a successful bearer request does not establish that the original IAM role is authorized. The fixtures validate credential selection and protocol behavior without contacting AWS; live account/model access remains unverified.

The gateway also preserves OpenAI/Anthropic keyless compatible endpoints by normalizing existing whitespace-only provider keys to an explicit empty string, without creating missing keys. The obsolete --profile full guidance is corrected, and the production overlay inherits the same startup wrapper.

Tenant BYOK

  • Bind endpoint, model and credential source in API Explain, agents Explain, automatic triage, connection probes and tenant-aware status.
  • Inherit deployment credentials only when the complete API base matches. Prevent ambient SDK credentials from filling intentionally keyless tenant routes; the tenant custom/local route uses a non-secret aisoc-no-auth placeholder.
  • Keep hosted OpenAI/Anthropic defaults and existing blank-model fallback semantics. For a custom destination, document and show that users should explicitly set its model or alias: Explain/probe otherwise use the deployment model, while triage retains its task-role model.
  • Align URL credential redaction and airgap policy checks. Add focused regression coverage, live PostgreSQL/vault/provider-HTTP cases, and one real-login CORE Settings model-guidance browser case.

How this relates to Bedrock in the UI: leave BYOK disabled when a tenant should inherit the bundled gateway's Bedrock configuration. A tenant can also select local-litellm or custom with a gateway's OpenAI-compatible URL, served model/alias and gateway key; a raw native Bedrock runtime URL is not that API.

For models supporting AWS's Chat Completions API, the existing custom flow can instead be configured with the regional /openai/v1 base, an explicit compatible model ID and a Bedrock API key. This is a documented protocol-compatible path requiring live acceptance in the target account, not a new native AWS client or a claim of live validation. AWS's model/API compatibility table does not list Anthropic models as Chat Completions compatible; those native Bedrock deployments use the gateway path. The direct custom flow does not obtain IAM-role credentials or sign SigV4 requests.

Scope: gateway runtime changes cover Compose and its production overlay; Helm runtime configuration is unchanged. No native Bedrock UI provider or SigV4 client was added. Other AI workloads retain deployment routing. Demo seeding, case UUID, investigation-ledger and replay repairs were excluded from this consolidation.

Testing

This is the single consolidated validation draft in the contributor fork, intended for beenuar/AiSOC:main. The rebased branch contains 2 commits, 45 files (+3114/-324): exact three-dot diff.

Current head: 82115480b66cacb26eb232e717a8d6da5aa54481
Current upstream base: 2a0d550165b9f82108a3fe1c487c44aa96ed1b24
Verified source tree: 11ebda832dc7d2e4753ea9447fec714fc4d5de7d

Conflict resolution and current-head verification — 2026-11-17

Upstream advanced by 59 commits since the previous feature base. Rebase both feature commits onto that main. The only manual conflict was in services/agents/app/llm/factory.py: retain upstream's AISOC_LLM_THINKING option and extra_body composition together with the existing direct-Anthropic JSON/temperature/token guards. No additional provider behavior was introduced to resolve the conflict.

Independent review confirms that all 38 non-overlapping feature files are byte-identical to the previous published head 7f03fe3a; six auto-merged overlapping files exactly match independently recomputed three-way unions. The seventh overlapping file is the factory conflict above. Upstream's Settings API-key scopes (cases:read, alerts:read), Compose fusion/UEBA/Cyble/Prometheus wiring, Helm action pins and new case-resolution OpenAPI fields are preserved. The second commit is patch-identical, and both commits retain their DCO sign-offs. GitHub reports the branch mergeable against this upstream base.

Local checks ran on the rebased source tree, then the published commits were fetched and verified to have that exact tree. Commit IDs differ from the local rebase because publication recreated the commit metadata; tested source contents are identical.

Current-head verification Result
Whole-tree Ruff 0.16.10, within the repository pin Lint passes; 3,053 files pass the format check.
Full mypy 2.4.1 baseline, Python 3.11.18 21 configured trees plus unmanaged scope, 2,155 files checked; no baseline regression. Existing recorded findings remain.
OpenAPI from the API service's exact locked dependencies scripts/export_openapi.py --check passes.
API Explain / credential probe / resolver routing / status 145 passed, 0 skipped.
API-key UI-scope drift gate 2 passed, 0 skipped.
Agents factory / resolver / triage / Explain / alias suites 117 passed, 0 skipped, including all four upstream thinking tests and existing direct-Anthropic HTTP cases.
Combined thinking + BYOK SDK payload probe 6 cases passed: gateway/custom/direct Anthropic, with and without caller extra_body. Uses the actual ChatOpenAI/OpenAI SDK with httpx.MockTransport; confirms URL/model/key isolation and parameter composition. This is not live-provider acceptance evidence.
Startup / upgrade / tenant-routing helpers 45 passed, 5 skipped, 15 subtests passed. The five skipped PostgreSQL upgrade tests require Docker.
Web 127 test files / 1,265 tests passed, 0 failures, including coverage floors; Node 22.23.5 / pnpm 8.15.1, frozen lockfile. Typecheck passes. ESLint exits 0 with 0 errors and 95 existing warnings; the six Settings warnings exactly match upstream and the other three changed web files have none.
Public eval Current upstream and published head's source tree both pass 11/11 axes; all non-timing suite values/details are identical. Full suite result blocks below.
Workflow / repository checks Six changed workflow YAML files parse; 386 embedded shell blocks pass bash -n. Workflow concurrency, required-check substance (static), merge-marker and whitespace checks pass.

Python tests used separate Python 3.11 environments installed exclusively from each service's generated locked requirements with --no-deps. Inherited proxy variables were removed only for the local test subprocesses; dependencies and fixtures were not changed. Existing deprecation/instrumentation warnings are not hidden. Full Python suites and Docker-backed acceptance were not rerun locally.

Fresh CI for 82115480

At 2026-11-17 10:04 Taiwan time (UTC+8), an exact-head snapshot shows 19 successful workflow runs, 13 running and 35 queued; no failed checks observed. OpenAPI schema and breaking-change checks have passed. Full CI is not yet complete.

The links above track final results; the passing historical counts below must not be read as results for this new head.

Local Docker is unavailable, so fresh gateway-image HTTP/Compose reload, live PostgreSQL BYOK, full Compose smoke and real-login Settings browser evidence must come from this head's CI. No live AWS account/model acceptance is claimed.

Historical CI at 7f03fe3a — completed, not evidence for the new head

An exact-head_sha audit found 68/68 workflow runs completed successfully. Main CI completed on 2026-11-15 00:33 Taiwan time (UTC+8). These results replace the stale “31/32 jobs, Python still running” snapshot; they precede the current upstream rebase.

Historical check Verified result / evidence
Main CI 32/32 jobs successful.
Integration 9/9 jobs successful.
Real-image gateway HTTP and Compose reload 32 passed, 0 skipped, plus three successful negative controls for generic-key bearer masking, keyless endpoint auth and stale reload.
Live PostgreSQL / provider-HTTP BYOK 9 passed, 0 skipped, plus a negative control that correctly rejects the restored probe credential mismatch.
Golden pipeline / Settings browser Golden 10/10 stages, subsequent smoke 10/10, real-login Settings browser 1 passed.
Compose smoke 10/10 stages.
Web Vitest 126 files, 1,261 tests passed.
Full Python suites API 5,462 passed / 249 skipped / 1 xfailed; agents 2,623 passed / 20 skipped plus 219 subtests; repository 2,036 passed / 28 skipped plus 15 subtests.

The intentional pytest failures inside successful negative-control steps demonstrate that the restored bugs are detected; they are not failed CI jobs.

  • Unit tests added / updated
  • Integration tests pass locally
  • Manually tested

Screenshots (if UI changes)

Historical 7f03fe3a real-login Settings browser artifact contains the OpenAI and local-Ollama model-guidance screenshots. It was available and unexpired when checked on 2026-11-17. This browser case covers login and guidance; provider save/probe transport is covered separately. No new-head manual browser inspection or live AWS UI test is claimed.

Checklist

  • My code follows the style guidelines of this project (ruff, eslint, gofmt)
  • I have performed a self-review of my own code
  • I have added tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally
  • I have updated documentation as needed (README, CHANGELOG, CONTRIBUTING)
  • My changes do not introduce new linter warnings
  • I have checked for sensitive data / credentials in my diff

Eval harness (required for substrate / playbook / detection changes)

Ran all suites against base 2a0d550165b9f82108a3fe1c487c44aa96ed1b24 and the exact source tree published as 82115480b66cacb26eb232e717a8d6da5aa54481, using the same locked agents environment on Python 3.11.18 with PYTHONHASHSEED=0:

python scripts/run_evals.py --ci --json --out /tmp/report.json

Both commands exited 0. All 11 axes passed, and every non-timing suite result is identical. The following are the complete suite blocks from each saved JSON report, with dataset and pass status; the per-investigation telemetry section is omitted. Timings naturally differ. These are deterministic evaluations against synthetic substrate fixtures, not live LLM or AWS performance measurements.

The pre-existing failing MITRE templates remain outlook-auto-forward-rule and compromised-ci-runner; they do not represent a new regression. Macro accuracy/completeness/response scores remain 0.9636 / 0.9429 / 1.0000.

Before — upstream 2a0d550
{
  "generated_at": "2026-11-17T02:00:11.632228+00:00",
  "dataset": "synthetic_incidents.json (200 cases, deterministic)",
  "suite_filter": "all",
  "suites": {
    "mitre_accuracy": {
      "metric": "accuracy",
      "value": 0.97,
      "target": 0.8,
      "passed": true,
      "duration_ms": 6.3,
      "details": {
        "incidents": 200,
        "correct": 194,
        "precision": 0.7058,
        "recall": 0.85,
        "f1": 0.7712
      },
      "per_template": {
        "metric": "macro_accuracy",
        "value": 0.9636,
        "target": 0.8,
        "passed": true,
        "template_count": 55,
        "failing_templates": [
          "outlook-auto-forward-rule",
          "compromised-ci-runner"
        ]
      }
    },
    "alert_reduction": {
      "metric": "reduction_ratio",
      "value": 0.753,
      "target": 0.7,
      "passed": true,
      "duration_ms": 13.0,
      "details": {
        "alerts_in": 1000,
        "incidents_out": 247,
        "reduction_pct": "75.3%",
        "storm_incidents": 16
      }
    },
    "investigation_completeness": {
      "metric": "mean_keyword_coverage",
      "value": 0.9425,
      "target": 0.85,
      "passed": true,
      "duration_ms": 4.6,
      "details": {
        "incidents": 200,
        "fully_covered": 134,
        "fully_covered_pct": 0.67
      },
      "per_template": {
        "metric": "macro_completeness",
        "value": 0.9429,
        "target": 0.8,
        "passed": true,
        "template_count": 55,
        "failing_templates": []
      }
    },
    "response_quality": {
      "metric": "mean_rubric_score",
      "value": 1.0,
      "target": 0.8,
      "passed": true,
      "duration_ms": 2.1,
      "details": {
        "incidents": 200,
        "criteria": {
          "action_aligned": 1.0,
          "severity_aware": 1.0,
          "mitre_aligned": 1.0,
          "evidence_grounded": 1.0,
          "actionable": 1.0
        }
      },
      "per_template": {
        "metric": "macro_score",
        "value": 1.0,
        "target": 0.75,
        "passed": true,
        "template_count": 55,
        "failing_templates": []
      }
    },
    "hunt_corpus": {
      "metric": "positive_scenario_catch_rate",
      "value": 1.0,
      "target": 1.0,
      "passed": true,
      "duration_ms": 129.3,
      "details": {
        "hunts": 68,
        "positives_expected": 68,
        "positives_caught": 68,
        "negatives_expected": 68,
        "false_positives": 0,
        "false_positive_rate": 0.0,
        "negative_ceiling": 0.0,
        "orphan_incident_ids": [],
        "misses": [],
        "false_positive_details": [],
        "positive_pass": true,
        "negative_pass": true,
        "no_orphans": true
      }
    },
    "adversary_eval": {
      "metric": "graceful_degradation_catch_rate",
      "value": 0.475,
      "target": 0.4,
      "passed": true,
      "duration_ms": 8.2,
      "details": {
        "incidents": 200,
        "correct": 95,
        "lost_all_tactics": 105,
        "buckets": {
          "heavy": {
            "incidents": 93,
            "correct": 4,
            "accuracy": 0.043
          },
          "medium": {
            "incidents": 69,
            "correct": 55,
            "accuracy": 0.7971
          },
          "light": {
            "incidents": 38,
            "correct": 36,
            "accuracy": 0.9474
          }
        },
        "light_floor": 0.85,
        "heavy_ceiling": 0.5,
        "light_pass": true,
        "heavy_pass": true
      }
    },
    "confidence_calibration": {
      "metric": "investigation_brier_score",
      "value": 0.0647,
      "target": 0.18,
      "passed": true,
      "duration_ms": 101.4,
      "details": {
        "lower_is_better": true,
        "investigation_brier": 0.0647,
        "investigation_brier_max": 0.18,
        "investigation_ece": 0.1712,
        "investigation_ece_max": 0.18,
        "triage_brier": 0.0706,
        "triage_brier_max": 0.2,
        "triage_ece": 0.1675,
        "triage_ece_max": 0.2,
        "investigation_separation": 0.6313,
        "triage_separation": 0.6122,
        "mean_confidence_positive_investigation": 0.6922,
        "mean_confidence_benign_investigation": 0.0609,
        "investigation_brier_pass": true,
        "investigation_ece_pass": true,
        "triage_brier_pass": true,
        "triage_ece_pass": true
      }
    },
    "memory_recall": {
      "metric": "memory_recall_accuracy",
      "value": 1.0,
      "target": 1.0,
      "passed": true,
      "duration_ms": 1.2,
      "details": {
        "total_cases": 14,
        "passed_cases": 14,
        "failed_cases": []
      }
    },
    "override_accuracy": {
      "metric": "override_accuracy",
      "value": 1.0,
      "target": 1.0,
      "passed": true,
      "duration_ms": 1.4,
      "details": {
        "total_cases": 6,
        "passed_cases": 6,
        "failed_cases": []
      }
    },
    "playbook_completion_rate": {
      "metric": "completion_rate",
      "value": 0.735,
      "target": 0.5,
      "passed": true,
      "duration_ms": 28.4,
      "details": {
        "incidents": 200,
        "covered": 147,
        "aligned": 132,
        "overall_completion_rate": 0.735,
        "overall_completion_pass": true,
        "high_critical_completion_rate_mapped": 1.0,
        "high_critical_completion_rate_raw": 0.8122,
        "high_critical_covered_mapped": 147,
        "high_critical_total_mapped": 147,
        "high_critical_floor": 0.95,
        "high_critical_pass": true,
        "action_alignment_rate": 0.898,
        "action_alignment_floor": 0.85,
        "action_alignment_pass": true,
        "per_severity": {
          "high": {
            "incidents": 93,
            "covered": 63,
            "aligned": 55
          },
          "critical": {
            "incidents": 88,
            "covered": 84,
            "aligned": 77
          },
          "medium": {
            "incidents": 19,
            "covered": 0,
            "aligned": 0
          }
        },
        "per_category": {
          "account-takeover": {
            "incidents": 27,
            "covered": 98,
            "aligned": 98
          },
          "ai-agent-kill-switch": {
            "incidents": 0,
            "covered": 0,
            "aligned": 0
          },
          "anomalous-data": {
            "incidents": 12,
            "covered": 12,
            "aligned": 8
          },
          "bec": {
            "incidents": 14,
            "covered": 45,
            "aligned": 23
          },
          "brute-force": {
            "incidents": 7,
            "covered": 7,
            "aligned": 7
          },
          "cloud-account-takeover": {
            "incidents": 10,
            "covered": 10,
            "aligned": 7
          },
          "cloud-misconfig": {
            "incidents": 9,
            "covered": 66,
            "aligned": 6
          },
          "container-escape": {
            "incidents": 8,
            "covered": 8,
            "aligned": 8
          },
          "data-exfil": {
            "incidents": 22,
            "covered": 47,
            "aligned": 12
          },
          "ddos": {
            "incidents": 3,
            "covered": 12,
            "aligned": 6
          },
          "endpoint-isolation": {
            "incidents": 12,
            "covered": 12,
            "aligned": 12
          },
          "ids-critical": {
            "incidents": 6,
            "covered": 6,
            "aligned": 6
          },
          "insider-risk": {
            "incidents": 14,
            "covered": 34,
            "aligned": 14
          },
          "lateral-movement": {
            "incidents": 24,
            "covered": 72,
            "aligned": 36
          },
          "malware": {
            "incidents": 8,
            "covered": 8,
            "aligned": 8
          },
          "network-containment": {
            "incidents": 3,
            "covered": 3,
            "aligned": 3
          },
          "phishing": {
            "incidents": 12,
            "covered": 12,
            "aligned": 12
          },
          "privilege-escalation": {
            "incidents": 12,
            "covered": 12,
            "aligned": 0
          },
          "ransomware": {
            "incidents": 3,
            "covered": 15,
            "aligned": 15
          },
          "supply-chain": {
            "incidents": 13,
            "covered": 45,
            "aligned": 3
          },
          "suspicious-signin": {
            "incidents": 9,
            "covered": 15,
            "aligned": 15
          },
          "web-compromise": {
            "incidents": 8,
            "covered": 8,
            "aligned": 8
          }
        },
        "orphan_templates": [],
        "orphan_playbooks": [],
        "no_orphan_playbooks": true,
        "no_orphan_templates": true
      }
    },
    "detection_fp_rate": {
      "metric": "worst_per_rule_fp_rate",
      "value": 0.0031,
      "target": 0.05,
      "passed": true,
      "duration_ms": 5117.1,
      "details": {
        "lower_is_better": true,
        "rules_evaluated": 1293,
        "rules_passed": 1293,
        "rules_failed": 0,
        "mean_fpr": 0.0,
        "worst_fpr": 0.0031,
        "ceiling": 0.05,
        "failing_rules_sample": []
      }
    }
  },
  "all_passed": true
}
After — 8211548
{
  "generated_at": "2026-11-17T02:00:11.616253+00:00",
  "dataset": "synthetic_incidents.json (200 cases, deterministic)",
  "suite_filter": "all",
  "suites": {
    "mitre_accuracy": {
      "metric": "accuracy",
      "value": 0.97,
      "target": 0.8,
      "passed": true,
      "duration_ms": 6.8,
      "details": {
        "incidents": 200,
        "correct": 194,
        "precision": 0.7058,
        "recall": 0.85,
        "f1": 0.7712
      },
      "per_template": {
        "metric": "macro_accuracy",
        "value": 0.9636,
        "target": 0.8,
        "passed": true,
        "template_count": 55,
        "failing_templates": [
          "outlook-auto-forward-rule",
          "compromised-ci-runner"
        ]
      }
    },
    "alert_reduction": {
      "metric": "reduction_ratio",
      "value": 0.753,
      "target": 0.7,
      "passed": true,
      "duration_ms": 12.5,
      "details": {
        "alerts_in": 1000,
        "incidents_out": 247,
        "reduction_pct": "75.3%",
        "storm_incidents": 16
      }
    },
    "investigation_completeness": {
      "metric": "mean_keyword_coverage",
      "value": 0.9425,
      "target": 0.85,
      "passed": true,
      "duration_ms": 4.9,
      "details": {
        "incidents": 200,
        "fully_covered": 134,
        "fully_covered_pct": 0.67
      },
      "per_template": {
        "metric": "macro_completeness",
        "value": 0.9429,
        "target": 0.8,
        "passed": true,
        "template_count": 55,
        "failing_templates": []
      }
    },
    "response_quality": {
      "metric": "mean_rubric_score",
      "value": 1.0,
      "target": 0.8,
      "passed": true,
      "duration_ms": 2.4,
      "details": {
        "incidents": 200,
        "criteria": {
          "action_aligned": 1.0,
          "severity_aware": 1.0,
          "mitre_aligned": 1.0,
          "evidence_grounded": 1.0,
          "actionable": 1.0
        }
      },
      "per_template": {
        "metric": "macro_score",
        "value": 1.0,
        "target": 0.75,
        "passed": true,
        "template_count": 55,
        "failing_templates": []
      }
    },
    "hunt_corpus": {
      "metric": "positive_scenario_catch_rate",
      "value": 1.0,
      "target": 1.0,
      "passed": true,
      "duration_ms": 158.0,
      "details": {
        "hunts": 68,
        "positives_expected": 68,
        "positives_caught": 68,
        "negatives_expected": 68,
        "false_positives": 0,
        "false_positive_rate": 0.0,
        "negative_ceiling": 0.0,
        "orphan_incident_ids": [],
        "misses": [],
        "false_positive_details": [],
        "positive_pass": true,
        "negative_pass": true,
        "no_orphans": true
      }
    },
    "adversary_eval": {
      "metric": "graceful_degradation_catch_rate",
      "value": 0.475,
      "target": 0.4,
      "passed": true,
      "duration_ms": 8.4,
      "details": {
        "incidents": 200,
        "correct": 95,
        "lost_all_tactics": 105,
        "buckets": {
          "heavy": {
            "incidents": 93,
            "correct": 4,
            "accuracy": 0.043
          },
          "medium": {
            "incidents": 69,
            "correct": 55,
            "accuracy": 0.7971
          },
          "light": {
            "incidents": 38,
            "correct": 36,
            "accuracy": 0.9474
          }
        },
        "light_floor": 0.85,
        "heavy_ceiling": 0.5,
        "light_pass": true,
        "heavy_pass": true
      }
    },
    "confidence_calibration": {
      "metric": "investigation_brier_score",
      "value": 0.0647,
      "target": 0.18,
      "passed": true,
      "duration_ms": 98.5,
      "details": {
        "lower_is_better": true,
        "investigation_brier": 0.0647,
        "investigation_brier_max": 0.18,
        "investigation_ece": 0.1712,
        "investigation_ece_max": 0.18,
        "triage_brier": 0.0706,
        "triage_brier_max": 0.2,
        "triage_ece": 0.1675,
        "triage_ece_max": 0.2,
        "investigation_separation": 0.6313,
        "triage_separation": 0.6122,
        "mean_confidence_positive_investigation": 0.6922,
        "mean_confidence_benign_investigation": 0.0609,
        "investigation_brier_pass": true,
        "investigation_ece_pass": true,
        "triage_brier_pass": true,
        "triage_ece_pass": true
      }
    },
    "memory_recall": {
      "metric": "memory_recall_accuracy",
      "value": 1.0,
      "target": 1.0,
      "passed": true,
      "duration_ms": 1.2,
      "details": {
        "total_cases": 14,
        "passed_cases": 14,
        "failed_cases": []
      }
    },
    "override_accuracy": {
      "metric": "override_accuracy",
      "value": 1.0,
      "target": 1.0,
      "passed": true,
      "duration_ms": 1.5,
      "details": {
        "total_cases": 6,
        "passed_cases": 6,
        "failed_cases": []
      }
    },
    "playbook_completion_rate": {
      "metric": "completion_rate",
      "value": 0.735,
      "target": 0.5,
      "passed": true,
      "duration_ms": 28.8,
      "details": {
        "incidents": 200,
        "covered": 147,
        "aligned": 132,
        "overall_completion_rate": 0.735,
        "overall_completion_pass": true,
        "high_critical_completion_rate_mapped": 1.0,
        "high_critical_completion_rate_raw": 0.8122,
        "high_critical_covered_mapped": 147,
        "high_critical_total_mapped": 147,
        "high_critical_floor": 0.95,
        "high_critical_pass": true,
        "action_alignment_rate": 0.898,
        "action_alignment_floor": 0.85,
        "action_alignment_pass": true,
        "per_severity": {
          "high": {
            "incidents": 93,
            "covered": 63,
            "aligned": 55
          },
          "critical": {
            "incidents": 88,
            "covered": 84,
            "aligned": 77
          },
          "medium": {
            "incidents": 19,
            "covered": 0,
            "aligned": 0
          }
        },
        "per_category": {
          "account-takeover": {
            "incidents": 27,
            "covered": 98,
            "aligned": 98
          },
          "ai-agent-kill-switch": {
            "incidents": 0,
            "covered": 0,
            "aligned": 0
          },
          "anomalous-data": {
            "incidents": 12,
            "covered": 12,
            "aligned": 8
          },
          "bec": {
            "incidents": 14,
            "covered": 45,
            "aligned": 23
          },
          "brute-force": {
            "incidents": 7,
            "covered": 7,
            "aligned": 7
          },
          "cloud-account-takeover": {
            "incidents": 10,
            "covered": 10,
            "aligned": 7
          },
          "cloud-misconfig": {
            "incidents": 9,
            "covered": 66,
            "aligned": 6
          },
          "container-escape": {
            "incidents": 8,
            "covered": 8,
            "aligned": 8
          },
          "data-exfil": {
            "incidents": 22,
            "covered": 47,
            "aligned": 12
          },
          "ddos": {
            "incidents": 3,
            "covered": 12,
            "aligned": 6
          },
          "endpoint-isolation": {
            "incidents": 12,
            "covered": 12,
            "aligned": 12
          },
          "ids-critical": {
            "incidents": 6,
            "covered": 6,
            "aligned": 6
          },
          "insider-risk": {
            "incidents": 14,
            "covered": 34,
            "aligned": 14
          },
          "lateral-movement": {
            "incidents": 24,
            "covered": 72,
            "aligned": 36
          },
          "malware": {
            "incidents": 8,
            "covered": 8,
            "aligned": 8
          },
          "network-containment": {
            "incidents": 3,
            "covered": 3,
            "aligned": 3
          },
          "phishing": {
            "incidents": 12,
            "covered": 12,
            "aligned": 12
          },
          "privilege-escalation": {
            "incidents": 12,
            "covered": 12,
            "aligned": 0
          },
          "ransomware": {
            "incidents": 3,
            "covered": 15,
            "aligned": 15
          },
          "supply-chain": {
            "incidents": 13,
            "covered": 45,
            "aligned": 3
          },
          "suspicious-signin": {
            "incidents": 9,
            "covered": 15,
            "aligned": 15
          },
          "web-compromise": {
            "incidents": 8,
            "covered": 8,
            "aligned": 8
          }
        },
        "orphan_templates": [],
        "orphan_playbooks": [],
        "no_orphan_playbooks": true,
        "no_orphan_templates": true
      }
    },
    "detection_fp_rate": {
      "metric": "worst_per_rule_fp_rate",
      "value": 0.0031,
      "target": 0.05,
      "passed": true,
      "duration_ms": 4901.0,
      "details": {
        "lower_is_better": true,
        "rules_evaluated": 1293,
        "rules_passed": 1293,
        "rules_failed": 0,
        "mean_fpr": 0.0,
        "worst_fpr": 0.0031,
        "ceiling": 0.05,
        "failing_rules_sample": []
      }
    }
  },
  "all_passed": true
}
  • Not applicable
  • No axis regressed
  • An axis regressed

Risk and rollback

Gateway recreation briefly interrupts in-flight requests. Reverting the change and recreating LiteLLM restores the previous behavior and defects. Image upgrades must rerun the gateway tests to verify the pinned entrypoint path. Real AWS permissions remain account/model-specific.

@coderabbitai

coderabbitai Bot commented Nov 13, 2026

Copy link
Copy Markdown

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

  • Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

Comment thread tests/test_litellm_reload.py Fixed
Preserve CORE/Ollama defaults while resolving provider-native gateway
credentials and reloading changed configuration during upgrades. Bind each
tenant endpoint to its credential source across runtime consumers, probes
and status, with explicit provider model guidance.

Combine the related AI configuration fixes on the current upstream base.
Keep demo seeding and investigation-ledger repairs outside this change.
Retain real gateway, PostgreSQL, provider-HTTP and Settings browser coverage.

Signed-off-by: Albert Hsieh <knowlet@pm.me>
Keep the gateway state helper's timeout path explicitly terminating.
Preserve the existing error message and failure semantics.

Signed-off-by: Albert Hsieh <knowlet@pm.me>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants