Repository navigation
Conversation
…y-scan remediation and threat model ## Summary This commit carries five workstreams: 1. **Cost model.** Every model call is priced individually, with long-context tiers and a geo-route multiplier. Arms report all-tier cost totals, and stored runs can be re-priced offline. 2. **Prompt limits and cost caps.** Oversized prompts are refused per tier rather than trimmed, and live sweeps require a cost cap. 3. **Identity and access hardening.** The identity envelope fails closed when `principal_type` is missing. The API client allowlist moves to a `SecureString` under its own KMS key. HSTS gains `includeSubDomains` and `preload`. The eval sweep re-checks run ownership on every step that writes. 4. **Security review.** - Every finding from an automated security scan was fixed or suppressed in code with a reason; none remain open. - A new `SECURITY.md` holds the project's threat model, with OWASP Top 10 (2025) and OWASP Top 10 for LLM Applications (2025) mappings. - An external automated threat model was checked claim by claim against the code. 5. **Documentation and results.** The docs and figures cover the new cost model, limits and controls. `results/matrix.md` is refreshed from a 2-arm × 5-repeat live sweep. ## Cost model - `config/pricing.yaml` moves to schema 2: - `long_context` tiers: Haiku 5.5 above 100K input tokens, OpenAI models above 272K; - a single geo-route multiplier for cross-region inference-profile routing. - Every model call is priced individually, and its pricing tier and route are recorded. - Cached prompt tokens count toward the prompt size, and so toward the long-context threshold. - An unknown inference-profile prefix is refused rather than priced at a guess. - Each arm publishes `arm_cost_total` and `arm_long_context_invocation_count` over every tier. `model_metrics` is labelled supervisor-scoped in `matrix.md` and the eval UI. - Long-context call counts appear in the matrix renderers, the tier-comparison UI and the chat usage meter. - `python -m src.eval.reprice --raw-dir …` re-prices a stored run record offline, without a model call. It refuses a per-call archive whose per-tier counts or token sums disagree with the arm archive. ## Prompt limits and budgets - A request prompt above the tier's limit is refused, never trimmed. The limits are 270K tokens for the supervisor and subagents, and 95K for the tool tier. - The turn budget is raised to 1M tokens. - A tree-wide guard keeps the prompt-limit check on every model call path. It covers binders, an explicit `None` and aliased imports. ## Eval harness and sweeps - **Cost cap.** `--live` requires a positive `--max-cost-usd`. Before each unit, the harness stops if spent plus the largest completed unit would exceed the cap. Teardown still runs, no matrix is written, and the exit code is 4. - **Actor validation.** Golden-task actors are validated when the tasks load, before any tenant is seeded. - **Retry.** A live OpenAI turn whose stream fails with an in-band `service_unavailable` is retried and recorded as a throttle exclusion, the same as Bedrock's `ServiceUnavailableException`. It previously aborted the sweep on the first attempt. - **Run ownership.** - The `eval_sweep` state machine now passes the execution ARN to the Seed and Aggregate steps. - Both steps check run ownership before any write, as Teardown and Finalize already did. For a run another execution now owns, such as after an admin unlock, they raise `NotRunOwner`. - The error is not retried. It routes through Teardown and Finalize, which also refuse a non-owner, so the stale execution ends without touching the run. ## Identity and access ### Identity envelope - On the `gateway` and `agent` channels, an absent or invalid `principal_type` is refused with `401`; there is no default. - The relay sends `principal_type` on both turn and cancel. - The golden task that exercises the MCP path now runs as a service principal. - A tree-wide guard checks that agent-channel identity literals and actor fixtures declare `principal_type`. - Gateway interceptor denials are logged at WARNING. Exception cause walks follow `__cause__` only. ### API client allowlist - The allowlist SSM parameter is now a `SecureString` under a dedicated customer-managed KMS key, with rotation on. - Consumers may decrypt it only through SSM, under this parameter's encryption context. Each is explicitly denied decryption through SSM under any other key, `aws/ssm` included. - A parameter that is not a `SecureString` is refused, so a downgraded parameter fails closed. ### Browser and edge - Sign-out also ends the Hosted UI session (`/logout`) when the Hosted UI is enabled. - CloudFront's `Strict-Transport-Security` header adds `includeSubDomains` and `preload` to the one-year max-age. ## Security review ### Scan remediation - **Fixed in code:** - the log-redaction test mints its fake JWT at test time instead of embedding a literal; - `deploy.sh`'s placeholder loop is parsed line by line instead of with a nested-quantifier regex; - each test file uses a single interpolated documentation placeholder account id; - the metering per-million divisor is renamed. - **Suppressed in code with a reason:** non-credential fixture strings carry `# nosec`, and Semgrep false positives carry `# nosemgrep`. - The final re-scan reports no open finding at any severity. `docs/security-baseline.md` records each disposition. ### Threat model (`SECURITY.md`) - **Contents:** - assets, trust boundaries and threat actors; - the security invariants the code enforces; - 20 STRIDE threats with mitigations and residual risk; - accepted risks; - how to report a vulnerability. - **OWASP mappings:** one row per category for the OWASP Top 10 (2025) and the OWASP Top 10 for LLM Applications (2025). - **External threat model.** An automated STRIDE threat model reported 22 threats, 14 High and 8 Medium. Every claim was checked against the code: - 9 are false positives; - 12 are accepted residuals, each with a stated reason; - 1 was a real gap, the sweep ownership check above, which is fixed here. - `SECURITY.md` is linked from the README documentation map. ### Not changed, by decision - No finding was addressed by requiring MFA. MFA stays optional. ## Smoke and verification tooling - `verify_deployment.sh --only` runs write a separate PARTIAL inventory and leave the full-run inventory untouched. - The deployed stack's refusal of an absent `principal_type` is smoke-tested on both channels. - The CloudFront smoke now: - matches inline scripts case-insensitively in its CSP check; - asserts the served HSTS header on every probed behaviour. - The allowlist key's IAM rows are simulated on the identity side only. The policy simulator rejects a KMS key policy as a resource policy, so the key policy itself stays asserted structurally. - The plan fixtures resolve the allowlist key, and they are recaptured for the new infrastructure. ## Documentation - `README.md`, `docs/eval-methodology.md`, `docs/runbook.md`, `docs/development.md` and `docs/security-baseline.md` cover: - the per-call cost model and the prompt limits; - `--max-cost-usd` and offline re-pricing; - `CRM_EXPECTED_ACCOUNT_ID`; - the PARTIAL inventory; - Hosted UI sign-out; - the allowlist key; - the in-band errors the live wrapper retries. - The user-plane and network figures show the allowlist key and Hosted UI sign-out. - New `SECURITY.md`, described above. - `results/matrix.md` is re-rendered from the closing live sweep's stored records. ## Testing - `make verify` (lint, format, type-check, unit, stub, integration, gates, shellcheck): green. Unit tests: 16,325 passed. Plan-time security baseline: 1,458 checks, 0 failed. - Local reproduction of the GitLab CI jobs on a fresh clone of the branch: green. The frontend job runs on linux/amd64 and the verify job on linux/arm64. - Against a deployed dev stack: - targeted post-deploy smoke of the changed resources, with 0 failures beyond previously documented deferrals; - the stub sweep through the deployed state machine: 38 of 38 checks passed, covering the rehearsal, cancellation, tenant teardown and IAM role simulations; - the CloudFront smoke confirmed the HSTS header on every behaviour. - A live sweep ran across both model families, 5 repeats of every golden task, under a $30 cap: - it cost $4.45; - the `demo` tenant was untouched, and every write landed in a disposable tenant; - all artefacts were asserted; - `results/matrix.md` is its re-render, byte-identical to the sweep's own output. - A run losing its ownership mid-sweep cannot be triggered on demand against a live stack. Unit tests and tests that execute the state-machine definition cover that path. ## Upgrade notes - Run `scripts/package_lambdas.sh` and `scripts/package_relay.sh`, then `scripts/deploy.sh`. - Packaging refuses first-party source that differs from the tracked tree. - `deploy.sh` needs `CRM_EXPECTED_ACCOUNT_ID`. - The apply creates the allowlist KMS key, converts the allowlist parameter to `SecureString`, and updates the Lambdas, the `eval_sweep` state machine and the CloudFront headers policy. - `python -m src.eval.harness --live` now exits 2 without a positive `--max-cost-usd`. - A direct invoke on the `gateway` or `agent` channel without `principal_type` is now refused with `401`. ## Known limitations - No AWS WAF, and no per-principal rate or spend quota. - CloudFront's minimum TLS version needs a custom ACM certificate. - CloudTrail has no alarms. - No SCA job runs in CI. - Hosted UI sign-out is dormant on the deployed SPA, which has no Hosted UI domain configured. - The Runtime image still carries `src/relay/node_modules`; its packaging needs the same exclusion the Lambda zips have.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
feat: per-call cost model, prompt limits, identity hardening, security-scan remediation and threat model
Summary
This PR carries five workstreams:
principal_typeis missing. The API client allowlist moves to aSecureStringunder its own KMS key. HSTS gainsincludeSubDomainsandpreload. The eval sweep re-checks run ownership on every step that writes.SECURITY.mdholds the project's threat model, with OWASP Top 10 (2025) and OWASP Top 10 for LLM Applications (2025) mappings.results/matrix.mdis refreshed from a 2-arm × 5-repeat live sweep.Cost model
config/pricing.yamlmoves to schema 2:long_contexttiers: Haiku 5.5 above 100K input tokens, OpenAI models above 272K;arm_cost_totalandarm_long_context_invocation_countover every tier.model_metricsis labelled supervisor-scoped inmatrix.mdand the eval UI.python -m src.eval.reprice --raw-dir …re-prices a stored run record offline, without a model call. It refuses a per-call archive whose per-tier counts or token sums disagree with the arm archive.Prompt limits and budgets
Noneand aliased imports.Eval harness and sweeps
--liverequires a positive--max-cost-usd. Before each unit, the harness stops if spent plus the largest completed unit would exceed the cap. Teardown still runs, no matrix is written, and the exit code is 4.service_unavailableis retried and recorded as a throttle exclusion, the same as Bedrock'sServiceUnavailableException. It previously aborted the sweep on the first attempt.eval_sweepstate machine now passes the execution ARN to the Seed and Aggregate steps.NotRunOwner.Identity and access
Identity envelope
gatewayandagentchannels, an absent or invalidprincipal_typeis refused with401; there is no default.principal_typeon both turn and cancel.principal_type.__cause__only.API client allowlist
SecureStringunder a dedicated customer-managed KMS key, with rotation on.aws/ssmincluded.SecureStringis refused, so a downgraded parameter fails closed.Browser and edge
/logout) when the Hosted UI is enabled.Strict-Transport-Securityheader addsincludeSubDomainsandpreloadto the one-year max-age.Security review
Scan remediation
deploy.sh's placeholder loop is parsed line by line instead of with a nested-quantifier regex;# nosec, and Semgrep false positives carry# nosemgrep.docs/security-baseline.mdrecords each disposition.Threat model (
SECURITY.md)SECURITY.mdis linked from the README documentation map.Not changed, by decision
Smoke and verification tooling
verify_deployment.sh --onlyruns write a separate PARTIAL inventory and leave the full-run inventory untouched.principal_typeis smoke-tested on both channels.Documentation
README.md,docs/eval-methodology.md,docs/runbook.md,docs/development.mdanddocs/security-baseline.mdcover:--max-cost-usdand offline re-pricing;CRM_EXPECTED_ACCOUNT_ID;SECURITY.md, described above.results/matrix.mdis re-rendered from the closing live sweep's stored records.Testing
make verify(lint, format, type-check, unit, stub, integration, gates, shellcheck): green. Unit tests: 16,325 passed. Plan-time security baseline: 1,458 checks, 0 failed.demotenant was untouched, and every write landed in a disposable tenant;results/matrix.mdis its re-render, byte-identical to the sweep's own output.Upgrade notes
scripts/package_lambdas.shandscripts/package_relay.sh, thenscripts/deploy.sh.deploy.shneedsCRM_EXPECTED_ACCOUNT_ID.SecureString, and updates the Lambdas, theeval_sweepstate machine and the CloudFront headers policy.python -m src.eval.harness --livenow exits 2 without a positive--max-cost-usd.gatewayoragentchannel withoutprincipal_typeis now refused with401.Known limitations
src/relay/node_modules; its packaging needs the same exclusion the Lambda zips have.