Skip to content

feat: per-call cost model, prompt limits, and threat model - #4

Merged
khanrubd merged 1 commit into
mainfrom
develop
Nov 11, 2026
Merged

khanrubd merged 1 commit into
mainfrom
develop

Conversation

@khanrubd

Copy link
Copy Markdown
Contributor

feat: per-call cost model, prompt limits, identity hardening, security-scan remediation and threat model

Summary

This PR carries five workstreams:

  1. Cost model. Every model call is priced individually, with long-context tiers and a geo-route multiplier. Arms report all-tier cost totals, and stored runs can be re-priced offline.
  2. Prompt limits and cost caps. Oversized prompts are refused per tier rather than trimmed, and live sweeps require a cost cap.
  3. Identity and access hardening. The identity envelope fails closed when principal_type is missing. The API client allowlist moves to a SecureString under its own KMS key. HSTS gains includeSubDomains and preload. The eval sweep re-checks run ownership on every step that writes.
  4. Security review.
    • Every finding from an automated security scan was fixed or suppressed in code with a reason; none remain open.
    • A new SECURITY.md holds the project's threat model, with OWASP Top 10 (2025) and OWASP Top 10 for LLM Applications (2025) mappings.
    • An external automated threat model was checked claim by claim against the code.
  5. Documentation and results. The docs and figures cover the new cost model, limits and controls. results/matrix.md is refreshed from a 2-arm × 5-repeat live sweep.

Cost model

  • config/pricing.yaml moves to schema 2:
    • long_context tiers: Haiku 5.5 above 100K input tokens, OpenAI models above 272K;
    • a single geo-route multiplier for cross-region inference-profile routing.
  • Every model call is priced individually, and its pricing tier and route are recorded.
  • Cached prompt tokens count toward the prompt size, and so toward the long-context threshold.
  • An unknown inference-profile prefix is refused rather than priced at a guess.
  • Each arm publishes arm_cost_total and arm_long_context_invocation_count over every tier. model_metrics is labelled supervisor-scoped in matrix.md and the eval UI.
  • Long-context call counts appear in the matrix renderers, the tier-comparison UI and the chat usage meter.
  • python -m src.eval.reprice --raw-dir … re-prices a stored run record offline, without a model call. It refuses a per-call archive whose per-tier counts or token sums disagree with the arm archive.

Prompt limits and budgets

  • A request prompt above the tier's limit is refused, never trimmed. The limits are 270K tokens for the supervisor and subagents, and 95K for the tool tier.
  • The turn budget is raised to 1M tokens.
  • A tree-wide guard keeps the prompt-limit check on every model call path. It covers binders, an explicit None and aliased imports.

Eval harness and sweeps

  • Cost cap. --live requires a positive --max-cost-usd. Before each unit, the harness stops if spent plus the largest completed unit would exceed the cap. Teardown still runs, no matrix is written, and the exit code is 4.
  • Actor validation. Golden-task actors are validated when the tasks load, before any tenant is seeded.
  • Retry. A live OpenAI turn whose stream fails with an in-band service_unavailable is retried and recorded as a throttle exclusion, the same as Bedrock's ServiceUnavailableException. It previously aborted the sweep on the first attempt.
  • Run ownership.
    • The eval_sweep state machine now passes the execution ARN to the Seed and Aggregate steps.
    • Both steps check run ownership before any write, as Teardown and Finalize already did. For a run another execution now owns, such as after an admin unlock, they raise NotRunOwner.
    • The error is not retried. It routes through Teardown and Finalize, which also refuse a non-owner, so the stale execution ends without touching the run.

Identity and access

Identity envelope

  • On the gateway and agent channels, an absent or invalid principal_type is refused with 401; there is no default.
  • The relay sends principal_type on both turn and cancel.
  • The golden task that exercises the MCP path now runs as a service principal.
  • A tree-wide guard checks that agent-channel identity literals and actor fixtures declare principal_type.
  • Gateway interceptor denials are logged at WARNING. Exception cause walks follow __cause__ only.

API client allowlist

  • The allowlist SSM parameter is now a SecureString under a dedicated customer-managed KMS key, with rotation on.
  • Consumers may decrypt it only through SSM, under this parameter's encryption context. Each is explicitly denied decryption through SSM under any other key, aws/ssm included.
  • A parameter that is not a SecureString is refused, so a downgraded parameter fails closed.

Browser and edge

  • Sign-out also ends the Hosted UI session (/logout) when the Hosted UI is enabled.
  • CloudFront's Strict-Transport-Security header adds includeSubDomains and preload to the one-year max-age.

Security review

Scan remediation

  • Fixed in code:
    • the log-redaction test mints its fake JWT at test time instead of embedding a literal;
    • deploy.sh's placeholder loop is parsed line by line instead of with a nested-quantifier regex;
    • each test file uses a single interpolated documentation placeholder account id;
    • the metering per-million divisor is renamed.
  • Suppressed in code with a reason: non-credential fixture strings carry # nosec, and Semgrep false positives carry # nosemgrep.
  • The final re-scan reports no open finding at any severity. docs/security-baseline.md records each disposition.

Threat model (SECURITY.md)

  • Contents:
    • assets, trust boundaries and threat actors;
    • the security invariants the code enforces;
    • 20 STRIDE threats with mitigations and residual risk;
    • accepted risks;
    • how to report a vulnerability.
  • OWASP mappings: one row per category for the OWASP Top 10 (2025) and the OWASP Top 10 for LLM Applications (2025).
  • External threat model. An automated STRIDE threat model reported 22 threats, 14 High and 8 Medium. Every claim was checked against the code:
    • 9 are false positives;
    • 12 are accepted residuals, each with a stated reason;
    • 1 was a real gap, the sweep ownership check above, which is fixed here.
  • SECURITY.md is linked from the README documentation map.

Not changed, by decision

  • No finding was addressed by requiring MFA. MFA stays optional.

Smoke and verification tooling

  • verify_deployment.sh --only runs write a separate PARTIAL inventory and leave the full-run inventory untouched.
  • The deployed stack's refusal of an absent principal_type is smoke-tested on both channels.
  • The CloudFront smoke now:
    • matches inline scripts case-insensitively in its CSP check;
    • asserts the served HSTS header on every probed behaviour.
  • The allowlist key's IAM rows are simulated on the identity side only. The policy simulator rejects a KMS key policy as a resource policy, so the key policy itself stays asserted structurally.
  • The plan fixtures resolve the allowlist key, and they are recaptured for the new infrastructure.

Documentation

  • README.md, docs/eval-methodology.md, docs/runbook.md, docs/development.md and docs/security-baseline.md cover:
    • the per-call cost model and the prompt limits;
    • --max-cost-usd and offline re-pricing;
    • CRM_EXPECTED_ACCOUNT_ID;
    • the PARTIAL inventory;
    • Hosted UI sign-out;
    • the allowlist key;
    • the in-band errors the live wrapper retries.
  • The user-plane and network figures show the allowlist key and Hosted UI sign-out.
  • New SECURITY.md, described above.
  • results/matrix.md is re-rendered from the closing live sweep's stored records.

Testing

  • make verify (lint, format, type-check, unit, stub, integration, gates, shellcheck): green. Unit tests: 16,325 passed. Plan-time security baseline: 1,458 checks, 0 failed.
  • Local reproduction of the GitLab CI jobs on a fresh clone of the branch: green. The frontend job runs on linux/amd64 and the verify job on linux/arm64.
  • Against a deployed dev stack:
    • targeted post-deploy smoke of the changed resources, with 0 failures beyond previously documented deferrals;
    • the stub sweep through the deployed state machine: 38 of 38 checks passed, covering the rehearsal, cancellation, tenant teardown and IAM role simulations;
    • the CloudFront smoke confirmed the HSTS header on every behaviour.
  • A live sweep ran across both model families, 5 repeats of every golden task, under a $30 cap:
    • it cost $4.45;
    • the demo tenant was untouched, and every write landed in a disposable tenant;
    • all artefacts were asserted;
    • results/matrix.md is its re-render, byte-identical to the sweep's own output.
  • A run losing its ownership mid-sweep cannot be triggered on demand against a live stack. Unit tests and tests that execute the state-machine definition cover that path.

Upgrade notes

  • Run scripts/package_lambdas.sh and scripts/package_relay.sh, then scripts/deploy.sh.
    • Packaging refuses first-party source that differs from the tracked tree.
    • deploy.sh needs CRM_EXPECTED_ACCOUNT_ID.
  • The apply creates the allowlist KMS key, converts the allowlist parameter to SecureString, and updates the Lambdas, the eval_sweep state machine and the CloudFront headers policy.
  • python -m src.eval.harness --live now exits 2 without a positive --max-cost-usd.
  • A direct invoke on the gateway or agent channel without principal_type is now refused with 401.

Known limitations

  • No AWS WAF, and no per-principal rate or spend quota.
  • CloudFront's minimum TLS version needs a custom ACM certificate.
  • CloudTrail has no alarms.
  • No SCA job runs in CI.
  • Hosted UI sign-out is dormant on the deployed SPA, which has no Hosted UI domain configured.
  • The Runtime image still carries src/relay/node_modules; its packaging needs the same exclusion the Lambda zips have.

…y-scan remediation and threat model

## Summary

This commit carries five workstreams:

1. **Cost model.** Every model call is priced individually, with long-context tiers and a geo-route multiplier. Arms report all-tier cost totals, and stored runs can be re-priced offline.
2. **Prompt limits and cost caps.** Oversized prompts are refused per tier rather than trimmed, and live sweeps require a cost cap.
3. **Identity and access hardening.** The identity envelope fails closed when `principal_type` is missing. The API client allowlist moves to a `SecureString` under its own KMS key. HSTS gains `includeSubDomains` and `preload`. The eval sweep re-checks run ownership on every step that writes.
4. **Security review.**
   - Every finding from an automated security scan was fixed or suppressed in code with a reason; none remain open.
   - A new `SECURITY.md` holds the project's threat model, with OWASP Top 10 (2025) and OWASP Top 10 for LLM Applications (2025) mappings.
   - An external automated threat model was checked claim by claim against the code.
5. **Documentation and results.** The docs and figures cover the new cost model, limits and controls. `results/matrix.md` is refreshed from a 2-arm × 5-repeat live sweep.

## Cost model

- `config/pricing.yaml` moves to schema 2:
  - `long_context` tiers: Haiku 5.5 above 100K input tokens, OpenAI models above 272K;
  - a single geo-route multiplier for cross-region inference-profile routing.
- Every model call is priced individually, and its pricing tier and route are recorded.
- Cached prompt tokens count toward the prompt size, and so toward the long-context threshold.
- An unknown inference-profile prefix is refused rather than priced at a guess.
- Each arm publishes `arm_cost_total` and `arm_long_context_invocation_count` over every tier. `model_metrics` is labelled supervisor-scoped in `matrix.md` and the eval UI.
- Long-context call counts appear in the matrix renderers, the tier-comparison UI and the chat usage meter.
- `python -m src.eval.reprice --raw-dir …` re-prices a stored run record offline, without a model call. It refuses a per-call archive whose per-tier counts or token sums disagree with the arm archive.

## Prompt limits and budgets

- A request prompt above the tier's limit is refused, never trimmed. The limits are 270K tokens for the supervisor and subagents, and 95K for the tool tier.
- The turn budget is raised to 1M tokens.
- A tree-wide guard keeps the prompt-limit check on every model call path. It covers binders, an explicit `None` and aliased imports.

## Eval harness and sweeps

- **Cost cap.** `--live` requires a positive `--max-cost-usd`. Before each unit, the harness stops if spent plus the largest completed unit would exceed the cap. Teardown still runs, no matrix is written, and the exit code is 4.
- **Actor validation.** Golden-task actors are validated when the tasks load, before any tenant is seeded.
- **Retry.** A live OpenAI turn whose stream fails with an in-band `service_unavailable` is retried and recorded as a throttle exclusion, the same as Bedrock's `ServiceUnavailableException`. It previously aborted the sweep on the first attempt.
- **Run ownership.**
  - The `eval_sweep` state machine now passes the execution ARN to the Seed and Aggregate steps.
  - Both steps check run ownership before any write, as Teardown and Finalize already did. For a run another execution now owns, such as after an admin unlock, they raise `NotRunOwner`.
  - The error is not retried. It routes through Teardown and Finalize, which also refuse a non-owner, so the stale execution ends without touching the run.

## Identity and access

### Identity envelope
- On the `gateway` and `agent` channels, an absent or invalid `principal_type` is refused with `401`; there is no default.
- The relay sends `principal_type` on both turn and cancel.
- The golden task that exercises the MCP path now runs as a service principal.
- A tree-wide guard checks that agent-channel identity literals and actor fixtures declare `principal_type`.
- Gateway interceptor denials are logged at WARNING. Exception cause walks follow `__cause__` only.

### API client allowlist
- The allowlist SSM parameter is now a `SecureString` under a dedicated customer-managed KMS key, with rotation on.
- Consumers may decrypt it only through SSM, under this parameter's encryption context. Each is explicitly denied decryption through SSM under any other key, `aws/ssm` included.
- A parameter that is not a `SecureString` is refused, so a downgraded parameter fails closed.

### Browser and edge
- Sign-out also ends the Hosted UI session (`/logout`) when the Hosted UI is enabled.
- CloudFront's `Strict-Transport-Security` header adds `includeSubDomains` and `preload` to the one-year max-age.

## Security review

### Scan remediation
- **Fixed in code:**
  - the log-redaction test mints its fake JWT at test time instead of embedding a literal;
  - `deploy.sh`'s placeholder loop is parsed line by line instead of with a nested-quantifier regex;
  - each test file uses a single interpolated documentation placeholder account id;
  - the metering per-million divisor is renamed.
- **Suppressed in code with a reason:** non-credential fixture strings carry `# nosec`, and Semgrep false positives carry `# nosemgrep`.
- The final re-scan reports no open finding at any severity. `docs/security-baseline.md` records each disposition.

### Threat model (`SECURITY.md`)
- **Contents:**
  - assets, trust boundaries and threat actors;
  - the security invariants the code enforces;
  - 20 STRIDE threats with mitigations and residual risk;
  - accepted risks;
  - how to report a vulnerability.
- **OWASP mappings:** one row per category for the OWASP Top 10 (2025) and the OWASP Top 10 for LLM Applications (2025).
- **External threat model.** An automated STRIDE threat model reported 22 threats, 14 High and 8 Medium. Every claim was checked against the code:
  - 9 are false positives;
  - 12 are accepted residuals, each with a stated reason;
  - 1 was a real gap, the sweep ownership check above, which is fixed here.
- `SECURITY.md` is linked from the README documentation map.

### Not changed, by decision
- No finding was addressed by requiring MFA. MFA stays optional.

## Smoke and verification tooling

- `verify_deployment.sh --only` runs write a separate PARTIAL inventory and leave the full-run inventory untouched.
- The deployed stack's refusal of an absent `principal_type` is smoke-tested on both channels.
- The CloudFront smoke now:
  - matches inline scripts case-insensitively in its CSP check;
  - asserts the served HSTS header on every probed behaviour.
- The allowlist key's IAM rows are simulated on the identity side only. The policy simulator rejects a KMS key policy as a resource policy, so the key policy itself stays asserted structurally.
- The plan fixtures resolve the allowlist key, and they are recaptured for the new infrastructure.

## Documentation

- `README.md`, `docs/eval-methodology.md`, `docs/runbook.md`, `docs/development.md` and `docs/security-baseline.md` cover:
  - the per-call cost model and the prompt limits;
  - `--max-cost-usd` and offline re-pricing;
  - `CRM_EXPECTED_ACCOUNT_ID`;
  - the PARTIAL inventory;
  - Hosted UI sign-out;
  - the allowlist key;
  - the in-band errors the live wrapper retries.
- The user-plane and network figures show the allowlist key and Hosted UI sign-out.
- New `SECURITY.md`, described above.
- `results/matrix.md` is re-rendered from the closing live sweep's stored records.

## Testing

- `make verify` (lint, format, type-check, unit, stub, integration, gates, shellcheck): green. Unit tests: 16,325 passed. Plan-time security baseline: 1,458 checks, 0 failed.
- Local reproduction of the GitLab CI jobs on a fresh clone of the branch: green. The frontend job runs on linux/amd64 and the verify job on linux/arm64.
- Against a deployed dev stack:
  - targeted post-deploy smoke of the changed resources, with 0 failures beyond previously documented deferrals;
  - the stub sweep through the deployed state machine: 38 of 38 checks passed, covering the rehearsal, cancellation, tenant teardown and IAM role simulations;
  - the CloudFront smoke confirmed the HSTS header on every behaviour.
- A live sweep ran across both model families, 5 repeats of every golden task, under a $30 cap:
  - it cost $4.45;
  - the `demo` tenant was untouched, and every write landed in a disposable tenant;
  - all artefacts were asserted;
  - `results/matrix.md` is its re-render, byte-identical to the sweep's own output.
- A run losing its ownership mid-sweep cannot be triggered on demand against a live stack. Unit tests and tests that execute the state-machine definition cover that path.

## Upgrade notes

- Run `scripts/package_lambdas.sh` and `scripts/package_relay.sh`, then `scripts/deploy.sh`.
  - Packaging refuses first-party source that differs from the tracked tree.
  - `deploy.sh` needs `CRM_EXPECTED_ACCOUNT_ID`.
- The apply creates the allowlist KMS key, converts the allowlist parameter to `SecureString`, and updates the Lambdas, the `eval_sweep` state machine and the CloudFront headers policy.
- `python -m src.eval.harness --live` now exits 2 without a positive `--max-cost-usd`.
- A direct invoke on the `gateway` or `agent` channel without `principal_type` is now refused with `401`.

## Known limitations

- No AWS WAF, and no per-principal rate or spend quota.
- CloudFront's minimum TLS version needs a custom ACM certificate.
- CloudTrail has no alarms.
- No SCA job runs in CI.
- Hosted UI sign-out is dormant on the deployed SPA, which has no Hosted UI domain configured.
- The Runtime image still carries `src/relay/node_modules`; its packaging needs the same exclusion the Lambda zips have.
Comment thread scripts/smoke/cloudfront.py Dismissed
@khanrubd
khanrubd merged commit fcfca10 into main Nov 11, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants