Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 12 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@ This is a demo and a research instrument, not a production CRM.

![Concept overview: the React SPA (served from S3 via CloudFront), external MCP clients and the in-app Strands agents reach one enforcement boundary in the CRM action Lambdas through the apigw, gateway and agent identity channels; decode_identity, the input validator, authz and store.py lead to tenant-partitioned DynamoDB; the toolspec generates the validator, the MCP tool definitions and the agent tool registry; the eval run_sweep drives agent turns and writes results](docs/figures/readme-concept-overview.svg)

![AWS user plane: the browser runs the React SPA it downloads through CloudFront. CloudFront's default behaviour serves the built SPA's static files from a private S3 bucket (OAC, SSE-KMS), which is where the UI is hosted; CloudFront also sends /api/* through a VPC origin, internal ALB and execute-api endpoint to the private REST API and the CRM action Lambdas on DynamoDB, which read their API client allowlist from Parameter Store, and sends /agent/* to the relay Lambda, the AgentCore Runtime and Amazon Bedrock; every behaviour carries an enforced security-headers and CSP policy; Cognito signs users in](docs/figures/readme-aws-user-plane.svg)
![AWS user plane: the browser runs the React SPA it downloads through CloudFront. CloudFront's default behaviour serves the built SPA's static files from a private S3 bucket (OAC, SSE-KMS), which is where the UI is hosted; CloudFront also sends /api/* through a VPC origin, internal ALB and execute-api endpoint to the private REST API and the CRM action Lambdas on DynamoDB, which read their API client allowlist, a SecureString, from Parameter Store, and sends /agent/* to the relay Lambda, the AgentCore Runtime and Amazon Bedrock; every behaviour carries an enforced security-headers and CSP policy; Cognito signs users in, and signing out also ends the Cognito Hosted UI session](docs/figures/readme-aws-user-plane.svg)

![AWS agent plane: MCP clients get a Cognito token and call two AgentCore Gateways; the CRM tools Gateway runs an identity interceptor and invokes one action Lambda per tool, the chat Gateway runs its own interceptor and invokes chat_with_supervisor, which calls the AgentCore Runtime; the Runtime invokes the action Lambdas and Bedrock models and pulls its image from ECR, built by CodeBuild; eval_trigger starts the eval_sweep state machine, whose sweep_task sends eval turns to the Runtime and whose sweep_control writes artefacts to S3](docs/figures/readme-aws-agent-plane.svg)

Expand Down Expand Up @@ -408,11 +408,17 @@ instruments. `docs/eval-methodology.md` describes the bundle and the prompt-cach

# Live mode: billable. Real models, the deployed tables and action Lambdas, a disposable tenant.
EVAL_SUBSTRATE=live-aws CRM_TABLE_PREFIX=crm-agent-eval ACTION_LAMBDA_PREFIX=crm-agent-eval \
.venv/bin/python3 -m src.eval.harness --live --assert-artifacts
.venv/bin/python3 -m src.eval.harness --live --assert-artifacts --max-cost-usd <usd>
```

`--live` without `EVAL_SUBSTRATE=live-aws` (or `--substrate live-aws`) exits 2. Other flags include
`--dataset-seed`, `--repeats` and `--archive-raw`. `docs/eval-methodology.md` defines each metric.
`--live` without `EVAL_SUBSTRATE=live-aws` (or `--substrate live-aws`) exits 2. `--max-cost-usd` is
required with `--live`: before each (arm, repeat) unit, a sweep whose priced spend plus its largest
completed unit would exceed the cap stops, tears down its tenants, writes no matrix and exits 4.
Other flags include `--dataset-seed`, `--repeats` and `--archive-raw`. A sweep archived with
`--archive-raw` can be re-priced against the current price table without calling a model:
`.venv/bin/python3 -m src.eval.reprice --raw-dir <sweep>/raw --output-dir <out>`.
`docs/eval-methodology.md` defines each metric, the per-call pricing and the per-request prompt
limits.

## Teardown

Expand Down Expand Up @@ -535,6 +541,8 @@ docs/ guides
- [`docs/runbook.md`](docs/runbook.md): build, deploy, verify, seed, troubleshoot and tear down
- [`docs/security-baseline.md`](docs/security-baseline.md): the security controls the deployed stack
must satisfy
- [`SECURITY.md`](SECURITY.md): the threat model, OWASP mappings, accepted risks and how to report
a vulnerability
- [`docs/relay-artefact.md`](docs/relay-artefact.md): building and staleness-checking the relay zip

**Develop**
Expand Down
226 changes: 226 additions & 0 deletions SECURITY.md

Large diffs are not rendered by default.

8 changes: 8 additions & 0 deletions config/models.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,14 @@ families:
model: openai.gpt-6-luna
tuning: { effort: low }

# Largest estimated prompt, in tokens, a single model request may carry at each tier. Each tier is
# an independent setting; a request over its tier's limit is refused before it is sent. Every limit
# must fit the context window of every model in model_capabilities.yaml.
request_prompt_limits:
supervisor: 270000
subagent: 270000
tool: 95000

# All inference goes through the bedrock-runtime endpoint, which serves the Messages API and the
# Responses API. bedrock-mantle is not used.
endpoint: bedrock-runtime
Expand Down
65 changes: 59 additions & 6 deletions config/pricing.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,11 @@
# Versioned token price table. Cost is computed from recorded token counts against this table,
# never estimated or read from a live pricing API at measurement time.
#
# Rates are the global-route per-1M-token prices for the models this system uses. While
# Rates are the global-route per-1M-token prices for the models this system uses. A model's optional
# `long_context` block replaces its standard rates for a whole request (every token class, output
# included) whose prompt size (input + cache read + cache write) exceeds `above_prompt_tokens`; a
# request exactly at the threshold is standard. Every rate, standard or long-context, is multiplied by
# the `routes` multiplier of the route its inference profile resolves to. While
# `provisional: false`, absolute currency figures may appear in published artifacts;
# scripts/check_cost_withheld.sh fails the build if `provisional` is `true` and a figure appears
# in results/ or README.md.
Expand All @@ -16,12 +20,20 @@
# cache-write rate is priced here. The OpenAI models publish only a 30-minute cache-write figure,
# which is priced for them.

schema_version: 1
price_table_version: "1.2.0"
schema_version: 2
price_table_version: "1.3.0"
provisional: false
currency: USD
unit: per_1m_tokens

# Route multipliers applied to every rate. `global.` profiles are the global route; the geographic
# prefixes recognised by src/crm/inference_route.py are the geo route.
routes:
global:
multiplier: 1.0
geo:
multiplier: 1.10

sourced_from: >-
Global-route on-demand Standard rates, short context. Opus 5.5, Sonnet 5.5, Haiku 5.5 and GPT-6
Astra from the live AWS Price List: `aws pricing get-products --profile default --region
Expand All @@ -35,11 +47,21 @@ sourced_from: >-
Bedrock model card (model-card-openai-gpt-6-1-terra.html; Global CRIS, short context), read
2026-10-27. The openai tier defaults are Sol, Terra and Luna; Astra stays priced as a
selectable non-default.
Long-context rates (Haiku 5.5 above 100,000 prompt tokens; GPT-6.1 Sol, GPT-6.1 Terra, GPT-6
Luna and GPT-6 Astra above 272,000) and the geo route multiplier (1.10, Geo/In-region cross-Region
inference) from the AWS Bedrock model cards (model-card-openai-gpt-6-1-sol.html,
model-card-openai-gpt-6-1-terra.html, model-card-openai-gpt-6-luna.html, and the GPT-6 Astra and
Claude Haiku, Sonnet and Opus 5.5 model cards) and
the Amazon Bedrock pricing page (Global and Geo/In-region cross-Region inference, US East (Ohio)),
read 2026-11-08. The Haiku 5.5 and GPT-6 Astra long-context rates are as provided by the
repository owner from the Bedrock pricing page. Sonnet 5.5 and Opus 5.5 carry no long-context
tier.
Transcribed by hand; see docs/eval-methodology.md.
sourced_on: "2026-10-27"
sourced_on: "2026-11-08"
sourced_by: "repository owner"

# Per-1M-token rates, global route. cache_write is the 5-minute (Anthropic) / 30-minute (GPT-6) rate.
# Per-1M-token rates, global route, standard context unless under `long_context`. cache_write is
# the 5-minute (Anthropic) / 30-minute (GPT-6) rate.
models:
anthropic.claude-opus-5-5:
input: 4.00
Expand All @@ -56,28 +78,59 @@ models:
output: 0.50
cache_write: 0.125
cache_read: 0.01
long_context:
above_prompt_tokens: 100000
input: 0.50
output: 2.50
cache_write: 0.625
cache_read: 0.05
openai.gpt-6-astra:
input: 10.00
output: 50.00
cache_write: 12.50
cache_read: 1.00
long_context:
above_prompt_tokens: 272000
input: 20.00
output: 75.00
cache_write: 25.00
cache_read: 2.00
openai.gpt-6.1-sol:
input: 2.00
output: 10.00
cache_write: 2.50
cache_read: 0.10
long_context:
above_prompt_tokens: 272000
input: 4.00
output: 15.00
cache_write: 5.00
cache_read: 0.20
openai.gpt-6.1-terra:
input: 1.00
output: 5.00
cache_write: 1.25
cache_read: 0.05
long_context:
above_prompt_tokens: 272000
input: 2.00
output: 7.50
cache_write: 2.50
cache_read: 0.10
openai.gpt-6-luna:
input: 0.10
output: 0.50
cache_write: 0.125
cache_read: 0.01
long_context:
above_prompt_tokens: 272000
input: 0.20
output: 0.75
cache_write: 0.25
cache_read: 0.02

# Reasoning and thinking tokens are billed as output tokens and recorded separately.
# Reasoning and thinking tokens are billed as output tokens and are counted inside output_tokens;
# they are not recorded separately.
reasoning_tokens_billed_as: output

# Per-model prompt-cache minimum checkpoint sizes live in model_capabilities.yaml and are
Expand Down
2 changes: 1 addition & 1 deletion docs/development.md
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@ git config core.hooksPath githooks # or: pre-commit install
- The git hooks run `check_no_account_id`, `check_no_vendor_keys`, `check_log_redaction` and
gitleaks against staged content, then `check_secret_scan` over the working tree. Any non-zero
`check_secret_scan` status fails the hook, including 3 (gitleaks absent).
- Figures under `docs/figures/` are drawn and exported with draw.io desktop 32.5.5 (Homebrew
- Figures under `docs/figures/` are drawn and exported with draw.io desktop 32.6.10 (Homebrew
cask `drawio`, whose CLI is `drawio`), using `drawio -x -f svg --embed-svg-images --theme light`;
see [`docs/figures/README.md`](figures/README.md).
- Security-relevant pins: `cryptography` 50.0.2, `PyJWT` 2.15.1, `urllib3` 2.8.0 and
Expand Down
Loading
Loading