From acdf914c9dfb8ada053398b03f7f900b88b5cab8 Mon Sep 17 00:00:00 2001 From: Rubab Khan Date: Tue, 10 Nov 2026 20:12:25 -0800 Subject: [PATCH] feat: per-call cost model, prompt limits, identity hardening, security-scan remediation and threat model MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Summary This commit carries five workstreams: 1. **Cost model.** Every model call is priced individually, with long-context tiers and a geo-route multiplier. Arms report all-tier cost totals, and stored runs can be re-priced offline. 2. **Prompt limits and cost caps.** Oversized prompts are refused per tier rather than trimmed, and live sweeps require a cost cap. 3. **Identity and access hardening.** The identity envelope fails closed when `principal_type` is missing. The API client allowlist moves to a `SecureString` under its own KMS key. HSTS gains `includeSubDomains` and `preload`. The eval sweep re-checks run ownership on every step that writes. 4. **Security review.** - Every finding from an automated security scan was fixed or suppressed in code with a reason; none remain open. - A new `SECURITY.md` holds the project's threat model, with OWASP Top 10 (2025) and OWASP Top 10 for LLM Applications (2025) mappings. - An external automated threat model was checked claim by claim against the code. 5. **Documentation and results.** The docs and figures cover the new cost model, limits and controls. `results/matrix.md` is refreshed from a 2-arm × 5-repeat live sweep. ## Cost model - `config/pricing.yaml` moves to schema 2: - `long_context` tiers: Haiku 5.5 above 100K input tokens, OpenAI models above 272K; - a single geo-route multiplier for cross-region inference-profile routing. - Every model call is priced individually, and its pricing tier and route are recorded. - Cached prompt tokens count toward the prompt size, and so toward the long-context threshold. - An unknown inference-profile prefix is refused rather than priced at a guess. - Each arm publishes `arm_cost_total` and `arm_long_context_invocation_count` over every tier. `model_metrics` is labelled supervisor-scoped in `matrix.md` and the eval UI. - Long-context call counts appear in the matrix renderers, the tier-comparison UI and the chat usage meter. - `python -m src.eval.reprice --raw-dir …` re-prices a stored run record offline, without a model call. It refuses a per-call archive whose per-tier counts or token sums disagree with the arm archive. ## Prompt limits and budgets - A request prompt above the tier's limit is refused, never trimmed. The limits are 270K tokens for the supervisor and subagents, and 95K for the tool tier. - The turn budget is raised to 1M tokens. - A tree-wide guard keeps the prompt-limit check on every model call path. It covers binders, an explicit `None` and aliased imports. ## Eval harness and sweeps - **Cost cap.** `--live` requires a positive `--max-cost-usd`. Before each unit, the harness stops if spent plus the largest completed unit would exceed the cap. Teardown still runs, no matrix is written, and the exit code is 4. - **Actor validation.** Golden-task actors are validated when the tasks load, before any tenant is seeded. - **Retry.** A live OpenAI turn whose stream fails with an in-band `service_unavailable` is retried and recorded as a throttle exclusion, the same as Bedrock's `ServiceUnavailableException`. It previously aborted the sweep on the first attempt. - **Run ownership.** - The `eval_sweep` state machine now passes the execution ARN to the Seed and Aggregate steps. - Both steps check run ownership before any write, as Teardown and Finalize already did. For a run another execution now owns, such as after an admin unlock, they raise `NotRunOwner`. - The error is not retried. It routes through Teardown and Finalize, which also refuse a non-owner, so the stale execution ends without touching the run. ## Identity and access ### Identity envelope - On the `gateway` and `agent` channels, an absent or invalid `principal_type` is refused with `401`; there is no default. - The relay sends `principal_type` on both turn and cancel. - The golden task that exercises the MCP path now runs as a service principal. - A tree-wide guard checks that agent-channel identity literals and actor fixtures declare `principal_type`. - Gateway interceptor denials are logged at WARNING. Exception cause walks follow `__cause__` only. ### API client allowlist - The allowlist SSM parameter is now a `SecureString` under a dedicated customer-managed KMS key, with rotation on. - Consumers may decrypt it only through SSM, under this parameter's encryption context. Each is explicitly denied decryption through SSM under any other key, `aws/ssm` included. - A parameter that is not a `SecureString` is refused, so a downgraded parameter fails closed. ### Browser and edge - Sign-out also ends the Hosted UI session (`/logout`) when the Hosted UI is enabled. - CloudFront's `Strict-Transport-Security` header adds `includeSubDomains` and `preload` to the one-year max-age. ## Security review ### Scan remediation - **Fixed in code:** - the log-redaction test mints its fake JWT at test time instead of embedding a literal; - `deploy.sh`'s placeholder loop is parsed line by line instead of with a nested-quantifier regex; - each test file uses a single interpolated documentation placeholder account id; - the metering per-million divisor is renamed. - **Suppressed in code with a reason:** non-credential fixture strings carry `# nosec`, and Semgrep false positives carry `# nosemgrep`. - The final re-scan reports no open finding at any severity. `docs/security-baseline.md` records each disposition. ### Threat model (`SECURITY.md`) - **Contents:** - assets, trust boundaries and threat actors; - the security invariants the code enforces; - 20 STRIDE threats with mitigations and residual risk; - accepted risks; - how to report a vulnerability. - **OWASP mappings:** one row per category for the OWASP Top 10 (2025) and the OWASP Top 10 for LLM Applications (2025). - **External threat model.** An automated STRIDE threat model reported 22 threats, 14 High and 8 Medium. Every claim was checked against the code: - 9 are false positives; - 12 are accepted residuals, each with a stated reason; - 1 was a real gap, the sweep ownership check above, which is fixed here. - `SECURITY.md` is linked from the README documentation map. ### Not changed, by decision - No finding was addressed by requiring MFA. MFA stays optional. ## Smoke and verification tooling - `verify_deployment.sh --only` runs write a separate PARTIAL inventory and leave the full-run inventory untouched. - The deployed stack's refusal of an absent `principal_type` is smoke-tested on both channels. - The CloudFront smoke now: - matches inline scripts case-insensitively in its CSP check; - asserts the served HSTS header on every probed behaviour. - The allowlist key's IAM rows are simulated on the identity side only. The policy simulator rejects a KMS key policy as a resource policy, so the key policy itself stays asserted structurally. - The plan fixtures resolve the allowlist key, and they are recaptured for the new infrastructure. ## Documentation - `README.md`, `docs/eval-methodology.md`, `docs/runbook.md`, `docs/development.md` and `docs/security-baseline.md` cover: - the per-call cost model and the prompt limits; - `--max-cost-usd` and offline re-pricing; - `CRM_EXPECTED_ACCOUNT_ID`; - the PARTIAL inventory; - Hosted UI sign-out; - the allowlist key; - the in-band errors the live wrapper retries. - The user-plane and network figures show the allowlist key and Hosted UI sign-out. - New `SECURITY.md`, described above. - `results/matrix.md` is re-rendered from the closing live sweep's stored records. ## Testing - `make verify` (lint, format, type-check, unit, stub, integration, gates, shellcheck): green. Unit tests: 16,325 passed. Plan-time security baseline: 1,458 checks, 0 failed. - Local reproduction of the GitLab CI jobs on a fresh clone of the branch: green. The frontend job runs on linux/amd64 and the verify job on linux/arm64. - Against a deployed dev stack: - targeted post-deploy smoke of the changed resources, with 0 failures beyond previously documented deferrals; - the stub sweep through the deployed state machine: 38 of 38 checks passed, covering the rehearsal, cancellation, tenant teardown and IAM role simulations; - the CloudFront smoke confirmed the HSTS header on every behaviour. - A live sweep ran across both model families, 5 repeats of every golden task, under a $30 cap: - it cost $4.45; - the `demo` tenant was untouched, and every write landed in a disposable tenant; - all artefacts were asserted; - `results/matrix.md` is its re-render, byte-identical to the sweep's own output. - A run losing its ownership mid-sweep cannot be triggered on demand against a live stack. Unit tests and tests that execute the state-machine definition cover that path. ## Upgrade notes - Run `scripts/package_lambdas.sh` and `scripts/package_relay.sh`, then `scripts/deploy.sh`. - Packaging refuses first-party source that differs from the tracked tree. - `deploy.sh` needs `CRM_EXPECTED_ACCOUNT_ID`. - The apply creates the allowlist KMS key, converts the allowlist parameter to `SecureString`, and updates the Lambdas, the `eval_sweep` state machine and the CloudFront headers policy. - `python -m src.eval.harness --live` now exits 2 without a positive `--max-cost-usd`. - A direct invoke on the `gateway` or `agent` channel without `principal_type` is now refused with `401`. ## Known limitations - No AWS WAF, and no per-principal rate or spend quota. - CloudFront's minimum TLS version needs a custom ACM certificate. - CloudTrail has no alarms. - No SCA job runs in CI. - Hosted UI sign-out is dormant on the deployed SPA, which has no Hosted UI domain configured. - The Runtime image still carries `src/relay/node_modules`; its packaging needs the same exclusion the Lambda zips have. --- README.md | 16 +- SECURITY.md | 226 +++++++ config/models.yaml | 8 + config/pricing.yaml | 65 +- docs/development.md | 2 +- docs/eval-methodology.md | 108 +++- docs/figures/README.md | 2 +- docs/figures/readme-aws-user-plane.drawio | 8 +- docs/figures/readme-aws-user-plane.svg | 2 +- .../security-baseline-aws-network.drawio | 2 +- .../figures/security-baseline-aws-network.svg | 2 +- docs/runbook.md | 19 +- docs/security-baseline.md | 104 ++- frontend/src/App.test.tsx | 14 + frontend/src/App.tsx | 4 + frontend/src/auth/AuthProvider.test.tsx | 137 +++- frontend/src/auth/AuthProvider.tsx | 42 +- frontend/src/auth/CognitoAuthSession.test.ts | 74 ++- frontend/src/auth/CognitoAuthSession.ts | 52 +- frontend/src/auth/hostedUi.test.ts | 53 +- frontend/src/auth/hostedUi.ts | 42 ++ .../AssistantPanel/UsageMeter.test.tsx | 74 +++ .../components/AssistantPanel/UsageMeter.tsx | 36 +- .../AssistantPanel/eventFrame.test.ts | 1 + frontend/src/pages/evals/EvalArmResults.tsx | 175 ++++-- frontend/src/pages/evals/README.md | 3 +- .../evals/TierComparisonSummary.test.tsx | 66 ++ .../evals/__fixtures__/tierComparison.json | 31 +- .../src/pages/evals/evals.armTotals.test.tsx | 136 ++++ frontend/src/pages/evals/types.ts | 14 + infra/main.tf | 169 ++++- infra/modules/frontend_cdn/cloudfront.tf | 7 +- infra/modules/lambda_action/README.md | 6 +- infra/modules/lambda_action/main.tf | 50 ++ infra/modules/lambda_action/variables.tf | 21 + infra/outputs.tf | 5 + infra/templates/eval_sweep.asl.json | 4 +- infra/user_settings.tf | 5 +- results/matrix.md | 317 +++++----- scripts/check_canary_tenant_empty.py | 4 +- scripts/check_cost_withheld.py | 72 ++- scripts/check_plan_security_baseline.py | 309 ++++++++- scripts/check_relay_thinness.py | 72 ++- scripts/seed_demo_tenant.py | 4 +- scripts/smoke/api_routes.py | 6 +- scripts/smoke/cloudfront.py | 55 +- scripts/smoke/cognito_claims.py | 130 +++- scripts/smoke/gateway_tools.py | 187 +++++- scripts/smoke/harness.py | 2 +- scripts/smoke/iam_roles.py | 526 +++++++++++++++- scripts/smoke/lambda_actions.py | 108 +++- scripts/smoke/mcp_client_compat.py | 3 +- scripts/smoke/three_tier_chat.py | 35 +- scripts/verify_deployment.sh | 22 +- src/agents/budget.py | 89 ++- src/agents/events.py | 5 +- src/agents/prompt_estimate.py | 64 ++ src/agents/runtime_entrypoint.py | 7 +- src/agents/subagent_runner.py | 44 +- src/agents/tool_actor.py | 25 +- src/agents/turn_costing.py | 95 ++- src/agents/turn_driver.py | 34 +- src/crm/envelope.py | 68 +- src/crm/inference_route.py | 47 ++ src/crm/metering.py | 81 ++- src/eval/actor.py | 45 ++ src/eval/golden_tasks.yaml | 30 +- src/eval/harness.py | 200 +++++- src/eval/live.py | 4 +- src/eval/matrix.py | 108 +++- src/eval/metrics.py | 82 ++- src/eval/pricing.py | 27 +- src/eval/reprice.py | 551 ++++++++++++++++ src/eval/runs.py | 7 + src/eval/sfn_handlers.py | 42 +- src/eval/sweep_units.py | 75 ++- src/gateway/identity_interceptor.py | 20 +- src/providers/capabilities.py | 82 ++- src/providers/openai_adapter.py | 3 +- src/relay/src/generated/events.ts | 1 + src/relay/src/runtime.ts | 21 +- src/relay/tests/index.test.ts | 2 + src/relay/tests/runtime.test.ts | 31 +- tests/e2e/browser/journeys/01-signin.spec.ts | 151 +++++ .../journeys/20-chat-cost/chat-cost.spec.ts | 35 +- .../tier-comparison.spec.ts | 11 +- tests/e2e/browser/scripts/check-executed.mjs | 15 + tests/fixtures/plans/compliant.base.ndjson | 143 +++-- .../fixtures/resolved_efforts.json | 12 + tests/integration/oracle/test_oracle_gate.py | 7 +- tests/integration/test_eval_turn_contract.py | 40 +- .../test_fault_injection_unreachable.py | 1 + .../test_gt15_service_principal.py | 136 ++++ .../test_interceptor_local_jwks.py | 5 +- tests/integration/test_listing_scope_ddb.py | 6 +- tests/integration/test_local_turn.py | 2 +- .../test_mcp_target_runtime_contract.py | 36 ++ tests/integration/test_prompt_cap_contract.py | 1 + tests/integration/test_reprice_roundtrip.py | 92 +++ tests/integration/test_sfn_asl_contract.py | 86 ++- .../test_tier_comparison_local_sweep.py | 1 + tests/unit/test_arm_tier_cost_invariant.py | 174 +++++ tests/unit/test_bad_write_rate_falsifiable.py | 7 +- tests/unit/test_budget.py | 69 +- tests/unit/test_check_canary_tenant_empty.py | 22 + tests/unit/test_check_cost_withheld.py | 148 +++++ tests/unit/test_check_relay_thinness.py | 105 +++- tests/unit/test_conversation_history_wired.py | 2 +- tests/unit/test_cost_withheld_gate.py | 96 +++ tests/unit/test_deploy_script.py | 69 +- tests/unit/test_dispatch.py | 12 +- tests/unit/test_envelope.py | 140 ++++- tests/unit/test_envelope_groups.py | 11 +- tests/unit/test_eval_actor.py | 219 +++++++ tests/unit/test_eval_live_estimate_config.py | 4 +- tests/unit/test_eval_trigger.py | 7 +- tests/unit/test_eval_trigger_authz.py | 2 + tests/unit/test_event_schema.py | 1 + tests/unit/test_event_schema_arrays.py | 1 + tests/unit/test_events.py | 11 +- .../test_forbidden_outcomes_falsifiable.py | 7 +- tests/unit/test_gateway_identity_producers.py | 338 ++++++++++ tests/unit/test_golden_task_localization.py | 7 +- tests/unit/test_handler_capabilities.py | 8 +- tests/unit/test_handler_error_logging.py | 15 +- tests/unit/test_handler_input_validation.py | 2 + tests/unit/test_handlers_apigw_proxy.py | 12 +- tests/unit/test_harness.py | 85 +++ tests/unit/test_harness_cost_cap.py | 252 ++++++++ tests/unit/test_harness_drives_provider.py | 42 +- tests/unit/test_harness_live_mode.py | 34 + tests/unit/test_harness_live_substrate.py | 23 +- .../test_harness_measurement_integrity.py | 44 +- .../unit/test_harness_multirole_rehearsal.py | 7 +- tests/unit/test_harness_unit_tenants.py | 7 +- tests/unit/test_heartbeat.py | 1 + tests/unit/test_identity_channel_shadowing.py | 14 +- tests/unit/test_identity_interceptor.py | 156 ++++- tests/unit/test_inference_route.py | 79 +++ tests/unit/test_infra_allowlist_key.py | 592 +++++++++++++++++ tests/unit/test_infra_api.py | 39 +- tests/unit/test_infra_csp.py | 13 +- tests/unit/test_infra_cursor_key.py | 6 +- tests/unit/test_infra_eval_sfn.py | 5 +- tests/unit/test_infra_inline_policy_budget.py | 7 +- tests/unit/test_infra_user_settings.py | 27 + tests/unit/test_matrix.py | 97 +++ tests/unit/test_matrix_tier_comparison.py | 118 +++- tests/unit/test_matrix_tier_summary.py | 1 + tests/unit/test_metering.py | 51 +- tests/unit/test_metering_long_context.py | 229 +++++++ tests/unit/test_metrics.py | 97 ++- tests/unit/test_model_capabilities.py | 225 +++++++ tests/unit/test_page_context_contract.py | 2 +- tests/unit/test_plan_fixtures.py | 8 +- tests/unit/test_plan_security_baseline.py | 16 +- tests/unit/test_pricing.py | 71 +++ tests/unit/test_prompt_estimate.py | 98 +++ tests/unit/test_proposal_decision_channel.py | 5 +- tests/unit/test_proposal_payload_allowlist.py | 6 +- tests/unit/test_render_agentcore_env.py | 28 +- tests/unit/test_reprice.py | 358 +++++++++++ tests/unit/test_request_prompt_limit.py | 594 ++++++++++++++++++ .../test_run_browser_journeys_container.py | 19 +- tests/unit/test_runs_estimate.py | 19 +- tests/unit/test_runtime_eval_turn.py | 1 + tests/unit/test_runtime_server.py | 21 +- tests/unit/test_runtime_server_turn.py | 9 +- tests/unit/test_runtime_snapshot_safety.py | 2 + tests/unit/test_scoring.py | 7 +- tests/unit/test_scoring_out_of_pattern_ids.py | 7 +- tests/unit/test_seed_demo_tenant.py | 20 + tests/unit/test_sfn_handlers.py | 72 ++- tests/unit/test_sfn_tier_task_roundtrip.py | 2 +- tests/unit/test_smoke_api_routes.py | 40 +- tests/unit/test_smoke_cloudfront.py | 163 ++++- tests/unit/test_smoke_cognito_claims.py | 142 ++++- tests/unit/test_smoke_gateway_tools.py | 110 ++++ tests/unit/test_smoke_harness.py | 22 + tests/unit/test_smoke_iam_roles.py | 482 +++++++++++++- tests/unit/test_smoke_lambda_actions.py | 116 ++++ tests/unit/test_smoke_mcp_client_compat.py | 44 ++ tests/unit/test_smoke_three_tier_chat.py | 40 +- tests/unit/test_store_transport_errors.py | 2 +- tests/unit/test_stream_contract.py | 14 + tests/unit/test_sweep_units.py | 121 +++- .../unit/test_teardown_deletion_protection.py | 11 +- tests/unit/test_token_accounting.py | 29 +- tests/unit/test_tool_actor.py | 5 +- tests/unit/test_turn_costing.py | 125 +++- tests/unit/test_verify_deployment.py | 70 ++- 191 files changed, 12102 insertions(+), 846 deletions(-) create mode 100644 SECURITY.md create mode 100644 frontend/src/pages/evals/evals.armTotals.test.tsx create mode 100644 src/agents/prompt_estimate.py create mode 100644 src/crm/inference_route.py create mode 100644 src/eval/actor.py create mode 100644 src/eval/reprice.py create mode 100644 tests/integration/test_gt15_service_principal.py create mode 100644 tests/integration/test_reprice_roundtrip.py create mode 100644 tests/unit/test_arm_tier_cost_invariant.py create mode 100644 tests/unit/test_eval_actor.py create mode 100644 tests/unit/test_gateway_identity_producers.py create mode 100644 tests/unit/test_harness_cost_cap.py create mode 100644 tests/unit/test_inference_route.py create mode 100644 tests/unit/test_infra_allowlist_key.py create mode 100644 tests/unit/test_metering_long_context.py create mode 100644 tests/unit/test_prompt_estimate.py create mode 100644 tests/unit/test_reprice.py create mode 100644 tests/unit/test_request_prompt_limit.py diff --git a/README.md b/README.md index 6465b92..81294a1 100644 --- a/README.md +++ b/README.md @@ -31,7 +31,7 @@ This is a demo and a research instrument, not a production CRM. ![Concept overview: the React SPA (served from S3 via CloudFront), external MCP clients and the in-app Strands agents reach one enforcement boundary in the CRM action Lambdas through the apigw, gateway and agent identity channels; decode_identity, the input validator, authz and store.py lead to tenant-partitioned DynamoDB; the toolspec generates the validator, the MCP tool definitions and the agent tool registry; the eval run_sweep drives agent turns and writes results](docs/figures/readme-concept-overview.svg) -![AWS user plane: the browser runs the React SPA it downloads through CloudFront. CloudFront's default behaviour serves the built SPA's static files from a private S3 bucket (OAC, SSE-KMS), which is where the UI is hosted; CloudFront also sends /api/* through a VPC origin, internal ALB and execute-api endpoint to the private REST API and the CRM action Lambdas on DynamoDB, which read their API client allowlist from Parameter Store, and sends /agent/* to the relay Lambda, the AgentCore Runtime and Amazon Bedrock; every behaviour carries an enforced security-headers and CSP policy; Cognito signs users in](docs/figures/readme-aws-user-plane.svg) +![AWS user plane: the browser runs the React SPA it downloads through CloudFront. CloudFront's default behaviour serves the built SPA's static files from a private S3 bucket (OAC, SSE-KMS), which is where the UI is hosted; CloudFront also sends /api/* through a VPC origin, internal ALB and execute-api endpoint to the private REST API and the CRM action Lambdas on DynamoDB, which read their API client allowlist, a SecureString, from Parameter Store, and sends /agent/* to the relay Lambda, the AgentCore Runtime and Amazon Bedrock; every behaviour carries an enforced security-headers and CSP policy; Cognito signs users in, and signing out also ends the Cognito Hosted UI session](docs/figures/readme-aws-user-plane.svg) ![AWS agent plane: MCP clients get a Cognito token and call two AgentCore Gateways; the CRM tools Gateway runs an identity interceptor and invokes one action Lambda per tool, the chat Gateway runs its own interceptor and invokes chat_with_supervisor, which calls the AgentCore Runtime; the Runtime invokes the action Lambdas and Bedrock models and pulls its image from ECR, built by CodeBuild; eval_trigger starts the eval_sweep state machine, whose sweep_task sends eval turns to the Runtime and whose sweep_control writes artefacts to S3](docs/figures/readme-aws-agent-plane.svg) @@ -408,11 +408,17 @@ instruments. `docs/eval-methodology.md` describes the bundle and the prompt-cach # Live mode: billable. Real models, the deployed tables and action Lambdas, a disposable tenant. EVAL_SUBSTRATE=live-aws CRM_TABLE_PREFIX=crm-agent-eval ACTION_LAMBDA_PREFIX=crm-agent-eval \ - .venv/bin/python3 -m src.eval.harness --live --assert-artifacts + .venv/bin/python3 -m src.eval.harness --live --assert-artifacts --max-cost-usd ``` -`--live` without `EVAL_SUBSTRATE=live-aws` (or `--substrate live-aws`) exits 2. Other flags include -`--dataset-seed`, `--repeats` and `--archive-raw`. `docs/eval-methodology.md` defines each metric. +`--live` without `EVAL_SUBSTRATE=live-aws` (or `--substrate live-aws`) exits 2. `--max-cost-usd` is +required with `--live`: before each (arm, repeat) unit, a sweep whose priced spend plus its largest +completed unit would exceed the cap stops, tears down its tenants, writes no matrix and exits 4. +Other flags include `--dataset-seed`, `--repeats` and `--archive-raw`. A sweep archived with +`--archive-raw` can be re-priced against the current price table without calling a model: +`.venv/bin/python3 -m src.eval.reprice --raw-dir /raw --output-dir `. +`docs/eval-methodology.md` defines each metric, the per-call pricing and the per-request prompt +limits. ## Teardown @@ -535,6 +541,8 @@ docs/ guides - [`docs/runbook.md`](docs/runbook.md): build, deploy, verify, seed, troubleshoot and tear down - [`docs/security-baseline.md`](docs/security-baseline.md): the security controls the deployed stack must satisfy +- [`SECURITY.md`](SECURITY.md): the threat model, OWASP mappings, accepted risks and how to report + a vulnerability - [`docs/relay-artefact.md`](docs/relay-artefact.md): building and staleness-checking the relay zip **Develop** diff --git a/SECURITY.md b/SECURITY.md new file mode 100644 index 0000000..69fe295 --- /dev/null +++ b/SECURITY.md @@ -0,0 +1,226 @@ +# Security + +See also: [documentation map](README.md#documentation) · [security baseline](docs/security-baseline.md) · [agent architecture](docs/agent-architecture.md) · [MCP client guide](docs/mcp-client-guide.md) · [runbook](docs/runbook.md) + +This page is the project's threat model: what the system protects, who might attack it and how, +which controls stand in the way, and which risks remain. It covers the OWASP Top 10 (2025) and the +OWASP Top 10 for LLM Applications (2025). The infrastructure controls themselves, each with its +verification command, live in [`docs/security-baseline.md`](docs/security-baseline.md). + +## Reporting a vulnerability + +If you discover a potential security issue in this project, notify AWS/Amazon Security via the +[vulnerability reporting page](http://aws.amazon.com/security/vulnerability-reporting/). Please do +**not** create a public GitHub issue. + +## Scope and status + +The stack is a demonstration and measurement instrument, not a production CRM (see +[Limitations](README.md#limitations)). This model assumes: + +- one AWS account, deployed with `scripts/deploy.sh` by a small set of operators; +- synthetic CRM data (the fixture data set and seeded eval tenants), with no real customer records; +- users created by an administrator, never by self sign-up; +- the AWS services themselves (Cognito, CloudFront, API Gateway, Lambda, DynamoDB, Bedrock, + AgentCore, S3, KMS) working as documented. + +Anyone deploying it with real data must revisit the accepted risks below first. + +## System overview + +```text +Browser SPA ──HTTPS──▶ CloudFront ─┬─ /* ▶ S3 origin (OAC, private) + ├─ /api/* ▶ VPC origin ▶ internal ALB ▶ private REST API ▶ action Lambdas + └─ /agent/* ▶ OAC (SigV4) ▶ relay Function URL (AWS_IAM) ▶ AgentCore Runtime +MCP client ──HTTPS──▶ AgentCore Gateway #1 (CRM tools) / #2 (chat_with_supervisor) + └─ CUSTOM_JWT authorizer ▶ REQUEST interceptor ▶ action Lambdas / chat target +AgentCore Runtime ── supervisor, subagents, tool tier ──▶ Bedrock (SigV4, no API keys) + └─▶ action Lambdas (lambda:InvokeFunction) +Admin /evals page ──▶ eval_trigger ──▶ Step Functions sweep ──▶ Runtime (eval_turn) ──▶ eval-artefacts S3 +``` + +All three entry planes, the SPA, external MCP clients and the in-app agents, end at the same CRM +action Lambdas. Authorization is decided there and nowhere else. + +### Assets + +| Asset | Where it lives | Why it matters | +|---|---|---| +| CRM records (accounts, contacts, opportunities, activities, proposals) | DynamoDB, KMS CMK, PITR | confidentiality and integrity per tenant and per rep book | +| Audit records | DynamoDB, written in the same transaction as each mutation | non-repudiation of writes and approvals | +| User identities and tokens | Cognito user pool, browser session storage, MCP client credential store | every authorization decision rests on them | +| MCP client secret | the file written by `scripts/rotate_mcp_client_secret.py` | admits a machine caller to both Gateways | +| Model spend and quota | Bedrock in the deploying account | direct cost and availability | +| Eval artefacts (matrices, transcripts, raw records) | eval-artefacts S3 bucket, KMS CMK | integrity of published measurements | +| Deployment authority | operator roles, Terraform state bucket and lock table | controls every other asset | + +### Trust boundaries + +1. **Internet to edge.** CloudFront and the two Gateway endpoints are the only internet-facing + surfaces. No security group admits internet ingress, and the API, relay and Runtime are reachable + only through these edges. +2. **Browser to backend.** Everything in the browser is untrusted. Client-side checks (tuning + validation, admin-only UI controls, decoded identity) are presentation only; the backend repeats + every check. +3. **Edge to identity.** Each edge verifies a Cognito JWT itself and then re-derives the caller's + identity. Identity is never forwarded as the caller sent it. +4. **Model to tools.** Model output is untrusted. The tool tier's arguments are validated against + the generated toolspec, and identity keys written by the model are stripped before dispatch. +5. **Account to model provider.** Prompts, page context and tool results leave the workload for + Bedrock over SigV4. +6. **Operator to account.** Operators hold deployment and data authority outside every in-app + control. Resource policies narrow which principals may touch each data store. + +### Threat actors + +| Actor | Starting position | +|---|---| +| Anonymous internet client | can reach CloudFront and the Gateway endpoints; holds no token | +| Signed-in rep | valid SPA token for one tenant and one book; may be curious or malicious | +| Stolen-token holder | replays a rep or machine token until it expires | +| MCP client operator | holds the machine client secret | +| Content author | plants text in CRM fields that an agent later reads (indirect prompt injection) | +| Compromised dependency | runs inside the build, the browser or a Lambda | +| In-account principal | has IAM rights in the deploying account but is not an operator | + +## Security design invariants + +These properties are enforced in code and checked by tests or gates. They are what the threat +table below relies on. + +- **One enforcement boundary.** The action Lambdas decide authorization through one scope predicate + (`src/crm/authz.py`) used by get, list, search and aggregates alike. A record hidden from `get` + cannot leak through a rollup. +- **Identity is channel-decoded, never constructed.** `src/crm/envelope.py:decode_identity` selects + exactly one of three channels (`apigw`, `gateway`, `agent`) from top-level event keys. There is no + default; an identity that cannot be established is refused. On `apigw`, the token's app client must + be in an SSM `SecureString` allowlist, and an unreadable or empty allowlist admits nobody. +- **Every hop strips and rewrites identity.** The relay replaces the caller's claims with its own + verified `_agent_identity`. The Gateway interceptor removes every identity-channel key from tool + arguments before writing its own. The tool tier removes identity keys the model wrote. +- **Tenant isolation is structural.** Every GSI partition key is tenant-prefixed and every index + projects `tenant` (`src/crm/store/keys.py`, `src/crm/store/schema.py`). Eval sweeps run in + disposable `eval-` tenants and may not write `demo` or `canary`. +- **No direct high-risk writes.** High-risk changes go through `submit_proposal` and a human approve + or reject in the UI. The agent channel and service principals cannot approve, reject or revert. The + proposal card renders from the stored proposal, not from model text. +- **No vendor or Bedrock API keys.** Model calls are SigV4-signed from the workload role. The SDK + bearer-token fallbacks are refused, and `scripts/check_no_vendor_keys.sh` and gitleaks gate the + tree. +- **Fail closed.** No raw `botocore` exception crosses a module boundary (`src/crm/errors.py`). + Missing configuration, allowlists, cursor keys or scope attributes refuse the request rather than + widen it. + +## Threats and mitigations + +Severity reflects this deployment's scope: admin-provisioned users and synthetic data in one +account. + +| Id | STRIDE | Threat | Mitigations | Residual and severity | +|---|---|---|---|---| +| TM-01 | Spoofing | Forged or unsigned JWT at an edge | relay, REST authorizer and Gateway interceptor each verify the RS256 signature against the pool JWKS, expiry and app client; Gateway `CUSTOM_JWT` checks allowed clients and audiences | none known; Low | +| TM-02 | Spoofing | Token stolen from the browser or a client store and replayed | token expiry; refresh-token revocation at sign-out; enforced CSP with no `unsafe-inline` script; markdown rendered without raw HTML; each surface accepts only its own app client | a stolen token works until expiry; Medium | +| TM-03 | Spoofing | Caller-supplied `_identity` or `_agent_identity` in a request body | channel chosen from top-level event keys; interceptor and tool tier strip identity keys; Runtime resource policy admits only the relay, sweep and chat-target roles | in-account principals with `lambda:InvokeFunction` on action Lambdas can hand-build an envelope, which the local live harness does by design; Low | +| TM-04 | Spoofing | Self-registration or username chosen to collide with a rep | `allow_admin_create_user_only`; pre-token trigger keys on the immutable Cognito username; only operators write `reps` rows | operator misuse; Low | +| TM-05 | Tampering | Cross-tenant or out-of-book read or write | structural tenant keys; `require_scope` / `require_record_scope` in every handler touching an existing record; caller-bound KMS-encrypted pagination cursors | none known; Low | +| TM-06 | Tampering | Indirect prompt injection steers a write | proposals for high-risk changes; human approval bound to the stored proposal; writes limited to the rep's own book; per-turn tool-call and time budgets | injected text can still steer low-risk writes inside the rep's book; Medium | +| TM-07 | Tampering | Tampered SPA assets or `/config.json` | origin bucket private, OAC-only, writes denied to all but operators; CSP `connect-src` pins API and Cognito origins | anyone who can write the origin can replace the JavaScript too; Low | +| TM-08 | Tampering | Eval sweep writes outside its run | disposable tenants; `demo`/`canary` writes refused; every sweep step that writes (seed, aggregate, teardown, finalize) first re-checks that its execution still owns the run, and a refused step ends the execution without writing | none known; Low | +| TM-09 | Repudiation | A user denies a write or approval | audit row in the same transaction as each mutation, naming the verified rep; API Gateway access logs; CloudTrail | audit rows omit the entry channel; MCP callers share one service principal; relay log lines carry no rep id; Low | +| TM-10 | Information disclosure | Bulk extraction through the agent | scope predicate on every read; per-call result and read caps; per-turn tool-call cap | a rep can summarise their whole book, which they may already read; Low | +| TM-11 | Information disclosure | CRM data sent to the model provider | Bedrock in the deploying account over SigV4; data retention governed by the account's Bedrock setting; synthetic data | no per-field redaction, which would break the measurement; Low with synthetic data, revisit for real data | +| TM-12 | Information disclosure | Leaked artefact URL | presigned GET of one object for 300 s, issued only to admins or full-visibility users; keys built server-side from the run id, `..` refused; S3 access logs | bearer link until expiry; Low | +| TM-13 | Information disclosure | Session or conversation takeover | session ownership (rep and tenant) checked before resume or cancel; MCP conversation ids minted server-side | all holders of the one MCP secret share a principal; process-local session state fails closed on another container; Low | +| TM-14 | Denial of service | Request or cost flooding by a signed-in caller | API Gateway stage throttle; reserved concurrency on every function; per-turn token, tool-call and time budgets; prompt length cap; no self sign-up | no per-principal quota or spend ceiling, and no WAF; Medium | +| TM-15 | Denial of service | Runaway or repeated eval sweeps | live sweeps admin-only with a confirmed estimate; daily live cap; one sweep at a time; sweep tasks run sequentially; dead lock holders released on the next submit | no hard dollar ceiling on the state-machine live path; Low | +| TM-16 | Denial of service | Dependency outage at sign-in or on the request path | pre-token trigger falls back to an empty book without full visibility (least privilege); JWKS and allowlist cached per warm environment and retried after failure | a failed reps lookup logs at INFO with no alarm; Low | +| TM-17 | Elevation of privilege | Machine token with broad scope | interceptor maps every machine token to one service principal limited to the demo tenant and without full visibility; service principals cannot decide proposals | one principal for every MCP client; Medium if a second client is ever added | +| TM-18 | Elevation of privilege | Non-admin drives privileged eval actions | live submit, cancel and unlock need the admin group; stub sweeps and run reads need admin or full visibility, enforced server-side | none beyond the documented policy; Low | +| TM-19 | Elevation of privilege | Compromised dependency or build input | exact pins with a frozen lock installed `--no-deps`; one lockfile per npm project; SDKs only in the Runtime image; gitleaks and vendor-key gates; dev mock server applied only to `vite serve` and absent from the build | no CI job runs `pip-audit`, `npm audit` or other SCA; Medium | +| TM-20 | Information disclosure | Bearer headers on internal hops | CloudFront to API traffic stays inside the VPC on a VPC origin; the ALB is internal and admits only security groups; ALB to execute-api is HTTPS | the CloudFront-to-ALB hop is HTTP inside the VPC, with no certificate to terminate TLS there; Low | + +## OWASP Top 10 (2025) + +| Category | How it applies | Controls | Residual | +|---|---|---|---| +| A01 Broken Access Control | tenant, book and role scope on every record; admin-only eval actions | one scope predicate in the action Lambdas; structural tenant keys; channel-decoded identity; authorizer on every API route; resource policies on every data store | TM-03, TM-17 | +| A02 Security Misconfiguration | CloudFront, S3, API Gateway, IAM and KMS configuration | plan-time `check_plan_security_baseline`; Layer 4–6 live smoke; private origins with Block Public Access; no internet-facing ingress; enforced CSP and HSTS | no WAF; CloudFront minimum TLS deferred until a custom certificate is configured | +| A03 Software Supply Chain Failures | Python, npm and container dependencies | exact pins and frozen locks (`check_pin_drift`); `--no-deps` installs; lockfile per npm project; SDKs confined to the Runtime image | no SCA in CI (TM-19) | +| A04 Cryptographic Failures | data at rest and in transit | KMS CMKs with rotation on tables, buckets, logs and parameters; TLS-only bucket policies; HTTPS at the viewer; KMS-encrypted caller-bound cursors | TM-20; viewer TLS minimum deferred | +| A05 Injection | DynamoDB expressions, markdown rendering, shell in scripts | DynamoDB calls only through `src/crm/store/store.py` with expression attribute values; generated input validators with `maxLength`; markdown without raw HTML; `shellcheck` over every shell file | CRM strings accept control characters | +| A06 Insecure Design | agent writes and delegation | proposals for high-risk writes; four exclusive delegation outcomes so a denial is never reported as data; measurement-only eval with no verdict | TM-06 | +| A07 Authentication Failures | Cognito sign-in, machine tokens | admin-only sign-up; password policy; MFA available (TOTP, optional); token revocation; per-surface app-client allowlists | MFA is optional by design; TM-02 | +| A08 Software or Data Integrity Failures | SPA assets, eval artefacts, toolspec consumers | operator-only origin writes; generated consumers checked for parity (`check_toolspec_parity`, `check_sse_schema_parity`); fixture data regenerated byte for byte | artefacts are not signed | +| A09 Security Logging and Alerting Failures | detection and attribution | CloudTrail; API Gateway access logs; KMS-encrypted log groups with retention; `check_log_redaction` | no CloudTrail metric filters or alarms; TM-09 | +| A10 Mishandling of Exceptional Conditions | AWS errors, misconfiguration | error taxonomy at every module boundary; handler failures returned as classified payloads; fail-closed allowlist, cursor and identity paths | TM-16 | + +## OWASP Top 10 for LLM Applications (2025) + +| Category | How it applies | Controls | Residual | +|---|---|---|---| +| LLM01 Prompt Injection | rep prompts, MCP arguments and CRM text returned as tool output | the model never holds authority: identity, tenant and scope come from verified tokens; tool arguments validated against the toolspec; high-risk writes need human approval | TM-06 | +| LLM02 Sensitive Information Disclosure | CRM data in prompts and answers | answers can only contain what scoped tools returned; images in answers are never fetched, so markdown cannot exfiltrate through a URL; log redaction | TM-11 | +| LLM03 Supply Chain | model access and SDKs | models reached only through Bedrock inference profiles in the account; pinned vendor SDKs in the Runtime image only; model ids in `config/model_capabilities.yaml` | as A03 | +| LLM04 Data and Model Poisoning | golden tasks and seeded data | no fine-tuning or training; golden tasks and fixtures are tracked and reviewed; eval tenants are disposable | planted CRM text is handled as LLM01 | +| LLM05 Improper Output Handling | model output rendered in the SPA and passed to tools | markdown without raw HTML, `https:` and relative links only, no image fetch; tool calls schema-validated; SSE frames generated from one schema | none known | +| LLM06 Excessive Agency | agent write tools | role-scoped subagents with only the tools they need; no tool commits a high-risk change; the agent cannot approve, reject or revert; per-turn tool-call and time budgets | low-risk writes in the rep's own book (TM-06) | +| LLM07 System Prompt Leakage | supervisor and subagent prompts | prompts hold no secret or authority and are published in `config/agent_prompts/`; authorization never depends on prompt text | none | +| LLM08 Vector and Embedding Weaknesses | not applicable | the system has no vector store or retrieval index | none | +| LLM09 Misinformation | fabricated CRM facts in answers | delegation outcomes are typed (`data`, `denied`, `tool_failure`, `no_match`), so a denial or failure is never turned into data; the UI shows the tool trace beside the answer | a model can still misstate data it was given | +| LLM10 Unbounded Consumption | token and cost exhaustion | per-request prompt limit (refused, not trimmed); per-turn token, tool-call and time budgets; eval cost cap and daily live cap; throttles and reserved concurrency | TM-14, TM-15 | + +## Accepted risks and open items + +The residuals named in the tables above are summarised here. The infrastructure deferrals are also +listed under [Limitations](README.md#limitations). + +- No AWS WAF, and no per-principal rate or spend quota (TM-14). +- CloudFront minimum TLS 1.2 needs a custom ACM certificate (see the security baseline's Transport + row). +- MFA stays optional by owner policy. Deployers handling real data should require it. +- No SCA job in CI (TM-19). +- CloudTrail has no alarms; audit rows do not record the entry channel (TM-09). +- One shared MCP service principal (TM-17). +- Prompts and tool results reach Bedrock unredacted; acceptable only for synthetic data (TM-11). + +## External threat model review + +An automated STRIDE threat model of this repository, run on 2026-11-10, reported 22 threats (14 High, +8 Medium). Each claim was checked against the code. None reached High once checked: most rest on a +premise the code does not bear out, and the real residuals are recorded above. + +| Reported threat | Disposition | Here | +|---|---|---| +| Tampered `/config.json` or SPA origin | false positive: operator-only writes, CSP pins origins, other pools' tokens refused | TM-07 | +| Dev server code execution or flooding | false positive: the dev server binds to localhost and runs only under `vite serve` | TM-19 | +| Eval writes outside the run tenant | mostly false; the one real gap, missing ownership checks in the seed and aggregate steps, is fixed | TM-08 | +| Non-admin drives eval actions | false positive: live, cancel and unlock are admin-only; app client checked | TM-18 | +| Flooding with no per-principal throttle | accepted: aggregate throttles and budgets bound it | TM-14 | +| CRM data sent to hosted models | accepted for synthetic data | TM-11 | +| Bulk extraction and tool enumeration | accepted: scoped, capped, authenticated | TM-10 | +| Session or conversation hijack | accepted: ownership checked; MCP ids minted server-side | TM-13 | +| Chosen username or reps-table writes | false positive: admin-only sign-up, immutable username | TM-04 | +| Book scope not enforced centrally | false positive: every handler that touches a record checks scope | TM-05 | +| Prompt injection into writes and proposals | accepted for low-risk writes; proposal claims false | TM-06 | +| Broad MCP service principal; caller `_identity` | accepted; caller `_identity` is stripped | TM-17 | +| Unsigned identity envelopes | accepted: in-account IAM only; body `_identity` never selects a channel | TM-03 | +| Token replay; audience; MFA; plaintext hops | accepted: per-surface client allowlists; MFA optional by policy | TM-02, TM-20 | +| Repudiation of writes and approvals | accepted: audit names the verified rep | TM-09 | +| Dev stub `alg: none` tokens | false positive: dev-only, absent from the build | TM-19 | +| Tampered run row or artefact keys | accepted: keys built server-side, `..` refused | TM-12 | +| Leaked presigned URL | accepted: 300 s, one object | TM-12 | +| Maximal or repeated sweeps | accepted: no fan-out, daily cap, lock release | TM-15 | +| Crafted settings or CRM payloads | accepted: strict server parsing, length caps | A05 | +| `transaction_key` collision | false positive: keys are random per action; a replay returns success | TM-09 | +| Cold-start floods, outages, fail-open trigger | accepted: the trigger falls back to least privilege | TM-16 | + +## Verifying the controls + +- `make verify` runs the offline gates (`scripts/run_gates.py`), including the plan-time security + baseline, secret and vendor-key scans, log redaction, toolspec and SSE schema parity, and the + identity, authorization and tenancy unit tests. +- `make verify-deploy` runs the live Layer 4–6 smoke against a deployed stack: IAM simulation, + denied probes against each data store, authorized and unauthenticated calls on every route, and + CloudFront headers. It is billable. +- [`docs/security-baseline.md`](docs/security-baseline.md) lists each infrastructure control with + its verification command and class. diff --git a/config/models.yaml b/config/models.yaml index bc0e9eb..bcae6b8 100644 --- a/config/models.yaml +++ b/config/models.yaml @@ -39,6 +39,14 @@ families: model: openai.gpt-6-luna tuning: { effort: low } +# Largest estimated prompt, in tokens, a single model request may carry at each tier. Each tier is +# an independent setting; a request over its tier's limit is refused before it is sent. Every limit +# must fit the context window of every model in model_capabilities.yaml. +request_prompt_limits: + supervisor: 270000 + subagent: 270000 + tool: 95000 + # All inference goes through the bedrock-runtime endpoint, which serves the Messages API and the # Responses API. bedrock-mantle is not used. endpoint: bedrock-runtime diff --git a/config/pricing.yaml b/config/pricing.yaml index b69c5b4..1b6629a 100644 --- a/config/pricing.yaml +++ b/config/pricing.yaml @@ -3,7 +3,11 @@ # Versioned token price table. Cost is computed from recorded token counts against this table, # never estimated or read from a live pricing API at measurement time. # -# Rates are the global-route per-1M-token prices for the models this system uses. While +# Rates are the global-route per-1M-token prices for the models this system uses. A model's optional +# `long_context` block replaces its standard rates for a whole request (every token class, output +# included) whose prompt size (input + cache read + cache write) exceeds `above_prompt_tokens`; a +# request exactly at the threshold is standard. Every rate, standard or long-context, is multiplied by +# the `routes` multiplier of the route its inference profile resolves to. While # `provisional: false`, absolute currency figures may appear in published artifacts; # scripts/check_cost_withheld.sh fails the build if `provisional` is `true` and a figure appears # in results/ or README.md. @@ -16,12 +20,20 @@ # cache-write rate is priced here. The OpenAI models publish only a 30-minute cache-write figure, # which is priced for them. -schema_version: 1 -price_table_version: "1.2.0" +schema_version: 2 +price_table_version: "1.3.0" provisional: false currency: USD unit: per_1m_tokens +# Route multipliers applied to every rate. `global.` profiles are the global route; the geographic +# prefixes recognised by src/crm/inference_route.py are the geo route. +routes: + global: + multiplier: 1.0 + geo: + multiplier: 1.10 + sourced_from: >- Global-route on-demand Standard rates, short context. Opus 5.5, Sonnet 5.5, Haiku 5.5 and GPT-6 Astra from the live AWS Price List: `aws pricing get-products --profile default --region @@ -35,11 +47,21 @@ sourced_from: >- Bedrock model card (model-card-openai-gpt-6-1-terra.html; Global CRIS, short context), read 2026-10-27. The openai tier defaults are Sol, Terra and Luna; Astra stays priced as a selectable non-default. + Long-context rates (Haiku 5.5 above 100,000 prompt tokens; GPT-6.1 Sol, GPT-6.1 Terra, GPT-6 + Luna and GPT-6 Astra above 272,000) and the geo route multiplier (1.10, Geo/In-region cross-Region + inference) from the AWS Bedrock model cards (model-card-openai-gpt-6-1-sol.html, + model-card-openai-gpt-6-1-terra.html, model-card-openai-gpt-6-luna.html, and the GPT-6 Astra and + Claude Haiku, Sonnet and Opus 5.5 model cards) and + the Amazon Bedrock pricing page (Global and Geo/In-region cross-Region inference, US East (Ohio)), + read 2026-11-08. The Haiku 5.5 and GPT-6 Astra long-context rates are as provided by the + repository owner from the Bedrock pricing page. Sonnet 5.5 and Opus 5.5 carry no long-context + tier. Transcribed by hand; see docs/eval-methodology.md. -sourced_on: "2026-10-27" +sourced_on: "2026-11-08" sourced_by: "repository owner" -# Per-1M-token rates, global route. cache_write is the 5-minute (Anthropic) / 30-minute (GPT-6) rate. +# Per-1M-token rates, global route, standard context unless under `long_context`. cache_write is +# the 5-minute (Anthropic) / 30-minute (GPT-6) rate. models: anthropic.claude-opus-5-5: input: 4.00 @@ -56,28 +78,59 @@ models: output: 0.50 cache_write: 0.125 cache_read: 0.01 + long_context: + above_prompt_tokens: 100000 + input: 0.50 + output: 2.50 + cache_write: 0.625 + cache_read: 0.05 openai.gpt-6-astra: input: 10.00 output: 50.00 cache_write: 12.50 cache_read: 1.00 + long_context: + above_prompt_tokens: 272000 + input: 20.00 + output: 75.00 + cache_write: 25.00 + cache_read: 2.00 openai.gpt-6.1-sol: input: 2.00 output: 10.00 cache_write: 2.50 cache_read: 0.10 + long_context: + above_prompt_tokens: 272000 + input: 4.00 + output: 15.00 + cache_write: 5.00 + cache_read: 0.20 openai.gpt-6.1-terra: input: 1.00 output: 5.00 cache_write: 1.25 cache_read: 0.05 + long_context: + above_prompt_tokens: 272000 + input: 2.00 + output: 7.50 + cache_write: 2.50 + cache_read: 0.10 openai.gpt-6-luna: input: 0.10 output: 0.50 cache_write: 0.125 cache_read: 0.01 + long_context: + above_prompt_tokens: 272000 + input: 0.20 + output: 0.75 + cache_write: 0.25 + cache_read: 0.02 -# Reasoning and thinking tokens are billed as output tokens and recorded separately. +# Reasoning and thinking tokens are billed as output tokens and are counted inside output_tokens; +# they are not recorded separately. reasoning_tokens_billed_as: output # Per-model prompt-cache minimum checkpoint sizes live in model_capabilities.yaml and are diff --git a/docs/development.md b/docs/development.md index 3e0a44b..f0149e7 100644 --- a/docs/development.md +++ b/docs/development.md @@ -35,7 +35,7 @@ git config core.hooksPath githooks # or: pre-commit install - The git hooks run `check_no_account_id`, `check_no_vendor_keys`, `check_log_redaction` and gitleaks against staged content, then `check_secret_scan` over the working tree. Any non-zero `check_secret_scan` status fails the hook, including 3 (gitleaks absent). -- Figures under `docs/figures/` are drawn and exported with draw.io desktop 32.5.5 (Homebrew +- Figures under `docs/figures/` are drawn and exported with draw.io desktop 32.6.10 (Homebrew cask `drawio`, whose CLI is `drawio`), using `drawio -x -f svg --embed-svg-images --theme light`; see [`docs/figures/README.md`](figures/README.md). - Security-relevant pins: `cryptography` 50.0.2, `PyJWT` 2.15.1, `urllib3` 2.8.0 and diff --git a/docs/eval-methodology.md b/docs/eval-methodology.md index ee182e4..7edf0d6 100644 --- a/docs/eval-methodology.md +++ b/docs/eval-methodology.md @@ -123,9 +123,15 @@ Billable command (real model inference plus writes to deployed tables under a di EVAL_SUBSTRATE=live-aws \ CRM_TABLE_PREFIX=crm-agent-eval \ ACTION_LAMBDA_PREFIX=crm-agent-eval \ - .venv/bin/python3 -m src.eval.harness --live --assert-artifacts + .venv/bin/python3 -m src.eval.harness --live --assert-artifacts --max-cost-usd ``` +**Cost cap.** `--max-cost-usd` is required with `--live`. Before each (arm, repeat) unit starts, the +harness compares the priced spend of the completed units plus the most expensive completed unit with +the cap. When that sum exceeds the cap, no further unit starts, every tenant is torn down, no matrix +is written, and the exit code is 4. Priced cost is computed from recorded tokens against +`config/pricing.yaml`, so it estimates the bill; the Bedrock bill is the authority. + **Account precondition.** `--live` needs the account's Bedrock data-retention mode to permit the OpenAI models; at `mode: none` every GPT-6 invocation is rejected. Check it read-only with `aws bedrock get-account-data-retention --region --no-cli-pager`. The setting is @@ -133,10 +139,14 @@ account-wide, so changing it is the account owner's decision. See `docs/runbook. **Transient failures.** `eval.live.ThrottleRetryingProvider` retries a throttled turn with bounded exponential backoff. Every retried attempt, whether retried by the vendor SDK or by this wrapper, -becomes exactly one exclusion, categorised by its cause: `throttle` for a throttle, `retry` otherwise. A -failure outside the closed exclusion taxonomy (cold start, retry, throttle) raises -`eval.live.LiveInvocationError` and aborts the sweep, so no artifact is published with an -uncategorised gap. +becomes exactly one exclusion, categorised by its cause: `throttle` for a throttle, `retry` +otherwise. The wrapper retries an in-band error by its leading code: throttle and availability +codes (Bedrock's `ThrottlingException`, `ServiceUnavailableException` and similar, and the Responses +surface's `service_unavailable`) count as `throttle`; transient server faults (Bedrock's +`InternalServerException`, `ModelErrorException`, `ModelStreamErrorException`, and the Responses +surface's `server_error`) count as `retry`. A failure outside the closed exclusion taxonomy (cold +start, retry, throttle) raises `eval.live.LiveInvocationError` and aborts the sweep, so no artifact +is published with an uncategorised gap. **Runtime coverage.** A CLI `--live` sweep records `runtime_identifier: live-aws-lambda-dispatch`: live models, live DynamoDB and real Lambda dispatch are exercised; the deployed AgentCore Runtime's @@ -148,7 +158,7 @@ own turn loop is not. A deployed sweep (the `eval_sweep` state machine) runs eac A deployed sweep (the `eval_sweep` state machine, see `docs/runbook.md`) runs each task as one `eval_turn` on the AgentCore Runtime, under the caps in `src/agents/budget.py`: at most 25 tool -calls, 200,000 tokens and a 120-second wall clock. The 120-second cap is soft. When it is reached, +calls, 1,000,000 tokens and a 120-second wall clock. The 120-second cap is soft. When it is reached, every pending model wait ends, the turn stops with `error_kind` `budget_exhausted`, and the partial turn is scored against committed state like any other, with `budget_exhausted: true` on its result. At 150 seconds (30 seconds past the soft cap) the Runtime cancels the turn at its next await and @@ -160,6 +170,29 @@ its usage arrived is counted as metered, and does not make the usage a lower bou `usage_lower_bound: true`, and each affected arm lists those task runs in `usage_lower_bound_task_ids`, with a note under its figures in `matrix.md`. +### Per-request prompt limits + +Every model request, chat or eval, is checked before it is sent against its tier's limit in +`config/models.yaml` `request_prompt_limits`: 270,000 estimated prompt tokens at the supervisor and +subagent tiers, and 95,000 at the tool tier. The three are independent settings, and each must fit +the context window (`context_window_tokens` in `config/model_capabilities.yaml`) of every model that +can be assigned to its tier; the capability matrix refuses to load otherwise. A request over its +limit is **refused, never trimmed**: nothing is sent, and the turn ends with its budget exhausted +on the `request_prompt_limit` dimension, naming the estimate, the limit and the tier. +Separately, a turn may use at most 1,000,000 input plus output tokens in total. + +The estimate is `ceil(utf8_bytes / 3)` over the compact JSON of the messages and tool schemas +handed to the model call (`src/agents/prompt_estimate.py`). It uses no tokenizer, so it is the same +for every family, and it omits framing an adapter adds afterwards (its own system blocks or +tool-choice wrappers). Error bound: English prose and JSON tokenize at about 3.5 to 4.5 bytes per +token, so for them the estimate runs about 17 to 50 percent high and refuses early: a real prompt +of roughly 180,000 to 231,400 tokens can be refused at an estimate of 270,000. Dense numerics, ids or +non-Latin text can tokenize below 3 bytes per token, so the estimate can run low and admit a request +larger than the limit. Such a request is still priced from the usage the model reports, at +long-context rates when it crosses the model's threshold. The tool limit sits below Haiku 5.5's +100,000-token threshold, so a tool call the estimate admits is priced at standard rates unless the +estimate under-counts. + ## What the models are sent Every family receives the same texts; no prompt is selected by family. @@ -203,7 +236,8 @@ unchanged. `matrix.json` carries the same figures as its first key, `tier_compar `summary_*` columns after every existing column, with each arm's values repeated on each of its task rows. `eval.matrix.tier_comparison` builds all three from the arm records. The summary tables themselves come from `eval.matrix.tier_comparison_table`, which the CLI's `matrix.md` and the eval -dashboard both render. +dashboard both render. The dashboard shows every header, cell and note as verbatim text; the only +markdown it honours is a note's inline code spans, which it renders as code. There are three tables, Supervisor, Subagent and ToolCall, each with one column per arm in run order. Columns are never re-sorted. A column is headed by the arm's family. When a family has more than one @@ -223,6 +257,7 @@ Every table has these rows, in this order: | P95 latency per completed task (ms) | Per task completed at the tier, the sum of that tier's measured call latencies in the task; nearest-rank P95 over those tasks, shown with n. | | P50 latency per completed task (ms) | As above, at P50. | | Model invocations | Included invocations at the tier. | +| Long-context invocations | The tier's included invocations priced at the invoked model's long-context rates (see Usage and cost); `—` for a run recorded before the count existed. | | Input tokens | The tier's input tokens, in each family's own accounting. | | Output tokens | The tier's output tokens. | | Cost per task, all tasks | The tier's total cost divided by the number of tasks the arm ran. | @@ -314,13 +349,60 @@ Defined in `src/eval/metrics.py` (`ModelMetrics.to_report_fields`). Per arm: `cost_excluding_cache`, `cost_cache_total`, `cost_total`, `price_table_version`), derived from recorded tokens against `config/pricing.yaml`. -Rates come from `config/pricing.yaml` (`price_table_version` 1.2.0), global-route on-demand -Standard rates for short context. The Anthropic models and GPT-6 Astra are read from the AWS Price -List (`aws pricing get-products --service-code AmazonBedrockFoundationModels`, region us-east-1). +**Per-call pricing.** Every model call is priced on its own. Its prompt size is input plus cache +read plus cache write tokens. When the invoked model has a `long_context` block in +`config/pricing.yaml` and the prompt size exceeds its `above_prompt_tokens`, every token class of +that call, output included, is priced at the long-context rates; a call exactly at the threshold is +standard. The thresholds are: + +| Model | Long-context above (prompt tokens) | +|---|---| +| GPT-6.1 Sol, GPT-6.1 Terra, GPT-6 Luna, GPT-6 Astra | 272,000 | +| Claude Haiku 5.5 | 100,000 | +| Claude Sonnet 5.5, Claude Opus 5.5 | none (no long-context tier) | + +Every rate, standard or long-context, is then multiplied by the route of the call's inference +profile (`src/crm/inference_route.py`): a `global.` profile is the global route at the base rate +(`routes.global.multiplier` 1.0), and a geographic prefix (`us.`, `eu.`, `apac.`, `ca.`, `jp.`, `au.`) +is the geo route at `routes.geo.multiplier` 1.10. A profile with any other prefix, or one naming a +different model than the call's, is refused rather than priced at a guessed rate: the capability +matrix refuses to load it, and re-pricing refuses the call. + +Each cost block also counts `long_context_invocation_count`, its included invocations priced at +long-context rates, per arm, per tier and per (tier, role, model) row. + +**Arm figures.** An arm's `model_metrics` (its Usage and cost block in `matrix.md`) covers only the +supervisor-tier calls to the arm's model, and is labelled as such where it is rendered. The arm +totals, `arm_cost_total` and `arm_long_context_invocation_count`, cover every call the arm made at +every tier, so `arm_cost_total` equals the sum of the three tiers' costs in the tier comparison. +`matrix.md` shows them as an "All tiers" table under each arm. `matrix.json` carries them as +arm-level keys of those names; `matrix.csv` carries them as the `arm_cost` and +`arm_long_context_invocation_count` columns. + +Rates come from `config/pricing.yaml` (`price_table_version` 1.3.0), global-route on-demand +Standard rates for short context, plus the long-context rates and route multipliers above. The +Anthropic models and GPT-6 Astra are read from the AWS Price List +(`aws pricing get-products --service-code AmazonBedrockFoundationModels`, region us-east-1). GPT-6.1 Sol, GPT-6.1 Terra and GPT-6 Luna are absent from the us-east-1 Price List, so their rates -come from the AWS Bedrock model cards (Global CRIS, short context). The file's `sourced_from` and `sourced_on` name the exact query and pages. -To refresh the table, re-read the same sources, transcribe the rates, bump `price_table_version`, -and update `sourced_from` and `sourced_on`. +come from the AWS Bedrock model cards (Global CRIS, short context). The file's `sourced_from` and +`sourced_on` name the exact query and pages. +To refresh the table, re-read the same sources, transcribe the standard and long-context rates, the +long-context thresholds and the route multipliers, bump `price_table_version`, and update +`sourced_from` and `sourced_on`. + +**Re-pricing a stored sweep.** A sweep archived with `--archive-raw` keeps each call's tokens and +inference profile, so it can be re-priced against the current table without calling a model: + +```bash +.venv/bin/python3 -m src.eval.reprice --raw-dir /raw --output-dir +``` + +It writes `matrix.{json,md,csv}` and `reprice.json` into ``. A call recorded without an +inference profile is refused unless `--assume-inference-profile-prefix ` is given, in which +case its profile is ``; the assumption is printed to stderr and recorded in +`reprice.json`. An arm whose per-call records disagree with its archived token totals, per +(tier, role, model) row and per tier, is refused. Exit code 0 means the files were written, 2 that +the input was refused. `config/pricing.yaml` carries `provisional: false`, so absolute cost is published and `cost_withheld_provisional` is `false`. If the table is set back to `provisional: true`, cost fields diff --git a/docs/figures/README.md b/docs/figures/README.md index dea8ade..00833d9 100644 --- a/docs/figures/README.md +++ b/docs/figures/README.md @@ -5,7 +5,7 @@ See also: [documentation map](../../README.md#documentation). Pages embedding th Each figure is a draw.io source (`.drawio`) next to its SVG export (`.svg`); the markdown pages link the SVG. Edit the source, then re-export; never edit an SVG by hand. -- **Toolchain.** draw.io desktop 32.5.5 (`brew install --cask drawio`). Export with: +- **Toolchain.** draw.io desktop 32.6.10 (`brew install --cask drawio`). Export with: ```bash DRAWIO=/Applications/draw.io.app/Contents/MacOS/draw.io diff --git a/docs/figures/readme-aws-user-plane.drawio b/docs/figures/readme-aws-user-plane.drawio index 8159aaa..e1a3fb1 100644 --- a/docs/figures/readme-aws-user-plane.drawio +++ b/docs/figures/readme-aws-user-plane.drawio @@ -14,12 +14,12 @@ - + - + @@ -28,9 +28,9 @@ - + - + diff --git a/docs/figures/readme-aws-user-plane.svg b/docs/figures/readme-aws-user-plane.svg index d6b459d..8402f57 100644 --- a/docs/figures/readme-aws-user-plane.svg +++ b/docs/figures/readme-aws-user-plane.svg @@ -1,3 +1,3 @@ -CRM Agent Eval — user plane on AWSAWS CloudAWS RegionVPCPrivate subnetsRuntime subnets (no NAT route)Browser –React SPA,downloadedvia CloudFrontAmazon CloudFront –CDN, 3 behavioursdeferred, not enforced:AWS WAFTLS 1.2 min. (viewer)enforced:security headers + CSP(response-headers policy)Amazon S3 – UI hostingbuilt React SPA (static files)private bucket, OAC, SSE-KMSAmazon API Gateway –private REST API(Cognito authorizer)AWS KMS –customer managed keysElastic Load Balancing –internal ALBVPC interface endpoint –execute-apiAWS Lambda –CRM action handlersAmazon DynamoDB –CRM tablesAWS Systems Manager –Parameter Store(API client allowlist)AWS Lambda –streaming relayAmazon Bedrock AgentCore –RuntimeAmazon Bedrock –modelsAWS Lambda –pre-token generationAmazon Cognito –user poolHTTPSdefault · OAC + SigV4serves the SPA files/api/* · VPC origincarries Cognito JWT/agent/* · OAC + SigV4Function URL (AWS_IAM)carries Cognito JWTHTTPSHTTPSprivate APILambda proxy(IAM)IAM · gatewayendpointIAM · ssm:GetParameter(interface endpoint)SSE-KMSSigV4 · InvokeAgentRuntime(interface endpoint)SigV4 · bedrock-runtime(interface endpoint)HTTPS · OAuth code flowinvoke (IAM) \ No newline at end of file +CRM Agent Eval — user plane on AWSAWS CloudAWS RegionVPCPrivate subnetsRuntime subnets (no NAT route)Browser –React SPA,downloadedvia CloudFrontAmazon CloudFront –CDN, 3 behavioursdeferred, not enforced:AWS WAFTLS 1.2 min. (viewer)enforced:security headers + CSP(response-headers policy)Amazon S3 – UI hostingbuilt React SPA (static files)private bucket, OAC, SSE-KMSAmazon API Gateway –private REST API(Cognito authorizer)AWS KMS –customer managed keysElastic Load Balancing –internal ALBVPC interface endpoint –execute-apiAWS Lambda –CRM action handlersAmazon DynamoDB –CRM tablesAWS Systems Manager –Parameter Store(API client allowlist,SecureString)AWS Lambda –streaming relayAmazon Bedrock AgentCore –RuntimeAmazon Bedrock –modelsAWS Lambda –pre-token generationAmazon Cognito –user pool + Hosted UIHTTPSdefault · OAC + SigV4serves the SPA files/api/* · VPC origincarries Cognito JWT/agent/* · OAC + SigV4Function URL (AWS_IAM)carries Cognito JWTHTTPSHTTPSprivate APILambda proxy(IAM)IAM · gatewayendpointIAM · ssm:GetParameter(interface endpoint)SSE-KMSSSE-KMSSigV4 · InvokeAgentRuntime(interface endpoint)SigV4 · bedrock-runtime(interface endpoint)HTTPS · OAuth code flow,Hosted UI sign-outinvoke (IAM) \ No newline at end of file diff --git a/docs/figures/security-baseline-aws-network.drawio b/docs/figures/security-baseline-aws-network.drawio index e5122dd..521d12a 100644 --- a/docs/figures/security-baseline-aws-network.drawio +++ b/docs/figures/security-baseline-aws-network.drawio @@ -18,7 +18,7 @@ - + diff --git a/docs/figures/security-baseline-aws-network.svg b/docs/figures/security-baseline-aws-network.svg index 005934a..e854c10 100644 --- a/docs/figures/security-baseline-aws-network.svg +++ b/docs/figures/security-baseline-aws-network.svg @@ -1,3 +1,3 @@ -CRM Agent Eval — AWS network security baselineAWS CloudAWS RegionVPC (adopted; this stack adds endpoints, security groups, flow log, optional NAT)Public subnet (adopted)Private subnets – Lambda and endpoint ENIsRuntime subnets (no NAT route)Amazon CloudFront –/api/* VPC origindeferred, not enforced:AWS WAFTLS 1.2 min. (viewer)enforced:security headers + CSP(response-headers policy)NAT gateway –optional, egress onlyAWS Lambda –all functions(SG: no ingress)Elastic Load Balancing –internal ALB (SG: in fromCloudFront VPC origin only)VPC interface endpointsbedrock-agentcore · bedrock-runtimeecr.api · ecr.dkr · execute-api *kms · lambda · logs · ssm · statesSG: 443 in from Lambda, Runtime, ALB* private DNS offAmazon Bedrock AgentCore –Runtime (SG: no ingress)Runtime resource policy:invoke by the invoker roles only;Deny unless aws:SourceVpce =our bedrock-agentcore endpoint(stage-2 toggle, off by default)VPC flow logs –ALL trafficVPC gateway endpoints –S3, DynamoDB (Lambdaand Runtime route tables)Amazon Cognito –user pool JWKSAmazon CloudWatch –Logs (CMK)AWS KMS – CMKs:DynamoDB, SPA bucket, logs,CloudTrail, eval artefacts,cursors, Terraform stateAmazon DynamoDB –CRM tablesAmazon S3 –buckets, ECR layersAWS CloudTrail –multi-Region trailAmazon S3 –trail bucket (+ access logs)/api/* · VPC originHTTP 80 (in VPC)HTTPS 443execute-apiHTTPS 443HTTPS 443bedrock-runtime · ECRHTTPS 443 · Cognito JWKSfetch (relay, Gatewayinterceptors)HTTPS · internet egress(no inbound path)delivery(IAM role)SSE-KMSSSE-KMSIAMIAMSSE-KMS · log file validation \ No newline at end of file +CRM Agent Eval — AWS network security baselineAWS CloudAWS RegionVPC (adopted; this stack adds endpoints, security groups, flow log, optional NAT)Public subnet (adopted)Private subnets – Lambda and endpoint ENIsRuntime subnets (no NAT route)Amazon CloudFront –/api/* VPC origindeferred, not enforced:AWS WAFTLS 1.2 min. (viewer)enforced:security headers + CSP(response-headers policy)NAT gateway –optional, egress onlyAWS Lambda –all functions(SG: no ingress)Elastic Load Balancing –internal ALB (SG: in fromCloudFront VPC origin only)VPC interface endpointsbedrock-agentcore · bedrock-runtimeecr.api · ecr.dkr · execute-api *kms · lambda · logs · ssm · statesSG: 443 in from Lambda, Runtime, ALB* private DNS offAmazon Bedrock AgentCore –Runtime (SG: no ingress)Runtime resource policy:invoke by the invoker roles only;Deny unless aws:SourceVpce =our bedrock-agentcore endpoint(stage-2 toggle, off by default)VPC flow logs –ALL trafficVPC gateway endpoints –S3, DynamoDB (Lambdaand Runtime route tables)Amazon Cognito –user pool JWKSAmazon CloudWatch –Logs (CMK)AWS KMS – CMKs:DynamoDB, SPA bucket, logs,CloudTrail, eval artefacts,cursors, API allowlist,Terraform stateAmazon DynamoDB –CRM tablesAmazon S3 –buckets, ECR layersAWS CloudTrail –multi-Region trailAmazon S3 –trail bucket (+ access logs)/api/* · VPC originHTTP 80 (in VPC)HTTPS 443execute-apiHTTPS 443HTTPS 443bedrock-runtime · ECRHTTPS 443 · Cognito JWKSfetch (relay, Gatewayinterceptors)HTTPS · internet egress(no inbound path)delivery(IAM role)SSE-KMSSSE-KMSIAMIAMSSE-KMS · log file validation \ No newline at end of file diff --git a/docs/runbook.md b/docs/runbook.md index d64450c..a93465d 100644 --- a/docs/runbook.md +++ b/docs/runbook.md @@ -576,7 +576,7 @@ make verify-deploy VERIFY_DEPLOY_ARGS="--layer 4 --i-understand-this-is-billable | `--i-understand-this-is-billable` | required for a live run; without it (or `--dry-run`) the script exits 2 before any AWS call | | `--dry-run` | print each module's exact command line; no AWS call, no tenant | | `--layer 4\|5\|6\|inventory` | run one layer only, or consolidate (see below). `--layer 4`, `5` or `6` is **PARTIAL**, exactly as `--only` is, with `"partial_selector": "--layer N"` in the per-run inventory | -| `--only MOD[,MOD...]` | run only the named modules (short `iam_roles` or full `scripts.smoke.iam_roles`; repeatable), in layer order. An unknown name exits 2 before any AWS call. With `--layer N` every name must be in layer N; not allowed with `--layer inventory`. The run is **PARTIAL**: the summary says so, and `verification-inventory.run.json` becomes `{"partial": true, "partial_selector", "selected_modules", "not_run_modules", "results"}` instead of the full run's array. A partial run is never a full-deployment pass | +| `--only MOD[,MOD...]` | run only the named modules (short `iam_roles` or full `scripts.smoke.iam_roles`; repeatable), in layer order. An unknown name exits 2 before any AWS call. With `--layer N` every name must be in layer N; not allowed with `--layer inventory`. The run is **PARTIAL**: the summary says so, and its inventory goes to `verification-inventory.partial.run.json` as `{"partial": true, "partial_selector", "selected_modules", "not_run_modules", "results"}`, never to the full run's `verification-inventory.run.json`. A partial run is never a full-deployment pass | | `--profile`, `--region` | target; flag, then `AWS_PROFILE`/`AWS_REGION`, then `default`/`us-east-1` | | `--outputs PATH` | Terraform outputs document (default `results/tf-outputs.json`) | | `--token JWT` | bearer token for authenticated checks (also `CRM_SMOKE_TOKEN`); never passed to the Gateway modules, which mint their own machine token. A token 45 minutes old (`VERIFY_SMOKE_TOKEN_MAX_AGE_SECONDS`, default 2700) is replaced by a freshly minted one before the next token-taking module, and the previous minted user is removed | @@ -599,7 +599,10 @@ What a live run does: that output is empty, so set the variable to the `--secret-out` file before running `gateway_tools`, `integration_gateway_scope` or `mcp_chat_gateway`; - tears down the tenant and the smoke user on exit, failure or interrupt; -- writes a per-run rendering to `results/verification-inventory.run.{json,md}`. +- writes a per-run rendering: a full run to `results/verification-inventory.run.{json,md}`, a PARTIAL + run (`--only` or `--layer 4|5|6`) to `results/verification-inventory.partial.run.{json,md}`. A + partial run never touches the full-run files, so the last full run's rendering survives targeted + smoke. Exit codes: `0` every check reported `pass`, `skip` or `defer`; `1` a layer or the teardown failed; `2` a usage or environment error. A `defer` is a disclosed control that is not verified, carrying a @@ -813,7 +816,8 @@ bash scripts/verify_deployment.sh --layer inventory --i-understand-this-is-billa --profile default --region ``` -The layer runs write only the per-run `results/verification-inventory.run.*` files. The consolidator +The layer runs write only the per-run `results/verification-inventory.run.*` (full) or +`results/verification-inventory.partial.run.*` (PARTIAL) files. The consolidator reads only per-class artifacts and has no list of classes it expects, so a class with no file there is silently absent. Write each class's `--json` output to `results/verification-inventory/.json`, and confirm the file count before publishing: @@ -1246,7 +1250,7 @@ it later. Exit 0 means every resource is absent or pending deletion. A later dep | `cloudfront` reports TLS minimum as `defer` | `minimum_protocol_version` has no effect on the default `cloudfront.net` certificate; TLS 1.2 is not enforced without a custom ACM certificate | | `/agent/*` returns 401 | the relay rejected the Cognito token: it expired, or it was issued for a client the relay does not allow | | Gateway `tools/call` requests return JSON-RPC error `-32001`, logged as `gateway_auth_denied` with reason `PyJWKClientError`, for up to 30 s after a Cognito signing-key rotation | the Gateway interceptor (`src/gateway/identity_interceptor.py`, PyJWT 2.15 `PyJWKClient`) re-fetches the JWKS on an unknown `kid` at most once per 30 s cooldown, measured from its last fetch. Retry after the cooldown; a miss that persists means the `kid` is not in the pool's JWKS | -| Gateway requests return JSON-RPC error `-32001`, logged as `gateway_auth_denied` with reason `body_not_an_object`, `params_not_an_object` or `unexpected_error:` | the identity interceptor (`src/gateway/identity_interceptor.py`) refuses malformed requests and fails closed. `body_not_an_object`: the JSON-RPC body is not a JSON object (an array or a scalar, for example). `params_not_an_object`: a method other than `tools/list` sent `params` that is not an object. Fix the client to send one JSON-RPC request object with object `params`. `unexpected_error:` (logged at warning): the interceptor raised while handling the request, for example on a configuration error. No traceback is logged. Read the exception type, then check the interceptor's environment and its recent deploy. None of these reach a Lambda target | +| Gateway requests return JSON-RPC error `-32001`, logged as `gateway_auth_denied` with reason `body_not_an_object`, `params_not_an_object` or `unexpected_error:` | the identity interceptor (`src/gateway/identity_interceptor.py`) refuses malformed requests and fails closed. `body_not_an_object`: the JSON-RPC body is not a JSON object (an array or a scalar, for example). `params_not_an_object`: a method other than `tools/list` sent `params` that is not an object. Fix the client to send one JSON-RPC request object with object `params`. `unexpected_error:`: the interceptor raised while handling the request, for example on a configuration error. No traceback is logged. Read the exception type, then check the interceptor's environment and its recent deploy. None of these reach a Lambda target. Every `gateway_auth_denied` reason, and every `gateway_service_identity_*` refusal, is logged at WARNING; a verified request (`gateway_auth_verified`) is logged at INFO | | A direct `InvokeAgentRuntime` call fails with `AccessDeniedException` | the Runtime uses IAM (SigV4) inbound auth. Its resource policies allow only the relay's, `sweep_task`'s and `chat_target`'s execution roles and explicitly deny the data-plane actions to every other principal, including same-account admins; call it through the relay | | `pip install -e .` fails with `requires a different Python` | the venv was built with a host `python3` that is not the `.python-version` pin; rebuild it as in "Prerequisites" | | `bootstrap_state_backend.sh` refuses `CRM_EXPECTED_ACCOUNT_ID ... is required` or `WRONG ACCOUNT` | set `CRM_EXPECTED_ACCOUNT_ID` (or `--expected-account-id`) to the account the backend belongs in, and pass the profile that resolves to it | @@ -1593,6 +1597,13 @@ to the whole `module.lambda_action` would make the key depend on the functions i ### Changing operators +Changing `crm_operator_principal_arns` is always two steps, because only a principal in the +**deployed** list can apply the change. The plan-time precondition checks the caller against the list +in the plan being run, so a new operator who adds only their own ARN passes it. The apply is then +refused: the deployed state-store and key policies, the API allowlist key's included, deny +administration to every principal outside the deployed list, and the policy updates fail with an +explicit deny. A new operator acting alone is therefore refused loudly. + 1. Add the new ARN to `crm_operator_principal_arns` and apply as the **old** operator, which is still listed. 2. Re-run `infra/bootstrap_state_backend.sh`, still as the old operator, with the new full list in diff --git a/docs/security-baseline.md b/docs/security-baseline.md index 94729be..641a752 100644 --- a/docs/security-baseline.md +++ b/docs/security-baseline.md @@ -64,13 +64,54 @@ These controls narrow trust the stack already grants. Each fails closed when mis token to be issued to an allowed app client (`aud` on an ID token, `client_id` on an access token). The allowed client ids come from the SSM parameter named by `CRM_API_ALLOWED_CLIENT_IDS_PARAMETER`, read once per process. That parameter, `aws_ssm_parameter.api_allowed_client_ids` in `infra/main.tf`, - is a `String` named `//api/allowed-client-ids`, holding the SPA client id. A token from any other client gets `401`. An unset, unreadable or empty - allowlist admits nobody: the request fails as a server error, and neither a failed read nor an empty value is cached. - Unit: `tests/unit/test_envelope.py`. + is a Standard-tier `SecureString` named `//api/allowed-client-ids`, holding the SPA + client id. A token from any other client gets `401`. An unset, unreadable or empty allowlist admits + nobody: the request fails as a server error, and neither a failed read nor an empty value is cached. + The reader also refuses a parameter whose type is not `SecureString`, so a downgraded parameter is + refused rather than trusted. Unit: `tests/unit/test_envelope.py`. +- **API allowlist key.** The parameter is encrypted under its own customer-managed key, + `aws_kms_key.api_allowlist` in `infra/main.tf` (alias `alias/-api-allowlist`), with + rotation on and `data-classification = internal`. Its key policy: + - lets the operators in `crm_operator_principal_arns` encrypt and decrypt only through SSM; + - lets the consumer roles (every action Lambda that receives the parameter, plus `eval_trigger`, + `capabilities` and `user_settings`) decrypt only through SSM, and only under this parameter's + `PARAMETER_ARN` encryption context; + - denies encryption and data-key generation to every non-service principal outside the operators; + - denies all key use to every non-service principal outside the consumers and operators; + - denies key administration to every principal outside the operators and the account root. + + Each consumer's own `-config-kms` policy allows that one decrypt and explicitly denies + any decrypt through SSM under another key, `aws/ssm` included. No consumer may write, delete or + label the parameter. Plan time: `tests/unit/test_infra_allowlist_key.py`. Live: the Layer 4 + `iam_roles` smoke simulates each consumer role. The deployed stack was also checked once by hand: + a run-scoped role outside the key policy's principals was refused decryption with an explicit deny + in the key policy. +- **Known gap: parameter downgrade.** SSM accepts an overwrite that changes the parameter's type to + `String`, or re-encrypts it under `aws/ssm`, from any principal allowed `ssm:PutParameter` on it. + No project role is granted that; the operators, and any other principal whose own IAM policies + allow SSM writes in the account, can do it. Neither overwrite uses the allowlist key, so its key + policy does not stop them, and no control in this stack prevents it. What holds is that a + downgraded parameter fails closed: consumers are explicitly denied decryption under `aws/ssm`, and + the reader refuses any type other than `SecureString`, so every request is refused until the + parameter is restored. A `String` value is stored unencrypted and is readable by any principal + allowed `ssm:GetParameter` on it. Restoring it is an operator apply (`scripts/deploy.sh`), which + rewrites the parameter as a `SecureString` under the allowlist key. +- **Declared principal type.** On the `gateway` and `agent` identity channels, `crm.envelope` requires + the identity to declare `principal_type` as `user` or `service`. An absent, empty or other value is + refused `401 authentication_failed`; there is no default. The Gateway interceptor and the relay + write it, and the in-app agents forward it. Unit: `tests/unit/test_envelope.py`. Live: the Layer 4 + `lambda_actions` smoke invokes an action Lambda with each channel's identity minus the key and + expects the refusal, next to a control call that declares it. - **Refresh-token revocation at sign-out.** Every app client has token revocation enabled (`enable_token_revocation` in `infra/modules/cognito/main.tf`). Signing out of the SPA starts revoking the refresh token and clears the cached Cognito tokens at once (`frontend/src/auth/CognitoAuthSession.ts`); a failed revocation is logged to the browser console only, and the sign-out still completes. +- **Hosted UI sign-out.** When the Hosted UI is enabled, signing out also navigates to the Cognito + domain's `/logout` endpoint (`frontend/src/auth/hostedUi.ts`), which ends the Hosted UI session, so + the next sign-in asks for credentials again. Cognito returns to the SPA's `/logout` path, which must + be registered as a logout URL on the SPA client (`spa_logout_urls`, `logout_urls` in + `infra/modules/cognito/main.tf`); an unregistered return URL is refused by Cognito. The Layer 4 + `cognito_claims` smoke checks the registered and an unregistered logout URL. - **Free-text length caps.** Every free-text string a tool accepts carries a `maxLength` (`src/crm/toolspec/definitions.py`): 4000 characters for `description`, `notes`, `body` and `reason`, 4000 for `cursor`, 1024 for the `revert_audit_record` reason, and 256 for any other string without a @@ -231,8 +272,11 @@ scan-2 rows they account for therefore no longer apply: all of scan-2 H-2, and t of scan-2 H-4, H-5 and H-6. The 1 source row in each of those three still stands. The counts above are scan 2's and have not been re-measured. -Scan 2 measured how Holmes treats inline suppressions. It honours bandit `# nosec Bxxx`. It ignores -Semgrep `# nosemgrep` (92 of the 107 subprocess rows carry one) and `checkov:skip`. +Scan 2 measured how Holmes treats inline suppressions. It honoured bandit `# nosec Bxxx` and ignored +Semgrep `# nosemgrep` (92 of the 107 subprocess rows carried one) and `checkov:skip`. The re-scan of +2026-11-10 (below) reports every row carrying `# nosemgrep` as suppressed, but with an empty status +message. Its suppressed Checkov rows, by contrast, each carry their `checkov:skip` reason. So the re-scan +cannot tell whether the marker or a portal action suppressed the Semgrep rows. On 2026-10-25, semgrep 1.172.0 reported 0 `dangerous-subprocess-use-audit` rows in the tree, run from the repository root: @@ -279,6 +323,56 @@ The other Semgrep rows that scan 2 leaves standing are: All of these Semgrep rows are filed in Holmes as false positives. +**Holmes scan of 2026-11-09** (scan `84eb6441-08a2-40b4-a2e1-7f55f257b27a`). + +Baseline `HolmesContentSecurityReviewBaselinePolicy` v3 reported 45 rows. 19 are Checkov Highs that +Holmes already shows as `suppressed`: H-4 to H-10 below cover them, and no code changed for them. The +other 26 rows (8 High, 2 Medium, 16 Low) are dispositioned below. Row numbers are the scan's +`findingId`. + +Fixed in code. The local scanner no longer reports these rows, and the code before the fix still does: + +| Rows | Rule | Site | Fix | +|---|---|---|---| +| 11 | `CKV_AWS_259` | `infra/modules/frontend_cdn/cloudfront.tf`, response-headers policy | HSTS sets `include_subdomains` and `preload` to true. `tests/unit/test_infra_csp.py` asserts the planned value. `scripts/smoke/cloudfront.py` asserts the served `Strict-Transport-Security` (max-age=31536000, includeSubDomains, preload) on every probed behaviour | +| 21, 22 | `B105`, `detected-jwt-token` | `tests/unit/test_handler_error_logging.py` | the fake JWT is minted when the test module is imported, so no token literal remains | +| 24 | `regex_dos` | `tests/unit/test_deploy_script.py` | the nested-quantifier regex is replaced by a line parser | +| 25 to 34, 36 | `detected-aws-account-id` | `tests/unit/test_render_agentcore_env.py`, `tests/unit/test_teardown_deletion_protection.py` | the stubs interpolate one named placeholder constant per file | +| 38 | gitleaks `generic-api-key` | `src/crm/metering.py` | the per-million divisor is renamed | + +Suppressed with `# nosec`, which Holmes honours. Each value is a test fixture or an empty or sentinel +string, not a credential: + +| Row | Rule | Site | +|---|---|---| +| 37 | `B105` | `scripts/smoke/cognito_claims.py` | +| 40 | `B105` | `tests/unit/test_handlers_apigw_proxy.py` | +| 41 | `B106` | `tests/unit/test_identity_interceptor.py` | +| 43 | `B105` | `tests/unit/test_smoke_cognito_claims.py` | +| 45 | `B105` | `tests/unit/test_store_transport_errors.py` | + +Four more lines that local bandit flags under the same rules carry the same marker: +`tests/unit/test_identity_interceptor.py` (2), `tests/unit/test_proposal_payload_allowlist.py` and +`tests/unit/test_smoke_cognito_claims.py`. + +Residual rows. Each carries `# nosemgrep`, and the re-scan below reports each one it lists as +suppressed: + +| Row | Severity | Rule | Site | Reason | +|---|---|---|---|---| +| 12 | Medium | `aws-insecure-cloudfront-distribution-tls-version` | `infra/modules/frontend_cdn/cloudfront.tf`, distribution | ACCEPT, see H-5. With the default `*.cloudfront.net` certificate, CloudFront fixes the minimum protocol. Only a custom-domain certificate could set one | +| 35 | Low | `detected-aws-account-id` | `tests/unit/test_teardown_deletion_protection.py`, `ACCOUNT` | FP. The value is the AWS documentation example account id | +| none | Low | `detected-aws-account-id` | `tests/unit/test_render_agentcore_env.py`, `SANDBOX_ACCOUNT` | FP, the same example id. The line is new and is not in this scan, so it may appear in the next one | +| 39 | Low | gitleaks `generic-api-key` | `tests/integration/test_local_turn.py`, `transaction_key` | FP. A test value for an idempotency token, not a secret | +| 42 | Low | gitleaks `generic-api-key` | `tests/unit/test_proposal_payload_allowlist.py`, `transaction_key` | FP, the same kind of test id | +| 44 | Low | gitleaks `generic-api-key` | `tests/unit/test_smoke_iam_roles.py`, the in-scope account record key in the own-row test | FP. A test record key, not a secret | + +**Re-scan of 2026-11-10** (scan `f4c00713-dc00-4284-a2f1-7a91efae47f7`). This scan read the tree in +which this block was first written. It reported 25 rows, 24 of them suppressed: the 19 Checkov rows, +plus rows 12, 35, 39, 42 and 44, all of which carry `# nosemgrep`. `SANDBOX_ACCOUNT` was not reported. +The one open row was a gitleaks `generic-api-key` Low on this document: the row-44 entry above had +quoted the test's key assignment. That entry now describes the key instead of quoting it. + ### H-4 - **Rule:** `CKV_AWS_355`. diff --git a/frontend/src/App.test.tsx b/frontend/src/App.test.tsx index 5d8e9d8..715db0d 100644 --- a/frontend/src/App.test.tsx +++ b/frontend/src/App.test.tsx @@ -147,6 +147,13 @@ describe("App auth gating", () => { expect(screen.getByText("Loading…")).toBeInTheDocument(); }); + it("renders the sign-in form on the /logout landing page when signed out", () => { + stubAuth({ status: "signed-out", identity: undefined }); + renderAt("/logout"); + expect(screen.getByRole("heading", { name: "page:sign-in" })).toBeInTheDocument(); + expect(screen.queryByRole("heading", { name: "Page not found" })).not.toBeInTheDocument(); + }); + it("does not render any page route while signed out", () => { stubAuth({ status: "signed-out", identity: undefined }); renderAt("/audit"); @@ -188,6 +195,13 @@ describe("route table", () => { expect(screen.getByRole("heading", { name: "page:pipeline" })).toBeInTheDocument(); }); + it("sends a signed-in user on /logout to the pipeline instead of a not-found page", () => { + stubAuth(); + renderAt("/logout"); + expect(screen.getByRole("heading", { name: "page:pipeline" })).toBeInTheDocument(); + expect(screen.queryByRole("heading", { name: "Page not found" })).not.toBeInTheDocument(); + }); + it("passes the runtime config's apiBaseUrl to the eval dashboard", () => { stubAuth(); renderAt("/evals"); diff --git a/frontend/src/App.tsx b/frontend/src/App.tsx index b22465c..ddd0bad 100644 --- a/frontend/src/App.tsx +++ b/frontend/src/App.tsx @@ -8,6 +8,9 @@ * `getRuntimeConfig()` is called synchronously: `AuthProvider` only reports `signed-in` after * `loadRuntimeConfig()` has resolved, so the cache is warm by the time this tree renders. * + * `/logout` is the Hosted UI sign-out landing page: signed out it renders the sign-in form like + * every other path, and a signed-in user there is sent to the pipeline rather than a 404. + * * The signed-in layout is `AppShell`, which mounts the assistant panel as a sibling of the * ``. The panel is the only assistant surface; there is no `/chat` route. */ @@ -71,6 +74,7 @@ export function App(): JSX.Element { return ( + } /> }> } /> } /> diff --git a/frontend/src/auth/AuthProvider.test.tsx b/frontend/src/auth/AuthProvider.test.tsx index 14940df..11d3398 100644 --- a/frontend/src/auth/AuthProvider.test.tsx +++ b/frontend/src/auth/AuthProvider.test.tsx @@ -1,7 +1,9 @@ // SPDX-License-Identifier: MIT /** * Tests for the auth context's sign-out: it clears the saved-settings store, so the next user to - * sign in on the same page load fetches their own settings and never sees the previous user's. + * sign in on the same page load fetches their own settings and never sees the previous user's; and + * it waits for the session's revocation before clearing local state and ending the Hosted UI + * session. */ import { act, renderHook } from "@testing-library/react"; import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; @@ -22,13 +24,23 @@ import { AuthProvider, useAuth, type AuthContextValue } from "./AuthProvider"; import type { AuthSession } from "./CognitoAuthSession"; import type { Identity } from "./identity"; -const { signInMock, restoreMock } = vi.hoisted(() => ({ - signInMock: vi.fn(), - restoreMock: vi.fn(), -})); +const { signInMock, restoreMock, loadRuntimeConfigMock, endHostedUiSessionMock } = vi.hoisted( + () => ({ + signInMock: vi.fn(), + restoreMock: vi.fn(), + loadRuntimeConfigMock: vi.fn(), + endHostedUiSessionMock: vi.fn(), + }), +); + +const RUNTIME_CONFIG = { cognitoHostedUiDomain: "crm-example.auth.us-east-1.amazoncognito.com" }; vi.mock("../config/runtimeConfig", () => ({ - loadRuntimeConfig: vi.fn().mockResolvedValue({}), + loadRuntimeConfig: loadRuntimeConfigMock, +})); + +vi.mock("./hostedUi", () => ({ + endHostedUiSession: endHostedUiSessionMock, })); vi.mock("./CognitoAuthSession", () => ({ @@ -46,7 +58,7 @@ function fakeSession(userId: string): AuthSession { return { getIdentity: () => identity, getAccessToken: () => Promise.resolve(`token-${userId}`), - signOut: vi.fn(), + signOut: vi.fn(() => Promise.resolve()), }; } @@ -86,10 +98,20 @@ async function signInAs(auth: { current: () => AuthContextValue }, userId: strin }); } +/** Sign out and let the asynchronous revoke → clear → redirect sequence finish. */ +async function signOutAndSettle(auth: { current: () => AuthContextValue }): Promise { + await act(async () => { + auth.current().signOut(); + await new Promise((resolve) => setTimeout(resolve, 0)); + }); +} + beforeEach(() => { __resetSessionTuningForTests(); signInMock.mockReset(); restoreMock.mockReset().mockResolvedValue(undefined); + loadRuntimeConfigMock.mockReset().mockResolvedValue(RUNTIME_CONFIG); + endHostedUiSessionMock.mockReset().mockReturnValue(true); }); afterEach(() => { @@ -108,7 +130,7 @@ describe("AuthProvider sign-out", () => { }); expect(getSessionTuning().tiers.supervisor.model).toBe("model-of-a"); - act(() => auth.current().signOut()); + await signOutAndSettle(auth); expect(auth.current().status).toBe("signed-out"); expect(getSessionTuning().tiers.supervisor).toEqual(EMPTY_TIER_SELECTION); @@ -129,10 +151,105 @@ describe("AuthProvider sign-out", () => { await auth.current().signIn("user-a", "pw"); }); - act(() => auth.current().signOut()); + await signOutAndSettle(auth); expect(session.signOut).toHaveBeenCalledTimes(1); }); + + it("waits for the session's revocation before clearing state, then ends the Hosted UI session", async () => { + const auth = await renderAuth(); + const revoked = deferred(); + const session = { ...fakeSession("user-a"), signOut: vi.fn(() => revoked.promise) }; + signInMock.mockResolvedValueOnce(session); + await act(async () => { + await auth.current().signIn("user-a", "pw"); + }); + await act(async () => { + await loadUserSettings(settingsClient(() => Promise.resolve(viewFor("model-of-a")))); + }); + + await act(async () => { + auth.current().signOut(); + await new Promise((resolve) => setTimeout(resolve, 0)); + }); + expect(session.signOut).toHaveBeenCalledTimes(1); + expect(auth.current().status).toBe("signed-in"); + expect(getSessionTuning().tiers.supervisor.model).toBe("model-of-a"); + expect(endHostedUiSessionMock).not.toHaveBeenCalled(); + + await act(async () => { + revoked.resolve(undefined); + await new Promise((resolve) => setTimeout(resolve, 0)); + }); + expect(auth.current().status).toBe("signed-out"); + expect(getSessionTuning().tiers.supervisor).toEqual(EMPTY_TIER_SELECTION); + expect(endHostedUiSessionMock).toHaveBeenCalledTimes(1); + expect(endHostedUiSessionMock).toHaveBeenCalledWith(RUNTIME_CONFIG); + }); + + it("ignores a second sign-out while the first is still revoking", async () => { + const auth = await renderAuth(); + const revoked = deferred(); + const session = { ...fakeSession("user-a"), signOut: vi.fn(() => revoked.promise) }; + signInMock.mockResolvedValueOnce(session); + await act(async () => { + await auth.current().signIn("user-a", "pw"); + }); + + await act(async () => { + auth.current().signOut(); + auth.current().signOut(); + revoked.resolve(undefined); + await new Promise((resolve) => setTimeout(resolve, 0)); + }); + + expect(session.signOut).toHaveBeenCalledTimes(1); + expect(endHostedUiSessionMock).toHaveBeenCalledTimes(1); + }); + + it("still signs out locally and ends the Hosted UI session when the session's sign-out rejects", async () => { + const warn = vi.spyOn(console, "warn").mockImplementation(() => {}); + const auth = await renderAuth(); + const session = { + ...fakeSession("user-a"), + signOut: vi.fn(() => Promise.reject(new Error("revoke exploded"))), + }; + signInMock.mockResolvedValueOnce(session); + await act(async () => { + await auth.current().signIn("user-a", "pw"); + }); + + await signOutAndSettle(auth); + + expect(auth.current().status).toBe("signed-out"); + expect(endHostedUiSessionMock).toHaveBeenCalledTimes(1); + expect(warn).toHaveBeenCalledTimes(1); + warn.mockRestore(); + }); + + it("signs out locally without redirecting when the runtime config cannot load", async () => { + const warn = vi.spyOn(console, "warn").mockImplementation(() => {}); + const auth = await renderAuth(); + await signInAs(auth, "user-a"); + loadRuntimeConfigMock.mockRejectedValueOnce(new Error("runtime config: /config.json responded 503")); + + await signOutAndSettle(auth); + + expect(auth.current().status).toBe("signed-out"); + expect(endHostedUiSessionMock).not.toHaveBeenCalled(); + expect(warn).toHaveBeenCalledTimes(1); + warn.mockRestore(); + }); + + it("ends the Hosted UI session even when no local session exists", async () => { + const auth = await renderAuth(); + expect(auth.current().status).toBe("signed-out"); + + await signOutAndSettle(auth); + + expect(auth.current().status).toBe("signed-out"); + expect(endHostedUiSessionMock).toHaveBeenCalledWith(RUNTIME_CONFIG); + }); }); describe("resetUserSettingsState", () => { @@ -218,7 +335,7 @@ describe("a save or reset in flight at sign-out", () => { }); const writeA = write(clientA); - act(() => auth.current().signOut()); + await signOutAndSettle(auth); await act(async () => { pendingA.resolve(viewFor("model-of-a-written")); await writeA; diff --git a/frontend/src/auth/AuthProvider.tsx b/frontend/src/auth/AuthProvider.tsx index 1e8fb47..6dceb8f 100644 --- a/frontend/src/auth/AuthProvider.tsx +++ b/frontend/src/auth/AuthProvider.tsx @@ -3,12 +3,13 @@ * React auth context. Every page and hook that needs the current identity or an access token * consumes this via {@link useAuth} — never `CognitoAuthSession` directly. */ -import { createContext, useCallback, useContext, useEffect, useMemo, useState } from "react"; +import { createContext, useCallback, useContext, useEffect, useMemo, useRef, useState } from "react"; import type { JSX, ReactNode } from "react"; import { resetUserSettingsState } from "../components/tuning/sessionTuning"; -import { loadRuntimeConfig } from "../config/runtimeConfig"; +import { loadRuntimeConfig, type RuntimeConfig } from "../config/runtimeConfig"; import { CognitoAuthSession, type AuthSession } from "./CognitoAuthSession"; +import { endHostedUiSession } from "./hostedUi"; import type { Identity } from "./identity"; export type AuthStatus = "loading" | "signed-out" | "signed-in"; @@ -18,6 +19,11 @@ export interface AuthContextValue { /** Present only when `status === "signed-in"`. */ readonly identity: Identity | undefined; signIn(username: string, password: string): Promise; + /** + * Revokes and clears the session and resets saved settings, then, when Hosted UI is configured, + * navigates to its `/logout` so the Cognito domain session ends too. Repeat calls while one is in + * flight are ignored. + */ signOut(): void; /** Resolves a fresh, non-expired access token. Throws if not signed in. */ getAccessToken(): Promise; @@ -57,12 +63,34 @@ export function AuthProvider({ children }: { children: ReactNode }): JSX.Element setStatus("signed-in"); }, []); + const signingOut = useRef(false); const signOut = useCallback(() => { - session?.signOut(); - resetUserSettingsState(); - setSession(undefined); - setIdentity(undefined); - setStatus("signed-out"); + if (signingOut.current) return; + signingOut.current = true; + void (async () => { + try { + await session?.signOut(); + } catch (err) { + console.warn("Sign-out did not finish cleanly", err instanceof Error ? err.name : String(err)); + } + resetUserSettingsState(); + setSession(undefined); + setIdentity(undefined); + setStatus("signed-out"); + signingOut.current = false; + + let config: RuntimeConfig; + try { + config = await loadRuntimeConfig(); + } catch (err) { + console.warn( + "Signed out locally; could not load the runtime config to end the Hosted UI session", + err instanceof Error ? err.message : String(err), + ); + return; + } + endHostedUiSession(config); + })(); }, [session]); const getAccessToken = useCallback(async () => { diff --git a/frontend/src/auth/CognitoAuthSession.test.ts b/frontend/src/auth/CognitoAuthSession.test.ts index 4e24baa..34302bd 100644 --- a/frontend/src/auth/CognitoAuthSession.test.ts +++ b/frontend/src/auth/CognitoAuthSession.test.ts @@ -2,13 +2,14 @@ /** * Tests for sign-out against the real `amazon-cognito-identity-js`: a session restored from seeded * browser storage, with `fetch` stubbed so the only observable network effect is a `RevokeToken` - * request. Asserts the refresh token is revoked when revocable, that cached tokens are gone before - * `signOut` returns, and that a non-revocable token makes no request at all. + * request. Asserts the refresh token is revoked when revocable and before the cache is cleared, that + * revocation is bounded at `REVOKE_TIMEOUT_MS`, that cached tokens are gone once `signOut` resolves, + * and that a non-revocable token makes no request at all. */ import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; import type { RuntimeConfig } from "../config/runtimeConfig"; -import { CognitoAuthSession } from "./CognitoAuthSession"; +import { CognitoAuthSession, REVOKE_TIMEOUT_MS } from "./CognitoAuthSession"; const CLIENT_ID = "testclient123"; const USERNAME = "rep-042"; @@ -85,10 +86,11 @@ describe("CognitoAuthSession.signOut", () => { }); it("sends one RevokeToken request carrying the refresh token", async () => { + fetchMock.mockImplementation(() => Promise.resolve(new Response("{}", { status: 200 }))); seedCachedSession({ revocable: true }); const session = await restoredSession(); - session.signOut(); + await session.signOut(); const requests = revokeRequests(fetchMock); expect(requests).toHaveLength(1); @@ -98,27 +100,73 @@ describe("CognitoAuthSession.signOut", () => { }); }); - it("clears every cached Cognito key before it returns, with the revocation still pending", async () => { + it("keeps the cache until revocation succeeds, then clears it without waiting for the bound", async () => { + let respond: (response: Response) => void = () => {}; + fetchMock.mockImplementation( + () => + new Promise((resolve) => { + respond = resolve; + }), + ); seedCachedSession({ revocable: true }); const session = await restoredSession(); - session.signOut(); + vi.useFakeTimers({ toFake: ["setTimeout"] }); + try { + let resolved = false; + const signingOut = session.signOut().then(() => { + resolved = true; + }); + await vi.advanceTimersByTimeAsync(REVOKE_TIMEOUT_MS / 2); + expect(resolved).toBe(false); + expect(cognitoKeys().length).toBeGreaterThan(0); + + respond(new Response("{}", { status: 200 })); + await signingOut; + expect(cognitoKeys()).toEqual([]); + } finally { + vi.useRealTimers(); + } + }); + + it("gives up on a revocation that hangs after the bound, warning once, and clears the cache", async () => { + const warn = vi.spyOn(console, "warn").mockImplementation(() => {}); + seedCachedSession({ revocable: true }); + const session = await restoredSession(); + + vi.useFakeTimers({ toFake: ["setTimeout"] }); + try { + let resolved = false; + const signingOut = session.signOut().then(() => { + resolved = true; + }); + await vi.advanceTimersByTimeAsync(REVOKE_TIMEOUT_MS - 1); + expect(resolved).toBe(false); + expect(cognitoKeys().length).toBeGreaterThan(0); + + await vi.advanceTimersByTimeAsync(1); + await signingOut; + } finally { + vi.useRealTimers(); + } - expect(cognitoKeys()).toEqual([]); expect(revokeRequests(fetchMock)).toHaveLength(1); + expect(cognitoKeys()).toEqual([]); + expect(warn).toHaveBeenCalledTimes(1); + expect(JSON.stringify(warn.mock.calls)).not.toContain(REFRESH_TOKEN); }); it("makes no network call for a token without origin_jti, and still clears the cache", async () => { seedCachedSession({ revocable: false }); const session = await restoredSession(); - session.signOut(); + await session.signOut(); expect(fetchMock).not.toHaveBeenCalled(); expect(cognitoKeys()).toEqual([]); }); - it("reports a failed revocation without the token and keeps the cache cleared", async () => { + it("reports a failed revocation once, without the token, and clears the cache", async () => { fetchMock.mockImplementation(() => Promise.reject(new TypeError("network down"))); const warn = vi.spyOn(console, "warn").mockImplementation(() => {}); seedCachedSession({ revocable: true }); @@ -126,24 +174,24 @@ describe("CognitoAuthSession.signOut", () => { vi.useFakeTimers({ toFake: ["setTimeout"] }); try { - session.signOut(); + const signingOut = session.signOut(); await vi.advanceTimersByTimeAsync(120_000); + await signingOut; } finally { vi.useRealTimers(); } expect(warn).toHaveBeenCalledTimes(1); - expect(JSON.stringify(warn.mock.calls)).not.toContain(REFRESH_TOKEN); expect(cognitoKeys()).toEqual([]); }); it("does not navigate away from the page", async () => { - seedCachedSession({ revocable: true }); + seedCachedSession({ revocable: false }); const session = await restoredSession(); const before = window.location.href; - session.signOut(); + await session.signOut(); expect(window.location.href).toBe(before); }); diff --git a/frontend/src/auth/CognitoAuthSession.ts b/frontend/src/auth/CognitoAuthSession.ts index 43bf283..d6e3368 100644 --- a/frontend/src/auth/CognitoAuthSession.ts +++ b/frontend/src/auth/CognitoAuthSession.ts @@ -38,12 +38,15 @@ export interface AuthSession { /** Returns a valid, non-expired Cognito ID token, refreshing first if necessary. */ getAccessToken(): Promise; /** - * Signs the user out: starts revoking the refresh token, and clears all cached Cognito state for - * this pool before returning. + * Signs the user out: revokes the refresh token, waiting at most {@link REVOKE_TIMEOUT_MS}, then + * clears all cached Cognito state for this pool. Never rejects. */ - signOut(): void; + signOut(): Promise; } +/** The longest sign-out waits for refresh-token revocation before clearing local state anyway. */ +export const REVOKE_TIMEOUT_MS = 3_000; + /** `CognitoUser.revokeTokens`, present at runtime but missing from the library's type declarations. */ interface RevocableUser { revokeTokens(callback: (err?: Error) => void): void; @@ -134,14 +137,43 @@ export class CognitoAuthSession implements AuthSession { } /** - * Start revoking the refresh token, then clear every cached Cognito token synchronously. The - * revocation runs in the background; a token without `origin_jti` is not revocable and makes no - * network call, and a failed revocation is reported to the console only. + * Revoke the refresh token, then clear every cached Cognito token. Revocation is bounded at + * {@link REVOKE_TIMEOUT_MS}; a token without `origin_jti` is not revocable and makes no network + * call. A failed or timed-out revocation is reported to the console only, and the cache is + * cleared regardless. */ - signOut(): void { - (this.user as unknown as RevocableUser).revokeTokens((err?: Error) => { - if (err) console.warn("Could not revoke the refresh token at sign-out", err.name); + async signOut(): Promise { + try { + await this.revokeRefreshToken(); + } finally { + this.user.signOut(); + } + } + + /** Resolves when revocation settles or times out; reports at most one warning, never rejects. */ + private revokeRefreshToken(): Promise { + return new Promise((resolve) => { + let settled = false; + const settle = (warning?: string): void => { + if (settled) return; + settled = true; + clearTimeout(timer); + if (warning !== undefined) console.warn(warning); + resolve(); + }; + const timer = setTimeout( + () => settle("Refresh-token revocation did not finish in time; signed out locally"), + REVOKE_TIMEOUT_MS, + ); + try { + (this.user as unknown as RevocableUser).revokeTokens((err?: Error) => { + settle(err ? `Could not revoke the refresh token at sign-out: ${err.name}` : undefined); + }); + } catch (err) { + settle( + `Could not revoke the refresh token at sign-out: ${err instanceof Error ? err.name : "unknown"}`, + ); + } }); - this.user.signOut(); } } diff --git a/frontend/src/auth/hostedUi.test.ts b/frontend/src/auth/hostedUi.test.ts index 0346660..7021bac 100644 --- a/frontend/src/auth/hostedUi.test.ts +++ b/frontend/src/auth/hostedUi.test.ts @@ -6,12 +6,14 @@ * attempt deletes it, and a replayed callback is rejected. * 2. Every token-exchange failure path is a `HostedUiError` whose message names what happened and * what to do. + * 3. Sign-out navigates to the Hosted UI `/logout` with the SPA client and the `/logout` return URI, + * and only when Hosted UI is configured. */ import { cleanup, render, screen, waitFor } from "@testing-library/react"; import userEvent from "@testing-library/user-event"; import { createElement } from "react"; -import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; +import { afterEach, beforeEach, describe, expect, it, vi, type Mock } from "vitest"; import { HostedUiError, @@ -21,8 +23,11 @@ import { createCodeChallenge, createCodeVerifier, createState, + endHostedUiSession, exchangeCodeForTokens, extractAuthorizationCode, + hostedUiLogoutRedirectUri, + hostedUiLogoutUrl, hostedUiRedirectUri, isHostedUiCallback, isHostedUiEnabled, @@ -449,6 +454,52 @@ describe("every failure is actionable, never blank", () => { }); }); +describe("Hosted UI logout", () => { + const ORIGIN = "https://crm.example"; + + function fakeLocation(): { origin: string; assign: Mock<(url: string | URL) => void> } { + return { origin: ORIGIN, assign: vi.fn<(url: string | URL) => void>() }; + } + + it("builds the /logout URL on the Hosted UI domain with the client and the /logout return URI", () => { + const url = new URL(hostedUiLogoutUrl(config, ORIGIN)); + + expect(url.origin).toBe(`https://${config.cognitoHostedUiDomain}`); + expect(url.pathname).toBe("/logout"); + expect(url.searchParams.get("client_id")).toBe(config.cognitoClientId); + expect(url.searchParams.get("logout_uri")).toBe(`${ORIGIN}/logout`); + expect([...url.searchParams.keys()].sort()).toEqual(["client_id", "logout_uri"]); + }); + + it("returns to the SPA's /logout path, not to the sign-in callback", () => { + expect(hostedUiLogoutRedirectUri(ORIGIN)).toBe(`${ORIGIN}/logout`); + expect(hostedUiLogoutRedirectUri(ORIGIN)).not.toBe(hostedUiRedirectUri(ORIGIN)); + }); + + it("refuses to build a logout URL when Hosted UI is not configured", () => { + expect(() => hostedUiLogoutUrl({ ...config, cognitoHostedUiDomain: "" }, ORIGIN)).toThrow( + HostedUiError, + ); + }); + + it("navigates to the logout URL when Hosted UI is configured", () => { + const location = fakeLocation(); + + expect(endHostedUiSession(config, location)).toBe(true); + + expect(location.assign).toHaveBeenCalledTimes(1); + expect(location.assign).toHaveBeenCalledWith(hostedUiLogoutUrl(config, ORIGIN)); + }); + + it("does not navigate when Hosted UI is not configured", () => { + const location = fakeLocation(); + + expect(endHostedUiSession({ ...config, cognitoHostedUiDomain: "" }, location)).toBe(false); + + expect(location.assign).not.toHaveBeenCalled(); + }); +}); + describe("callback and config helpers", () => { it("recognises a callback carrying a code or an error, and nothing else", () => { expect(isHostedUiCallback(`${REDIRECT_URI}?code=c&state=s`)).toBe(true); diff --git a/frontend/src/auth/hostedUi.ts b/frontend/src/auth/hostedUi.ts index 68d19a7..0300cf2 100644 --- a/frontend/src/auth/hostedUi.ts +++ b/frontend/src/auth/hostedUi.ts @@ -7,6 +7,7 @@ * * The flow is three calls across a full-page redirect: * `beginHostedUiSignIn` → (browser leaves) → `completeHostedUiSignIn` on the callback URL. + * Sign-out calls `endHostedUiSession`, navigating to `hostedUiLogoutUrl`, which ends the Hosted UI session and returns to `/logout`. * * - A verifier is generated per attempt and consumed exactly once: `takePkceAttempt` deletes it * before it returns, so a replayed callback URL cannot reuse it. @@ -189,6 +190,47 @@ export async function beginHostedUiSignIn( return `https://${config.cognitoHostedUiDomain}/login?${params.toString()}`; } +/* --------------------------------------------------------------------------- logout */ + +/** The SPA path the Hosted UI returns to after ending its session; registered as a logout URL. */ +export const HOSTED_UI_LOGOUT_PATH = "/logout"; + +/** The `logout_uri` sent to the Hosted UI; must exactly match a registered logout URL. */ +export function hostedUiLogoutRedirectUri(origin: string): string { + return `${origin}${HOSTED_UI_LOGOUT_PATH}`; +} + +/** + * The Hosted UI `/logout` URL that ends the Cognito domain session and returns to + * {@link hostedUiLogoutRedirectUri}. Throws `not_configured` when Hosted UI is disabled. + */ +export function hostedUiLogoutUrl(config: RuntimeConfig, origin: string): string { + if (!isHostedUiEnabled(config)) { + throw new HostedUiError("not_configured", "Hosted sign-in is not configured for this deployment."); + } + const params = new URLSearchParams({ + client_id: config.cognitoClientId, + logout_uri: hostedUiLogoutRedirectUri(origin), + }); + return `https://${config.cognitoHostedUiDomain}/logout?${params.toString()}`; +} + +/** The part of `window.location` that {@link endHostedUiSession} uses. */ +export type LogoutNavigator = Pick; + +/** + * Navigates to the Hosted UI `/logout` when Hosted UI is configured and reports whether it did. + * Does nothing and returns `false` otherwise. + */ +export function endHostedUiSession( + config: RuntimeConfig, + location: LogoutNavigator = window.location, +): boolean { + if (!isHostedUiEnabled(config)) return false; + location.assign(hostedUiLogoutUrl(config, location.origin)); + return true; +} + /* ------------------------------------------------------------------------ step two */ /** Extract the `code` query parameter from a Hosted UI redirect callback URL, if present. */ diff --git a/frontend/src/components/AssistantPanel/UsageMeter.test.tsx b/frontend/src/components/AssistantPanel/UsageMeter.test.tsx index f9e0433..4906c28 100644 --- a/frontend/src/components/AssistantPanel/UsageMeter.test.tsx +++ b/frontend/src/components/AssistantPanel/UsageMeter.test.tsx @@ -94,6 +94,7 @@ const PRICED_LINES = [ output_tokens: 200, cache_read_tokens: 0, cache_write_tokens: 0, + long_context_calls: 0, cost_usd: 0.0123, }, { @@ -104,6 +105,7 @@ const PRICED_LINES = [ output_tokens: 90, cache_read_tokens: 0, cache_write_tokens: 0, + long_context_calls: 0, cost_usd: 0.0007, }, ] as const; @@ -187,6 +189,78 @@ describe("per-reply cost and its per-tier lines", () => { }); }); +describe("calls priced at long-context rates", () => { + const withLongContext = [ + { ...PRICED_LINES[0], long_context_calls: 2 }, + { ...PRICED_LINES[1], long_context_calls: 1 }, + ]; + + it("shows the turn's long-context call count, and which line it came from, when above zero", () => { + const events = eventsFrom({ + kind: "completed", + cost_withheld: false, + cost_usd: 0.013, + cost_lines: withLongContext, + }); + + render(); + + expect(screen.getByTestId("usage-long-context-calls")).toHaveTextContent("3"); + const notes = screen.getAllByTestId("usage-cost-line-long-context"); + expect(notes.map((note) => note.textContent)).toEqual([ + "2 calls at long-context rates", + "1 call at long-context rates", + ]); + const rows = screen.getAllByTestId("usage-cost-line"); + expect(rows[0]).toContainElement(notes[0] ?? null); + }); + + it("shows nothing about long context when no call used it", () => { + const events = eventsFrom({ + kind: "completed", + cost_withheld: false, + cost_usd: 0.013, + cost_lines: [...PRICED_LINES], + }); + + render(); + + expect(screen.queryByTestId("usage-long-context-calls")).not.toBeInTheDocument(); + expect(screen.queryByTestId("usage-cost-line-long-context")).not.toBeInTheDocument(); + expect(screen.getAllByTestId("usage-cost-line")).toHaveLength(2); + }); + + it("still shows the count while the cost is withheld", () => { + const events = eventsFrom({ + kind: "completed", + cost_withheld: true, + cost_usd: null, + cost_lines: withLongContext.map((line) => ({ ...line, cost_usd: null })), + }); + + render(); + + expect(screen.getByTestId("usage-cost-withheld")).toBeInTheDocument(); + expect(screen.getByTestId("usage-long-context-calls")).toHaveTextContent("3"); + }); + + it("ignores a line from a frame that predates the field", () => { + const legacy: Record = { ...PRICED_LINES[0] }; + delete legacy.long_context_calls; + const events = eventsFrom({ + kind: "completed", + cost_withheld: false, + cost_usd: 0.013, + cost_lines: [legacy, withLongContext[1]] as unknown as typeof withLongContext, + }); + + render(); + + expect(screen.getByTestId("usage-long-context-calls")).toHaveTextContent("1"); + expect(screen.getAllByTestId("usage-cost-line-long-context")).toHaveLength(1); + }); +}); + describe("the conversation running total", () => { function turn(...frames: FrameOverrides[]) { return { prompt: "q", frames: eventsFrom(...frames), terminalState: null }; diff --git a/frontend/src/components/AssistantPanel/UsageMeter.tsx b/frontend/src/components/AssistantPanel/UsageMeter.tsx index 11bed06..a5cb149 100644 --- a/frontend/src/components/AssistantPanel/UsageMeter.tsx +++ b/frontend/src/components/AssistantPanel/UsageMeter.tsx @@ -7,7 +7,8 @@ * `cost_withheld === true`. * * Cache counts are shown only when the model reported them, so a reported `0` is distinct from - * "not reported". + * "not reported". Calls priced at long-context rates are counted only when there is at least one, + * whether or not the cost is withheld. */ import type { JSX } from "react"; @@ -98,6 +99,18 @@ export function turnCost(events: readonly AgentEventFrame[]): TurnCost | undefin return undefined; } +/** How many of the turn's calls were priced at long-context rates, summed over its cost lines. */ +function longContextCalls(lines: readonly CostLine[]): number { + return lines.reduce( + (total, line) => (line.long_context_calls > 0 ? total + line.long_context_calls : total), + 0, + ); +} + +function longContextNote(count: number): string { + return count === 1 ? "1 call at long-context rates" : `${count} calls at long-context rates`; +} + function CostBreakdown({ lines, withheld }: { lines: readonly CostLine[]; withheld: boolean }) { return (
@@ -124,7 +137,17 @@ function CostBreakdown({ lines, withheld }: { lines: readonly CostLine[]; withhe > {TIER_LABEL[line.tier] ?? line.tier} {line.role} - {line.model_id} + + {line.model_id} + {line.long_context_calls > 0 ? ( + + {longContextNote(line.long_context_calls)} + + ) : null} + {line.input_tokens.toLocaleString()} {line.output_tokens.toLocaleString()} {withheld ? null : ( @@ -142,6 +165,7 @@ export function UsageMeter({ events }: { events: readonly AgentEventFrame[] }): const totals = usageTotals(events); const cost = turnCost(events); if (totals === undefined && cost === undefined) return null; + const longContext = cost === undefined ? 0 : longContextCalls(cost.lines); return (
)} + {longContext === 0 ? null : ( +
+
Long-context calls
+
+ {longContext.toLocaleString()} +
+
+ )} {cost === undefined || cost.totalUsd === null ? null : (
Cost
diff --git a/frontend/src/components/AssistantPanel/eventFrame.test.ts b/frontend/src/components/AssistantPanel/eventFrame.test.ts index bb4e477..9585da1 100644 --- a/frontend/src/components/AssistantPanel/eventFrame.test.ts +++ b/frontend/src/components/AssistantPanel/eventFrame.test.ts @@ -109,6 +109,7 @@ describe("toEventFrame", () => { output_tokens: 22, cache_read_tokens: 33, cache_write_tokens: 44, + long_context_calls: 0, cost_usd: 0.5, }, ], diff --git a/frontend/src/pages/evals/EvalArmResults.tsx b/frontend/src/pages/evals/EvalArmResults.tsx index 2e3edd1..dbf0919 100644 --- a/frontend/src/pages/evals/EvalArmResults.tsx +++ b/frontend/src/pages/evals/EvalArmResults.tsx @@ -583,10 +583,6 @@ export function EvalArmResults({ arm, tasksFailed = 0 }: EvalArmResultsProps): J } /> ) : null} - + - {/* Prompt-cache attribution. Checkpoint minimums differ by model, so an identical system - prompt can cache on one arm and not another; the threshold and the cache token counts sit - beside the two cost fields. */} - +

+ Supervisor tier +

+

+ These figures count the supervisor’s own model calls. Subagent and tool calls are + in the tier comparison, and in the all-tier figures below when the run reports them. +

+
+ + + {/* Prompt-cache attribution. Checkpoint minimums differ by model, so an identical system + prompt can cache on one arm and not another; the threshold and the cache token counts sit + beside the two cost fields. */} + + This model’s own minimum. An identical prompt can reach a checkpoint on one arm + and not on another, so a cost difference can be cache-driven rather than model-driven. + + } + /> + + + + {/* Cost is rendered in three states, keyed on the backend's `cost_withheld_provisional` + flag: + - `true` → withheld disclosure, no figure (price table provisional). + - `false` → the two cost fields the backend emits, cache-attributable apart from + excluding-cache, never summed. + - absent → "not displayed": no figure is shown. */} + {costWithheld ? ( <> - This model’s own minimum. An identical prompt can reach a checkpoint on one arm - and not on another, so a cost difference can be cache-driven rather than model-driven. + + - } - /> - - - - {/* Cost is rendered in three states, keyed on the backend's `cost_withheld_provisional` - flag: - - `true` → withheld disclosure, no figure (price table provisional). - - `false` → the two cost fields the backend emits, cache-attributable apart from - excluding-cache, never summed. - - absent → "not displayed": no figure is shown. */} - {costWithheld ? ( - <> - + ) : costShown ? ( + <> + {(() => { + const cache = formatCostField(modelMetrics.cost_cache_total); + const excluding = formatCostField(modelMetrics.cost_excluding_cache); + return ( + <> + + Cache-attributable spend, shown apart from the cost excluding cache beside it. + The two are never summed into one figure — a cost gap that tracks cache + eligibility is not a gap between the models. + + } + /> + + + ); + })()} + + ) : ( - + )} + +
+
+ +
+ {/* All-tier arm figures, present only on a run recorded after they were published. */} + {arm.arm_cost_total === undefined ? null : costWithheld ? ( + ) : costShown ? ( - <> - {(() => { - const cache = formatCostField(modelMetrics.cost_cache_total); - const excluding = formatCostField(modelMetrics.cost_excluding_cache); - return ( - <> - - Cache-attributable spend, shown apart from the cost excluding cache beside it. - The two are never summed into one figure — a cost gap that tracks cache - eligibility is not a gap between the models. - - } - /> - - - ); - })()} - + (() => { + const total = formatCostField(arm.arm_cost_total); + return ( + Every tier’s model calls: supervisor, subagent and tool.} + /> + ); + })() ) : ( )} + {typeof arm.arm_long_context_invocation_count === "number" ? ( + + ) : null} {/* Cost-completeness disclosures, worded as matrix.md words them. */} {lowerBoundNote !== undefined ? ( diff --git a/frontend/src/pages/evals/README.md b/frontend/src/pages/evals/README.md index 3bb45ae..8376fb0 100644 --- a/frontend/src/pages/evals/README.md +++ b/frontend/src/pages/evals/README.md @@ -16,7 +16,7 @@ each property below is enforced by a test. | **Repeats.** | A "Repeats" select (1 to `MAX_SWEEP_REPEATS`, mirroring `eval.runs.MAX_SWEEP_REPEATS`; default 1) sits above the stub button, and another inside `LiveSweepDialog`. Both submits, and the dialog's `estimate`, add `repeats` only when it is not 1. The dialog re-fetches the estimate for each selection and keeps "Start live sweep" disabled until that selection's estimate has loaded; the sweep size it shows multiplies tasks × arms × the estimate's `repeats`. A run payload without `repeats` reads as 1 (`types.ts` `runRepeats`). | `evals.test.tsx` ("Repeats") | | **Live sweep, admins only.** | "Run live sweep…" opens `LiveSweepDialog`, which fetches `estimate` first. "Start live sweep" stays disabled until an estimate has loaded, then sends `{action: "submit", mode: "live", confirmed: true}`. The sweep size is derived from the estimate. When the estimate's figures were measured at a supervisor effort other than the current default (`eval.sweep_units.DEFAULT_ARM_EFFORT`), the payload carries `effort_caveat` (`eval.runs.ESTIMATE_EFFORT_CAVEAT`), and the dialog shows it under the basis line; with no caveat, nothing is shown. A failed start keeps the dialog open with the reason. | `evals.test.tsx` ("live sweep, admins only", the effort-caveat cells), `src/auth/adminGroup.test.ts`, `tests/unit/test_runs_estimate_effort_caveat.py` | | **Cancel, admins only.** | "Cancel sweep" appears on a `queued` or `running` live run with no cancel requested yet. It is cooperative: the task in flight finishes, and the rest are skipped. | `evals.test.tsx` ("cancel") | -| **Tier comparison summary.** | Above the per-arm results, three tables (Supervisor, Subagent, ToolCall) headed `Metric` plus one column per arm in run order, then the notes that define each row. Every header, cell and note is the run payload's `tier_comparison_table` string (`eval.matrix.tier_comparison_table`, the same function that writes `matrix.md`'s `## Tier comparison summary`), rendered verbatim. A run without the table, or with one `EvalRunClient` cannot read, shows "Tier comparison summary is not available for this run." Cancelled runs with partial results show it as well. | `TierComparisonSummary.test.tsx`, `tests/integration/test_tier_comparison_ui_contract.py` (writes `__fixtures__/tierComparison.json`) | +| **Tier comparison summary.** | Above the per-arm results, three tables (Supervisor, Subagent, ToolCall) headed `Metric` plus one column per arm in run order, then the notes that define each row. Every header, cell and note is the run payload's `tier_comparison_table` string (`eval.matrix.tier_comparison_table`, the same function that writes `matrix.md`'s `## Tier comparison summary`), rendered as verbatim text; the one markdown feature honoured is a note's inline code spans, which render as code. Each table includes the `Long-context invocations` row. A run without the table, or with one `EvalRunClient` cannot read, shows "Tier comparison summary is not available for this run." Cancelled runs with partial results show it as well. | `TierComparisonSummary.test.tsx`, `tests/integration/test_tier_comparison_ui_contract.py` (writes `__fixtures__/tierComparison.json`) | | **Download matrix.md.** | "Download matrix.md" beside the matrix.md control asks `artefact_url` with `download: true`, which signs the URL with an attachment disposition (`matrix-.md`), and navigates to it, so the browser saves the file and the page stays. A refusal is shown in the artifact list's alert. | `evals.artefacts.test.tsx` | | **Cancelled and failed runs.** | A cancelled run reads `Cancelled after N/total tasks` with its spend, plus partial results when any task finished. A failed run shows the backend's failure summary. | `evals.test.tsx` | @@ -31,6 +31,7 @@ Admin status comes from the ID token's `cognito:groups` (`src/auth/adminGroup.ts | **Effort labels are not comparable across families.** `low` names a different setting on each family's own ladder. | An annotation inside the Effort field of every arm. | `evals.annotations.test.tsx` | | **Prompt-cache minimums are per model**, and cache-attributable cost is reported apart from cost excluding cache. | The arm's `cache_min_checkpoint_tokens`, the cache read/write token counts, and two separate cost fields — never one summed figure. | `evals.annotations.test.tsx` | | **Cost is shown only when the backend marks it publishable.** | Keyed on each record's `cost_withheld_provisional` flag: `true` renders a withheld disclosure, `false` renders the backend-emitted cost fields, absent renders "not displayed". This covers the per-tier summary's `cost_total`. The page computes no cost figure from token counts. | `evals.integrity.test.tsx` (adversarial payloads), `scripts/check_cost_withheld.sh --frontend-dir` (active while `config/pricing.yaml` is `provisional: true`) | +| **Supervisor-tier and all-tier arm figures are kept apart.** | The arm's usage and cost block (`model_metrics`) sits under a "Supervisor tier" heading that says it counts the supervisor's own model calls. Below it, "Arm cost, all tiers" (`arm_cost_total`) and "Long-context invocations, all tiers" (`arm_long_context_invocation_count`) cover every tier's calls, and appear only on a run recorded with them. The all-tier cost follows the same withheld and not-displayed rules as every other cost field. | `evals.armTotals.test.tsx` | | **Run spend is metered spend, live runs only.** | A live run's `cost_spent_usd` is shown as the backend sent it, except a live run with no measured item (`tasks_done` 0, `tasks_failed` above 0), which reads "not measured" rather than a zero figure, beside the errored-items note. A stub run reads "no inference cost (stub)" rather than a zero figure. An unrecognised mode shows no figure. | `evals.test.tsx`, `evals.allFailed.test.tsx` | | **No verdict, winner, or ranking.** | Arms render in backend order with identical styling and no sort affordance; nothing labels a model. The tier comparison columns share one style and have no sort control. The history table has no sort affordance either. | `evals.test.tsx` (including a source scan) + `evals.integrity.test.tsx` | diff --git a/frontend/src/pages/evals/TierComparisonSummary.test.tsx b/frontend/src/pages/evals/TierComparisonSummary.test.tsx index 905b554..0f5808a 100644 --- a/frontend/src/pages/evals/TierComparisonSummary.test.tsx +++ b/frontend/src/pages/evals/TierComparisonSummary.test.tsx @@ -165,6 +165,72 @@ describe("TierComparisonSummary — verbatim cells", () => { }); }); +const LONG_CONTEXT_ROW = "Long-context invocations"; + +function rowLabels(element: HTMLElement): string[] { + return within(element) + .getAllByRole("rowheader") + .map((cell) => cell.textContent ?? ""); +} + +describe("TierComparisonSummary — long-context invocations", () => { + it.each(["multi_arm", "single_arm"])( + "shows the row in every tier of the %s fixture, directly after Model invocations", + (name) => { + render(); + + for (const tier of ["supervisor", "subagent", "tool"]) { + const labels = rowLabels(screen.getByTestId(`tier-comparison-table-${tier}`)); + const at = labels.indexOf(LONG_CONTEXT_ROW); + expect(at, tier).toBeGreaterThan(0); + expect(labels[at - 1], tier).toBe("Model invocations"); + } + }, + ); + + it("shows an em dash for every arm of a run recorded without the count", () => { + const table = fixtureTable("multi_arm"); + render(); + + for (const section of table.sections) { + const row = within(screen.getByTestId(`tier-comparison-table-${section.tier}`)) + .getByRole("rowheader", { name: LONG_CONTEXT_ROW }) + .closest("tr"); + expect(row).not.toBeNull(); + const cells = within(row as HTMLElement).getAllByRole("cell").map((cell) => cell.textContent); + expect(cells).toEqual(table.headers.slice(1).map(() => "—")); + } + }); + + it("renders a recorded count exactly as sent, zero included", () => { + const base = fixtureTable("multi_arm"); + const table: EvalTierComparisonTable = { + ...base, + sections: base.sections.map((section) => ({ + ...section, + rows: section.rows.map((row) => + row[0] === LONG_CONTEXT_ROW ? [LONG_CONTEXT_ROW, "0", "3"] : row, + ), + })), + }; + render(); + + const row = within(screen.getByTestId("tier-comparison-table-subagent")) + .getByRole("rowheader", { name: LONG_CONTEXT_ROW }) + .closest("tr"); + expect(within(row as HTMLElement).getAllByRole("cell").map((cell) => cell.textContent)).toEqual([ + "0", + "3", + ]); + }); + + it("defines the row in the notes, matrix.md's wording", () => { + render(); + const notes = screen.getByRole("list", { name: /notes on the tier comparison/i }); + expect(notes).toHaveTextContent(/Long-context invocations counts those priced at the model's long-context rates/); + }); +}); + describe("TierComparisonSummary — markdown code spans in notes", () => { const CODE_NOTE = 'Effort labels are not equivalent across families; see `effort` under "Reading these figures".'; diff --git a/frontend/src/pages/evals/__fixtures__/tierComparison.json b/frontend/src/pages/evals/__fixtures__/tierComparison.json index dab09dd..ad67682 100644 --- a/frontend/src/pages/evals/__fixtures__/tierComparison.json +++ b/frontend/src/pages/evals/__fixtures__/tierComparison.json @@ -56,6 +56,11 @@ "2", "2" ], + [ + "Long-context invocations", + "—", + "—" + ], [ "Input tokens", "100", @@ -132,6 +137,11 @@ "2", "2" ], + [ + "Long-context invocations", + "—", + "—" + ], [ "Input tokens", "200", @@ -228,6 +238,11 @@ "2", "2" ], + [ + "Long-context invocations", + "—", + "—" + ], [ "Input tokens", "300", @@ -257,7 +272,7 @@ } ], "notes": [ - "Model lists the tier's configured model and every model that spent tokens at the tier; Effort lists the efforts the tier's turns ran at, with unsupported for a model that takes no effort setting. Task completion rate is rate (met/applicable) over the arm's tasks applicable at the tier, where n is the applicable task count: at Supervisor a task's correctness is 1.0, at Subagent the observed delegation outcome equals the expected one, and at ToolCall an expected tool was called with schema-valid arguments. Task accuracy is the arm's mean_correctness at Supervisor and the completion rate at Subagent and ToolCall. Task completion over repeats is rate (completed/applicable) over the arm's tasks applicable at the tier in any repeat, where a task counts when its tier expectation was met in a strict majority of its applicable repeats, and N is the per-task repeat count. Cost per completed task divides the tier's total cost by its completed task count, so spend on tasks that did not complete is included; Cost per task, all tasks divides the same total by every task the arm ran. Latency per completed task sums the tier's measured call latencies within each completed task and takes the nearest-rank percentile over those tasks; a completed task with no measured call at the tier contributes no sample, and n counts the contributing tasks. Model invocations sums the tier's included invocations. Self-corrected retries counts the tier's tool calls whose tool_failure result was followed, later in the same task, by a call to the same tool from the same role that returned data or no_match; such a call does not set the task's delegation outcome. ToolCall makes one call per invocation, so that row is not applicable there.", + "Model lists the tier's configured model and every model that spent tokens at the tier; Effort lists the efforts the tier's turns ran at, with unsupported for a model that takes no effort setting. Task completion rate is rate (met/applicable) over the arm's tasks applicable at the tier, where n is the applicable task count: at Supervisor a task's correctness is 1.0, at Subagent the observed delegation outcome equals the expected one, and at ToolCall an expected tool was called with schema-valid arguments. Task accuracy is the arm's mean_correctness at Supervisor and the completion rate at Subagent and ToolCall. Task completion over repeats is rate (completed/applicable) over the arm's tasks applicable at the tier in any repeat, where a task counts when its tier expectation was met in a strict majority of its applicable repeats, and N is the per-task repeat count. Cost per completed task divides the tier's total cost by its completed task count, so spend on tasks that did not complete is included; Cost per task, all tasks divides the same total by every task the arm ran. Latency per completed task sums the tier's measured call latencies within each completed task and takes the nearest-rank percentile over those tasks; a completed task with no measured call at the tier contributes no sample, and n counts the contributing tasks. Model invocations sums the tier's included invocations; Long-context invocations counts those priced at the model's long-context rates, and reads — for a run recorded before that count existed. Self-corrected retries counts the tier's tool calls whose tool_failure result was followed, later in the same task, by a call to the same tool from the same role that returned data or no_match; such a call does not set the task's delegation outcome. ToolCall makes one call per invocation, so that row is not applicable there.", "Columns are the sweep's arms in run order, not sorted by any result, and no difference or ratio between columns is computed.", "Effort labels are not equivalent across families: each family declares its own effort scale (config/model_capabilities.yaml, effort_scale_comparability); see `effort` under \"Reading these figures\".", "Input tokens are each family's own reported count and are not like-for-like across families (token_accounting_disclosure, src/eval/matrix.py); see `input_tokens` comparability under \"Reading these figures\".", @@ -313,6 +328,10 @@ "Model invocations", "2" ], + [ + "Long-context invocations", + "—" + ], [ "Input tokens", "100" @@ -375,6 +394,10 @@ "Model invocations", "2" ], + [ + "Long-context invocations", + "—" + ], [ "Input tokens", "200" @@ -453,6 +476,10 @@ "Model invocations", "2" ], + [ + "Long-context invocations", + "—" + ], [ "Input tokens", "300" @@ -477,7 +504,7 @@ } ], "notes": [ - "Model lists the tier's configured model and every model that spent tokens at the tier; Effort lists the efforts the tier's turns ran at, with unsupported for a model that takes no effort setting. Task completion rate is rate (met/applicable) over the arm's tasks applicable at the tier, where n is the applicable task count: at Supervisor a task's correctness is 1.0, at Subagent the observed delegation outcome equals the expected one, and at ToolCall an expected tool was called with schema-valid arguments. Task accuracy is the arm's mean_correctness at Supervisor and the completion rate at Subagent and ToolCall. Task completion over repeats is rate (completed/applicable) over the arm's tasks applicable at the tier in any repeat, where a task counts when its tier expectation was met in a strict majority of its applicable repeats, and N is the per-task repeat count. Cost per completed task divides the tier's total cost by its completed task count, so spend on tasks that did not complete is included; Cost per task, all tasks divides the same total by every task the arm ran. Latency per completed task sums the tier's measured call latencies within each completed task and takes the nearest-rank percentile over those tasks; a completed task with no measured call at the tier contributes no sample, and n counts the contributing tasks. Model invocations sums the tier's included invocations. Self-corrected retries counts the tier's tool calls whose tool_failure result was followed, later in the same task, by a call to the same tool from the same role that returned data or no_match; such a call does not set the task's delegation outcome. ToolCall makes one call per invocation, so that row is not applicable there.", + "Model lists the tier's configured model and every model that spent tokens at the tier; Effort lists the efforts the tier's turns ran at, with unsupported for a model that takes no effort setting. Task completion rate is rate (met/applicable) over the arm's tasks applicable at the tier, where n is the applicable task count: at Supervisor a task's correctness is 1.0, at Subagent the observed delegation outcome equals the expected one, and at ToolCall an expected tool was called with schema-valid arguments. Task accuracy is the arm's mean_correctness at Supervisor and the completion rate at Subagent and ToolCall. Task completion over repeats is rate (completed/applicable) over the arm's tasks applicable at the tier in any repeat, where a task counts when its tier expectation was met in a strict majority of its applicable repeats, and N is the per-task repeat count. Cost per completed task divides the tier's total cost by its completed task count, so spend on tasks that did not complete is included; Cost per task, all tasks divides the same total by every task the arm ran. Latency per completed task sums the tier's measured call latencies within each completed task and takes the nearest-rank percentile over those tasks; a completed task with no measured call at the tier contributes no sample, and n counts the contributing tasks. Model invocations sums the tier's included invocations; Long-context invocations counts those priced at the model's long-context rates, and reads — for a run recorded before that count existed. Self-corrected retries counts the tier's tool calls whose tool_failure result was followed, later in the same task, by a call to the same tool from the same role that returned data or no_match; such a call does not set the task's delegation outcome. ToolCall makes one call per invocation, so that row is not applicable there.", "Columns are the sweep's arms in run order, not sorted by any result, and no difference or ratio between columns is computed.", "Effort labels are not equivalent across families: each family declares its own effort scale (config/model_capabilities.yaml, effort_scale_comparability); see `effort` under \"Reading these figures\".", "Input tokens are each family's own reported count and are not like-for-like across families (token_accounting_disclosure, src/eval/matrix.py); see `input_tokens` comparability under \"Reading these figures\".", diff --git a/frontend/src/pages/evals/evals.armTotals.test.tsx b/frontend/src/pages/evals/evals.armTotals.test.tsx new file mode 100644 index 0000000..e083226 --- /dev/null +++ b/frontend/src/pages/evals/evals.armTotals.test.tsx @@ -0,0 +1,136 @@ +// SPDX-License-Identifier: MIT +/** + * The arm's `model_metrics` figures are labelled as the supervisor tier's, and the arm's all-tier + * `arm_cost_total` and `arm_long_context_invocation_count` are shown beside them when the run + * publishes them, withheld while pricing is provisional and absent on an older record. + */ +import { cleanup, render, screen, within } from "@testing-library/react"; +import { afterEach, describe, expect, it } from "vitest"; + +import { EvalArmResults } from "./EvalArmResults"; +import type { EvalArm } from "./types"; + +const MODEL = "anthropic.claude-sonnet-5-5"; + +afterEach(() => { + cleanup(); +}); + +function arm(overrides: Partial = {}, metrics: Partial = {}): EvalArm { + return { + provenance: { + mode: "live", + timestamp: "2026-11-08T00:00:00Z", + model_id: MODEL, + effort: "medium", + dataset_seed: 20261108, + git_revision: "not-a-commit", + runtime_identifier: "agentcore", + price_table_version: "1.3.0", + }, + model_metrics: { + model_id: MODEL, + included_invocation_count: 12, + cache_read_tokens: 3000, + cache_write_tokens: 900, + cost_withheld_provisional: false, + cost_cache_total: 0.12, + cost_excluding_cache: 0.35, + ...metrics, + }, + aggregate_score: { mean_correctness: 0.5, scored_task_count: 2, unscored_task_count: 0, bad_write_rate: 0 }, + task_scores: [{ task_id: "gt-01", correctness: 1, bad_write: false }], + ...overrides, + }; +} + +/** The value element of the `
` labelled exactly `label` within `container`. */ +function fieldValue(container: HTMLElement, label: string): string { + const term = within(container).getByText(label, { selector: "dt" }); + return term.nextElementSibling?.firstElementChild?.textContent ?? ""; +} + +function supervisorGroup(): HTMLElement { + return screen.getByRole("region", { name: "Supervisor tier" }); +} + +describe("supervisor-scoped arm figures", () => { + it("places the cache token and cost figures under a Supervisor tier heading", () => { + render(); + + const group = supervisorGroup(); + expect(within(group).getByRole("heading", { level: 4 })).toHaveTextContent("Supervisor tier"); + expect(fieldValue(group, "Cache read tokens")).toBe("3000 tokens"); + expect(fieldValue(group, "Cache write tokens")).toBe("900 tokens"); + expect(fieldValue(group, "Cost, cache-attributable")).toBe("$0.12"); + expect(fieldValue(group, "Cost, excluding cache")).toBe("$0.35"); + expect(fieldValue(group, "Included invocations")).toBe("12"); + }); + + it("keeps arm-wide figures outside the supervisor group", () => { + render(); + + const group = supervisorGroup(); + expect(within(group).queryByText("Mean correctness")).toBeNull(); + expect(within(group).queryByText("Bad-write rate")).toBeNull(); + expect(within(screen.getByRole("region", { name: MODEL })).getByText("Mean correctness")).toBeInTheDocument(); + }); + + it("labels the withheld cost fields as supervisor-tier too", () => { + render(); + + expect(fieldValue(supervisorGroup(), "Cost, cache-attributable")).toMatch(/withheld/); + }); +}); + +describe("all-tier arm figures", () => { + it("shows the all-tier arm cost and long-context count outside the supervisor group", () => { + render( + , + ); + + const section = screen.getByRole("region", { name: MODEL }); + expect(fieldValue(section, "Arm cost, all tiers")).toBe("$1.25"); + expect(fieldValue(section, "Long-context invocations, all tiers")).toBe("3"); + const group = supervisorGroup(); + expect(within(group).queryByText("Arm cost, all tiers")).toBeNull(); + expect(within(group).queryByText("Long-context invocations, all tiers")).toBeNull(); + }); + + it("shows a zero long-context count as 0, not as absent", () => { + render(); + + expect(fieldValue(screen.getByRole("region", { name: MODEL }), "Long-context invocations, all tiers")).toBe("0"); + }); + + it("withholds the all-tier cost while pricing is provisional, still showing the count", () => { + render( + , + ); + + const section = screen.getByRole("region", { name: MODEL }); + const shown = fieldValue(section, "Arm cost, all tiers"); + expect(shown).toMatch(/withheld/); + expect(shown).not.toMatch(/\$/); + expect(fieldValue(section, "Long-context invocations, all tiers")).toBe("2"); + }); + + it("renders an unpriced all-tier cost as not reported, never as $0", () => { + render(); + + expect(fieldValue(screen.getByRole("region", { name: MODEL }), "Arm cost, all tiers")).toBe("not reported"); + }); + + it("shows neither all-tier figure for a run recorded before they were published", () => { + render(); + + const section = screen.getByRole("region", { name: MODEL }); + expect(within(section).queryByText("Arm cost, all tiers")).toBeNull(); + expect(within(section).queryByText("Long-context invocations, all tiers")).toBeNull(); + }); +}); diff --git a/frontend/src/pages/evals/types.ts b/frontend/src/pages/evals/types.ts index 44f8cd2..1e6648d 100644 --- a/frontend/src/pages/evals/types.ts +++ b/frontend/src/pages/evals/types.ts @@ -24,6 +24,9 @@ export interface EvalTaskScore { export interface EvalModelMetrics { readonly model_id: string; readonly included_invocation_count: number; + /** Invocations priced at the model's long-context rates; absent on a matrix written before the + * count existed. */ + readonly long_context_invocation_count?: number; readonly input_tokens?: number; readonly output_tokens?: number; /** Tokens written to a new prompt-cache checkpoint; published even while cost is withheld. */ @@ -121,6 +124,9 @@ export interface EvalTierRoleModelMetric { readonly role: string; readonly model_id: string; readonly included_invocation_count: number; + /** Invocations priced at the model's long-context rates; absent on a matrix written before the + * count existed. */ + readonly long_context_invocation_count?: number; readonly input_tokens: number; readonly output_tokens: number; readonly cache_read_tokens: number; @@ -185,6 +191,9 @@ export interface EvalTierSummaryEntry { readonly latency_p50_ms: number | null; readonly latency_p95_ms: number | null; readonly success: EvalTierSuccess; + /** Invocations priced at the model's long-context rates; absent on a matrix written before the + * count existed. */ + readonly long_context_invocation_count?: number; } /** `eval.metrics.TierSummary.to_report_fields()`, keyed by tier. The writer emits every tier; a tier @@ -207,6 +216,11 @@ export interface EvalArm { readonly task_spread?: readonly EvalTaskSpread[]; /** The per-tier summary; absent on a matrix written before it existed. */ readonly tier_summary?: EvalTierSummary; + /** The arm's cost over every tier's calls (`model_metrics` covers the supervisor tier only); + * absent on a matrix written before it existed, `null` when not priced. */ + readonly arm_cost_total?: number | null; + /** The arm's long-context invocations over every tier; absent on an older matrix. */ + readonly arm_long_context_invocation_count?: number; /** * `eval.matrix.USAGE_LOWER_BOUND_TASK_IDS_KEY`: the task runs, in run order, in which a model call * ended before reporting token usage. Absent when every run reported its usage, and on a matrix diff --git a/infra/main.tf b/infra/main.tf index 19e5ce9..c555927 100644 --- a/infra/main.tf +++ b/infra/main.tf @@ -657,7 +657,8 @@ module "lambda_action" { ) environment_kms_key_arn = module.observability.cloudwatch_logs_kms_key_arn - config_parameter_arns = contains(local.routed_tool_names, each.key) ? [local.api_allowed_client_ids_parameter_arn] : [] + config_parameter_arns = contains(local.routed_tool_names, each.key) ? [local.api_allowed_client_ids_parameter_arn] : [] + config_parameter_kms_key_arn = contains(local.routed_tool_names, each.key) ? aws_kms_key.api_allowlist.arn : null cursor_kms_key_arn = contains(local.cursor_tools, each.key) ? aws_kms_key.cursor.arn : null cursor_kms_grant_enabled = contains(local.cursor_tools, each.key) @@ -886,8 +887,9 @@ module "lambda_action_eval_trigger" { EVAL_ARTIFACTS_BUCKET = module.eval_artifacts.bucket_name EVAL_ARTIFACTS_PREFIX = module.eval_artifacts.artifact_prefix } - environment_kms_key_arn = module.observability.cloudwatch_logs_kms_key_arn - config_parameter_arns = [local.api_allowed_client_ids_parameter_arn] + environment_kms_key_arn = module.observability.cloudwatch_logs_kms_key_arn + config_parameter_arns = [local.api_allowed_client_ids_parameter_arn] + config_parameter_kms_key_arn = aws_kms_key.api_allowlist.arn reserved_concurrent_executions = local.reserved_concurrency["eval_trigger"] @@ -941,8 +943,9 @@ module "lambda_action_capabilities" { # Required for this zip's layout: the first-party packages live under `src/`. PYTHONPATH = "/var/task/src" } - environment_kms_key_arn = module.observability.cloudwatch_logs_kms_key_arn - config_parameter_arns = [local.api_allowed_client_ids_parameter_arn] + environment_kms_key_arn = module.observability.cloudwatch_logs_kms_key_arn + config_parameter_arns = [local.api_allowed_client_ids_parameter_arn] + config_parameter_kms_key_arn = aws_kms_key.api_allowlist.arn reserved_concurrent_executions = local.reserved_concurrency["capabilities"] @@ -1585,16 +1588,164 @@ locals { } resource "aws_ssm_parameter" "api_allowed_client_ids" { - # checkov:skip=CKV2_AWS_34: holds the SPA client id, a public value, not a secret - name = local.api_allowed_client_ids_parameter_name - type = "String" - value = module.cognito.spa_client_id + name = local.api_allowed_client_ids_parameter_name + type = "SecureString" + tier = "Standard" + key_id = aws_kms_key.api_allowlist.arn + value = module.cognito.spa_client_id + + tags = { + "data-classification" = "internal" + } + + depends_on = [aws_kms_key_policy.api_allowlist] +} + +# --------------------------------------------------------------------------- +# api_allowlist — the CMK encrypting the allowlist SecureString. Operators write and read it through +# SSM; the consumer roles may only decrypt it through SSM under the parameter's encryption context. +# --------------------------------------------------------------------------- + +locals { + api_allowlist_ssm_service = "ssm.${var.aws_region}.amazonaws.com" + + # Every role passed the allowlist parameter in `config_parameter_arns`. + api_allowlist_key_consumer_arns = concat( + [for name in local.routed_tool_names : module.lambda_action[name].role_arn], + [ + module.lambda_action_eval_trigger.role_arn, + module.lambda_action_capabilities.role_arn, + module.lambda_action_user_settings.role_arn, + ], + ) +} + +resource "aws_kms_key" "api_allowlist" { + # checkov:skip=CKV2_AWS_64: policy managed by aws_kms_key_policy.api_allowlist + description = "${local.name_prefix} API client allowlist parameter" + enable_key_rotation = true + deletion_window_in_days = 7 tags = { "data-classification" = "internal" } } +resource "aws_kms_alias" "api_allowlist" { + name = "alias/${local.name_prefix}-api-allowlist" + target_key_id = aws_kms_key.api_allowlist.key_id +} + +# Keeps the account-root statement, admits the operators to write and read the parameter and the +# consumers to read it, both only through SSM, and denies writes, key use and key administration +# outside those lists. +resource "aws_kms_key_policy" "api_allowlist" { + key_id = aws_kms_key.api_allowlist.id + bypass_policy_lockout_safety_check = false + + lifecycle { + precondition { + condition = contains(concat(var.crm_operator_principal_arns, ["arn:aws:iam::${data.aws_caller_identity.current.account_id}:root"]), data.aws_iam_session_context.current.issuer_arn) && data.aws_iam_session_context.current.issuer_arn != "arn:aws:iam::${data.aws_caller_identity.current.account_id}:root" + error_message = "The caller running this plan is not an operator. Add its role or user ARN to crm_operator_principal_arns in terraform.tfvars, or run as a listed operator; nothing was applied." + } + } + + policy = jsonencode({ + Version = "2012-10-17" + Statement = [ + { + Sid = "EnableIAMUserPermissions" + Effect = "Allow" + Principal = { AWS = "arn:aws:iam::${data.aws_caller_identity.current.account_id}:root" } + Action = "kms:*" + Resource = "*" + }, + { + Sid = "AllowOperatorsUseThroughSsm" + Effect = "Allow" + Principal = { AWS = var.crm_operator_principal_arns } + Action = [ + "kms:Encrypt", + "kms:Decrypt", + "kms:DescribeKey", + ] + Resource = "*" + Condition = { + StringEquals = { "kms:ViaService" = local.api_allowlist_ssm_service } + } + }, + { + Sid = "AllowConsumersDecryptTheParameter" + Effect = "Allow" + Principal = { AWS = local.api_allowlist_key_consumer_arns } + Action = "kms:Decrypt" + Resource = "*" + Condition = { + StringEquals = { + "kms:ViaService" = local.api_allowlist_ssm_service + "kms:EncryptionContext:PARAMETER_ARN" = local.api_allowed_client_ids_parameter_arn + } + } + }, + { + Sid = "DenyKeyWriteToNonOperators" + Effect = "Deny" + Principal = "*" + Action = [ + "kms:Encrypt", + "kms:GenerateDataKey*", + "kms:ReEncrypt*", + "kms:CreateGrant", + ] + Resource = "*" + Condition = { + ArnNotEquals = { "aws:PrincipalArn" = var.crm_operator_principal_arns } + Bool = { "aws:PrincipalIsAWSService" = "false" } + } + }, + { + Sid = "DenyKeyUseToNonProjectPrincipals" + Effect = "Deny" + Principal = "*" + Action = [ + "kms:Decrypt", + "kms:Encrypt", + "kms:GenerateDataKey*", + "kms:ReEncrypt*", + "kms:CreateGrant", + ] + Resource = "*" + Condition = { + ArnNotEquals = { "aws:PrincipalArn" = concat(local.api_allowlist_key_consumer_arns, var.crm_operator_principal_arns) } + Bool = { "aws:PrincipalIsAWSService" = "false" } + } + }, + { + Sid = "DenyKeyAdministrationToNonOperators" + Effect = "Deny" + Principal = "*" + Action = [ + "kms:PutKeyPolicy", + "kms:ScheduleKeyDeletion", + "kms:DisableKey", + "kms:RevokeGrant", + "kms:DisableKeyRotation", + "kms:CreateAlias", + "kms:UpdateAlias", + "kms:DeleteAlias", + "kms:TagResource", + "kms:UntagResource", + "kms:UpdateKeyDescription", + ] + Resource = "*" + Condition = { + ArnNotEquals = { "aws:PrincipalArn" = concat(var.crm_operator_principal_arns, ["arn:aws:iam::${data.aws_caller_identity.current.account_id}:root"]) } + } + }, + ] + }) +} + resource "aws_ssm_parameter" "relay_cognito_audience" { # checkov:skip=CKV2_AWS_34: holds the SPA client id, a public value, not a secret name = local.relay_cognito_audience_parameter_name diff --git a/infra/modules/frontend_cdn/cloudfront.tf b/infra/modules/frontend_cdn/cloudfront.tf index b515265..e015423 100644 --- a/infra/modules/frontend_cdn/cloudfront.tf +++ b/infra/modules/frontend_cdn/cloudfront.tf @@ -124,8 +124,8 @@ resource "aws_cloudfront_response_headers_policy" "security_headers" { strict_transport_security { access_control_max_age_sec = 31536000 - include_subdomains = false - preload = false + include_subdomains = true + preload = true override = true } @@ -164,6 +164,9 @@ resource "aws_cloudfront_function" "spa_history_fallback" { code = file("${path.module}/functions/spa_history_fallback.js") } +# The default *.cloudfront.net certificate fixes the viewer minimum protocol; only a custom-domain +# certificate could set one. +# nosemgrep: terraform.aws.security.aws-insecure-cloudfront-distribution-tls-version, terraform.aws.security.aws-cloudfront-insecure-tls.aws-insecure-cloudfront-distribution-tls-version resource "aws_cloudfront_distribution" "frontend" { # checkov:skip=CKV_AWS_68: no WAF on the distribution, an open known limitation # checkov:skip=CKV2_AWS_47: no WAF on the distribution, an open known limitation diff --git a/infra/modules/lambda_action/README.md b/infra/modules/lambda_action/README.md index e2571d7..588bfee 100644 --- a/infra/modules/lambda_action/README.md +++ b/infra/modules/lambda_action/README.md @@ -22,7 +22,10 @@ baseline](../../../docs/security-baseline.md#h-7). Modules named here: [`api`](. `config_parameter_arns` when set (`logs`); manage its own Hyperplane ENIs and write X-Ray segments (`vpc_access`, the two action sets that take `Resource = "*"`); DynamoDB grants and denies (when any are configured and `dynamodb_policy_managed = false`); DynamoDB KMS grant, plus the - context-bound cursor-key grant when `cursor_kms_grant_enabled` (`dynamodb_kms`, when enabled). + context-bound cursor-key grant when `cursor_kms_grant_enabled` (`dynamodb_kms`, when enabled); + and, when `config_parameter_arns` is set, `kms:Decrypt` on `config_parameter_kms_key_arn` through SSM + under each parameter's `PARAMETER_ARN` encryption context, with a Deny on `kms:Decrypt` through SSM + under any other key (`config_parameter_kms`). - `aws_iam_policy.dynamodb` (keyed `allow`/`deny`, plus `allow-2`, ... when `dynamodb_allow_statements_per_policy` splits the Allows) and `aws_iam_role_policy_attachment.dynamodb_deny` / `.dynamodb_allow[*]`, when @@ -63,6 +66,7 @@ baseline](../../../docs/security-baseline.md#h-7). Modules named here: [`api`](. | `cursor_kms_key_arn` | no (`null`) | The pagination-cursor CMK (`aws_kms_key.cursor` in `infra/main.tf`). Required when `cursor_kms_grant_enabled` is true (validation). | | `cursor_kms_grant_enabled` | no (`false`) | Adds a `SealPaginationCursors` statement to the `-dynamodb-kms` policy: `kms:Encrypt` and `kms:Decrypt` on `cursor_kms_key_arn`, only with `kms:EncryptionContext:purpose = crm-cursor`. Requires `dynamodb_kms_grant_enabled` (validation). The root sets it for the tools in `local.cursor_tools`, which also get `CRM_CURSOR_KMS_KEY_ARN`. Their roles are listed in `local.cursor_key_principal_arns`, because the key policy denies key use to every other role. | | `config_parameter_arns` | no (`[]`) | SSM parameter ARNs the function reads at runtime. Non-empty adds a `ReadConfigParameters` statement (`ssm:GetParameter` on exactly these ARNs) to the `-logs` policy. Wildcards are rejected by validation. The root passes `local.api_allowed_client_ids_parameter_arn` to every API-Gateway-integrated function, beside `CRM_API_ALLOWED_CLIENT_IDS_PARAMETER`. | +| `config_parameter_kms_key_arn` | when `config_parameter_arns` is set (`null`) | The CMK encrypting the SecureString `config_parameter_arns`. Adds the `-config-kms` policy: a `DecryptConfigParameters` Allow (`kms:Decrypt` on this key, `kms:ViaService = ssm..amazonaws.com`, `kms:EncryptionContext:PARAMETER_ARN` in `config_parameter_arns`) and a `DenyConfigParameterDecryptUnderOtherKeys` Deny (`kms:Decrypt` on every other key through `ssm.*.amazonaws.com`), so a parameter re-encrypted under `aws/ssm` is unreadable. Validation rejects a wildcard ARN and a null key beside non-empty `config_parameter_arns`. The root passes `aws_kms_key.api_allowlist.arn`, and lists the same roles in `local.api_allowlist_key_consumer_arns` for the key policy. | | `gateway_target_role_arn` | no (`null`) | The Gateway target execution role admitted to invoke this function. | | `attach_gateway_target_grant` | no (`false`) | Drives the Gateway permission's `count`. Must agree with `gateway_target_role_arn` being non-null (enforced by a precondition). A flag rather than a null check on the ARN so an empty-state plan has a known `count`. | | `table_action_role_matrix_entries` | no (`[]`) | This instantiation's rows of the isolation matrix (below). Not read at plan time; echoed by `table_action_role_matrix_rows`. | diff --git a/infra/modules/lambda_action/main.tf b/infra/modules/lambda_action/main.tf index e902b6a..5a0a6e6 100644 --- a/infra/modules/lambda_action/main.tf +++ b/infra/modules/lambda_action/main.tf @@ -55,6 +55,56 @@ resource "aws_iam_role_policy" "logs" { policy = data.aws_iam_policy_document.logs.json } +locals { + config_parameter_region = length(var.config_parameter_arns) > 0 ? split(":", var.config_parameter_arns[0])[3] : null +} + +# Decrypt for the SecureString config parameters, only through SSM under each parameter's encryption +# context, and a refusal to decrypt through SSM under any other key. +data "aws_iam_policy_document" "config_parameter_kms" { + count = length(var.config_parameter_arns) > 0 ? 1 : 0 + + statement { + sid = "DecryptConfigParameters" + effect = "Allow" + actions = ["kms:Decrypt"] + resources = [var.config_parameter_kms_key_arn] + + condition { + test = "StringEquals" + variable = "kms:ViaService" + values = ["ssm.${local.config_parameter_region}.amazonaws.com"] + } + + condition { + test = "StringEquals" + variable = "kms:EncryptionContext:PARAMETER_ARN" + values = var.config_parameter_arns + } + } + + statement { + sid = "DenyConfigParameterDecryptUnderOtherKeys" + effect = "Deny" + actions = ["kms:Decrypt"] + not_resources = [var.config_parameter_kms_key_arn] + + condition { + test = "StringLike" + variable = "kms:ViaService" + values = ["ssm.*.amazonaws.com"] + } + } +} + +resource "aws_iam_role_policy" "config_parameter_kms" { + count = length(var.config_parameter_arns) > 0 ? 1 : 0 + + name = "${var.function_name}-config-kms" + role = aws_iam_role.this.id + policy = data.aws_iam_policy_document.config_parameter_kms[0].json +} + # --------------------------------------------------------------------------- # Grants that take no resource-level scoping: Hyperplane ENI management for `vpc_config`, and # X-Ray trace writes for `tracing_config`. diff --git a/infra/modules/lambda_action/variables.tf b/infra/modules/lambda_action/variables.tf index 53efba0..4ce756d 100644 --- a/infra/modules/lambda_action/variables.tf +++ b/infra/modules/lambda_action/variables.tf @@ -238,6 +238,27 @@ variable "config_parameter_arns" { } } +variable "config_parameter_kms_key_arn" { + description = <<-EOT + ARN of the customer-managed KMS key encrypting the SecureString `config_parameter_arns`. The + `-config-kms` policy allows `kms:Decrypt` on it through SSM under each parameter's + `PARAMETER_ARN` encryption context, and denies `kms:Decrypt` through SSM on every other key. + Required when `config_parameter_arns` is non-empty. + EOT + type = string + default = null + + validation { + condition = var.config_parameter_kms_key_arn == null || can(regex("^arn:aws[a-z-]*:kms:[a-z0-9-]+:[0-9]{12}:key/[A-Za-z0-9-]+$", var.config_parameter_kms_key_arn)) + error_message = "config_parameter_kms_key_arn must be null or one fully-qualified KMS key ARN with no wildcard." + } + + validation { + condition = length(var.config_parameter_arns) == 0 || var.config_parameter_kms_key_arn != null + error_message = "config_parameter_arns is non-empty but config_parameter_kms_key_arn is null: the parameters could not be decrypted." + } +} + variable "cursor_kms_key_arn" { description = <<-EOT ARN of the customer-managed KMS key that seals pagination cursors. The role's diff --git a/infra/outputs.tf b/infra/outputs.tf index 38f80a7..d1cf8a1 100644 --- a/infra/outputs.tf +++ b/infra/outputs.tf @@ -228,3 +228,8 @@ output "eval_trigger_function_name" { value = module.lambda_action_eval_trigger.function_name } + +output "api_allowlist_kms_key_arn" { + description = "The CMK encrypting the API client-allowlist SecureString parameter. Read by the smoke checks to assert the parameter's key, the key policy and rotation, and to simulate the roles' KMS access." + value = aws_kms_key.api_allowlist.arn +} diff --git a/infra/templates/eval_sweep.asl.json b/infra/templates/eval_sweep.asl.json index 05cb132..d699c84 100644 --- a/infra/templates/eval_sweep.asl.json +++ b/infra/templates/eval_sweep.asl.json @@ -80,7 +80,8 @@ "FunctionName": "${sweep_control_arn}", "Payload": { "step": "seed", - "run_id.$": "$.run_id" + "run_id.$": "$.run_id", + "execution_arn.$": "$$.Execution.Id" } }, "ResultPath": null, @@ -205,6 +206,7 @@ "Payload": { "step": "aggregate", "run_id.$": "$.run_id", + "execution_arn.$": "$$.Execution.Id", "error.$": "$.error" } }, diff --git a/infra/user_settings.tf b/infra/user_settings.tf index 7440508..18977fc 100644 --- a/infra/user_settings.tf +++ b/infra/user_settings.tf @@ -27,8 +27,9 @@ module "lambda_action_user_settings" { # Required for this zip's layout: the first-party packages live under `src/`. PYTHONPATH = "/var/task/src" } - environment_kms_key_arn = module.observability.cloudwatch_logs_kms_key_arn - config_parameter_arns = [local.api_allowed_client_ids_parameter_arn] + environment_kms_key_arn = module.observability.cloudwatch_logs_kms_key_arn + config_parameter_arns = [local.api_allowed_client_ids_parameter_arn] + config_parameter_kms_key_arn = aws_kms_key.api_allowlist.arn reserved_concurrent_executions = local.reserved_concurrency["user_settings"] diff --git a/results/matrix.md b/results/matrix.md index f3c8af1..23782a7 100644 --- a/results/matrix.md +++ b/results/matrix.md @@ -10,18 +10,19 @@ Per-tier figures for each arm, one column per arm in run order; the notes below | --- | --- | --- | | Model | anthropic.claude-sonnet-5-5 | openai.gpt-6.1-sol | | Effort | medium | medium | -| Task completion rate | 0.933333 (112/120) | 1.000000 (120/120) | +| Task completion rate | 0.941667 (113/120) | 0.991667 (119/120) | | Task completion over repeats | 0.958333 (23/24, majority of 5) | 1.000000 (24/24, majority of 5) | -| Task accuracy | 0.982130 | 1.000000 | -| Cost per completed task | 0.013337 | 0.005343 | -| P95 latency per completed task (ms) | 11197.106291 (n=111) | 11562.644292 (n=115) | -| P50 latency per completed task (ms) | 4035.949583 (n=111) | 4684.243208 (n=115) | -| Model invocations | 281 | 242 | -| Input tokens | 437073 | 692274 | -| Output tokens | 49150 | 23443 | -| Cost per task, all tasks | 0.012448 | 0.005343 | +| Task accuracy | 0.984167 | 0.997500 | +| Cost per completed task | 0.011594 | 0.005498 | +| P95 latency per completed task (ms) | 10541.749459 (n=110) | 10262.295082 (n=114) | +| P50 latency per completed task (ms) | 4630.674541 (n=110) | 4112.333292 (n=114) | +| Model invocations | 273 | 242 | +| Long-context invocations | 0 | 0 | +| Input tokens | 350986 | 727458 | +| Output tokens | 47818 | 23719 | +| Cost per task, all tasks | 0.010917 | 0.005452 | | Self-corrected retries | 0 | 0 | -| Budget-exhausted tasks | 4 | 5 | +| Budget-exhausted tasks | 5 | 5 | ### Subagent @@ -32,14 +33,15 @@ Per-tier figures for each arm, one column per arm in run order; the notes below | Task completion rate | 1.000000 (115/115) | 1.000000 (115/115) | | Task completion over repeats | 1.000000 (23/23, majority of 5) | 1.000000 (23/23, majority of 5) | | Task accuracy | 1.000000 | 1.000000 | -| Cost per completed task | 0.015107 | 0.007416 | -| P95 latency per completed task (ms) | 32948.108208 (n=54) | 12691.750541 (n=65) | -| P50 latency per completed task (ms) | 4681.970250 (n=54) | 2412.751916 (n=65) | -| Model invocations | 124 | 193 | -| Input tokens | 558126 | 1760204 | -| Output tokens | 52672 | 28510 | -| Cost per task, all tasks | 0.014478 | 0.007107 | -| Self-corrected retries | 3 | 0 | +| Cost per completed task | 0.013680 | 0.007523 | +| P95 latency per completed task (ms) | 22778.526041 (n=55) | 14190.514624 (n=68) | +| P50 latency per completed task (ms) | 5047.331292 (n=55) | 2678.756750 (n=68) | +| Model invocations | 128 | 195 | +| Long-context invocations | 0 | 0 | +| Input tokens | 532299 | 1873711 | +| Output tokens | 41494 | 29117 | +| Cost per task, all tasks | 0.013110 | 0.007210 | +| Self-corrected retries | 0 | 0 | | Delegation outcome: data | 95 | 95 | | Delegation outcome: no_match | 5 | 5 | | Delegation outcome: denied | 15 | 15 | @@ -55,19 +57,20 @@ Per-tier figures for each arm, one column per arm in run order; the notes below | Task completion rate | 1.000000 (95/95) | 1.000000 (95/95) | | Task completion over repeats | 1.000000 (19/19, majority of 5) | 1.000000 (19/19, majority of 5) | | Task accuracy | 1.000000 | 1.000000 | -| Cost per completed task | 0.000323 | 0.000176 | -| P95 latency per completed task (ms) | 46802.357543 (n=54) | 30112.864834 (n=60) | -| P50 latency per completed task (ms) | 1909.853791 (n=54) | 1337.665750 (n=60) | -| Model invocations | 235 | 336 | -| Input tokens | 154440 | 68109 | -| Output tokens | 30555 | 19848 | -| Cost per task, all tasks | 0.000256 | 0.000139 | +| Cost per completed task | 0.000331 | 0.000184 | +| P95 latency per completed task (ms) | 61764.406877 (n=55) | 32371.663457 (n=63) | +| P50 latency per completed task (ms) | 2117.928416 (n=55) | 1500.936292 (n=63) | +| Model invocations | 240 | 336 | +| Long-context invocations | 0 | 0 | +| Input tokens | 159249 | 72244 | +| Output tokens | 31113 | 20463 | +| Cost per task, all tasks | 0.000262 | 0.000145 | | Self-corrected retries | not applicable | not applicable | | Truncated tasks | 0 | 0 | Notes: -- Model lists the tier's configured model and every model that spent tokens at the tier; Effort lists the efforts the tier's turns ran at, with unsupported for a model that takes no effort setting. Task completion rate is rate (met/applicable) over the arm's tasks applicable at the tier, where n is the applicable task count: at Supervisor a task's correctness is 1.0, at Subagent the observed delegation outcome equals the expected one, and at ToolCall an expected tool was called with schema-valid arguments. Task accuracy is the arm's mean_correctness at Supervisor and the completion rate at Subagent and ToolCall. Task completion over repeats is rate (completed/applicable) over the arm's tasks applicable at the tier in any repeat, where a task counts when its tier expectation was met in a strict majority of its applicable repeats, and N is the per-task repeat count. Cost per completed task divides the tier's total cost by its completed task count, so spend on tasks that did not complete is included; Cost per task, all tasks divides the same total by every task the arm ran. Latency per completed task sums the tier's measured call latencies within each completed task and takes the nearest-rank percentile over those tasks; a completed task with no measured call at the tier contributes no sample, and n counts the contributing tasks. Model invocations sums the tier's included invocations. Self-corrected retries counts the tier's tool calls whose tool_failure result was followed, later in the same task, by a call to the same tool from the same role that returned data or no_match; such a call does not set the task's delegation outcome. ToolCall makes one call per invocation, so that row is not applicable there. +- Model lists the tier's configured model and every model that spent tokens at the tier; Effort lists the efforts the tier's turns ran at, with unsupported for a model that takes no effort setting. Task completion rate is rate (met/applicable) over the arm's tasks applicable at the tier, where n is the applicable task count: at Supervisor a task's correctness is 1.0, at Subagent the observed delegation outcome equals the expected one, and at ToolCall an expected tool was called with schema-valid arguments. Task accuracy is the arm's mean_correctness at Supervisor and the completion rate at Subagent and ToolCall. Task completion over repeats is rate (completed/applicable) over the arm's tasks applicable at the tier in any repeat, where a task counts when its tier expectation was met in a strict majority of its applicable repeats, and N is the per-task repeat count. Cost per completed task divides the tier's total cost by its completed task count, so spend on tasks that did not complete is included; Cost per task, all tasks divides the same total by every task the arm ran. Latency per completed task sums the tier's measured call latencies within each completed task and takes the nearest-rank percentile over those tasks; a completed task with no measured call at the tier contributes no sample, and n counts the contributing tasks. Model invocations sums the tier's included invocations; Long-context invocations counts those priced at the model's long-context rates, and reads — for a run recorded before that count existed. Self-corrected retries counts the tier's tool calls whose tool_failure result was followed, later in the same task, by a call to the same tool from the same role that returned data or no_match; such a call does not set the task's delegation outcome. ToolCall makes one call per invocation, so that row is not applicable there. - Columns are the sweep's arms in run order, not sorted by any result, and no difference or ratio between columns is computed. - Effort labels are not equivalent across families: each family declares its own effort scale (config/model_capabilities.yaml, effort_scale_comparability); see `effort` under "Reading these figures". - Input tokens are each family's own reported count and are not like-for-like across families (token_accounting_disclosure, src/eval/matrix.py); see `input_tokens` comparability under "Reading these figures". @@ -85,8 +88,10 @@ Each figure below names one narrow measured signal. Its boundary is stated here - `tool_call_correct_rate` — Reports that each scored turn issued a tool call which dispatched against an in-scope target. It is derived from exactly three conditions — a turn that issued no tool call at all, a proposal submitted on a task whose proposal limit is zero, and a proposal targeting a record other than the expected one — and from nothing else. It therefore does not report whether the turn chose the most apt tool for the task; where a task names the retrieval its answer requires, an `expected_reads` assertion checks that at least one call to a named tool reached the named record, and contributes to `mean_correctness` and deliberately not to this figure. An entry whose `target_id` is null checks only that one of the named tools was called at all, never that any particular record was reached, because the tools it names take no record id. Read to its exact boundary: it is satisfied incidentally by a turn that calls the named tool on the named record among many other calls, it says nothing about what the turn then reported, and it is therefore a necessary condition for an apt retrieval and never a sufficient one (src/eval/scoring.py, `_check_expected_reads`). - report quality — Not measured anywhere in this project, by any figure in this artifact. Two tasks score retrieval in place of report quality: grading a multi-part prose report needs a judge model, which this project forbids, so their assertions check that the turn read the records the answer requires and nothing checks what the turn then wrote. A turn that reads exactly the right records and then reports them incompletely, or wrongly, still satisfies those assertions and still counts toward `mean_correctness`. The `lookup` tasks' positive signal is retrieval on the same terms, and for two of the four it is weaker still: their `expected_reads` entry carries a null `target_id`, so the satisfied assertion evidences only that the named whole-book tool was called at all, not that any particular record was read. The narrowing was authorised deliberately and each narrowed task's own `notes` in golden_tasks.yaml restate the superseded reasoning at length. The tasks narrowed to retrieval are: `gt-04-investigation-turn-budget-partial`, `gt-05-investigation-large-tool-result`. - retrieval narrowing — A task narrowed to retrieval names the specific retrieval its assertions check, and names only the retrievals that expose an observable target id. A turn that answers such a task by an equally apt retrieval which exposes no target id therefore satisfies none of that task's positive assertions, for a reason unrelated to what the model did. The deduction that conservatism makes to `mean_correctness` is quantified per arm with that arm's aggregate figures, as an upper bound derived from the arm's own per-task records, so a reader of this artifact alone can size it against the figures it appears beside. -- Retrieval-narrowing magnitude: the largest deduction any one arm's retrieval-narrowed tasks account for is 0.000000 of that arm's `mean_correctness`, which is smaller than the largest absolute difference between any two arms' `mean_correctness` in this sweep (0.017870). Which arm sits on which side of that difference is deliberately not computed or stated anywhere in this artifact; the comparison is published because a disclosed measurement conservatism of this size bounds how much the difference between arms can be read as a property of the models (src/eval/golden_tasks.yaml). +- Retrieval-narrowing magnitude: the largest deduction any one arm's retrieval-narrowed tasks account for is 0.000000 of that arm's `mean_correctness`, which is smaller than the largest absolute difference between any two arms' `mean_correctness` in this sweep (0.013333). Which arm sits on which side of that difference is deliberately not computed or stated anywhere in this artifact; the comparison is published because a disclosed measurement conservatism of this size bounds how much the difference between arms can be read as a property of the models (src/eval/golden_tasks.yaml). - `effort` (provenance) — The reasoning-effort string sent to each arm's supervisor tier, pinned identically across arms so the sweep is reproducible; the subagent and tool tiers run their config/models.yaml tier effort. It is not a claim that the arms were given equivalent reasoning budgets: each family declares its own effort scale with its own rung count and its own default, so an identical label sits at a different position on each family's scale. The declared scales, and this non-equivalence, are recorded in config/model_capabilities.yaml under `effort_scale_comparability.compared_models`, per tier. Sweeps recorded before tier efforts were configured ran the subagent and tool tiers at each model's own default effort (the Anthropic subagent at `high`), so their subagent and tool tiers are not effort-comparable with this artifact's. +- `long_context_invocation_count` — counts the model invocations priced at the invoked model's long-context rates, i.e. a call whose prompt (input plus cache-read plus cache-write tokens) exceeded the model's long-context threshold in config/pricing.yaml; such a call's every token class is priced at those rates. It is a count of calls, not of tokens, and a model with no long-context tier never contributes. A run recorded before the count existed shows —, never 0. +- arm totals — an arm's Usage and Cost (model_metrics) covers only the supervisor-tier calls to the arm's model; the arm totals (the arm cost and arm_long_context_invocation_count) cover every call the arm made at every tier, so the arm cost equals the sum of the three tiers' costs. - `input_tokens` comparability — Each arm's `input_tokens` is the count its own provider family reported, and the families report cached prompt reads in differently-shaped usage objects: one as a sibling of `input_tokens`, the other nested inside its breakdown. Whether a family's `input_tokens` already contains the `cache_read_tokens` printed beside it is therefore a per-family question, and this project declares it per family with the evidence behind the declaration rather than assuming the two figures count the same thing. Two consequences, both exact: the arms' `input_tokens` are not established to be like-for-like totals and should not be differenced or ratioed as though they were; and while any family's accounting is declared unconfirmed, pricing refuses to produce a publishable currency figure for it at all, so no cost comparison can rest on the ambiguity. No token figure in this artifact is adjusted by this disclosure — each is the count that was reported. (src/crm/metering.py) - `anthropic`: cached reads read from `usage.cache_read_input_tokens (Messages API message_start)`; accounting priced under: `exclusive`; direction the evidence points: `exclusive`. Exclusive (disjoint): inputTokens excludes both cacheReadInputTokens and cacheWriteInputTokens; the total input is their sum. AWS's own Bedrock prompt-caching walkthrough shows a cache-read response as {"inputTokens": 28, "outputTokens": 294, "cacheWriteInputTokens": 0, "cacheReadInputTokens": 1898} and states "Only 28 tokens (the question itself) were processed as standard input" while the 1,898 reused tokens are reported separately in cacheReadInputTokens — so the cached count is not inside inputTokens. Source: https://aws.amazon.com/blogs/machine-learning/optimizing-cost-and-latency-with-amazon-bedrock-prompt-caching/ (verified 2026-09-28). The Messages API message_start path the adapter reads carries the same disjoint cache_read_input_tokens / cache_creation_input_tokens siblings. - `openai`: cached reads read from `usage.input_tokens_details.cached_tokens (Responses API response.completed)`; accounting priced under: `inclusive`; direction the evidence points: `inclusive`. Inclusive (subset): input_tokens is the full input count and both the cached reads and the cache writes are part of it, not added on top. AWS's Bedrock explicit-caching documentation for the OpenAI models states verbatim "input_tokens = cached_tokens + cache_write_tokens + non-cached remainder" and "Because the cached and written counts are part of input_tokens rather than added on top of it, a token served from the cache is counted once and billed once at the cache-read rate. You are never charged twice for the same token." Its worked example confirms the arithmetic (input_tokens 3682 = cached 0 + cache_write 3626 + new 56). Source: https://aws.amazon.com/blogs/machine-learning/introducing-explicit-prompt-caching-for-openai-gpt-5-6-models-on-amazon-bedrock/ (verified 2026-09-28). So both cached_tokens and cache_write_tokens must be subtracted from input_tokens to obtain the tokens billed at the standard input rate. CAVEAT: that AWS source describes the GPT-5.6 models; the OpenAI models in the matrix use the same Responses API usage shape (input_tokens with input_tokens_details.cached_tokens), and no vendor statement of this accounting specific to them was found, so the reading is carried forward by shape. The first --live sweep that records a non-zero cached read should be reconciled against the actual Bedrock bill to confirm it holds for them. @@ -98,90 +103,98 @@ Each figure below names one narrow measured signal. Its boundary is stated here | field | value | | --- | --- | | mode | live | -| timestamp | 2026-10-29T03:39:53.444033+00:00 | +| timestamp | 2026-11-10T20:18:32.528771+00:00 | | model_id | anthropic.claude-sonnet-5-5 | | effort | medium | | dataset_seed | 20260101 | | runtime_identifier | live-aws-lambda-dispatch | -| price_table_version | 1.2.0 | +| price_table_version | 1.3.0 | | answer_style_sha256 | 953efa6d0d26da1fb09cf2cdf625c78f0bd8f7d04da7652e581e623edea67f47 | -| prompt_bundle_sha256 | 77723af1ff2c9b216e100c7b352e9729dc4063fe2edd0bd7f3c12754fdb1ce1d | +| prompt_bundle_sha256 | 2959cde8d2bde46cfdf198614a890110d40d4753501f6151fb49f59285e3cec1 | ### Usage and Cost +Supervisor tier only: these figures cover the arm's supervisor-tier calls to anthropic.claude-sonnet-5-5. + | field | value | | --- | --- | | model_id | anthropic.claude-sonnet-5-5 | -| included_invocation_count | 119 | -| input_tokens | 437073 | -| output_tokens | 49150 | -| cache_write_tokens | 4199 | -| cache_read_tokens | 1175720 | +| included_invocation_count | 273 | +| long_context_invocation_count | 0 | +| input_tokens | 350986 | +| output_tokens | 47818 | +| cache_write_tokens | 4375 | +| cache_read_tokens | 1190000 | | cache_min_checkpoint_tokens | 512 | | latency_measured | yes | | latency_percentile_method | nearest-rank | -| task_latency_ms_p50 | 6724.061625 | -| task_latency_ms_p95 | 26708.118958 | +| task_latency_ms_p50 | 7596.169667 | +| task_latency_ms_p95 | 24924.877333 | | cost_withheld_provisional | no | -| cost_input | 0.874146 | -| cost_output | 0.491500 | -| cost_cache_write | 0.010497 | -| cost_cache_read | 0.117572 | -| cost_excluding_cache | 1.365646 | -| cost_cache_total | 0.128070 | -| cost_total | 1.493715 | -| price_table_version | 1.2.0 | +| cost_input | 0.701972 | +| cost_output | 0.478180 | +| cost_cache_write | 0.010938 | +| cost_cache_read | 0.119000 | +| cost_excluding_cache | 1.180152 | +| cost_cache_total | 0.129938 | +| cost_total | 1.310090 | +| price_table_version | 1.3.0 | + +All tiers (every call the arm made): + +| field | value | +| --- | --- | +| arm_cost_total | 2.914799 | +| arm_long_context_invocation_count | 0 | -Usage and cost for this arm are a lower bound: in 4 task run(s) (gt-04-investigation-turn-budget-partial, gt-04-investigation-turn-budget-partial, gt-04-investigation-turn-budget-partial, gt-04-investigation-turn-budget-partial) a model call ended before reporting token usage, so its tokens and cost are not counted in the figures above. +Usage and cost for this arm are a lower bound: in 5 task run(s) (gt-04-investigation-turn-budget-partial, gt-04-investigation-turn-budget-partial, gt-04-investigation-turn-budget-partial, gt-04-investigation-turn-budget-partial, gt-04-investigation-turn-budget-partial) a model call ended before reporting token usage, so its tokens and cost are not counted in the figures above. Call latency by tier (ms): | tier | p50 | p95 | count | | --- | --- | --- | --- | -| supervisor | 1679.674291 | 3767.128250 | 281 | -| subagent | 2124.368833 | 15145.678916 | 124 | -| tool | 1160.205458 | 2030.019042 | 235 | +| supervisor | 1803.318791 | 4399.364166 | 273 | +| subagent | 2267.182459 | 7086.334042 | 128 | +| tool | 1576.261125 | 2854.407583 | 240 | Per tier / role / model: -| tier | role | model_id | included_invocation_count | input_tokens | output_tokens | cache_read_tokens | cache_write_tokens | call_latency_measured | call_latency_ms_p50 | call_latency_ms_p95 | cost_withheld_provisional | cost_input | cost_output | cost_cache_read | cost_cache_write | cost_excluding_cache | cost_cache_total | cost_total | price_table_version | -| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | -| subagent | mutation | anthropic.claude-sonnet-5-5 | 111 | 117986 | 25524 | 669900 | 6090 | yes | 2093.161167 | 3460.283042 | no | 0.235972 | 0.255240 | 0.066990 | 0.015225 | 0.491212 | 0.082215 | 0.573427 | 1.2.0 | -| subagent | research | anthropic.claude-sonnet-5-5 | 13 | 440140 | 27148 | 21923 | 3986 | yes | 15145.678916 | 21632.722500 | no | 0.880280 | 0.271480 | 0.002192 | 0.009965 | 1.151760 | 0.012157 | 1.163917 | 1.2.0 | -| supervisor | supervisor | anthropic.claude-sonnet-5-5 | 281 | 437073 | 49150 | 1175720 | 4199 | yes | 1679.674291 | 3767.128250 | no | 0.874146 | 0.491500 | 0.117572 | 0.010497 | 1.365646 | 0.128070 | 1.493716 | 1.2.0 | -| tool | mutation | anthropic.claude-haiku-5-5 | 69 | 62368 | 16048 | 0 | 0 | yes | 1644.373833 | 3394.509750 | no | 0.006237 | 0.008024 | 0.000000 | 0.000000 | 0.014261 | 0.000000 | 0.014261 | 1.2.0 | -| tool | research | anthropic.claude-haiku-5-5 | 166 | 92072 | 14507 | 0 | 0 | yes | 1122.135500 | 1804.872000 | no | 0.009207 | 0.007253 | 0.000000 | 0.000000 | 0.016461 | 0.000000 | 0.016461 | 1.2.0 | +| tier | role | model_id | included_invocation_count | long_context_invocation_count | input_tokens | output_tokens | cache_read_tokens | cache_write_tokens | call_latency_measured | call_latency_ms_p50 | call_latency_ms_p95 | cost_withheld_provisional | cost_input | cost_output | cost_cache_read | cost_cache_write | cost_excluding_cache | cost_cache_total | cost_total | price_table_version | +| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | +| subagent | mutation | anthropic.claude-sonnet-5-5 | 105 | 0 | 94965 | 23596 | 674544 | 6486 | yes | 2150.364500 | 4162.445542 | no | 0.189930 | 0.235960 | 0.067454 | 0.016215 | 0.425890 | 0.083669 | 0.509559 | 1.3.0 | +| subagent | research | anthropic.claude-sonnet-5-5 | 23 | 0 | 437334 | 17898 | 46904 | 2132 | yes | 5047.587833 | 11582.232958 | no | 0.874668 | 0.178980 | 0.004690 | 0.005330 | 1.053648 | 0.010020 | 1.063668 | 1.3.0 | +| supervisor | supervisor | anthropic.claude-sonnet-5-5 | 273 | 0 | 350986 | 47818 | 1190000 | 4375 | yes | 1803.318791 | 4399.364166 | no | 0.701972 | 0.478180 | 0.119000 | 0.010938 | 1.180152 | 0.129937 | 1.310090 | 1.3.0 | +| tool | mutation | anthropic.claude-haiku-5-5 | 62 | 0 | 58185 | 15131 | 0 | 0 | yes | 1906.717834 | 2801.624333 | no | 0.005818 | 0.007565 | 0.000000 | 0.000000 | 0.013384 | 0.000000 | 0.013384 | 1.3.0 | +| tool | research | anthropic.claude-haiku-5-5 | 178 | 0 | 101064 | 15982 | 0 | 0 | yes | 1518.533375 | 2929.801709 | no | 0.010106 | 0.007991 | 0.000000 | 0.000000 | 0.018097 | 0.000000 | 0.018097 | 1.3.0 | Per-tier summary: | tier | input_tokens | output_tokens | cost | latency_p50_ms | latency_p95_ms | success (met/applicable) | success_rate | | --- | --- | --- | --- | --- | --- | --- | --- | -| supervisor | 437073 | 49150 | 1.493716 | 1679.674291 | 3767.128250 | 112/120 | 0.933333 | -| subagent | 558126 | 52672 | 1.737344 | 2124.368833 | 15145.678916 | 115/115 | 1.000000 | -| tool | 154440 | 30555 | 0.030722 | 1160.205458 | 2030.019042 | 95/95 | 1.000000 | +| supervisor | 350986 | 47818 | 1.310090 | 1803.318791 | 4399.364166 | 113/120 | 0.941667 | +| subagent | 532299 | 41494 | 1.573228 | 2267.182459 | 7086.334042 | 115/115 | 1.000000 | +| tool | 159249 | 31113 | 0.031481 | 1576.261125 | 2854.407583 | 95/95 | 1.000000 | Per tool: | tool_name | role | count | call_latency_ms_p50 | call_latency_ms_p95 | truncated_count | outcome_data | outcome_no_match | outcome_denied | outcome_tool_failure | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | -| create_note | mutation | 10 | 395.405625 | 1955.169209 | 0 | 10 | 0 | 0 | 0 | -| get_account | research | 10 | 252.081541 | 330.135792 | 0 | 10 | 0 | 0 | 0 | -| get_account | supervisor | 36 | 267.047958 | 395.243083 | 0 | 36 | 0 | 0 | 0 | -| get_contact | research | 1 | 249.045417 | 249.045417 | 0 | 1 | 0 | 0 | 0 | -| get_contact | supervisor | 23 | 249.728875 | 351.584416 | 0 | 23 | 0 | 0 | 0 | -| get_opportunity | research | 152 | 301.124041 | 402.877000 | 0 | 152 | 0 | 0 | 0 | -| get_opportunity | supervisor | 47 | 313.865375 | 435.351666 | 0 | 37 | 5 | 5 | 0 | -| list_activities | supervisor | 10 | 0.647250 | 2009.227291 | 0 | 5 | 0 | 0 | 5 | -| list_proposals | mutation | 3 | 267.774708 | 268.921833 | 0 | 3 | 0 | 0 | 0 | -| list_proposals | supervisor | 5 | 311.241542 | 1960.959333 | 0 | 5 | 0 | 0 | 0 | -| log_activity | mutation | 10 | 391.168458 | 1942.776875 | 0 | 10 | 0 | 0 | 0 | -| pipeline_aggregate | supervisor | 5 | 1090.526916 | 2696.057917 | 0 | 5 | 0 | 0 | 0 | -| search_accounts | supervisor | 5 | 435.152875 | 1877.212750 | 0 | 5 | 0 | 0 | 0 | -| search_opportunities | research | 3 | 268.024333 | 292.754000 | 0 | 3 | 0 | 0 | 0 | -| search_opportunities | supervisor | 19 | 126.470875 | 413.476084 | 0 | 9 | 0 | 10 | 0 | -| submit_proposal | mutation | 36 | 308.012167 | 403.327958 | 0 | 33 | 0 | 0 | 3 | -| update_account | mutation | 5 | 1921.012375 | 2059.647458 | 0 | 5 | 0 | 0 | 0 | -| update_contact | mutation | 5 | 394.912625 | 2011.498792 | 0 | 5 | 0 | 0 | 0 | +| create_note | mutation | 10 | 350.102125 | 1911.414917 | 0 | 10 | 0 | 0 | 0 | +| get_account | research | 16 | 268.130375 | 295.391167 | 0 | 16 | 0 | 0 | 0 | +| get_account | supervisor | 20 | 287.268959 | 388.594375 | 0 | 20 | 0 | 0 | 0 | +| get_contact | supervisor | 23 | 258.644000 | 368.531209 | 0 | 23 | 0 | 0 | 0 | +| get_opportunity | research | 158 | 309.076709 | 369.213583 | 0 | 158 | 0 | 0 | 0 | +| get_opportunity | supervisor | 46 | 305.912291 | 359.309000 | 0 | 36 | 5 | 5 | 0 | +| list_activities | supervisor | 10 | 2.323250 | 2217.895625 | 0 | 5 | 0 | 0 | 5 | +| list_proposals | supervisor | 5 | 1999.800167 | 2408.586416 | 0 | 5 | 0 | 0 | 0 | +| log_activity | mutation | 10 | 339.626333 | 2101.602416 | 0 | 10 | 0 | 0 | 0 | +| pipeline_aggregate | supervisor | 5 | 2268.992584 | 2705.675875 | 0 | 5 | 0 | 0 | 0 | +| search_accounts | supervisor | 5 | 2101.000834 | 2280.018000 | 0 | 5 | 0 | 0 | 0 | +| search_opportunities | research | 4 | 277.645875 | 2031.635792 | 0 | 4 | 0 | 0 | 0 | +| search_opportunities | supervisor | 16 | 136.441084 | 299.789042 | 0 | 6 | 0 | 10 | 0 | +| submit_proposal | mutation | 32 | 308.653708 | 436.228375 | 0 | 32 | 0 | 0 | 0 | +| update_account | mutation | 5 | 2057.290125 | 2071.808959 | 0 | 5 | 0 | 0 | 0 | +| update_contact | mutation | 5 | 1987.479208 | 2078.091875 | 0 | 5 | 0 | 0 | 0 | Excluded invocations: 0 total (cold_start: 0, retry: 0, throttle: 0) @@ -191,13 +204,13 @@ Excluded invocations: 0 total (cold_start: 0, retry: 0, throttle: 0) | --- | --- | | scored_task_count | 120 | | unscored_task_count | 0 | -| mean_correctness | 0.982130 | +| mean_correctness | 0.984167 | | bad_write_rate | 0.000000 | | tool_call_correct_rate | 1.000000 | `bad_write_rate` basis: all 250 declared `must_not_change` assertion(s) in this arm had a captured pre-turn baseline, so the figure is a measurement over them (src/eval/matrix.py). -`mean_correctness` composition: negative guards a do-nothing turn satisfies are 0.8162 of the assertion mass this figure averages — the mean over the 120 scored tasks of each task's own guard share, weighted exactly as `mean_correctness` itself is, rather than the ratio pooled over assertions (pooled totals, for context and not the basis of the figure just quoted: 683 guard assertions and 145 positive ones across those tasks). So the figure is not a task-achievement rate. Taken alone, 138 of 145 positive assertions were satisfied (0.9517); the guards alone were satisfied at 0.9795 (src/eval/matrix.py). +`mean_correctness` composition: negative guards a do-nothing turn satisfies are 0.8161 of the assertion mass this figure averages — the mean over the 120 scored tasks of each task's own guard share, weighted exactly as `mean_correctness` itself is, rather than the ratio pooled over assertions (pooled totals, for context and not the basis of the figure just quoted: 682 guard assertions and 145 positive ones across those tasks). So the figure is not a task-achievement rate. Taken alone, 140 of 145 positive assertions were satisfied (0.9655); the guards alone were satisfied at 0.9795 (src/eval/matrix.py). Retrieval-narrowed task deduction: the assertions this arm did not satisfy on the tasks narrowed to retrieval account for at most 0.000000 of its `mean_correctness` (`gt-04-investigation-turn-budget-partial` scored 1.000000, at most 0.000000; `gt-05-investigation-large-tool-result` scored 1.000000, at most 0.000000; `gt-04-investigation-turn-budget-partial` scored 1.000000, at most 0.000000; `gt-05-investigation-large-tool-result` scored 1.000000, at most 0.000000; `gt-04-investigation-turn-budget-partial` scored 1.000000, at most 0.000000; `gt-05-investigation-large-tool-result` scored 1.000000, at most 0.000000; `gt-04-investigation-turn-budget-partial` scored 1.000000, at most 0.000000; `gt-05-investigation-large-tool-result` scored 1.000000, at most 0.000000; `gt-04-investigation-turn-budget-partial` scored 1.000000, at most 0.000000; `gt-05-investigation-large-tool-result` scored 1.000000, at most 0.000000), an upper bound over the 120 scored tasks derived from this arm's own per-task records. The figure itself is unchanged by this disclosure (src/eval/golden_tasks.yaml). @@ -230,14 +243,14 @@ Retrieval-narrowed task deduction: the assertions this arm did not satisfy on th | gt-23-investigation-pd06-ambiguous-contact | machine | 1.000000 | — | no | yes | no | no | | gt-24-enrichment-pd07-multi-step-write | machine | 1.000000 | — | no | yes | no | no | | gt-01-investigation-pd01-successor | machine | 1.000000 | — | no | yes | no | no | -| gt-02-investigation-pd02-stale-forecast | machine | 0.777778 | — | no | yes | no | no | +| gt-02-investigation-pd02-stale-forecast | machine | 1.000000 | — | no | yes | no | no | | gt-03-investigation-pd03-scoped-lookup | machine | 1.000000 | — | no | yes | no | no | | gt-04-investigation-turn-budget-partial | machine | 1.000000 | — | no | yes | no | yes | | gt-05-investigation-large-tool-result | machine | 1.000000 | — | no | yes | no | no | | gt-06-enrichment-create-note | machine | 1.000000 | — | no | yes | no | no | | gt-07-enrichment-log-activity-last-touch | machine | 1.000000 | — | no | yes | no | no | | gt-08-enrichment-stage-advance-proposal | machine | 1.000000 | — | no | yes | no | no | -| gt-09-enrichment-value-and-close-date-proposal | machine | 1.000000 | — | no | yes | no | no | +| gt-09-enrichment-value-and-close-date-proposal | machine | 0.800000 | — | no | yes | no | no | | gt-10-enrichment-update-account-fields | machine | 1.000000 | — | no | yes | no | no | | gt-11-denial-out-of-book-target | machine | 1.000000 | — | no | yes | no | no | | gt-12-denial-nonexistent-target | machine | 1.000000 | — | no | yes | no | no | @@ -256,12 +269,12 @@ Retrieval-narrowed task deduction: the assertions this arm did not satisfy on th | gt-01-investigation-pd01-successor | machine | 1.000000 | — | no | yes | no | no | | gt-02-investigation-pd02-stale-forecast | machine | 1.000000 | — | no | yes | no | no | | gt-03-investigation-pd03-scoped-lookup | machine | 1.000000 | — | no | yes | no | no | -| gt-04-investigation-turn-budget-partial | machine | 1.000000 | — | no | yes | no | no | +| gt-04-investigation-turn-budget-partial | machine | 1.000000 | — | no | yes | no | yes | | gt-05-investigation-large-tool-result | machine | 1.000000 | — | no | yes | no | no | | gt-06-enrichment-create-note | machine | 1.000000 | — | no | yes | no | no | | gt-07-enrichment-log-activity-last-touch | machine | 1.000000 | — | no | yes | no | no | | gt-08-enrichment-stage-advance-proposal | machine | 1.000000 | — | no | yes | no | no | -| gt-09-enrichment-value-and-close-date-proposal | machine | 1.000000 | — | no | yes | no | no | +| gt-09-enrichment-value-and-close-date-proposal | machine | 0.800000 | — | no | yes | no | no | | gt-10-enrichment-update-account-fields | machine | 1.000000 | — | no | yes | no | no | | gt-11-denial-out-of-book-target | machine | 1.000000 | — | no | yes | no | no | | gt-12-denial-nonexistent-target | machine | 1.000000 | — | no | yes | no | no | @@ -278,14 +291,14 @@ Retrieval-narrowed task deduction: the assertions this arm did not satisfy on th | gt-23-investigation-pd06-ambiguous-contact | machine | 1.000000 | — | no | yes | no | no | | gt-24-enrichment-pd07-multi-step-write | machine | 1.000000 | — | no | yes | no | no | | gt-01-investigation-pd01-successor | machine | 1.000000 | — | no | yes | no | no | -| gt-02-investigation-pd02-stale-forecast | machine | 0.777778 | — | no | yes | no | no | +| gt-02-investigation-pd02-stale-forecast | machine | 1.000000 | — | no | yes | no | no | | gt-03-investigation-pd03-scoped-lookup | machine | 1.000000 | — | no | yes | no | no | | gt-04-investigation-turn-budget-partial | machine | 1.000000 | — | no | yes | no | yes | | gt-05-investigation-large-tool-result | machine | 1.000000 | — | no | yes | no | no | | gt-06-enrichment-create-note | machine | 1.000000 | — | no | yes | no | no | | gt-07-enrichment-log-activity-last-touch | machine | 1.000000 | — | no | yes | no | no | | gt-08-enrichment-stage-advance-proposal | machine | 1.000000 | — | no | yes | no | no | -| gt-09-enrichment-value-and-close-date-proposal | machine | 0.800000 | — | no | yes | no | no | +| gt-09-enrichment-value-and-close-date-proposal | machine | 1.000000 | — | no | yes | no | no | | gt-10-enrichment-update-account-fields | machine | 1.000000 | — | no | yes | no | no | | gt-11-denial-out-of-book-target | machine | 1.000000 | — | no | yes | no | no | | gt-12-denial-nonexistent-target | machine | 1.000000 | — | no | yes | no | no | @@ -331,14 +344,14 @@ Per-task correctness spread across repeats: | task_id | repeats | correctness_min | correctness_mean | correctness_max | success_fraction | | --- | --- | --- | --- | --- | --- | | gt-01-investigation-pd01-successor | 5 | 1.000000 | 1.000000 | 1.000000 | 1.000000 | -| gt-02-investigation-pd02-stale-forecast | 5 | 0.777778 | 0.911111 | 1.000000 | 0.600000 | +| gt-02-investigation-pd02-stale-forecast | 5 | 1.000000 | 1.000000 | 1.000000 | 1.000000 | | gt-03-investigation-pd03-scoped-lookup | 5 | 1.000000 | 1.000000 | 1.000000 | 1.000000 | | gt-04-investigation-turn-budget-partial | 5 | 1.000000 | 1.000000 | 1.000000 | 1.000000 | | gt-05-investigation-large-tool-result | 5 | 1.000000 | 1.000000 | 1.000000 | 1.000000 | | gt-06-enrichment-create-note | 5 | 1.000000 | 1.000000 | 1.000000 | 1.000000 | | gt-07-enrichment-log-activity-last-touch | 5 | 1.000000 | 1.000000 | 1.000000 | 1.000000 | | gt-08-enrichment-stage-advance-proposal | 5 | 1.000000 | 1.000000 | 1.000000 | 1.000000 | -| gt-09-enrichment-value-and-close-date-proposal | 5 | 0.800000 | 0.960000 | 1.000000 | 0.800000 | +| gt-09-enrichment-value-and-close-date-proposal | 5 | 0.800000 | 0.920000 | 1.000000 | 0.600000 | | gt-10-enrichment-update-account-fields | 5 | 1.000000 | 1.000000 | 1.000000 | 1.000000 | | gt-11-denial-out-of-book-target | 5 | 1.000000 | 1.000000 | 1.000000 | 1.000000 | | gt-12-denial-nonexistent-target | 5 | 1.000000 | 1.000000 | 1.000000 | 1.000000 | @@ -362,39 +375,49 @@ Per-task correctness spread across repeats: | field | value | | --- | --- | | mode | live | -| timestamp | 2026-10-29T04:02:16.854726+00:00 | +| timestamp | 2026-11-10T20:40:59.870904+00:00 | | model_id | openai.gpt-6.1-sol | | effort | medium | | dataset_seed | 20260101 | | runtime_identifier | live-aws-lambda-dispatch | -| price_table_version | 1.2.0 | +| price_table_version | 1.3.0 | | answer_style_sha256 | 953efa6d0d26da1fb09cf2cdf625c78f0bd8f7d04da7652e581e623edea67f47 | -| prompt_bundle_sha256 | 77723af1ff2c9b216e100c7b352e9729dc4063fe2edd0bd7f3c12754fdb1ce1d | +| prompt_bundle_sha256 | 2959cde8d2bde46cfdf198614a890110d40d4753501f6151fb49f59285e3cec1 | ### Usage and Cost +Supervisor tier only: these figures cover the arm's supervisor-tier calls to openai.gpt-6.1-sol. + | field | value | | --- | --- | | model_id | openai.gpt-6.1-sol | -| included_invocation_count | 115 | -| input_tokens | 692274 | -| output_tokens | 23443 | -| cache_write_tokens | 140222 | -| cache_read_tokens | 551568 | +| included_invocation_count | 242 | +| long_context_invocation_count | 0 | +| input_tokens | 727458 | +| output_tokens | 23719 | +| cache_write_tokens | 143097 | +| cache_read_tokens | 583877 | | cache_min_checkpoint_tokens | 1024 | | latency_measured | yes | | latency_percentile_method | nearest-rank | -| task_latency_ms_p50 | 7237.658750 | -| task_latency_ms_p95 | 35419.177416 | +| task_latency_ms_p50 | 7513.681541 | +| task_latency_ms_p95 | 29048.906041 | | cost_withheld_provisional | no | | cost_input | 0.000968 | -| cost_output | 0.234430 | -| cost_cache_write | 0.350555 | -| cost_cache_read | 0.055157 | -| cost_excluding_cache | 0.235398 | -| cost_cache_total | 0.405712 | -| cost_total | 0.641110 | -| price_table_version | 1.2.0 | +| cost_output | 0.237190 | +| cost_cache_write | 0.357743 | +| cost_cache_read | 0.058388 | +| cost_excluding_cache | 0.238158 | +| cost_cache_total | 0.416130 | +| cost_total | 0.654288 | +| price_table_version | 1.3.0 | + +All tiers (every call the arm made): + +| field | value | +| --- | --- | +| arm_cost_total | 1.536942 | +| arm_long_context_invocation_count | 0 | Usage and cost for this arm are a lower bound: in 5 task run(s) (gt-04-investigation-turn-budget-partial, gt-04-investigation-turn-budget-partial, gt-04-investigation-turn-budget-partial, gt-04-investigation-turn-budget-partial, gt-04-investigation-turn-budget-partial) a model call ended before reporting token usage, so its tokens and cost are not counted in the figures above. @@ -402,58 +425,58 @@ Call latency by tier (ms): | tier | p50 | p95 | count | | --- | --- | --- | --- | -| supervisor | 2108.646875 | 5129.901250 | 242 | -| subagent | 1022.857542 | 2781.642875 | 193 | -| tool | 696.077333 | 1163.845291 | 336 | +| supervisor | 2064.151709 | 3983.399916 | 242 | +| subagent | 1081.820291 | 3070.852542 | 195 | +| tool | 724.084167 | 1479.331417 | 336 | Per tier / role / model: -| tier | role | model_id | included_invocation_count | input_tokens | output_tokens | cache_read_tokens | cache_write_tokens | call_latency_measured | call_latency_ms_p50 | call_latency_ms_p95 | cost_withheld_provisional | cost_input | cost_output | cost_cache_read | cost_cache_write | cost_excluding_cache | cost_cache_total | cost_total | price_table_version | -| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | -| subagent | analytics | openai.gpt-6.1-terra | 10 | 20544 | 1225 | 0 | 15656 | yes | 1022.857542 | 1797.096875 | no | 0.004888 | 0.006125 | 0.000000 | 0.019570 | 0.011013 | 0.019570 | 0.030583 | 1.2.0 | -| subagent | mutation | openai.gpt-6.1-terra | 132 | 489031 | 10935 | 417522 | 71245 | yes | 984.310083 | 1652.462500 | no | 0.000264 | 0.054675 | 0.020876 | 0.089056 | 0.054939 | 0.109932 | 0.164871 | 1.2.0 | -| subagent | research | openai.gpt-6.1-terra | 51 | 1250629 | 16350 | 821589 | 421947 | yes | 1263.867083 | 6111.557083 | no | 0.007093 | 0.081750 | 0.041079 | 0.527434 | 0.088843 | 0.568513 | 0.657356 | 1.2.0 | -| supervisor | supervisor | openai.gpt-6.1-sol | 242 | 692274 | 23443 | 551568 | 140222 | yes | 2108.646875 | 5129.901250 | no | 0.000968 | 0.234430 | 0.055157 | 0.350555 | 0.235398 | 0.405712 | 0.641110 | 1.2.0 | -| tool | analytics | openai.gpt-6-luna | 10 | 2980 | 480 | 0 | 0 | yes | 638.964667 | 824.001792 | no | 0.000298 | 0.000240 | 0.000000 | 0.000000 | 0.000538 | 0.000000 | 0.000538 | 1.2.0 | -| tool | mutation | openai.gpt-6-luna | 79 | 25004 | 7468 | 0 | 0 | yes | 913.225541 | 1470.757250 | no | 0.002500 | 0.003734 | 0.000000 | 0.000000 | 0.006234 | 0.000000 | 0.006234 | 1.2.0 | -| tool | research | openai.gpt-6-luna | 247 | 40125 | 11900 | 0 | 0 | yes | 670.447292 | 941.605208 | no | 0.004012 | 0.005950 | 0.000000 | 0.000000 | 0.009963 | 0.000000 | 0.009963 | 1.2.0 | +| tier | role | model_id | included_invocation_count | long_context_invocation_count | input_tokens | output_tokens | cache_read_tokens | cache_write_tokens | call_latency_measured | call_latency_ms_p50 | call_latency_ms_p95 | cost_withheld_provisional | cost_input | cost_output | cost_cache_read | cost_cache_write | cost_excluding_cache | cost_cache_total | cost_total | price_table_version | +| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | +| subagent | analytics | openai.gpt-6.1-terra | 10 | 0 | 21757 | 1174 | 5366 | 16371 | yes | 1081.820291 | 1867.854583 | no | 0.000020 | 0.005870 | 0.000268 | 0.020464 | 0.005890 | 0.020732 | 0.026622 | 1.3.0 | +| subagent | mutation | openai.gpt-6.1-terra | 135 | 0 | 543002 | 11611 | 468729 | 74003 | yes | 964.570000 | 1789.223250 | no | 0.000270 | 0.058055 | 0.023436 | 0.092504 | 0.058325 | 0.115940 | 0.174265 | 1.3.0 | +| subagent | research | openai.gpt-6.1-terra | 50 | 0 | 1308952 | 16332 | 877929 | 430923 | yes | 1371.243291 | 7899.265125 | no | 0.000100 | 0.081660 | 0.043896 | 0.538654 | 0.081760 | 0.582550 | 0.664310 | 1.3.0 | +| supervisor | supervisor | openai.gpt-6.1-sol | 242 | 0 | 727458 | 23719 | 583877 | 143097 | yes | 2064.151709 | 3983.399916 | no | 0.000968 | 0.237190 | 0.058388 | 0.357743 | 0.238158 | 0.416130 | 0.654288 | 1.3.0 | +| tool | analytics | openai.gpt-6-luna | 10 | 0 | 3270 | 490 | 0 | 0 | yes | 694.980750 | 1042.591792 | no | 0.000327 | 0.000245 | 0.000000 | 0.000000 | 0.000572 | 0.000000 | 0.000572 | 1.3.0 | +| tool | mutation | openai.gpt-6-luna | 80 | 0 | 26761 | 7867 | 0 | 0 | yes | 969.241042 | 1678.421334 | no | 0.002676 | 0.003934 | 0.000000 | 0.000000 | 0.006610 | 0.000000 | 0.006610 | 1.3.0 | +| tool | research | openai.gpt-6-luna | 246 | 0 | 42213 | 12106 | 0 | 0 | yes | 699.093875 | 1096.752833 | no | 0.004221 | 0.006053 | 0.000000 | 0.000000 | 0.010274 | 0.000000 | 0.010274 | 1.3.0 | Per-tier summary: | tier | input_tokens | output_tokens | cost | latency_p50_ms | latency_p95_ms | success (met/applicable) | success_rate | | --- | --- | --- | --- | --- | --- | --- | --- | -| supervisor | 692274 | 23443 | 0.641110 | 2108.646875 | 5129.901250 | 120/120 | 1.000000 | -| subagent | 1760204 | 28510 | 0.852811 | 1022.857542 | 2781.642875 | 115/115 | 1.000000 | -| tool | 68109 | 19848 | 0.016735 | 696.077333 | 1163.845291 | 95/95 | 1.000000 | +| supervisor | 727458 | 23719 | 0.654288 | 2064.151709 | 3983.399916 | 119/120 | 0.991667 | +| subagent | 1873711 | 29117 | 0.865197 | 1081.820291 | 3070.852542 | 115/115 | 1.000000 | +| tool | 72244 | 20463 | 0.017456 | 724.084167 | 1479.331417 | 95/95 | 1.000000 | Per tool: | tool_name | role | count | call_latency_ms_p50 | call_latency_ms_p95 | truncated_count | outcome_data | outcome_no_match | outcome_denied | outcome_tool_failure | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | -| create_note | mutation | 10 | 369.718916 | 1233.895667 | 0 | 10 | 0 | 0 | 0 | -| get_account | mutation | 11 | 269.546416 | 339.247000 | 0 | 11 | 0 | 0 | 0 | -| get_account | research | 10 | 268.442625 | 396.207792 | 0 | 10 | 0 | 0 | 0 | -| get_account | supervisor | 5 | 282.450916 | 322.254833 | 0 | 5 | 0 | 0 | 0 | -| get_contact | mutation | 1 | 223.148959 | 223.148959 | 0 | 1 | 0 | 0 | 0 | -| get_contact | research | 29 | 242.467583 | 296.410042 | 0 | 29 | 0 | 0 | 0 | -| get_contact | supervisor | 5 | 245.258292 | 273.146083 | 0 | 5 | 0 | 0 | 0 | -| get_opportunity | mutation | 6 | 312.354916 | 341.645416 | 0 | 6 | 0 | 0 | 0 | -| get_opportunity | research | 197 | 296.631708 | 403.424458 | 0 | 197 | 0 | 0 | 0 | -| get_opportunity | supervisor | 25 | 289.559583 | 354.792917 | 0 | 15 | 5 | 5 | 0 | -| list_activities | mutation | 1 | 290.664542 | 290.664542 | 0 | 1 | 0 | 0 | 0 | -| list_activities | research | 5 | 313.815458 | 400.238208 | 0 | 5 | 0 | 0 | 0 | -| list_activities | supervisor | 10 | 0.467334 | 343.737958 | 0 | 5 | 0 | 0 | 5 | -| list_audit_records | research | 1 | 1839.384667 | 1839.384667 | 0 | 1 | 0 | 0 | 0 | -| list_proposals | supervisor | 5 | 304.312333 | 415.105875 | 0 | 5 | 0 | 0 | 0 | -| log_activity | mutation | 10 | 380.720084 | 2017.808000 | 0 | 10 | 0 | 0 | 0 | -| pipeline_aggregate | supervisor | 5 | 1055.497542 | 1315.181125 | 0 | 5 | 0 | 0 | 0 | -| search_accounts | mutation | 5 | 310.584750 | 1262.437083 | 0 | 5 | 0 | 0 | 0 | -| search_opportunities | analytics | 10 | 127.048375 | 301.478584 | 0 | 5 | 0 | 5 | 0 | -| search_opportunities | research | 5 | 278.759959 | 291.046292 | 0 | 5 | 0 | 0 | 0 | -| search_opportunities | supervisor | 5 | 112.732250 | 114.818459 | 0 | 0 | 0 | 5 | 0 | -| submit_proposal | mutation | 25 | 317.173042 | 387.050833 | 0 | 25 | 0 | 0 | 0 | -| update_account | mutation | 5 | 442.582583 | 594.826500 | 0 | 5 | 0 | 0 | 0 | -| update_contact | mutation | 5 | 367.301833 | 2018.741333 | 0 | 5 | 0 | 0 | 0 | +| create_note | mutation | 10 | 323.062041 | 394.156750 | 0 | 10 | 0 | 0 | 0 | +| get_account | mutation | 10 | 274.082500 | 389.635750 | 0 | 10 | 0 | 0 | 0 | +| get_account | research | 10 | 267.483959 | 292.097791 | 0 | 10 | 0 | 0 | 0 | +| get_account | supervisor | 5 | 296.824750 | 320.795250 | 0 | 5 | 0 | 0 | 0 | +| get_contact | research | 26 | 247.368958 | 294.216417 | 0 | 26 | 0 | 0 | 0 | +| get_contact | supervisor | 5 | 250.503333 | 273.101208 | 0 | 5 | 0 | 0 | 0 | +| get_opportunity | mutation | 6 | 303.956875 | 395.329333 | 0 | 6 | 0 | 0 | 0 | +| get_opportunity | research | 196 | 312.354917 | 414.298959 | 0 | 196 | 0 | 0 | 0 | +| get_opportunity | supervisor | 25 | 305.323792 | 412.832458 | 0 | 15 | 5 | 5 | 0 | +| list_activities | research | 8 | 280.460500 | 511.758250 | 0 | 8 | 0 | 0 | 0 | +| list_activities | supervisor | 8 | 1.360375 | 306.790416 | 0 | 3 | 0 | 0 | 5 | +| list_audit_records | mutation | 2 | 252.085416 | 1976.990250 | 0 | 2 | 0 | 0 | 0 | +| list_audit_records | research | 1 | 1624.517250 | 1624.517250 | 0 | 1 | 0 | 0 | 0 | +| list_proposals | mutation | 1 | 278.802750 | 278.802750 | 0 | 1 | 0 | 0 | 0 | +| list_proposals | supervisor | 5 | 328.874916 | 411.687750 | 0 | 5 | 0 | 0 | 0 | +| log_activity | mutation | 10 | 336.799833 | 400.390917 | 0 | 10 | 0 | 0 | 0 | +| pipeline_aggregate | supervisor | 5 | 499.190875 | 552.098542 | 0 | 5 | 0 | 0 | 0 | +| search_accounts | mutation | 5 | 315.369167 | 1987.302333 | 0 | 5 | 0 | 0 | 0 | +| search_opportunities | analytics | 10 | 125.000666 | 284.981083 | 0 | 5 | 0 | 5 | 0 | +| search_opportunities | research | 5 | 278.005416 | 301.715834 | 0 | 5 | 0 | 0 | 0 | +| search_opportunities | supervisor | 5 | 120.459958 | 130.287458 | 0 | 0 | 0 | 5 | 0 | +| submit_proposal | mutation | 26 | 315.181833 | 352.988250 | 0 | 26 | 0 | 0 | 0 | +| update_account | mutation | 5 | 346.125458 | 2066.207083 | 0 | 5 | 0 | 0 | 0 | +| update_contact | mutation | 5 | 327.592000 | 1943.571542 | 0 | 5 | 0 | 0 | 0 | Excluded invocations: 0 total (cold_start: 0, retry: 0, throttle: 0) @@ -463,13 +486,13 @@ Excluded invocations: 0 total (cold_start: 0, retry: 0, throttle: 0) | --- | --- | | scored_task_count | 120 | | unscored_task_count | 0 | -| mean_correctness | 1.000000 | +| mean_correctness | 0.997500 | | bad_write_rate | 0.000000 | | tool_call_correct_rate | 1.000000 | `bad_write_rate` basis: all 250 declared `must_not_change` assertion(s) in this arm had a captured pre-turn baseline, so the figure is a measurement over them (src/eval/matrix.py). -`mean_correctness` composition: negative guards a do-nothing turn satisfies are 0.8154 of the assertion mass this figure averages — the mean over the 120 scored tasks of each task's own guard share, weighted exactly as `mean_correctness` itself is, rather than the ratio pooled over assertions (pooled totals, for context and not the basis of the figure just quoted: 675 guard assertions and 145 positive ones across those tasks). So the figure is not a task-achievement rate. Taken alone, 145 of 145 positive assertions were satisfied (1.0000); the guards alone were satisfied at 1.0000 (src/eval/matrix.py). +`mean_correctness` composition: negative guards a do-nothing turn satisfies are 0.8155 of the assertion mass this figure averages — the mean over the 120 scored tasks of each task's own guard share, weighted exactly as `mean_correctness` itself is, rather than the ratio pooled over assertions (pooled totals, for context and not the basis of the figure just quoted: 676 guard assertions and 145 positive ones across those tasks). So the figure is not a task-achievement rate. Taken alone, 144 of 145 positive assertions were satisfied (0.9931); the guards alone were satisfied at 0.9970 (src/eval/matrix.py). Retrieval-narrowed task deduction: the assertions this arm did not satisfy on the tasks narrowed to retrieval account for at most 0.000000 of its `mean_correctness` (`gt-04-investigation-turn-budget-partial` scored 1.000000, at most 0.000000; `gt-05-investigation-large-tool-result` scored 1.000000, at most 0.000000; `gt-04-investigation-turn-budget-partial` scored 1.000000, at most 0.000000; `gt-05-investigation-large-tool-result` scored 1.000000, at most 0.000000; `gt-04-investigation-turn-budget-partial` scored 1.000000, at most 0.000000; `gt-05-investigation-large-tool-result` scored 1.000000, at most 0.000000; `gt-04-investigation-turn-budget-partial` scored 1.000000, at most 0.000000; `gt-05-investigation-large-tool-result` scored 1.000000, at most 0.000000; `gt-04-investigation-turn-budget-partial` scored 1.000000, at most 0.000000; `gt-05-investigation-large-tool-result` scored 1.000000, at most 0.000000), an upper bound over the 120 scored tasks derived from this arm's own per-task records. The figure itself is unchanged by this disclosure (src/eval/golden_tasks.yaml). @@ -521,7 +544,7 @@ Retrieval-narrowed task deduction: the assertions this arm did not satisfy on th | gt-18-lookup-account-contacts | machine | 1.000000 | — | no | yes | no | no | | gt-19-lookup-pipeline-aggregate | machine | 1.000000 | — | no | yes | no | no | | gt-20-lookup-proposal-queue | machine | 1.000000 | — | no | yes | no | no | -| gt-21-enrichment-pd04-multi-field-proposal | machine | 1.000000 | — | no | yes | no | no | +| gt-21-enrichment-pd04-multi-field-proposal | machine | 0.700000 | — | no | yes | no | no | | gt-22-investigation-pd05-report-only | machine | 1.000000 | — | no | yes | no | no | | gt-23-investigation-pd06-ambiguous-contact | machine | 1.000000 | — | no | yes | no | no | | gt-24-enrichment-pd07-multi-step-write | machine | 1.000000 | — | no | yes | no | no | @@ -622,7 +645,7 @@ Per-task correctness spread across repeats: | gt-18-lookup-account-contacts | 5 | 1.000000 | 1.000000 | 1.000000 | 1.000000 | | gt-19-lookup-pipeline-aggregate | 5 | 1.000000 | 1.000000 | 1.000000 | 1.000000 | | gt-20-lookup-proposal-queue | 5 | 1.000000 | 1.000000 | 1.000000 | 1.000000 | -| gt-21-enrichment-pd04-multi-field-proposal | 5 | 1.000000 | 1.000000 | 1.000000 | 1.000000 | +| gt-21-enrichment-pd04-multi-field-proposal | 5 | 0.700000 | 0.940000 | 1.000000 | 0.800000 | | gt-22-investigation-pd05-report-only | 5 | 1.000000 | 1.000000 | 1.000000 | 1.000000 | | gt-23-investigation-pd06-ambiguous-contact | 5 | 1.000000 | 1.000000 | 1.000000 | 1.000000 | | gt-24-enrichment-pd07-multi-step-write | 5 | 1.000000 | 1.000000 | 1.000000 | 1.000000 | diff --git a/scripts/check_canary_tenant_empty.py b/scripts/check_canary_tenant_empty.py index 78bd541..20b54f5 100644 --- a/scripts/check_canary_tenant_empty.py +++ b/scripts/check_canary_tenant_empty.py @@ -113,12 +113,12 @@ def _raised_from_connection_failure(exc: BaseException) -> bool: seen: set[int] = set() - current: BaseException | None = exc.__cause__ or exc.__context__ + current: BaseException | None = exc.__cause__ while current is not None and id(current) not in seen: if isinstance(current, _CONNECTION_CAUSES): return True seen.add(id(current)) - current = current.__cause__ or current.__context__ + current = current.__cause__ return False diff --git a/scripts/check_cost_withheld.py b/scripts/check_cost_withheld.py index 523b7d0..d2869c4 100755 --- a/scripts/check_cost_withheld.py +++ b/scripts/check_cost_withheld.py @@ -14,8 +14,8 @@ detection (:data:`_CURRENCY_SHAPE_PATTERN`, :data:`_CURRENCY_FIELD_NAMES`). 2. The same, on the rendered eval dashboard source (``--frontend-dir``; :func:`find_currency_rendering_in_frontend`). -3. While every rate in the price table's ``models`` block is zero, no arm in ``matrix.json`` may - render with ``cost_withheld_provisional`` set to ``false``. +3. While every rate in the price table's ``models`` block, flat and ``long_context``, is zero, no + arm in ``matrix.json`` may render with ``cost_withheld_provisional`` set to ``false``. When ``provisional`` is ``false``, every check is skipped and the gate exits 0. When it is ``true`` and no artifact or consumer was examined, the gate exits 3. @@ -194,24 +194,59 @@ def is_provisional(price_table: dict[str, Any]) -> bool: return bool(price_table.get("provisional", False)) -def all_rates_are_zero(price_table: dict[str, Any]) -> bool: - """Return whether every numeric rate in the price table's ``models`` block is zero.""" +#: Key of a model's optional long-context rate block in the price table. +_LONG_CONTEXT_BLOCK: Final[str] = "long_context" + +#: Non-rate fields of a long-context block: the prompt-token threshold above which it applies. +_LONG_CONTEXT_NON_RATE_FIELDS: Final[frozenset[str]] = frozenset({"above_prompt_tokens"}) + + +def _numeric_rate(model_id: str, field_path: str, value: Any) -> float: + """Return ``value`` as a float, or raise naming the offending price-table field.""" + try: + return float(value) + except (TypeError, ValueError) as exc: + raise CostWithholdingGateError( + f"price table entry {model_id!r}.{field_path!r} is not numeric: {value!r}" + ) from exc + + +def iter_rates(price_table: dict[str, Any]) -> list[tuple[str, str, float]]: + """Return ``(model_id, field_path, rate)`` for every rate in the ``models`` block. + + Covers each model's flat rates and the rates of its optional ``long_context`` block, whose + ``above_prompt_tokens`` threshold is not a rate. The ``routes`` block holds multipliers, not + rates, and is not read. Raises :class:`CostWithholdingGateError` on a non-mapping entry or + block, or a non-numeric rate. + """ models = price_table.get("models", {}) if not isinstance(models, dict): raise CostWithholdingGateError("price table 'models' block is not a mapping") - for model_id, rates in models.items(): - if not isinstance(rates, dict): + rates: list[tuple[str, str, float]] = [] + for model_id, entry in models.items(): + if not isinstance(entry, dict): raise CostWithholdingGateError(f"price table entry for {model_id!r} is not a mapping") - for field_name, value in rates.items(): - try: - numeric_value = float(value) - except (TypeError, ValueError) as exc: + for field_name, value in entry.items(): + if field_name != _LONG_CONTEXT_BLOCK: + rates.append((model_id, field_name, _numeric_rate(model_id, field_name, value))) + continue + if not isinstance(value, dict): raise CostWithholdingGateError( - f"price table entry {model_id!r}.{field_name!r} is not numeric: {value!r}" - ) from exc - if numeric_value != 0.0: - return False - return True + f"price table entry {model_id!r}.{field_name!r} is not a mapping" + ) + for tier_field, tier_value in value.items(): + if tier_field in _LONG_CONTEXT_NON_RATE_FIELDS: + continue + field_path = f"{field_name}.{tier_field}" + rates.append( + (model_id, field_path, _numeric_rate(model_id, field_path, tier_value)) + ) + return rates + + +def all_rates_are_zero(price_table: dict[str, Any]) -> bool: + """Return whether every rate :func:`iter_rates` yields, flat and long-context, is zero.""" + return all(rate == 0.0 for _, _, rate in iter_rates(price_table)) #: Characters removed, and whitespace normalised, before a line is matched, so `$0.47` reads @@ -384,6 +419,13 @@ def find_currency_figures_in_matrix_json(text: str, *, source: str) -> list[str] "narrowed_tasks_scored", # Counts and provenance. "included_invocation_count", + "long_context_invocation_count", + "arm_long_context_invocation_count", + *( + f"{prefix}_{tier}_long_context_invocation_count" + for prefix in ("tier", "summary") + for tier in ("supervisor", "subagent", "tool") + ), "scored_task_count", "unscored_task_count", "total_excluded", diff --git a/scripts/check_plan_security_baseline.py b/scripts/check_plan_security_baseline.py index 91f0e75..384b2dd 100644 --- a/scripts/check_plan_security_baseline.py +++ b/scripts/check_plan_security_baseline.py @@ -3080,8 +3080,9 @@ def check_relay_denies_command_execution(_plan: Plan) -> list[Finding]: #: whereas `EXPECTED_RESOURCE_COUNTS` counts plan instances. DECLARED_INFRA_BLOCK_COUNTS: dict[str, tuple[int, str]] = { 'resource "aws_iam_role_policy"': ( - 22, - "`eval_trigger_artifacts` is one. One was removed with eval_worker: " + 23, + "The lambda_action module's `config_parameter_kms` is one, decrypting the allowlist " + "SecureString. `eval_trigger_artifacts` is one. One was removed with eval_worker: " "`eval_trigger_invoke_worker`. Six more are `sweep_control_artifacts`, " "`sweep_task_invoke_runtime` and `eval_sweep` (`eval_sfn.tf`), `chat_target_invoke_runtime` " "and the mcp_chat_gateway module's `gateway`, and `eval_trigger_states`; the plan was " @@ -3104,14 +3105,14 @@ def check_relay_denies_command_execution(_plan: Plan) -> list[Finding]: "plan re-capture is invisible to EXPECTED_RESOURCE_COUNTS, which is why this count exists.", ), 'resource "aws_kms_key_policy"': ( - 4, - "`aws_kms_key_policy.eval_artifacts` and `aws_kms_key_policy.cursor` at the root, beside " - "`dynamodb` and the frontend module's `frontend`; `EXPECTED_RESOURCE_COUNTS` expects the " - "same four in the plan.", + 5, + "`aws_kms_key_policy.eval_artifacts`, `aws_kms_key_policy.cursor` and " + "`aws_kms_key_policy.api_allowlist` at the root, beside `dynamodb` and the frontend " + "module's `frontend`; `EXPECTED_RESOURCE_COUNTS` expects the same five in the plan.", ), 'data "aws_iam_policy_document"': ( - 43, - "`eval_trigger_artifacts` is one. Four were removed with eval_worker: " + 44, + "The lambda_action module's `config_parameter_kms` is one. `eval_trigger_artifacts` is one. Four were removed with eval_worker: " "`eval_trigger_invoke_worker`, " "`eval_worker_invoke_tools` and the deleted async_failure_destination module's two. " "Nine more are `sweep_control_artifacts`, `sweep_task_invoke_runtime`, " @@ -4505,6 +4506,57 @@ def _bucket_denies( ) +#: Root locals naming the allowlist CMK's consumer roles, its SSM service and the parameter ARN. +ALLOWLIST_KEY_CONSUMERS_LOCAL = "api_allowlist_key_consumer_arns" +ALLOWLIST_SSM_SERVICE_LOCAL = "api_allowlist_ssm_service" +ALLOWLIST_PARAMETER_ARN_LOCAL = "api_allowed_client_ids_parameter_arn" +ALLOWLIST_KEY_REFERENCE = "aws_kms_key.api_allowlist.arn" +ALLOWLIST_OPERATOR_SID = "AllowOperatorsUseThroughSsm" +ALLOWLIST_OPERATOR_ACTIONS: tuple[str, ...] = ("kms:Encrypt", "kms:Decrypt", "kms:DescribeKey") +ALLOWLIST_CONSUMER_SID = "AllowConsumersDecryptTheParameter" +ALLOWLIST_WRITE_SID = "DenyKeyWriteToNonOperators" +#: Every operation that writes under the allowlist CMK; refused to every principal but the operators. +ALLOWLIST_WRITE_ACTIONS: tuple[str, ...] = ( + "kms:Encrypt", + "kms:GenerateDataKey*", + "kms:ReEncrypt*", + "kms:CreateGrant", +) +ALLOWLIST_GRANT_SID = "DecryptConfigParameters" +ALLOWLIST_GRANT_DENY_SID = "DenyConfigParameterDecryptUnderOtherKeys" + +#: The allowlist CMK's policy, read like `STORE_POLICIES` plus its operator write Deny. +ALLOWLIST_KEY_POLICY = StorePolicy( + "allowlist_key", + "aws_kms_key_policy", + ".", + "api_allowlist", + ( + "EnableIAMUserPermissions", + ALLOWLIST_OPERATOR_SID, + ALLOWLIST_CONSUMER_SID, + ALLOWLIST_WRITE_SID, + *_KEY_DENY_SIDS, + ), + ( + StoreDeny( + ALLOWLIST_WRITE_SID, + ALLOWLIST_WRITE_ACTIONS, + ("*",), + CRM_OPERATOR_PRINCIPALS_VAR, + STORE_KEY_EXEMPTION_OPERATOR, + ), + *_key_denies( + f"concat(local.{ALLOWLIST_KEY_CONSUMERS_LOCAL}, {CRM_OPERATOR_PRINCIPALS_VAR})", + _STORE_ADMIN_ROOT, + ), + ), + root_sid="EnableIAMUserPermissions", + root_wired=True, + lockout_flag=True, +) + + def _store_root_problems(statement: RenderedStatement) -> list[str]: problems: list[str] = [] if statement.effect != "Allow": @@ -5009,11 +5061,194 @@ def _cursor_principals_findings() -> list[Finding]: ] +def _allowlist_allow_problems( + statement: RenderedStatement | None, + principal: str, + actions: tuple[str, ...], + condition: dict[str, tuple[HclValue, ...]], +) -> list[str]: + if statement is None: + return ["statement absent"] + problems: list[str] = [] + if statement.effect != "Allow": + problems.append(f"effect={statement.effect!r}, expected 'Allow'") + if statement.principal != {"AWS": HclExpr(principal)}: + problems.append(f"principal={statement.principal!r}, expected {{'AWS': {principal}}}") + if statement.actions != actions: + problems.append(f"actions={list(statement.actions)!r}, expected exactly {list(actions)!r}") + if statement.resources != ("*",): + problems.append(f"resources={list(statement.resources)!r}, expected exactly ['*']") + if statement.conditions != {"StringEquals": condition}: + problems.append( + f"conditions={statement.conditions!r}, expected exactly StringEquals {condition!r}" + ) + return problems + + +def _allowlist_allow_findings() -> list[Finding]: + """The allowlist CMK's two Allows: operators use it through SSM; consumers decrypt the parameter.""" + address = f"./aws_kms_key_policy.{ALLOWLIST_KEY_POLICY.name}" + blocks = [ + b + for d, n, b in tf_resource_blocks("aws_kms_key_policy") + if d == "." and n == ALLOWLIST_KEY_POLICY.name + ] + if len(blocks) != 1: + return [Finding(_STORE_CONTROL, address, "fail", f"found {len(blocks)} policy blocks")] + try: + statements = {s.sid: s for s in rendered_policy_statements(".", blocks[0])} + except ValueError as exc: + return [Finding(_STORE_CONTROL, address, "fail", f"policy unreadable: {exc}")] + via_ssm = HclExpr(f"local.{ALLOWLIST_SSM_SERVICE_LOCAL}") + expected = { + ALLOWLIST_OPERATOR_SID: _allowlist_allow_problems( + statements.get(ALLOWLIST_OPERATOR_SID), + CRM_OPERATOR_PRINCIPALS_VAR, + ALLOWLIST_OPERATOR_ACTIONS, + {"kms:ViaService": (via_ssm,)}, + ), + ALLOWLIST_CONSUMER_SID: _allowlist_allow_problems( + statements.get(ALLOWLIST_CONSUMER_SID), + f"local.{ALLOWLIST_KEY_CONSUMERS_LOCAL}", + ("kms:Decrypt",), + { + "kms:ViaService": (via_ssm,), + "kms:EncryptionContext:PARAMETER_ARN": ( + HclExpr(f"local.{ALLOWLIST_PARAMETER_ARN_LOCAL}"), + ), + }, + ), + } + return [ + Finding( + _STORE_CONTROL, + f"{address}#{sid}", + "fail" if problems else "pass", + "; ".join(problems) or "matches the expected Allow statement", + ) + for sid, problems in expected.items() + ] + + +def _allowlist_parameter_finding() -> Finding: + """The allowlist parameter is a SecureString under the allowlist CMK.""" + address = "./aws_ssm_parameter.api_allowed_client_ids" + blocks = [ + b + for d, n, b in tf_resource_blocks("aws_ssm_parameter") + if d == "." and n == "api_allowed_client_ids" + ] + if len(blocks) != 1: + return Finding(_STORE_CONTROL, address, "fail", f"found {len(blocks)}, expected 1") + problems: list[str] = [] + kind = _hcl_expression(blocks[0], "type") + if kind != "SecureString": + problems.append(f"type={kind!r}, expected 'SecureString'") + key = _hcl_expression(blocks[0], "key_id") + if key != ALLOWLIST_KEY_REFERENCE: + problems.append(f"key_id={key!r}, expected {ALLOWLIST_KEY_REFERENCE}") + return Finding( + _STORE_CONTROL, + address, + "fail" if problems else "pass", + "; ".join(problems) or "a SecureString under the allowlist CMK", + ) + + +def _allowlist_grant_finding() -> Finding: + """The `config_parameter_kms` document: Decrypt through SSM under the context, Deny other keys.""" + address = "modules/lambda_action/data.aws_iam_policy_document.config_parameter_kms" + blocks = [ + b + for d, n, b in tf_resource_blocks("aws_iam_policy_document", kind="data") + if d == "modules/lambda_action" and n == "config_parameter_kms" + ] + if len(blocks) != 1: + return Finding(_STORE_CONTROL, address, "fail", f"found {len(blocks)}, expected 1") + source = re.sub(r"\s*=\s*", " = ", _normalised_expression(blank_comments(blocks[0]))) + expected = ( + "statement { " + f'sid = "{ALLOWLIST_GRANT_SID}" effect = "Allow" actions = ["kms:Decrypt"] ' + "resources = [var.config_parameter_kms_key_arn] " + 'condition { test = "StringEquals" variable = "kms:ViaService" ' + 'values = ["ssm.${local.config_parameter_region}.amazonaws.com"] } ' + 'condition { test = "StringEquals" variable = "kms:EncryptionContext:PARAMETER_ARN" ' + "values = var.config_parameter_arns } } " + "statement { " + f'sid = "{ALLOWLIST_GRANT_DENY_SID}" effect = "Deny" actions = ["kms:Decrypt"] ' + "not_resources = [var.config_parameter_kms_key_arn] " + 'condition { test = "StringLike" variable = "kms:ViaService" ' + 'values = ["ssm.*.amazonaws.com"] } }' + ) + found = expected in source and source.count("statement {") == 2 + return Finding( + _STORE_CONTROL, + address, + "pass" if found else "fail", + "Decrypt on the one key through SSM under each parameter's context, and a Deny on " + "decrypting through SSM under any other key" + if found + else "the document is not exactly the Decrypt-through-SSM statement under " + "kms:EncryptionContext:PARAMETER_ARN = var.config_parameter_arns plus the NotResource Deny " + "on kms:ViaService ssm.*.amazonaws.com", + ) + + +def _allowlist_consumers_finding() -> Finding: + """Every module call granted the parameter gets the key, and the key's consumer list is theirs.""" + resource = f"local.{ALLOWLIST_KEY_CONSUMERS_LOCAL}" + try: + expression = _normalised_expression(_root_local_expression(ALLOWLIST_KEY_CONSUMERS_LOCAL)) + except ValueError as exc: + return Finding(_STORE_CONTROL, resource, "fail", str(exc)) + parameter = f"local.{ALLOWLIST_PARAMETER_ARN_LOCAL}" + problems: list[str] = [] + readers: set[str] = set() + for name, body in _root_module_calls().items(): + granted = _normalised_expression(_hcl_expression(body, "config_parameter_arns")) + key = _normalised_expression(_hcl_expression(body, "config_parameter_kms_key_arn")) + if parameter not in granted: + if key: + problems.append(f"module.{name} receives the key without the parameter") + continue + readers.add(name) + if name == "lambda_action": + wanted_key = ( + f"contains(local.routed_tool_names, each.key) ? {ALLOWLIST_KEY_REFERENCE} : null" + ) + else: + wanted_key = ALLOWLIST_KEY_REFERENCE + if key != wanted_key: + problems.append( + f"module.{name} config_parameter_kms_key_arn={key!r}, expected {wanted_key!r}" + ) + listed = set(re.findall(r"module\.(lambda_action_\w+)\.role_arn", expression)) + if "lambda_action" in readers: + loop = "[for name in local.routed_tool_names : module.lambda_action[name].role_arn]" + if loop not in expression: + problems.append(f"{resource} does not name the routed tools' roles by {loop}") + if listed != readers - {"lambda_action"}: + problems.append( + f"{resource} names {sorted(listed)!r}, the parameter readers are " + f"{sorted(readers - {'lambda_action'})!r}" + ) + if not readers: + problems.append("no module call is granted the parameter, so this verified nothing") + return Finding( + _STORE_CONTROL, + resource, + "fail" if problems else "pass", + "; ".join(problems) + or "names exactly the roles granted the parameter, and each of them receives the key", + ) + + def check_store_principal_deny(_plan: Plan) -> list[Finding]: - """The logs, CloudTrail, frontend, eval-artefacts and cursor key and bucket policies deny the - data plane, key use and administration outside each store's allowlist and the operators, the - cursor key refuses use outside its encryption context, and every key rotates. Read from - source.""" + """The logs, CloudTrail, frontend, eval-artefacts, cursor and allowlist key and bucket policies + deny the data plane, key use and administration outside each store's allowlist and the + operators, the cursor key refuses use outside its encryption context, the allowlist key admits + writes from the operators only and decryption only through SSM, and every key rotates. Read + from source.""" findings: list[Finding] = [] for policy in STORE_POLICIES: findings += _store_policy_findings(policy) @@ -5024,6 +5259,11 @@ def check_store_principal_deny(_plan: Plan) -> list[Finding]: findings.append(_cursor_purpose_finding()) findings.append(_cursor_grant_finding()) findings += _cursor_principals_findings() + findings += _store_policy_findings(ALLOWLIST_KEY_POLICY) + findings += _allowlist_allow_findings() + findings.append(_allowlist_parameter_finding()) + findings.append(_allowlist_grant_finding()) + findings.append(_allowlist_consumers_finding()) findings += _key_rotation_findings() return findings @@ -6539,11 +6779,13 @@ def _vpc_endpoint_count() -> int: "Lambda groups, so the total exceeds the function count by five.", ), "aws_iam_role_policy": ExpectedCount( - per_tool=4, - fixed=51, - basis="FOUR per action Lambda -- `dynamodb`, `dynamodb_kms`, `logs`, and the `vpc_access` " - "Hyperplane ENI grant, without which the apply is green and every invocation fails " - "-- plus 51 elsewhere: four each on the interceptor, pre-token-generation, eval_trigger, " + per_tool=5, + fixed=53, + basis="FIVE per action Lambda -- `dynamodb`, `dynamodb_kms`, `logs`, `config_parameter_kms` " + "(decrypting the allowlist SecureString), and the `vpc_access` Hyperplane ENI grant, without " + "which the apply is green and every invocation fails -- plus 53 elsewhere: the " + "`config_parameter_kms` policies of eval_trigger, capabilities and user_settings (3), less " + "one for the agent-only `get_page_context`, which reads no parameter; and the 51 that follow: four each on the interceptor, pre-token-generation, eval_trigger, " "chat_interceptor, chat_target, user_settings, sweep_control and sweep_task lambda_action " "instantiations (32), capabilities' `logs`, `vpc_access` and `dynamodb` (3), the relay's four " "(invoke-runtime, logs, read-audience-parameter, vpc_access), nine at the root " @@ -6558,10 +6800,12 @@ def _vpc_endpoint_count() -> int: per_managed_dynamodb=-1, ), "aws_iam_policy_document": ExpectedCount( - per_tool=3, - fixed=51, - basis="one document behind each of an action Lambda's `dynamodb`, `dynamodb_kms` and `logs` " - "role policies, plus 51: ten at the root (agentcore_gateway permissions, agentcore_runtime " + per_tool=4, + fixed=53, + basis="one document behind each of an action Lambda's `dynamodb`, `dynamodb_kms`, `logs` and " + "`config_parameter_kms` role policies, plus 53: the `config_parameter_kms` documents of " + "eval_trigger, capabilities and user_settings (3), less one for the agent-only " + "`get_page_context`; and the 51 that follow: ten at the root (agentcore_gateway permissions, agentcore_runtime " "permissions and logs, the two agent-runtime relay-only resource policies, " "chat_target_invoke_runtime, eval_sweep_permissions, eval_trigger_artifacts, sweep_control_artifacts, " "sweep_task_invoke_runtime), two in agent_relay, two in agentcore, one in api (its resource " @@ -6646,14 +6890,15 @@ def _vpc_endpoint_count() -> int: "uploads so the current SPA build is never expired", ), "aws_kms_key": ExpectedCount( - fixed=6, + fixed=7, basis="one CMK each for DynamoDB, the frontend, CloudTrail, CloudWatch Logs, the " - "eval-artifacts bucket and the pagination cursors. The last two are declared at the ROOT " - "(aws_kms_key.eval_artifacts, aws_kms_key.cursor) rather than inside a module, so their " - "policies render at plan time", + "eval-artifacts bucket, the pagination cursors and the API client-allowlist SecureString. " + "The last three are declared at the ROOT (aws_kms_key.eval_artifacts, aws_kms_key.cursor, " + "aws_kms_key.api_allowlist) rather than inside a module, so their policies render at plan " + "time", ), "aws_kms_alias": ExpectedCount( - fixed=6, basis="one alias per CMK above, so each key is addressable by a stable name" + fixed=7, basis="one alias per CMK above, so each key is addressable by a stable name" ), # --- relay, runtime, frontend and supporting resources --- "aws_bedrockagentcore_resource_policy": ExpectedCount( @@ -6669,14 +6914,16 @@ def _vpc_endpoint_count() -> int: "secret and a JWT check of its own", ), "aws_kms_key_policy": ExpectedCount( - fixed=4, - basis="four, each carrying the account-root statement and the Denies on key use and key " + fixed=5, + basis="five, each carrying the account-root statement and the Denies on key use and key " "administration outside its principal lists: the frontend CMK's, which also grants " "CloudFront's service principal decrypt against this distribution only; the DynamoDB table " "CMK's; the eval-artefacts CMK's, separate from its key because an inline policy would " - "cycle through the CodeBuild role; and the cursor CMK's, which also refuses every use " - "outside the `crm-cursor` encryption context. The logs and CloudTrail CMKs carry their " - "policy inline on the key resource, which is why this is 4 rather than 6 -- a discrepancy " + "cycle through the CodeBuild role; the cursor CMK's, which also refuses every use " + "outside the `crm-cursor` encryption context; and the allowlist CMK's, which admits writes " + "from the operators only and decryption only through SSM. The logs and CloudTrail CMKs " + "carry their policy inline on the key resource, which is why this is 5 rather than 7 -- a " + "discrepancy " "worth keeping visible in the basis", ), "aws_cloudfront_function": ExpectedCount( diff --git a/scripts/check_relay_thinness.py b/scripts/check_relay_thinness.py index b72f7e4..2637f42 100644 --- a/scripts/check_relay_thinness.py +++ b/scripts/check_relay_thinness.py @@ -3,7 +3,7 @@ """Gate: the streaming relay stays thin. The relay validates a Cognito JWT, calls `InvokeAgentRuntime` with SigV4, and re-emits frames. It -holds no business logic, no tool dispatch, and constructs no `Identity`. Three checks: +holds no business logic, no tool dispatch, and constructs no `Identity`. Four checks: 1. **No domain import.** Nothing under `src/relay/` may import from `src/crm/`, `src/agents/`, `src/gateway/`, or `src/eval/`, or reach outside its own package by relative path. @@ -11,6 +11,9 @@ `identity_from_claims`, and no assembly of an identity-shaped object literal. 3. **A size ceiling on production source** under `src/`. Test files and generated files are exempt from the ceiling; generated files are still subject to checks 1 and 2. +4. **No identity attribute of its own.** Production source under `src/` sets no `principal_type`, + `rep_id` or `full_visibility` key, with exactly one admitted exception: `src/runtime.ts` sets + `principal_type: RELAY_PRINCIPAL_TYPE` once and declares `RELAY_PRINCIPAL_TYPE = "user"` once. Reading `frontend/package.json` as data is not an import and is not flagged. @@ -70,6 +73,17 @@ re.DOTALL, ) +#: An object key naming an identity attribute, quoted or bare, with the expression it is set to. +_IDENTITY_ATTRIBUTE_KEY = re.compile( + r"""(?principal_type|rep_id|full_visibility)['"]?\s*:\s*(?P[^,}\n]*)""" +) + +#: The single identity attribute the relay may set: its channel's declared principal type. +ADMITTED_ATTRIBUTE_FILE = Path("src") / "runtime.ts" +ADMITTED_ATTRIBUTE = ("principal_type", "RELAY_PRINCIPAL_TYPE") +_ADMITTED_CONSTANT = re.compile(r"""\bconst\s+RELAY_PRINCIPAL_TYPE\s*=\s*(?P[^;\n]*)""") +ADMITTED_CONSTANT_VALUE = '"user"' + _LINE_COMMENT = re.compile(r"//[^\n]*") _BLOCK_COMMENT = re.compile(r"/\*.*?\*/", re.DOTALL) @@ -175,6 +189,57 @@ def find_identity_construction(path: Path, code: str) -> list[Violation]: return violations +def find_identity_attributes(path: Path, code: str, relay_dir: Path) -> list[Violation]: + """Report identity attribute keys in production source beyond the one admitted declaration. + + Only files under `src/` are scanned. In :data:`ADMITTED_ATTRIBUTE_FILE`, exactly one + `principal_type: RELAY_PRINCIPAL_TYPE` and exactly one `RELAY_PRINCIPAL_TYPE = "user"` are + required; anything else, there or elsewhere, is a violation. + """ + relative = path.relative_to(relay_dir) + if relative.parts[0] != "src": + return [] + admitted_here = relative == ADMITTED_ATTRIBUTE_FILE + violations: list[Violation] = [] + admitted_count = 0 + for match in _IDENTITY_ATTRIBUTE_KEY.finditer(code): + key, value = match.group("key"), match.group("value").strip() + if admitted_here and (key, value) == ADMITTED_ATTRIBUTE: + admitted_count += 1 + continue + violations.append( + Violation( + path, + "identity-attribute", + f"sets the identity attribute `{key}: {value}`. The relay forwards verified " + "claims verbatim; the only attribute it declares is " + f"`{ADMITTED_ATTRIBUTE[0]}: {ADMITTED_ATTRIBUTE[1]}` in {ADMITTED_ATTRIBUTE_FILE}.", + ) + ) + if not admitted_here: + return violations + if admitted_count != 1: + violations.append( + Violation( + path, + "identity-attribute", + f"sets `{ADMITTED_ATTRIBUTE[0]}: {ADMITTED_ATTRIBUTE[1]}` {admitted_count} times; " + "the Runtime payload must declare it exactly once.", + ) + ) + constants = [m.group("value").strip() for m in _ADMITTED_CONSTANT.finditer(code)] + if constants != [ADMITTED_CONSTANT_VALUE]: + violations.append( + Violation( + path, + "identity-attribute", + f"declares RELAY_PRINCIPAL_TYPE as {constants!r}; it must be declared exactly once " + f"as {ADMITTED_CONSTANT_VALUE}, the only principal the relay serves.", + ) + ) + return violations + + def find_oversize(path: Path, text: str, relay_dir: Path, max_lines: int) -> list[Violation]: """Report a hand-written production relay file past the line ceiling. @@ -198,13 +263,14 @@ def find_oversize(path: Path, text: str, relay_dir: Path, max_lines: int) -> lis def scan_relay(relay_dir: Path, max_lines: int) -> list[Violation]: - """Run all three checks over every relay source file.""" + """Run all four checks over every relay source file.""" violations: list[Violation] = [] for path in collect_relay_sources(relay_dir): text = path.read_text(encoding="utf-8", errors="replace") code = strip_comments(text) violations += find_domain_imports(path, code, relay_dir) violations += find_identity_construction(path, code) + violations += find_identity_attributes(path, code, relay_dir) violations += find_oversize(path, text, relay_dir, max_lines) return violations @@ -255,7 +321,7 @@ def main(argv: list[str] | None = None) -> int: longest = max((len(p.read_text(errors="replace").splitlines()) for p in production), default=0) print( f"check_relay_thinness: OK — {len(sources)} relay source file(s) scanned for domain imports " - f"and `Identity` construction; {len(production)} hand-written production file(s), longest " + f"and `Identity` construction or attributes; {len(production)} hand-written production file(s), longest " f"{longest} lines (ceiling {args.max_lines}, {args.max_lines - longest} to spare)." ) return 0 diff --git a/scripts/seed_demo_tenant.py b/scripts/seed_demo_tenant.py index 039bbb5..7adccfe 100644 --- a/scripts/seed_demo_tenant.py +++ b/scripts/seed_demo_tenant.py @@ -110,12 +110,12 @@ def _raised_from_connection_failure(exc: BaseException) -> bool: seen: set[int] = set() - current: BaseException | None = exc.__cause__ or exc.__context__ + current: BaseException | None = exc.__cause__ while current is not None and id(current) not in seen: if isinstance(current, _CONNECTION_CAUSES): return True seen.add(id(current)) - current = current.__cause__ or current.__context__ + current = current.__cause__ return False diff --git a/scripts/smoke/api_routes.py b/scripts/smoke/api_routes.py index 4001ec5..9ea640e 100644 --- a/scripts/smoke/api_routes.py +++ b/scripts/smoke/api_routes.py @@ -1312,7 +1312,7 @@ def allowlist_checks( (name,) = values spa_client_id = ctx.outputs.require_str("cognito_spa_client_id") ssm = ctx.client("ssm") - parameter = ssm.get_parameter(Name=name)["Parameter"] + parameter = ssm.get_parameter(Name=name, WithDecryption=True)["Parameter"] raw_value = str(parameter.get("Value", "")) listed = [part.strip() for part in raw_value.split(",") if part.strip()] yield CheckResult( @@ -1325,8 +1325,8 @@ def allowlist_checks( parameter_type = str(parameter.get("Type", "")) yield CheckResult( resource=f"parameter:{name}", - check="allowlist_parameter_type_string", - status="pass" if parameter_type == "String" else "fail", + check="allowlist_parameter_type_securestring", + status="pass" if parameter_type == "SecureString" else "fail", detail=f"Type={parameter_type!r}", ) tags = ssm.list_tags_for_resource(ResourceType="Parameter", ResourceId=name).get("TagList", []) diff --git a/scripts/smoke/cloudfront.py b/scripts/smoke/cloudfront.py index 402b421..d479d05 100644 --- a/scripts/smoke/cloudfront.py +++ b/scripts/smoke/cloudfront.py @@ -14,10 +14,11 @@ data-plane origin, a viewer-request function association, and response compression. Asserts one custom response-headers policy on every behaviour, carrying an enforced -`Content-Security-Policy` (`Override` on) and `X-Frame-Options: DENY`, and that the served header on -`/`, an unauthenticated `/api/*` response and an unauthenticated `/agent/*` response is that policy: -the designed directive set, Cognito in `connect-src`, and a `sha256-` source for every inline script -the served document carries. +`Content-Security-Policy` (`Override` on) and `X-Frame-Options: DENY`; that each probed behaviour +serves `Strict-Transport-Security` with max-age=31536000, includeSubDomains and preload; and that the +served CSP on `/`, an unauthenticated `/api/*` response and an unauthenticated `/agent/*` response is +that policy: the designed directive set, Cognito in `connect-src`, and a `sha256-` source for every +inline script the served document carries. Then drives the distribution over the network: a client-side deep link must return the built SPA document (and a missing asset path must not), while an unauthenticated `/api/*` request must still @@ -797,8 +798,10 @@ def _check_api_negative_auth_preserved( #: A `script-src` hash source. _SCRIPT_HASH_SOURCE = re.compile(r"^'sha256-[A-Za-z0-9+/]{43}='$") -#: An inline `", re.DOTALL) +#: An inline `", + '', + "", + '", + "", + '', + "", + '" + assert _INLINE_SCRIPT.findall(document) == ["window.x = 1;"] + + +@pytest.mark.parametrize( + "tag", + [ + '', + '', + '', + ], + ids=["module-src", "uppercase-src", "spaced-src"], +) +def test_a_script_with_a_src_attribute_is_not_hashed(tag: str) -> None: + extra = SERVED_INDEX.replace("", f"{tag}") + rows = _csp_results(header_caller=_header_caller(index_body=extra)) + assert _failing(rows) == set() + + +def test_only_src_scripts_still_verify_nothing() -> None: + only_src = SERVED_INDEX.replace(f"", "").replace( + "", '' + ) + rows = _csp_results(header_caller=_header_caller(index_body=only_src)) + row = rows[("distribution", "served_csp_allows_every_inline_script")] + assert row.status == "fail" + assert "verified nothing" in row.detail + + +def test_an_uppercase_theme_script_is_hashed_like_the_lowercase_one() -> None: + shouted = SERVED_INDEX.replace( + f"", f"" + ) + rows = _csp_results(header_caller=_header_caller(index_body=shouted)) + assert _failing(rows) == set() + + def test_a_document_with_no_inline_script_fails_rather_than_passing_vacuously() -> None: bare = SERVED_INDEX.replace(f"", "") rows = _csp_results(header_caller=_header_caller(index_body=bare)) @@ -1814,3 +1898,78 @@ def test_a_policy_property_broken_fails_exactly_the_policy_row( rows = _csp_results(**client_kwargs) assert _failing(rows) == {("distribution", "response_headers_policy_enforces_csp")} assert fragment in rows[("distribution", "response_headers_policy_enforces_csp")].detail + + +# --------------------------------------------------------------------------- +# The served Strict-Transport-Security header. +# --------------------------------------------------------------------------- + +#: The probe path and resource of each `hsts_header_served` row. +HSTS_PROBES: tuple[tuple[str, str], ...] = ( + ("/", "behavior:/*"), + (_UNAUTHENTICATED_API_PATH, "behavior:/api/*"), + ("/agent/csp-probe", "behavior:/agent/*"), +) + + +def _hsts_rows(header_caller: HeaderCaller) -> dict[str, CheckResult]: + """Run `checks()` and return each behaviour's `hsts_header_served` row, asserting exactly one.""" + results = list( + checks( + _make_ctx(_distribution_config()), + http_caller=_healthy_caller, + http_method_caller=_as_method_caller(_healthy_caller), + tls_probe=_refused_probe, + http_header_caller=header_caller, + ) + ) + return {resource: _one(results, resource, "hsts_header_served") for _, resource in HSTS_PROBES} + + +def test_every_behaviour_serving_the_designed_hsts_passes() -> None: + rows = _hsts_rows(_healthy_header_caller) + assert {resource: row.status for resource, row in rows.items()} == { + resource: "pass" for _, resource in HSTS_PROBES + } + + +@pytest.mark.parametrize( + ("value", "reason"), + [ + (None, "no Strict-Transport-Security"), + ("max-age=31536000", "includesubdomains is absent"), + ("max-age=31536000; includeSubDomains", "preload is absent"), + ("max-age=31536000; preload", "includesubdomains is absent"), + ("max-age=86400; includeSubDomains; preload", "max-age is '86400'"), + ("includeSubDomains; preload", "max-age is None"), + ], + ids=[ + "absent", + "max-age-only", + "no-preload", + "no-include-subdomains", + "short-max-age", + "no-max-age", + ], +) +@pytest.mark.parametrize(("path", "resource"), HSTS_PROBES) +def test_a_behaviour_serving_a_weaker_hsts_fails_exactly_its_own_row( + path: str, resource: str, value: str | None, reason: str +) -> None: + rows = _hsts_rows(_header_caller(hsts_by_path={path: value})) + failing = {name for name, row in rows.items() if row.status != "pass"} + assert failing == {resource} + assert reason in rows[resource].detail + + +def test_hsts_directive_names_are_case_insensitive() -> None: + rows = _hsts_rows( + _header_caller(hsts_by_path={"/": "MAX-AGE=31536000;INCLUDESUBDOMAINS;PRELOAD"}) + ) + assert rows["behavior:/*"].status == "pass", rows["behavior:/*"].detail + + +def test_a_transport_failure_fails_the_hsts_row_too() -> None: + rows = _hsts_rows(_header_caller(transport_failure_paths=["/agent/csp-probe"])) + assert rows["behavior:/agent/*"].status == "fail" + assert "transport error" in rows["behavior:/agent/*"].detail diff --git a/tests/unit/test_smoke_cognito_claims.py b/tests/unit/test_smoke_cognito_claims.py index 0d55d6b..8de58d1 100644 --- a/tests/unit/test_smoke_cognito_claims.py +++ b/tests/unit/test_smoke_cognito_claims.py @@ -813,7 +813,18 @@ def _checks_with_provisioner( queue_revocation: bool = True, ) -> list[Any]: client, stubber = _idp_client() - _add_pool(stubber, _compliant_pool(), spa_client=spa_client) + _add_pool(stubber, _compliant_pool(Domain=_DOMAIN), spa_client=spa_client) + stubber.add_response( + "describe_user_pool_client", + { + "UserPoolClient": { + "UserPoolId": _USER_POOL_ID, + "ClientId": _SPA_CLIENT_ID, + "LogoutURLs": [_LOGOUT_URL], + } + }, + {"UserPoolId": _USER_POOL_ID, "ClientId": _SPA_CLIENT_ID}, + ) if queue_revocation: _queue_revocation(stubber, revoke_error=revoke_error, refresh_outcome=refresh_outcome) with stubber: @@ -824,6 +835,7 @@ def _checks_with_provisioner( provisioner=provisioner, decode_id_token=_live_decode_for(ctx, jwks_url=test_key.jwks_url()), signup_client=_refusing_signup_client(), + logout_probe=_Probe(), ) ) @@ -1201,7 +1213,7 @@ def test_token_revocation_not_enabled_reds_exactly_its_row(value: object) -> Non ({"refresh_outcome": "InvalidParameterException"}, {}), ({"refresh_outcome": "TooManyRequestsException"}, {}), ({"revoke_error": "UnsupportedTokenTypeException"}, {}), - ({"queue_revocation": False}, {"refresh_token": ""}), + ({"queue_revocation": False}, {"refresh_token": ""}), # nosec B105 # empty-token fixture ], ids=["refresh-accepted", "other-error", "throttled", "revoke-failed", "no-refresh-token"], ) @@ -1241,5 +1253,129 @@ def test_the_live_provisioner_carries_the_issued_refresh_token() -> None: id_token, _secret = provisioner._one_auth_attempt( username="u", password=_password(), secret=None ) - assert id_token == "id" + assert id_token == "id" # nosec B105 # a stub token value, not a credential assert provisioner._refresh_token == _REFRESH_TOKEN + + +# --------------------------------------------------------------------------- +# Hosted UI sign-out: registered logout_uri -> 302 to it; unregistered -> no redirect to it. +# --------------------------------------------------------------------------- + +_LOGOUT_URL = "https://d111111abcdef8.cloudfront.net/logout" +_DOMAIN = "crm-fixture" +_LOGOUT_BASE = f"https://{_DOMAIN}.auth.us-east-1.amazoncognito.com/logout" + + +class _Probe: + """Answers like Cognito: a registered logout_uri redirects there, anything else to /error.""" + + def __init__(self, registered: str = _LOGOUT_URL, *, echo: bool = False) -> None: + self.registered = registered + self.echo = echo + self.urls: list[str] = [] + + def __call__(self, url: str) -> tuple[int, str]: + from urllib.parse import parse_qs, urlsplit + + self.urls.append(url) + query = parse_qs(urlsplit(url).query) + uri = query["logout_uri"][0] + if uri == self.registered or self.echo: + return 302, uri + return ( + 302, + f"https://{_DOMAIN}.auth.us-east-1.amazoncognito.com/error?error=redirect_mismatch", + ) + + +def _logout_rows( + *, + probe: Any, + domain: str | None = _DOMAIN, + logout_urls: list[str] | None = None, +) -> dict[str, Any]: + client, stubber = _idp_client() + pool = _compliant_pool(**({"Domain": domain} if domain is not None else {})) + _add_pool(stubber, pool) + if domain: + stubber.add_response( + "describe_user_pool_client", + { + "UserPoolClient": { + "UserPoolId": _USER_POOL_ID, + "ClientId": _SPA_CLIENT_ID, + "LogoutURLs": [_LOGOUT_URL] if logout_urls is None else logout_urls, + } + }, + {"UserPoolId": _USER_POOL_ID, "ClientId": _SPA_CLIENT_ID}, + ) + with stubber: + ctx = context_for_test(outputs=_outputs(), clients={"cognito-idp": client}) + results = list(checks(ctx, logout_probe=probe)) + return {r.check: r for r in results if r.check.startswith("hosted_ui_logout_")} + + +def test_the_registered_logout_redirects_and_the_unregistered_one_does_not() -> None: + probe = _Probe() + rows = _logout_rows(probe=probe) + assert {c: r.status for c, r in rows.items()} == { + "hosted_ui_logout_registered_redirects": "pass", + "hosted_ui_logout_unregistered_refused": "pass", + } + assert all(url.startswith(f"{_LOGOUT_BASE}?") for url in probe.urls) + assert f"client_id={_SPA_CLIENT_ID}" in probe.urls[0] + + +def test_a_registered_logout_that_does_not_redirect_there_fails() -> None: + rows = _logout_rows(probe=_Probe(registered="https://elsewhere.example/logout")) + assert rows["hosted_ui_logout_registered_redirects"].status == "fail" + assert rows["hosted_ui_logout_unregistered_refused"].status == "pass" + + +def test_an_endpoint_that_redirects_to_any_logout_uri_fails_the_unregistered_row() -> None: + rows = _logout_rows(probe=_Probe(echo=True)) + assert rows["hosted_ui_logout_registered_redirects"].status == "pass" + assert rows["hosted_ui_logout_unregistered_refused"].status == "fail" + + +def test_an_unreachable_hosted_ui_defers_both_rows_with_a_reason() -> None: + def _down(url: str) -> tuple[int, str]: + raise OSError("connection refused") + + rows = _logout_rows(probe=_down) + assert {r.status for r in rows.values()} == {"defer"} + assert {r.defer_reason for r in rows.values()} == {"hosted_ui_unreachable"} + + +def test_a_pool_without_a_domain_defers_both_rows_with_a_reason() -> None: + rows = _logout_rows(probe=_Probe(), domain=None) + assert {r.status for r in rows.values()} == {"defer"} + assert {r.defer_reason for r in rows.values()} == {"hosted_ui_domain_absent"} + + +def test_a_client_with_no_logout_url_fails_both_rows() -> None: + rows = _logout_rows(probe=_Probe(), logout_urls=[]) + assert {r.status for r in rows.values()} == {"fail"} + + +def test_no_probe_skips_both_rows_never_passes() -> None: + results = _run_checks_with_describe(_compliant_pool()) + rows = {r.check: r.status for r in results if r.check.startswith("hosted_ui_logout_")} + assert rows == { + "hosted_ui_logout_registered_redirects": "skip", + "hosted_ui_logout_unregistered_refused": "skip", + } + + +def test_the_live_probe_refuses_a_non_https_url() -> None: + with pytest.raises(ValueError, match="non-HTTPS"): + cognito_claims._live_logout_probe( + "http://crm-fixture.auth.us-east-1.amazoncognito.com/logout" + ) + + +def test_the_live_entry_point_supplies_the_logout_probe() -> None: + import inspect + + source = inspect.getsource(cognito_claims.main) + assert "logout_probe=_live_logout_probe" in source diff --git a/tests/unit/test_smoke_gateway_tools.py b/tests/unit/test_smoke_gateway_tools.py index a3b0d17..3bce366 100644 --- a/tests/unit/test_smoke_gateway_tools.py +++ b/tests/unit/test_smoke_gateway_tools.py @@ -1300,6 +1300,7 @@ def _record(_ctx: Any, **kwargs: Any) -> list[CheckResult]: assert isinstance(captured["provisioner"], LiveServicePrincipalProvisioner) assert isinstance(captured["user_provisioner"], LiveBareMfaUserProvisioner) assert callable(captured["expired_token_minter"]) + assert callable(captured["denial_log_reader"]) def test_the_underscoped_user_is_cleaned_up_even_when_its_provision_is_interrupted() -> None: @@ -1873,3 +1874,112 @@ def test_no_token_fails_the_array_body_row_without_sending_it() -> None: rows = _array_rows(raw, token=None) assert rows[_ARRAY_ROW].status == "fail" assert raw.calls == [] + + +# -------------------------------------------------------------------------------------- +# The interceptor's denial decisions are logged at WARNING. +# -------------------------------------------------------------------------------------- + +from scripts.smoke import gateway_tools as _gateway_tools # noqa: E402 + +_WARN_LINE = ( + '[WARNING]\t2026-11-08T00:00:00Z\treq-1\t{"event": "gateway_auth_denied", "reason": "x"}' +) +_INFO_LINE = '[INFO]\t2026-11-08T00:00:00Z\treq-1\t{"event": "gateway_auth_denied", "reason": "x"}' + + +class _Reader: + def __init__(self, pages: list[list[str]]) -> None: + self.pages = pages + self.starts: list[int] = [] + + def __call__(self, start_ms: int) -> list[str]: + self.starts.append(start_ms) + return self.pages[min(len(self.starts) - 1, len(self.pages) - 1)] + + +def _denial_row(reader: Any) -> CheckResult: + sleeps: list[float] = [] + return _gateway_tools._check_interceptor_denial_logged( + reader, start_ms=1_000, sleeper=sleeps.append + ) + + +def test_a_denial_at_warning_passes() -> None: + row = _denial_row(_Reader([[_WARN_LINE]])) + assert row.status == "pass" + + +def test_a_warning_that_appears_on_a_later_poll_passes() -> None: + reader = _Reader([[], [], [_WARN_LINE]]) + assert _denial_row(reader).status == "pass" + assert reader.starts == [1_000, 1_000, 1_000] + + +def test_denials_only_at_info_fail() -> None: + row = _denial_row(_Reader([[_INFO_LINE]])) + assert row.status == "fail" + assert "INFO" in row.detail + + +def test_a_json_format_warning_passes() -> None: + line = '{"level": "WARN", "message": "{\\"event\\": \\"gateway_service_identity_absent\\"}"}' + assert _denial_row(_Reader([[line]])).status == "pass" + + +def test_no_denial_line_defers_naming_the_unreached_path() -> None: + reader = _Reader([["[WARNING]\tunrelated line"]]) + row = _denial_row(reader) + assert row.status == "defer" + assert row.defer_reason == "interceptor_denial_not_reached" + assert len(reader.starts) == _gateway_tools.DENIAL_LOG_POLL_ATTEMPTS + + +def test_the_unreached_defer_states_the_observation_not_a_cause() -> None: + row = _denial_row(_Reader([["[WARNING]\tunrelated line"]])) + window = ( + _gateway_tools.DENIAL_LOG_POLL_ATTEMPTS - 1 + ) * _gateway_tools.DENIAL_LOG_POLL_INTERVAL_SECONDS + assert row.detail.startswith(f"no interceptor denial line observed within {window:g} s") + assert "expected to be refused before the interceptor" in row.detail + assert "were refused" not in row.detail + assert "was reached" not in row.detail + + +def test_an_unreadable_log_group_fails() -> None: + def _broken(start_ms: int) -> list[str]: + raise RuntimeError("AccessDenied") + + assert _denial_row(_broken).status == "fail" + + +def test_no_reader_skips_never_passes() -> None: + assert _denial_row(None).status == "skip" + + +def test_checks_emit_the_denial_row_after_the_probes() -> None: + reader = _Reader([[_WARN_LINE]]) + results = list( + checks( + context_for_test(outputs=_OUTPUTS), + mcp_caller=_FakeCaller(tools_list=_tools_list_result()), + token="tok", # nosec B106 # test placeholder, not a credential + denial_log_reader=reader, + ) + ) + assert results[-1].check == "interceptor_denial_logged_at_warning" + assert results[-1].status == "pass" + + +def test_the_live_entry_point_supplies_the_denial_reader() -> None: + import inspect + + assert "denial_log_reader=live_denial_log_reader(ctx)" in inspect.getsource(_gateway_tools.main) + + +def test_the_array_body_premise_says_the_gateway_refuses_it_itself() -> None: + doc = _gateway_tools.__doc__ or "" + assert "Gateway refuses an array body itself" in doc + assert "the Gateway refuses it itself" in ( + _gateway_tools._check_array_body_answers_no_tool_call.__doc__ or "" + ) diff --git a/tests/unit/test_smoke_harness.py b/tests/unit/test_smoke_harness.py index 078a3b9..a40a31e 100644 --- a/tests/unit/test_smoke_harness.py +++ b/tests/unit/test_smoke_harness.py @@ -2174,3 +2174,25 @@ def _raise() -> None: _raise() assert harness.is_timeout(raised.value) is True assert harness.is_timeout(StoreBusyError("busy")) is False + + +def test_is_timeout_does_not_follow_a_context_only_chain() -> None: + """A transport error that is only the implicit ``__context__`` is not a cause of the failure.""" + + class EndpointConnectionError(Exception): + """Named to match botocore's type, which is how `is_timeout` recognises it.""" + + class StoreBusyError(Exception): + """Stands in for an unrelated failure raised while handling the transport error.""" + + def _raise() -> None: + try: + raise EndpointConnectionError("unreachable") + except EndpointConnectionError: + raise StoreBusyError("busy") # noqa: B904 + + with pytest.raises(StoreBusyError) as raised: + _raise() + assert raised.value.__cause__ is None + assert isinstance(raised.value.__context__, EndpointConnectionError) + assert harness.is_timeout(raised.value) is False diff --git a/tests/unit/test_smoke_iam_roles.py b/tests/unit/test_smoke_iam_roles.py index c4573c9..84023b2 100644 --- a/tests/unit/test_smoke_iam_roles.py +++ b/tests/unit/test_smoke_iam_roles.py @@ -96,14 +96,16 @@ } -#: The real supplementary rows, kept before the autouse fixture below replaces them. +#: The real supplementary and allowlist-key rows, kept before the autouse fixture replaces them. _REAL_SUPPLEMENTARY_CHECKS: Final = iam_roles.supplementary_checks +_REAL_ALLOWLIST_KEY_CHECKS: Final = iam_roles.allowlist_key_checks @pytest.fixture(autouse=True) def _no_supplementary_rows(monkeypatch: pytest.MonkeyPatch) -> None: - """Matrix tests stub only the IAM client; the supplementary rows are tested on their own below.""" + """Matrix tests stub only the IAM client; the extra row sets are tested on their own below.""" monkeypatch.setattr(iam_roles, "supplementary_checks", lambda _ctx: iter(())) + monkeypatch.setattr(iam_roles, "allowlist_key_checks", lambda _ctx: iter(())) @pytest.fixture(autouse=True) @@ -1352,6 +1354,7 @@ def test_expected_check_count_is_derived_from_the_real_files_not_hardcoded( + conditioned_rows + roles * 4 + iam_roles.SUPPLEMENTARY_ROW_COUNT + + iam_roles.ALLOWLIST_KEY_ROW_COUNT ) assert str(roles) in expectation.basis and str(triples) in expectation.basis assert f"{scope_rows} index_scope_denied" in expectation.basis @@ -1388,6 +1391,11 @@ def test_a_complete_run_satisfies_its_own_floor(monkeypatch: pytest.MonkeyPatch) for n in range(iam_roles.SUPPLEMENTARY_ROW_COUNT) ] monkeypatch.setattr(iam_roles, "supplementary_checks", lambda _ctx: iter(supplementary)) + allowlist = [ + CheckResult(resource="a", check=check, status="pass", detail="") + for check in iam_roles.ALLOWLIST_KEY_CHECKS + ] + monkeypatch.setattr(iam_roles, "allowlist_key_checks", lambda _ctx: iter(allowlist)) clients = _StubbedClientSet() for role_function in _SORTED_FUNCTIONS: clients.add_role_for(role_function) @@ -2638,7 +2646,7 @@ def _own(self, iam: _ConditionedTableIam) -> Any: return _row(_run_with_arn_deciding_iam(iam), "create_account", self._OWN_ROW) def test_the_own_row_sends_the_in_scope_key(self, monkeypatch: pytest.MonkeyPatch) -> None: - key = "acct-eval-0001" # gitleaks:allow + key = "acct-eval-0001" # gitleaks:allow # nosemgrep: generic.secrets.gitleaks.generic-api-key monkeypatch.setattr( iam_roles, "conditioned_own_keys", @@ -4839,6 +4847,472 @@ def test_checks_appends_the_supplementary_rows_after_the_matrix( marker = CheckResult(resource="m", check="matrix_row", status="pass", detail="") extra = CheckResult(resource="s", check="supplementary_row", status="pass", detail="") monkeypatch.setattr(iam_roles, "_role_matrix_checks", lambda _ctx: iter([marker])) + allowlist = CheckResult(resource="a", check="allowlist_row", status="pass", detail="") monkeypatch.setattr(iam_roles, "supplementary_checks", lambda _ctx: iter([extra])) + monkeypatch.setattr(iam_roles, "allowlist_key_checks", lambda _ctx: iter([allowlist])) ctx = context_for_test(clients={}, outputs={}) - assert [r.check for r in iam_roles.checks(ctx)] == ["matrix_row", "supplementary_row"] + assert [r.check for r in iam_roles.checks(ctx)] == [ + "matrix_row", + "supplementary_row", + "allowlist_row", + ] + + +# --------------------------------------------------------------------------- +# allowlist_key_checks: the API client-allowlist SecureString and its CMK. +# --------------------------------------------------------------------------- + +_AK_KEY: Final = "arn:aws:kms:us-east-1:123456789012:key/aaaa1111-2222-3333-4444-555566667777" +_AK_MANAGED: Final = "arn:aws:kms:us-east-1:123456789012:key/bbbb1111-2222-3333-4444-555566667777" +_AK_PARAMETER: Final = "/crm-agent-eval/api/allowed-client-ids" +_AK_PARAMETER_ARN: Final = f"arn:aws:ssm:us-east-1:123456789012:parameter{_AK_PARAMETER}" +_AK_CONSUMERS: Final = { + "crm-agent-eval-search_accounts": _sup_role_arn("crm-agent-eval-search_accounts-role"), + "crm-agent-eval-eval-trigger": _sup_role_arn("crm-agent-eval-eval-trigger-role"), +} +_AK_OPERATOR: Final = _sup_role_arn("crm-agent-eval-operator") +_AK_ROLES: Final = ( + "crm-agent-eval-search_accounts-role", + "crm-agent-eval-eval-trigger-role", + "crm-agent-eval-sweep-task-role", + "crm-agent-eval-operator", + "unrelated-role", +) +_AK_TABLE_KEY: Final = "arn:aws:kms:us-east-1:123456789012:key/cccc1111-2222-3333-4444-555566667777" +_AK_CURSOR_KEY: Final = ( + "arn:aws:kms:us-east-1:123456789012:key/dddd1111-2222-3333-4444-555566667777" +) +_AK_TABLE: Final = "crm-agent-evalreps" + + +def _ak_policy(*, drop: str | None = None, use_deny_extra: Sequence[str] = ()) -> str: + principals = sorted([*_AK_CONSUMERS.values(), _AK_OPERATOR, *use_deny_extra]) + statements: list[dict[str, Any]] = [ + { + "Sid": "EnableIAMUserPermissions", + "Effect": "Allow", + "Principal": {"AWS": "arn:aws:iam::123456789012:root"}, + "Action": "kms:*", + "Resource": "*", + }, + { + "Sid": "AllowOperatorsUseThroughSsm", + "Effect": "Allow", + "Principal": {"AWS": _AK_OPERATOR}, + }, + { + "Sid": "AllowConsumersDecryptTheParameter", + "Effect": "Allow", + "Principal": {"AWS": sorted(_AK_CONSUMERS.values())}, + "Action": "kms:Decrypt", + "Resource": "*", + }, + {"Sid": "DenyKeyWriteToNonOperators", "Effect": "Deny", "Principal": "*"}, + { + "Sid": "DenyKeyUseToNonProjectPrincipals", + "Effect": "Deny", + "Principal": "*", + "Condition": {"ArnNotEquals": {"aws:PrincipalArn": principals}}, + }, + {"Sid": "DenyKeyAdministrationToNonOperators", "Effect": "Deny", "Principal": "*"}, + ] + return json.dumps( + {"Version": "2012-10-17", "Statement": [s for s in statements if s["Sid"] != drop]} + ) + + +_AK_POLICY: Final = _ak_policy() +_AK_TAGS: Final = { + "service": "crm-agent-eval", + "environment": "dev", + "owner": "team", + "cost-center": "crm-agent-eval", + "data-classification": "internal", + "AppName": "crm-agent-eval", +} + + +class _AkPaginator: + def __init__(self, pages: list[dict[str, Any]]) -> None: + self.pages = pages + + def paginate(self) -> Any: + return iter(self.pages) + + +class _AkIam: + """Simulates the healthy deployment unless `overrides` maps (role, action, resource) elsewhere.""" + + def __init__( + self, overrides: dict[tuple[str, str, str], str], *, context_blind: bool = False + ) -> None: + self.overrides = overrides + self.context_blind = context_blind + self.requests: list[dict[str, Any]] = [] + + def get_paginator(self, name: str) -> Any: + assert name == "list_roles" + return _AkPaginator( + [{"Roles": [{"RoleName": r, "Arn": _sup_role_arn(r)} for r in _AK_ROLES]}] + ) + + def simulate_principal_policy(self, **request: Any) -> dict[str, Any]: + self.requests.append(request) + role, action, resource = ( + request["PolicySourceArn"], + request["ActionNames"][0], + request["ResourceArns"][0], + ) + context = { + e["ContextKeyName"]: e["ContextKeyValues"][0] for e in request.get("ContextEntries", []) + } + decision = "implicitDeny" + if (action == "kms:Encrypt" and "ResourcePolicy" in request) or ( + action == "kms:Decrypt" and resource == _AK_MANAGED + ): + decision = "explicitDeny" + elif ( + ( + action == "kms:Decrypt" + and resource == _AK_KEY + and role in _AK_CONSUMERS.values() + and ( + self.context_blind + or ( + context.get("kms:EncryptionContext:PARAMETER_ARN") == _AK_PARAMETER_ARN + and context.get("kms:ViaService") == "ssm.us-east-1.amazonaws.com" + ) + ) + ) + or ( + resource == _AK_TABLE_KEY + and action in {"kms:Decrypt", "kms:GenerateDataKey"} + and context.get("kms:ViaService") == "dynamodb.us-east-1.amazonaws.com" + ) + or ( + resource == _AK_CURSOR_KEY + and action in {"kms:Encrypt", "kms:Decrypt"} + and context.get("kms:EncryptionContext:purpose") == "crm-cursor" + ) + ): + decision = "allowed" + decision = self.overrides.get((role, action, resource), decision) + return {"EvaluationResults": [{"EvalDecision": decision, "MissingContextValues": []}]} + + +class _AkLambda: + def __init__(self, names: dict[str, str]) -> None: + self.names = names + + def get_paginator(self, name: str) -> Any: + assert name == "list_functions" + functions = [ + { + "FunctionName": fn, + "Role": role, + "Environment": { + "Variables": { + iam_roles.ALLOWLIST_PARAMETER_ENV: self.names.get(fn, _AK_PARAMETER), + iam_roles.TABLE_PREFIX_ENV: "crm-agent-eval", + **( + {iam_roles.CURSOR_KEY_ENV: _AK_CURSOR_KEY} + if fn.endswith("search_accounts") + else {} + ), + } + }, + } + for fn, role in _AK_CONSUMERS.items() + ] + functions.append({"FunctionName": "crm-agent-eval-get_page_context", "Role": "r"}) + return _AkPaginator([{"Functions": functions}]) + + +class _AkSsm: + def __init__(self, kind: str, key_id: str | None, value: str, tier: str = "Standard") -> None: + self.kind, self.key_id, self.value, self.tier = kind, key_id, value, tier + self.reads: list[dict[str, Any]] = [] + + def describe_parameters(self, ParameterFilters: Any) -> dict[str, Any]: # noqa: N803 + assert ParameterFilters[0]["Values"] == [_AK_PARAMETER] + meta: dict[str, Any] = {"Name": _AK_PARAMETER, "Type": self.kind, "Tier": self.tier} + if self.key_id is not None: + meta["KeyId"] = self.key_id + return {"Parameters": [meta]} + + def get_parameter(self, Name: str, WithDecryption: bool) -> dict[str, Any]: # noqa: N803 + self.reads.append({"Name": Name, "WithDecryption": WithDecryption}) + return {"Parameter": {"Type": self.kind, "Value": self.value}} + + +class _AkKms: + def __init__(self, rotation: bool, tags: dict[str, str], policy: str, managed: bool) -> None: + self.rotation, self.tags, self.policy, self.managed = rotation, tags, policy, managed + + def describe_key(self, KeyId: str) -> dict[str, Any]: # noqa: N803 + if KeyId == iam_roles.AWS_SSM_KEY_ALIAS: + if not self.managed: + raise RuntimeError("NotFoundException") + return {"KeyMetadata": {"Arn": _AK_MANAGED}} + if KeyId in {_AK_KEY, "alias/crm-agent-eval-api-allowlist"}: + return {"KeyMetadata": {"Arn": _AK_KEY}} + return {"KeyMetadata": {"Arn": KeyId}} + + def get_key_rotation_status(self, KeyId: str) -> dict[str, Any]: # noqa: N803 + return {"KeyRotationEnabled": self.rotation} + + def list_resource_tags(self, KeyId: str) -> dict[str, Any]: # noqa: N803 + return {"Tags": [{"TagKey": k, "TagValue": v} for k, v in self.tags.items()]} + + def get_key_policy(self, KeyId: str, PolicyName: str) -> dict[str, Any]: # noqa: N803 + assert (KeyId, PolicyName) == (_AK_KEY, "default") + return {"Policy": self.policy} + + +class _AkDynamo: + def __init__(self, key: str | None) -> None: + self.key = key + + def describe_table(self, TableName: str) -> dict[str, Any]: # noqa: N803 + assert TableName == _AK_TABLE + sse = {"KMSMasterKeyArn": self.key} if self.key else {} + return {"Table": {"TableName": TableName, "SSEDescription": sse}} + + +def _ak_run( + *, + overrides: dict[tuple[str, str, str], str] | None = None, + context_blind: bool = False, + tier: str = "Standard", + table_key: str | None = _AK_TABLE_KEY, + kind: str = "SecureString", + key_id: str | None = "alias/crm-agent-eval-api-allowlist", + value: str = "client-a", + rotation: bool = True, + tags: dict[str, str] | None = None, + policy: str = _AK_POLICY, + managed: bool = True, + names: dict[str, str] | None = None, + outputs: dict[str, Any] | None = None, +) -> tuple[dict[str, Any], _AkIam, _AkSsm]: + iam = _AkIam(overrides or {}, context_blind=context_blind) + ssm = _AkSsm(kind, key_id, value, tier) + ctx = context_for_test( + outputs={ + iam_roles.ALLOWLIST_KEY_OUTPUT: _AK_KEY, + "crm_operator_principal_arns": [_AK_OPERATOR], + "dynamodb_table_names": {"reps": _AK_TABLE}, + } + if outputs is None + else outputs, + clients={ + "iam": iam, + "sts": _SupSts(), + "lambda": _AkLambda(names or {}), + "ssm": ssm, + "kms": _AkKms(rotation, _AK_TAGS if tags is None else tags, policy, managed), + "dynamodb": _AkDynamo(table_key), + }, + ) + results = list(_REAL_ALLOWLIST_KEY_CHECKS(ctx)) + assert [r.check for r in results] == list(iam_roles.ALLOWLIST_KEY_CHECKS) + return {r.check: r for r in results}, iam, ssm + + +def _ak_not_passing(rows: dict[str, Any]) -> set[str]: + return {name for name, row in rows.items() if row.status != "pass"} + + +def test_allowlist_rows_all_pass_on_a_healthy_deployment() -> None: + rows, _, _ = _ak_run() + assert _ak_not_passing(rows) == set(), {n: r.detail for n, r in rows.items()} + + +def test_the_parameter_is_read_with_decryption() -> None: + _, _, ssm = _ak_run() + assert ssm.reads == [{"Name": _AK_PARAMETER, "WithDecryption": True}] + + +def test_the_kms_simulations_are_identity_only_and_send_the_ssm_context() -> None: + """The simulator rejects a key policy (`Resource: *`) as `ResourcePolicy`, so none is sent.""" + _, iam, _ = _ak_run() + assert all("ResourcePolicy" not in r for r in iam.requests) + kms = [ + r + for r in iam.requests + if r["ResourceArns"] == [_AK_KEY] + and {e["ContextKeyName"] for e in r.get("ContextEntries", [])} >= {"aws:PrincipalArn"} + ] + assert {r["ActionNames"][0] for r in kms} == {"kms:Encrypt", "kms:Decrypt"} + for request in kms: + context = {e["ContextKeyName"]: e["ContextKeyValues"] for e in request["ContextEntries"]} + assert context["kms:ViaService"] == ["ssm.us-east-1.amazonaws.com"] + assert context["aws:PrincipalArn"] == [request["PolicySourceArn"]] + + +class _RejectsResourcePolicy(_AkIam): + def simulate_principal_policy(self, **request: Any) -> dict[str, Any]: + if "ResourcePolicy" in request: + raise RuntimeError("InvalidInput: ResourceNames are not in a valid ARN format: *.") + return super().simulate_principal_policy(**request) + + +def test_rows_pass_against_a_simulator_that_rejects_a_key_policy_as_resource_policy() -> None: + iam = _RejectsResourcePolicy({}) + ctx = context_for_test( + outputs={ + iam_roles.ALLOWLIST_KEY_OUTPUT: _AK_KEY, + "crm_operator_principal_arns": [_AK_OPERATOR], + "dynamodb_table_names": {"reps": _AK_TABLE}, + }, + clients={ + "iam": iam, + "sts": _SupSts(), + "lambda": _AkLambda({}), + "ssm": _AkSsm("SecureString", "alias/crm-agent-eval-api-allowlist", "client-a"), + "kms": _AkKms(True, _AK_TAGS, _AK_POLICY, True), + "dynamodb": _AkDynamo(_AK_TABLE_KEY), + }, + ) + rows = {r.check: r for r in _REAL_ALLOWLIST_KEY_CHECKS(ctx)} + assert _ak_not_passing(rows) == set(), {n: r.detail for n, r in rows.items()} + + +def test_the_operators_and_unrelated_roles_are_not_simulated_as_project_roles() -> None: + _, iam, _ = _ak_run() + simulated = {r["PolicySourceArn"] for r in iam.requests} + assert _AK_OPERATOR not in simulated + assert _sup_role_arn("unrelated-role") not in simulated + assert _sup_role_arn("crm-agent-eval-sweep-task-role") in simulated + + +_AK_SWEEP: Final = _sup_role_arn("crm-agent-eval-sweep-task-role") +_AK_SEARCH: Final = _AK_CONSUMERS["crm-agent-eval-search_accounts"] + + +@pytest.mark.parametrize( + ("kwargs", "expected"), + [ + ( + {"kind": "String", "key_id": None}, + { + "allowlist_parameter_securestring_under_cmk", + "allowlist_parameter_read_with_decryption", + }, + ), + ( + {"key_id": "alias/aws/ssm"}, + {"allowlist_parameter_securestring_under_cmk"}, + ), + ({"value": ""}, {"allowlist_parameter_read_with_decryption"}), + ({"rotation": False}, {"allowlist_key_rotation_enabled"}), + ( + {"tags": {k: v for k, v in _AK_TAGS.items() if k != "data-classification"}}, + {"allowlist_key_tags"}, + ), + ({"tags": {**_AK_TAGS, "data-classification": "public"}}, {"allowlist_key_tags"}), + ( + { + "policy": json.dumps( + {"Statement": [{"Sid": "Open", "Effect": "Allow", "Principal": {"AWS": "*"}}]} + ) + }, + {"allowlist_key_policy_no_wildcard_allow", "allowlist_key_policy_statements"}, + ), + ({"tier": "Advanced"}, {"allowlist_parameter_securestring_under_cmk"}), + ( + {"policy": _ak_policy(drop="DenyKeyWriteToNonOperators")}, + {"allowlist_key_policy_statements"}, + ), + ( + {"policy": _ak_policy(use_deny_extra=[_AK_SWEEP])}, + {"allowlist_key_policy_statements"}, + ), + ( + {"overrides": {(_AK_SEARCH, "kms:GenerateDataKey", _AK_KEY): "allowed"}}, + {"allowlist_consumer_other_key_operations_denied"}, + ), + ( + {"overrides": {(_AK_SEARCH, "kms:GenerateDataKey", _AK_TABLE_KEY): "explicitDeny"}}, + {"allowlist_consumer_dynamodb_key_unaffected"}, + ), + ({"table_key": None}, {"allowlist_consumer_dynamodb_key_unaffected"}), + ( + {"overrides": {(_AK_SEARCH, "kms:Decrypt", _AK_CURSOR_KEY): "explicitDeny"}}, + {"allowlist_cursor_key_unaffected"}, + ), + ( + {"overrides": {(_AK_SWEEP, "ssm:PutParameter", _AK_PARAMETER_ARN): "allowed"}}, + {"allowlist_parameter_write_denied:project_roles"}, + ), + ( + {"overrides": {(_AK_SWEEP, "ssm:DeleteParameter", _AK_PARAMETER_ARN): "allowed"}}, + {"allowlist_parameter_write_denied:project_roles"}, + ), + ( + { + "overrides": { + (_AK_SEARCH, "ssm:LabelParameterVersion", _AK_PARAMETER_ARN): "allowed" + } + }, + {"allowlist_parameter_write_denied:project_roles"}, + ), + ( + {"overrides": {(_AK_SEARCH, "kms:Encrypt", _AK_KEY): "allowed"}}, + { + "allowlist_key_encrypt_denied:project_roles", + "allowlist_consumer_other_key_operations_denied", + }, + ), + ( + {"overrides": {(_AK_SEARCH, "kms:Decrypt", _AK_KEY): "implicitDeny"}}, + { + "allowlist_consumer_decrypt_allowed", + "allowlist_consumer_decrypt_needs_parameter_context", + }, + ), + ( + {"overrides": {(_AK_SEARCH, "kms:Decrypt", _AK_MANAGED): "implicitDeny"}}, + {"allowlist_consumer_aws_ssm_decrypt_denied"}, + ), + ], +) +def test_each_defect_fails_exactly_its_own_row(kwargs: dict[str, Any], expected: set[str]) -> None: + rows, _, _ = _ak_run(**kwargs) + assert _ak_not_passing(rows) == expected, {n: r.detail for n, r in rows.items()} + + +def test_a_context_blind_decrypt_fails_the_other_parameter_row_and_the_control() -> None: + rows, _, _ = _ak_run(context_blind=True) + assert _ak_not_passing(rows) == { + "allowlist_consumer_decrypt_other_parameter_denied", + "allowlist_consumer_decrypt_needs_parameter_context", + } + + +def test_an_absent_managed_ssm_key_defers_with_a_reason_never_passes() -> None: + rows, _, _ = _ak_run(managed=False) + row = rows["allowlist_consumer_aws_ssm_decrypt_denied"] + assert row.status == "defer" + assert row.defer_reason == "aws_managed_ssm_key_absent" + assert iam_roles.AWS_SSM_KEY_ALIAS in row.detail + assert _ak_not_passing(rows) == {"allowlist_consumer_aws_ssm_decrypt_denied"} + + +def test_a_missing_key_output_fails_every_row() -> None: + rows, _, _ = _ak_run(outputs={}) + assert {r.status for r in rows.values()} == {"fail"} + assert iam_roles.ALLOWLIST_KEY_OUTPUT in rows["allowlist_key_tags"].detail + + +def test_disagreeing_parameter_names_fail_every_row() -> None: + rows, _, _ = _ak_run(names={"crm-agent-eval-eval-trigger": "/other/name"}) + assert {r.status for r in rows.values()} == {"fail"} + + +def test_the_key_output_is_declared_in_infra() -> None: + root = Path(__file__).resolve().parents[2] + outputs = (root / "infra" / "outputs.tf").read_text(encoding="utf-8") + assert f'output "{iam_roles.ALLOWLIST_KEY_OUTPUT}"' in outputs + source = (root / "scripts" / "smoke" / "iam_roles.py").read_text(encoding="utf-8") + assert f'require_str("{iam_roles.ALLOWLIST_KEY_OUTPUT}")' in source diff --git a/tests/unit/test_smoke_lambda_actions.py b/tests/unit/test_smoke_lambda_actions.py index e7a38f6..10f3f54 100644 --- a/tests/unit/test_smoke_lambda_actions.py +++ b/tests/unit/test_smoke_lambda_actions.py @@ -22,6 +22,8 @@ _PRIMARY_CONTACT_CHECKS, _PRIMARY_CONTACT_TOOL, _T3_PENDING_TOOLS, + AGENT_PRINCIPAL_TYPE_ABSENT_CHECK, + AGENT_PRINCIPAL_TYPE_DECLARED_CHECK, CONTACT_DEPARTED_MISSING_SCOPE, CURSOR_ENV_CHECK, CURSOR_KEY_ARN_CHECK, @@ -41,6 +43,7 @@ EMPTY_BOOK_MISSING_SCOPE, EXPLICIT_FILTER_CHECK, FUNCTION_NAME_PREFIX, + GATEWAY_PRINCIPAL_TYPE_ABSENT_CHECK, KMS_ADMIN_DENIED_CHECK, KMS_NO_CONTEXT_CHECK, KMS_OTHER_ACTIONS_CHECK, @@ -52,6 +55,7 @@ MALFORMED_PAGE_CONTEXT_CHECK, NO_SECOND_PAGE_DEFER_REASON, NON_CURSOR_PROBE_TOOL, + PRINCIPAL_TYPE_PROBE_TOOL, SMOKE_PHONE, _check_explicit_filter_denied, _check_foreign_primary_contact_refused, @@ -70,6 +74,7 @@ out_of_scope_event, page_context_tools, primary_contact_event, + principal_type_events, seeded_active_contact_id, seeded_departed_contact_id, seeded_in_account_contact_id, @@ -81,6 +86,8 @@ from scripts.smoke.tenant import SmokeTenant, TenantTeardownError from crm.cursor import CURSOR_KMS_KEY_ENV, CURSOR_PURPOSE +from crm.envelope import AGENT_IDENTITY_KEY, IDENTITY_KEY, decode_identity +from crm.errors import AuthenticationFailed from crm.toolspec.definitions import ALL_TOOLS, LONG_TEXT_MAX_LENGTH, ToolDefinition _TEST_TENANT = "eval-smoke-testfixedrunid" @@ -273,6 +280,13 @@ def _authentication_failed_body() -> dict[str, Any]: } +def _principal_type_refusal() -> dict[str, Any]: + return { + "statusCode": 401, + "error": {"code": "authentication_failed", "message": "Authentication is required."}, + } + + def _internal_error_body() -> dict[str, Any]: return {"statusCode": 500, "error": {"code": "internal_error", "message": "x"}} @@ -1499,6 +1513,9 @@ def __init__( refusal_body: dict[str, Any] | None = None, env_overrides: dict[str, str | None] | None = None, long_text_body: dict[str, Any] | None = None, + accept_absent_principal_type: bool = False, + refuse_agent_channel: bool = False, + absent_principal_type_body: dict[str, Any] | None = None, ) -> None: self.issuing = set(issuing) self.page_two_error = set(page_two_error) @@ -1508,6 +1525,9 @@ def __init__( self.refusal_body = refusal_body or _cursor_refusal_body() self.env_overrides = env_overrides or {} self.long_text_body = long_text_body or _validation_failed_body() + self.accept_absent_principal_type = accept_absent_principal_type + self.refuse_agent_channel = refuse_agent_channel + self.absent_principal_type_body = absent_principal_type_body or _principal_type_refusal() self.events: list[tuple[str, dict[str, Any]]] = [] def invoke(self, **kwargs: Any) -> dict[str, Any]: @@ -1517,6 +1537,15 @@ def invoke(self, **kwargs: Any) -> dict[str, Any]: return _payload_response(self._answer(tool, event)) def _answer(self, tool: str, event: dict[str, Any]) -> dict[str, Any]: + channel = event.get(IDENTITY_KEY) or event.get(AGENT_IDENTITY_KEY) + if ( + isinstance(channel, dict) + and "principal_type" not in channel + and not self.accept_absent_principal_type + ): + return self.absent_principal_type_body + if AGENT_IDENTITY_KEY in event and self.refuse_agent_channel: + return _principal_type_refusal() if tool == "create_note": return self.long_text_body cursor = event.get("cursor") @@ -1693,10 +1722,14 @@ def test_every_extra_row_passes_against_the_intended_deployment() -> None: CURSOR_KEY_SHAPE_CHECK, CURSOR_KEY_TAG_CHECK, } == checks_by_resource[CURSOR_KEY_RESOURCE] + assert PRINCIPAL_TYPE_PROBE_TOOL == NON_CURSOR_PROBE_TOOL assert { CURSOR_ENV_CHECK, KMS_OUTSIDE_IDENTITY_CHECK, KMS_OUTSIDE_KEY_POLICY_CHECK, + GATEWAY_PRINCIPAL_TYPE_ABSENT_CHECK, + AGENT_PRINCIPAL_TYPE_ABSENT_CHECK, + AGENT_PRINCIPAL_TYPE_DECLARED_CHECK, } == checks_by_resource[NON_CURSOR_PROBE_TOOL] assert LONG_TEXT_CAP_CHECK in checks_by_resource["create_note"] every_tool = {tool.name for tool in ALL_TOOLS} @@ -1981,3 +2014,86 @@ def recorder(ctx: Any, *, tenant: str) -> Any: stubber.deactivate() assert seen == [_TEST_TENANT] assert (results[-1].resource, results[-1].check) == ("sentinel", "extra") + + +# --------------------------------------------------------------------------- +# Absent principal_type is refused on the gateway and agent channels +# --------------------------------------------------------------------------- + +_PRINCIPAL_TYPE_CHECKS = ( + GATEWAY_PRINCIPAL_TYPE_ABSENT_CHECK, + AGENT_PRINCIPAL_TYPE_ABSENT_CHECK, + AGENT_PRINCIPAL_TYPE_DECLARED_CHECK, +) + + +def _principal_type_rows(lam: _CursorLambda) -> dict[str, Any]: + rows = _rows(_extra_ctx(lam=lam)) + return {r.check: r for r in rows if r.check in _PRINCIPAL_TYPE_CHECKS} + + +def test_the_principal_type_rows_invoke_the_probe_tool_once_each_on_one_channel() -> None: + lam = _CursorLambda() + rows = _principal_type_rows(lam) + assert {check: row.status for check, row in rows.items()} == dict.fromkeys( + _PRINCIPAL_TYPE_CHECKS, "pass" + ) + sent = [event for tool, event in lam.events if tool == PRINCIPAL_TYPE_PROBE_TOOL] + assert len(sent) == 3 + for event in sent: + assert (IDENTITY_KEY in event) != (AGENT_IDENTITY_KEY in event) + + +def test_the_absent_principal_type_events_are_refused_by_the_real_decoder() -> None: + events = principal_type_events(tenant=_TEST_TENANT) + assert set(events) == set(_PRINCIPAL_TYPE_CHECKS) + for check in (GATEWAY_PRINCIPAL_TYPE_ABSENT_CHECK, AGENT_PRINCIPAL_TYPE_ABSENT_CHECK): + with pytest.raises(AuthenticationFailed): + decode_identity(dict(events[check])) + identity = decode_identity(dict(events[AGENT_PRINCIPAL_TYPE_DECLARED_CHECK])) + assert identity.principal_type == "user" + assert identity.source == "agent" + assert identity.tenant == _TEST_TENANT + + +def test_the_absent_principal_type_events_differ_from_a_decodable_one_only_in_that_key() -> None: + events = principal_type_events(tenant=_TEST_TENANT) + gateway = dict(events[GATEWAY_PRINCIPAL_TYPE_ABSENT_CHECK][IDENTITY_KEY]) + decode_identity({IDENTITY_KEY: {**gateway, "principal_type": "user"}}) + absent_agent = dict(events[AGENT_PRINCIPAL_TYPE_ABSENT_CHECK][AGENT_IDENTITY_KEY]) + declared_agent = dict(events[AGENT_PRINCIPAL_TYPE_DECLARED_CHECK][AGENT_IDENTITY_KEY]) + assert {**absent_agent, "principal_type": "user"} == declared_agent + + +def test_a_decoder_that_defaults_an_absent_principal_type_fails_both_refusal_rows() -> None: + rows = _principal_type_rows(_CursorLambda(accept_absent_principal_type=True)) + assert rows[GATEWAY_PRINCIPAL_TYPE_ABSENT_CHECK].status == "fail" + assert rows[AGENT_PRINCIPAL_TYPE_ABSENT_CHECK].status == "fail" + assert rows[AGENT_PRINCIPAL_TYPE_DECLARED_CHECK].status == "pass" + + +@pytest.mark.parametrize( + "body", + [ + _scope_denied_body(), + _validation_failed_body(), + _internal_error_body(), + { + "statusCode": 401, + "error": {"code": "authentication_failed", "message": "x", "detail": "leaked"}, + }, + ], + ids=["scope_denied", "validation_failed", "internal_error", "leaky_auth_failed"], +) +def test_a_refusal_other_than_a_clean_authentication_failed_fails_the_rows( + body: dict[str, Any], +) -> None: + rows = _principal_type_rows(_CursorLambda(absent_principal_type_body=body)) + assert rows[GATEWAY_PRINCIPAL_TYPE_ABSENT_CHECK].status == "fail" + assert rows[AGENT_PRINCIPAL_TYPE_ABSENT_CHECK].status == "fail" + + +def test_a_refused_declared_agent_identity_fails_the_control_row() -> None: + rows = _principal_type_rows(_CursorLambda(refuse_agent_channel=True)) + assert rows[AGENT_PRINCIPAL_TYPE_DECLARED_CHECK].status == "fail" + assert rows[AGENT_PRINCIPAL_TYPE_ABSENT_CHECK].status == "pass" diff --git a/tests/unit/test_smoke_mcp_client_compat.py b/tests/unit/test_smoke_mcp_client_compat.py index 4bd0bfd..5080976 100644 --- a/tests/unit/test_smoke_mcp_client_compat.py +++ b/tests/unit/test_smoke_mcp_client_compat.py @@ -837,3 +837,47 @@ def test_the_cli_fails_loudly_on_missing_outputs(tmp_path: Path) -> None: payload = json.loads(result.stdout) assert payload["resource_class"] == module.RESOURCE_CLASS assert [c["status"] for c in payload["checks"]] == ["fail"] + + +class HTTPStatusError(Exception): + """Named to match the HTTP client's status error, which is how the walk recognises it.""" + + def __init__(self, status_code: int) -> None: + super().__init__(f"status {status_code}") + self.response = type("_Response", (), {"status_code": status_code})() + + +def _raised_from(outer: Exception, inner: BaseException, *, explicit: bool) -> Exception: + try: + try: + raise inner + except BaseException as exc: + if explicit: + raise outer from exc + raise outer # noqa: B904 + except Exception as raised: + return raised + + +def test_http_status_of_follows_an_explicit_cause() -> None: + assert ( + module._http_status_of(_raised_from(RuntimeError("x"), HTTPStatusError(403), explicit=True)) + == 403 + ) + + +def test_http_status_of_traverses_exception_group_members_and_their_causes() -> None: + member = _raised_from(RuntimeError("member"), HTTPStatusError(401), explicit=True) + group = ExceptionGroup("task group", [ValueError("other"), member]) + assert module._http_status_of(group) == 401 + assert module._http_status_of(ExceptionGroup("g", [HTTPStatusError(429)])) == 429 + + +def test_http_status_of_does_not_follow_a_context_only_chain() -> None: + """A status error that is only the implicit ``__context__`` is not the failure's cause.""" + raised = _raised_from(RuntimeError("x"), HTTPStatusError(403), explicit=False) + assert raised.__cause__ is None + assert isinstance(raised.__context__, HTTPStatusError) + assert module._http_status_of(raised) is None + group = ExceptionGroup("task group", [raised]) + assert module._http_status_of(group) is None diff --git a/tests/unit/test_smoke_three_tier_chat.py b/tests/unit/test_smoke_three_tier_chat.py index 7eeeef0..ab667cd 100644 --- a/tests/unit/test_smoke_three_tier_chat.py +++ b/tests/unit/test_smoke_three_tier_chat.py @@ -359,6 +359,7 @@ def _priced_frame(tuning: Mapping[str, Any]) -> dict[str, Any]: "cost_usd": 0.01, "cache_read_tokens": 1024, "cache_write_tokens": 0, + "long_context_calls": 0, } for tier, model in models.items() ], @@ -511,6 +512,7 @@ def test_checks_happy_path_all_pass_and_both_users_cleaned_up() -> None: assert by_check["owner_session_reuse_control"] == "pass" assert by_check["foreign_session_reuse_refused"] == "pass" assert by_check["cost_lines_present"] == "pass" + assert by_check["cost_lines_carry_long_context_calls"] == "pass" assert by_check["cost_usd_positive"] == "pass" assert by_check["resolved_tiers_match_tuning"] == "pass" for tier, model in expected_tier_models(MIXED_FAMILIES).items(): @@ -520,7 +522,7 @@ def test_checks_happy_path_all_pass_and_both_users_cleaned_up() -> None: attribution_rows = {c: s for c, s in by_check.items() if c.endswith(ATTRIBUTION_ROW_SUFFIX)} families = select_attribution_mapping().families assert families is not None - assert len(attribution_rows) == 3 + 2 * len(families) + assert len(attribution_rows) == 4 + 2 * len(families) assert set(attribution_rows.values()) == {"pass"} assert by_check["stale_temperature_tuning_refused"] == "pass" assert by_check["model_backed_turns_within_envelope"] == "pass" @@ -650,8 +652,14 @@ def test_a_priced_mixed_frame_passes_every_cost_row() -> None: @pytest.mark.parametrize( ("override", "red"), [ - ({"cost_lines": []}, ["cost_lines_present", *_MODEL_ROWS]), - ({"cost_lines": None}, ["cost_lines_present", *_MODEL_ROWS]), + ( + {"cost_lines": []}, + ["cost_lines_present", "cost_lines_carry_long_context_calls", *_MODEL_ROWS], + ), + ( + {"cost_lines": None}, + ["cost_lines_present", "cost_lines_carry_long_context_calls", *_MODEL_ROWS], + ), ({"cost_usd": 0}, ["cost_usd_positive"]), ({"cost_usd": None, "cost_withheld": True}, ["cost_usd_positive"]), ({"cost_withheld": True}, ["cost_usd_positive"]), @@ -680,6 +688,32 @@ def test_a_tier_priced_on_another_model_reds_only_its_model_row(tier: str) -> No assert failed == [f"{tier}_cost_line_names_{model}"] +@pytest.mark.parametrize("value", [None, -1, True, "2", 1.0]) +def test_a_line_without_a_valid_long_context_count_reds_only_that_row(value: Any) -> None: + frame = _priced_frame(tuning_block(MIXED_FAMILIES)) + frame["cost_lines"] = [ + {**line, "long_context_calls": value} if line["tier"] == "tool" else line + for line in frame["cost_lines"] + ] + failed = [check for check, status in _cost_rows(frame).items() if status == "fail"] + assert failed == ["cost_lines_carry_long_context_calls"] + + +def test_a_line_missing_the_long_context_key_reds_only_that_row() -> None: + frame = _priced_frame(tuning_block(MIXED_FAMILIES)) + frame["cost_lines"] = [ + {k: v for k, v in line.items() if k != "long_context_calls"} for line in frame["cost_lines"] + ] + failed = [check for check, status in _cost_rows(frame).items() if status == "fail"] + assert failed == ["cost_lines_carry_long_context_calls"] + + +def test_a_positive_long_context_count_passes() -> None: + frame = _priced_frame(tuning_block(MIXED_FAMILIES)) + frame["cost_lines"] = [{**line, "long_context_calls": 3} for line in frame["cost_lines"]] + assert _cost_rows(frame)["cost_lines_carry_long_context_calls"] == "pass" + + def test_an_unpriced_line_reds_its_model_row() -> None: frame = _priced_frame(tuning_block(MIXED_FAMILIES)) frame["cost_lines"] = [ diff --git a/tests/unit/test_store_transport_errors.py b/tests/unit/test_store_transport_errors.py index 26a3d31..4dbcd7f 100644 --- a/tests/unit/test_store_transport_errors.py +++ b/tests/unit/test_store_transport_errors.py @@ -32,7 +32,7 @@ pytestmark = pytest.mark.filterwarnings("ignore::DeprecationWarning") _ENDPOINT = "https://dynamodb.us-east-1.amazonaws.com" -_SECRET_TEXT = "internal-endpoint-detail" +_SECRET_TEXT = "internal-endpoint-detail" # nosec B105 # a redaction sentinel, not a credential _TENANT = "demo" _ACCOUNT = "acct-1" diff --git a/tests/unit/test_stream_contract.py b/tests/unit/test_stream_contract.py index b805afa..9f72479 100644 --- a/tests/unit/test_stream_contract.py +++ b/tests/unit/test_stream_contract.py @@ -419,3 +419,17 @@ def test_registry_size_tracks_all_tools_when_a_definition_is_added( patched_registry = registry_module.full_registry() assert len(patched_registry) == len(patched_tools) + + +def test_a_refused_request_ends_the_stream_with_the_budget_exhausted_terminal() -> None: + """A request refused for its prompt size is reported like any other exhausted dimension.""" + budget = TurnBudget(max_tool_calls=EVAL_MAX_TOOL_CALLS) + budget.record_tool_call("called get_account") + budget.refuse_request( + estimated_prompt_tokens=95_001, limit=95_000, tier="tool", role="research" + ) + event = budget.exhausted_event() + assert event.kind == "error" + assert event.error_kind == "budget_exhausted" + assert event.partial_work == "called get_account" + assert "request_prompt_limit: estimated 95001 > 95000" in (event.error_message or "") diff --git a/tests/unit/test_sweep_units.py b/tests/unit/test_sweep_units.py index fe6fe1b..2f89995 100644 --- a/tests/unit/test_sweep_units.py +++ b/tests/unit/test_sweep_units.py @@ -28,6 +28,7 @@ run_task_unit, task_actor, task_refs, + usage_records_for_turn, ) from eval.turn_observation import ObservedTurn from providers.base import ModelProvider, ProviderEvent, ResolvedInvocation @@ -229,8 +230,8 @@ def runner(runner_task: Mapping[str, object], tenant: str) -> ObservedTurn: assert unit.task_id == task["id"] assert unit.task_score.task_id == task["id"] assert unit.tier_scores.task_id == task["id"] - assert len(unit.usage_records) == 1 - assert unit.usage_records[0].usage.input_tokens == 20 + assert len(unit.usage_records) == 2 + assert [record.usage.input_tokens for record in unit.usage_records] == [10, 10] def test_a_task_without_expected_tiers_is_not_applicable_at_every_tier(store: Store) -> None: @@ -278,7 +279,7 @@ def test_assemble_arm_without_an_arm_counter_checks_against_the_units(store: Sto invocations=_turn(stream_calls=2).invocations, stream_calls=0, ) - with pytest.raises(AssertionError, match="included_invocation_count"): + with pytest.raises(AssertionError, match="double-recorded usage"): _assemble([_unit(store, _golden(1)[0], overcounted)]) @@ -309,3 +310,117 @@ def test_assemble_arm_takes_latency_only_from_live_turns(store: Store) -> None: assert _assemble(units, mode="stub").model_metrics.latency.measured is False live = _assemble(units, mode="live").model_metrics.latency assert live.measured is True and live.task_p50_ms == pytest.approx(900.0) + + +# --- per-call pricing -------------------------------------------------------------------------- + + +def _haiku_call(input_tokens: int) -> InvocationAttribution: + return InvocationAttribution( + tier="tool", + role="research", + model_id=MATRIX.model_for("anthropic", "tool").model_id, + tokens=TokenCounts(input_tokens=input_tokens, output_tokens=5), + cache=CacheCounts(), + ) + + +def test_usage_records_for_turn_prices_each_attribution_on_its_own() -> None: + """Two 60K calls stay standard; summed they would cross Haiku's 100K threshold.""" + records = usage_records_for_turn( + (_haiku_call(60_000), _haiku_call(60_000)), + task_id="t-1", + rep_id="rep-1", + family="anthropic", + ) + assert [r.usage.input_tokens for r in records] == [60_000, 60_000] + assert [r.cost.pricing_tier for r in records] == ["standard", "standard"] + assert {r.cost.route for r in records} == {"global"} + assert len({r.invocation_id for r in records}) == 2 + + +def test_usage_records_for_turn_prices_under_the_configured_profile( + monkeypatch: pytest.MonkeyPatch, +) -> None: + """The record's route follows the model's configured inference profile.""" + monkeypatch.setattr(sweep_units, "inference_profile_for", lambda model_id: f"eu.{model_id}") + (record,) = usage_records_for_turn( + (_haiku_call(10_000),), task_id="t-1", rep_id="rep-1", family="anthropic" + ) + assert record.cost.route == "geo" + assert record.cost.input == pytest.approx(10_000 / 1e6 * 0.10 * 1.10) + + +def test_assemble_arm_reports_long_context_counts_per_tier_and_row(store: Store) -> None: + """The tool tier's summary and its row carry the count of long-context calls.""" + base = _turn() + turn = ObservedTurn( + outcome=base.outcome, + tool_arguments=base.tool_arguments, + invocations=(*base.invocations, _haiku_call(30_000), _haiku_call(100_001)), + stream_calls=3, + ) + arm = _assemble([_unit(store, _golden(1)[0], turn)]) + summary = arm.arm_result.tier_summary_fields + assert summary["tool"]["long_context_invocation_count"] == 1 + assert summary["supervisor"]["long_context_invocation_count"] == 0 + assert arm.arm_result.model_metrics_fields["long_context_invocation_count"] == 0 + + +# --- attribution bound ------------------------------------------------------------------------- + + +def _delegating_turn(*, duplicate_supervisor: bool) -> ObservedTurn: + """Two supervisor rounds delegating to a subagent round and a tool call: four stream calls.""" + base = _turn(stream_calls=2) + subagent = InvocationAttribution( + tier="subagent", + role="research", + model_id=MATRIX.model_for("anthropic", "subagent").model_id, + tokens=TokenCounts(input_tokens=40, output_tokens=4), + cache=CacheCounts(), + ) + supervisor = base.invocations[:1] * (2 if duplicate_supervisor else 1) + return ObservedTurn( + outcome=base.outcome, + tool_arguments=base.tool_arguments, + invocations=(*supervisor, *base.invocations[1:], subagent, _haiku_call(500)), + stream_calls=4, + ) + + +def test_a_delegating_turn_with_every_call_attributed_once_assembles(store: Store) -> None: + turn = _delegating_turn(duplicate_supervisor=False) + assert len(turn.invocations) == turn.stream_calls == 4 + arm = _assemble([_unit(store, _golden(1)[0], turn)], provider_stream_calls=4) + assert arm.arm_result.model_metrics_fields["included_invocation_count"] == 2 + + +def test_a_duplicated_supervisor_attribution_cannot_hide_under_delegated_calls( + store: Store, +) -> None: + """One extra supervisor usage on a delegating turn: 3 supervisor records against 4 calls + passed the old supervisor-only bound, and the all-tier bound refuses it.""" + turn = _delegating_turn(duplicate_supervisor=True) + supervisor_records = sum(1 for a in turn.invocations if a.tier == "supervisor") + assert supervisor_records < turn.stream_calls < len(turn.invocations) + with pytest.raises(AssertionError, match="task unit .* double-recorded usage"): + _assemble([_unit(store, _golden(1)[0], turn)], provider_stream_calls=4) + + +def test_the_bound_is_per_unit_so_another_turns_slack_cannot_absorb_a_surplus( + store: Store, +) -> None: + """A turn with a usage-less call does not cover a duplicate recorded in a different turn.""" + tasks = _golden(2) + surplus = _delegating_turn(duplicate_supervisor=True) + slack = ObservedTurn( + outcome=_turn().outcome, + tool_arguments=_turn().tool_arguments, + invocations=_turn().invocations, + stream_calls=2, + ) + units = [_unit(store, tasks[0], surplus), _unit(store, tasks[1], slack)] + assert sum(len(u.turn.invocations) for u in units) == sum(u.turn.stream_calls for u in units) + with pytest.raises(AssertionError, match="task unit .* double-recorded usage"): + _assemble(units, provider_stream_calls=6) diff --git a/tests/unit/test_teardown_deletion_protection.py b/tests/unit/test_teardown_deletion_protection.py index 8d6b9d5..3e28f53 100644 --- a/tests/unit/test_teardown_deletion_protection.py +++ b/tests/unit/test_teardown_deletion_protection.py @@ -18,7 +18,7 @@ TEARDOWN_SH = REPO_ROOT / "scripts" / "teardown.sh" RENDER_SCRIPT = REPO_ROOT / "scripts" / "render_agentcore_env.py" -ACCOUNT = "111122223333" +ACCOUNT = "111122223333" # nosemgrep: generic.secrets.security.detected-aws-account-id REGION = "eu-west-2" PROFILE = "teardown-test-profile" ALB_ARN = f"arn:aws:elasticloadbalancing:{REGION}:{ACCOUNT}:loadbalancer/app/crm-api-origin/50dc6c495c0c9188" @@ -32,11 +32,13 @@ ] ) -_FAKE_AWS = """#!/usr/bin/env bash +_FAKE_AWS = ( + """#!/usr/bin/env bash printf 'aws %s\\n' "$*" >> "$CALL_LOG" if [[ "$*" == *"get-caller-identity"* ]]; then - echo '{"Account": "111122223333"}' - exit 0 +""" + + f' echo \'{{"Account": "{ACCOUNT}"}}\'\n' + + """ exit 0 fi if [[ "$1" == "elbv2" && "$2" == "describe-load-balancers" ]]; then case "${FAKE_ALB_AFTER:-gone}" in @@ -53,6 +55,7 @@ fi exit 0 """ +) _FAKE_TERRAFORM = """#!/usr/bin/env bash printf 'terraform %s\\n' "$*" >> "$CALL_LOG" diff --git a/tests/unit/test_token_accounting.py b/tests/unit/test_token_accounting.py index 539aa81..ade4d8e 100644 --- a/tests/unit/test_token_accounting.py +++ b/tests/unit/test_token_accounting.py @@ -39,6 +39,8 @@ _MODEL_ID: Final[str] = "anthropic.claude-opus-5-5" _OPENAI_MODEL_ID: Final[str] = "openai.gpt-6-astra" +_PROFILE: Final[str] = f"global.{_MODEL_ID}" +_OPENAI_PROFILE: Final[str] = f"global.{_OPENAI_MODEL_ID}" _USAGE: Final[TokenUsage] = TokenUsage( input_tokens=1_000_000, output_tokens=100_000, @@ -56,11 +58,12 @@ def _price_table(tmp_path: Path, *, provisional: bool) -> Path: "cache_read": _CACHE_READ_RATE, } payload: dict[str, Any] = { - "schema_version": 1, + "schema_version": 2, "price_table_version": "test-1", "provisional": provisional, "currency": "USD", "unit": "per_1m_tokens", + "routes": {"global": {"multiplier": 1.0}, "geo": {"multiplier": 1.10}}, "models": {_MODEL_ID: dict(rates), _OPENAI_MODEL_ID: dict(rates)}, } path = tmp_path / "pricing.yaml" @@ -145,7 +148,7 @@ def test_inclusive_accounting_subtracts_both_cached_read_and_cache_write_from_th _declare(monkeypatch, "anthropic", "inclusive") path = _price_table(tmp_path, provisional=True) - cost = cost_for_usage(_USAGE, _MODEL_ID, pricing_path=path) + cost = cost_for_usage(_USAGE, _MODEL_ID, inference_profile=_PROFILE, pricing_path=path) # 1_000_000 - 400_000 cache_read - 50_000 cache_write = 550_000 assert billable_input_tokens(_USAGE, "anthropic") == 550_000 @@ -162,7 +165,7 @@ def test_exclusive_accounting_charges_input_cache_read_and_cache_write_additivel _declare(monkeypatch, "anthropic", "exclusive") path = _price_table(tmp_path, provisional=True) - cost = cost_for_usage(_USAGE, _MODEL_ID, pricing_path=path) + cost = cost_for_usage(_USAGE, _MODEL_ID, inference_profile=_PROFILE, pricing_path=path) assert billable_input_tokens(_USAGE, "anthropic") == 1_000_000 assert cost.input == pytest.approx(1.0 * _INPUT_RATE) @@ -174,7 +177,7 @@ def test_exclusive_accounting_charges_input_cache_read_and_cache_write_additivel def test_shipped_anthropic_prices_exclusively_with_a_non_zero_cached_read(tmp_path: Path) -> None: """Under the shipped declaration a non-zero cached read does not reduce Anthropic's input base.""" path = _price_table(tmp_path, provisional=False) - cost = cost_for_usage(_USAGE, _MODEL_ID, pricing_path=path) + cost = cost_for_usage(_USAGE, _MODEL_ID, inference_profile=_PROFILE, pricing_path=path) assert billable_input_tokens(_USAGE, "anthropic") == 1_000_000 assert cost.input == pytest.approx(1.0 * _INPUT_RATE) assert cost.cache_read_accounting == "exclusive" @@ -183,7 +186,9 @@ def test_shipped_anthropic_prices_exclusively_with_a_non_zero_cached_read(tmp_pa def test_shipped_openai_prices_inclusively_with_a_non_zero_cached_read(tmp_path: Path) -> None: """Under the shipped declaration cached reads and writes are subtracted from OpenAI's input base.""" path = _price_table(tmp_path, provisional=False) - cost = cost_for_usage(_USAGE, _OPENAI_MODEL_ID, pricing_path=path) + cost = cost_for_usage( + _USAGE, _OPENAI_MODEL_ID, inference_profile=_OPENAI_PROFILE, pricing_path=path + ) assert billable_input_tokens(_USAGE, "openai") == 550_000 assert cost.input == pytest.approx(0.55 * _INPUT_RATE) assert cost.cache_read == pytest.approx(0.4 * _CACHE_READ_RATE) @@ -198,7 +203,7 @@ def test_unconfirmed_accounting_prices_additively_while_the_table_is_provisional _declare(monkeypatch, "anthropic", "unconfirmed") path = _price_table(tmp_path, provisional=True) - cost = cost_for_usage(_USAGE, _MODEL_ID, pricing_path=path) + cost = cost_for_usage(_USAGE, _MODEL_ID, inference_profile=_PROFILE, pricing_path=path) assert billable_input_tokens(_USAGE, "anthropic") == 1_000_000 assert cost.input == pytest.approx(1.0 * _INPUT_RATE) @@ -230,7 +235,7 @@ def test_clearing_provisional_under_an_unconfirmed_accounting_stops_rather_than_ path = _price_table(tmp_path, provisional=False) with pytest.raises(UnconfirmedTokenAccountingError) as excinfo: - cost_for_usage(_USAGE, _MODEL_ID, pricing_path=path) + cost_for_usage(_USAGE, _MODEL_ID, inference_profile=_PROFILE, pricing_path=path) assert "'anthropic'" in str(excinfo.value) assert "TOKEN_ACCOUNTING" in str(excinfo.value) @@ -253,6 +258,7 @@ def test_the_guard_reaches_the_sole_record_constructor_path( family="anthropic", tier="supervisor", usage=_USAGE, + inference_profile=_PROFILE, pricing_path=path, ) @@ -263,10 +269,14 @@ def test_the_shipped_settled_families_price_a_publishable_table_without_raising( """Both shipped families price a non-provisional table under their declared accounting.""" path = _price_table(tmp_path, provisional=False) - anthropic_cost = cost_for_usage(_USAGE, _MODEL_ID, pricing_path=path) + anthropic_cost = cost_for_usage( + _USAGE, _MODEL_ID, inference_profile=_PROFILE, pricing_path=path + ) assert anthropic_cost.input == pytest.approx(1.0 * _INPUT_RATE) # exclusive: whole input base - openai_cost = cost_for_usage(_USAGE, _OPENAI_MODEL_ID, pricing_path=path) + openai_cost = cost_for_usage( + _USAGE, _OPENAI_MODEL_ID, inference_profile=_OPENAI_PROFILE, pricing_path=path + ) assert openai_cost.input == pytest.approx(0.55 * _INPUT_RATE) # inclusive: read+write removed @@ -283,6 +293,7 @@ def test_a_cost_figure_never_travels_without_the_accounting_that_produced_it( family="anthropic", tier="supervisor", usage=_USAGE, + inference_profile=_PROFILE, pricing_path=_price_table(tmp_path, provisional=True), ) assert record.cost.cache_read_accounting == "exclusive" diff --git a/tests/unit/test_tool_actor.py b/tests/unit/test_tool_actor.py index 5fedee4..53ef8f1 100644 --- a/tests/unit/test_tool_actor.py +++ b/tests/unit/test_tool_actor.py @@ -254,7 +254,10 @@ def test_production_dispatch_strips_identity_keys_and_appends_the_trusted_block( identity = Identity(rep_id="rep-1", book=frozenset({"acc-1"}), tenant="demo") spoofed_args: dict[str, object] = { "account_id": "acc-1", - AGENT_IDENTITY_KEY: {"claims": {CLAIM_REP_ID: "attacker", CLAIM_TENANT: "other"}}, + AGENT_IDENTITY_KEY: { + "claims": {CLAIM_REP_ID: "attacker", CLAIM_TENANT: "other"}, + "principal_type": "user", + }, } event = agent_identity_event(identity, spoofed_args) diff --git a/tests/unit/test_turn_costing.py b/tests/unit/test_turn_costing.py index a02a713..52f0f8c 100644 --- a/tests/unit/test_turn_costing.py +++ b/tests/unit/test_turn_costing.py @@ -3,22 +3,30 @@ from __future__ import annotations +import dataclasses import math from pathlib import Path import pytest +import yaml from agents.events import CostLine from agents.tier_attribution import CacheCounts, InvocationAttribution, TokenCounts from agents.turn_costing import DEFAULT_PRICING_PATH, cost_lines, is_cost_withheld, turn_cost +from crm.inference_route import UnknownInferenceRoute from crm.metering import TokenUsage, cost_for_usage -from providers.capabilities import load_capability_matrix +from providers.capabilities import CapabilityMatrix, load_capability_matrix _MATRIX = load_capability_matrix() OPUS = _MATRIX.model_for("anthropic", "supervisor").model_id HAIKU = _MATRIX.model_for("anthropic", "tool").model_id LUNA = _MATRIX.model_for("openai", "tool").model_id +#: Haiku's long-context threshold in the shipped price table. +HAIKU_LONG_CONTEXT_ABOVE: int = yaml.safe_load(DEFAULT_PRICING_PATH.read_text(encoding="utf-8"))[ + "models" +][HAIKU]["long_context"]["above_prompt_tokens"] + def _inv( tier: str, role: str, model_id: str, tokens_in: int, tokens_out: int, cache_read: int = 0 @@ -32,11 +40,25 @@ def _inv( ) -def _price(model_id: str, tokens_in: int, tokens_out: int, cache_read: int = 0) -> float: +def _price( + model_id: str, + tokens_in: int, + tokens_out: int, + cache_read: int = 0, + *, + inference_profile: str | None = None, +) -> float: usage = TokenUsage( input_tokens=tokens_in, output_tokens=tokens_out, cache_read_tokens=cache_read ) - return cost_for_usage(usage, model_id).total + profile = inference_profile or _MATRIX.models[model_id].inference_profile + return cost_for_usage(usage, model_id, inference_profile=profile).total + + +def _with_profile(model_id: str, profile: str) -> CapabilityMatrix: + models = dict(_MATRIX.models) + models[model_id] = dataclasses.replace(models[model_id], inference_profile=profile) + return dataclasses.replace(_MATRIX, models=models) def _table(tmp_path: Path, *, provisional: object) -> Path: @@ -93,13 +115,106 @@ def test_withheld_pricing_leaves_every_cost_unset() -> None: lines = cost_lines([_inv("supervisor", "supervisor", OPUS, 10, 10)], withheld=True) assert lines[0]["cost_usd"] is None assert lines[0]["input_tokens"] == 10 + assert lines[0]["long_context_calls"] == 0 assert turn_cost(lines, withheld=True) is None -def test_unpriced_model_raises() -> None: +def test_model_absent_from_the_capability_matrix_raises() -> None: + """A model with no configured inference profile is an error, never priced at a guessed route.""" + with pytest.raises(ValueError, match="capability matrix"): + cost_lines([_inv("tool", "research", "vendor.unknown-model", 1, 1)], withheld=False) + + +def test_unpriced_model_raises(tmp_path: Path) -> None: """A model missing from the price table is an error, never a silent zero.""" + table = yaml.safe_load(DEFAULT_PRICING_PATH.read_text(encoding="utf-8")) + del table["models"][LUNA] + path = tmp_path / "pricing.yaml" + path.write_text(yaml.safe_dump(table), encoding="utf-8") with pytest.raises(KeyError): - cost_lines([_inv("tool", "research", "vendor.unknown-model", 1, 1)], withheld=False) + cost_lines([_inv("tool", "research", LUNA, 1, 1)], withheld=False, pricing_path=path) + + +def test_each_invocation_is_priced_at_its_configured_profile_route() -> None: + """A geo profile in the matrix prices at the geo route, not the global default.""" + geo = _with_profile(HAIKU, f"us.{HAIKU}") + inv = _inv("tool", "research", HAIKU, 1_000, 200) + + geo_cost = cost_lines([inv], withheld=False, matrix=geo)[0]["cost_usd"] + global_cost = cost_lines([inv], withheld=False)[0]["cost_usd"] + + assert geo_cost == pytest.approx(_price(HAIKU, 1_000, 200, inference_profile=f"us.{HAIKU}")) + assert global_cost == pytest.approx(_price(HAIKU, 1_000, 200)) + assert geo_cost != pytest.approx(global_cost) + + +def test_the_default_matrix_is_the_shipped_capability_matrix() -> None: + """Omitting ``matrix`` prices exactly as passing the loaded ``config/`` matrix.""" + invocations = [_inv("supervisor", "supervisor", OPUS, 30, 5), _inv("tool", "x", LUNA, 9, 2)] + assert cost_lines(invocations, withheld=False) == cost_lines( + invocations, withheld=False, matrix=load_capability_matrix() + ) + + +def test_a_profile_naming_another_model_is_refused() -> None: + """The configured profile must route to the invocation's own model.""" + with pytest.raises(UnknownInferenceRoute): + cost_lines( + [_inv("tool", "research", HAIKU, 1, 1)], + withheld=False, + matrix=_with_profile(HAIKU, f"global.{LUNA}"), + ) + + +def test_long_context_calls_count_members_priced_at_long_context_rates() -> None: + """Each member is priced alone: one call over the threshold counts, smaller calls do not.""" + above = HAIKU_LONG_CONTEXT_ABOVE + 1 + small = HAIKU_LONG_CONTEXT_ABOVE // 2 + invocations = [ + _inv("tool", "research", HAIKU, small, 10), + _inv("tool", "research", HAIKU, small, 10), + _inv("tool", "research", HAIKU, above, 10), + ] + + (line,) = cost_lines(invocations, withheld=False) + + assert line["long_context_calls"] == 1 + assert line["cost_usd"] == pytest.approx( + 2 * _price(HAIKU, small, 10) + _price(HAIKU, above, 10) + ) + assert line["cost_usd"] != pytest.approx(_price(HAIKU, 2 * small + above, 30)) + + +def test_a_call_at_the_threshold_is_not_long_context() -> None: + """The long-context rates apply strictly above ``above_prompt_tokens``.""" + (line,) = cost_lines( + [_inv("tool", "research", HAIKU, HAIKU_LONG_CONTEXT_ABOVE, 1)], withheld=False + ) + assert line["long_context_calls"] == 0 + + +def test_cache_tokens_count_toward_the_long_context_prompt() -> None: + """Cache reads are part of the prompt the threshold is measured against.""" + (line,) = cost_lines( + [_inv("tool", "research", HAIKU, HAIKU_LONG_CONTEXT_ABOVE, 1, cache_read=1)], + withheld=False, + ) + assert line["long_context_calls"] == 1 + + +def test_a_model_without_long_context_rates_never_counts() -> None: + """Opus has no long-context block, so even a very large prompt counts zero.""" + (line,) = cost_lines([_inv("supervisor", "supervisor", OPUS, 900_000, 1)], withheld=False) + assert line["long_context_calls"] == 0 + + +def test_long_context_calls_are_reported_while_withheld() -> None: + """Withholding drops the figure, not the count.""" + (line,) = cost_lines( + [_inv("tool", "research", HAIKU, HAIKU_LONG_CONTEXT_ABOVE + 1, 1)], withheld=True + ) + assert line["cost_usd"] is None + assert line["long_context_calls"] == 1 def test_turn_cost_refuses_an_unpriced_line_when_not_withheld() -> None: diff --git a/tests/unit/test_verify_deployment.py b/tests/unit/test_verify_deployment.py index b8f3047..8c36689 100644 --- a/tests/unit/test_verify_deployment.py +++ b/tests/unit/test_verify_deployment.py @@ -1059,7 +1059,7 @@ def test_run_rendering_is_labelled_unguarded_and_tallies_skip_and_defer( assert result.returncode == 0, result.stdout + result.stderr published = tmp_path / "results" / "verification-inventory.md" - run_md = tmp_path / "results" / "verification-inventory.run.md" + run_md = tmp_path / "results" / "verification-inventory.partial.run.md" assert run_md.exists(), result.stdout + result.stderr assert not published.exists(), "a --layer 4 run wrote the guarded published document" @@ -1082,7 +1082,14 @@ def test_inventory_artifacts_are_never_written_into_the_real_repo_by_this_suite( "would write its inventory into the real repository" ) script_text = script_path.read_text(encoding="utf-8") - for variable in ("INVENTORY_JSON", "INVENTORY_MD", "RUN_INVENTORY_JSON", "RUN_INVENTORY_MD"): + for variable in ( + "INVENTORY_JSON", + "INVENTORY_MD", + "RUN_INVENTORY_JSON", + "RUN_INVENTORY_MD", + "PARTIAL_RUN_INVENTORY_JSON", + "PARTIAL_RUN_INVENTORY_MD", + ): assert f'{variable}="${{REPO_ROOT}}/results/' in script_text, ( f"{variable} is no longer derived from REPO_ROOT; the tmp_path isolation this suite " "relies on would no longer hold" @@ -2501,7 +2508,7 @@ class whose synthetic CheckResult names the stderr log. ) assert result.returncode != 0 - inventory_json = tmp_path / "results" / "verification-inventory.run.json" + inventory_json = tmp_path / "results" / "verification-inventory.partial.run.json" assert inventory_json.exists(), result.stdout + result.stderr document = json.loads(inventory_json.read_text(encoding="utf-8")) assert document["partial"] is True and document["partial_selector"] == "--layer 4" @@ -2753,14 +2760,18 @@ def test_a_partial_run_is_marked_partial_and_lists_what_did_not_run( assert "NOT a full-deployment verification" in result.stderr document = json.loads( - (tmp_path / "results" / "verification-inventory.run.json").read_text(encoding="utf-8") + (tmp_path / "results" / "verification-inventory.partial.run.json").read_text( + encoding="utf-8" + ) ) assert isinstance(document, dict) and document["partial"] is True assert document["selected_modules"] == ["scripts.smoke.iam_roles"] assert set(document["not_run_modules"]) == _all_declared_modules() - {"scripts.smoke.iam_roles"} assert [entry["resource_class"] for entry in document["results"]] == ["iam_roles"] - md_text = (tmp_path / "results" / "verification-inventory.run.md").read_text(encoding="utf-8") + md_text = (tmp_path / "results" / "verification-inventory.partial.run.md").read_text( + encoding="utf-8" + ) assert "**PARTIAL RUN (--only)**" in md_text and "scripts.smoke.eval_sfn" in md_text @@ -2775,6 +2786,43 @@ def test_a_full_run_inventory_stays_an_unmarked_array( ) assert isinstance(document, list) assert "PARTIAL" not in result.stdout + result.stderr + assert not (tmp_path / "results" / "verification-inventory.partial.run.json").exists() + assert not (tmp_path / "results" / "verification-inventory.partial.run.md").exists() + + +@pytest.mark.parametrize( + "selector", + [("--only", "iam_roles"), ("--layer", "5"), ("--layer", "4", "--only", "iam_roles")], + ids=["only", "layer", "layer-and-only"], +) +def test_a_partial_run_leaves_the_full_run_inventory_bytes_untouched( + stubbed: tuple[Path, dict[str, str]], + outputs_file: Path, + tmp_path: Path, + selector: tuple[str, ...], +) -> None: + """A PARTIAL run writes only the partial-run paths; seeded full-run files keep their bytes.""" + script_path, env = stubbed + results_dir = tmp_path / "results" + results_dir.mkdir(parents=True, exist_ok=True) + full_json = results_dir / "verification-inventory.run.json" + full_md = results_dir / "verification-inventory.run.md" + seeded_json = b'[{"resource_class": "seeded-full-run", "checks": []}]\n' + seeded_md = b"# seeded full-run rendering\n" + full_json.write_bytes(seeded_json) + full_md.write_bytes(seeded_md) + + result = _run(script_path, _billable(outputs_file, *selector), env, tmp_path) + assert result.returncode == 0, result.stdout + result.stderr + + assert full_json.read_bytes() == seeded_json, "a PARTIAL run overwrote the full-run JSON" + assert full_md.read_bytes() == seeded_md, "a PARTIAL run overwrote the full-run rendering" + partial_json = results_dir / "verification-inventory.partial.run.json" + partial_md = results_dir / "verification-inventory.partial.run.md" + document = json.loads(partial_json.read_text(encoding="utf-8")) + assert isinstance(document, dict) and document["partial"] is True + assert "**PARTIAL RUN (" in partial_md.read_text(encoding="utf-8") + assert f"wrote {partial_json} and {partial_md}" in result.stdout def test_a_partial_run_that_stops_at_a_failing_layer_lists_the_unrun_selection( @@ -2790,7 +2838,9 @@ def test_a_partial_run_that_stops_at_a_failing_layer_lists_the_unrun_selection( "scripts.smoke.iam_roles" ] document = json.loads( - (tmp_path / "results" / "verification-inventory.run.json").read_text(encoding="utf-8") + (tmp_path / "results" / "verification-inventory.partial.run.json").read_text( + encoding="utf-8" + ) ) assert "scripts.smoke.eval_sfn" in document["not_run_modules"] assert "PARTIAL RUN: ONE OR MORE SELECTED MODULES FAILED" in result.stderr @@ -2813,7 +2863,9 @@ def test_a_layer_run_is_marked_partial_like_only( assert "PARTIAL RUN (--layer 5) — NOT a full-deployment verification." in result.stderr document = json.loads( - (tmp_path / "results" / "verification-inventory.run.json").read_text(encoding="utf-8") + (tmp_path / "results" / "verification-inventory.partial.run.json").read_text( + encoding="utf-8" + ) ) assert isinstance(document, dict) and document["partial"] is True assert document["partial_selector"] == "--layer 5" @@ -2822,7 +2874,9 @@ def test_a_layer_run_is_marked_partial_like_only( } assert set(document["selected_modules"]) == layer_5 assert set(document["not_run_modules"]) == _all_declared_modules() - layer_5 - md_text = (tmp_path / "results" / "verification-inventory.run.md").read_text(encoding="utf-8") + md_text = (tmp_path / "results" / "verification-inventory.partial.run.md").read_text( + encoding="utf-8" + ) assert "**PARTIAL RUN (--layer 5)**" in md_text