Repository navigation
Merged
Conversation
…ble tier tables, durable chat history, real truncation and in-flight budget
## Summary
This PR carries three bodies of work.
**Security hardening from code scanning.**
1. The frontend and browser journeys draw every token and idempotency key from Web Crypto instead of `Math.random`, and a unit guard keeps `Math.random` calls out.
2. Several regular expressions that could backtrack, or that misread HTML tags, are rewritten, and each has tests.
3. `minimatch` is pinned to a fixed release through npm overrides.
**Platform upgrade.**
4. **Python 3.14.** The toolchain, every Lambda runtime, Lambda packaging and the CI images move from Python 3.13.18 to 3.14.10.
5. **Infrastructure tooling.** Terraform moves from 1.17.0 to 1.17.2 and the `hashicorp/aws` provider from 6.71.0 to 6.72.0.
6. **Dependencies.** Python, relay and frontend packages get patch and minor upgrades only.
7. **Smoke error redaction.** No smoke check reports an AWS error message any more, because such a message can name the caller's ARN and with it the account id. Every AWS failure is reported as its operation and error code only.
8. **Runtime image content.** The AgentCore agent source no longer ships `src/relay/` and its `node_modules`.
**Eval measurement and chat behaviour.** An independent re-pricing of every recorded call showed the arithmetic in `results/matrix.md` was exact. The figures still misled, for four reasons, and each is fixed:
9. **Per-task cost denominator.** "Cost per task, all tasks" divided by every task the arm ran. It now divides by the tasks that apply at the tier, so at 100% completion the two cost rows are equal.
10. **Token rows were not like-for-like.** Anthropic reports input excluding cache reads and writes, while OpenAI reports it including them. Each family's reported input count therefore made Anthropic look impossibly low. Each tier table now shows total input, cache read, cache write and uncached input tokens, computed the same way for both families, so that total input = cache read + cache write + uncached input.
11. **Caching was asymmetric.** OpenAI caches the growing conversation, but the Anthropic adapter cached only the system prefix. It now also places a cache point on the latest message.
12. **Budget exhaustion and truncation were per-task flags shown on one table each.** The tier where a budget ran out is now recorded, and each tier table counts its own budget-exhausted tasks, truncated tool calls and truncated tasks.
Analysing the matrix also uncovered three behaviour problems, which are fixed:
13. **A chat follow-up turn could be replayed instead of run.** A same-session follow-up with no resume position was answered by replaying the previous turn's recorded events. That covered every MCP follow-up. A follow-up is now always a fresh turn. Replay happens only when the client sends `resume: true`, which the SPA's reconnect does. Conversation history is now durable in DynamoDB and survives a Runtime restart.
14. **`gt-05-investigation-large-tool-result` could never truncate.** Its data now exceeds the tool-output bound. The bound applies on every dispatch path, and the truncation flag reaches the tier tables.
15. **`gt-04-investigation-turn-budget-partial` measured the delegation choice, not the budget.** Leaf calls inside a delegation were charged only when the delegation returned, so a delegation could overrun the turn cap. They are now charged as they are dispatched, and the task is resized to fit the budget under both strategies.
`SECURITY.md` now maps the threat model to the OWASP Top 10 for LLM Applications 2026 and the OWASP Top 10 for Agentic Applications 2026. `results/matrix.md` is re-rendered from a new live sweep.
## Security hardening
- **Random tokens.**
- A new `frontend/src/lib/randomToken.ts` provides `randomToken(length)`, an unbiased base-36 token from `crypto.getRandomValues`, and `newTransactionKey(prefix)`, which prefers `crypto.randomUUID()`.
- Session ids, transcript keys and every form's and board's `transaction_key` use it.
- The browser journeys' run ids and keys use the same scheme, from `tests/e2e/browser/fixtures/randomToken.ts`.
- `tests/unit/test_no_math_random.py` fails on any `Math.random` call under `frontend/src` or `tests/e2e/browser`.
- **Regular expressions.**
- The SSE schema parity gate's union pattern and the `iam_roles` smoke's list pattern no longer nest optional repeats.
- The test that reads the API route map uses a single-line pattern, and a test asserts it stays linear on an unterminated input.
- The CloudFront smoke and the CSP tests match a closing `</script>` tag with trailing content or in any case.
- A new test fails when `index.html` carries a script tag that the case-sensitive Terraform hash expression would read differently.
- **Test fixtures.**
- The relay's JWT test builds its Basic credential at run time instead of embedding an encoded literal.
- The pricing-evidence test checks that every cited URL's host is `aws.amazon.com`, rather than looking for a substring.
- **Dependencies.** The frontend pins `minimatch` 11.0.8 through `overrides`.
## Python 3.14
- **Pins.**
- `.python-version` is 3.14.10.
- `requires-python` is `==3.14.*`.
- mypy targets 3.14.
- ruff targets `py314`. Its reformat rewrites only parenthesised `except` clauses into the PEP 758 form.
- **Lambda runtimes.** Every Python Lambda and the `lambda_action` module default move to `python3.14`. The relay stays on `nodejs24.x`.
- **Packaging.**
- `scripts/package_lambdas.sh` and `check_lambda_zip_imports` target CPython 3.14 (`cp314`).
- The runtime-provided distribution set was re-measured in the `python:3.14` Lambda base image. It is the boto3 closure, unchanged from 3.13. `setuptools` is no longer claimed, because neither base image ships it.
- **Images.** The Runtime image and the CI images use `python:3.14.10-slim` with pip 26.3.
- **Interpreter resolution.** A new `scripts/repo_python.sh` resolves the interpreter in this order:
1. `<repo>/.venv/bin/python3`;
2. an ambient `python3` at or above the `.python-version` major.minor;
3. otherwise, it exits 2 naming both versions.
Shell scripts, the git hooks, pre-commit and the frontend type generator now call it instead of a bare `python3`. It also pins `LC_ALL=C` and refuses a `.venv` below the required version.
- **Vendor SDK seam.**
- `sdk_transport` now refuses `ANTHROPIC_CUSTOM_HEADERS`.
- It also checks the Anthropic client's default headers after construction.
- Tests follow anthropic 2.8.0's move of its AWS auth module.
## Infrastructure tooling
- Every `hashicorp/aws` pin moves from 6.71.0 to 6.72.0. Both provider lock files are regenerated over the same platform hash set.
- Terraform moves from 1.17.0 to 1.17.2 in CI and the plan-fixture tooling. CI's relay Node moves to 24.22.3.
- `tests/fixtures/plans/compliant.base.ndjson` is recaptured. Apart from the following, it is unchanged:
- the runtimes and the provider constraint;
- the CloudTrail `recursive_logging` attribute, which the provider now computes;
- build-artefact content hashes.
Its recorded infra digest matches the current tree.
- **Provider scan.** A grype scan of the pinned 6.72.0 provider binary found one match: the Go standard library it was built with (go1.27.3, GO-2026-6859, fixed in go1.27.4). The binary is local Terraform tooling and is never deployed. The README's limitations record the result.
## Dependencies
| Area | Package | From | To |
|---|---|---|---|
| Python | anthropic | 2.6.0 | 2.8.0 |
| Python | boto3 / botocore | 1.43.125 | 1.44.1 |
| Python | openai | 3.46.0 | 3.49.0 |
| Python | sse-starlette | 3.6.0 | 3.7.0 |
| Python | types-PyYAML | 6.0.12.20261103 | 6.0.12.20261111 |
| Relay | @aws-sdk/client-bedrock-agentcore, @aws-sdk/client-ssm | 3.1163.0 | 3.1167.0 |
| Frontend | vite | 8.3.6 | 8.3.8 |
| Frontend | @vitejs/plugin-react | 6.2.1 | 6.3.0 |
| Frontend | typescript-eslint | 8.72.1 | 8.72.2 |
| Frontend | @testing-library/user-event | 14.6.8 | 14.6.9 |
| Frontend | minimatch (override) | — | 11.0.8 |
| Frontend | Node (`.nvmrc`, `engines.node`) | 26.13.0 | 26.13.1 |
`requirements.txt` is re-frozen under 3.14.10 and stays the sole install authority. Every lock file is regenerated with `npm ci`.
Major upgrades are deliberately deferred:
- `python-dateutil` 4, because botocore still caps it below 3;
- `watchdog` 7, which strands caps;
- `@types/node` 26, because the relay runs on `nodejs24.x`.
## Smoke error redaction
- **Shared helpers in `scripts/smoke/harness.py`.**
- `aws_error_code(exc)` returns an AWS error's code, or its class name when it has none. It never returns the message.
- `aws_error_detail(op, exc)` formats `"<operation> failed (<code>)"`.
- `exception_detail(exc)` keeps a local error's own text, but masks any AWS error found in its cause or context chain.
- **Every smoke module converted.** 32 modules under `scripts/smoke/` now report AWS failures through these helpers. This covers the harness's account check, its expected-count deriver and the `run_main` catch-all.
- Local errors raised from an AWS failure no longer embed the AWS text. They chain it with `raise … from`.
- A failed Lambda invoke is reported as its `FunctionError` and an identifier-shaped `errorType`, never `errorMessage` or `stackTrace`.
- The eval smoke reports a failed run's `error` field only as `present (<Class>)`, never its text, which is a failed step's raw error message.
- DynamoDB deny probes report the denial code, its classification and the probe role name instead of the denial message.
- `deployed_parity` reads every project function's runtime from the account (`lambda_runtime`), and requires every parity target plus the relay to be covered (`lambda_runtime_coverage`). The missing-`CodeSha256` detail lists only the response's key names.
- **Tests.**
- Each converted module has tests that plant an ARN with a placeholder account in the AWS error, then assert it appears in no detail.
- A new `tests/unit/test_smoke_no_raw_aws_error.py` walks every smoke module's syntax tree. It fails when a broad or AWS-error handler interpolates the exception itself, and it has no allow-marker.
## Eval, agents and chat
- **Prompt caching (`src/providers/`).**
- The Anthropic adapter adds a message-level `cache_control` point on the latest message, beside the system-prefix point.
- It stays within Anthropic's limit of four points and omits any point below the model's minimum cacheable size.
- The caching method per family is declared in `providers/prompt_caching.CACHING` and recorded in each run's provenance.
- **Chat turns and history (`src/agents/`, `src/relay/`, `frontend/`).**
- A same-session follow-up runs as a fresh turn. The Runtime replays recorded events only when the request carries `resume: true` and a `resume_from` before the last frame.
- The SPA sends `resume: true` from both its automatic and manual reconnect, and the relay forwards the field unchanged.
- History is recorded before the turn's `completed` event.
- **New store (`src/agents/`, through `src/crm/store/store.py`):** a DynamoDB-backed conversation history store. It writes one `history#<tenant>#<session>` item per session in the existing `reps` table.
- Ownership is checked: another `(rep_id, tenant)` reads empty, and its write is refused.
- It keeps the existing redaction and turn and field caps.
- Each item carries a 24-hour TTL.
- Every store call is time-bounded, and a history write that fails is logged without content and never blocks the answer.
- **IAM and infrastructure (`infra/`).**
- The Runtime role gains an inline policy that allows only the item actions the store calls, on `reps`, keyed to `history#` items.
- TTL is enabled on `reps.expires_at`.
- **Tool-output bound (`src/agents/`).**
- Every tool result is bounded at 32,768 bytes on every path: the supervisor's direct calls, leaf calls inside a delegation, and provider-reported tool events.
- A bounded result keeps whole records and carries `truncated: true`. The same flag is recorded on the call's attribution.
- A delegation with a bounded leaf reports itself truncated.
- **Turn budget (`src/agents/`).** Each leaf call inside a delegation is charged to the turn budget as it is dispatched. A delegation therefore stops at the cap, and the refused call is attributed to the ToolCall tier.
- **Golden tasks and fixtures (`fixtures/`, `src/eval/golden_tasks.yaml`).**
- Planted defect PD-08 adds 55 activities to `opp-0000`, so a default activity page exceeds the bound.
- Tasks that read `opp-0000` have updated expectations.
- `gt-04` is resized to one account by prompt.
- **Per-call and per-task records (`src/eval/`).**
- Every per-invocation and per-tool-call record carries its task id and repeat.
- Each task score now also records its repeat and the tier where its budget ran out. The record shape is now version 3.
- Older records are refused by the renderer, the `matrix.json`/`matrix.csv` export and the Evals UI with a clear error. Past deployed runs are listed as recorded in an earlier format.
- Re-pricing checks unit ids on every stamped entry.
- `check_cost_withheld` accepts pre-aggregation archives' non-currency keys.
- **Tier tables (`src/eval/matrix.py`, Evals UI).**
- The per-task cost row is renamed and uses the applicable-task denominator.
- The token rows are, in order: total input, cache read, cache write and uncached input tokens, then total output tokens. Total input = cache read + cache write + uncached input in every column.
- The family-specific reported input row is gone. `matrix.json` and `matrix.csv` keep the raw `input_tokens` and `total_prompt_tokens` and add `uncached_input_tokens`.
- New rows: budget-exhausted tasks, truncated tool calls and truncated tasks.
- The Delegation outcome rows leave the Subagent table; the counts stay in JSON and CSV.
- P50 latency comes before P95.
- The in-app Tier comparison view renders the same rows.
## Packaging
- `scripts/package_lambdas.sh` leaves `src/relay/` out of `agent-source.zip`, the Runtime image's build context. The Python Runtime never loads the relay, and the archive drops from more than 6,000 entries to 150.
- A new `tests/unit/test_package_lambdas_agent_source.py` runs the script's staging step and reads the built archive. It fails if any `src/relay/` entry is present.
## Checks
- **CI.**
- The frontend job fails when the regenerated tier-comparison fixture differs from the committed copy.
- It also runs the reconnect `resume` contract.
- **Unit tests** gate the Step Functions repeat-index wiring and the two truncation keys.
- **A new executing contract** covers the path from SPA through relay to Runtime for `resume` and `resume_from`.
- **Smoke:**
- a cross-turn history check in the three-tier chat smoke, in content and in tokens;
- a Runtime history IAM smoke, with allowed and unused actions and a cross-tenant denial;
- the agents smoke deletes the history item its turn wrote;
- `verify_deployment.sh` removes its stderr temp file on exit.
- **Journey 22** asserts chat recall across four turns, and that a new conversation does not recall.
## Threat model
- `SECURITY.md` maps the threat model to the OWASP Top 10 for LLM Applications 2026 and the OWASP Top 10 for Agentic Applications 2026.
- The OWASP Top 10 (web) stays on 2025, its current edition.
- Three new residual risks are listed under "Accepted risks and open items".
## Documentation
- `README.md`, `docs/development.md` and `docs/runbook.md` state Python 3.14.10, Node 26.13.1, Terraform 1.17.2 and provider 6.72.0. The README's plan-fixture provenance names the same versions, and its limitations record the provider scan.
- The runbook says to rewrite the browser-journey credential file after `provision_demo_user.py` resets a password. A stale file that gets sourced overrides the current password, and every signed-in journey then fails.
- `README.md`, `docs/runbook.md` and `docs/eval-methodology.md` cover:
- durable history, its TTL and operator deletion;
- the tool-output bound and leaf-call charging;
- the reworked tier rows and the matched caching.
## Testing
**Against a deployed dev stack**, deployed from this branch.
- **Deployed code.** The deployed code differs from the final commit only in `SECURITY.md`.
- **Which checks ran on which deploy.** The full verification, the full journey run and the sweep below ran on an earlier deploy. The final deploy changed only the tier tables' token rows, so it was checked with a targeted smoke and journey 21.
- **After the final deploy:**
- the eval smoke passed 311 checks with 0 failed;
- journey 21 (Tier comparison) passed 9/9 and showed the five new token rows on the deployed Evals page.
- **Full Layer 4–6 verification:** 13,443 checks passed and 0 failed. 182 were skipped and 38 deferred, all for documented reasons.
- Every Python function reports `python3.14` and the relay reports `nodejs24.x`, and the runtime coverage check passes.
- The unauthenticated and wrong-tenant paths are refused (401/403).
- The relay's Function URL enforces `AWS_IAM`.
- The canary tenant holds 0 rows.
- **History survives a Runtime restart.** After a 1,020 s idle, which spans the Runtime's idle shutdown, a follow-up turn with no resume flag recalled turn 1's fact.
- **T4 browser journeys** (`scripts/run_browser_journeys_container.sh`): 66 passed, 0 failed, 0 flaky. Journey 22 (chat recall) passed. 4 were skipped as declared:
- two Hosted UI tests, because no Hosted UI domain is configured;
- two live-sweep tests, which need a separate opt-in.
- **Live sweep:** 2 arms (Claude Sonnet 5.5 and GPT-6.1 Sol, both at medium effort) × 24 golden tasks × 5 repeats, for $5.72 against a $30 cap.
- Pre-flight checks confirmed the data-retention mode and an empty canary tenant.
- `results/matrix.md` is re-rendered from the sweep's stored raw record with the final renderer. An independent re-render is byte-equal to it.
- `gt-05` returned a truncated, flagged tool result on all 10 runs, and none hid the truncation.
- `gt-04` stayed within budget on all 10 runs, and no task in the sweep exhausted a budget.
## Upgrade notes
- **Python.** Recreate the virtualenv on Python 3.14.10:
```bash
python3.14 -m venv .venv
.venv/bin/pip install --no-input --no-deps -r requirements.txt
.venv/bin/pip install --no-input --no-deps -e .
```
- **npm.** Run `npm ci` in `frontend/`, `src/relay/` and `tests/e2e/browser/`, using Node 26.13.1 for the frontend.
- **Terraform.** Run `terraform init -upgrade` under Terraform 1.17.2 to pick up provider 6.72.0.
- **Deploy.** Run `scripts/package_lambdas.sh` and `scripts/package_relay.sh`, then `scripts/deploy.sh` (with `CRM_EXPECTED_ACCOUNT_ID`). The apply:
- updates every Python Lambda to the `python3.14` runtime;
- rebuilds the Runtime image;
- adds the Runtime history policy and enables TTL on `reps`;
- updates the eval Lambdas and the relay.
- **Old eval runs.** Eval runs recorded before this change stay in the deployed history but are not rendered. Local stored sweep records from before this change no longer render; re-run a sweep to produce new ones.
- **Chat history** now persists for 24 hours after a session's last turn. See `docs/runbook.md`, "Conversation history", for operator deletion.
## Known limitations
- **The tool-output bound and pagination disagree.** When the bound trims a page, `next_cursor` still resumes after the full page, so a model that follows the cursor never sees the trimmed records. The trim is flagged as truncated, and a smaller `limit` recovers them. On `opp-0000`'s default activity page, 4 of 50 records are affected.
- **No deployed check sends `resume: true`.** The reconnect replay path is covered only by unit tests and the executing contract.
- **The relay's prompt-cap contract test does not run in CI.** It passes when run with the relay's dependencies installed.
- **Some smoke modules leave chat-history items in disposable smoke tenants.** Four modules don't yet delete the history items their turns write: the agent-stream, proposal-card, text-delta and MCP chat smokes. The items expire by TTL.
- **A single unclassified stream `ReadError` stops a live sweep.** The interrupted sweep exits without results and has to be re-run.
- **Dollar figures are not yet checked against Cost Explorer.** Token counts reconciled exactly with CloudWatch in an earlier sweep.
- **A budget-exhausted run's usage is a lower bound,** because an abandoned round is unmetered.
- **The eval run record still stores a failed step's raw error message.** The sweep Lambdas write it to the run record's `error` field, and the eval API returns it to the signed-in admin. An unwrapped AWS error there can name the caller's role ARN. The smoke no longer echoes it, but changing the stored value means changing deployed code.
- **Runs recorded before this change show the new token rows under their stored notes.** The figures are correct, but those notes still describe the earlier rows and don't define uncached input tokens.
- **Provider 6.72.0 carries one Go standard-library match** (GO-2026-6859), fixed upstream in go1.27.4. It is in local tooling only and is never deployed.
- The deferred major upgrades listed under Dependencies.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR carries three bodies of work.
Security hardening from code scanning.
Math.random, and a unit guard keepsMath.randomcalls out.minimatchis pinned to a fixed release through npm overrides.Platform upgrade.
hashicorp/awsprovider from 6.71.0 to 6.72.0.src/relay/and itsnode_modules.Eval measurement and chat behaviour. An independent re-pricing of every recorded call showed the arithmetic in
results/matrix.mdwas exact. The figures still misled, for four reasons, and each is fixed:Analysing the matrix also uncovered three behaviour problems, which are fixed:
resume: true, which the SPA's reconnect does. Conversation history is now durable in DynamoDB and survives a Runtime restart.gt-05-investigation-large-tool-resultcould never truncate. Its data now exceeds the tool-output bound. The bound applies on every dispatch path, and the truncation flag reaches the tier tables.gt-04-investigation-turn-budget-partialmeasured the delegation choice, not the budget. Leaf calls inside a delegation were charged only when the delegation returned, so a delegation could overrun the turn cap. They are now charged as they are dispatched, and the task is resized to fit the budget under both strategies.SECURITY.mdnow maps the threat model to the OWASP Top 10 for LLM Applications 2026 and the OWASP Top 10 for Agentic Applications 2026.results/matrix.mdis re-rendered from a new live sweep.Security hardening
frontend/src/lib/randomToken.tsprovidesrandomToken(length), an unbiased base-36 token fromcrypto.getRandomValues, andnewTransactionKey(prefix), which preferscrypto.randomUUID().transaction_keyuse it.tests/e2e/browser/fixtures/randomToken.ts.tests/unit/test_no_math_random.pyfails on anyMath.randomcall underfrontend/srcortests/e2e/browser.iam_rolessmoke's list pattern no longer nest optional repeats.</script>tag with trailing content or in any case.index.htmlcarries a script tag that the case-sensitive Terraform hash expression would read differently.aws.amazon.com, rather than looking for a substring.minimatch11.0.8 throughoverrides.Python 3.14
Pins.
.python-versionis 3.14.10.requires-pythonis==3.14.*.py314. Its reformat rewrites only parenthesisedexceptclauses into the PEP 758 form.Lambda runtimes. Every Python Lambda and the
lambda_actionmodule default move topython3.14. The relay stays onnodejs24.x.Packaging.
scripts/package_lambdas.shandcheck_lambda_zip_importstarget CPython 3.14 (cp314).python:3.14Lambda base image. It is the boto3 closure, unchanged from 3.13.setuptoolsis no longer claimed, because neither base image ships it.Images. The Runtime image and the CI images use
python:3.14.10-slimwith pip 26.3.Interpreter resolution. A new
scripts/repo_python.shresolves the interpreter in this order:<repo>/.venv/bin/python3;python3at or above the.python-versionmajor.minor;Shell scripts, the git hooks, pre-commit and the frontend type generator now call it instead of a bare
python3. It also pinsLC_ALL=Cand refuses a.venvbelow the required version.Vendor SDK seam.
sdk_transportnow refusesANTHROPIC_CUSTOM_HEADERS.Infrastructure tooling
Every
hashicorp/awspin moves from 6.71.0 to 6.72.0. Both provider lock files are regenerated over the same platform hash set.Terraform moves from 1.17.0 to 1.17.2 in CI and the plan-fixture tooling. CI's relay Node moves to 24.22.3.
tests/fixtures/plans/compliant.base.ndjsonis recaptured. Apart from the following, it is unchanged:recursive_loggingattribute, which the provider now computes;Its recorded infra digest matches the current tree.
Provider scan. A grype scan of the pinned 6.72.0 provider binary found one match: the Go standard library it was built with (go1.27.3, GO-2026-6859, fixed in go1.27.4). The binary is local Terraform tooling and is never deployed. The README's limitations record the result.
Dependencies
.nvmrc,engines.node)requirements.txtis re-frozen under 3.14.10 and stays the sole install authority. Every lock file is regenerated withnpm ci.Major upgrades are deliberately deferred:
python-dateutil4, because botocore still caps it below 3;watchdog7, which strands caps;@types/node26, because the relay runs onnodejs24.x.Smoke error redaction
scripts/smoke/harness.py.aws_error_code(exc)returns an AWS error's code, or its class name when it has none. It never returns the message.aws_error_detail(op, exc)formats"<operation> failed (<code>)".exception_detail(exc)keeps a local error's own text, but masks any AWS error found in its cause or context chain.scripts/smoke/now report AWS failures through these helpers. This covers the harness's account check, its expected-count deriver and therun_maincatch-all.raise … from.FunctionErrorand an identifier-shapederrorType, nevererrorMessageorstackTrace.errorfield only aspresent (<Class>), never its text, which is a failed step's raw error message.deployed_parityreads every project function's runtime from the account (lambda_runtime), and requires every parity target plus the relay to be covered (lambda_runtime_coverage). The missing-CodeSha256detail lists only the response's key names.tests/unit/test_smoke_no_raw_aws_error.pywalks every smoke module's syntax tree. It fails when a broad or AWS-error handler interpolates the exception itself, and it has no allow-marker.Eval, agents and chat
src/providers/).cache_controlpoint on the latest message, beside the system-prefix point.providers/prompt_caching.CACHINGand recorded in each run's provenance.src/agents/,src/relay/,frontend/).resume: trueand aresume_frombefore the last frame.resume: truefrom both its automatic and manual reconnect, and the relay forwards the field unchanged.completedevent.src/agents/, throughsrc/crm/store/store.py): a DynamoDB-backed conversation history store. It writes onehistory#<tenant>#<session>item per session in the existingrepstable.(rep_id, tenant)reads empty, and its write is refused.infra/).reps, keyed tohistory#items.reps.expires_at.src/agents/).truncated: true. The same flag is recorded on the call's attribution.src/agents/). Each leaf call inside a delegation is charged to the turn budget as it is dispatched. A delegation therefore stops at the cap, and the refused call is attributed to the ToolCall tier.fixtures/,src/eval/golden_tasks.yaml).opp-0000, so a default activity page exceeds the bound.opp-0000have updated expectations.gt-04is resized to one account by prompt.src/eval/).matrix.json/matrix.csvexport and the Evals UI with a clear error. Past deployed runs are listed as recorded in an earlier format.check_cost_withheldaccepts pre-aggregation archives' non-currency keys.src/eval/matrix.py, Evals UI).matrix.jsonandmatrix.csvkeep the rawinput_tokensandtotal_prompt_tokensand adduncached_input_tokens.Packaging
scripts/package_lambdas.shleavessrc/relay/out ofagent-source.zip, the Runtime image's build context. The Python Runtime never loads the relay, and the archive drops from more than 6,000 entries to 150.tests/unit/test_package_lambdas_agent_source.pyruns the script's staging step and reads the built archive. It fails if anysrc/relay/entry is present.Checks
resumecontract.resumeandresume_from.verify_deployment.shremoves its stderr temp file on exit.Threat model
SECURITY.mdmaps the threat model to the OWASP Top 10 for LLM Applications 2026 and the OWASP Top 10 for Agentic Applications 2026.Documentation
README.md,docs/development.mdanddocs/runbook.mdstate Python 3.14.10, Node 26.13.1, Terraform 1.17.2 and provider 6.72.0. The README's plan-fixture provenance names the same versions, and its limitations record the provider scan.provision_demo_user.pyresets a password. A stale file that gets sourced overrides the current password, and every signed-in journey then fails.README.md,docs/runbook.mdanddocs/eval-methodology.mdcover:Testing
Against a deployed dev stack, deployed from this branch.
SECURITY.md.python3.14and the relay reportsnodejs24.x, and the runtime coverage check passes.AWS_IAM.scripts/run_browser_journeys_container.sh): 66 passed, 0 failed, 0 flaky. Journey 22 (chat recall) passed. 4 were skipped as declared:results/matrix.mdis re-rendered from the sweep's stored raw record with the final renderer. An independent re-render is byte-equal to it.gt-05returned a truncated, flagged tool result on all 10 runs, and none hid the truncation.gt-04stayed within budget on all 10 runs, and no task in the sweep exhausted a budget.Upgrade notes
python3.14 -m venv .venv .venv/bin/pip install --no-input --no-deps -r requirements.txt .venv/bin/pip install --no-input --no-deps -e .npm ciinfrontend/,src/relay/andtests/e2e/browser/, using Node 26.13.1 for the frontend.terraform init -upgradeunder Terraform 1.17.2 to pick up provider 6.72.0.scripts/package_lambdas.shandscripts/package_relay.sh, thenscripts/deploy.sh(withCRM_EXPECTED_ACCOUNT_ID). The apply:python3.14runtime;reps;docs/runbook.md, "Conversation history", for operator deletion.Known limitations
next_cursorstill resumes after the full page, so a model that follows the cursor never sees the trimmed records. The trim is flagged as truncated, and a smallerlimitrecovers them. Onopp-0000's default activity page, 4 of 50 records are affected.resume: true. The reconnect replay path is covered only by unit tests and the executing contract.ReadErrorstops a live sweep. The interrupted sweep exits without results and has to be re-run.errorfield, and the eval API returns it to the signed-in admin. An unwrapped AWS error there can name the caller's role ARN. The smoke no longer echoes it, but changing the stored value means changing deployed code.