Skip to content

fix(flags): reject unknown keys in schema-backed JSON flags - #2379

Draft
jariy17 wants to merge 1 commit into
refactorfrom
fix/strict-json-flag-schemas
Draft

jariy17 wants to merge 1 commit into
refactorfrom
fix/strict-json-flag-schemas

Conversation

@jariy17

@jariy17 jariy17 commented Sep 22, 2026

Copy link
Copy Markdown
Contributor

Problem

Zod strips keys a schema does not declare. So a misspelled key in a JSON flag body parses cleanly, silently disappears, and on an update path the surviving object replaces the stored configuration — the typo destroys the value the user was trying to edit instead of reporting an error.

The typo cannot be caught any later. The AWS SDK serializes only the members its own schema declares, so the bad key never reaches the service and no server-side validation can see it:

input:  model: { modlId: "anthropic.claude-sonnet-4" }
wire:   {"clientToken":"...","model":{}}

Change

One helper, parseJsonFlagWithSchema, applies .strict() to the schema it was handed:

// Top level only: a nested typo (model.bedrockModelConfig.modelIdd) still slips through.
function strictened<T>(schema: z.ZodType<T>): z.ZodType<T> {
  return schema instanceof z.ZodObject ? (schema.strict() as unknown as z.ZodType<T>) : schema;
}

Applied in the helper rather than at each schema definition so a schema author cannot forget it, and so the schemas in projectSchemas/ stay untouched. The instanceof z.ZodObject check is what leaves z.record schemas (e.g. TagsSchema, where every key is legal by definition) and array-rooted schemas alone.

Verified against the real CLI

Same command, same input, with the change stashed vs applied:

$ agentcore project add evaluator llm-as-a-judge --name e9 --instructions inst \
    --level SESSION --model anthropic.claude-sonnet-4-20250514-v1:0 \
    --rating-scale '{"numerical":[{"value":1,"label":"good","definition":"d"}],"bogus":1}'

before:  added evaluator 'e9' to 'bb'          # exit 0; "bogus" silently stripped from agentcore.json
after:   Error: Invalid value for option '--rating-scale': Unrecognized key: "bogus"

The before run wrote the evaluator to agentcore.json and reported success.

Blast radius

All 15 flags that route through this helper were exercised against the real CLI, before and after. Exactly one changes behaviour — the rest were already guarded:

Flag Guard today Changed
project add evaluator llm-as-a-judge --rating-scale none yes
memory event list / memory record list --metadata-filters element .strict() no
project add gateway --authorizer-configuration .strict() at call site no
project add config-bundle --components, eval config-bundle create --components record of strict no
project add memory --strategies / --indexed-keys / --stream-delivery-resources custom field check no
project add gateway-target --target-configuration, gateway-connector --connector-configuration union branch check no
--tags (×4) z.record — every key legal no

The rest of the value is forward-looking: any schema passed to this helper from now on is strict without the author remembering.

Known limitation

Top level only, as the comment says. A nested typo (model.bedrockModelConfig.modelIdd) still slips through, because zod's strictness is per-object. Recursing means rebuilding the schema tree, which drops the .refine checks projectSchemas/ relies on — zod v4 throws on .extend() of a refined object for exactly that reason, and zod's maintainer rejects a built-in deepStrict on the same grounds (colinhacks/zod#2062).

Also out of scope: the ~90 call sites still using parseJsonFlag / parseJsonObjectFlag / parseJsonArrayFlag, which cast without any schema and so have nothing to strictify.

Testing

  • bun test — 3555 pass, 0 fail (242 files)
  • tsc --noEmit clean, oxlint and prettier clean
  • 5 new tests in src/handlers/utils.test.tsx: typo rejected, schema refinements still enforced, z.record stays permissive, valid input unchanged, omitted flag returns undefined

Zod strips keys a schema does not declare, so a misspelled key in a JSON
flag body parses cleanly and then vanishes. On an update path the surviving
object replaces the stored configuration, so the typo destroys the value the
user was trying to edit rather than reporting an error.

The typo cannot be caught later either: the AWS SDK serializes only the
members its own schema declares, so the bad key never reaches the service
and no server-side validation can see it.

Applying strict in the helper rather than at each schema definition means a
schema author cannot forget it, and leaves the schemas in projectSchemas/
untouched.

Verified against the real CLI. `project add evaluator llm-as-a-judge
--rating-scale '{"numerical":[{"value":1,"label":"good","definition":"d"}],
"bogus":1}'` previously printed "added evaluator 'e9'" and wrote the
evaluator with "bogus" silently dropped; it now fails with
`Unrecognized key: "bogus"` and writes nothing.
@github-actions github-actions Bot added the size/s PR size: S label Sep 22, 2026
@agentcore-devx-automation agentcore-devx-automation Bot added agentcore-harness-reviewing AgentCore Harness review in progress claude-security-reviewing Claude Code /security-review in progress labels Sep 22, 2026
@agentcore-devx-automation

Copy link
Copy Markdown
Contributor

Claude Security Review: no high-confidence findings. (run)

@agentcore-devx-automation agentcore-devx-automation Bot removed the claude-security-reviewing Claude Code /security-review in progress label Sep 22, 2026

@agentcore-devx-automation agentcore-devx-automation Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AgentCore Harness Review

Verdict: Looks good

The change is a small, well-scoped hardening: parseJsonFlagWithSchema now applies .strict() when the top-level schema is a ZodObject, so typos like modlId fail loudly instead of being silently dropped. Tests cover the important cases (valid input, unknown key rejected, refinements preserved, records left permissive), and strictened correctly skips non-object schemas so z.record(...), z.array(...), and custom-pipe wrappers (e.g. projectMemoryObject, componentMapSchema, TagsSchema) keep working.

I walked every existing callsite (gateway-connector, gateway-target, gateway, memory, evaluator/*, config-bundle, metadataFilters, online-eval/create) and none of them regress:

  • Schemas that already call .strict() (e.g. AgentCoreGatewayTargetSchema, GatewayAuthorizerConfigurationInputSchema) are unaffected — .strict() is idempotent.
  • Array/record schemas (indexed keys, strategies, tags, components, metadata filters) fall through strictened unchanged, as intended.
  • Zod v4's .refine() / .superRefine() still return ZodObject instances, so RatingScaleSchema and the tested ModelSchema correctly get strictened.

The inline comment on strictened honestly calls out the "top level only" limitation, and the callers that care about deep validation (memory input via projectMemoryObject, gateway target via nested .strict() blocks) already provide their own deep checks. This is a pure utility change so no telemetry is needed, and there is no mocking in the added tests.

Nothing to change — good to merge.

@agentcore-devx-automation agentcore-devx-automation Bot removed the agentcore-harness-reviewing AgentCore Harness review in progress label Sep 22, 2026
@codecov-commenter

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 97.25%. Comparing base (74fddae) to head (6b1d3aa).
⚠️ Report is 2 commits behind head on refactor.

Additional details and impacted files
@@             Coverage Diff              @@
##           refactor    #2379      +/-   ##
============================================
- Coverage     97.25%   97.25%   -0.01%     
============================================
  Files           613      613              
  Lines         41030    41033       +3     
============================================
+ Hits          39905    39907       +2     
- Misses         1125     1126       +1     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size/s PR size: S

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants