Repository navigation
test(conformance): fix the custom catalog and multi-catalog fixtures - #3267
Conversation
1. `Chart.caption` in `custom_catalog_v1_0.json` is a `DynamicString`, as a caption in a published catalog would be. The prompt goldens that embed the catalog are updated. 2. The Express case that binds a path to a property with a static type uses `Gauge($/value)`, since `Chart.caption` now accepts a binding. The TypeScript unit tests that used the old fixture follow. 3. The inline catalogs in `multi_catalog_formats.yaml` reference `common_types.json`, the file the protocol publishes, instead of `common.json`, which doesn't exist. Signed-off-by: Nan Yu <nanyu@google.com>
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request updates the caption property of the Chart component in the custom catalog from a static string to a dynamic string (DynamicString from common_types.json). Consequently, the conformance tests, prompt generator expectations, and compiler unit tests have been updated to reflect this change (e.g., using Gauge($/value) instead of Chart to test forbidden databindings on static properties). Additionally, schema references in multi_catalog_formats.yaml have been corrected from common.json to common_types.json. There are no review comments, and I have no feedback to provide.
There was a problem hiding this comment.
Is .txt the best extension for these files?
There was a problem hiding this comment.
I think .txt fits: these are verbatim goldens that every SDK compares byte-for-byte, and the content is headings mixed with raw JSON rather than real Markdown, so .md would just get rendered on GitHub. It also matches the existing prompt goldens in conformance/test_data/skills/ (express_base_rules.txt, express_catalog_instructions.txt).
|
|
||
| ### Catalog Schema: | ||
| {"$schema":"https://json-schema.org/draft/2020-12/schema","$id":"https://a2ui.org/conformance/test_data/catalogs/custom_catalog_v1_0.json","protocolVersion":"1.0","title":"Conformance Custom Catalog (v1.0)","description":"A second catalog used by the multi catalog and capability negotiation cases. Its component and function names do not overlap with the basic catalog, so a case can tell the two apart by name alone.","catalogId":"conformance/custom","components":{"Chart":{"type":"object","description":"Plots a series of numbers.","properties":{"component":{"const":"Chart"},"values":{"type":"array","items":{"type":"number"}},"caption":{"type":"string"}},"required":["component","values"]},"Gauge":{"type":"object","description":"Shows a single value on a scale.","properties":{"component":{"const":"Gauge"},"value":{"type":"number"}},"required":["component","value"]}},"functions":{"percent":{"type":"object","description":"Renders a ratio as a percentage string.","returnType":"string","properties":{"@call":{"const":"percent"},"args":{"type":"object","properties":{"value":{"type":"number"}},"required":["value"],"unevaluatedProperties":false}},"required":["@call","args"]}},"$defs":{"anyComponent":{"oneOf":[{"$ref":"#/components/Chart"},{"$ref":"#/components/Gauge"}]},"anyFunction":{"oneOf":[{"$ref":"#/functions/percent"}]}}} | ||
| {"$schema":"https://json-schema.org/draft/2020-12/schema","$id":"https://a2ui.org/conformance/test_data/catalogs/custom_catalog_v1_0.json","protocolVersion":"1.0","title":"Conformance Custom Catalog (v1.0)","description":"A second catalog used by the multi catalog and capability negotiation cases. Its component and function names do not overlap with the basic catalog, so a case can tell the two apart by name alone.","catalogId":"conformance/custom","components":{"Chart":{"type":"object","description":"Plots a series of numbers.","properties":{"component":{"const":"Chart"},"values":{"type":"array","items":{"type":"number"}},"caption":{"$ref":"common_types.json#/$defs/DynamicString"}},"required":["component","values"]},"Gauge":{"type":"object","description":"Shows a single value on a scale.","properties":{"component":{"const":"Gauge"},"value":{"type":"number"}},"required":["component","value"]}},"functions":{"percent":{"type":"object","description":"Renders a ratio as a percentage string.","returnType":"string","properties":{"@call":{"const":"percent"},"args":{"type":"object","properties":{"value":{"type":"number"}},"required":["value"],"unevaluatedProperties":false}},"required":["@call","args"]}},"$defs":{"anyComponent":{"oneOf":[{"$ref":"#/components/Chart"},{"$ref":"#/components/Gauge"}]},"anyFunction":{"oneOf":[{"$ref":"#/functions/percent"}]}}} |
There was a problem hiding this comment.
Because these schemas are oneliners, it's suuuuper hard to review what's changing here. Is this generated or maintained by hand?
There was a problem hiding this comment.
They're SDK output: the Direct JSON prompt generator serializes schemas compactly on purpose to save tokens, and all three SDKs compare the rendered prompt byte-for-byte against these files, so a bad edit fails CI everywhere. In this PR the only change in each of the four goldens is Chart.caption: "type":"string" → "$ref":"common_types.json#/$defs/DynamicString", mirroring the fixture change in custom_catalog_v1_0.json. git diff --word-diff makes it easy to spot.
2 participants
Fixes to the shared conformance fixtures (the YAML cases under
conformance/agent/and the data underconformance/test_data/).Chart.captionincustom_catalog_v1_0.jsonis aDynamicString, as a caption in a published catalog would be. Two Express cases inmulti_catalog_formats.yaml(test_express_compile_variable_holding_function_call_with_catalog_overrideandtest_express_compile_function_catalog_override_in_expression_and_action) pass aformatString(...)call to the caption. They pass on main only because Express output skips some catalog checks. Once feat(python/agent)!: make the Parser stateless; always validate; add ResponseStream #3269 checks it like direct JSON output, a plain-string caption rejects them. The four direct JSON prompt goldens that embed the catalog are updated.test_compile_express_binding_a_static_property_is_a_validation_errorcan no longer useChart([1, 2], $/caption)to show that binding a static property fails. It bindsGauge.value, which is still a plain number, so the case keeps testing the same rule. The TypeScript unit tests that used the old input and the old prompt signature follow.multi_catalog_formats.yamlreferencecommon_types.json, the file the protocol publishes, instead ofcommon.json, which doesn't exist.Testing
uv run pytest python/a2ui_agent/tests(1290 passed).yarn testintypescript/agents/a2ui_agent(757 passed).dart testindart/a2ui_agent.