Skip to content

test(conformance): cover reference reading and composition cost in the core suites - #3344

Merged
nan-yu merged 4 commits into
mainfrom
conformance-graph-cases
Oct 20, 2026
Merged

nan-yu merged 4 commits into
mainfrom
conformance-graph-cases

Conversation

@polina-c

@polina-c polina-c commented Oct 20, 2026 •

Copy link
Copy Markdown
Collaborator

This PR adds conformance cases for three behaviours that came up in the reviews of #3326 and #3327. No suite covers them today. The Dart core passes the new timing case since #3345. Each SDK's results are below, and the cases an SDK still fails on main are listed as known gaps in its harness.

The cases

  1. core/validator_v0_8.yaml gets two v0.8 counterparts of test_v09_topology_plain_string_property_not_treated_as_child_ref. In both, a Text has usageHint: h1 and another component has the id h1.
    • test_v08_topology_plain_string_property_not_treated_as_child_ref expects the error for the orphan h1.
    • test_v08_topology_plain_string_property_does_not_close_cycle expects root -> h1 -> title to pass. If usageHint were read as a reference, it would close a cycle.
  2. core/validator_v0_9.yaml and core/validator_v1_0.yaml each get two cases. Both use a custom Stack whose slots property is an array of ComponentId. The property name matches no naming convention.
    • test_v09_topology_component_id_array_dangling_reference_error and its v1.0 counterpart expect a dangling-reference error when the only item names no component.
    • test_v09_topology_component_id_array_child_reachable and its v1.0 counterpart expect the component an item names to be reachable from the root.
  3. core/message_processor_scaling.yaml gets test_scaling_process_single_component_messages_composition_v10. It times n messages of one Text each into a root Column that names all n, under the relaxed config.

Results

Every harness in the table reads the shared suites from conformance/, so the new cases run without harness changes. In Dart:

  1. dart/a2ui_core/test/conformance/validator_conformance_test.dart runs validator_v0_9.yaml and validator_v1_0.yaml, so the new ComponentId array cases run in dart test.
  2. dart/a2ui_core/test/conformance/message_processor_scaling_conformance_test.dart runs message_processor_scaling.yaml, so the timing case runs in dart test -P performance. The dart_a2ui_core (stable, performance) job of Flutter CI runs that.

Python is the only harness this PR changes, to list the cases it fails on main as known gaps.

Harness v0.8 plain string ComponentId array Composition timing
Python core, main Both fail, listed as known gaps; #3326 fixes them Dangling cases fail, listed as known gaps, #3326 fixes them; reachable cases pass Not run: the harness skips the suite until #3326
Python core, #3326 Pass Pass Passes
TypeScript web core, main Pass Pass Not run: no timing runner until #3327
TypeScript web core, #3327 Pass Pass Fails: ratio 11.92; #3327 needs the change in its review
Dart core, main Not run: a2ui_core does not implement v0.8 Pass Passes since #3345 (ratio 14.9 before it)
Swift core Not run: no v0.8 validator cases Pass Not run: no timing runner
Kotlin legacy agent Pass Pass (v0.9 only) Not run: no timing runner

The timing numbers are median times on the same machine, at 1000 and 4000:

  1. TypeScript web core with perf(web_core): check updateComponents incrementally under deferred graph checks #3327: 32.8 ms and 390.4 ms.
  2. Dart core before perf(dart/core): check composition against what an update changes #3345: 302 ms and 4494 ms.

#3345 adds a Dart-only timing test for the same container, and says the case can move into the shared suite once #3326 and #3327 land. This PR adds it to the shared suite now. The Python core harness skips the whole suite on main, and the TypeScript harness has no timing runner there, so the case runs only in Dart until those PRs land. Each of them then needs to pass it or list it.

The #3345 shortcut applies only under a validation config that defers the graph checks. A processor with no config, or a strict one, still checks every reference on the surface for each message once a catalog declares constraints. The shared case uses the relaxed config, so it doesn't cover that path.

Known gaps

  1. Python core: on main, the topology checks read references using the ids the surface holds.

    • A v0.8 string property counts as a reference when its value names a held component.
    • An array of ComponentId counts only when one of its items names a held component.

    perf(python/core): check updateComponents incrementally under deferred graph checks #3326 reads both from the catalog alone and passes all four cases, so it should drop the four SKIP_TEST_NAMES entries added here.

  2. TypeScript web core: main skips the timing case, so nothing is listed. perf(web_core): check updateComponents incrementally under deferred graph checks #3327 adds a timing runner, which fails this case until it checks a parent's allowedChildren only against the components of the batch, as perf(python/core): check updateComponents incrementally under deferred graph checks #3326 does.

This PR is a draft because the v0.8 cases decide a question that #3326 raises: whether a property that is not a reference field becomes one when its value names a component. They follow the v0.9 and v1.0 cases and #3326: it does not. If main's reading is intended for v0.8, those two cases should go.

Testing

  1. uv run pytest python/a2ui_core/tests/test_conformance.py conformance/tests, after merging main: 642 passed, 17 skipped.
  2. In dart/a2ui_core, after merging main, dart test passes (1826 tests). dart test -P performance passes, the new case included, in three runs out of three.
  3. dart test in dart/a2ui_agent passes (1105 tests).
  4. node tests/conformance/conformance_test.mjs in typescript/core/web_core: passes.
  5. swift test --filter ValidatorConformanceTests in swift/core: passes.
  6. ./gradlew test --tests '*ConformanceTest*' in kotlin/agent_sdk_legacy: 173 tests pass.
  7. The perf(python/core): check updateComponents incrementally under deferred graph checks #3326 and perf(web_core): check updateComponents incrementally under deferred graph checks #3327 results come from their branches, run against these suites.

🤖 Agent-generated

…e core suites

Add v0.8 cases for a plain string property that names a component id,
v0.9 and v1.0 cases for an array property whose items are ComponentId,
and a timing case for processing messages with a catalog that declares
allowedParents and allowedChildren. List the cases the Python and Dart
cores fail on main as known gaps.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces several new conformance tests across different protocol versions (v0.8, v0.9, and v1.0) to validate topology rules, specifically ensuring that plain string properties are not incorrectly treated as child references and that array properties containing ComponentId items are properly validated. Additionally, a new scaling performance test and its corresponding test catalog have been added to measure message processing under composition constraints. Finally, the Dart and Python test suites have been updated to skip these new tests where they are currently known gaps. As there are no review comments, I have no additional feedback to provide.

@polina-c polina-c changed the title test(conformance): cover reference reading and composition cost in the core suites test(conformance): cover reference reading and composition cost; make the Dart composition check linear Oct 20, 2026
…hanges

MessageProcessor checked every reference on the surface against the
composition constraints for each updateComponents, so a surface sent as
n messages took time in proportion to n squared once a catalog declared
allowedParents or allowedChildren. It now keeps the references the last
check passed and checks a batch against the root, the references from
its components and the references into them, falling back to the
whole-surface check when the components changed outside the processor,
a batch moves a component to another catalog, or the reference fields or
constraints change. The composition timing case now passes.
Main now has #3345, which makes the Dart composition check incremental
under a deferring validation config. Take its Dart core files in place of
this branch's own fix, which did the same. The shared composition timing
case passes with it.
@polina-c polina-c changed the title test(conformance): cover reference reading and composition cost; make the Dart composition check linear test(conformance): cover reference reading and composition cost in the core suites Oct 20, 2026
@polina-c
polina-c marked this pull request as ready for review October 20, 2026 19:46
@polina-c
polina-c requested a review from nan-yu October 20, 2026 20:01
@nan-yu
nan-yu merged commit 346d9b1 into main Oct 20, 2026
46 of 47 checks passed
@nan-yu
nan-yu deleted the conformance-graph-cases branch October 20, 2026 20:41
@github-project-automation github-project-automation Bot moved this from Todo to Done in A2UI Oct 20, 2026
nan-yu added a commit that referenced this pull request Oct 20, 2026
#3344 added a dangling reference case and a reachable child case for
arrays of ComponentId to the v0.9 and v1.0 validator suites. Drop this
branch's duplicates of those two and keep its partial dangling and
circular reference cases. Python now passes main's dangling cases, so
they leave its skip list, and Kotlin passes main's v0.9 dangling case,
which checks only the missing id.

Signed-off-by: Nan Yu <nanyu@google.com>
nan-yu added a commit that referenced this pull request Oct 20, 2026
GitHub did not pick up the previous push (80bc113, which merges main and
resolves the conflict with #3344), so the pull request still showed the old
head as conflicting.

Signed-off-by: Nan Yu <nanyu@google.com>
nan-yu added a commit that referenced this pull request Oct 20, 2026
… test entry covers the fix

- Move the partial-dangling and circular ComponentId array cases to just
  after #3344's dangling and child-reachable cases in validator_v0_9.yaml and
  validator_v1_0.yaml, so the four cases sit together. The cases are
  unchanged.
- test_reference_map.py: say that `external` passed before the fix (only
  `#/` pointers are resolved) and that `slots` is the shape a loaded catalog
  has, which the fix covers.

Signed-off-by: Nan Yu <nanyu@google.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants