Repository navigation
Conversation
…:timing Add @a2ui/conformance-timing, a private workspace that runs the measure_growth and measure_overhead cases of the conformance suites: it reads a suite, writes a case's payload at a size, times each operation after one untimed run and compares the medians. A package supplies the operation to time, so typescript/agents/a2ui_agent can run its scaling suites through it too. web_core runs conformance/core/message_processor_scaling.yaml with yarn test:timing, which yarn test leaves out, and the Web CI workflow runs it in a job of its own. Both cases fail on main: processing a surface sent as one message per component takes time that grows with the square of the surface. Signed-off-by: Nan Yu <nanyu@google.com>
…raph checks Under a validation config that allows dangling references and orphans or a missing root, as the agent's RendererState configures it, MessageProcessor no longer copies every component of the surface into a candidate model and checks the whole surface on each updateComponents. It keeps what the last check found (each component's references, its parents, and a bound on the longest chain below it), as the Dart SurfaceComponentsModel does since #3314, and checks cycles, depth and the composition constraints of the components the message affects. When the record cannot show the message passes, or the components changed outside the processor, it checks the whole surface as before, so a failing message reports the same error. Strict configs and no config keep the whole-surface checks; the composition check returns early when no catalog involved declares allowedParents or allowedChildren, which it would not check anyway. Signed-off-by: Nan Yu <nanyu@google.com>
There was a problem hiding this comment.
Code Review
This pull request introduces a new private workspace package, @a2ui/conformance-timing, to run performance and timing conformance tests for TypeScript packages. It optimizes the MessageProcessor to perform incremental topology and composition checks on updated components rather than re-validating the entire surface, reducing the processing time from quadratic to linear under deferred validation configurations. To support this, ComponentGraphBound and ComponentsWatcher are introduced. Feedback suggests optimizing the DFS implementation in ComponentGraphBound.of to avoid O(N^2) worst-case child lookups by tracking the child index on the stack, achieving linear O(V + E) time complexity, and replacing a non-null assertion with a safer nullish coalescing operator.
publish_npm added every internal workspace dependency, including devDependencies, to the publish set. web_core now has a devDependency on the private @a2ui/conformance-timing workspace, so the script tried to version-check and publish it. Private packages are never published, so skip them when they are development-only dependencies and fail clearly if a published package needs one at runtime. Signed-off-by: Nan Yu <nanyu@google.com>
…te where the SDKs differ - a2ui_core: RendererState's atomic application undoes a rejected message from an undo log rather than rolling back to a copy. Requirement 4 now states only what a processor must do under a ValidationConfig that defers the graph checks, with no SDK status. - a2ui_core: the agent example shows the agent using RendererState (apply, then checkTopology) and, separately, the MessageProcessor calls RendererState makes inside, since an agent does not call MessageProcessor itself. - a2ui_agent: "Where the SDKs differ" assumes the pending Python (#3326, #3268-#3271) and TypeScript (#3327, #3278, #3279) pull requests. Both meet the three requirements of "Cost of a response"; the remaining entries are Dart's, plus Python being the only SDK with a v0.8 state. Signed-off-by: Nan Yu <nanyu@google.com>
…se-stream #3327 moved the shared timing runner from the @a2ui/conformance-timing workspace into typescript/core/web_core/tests/timing/timing.mjs. The agent's timing tests now import it from there (a test-only import), and the agent drops its devDependency on the removed workspace.
polina-c
left a comment
There was a problem hiding this comment.
I checked out the branch and ran yarn test in web_core (993 pass) and yarn test:timing (ratios 4.09 and 4.73). I also tried the incremental path on cases the timing suite doesn't cover.
The bound and the fallback to the whole-surface check look right, and the differential tests cover what I would have asked for. One case is still quadratic: when a catalog declares allowedChildren, a message that adds a child to a large container reads every child of that container. The inline comment has the numbers. Python #3326 is linear on the same case.
This PR and #3326 both rewrite the same sentence of conformance/README.md, and each names the other SDK as the one with no runner. Whichever lands second should drop that example.
🤖 Agent-generated
…eferred graph checks A component outside the batch that references a component of the batch is now checked against only the children of the batch it references, as the Python core does with its holding index, so a container whose catalog declares allowedChildren, receiving one child per message, is no longer checked against all its children each time. Adds an SDK-local timing case for it under yarn test:timing. docs(conformance): end the timing runner sentence without naming the SDK that has no runner, so #3326 and #3327 make the same change there.
Restores the known component ids in validate_component_integrity, analyze_topology and SurfaceComponentsModel.get_child_references, so the legacy v0.8 heuristic reads the same references as on main, and restores _AnyId for ComponentGraphBound, whose topology edges over-approximate the ones the whole-surface check reads. Composition keeps reading the references without known ids, as on main, in a map of its own. Keeps the cleanup of empty parent sets from 7980442. Drops the reference_map branch that read an array of ComponentId items as a list of references; it changed the reference map under every config and is not needed for the incremental check. Adds tests for the v0.8 usageHint cases from review, a Slots type to the differential test, and a performance test for a container with allowedChildren receiving one child per message. docs(conformance): end the timing runner sentence without naming the SDK that has no runner, so #3326 and #3327 make the same change there.
…mposition cases Adds the allowedChildren case to conformance/core/message_processor_scaling.yaml (a Column whose catalog allows only Text children, receiving one child per message under the relaxed config, at 500 and 2000 children) with its catalog fixture, so the Python, TypeScript and Dart cores time the same payload. It replaces tests/timing/composition_scaling_test.mjs, which timed the same scenario. Ports the named composition cases of the Dart core's incremental check (#3345) as tests comparing the incremental and whole-surface checks, and adds a random comparison over two catalogs that both declare constraints on the same type names. Signed-off-by: Nan Yu <nanyu@google.com>
|
Thanks for the review. The |
Signed-off-by: Nan Yu <nanyu@google.com> # Conflicts: # conformance/core/message_processor_scaling.yaml
polina-c
left a comment
There was a problem hiding this comment.
I pulled 3a3a4cb and ran yarn test (1010 pass), yarn test:timing (4 pass), yarn lint (no errors) and yarn format:check in web_core, and yarn test in @a2ui/agent (764 pass).
The allowedChildren case from my last review is fixed. On test_scaling_process_single_component_messages_composition_v10 at 1000 and 4000, forcing the whole-surface check before every message gives a ratio of 17.4. This branch gives 3.8.
I also compared this branch with a processor that drops its record before every message, on random input. It used two catalogs with different constraints on the same type names, batches of up to three components, and four deferring configs. Every so often a component was added, removed or changed outside the processor. Over 144,000 steps, about 70% of them with a recorded bound, the errors and the components held matched every time.
One thing to fix before merge: the runner's own tests in tests/timing/timing.test.mjs time real spins inside yarn test, and they fail when the machine is busy (inline). The other two comments are small doc fixes.
Two things in the PR description:
-
"The shared case still lets it replaces the web-core-only
composition_scaling_test.mjs" is missing words. -
The "Numbers" tables would be easier to read as a line naming the measurement and sizes, then a before/after list. For example:
test_scaling_process_single_component_messages_relaxed_v10, median of five runs at 200 and 800:- before: 136.6 ms and 1890.9 ms, ratio 13.84 (fails)
- after: 12.3 ms and 52.7 ms, ratio 4.29
🤖 Agent-generated
compareMedianTimes takes the clock as a `now` option, defaulting to performance.now(), and measureGrowth and measureOverhead pass it through. The runner's unit tests, which `yarn test` runs alongside every other workspace's tests, advance a fake clock instead of spinning, so their times are exact under any load. Adds tests for the median and for measureOverhead. Updates the README and CHANGELOG now that tests/timing holds no timing cases of the web core's own. Signed-off-by: Nan Yu <nanyu@google.com>
Signed-off-by: Nan Yu <nanyu@google.com> # Conflicts: # typescript/core/web_core/CHANGELOG.md
Part of #3315
Under a
ValidationConfigthat defers the graph checks (it allows dangling references, and orphans or a missing root, as the agent'sRendererStateconfigures it), the web coreMessageProcessorcopied every component of the surface into a candidate model and checked the whole surface on everyupdateComponents. A surface sent as n messages of one component each therefore took time that grew with n squared. This PR checks each message against the components it changes, following the Dart approach from #3314, and adds a runner for the conformance timing cases so the TypeScript packages can check this.It implements requirement 1 of "Cost of a response" in the consolidated blueprint (#3283,
a2ui_agent.blueprint.md) and item 4 of the agent use ofMessageProcessorina2ui_core.blueprint.md. Once this merges, the TypeScript half of entry 2 of "Where the SDKs differ" can be removed.What changed
component-graph-bound.tsis a port of the DartComponentGraphBound(component_graph_bound.dart). It records each component's references, the components that reference it, and an upper bound on the longest chain below it.admits(batch)updates the record with a batch and raises the bounds of the components above the ones the batch changed. If no bound passesmaxDepth, the graph has no cycle and no chain that is too deep. A self-reference, or a bound past the limit, means the record cannot show that the batch passes.validation-helpers.tsextracts the component graph, candidate topology, and composition constraint validation helpers out ofmessage-processor.ts. Under a deferring config,validateComponentGraphchecks a batch against the record left by the previous message. The cycle and depth checks go throughadmits. TheallowedParentsandallowedChildrenchecks run only on the components whose parents, children or types the batch changes (compositionHolds). A component of the batch is checked against its parents and all of its children. A child it references before or after the batch, outside the batch, is checked against its parents only. A component outside the batch that references components of the batch is checked against only those children, as the Python core'sholdingindex does. A container whose catalog declaresallowedChildren, receiving one child per message, is then checked against one child per message rather than all of them. Duplicate ids and the per-component catalog checks run on the batch as before. The record is saved only after the batch has been applied.validateComponentGraphfalls back to the whole-surface check in four cases: there is no record yet, the components changed outside the processor since the record was saved, the record cannot show that the batch passes, or a component in the batch has no type or comes from a catalog that declares no components (the topology and composition checks read references differently there). A message that fails therefore reports the same error as onmain, and a message is still checked whole before it changes anything. AComponentsWatchercounts changes through the components model'sonCreated,onDeletedandonUpdatedevents. It is attached only under a deferring config.a2ui-lit-element.test.tschecks this. One shortcut applies to every config: the composition check returns early when no catalog that the surface's components come from declaresallowedParentsorallowedChildren, because the check would find nothing in that case. To find a component added from outside the processor with a catalog of its own, the early return still looks at each component's catalog. That costs one set lookup per component, much less than the reference extraction it skips.typescript/core/web_core/tests/timing/timing.mjsruns themeasure_growthandmeasure_overheadactions ofconformance_schema.json. It reads a suite, writes aPayloadTemplateat a size, runs each operation once untimed and then five times in turn (collecting garbage first under--expose-gc), and compares the medians againstmax_ratio.conformance/core/message_processor_scaling.yamlwithyarn test:timing.yarn testdoes not include it. A newci-timingjob inweb_ci.ymlrunstest:timingfor each workspace in its matrix, which holds@a2ui/web_corefor now. The default conformance harness still skipsmeasure_growth, with a reason that points totest:timing. The composition case of the suite is main'stest_scaling_process_single_component_messages_composition_v10, added by test(conformance): cover reference reading and composition cost in the core suites #3344 with its catalogcomposition_catalog_v1_0.json: aColumnnames n children, which then arrive one per message, at n=1000 and 4000. It is the scenario of theallowedChildrencase this PR first added, so after merging main that case and itsallowed_children_catalog_v1_0.jsonfixture are dropped, the same way in perf(python/core): check updateComponents incrementally under deferred graph checks #3326. The shared case replaces the web-core-onlycomposition_scaling_test.mjs, which timed the same scenario. Python and TypeScript both pass it (TypeScript ratio about 2.5 at 1000 and 4000).Numbers
These were measured on the same machine with Node 22.22.1, using the median of five runs.
test_scaling_process_single_component_messages_relaxed_v10(yarn test:timing), at 200 and 800:test_scaling_process_single_component_messages_v10(no config), at 200 and 800:Before this change the no-config case was quadratic too, because the composition check walked every component on every message even when no catalog declares any constraints.
test_scaling_process_single_component_messages_composition_v10(aColumnwhose catalog declaresallowedChildrenreceives one child per message, relaxed config), at 1000 and 4000, as measured in review. "Before" forces the whole-surface composition check on every message:n
updateComponentsmessages of oneTexteach on one surface, after a rootColumnthat names all n children, with each message passed to its ownprocessMessagescall, at n = 100, 200 and 400:RendererState: dangling references, missing root and orphans allowed, unknown elements rejected), before: 39.1 ms, 125.6 ms and 406.9 msTesting
component-graph-bound.test.tscovers the bound itself and processor cases where only the incremental check would catch the error: a cycle closed by a later message, a self-reference, a chain pastmaxDepthbuilt over several messages, a reference added to the surface outside the processor, a child placed under a disallowed parent, a parent that changes type, and a component left without the parent its type requires. It also runs a differential test on random message sequences, under three deferring configs and with and without composition constraints. Each sequence is checked against a processor whose record is dropped before every message, and the two must accept and reject the same messages with the same errors and end up holding the same components. I checked that removing the parents or the previous children from the set of affected components makes these tests fail.A second group ports the named composition cases of the Dart core's incremental check (perf(dart/core): check composition against what an update changes #3345), on Dart's catalog, under its two configs: a child that arrives after its container, a container that arrives after its child, a child added to a container in a later message, a container rewritten to drop a child it disallows, a held child whose type changes, a holder its child's
allowedParentsomits, and a root the surface may not hold. Each step is sent to the incremental processor and to one that checks the whole surface, and both must give the same error class, code and message and hold the same components and catalogs. A random comparison runs over two catalogs that declare different constraints on the same type names, with each component drawn from either. MakingcompositionHoldsalways pass, or dropping itsholdingormovedchecks, makes these tests fail (23, 9 and 9 failures).yarn test(1014 tests at d768f18, conformance harness included),yarn test:timing(4 passed),yarn lint(no errors) andyarn format:checkintypescript/core/web_core.yarn testruns, advance a fake clock passed ascompareMedianTimes'snowoption instead of timing real work, so their times are exact. They passed 30 of 30 runs with 72 busy-loop processes on 24 cores, and 30 of 30 idle.yarn testin@a2ui/markdown-it,@a2ui/angularand@a2ui/agent, andyarn test:unitin@a2ui/litand@a2ui/react. After theallowedChildrenchange, and again at d768f18, I re-ran@a2ui/agentonly (764 passed at d768f18).