feat(python/agent)!: make the Parser stateless; always validate; add ResponseStream

[nan-yu]

Part of #3262 and #3263. Stacked on #3268.

Important

Like #3268, this PR needs the core change in #3326, whose branch is merged in through #3268. The core part of the diff disappears once #3326 lands.

The Parser read whole responses, which needs no state, and streamed responses, which needs the buffers of one response, on the same object. It also validated inside compile, so translation and validation could not be separated, and some paths skipped validation while others repeated it.

Parser

1. The parser holds no state. compile, decompile, wrap and unwrap only translate.
2. parse_response(content, wrapped, state=) compiles every block, applies them to a RendererState inside a transaction, and checks the trees of the surfaces they touch once every block is applied. A component may name a child that a later block adds. Without a state, the response is checked against a new, empty one, as a chat turn that shows new UI.
3. open_stream(wrapped=, state=) returns a ResponseStream for one response, with feed and close. Each message it emits is applied to the state, so the state matches what the client received even when the response fails part way. close emits text still held back as a possible tag, raises A2uiParseError if a block is still open, and checks the tree of an unwrapped stream.
4. The direct JSON stream (direct_json/response_stream.py) checks each message against its state instead of calling validate_payload for every message on every chunk, and no longer keeps its own map of surfaces to catalogs. The block reading it shares with the parser moves to direct_json/blocks.py.
5. Express, Elemental and Atom output is checked the same way as direct JSON output.

Reading each chunk once

A stream must take time in proportion to the response (blueprint, "Cost of a response"). The direct JSON stream used to scan the open block for the close tag from its start, read the whole open block again, and apply a growing message again whole, on every chunk. Now:

1. BlockScanner resumes the close-tag scan where the previous chunk stopped.
2. read_partial_message_items takes a PartialReadCache: a list or map that closed in an earlier read is taken from the cache, and an open list, the list of messages included, is read on from its last closed item. The reader ports the Dart reader from Fix performance and create conformance tests. #3314.
3. A message that has closed is not read, applied or checked again. A growing updateComponents applies only the components it gained, plus its last component again while that one is healed before v1.0 (_check_added); an id repeated across those batches is rejected as a duplicate. A growing updateDataModel is emitted each time its value changes and applied again once it closes (_check_grown).
4. When a block closes, the read goes on from the cache. Only a block that does not read as complete JSON is compiled whole, so that it raises as compile does (_final_messages).

Only updateComponents and updateDataModel (v0.8's surfaceUpdate and dataModelUpdate) are emitted while they grow. Any other message, callRendererFunction included, now waits for its object to close, and the messages after it wait with it, as the blueprint's DirectJsonResponseStream describes. Before, only createSurface, beginRendering and deleteSurface waited.

Streaming 40-character chunks with a v1.0 basic catalog, best of three:

payload	n=100	n=200	n=400	400/100
one message of n components, before	667 ms	2595 ms	10264 ms	15.4
after	42 ms	85 ms	193 ms	4.6
n one-component messages, before	895 ms	3582 ms	14312 ms	16.0
after	45 ms	97 ms	208 ms	4.7

The "before" numbers are this branch with #3268 and #3326 merged, so they measure the stream alone.

DirectJsonParser.compile now returns AgentToRendererMessage models rather than decoded mappings, as the blueprint's compile and the other formats do. A message that matches no protocol version's message schema raises A2uiValidationError there. Every caller in the stack already accepted models.

Removed

1. Parser.parse_chunk and validates_output, and the formats' parser property (use create_parser()).
2. validate_payload(..., surface_catalog_ids=...).
3. In a2ui-a2a, parse_stream_chunk_to_parts and the deprecated parse_response_to_parts. stream_response_to_parts takes a ResponseStream, and the new to_a2a_parts converts parsed parts. These move with the parser because they called parse_chunk.

Behaviour changes

1. An update to a surface the response does not create is now rejected, by A2uiRequestProcessor.parse_response and A2uiPartConverter too. A canvas agent passes the RendererState it keeps; the next PR lets the processor hold it. validate_payload still assumes such a surface exists.
2. Express, Elemental and Atom output that only passed because those formats skipped some checks is rejected: inline child ids that collide with assigned variables, check lists on components that declare none, a function call given to a property with a static type, and Elemental output whose top-level element has no id="root".

Conformance and evaluation

1. The shared conformance cases describe no renderer history. A streaming or parse_response case is read as one chat turn against a renderer that holds no surfaces, so a case that updates a surface creates it in its input: streaming cases start with a block that creates the surface, and the parse_response case creates it in the same block. The Python runner drives streaming cases through open_stream, feed and close.
2. Compile and decompile cases check a block on its own, the way validate_payload does, so a surface the block only updates is assumed to exist. The Python runner compiles the block and calls validate_payload.
3. Two streaming cases now expect an update to a surface the renderer doesn't hold to be held back rather than emitted.
4. The two v0.8 data model streaming cases begin rendering their surface in a first block, as the v0.9 cases create theirs, since a v0.8 response that never sends beginRendering is now rejected (feat(python/agent): add RendererState; build validate_payload on it #3268).
5. The agent harness runs the timing suites agent/direct_json/scaling.yaml and agent/express/scaling.yaml in test_scaling_conformance.py, marked performance, under the timing harness from perf(python/core): check updateComponents incrementally under deferred graph checks #3326. test_scaling_cases_have_handlers runs in the default run. A payload that updates a surface it does not create (the v0.9 healed case) streams into a state that holds that surface.
6. The Python evaluation bridge parses once, instead of parsing and then running a second strict MessageProcessor.

The Dart runner passes the updated cases as it is today. The current TypeScript stream processor now passes seven streaming cases it listed as known failures, so they come off its list.

The samples under samples/ still call parse_chunk and parse_response_to_parts; the sample PRs at the top of this stack move them over.

Testing

1. New test_response_stream.py and test_renderer_state_parsing.py cover chat and canvas turns, rollback of a failed response, streams that fail part way, close, and a block that names a child a later block adds. test_parse_chunk.py is replaced by them. New python/a2ui_a2a/tests/test_parts.py.
2. uv run pytest python/a2ui_agent/tests python/a2ui_a2a/tests python/a2ui_adk/tests samples/agent/adk/tests eval/tests: all pass except the TypeScript evaluation bridge tests, which fail the same way on main in my checkout (@a2ui/agent not resolvable from node_modules).
3. uv run mypy python/a2ui_agent/src python/a2ui_a2a/src: no issues.
4. TypeScript yarn test and Dart dart test still pass with the updated conformance YAML.
5. A new test streams a fenced block one character at a time.
6. New test_response_stream_reads.py checks that each closed message is applied once, that a growing message applies only what it gains, that a healed component is applied again only while it grows, that a data model update is applied when it first reads and once it closes, that a function call waits for its object to close, that a repeated id in a growing message is rejected, and that a close tag inside a string does not end the block however the response is cut.
7. uv run pytest python/ samples/agent/adk/tests: 2855 passed. uv run pytest -m performance python/a2ui_core/tests python/a2ui_agent/tests: 12 passed, including the 8 agent scaling cases. The old stream's ratios in the table above are over their limit of 8.
8. test_streaming_delete_surface.py, which main added against parse_chunk, is ported to open_stream.

[gemini-code-assist]

Code Review

This pull request refactors the A2UI parsing and streaming architecture to be stateless, introducing ResponseStream and DirectJsonResponseStream for incremental chunk-by-chunk processing and utilizing RendererState for validation. It also removes several deprecated streaming APIs across the a2ui_a2a and a2ui_adk packages. Feedback on the changes highlights a bug in clean_start where single backticks are incorrectly handled compared to triple backticks, and suggests a performance optimization in possible_tag_start to avoid O(N^2) string slicing by leveraging the pos parameter in re.match without the redundant ^ anchor.

[gspencergoog]

This PR makes the Python Parser stateless, moves catalog and topology checks out of compile into parse_response and the new open_stream/ResponseStream backed by a RendererState, and updates the a2ui.a2a helpers, the ADK converter, the eval bridge, the shared conformance suites and the tests. The streaming state machine and the transactional whole-response path hold up under review: a failed chunk rolls back to what was emitted, createSurface/deleteSurface are held until their object closes, and the open review threads from @ditman are addressed at 38c643f.

Requesting changes for one reason: the two rewritten "update before create" cases in response_streaming.yaml change the shared contract, and the Dart runner does not implement the hold-back, so dart_a2ui_agent (stable) fails on this PR and would stay red after the rebase onto main (the body says the Dart runner passes the updated cases). The sample breakage the body acknowledges is confirmed to be fixed by #3272 for the two workspace samples; the remaining notes below are about the gap between the two merges and one question on truncated blocks in canvas mode.

Broader Context & Non-Diff Observations

The following observations relate to surrounding or referenced code outside the active diff hunks:

• samples/agent/adk/restaurant_finder/agent.py:304 — Workspace samples import a removed symbol until #3272 lands

  samples/agent/adk/restaurant_finder/agent.py:42,304 and samples/agent/adk/custom-components-example/agent.py:42,528 import the removed parse_response_to_parts and pass a Parser to stream_response_to_parts, so they fail at import, and streaming would raise AttributeError, from the moment this PR merges until #3272 merges. ci-samples was skipped on this PR, so CI does not show it. The gemini_enterprise samples depend on published releases and are unaffected.

_{🤖 Agent-generated}

[ditman]

More nits. I think I've seen the update to the samples coming later in the stack, so that's probably not too bad!