Repository navigation
feat(mcp): let a session-bound app tool hand back one capped PNG - #38331
chenmingwei23 wants to merge 1 commit into
Conversation
UX Review (Opus 5.5) — ⏭️ skippedRevision |
Design Review (Opus 5.5) — ✅ PASSDesign-level review of Design-Verdict: PASS Nothing blocks: the picture goes out through the one response exit, it is cleared in the same worker thread's Notes
[DESIGN-REVIEWED] 4e060b4 Description read[DESCRIPTION-READ] 59b5176cf20afa1910ed4a4d784b46e652f8d4569f85fd2f63d2188805322225 That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
First Principles Review (Opus 5.5) — 🔴 BLOCKPremise-level review of First-Principles-Verdict: BLOCK The PR adds an image block to Notes
What this change shipsInventory (5 items) — 0 justifiedIntent: let an app tool that acts on the caller's own page show the model a picture of the result. This is an ADDITION.
Blockers1. A recorded transport decision is reversed without being mentioned — undeclared, and a duplicate of 2. A public manifest answer shape ships with no app using it — zero consumers [FIRST-PRINCIPLES-REVIEWED] 4e060b4 Description read[DESCRIPTION-READ] bab3d0c6d9f0d4c1016a682928251c457042c2306ec32dcb286497291c5ffb27 That sha256 names the PR description this verdict read. Recompute it with |
Opus 5.5 Review — 🔴 changes requested (blocking)Reviewed A test leaves a picture in the shared per-thread image slot, so a neighbouring test's BLOCKING — test/test_mcp_tool_image.py:68 [BLOCK-MERGE] 4e060b4 Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
GPT 6.1 Review — 🔴 changes requested (blocking)GPT 6.1 found at least one blocking issue that must be resolved before merging This comment is updated in place on each push. BLOCKING -- test/test_mcp_tool_image.py:68 -- The byte-cap test leaks an image into subsequent tests
[BLOCK-MERGE] 4e060b4 Adjudication (Opus 5.5) — is blocking on each finding proportionate?F1: MEDIUM rung, a test that passes or fails depending on run order. Confirmed in the code: False positive or not applicable? A repository writer can comment: |
|
Withdrawing this PR per the First Principles review: tool pictures keep the existing approach of writing a PNG file and returning its path (as dashboard_snapshot and computer-use do), so no inline image block is needed on the MCP transport. |
Problem / Motivation
Goal: A session-bound app tool can show the model one picture of the caller's own page beside its text answer.
Today every app tool answer reaches the model as text only. A browser app that changes a page for the agent has no way to let the model see the result, so the model edits blind and cannot check its own work.
Why it matters
Agents that drive a page in the caller's own session (Browser Collab style apps) need to look at what they made to fix layout mistakes. Without a picture they guess, and users see broken edits the model could have caught.
Not a goal
strictSession(third-party apps, process-backed apps, plain core tools).shape_answer) that feat(apps): list a third-party app's agentTools on an opt-in prompted kirocrew-apps server #17015 adds.What changed (motivation -> approach -> change)
build_tool_response; the smallest change is to let that frame carry one MCP image block after the text.mcp_tool_imagekeeps the picture in a thread-local.app_agent_tool(strictSession branch only, after the post) callsattach()when aresult: "text"tool answers exactly{"text", "image"}; the dispatch loop callstake()in the same worker thread when the call returns, so a picture can never ride on another call's answer.attach()refuses anything that is not a real PNG within 1,500,000 bytes and 2000 px per edge (read from IHDR), and refuses oversized base64 before decoding; a refused picture is dropped and the text goes alone. An error answer never carries a picture. The gateway relay image budget still applies on top.{text, image}answer.mcp_tools/app_tools.pyand the same two docs. A trial merge against its head shows no conflict; whichever PR lands second rebases.Backwards compatibility
Compatible: an answer of
{"text": ...}and every JSON answer behave as before;{"text", "image"}was previously relayed as its JSON and is only reachable from a strictSession tool, which no shipped app sends today.Removes nothing: the change only adds an optional image block to the tool response frame.
Tests
test_mcp_tool_image.py: a PNG inside the caps is kept once; over the byte cap is refused; over the edge cap or not a PNG is refused; a refused picture replaces an earlier one; an error answer carries no picture; the picture rides only on the call that attached it (two calls in the dispatch loop).test_app_agent_tools.py: a strictSession text tool hands back one picture; a tool that is not session bound cannot hand back a picture; an oversized picture sends the text alone.Proof: not provable -- other: reverting all hunks deletes the new module so collection errors; reverting the app_tools hunk alone fails 2 of the 3 picture tests, and reverting the dispatch hunk alone fails the one-call-only test.
Ran serially: test_mcp_tool_image.py and test_app_agent_tools.py, 108 passed. flake8, isort and scripts/docs_lint.py clean.
Manual verification
N/A -- unit coverage sufficient: the change is a backend response frame with no UI; the end-to-end use comes with the browser app that sends pictures.
Related Issues
N/A
Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)Contribution License Agreement
Contributions are accepted under the project's Apache-2.0 license.