Skip to content

feat(mcp): let a session-bound app tool hand back one capped PNG - #38331

Closed
chenmingwei23 wants to merge 1 commit into
mainfrom
feat/browser-fix-f4-tool-image
Closed

chenmingwei23 wants to merge 1 commit into
mainfrom
feat/browser-fix-f4-tool-image

Conversation

@chenmingwei23

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: A session-bound app tool can show the model one picture of the caller's own page beside its text answer.

Today every app tool answer reaches the model as text only. A browser app that changes a page for the agent has no way to let the model see the result, so the model edits blind and cannot check its own work.

Why it matters

Agents that drive a page in the caller's own session (Browser Collab style apps) need to look at what they made to fix layout mistakes. Without a picture they guess, and users see broken edits the model could have caught.

Not a goal

What changed (motivation -> approach -> change)

  • Goal -> the MCP transport already frames one answer per call through build_tool_response; the smallest change is to let that frame carry one MCP image block after the text.
  • Approach: a new module mcp_tool_image keeps the picture in a thread-local. app_agent_tool (strictSession branch only, after the post) calls attach() when a result: "text" tool answers exactly {"text", "image"}; the dispatch loop calls take() in the same worker thread when the call returns, so a picture can never ride on another call's answer. attach() refuses anything that is not a real PNG within 1,500,000 bytes and 2000 px per edge (read from IHDR), and refuses oversized base64 before decoding; a refused picture is dropped and the text goes alone. An error answer never carries a picture. The gateway relay image budget still applies on top.
  • Docs: one sentence each in the manifest reference and the app-kit platform spec on the {text, image} answer.
  • Note: feat(apps): list a third-party app's agentTools on an opt-in prompted kirocrew-apps server #17015 also edits mcp_tools/app_tools.py and the same two docs. A trial merge against its head shows no conflict; whichever PR lands second rebases.

Backwards compatibility

Compatible: an answer of {"text": ...} and every JSON answer behave as before; {"text", "image"} was previously relayed as its JSON and is only reachable from a strictSession tool, which no shipped app sends today.

Removes nothing: the change only adds an optional image block to the tool response frame.

Tests

  • test_mcp_tool_image.py: a PNG inside the caps is kept once; over the byte cap is refused; over the edge cap or not a PNG is refused; a refused picture replaces an earlier one; an error answer carries no picture; the picture rides only on the call that attached it (two calls in the dispatch loop).
  • test_app_agent_tools.py: a strictSession text tool hands back one picture; a tool that is not session bound cannot hand back a picture; an oversized picture sends the text alone.

Proof: not provable -- other: reverting all hunks deletes the new module so collection errors; reverting the app_tools hunk alone fails 2 of the 3 picture tests, and reverting the dispatch hunk alone fails the one-call-only test.

Ran serially: test_mcp_tool_image.py and test_app_agent_tools.py, 108 passed. flake8, isort and scripts/docs_lint.py clean.

Manual verification

N/A -- unit coverage sufficient: the change is a backend response frame with no UI; the end-to-end use comes with the browser app that sends pictures.

Related Issues

N/A

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

Contribution License Agreement

Contributions are accepted under the project's Apache-2.0 license.

@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Nov 17, 2026
@github-actions

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision 4e060b450bd0ea3b25db97ce5ee7f9b56e6ea4e3 touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of 4e060b450bd0ea3b25db97ce5ee7f9b56e6ea4e3 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

Nothing blocks: the picture goes out through the one response exit, it is cleared in the same worker thread's finally, and the patch removes nothing.

Notes

  • The image is passed to the model without redaction, unlike the text beside it. Say in the app-kit platform spec that a picture is untrusted app content and is not redacted, so the next reader does not assume it gets the same treatment as text.

[DESIGN-REVIEWED] 4e060b4

Description read

[DESCRIPTION-READ] 59b5176cf20afa1910ed4a4d784b46e652f8d4569f85fd2f63d2188805322225

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — 🔴 BLOCK

Premise-level review of 4e060b450bd0ea3b25db97ce5ee7f9b56e6ea4e3 — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

First-Principles-Verdict: BLOCK

The PR adds an image block to build_tool_response, the one exit every tool result goes through. A recorded design decision says that exit is text only, and the description never mentions overturning it. It also opens a public manifest answer shape that no app uses.

Notes

  • Item 3 — symptom-level: the thread-local picture slot runs on every tool call in the shared stdio loop, but it exists for one tool branch. Subtraction: goes away with Blocker 1.

What this change ships

Inventory (5 items) — 0 justified

Intent: let an app tool that acts on the caller's own page show the model a picture of the result. This is an ADDITION.

  1. A strictSession text tool can answer {text, image} and the model gets one PNG — duplicate of mcp_panel_snapshot.py (path relay)
  2. Every MCP tool response frame can now carry an image block — undeclared reversal of computer-use.md:444
  3. The dispatch loop takes a thread-local picture after every tool call — symptom-level
  4. A PNG is refused if it is over 1.5 MB or wider or taller than 2000 px — zero consumers
  5. The manifest reference documents the {text, image} answer — zero consumers

Blockers

1. A recorded transport decision is reversed without being mentioned — undeclared, and a duplicate of mcp_panel_snapshot.py
Trigger: computer-use.md:452-455 reads "There is no isError field and no image block — an image block is not expressible on this transport, so 'tree-first, relay the screenshot as a path' is a property of the transport rather than a policy someone can regress." The same doc measured one inline screenshot at about 109K tokens. mcp_panel_snapshot.py:6-8 already shows "a picture of the crewmate's own page" by passing a file path. The diff adds frame["content"].append(content_block(image)). The description says only "the smallest change is to let that frame carry one MCP image block" and leaves that spec untouched.
Outcome: the transport stops guaranteeing text only, and the spec now says something false. The up to 1.5 MB (about 2 MB base64) image rides inside the model's context instead of staying a path.
Subtraction: delete mcp_tool_image.py and the image parameter. The app returns a PNG path in its text, the way dashboard_snapshot does.
Clears when: the image block is removed and the picture travels as a path, or the description names the decision it reverses and gives the evidence that the path approach fails for app tools.

2. A public manifest answer shape ships with no app using it — zero consumers
Trigger: the description says {text, image} "is only reachable from a strictSession tool, which no shipped app sends today". The patch has no non-test caller that sends image, and manifest-reference.md makes the shape part of the public App Kit contract.
Outcome: the core has to keep honoring a public answer shape that nothing uses.
Subtraction: drop the {text, image} branch in app_tools.py and the manifest sentence, and add them back with the browser app that needs them.
Clears when: the branch and the doc row are removed, or a real app sending {text, image} lands alongside them.

[FIRST-PRINCIPLES-REVIEWED] 4e060b4

Description read

[DESCRIPTION-READ] bab3d0c6d9f0d4c1016a682928251c457042c2306ec32dcb286497291c5ffb27

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — 🔴 changes requested (blocking)

Reviewed 4e060b450bd0ea3b25db97ce5ee7f9b56e6ea4e3 — this comment is updated in place on each push.

A test leaves a picture in the shared per-thread image slot, so a neighbouring test's take() is None check fails whenever the two run in reverse order, such as in a shuffled run.

BLOCKING — test/test_mcp_tool_image.py:68
assert mcp_tool_image.attach(_png(pad=mcp_tool_image.MAX_PNG_BYTES - 200))
Trigger: this is the last line of test_a_png_over_the_byte_cap_is_refused, and the test never calls take() afterwards. The picture stays in the module-level _local = threading.local() slot on the worker's main thread, and no conftest fixture clears it. If test_app_agent_tools.py::test_a_tool_that_is_not_session_bound_cannot_hand_back_a_picture runs later on the same worker, as in the shuffled-order proof that testing-conventions requires, its non-strict branch never calls attach. So assert mcp_tool_image.take() is None (test_app_agent_tools.py:824) reads the leftover picture.
Outcome: that test fails or passes depending on which tests ran before it on its worker. This is the order-dependence the rule forbids.
Anchor: tests-are-deterministic
Fix: end the byte-cap test with assert mcp_tool_image.take() is not None, or add a fixture to both files that clears the slot (mcp_tool_image.take()) before and after each test.

[BLOCK-MERGE] 4e060b4
[OPUS-REVIEWED] 4e060b4

Verdict parsed from the review's SHA-scoped output markers for commit 4e060b450bd0ea3b25db97ce5ee7f9b56e6ea4e3.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 4e060b450bd0ea3b25db97ce5ee7f9b56e6ea4e3 false-positive: <reason, 20 to 500 characters>

@github-actions

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — 🔴 changes requested (blocking)

GPT 6.1 found at least one blocking issue that must be resolved before merging 4e060b450bd0ea3b25db97ce5ee7f9b56e6ea4e3. Adjudication upheld 1 of 1 blocking finding(s); those must be resolved before merging.

This comment is updated in place on each push.

BLOCKING -- test/test_mcp_tool_image.py:68 -- The byte-cap test leaks an image into subsequent tests

assert mcp_tool_image.attach(_png(pad=mcp_tool_image.MAX_PNG_BYTES - 200))
Trigger: Run this test immediately before test_a_tool_that_is_not_session_bound_cannot_hand_back_a_picture on the same serial pytest worker. The final attach() leaves the thread-local image populated; neither the fixtures nor the subsequent direct, non-strict app_agent_tool() call clears it.
Outcome: The subsequent test fails its take() is None assertion despite correctly refusing the picture, making the suite order-dependent.
Anchor: tests-are-deterministic
Fix: Consume the successfully attached image with take() before this test returns.

[BLOCK-MERGE] 4e060b4
[GPT-REVIEWED] 4e060b4

Adjudication (Opus 5.5) — is blocking on each finding proportionate?

F1: MEDIUM rung, a test that passes or fails depending on run order. Confirmed in the code: test/test_mcp_tool_image.py:68 ends on a successful attach() with no take() after it. src/kiro_crew/mcp_tool_image.py:53 stores the image on a thread-local that persists between tests on the same worker thread. test/test_app_agent_tools.py:823 asserts take() is None on that thread, and its error path never calls attach(), so it would read the leftover image. The finding is anchored to tests-are-deterministic, which carries blocking: true in .review-base-rules/AUTOSDE.yaml:376. The real fix is one take() line.

[ADJUDICATION] 4e060b450bd0ea3b25db97ce5ee7f9b56e6ea4e3 total=1 uphold=1 downgrade=0
UPHOLD F1 test/test_mcp_tool_image.py:68 reason=autosde-blocking-rule
[GPT-ADJUDICATED] 4e060b450bd0ea3b25db97ce5ee7f9b56e6ea4e3

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 4e060b450bd0ea3b25db97ce5ee7f9b56e6ea4e3 false-positive: <reason, 20 to 500 characters>

@chenmingwei23

Copy link
Copy Markdown
Contributor Author

Withdrawing this PR per the First Principles review: tool pictures keep the existing approach of writing a PNG file and returning its path (as dashboard_snapshot and computer-use do), so no inline image block is needed on the MCP transport.

@github-actions github-actions Bot removed the readiness: checking Automated validation is still running label Nov 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant