Skip to content

feat(mcp): add an owner-only Try a tool panel behind mcp.tool_tester - #37807

Open
javenciu wants to merge 1 commit into
mainfrom
feat/mcp-try-a-tool
Open

javenciu wants to merge 1 commit into
mainfrom
feat/mcp-try-a-tool

Conversation

@javenciu

@javenciu javenciu commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: the owner can run one tool of a configured MCP server from the MCP Servers page and see its raw result, without a chat turn.

POST /api/mcp/probe lists a server's tools, but no route calls one. Today tools/call runs only through an agent turn, behind approvals, and the owner reads the model's summary instead of the result.

Why it matters

Someone who has just added or fixed a server needs to know whether a tool works with exact arguments. A chat turn costs model time and may pick others.

Not a goal

  • Remote (url) servers: the run starts a server's command, so only local ones are listed.
  • A new agent approval path, or a server kept running between presses.

What changed (motivation → approach → change)

Round 9 (head 5aa6a6c): a timed-out run's backstop waits 20 s past the run's limit, so the probe's own teardown (kill, group reap, temp sweep) is never cancelled partway, and the run's limit also bounds the handshake reads; raw JSON refuses a number too large to be finite (1e400 would be sent as null); a Windows line-ending assertion is fixed. Product code is about 850 lines; a split puts the setting, probe step, route and sink row (about 590) first and the panel (about 260) second.

  • mcp.tool_tester (config/integration_sections.py, section_builders.py, config_all_settings.py): off by default, only a real true turns it on, read per request, so no restart is needed.
  • mcp_discovery_probe.py: probe_server takes an optional tool_call step that sends one tools/call after a good tools/list on the same pipe, so a run shares the probe's sandboxed spawn, handshake, cancel forwarding and finally, which reaps the whole process group. A cancel queued while the step chooses lands before anything is written. The handshake and the answer are read against the run's 30-second limit; meanwhile stderr is drained, keeping its first 32 KiB as a probe does, so a verbose server still answers. A plain probe is unchanged (a test that passes on main pins it).
  • dashboard/routes/agent_config.py: POST /api/mcp/try-tool is always registered through a small owner-gated handler that reads the setting off the loop, answers 404 while it is off, and imports the tester module only once it is on, so start-up reads no config.
  • dashboard/handlers/mcp_try_tool.py (new): lists tools or calls one once through that step. A tool name the redactor would change is counted as withheld, never listed or run (403 tool_name_withheld when asked for). A schema that is not an object, is over the size cap or stops being JSON once redacted is sent as null, not {}. A tool not marked readOnlyHint: true needs confirm: true; a tool or server switched off on the page is refused. A tool the governance ceiling or the profile bound to the dashboard surface denies is refused with 403 governance_denied (gate_decision for @server/tool in the mcp scope, asked under dashboard:mcp-try-tool, which classifies as dashboard); a check that cannot run refuses with 503. Every run writes a critical security event before the call and one with the outcome; if the first cannot be written nothing runs. Every error carries a code; output is dumped, redacted and then cut to 65,536 characters in a worker thread; the module is a registered redaction sink.
  • website/src/pages/overview/McpTryTool.tsx (new, mounted in McpTab.tsx): the panel. For a tool that can change data the button reads "Run tool…" with "Can change data. You'll be asked to confirm." beside it; a read-only tool keeps "Run tool". A run in flight shows the button busy, not switched off. A tool error leads with the server's own text. A schema using allOf, anyOf, oneOf, not or $ref, or a null one, gets the raw JSON box. A line counts withheld tools; a failed run shows "The run did not finish." above the server's text. The first tool not switched off is selected (none when all are). An error, hint, answer or confirm does not outlive its tool or server.
  • Catalogs in 12 languages and the pseudo-locale; docs/configuration.md; regenerated config-baseline.json.

needs a confirm, before the press, light
the inline confirm, light
needs a confirm, before the press, dark
result, light
tool error with the server's text, light
raw JSON box, dark
run failed, titled notice, light
run failed, titled notice, dark
names withheld, light
discarded schema gets the raw box, dark
first state, light
first state, dark

Backwards compatibility

With the setting off (the default) the route answers as an unknown path does (a test that passes on main pins it), start-up does no new work, and the panel is not drawn. A probe without the new step behaves as on main.

Tests

  • Before (round 9 tests on the round 8 source): a server that closes its stdout and ignores its closed stdin is still running after the route answers 504; a handshake read waits the raised 60 s probe timeout; {"value":1e400} goes out as null.
  • After: 61 backend and 33 panel tests pass, three runs in a row. Related suites (MCP discovery and probe, security posture, facade, error codes, body guard, owner gate, config, feature map): 1,305 passed. Whole backend suite: 275,827 passed; its failing ids are main's own. src/i18n/: 1,262 passed.
  • Reproduce: python -m pytest test/test_mcp_try_tool.py -n0 and npx vitest run src/pages/overview/McpTryTool.test.tsx
  • Proof: PROVEN test/test_mcp_try_tool.py::test_a_timed_out_run_is_answered_only_once_its_teardown_stopped_the_server test/test_mcp_try_tool.py::test_a_server_that_never_answers_the_handshake_stops_at_the_run_limit (head 5aa6a6c)

Manual verification

The real panel on a loopback vite page with fake answers, light and dark: every state above.

Sibling sweep

Sibling: dashboard/handlers/mcp_probe.py -- not affected -- no tool step
Sibling: mcp_gateway/app_call.py -- not affected -- it asks the same governance question

Pattern harvest

Rule candidate: review-prompt
Pattern: a new way to invoke a tool asks the governance question under its own surface's session key.

Related Issues

None filed. Searched open PRs and issues for "try a tool", "tool tester" and "mcp probe".

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

Contribution License Agreement

@javenciu
javenciu requested review from a team and KiroCrewCaptain as code owners November 16, 2026 04:30
@javenciu
javenciu requested a review from mbriones98 November 16, 2026 04:30
@javenciu
javenciu enabled auto-merge (squash) November 16, 2026 04:30
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Nov 16, 2026
@github-actions

github-actions Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of 5aa6a6ca915469b472a726474268debff8c8ea5d — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

Nothing blocks: the feature is additive and off by default, the run reuses the probe's sandboxed spawn and teardown, and governance is still checked under the dashboard surface.

[DESIGN-REVIEWED] 5aa6a6c

Description read

[DESCRIPTION-READ] d145f6441717d2d4ccc3d62f26ab091a905cf13bd951fa504e8acac9ed524ec7

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ✅ PASS (a BLOCK with no Trigger and no Outcome does not block)

UX-level review of 5aa6a6ca915469b472a726474268debff8c8ea5d — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

UX-Verdict: PASS

cannot evaluate: missing the panel's empty, limit, truncation, field-error, config-error and in-flight states. The diff adds strings for each of these, and none of the 12 screenshots shows them.

Evidence gaps

  • no_tools "This server lists no tools." (the empty state): no screenshot. A capture of a server that lists no tools would close it.
  • left_out "Tools not listed because the list is over the limit: {{count}}": no screenshot. Only withheld is shown (shot-09).
  • truncated "The result was too long, so only its start is shown.": no screenshot of a result that was cut.
  • invalid_field "Check the value of {{name}}." on the form path: no screenshot. Only the raw-JSON hint appears (shot-06).
  • config_failed notice, both the full-panel ErrorNotice and the refresh-error impact="partial" one: no screenshot.
  • In-flight state: "Load tools" busy and "Run tool" busy while a run is in progress. No screenshot.

Notes

  • The server NativeSelect has only an aria-label ("Server to try"). The blind reader said: "it has no label. I am only guessing that it is the server." Show "Server to try" as a visible label.
  • invalid_json "The arguments are not valid JSON." left the reader with "no idea how to fix it". Say what the box expects, e.g. "Enter the tool's arguments as one JSON object, like {"name": "value"}."
  • withheld "…their names look like secrets: 2" was read as "unsettling" and clashes with "Secrets are hidden from the result" in help. Reword it to "Tools hidden because their names may contain a password or key: {{count}}".

[UX-REVIEWED] 5aa6a6c

Description read

[DESCRIPTION-READ] 6ede0898c08391205533efb7e7cbd6d0d64dbcb7cddd82433a8eafd3452e86d7

That sha256 names everything this verdict read: the PR description, plus the 15 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ✅ PASS

Premise-level review of 5aa6a6ca915469b472a726474268debff8c8ea5d — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

First-Principles-Verdict: PASS

Nothing blocks: the panel is opt-in, owner-only and new, with no existing owner route that calls a tool. One part goes further than the Goal needs: the panel builds a form from each tool's schema.

Notes

  • Item 3 — oversized: the Goal is to run a tool "with exact arguments" and see the raw result. The raw JSON box the panel already has does that. schemaFields/argumentsFrom add a second way to type arguments, with its own coercion rules (safe integers, empty enum means "not set", number trimming). Subtraction: delete schemaFields, argumentsFrom and the field form in McpTryTool.tsx, and always show the raw JSON box. Clears when: the mechanism is removed, or a human amends the Goal.

What this change ships

Inventory (8 items) — 7 justified

Intent: the owner checks that one tool of a local MCP server works with exact arguments, without a chat turn. ADDITION.

  1. New mcp.tool_tester setting, off by default, no restart needed — justified
  2. POST /api/mcp/try-tool: owner-only, answers 404 while the setting is off, lists tools or runs one — justified
  3. Panel on the MCP Servers page with a form built from the tool's schema, plus a raw JSON box — oversized
  4. A tool not marked read-only needs a confirm; tools or servers switched off on the page are refused — justified
  5. The governance ceiling is checked under dashboard:mcp-try-tool, and a security event is written before and after each run — justified
  6. The probe gains an optional one-call step that reuses its spawn and teardown; a plain probe is unchanged — justified
  7. While a call waits, the server's stderr is drained and its first 32 KiB kept, with a "more not shown" note — justified
  8. Redaction-sink row, feature-map row, config docs, catalogs in 12 languages — justified

[FIRST-PRINCIPLES-REVIEWED] 5aa6a6c

Description read

[DESCRIPTION-READ] 73372bf4f6c9136a15506ad6bee56e8c29972448b2d3330f05d839d71ccc2003

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed 5aa6a6ca915469b472a726474268debff8c8ea5d — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 5aa6a6c

Verdict parsed from the review's SHA-scoped output markers for commit 5aa6a6ca915469b472a726474268debff8c8ea5d.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 5aa6a6ca915469b472a726474268debff8c8ea5d false-positive: <reason, 20 to 500 characters>

@github-actions

github-actions Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — 🔴 changes requested (blocking)

GPT 6.1 found at least one blocking issue that must be resolved before merging 5aa6a6ca915469b472a726474268debff8c8ea5d. 1 of 2 blocking finding(s) are security-class and were withheld from adjudication, so the blocking verdict stands.

This comment is updated in place on each push.

BLOCKING -- website/src/pages/overview/McpTryTool.tsx:208 -- Prototype-named arguments crash the tool form

(values[f.name] ?? '').split('\n').length
Trigger: With the panel enabled, Load tools selects a tool whose schema declares a string property named constructor, toString, or __proto__; TryToolPanel renders that field with values initialized to {}.
Outcome: The lookup returns an inherited function or object, so .split() throws TypeError, preventing the owner from editing or running the tool.
Anchor: residual/crash-data-loss-corruption
Fix: Read only own entries throughout the form and construct outgoing arguments using a prototype-free map.
Sites: website/src/pages/overview/McpTryTool.tsx:47, website/src/pages/overview/McpTryTool.tsx:201, and website/src/pages/overview/McpTryTool.tsx:209 also read inherited entries; website/src/pages/overview/McpTryTool.tsx:56 drops an argument named __proto__.

BLOCKING -- website/src/pages/overview/McpTryTool.tsx:23 -- Dynamic argument schemas become unusable empty forms

const props = (schema.properties ?? {}) as Record<string, { type?: unknown; enum?: unknown }>
Trigger: Load tools returns {"type":"object","additionalProperties":{"type":"string"},"minProperties":1}; schemaFields returns [], and TryToolPanel selects the generated form rather than the raw JSON editor.
Outcome: The owner receives no argument editor, and Run tool sends only {}, making it impossible to supply arguments satisfying this schema.
Anchor: defect
Fix: Fall back to raw JSON for explicit dynamic-property schemas and required keys the generated fields cannot represent.
Sites: patternProperties schemas follow the same path at website/src/pages/overview/McpTryTool.tsx:23; required keys absent from properties are also ignored at website/src/pages/overview/McpTryTool.tsx:25.

[BLOCK-MERGE] 5aa6a6c
[GPT-REVIEWED] 5aa6a6c

Adjudication (Opus 5.5) — is blocking on each finding proportionate?

F2 (MEDIUM): this is a common path that fails without crashing. Conditions confirmed: McpTryTool.tsx:22 sends only composed or $ref schemas to raw JSON. :23-24 treats a schema with no properties but with additionalProperties/patternProperties as a valid object. :27 then yields [], and :132/:197 render that empty field list in place of the raw box. Recovery: none in the panel. :162 always sends {}, so a tool with dynamic keys can never be given arguments; the server's error at :234 is the only sign. The real fix is cheap: two or three more conditions on the existing return null guard at :22/:25, with no new state or contract. Cost does not exceed harm.

F1 (fenced): the panel crashes, but no data is lost or corrupted. Conditions confirmed: values starts as {} (:117, :151). :208 values[f.name] ?? '' returns the inherited Object.prototype member for a field named constructor/toString, and .split throws during render. A schema property must carry that exact name (:27). The panel only exists while the owner has opted into mcp.tool_tester (:106). Recovery: reload, or switch the panel off. Nothing persists, and the crash is visible, not silent.

[ADJUDICATION] 5aa6a6ca915469b472a726474268debff8c8ea5d total=1 uphold=1 downgrade=0
UPHOLD F2 website/src/pages/overview/McpTryTool.tsx:23 reason=harm-warrants-remedy
[GPT-ADJUDICATED] 5aa6a6ca915469b472a726474268debff8c8ea5d

[ADJUDICATION-FENCED] 5aa6a6ca915469b472a726474268debff8c8ea5d fenced=1 flagged=1
FLAG F1 website/src/pages/overview/McpTryTool.tsx:208 -- The worst outcome is a visible render crash of an owner-only, opt-in debug panel (gated at :106), and only when a tool's schema names a field after an Object.prototype member (:27, :208); nothing is persisted, so a reload or turning off mcp.tool_tester recovers fully with no data loss or corruption.
[GPT-ADJUDICATED-FENCED] 5aa6a6ca915469b472a726474268debff8c8ea5d

🏷️ Fenced finding(s) machine-flagged as likely edge case

The security fence keeps these findings blocking regardless of adjudication; the only clearance path is a human override. The annotate-only pass judged the condition combination each one requires extreme, and pre-drafted the override rationale below. A repository writer must independently verify a rationale before posting it — it is machine-authored, and a wrong override on a security-class finding ships exactly the class the fence exists to stop.

  • F1 website/src/pages/overview/McpTryTool.tsx:208 — The worst outcome is a visible render crash of an owner-only, opt-in debug panel (gated at :106), and only when a tool's schema names a field after an Object.prototype member (:27, :208); nothing is persisted, so a reload or turning off mcp.tool_tester recovers fully with no data loss or corruption.

    /ai-review override gpt 5aa6a6ca915469b472a726474268debff8c8ea5d <class>: The worst outcome is a visible render crash of an owner-only, opt-in debug panel (gated at :106), and only when a tool's schema names a field after an Object.prototype member (:27, :208); nothing is persisted, so a reload or turning off mcp.tool_tester recovers fully with no data loss or corruption.
    

Replace <class> with the class that records what you did. These findings are real, so for an accepted risk use owner-decision (link the decision comment in the reason) or deferred #<N> (a tracked follow-up issue); false-positive only if you verified the finding is wrong. Not not-applicable: that class files the override as a rule mismatch in the audit.

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 5aa6a6ca915469b472a726474268debff8c8ea5d false-positive: <reason, 20 to 500 characters>

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Nov 16, 2026
@javenciu
javenciu force-pushed the feat/mcp-try-a-tool branch from 21c1105 to 7d2f148 Compare November 16, 2026 07:17
@javenciu

Copy link
Copy Markdown
Contributor Author

Round 1 answers, head 7d2f148.

1. Static Ratchet Gate (error code, redaction sink). Fixed in 7d2f148. Every refusal and failure of /api/mcp/try-tool now leaves through one json_response({"error": ..., "code": ...}, status=status) with a literal dict, so the contract scan sees a coded body (it counted the old json_response(payload, status=status) as a dynamic status). dashboard/handlers/mcp_try_tool.py is registered in security_posture_sinks_gateway.py as "The Try a tool panel". test_error_code_contract.py::test_no_new_error_response_without_a_code and test_security_posture.py::TestOmissionDetection::test_every_redactor_call_site_is_a_registered_sink_or_allowlisted both fail on the old head and pass now. The Await Fast Gate and Coverage Gate logs only report that the Fast Gate failed, so they follow from this fix. All seven Static Ratchet Gate files pass locally (393 related files, 13,761 passed).

2. UX block: the confirm step. Fixed in 7d2f148. The warning reads: This runs "tool" on the real "server" server and can change its data. This panel cannot undo it. The button reads Run "tool" for real, and the quoted names are pinned in destructiveConfirm.test.ts. The help text now says what a run returns and that secrets are hidden, and the result no longer breaks words mid-word. Stills of the raw JSON box, a bad field, a failed run, a tool error, no tools, the left-out and cut notes, and the panel before Load tools are in shots/, light and dark. Load tools stays a deliberate press: it starts the server's command, which should not happen on selecting a name.

3. McpTryTool.tsx:49 (server starts as ''). Fixed in 7d2f148. The pick is kept only while it is still listed, otherwise the first server is used, for the select, the Load button and every request. McpTryTool.test.tsx "uses the first server once the list arrives after the panel is drawn" fails on the old head.

4. McpTryTool.tsx:67 (stale listing). Fixed in 7d2f148. A listing carries the server it was requested for and shows only for that server, and an answer for a server the user has left is dropped before it sets tools, a result or an error. "drops a listing that lands after the user chose another server" fails on the old head.

5. McpTryTool.tsx:33 (trimmed strings). Fixed in 7d2f148. Strings and enums go out as typed; only numbers are trimmed. A field never touched is left out (and an empty enum or boolean choice reads as not set), while a string emptied on purpose is sent as "". Once an optional text field has been typed in, clearing it sends "". "sends strings and enums as typed and tells an unset field from an empty string" fails on the old head.

6. mcp_try_tool.py:164 (process tree). Fixed in 7d2f148. McpToolClient takes own_process_group, which starts the server in its own session on POSIX, and McpToolClient.stop_tree stops the tree through platform_compat.kill_process_tree (SIGKILL), falling back to the launcher if the tree cannot be reached and leaving an already reaped launcher alone. The route calls it on timeout and on cancel, and close() calls it on completion and when start-up fails. The tests patch platform_compat.kill_process_tree and assert the call (test_a_run_past_the_time_limit_stops_the_server, test_a_cancelled_request_stops_the_server, the three stop_tree tests and test_a_client_owns_a_process_group_only_when_asked); no test touches os.killpg. They fail on the old head.

7. McpTryTool.tsx:120 (error surface). Fixed in 7d2f148. A tool's isError result renders an ErrorNotice above the raw result, which stays visible. A failed read of the setting renders an ErrorNotice instead of hiding the panel. A bad argument or invalid JSON is a separate hint under the form. Each notice carries a No hand-off comment naming what it protects. The panel tests for the three states fail on the old head.

8. mcp_try_tool.py:68 (silent overflow). Fixed in 7d2f148. A tool past 200, past the listing budget, or with a name too long to show is counted. The listing returns leftOut, the panel shows the count, and running such a tool answers 404 tool_not_listed with "not listed: over the limit"; a name the server never announced still answers unknown_tool. test_tools_past_the_cap_are_counted_and_named_when_run and the panel's left-out test fail on the old head.

Also. Round 0's Bengali, Hindi and Russian strings used the formal register, which I18N_BASE_REF=<merge-base> npx vitest run src/i18n/style/ rejects; they now use the informal one. test_mcp_handler_facade.py lists mcp_try_tool among the route modules that are not facade owners. Product code is now about 445 added lines (it was 342). If a split is preferred, the setting, route, client and sink row (about 270) can land first and the panel (about 175) second.

@javenciu

Copy link
Copy Markdown
Contributor Author

Sensitive change review

  • Head commit: 7d2f148
  • What rule changed: src/kiro_crew/security_posture_sinks_gateway.py gains one row in the redaction sink registry, naming dashboard/handlers/mcp_try_tool.py as an output boundary. No redaction rule, pattern or allowlist changes.
  • Before -> after (what was allowed/blocked, now): before, the omission scan found a redactor call in mcp_try_tool.py that no row named, so the Static Ratchet Gate failed. After, the module is a registered sink and the Settings -> Security panel counts it. The module already redacted every string it returns (results, tool descriptions, input schemas, start-up errors) before this row existed.
  • Worst case if wrong (breaks commands / leaks secret / locks user out): the row only describes an existing redaction, so a wrong row cannot let text through. A mistake in the module's own redaction could show a server's credential to the dashboard owner, who started that server; test_a_credential_in_the_result_is_redacted and test_a_listing_is_redacted_and_bounded pin it with a synthetic credential.
  • Evidence checked (which sandbox result proves it): test/test_security_posture.py and the other six Static Ratchet Gate files pass on this head; test_every_redactor_call_site_is_a_registered_sink_or_allowlisted fails on the previous head and passes now.
  • No regression:
    • Tools still work (which tools/commands were run, result): test/test_mcp_try_tool.py (29 passed) and the 393 related test files (13,761 passed) run on this head.
    • Backward compatible (old config / old data / old callers still work, how checked): the route is registered only while mcp.tool_tester is on, which defaults off; with it off nothing reads the new row's module. test_off_the_path_answers_as_an_unknown_route passes on main and on this head.
    • Existing tests (which suites passed on this head): test_security_posture*.py are inside the 393 files; test_denied_commands_api.py, test_dashboard_handlers_core_coverage_security_endpoints.py and test_security_posture_facade.py ran separately on this head: 130 passed.
  • How to undo: revert the commit. The registry row is removed with it, and the setting stays off.

@github-actions

github-actions Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

Sensitive Change Review

Sensitive files changed:

  • src/kiro_crew/security_posture_sinks_gateway.py

  • @javenciu posted a complete reasoning template for 5aa6a6c.

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Nov 16, 2026
@javenciu
javenciu force-pushed the feat/mcp-try-a-tool branch from 7d2f148 to ef52b82 Compare November 16, 2026 11:25
@javenciu

Copy link
Copy Markdown
Contributor Author

Round 2 answers, head ef52b82.

1. Process teardown (Opus and GPT mcp_try_tool.py:181, GPT cron_script_mcp.py:330, CI test_kill_chokepoint_ratchet.py). Fixed in ef52b82 by dropping this PR's own teardown. The cron_script_mcp.py changes (own_process_group, stop_tree, on_spawn, rpc) are gone, so that file matches main. The tester now runs through the discovery probe. probe_server takes an optional tool_call=ToolCallStep(choose), which sends one tools/call after a good tools/list on the same pipe. The run therefore uses the probe's sandboxed spawn on its private loop in the pooled probe worker, its handshake, its cancel forwarding and its finally, which reaps the whole process group (POSIX killpg, Windows kill_process_tree through asyncio.to_thread). Nothing is killed on the gateway loop, and the group is reaped even after the launcher has exited. A plain probe is unchanged: test_a_plain_probe_sends_no_tool_call passes on main and pins initialize, notifications/initialized, tools/list and nothing more. test_a_cancelled_request_stops_the_server_and_what_it_started starts a real server that forks a sleeper, cancels the request mid-call and waits until both are gone. test_a_probe_step_calls_one_tool_after_the_listing fails on the old head, and the kill-chokepoint ratchet that failed in CI passes now.

2. Opus BLOCKING mcp_try_tool.py:123 (no SEL record). Fixed in ef52b82. Each run writes sel().log_tool_invocation(session_key="dashboard", source="dashboard", tool_name="server/tool", tool_kind="dashboard_mcp_tool", metadata={"confirm": ...}, critical=True) as invoked before the call is sent, from the probe's worker thread. When that write fails, the call is not sent and the route answers 503 audit_unavailable. A second critical record carries the outcome (completed, error or timeout). The confirm-required, switched-off, unknown-tool and over-the-limit refusals are written as denied with their code. That write is best effort, the way the auto-nudge fire route treats its refusals, because nothing runs either way. test_every_call_is_in_the_security_log_written_ahead, test_a_call_that_cannot_be_recorded_never_runs, test_a_tool_error_is_recorded_as_an_error and the refusal assertions fail on the old head.

3. GPT mcp_try_tool.py:109 (a listing error reads as no tools). Fixed in ef52b82. The probe reports a tools/list RPC error as a failed probe (tools/list: <message>). The route returns it as 502 server_failed with the text redacted, for a listing and for a named run. test_a_listing_error_is_a_coded_redacted_failure uses a synthetic credential in the server's message. It fails on the old head, which answered 200 with an empty list.

4. CI test_json_object_body_guard.py. Fixed in ef52b82. handlers/mcp_try_tool.py::api_mcp_try_tool has a register row with the shared default cap and its reason: a server and tool name, a confirm flag and the arguments typed into one tool's form. The route passes no cap, so the default applies.

5. Frontend. Fixed in ef52b82.

  • ApiClient.refactor.surface.test.ts: mcpTryTool is listed after mcpProbe, where the facade spreads it.
  • McpTryTool.tsx:56: a failed first read of the setting renders a blocking ErrorNotice with Ask the agent (nothing is typed yet), and its No hand-off comment is removed. A failed background refresh keeps the panel mounted, with a partial notice inside it. Its No hand-off comment names the typed arguments it protects. A tool's isError notice now uses the default blocking impact.
  • :88: every request carries a number, and picking a tool or a server makes any answer still in flight stale. While a run is in flight, the tool picker is marked busy (aria-busy, aria-disabled, the primitive's busy look) and refuses changes.
  • :104: argumentsFrom returns { ok: true, args } | { ok: false, invalid }, so an argument named invalid is sent as typed.
  • The panel tests for these states fail on the old head (8 of 15).

6. Whole suites and size. The whole backend suite on this patch fails only where plain main fails on this machine, plus one timing test (test_push_guard_diagnostics.py, a signal during a child wait) that passes on rerun. The whole website vitest passes (71,261 tests). Product code is about 557 lines. If a split is wanted, the setting, the probe step, the route and the sink row (about 367) can land first, and the panel (about 190) second.

@javenciu

Copy link
Copy Markdown
Contributor Author

Sensitive change review

  • Head commit: ef52b82
  • What rule changed: src/kiro_crew/security_posture_sinks_gateway.py gains one row in the redaction sink registry, naming dashboard/handlers/mcp_try_tool.py as an output boundary. No redaction rule, pattern or allowlist changes. Round 2 does not change this row.
  • Before -> after (what was allowed/blocked, now): before, the omission scan found a redactor call in mcp_try_tool.py that no row named, so the Static Ratchet Gate failed. After, the module is a registered sink and the Settings -> Security panel counts it. The module redacts every string it returns (results, tool descriptions, input schemas, start-up and listing errors) whether or not this row exists.
  • Worst case if wrong (breaks commands / leaks secret / locks user out): the row only describes an existing redaction, so a wrong row cannot let text through. A mistake in the module's own redaction could show a server's credential to the dashboard owner, who started that server; test_a_credential_in_the_result_is_redacted, test_a_listing_is_redacted_and_bounded and test_a_listing_error_is_a_coded_redacted_failure pin it with synthetic credentials.
  • Evidence checked (which sandbox result proves it): test/test_security_posture.py and the other six Static Ratchet Gate files pass on this head (lane_ci_checks.sh: CI-CHECKS PASS); test_every_redactor_call_site_is_a_registered_sink_or_allowlisted fails without the row and passes with it.
  • No regression:
    • Tools still work (which tools/commands were run, result): test/test_mcp_try_tool.py (30 passed), and the related suites (the MCP discovery and probe files, cron script, config, SEL, facade, error-code, body-cap, kill-chokepoint and security posture files) on this head: 5,489 passed, 9 skipped.
    • Backward compatible (old config / old data / old callers still work, how checked): the route is registered only while mcp.tool_tester is on, which defaults off; with it off nothing reads the new row's module. test_off_the_path_answers_as_an_unknown_route and test_a_plain_probe_sends_no_tool_call pass on main and on this head.
    • Existing tests (which suites passed on this head): the whole backend suite on this patch shows the same failures as plain main on this machine plus one timing test that passes on rerun; the whole website vitest passed (71,261 tests).
  • How to undo: revert the commit. The registry row is removed with it, and the setting stays off.

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Nov 16, 2026
@javenciu
javenciu force-pushed the feat/mcp-try-a-tool branch from ef52b82 to 919949b Compare November 16, 2026 15:15
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Nov 16, 2026
@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention merge conflict Branch has merge conflicts with its base — author must resolve before merge and removed readiness: checking Automated validation is still running labels Nov 16, 2026
@javenciu
javenciu force-pushed the feat/mcp-try-a-tool branch from b1f2f9c to 4a70ac0 Compare November 17, 2026 00:15
@javenciu

Copy link
Copy Markdown
Contributor Author

Round 6 answers, head 4a70ac0 (rebased on main 5ff3fed).

1. Opus mcp_try_tool.py:401: a big result is dumped and redacted on the gateway loop. Fixed. The json.dumps of the result and its redaction now run in one asyncio.to_thread call (_result_text), and the text is cut afterwards, so the redactor still sees the text from its first character and a credential that straddles the cut cannot get through. test_the_result_is_dumped_and_redacted_off_the_gateway_loop patches asyncio.to_thread, records the callee and checks the result stays redacted and cut; it fails on the round 5 head (the callee is not there). test_a_credential_across_the_cut_is_redacted_before_the_text_is_cut puts the cut inside a credential.

2. Opus McpTryTool.tsx:122: Load tools selects a switched-off first tool. Fixed. Load tools selects the first tool whose switchedOff is false, and nothing when every tool is switched off, so Run stays disabled. selects the first tool that is not switched off, and runs it and selects nothing and keeps Run disabled when every tool is switched off fail on the round 5 head.

3. GPT mcp_discovery_probe.py:335: an unread stderr can stall the tool call. Fixed. While _call_one_tool waits for the answer, a drain task reads the server's stderr into a tail bounded by the same byte count _stderr_text reads (_STDERR_TAIL_BYTES, 8 reads of 4 KiB). The drain is cancelled and awaited in a finally, so no task is left after an answer, a timeout or a cancel. A failed probe still appends a stderr tail: _stderr_text now starts from what the drain kept and then reads what is left in the pipe. A probe without a step is unchanged. Tests: test_a_server_that_fills_its_stderr_before_it_answers_still_answers (a real server writing 1 MiB to stderr before it answers; the reader's buffer is set small through KIROCREW_MCP_READ_LIMIT, because the default 64 MiB buffer would absorb 1 MiB) fails on the round 5 head with a timeout; test_the_drain_is_gone_once_the_answer_arrives, _after_a_timeout and _after_a_cancel check that no drain task survives; test_a_call_that_fails_after_noisy_stderr_still_shows_its_tail and test_the_failure_report_starts_with_what_the_drain_took cover the failure report.

4. GPT McpTryTool.tsx:117: feedback outlives the selection it belongs to. Fixed. The error, the run-failed title, the hint, the answer and the confirm are cleared whenever the effective server or the tool changes, including a server list refresh that changes the effective server. The tests for a tool change after a failed run, an answer cleared by a refresh that changes the server, and a failed run cleared the same way fail on the round 5 head.

5. UX lane: the inline confirm state is not shown. Done. The light still of the inline confirm (after pressing "Run tool…", with "Run … for real" and Cancel) is back in the description, which stays under 8,000 bytes after upload.

6. CI. The two failing tests, test_kas_orchestrator_memory.py::TestTheRealSessionNewRequest::test_an_unconfirmed_resume_is_evicted_without_deleting_its_record and test_acp_runtime_load_session.py::TestAcpRuntimeLoadSession::test_load_session_keeps_the_transcript_path_alongside_the_agents, fail on the previous base commit 3c9f33c (4 failed, with DID NOT RAISE and KeyError: 'settings', the same errors as in the CI logs) and pass on current main 5ff3fed and on this branch after the rebase (4 passed each). They failed on main until #38108 landed, so they are main-side and no change was needed here. The Windows shards and the Coverage Gate failed on the same two tests (Coverage Gate fails closed when a backend shard fails).

Whole suites: the backend suite fails only where plain main fails on this machine; the one id not in the earlier known sets (a find-UI test) fails the same on a clean main with the website deps installed. The website vitest passes in four shards (71,396 tests).

@javenciu

Copy link
Copy Markdown
Contributor Author

Sensitive change review

  • Head commit: 4a70ac0
  • What rule changed: src/kiro_crew/security_posture_sinks_gateway.py gains one row in the redaction sink registry, naming dashboard/handlers/mcp_try_tool.py as an output boundary. No redaction rule, pattern or allowlist changes. Rounds 2 to 6 do not change this row.
  • Before -> after (what was allowed/blocked, now): before, the omission scan found a redactor call in mcp_try_tool.py that no row named, so the Static Ratchet Gate failed. After, the module is a registered sink and the Settings -> Security panel counts it. The module redacts every string it returns (results, tool descriptions, input schemas, start-up and listing errors) whether or not this row exists, and since round 5 a tool whose name the redactor would change is neither listed nor run (403 tool_name_withheld).
  • Worst case if wrong (breaks commands / leaks secret / locks user out): the row only describes an existing redaction, so a wrong row cannot let text through. A mistake in the module's own redaction could show a server's credential to the dashboard owner, who started that server; test_a_credential_in_the_result_is_redacted, test_a_listing_is_redacted_and_bounded and test_a_listing_error_is_a_coded_redacted_failure, test_a_tool_name_the_redactor_would_change_is_counted_and_never_listed, test_a_withheld_tool_name_is_refused_when_it_is_the_tool_asked_for, test_the_result_is_dumped_and_redacted_off_the_gateway_loop and test_a_credential_across_the_cut_is_redacted_before_the_text_is_cut pin it with synthetic credentials. Round 6 moves the dump and redaction of a result into a worker thread and keeps redact-then-cut; the redactor sees the same text.
  • Evidence checked (which sandbox result proves it): test/test_security_posture.py and the other six Static Ratchet Gate files pass on this head (lane_ci_checks.sh: CI-CHECKS PASS); test_every_redactor_call_site_is_a_registered_sink_or_allowlisted fails without the row and passes with it.
  • No regression:
    • Tools still work (which tools/commands were run, result): test/test_mcp_try_tool.py (53 passed), its related set (owner-gate invariant, facade guards, body caps, error codes, security posture, probe cache, boot-path probe, flake and wait ratchets: re-run on this head through lane_ci_checks.sh and the whole backend suite), and the whole backend suite, which fails only where plain main fails on this machine (208 ids, one of them not in the earlier known sets and failing the same on a clean main).
    • Backward compatible (old config / old data / old callers still work, how checked): mcp.tool_tester defaults off; with it off the route answers as an unknown path does and start-up neither reads the config nor imports the row's module. test_off_the_path_answers_as_an_unknown_route, test_a_plain_probe_sends_no_tool_call and test_registering_the_routes_loads_no_config_and_no_tester_module pass on this head.
    • Existing tests (which suites passed on this head): the whole website vitest passed in four shards (71,396 tests); lane_ci_checks.sh printed CI-CHECKS PASS on this head.
  • How to undo: revert the commit. The registry row is removed with it, and the setting stays off.

@github-actions github-actions Bot removed the merge conflict Branch has merge conflicts with its base — author must resolve before merge label Nov 17, 2026
@javenciu
javenciu force-pushed the feat/mcp-try-a-tool branch from 4a70ac0 to d70295a Compare November 17, 2026 02:15
@javenciu

Copy link
Copy Markdown
Contributor Author

Round 7 (head d70295a), on current main c87ea0c. Answers the three blocking GPT findings for 4a70ac0; nothing else changed (no website file).

mcp_try_tool.py:286, teardown discards an answer that already arrived. Fixed in _try: when the outer 30-second limit fires and step.answer is already set, the run takes the answered path (the result, completed or error in the audit) instead of returning 504. The probe's cancelled task still reaps the process tree in its finally, as before; with no answer the 504 and the timeout record are unchanged. Test that fails on the previous head with a 504: test_an_answer_that_came_before_the_limit_is_kept_when_the_teardown_overruns (the stand-in server sets the answer and then never ends, as one that ignores its closed stdin does, with the limit patched so the deadline fires during that wait; the route returns the answer, the audit reads invoked then completed, and the probe task was cancelled). test_a_run_past_the_time_limit_is_a_timeout_in_the_log pins the unanswered case. The docs page says an answer that arrived in time is shown even when the server is slow to stop.

test_mcp_try_tool.py:956, drain tests hang instead of failing. Every await a drain test must unblock now goes through _within, which uses asyncio.wait_for with FALLBACK_TIMEOUT_SECS from kiro_crew.testing.wait (60 s, half of the 120 s test timeout) and fails naming what did not finish. The fake reply reader honours the timeout it is given (asyncio.wait_for(stderr.waiting.wait(), timeout)), and the call task the cancel test creates is cancelled and awaited in a finally, as is any drain a failing run leaves behind. Proof: with _drain_stderr made to return at once, the old drain tests hang until the 30 s limit I gave pytest (red-backend-r7-item2-old-tests.log), and the new ones fail by name at the 60 s bound (red-backend-r7-item2.log).

test_mcp_try_tool.py:238, interpreter launches per test (D8). The module is one xdist_group (mcp_try_tool), and the cases that do not need a child process moved to an in-memory stand-in for the probe (_FakeServer, patched through the mcp_discovery facade), which runs the real _Run.choose, _tool_rows, _failure, the governance and audit code and the route. Counted by a plugin that wraps subprocess.Popen (which asyncio subprocesses also use): before, 34 of 53 tests started a child, 44 starts in all; now 6 of 55 do, 8 starts. These still start a real child, because the spawn, the handshake bytes, the teardown, a cancel or the stderr pipe is what they check: test_a_probe_sends_a_tool_call_only_when_it_is_given_a_step (handshake unchanged without a step, one tools/call with one), test_a_tool_is_confirmed_and_run_on_a_real_server (spawn and the refusal after the listing, through the route), test_a_run_past_the_time_limit_stops_the_server and test_a_cancelled_request_stops_the_server_and_what_it_started (teardown reaps the process group), and the two stderr tests (a full pipe and a server that dies after a flood). The route-registration test no longer starts an interpreter: it records the imports and the config reads that register makes in this process. One module-scoped interpreter cannot serve the real cases, because probe_server spawns the server itself and each case needs its own script and config; test/test_mcp_discovery_probe_stdio.py on main, which starts one per case for the same reason, is the established pattern for code that spawns its own server. Three runs in a row: 55 passed each time.

@javenciu

Copy link
Copy Markdown
Contributor Author

Sensitive change review

  • Head commit: d70295a
  • What rule changed: src/kiro_crew/security_posture_sinks_gateway.py gains one row in the redaction sink registry, naming dashboard/handlers/mcp_try_tool.py as an output boundary. No redaction rule, pattern or allowlist changes. Rounds 2 to 7 do not change this row.
  • Before -> after (what was allowed/blocked, now): before, the omission scan found a redactor call in mcp_try_tool.py that no row named, so the Static Ratchet Gate failed. After, the module is a registered sink and the Settings -> Security panel counts it. The module redacts every string it returns (results, tool descriptions, input schemas, start-up and listing errors) whether or not this row exists, and a tool whose name the redactor would change is neither listed nor run (403 tool_name_withheld). Round 7 changes only which response a run gets when its limit fires after the tool answered: the answer, redacted and cut as every answer is, instead of a 504.
  • Worst case if wrong (breaks commands / leaks secret / locks user out): the row only describes an existing redaction, so a wrong row cannot let text through. A mistake in the module's own redaction could show a server's credential to the dashboard owner, who started that server; test_a_credential_in_the_result_is_redacted, test_a_listing_is_redacted_and_bounded, test_a_listing_error_is_a_coded_redacted_failure, test_a_tool_name_the_redactor_would_change_is_counted_and_never_listed, test_a_withheld_tool_name_is_refused_when_it_is_the_tool_asked_for, test_the_result_is_dumped_and_redacted_off_the_gateway_loop and test_a_credential_across_the_cut_is_redacted_before_the_text_is_cut pin it with synthetic credentials. The answer returned after a late limit takes the same path (_result_text), so the redactor sees the same text.
  • Evidence checked (which sandbox result proves it): test/test_security_posture.py and the other six Static Ratchet Gate files pass on this head (lane_ci_checks.sh: CI-CHECKS PASS); test_every_redactor_call_site_is_a_registered_sink_or_allowlisted fails without the row and passes with it.
  • No regression:
    • Tools still work (which tools/commands were run, result): test/test_mcp_try_tool.py (55 passed, three runs in a row), and its related set (MCP discovery and probe, security posture, facade guards, error codes, body guard, owner gate, config, feature map: 1,003 passed) plus the flake and wait ratchets (125 passed).
    • Backward compatible (old config / old data / old callers still work, how checked): mcp.tool_tester defaults off; with it off the route answers as an unknown path does and start-up neither reads the config nor imports the row's module. test_off_the_path_answers_as_an_unknown_route, test_a_probe_sends_a_tool_call_only_when_it_is_given_a_step (a probe without the step sends no tools/call) and test_registering_the_routes_loads_no_config_and_no_tester_module pass on this head.
    • Existing tests (which suites passed on this head): the related set above, the 648 docs tests, and lane_ci_checks.sh (CI-CHECKS PASS on this head). This round changes no website file and no Python source besides mcp_try_tool.py, so round 6's whole backend and website runs are not repeated.
  • How to undo: revert the commit. The registry row is removed with it, and the setting stays off.

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Nov 17, 2026
@javenciu
javenciu force-pushed the feat/mcp-try-a-tool branch from d70295a to 24afa04 Compare November 17, 2026 04:52
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Nov 17, 2026
@javenciu

Copy link
Copy Markdown
Contributor Author

Round 8 (head 24afa04), rebased on main cb6cffb. Answers the two blocking GPT findings for d70295a; nothing else changed.

mcp_discovery_probe.py:158, the stderr tail can drop a private key's BEGIN line. Fixed. While a tool call waits, the drain now keeps the first 32 KiB of what the server writes, the same bytes a probe without a call keeps (_stderr_text reads "from its first byte"), and goes on reading and discarding the rest so the pipe never fills. It records that it dropped bytes. The failure report starts with the kept bytes, appends nothing that follows a dropped stretch, and says "... (more stderr not shown)" after the text when a bound stopped the read with more left. The redactor therefore sees a block from its BEGIN line on and takes the rest of it. Tests that fail on the previous head: test_a_private_key_longer_than_what_is_kept_is_never_shown_in_part (real server, through the route: a 52 KB armored block on stderr, then no answer; on the old head base64 lines of the key are shown, now none are and the BEGIN line is redacted with the block), test_a_drain_keeps_the_start_of_what_a_server_wrote_and_says_what_it_dropped, test_nothing_is_appended_after_what_the_drain_dropped, test_a_full_report_says_so_only_when_more_was_left_in_the_pipe and test_a_call_that_fails_after_noisy_stderr_shows_its_start_and_says_more_was_cut. A server that writes a megabyte to stderr and then answers still gets its answer (test_a_server_that_fills_its_stderr_before_it_answers_still_answers). A plain probe is unchanged.

McpTryTool.tsx:54 and :135, integers outside the safe range change silently. Fixed in both places. In the form, an integer field whose value is not a safe integer is invalid and shows the existing field hint (a number field still takes any finite number). In the raw JSON box, a small scan with plain string methods (no regular expression) finds an integer literal outside quotes whose value is not a safe integer and refuses the run with the existing invalid JSON hint; a number inside a string, a fraction and an exponent are left alone. Tests that fail on the previous head: names an integer outside the safe range instead of sending a different number, finds an integer literal past the safe range outside strings only and refuses a raw JSON integer outside the safe range with the invalid JSON hint.

Maintainer review: the whole-design, first-principles, UX and Opus lanes have passed for several rounds, and each recent GPT round has found a new edge case in a large surface. Could a maintainer look at this PR? From here, further automated findings will be answered in one batch after a maintainer has looked, rather than round by round.

@javenciu

Copy link
Copy Markdown
Contributor Author

Sensitive change review

  • Head commit: 24afa04
  • What rule changed: src/kiro_crew/security_posture_sinks_gateway.py gains one row in the redaction sink registry, naming dashboard/handlers/mcp_try_tool.py as an output boundary. No redaction rule, pattern or allowlist changes. Rounds 2 to 8 do not change this row.
  • Before -> after (what was allowed/blocked, now): before, the omission scan found a redactor call in mcp_try_tool.py that no row named, so the Static Ratchet Gate failed. After, the module is a registered sink and the Settings -> Security panel counts it. The module redacts every string it returns (results, tool descriptions, input schemas, start-up and listing errors) whether or not this row exists, and a tool whose name the redactor would change is neither listed nor run (403 tool_name_withheld). Round 8 changes what a failed run's error carries from the server's stderr: the first 32 KiB the server wrote (as a probe without a call keeps), with a note when more was dropped, instead of the last 32 KiB. The redactor sees a private key from its BEGIN line, which the last bytes could cut away; test_a_private_key_longer_than_what_is_kept_is_never_shown_in_part pins it with a synthetic block.
  • Worst case if wrong (breaks commands / leaks secret / locks user out): the row only describes an existing redaction, so a wrong row cannot let text through. A mistake in the module's own redaction could show a server's credential to the dashboard owner, who started that server; test_a_credential_in_the_result_is_redacted, test_a_listing_is_redacted_and_bounded, test_a_listing_error_is_a_coded_redacted_failure, test_a_tool_name_the_redactor_would_change_is_counted_and_never_listed, test_a_withheld_tool_name_is_refused_when_it_is_the_tool_asked_for, test_the_result_is_dumped_and_redacted_off_the_gateway_loop and test_a_credential_across_the_cut_is_redacted_before_the_text_is_cut pin it with synthetic credentials. The answer returned after a late limit takes the same path (_result_text), so the redactor sees the same text.
  • Evidence checked (which sandbox result proves it): test/test_security_posture.py and the other six Static Ratchet Gate files pass on this head (lane_ci_checks.sh: CI-CHECKS PASS); test_every_redactor_call_site_is_a_registered_sink_or_allowlisted fails without the row and passes with it.
  • No regression:
    • Tools still work (which tools/commands were run, result): test/test_mcp_try_tool.py (58 passed, three runs in a row), and its related set (MCP discovery and probe, security posture, facade guards, error codes, body guard, owner gate, config, feature map: 1,260 passed) plus the flake and wait ratchets (125 passed).
    • Backward compatible (old config / old data / old callers still work, how checked): mcp.tool_tester defaults off; with it off the route answers as an unknown path does and start-up neither reads the config nor imports the row's module. test_off_the_path_answers_as_an_unknown_route, test_a_probe_sends_a_tool_call_only_when_it_is_given_a_step (a probe without the step sends no tools/call) and test_registering_the_routes_loads_no_config_and_no_tester_module pass on this head.
    • Existing tests (which suites passed on this head): the related set above, the whole backend suite (275,672 passed; its 209 failing ids are the ids already failing on main plus one load-timing case in test_crew_mode_board.py that passes alone and three runs in a row), the website tests for the panel and src/i18n/ (1,262 passed), and lane_ci_checks.sh (CI-CHECKS PASS on this head).
  • How to undo: revert the commit. The registry row is removed with it, and the setting stays off.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Nov 17, 2026
The MCP Servers page lists a server's tools but gives the owner no way to run one and see what it returns. With the new mcp.tool_tester setting on (off by default, applies at restart), POST /api/mcp/try-tool is registered. It is owner-only. It runs one local server through the discovery probe, which gains an optional one-call step after its listing (ToolCallStep), so the run shares the probe's sandboxed spawn on a private loop, its handshake and its process-group teardown; a plain probe is unchanged. The step's answer is read against the run's 30-second limit, not the shorter handshake timeout. A muted or remote server is refused before it starts, a tool not declared readOnlyHint needs a confirm, a switched-off tool is refused, and so is a tool the governance ceiling or the dashboard's profile denies (the same mcp-scope check an agent's call meets; a check that cannot run refuses). Every run is written to the security event log ahead of the call, which is refused when that write fails. The result is redacted and then cut to 65,536 characters with a note. The panel builds a form from the tool's input schema for simple types, with a multi-line box for strings, and falls back to a raw JSON box.
@javenciu
javenciu force-pushed the feat/mcp-try-a-tool branch from 24afa04 to 5aa6a6c Compare November 17, 2026 07:15
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Nov 17, 2026
@javenciu

Copy link
Copy Markdown
Contributor Author

Round 9 (head 5aa6a6c), rebased on main 1eb2dd7. It fixes this PR's own Windows test failure and answers the two blocking findings for 24afa04; nothing else changed.

Windows CI, test_a_private_key_longer_than_what_is_kept_is_never_shown_in_part. Fixed in the test. On Windows the child ends its stderr lines with CRLF, so the error ended -----BEGIN RSA PRIVATE KEY-----\r\n[REDACTED: credential] and the last assertion, which expects \n, failed. The redaction held: no line of the key was shown. The test now compares after turning \r\n into \n. No other job in run 44020117935 failed.

mcp_try_tool.py:281, a timed-out run's teardown is cancelled partway. Fixed in two parts. The backstop now waits _CALL_TIMEOUT_SECS + _TEARDOWN_MARGIN_SECS (20 s, which covers the stderr read, the exit wait and the wait after the kill, and the Windows tree kill), so the probe's own limit ends the run and its finally (kill, group or tree reap, profile unlink, temp sweep) runs to the end before the route answers. The run's limit also bounds the handshake: the initialize and tools/list reads wait min(probe timeout, time left) when a call step is given, so a raised probe timeout cannot carry the handshake past the limit. A probe without a step is unchanged. A timed-out run is still a timeout in the security log. Tests that fail on the previous head: test_a_timed_out_run_is_answered_only_once_its_teardown_stopped_the_server (real server that closes its stdout instead of answering and ignores its closed stdin; on the old head the route answers 504 and the server is still running, three runs out of three; now it answers after the teardown and the server is gone, three out of three), test_a_server_that_never_answers_the_handshake_stops_at_the_run_limit (probe timeout raised to 60 s; it also fails with the backstop change alone, where the handshake read waits 60 s) and test_the_backstop_waits_longer_than_the_probes_teardown_can_take (sums the probe's teardown waits from its constants, so a longer wait cannot reopen this quietly). probe_server's finally is unchanged.

McpTryTool.tsx:162, raw JSON turns an overflowing number into null. Fixed. The raw JSON scan, now hasUnsafeNumber, also refuses a number literal outside strings with a fraction or an exponent whose value is not finite (1e400, -1e400, nested copies), with the existing invalid JSON hint. A plain run of digits keeps the safe-integer check; "1e400" in a string, 1e21 and 1.5 still go out. Plain string methods, no regular expression. Tests that fail on the previous head: finds a fraction or an exponent too large to be finite outside strings only and refuses a raw JSON number too large to be finite with the invalid JSON hint (a run of {"value":1e400} sends nothing).

First-state stills. The description now shows the panel before "Load tools" is pressed, in light and dark: the title, the one-line explanation, the server picker and the Load tools button. No string or look changed.

The request in the round 8 reply for a maintainer review still stands.

@javenciu

Copy link
Copy Markdown
Contributor Author

Sensitive change review

  • Head commit: 5aa6a6c
  • What rule changed: src/kiro_crew/security_posture_sinks_gateway.py gains one row in the redaction sink registry, naming dashboard/handlers/mcp_try_tool.py as an output boundary. No redaction rule, pattern or allowlist changes. Rounds 2 to 9 do not change this row.
  • Before -> after (what was allowed/blocked, now): before, the omission scan found a redactor call in mcp_try_tool.py that no row named, so the Static Ratchet Gate failed. After, the module is a registered sink and the Settings -> Security panel counts it. The module redacts every string it returns (results, tool descriptions, input schemas, start-up and listing errors) whether or not this row exists, and a tool whose name the redactor would change is neither listed nor run (403 tool_name_withheld). Round 9 changes when a run ends, not what it shows: the backstop now waits 20 s past the run's limit, so the probe's own teardown (kill, group or tree reap, profile unlink, temp sweep) runs to the end before the route answers, and the run's limit also bounds the handshake reads, so a raised probe timeout cannot carry a run past it. The raw JSON box also refuses a number too large to be finite. Round 8's stderr handling (the first 32 KiB, with a note when more was dropped) is unchanged; test_a_private_key_longer_than_what_is_kept_is_never_shown_in_part still pins it, now with Windows line endings.
  • Worst case if wrong (breaks commands / leaks secret / locks user out): the row only describes an existing redaction, so a wrong row cannot let text through. A mistake in the module's own redaction could show a server's credential to the dashboard owner, who started that server; test_a_credential_in_the_result_is_redacted, test_a_listing_is_redacted_and_bounded, test_a_listing_error_is_a_coded_redacted_failure, test_a_tool_name_the_redactor_would_change_is_counted_and_never_listed, test_a_withheld_tool_name_is_refused_when_it_is_the_tool_asked_for, test_the_result_is_dumped_and_redacted_off_the_gateway_loop and test_a_credential_across_the_cut_is_redacted_before_the_text_is_cut pin it with synthetic credentials. The answer returned after a late limit takes the same path (_result_text), so the redactor sees the same text.
  • Evidence checked (which sandbox result proves it): test/test_security_posture.py and the other six Static Ratchet Gate files pass on this head (lane_ci_checks.sh: CI-CHECKS PASS); test_every_redactor_call_site_is_a_registered_sink_or_allowlisted fails without the row and passes with it.
  • No regression:
    • Tools still work (which tools/commands were run, result): test/test_mcp_try_tool.py (61 passed, three runs in a row), and its related set (MCP discovery and probe, security posture, facade guards, error codes, body guard, owner gate, config, feature map: 1,305 passed) plus the flake and wait ratchets (125 passed).
    • Backward compatible (old config / old data / old callers still work, how checked): mcp.tool_tester defaults off; with it off the route answers as an unknown path does and start-up neither reads the config nor imports the row's module. test_off_the_path_answers_as_an_unknown_route, test_a_probe_sends_a_tool_call_only_when_it_is_given_a_step (a probe without the step sends no tools/call) and test_registering_the_routes_loads_no_config_and_no_tester_module pass on this head.
    • Existing tests (which suites passed on this head): the related set above, the whole backend suite (275,827 passed; its 209 failing ids are the ids already failing on main, plus the flake ratchet run on the file before its last comment, which passes on this head), the website tests for the panel and src/i18n/ (1,262 passed), and lane_ci_checks.sh (CI-CHECKS PASS on this head).
  • How to undo: revert the commit. The registry row is removed with it, and the setting stays off.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention merge conflict Branch has merge conflicts with its base — author must resolve before merge and removed readiness: checking Automated validation is still running labels Nov 17, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

merge conflict Branch has merge conflicts with its base — author must resolve before merge readiness: action required A blocking check or review needs attention

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant