Skip to content

fix(subagent): flag sub-agent runs with no tool calls in completions - #18633

Merged
cruisercohen merged 1 commit into
mainfrom
fix/subagent-claims-verify-13012
Oct 11, 2026
Merged

cruisercohen merged 1 commit into
mainfrom
fix/subagent-claims-verify-13012

Conversation

@cruisercohen

@cruisercohen cruisercohen commented Oct 9, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: A parent agent is told, in the message it reads a sub-agent's result from, when that run cannot have done on-disk work, so it checks claimed changes on disk instead of reporting a sub-agent's self-report as done.

A spawn_run that hands file-writing, git or build work to a reasoning-only agent type (kirocrew-lite, kirocrew-knowledge, whose specs declare "tools": []) gets a completion event that narrates the work, sometimes with invented command output and commit SHAs, while nothing changed on disk. The parent then reports that work to the user as done.

Why it matters

Delegating anything that must land on disk becomes untrustworthy: the user reads a success report for files, commits or PRs that were never made, and only finds out by checking by hand. The issue reports it reproduced across several waves (worktrees left pristine, 22 MDX rewrites reported done with an empty git status).

Not a goal

  • Refusing a dispatch to a zero-tool agent. Whether a task needs writes is not decidable from its prose, and these agents are the right choice for pure reasoning work.
  • Attaching a machine-computed disk/git delta to every completion event, or a three-stage enqueued/accepted/applied lifecycle. That is a larger change to the completion contract.
  • Changing when the synthesis turn arms or fires (fix(chat): skip sub-agent synthesis when one turn reported the batch #17307 covers the one-turn batch).

What changed (motivation → approach → change)

Symptom → cause: nothing tells the parent, where it reads the narration, that the run made no tool call at all, so the narration reads like a report. A lone sub-agent or a single wave digest gets no synthesis turn (docs/system-specs/common/injected-messages.md), so the note has to ride on the completion itself rather than on the synthesis prompt.

Options considered:

  • A. Refuse write-shaped tasks to zero-tool agents at dispatch. Rejected: needs a classifier over free-text tasks; a false positive refuses legitimate reasoning work, a false negative leaves the bug.
  • B. Gateway-computed proof of work on every completion (git delta, SHA check). Rejected for this slice: the child's working tree is not generally known to the gateway, and it changes the completion contract for every agent type.
  • C. Note on the spawn_run dispatch reply. Rejected: the dispatch reply is several turns before the completion event the narration arrives in.
  • D. Fire a synthesis turn for every lone or single-wave completion. Rejected: an extra model turn per spawn, and it changes the synthesis arming contract fix(chat): skip sub-agent synthesis when one turn reported the batch #17307 owns.
  • E. Predict from the agent's spec and the backend whether the run could have had tools. Rejected after trying it: the answer depends on which spec a session resolves (checkout vs user level, mirrored-session trust, user-level-only backends) and on whether the backend enforces tools: [], so it needs that whole resolution chain, says nothing on backends that do not enforce the ban, and misses an agent with tools that only narrated.
  • F. (chosen) Use the run's own observed tool activity. SubagentInfo.made_no_tool_calls is true when the run emitted no tool_call frame (tool_count) and raised no permission request (turns), the same pair the transient-retry and cancel-respawn replay gates already read as "no side effect happened". Counting permission requests too means a backend that surfaces a gated tool only as a permission request still counts as having called a tool. Such a run wrote nothing, on every backend, and the check reads no spec or config.

Change:

  • subagent.py: SubagentInfo.made_no_tool_calls, the one predicate.
  • slack/gateway.py (the single completion consumer): a completed run for which it is true ends its [Subagent completion event] with ⚠ Agent `<id>` made no tool calls in this run, so it did not itself write a file, run a command or make a commit. Treat any such change it reports as a claim: check it on disk before relying on it.; in a wave digest the same line sits under that member's entry. Failed and stopped runs get no note (their text already says the output is not a finished result).
  • dashboard/messaging_api/run_views.py: GET /api/spawn/{id} reports made_no_tool_calls for a completed live run (not for a stopped or failed one).
  • mcp_tools/spawn.py: spawn_sub_agents adds an unverified field carrying the note to a completed record whose status reports made_no_tool_calls: true; a status read without it (a run recovered from disk, a stopped or failed run) attaches nothing. The spawn_run dispatch reply carries no note.
  • constants.py: NO_TOOL_CALLS_NOTE, shared by the gateway and the tool.
  • Specs: subagent.md and injected-messages.md describe the note on each path.

Review findings and disposition

  • Design CONCERNS (note missing from the completion event; lone and single-wave paths get no synthesis turn): fixed. The note is in the completion event and in each digest entry, with tests that drive a lone run and a single wave through the gateway's completion consumer. The dispatch-reply line is dropped, as the Design lane suggested, so the completion event, the digest and spawn_sub_agents answer one question.
  • First Principles CONCERNS (warning predicted from the spec while the gateway already counts the run's real tool calls): fixed by taking that recommendation. The spec/backend predicate, its scope and mirror handling, and the extra session-spec read are gone; the note keys on the run's observed tool activity. This also covers backends that do not enforce tools: [] and agents with tools that only narrated.
  • First Principles CONCERNS (a claim-check sentence in the synthesis prompt covered only runs that did call tools, with no reported defect there): fixed by taking that subtraction. The synthesis prompt is unchanged from main; every zero-tool completion already arrives flagged before any synthesis turn.
  • Design and First Principles watch (every backend, Claude included, must surface gated tools): the predicate also counts the run's permission requests, so a backend that surfaces a gated tool only as a permission request is not flagged; a test pins it. Both counts are the pair the replay gates already rely on.
  • Opus advisory (a user-stopped run could reach spawn_sub_agents as completed and get the note): fixed. The status read sends made_no_tool_calls only for a completed run; tests pin that a stopped or failed run carries none, in the status read and in the completion event.
  • GPT advisories (KAS user-level scope; mirrored-backend checkout shadowing) and First Principles (duplicated tools: [] predicate; an export with no outside caller): resolved by the same change. No spec is read for the note, so there is no lookup scope to get wrong and no second predicate.
  • Opus advisory (spawn_sub_agents status read carries no agent name): resolved. The tool now keys on the status read's made_no_tool_calls, which GET /api/spawn/{id} reports for a completed run.

Backwards compatibility

Compatible: no dispatch is refused and no field is removed; a completed run with no tool call gains one advisory line in its completion event and digest entry, a spawn_sub_agents record from one gains one extra key, and GET /api/spawn/{id} gains one field for a completed run. Runs that called tools see no change.

Removes nothing that callers depend on: the advisory line that was on the unreleased dispatch reply moves to the completion event.

Tests

test/test_subagent_no_tool_calls_note.py:

  • Completion event: a lone run with no tool call ends its event with the note; a run that called tools, one whose tools arrived only as permission requests, a stopped run and a failed run carry none; a single wave digest flags only the member with no tool call.
  • spawn_sub_agents adds unverified only for a status reporting made_no_tool_calls: true, not for false, absent or a non-bool value.
  • GET /api/spawn/{id} reports made_no_tool_calls for a completed run (true with no activity, false after a tool call or a permission request) and omits it for a stopped or failed run.
  • spawn_run still dispatches and its reply carries no note.

The lone, single-wave and status tests fail on the previous head and pass with this change. The sub-agent, spawn, gateway, messaging handler and dashboard chat suites pass. test/test_approval_decide_owner_bound.py fails on origin/main without this change and is unrelated.

Manual verification

N/A — unit coverage sufficient: the change is a count check plus text appended to a completion event, a digest entry and a tool record, and the tests drive the real gateway completion consumer, status handler and tool handlers.

Related Issues

Fixes #13012
Related: #17307

Pattern harvest

Not generalizable: the gap is one specific missing signal on the delegation path, not a recurring code shape.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

Contribution License Agreement

Contributions are accepted under the project's Apache-2.0 license.

@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Oct 9, 2026
@github-actions

github-actions Bot commented Oct 9, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision d9bf56760144ae1589d9a972900f35dae598b322 touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Advisory — does not block merge.

@github-actions

github-actions Bot commented Oct 9, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ✅ PASS

Premise-level review of d9bf56760144ae1589d9a972900f35dae598b322 — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: PASS

Before merge, check that tool_count/turns go up on every selectable backend. If one doesn't count its tool calls, a run that did real work gets flagged as making no tool calls.

What this change ships

Inventory (6 items) — 6 justified

Intent: let a parent agent know when a sub-agent's reported disk work can't have happened, so it checks before telling the user the work is done. FIX.

  1. A lone [Subagent completion event] from a completed run with no tool calls now ends with a ⚠ claim-check line — justified
  2. In a wave digest, that member's entry carries the same ⚠ line — justified
  3. A spawn_sub_agents completed record gains an unverified field for such a run — justified
  4. GET /api/spawn/{id} reports made_no_tool_calls for a completed live run — justified
  5. SubagentInfo.made_no_tool_calls is a new property, read by the gateway and run_views (2 consumers) — justified
  6. constants.NO_TOOL_CALLS_NOTE is shared note text, read by gateway.py and spawn.py (2 consumers) — justified

[FIRST-PRINCIPLES-REVIEWED] d9bf567

Description read

[DESCRIPTION-READ] d291cda34bb608ace7cfa11b289966a785e5f38e8a0615b49b74dd958561580a

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Oct 9, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of d9bf56760144ae1589d9a972900f35dae598b322 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Check that every selectable ACP backend emits a tool_call frame or a permission request for each tool it runs. If one doesn't, the note will wrongly say "did not itself write a file".

[DESIGN-REVIEWED] d9bf567

Description read

[DESCRIPTION-READ] 54cd94abc1c7ae05c41880cefe84cd20b49b0a0ed8cd6a3e31a04c89e4d22794

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 9, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed d9bf56760144ae1589d9a972900f35dae598b322 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] d9bf567

Verdict parsed from the review's SHA-scoped output markers for commit d9bf56760144ae1589d9a972900f35dae598b322.

False positive or not applicable? A repository writer can comment:
/ai-review override fable d9bf56760144ae1589d9a972900f35dae598b322: <one-sentence reason>

@github-actions

github-actions Bot commented Oct 9, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of d9bf56760144ae1589d9a972900f35dae598b322 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] d9bf567

False positive or not applicable? A repository writer can comment:
/ai-review override gpt d9bf56760144ae1589d9a972900f35dae598b322: <one-sentence reason>

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running and removed readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention labels Oct 9, 2026
@cruisercohen
cruisercohen force-pushed the fix/subagent-claims-verify-13012 branch from defe32b to 51c03d0 Compare October 9, 2026 23:42
@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 10, 2026
@cruisercohen
cruisercohen marked this pull request as ready for review October 10, 2026 00:08
@cruisercohen
cruisercohen requested review from a team and KiroCrewCaptain as code owners October 10, 2026 00:08
@cruisercohen
cruisercohen force-pushed the fix/subagent-claims-verify-13012 branch from 51c03d0 to 9b5de75 Compare October 10, 2026 00:42
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Oct 10, 2026
@cruisercohen
cruisercohen enabled auto-merge (squash) October 10, 2026 01:12

@iamwhatever iamwhatever left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This PR tries to stop a parent agent from reporting a zero-tool sub-agent's made-up work as done. It covers only half of the case it targets. I read the full diff and the lane verdicts on head 9b5de75. Design CONCERNS is real and has no disposition. Per docs/system-specs/common/injected-messages.md, a lone sub-agent or a single wave digest normally gets no synthesis turn, so the new claim-check in SUBAGENT_SYNTHESIS_PROMPT (src/kiro_crew/dashboard/state.py:2014) never runs there. The only warning left is the dispatch-reply line added at src/kiro_crew/mcp_tools/spawn.py:1044, which is several turns before the [Subagent completion event] where the made-up narration arrives. That is the common way #13012 reproduces. Suggested fix: reuse _declares_no_tools in the completion-envelope builder (subagent_manager terminal/monitoring path) and append NO_TOOLS_AGENT_NOTE to a zero-tool agent's completion event. Add a test that runs one zero-tool spawn_run to completion and asserts the note is in the envelope. Also answer the two advisory items: the GPT finding that KAS reads only user-level specs, so the work_dir lookup at src/kiro_crew/agent_sdk/drivers/acp.py:234 can flag an agent that does have tools, and the First Principles item that the predicate duplicates AcpClient.authored_spec_declares_zero_tools. Note that auto-merge is armed while these concerns have no answer. — crew-pr-reviewer

@cruisercohen
cruisercohen force-pushed the fix/subagent-claims-verify-13012 branch from 9b5de75 to 475d7d8 Compare October 10, 2026 19:14
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: passed Eligible automated validation passed for the current revision labels Oct 10, 2026
@cruisercohen cruisercohen changed the title fix(subagent): flag zero-tool sub-agents, check claims before synthesis fix(subagent): flag zero-tool sub-agent claims in completion events Oct 10, 2026
@cruisercohen
cruisercohen force-pushed the fix/subagent-claims-verify-13012 branch 2 times, most recently from b3e5832 to f953977 Compare October 10, 2026 20:28
@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 10, 2026
@cruisercohen
cruisercohen force-pushed the fix/subagent-claims-verify-13012 branch from 6bff554 to 8387081 Compare October 10, 2026 22:11
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Oct 10, 2026
@cruisercohen
cruisercohen force-pushed the fix/subagent-claims-verify-13012 branch from 8387081 to ecc9e54 Compare October 11, 2026 00:18
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Oct 11, 2026
@cruisercohen
cruisercohen force-pushed the fix/subagent-claims-verify-13012 branch from ecc9e54 to 3ceb01d Compare October 11, 2026 00:47
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Oct 11, 2026
A sub-agent run that made no tool call wrote no file, ran no command and
made no commit, yet a reasoning-only agent handed write work narrates it
as done. A lone run or a single wave digest gets no synthesis turn, so the
claim note rides on the completion event and each digest entry the parent
reads the narration from. The signal is the run's own observed tool calls
(SubagentInfo.tool_count), so it holds on every backend without reading
specs; spawn_sub_agents reads the same count from the spawn status.

Fixes #13012
@cruisercohen
cruisercohen force-pushed the fix/subagent-claims-verify-13012 branch from 3ceb01d to d9bf567 Compare October 11, 2026 03:03
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Oct 11, 2026

@iamwhatever iamwhatever left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This PR warns a parent, inside the [Subagent completion event] it reads, that a completed sub-agent made no tool calls and so cannot have written files, run commands or made commits. That closes the #13012 gap where a zero-tool agent's invented narration got reported as done. Our CHANGES_REQUESTED on 9b5de75 is met. The note now rides on the completion event and on each wave-digest entry (slack/gateway.py _inject_with_retry), so lone and single-wave runs that get no synthesis turn are covered, and the dispatch-reply line is gone. The spec-based predicate, along with its KAS scope and duplicate-predicate problems, is replaced by SubagentInfo.made_no_tool_calls (tool_count == 0 and turns == 0). I checked that both counters come from the shared run loop: every normalized EVENT_TOOL_CALL and approved child escalation bumps tool_count, and every own permission request bumps turns. The replay gates already trust this pair as "no side effect", so the Design/First Principles backend worry applies equally to existing safety code and is no new risk. I tried to make it lie or fire wrongly. Stopped and failed runs and flush-only records get no note, GET /api/spawn/{id} sends the field only for outcome == "completed", and spawn_sub_agents attaches unverified only on an exact True. The change is purely advisory text, so the worst case is a false warning, never a refused dispatch. Checks are green and the base is main. — crew-pr-reviewer (deep re-check)

@cruisercohen
cruisercohen merged commit 04165c0 into main Oct 11, 2026
102 checks passed
@cruisercohen
cruisercohen deleted the fix/subagent-claims-verify-13012 branch October 11, 2026 06:04
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Oct 11, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Reasoning-only agent types accept write tasks and self-report success; synthesis prompt presumes success with no disk verification

2 participants