Skip to content

perf: run the skill dedupe and merge judges on a fresh conversation - #36730

Merged
KiroCrewCaptain merged 1 commit into
mainfrom
fix/30197-fresh-skill-judges
Nov 13, 2026
Merged

KiroCrewCaptain merged 1 commit into
mainfrom
fix/30197-fresh-skill-judges

Conversation

@bolichen97

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: The skill dedupe and merge judges stop resending earlier judges' prompts and replies with every request.

Both judges run on Crew's shared background session _bg. Each judge prompt is self-contained, but each call was sent as one more turn of _bg's conversation, so every request also carried every earlier judge's prompt and reply until _bg recycled at 70% context or 40 prompts. The next _bg job, such as a channel auto-title, inherited that history too. This is the first of the two places #30197 describes; the knowledge pool is the second.

Why it matters

In live Crew traffic about 81% of the skill dedupe judge's input is resent history (#30197). It costs tokens, makes every request larger, and shows the judge earlier candidates and an older skill list it was never meant to read.

Not a goal

  • The knowledge extraction pool (knowledge/llm_pool.py). Its AcpWorker has no cheap in-place reset yet, so a per-call reset there would restart the process for every chunk; that is the second half of [Crew Pulse] Skill judges and knowledge extraction resend their conversation history on every call (~76% of their input) #30197 and a separate PR.
  • Any other _bg caller. Channel auto-titles (messaging/auto_title.py) stack on _bg the same way; they keep the shared conversation here.
  • How long a new conversation takes to become ready. kirocrew-lite has no MCP servers to report, so a new session waits out the MCP-readiness ceiling (_MCP_DRAIN_NO_REPORT_CEILING, 6 s). Each judge now pays that while it holds _bg, so a channel auto-title queued behind a judge can wait up to about 6 s longer; every consolidation job already pays it on its own key. Shortening it is separate.
  • Backends whose _bg provider cannot swap conversations in place. Their judges keep today's shared conversation and recycle.

What changed (motivation → approach → change)

Each skill judge now starts from an empty conversation, so its request carries only its own prompt.

Consolidation already solved the same problem: HistoryConsolidator._call_llm asks background_turn for fresh_conversation=True, which opens a new conversation on the running process before the session permit is released. _dedupe_judge and _merge_skill_update in src/kiro_crew/history_consolidation_skills.py pass the same flag. Each judge keeps what _bg was chosen for: the same warm runtime, the same queue and the same cheap model. Only the history is dropped, and the context recycle no longer runs after a judge because its conversation is already empty.

On _bg the flag applies only when the provider swaps conversations in place (swaps_conversation_in_place, the shared runtime of the session-evicting backends). On a backend with no shared runtime, _bg's title, summary and tag one-liners wait on the _bg entry itself, and the reset fallback would pop it under them; there background_turn (llm_helpers_parts/background_turns.py) recycles as before. docs/system-specs/modules/history.md and session.md say both.

Backwards compatibility

Compatible: the judges' prompts, replies, model and session key are unchanged; only the conversation each one runs in is new.

Removes nothing: no field, key, config value or caller is removed or renamed; background_turn already accepted fresh_conversation.

Tests

  • test/test_consolidation_fresh_conversation.py, on the same real-path harness as consolidation's own tests (a real SessionManager spawning the packaged fake ACP backend, which records each request):
    • a dedupe judge, a merge judge and a chat-title turn on _bg each send turn 1 of their own conversation on one process, and none carries an earlier judge's text;
    • with new_conversation failing, the reset fallback on _bg starts new conversations with none of the judges' history.
  • test/test_consolidation_fresh_conversation.py, seam test: on _bg, a provider that cannot swap in place gets no new_conversation and no reset, only the recycle.
  • test/test_history_coverage.py: the two judge tests that asserted recycle_background was awaited now assert a fresh conversation instead.

All five changed tests fail without the fix: prove.py, which reverts the production hunks and re-runs them, reports PROVEN for test_skill_judges_each_send_only_their_own_prompt_on_bg, test_the_reset_fallback_on_bg_does_not_resume_the_judges_history, test_a_bg_provider_that_cannot_swap_in_place_is_recycled_not_reset, TestDedupeJudge::test_returns_text_and_releases_session and TestMergeSkillUpdate::test_returns_merged_body_and_prompts_with_all_inputs.

Determinism: 20 of 20 runs at -n0, 10 of 10 at -n 4 --dist loadgroup with both files (220 tests), and both files in a shuffled order (seed 16057, 220 passed).

Run on Ubuntu 24.04 (Linux x86_64) from this tree: the profile's setup and all 62 gates, scripts/local-gate.py (related backend and frontend sets), semgrep 1.78.0 on the diff (0 findings) and the flake-pattern ratchet pass. verify_vendor_manifest.py was red only on __pycache__ files the test runs themselves wrote, and green on a clean checkout.

Manual verification

Ran on Ubuntu 24.04 (Linux x86_64), on the real path: a real SessionManager and provider factory spawning the packaged fake ACP backend as a child process, with an isolated data home. A signed-in model cannot be driven unattended there, so the fake backend records each request instead. The script runs a dedupe judge, a merge judge and a chat-title turn on _bg:

run dedupe merge chat title same process
main (c22557cd83) turn 1 turn 2, carries the dedupe prompt turn 3, carries both judges (622 B) yes
this PR turn 1 turn 1, own prompt only turn 1 (12 B) yes
this PR, provider that cannot swap in place turn 1 turn 2 turn 3 yes, no reset

Each judge call takes about 6 s longer on this backend (the merge judge goes from 0.015 s to 6.04 s). All of it is the new session's MCP-readiness wait (see Not a goal); consolidation's fresh conversation already pays the same 6 s on main.

Screenshots / video

N/A: backend-only change, no user-visible surface.

Related Issues

Part of #30197

Builds on #30878 by @jasonzzj97.

Sibling sweep

Swept: git grep -n "background_turn(" -- src/kiro_crew
Sibling: src/kiro_crew/history_consolidation_pass.py -- not affected -- consolidation already passes fresh_conversation=True
Sibling: src/kiro_crew/messaging/auto_title.py -- tracked in #30197
Sibling: src/kiro_crew/knowledge/llm_pool.py -- tracked in #30197

Pattern harvest

Not generalizable: each remaining caller of background_turn on a shared key needs a judgement on whether its prompt is self-contained, which a lint cannot make.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

Contribution License Agreement

Contributions are accepted under the project's Apache-2.0 license.

The skill dedupe and merge judges run on the shared background session
_bg, and each call was sent as one more turn of its conversation. Every
request therefore resent every earlier judge's prompt and reply until
the recycle at 70% context or 40 prompts, about 81% of the dedupe
judge's input, and the next _bg job inherited that history too.

Both judges now pass fresh_conversation=True to background_turn, the
same flag consolidation already uses: each judge leaves _bg on an empty
conversation of the same warm runtime before its permit is released,
and the context recycle no longer runs after it.

On _bg the flag applies only when the provider swaps conversations in
place. On a backend with no shared runtime, _bg's one-liner handles
wait on the _bg entry itself, and the reset fallback would pop it under
them, so such a _bg turn recycles as before.

Builds on #30878 by @jasonzzj97.

Part of #30197

Co-authored-by: Ziji Zhang <42743036+jasonzzj97@users.noreply.github.com>
@github-actions

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision 1946a546aeeb2d4b99bae1b87b82fd9780b94e3e touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ✅ PASS

Premise-level review of 1946a546aeeb2d4b99bae1b87b82fd9780b94e3e — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

First-Principles-Verdict: PASS

Nothing blocks. The two judge call sites reuse consolidation's existing fresh_conversation path, and the new _bg guard is what stops the reset fallback from removing the entry _bg's one-liners wait on.

What this change ships

Inventory (4 items) — 4 justified

Intent: stop the skill dedupe and merge judges from resending earlier judges' history with every request. This is an ADDITION (perf).

  1. The skill dedupe judge starts each call on an empty _bg conversation instead of the shared one — justified
  2. The skill merge judge starts each call on an empty _bg conversation instead of the shared one — justified
  3. On _bg, fresh_conversation takes effect only when the provider has the existing swaps_conversation_in_place property; otherwise _bg recycles as before — justified
  4. A channel auto-title queued behind a judge can wait about 6 s longer, because each new conversation waits for MCP readiness (stated under Not a goal) — justified

[FIRST-PRINCIPLES-REVIEWED] 1946a54

Description read

[DESCRIPTION-READ] ae7022ad57c6c335502a78951ea288eb547b2c8d5ae3bf9fd42e4eb6095fbef5

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of 1946a546aeeb2d4b99bae1b87b82fd9780b94e3e — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

Nothing blocks: the change reuses consolidation's existing fresh_conversation path, and it is held to providers that can swap conversations in place.

Notes

  • Each judge now holds _bg through the roughly 6 s MCP-readiness wait, and channel auto-titles wait behind it. That cost exists only because the judges share _bg, so it is the reason to shorten _MCP_DRAIN_NO_REPORT_CEILING for kirocrew-lite in a follow-up.

[DESIGN-REVIEWED] 1946a54

Description read

[DESCRIPTION-READ] 25b76ad9f4d6409e68b5a0e00e785867894da3a78cc9a65a6d6f81879714c627

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed 1946a546aeeb2d4b99bae1b87b82fd9780b94e3e — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 1946a54

Verdict parsed from the review's SHA-scoped output markers for commit 1946a546aeeb2d4b99bae1b87b82fd9780b94e3e.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 1946a546aeeb2d4b99bae1b87b82fd9780b94e3e false-positive: <reason, 20 to 500 characters>

@bolichen97
bolichen97 enabled auto-merge (squash) November 13, 2026 16:21
@github-actions

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of 1946a546aeeb2d4b99bae1b87b82fd9780b94e3e and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 1946a54

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 1946a546aeeb2d4b99bae1b87b82fd9780b94e3e false-positive: <reason, 20 to 500 characters>

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Nov 13, 2026

@KiroCrewCaptain KiroCrewCaptain left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving at the maintainer's request.

@KiroCrewCaptain
KiroCrewCaptain merged commit f87bc9a into main Nov 13, 2026
96 checks passed
@KiroCrewCaptain
KiroCrewCaptain deleted the fix/30197-fresh-skill-judges branch November 13, 2026 17:46
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Nov 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants