Repository navigation
perf: run the skill dedupe and merge judges on a fresh conversation - #36730
Conversation
The skill dedupe and merge judges run on the shared background session _bg, and each call was sent as one more turn of its conversation. Every request therefore resent every earlier judge's prompt and reply until the recycle at 70% context or 40 prompts, about 81% of the dedupe judge's input, and the next _bg job inherited that history too. Both judges now pass fresh_conversation=True to background_turn, the same flag consolidation already uses: each judge leaves _bg on an empty conversation of the same warm runtime before its permit is released, and the context recycle no longer runs after it. On _bg the flag applies only when the provider swaps conversations in place. On a backend with no shared runtime, _bg's one-liner handles wait on the _bg entry itself, and the reset fallback would pop it under them, so such a _bg turn recycles as before. Builds on #30878 by @jasonzzj97. Part of #30197 Co-authored-by: Ziji Zhang <42743036+jasonzzj97@users.noreply.github.com>
UX Review (Opus 5.5) — ⏭️ skippedRevision |
First Principles Review (Opus 5.5) — ✅ PASSPremise-level review of First-Principles-Verdict: PASS Nothing blocks. The two judge call sites reuse consolidation's existing What this change shipsInventory (4 items) — 4 justifiedIntent: stop the skill dedupe and merge judges from resending earlier judges' history with every request. This is an ADDITION (
[FIRST-PRINCIPLES-REVIEWED] 1946a54 Description read[DESCRIPTION-READ] ae7022ad57c6c335502a78951ea288eb547b2c8d5ae3bf9fd42e4eb6095fbef5 That sha256 names the PR description this verdict read. Recompute it with |
Design Review (Opus 5.5) — ✅ PASSDesign-level review of Design-Verdict: PASS Nothing blocks: the change reuses consolidation's existing Notes
[DESIGN-REVIEWED] 1946a54 Description read[DESCRIPTION-READ] 25b76ad9f4d6409e68b5a0e00e785867894da3a78cc9a65a6d6f81879714c627 That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
Opus 5.5 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
GPT 6.1 Review — ✅ no blocking findingsGPT 6.1 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
KiroCrewCaptain
left a comment
There was a problem hiding this comment.
Approving at the maintainer's request.
Problem / Motivation
Goal: The skill dedupe and merge judges stop resending earlier judges' prompts and replies with every request.
Both judges run on Crew's shared background session
_bg. Each judge prompt is self-contained, but each call was sent as one more turn of_bg's conversation, so every request also carried every earlier judge's prompt and reply until_bgrecycled at 70% context or 40 prompts. The next_bgjob, such as a channel auto-title, inherited that history too. This is the first of the two places #30197 describes; the knowledge pool is the second.Why it matters
In live Crew traffic about 81% of the skill dedupe judge's input is resent history (#30197). It costs tokens, makes every request larger, and shows the judge earlier candidates and an older skill list it was never meant to read.
Not a goal
knowledge/llm_pool.py). ItsAcpWorkerhas no cheap in-place reset yet, so a per-call reset there would restart the process for every chunk; that is the second half of [Crew Pulse] Skill judges and knowledge extraction resend their conversation history on every call (~76% of their input) #30197 and a separate PR._bgcaller. Channel auto-titles (messaging/auto_title.py) stack on_bgthe same way; they keep the shared conversation here.kirocrew-litehas no MCP servers to report, so a new session waits out the MCP-readiness ceiling (_MCP_DRAIN_NO_REPORT_CEILING, 6 s). Each judge now pays that while it holds_bg, so a channel auto-title queued behind a judge can wait up to about 6 s longer; every consolidation job already pays it on its own key. Shortening it is separate._bgprovider cannot swap conversations in place. Their judges keep today's shared conversation and recycle.What changed (motivation → approach → change)
Each skill judge now starts from an empty conversation, so its request carries only its own prompt.
Consolidation already solved the same problem:
HistoryConsolidator._call_llmasksbackground_turnforfresh_conversation=True, which opens a new conversation on the running process before the session permit is released._dedupe_judgeand_merge_skill_updateinsrc/kiro_crew/history_consolidation_skills.pypass the same flag. Each judge keeps what_bgwas chosen for: the same warm runtime, the same queue and the same cheap model. Only the history is dropped, and the context recycle no longer runs after a judge because its conversation is already empty.On
_bgthe flag applies only when the provider swaps conversations in place (swaps_conversation_in_place, the shared runtime of the session-evicting backends). On a backend with no shared runtime,_bg's title, summary and tag one-liners wait on the_bgentry itself, and the reset fallback would pop it under them; therebackground_turn(llm_helpers_parts/background_turns.py) recycles as before.docs/system-specs/modules/history.mdandsession.mdsay both.Backwards compatibility
Compatible: the judges' prompts, replies, model and session key are unchanged; only the conversation each one runs in is new.
Removes nothing: no field, key, config value or caller is removed or renamed;
background_turnalready acceptedfresh_conversation.Tests
test/test_consolidation_fresh_conversation.py, on the same real-path harness as consolidation's own tests (a realSessionManagerspawning the packaged fake ACP backend, which records each request):_bgeach send turn 1 of their own conversation on one process, and none carries an earlier judge's text;new_conversationfailing, the reset fallback on_bgstarts new conversations with none of the judges' history.test/test_consolidation_fresh_conversation.py, seam test: on_bg, a provider that cannot swap in place gets nonew_conversationand no reset, only the recycle.test/test_history_coverage.py: the two judge tests that assertedrecycle_backgroundwas awaited now assert a fresh conversation instead.All five changed tests fail without the fix:
prove.py, which reverts the production hunks and re-runs them, reports PROVEN fortest_skill_judges_each_send_only_their_own_prompt_on_bg,test_the_reset_fallback_on_bg_does_not_resume_the_judges_history,test_a_bg_provider_that_cannot_swap_in_place_is_recycled_not_reset,TestDedupeJudge::test_returns_text_and_releases_sessionandTestMergeSkillUpdate::test_returns_merged_body_and_prompts_with_all_inputs.Determinism: 20 of 20 runs at
-n0, 10 of 10 at-n 4 --dist loadgroupwith both files (220 tests), and both files in a shuffled order (seed 16057, 220 passed).Run on Ubuntu 24.04 (Linux x86_64) from this tree: the profile's setup and all 62 gates,
scripts/local-gate.py(related backend and frontend sets), semgrep 1.78.0 on the diff (0 findings) and the flake-pattern ratchet pass.verify_vendor_manifest.pywas red only on__pycache__files the test runs themselves wrote, and green on a clean checkout.Manual verification
Ran on Ubuntu 24.04 (Linux x86_64), on the real path: a real
SessionManagerand provider factory spawning the packaged fake ACP backend as a child process, with an isolated data home. A signed-in model cannot be driven unattended there, so the fake backend records each request instead. The script runs a dedupe judge, a merge judge and a chat-title turn on_bg:c22557cd83)Each judge call takes about 6 s longer on this backend (the merge judge goes from 0.015 s to 6.04 s). All of it is the new session's MCP-readiness wait (see Not a goal); consolidation's fresh conversation already pays the same 6 s on main.
Screenshots / video
N/A: backend-only change, no user-visible surface.
Related Issues
Part of #30197
Builds on #30878 by @jasonzzj97.
Sibling sweep
Swept: git grep -n "background_turn(" -- src/kiro_crew
Sibling: src/kiro_crew/history_consolidation_pass.py -- not affected -- consolidation already passes fresh_conversation=True
Sibling: src/kiro_crew/messaging/auto_title.py -- tracked in #30197
Sibling: src/kiro_crew/knowledge/llm_pool.py -- tracked in #30197
Pattern harvest
Not generalizable: each remaining caller of background_turn on a shared key needs a judgement on whether its prompt is self-contained, which a lint cannot make.
Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)Contribution License Agreement
Contributions are accepted under the project's Apache-2.0 license.