Repository navigation
fix: release the stale consolidation backoff once a turn is answered - #36632
Conversation
GPT 6.1 Review — ✅ no blocking findingsGPT 6.1 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
UX Review (Opus 5.5) — ⏭️ skippedRevision |
Design Review (Opus 5.5) — ✅ PASSDesign-level review of Design-Verdict: PASS Nothing blocks. Clearing the backoff when a provider answers deals with the cause, the stale host-wide wait. The once-per-session stamp keeps a session that fails on its own from being retried for free over and over. Notes
[DESIGN-REVIEWED] c1fb02f Description read[DESCRIPTION-READ] fdcd98d6d54866ce011254066a6190d43262b9facabfe4502b13f45865e2d4d6 That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
First Principles Review (Opus 5.5) — ✅ PASSPremise-level review of First-Principles-Verdict: PASS Nothing blocks: the release is the missing recovery signal for a durable host-wide backoff, and no existing path clears that backoff. What this change shipsInventory (6 items) — 6 justifiedIntent: after a host-wide outage ends, the sessions it refused should consolidate right away instead of waiting up to 24 hours. This is a FIX.
[FIRST-PRINCIPLES-REVIEWED] c1fb02f Description read[DESCRIPTION-READ] e4d710274f1076b4aa5ff4719915354b49f481bd6348bdf8388b6f78e8979236 That sha256 names the PR description this verdict read. Recompute it with |
Opus 5.5 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
Code change approver recommendation: A person should decideWhy: the decider thinks the change is probably fine but estimates a 22% chance that this change introduces a defect, above the 12% limit for approving on its own. Approve. When a consolidation turn is answered, this PR releases the durable environment-failure backoff on every tracked session. The release runs under the transcript lock, keeps each session's failure count and skips sessions with a billed attempt on their current span. A new stamp, cleared when the session next consolidates, limits each session to one release. A deferred flag makes the release fire once per recovery: at start, after the startup scan, and after each environment failure. It fires only on a real provider answer, not on turns that raised or were never dispatched. Every reader of the deadline reads it uncached, so a release takes effect at once. Tests cover each path. The CI failures are rate-limit and skipped-lane infrastructure problems. The remaining gaps are small: a startup-loaded session refused by the idle sweep can wait up to 3 hours after release, and a session that fails again before it consolidates is not released at the next recovery. Neither blocks. An automated recommendation, not a review: a maintainer still decides. It approves on its own only when the estimated chance of a defect is at most 12% and the change has at most 400 changed lines. Decider v3c (217fe607) · commit |
7ec27e8 to
d60abf3
Compare
d60abf3 to
fe00590
Compare
|
self-added: yes
|
A history consolidation that never reaches a provider (kiro-cli missing, or the lite spec refused before its session starts, as in #33345) arms the session's environment backoff, and that wait widens to a day. Nothing released it when the host recovered, so after the #33965 fix landed every session refused meanwhile still sat out up to 24 hours before consolidating. The first consolidation turn a provider answers after a process start, after the startup scan adds sessions, or after any environment failure now releases the environment backoff of every tracked session. A turn that raised releases nothing. Each session is released once until it next consolidates and keeps its failure count, so a session whose failure is its own resumes widening to the ceiling. A session with a billed attempt on its current span keeps its wait, and nothing is marked consolidated. Part of #33345
fe00590 to
c1fb02f
Compare
|
Rebased: main moved in docs/system-specs/modules/app-kit-platform.md, docs/system-specs/modules/cli.md, docs/system-specs/modules/computer-use.md, docs/system-specs/modules/config.md, docs/system-specs/modules/security.md, docs/system-specs/modules/session-control.md, docs/system-specs/modules/session.md, docs/system-specs/modules/subagent.md, src/kiro_crew/autonudge_selfarm.py, src/kiro_crew/cli_commands.py, src/kiro_crew/cli_commands_memory.py, src/kiro_crew/cli_parser_agent.py, +6 more since this head went green (d4b0a0b -> 3677e29); CI re-running. The base carries #36652, which fixes the test_mate_conductor_parity red. |
…ox tool (#33345) (#36777) * fix: rewrite the guest spec from its template before refusing a turn (#34253) A tool that writes its own MCP server into every agent spec under the kiro agents directory leaves kirocrew-guest.json mounting that server, and a deny_all_tools channel turn refuses before its spawn on a guest spec that mounts anything. Only a rebuild put the file back, so every non-operator turn was refused until then. When the pre-spawn read refuses, the channel pipeline now rewrites an existing kirocrew-guest.json that is not its template through the same governed writer the installer uses, resets every live session bound to the guest agent, and reads the spec again; that second read decides. A live guest process may still mount the injected server after the file reads clean, since kiro-cli applies a reload only at a turn boundary and not on every version. A turn whose spawn overlaps a repair, which the resets may not reach, is refused: the count of started repairs is read before the vouching read and the read after the start confirms only when no repair started since and none is running. Each reset is bounded so a hung teardown cannot keep every guest turn refused. A missing file is not created. _spec_drift takes its reader's operation label from the caller. Closes #33522 (cherry picked from commit 387ea66) * fix: rewrite the lite spec from its template before refusing a turn (#33965) A tool that writes its own MCP server into every agent spec under the agents directory leaves kirocrew-lite.json mounting a server. The background helper correctly refuses to start a session on such a spec, but nothing put the file back before the next rebuild, so every background turn, history consolidation included, was refused. Before refusing, the helper now rewrites an existing kirocrew-lite.json that is not its template, through the same governed writer the installer uses, and reads the spec again. That second read still decides; a spec that still mounts something is refused as before, and a missing file is never created. The rewrite and the session resets after it run through the same code the guest spec repair uses, so each reset is bounded and one hung teardown cannot hold background turns. A session that started before the rewrite may have loaded the old file, so only the read before the rewrite vouches for a session, and not while a rewrite runs. A rewrite resets the _bg and _consolidate sessions, the two that outlive a turn and can start without the pre-spawn read, the turn's own included, and those resets finish even when the turn is cancelled. On a backend with no prepared projection, which cannot tell such a session from a new one, the turn that rewrote the file is refused and its own session reset, and so is a turn whose pre-spawn read overlapped a rewrite. A _bg start still in flight when the rewrite resets _bg is shut down instead of registered, so no later turn can take a session that loaded the old file. The next turn starts on the rewritten file. Closes #33345 (cherry picked from commit ca95636) * fix: release the stale consolidation backoff once a turn is answered (#36632) A history consolidation that never reaches a provider (kiro-cli missing, or the lite spec refused before its session starts, as in #33345) arms the session's environment backoff, and that wait widens to a day. Nothing released it when the host recovered, so after the #33965 fix landed every session refused meanwhile still sat out up to 24 hours before consolidating. The first consolidation turn a provider answers after a process start, after the startup scan adds sessions, or after any environment failure now releases the environment backoff of every tracked session. A turn that raised releases nothing. Each session is released once until it next consolidates and keeps its failure count, so a session whose failure is its own resumes widening to the ceiling. A session with a billed attempt on its current span keeps its wait, and nothing is marked consolidated. Part of #33345 (cherry picked from commit 0e5b203)
Problem / Motivation
Goal: Sessions whose history consolidation was refused while the host could not run turns consolidate as soon as the host can again, not up to a day later.
A consolidation that never reaches a provider arms the session's environment backoff (
consolidation_env_failures,consolidation_retry_at). Each failure doubles the wait from 15 minutes up to the 24-hour ceiling. Nothing releases that wait when the host recovers. On a host hit by #33345, every session was refused for days, so most of them now sit at the ceiling: one reporter has 89 sessions in backoff with 7 to 11 environment failures each. Once #33965 ships, those sessions still wait up to 24 hours before their first pass.Why it matters
Everyone who hit #33345 (any host where a credentials sandbox tool writes into
kirocrew-lite.json) gets the fix but keeps no automatic memory for up to another day after it lands, and the "Summarize now" panel keeps showing the queue in retry backoff. The same wait follows any other host-wide outage that ends: a lapsed kiro-cli install, a missing binary.Not a goal
What changed (motivation → approach → change)
The wait exists so a broken host does not retry every minute. It stops being useful the moment a consolidation turn is answered again, so that answer is what releases it.
clear_consolidation_environment_backoff(history_marker.py, bound onConversationLoginhistory.py)consolidation_retry_atto 0 and stampsconsolidation_env_released. Skips a session with a billed attempt on its current span (the same span testconsolidation_retry_stateuses), one already released, or one that was deleted. Keeps the failure count._release_environment_backoff(history_consolidation.py)_consolidate(history_consolidation_pass.py)_CONSOLIDATION_META_KEYS(history_consolidation_span.py)consolidation_env_released, so a successful mark clears it with the rest of the retry accounting.Released once per session until it consolidates: a session whose failure is its own (a deleted member store) gets one free retry after recovery, then widens from its kept count up to the ceiling, instead of being released after every other session's success.
The spec in
docs/system-specs/modules/memory-skills-hooks.mdstates the release rule beside the environment backoff.Backwards compatibility
Compatible: the only new state is one metadata key that a successful mark removes. A session that never had an environment failure is never written. Release only lowers a deadline that the next environment failure raises again.
Removes nothing: no field, path or behaviour is taken away; the release only writes
consolidation_retry_atand the new stamp.Tests
test/test_history_consolidation_env_release.py:test_an_environment_backoff_is_released: a session at 8 environment failures (24 h) becomes eligible, keeps its count, and no message is retired.test_a_session_is_released_once_until_it_consolidates: a second failure is not released again and widens from the kept count; a successful mark clears the stamp.test_a_billed_backoff_holdsandtest_a_billed_count_from_a_gone_span_does_not_hold_it: a billed attempt on the current span holds the wait; one from a rotated-away span does not.test_a_session_without_an_environment_failure_is_untouched,test_a_deleted_session_is_not_resurrected.test_sessions_refused_while_the_host_was_broken_become_eligible: another session's answered turn releases the refused one, and the answering session keeps its own billed wait.test_a_turn_that_raised_inside_the_stream_releases_nothing,test_a_turn_that_never_reached_the_provider_releases_nothing.test_sessions_the_startup_scan_adds_late_are_still_released: a release spent before the startup scan runs again for the seeded sessions.test_the_release_runs_once_per_recovery.test/test_history_bindings.pylists the newConversationLogmember, andtest/test_history_consolidation_facade.pythe newHistoryConsolidatormember.Proof: PROVEN test/test_history_bindings.py::test_the_member_set_is_the_frozen_inventory, test/test_history_consolidation_env_release.py::TestAReachedProviderReleasesTheBackoff::test_sessions_refused_while_the_host_was_broken_become_eligible, test/test_history_consolidation_facade.py::test_the_class_keeps_every_base_member_with_its_kind_and_signature (head c1fb02f)
Determinism: the new file passed 20 runs at
-n0, and withtest_history_consolidation_retry_accounting.py10 runs at-n 4. No shuffle plugin is installed in this venv, so order was varied only by xdist scheduling.Manual verification
macOS (Apple Silicon), Python 3.12, a script in an isolated
HOMEthat drives the real installer-writtenkirocrew-lite.json, the realbackground_turnpre-spawn check and repair, a realConversationLogandHistoryConsolidator. Thecreds-agententries were copied from this Mac's own AIM-injected~/.kiro/agents/kirocrew-lite.json. The only stub is the session manager's provider, which answers one JSON reply. Session A is live; session B carries 8 environment failures, the reporter's state.Base (
origin/mainwith #33965) vs this head, same script:Gates on c1fb02f: isort, flake8, mypy on the changed modules exit 0; semgrep (
--baseline-commitmerge-base,--error) exit 0;test/test_flake_pattern_ratchet.pyexit 0;test/test_history*.py,test_messaging_dispatch.py,test_frontmatter.py,test_prompt_handoff.pyand the flake ratchet: 2,290 passed.scripts/local-gate.pybackend batch 1 ran 17,049 tests with 12 failures in ACP model-cache and child-reap tests; the same 12 node ids fail on a cleanorigin/mainon this host.Related Issues
Part of #33345
Sibling sweep
Swept: git grep -nE "record_consolidation_environment_failure|_prefs_retry" -- src/
Sibling: src/kiro_crew/history_consolidation.py -- not affected -- the preference-only window's
_prefs_retryis in memory and starts empty on every restartSibling: src/kiro_crew/history_marker.py -- changed here
Pattern harvest
Not generalizable: a backoff for a host-wide failure needs a release on recovery; this is the only durable host-wide backoff in the codebase.
Checklist