Skip to content

fix: release the stale consolidation backoff once a turn is answered - #36632

Merged
bolichen97 merged 1 commit into
mainfrom
fix/33345-release-env-backoff
Nov 13, 2026
Merged

bolichen97 merged 1 commit into
mainfrom
fix/33345-release-env-backoff

Conversation

@bolichen97

@bolichen97 bolichen97 commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: Sessions whose history consolidation was refused while the host could not run turns consolidate as soon as the host can again, not up to a day later.

A consolidation that never reaches a provider arms the session's environment backoff (consolidation_env_failures, consolidation_retry_at). Each failure doubles the wait from 15 minutes up to the 24-hour ceiling. Nothing releases that wait when the host recovers. On a host hit by #33345, every session was refused for days, so most of them now sit at the ceiling: one reporter has 89 sessions in backoff with 7 to 11 environment failures each. Once #33965 ships, those sessions still wait up to 24 hours before their first pass.

Why it matters

Everyone who hit #33345 (any host where a credentials sandbox tool writes into kirocrew-lite.json) gets the fix but keeps no automatic memory for up to another day after it lands, and the "Summarize now" panel keeps showing the queue in retry backoff. The same wait follows any other host-wide outage that ends: a lapsed kiro-cli install, a missing binary.

Not a goal

What changed (motivation → approach → change)

The wait exists so a broken host does not retry every minute. It stops being useful the moment a consolidation turn is answered again, so that answer is what releases it.

Part What it does
clear_consolidation_environment_backoff (history_marker.py, bound on ConversationLog in history.py) Under the transcript lock, sets consolidation_retry_at to 0 and stamps consolidation_env_released. Skips a session with a billed attempt on its current span (the same span test consolidation_retry_state uses), one already released, or one that was deleted. Keeps the failure count.
_release_environment_backoff (history_consolidation.py) Runs that release, off the loop, for every session the idle sweep tracks. It runs once per recovery: armed at start, after the startup scan adds sessions, and after every environment failure.
_consolidate (history_consolidation_pass.py) Calls it only when a provider answered: a usable result or an unusable reply. A turn that raised returns no answer and releases nothing.
_CONSOLIDATION_META_KEYS (history_consolidation_span.py) Carries consolidation_env_released, so a successful mark clears it with the rest of the retry accounting.

Released once per session until it consolidates: a session whose failure is its own (a deleted member store) gets one free retry after recovery, then widens from its kept count up to the ceiling, instead of being released after every other session's success.

The spec in docs/system-specs/modules/memory-skills-hooks.md states the release rule beside the environment backoff.

Backwards compatibility

Compatible: the only new state is one metadata key that a successful mark removes. A session that never had an environment failure is never written. Release only lowers a deadline that the next environment failure raises again.

Removes nothing: no field, path or behaviour is taken away; the release only writes consolidation_retry_at and the new stamp.

Tests

test/test_history_consolidation_env_release.py:

  • test_an_environment_backoff_is_released: a session at 8 environment failures (24 h) becomes eligible, keeps its count, and no message is retired.
  • test_a_session_is_released_once_until_it_consolidates: a second failure is not released again and widens from the kept count; a successful mark clears the stamp.
  • test_a_billed_backoff_holds and test_a_billed_count_from_a_gone_span_does_not_hold_it: a billed attempt on the current span holds the wait; one from a rotated-away span does not.
  • test_a_session_without_an_environment_failure_is_untouched, test_a_deleted_session_is_not_resurrected.
  • test_sessions_refused_while_the_host_was_broken_become_eligible: another session's answered turn releases the refused one, and the answering session keeps its own billed wait.
  • test_a_turn_that_raised_inside_the_stream_releases_nothing, test_a_turn_that_never_reached_the_provider_releases_nothing.
  • test_sessions_the_startup_scan_adds_late_are_still_released: a release spent before the startup scan runs again for the seeded sessions.
  • test_the_release_runs_once_per_recovery.

test/test_history_bindings.py lists the new ConversationLog member, and test/test_history_consolidation_facade.py the new HistoryConsolidator member.

Proof: PROVEN test/test_history_bindings.py::test_the_member_set_is_the_frozen_inventory, test/test_history_consolidation_env_release.py::TestAReachedProviderReleasesTheBackoff::test_sessions_refused_while_the_host_was_broken_become_eligible, test/test_history_consolidation_facade.py::test_the_class_keeps_every_base_member_with_its_kind_and_signature (head c1fb02f)

Determinism: the new file passed 20 runs at -n0, and with test_history_consolidation_retry_accounting.py 10 runs at -n 4. No shuffle plugin is installed in this venv, so order was varied only by xdist scheduling.

Manual verification

macOS (Apple Silicon), Python 3.12, a script in an isolated HOME that drives the real installer-written kirocrew-lite.json, the real background_turn pre-spawn check and repair, a real ConversationLog and HistoryConsolidator. The creds-agent entries were copied from this Mac's own AIM-injected ~/.kiro/agents/kirocrew-lite.json. The only stub is the session manager's provider, which answers one JSON reply. Session A is live; session B carries 8 environment failures, the reporter's state.

Base (origin/main with #33965) vs this head, same script:

                                    origin/main       this PR
lite spec rewritten from template   yes               yes
A turn 2 reaches the provider       yes               yes
B after A's answered turn           eligible=False    eligible=True, retry_at=0.0
                                    (24.0 h wait)     released=True, 4 messages unread
B consolidation runs a turn         no                yes

Gates on c1fb02f: isort, flake8, mypy on the changed modules exit 0; semgrep (--baseline-commit merge-base, --error) exit 0; test/test_flake_pattern_ratchet.py exit 0; test/test_history*.py, test_messaging_dispatch.py, test_frontmatter.py, test_prompt_handoff.py and the flake ratchet: 2,290 passed. scripts/local-gate.py backend batch 1 ran 17,049 tests with 12 failures in ACP model-cache and child-reap tests; the same 12 node ids fail on a clean origin/main on this host.

Related Issues

Part of #33345

Sibling sweep

Swept: git grep -nE "record_consolidation_environment_failure|_prefs_retry" -- src/
Sibling: src/kiro_crew/history_consolidation.py -- not affected -- the preference-only window's _prefs_retry is in memory and starts empty on every restart
Sibling: src/kiro_crew/history_marker.py -- changed here

Pattern harvest

Not generalizable: a backoff for a host-wide failure needs a release on recovery; this is the only durable host-wide backoff in the codebase.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@bolichen97
bolichen97 requested review from a team and KiroCrewCaptain as code owners November 13, 2026 13:11
@bolichen97
bolichen97 enabled auto-merge (squash) November 13, 2026 13:20
@github-actions

github-actions Bot commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of c1fb02f2077bfb7f3a08d3a7faf91da31eaae8e1 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] c1fb02f

False positive or not applicable? A repository writer can comment:
/ai-review override gpt c1fb02f2077bfb7f3a08d3a7faf91da31eaae8e1 false-positive: <reason, 20 to 500 characters>

@github-actions

github-actions Bot commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision c1fb02f2077bfb7f3a08d3a7faf91da31eaae8e1 touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions

github-actions Bot commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of c1fb02f2077bfb7f3a08d3a7faf91da31eaae8e1 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

Nothing blocks. Clearing the backoff when a provider answers deals with the cause, the stale host-wide wait. The once-per-session stamp keeps a session that fails on its own from being retried for free over and over.

Notes

  • The release only reaches sessions in _last_activity at that moment. A session the sweep starts tracking after the release has run, without a new environment failure to re-arm it, keeps its old deadline until it expires. That is the same day-long wait for a session the sweep picks up after recovery. Re-arming _env_backoff_pending when the sweep starts tracking a new key would close that gap.

[DESIGN-REVIEWED] c1fb02f

Description read

[DESCRIPTION-READ] fdcd98d6d54866ce011254066a6190d43262b9facabfe4502b13f45865e2d4d6

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ✅ PASS

Premise-level review of c1fb02f2077bfb7f3a08d3a7faf91da31eaae8e1 — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

First-Principles-Verdict: PASS

Nothing blocks: the release is the missing recovery signal for a durable host-wide backoff, and no existing path clears that backoff.

What this change ships

Inventory (6 items) — 6 justified

Intent: after a host-wide outage ends, the sessions it refused should consolidate right away instead of waiting up to 24 hours. This is a FIX.

  1. After a provider-answered consolidation turn, every tracked session sitting in environment backoff becomes eligible to retry at once — justified
  2. Each session gets that release once until it next consolidates, via the persisted consolidation_env_released stamp — justified
  3. A session with a billed attempt on its current span keeps its wait — justified
  4. A turn that raised, or never reached the provider, releases nothing — justified
  5. Sessions added by the startup scan after an early release get released too, because the scan re-arms it — justified
  6. The spec paragraph in memory-skills-hooks.md describes the release rule — justified

[FIRST-PRINCIPLES-REVIEWED] c1fb02f

Description read

[DESCRIPTION-READ] e4d710274f1076b4aa5ff4719915354b49f481bd6348bdf8388b6f78e8979236

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed c1fb02f2077bfb7f3a08d3a7faf91da31eaae8e1 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] c1fb02f

Verdict parsed from the review's SHA-scoped output markers for commit c1fb02f2077bfb7f3a08d3a7faf91da31eaae8e1.

False positive or not applicable? A repository writer can comment:
/ai-review override fable c1fb02f2077bfb7f3a08d3a7faf91da31eaae8e1 false-positive: <reason, 20 to 500 characters>

@KiroCrewCaptain

KiroCrewCaptain commented Nov 13, 2026 •

Copy link
Copy Markdown
Collaborator

Code change approver recommendation: A person should decide

Why: the decider thinks the change is probably fine but estimates a 22% chance that this change introduces a defect, above the 12% limit for approving on its own.

Approve. When a consolidation turn is answered, this PR releases the durable environment-failure backoff on every tracked session. The release runs under the transcript lock, keeps each session's failure count and skips sessions with a billed attempt on their current span. A new stamp, cleared when the session next consolidates, limits each session to one release. A deferred flag makes the release fire once per recovery: at start, after the startup scan, and after each environment failure. It fires only on a real provider answer, not on turns that raised or were never dispatched. Every reader of the deadline reads it uncached, so a release takes effect at once. Tests cover each path. The CI failures are rate-limit and skipped-lane infrastructure problems. The remaining gaps are small: a startup-loaded session refused by the idle sweep can wait up to 3 hours after release, and a session that fails again before it consolidates is not released at the next recovery. Neither blocks.

An automated recommendation, not a review: a maintainer still decides. It approves on its own only when the estimated chance of a defect is at most 12% and the change has at most 400 changed lines. Decider v3c (217fe607) · commit c1fb02f207 · 396 changed lines. If this recommendation is wrong, reply here: replies are how the approver is evaluated.

@bolichen97
bolichen97 force-pushed the fix/33345-release-env-backoff branch from 7ec27e8 to d60abf3 Compare November 13, 2026 13:56
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Nov 13, 2026
@bolichen97
bolichen97 force-pushed the fix/33345-release-env-backoff branch from d60abf3 to fe00590 Compare November 13, 2026 14:58
@bolichen97

Copy link
Copy Markdown
Contributor Author

Rebased: main moved in docs/system-specs/modules/cli.md, docs/system-specs/modules/workflows.md, src/kiro_crew/cli*.py, test/test_frontmatter.py, test/test_history_retention.py, test/test_prompt_handoff.py since this head went green (8a197a8 -> d4b0a0b); CI re-running.

@bolichen97

bolichen97 commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor Author

self-added: yes
mechanism: one-shot release of the environment backoff after an answered turn

  • span=e4634eec0079 released sessions keep the idle throttle; a lock timeout skips one key's release

Rebutted: both items in this span leave a session no worse than main, and nothing is marked or retired.
F1 (idle throttle): _history_consolidated holds a refused seed for one idle window (history_consolidation.py:1069-1070); after the release the next sweep past that window consolidates it, versus up to 24 h on main.
F2 (lock timeout): a HistoryLockTimeout during one key's release leaves that key on its old deadline, which is main's behaviour; re-arming on any contended key would re-scan every tracked session on each answered turn.
Both fixes widen the release for a window bounded by an existing setting; the adjudication on this head downgraded both as disproportionate-remedy.

A history consolidation that never reaches a provider (kiro-cli
missing, or the lite spec refused before its session starts, as in
#33345) arms the session's environment backoff, and that wait widens
to a day. Nothing released it when the host recovered, so after the
#33965 fix landed every session refused meanwhile still sat out up to
24 hours before consolidating.

The first consolidation turn a provider answers after a process start,
after the startup scan adds sessions, or after any environment failure
now releases the environment backoff of every tracked session. A turn
that raised releases nothing. Each session is released once until it
next consolidates and keeps its failure count, so a session whose
failure is its own resumes widening to the ceiling. A session with a
billed attempt on its current span keeps its wait, and nothing is
marked consolidated.

Part of #33345
@bolichen97
bolichen97 force-pushed the fix/33345-release-env-backoff branch from fe00590 to c1fb02f Compare November 13, 2026 15:55
@bolichen97

Copy link
Copy Markdown
Contributor Author

Rebased: main moved in docs/system-specs/modules/app-kit-platform.md, docs/system-specs/modules/cli.md, docs/system-specs/modules/computer-use.md, docs/system-specs/modules/config.md, docs/system-specs/modules/security.md, docs/system-specs/modules/session-control.md, docs/system-specs/modules/session.md, docs/system-specs/modules/subagent.md, src/kiro_crew/autonudge_selfarm.py, src/kiro_crew/cli_commands.py, src/kiro_crew/cli_commands_memory.py, src/kiro_crew/cli_parser_agent.py, +6 more since this head went green (d4b0a0b -> 3677e29); CI re-running. The base carries #36652, which fixes the test_mate_conductor_parity red.

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Nov 13, 2026
@bolichen97
bolichen97 merged commit 0e5b203 into main Nov 13, 2026
133 of 144 checks passed
@bolichen97
bolichen97 deleted the fix/33345-release-env-backoff branch November 13, 2026 17:28
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Nov 13, 2026
bolichen97 added a commit that referenced this pull request Nov 13, 2026
…ox tool (#33345) (#36777)

* fix: rewrite the guest spec from its template before refusing a turn (#34253)

A tool that writes its own MCP server into every agent spec under the
kiro agents directory leaves kirocrew-guest.json mounting that server,
and a deny_all_tools channel turn refuses before its spawn on a guest
spec that mounts anything. Only a rebuild put the file back, so every
non-operator turn was refused until then.

When the pre-spawn read refuses, the channel pipeline now rewrites an
existing kirocrew-guest.json that is not its template through the same
governed writer the installer uses, resets every live session bound to
the guest agent, and reads the spec again; that second read decides.
A live guest process may still mount the injected server after the
file reads clean, since kiro-cli applies a reload only at a turn
boundary and not on every version. A turn whose
spawn overlaps a repair, which the resets may not reach, is refused:
the count of started repairs is read before the vouching read and the
read after the start confirms only when no repair started since and
none is running. Each reset is bounded so a hung teardown cannot keep
every guest turn refused. A missing file is not created. _spec_drift takes its
reader's operation label from the caller.

Closes #33522

(cherry picked from commit 387ea66)

* fix: rewrite the lite spec from its template before refusing a turn (#33965)

A tool that writes its own MCP server into every agent spec under the
agents directory leaves kirocrew-lite.json mounting a server. The
background helper correctly refuses to start a session on such a spec,
but nothing put the file back before the next rebuild, so every
background turn, history consolidation included, was refused.

Before refusing, the helper now rewrites an existing kirocrew-lite.json
that is not its template, through the same governed writer the
installer uses, and reads the spec again. That second read still
decides; a spec that still mounts something is refused as before, and
a missing file is never created. The rewrite and the session resets
after it run through the same code the guest spec repair uses, so each
reset is bounded and one hung teardown cannot hold background turns.

A session that started before the rewrite may have loaded the old
file, so only the read before the rewrite vouches for a session, and
not while a rewrite runs. A rewrite resets the _bg and _consolidate
sessions, the two that outlive a turn and can start without the
pre-spawn read, the turn's own included, and those resets finish even
when the turn is cancelled. On a backend with no prepared projection,
which cannot tell such a session from a new one, the turn that rewrote
the file is refused and its own session reset, and so is a turn whose
pre-spawn read overlapped a rewrite. A _bg start still in flight when
the rewrite resets _bg is shut down instead of registered, so no later
turn can take a session that loaded the old file. The next turn starts
on the rewritten file.

Closes #33345

(cherry picked from commit ca95636)

* fix: release the stale consolidation backoff once a turn is answered (#36632)

A history consolidation that never reaches a provider (kiro-cli
missing, or the lite spec refused before its session starts, as in
#33345) arms the session's environment backoff, and that wait widens
to a day. Nothing released it when the host recovered, so after the
#33965 fix landed every session refused meanwhile still sat out up to
24 hours before consolidating.

The first consolidation turn a provider answers after a process start,
after the startup scan adds sessions, or after any environment failure
now releases the environment backoff of every tracked session. A turn
that raised releases nothing. Each session is released once until it
next consolidates and keeps its failure count, so a session whose
failure is its own resumes widening to the ceiling. A session with a
billed attempt on its current span keeps its wait, and nothing is
marked consolidated.

Part of #33345

(cherry picked from commit 0e5b203)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants