Skip to content

fix(history): isolate consolidation sessions - #9036

Closed
Premshay wants to merge 1 commit into
kirodotdev:mainfrom
Premshay:upstream/history-consolidation-session
Closed

Premshay wants to merge 1 commit into
kirodotdev:mainfrom
Premshay:upstream/history-consolidation-session

Conversation

@Premshay

@Premshay Premshay commented Sep 6, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: Long history consolidation no longer makes lightweight background jobs (cron, maintenance) wait behind it.

History consolidation uses the shared _bg session, coupling long memory extraction to latency-sensitive background micro-jobs.

Why it matters

Lightweight maintenance queues behind consolidation even though it neither needs its context nor its admission path.

Not a goal

  • Changing what consolidation extracts or how its prompt is built.
  • Changing the _bg session's behaviour for the jobs that keep using it.
  • Keeping a consolidation process alive between passes; it expires when idle.

What changed (motivation → approach → change)

History extraction runs on its own stateless _consolidate session instead of _bg, so its long turns no longer hold the semaphore the background micro-jobs share; _bg behaviour is unchanged.

  • The session is created lazily and is NOT in _PERSISTENT_KEYS: idle cleanup reclaims its process like any other session, and an active consolidation turn is never expired mid-turn (its semaphore is held). Recycling after a full context recycles its own provider.
  • Telemetry classifies it with one _TELEMETRY_EXACT_KEYS entry ("_consolidate": "consolidation"), the same table that already names _bg and _hb; no new prefix rule.

Backwards compatibility

Compatible: nothing that works today stops working — _bg sessions keep the same behaviour and admission path, and sel._infer_source maps the new _consolidate key to the same background surface, so attribution and spend accounting are unchanged.

Removes nothing: the change is additive — one new session key plus its stateless-expiry entry; no key, kind, or entry point that main accepts is renamed, rejected, or removed.

Pattern harvest

Rule candidate: long-running maintenance needs a distinct admission identity from latency-sensitive background work.

Tests

python -m pytest -q test/test_session.py test/test_background_turn_accounting.py test/metrics/test_startup_channel_attrs.py — 448 passed; the history/consolidation/session-background files — 732 passed. New coverage: the exact-key telemetry label, separate keys for extraction and the cheap judge, idle expiry and re-creation without resume, an active consolidation surviving the sweep, and recycling its own provider.

Manual verification

N/A — focused automated coverage is sufficient.

Related Issues

N/A.

Checklist

  • Existing tests pass and regression coverage is included.
  • Self-review completed; code follows project style guidelines.
  • Documentation updated where applicable.
  • No secrets, credentials, or internal references in the diff.

Contribution License Agreement

N/A — template placeholder; no CLA wording is supplied.

@Premshay
Premshay requested review from a team and hoang-phan98 September 6, 2026 13:56
@github-actions github-actions Bot added fork Pull request from a fork (external contributor) readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Sep 6, 2026
@github-actions

github-actions Bot commented Sep 6, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review (fork) — ✅ no blocking findings

Reviewed 6dafbb2621071c0190bb4951cb27607f3e08fb0e via the fork AI-review pipeline; updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 6dafbb2

@github-actions

github-actions Bot commented Sep 6, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5, fork) — 🟡 CONCERNS

Design-level review of 6dafbb2621071c0190bb4951cb27607f3e08fb0e via the fork AI-review pipeline — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: CONCERNS

Two unrelated test files were rewritten and the description doesn't mention either. One of them now checks permissions through a private helper instead of the scenario path.

Watch

  • Unexplained test rewrites. test/test_eval_harness.py replaces runner.run_scenario(scenario) with runner._run_turn(PermProvider(), …) in five permission tests. Those tests now bypass the scenario/session wiring, so a regression in how run_scenario routes permissions would no longer fail them. test_agent_panel_crew_log.py swaps its "damage the last log entry" setup for a patched now_ms, which changes what the eviction test proves. Neither change relates to consolidation, and the description doesn't explain them.
    Clears when: both rewrites move to their own PR with a stated reason, or are reverted here.
  • Governance change not listed in the description. governance_profiles.resolve_active_scope adds "_consolidate" to the keys that don't count as a proven identity. Any unattended surface reached from this session now gets deny-all. That matches how _bg is treated, but it changes what this session is allowed to do, and neither "What changed" nor "Backwards compatibility" mentions it.
    Clears when: the description names the governance hunk and the containment it adds.
  • Commit-hash lines in the docs. Stray ebb643681 (fix(history): …) lines were added to history.md and to the session table in session.md. The one in the table splits it and breaks how it renders.
    Clears when: both stray lines are removed.

[DESIGN-REVIEWED] 6dafbb2

Description read

[DESCRIPTION-READ] db098889dfaa64852a0540263dfc0dbf3935b70f94f186399111b8c43fcb1afc

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Sep 6, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review (fork) — ✅ no blocking findings

Reviewed 6dafbb2621071c0190bb4951cb27607f3e08fb0e via the fork AI-review pipeline; updated in place on each push.

Review details

No findings that block the merge.

FINDING — docs/system-specs/modules/session.md:106 — the added line + ebb643681 (fix(history): isolate consolidation sessions) sits between two pipe-table rows, so the session-key table ends there and the "Lesson extract", "Subagent", "TaskRunner step"… rows render as loose text; the same leftover line was also added at docs/system-specs/modules/history.md:29 → Fix: delete both ebb643681 (...) lines.

[OPUS-REVIEWED] 6dafbb2

@github-actions

github-actions Bot commented Sep 6, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5, fork) — 🟡 CONCERNS

Premise-level review of 6dafbb2621071c0190bb4951cb27607f3e08fb0e via the fork AI-review pipeline — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: CONCERNS

The defect itself has no evidence on record: nothing measures or links the claim that _bg jobs wait behind consolidation, yet the fix keeps a second provider process alive.

Not justified as shipped

  • Item 1 — inherited: "Lightweight maintenance queues behind consolidation" has no linked issue, no measured wait and no reproduction, only the removed comment's "queue contention behind other _bg consumers like chat_nav link-preview".
    Clears when: a DEBUG timing log or issue shows _bg acquisition stalled behind a consolidation turn.
  • Item 5 — one consumer, generalized: _STATELESS_KEYS = frozenset({CONSOLIDATE_KEY}) has one entry and is threaded through AllocationConstants and SessionLifecycleConstants.
    Subtraction: compare against the single key, the way background_key is compared, and delete the stateless_keys fields.
    Clears when: the frozenset and both dataclass fields are removed.
  • Item 7 — undeclared, rides along: test_eval_harness.py swaps run_scenario for the private _run_turn, and test_agent_panel_crew_log.py rewrites its damage setup. Neither has anything to do with the Goal.
    Subtraction: revert both test files from this PR.
  • Item 8 — undeclared: stray ebb643681 (fix(history): isolate consolidation sessions) lines in history.md and in the session.md table.
    Subtraction: delete both lines.

What this change ships

Inventory (8 items) — 3 justified

Intent: stop history extraction from making cron and maintenance jobs wait. FIX.

  1. History extraction runs on its own _consolidate session, so an extra resident provider exists while it is present — inherited
  2. Dedupe and merge judges stay on _bg — justified
  3. Telemetry labels it consolidation, and Spend groups it under bg — justified
  4. The audit log (SEL) attributes _consolidate as background — justified
  5. A generalized set of stateless keys is matched exactly — one consumer, generalized
  6. _consolidate is treated as an unattended sentinel by governance, so unattended surfaces deny it — undeclared
  7. Unrelated eval-harness and crew-log test rewrites — undeclared, rides along
  8. Commit-hash lines pasted into specs — undeclared

[FIRST-PRINCIPLES-REVIEWED] 6dafbb2

Description read

[DESCRIPTION-READ] bc9384dfc7d4eac96c3943b95eb41b40e53ce96f36937c6e03b8145b62f33b77

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Sep 6, 2026
@Premshay
Premshay force-pushed the upstream/history-consolidation-session branch 2 times, most recently from bcb40fa to 7ae8904 Compare September 6, 2026 16:06
@bolichen97
bolichen97 force-pushed the upstream/history-consolidation-session branch from 7ae8904 to f7e44aa Compare September 8, 2026 10:11
@bolichen97

Copy link
Copy Markdown
Contributor

Rebased onto main 6fbb06bc by a maintainer as part of the 2026-09-08 open-PR audit. Old head 7ae89048, new head f7e44aaf.

Clean rebase — no conflicts. The diff is unchanged (9 files, +102/-30); the only drift main introduced was the removal of the else: LLMProvider = Any fallback in session_background.py, ~500 lines from this PR's hunk, so main's version now stands.

Gates run locally on changed files only: isort pass, flake8 pass, pytest pass (test_background_turn_accounting.py, test_startup_channel_attrs.py, session tests, consolidation tests — 203 passed). black flags test/metrics/test_startup_channel_attrs.py, which is pre-existing (it is in .github/black-baseline.txt and main's own copy fails too).

Please review the rebase. A maintainer push makes the maintainer the last pusher, so a second approver is needed under the repo's last-push rule. Reply if anything looks wrong.

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention merge conflict Branch has merge conflicts with its base — author must resolve before merge and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 8, 2026
@Premshay
Premshay force-pushed the upstream/history-consolidation-session branch from f7e44aa to 3807d1f Compare September 13, 2026 09:08
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention merge conflict Branch has merge conflicts with its base — author must resolve before merge and removed readiness: action required A blocking check or review needs attention merge conflict Branch has merge conflicts with its base — author must resolve before merge readiness: checking Automated validation is still running labels Sep 13, 2026
@Premshay
Premshay force-pushed the upstream/history-consolidation-session branch from 3807d1f to 4c39ae9 Compare September 19, 2026 09:20
@github-actions github-actions Bot removed the readiness: action required A blocking check or review needs attention label Sep 19, 2026
@Premshay
Premshay force-pushed the upstream/history-consolidation-session branch from b553d68 to 4f1e8d1 Compare October 4, 2026 08:05
@github-actions github-actions Bot removed the merge conflict Branch has merge conflicts with its base — author must resolve before merge label Oct 4, 2026

Copy link
Copy Markdown
Contributor

Open-PR triage (2026-10-05), checked against origin/main a97659df.

Verdict: DISTINCT, keep. No breaking change and no P0 found. Nothing on main or in another open PR moves history extraction off _bg. git merge-tree against current main is clean, so you don't need to rebase for conflicts.

Two non-blocking items on the current head 4f1e8d10:

  1. _consolidate has no arm in sel._infer_source. In sel.py (main, ~L3556-3583), _bg maps to background and _hb maps to heartbeat, but _consolidate falls through to the legacy "slack" fallback. governance_profiles._infer_surface delegates to that function. So the extraction turn moves from the background surface, which is in _UNATTENDED_SURFACES and fails closed when unbound, to slack, and any SEL rows it emits are attributed to Slack. The impact is small: the turn runs with REJECT_ALL and kirocrew-lite has tools: []. Still, adding _consolidate -> "background" next to the _bg arm would keep the governance surface the same as before this PR.
  2. The dead _MAX_CONCURRENT_COLD_STARTS = 4 constant in session.py is now staler. Design Review and First Principles both flagged it. Since then, fix(acp): give interactive chat starts priority over subagent starts #16020 removed the constant from main and replaced _start_sem with a PrioritySemaphore. Re-adding it brings back a name main deliberately deleted, and its comment points at a function that doesn't exist.

Related, but not duplicates: #9900 (cross-process consolidation lease) and #14984 (merged; RSS recycling, which _consolidate now inherits through the generalized recycle_background).

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Oct 5, 2026
@Premshay
Premshay force-pushed the upstream/history-consolidation-session branch from 6c5a1b0 to f07cdaf Compare October 6, 2026 21:27
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Oct 7, 2026
@Premshay
Premshay force-pushed the upstream/history-consolidation-session branch from f07cdaf to ebb6436 Compare October 7, 2026 19:32
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention merge conflict Branch has merge conflicts with its base — author must resolve before merge and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Oct 7, 2026
@Premshay
Premshay force-pushed the upstream/history-consolidation-session branch from ebb6436 to 6de55aa Compare October 8, 2026 07:52
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Oct 8, 2026
@github-actions

github-actions Bot commented Oct 8, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5, fork) — ⏭️ skipped

Revision 6dafbb2621071c0190bb4951cb27607f3e08fb0e touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Advisory — does not block merge.

@iamwhatever

Copy link
Copy Markdown
Contributor

Thank you @Premshay for this fix! It is now carried by #19733, rebased onto current main with your commit and authorship kept in the git history (plus a Co-authored-by: trailer). The port needed only small mechanical changes: recycle_background now also keeps main's newer context_overflowed flag, the history spec keeps main's new "Row counts off the loop" section next to yours, and a test hang guard got a 30s ceiling to satisfy the wait-discipline ratchet. All checks, including PR Readiness, are green there. The bug is tracked as #19731.

Closing this one in favour of #19733. Your branch is left untouched.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

fork Pull request from a fork (external contributor) merge conflict Branch has merge conflicts with its base — author must resolve before merge

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants