Skip to content

fix(history): isolate consolidation sessions - #19733

Merged
iamwhatever merged 2 commits into
mainfrom
fix/9036-consolidation-session
Oct 14, 2026
Merged

iamwhatever merged 2 commits into
mainfrom
fix/9036-consolidation-session

Conversation

@iamwhatever

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: Long history consolidation no longer makes lightweight background jobs (cron, maintenance) wait behind it.

History consolidation uses the shared _bg session, coupling long memory extraction to latency-sensitive background micro-jobs.

Why it matters

Lightweight maintenance queues behind consolidation even though it neither needs its context nor its admission path.

Not a goal

  • Changing what consolidation extracts or how its prompt is built.
  • Changing the _bg session's behaviour for the jobs that keep using it.
  • Keeping a consolidation process alive between passes; it expires when idle.

What changed (motivation → approach → change)

History extraction runs on its own stateless _consolidate session instead of _bg, so its long turns no longer hold the semaphore the background micro-jobs share; _bg behaviour is unchanged.

  • The session is created lazily and is NOT in _PERSISTENT_KEYS: idle cleanup reclaims its process like any other session, and an active consolidation turn is never expired mid-turn (its semaphore is held). Recycling after a full context recycles its own provider.
  • Telemetry classifies it with one _TELEMETRY_EXACT_KEYS entry ("_consolidate": "consolidation"), the same table that already names _bg and _hb; no new prefix rule.

Backwards compatibility

Compatible: nothing that works today stops working — _bg sessions keep the same behaviour and admission path, and sel._infer_source maps the new _consolidate key to the same background surface, so attribution and spend accounting are unchanged.

Removes nothing: the change is additive — one new session key plus its stateless-expiry entry; no key, kind, or entry point that main accepts is renamed, rejected, or removed.

Tests

python -m pytest -q test/test_session.py test/test_background_turn_accounting.py test/metrics/test_startup_channel_attrs.py — 448 passed; the history/consolidation/session-background files — 732 passed. New coverage: the exact-key telemetry label, separate keys for extraction and the cheap judge, idle expiry and re-creation without resume, an active consolidation surviving the sweep, and recycling its own provider.

Manual verification

N/A — focused automated coverage is sufficient.

Screenshots / video

N/A — backend only.

Related Issues

Fixes #19731

Replaces #9036 — original work by @Premshay, cherry-picked with authorship kept (plus a Co-authored-by: trailer). Ported onto current main: the only conflict was recycle_background on SessionManager and BackgroundSessionRuntime, where main added a keyword-only context_overflowed flag; the port keeps both the new session_key/agent parameters and context_overflowed.

Pattern harvest

Rule candidate: long-running maintenance needs a distinct admission identity from latency-sensitive background work.

Checklist

  • Existing tests pass and regression coverage is included.
  • Self-review completed; code follows project style guidelines.
  • Documentation updated where applicable.
  • No secrets, credentials, or internal references in the diff.

Contribution License Agreement

Contributed under the project's Apache-2.0 license (see CONTRIBUTING.md).

@iamwhatever
iamwhatever requested review from a team and KiroCrewCaptain as code owners October 13, 2026 06:33
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Oct 13, 2026
@github-actions

github-actions Bot commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision 2e130dee60a901c176bc3e50b3ceddb593c0d28a touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Advisory — does not block merge.

@github-actions

github-actions Bot commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — 🟡 CONCERNS

Premise-level review of 2e130dee60a901c176bc3e50b3ceddb593c0d28a — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: CONCERNS

The new consolidation telemetry label adds a permanent label value and a usage.py entry, even though the PR books the same key as background everywhere else.

Not justified as shipped

  • Item 3 — rides along: the PR adds "_consolidate": "consolidation" as a new bounded label, so usage.py needs an extra "consolidation" entry to fold it back into bg. Meanwhile sel._infer_source maps the same key to the existing background source.
    Subtraction: map _consolidate to "background" in _TELEMETRY_EXACT_KEYS, and delete the "consolidation" line from _BACKGROUND_CHANNELS.
    Clears when: the PR names a consumer that needs consolidation turns told apart from _bg turns in telemetry, or the label is collapsed into background.
  • Item 5 — one consumer, generalized: stateless_keys: frozenset[str] is added to 2 constants dataclasses, and _STATELESS_KEYS has exactly 1 entry.
    Subtraction: replace it with a single consolidate_key: str field, checked next to the existing background_key/heartbeat_key test.
    Clears when: a second exact stateless key is counted, or the field is made singular.
  • Item 7 — undeclared, rides along: the rewrites of test_eval_harness.py (run_scenario → _run_turn) and test_agent_panel_crew_log.py (log rewrite → now_ms patch) are not mentioned in the description.

What this change ships

Inventory (7 items) — 3 justified

Intent: stop long history consolidation from making cron and maintenance background jobs wait. FIX (#19731).

  1. History extraction runs on its own lazily created _consolidate session, so _bg jobs no longer queue behind it — justified
  2. The extraction session expires when idle and is recreated without resume; context recycling targets its own provider — justified
  3. Telemetry gets a new consolidation channel label; Spend folds it into bg — rides along, a new label where background exists
  4. Governance treats _consolidate as an unattended sentinel (identity not proven) — justified, undeclared
  5. stateless_keys set added to the allocation and lifecycle constants — one consumer, generalized
  6. SEL attributes _consolidate to the background source — justified
  7. Unrelated eval-harness and crew-log test rewrites — undeclared, rides along

[FIRST-PRINCIPLES-REVIEWED] 2e130de

Description read

[DESCRIPTION-READ] c52214c6877c6b5032c7f80fa72ba9098f344e1f7a8342bd8b409ca213a0e5eb

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of 2e130dee60a901c176bc3e50b3ceddb593c0d28a — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Check that context overflow still recycles _consolidate on runtime-capable backends: recycle_background drops overflowed whenever _bg_backend_supports_runtime() is true.

[DESIGN-REVIEWED] 2e130de

Description read

[DESCRIPTION-READ] 2485e9a0f98ff6e484f1a4f604d960cb1a26c48b5d81005530e05cb20b011799

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed 2e130dee60a901c176bc3e50b3ceddb593c0d28a — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 2e130de

Verdict parsed from the review's SHA-scoped output markers for commit 2e130dee60a901c176bc3e50b3ceddb593c0d28a.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 2e130dee60a901c176bc3e50b3ceddb593c0d28a: <one-sentence reason>

@github-actions

github-actions Bot commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of 2e130dee60a901c176bc3e50b3ceddb593c0d28a and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 2e130de

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 2e130dee60a901c176bc3e50b3ceddb593c0d28a: <one-sentence reason>

@github-actions github-actions Bot added the merge conflict Branch has merge conflicts with its base — author must resolve before merge label Oct 13, 2026
@iamwhatever
iamwhatever force-pushed the fix/9036-consolidation-session branch from f8d85ac to 479772c Compare October 13, 2026 07:34
@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed merge conflict Branch has merge conflicts with its base — author must resolve before merge readiness: checking Automated validation is still running labels Oct 13, 2026

@buluoray buluoray left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: REQUEST CHANGES (head 479772c86d)

What it fixes

History extraction ran on the shared _bg session, so a long extraction turn held _bg's semaphore and every short background job (titles, link previews, dedupe judges) queued behind it. Extraction now gets its own lazily created _consolidate session; cheap judges stay on _bg. Root-cause fix, not a timeout bump.

What I verified

  • history_consolidation.py:4370-4377 passes session_key=CONSOLIDATE_KEY; llm_helpers.py:2480 selects it and :2586-2591 recycles that key with its own agent, while _bg keeps the old recycle_background() call.
  • session_allocation.py:2341 and session_lifecycle.py:3291,3317 treat _consolidate as stateless by exact key, so it never resumes from or writes the session map. Nothing persisted changes shape: history offsets, daily files and the session map load exactly as before on upgrade, and V2 memory-consolidation: cleanup is untouched.
  • Containment does not widen: governance_profiles.py:1428 keeps _consolidate an unproven, unattended identity like _bg, and sel.py:3560 still classifies it as background, matching what these turns were recorded as before.
  • Idle expiry: _consolidate is deliberately outside _PERSISTENT_KEYS (session.py:806). The held turn semaphore protects an active turn, and test_active_consolidation_survives_idle_sweep pins it. Error paths fail closed: a failed acquire still raises _ConsolidationNotDispatched and sends nothing.
  • Targeted suites pass: 888 passed. Mutations: dropping the stateless_keys check reddens test_consolidation_expires_and_recreates_without_resume. Forcing background_turn back onto _bg reddens test_active_consolidation_survives_idle_sweep, test_dedicated_key_owns_its_session_and_recycle and test_extraction_and_cheap_judge_use_separate_keys.

Blocking

  1. docs/system-specs/modules/session.md:4961 contains a stray commit-log line, ebb643681 (fix(history): isolate consolidation sessions), in the middle of the session-key table. It ends the table, so the Lesson extract, Subagent and TaskRunner rows below it no longer render as a table. Fix: delete that line.

Non-blocking follow-ups

  • No test covers _consolidate in the identity_proven exclusion: removing it from governance_profiles.py:1428 keeps 1443 governance tests green. Add a resolve_active_scope("_consolidate", <unattended surface>) deny-all assertion.
  • Other _bg exact-key sites were not extended: taskq/lanes.py:31 (_SYSTEM_KEYS), validation.py:398 (infer_use_case now returns unknown, which is the more restrictive answer), context_assembly/sections.py:74, and dashboard/session_memory.py:316 (shows the raw key as the title). None of these has a reachable effect today as far as I traced, but each is a place where the new key quietly falls into a default.
  • The test_agent_panel_crew_log.py and test_eval_harness.py rewrites have nothing to do with this change. The eval-harness tests now call the private _run_turn instead of run_scenario, which narrows what they exercise. Please justify this in the description or split it into a separate PR.

Not verified

  • I did not measure real-provider memory with the extra resident lite process, and did not check behaviour on runtime-capable backends (_bg_backend_supports_runtime).

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: passed Eligible automated validation passed for the current revision labels Oct 13, 2026
Premshay and others added 2 commits October 13, 2026 18:22
Fixes #19731

Co-authored-by: Premshay <28099628+Premshay@users.noreply.github.com>
The wait-discipline ratchet forbids a new wait_for under 30s; the guard
only names a hang, so the longer ceiling keeps its meaning.

Co-authored-by: Premshay <28099628+Premshay@users.noreply.github.com>
@iamwhatever
iamwhatever force-pushed the fix/9036-consolidation-session branch from 479772c to 2e130de Compare October 13, 2026 18:27
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Oct 13, 2026
@iamwhatever

Copy link
Copy Markdown
Contributor Author

@buluoray Thanks. The blocking item is fixed on head 2e130dee60a901c176bc3e50b3ceddb593c0d28a:

  • docs/system-specs/modules/session.md:4961: deleted the stray ebb643681 (fix(history): isolate consolidation sessions) line. The Resource Budget table (session.md:4953-4966) now runs unbroken, so the Lesson extract, Subagent and TaskRunner rows render as table rows again.

The fix is amended into the first commit. The second commit is unchanged, and no code changed. ./scripts/docs-lint.sh passes, and every check is green on the new head, PR Readiness included. The non-blocking follow-ups are untouched in this push.

@iamwhatever
iamwhatever requested a review from buluoray October 13, 2026 19:20
@bolichen97

Copy link
Copy Markdown
Contributor

@buluoray your change request is the only thing holding this PR. @iamwhatever says it is addressed on head 2e130dee60, and readiness passed there. Could you re-review that head?

Posted by an AI assistant at bolichen97's request.

@buluoray buluoray left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-review at 2e130de. This supersedes my earlier request changes.

My blocker is fixed: the stray ebb643681 (fix(history): isolate consolidation sessions) line is gone from docs/system-specs/modules/session.md, so the table below it renders again. Range-diff against the head I reviewed shows that line as the only change to the fix commit; the test commit is unchanged.

CI: 87 passed, 14 skipped, 0 failed.

@iamwhatever
iamwhatever merged commit cfd4de0 into main Oct 14, 2026
102 checks passed
@iamwhatever
iamwhatever deleted the fix/9036-consolidation-session branch October 14, 2026 01:34
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Oct 14, 2026
chenmingwei23 added a commit that referenced this pull request Oct 19, 2026
…0463)

Every global consolidation job shared the "_consolidate" conversation,
which was replaced only at 70% context or 40 prompts. One job is ~80 KB,
so a conversation stacked 40+ jobs: small-window models refused the
prompt and 1M-window models drew HTTP 429. The backend record of each
conversation (KAS sess_<id>) was never deleted.

- _call_llm mints a fresh "_consolidate:<uuid4 hex>" key per job;
  background_turn removes and destroys that session in a finally, on
  success or failure, instead of recycling it.
- A one-job provider (global or member path) is asked to delete its
  backend record at shutdown: the harness teardown for that session id
  runs before the runtime kill (KAS _kiro/session/delete; kiro-cli
  terminate plus the guarded cli transcript unlink).
- The job key is stateless; every job works in the shared "_consolidate"
  directory, so a job leaves no directory of its own.
- Global jobs still run one at a time and never take _bg (#19733).
- Usage rows stay under "_consolidate"; telemetry, audit-source and
  governance classify a job key like the shared key.
- The consolidation failure log names the served model and prompt size.
- The fake ACP backend gains an opt-in native-store emulation, used by a
  real-child-process test of two back-to-back jobs.
- RFC: docs/request-for-change/rfc-consolidation-job-session.md
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

History consolidation holds the shared _bg session, so light background jobs wait behind it

4 participants