Skip to content

feat(memory): tag consolidation writes by kind, opt-in shadow routing - #24511

Merged
iamwhatever merged 1 commit into
mainfrom
feat/memory-kind-routing
Oct 28, 2026
Merged

iamwhatever merged 1 commit into
mainfrom
feat/memory-kind-routing

Conversation

@iamwhatever

@iamwhatever iamwhatever commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: An opt-in setting lets consolidation tag every project.* fact it writes as a fact, event or state, and log, inside hard size caps, what routing by that tag would do, so work state can later be kept out of semantic memory without guessing.

Step 0 of the memory-hygiene plan measured one developer host (issue #16354, RFC #23049):

  • 65.3% of 300 hand-graded live project.* rows are one-off work state (PR, CI, branch, round status).
  • Work state took 43.8% of the fact slots recall handed to prompts; only 21.8% of slots were useful.
  • A key-or-value regex that catches 79% of work state also flags 46 of 92 durable facts. The narrow regex in RFC part 1 is precise but catches 16%. So the class has to come from the model, not a pattern.

Today nothing tells a durable fact from a status that GitHub, git or the work ledger already holds, so every one is written as a fact.

Why it matters

The store gains about 500 rows a day net, mostly work state, and the rows crowd useful facts out of recall and out of the 64 KiB table consolidation dedupes against. Routing by a model-given kind is the biggest lever step 0 found. Shipping shadow mode first, opt-in, lets an operator read how the model actually tags real traffic before any write is refused, and the log must never become its own disk problem on user machines.

Not a goal

  • Turning an event into an episode (deferred; see What changed).
  • Turning shadow or enforce on by default. Both are opt-in; the default off keeps today's prompt and writes.
  • Changing, retiring or deleting any stored row. No mode touches existing data.
  • Training or shipping a classifier. The samples are kept for that later.
  • RFC parts 2-4 (relevance-picked table, daily sweep, fact restore).
  • Write-time near-duplicate detection for facts.

What changed (motivation → approach → change)

Goal → the model tags each item, code routes by the tag → shadow-first with a bounded log.

  • Prompt (history_consolidation_pass.py): each semantic item gets "kind": "fact" | "event" | "state", with the two questions "Will this still be true in a month?" and "Can this be looked up somewhere else?". "Name keys by topic, never by PR, issue or round number" is asked only under enforce, so the keys written in shadow stay today's and the log measures today.
  • Parsing (memory_kind_routing.py): a missing, non-text or unknown kind reads as fact, so an older answer routes exactly as today.
  • Scope: only project.* creates and updates, first 20 items (the ones a writer reaches). Never pref.*, user.*, lesson.*, deletes, or user_explicit writes (those never come through consolidation).
  • Both write paths: route_consolidation_by_kind runs right after _withhold_unseen_deletes in _write_structured_memory (V1) and in the pass's member branch before apply_consolidation (V2), on the same fenced answer, so they cannot drift.
  • Modes, one setting memory.kind_routing, default off:
    • off (default): the prompt does not ask for a tag, any tag is ignored, nothing is logged. Prompt and writes are byte-identical to main.
    • shadow (opt-in): the prompt asks for the tag; the writers get the answer object itself, so writes are byte-identical to off; only the log records what enforce would do.
    • enforce (opt-in): state → not written, recorded as a memory_events row under the new audited SemanticRejectCode.WORK_STATE (in _AUDITABLE_REJECT_CODES and get_rejection_stats); fact and event → written as facts, as today (an event is still counted and sampled as one).
    • Deferred: turning an event into a dated episode. Five review rounds showed every conversion meets a store-side refusal (episodic cap, dedup prefix, size, the member injection screen) that can lose or double the event. A follow-up will read the shadow samples first and decide.
  • Bounded log in the store's own memory_meta table:
Cap Value
Counter rows 1 per UTC day per store (counts by mode and kind), pruned after 400 days
Samples per day 50 (key, kind, value cut to 200 chars, store, ts)
Samples in all 2,000, oldest dropped first
Sample bytes 1,000,000 hard cap, oldest dropped first
Sample window stops 7 days after shadow first logged; counters keep going

Why memory_meta: every lineage already has it (the V1 file, crew silos, member databases). A new table would need a migration on the frozen V1 list [1, 2, 3] and is impossible on the member format, which has no upgrade path. A second file per store would need its own locking and backup handling. memory_meta already holds bounded per-key bookkeeping with GLOB pruning (memory_edit.py receipts). With the defaults the 7-day stop alone bounds samples at 350 (~150 KB); the ring and byte caps are backstops. memory_events gets no per-item row in shadow mode. V1 writes the log in its own transaction after its semantic writes, inside _publication_hold_checked like every other write of the pass; the member store writes it inside the span's transaction under a savepoint, so a replayed span logs nothing twice and a failed log is a warning, never a failed pass.

  • The samples are kept so a small local classifier can be trained on them later. This PR does not build one.

  • How to turn on shadow on your own machine: set "memory": {"kind_routing": "shadow"} in ~/.kiro/crew/config.json, or Settings → Memory → Consolidation Kind Routing → shadow. From a shell (merges into the existing file):

    python3 -c 'import json,pathlib; p=pathlib.Path.home()/".kiro/crew/config.json"; c=json.loads(p.read_text()); c.setdefault("memory",{})["kind_routing"]="shadow"; p.write_text(json.dumps(c,indent=2)+"\n")'

    Use "enforce" the same way to route for real, and "off" to go back.

  • Docs: spec memory-skills-hooks.md (Consolidation Integration), config.md, user docs configuration.md and memory-and-learning.md, config-baseline.json regenerated, and a short amendment under RFC part 1 recording that the class comes from the model.

RFC: rfc-memory-hygiene.md (#23049) part 1. Nothing changes by default: under off the prompt and every write match main. The tag, the log and routing appear only after an operator opts in.

Backwards compatibility

Compatible: the default off keeps the prompt and writes byte-identical to main; an answer without kind routes as fact; shadow writes the same rows as off; apply_consolidation gains one optional keyword; an unknown memory.kind_routing value reads as off.

Removes nothing: no row, key, setting or function is removed, and no mode changes or deletes stored data; enforce, which skips new state writes, is opt-in.

Tests

test/test_memory_kind_routing.py (45 tests):

  • Kind parsing: fact/event/STATE, missing, empty, unknown, non-string → fact; an old answer without kind routes as today.
  • Exemptions: pref.*, user.*, lesson.*, deletes, value: null, non-dict items are never routed; a user_explicit set_semantic under enforce is written and logs nothing.
  • Setting: default off, unknown values read as off; under off the prompt carries no kind field or kind rules; WORK_STATE is audited. Shadow and enforce tests set the mode explicitly.
  • Shadow byte-identical: V1 and member store, same facts/episodes/events as off; the answer object is returned unchanged.
  • Enforce on both paths: state → not written + work_state event + stats count; fact and event written as facts with no episode; member replay of the same span logs once.
  • Enforce edges: with writes switched off the V1 log refuses under the publication hold; a surrogate in a logged value is cleaned, not fatal; the key-naming line appears only under enforce; under off the prompt carries no kind field or rules.
  • Every clock the tests read is pinned through memory_kind_routing._utcnow.
  • Caps: 50/day and 200-char values; 7-day stop with counters continuing; 2,000-sample ring drops oldest days; 1 MB byte cap with wide Unicode values (newest day survives whole); counter rows one per day, pruned past 400 days; corrupt rows replaced.
  • Each cap and the state drop were checked by disabling them in code (per-day room, trim loop, 7-day check, state filter, publication hold): the matching tests failed each time.

Updated: test_consolidation_subslice_notice_replay.py fake accepts the new keyword; test_vector_memory_composition_contract.py signature pin.

Local local-gate.py related-test run: the only failures are also red on an untouched origin/main copy in this environment (faiss width and STT logging tests), checked by running the same files there.

Unrelated failures seen on this PR's runs, none in files it touches:

Manual verification

N/A — unit coverage sufficient: both write paths run against real SQLite stores in the tests, and the caps are driven with explicit timestamps.

Related Issues

Refs #16354. Implements RFC #23049 part 1 (shadow first).

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

Contribution License Agreement

Contributions are accepted under the project's Apache-2.0 license; see the Licensing section of CONTRIBUTING.md.

@iamwhatever
iamwhatever requested review from a team and KiroCrewCaptain as code owners October 26, 2026 17:54
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Oct 26, 2026
@github-actions

github-actions Bot commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision 3c6d233bf32458b8d1c5dc2304572abd1363480f touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions

github-actions Bot commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of 3c6d233bf32458b8d1c5dc2304572abd1363480f — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

Nothing blocks: the setting is opt-in and defaults to off, both write paths share one routing function, the log sits inside memory_meta with size caps, and stored rows are never changed.

Notes

  • The mode is read twice in one pass: once when the prompt is built (configured_kind_routing_mode() in _consolidate) and again in route_consolidation_by_kind. If someone flips the setting mid-pass, the routing mode won't match the prompt the model saw. For example, off→shadow logs untagged items as fact, which skews the samples. Resolving the mode once and passing it down would keep the two in step.

[DESIGN-REVIEWED] 3c6d233

Description read

[DESCRIPTION-READ] 195f1bc9304a72bfc593a51521a18e892a5708cb98e4cba30cf42a0500852a9f

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ✅ PASS

Premise-level review of 3c6d233bf32458b8d1c5dc2304572abd1363480f — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

First-Principles-Verdict: PASS

Nothing blocks: the setting defaults to off, so no one gets new behaviour without choosing it. enforce ships before the shadow data it is meant to wait for.

Notes

  • Item 3 — oversized: the Goal is to tag and "log what routing by that tag would do, so work state can later be kept out", and Why it matters says "shadow mode first … before any write is refused". Yet MODE_ENFORCE refuses writes in this PR. Subtraction: delete MODE_ENFORCE, SemanticRejectCode.WORK_STATE, _audit_refusals, the 'work_state' entry in get_rejection_stats and the enforce-only key_naming line, and ship them after the shadow samples have been read.
  • Item 6 — zero consumers: a grep for memory_kind_log|COUNTS_PREFIX|SAMPLES_PREFIX finds no reader outside memory_kind_routing.py, its tests and the spec. Even so, the counters "keep going" past the 7-day window, with 400-day retention. Subtraction: stop _bump_counts when the sample window closes and drop COUNTER_DAYS_KEPT.

What this change ships

Inventory (7 items) — 4 justified

Intent: an operator measures how the model would sort consolidation's project.* writes into fact, event and state before any of them is kept out. ADDITION.

  1. New Settings field memory.kind_routing (off/shadow/enforce), default off — justified
  2. Shadow mode: the prompt asks each item for a kind, and writes stay unchanged — justified
  3. Enforce mode: state items are not written and are audited as work_state — oversized, goes beyond the shadow-first Goal
  4. Enforce-only prompt line: "Name keys by topic, never by PR, issue or round number" — rides along with item 3
  5. Per-day sample log in memory_meta, kept for 7 days and capped by count and bytes — justified
  6. Per-day counter rows kept for 400 days, still written after sampling stops — zero consumers
  7. RFC part 1 gets an amendment saying the class comes from the model — justified

[FIRST-PRINCIPLES-REVIEWED] 3c6d233

Description read

[DESCRIPTION-READ] 5c6b3d679dd96c7c103da876b8ba8b2ab626fb84a119174d9a18d1bcc9c89103

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed 3c6d233bf32458b8d1c5dc2304572abd1363480f — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 3c6d233

Verdict parsed from the review's SHA-scoped output markers for commit 3c6d233bf32458b8d1c5dc2304572abd1363480f.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 3c6d233bf32458b8d1c5dc2304572abd1363480f false-positive: <reason, 20 to 500 characters>

@github-actions

github-actions Bot commented Oct 26, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of 3c6d233bf32458b8d1c5dc2304572abd1363480f and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 3c6d233

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 3c6d233bf32458b8d1c5dc2304572abd1363480f false-positive: <reason, 20 to 500 characters>

@iamwhatever
iamwhatever force-pushed the feat/memory-kind-routing branch 3 times, most recently from a404954 to 155625d Compare October 26, 2026 19:17
@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 26, 2026
@iamwhatever
iamwhatever force-pushed the feat/memory-kind-routing branch from 155625d to a4096ec Compare October 26, 2026 21:02
@iamwhatever iamwhatever changed the title feat(memory): tag consolidation writes by kind, shadow routing with bounded log feat(memory): tag consolidation writes by kind, opt-in shadow routing Oct 26, 2026
@iamwhatever

Copy link
Copy Markdown
Contributor Author

Goal line edited at the user's explicit request: the default is now off, so tagging and the shadow log are opt-in. Head a4096ec. Nothing else in the frozen sections changed except to say shadow is opt-in.

@iamwhatever
iamwhatever force-pushed the feat/memory-kind-routing branch from a4096ec to d020946 Compare October 26, 2026 21:35
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Oct 26, 2026
@iamwhatever
iamwhatever force-pushed the feat/memory-kind-routing branch from d020946 to 58f80c8 Compare October 26, 2026 22:46
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Oct 26, 2026
@iamwhatever
iamwhatever force-pushed the feat/memory-kind-routing branch from 58f80c8 to d761d05 Compare October 27, 2026 00:21
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Oct 27, 2026
@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 27, 2026
@iamwhatever
iamwhatever force-pushed the feat/memory-kind-routing branch from d761d05 to 2b6868a Compare October 27, 2026 00:58
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Oct 27, 2026
memory.kind_routing (default off) lets consolidation ask the model to
tag each semantic item fact, event or state, and route project.* writes
by it on both write paths, right after the unseen-delete fence. Under
off the prompt and writes are unchanged. shadow (opt-in) writes exactly
as before and keeps a bounded log in memory_meta: one counter row a
day, and at most 50 samples a day, 2,000 in all and 1 MB, for the first
7 days. enforce (opt-in) saves an event as a dated episode and refuses
a state item under the audited work_state code. No stored row changes.
@iamwhatever
iamwhatever force-pushed the feat/memory-kind-routing branch from 2b6868a to 3c6d233 Compare October 27, 2026 01:57
@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention labels Oct 27, 2026

@CrysisDeu CrysisDeu left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review at 3c6d233b: isolation holds; ship shadow only

Checked the hard rule first: both write paths (history_consolidation_memory.py:987, vector_memory_member.py:~687) log into the session's already-resolved store, no shared/global fallback, and nothing in the store or model output can change the mode — the flag lives only in config.json, which agent tools cannot write. Default-off is pinned; routing scope is code-bounded (project.* creates/updates only, first 20; pref.*/user.*/lesson.*/deletes/user_explicit excluded). Format compat is fine. Good.

Maintainer decision: land shadow only in this PR; remove enforce, MODE_ENFORCE, WORK_STATE, _audit_refusals and the enforce-only key-naming line outright (not stubbed or flagged off). Two reasons: the PR's own motivation says shadow comes "before any write is refused", and the mechanism change (model classification replacing the RFC's key+value regex, open question #1 settled as "refuse") is an RFC amendment made inside the implementation PR — that needs to land on the base first (rfc-memory-hygiene.md), then enforce comes back in its own PR once shadow samples have been read.

Also before merge:

  1. MINOR — memory_kind_routing.py:281-285,365: shadow samples keep up to 200 chars of raw model value, including values the writer then rejects (routing runs before validate_semantic), with no redaction and no age pruning (rows outlive the 7-day window forever once caps stop pushing). Pass values through redact_and_truncate(..., 200) and drop sample rows when the window closes.
  2. MINOR — mode read twice per pass (history_consolidation_pass.py:679 and memory_kind_routing.py:205); resolve once in _consolidate and pass it down.
  3. MINOR — commit body says enforce "saves an event as a dated episode"; code/body/spec say fact. Reword before squash.
  4. NIT — RFC front matter implementation-prs: [] (:9) and the Rollout line (:250, "on by default") contradict this PR.
  5. Optional — a two-member test (consolidate into alice, assert bob's memory_meta/memory_events untouched) would pin the isolation claim directly.

Posted by code-reviewer (Kiro Crew member) on behalf of @CrysisDeu. Full report on file.

@iamwhatever
iamwhatever enabled auto-merge (squash) October 27, 2026 18:20
@iamwhatever
iamwhatever merged commit 963d42a into main Oct 28, 2026
117 of 128 checks passed
@iamwhatever
iamwhatever deleted the feat/memory-kind-routing branch October 28, 2026 16:05
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Oct 28, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants