Repository navigation
perf(memory): bound startup lesson injection with authored tiers - #12431
Conversation
36247eb to
b6589f1
Compare
GPT 5.6 Review — ✅ no blocking findingsGPT 5.6 completed its review of This comment is updated in place on each push. Review detailsFINDING -- docs/system-specs/modules/memory-skills-hooks.md:2779 -- False positive or not applicable? A repository writer can comment: |
Design Review (Fable 5) — 🟡 CONCERNSDesign-level review of Design-Verdict: CONCERNS Spec-pinned retain-every-rule startup default is reversed; the diff cannot own that decision — only an explicit maintainer override can. WatchThe old pin ( SuggestionsTake the deferred [DESIGN-REVIEWED] 13dab93 |
First Principles Review (Fable 5) — ✅ human override accepted@bolichen97 overrode this lane for |
Opus 4.8 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
b6589f1 to
3162cce
Compare
3162cce to
89c7080
Compare
89c7080 to
82861b6
Compare
82861b6 to
f65cd86
Compare
|
Answering the subtraction proposal on its merits, separately from the RFC blocker. The premise is correct and I said the same thing myself: on a store whose rows all predate the field, the reduction comes from the budget alone. Wiring the allowance into the It is not sufficient, and the reason is what the tier is for. A budget without it truncates rules and findings as ONE pool. Then:
So the apparatus is not decoration on top of the budget; it is what makes the truncation safe rather than arbitrary. Shipping the budget alone would deliver the measured character reduction and, on the same stores, start silently choosing which of the user's explicit rules to drop. That is a worse default than the one being replaced, not a smaller one. The honest counterweight, and it is real: on an all-legacy store the tier helps nothing today, because every row is unstated. The reduction there is the budget's, and classification only starts paying as new lessons are written with a stated value. That limit is in the spec rather than argued away, and no stored row is rewritten, reclassified or deleted to reach it. Items 2, 3 and 4 are named as components of the same decision, which is accurate — they are the safety half of it, not separable features. Item 7 (refuse-at-capacity) is the one I would defend independently of the shape: reporting |
|
/ai-review override first-principles 13d96af: keep-every-rule was a measured cost regression, not product direction. Bounded, reported in-prompt, no stored row rewritten or deleted. |
Human judgment recorded@bolichen97 marked the first-principles AI finding as false positive, not applicable, or explicitly accepted for
This decision applies only to this commit. A new push requires a new judgment. |
|
Answering the Design review's two Watch items and its suggestion. Watch 1 (maintainer acceptance of the bounded default) — applied on this head: the first-principles override is on record for Watch 2 (consolidation states no tier) — filed as #12495, with the reason it was not done here: deciding a tier per extracted item is a judgement about what the user meant, and the field's whole basis is that it is authored rather than derived. Suggestion ( Doing it consistently means adding I agree the suggestion is the same KIND of move as authored-first, and that it points at a real residual. It belongs with #12495 rather than ahead of it. |
A bare "hi" carried every stored lesson. The startup path computed the lessons cap, passed it, and then ignored it: get_lessons_context's background branch reads only hard_cap, which is the remaining protected room. At a 1M window that room is about 500,000 characters, so startup injection grew with both the lesson count and the model window, and an empty query changed nothing because background ranking reorders without dropping. Split startup lessons into two blocks with two separate, window-independent budgets, on an authored `applies` field: "always" is a standing rule under the existing lessons budget, "on_topic" is a past finding under a separate smaller one. A row with no value reads as unstated and is served as a rule, because demoting a real standing rule is the costlier mistake. Authored rules are ordered ahead of unstated rows within the rule block; ranked together, a crowd of untagged rows displaces exactly the rules the user was most explicit about. Every overflow names its counts, names the budget, says the limit is a budget rather than a judgement, and points at learn_list or memory_recall. `applies` is authored and never derived. No stored property separates a standing rule from a past finding: the consolidator can extract a real safety correction, a human can type a note about an outage, category describes subject matter, and "always" appears in both classes. So the write path accepts only the two literals and raises on a misspelling, while the read path treats anything unrecognized as unstated so one bad row cannot abort prompt assembly. Also make the JSONL prune tier-aware. A tier-blind [-200:] deletes the user's oldest authored rules from disk as cheap findings accumulate, which is worse than a prompt omission: an omission is announced and readable through learn_list, a pruned row is gone. Measured on identical synthetic JSONL stores, clean main against this branch, characters of rendered lessons (no token figure is claimed): 5 findings 2,234 -> 2,383; 60 findings 13,729 -> 2,698; 190 findings 41,079 -> 2,712. Before grows with volume, after is flat. Zero embedding calls on both sides. Compatibility limit, stated rather than papered over: a store whose rows all predate `applies` gets its reduction from the budget alone, and the overflow notice is what keeps that trade visible. No existing row is rewritten, reclassified or deleted.
|
/ai-review override first-principles 13dab93: bounded startup lesson budgets replace the retain-every-rule default deliberately; the cost is measured (41,079 chars at 190 findings), every omission is announced with counts and a tool to read the rest, callers naming no budget keep the two-stage contract, and the change is additive and revertible. |
Human judgment recorded@bolichen97 marked the first-principles AI finding as false positive, not applicable, or explicitly accepted for
This decision applies only to this commit. A new push requires a new judgment. |
buluoray
left a comment
There was a problem hiding this comment.
Verified at head 13dab936. No code defect — the mechanism is careful and I could not break it. I am not approving, because the thing this PR decides is a default that Design Review already said only a maintainer can own, and I can put a number on what it costs. That number is at the end.
What I verified rather than took on trust
- Tier assignment is authored, never derived.
normalize_lesson_appliesaccepts onlyalways/on_topicand raises on anything else rather than clamping; the read side never infers a tier fromcategory,ts,sourceor wording. I specifically checked the consolidation cron, since that is where a derived tier would sneak in:_candidate_appliesonly reads the field, andapi_lessons_createtakesappliesfrom its caller. Ordering within a tier is derived (relevance, newest-first as a stable tiebreak), which is the right split. - The omission is structurally un-silenceable.
render_lesson_tier's contract is that the notice is never dropped to fit: when there is not room for one entry it still renders header + notice + footer, because "a silently absent block is indistinguishable from this user has no rules". That is the failure mode I went looking for, and it is the one thing the function is built to always be able to produce. - One huge lesson cannot starve the rest — the selection loop
continues past an oversized entry instead ofbreaking, so smaller and more relevant rows still render. - No silent zero. The experience tier gets
max(1, hard_cap - len(directive_block))specifically so a directive block that ate the ceiling cannot hand it an unbounded budget. Worst case is a labelled "omitted all N" block. - Drop order is load-bearing, mutation-confirmed. Reversing
authored + unclassifiedtounclassified + authoredin a scratch copy reddens exactly the two precedence pins and nothing else. The truncation tests are not vacuous either — the 80-finding case asserts0 < kept < 80plus the notice, and the fixtures run well over budget.
The number
_LESSONS_CAP = _budget(0.226) = 7,458 chars, window-independent. On main the rule block was returned complete up to caps.protected_context, whose floor is _CONTEXT_BUDGET_BASE * 3 = 99,000 and which scales with the window above that. The four startup renderers now pass directive_budget=caps.lessons unconditionally, so the "complete below the ceiling" path survives only for a caller passing 0 — which the real path never does.
So the default for the standing-rule block goes from at least 99,000 chars to 7,458, a floor-to-floor reduction of more than 13x, and more on a large window.
And the mechanism that makes that cut safe is empty on day one. applies is new, so every existing lesson in every existing store is unstated. They therefore all land in the same tier, the tiering protects nothing yet, and the budget simply cuts. The authored tier only starts paying once rules are tagged — and cli.py's learn add has only rule, --category and --negative, with no --applies (GPT flagged the doc claiming otherwise; I confirmed the parser). The learn_add MCP tool does expose it. So a user who writes corrections from the CLI cannot put them in the protected tier at all.
To make this concrete rather than hypothetical: the lessons block injected into my own session right now already reports omitting 777 lessons at the current 99,000-char ceiling. After this change that surviving set is bounded at 7,458 chars — a small fraction of what the same store surfaces today, on the first boot after merge, with nothing tagged.
Why that is your call and not mine
Everything mitigating it is real and I verified each one: the loss is loud rather than silent, survivors are relevance-ordered (the JSONL path gains order_by_request_relevance here, which it did not have), no stored row is rewritten or deleted, applies is absent when unset, and it is a one-line revert. The PR body states the tradeoff openly and does not hide the measurement.
But this reverses a spec-pinned "retain every eligible in-scope rule" default, Design Review said in as many words that only a maintainer can own that, and the First Principles lane was cleared by the author's own override rather than by the lane. A reviewer approving on mechanism correctness would be deciding the default by omission — which is the specific thing I want to avoid doing to you here.
Two things that would change my answer, either one on its own: a floor that keeps the block complete below the old ceiling when nothing is tagged yet (so the cut arrives with the tiering rather than before it), or an explicit decision from you that 7,458 is the intended startup allowance and the omission notice plus memory_recall is the intended recovery path.
Unrelated and non-blocking: the body says the test file has 34 cases; it defines 48. Understated, harmless.
Why it matters
A new session that says nothing but
hicarried every stored lesson.The startup path computes the lessons cap, passes it, and then ignores it:
get_lessons_context'sbackgroundbranch reads onlyhard_cap, which iswhatever protected room is left. At a 1M window that room is about 500,000
characters. So startup injection grew with two things a user never chose: the
number of lessons they had accumulated, and the size of the model window. Moving
from a 200K to a 1M model multiplied it roughly fivefold for a user who changed
nothing. An empty query changed nothing either, because the background branch
reorders by relevance without dropping anything.
The existing code says why it kept everything, and the reason is real:
That is true, and it is why this PR does not try to infer the difference. It
stores it.
What changed
Two blocks, two separate window-independent budgets, split on an authored
appliesfield.applies"always"[Learned corrections]lessonscap (22.6% of the 33,000 base)"on_topic"[Learned experience]lesson_experiencecap (5%)[Learned corrections]Separate budgets rather than one shared allowance, so a user with many findings
cannot crowd out their own rules and vice versa. They are not summed into a
larger total, and a wider window enlarges neither. The model-safe protected
ceiling still bounds the two blocks together, and the rule block is served
first, so a finding yields to a rule rather than the two sharing a shortfall.
Authored rules are ordered ahead of unstated rows inside the rule block.
This is not cosmetic. Ranked together on recency, six authored directives all
fell out of the budget behind forty newer untagged rows — caught by a probe
while building this, and now pinned by a mutation-verified test.
Every overflow is announced. Each block's notice names its own
shown/omitted counts, names the budget it hit, states that the limit is a budget
rather than a judgement that the rules stopped applying, and points at
learn_list(rules) ormemory_recall(findings). A budget too small for evenone entry still renders the labelled block with that notice, because an empty
block is indistinguishable from "this user has no rules".
The JSONL prune is now tier-aware (findings, then unstated, then authored;
oldest-first within each class). A tier-blind
[-200:]deletes the user'soldest authored rules from disk as cheap findings accumulate, which is worse
than a prompt omission: an omission is announced and readable through
learn_list, a pruned row is gone.Why
appliesis authored and never derivedNo property of a stored row separates "a rule the user wants enforced in every
session" from "something we worked out once":
sourcedoes not — the consolidator can extract a real safety correction, anda human can type a note about last week's outage.
categorydoes not —tool/preference/knowledgedescribe subject matter.neither.
A reader that guessed from any of those would re-decide the user's intent on
every turn, and would fail silently in the direction that costs most. So
normalize_lesson_appliesaccepts only the two literals and raises on amisspelling at the write path — there is no safe clamp, since one choice injects
a note into every session and the other demotes a standing rule, and both are
silent. The read path never raises: anything unrecognized reads as unstated, so
one hand-edited row cannot abort prompt assembly for every other lesson.
The field is additive and absent when unset, so an unstated row is byte-identical
to what the writer produced before the field existed. On the enrich path it is
write-once like
categoryandrepo_scope, so enrichment cannot silently demotea rule; re-tiering is a delete + re-add.
The compatibility limit, stated rather than papered over
A store whose lessons all predate
appliesgets its reduction from the budgetalone, not from classification: every row is unstated, so they all compete for
the rule budget and the overflow notice is what tells the user which ones did not
fit. Classification only starts helping as new lessons are written with a stated
value.
"Retain an unbounded number of unstated rules" and "a small fixed startup budget"
cannot both hold. This is the side that was chosen, and the notice is the
mechanism that keeps the trade visible instead of silent. No existing row is
rewritten, reclassified or deleted.
Measured
Identical synthetic JSONL stores, clean
main(df4bf0546a, byte-identical tothis branch's merge-base on every file under test) against this branch.
Characters of rendered lessons. No token figure is claimed: the chars/4
estimate used elsewhere in this repo is not a tokenizer reading, and no
provider-reported usage was captured.
Before grows with volume; after is flat. At 190 findings that is a ~15x
reduction. The small case is 149 characters larger after, from the second
block's header — reported rather than hidden. Zero embedding calls on both sides
(startup still spends no inference).
Not measured: no real-harness first-turn capture, so this PR makes no claim
about any specific reported context number, and does not claim to have
identified the cause of any particular session's usage. The benchmark also does
not exercise the window-scaled ceiling difference, because the JSONL path takes
the ceiling as its cap directly and both fixtures sat below it.
Tests
test/test_lesson_startup_budget.py, 34 cases: the authored vocabulary(absent/unrecognized read as unstated, misspelling raises), budget arithmetic
(0 means unbounded and never wins a
min, empty tier renders no block, atoo-small budget still reports), and both render paths (vector and JSONL) for
bounded growth, authored-before-unstated ordering, legacy rows served as rules,
announced overflow, and byte-identity for a caller naming no budget.
Mutation-verified, both ways:
authored + unstatedtounstated + authoredfailstest_an_authored_rule_outranks_newer_unstated_rows;updated[-_MAX_LESSONS_TOTAL:]failstest_saving_past_the_cap_through_the_store_keeps_authored_rules.That second case exists because the direct-helper tests passed under the
reverted prune — they called
_prune_to_totaland left the store's own choiceof pruner unpinned. The route-level case is what actually guards the fix.
Two pre-existing tests changed, keeping every structural assertion and updating
only the overflow wording they pin; one (
test_over_cap_preserves_complete_explicit_rules)had its "all N rules appear" claim replaced with the invariant that still holds
and matters — trimming is by whole entry, so no partial rule is ever emitted.
Its presence check was also anchored on the rendered entry, because the fixture
rules are prefix-ambiguous (
0-xxx…is a substring of10-xxx…) and a bareinreported rules as present that were never emitted.Gates: black, isort, flake8, subprocess-encoding, docs-lint, comment-history all
clean. mypy reports 215 errors on this branch and 215 on a clean-main control —
zero introduced. 712 related backend tests pass.
Known gaps, deliberately not in this PR
learn_add,POST /api/lessonsand the CLInow pass
applies, but the consolidator does not: deciding a tier per extracteditem is a judgement about the user's intent, not a mechanical passthrough, and it
needs its own change to the extraction prompt. Its rows stay unstated, which is
the safe rule-tier path.
would also bound preferences, a separate behaviour change to a protected
document with its own truncation semantics.
it hit, so the constraint is visible; tuning is a follow-up.
user_explicitrows are not ordered ahead ofextracted ones. Design suggested it and I think it is right — it is ordering
bias, not the inference this PR rejects. Held back only to keep this diff to one
reviewable change; say the word and it goes in.
Smoke test — scale sweep, relevance, and no-regression
Reproducible harness, run against clean
mainand this branch with the samefixture, so every cell is a before/after pair. Budgets are read from
context.pyrather than restated, so the harness cannot drift from shipped values(rules 7,458 chars, findings 1,650).
Characters and operation counts only — no token figure is derived, because the
chars/4 estimate in this repo is not a tokenizer reading.
Per scale the harness plants, in one store: 3 authored standing rules, 1 finding
naming a rare topic the query also names, and filler naming unrelated topics. The
relevant finding is planted early on purpose — an old-but-relevant row is
exactly what a newest-first budget drops. It then measures the startup render, the
explicit-reader render (what
memory_recall/learn_listreach), and how manyembedding calls startup made.
Vector store (the primary tier)
JSONL fallback tier
Verdict
Regressions: none, across all 16 scale × store combinations.
reduction, and flat from 100 lessons upward. Before, the cost tracked the pile.
and is recallable through the explicit reader where it is not injected.
paid for with inference.
block's header), reported rather than hidden.
Two things the sweep found that the earlier probes did not
1. A real regression this PR introduced, now fixed. At 100 and 199 lessons the
JSONL tier stopped injecting the relevant finding: that tier has no relevance
score, so newest-first dropped an old relevant row while newer irrelevant ones
fitted. The vector tier was unaffected (it ranks against the request already).
Fixed by ordering the FINDINGS tier by lexical overlap with the request
(
order_findings_by_relevance— word tokens, no stemming, no embedding). Rules aredeliberately not reordered: a rule applies whatever the message is about, so
scoring one against the message is the inference this PR rejects. Pinned by three
tests and mutation-verified.
2. A pre-existing data-loss bug on
mainthat this PR fixes. At 500 and 2,000lessons,
mainkeeps 0 of 3 standing rules — the tier-blind[-200:]prunedeletes them from disk, because they are the oldest rows. This branch keeps 3/3.
That is disk loss, not a prompt omission: on
mainthose corrections are gone andlearn_listcannot show them.Harness limitations, stated
The fixture needed two corrections worth recording, because both silently made an
earlier run measure nothing. Filler sharing a sentence frame cleared
write_lesson's 50% keyword-overlap dedup, so 25 requested lessons became 5 storedand the vector sweep was vacuous; every filler token now carries its index. And the
harness initially omitted
query_texton the JSONL path, which is not whatcontext.pypasses — it reported the relevance regression as unfixed after the fixhad landed. There is no real-harness first-turn capture here, so this PR still makes
no claim about any specific reported context number.
Review round 2 — what changed
Three findings were legitimate defects my own diff introduced, so they were fixed
rather than rebutted (my standing override does not cover a defect the PR itself
exposes):
learn.py— at the 200-row cap a newon_topicrow is the firsteviction candidate, so the prune could drop the submission while
savestillreturned
"inserted". A write that did not land now returns"refused".vector_memory.py— the dedup scan's three delete branches decide ontext alone, so a longer overlapping
on_topicwrite could tombstone an authoredalwaysrule. Added a tier guard on the delete direction only: anon_topicwrite may not retire an
alwaysrow, analwayswrite still may retire afinding (promoting guidance to a standing rule is the direction the user asked
for), and a covered
on_topicsubmission is still declined assubstring_covered.asdictemitted"applies": nullonevery row, rewriting legacy lines on the next save and making the two stores
disagree about what absence means.
_serializablenow drops the key when unset.Also fixed GPT finding 3: the findings-tier overflow notice now carries
{limit}and the same budget-not-judgement sentence the rule notice has, which is what the
spec promises for every overflow.
First Principles item 2 (zero consumers) is addressed:
appliesnow has threeproduction writers — the
learn_addMCP tool (enum in its schema, plus adescription telling the model to decide from what the user actually said rather
than from wording),
POST /api/lessons, and the CLI reaching that route. The tieris reachable in production, so the machinery is no longer inert.
First Principles item 1 (no RFC) is a maintainer decision and is not mine to
make. It objects that this PR reverses a shipped default — keep every eligible
rule until the window-scaled ceiling — with no RFC covering it. That is a fair
reading: the base spec pinned that sentence and this diff rewrites it. Drafted
override reason for a maintainer who agrees the bounded budget is the right
default:
New tests: the prune-refusal path, the three tier-guard directions, and JSONL
serialization. All three fixes are mutation-verified (reverting each one fails its
own test). mypy is back at control parity (215 on both this branch and clean main).