feat(memory): add bounded cursor pagination for Memory entries #1656

Activity

1. github-project-automation

   added this to PowerContexton Sep 18, 2026
2. github-project-automation

   moved this to Todo in PowerContexton Sep 18, 2026
3. wutongyuonce commented on Sep 22, 2026

   

   wutongyuonce

   on Sep 22, 2026

   Last edited by wutongyuonce

   Contributor

   More actions

   Coordinated pagination SPEC proposal for #1656 and #1657

   @knqiufan I would like to implement #1656 first and #1657 second as independently reviewable PRs. I reviewed the existing cursor, authentication, OpenAPI, and migration patterns; those established conventions do not need separate maintainer decisions.

   Shared public contract

   • Add paged query operations rather than silently changing the existing full-result operations.
   • Proposed limit: default 50, maximum 100. A documented response-byte budget may end a page earlier and return continuation.
   • Reuse the existing signed opaque cursor codec, TTL, and stable cursor errors. Bind it to endpoint, Scope, filters, limit, ordering version, and pinned Memory revision.
   • Reauthorize every page. Permission changes stop continuation even when the cursor remains cryptographically valid.
   • Pin the Memory revision on page one; later pages traverse that immutable boundary, so concurrent writes do not skip or duplicate results.
   • Missing next_cursor means completion. Tampered, expired, cross-Scope, cross-endpoint, or query-mismatched cursors fail without disclosing private data.

   #1656 entry query

   • Inputs retain Scope, inactive-entry, and tag filters, plus limit and optional cursor.
   • New paged order: entry_id ASC; legacy full-list behavior remains unchanged.
   • Filtering occurs before page selection.
   • Results preserve exact Memory revision, entry/version identity, state, and citation.
   • Response assembly loads only page-sized entry bodies plus bounded lookahead.

   #1657 history query

   • Preserve the existing exclusive since_revision semantics, including 0, omitted/null, and positive nonexistent revisions.
   • Order by (revision ASC, change_ordinal ASC).
   • Count changes, not revisions, against the page limit and allow continuation within one revision.
   • Preserve change metadata and exact references without expanding entry bodies.

   Required persistence boundary

   The current entry path decodes the complete manifest, and the current history path loads Artifact revisions before filtering. HTTP-level slicing would not satisfy either Issue.

   I propose:

   1. A normalized entry-directory query index with revision validity (valid_from_revision / valid_to_revision, or an equivalent representation), permitting keyset reads at the pinned revision without copying a complete manifest per revision.
   2. Normalized change rows keyed by (scope, memory_artifact_id, revision, change_ordinal), permitting continuation inside a large revision.
   3. Atomic index updates with the authoritative Memory Artifact commit. Artifact/Memory content remains authoritative; the new tables are rebuildable query indexes.
   4. An explicit, resumable offline plan/apply/verify backfill for existing databases, following the repository processing-migration pattern. Startup does not silently perform an unbounded rebuild or fall back to unbounded reads.
   5. No retention, split/routing, tombstone compaction, or broader feat(memory): define capacity contract for long-lived Memories #1718 capacity policy in these PRs.

   Verification and PR split

   #1656 will cover the additive entries contract, shared cursor context, directory index/backfill, generated code, observable behavior tests, and a scale measurement based on loaded rows/items and serialized bytes. #1657 will reuse that seam and add change-row/backfill support plus large-single-revision coverage.

   The existing repository already determines cursor implementation, authorization, migration workflow shape, generated-code handling, and the two-PR split. The only coordination decision is whether the additive pinned-revision contract and normalized query-index boundary above are acceptable. Please confirm that boundary; unless maintainers identify a conflict, I will apply the existing conventions directly rather than ask for further implementation choices.
4. knqiufan commented on Sep 27, 2026

   

   knqiufan

   on Sep 27, 2026

   CollaboratorAuthor

   More actions

   @wutongyuonce Thanks for coordinating both issues. I support additive paged operations with a pinned Memory revision and rebuildable query indexes, keeping the existing full-result operations unchanged. Separate PRs for #1656 and #1657 make sense.

   Before freezing the contract, please clarify three points:

   1. Tag assignments can change without advancing the Memory revision. Please define how tag-filtered continuation handles those changes, such as explicit cursor invalidation/restart, so revision pinning does not imply a snapshot guarantee it cannot provide.
   2. For existing databases, document the behavior while backfill is incomplete, including whether only the new operations are unavailable or Server startup is blocked.
   3. Define page-byte and oversized-item behavior. Scale evidence should cover database query work and incremental index-write costs, in addition to loaded items and serialized bytes, preserving the improvements from perf(memory): update search projections incrementally per revision #1709.

   The normalized-index approach is acceptable in principle, provided authoritative records and exact historical references remain preserved. Shared decisions can stay in this issue and be reused by #1657.
5. wutongyuonce commented on Sep 27, 2026

   

   wutongyuonce

   on Sep 27, 2026

   Contributor

   More actions

   @knqiufan Thanks. I rechecked these points against the current tag transaction, cursor/error, migration, exact-detail, and #1709 projection boundaries. I propose the following contract refinements.

   1. Tag-filtered continuation

   A Memory revision alone cannot snapshot mutable tag assignments, so tag-filtered pagination will use a separate durable tag generation owned by the tag persistence authority. It will not reuse governance_generation or reinterpret a Memory revision.

   • Every effective normalized Memory-entry tag-set mutation increments the owning Memory tag generation in the same transaction. No-op replacements do not increment it.
   • Tag writes already serialize on the owning Artifact head; the generation update stays under that lock.
   • Page one records both the pinned Memory revision and current tag generation in the signed cursor.
   • A continuation validates the generation before and after its bounded tag/index read. A mismatch returns the existing invalid-cursor class with a documented tag_state_changed reason; the client restarts from page one.
   • Unfiltered cursors are not invalidated by tag-only changes.
   • We will not collect per-entry tag etags into the cursor or materialize an unbounded tag snapshot.

   This gives the pinned-revision guarantee for Memory state while explicitly making tag changes a restart boundary, so the contract does not imply a tag snapshot that the current model cannot provide.

   2. Existing databases while backfill is incomplete

   The migration is feature-scoped rather than a Server-wide startup gate.

   • Fresh databases initialize the query-index marker as complete.
   • Existing databases use an explicit, resumable offline plan/apply/verify backfill in bounded batches for SQLite and OceanBase. The procedure requires quiesced writes and rejects mixed old/new writers.
   • Until verification marks the index complete, only the new paged Memory directory/history operations return 503 memory_query_index_unavailable, after normal authentication and authorization. They never fall back to the old unbounded read path.
   • Existing full-result operations, exact detail, search, and ordinary Memory writes remain available outside the maintenance window. New-version writes maintain the new index even while historical backfill is incomplete, but the paged operations remain unavailable until full verification.
   • Failed or interrupted apply is resumable and does not advertise readiness. Existing authoritative Artifact/Memory data is never rewritten by the backfill.

   This keeps an optional new read capability from taking down unrelated Server behavior while still failing closed for the bounded-work claim.

   3. Page bytes, oversized items, and scale evidence

   I will narrow the new #1656 operation to a directory response. Each item contains the exact citation (memory_ref, entry_id, entry_version_id), logical version, kind, and state. It does not expand text, source_refs, or artifact_refs; callers use the existing exact-detail operation for the body. The legacy full-list response remains unchanged.

   • Item limit remains default 50, maximum 100.
   • The Server also applies a fixed 4 MiB encoded item-payload budget, consistent with the existing bounded Source listing pattern, plus a separately bounded response envelope/cursor.
   • Selection stops before the next item would exceed either limit and returns continuation from the last emitted key.
   • All legal v1 directory fields already have bounded stored identities/kind/state, so one legal directory item cannot approach the byte cap. If corrupt legacy data or a future incompatible shape produces a single item that cannot fit, the operation returns 413 memory_directory_item_too_large, with no partial page and no advanced cursor. Entry-body size does not affect directory traversal.
   • feat(memory): add bounded pagination for Memory changes and history #1657 reuses the same page budget. Its compact change item is also bounded by identifier and reason limits, and it still supports continuation within a revision.

   Scale evidence will cover 200, 1,000, and 5,000-entry fixtures on SQLite and OceanBase where available, recording:

   • query rows examined/returned as limit + bounded lookahead, independent of total directory/history size;
   • zero entry-body expansion for the directory operation;
   • encoded page bytes and final-page behavior;
   • incremental directory/change-index row mutations per Memory write, proportional to the changed entries/changes rather than the total manifest/history;
   • bounded backfill batch work separately from steady-state reads/writes; and
   • unchanged-entry projection rows remaining untouched, preserving the perf(memory): update search projections incrementally per revision #1709 guarantee.

   The normalized index remains rebuildable query state. Artifact Revisions, Memory manifests, entry versions, tag rows, and exact historical references remain authoritative.

   With these refinements, the shared contract remains additive and pinned by Memory revision, tag-filtered cursors have an explicit restart boundary, incomplete migration cannot masquerade as bounded browsing, and both response assembly and persistence work have measurable bounds. If this resolves the three points, I will implement #1656 first and reuse these shared rules in the separate #1657 PR.
6. github-project-automation

   moved this from Todo to In progress in PowerContexton Sep 28, 2026
7. PsiACE commented on Oct 1, 2026

   

   PsiACE

   on Oct 1, 2026

   Member

   More actions

   Note: We'll revisit next steps as the migration and Memory redesign become clearer; see this comment.