Awesome In-Parameter Memory

This repository accompanies our survey paper:

Towards In-Parameter Memory Augmentation for Large Language Models

---

This repository maintains a continuously updated collection of in-parameter memory (IPM) augmentation methods for large language models — methods that compose a memory-bearing parameter object $\phi$ (adapters, soft prompts, encoded KV banks, fast weights, online neural memory, memory tables) into the forward pass at inference time, whether it is acquired before or during deployment. We organize the literature using the two-dimensional taxonomy introduced in our survey:

🔷 Parameter Placement — where is the memory object composed into the transformer block? (Embedding / Attention / FFN / Hybrid)
🔷 Parameter Acquisition Time — when is the deployed memory object formed? (Online during serving / Offline before serving)

News

Contributing

🙋 If you would like to include your paper in this survey and repository, please feel free to submit a pull request, or open an issue with the paper's title and a brief summary highlighting its key techniques. You can also contact us via email.

🙋🏻‍♀️ Please let us know if you find any metadata, venue, classification, or link errors, or have suggestions for the repository. We greatly appreciate your feedback!

🌟 If you find this resource helpful for your work, please consider giving us a star and citing our research.

---

Quick Index

• Embedding Layer
  • Online Acquisition
  • Offline Acquisition
• Attention Layer
  • Online Acquisition
  • Offline Acquisition
• FFN Layer
  • Online Acquisition
  • Offline Acquisition
• Hybrid
  • Online Acquisition
  • Offline Acquisition
• Insights from the Survey
  • Trade-offs
  • Open Directions

---

Embedding Layer

The memory object $\phi$ enters as continuous prompt tokens, prefixes, or latent input states appended before the first transformer block. Typical memory objects: soft prompts, soft tokens, memory tables. Embedding-side objects are the cheapest to compose (the backbone stays frozen), but their influence is indirect, mediated by every layer above.

Embedding — Online Acquisition

The memory object is generated or updated while the model is serving, e.g., synthesized per request from the current hidden state or compressed from the context at test time.

Paper	Memory Object	Acquisition Operator $\mathcal{A}$	Code
MemGen: Weaving Generative Latent Memory for Self-Evolving Agents [Link]	Soft Tokens	RL-trained trigger + LoRA weaver	MemGen
MINT: Memory-Infused Prompt Tuning at Test-Time for CLIP [Link]	Soft Prompts	Test-time bank update
GradMem: Learning to Write Context into Memory with Test-Time Gradient Descent [Link]	Soft Tokens	Test-time gradient descent	gradmem
REFRAG: Rethinking RAG based Decoding [Link]	Soft Tokens	Selective expansion policy
LatentMem: Customizing Latent Memory for Multi-Agent Systems [Link]	Soft Tokens	LMPO-trained composer	LatentMem
LatentSeek: Seek in the Dark — Reasoning via Test-Time Instance-Level Policy Gradient in Latent Space [Link]	Soft Tokens	Test-time policy gradient	LatentSeek

↑ Back to Index ↑

Embedding — Offline Acquisition

The memory object is formed before serving and held fixed, but still composed into the forward pass at deployment.

Paper	Memory Object	Acquisition Operator $\mathcal{A}$	Code
TokMem: One-Token Procedural Memory for Large Language Models [Link]	Soft Tokens	Offline training	TokMem
Engram: Conditional Memory via Scalable Lookup — A New Axis of Sparsity for Large Language Models [Link]	Memory Tables	Offline training	Engram
The Power of Scale for Parameter-Efficient Prompt Tuning (Prompt-Tuning) [Link]	Soft Prompts	Gradient optimization	prompt-tuning
H²MT: Semantic Hierarchy-Aware Hierarchical Memory Transformer [Link]	Memory Tables	Bottom-up aggregation
Memory Tokens: Large Language Models Can Generate Reversible Sentence Embeddings [Link]	Soft Tokens	Gradient optimization	memory_token
Memory Grafting: Scaling Language Model Pre-Training via Offline Conditional Memory [Link]	Memory Tables	Offline grafting

↑ Back to Index ↑

---

Attention Layer

The memory object $\phi$ participates in attention as keys/values, attention-side adapters, or matrix-valued associative states. Typical memory objects: encoded KV banks, adapters, fast weights, online neural memory, soft prompts. Attention-side objects are content-addressable — query-key reads support associative lookup over large banks and recurrently updated states — at the cost of coupling to head structure and attention budget.

Attention — Online Acquisition

A large family of test-time training (TTT) methods maintains online neural memory or fast weights at attention layers; KV-bank methods compose encoded representations per request.

Paper	Memory Object	Acquisition Operator $\mathcal{A}$	Code
δ-mem: Efficient Online Memory for Large Language Models [Link]	Fast Weights	Gated delta rule	delta-Mem
Metis: Memory Foundation Model [Link]	Online Neural Memory	Online write/read	Metis
Cache-to-Cache (C2C): Direct Semantic Communication Between Large Language Models [Link]	Encoded KV Banks	Cache-to-cache transfer	C2C
Titans: Learning to Memorize at Test Time [Link]	Online Neural Memory	Surprise-driven update
ATLAS: Learning to Optimally Memorize the Context at Test Time [Link]	Online Neural Memory	Context-aware update
Text-to-LoRA: Instant Transformer Adaption [Link]	Adapters	One-pass hypernetwork	text-to-lora
PrefixMemory-Tuning: Modernizing Prefix-Tuning by Decoupling the Prefix from Attention [Link]	Soft Prompts	Test-time prefix generation	PrefixMemory-Tuning
MesaNet: Sequence Modeling by Locally Optimal Test-Time Training [Link]	Online Neural Memory	Test-time training	flash-linear-attention
qTTT: Let's (not) Just Put Things in Context — Test-Time Training for Long-Context LLMs [Link]	Online Neural Memory	Test-time training
Infini-Transformer: Leave No Context Behind — Efficient Infinite Context Transformers with Infini-attention [Link]	Fast Weights	Compressive memory update
M+: Extending MemoryLLM with Scalable Long-Term Memory [Link]	Encoded KV Banks	Test-time memory composition	MemoryLLM
CEPE: Long-Context Language Modeling with Parallel Context Encoding [Link]	Encoded KV Banks	Context encoder	CEPE
TTT Layers: Learning to (Learn at Test Time) [Link]	Online Neural Memory	Test-time training	ttt-lm-pytorch
Draft-KV: Learning Useful Latent Communication Between Language Models [Link]	Encoded KV Banks	Draft-then-verify	Draft-KV

↑ Back to Index ↑

Attention — Offline Acquisition

The KV representations or attention-side parameters are organized before serving, e.g., learned prefixes or encoder-produced KV banks, and read at inference.

Paper	Memory Object	Acquisition Operator $\mathcal{A}$	Code
Prefix-Tuning: Optimizing Continuous Prompts for Generation [Link]	Soft Prompts	Gradient optimization	PrefixTuning
KBLaM: Knowledge Base Augmented Language Model [Link]	Encoded KV Banks	Knowledge encoder	KBLaM
AtlasKV: Augmenting LLMs with Billion-Scale Knowledge Graphs in 20GB VRAM [Link]	Encoded KV Banks	Knowledge encoder	AtlasKV
MoC: Mixture of Chapters — Scaling Learnt Memory in Transformers [Link]	Memory Tables	Offline training	MoC
SR-KI: Scalable and Real-Time Knowledge Integration into LLMs via Supervised Attention [Link]	Encoded KV Banks	Offline training	SR-KI
Memory³: Language Modeling with Explicit Memory [Link]	Encoded KV Banks	Sparse KV selection
FILM: Adaptable and Interpretable Neural Memory over Symbolic Knowledge [Link]	Encoded KV Banks	Offline training
TOME: Mention Memory — Incorporating Textual Knowledge into Transformers through Entity Mention Attention [Link]	Encoded KV Banks	Offline training	mentionmemory
Memory Attention [Link]	Memory Tables	Offline training	memory-attention

↑ Back to Index ↑

---

FFN Layer

The memory object $\phi$ is stored in feed-forward parameters or FFN-substituting projections. Typical memory objects: adapters, memory tables, fast weights, online neural memory. FFN-side objects write into the layers where factual associations are stored, so edits can directly overwrite specific behaviors; however, the read is position-wise, with no content-based selection among entries.

FFN — Online Acquisition

Adapters are generated on the fly by hypernetworks, or FFN-side states are updated continually during deployment.

Paper	Memory Object	Acquisition Operator $\mathcal{A}$	Code
Locas: Your Models are Principled Initializers of Locally-Supported Parametric Memories [Link]	Adapters	Continual online edit
Doc-to-LoRA: Learning to Instantly Internalize Contexts [Link]	Adapters	One-pass hypernetwork	doc-to-lora
DyPRAG: Dynamic Parametric Retrieval Augmented Generation for Test-Time Knowledge Enhancement [Link]	Adapters	Test-time parametric generation	DyPRAG
WISE: Rethinking the Knowledge Memory for Lifelong Model Editing of Large Language Models [Link]	Adapters	Side-memory edit	EasyEdit
GRACE: Aging with GRACE — Lifelong Model Editing with Discrete Key-Value Adaptors [Link]	Adapters	Codebook update	GRACE
FwPKM: Fast-Weight Product Key Memory [Link]	Fast Weights	Outer-product write	FwPKM
LaCT: Test-Time Training Done Right [Link]	Online Neural Memory	Test-time training	LaCT
Sparse Memory Finetuning: Continual Learning via Sparse Memory Finetuning [Link]	Memory Tables	Selective slot update
TMEM: Scaling Self-Evolving Agents via Parametric Memory [Link]	Memory Tables	Test-time memory write

↑ Back to Index ↑

FFN — Offline Acquisition

Knowledge is encoded into FFN-side parameters before serving, e.g., document-parameterized LoRA adapters or pretrained memory layers, and plugged in at inference.

Paper	Memory Object	Acquisition Operator $\mathcal{A}$	Code
PRAG: Parametric Retrieval Augmented Generation [Link]	Adapters	Offline parametric RAG	PRAG
PolyPRAG: Parametric Retrieval-Augmented Generation Using Latent Routing of LoRA Adapters [Link]	Adapters	Offline parametric RAG
MLP Memory: A Retriever-Pretrained Memory for Large Language Models [Link]	Adapters	Offline training	MLPMemory
MemoryLLM: Plug-n-Play Interpretable Feed-Forward Memory for Transformers [Link]	Memory Tables	Context-free FFN training
Pretraining with Hierarchical Memories: Separating Long-Tail and Common Knowledge [Link]	Memory Tables	Offline pretraining	ml-memory-pretraining
Memory Layers at Scale [Link]	Memory Tables	Offline training	memory
PlugLM: Decouple Knowledge from Parameters for Plug-and-Play Language Modeling [Link]	Adapters	Offline training	PlugLM
RING: Retrieval-Internalized Generation for Continual Large-Scale Knowledge Injection [Link]	Memory Tables	RL routing policy
PKM: Large Memory Layers with Product Keys [Link]	Memory Tables	Offline training	XLM

↑ Back to Index ↑

---

Hybrid

The memory object $\phi$ is composed at two or more sites (e.g., both attention and FFN). Typical memory objects: adapters, online neural memory. Hybrid stacks are the most expressive — shaping both what the model reads and how it computes — but updates must be coordinated across heterogeneous objects.

Hybrid — Online Acquisition

Paper	Memory Object	Acquisition Operator $\mathcal{A}$	Code
SHINE: A Scalable In-Context Hypernetwork for Mapping Context to LoRA in a Single Pass [Link]	Adapters	In-context hypernetwork	SHINE
Temp-LoRA: With Greater Text Comes Greater Necessity — Inference-Time Training Helps Long Text Generation [Link]	Adapters	Inference-time training	Temp-LoRA
Absorber LLM: Harnessing Causal Synchronization for Test-Time Training [Link]	Online Neural Memory	Test-time sync
Doc-to-Atom: Learning to Compile and Compose Memory Atoms [Link]	Adapters	Document compiler
Code2LoRA: Hypernetwork-Generated Adapters for Code Language Models under Software Evolution [Link]	Adapters	Hypernetwork

↑ Back to Index ↑

Hybrid — Offline Acquisition

Paper	Memory Object	Acquisition Operator $\mathcal{A}$	Code
LoRA: Low-Rank Adaptation of Large Language Models [Link]	Adapters	Gradient optimization	LoRA 🌟
Macaron-V1: Routing Each Turn to One Specialist LoRA on a Frozen Base [Link]	Adapters	Offline training	Macaron-V1
LatentSkill: From In-Context Textual Skills to In-Weight Latent Skills for LLM Agents [Link]	Adapters	Offline compilation	LatentSkill
KnowLA: Enhancing Parameter-Efficient Finetuning with Knowledgeable Adaptation [Link]	Adapters	Gradient optimization	KnowLA
LAG: LoRA-Augmented Generation for Knowledge-Intensive Language Tasks [Link]	Adapters	Gradient optimization
GeoRA: Geometry-Aware Low-Rank Adaptation for RLVR [Link]	Adapters	Gradient optimization

↑ Back to Index ↑

---

Citation

If you find this repository or our survey helpful, please consider citing:

@article{huang2026towards,
  title={Towards In-Parameter Memory Augmentation for Large Language Models},
  author={Huang, Haoyu and Xie, Zhongwei and Bai, Jiaxin and Gao, Yisen and Tsang, Hong Ting and Song, Wuganjing and Jing, Huihao and Li, Yufei and Song, Yangqiu},
  journal={arXiv preprint arXiv:2610.08630},
  year={2026}
}

Acknowledgments

The organization and README style of this repository are inspired by Awesome-KV-Cache-Optimization — a survey repository on system-aware KV cache optimization (ACL 2026 Findings). We thank its authors for setting an excellent example of how to maintain a survey-accompanying paper collection.