Memory³

by Institute for Advanced Algorithms Research Shanghai / Peking University

System Card

OrganizationInstitute for Advanced Algorithms Research Shanghai / Peking University

Released2024-07

Architecturekv-cache-extension / Third memory form: explicit KV memory

DetailsIntroduces explicit memory as a third memory form (after implicit weights and context KVs). References are pre-written to disk as sparsified attention key-value vectors, then recalled during inference with a memory circuitry theory and two-stage pretraining.

Parameters—

Domainlong-contextknowledge-graph

Open SourceNo

PaperView Paper

explicit-memorykvsparsificationpretraining

Capability Profile

Benchmark Scores

6 of 14 benchmarks

Long-Context Retrieval

3/5

RULER

no data

NIAH

no data

74.532p

603p

77.231p

Multi-Turn Recall

0/2

LoCoMo

no data

MemoryBank

no data

Cross-Session Memory

1/1

51.30p

Multi-Hop QA

2/3

7328p

MultiHop-RAG

no data

57.67p

Agent Task Memory

0/1

AgentBench-Mem

no data

Personalization

0/1

PerLTQA

no data

Factuality / Grounding

0/1

RAGAS

no data

Sources:Memory³ paper (arXiv:2407.01178); evaluated on BABILong: Testing the Limits of LLMs with Long-Context Reasoning-in-a-Haystack (AIRI, 2406)Memory³ paper (arXiv:2407.01178); evaluated on HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering (Stanford / CMU, 1809)Memory³ paper (arXiv:2407.01178); evaluated on InfiniteBench: Extending Long Context Evaluation Beyond 100K Tokens (Tsinghua / OpenBMB, 2402)Memory³ paper (arXiv:2407.01178); evaluated on LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding (Tsinghua KEG, 2308)Memory³ paper (arXiv:2407.01178); evaluated on LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memory (Salesforce AI Research, 2410)Memory³ paper (arXiv:2407.01178); evaluated on LooGLE: Can Long-Context Language Models Understand Long Contexts? (Peking University, 2311)