ICAE

by Microsoft Research (Ge et al.)

System Card

OrganizationMicrosoft Research (Ge et al.)

Released2023-07

Architecturekv-cache-extension / LoRA encoder + frozen decoder compressed memory slots

DetailsLoRA-adapted encoder compresses long contexts into a few memory-slot tokens that the frozen base LLM can condition on. Pretrained with autoencoding + LM objectives, then instruction-tuned.

Parameters—

Domainlong-context

Open SourceYes

PaperView Paper

CodeRepository

iclr-2024autoencoder4x-compressionlora

Capability Profile

Benchmark Scores

6 of 14 benchmarks

Long-Context Retrieval

5/5

72.359p

75.138p

78.768p

603p

77.841p

Multi-Turn Recall

0/2

LoCoMo

no data

MemoryBank

no data

Cross-Session Memory

0/1

LongMemEval

no data

Multi-Hop QA

1/3

BABILong

70.810p

MultiHop-RAG

no data

HotpotQA

no data

Agent Task Memory

0/1

AgentBench-Mem

no data

Personalization

0/1

PerLTQA

no data

Factuality / Grounding

0/1

RAGAS

no data

Sources:ICAE paper (arXiv:2307.06945); evaluated on BABILong: Testing the Limits of LLMs with Long-Context Reasoning-in-a-Haystack (AIRI, 2406)ICAE paper (arXiv:2307.06945); evaluated on InfiniteBench: Extending Long Context Evaluation Beyond 100K Tokens (Tsinghua / OpenBMB, 2402)ICAE paper (arXiv:2307.06945); evaluated on LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding (Tsinghua KEG, 2308)ICAE paper (arXiv:2307.06945); evaluated on LooGLE: Can Long-Context Language Models Understand Long Contexts? (Peking University, 2311)ICAE paper (arXiv:2307.06945); evaluated on Needle in a Haystack (Greg Kamradt, 2024)ICAE paper (arXiv:2307.06945); evaluated on RULER: What's the Real Context Size of Your Long-Context Language Models (NVIDIA, 2404)