Back to Arena

LM-Infinite

by Illinois / Meta (Han et al.)

System Card

OrganizationIllinois / Meta (Han et al.)
Released2023-08
Architecturekv-cache-extension / Lambda-shaped attention mask + distance limit
DetailsZero-shot length generalization via a Lambda-shaped attention mask (to avoid over-attention) and a distance limit (to avoid unseen positional distances). No parameter updates required.
Parameters—
Domainlong-context
Open SourceYes
PaperView Paper
CodeRepository
naacl-2024outstanding-paperlength-generalizationzero-shot

Capability Profile

Benchmark Scores

6 of 14 benchmarks
Long-Context Retrieval
5/5
RULER
74.579p
NIAH
77.277p
LooGLE
77.759p
LongBench
603p
∞Bench
8588p
Multi-Turn Recall
0/2
LoCoMo
no data
MemoryBank
no data
Cross-Session Memory
0/1
LongMemEval
no data
Multi-Hop QA
1/3
BABILong
73.839p
MultiHop-RAG
no data
HotpotQA
no data
Agent Task Memory
0/1
AgentBench-Mem
no data
Personalization
0/1
PerLTQA
no data
Factuality / Grounding
0/1
RAGAS
no data
Sources:LM-Infinite paper (arXiv:2308.16137); evaluated on BABILong: Testing the Limits of LLMs with Long-Context Reasoning-in-a-Haystack (AIRI, 2406)LM-Infinite paper (arXiv:2308.16137); evaluated on InfiniteBench: Extending Long Context Evaluation Beyond 100K Tokens (Tsinghua / OpenBMB, 2402)LM-Infinite paper (arXiv:2308.16137); evaluated on LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding (Tsinghua KEG, 2308)LM-Infinite paper (arXiv:2308.16137); evaluated on LooGLE: Can Long-Context Language Models Understand Long Contexts? (Peking University, 2311)LM-Infinite paper (arXiv:2308.16137); evaluated on Needle in a Haystack (Greg Kamradt, 2024)LM-Infinite paper (arXiv:2308.16137); evaluated on RULER: What's the Real Context Size of Your Long-Context Language Models (NVIDIA, 2404)