Back to Arena

Landmark Attention

by EPFL (Mohtashami, Jaggi)

System Card

OrganizationEPFL (Mohtashami, Jaggi)
Released2023-05
Architecturekv-cache-extension / Block-level landmark tokens with direct attention retrieval
DetailsInserts landmark tokens representing each input block, and trains attention to use them for selecting relevant blocks. Retrieval flows through the model's own attention mechanism, preserving random access to the full context.
Parameters—
Domainlong-context
Open SourceYes
PaperView Paper
CodeRepository
neurips-2023random-accessblockretrieval-by-attention

Capability Profile

Benchmark Scores

6 of 14 benchmarks
Data Transparency:6 estimated
Long-Context Retrieval
5/5
RULER
75.685pEstimated
NIAH
77.585pEstimated
LooGLE
80.486pEstimated
LongBench
603pEstimated
∞Bench
79.563pEstimated
Multi-Turn Recall
0/2
LoCoMo
no data
MemoryBank
no data
Cross-Session Memory
0/1
LongMemEval
no data
Multi-Hop QA
1/3
BABILong
73.231pEstimated
MultiHop-RAG
no data
HotpotQA
no data
Agent Task Memory
0/1
AgentBench-Mem
no data
Personalization
0/1
PerLTQA
no data
Factuality / Grounding
0/1
RAGAS
no data
Sources:Arena estimate — derived from capability profile, not independently verified