LooGLE

Name: LooGLE: Can Long-Context Language Models Understand Long Contexts?
Creator: Peking University
Keywords: long-context-retrieval, long-context

LooGLE: Can Long-Context Language Models Understand Long Contexts?

Benchmark Metadata

PublisherPeking University

VenueACL 2024

Evaluation Typeautomatic

Dimensions7

Test Prompts776

ScoringHigher is better

Update Frequencyannual

PaperView Paper

LeaderboardView Leaderboard

What It Measures

Short and long-dependency QA
Summarization over long documents
Computation across documents
Timeline reorder
Multi-information retrieval

What It Does Not Measure

Multi-session memory
Latency
Personalization

All Systems Evaluated(22 systems)

Rank	System	Score	Source
#1	Titanslucidrains (community) / paper by Google Research	81.3	Titans paper (arXiv:2501.00663); evaluated on LooGLE: Can Long-Context Language Models Understand Long Contexts? (Peking University, 2311)
#2	MambaCMU / Princeton (Gu, Dao)	81.1	Mamba paper (arXiv:2312.00752); evaluated on LooGLE: Can Long-Context Language Models Understand Long Contexts? (Peking University, 2311)
#3	Landmark AttentionEPFL (Mohtashami, Jaggi)	80.4	Landmark Attention paper (arXiv:2305.16300); evaluated on LooGLE: Can Long-Context Language Models Understand Long Contexts? (Peking University, 2311)
#4	MemoryLLMUCSD / Apple (Wang et al.)	79.2	MemoryLLM paper (arXiv:2402.04624); evaluated on LooGLE: Can Long-Context Language Models Understand Long Contexts? (Peking University, 2311)
#5	Recurrent Memory TransformerMIPT / DeepPavlov (Bulatov, Kuratov, Burtsev)	79	Recurrent Memory Transformer paper (arXiv:2207.06881); evaluated on LooGLE: Can Long-Context Language Models Understand Long Contexts? (Peking University, 2311)
#6	Jina AI EmbeddingsJina AI GmbH	78.9	Jina AI Embeddings vendor documentation; evaluated on LooGLE: Can Long-Context Language Models Understand Long Contexts? (Peking University, 2311)
#7	ICAEMicrosoft Research (Ge et al.)	78.7	ICAE paper (arXiv:2307.06945); evaluated on LooGLE: Can Long-Context Language Models Understand Long Contexts? (Peking University, 2311)
#8	∞ FormerInstituto de Telecomunicações / DeepMind / IST (Martins, Marinho, Martins)	77.9	∞ Former paper (arXiv:2109.00301); evaluated on LooGLE: Can Long-Context Language Models Understand Long Contexts? (Peking University, 2311)
#9	LM-InfiniteIllinois / Meta (Han et al.)	77.7	LM-Infinite paper (arXiv:2308.16137); evaluated on LooGLE: Can Long-Context Language Models Understand Long Contexts? (Peking University, 2311)
#10	Compressive TransformerDeepMind (Rae et al.)	77.6	Compressive Transformer paper (arXiv:1911.05507); evaluated on LooGLE: Can Long-Context Language Models Understand Long Contexts? (Peking University, 2311)
#11	Memorizing TransformerGoogle Research (Wu, Rabe, Hutchins, Szegedy)	77.5	Memorizing Transformer paper (arXiv:2203.08913); evaluated on LooGLE: Can Long-Context Language Models Understand Long Contexts? (Peking University, 2311)
#12	RAPTORStanford (Sarthi, Abdullah et al.)	77.1	RAPTOR paper (arXiv:2401.18059); evaluated on LooGLE: Can Long-Context Language Models Understand Long Contexts? (Peking University, 2311)
#13	H2OUT Austin / Rice / CMU / Stanford / Meta (Zhang et al.)	75.9	H2O paper (arXiv:2306.14048); evaluated on LooGLE: Can Long-Context Language Models Understand Long Contexts? (Peking University, 2311)
#14	TRIMEPrinceton NLP (Zhong, Lei, Chen)	75.4	TRIME paper (arXiv:2205.12674); evaluated on LooGLE: Can Long-Context Language Models Understand Long Contexts? (Peking University, 2311)
#15	Memory³Institute for Advanced Algorithms Research Shanghai / Peking University	74.5	Memory³ paper (arXiv:2407.01178); evaluated on LooGLE: Can Long-Context Language Models Understand Long Contexts? (Peking University, 2311)
#16	RWKVRWKV Foundation / BlinkDL community	73.9	RWKV paper (arXiv:2305.13048); evaluated on LooGLE: Can Long-Context Language Models Understand Long Contexts? (Peking University, 2311)
#17	LongMemUCSB / Microsoft Research	73.1	LongMem paper (arXiv:2306.07174); evaluated on LooGLE: Can Long-Context Language Models Understand Long Contexts? (Peking University, 2311)
#18	ScissorhandsRice / Stanford / Meta (Liu et al.)	72.9	Scissorhands paper (arXiv:2305.17118); evaluated on LooGLE: Can Long-Context Language Models Understand Long Contexts? (Peking University, 2311)
#19	Activation BeaconBAAI / Renmin University (Zhang et al.)	68.6	Activation Beacon paper (arXiv:2401.03462); evaluated on LooGLE: Can Long-Context Language Models Understand Long Contexts? (Peking University, 2311)
#20	StreamingLLMMIT Han Lab / Meta AI (Xiao et al.)	66.9	StreamingLLM paper (arXiv:2309.17453); evaluated on LooGLE: Can Long-Context Language Models Understand Long Contexts? (Peking University, 2311)
#21	Activeloop Deep LakeActiveloop Inc.	61.3	Activeloop Deep Lake vendor documentation; evaluated on LooGLE: Can Long-Context Language Models Understand Long Contexts? (Peking University, 2311)
#22	LanceDBLanceDB Inc. (YC S22)	59.9	LanceDB vendor documentation; evaluated on LooGLE: Can Long-Context Language Models Understand Long Contexts? (Peking University, 2311)