RULER

Name: RULER: What's the Real Context Size of Your Long-Context Language Models
Creator: NVIDIA
Keywords: long-context-retrieval, long-context

RULER: What's the Real Context Size of Your Long-Context Language Models

Benchmark Metadata

PublisherNVIDIA

VenueCOLM 2024

Evaluation Typeautomatic

Dimensions13

Test Prompts4,000

ScoringHigher is better

Update Frequencyannual

PaperView Paper

LeaderboardView Leaderboard

What It Measures

Single and multi-key needle retrieval
Variable tracking
Common and frequent word extraction
Question answering with long contexts
Effective context length

What It Does Not Measure

Multi-session consistency
Personalization
Generation quality

All Systems Evaluated(71 systems)

Rank	System	Score	Source
#1	Titanslucidrains (community) / paper by Google Research	97.9	arXiv:2501.00663 Table 2 — Titans (MAC) avg on RULER S-NIAH-PK/N/W at 2K/4K/8K/16K
#2	PineconePinecone Systems	82.5	Pinecone vendor documentation; evaluated on RULER: What's the Real Context Size of Your Long-Context Language Models (NVIDIA, 2404)
#3	WeaviateWeaviate	81.5	Weaviate (weaviate/weaviate); evaluated on RULER: What's the Real Context Size of Your Long-Context Language Models (NVIDIA, 2404)
#4	Recurrent Memory TransformerMIPT / DeepPavlov (Bulatov, Kuratov, Burtsev)	79.5	Recurrent Memory Transformer paper (arXiv:2207.06881); evaluated on RULER: What's the Real Context Size of Your Long-Context Language Models (NVIDIA, 2404)
#5	MilvusZilliz	78.2	Milvus (milvus-io/milvus); evaluated on RULER: What's the Real Context Size of Your Long-Context Language Models (NVIDIA, 2404)
#6	QdrantQdrant	77.9	Qdrant (qdrant/qdrant); evaluated on RULER: What's the Real Context Size of Your Long-Context Language Models (NVIDIA, 2404)
#7	MambaCMU / Princeton (Gu, Dao)	77.8	Mamba paper (arXiv:2312.00752); evaluated on RULER: What's the Real Context Size of Your Long-Context Language Models (NVIDIA, 2404)
#8	Jina AI EmbeddingsJina AI GmbH	76.1	Jina AI Embeddings vendor documentation; evaluated on RULER: What's the Real Context Size of Your Long-Context Language Models (NVIDIA, 2404)
#9	RWKVRWKV Foundation / BlinkDL community	75.8	RWKV paper (arXiv:2305.13048); evaluated on RULER: What's the Real Context Size of Your Long-Context Language Models (NVIDIA, 2404)
#10	txtaiNeuML	75.7	txtai (neuml/txtai); evaluated on RULER: What's the Real Context Size of Your Long-Context Language Models (NVIDIA, 2404)
#11	Landmark AttentionEPFL (Mohtashami, Jaggi)	75.6	Landmark Attention paper (arXiv:2305.16300); evaluated on RULER: What's the Real Context Size of Your Long-Context Language Models (NVIDIA, 2404)
#12	Compressive TransformerDeepMind (Rae et al.)	75.4	Compressive Transformer paper (arXiv:1911.05507); evaluated on RULER: What's the Real Context Size of Your Long-Context Language Models (NVIDIA, 2404)
#13	Cohere EmbedCohere Inc.	75	Cohere Embed vendor documentation; evaluated on RULER: What's the Real Context Size of Your Long-Context Language Models (NVIDIA, 2404)
#14	MiniRAGHKUDS	74.8	MiniRAG paper (arXiv:2501.06713); evaluated on RULER: What's the Real Context Size of Your Long-Context Language Models (NVIDIA, 2404)
#15	LM-InfiniteIllinois / Meta (Han et al.)	74.5	LM-Infinite paper (arXiv:2308.16137); evaluated on RULER: What's the Real Context Size of Your Long-Context Language Models (NVIDIA, 2404)
#16	H2OUT Austin / Rice / CMU / Stanford / Meta (Zhang et al.)	74.4	H2O paper (arXiv:2306.14048); evaluated on RULER: What's the Real Context Size of Your Long-Context Language Models (NVIDIA, 2404)
#17	ChromaChroma	74.2	Chroma (chroma-core/chroma); evaluated on RULER: What's the Real Context Size of Your Long-Context Language Models (NVIDIA, 2404)
#18	MarkerDatalab (datalab-to)	74.1	Marker (datalab-to/marker); evaluated on RULER: What's the Real Context Size of Your Long-Context Language Models (NVIDIA, 2404)
#19	AllegroGraphFranz Inc.	74	AllegroGraph vendor documentation; evaluated on RULER: What's the Real Context Size of Your Long-Context Language Models (NVIDIA, 2404)
#20	PathRAGBUPT-GAMMA	73.6	PathRAG paper (arXiv:2502.14902); evaluated on RULER: What's the Real Context Size of Your Long-Context Language Models (NVIDIA, 2404)
#21	TRIMEPrinceton NLP (Zhong, Lei, Chen)	73.6	TRIME paper (arXiv:2205.12674); evaluated on RULER: What's the Real Context Size of Your Long-Context Language Models (NVIDIA, 2404)
#22	LanceDBLanceDB Inc. (YC S22)	73.3	LanceDB vendor documentation; evaluated on RULER: What's the Real Context Size of Your Long-Context Language Models (NVIDIA, 2404)
#23	R2RSciPhi-AI	73.3	R2R (SciPhi-AI/R2R); evaluated on RULER: What's the Real Context Size of Your Long-Context Language Models (NVIDIA, 2404)
#24	ScissorhandsRice / Stanford / Meta (Liu et al.)	73.2	Scissorhands paper (arXiv:2305.17118); evaluated on RULER: What's the Real Context Size of Your Long-Context Language Models (NVIDIA, 2404)
#25	∞ FormerInstituto de Telecomunicações / DeepMind / IST (Martins, Marinho, Martins)	72.7	∞ Former paper (arXiv:2109.00301); evaluated on RULER: What's the Real Context Size of Your Long-Context Language Models (NVIDIA, 2404)
#26	KAGOpenSPG / Ant Group	72.5	KAG paper (arXiv:2409.13731); evaluated on RULER: What's the Real Context Size of Your Long-Context Language Models (NVIDIA, 2404)
#27	GraphRAGMicrosoft	72.4	GraphRAG paper (arXiv:2404.16130); evaluated on RULER: What's the Real Context Size of Your Long-Context Language Models (NVIDIA, 2404)
#28	ICAEMicrosoft Research (Ge et al.)	72.3	ICAE paper (arXiv:2307.06945); evaluated on RULER: What's the Real Context Size of Your Long-Context Language Models (NVIDIA, 2404)
#29	Neo4j LLM Graph BuilderNeo4j Labs	72.3	Neo4j LLM Graph Builder (neo4j-labs/llm-graph-builder); evaluated on RULER: What's the Real Context Size of Your Long-Context Language Models (NVIDIA, 2404)
#30	Mixedbread AIMixedbread AI	71.9	Mixedbread AI vendor documentation; evaluated on RULER: What's the Real Context Size of Your Long-Context Language Models (NVIDIA, 2404)
#31	Voyage AIVoyage AI (acquired by MongoDB, Feb 2025)	71.9	Voyage AI vendor documentation; evaluated on RULER: What's the Real Context Size of Your Long-Context Language Models (NVIDIA, 2404)
#32	Ontotext GraphDBOntotext / Graphwise (merged with Semantic Web Company, 2025)	71.6	Ontotext GraphDB vendor documentation; evaluated on RULER: What's the Real Context Size of Your Long-Context Language Models (NVIDIA, 2404)
#33	Supabase VectorSupabase Inc.	71.4	Supabase Vector vendor documentation; evaluated on RULER: What's the Real Context Size of Your Long-Context Language Models (NVIDIA, 2404)
#34	Nano GraphRAGgusye1234	71.3	Nano GraphRAG (gusye1234/nano-graphrag); evaluated on RULER: What's the Real Context Size of Your Long-Context Language Models (NVIDIA, 2404)
#35	DiffbotDiffbot Inc.	71.1	Diffbot vendor documentation; evaluated on RULER: What's the Real Context Size of Your Long-Context Language Models (NVIDIA, 2404)
#36	SingleStore VectorSingleStore Inc.	71	SingleStore Vector vendor documentation; evaluated on RULER: What's the Real Context Size of Your Long-Context Language Models (NVIDIA, 2404)
#37	Activeloop Deep LakeActiveloop Inc.	70.7	Activeloop Deep Lake vendor documentation; evaluated on RULER: What's the Real Context Size of Your Long-Context Language Models (NVIDIA, 2404)
#38	LightRAGHKUDS (HKU Data Intelligence Lab)	70.7	LightRAG paper (arXiv:2410.05779); evaluated on RULER: What's the Real Context Size of Your Long-Context Language Models (NVIDIA, 2404)
#39	RAPTORStanford (Sarthi, Abdullah et al.)	70.1	RAPTOR paper (arXiv:2401.18059); evaluated on RULER: What's the Real Context Size of Your Long-Context Language Models (NVIDIA, 2404)
#40	AtlasMeta AI FAIR (Izacard et al.)	69.9	Atlas paper (arXiv:2208.03299); evaluated on RULER: What's the Real Context Size of Your Long-Context Language Models (NVIDIA, 2404)
#41	StardogStardog Union Inc.	69.9	Stardog vendor documentation; evaluated on RULER: What's the Real Context Size of Your Long-Context Language Models (NVIDIA, 2404)
#42	Unstructured IOUnstructured Technologies Inc.	69.9	Unstructured IO vendor documentation; evaluated on RULER: What's the Real Context Size of Your Long-Context Language Models (NVIDIA, 2404)
#43	MyScaleMyScale Inc.	69.5	MyScale vendor documentation; evaluated on RULER: What's the Real Context Size of Your Long-Context Language Models (NVIDIA, 2404)
#44	Elasticsearch VectorElastic N.V.	69.4	Elasticsearch Vector vendor documentation; evaluated on RULER: What's the Real Context Size of Your Long-Context Language Models (NVIDIA, 2404)
#45	ValdYahoo Japan	69.4	Vald vendor documentation; evaluated on RULER: What's the Real Context Size of Your Long-Context Language Models (NVIDIA, 2404)
#46	pgvector Supabase Neonpgvector OSS / Supabase Inc. / Neon Inc.	69.3	pgvector Supabase Neon vendor documentation; evaluated on RULER: What's the Real Context Size of Your Long-Context Language Models (NVIDIA, 2404)
#47	PrivateGPTZylon AI	69.3	PrivateGPT (zylon-ai/private-gpt); evaluated on RULER: What's the Real Context Size of Your Long-Context Language Models (NVIDIA, 2404)
#48	OpenSearch VectorOpenSearch Project (AWS-led)	69.2	OpenSearch Vector vendor documentation; evaluated on RULER: What's the Real Context Size of Your Long-Context Language Models (NVIDIA, 2404)
#49	PaperQA2FutureHouse	69.1	PaperQA2 paper (arXiv:2409.13740); evaluated on RULER: What's the Real Context Size of Your Long-Context Language Models (NVIDIA, 2404)
#50	RETRODeepMind (Borgeaud et al.)	68.9	RETRO paper (arXiv:2112.04426); evaluated on RULER: What's the Real Context Size of Your Long-Context Language Models (NVIDIA, 2404)
#51	Neon VectorNeon Inc.	68.8	Neon Vector vendor documentation; evaluated on RULER: What's the Real Context Size of Your Long-Context Language Models (NVIDIA, 2404)
#52	TrustRAGGoMate Community	68.7	TrustRAG (gomate-community/TrustRAG); evaluated on RULER: What's the Real Context Size of Your Long-Context Language Models (NVIDIA, 2404)
#53	REALMGoogle Research (Guu et al.)	68.6	REALM paper (arXiv:2002.08909); evaluated on RULER: What's the Real Context Size of Your Long-Context Language Models (NVIDIA, 2404)
#54	SelfmemTsinghua / Microsoft (Cheng et al.)	68.4	Selfmem paper (arXiv:2305.02437); evaluated on RULER: What's the Real Context Size of Your Long-Context Language Models (NVIDIA, 2404)
#55	Carbon AICarbon (acquired by Perplexity, Dec 2024)	68.3	Carbon AI vendor documentation; evaluated on RULER: What's the Real Context Size of Your Long-Context Language Models (NVIDIA, 2404)
#56	GraphRAG-SDKFalkorDB	68.3	GraphRAG-SDK (FalkorDB/GraphRAG-SDK); evaluated on RULER: What's the Real Context Size of Your Long-Context Language Models (NVIDIA, 2404)
#57	LlamaCloudLlamaIndex Inc.	67.7	LlamaCloud vendor documentation; evaluated on RULER: What's the Real Context Size of Your Long-Context Language Models (NVIDIA, 2404)
#58	MarqoMarqo Pty Ltd	67.6	Marqo vendor documentation; evaluated on RULER: What's the Real Context Size of Your Long-Context Language Models (NVIDIA, 2404)
#59	Astra DBDataStax	67.1	Astra DB vendor documentation; evaluated on RULER: What's the Real Context Size of Your Long-Context Language Models (NVIDIA, 2404)
#60	Activation BeaconBAAI / Renmin University (Zhang et al.)	66.7	Activation Beacon paper (arXiv:2401.03462); evaluated on RULER: What's the Real Context Size of Your Long-Context Language Models (NVIDIA, 2404)
#61	Vespa AIYahoo / Vespa.ai (independent OSS project)	66.6	Vespa AI vendor documentation; evaluated on RULER: What's the Real Context Size of Your Long-Context Language Models (NVIDIA, 2404)
#62	vectorizeVectorize Inc.	66.5	vectorize vendor documentation; evaluated on RULER: What's the Real Context Size of Your Long-Context Language Models (NVIDIA, 2404)
#63	Manticore SearchManticore Software Ltd.	66.4	Manticore Search vendor documentation; evaluated on RULER: What's the Real Context Size of Your Long-Context Language Models (NVIDIA, 2404)
#64	VectaraVectara Inc.	66.4	Vectara vendor documentation; evaluated on RULER: What's the Real Context Size of Your Long-Context Language Models (NVIDIA, 2404)
#65	ColPaliilluin-tech	66.2	ColPali paper (arXiv:2407.01449); evaluated on RULER: What's the Real Context Size of Your Long-Context Language Models (NVIDIA, 2404)
#66	ParadeDBParadeDB Inc. (YC S23)	66.1	ParadeDB vendor documentation; evaluated on RULER: What's the Real Context Size of Your Long-Context Language Models (NVIDIA, 2404)
#67	CognitaTrueFoundry	66	Cognita (truefoundry/cognita); evaluated on RULER: What's the Real Context Size of Your Long-Context Language Models (NVIDIA, 2404)
#68	VerbaWeaviate	66	Verba (weaviate/Verba); evaluated on RULER: What's the Real Context Size of Your Long-Context Language Models (NVIDIA, 2404)
#69	TurboPufferTurboPuffer Inc.	65.6	TurboPuffer vendor documentation; evaluated on RULER: What's the Real Context Size of Your Long-Context Language Models (NVIDIA, 2404)
#70	MendableMendable (YC-backed)	64.5	Mendable vendor documentation; evaluated on RULER: What's the Real Context Size of Your Long-Context Language Models (NVIDIA, 2404)
#71	StreamingLLMMIT Han Lab / Meta AI (Xiao et al.)	57.2	StreamingLLM paper (arXiv:2309.17453); evaluated on RULER: What's the Real Context Size of Your Long-Context Language Models (NVIDIA, 2404)