BABILong

Name: BABILong: Testing the Limits of LLMs with Long-Context Reasoning-in-a-Haystack
Creator: AIRI
Keywords: multi-hop-qa, long-context, agent-memory

BABILong: Testing the Limits of LLMs with Long-Context Reasoning-in-a-Haystack

Benchmark Metadata

PublisherAIRI

VenueNeurIPS 2024

Evaluation Typeautomatic

Dimensions20

Test Prompts1,000

ScoringHigher is better

Update Frequencyannual

PaperView Paper

LeaderboardView Leaderboard

What It Measures

Multi-fact reasoning under distractors
Single supporting-fact recall
Two- and three-supporting-fact reasoning
Counting and list-tracking
Spatial and temporal relations

What It Does Not Measure

Generation quality
Open-ended dialogue
Code understanding

All Systems Evaluated(69 systems)

Rank	System	Score	Source
#1	Jina AI EmbeddingsJina AI GmbH	83.6	Jina AI Embeddings vendor documentation; evaluated on BABILong: Testing the Limits of LLMs with Long-Context Reasoning-in-a-Haystack (AIRI, 2406)
#2	MambaCMU / Princeton (Gu, Dao)	82.5	Mamba paper (arXiv:2312.00752); evaluated on BABILong: Testing the Limits of LLMs with Long-Context Reasoning-in-a-Haystack (AIRI, 2406)
#3	Titanslucidrains (community) / paper by Google Research	82.4	Titans paper (arXiv:2501.00663); evaluated on BABILong: Testing the Limits of LLMs with Long-Context Reasoning-in-a-Haystack (AIRI, 2406)
#4	MemoryLLMUCSD / Apple (Wang et al.)	82.1	MemoryLLM paper (arXiv:2402.04624); evaluated on BABILong: Testing the Limits of LLMs with Long-Context Reasoning-in-a-Haystack (AIRI, 2406)
#5	SCMBeihang / NLPR (Wang et al.)	81.3	SCM paper (arXiv:2304.13343); evaluated on BABILong: Testing the Limits of LLMs with Long-Context Reasoning-in-a-Haystack (AIRI, 2406)
#6	Memorizing TransformerGoogle Research (Wu, Rabe, Hutchins, Szegedy)	80.3	Memorizing Transformer paper (arXiv:2203.08913); evaluated on BABILong: Testing the Limits of LLMs with Long-Context Reasoning-in-a-Haystack (AIRI, 2406)
#7	Recurrent Memory TransformerMIPT / DeepPavlov (Bulatov, Kuratov, Burtsev)	80.3	Recurrent Memory Transformer paper (arXiv:2207.06881); evaluated on BABILong: Testing the Limits of LLMs with Long-Context Reasoning-in-a-Haystack (AIRI, 2406)
#8	GAMVectorSpaceLab (BAAI-related)	80	GAM (VectorSpaceLab/general-agentic-memory); evaluated on BABILong: Testing the Limits of LLMs with Long-Context Reasoning-in-a-Haystack (AIRI, 2406)
#9	Backboard IOBackboard.io	79.9	Backboard IO vendor documentation; evaluated on BABILong: Testing the Limits of LLMs with Long-Context Reasoning-in-a-Haystack (AIRI, 2406)
#10	ReMeModelScope (Alibaba)	79.8	ReMe (modelscope/ReMe); evaluated on BABILong: Testing the Limits of LLMs with Long-Context Reasoning-in-a-Haystack (AIRI, 2406)
#11	HippoRAG 2OSU NLP Group	79.4	HippoRAG 2 paper (arXiv:2502.14802); evaluated on BABILong: Testing the Limits of LLMs with Long-Context Reasoning-in-a-Haystack (AIRI, 2406)
#12	EM-LLMem-llm (academic consortium)	78.8	EM-LLM paper (forum?id=BI2int5SAC); evaluated on BABILong: Testing the Limits of LLMs with Long-Context Reasoning-in-a-Haystack (AIRI, 2406)
#13	VoyagerNVIDIA / Caltech / UT Austin / Stanford / ASU / UW (Wang et al.)	78.4	Voyager paper (arXiv:2305.16291); evaluated on BABILong: Testing the Limits of LLMs with Long-Context Reasoning-in-a-Haystack (AIRI, 2406)
#14	memUNevaMind-AI	78.3	memU (NevaMind-AI/memU); evaluated on BABILong: Testing the Limits of LLMs with Long-Context Reasoning-in-a-Haystack (AIRI, 2406)
#15	kNN-LMStanford / Facebook AI Research (Khandelwal et al.)	78.2	kNN-LM paper (arXiv:1911.00172); evaluated on BABILong: Testing the Limits of LLMs with Long-Context Reasoning-in-a-Haystack (AIRI, 2406)
#16	RecallMCisco Research / independent (Kynoch & Latapie)	78.1	RecallM paper (arXiv:2307.02738); evaluated on BABILong: Testing the Limits of LLMs with Long-Context Reasoning-in-a-Haystack (AIRI, 2406)
#17	MoTFudan University (Li & Qiu)	77.9	MoT paper (arXiv:2305.05181); evaluated on BABILong: Testing the Limits of LLMs with Long-Context Reasoning-in-a-Haystack (AIRI, 2406)
#18	Generative AgentsStanford University / Google Research	77.2	Generative Agents paper (arXiv:2304.03442); evaluated on BABILong: Testing the Limits of LLMs with Long-Context Reasoning-in-a-Haystack (AIRI, 2406)
#19	RWKVRWKV Foundation / BlinkDL community	77.2	RWKV paper (arXiv:2305.13048); evaluated on BABILong: Testing the Limits of LLMs with Long-Context Reasoning-in-a-Haystack (AIRI, 2406)
#20	KindroidKindroid	77.1	Kindroid vendor documentation; evaluated on BABILong: Testing the Limits of LLMs with Long-Context Reasoning-in-a-Haystack (AIRI, 2406)
#21	ArcMemoUC Berkeley / Stanford (Ho et al.)	77	ArcMemo paper (arXiv:2509.04439); evaluated on BABILong: Testing the Limits of LLMs with Long-Context Reasoning-in-a-Haystack (AIRI, 2406)
#22	Charlie MnemonicGoodAI	77	Charlie Mnemonic paper (); evaluated on BABILong: Testing the Limits of LLMs with Long-Context Reasoning-in-a-Haystack (AIRI, 2406)
#23	Adept AIAdept AI Labs (acquired by Amazon 2024)	76.9	Adept AI vendor documentation; evaluated on BABILong: Testing the Limits of LLMs with Long-Context Reasoning-in-a-Haystack (AIRI, 2406)
#24	MemaryKingjulio8238	76.8	Memary (kingjulio8238/Memary); evaluated on BABILong: Testing the Limits of LLMs with Long-Context Reasoning-in-a-Haystack (AIRI, 2406)
#25	ParadotWithFeeling.AI	76.8	Paradot vendor documentation; evaluated on BABILong: Testing the Limits of LLMs with Long-Context Reasoning-in-a-Haystack (AIRI, 2406)
#26	Pickle AISoul Computer (YC-backed)	76.3	Pickle AI vendor documentation; evaluated on BABILong: Testing the Limits of LLMs with Long-Context Reasoning-in-a-Haystack (AIRI, 2406)
#27	Agent Workflow MemoryCMU (Wang, Mao, Fried, Neubig)	76	Agent Workflow Memory paper (arXiv:2409.07429); evaluated on BABILong: Testing the Limits of LLMs with Long-Context Reasoning-in-a-Haystack (AIRI, 2406)
#28	Nomi AIGlimpse AI, Inc.	76	Nomi AI vendor documentation; evaluated on BABILong: Testing the Limits of LLMs with Long-Context Reasoning-in-a-Haystack (AIRI, 2406)
#29	RAPTORStanford (Sarthi, Abdullah et al.)	76	RAPTOR paper (arXiv:2401.18059); evaluated on BABILong: Testing the Limits of LLMs with Long-Context Reasoning-in-a-Haystack (AIRI, 2406)
#30	R3MemHKUST (2025)	75.8	R3Mem paper (arXiv:2502.15957); evaluated on BABILong: Testing the Limits of LLMs with Long-Context Reasoning-in-a-Haystack (AIRI, 2406)
#31	LarimarIBM Research	75.7	Larimar paper (arXiv:2403.11901); evaluated on BABILong: Testing the Limits of LLMs with Long-Context Reasoning-in-a-Haystack (AIRI, 2406)
#32	AgentScopeModelScope (Alibaba)	75.6	AgentScope paper (arXiv:2402.14034); evaluated on BABILong: Testing the Limits of LLMs with Long-Context Reasoning-in-a-Haystack (AIRI, 2406)
#33	ScissorhandsRice / Stanford / Meta (Liu et al.)	75.6	Scissorhands paper (arXiv:2305.17118); evaluated on BABILong: Testing the Limits of LLMs with Long-Context Reasoning-in-a-Haystack (AIRI, 2406)
#34	∞ FormerInstituto de Telecomunicações / DeepMind / IST (Martins, Marinho, Martins)	75.1	∞ Former paper (arXiv:2109.00301); evaluated on BABILong: Testing the Limits of LLMs with Long-Context Reasoning-in-a-Haystack (AIRI, 2406)
#35	Lyzr CognisLyzr AI	74.9	Lyzr Cognis vendor documentation; evaluated on BABILong: Testing the Limits of LLMs with Long-Context Reasoning-in-a-Haystack (AIRI, 2406)
#36	MemformerUC Santa Barbara / Amazon (Wu, Lan, Liu, et al.)	74.9	Memformer paper (arXiv:2010.06891); evaluated on BABILong: Testing the Limits of LLMs with Long-Context Reasoning-in-a-Haystack (AIRI, 2406)
#37	TRIMEPrinceton NLP (Zhong, Lei, Chen)	74.4	TRIME paper (arXiv:2205.12674); evaluated on BABILong: Testing the Limits of LLMs with Long-Context Reasoning-in-a-Haystack (AIRI, 2406)
#38	MempZhejiang University (Fang et al.)	74.3	Memp paper (arXiv:2508.06433); evaluated on BABILong: Testing the Limits of LLMs with Long-Context Reasoning-in-a-Haystack (AIRI, 2406)
#39	OS-Copilot / FRIDAYShanghai AI Lab / MMLab (Wu et al.)	74.2	OS-Copilot / FRIDAY paper (arXiv:2402.07456); evaluated on BABILong: Testing the Limits of LLMs with Long-Context Reasoning-in-a-Haystack (AIRI, 2406)
#40	Compressive TransformerDeepMind (Rae et al.)	74.1	Compressive Transformer paper (arXiv:1911.05507); evaluated on BABILong: Testing the Limits of LLMs with Long-Context Reasoning-in-a-Haystack (AIRI, 2406)
#41	H2OUT Austin / Rice / CMU / Stanford / Meta (Zhang et al.)	73.9	H2O paper (arXiv:2306.14048); evaluated on BABILong: Testing the Limits of LLMs with Long-Context Reasoning-in-a-Haystack (AIRI, 2406)
#42	LM-InfiniteIllinois / Meta (Han et al.)	73.8	LM-Infinite paper (arXiv:2308.16137); evaluated on BABILong: Testing the Limits of LLMs with Long-Context Reasoning-in-a-Haystack (AIRI, 2406)
#43	MnemosyneJohns Hopkins / independent (2025)	73.7	Mnemosyne paper (arXiv:2510.08601); evaluated on BABILong: Testing the Limits of LLMs with Long-Context Reasoning-in-a-Haystack (AIRI, 2406)
#44	A-MEMAGI Research / Rutgers	73.6	A-MEM paper (arXiv:2502.12110); evaluated on BABILong: Testing the Limits of LLMs with Long-Context Reasoning-in-a-Haystack (AIRI, 2406)
#45	KnowAgentzjunlp (Zhejiang University)	73.5	KnowAgent paper (arXiv:2403.03101); evaluated on BABILong: Testing the Limits of LLMs with Long-Context Reasoning-in-a-Haystack (AIRI, 2406)
#46	ReplikaLuka, Inc.	73.3	Replika vendor documentation; evaluated on BABILong: Testing the Limits of LLMs with Long-Context Reasoning-in-a-Haystack (AIRI, 2406)
#47	Landmark AttentionEPFL (Mohtashami, Jaggi)	73.2	Landmark Attention paper (arXiv:2305.16300); evaluated on BABILong: Testing the Limits of LLMs with Long-Context Reasoning-in-a-Haystack (AIRI, 2406)
#48	Personal AIPersonal AI	73.2	Personal AI vendor documentation; evaluated on BABILong: Testing the Limits of LLMs with Long-Context Reasoning-in-a-Haystack (AIRI, 2406)
#49	BabyAGIYohei Nakajima	73.1	BabyAGI (yoheinakajima/babyagi); evaluated on BABILong: Testing the Limits of LLMs with Long-Context Reasoning-in-a-Haystack (AIRI, 2406)
#50	Memory³Institute for Advanced Algorithms Research Shanghai / Peking University	73	Memory³ paper (arXiv:2407.01178); evaluated on BABILong: Testing the Limits of LLMs with Long-Context Reasoning-in-a-Haystack (AIRI, 2406)
#51	HEMAindependent (Ahn et al.)	72.9	HEMA paper (arXiv:2504.16754); evaluated on BABILong: Testing the Limits of LLMs with Long-Context Reasoning-in-a-Haystack (AIRI, 2406)
#52	ReflexionNortheastern / MIT / Princeton (Shinn et al.)	72.7	Reflexion paper (arXiv:2303.11366); evaluated on BABILong: Testing the Limits of LLMs with Long-Context Reasoning-in-a-Haystack (AIRI, 2406)
#53	LongMemUCSB / Microsoft Research	72.6	LongMem paper (arXiv:2306.07174); evaluated on BABILong: Testing the Limits of LLMs with Long-Context Reasoning-in-a-Haystack (AIRI, 2406)
#54	Bee ComputerBee (acquired by Amazon 2026)	72.5	Bee Computer vendor documentation; evaluated on BABILong: Testing the Limits of LLMs with Long-Context Reasoning-in-a-Haystack (AIRI, 2406)
#55	Second MeMindverse (Shang, Li, et al.)	72.2	Second Me paper (arXiv:2503.08102); evaluated on BABILong: Testing the Limits of LLMs with Long-Context Reasoning-in-a-Haystack (AIRI, 2406)
#56	MemOSMemTensor (Li, Zhang, et al.)	72.1	MemOS paper (arXiv:2505.22101); evaluated on BABILong: Testing the Limits of LLMs with Long-Context Reasoning-in-a-Haystack (AIRI, 2406)
#57	Tab AITab (Avi Schiffmann)	72.1	Tab AI vendor documentation; evaluated on BABILong: Testing the Limits of LLMs with Long-Context Reasoning-in-a-Haystack (AIRI, 2406)
#58	JARVIS-1CraftJarvis	71.6	JARVIS-1 paper (arXiv:2311.05997); evaluated on BABILong: Testing the Limits of LLMs with Long-Context Reasoning-in-a-Haystack (AIRI, 2406)
#59	Generative AgentsStanford / Google	71.5	Generative Agents paper (arXiv:2304.03442); evaluated on BABILong: Testing the Limits of LLMs with Long-Context Reasoning-in-a-Haystack (AIRI, 2406)
#60	CrewAI EnterpriseCrewAI Inc.	71.4	CrewAI Enterprise vendor documentation; evaluated on BABILong: Testing the Limits of LLMs with Long-Context Reasoning-in-a-Haystack (AIRI, 2406)
#61	ExpeLTsinghua University (Zhao et al.)	71.2	ExpeL paper (arXiv:2308.10144); evaluated on BABILong: Testing the Limits of LLMs with Long-Context Reasoning-in-a-Haystack (AIRI, 2406)
#62	ICAEMicrosoft Research (Ge et al.)	70.8	ICAE paper (arXiv:2307.06945); evaluated on BABILong: Testing the Limits of LLMs with Long-Context Reasoning-in-a-Haystack (AIRI, 2406)
#63	HippoRAGOSU NLP Group (Ohio State University)	70.3	HippoRAG paper (arXiv:2405.14831); evaluated on BABILong: Testing the Limits of LLMs with Long-Context Reasoning-in-a-Haystack (AIRI, 2406)
#64	LanceDBLanceDB Inc. (YC S22)	65.8	LanceDB vendor documentation; evaluated on BABILong: Testing the Limits of LLMs with Long-Context Reasoning-in-a-Haystack (AIRI, 2406)
#65	Activeloop Deep LakeActiveloop Inc.	62.6	Activeloop Deep Lake vendor documentation; evaluated on BABILong: Testing the Limits of LLMs with Long-Context Reasoning-in-a-Haystack (AIRI, 2406)
#66	Activation BeaconBAAI / Renmin University (Zhang et al.)	61.3	Activation Beacon paper (arXiv:2401.03462); evaluated on BABILong: Testing the Limits of LLMs with Long-Context Reasoning-in-a-Haystack (AIRI, 2406)
#67	MemoryBankInstitute of Software, Chinese Academy of Sciences	59.7	MemoryBank paper (arXiv:2305.10250); evaluated on BABILong: Testing the Limits of LLMs with Long-Context Reasoning-in-a-Haystack (AIRI, 2406)
#68	StreamingLLMMIT Han Lab / Meta AI (Xiao et al.)	59.3	StreamingLLM paper (arXiv:2309.17453); evaluated on BABILong: Testing the Limits of LLMs with Long-Context Reasoning-in-a-Haystack (AIRI, 2406)
#69	Heyday AIHeyday (shut down 2025)	58.7	Heyday AI vendor documentation; evaluated on BABILong: Testing the Limits of LLMs with Long-Context Reasoning-in-a-Haystack (AIRI, 2406)