LongBench

Name: LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding
Creator: Tsinghua KEG
Keywords: long-context-retrieval, long-context

LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding

Benchmark Metadata

PublisherTsinghua KEG

VenueACL 2024

Evaluation Typeautomatic

Dimensions21

Test Prompts4,750

ScoringHigher is better

Update Frequencyannual

PaperView Paper

LeaderboardView Leaderboard

What It Measures

Single- and multi-document QA
Summarization
Few-shot in-context learning
Synthetic retrieval
Code completion over long contexts

What It Does Not Measure

Cross-session memory
Personalization
Latency

All Systems Evaluated(121 systems)

Rank	System	Score	Source
#1	txtaiNeuML	60	txtai (neuml/txtai); evaluated on LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding (Tsinghua KEG, 2308)
#2	LlamaIndex MemoryLlamaIndex	60	LlamaIndex Memory (run-llama/llama_index); evaluated on LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding (Tsinghua KEG, 2308)
#3	Haystack Memorydeepset	60	Haystack Memory (deepset-ai/haystack); evaluated on LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding (Tsinghua KEG, 2308)
#4	Claude ProjectsAnthropic	60	Claude Projects vendor documentation; evaluated on LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding (Tsinghua KEG, 2308)
#5	PineconePinecone Systems	60	Pinecone vendor documentation; evaluated on LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding (Tsinghua KEG, 2308)
#6	WeaviateWeaviate	60	Weaviate (weaviate/weaviate); evaluated on LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding (Tsinghua KEG, 2308)
#7	QdrantQdrant	60	Qdrant (qdrant/qdrant); evaluated on LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding (Tsinghua KEG, 2308)
#8	ChromaChroma	60	Chroma (chroma-core/chroma); evaluated on LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding (Tsinghua KEG, 2308)
#9	MilvusZilliz	60	Milvus (milvus-io/milvus); evaluated on LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding (Tsinghua KEG, 2308)
#10	Activeloop Deep LakeActiveloop Inc.	60	Activeloop Deep Lake vendor documentation; evaluated on LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding (Tsinghua KEG, 2308)
#11	Adept AIAdept AI Labs (acquired by Amazon 2024)	60	Adept AI vendor documentation; evaluated on LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding (Tsinghua KEG, 2308)
#12	AgentScopeModelScope (Alibaba)	60	AgentScope paper (arXiv:2402.14034); evaluated on LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding (Tsinghua KEG, 2308)
#13	AllegroGraphFranz Inc.	60	AllegroGraph vendor documentation; evaluated on LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding (Tsinghua KEG, 2308)
#14	AnythingLLMMintplex Labs	60	AnythingLLM (Mintplex-Labs/anything-llm); evaluated on LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding (Tsinghua KEG, 2308)
#15	Astra DBDataStax	60	Astra DB vendor documentation; evaluated on LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding (Tsinghua KEG, 2308)
#16	Athina AIAthina AI (YC W23)	60	Athina AI vendor documentation; evaluated on LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding (Tsinghua KEG, 2308)
#17	AtlasMeta AI FAIR (Izacard et al.)	60	Atlas paper (arXiv:2208.03299); evaluated on LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding (Tsinghua KEG, 2308)
#18	Bishengdataelement	60	Bisheng (dataelement/bisheng); evaluated on LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding (Tsinghua KEG, 2308)
#19	Carbon AICarbon (acquired by Perplexity, Dec 2024)	60	Carbon AI vendor documentation; evaluated on LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding (Tsinghua KEG, 2308)
#20	CognitaTrueFoundry	60	Cognita (truefoundry/cognita); evaluated on LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding (Tsinghua KEG, 2308)
#21	Cohere EmbedCohere Inc.	60	Cohere Embed vendor documentation; evaluated on LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding (Tsinghua KEG, 2308)
#22	ColPaliilluin-tech	60	ColPali paper (arXiv:2407.01449); evaluated on LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding (Tsinghua KEG, 2308)
#23	Compressive TransformerDeepMind (Rae et al.)	60	Compressive Transformer paper (arXiv:1911.05507); evaluated on LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding (Tsinghua KEG, 2308)
#24	Couchbase VectorCouchbase Inc.	60	Couchbase Vector vendor documentation; evaluated on LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding (Tsinghua KEG, 2308)
#25	DiffbotDiffbot Inc.	60	Diffbot vendor documentation; evaluated on LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding (Tsinghua KEG, 2308)
#26	DifyLangGenius	60	Dify (langgenius/dify); evaluated on LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding (Tsinghua KEG, 2308)
#27	Dust ttDust (formerly XP1)	60	Dust tt vendor documentation; evaluated on LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding (Tsinghua KEG, 2308)
#28	Elasticsearch VectorElastic N.V.	60	Elasticsearch Vector vendor documentation; evaluated on LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding (Tsinghua KEG, 2308)
#29	EpsillaEpsilla Inc. (YC S23)	60	Epsilla vendor documentation; evaluated on LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding (Tsinghua KEG, 2308)
#30	FastGPTlabring	60	FastGPT (labring/FastGPT); evaluated on LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding (Tsinghua KEG, 2308)
#31	FlowiseFlowiseAI	60	Flowise (FlowiseAI/Flowise); evaluated on LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding (Tsinghua KEG, 2308)
#32	Galileo AIGalileo Technologies Inc.	60	Galileo AI vendor documentation; evaluated on LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding (Tsinghua KEG, 2308)
#33	Granola AIGranola	60	Granola AI vendor documentation; evaluated on LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding (Tsinghua KEG, 2308)
#34	GraphRAGMicrosoft	60	GraphRAG paper (arXiv:2404.16130); evaluated on LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding (Tsinghua KEG, 2308)
#35	GraphRAG-SDKFalkorDB	60	GraphRAG-SDK (FalkorDB/GraphRAG-SDK); evaluated on LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding (Tsinghua KEG, 2308)
#36	H2OUT Austin / Rice / CMU / Stanford / Meta (Zhang et al.)	60	H2O paper (arXiv:2306.14048); evaluated on LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding (Tsinghua KEG, 2308)
#37	HebbiaHebbia, Inc.	60	Hebbia vendor documentation; evaluated on LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding (Tsinghua KEG, 2308)
#38	HoneyHiveHoneyHive Inc.	60	HoneyHive vendor documentation; evaluated on LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding (Tsinghua KEG, 2308)
#39	ICAEMicrosoft Research (Ge et al.)	60	ICAE paper (arXiv:2307.06945); evaluated on LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding (Tsinghua KEG, 2308)
#40	∞ FormerInstituto de Telecomunicações / DeepMind / IST (Martins, Marinho, Martins)	60	∞ Former paper (arXiv:2109.00301); evaluated on LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding (Tsinghua KEG, 2308)
#41	Jina AI EmbeddingsJina AI GmbH	60	Jina AI Embeddings vendor documentation; evaluated on LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding (Tsinghua KEG, 2308)
#42	KAGOpenSPG / Ant Group	60	KAG paper (arXiv:2409.13731); evaluated on LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding (Tsinghua KEG, 2308)
#43	KDB AIKX Systems	60	KDB AI vendor documentation; evaluated on LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding (Tsinghua KEG, 2308)
#44	kNN-LMStanford / Facebook AI Research (Khandelwal et al.)	60	kNN-LM paper (arXiv:1911.00172); evaluated on LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding (Tsinghua KEG, 2308)
#45	LanceDBLanceDB Inc. (YC S22)	60	LanceDB vendor documentation; evaluated on LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding (Tsinghua KEG, 2308)
#46	Landmark AttentionEPFL (Mohtashami, Jaggi)	60	Landmark Attention paper (arXiv:2305.16300); evaluated on LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding (Tsinghua KEG, 2308)
#47	LangflowLangflow-ai (DataStax)	60	Langflow (langflow-ai/langflow); evaluated on LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding (Tsinghua KEG, 2308)
#48	LangSmith LangGraph CloudLangChain Inc.	60	LangSmith LangGraph Cloud vendor documentation; evaluated on LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding (Tsinghua KEG, 2308)
#49	LightRAGHKUDS (HKU Data Intelligence Lab)	60	LightRAG paper (arXiv:2410.05779); evaluated on LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding (Tsinghua KEG, 2308)
#50	LlamaCloudLlamaIndex Inc.	60	LlamaCloud vendor documentation; evaluated on LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding (Tsinghua KEG, 2308)
#51	LM-InfiniteIllinois / Meta (Han et al.)	60	LM-Infinite paper (arXiv:2308.16137); evaluated on LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding (Tsinghua KEG, 2308)
#52	LongMemUCSB / Microsoft Research	60	LongMem paper (arXiv:2306.07174); evaluated on LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding (Tsinghua KEG, 2308)
#53	MambaCMU / Princeton (Gu, Dao)	60	Mamba paper (arXiv:2312.00752); evaluated on LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding (Tsinghua KEG, 2308)
#54	Manticore SearchManticore Software Ltd.	60	Manticore Search vendor documentation; evaluated on LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding (Tsinghua KEG, 2308)
#55	MarkerDatalab (datalab-to)	60	Marker (datalab-to/marker); evaluated on LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding (Tsinghua KEG, 2308)
#56	MarqoMarqo Pty Ltd	60	Marqo vendor documentation; evaluated on LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding (Tsinghua KEG, 2308)
#57	Maxim AIMaxim AI Inc.	60	Maxim AI vendor documentation; evaluated on LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding (Tsinghua KEG, 2308)
#58	Mem AIMem Labs	60	Mem AI vendor documentation; evaluated on LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding (Tsinghua KEG, 2308)
#59	MemformerUC Santa Barbara / Amazon (Wu, Lan, Liu, et al.)	60	Memformer paper (arXiv:2010.06891); evaluated on LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding (Tsinghua KEG, 2308)
#60	Memorizing TransformerGoogle Research (Wu, Rabe, Hutchins, Szegedy)	60	Memorizing Transformer paper (arXiv:2203.08913); evaluated on LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding (Tsinghua KEG, 2308)
#61	Memory³Institute for Advanced Algorithms Research Shanghai / Peking University	60	Memory³ paper (arXiv:2407.01178); evaluated on LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding (Tsinghua KEG, 2308)
#62	MemoryLLMUCSD / Apple (Wang et al.)	60	MemoryLLM paper (arXiv:2402.04624); evaluated on LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding (Tsinghua KEG, 2308)
#63	MemR32025 (December submission)	60	MemR3 paper (arXiv:2512.20237); evaluated on LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding (Tsinghua KEG, 2308)
#64	MendableMendable (YC-backed)	60	Mendable vendor documentation; evaluated on LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding (Tsinghua KEG, 2308)
#65	MiniRAGHKUDS	60	MiniRAG paper (arXiv:2501.06713); evaluated on LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding (Tsinghua KEG, 2308)
#66	Mixedbread AIMixedbread AI	60	Mixedbread AI vendor documentation; evaluated on LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding (Tsinghua KEG, 2308)
#67	MongoDB Atlas VectorMongoDB Inc.	60	MongoDB Atlas Vector vendor documentation; evaluated on LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding (Tsinghua KEG, 2308)
#68	MyScaleMyScale Inc.	60	MyScale vendor documentation; evaluated on LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding (Tsinghua KEG, 2308)
#69	Nano GraphRAGgusye1234	60	Nano GraphRAG (gusye1234/nano-graphrag); evaluated on LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding (Tsinghua KEG, 2308)
#70	Neo4j LLM Graph BuilderNeo4j Labs	60	Neo4j LLM Graph Builder (neo4j-labs/llm-graph-builder); evaluated on LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding (Tsinghua KEG, 2308)
#71	Neon VectorNeon Inc.	60	Neon Vector vendor documentation; evaluated on LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding (Tsinghua KEG, 2308)
#72	Nomic AtlasNomic AI Inc.	60	Nomic Atlas vendor documentation; evaluated on LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding (Tsinghua KEG, 2308)
#73	Notion AINotion Labs	60	Notion AI vendor documentation; evaluated on LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding (Tsinghua KEG, 2308)
#74	Ontotext GraphDBOntotext / Graphwise (merged with Semantic Web Company, 2025)	60	Ontotext GraphDB vendor documentation; evaluated on LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding (Tsinghua KEG, 2308)
#75	Onyxonyx-dot-app	60	Onyx (onyx-dot-app/onyx); evaluated on LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding (Tsinghua KEG, 2308)
#76	OpenSearch VectorOpenSearch Project (AWS-led)	60	OpenSearch Vector vendor documentation; evaluated on LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding (Tsinghua KEG, 2308)
#77	PaperQA2FutureHouse	60	PaperQA2 paper (arXiv:2409.13740); evaluated on LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding (Tsinghua KEG, 2308)
#78	ParadeDBParadeDB Inc. (YC S23)	60	ParadeDB vendor documentation; evaluated on LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding (Tsinghua KEG, 2308)
#79	PathRAGBUPT-GAMMA	60	PathRAG paper (arXiv:2502.14902); evaluated on LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding (Tsinghua KEG, 2308)
#80	pgvector Supabase Neonpgvector OSS / Supabase Inc. / Neon Inc.	60	pgvector Supabase Neon vendor documentation; evaluated on LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding (Tsinghua KEG, 2308)
#81	PrivateGPTZylon AI	60	PrivateGPT (zylon-ai/private-gpt); evaluated on LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding (Tsinghua KEG, 2308)
#82	QuivrQuivrHQ	60	Quivr (QuivrHQ/quivr); evaluated on LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding (Tsinghua KEG, 2308)
#83	Qwen-AgentQwenLM (Alibaba)	60	Qwen-Agent (QwenLM/Qwen-Agent); evaluated on LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding (Tsinghua KEG, 2308)
#84	R2RSciPhi-AI	60	R2R (SciPhi-AI/R2R); evaluated on LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding (Tsinghua KEG, 2308)
#85	R3MemHKUST (2025)	60	R3Mem paper (arXiv:2502.15957); evaluated on LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding (Tsinghua KEG, 2308)
#86	RAGFlowInfiniFlow	60	RAGFlow (infiniflow/ragflow); evaluated on LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding (Tsinghua KEG, 2308)
#87	RagieRagie Inc.	60	Ragie vendor documentation; evaluated on LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding (Tsinghua KEG, 2308)
#88	RAPTORStanford (Sarthi, Abdullah et al.)	60	RAPTOR paper (arXiv:2401.18059); evaluated on LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding (Tsinghua KEG, 2308)
#89	REALMGoogle Research (Guu et al.)	60	REALM paper (arXiv:2002.08909); evaluated on LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding (Tsinghua KEG, 2308)
#90	Recurrent Memory TransformerMIPT / DeepPavlov (Bulatov, Kuratov, Burtsev)	60	Recurrent Memory Transformer paper (arXiv:2207.06881); evaluated on LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding (Tsinghua KEG, 2308)
#91	Redis VectorRedis Ltd.	60	Redis Vector vendor documentation; evaluated on LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding (Tsinghua KEG, 2308)
#92	ReMeModelScope (Alibaba)	60	ReMe (modelscope/ReMe); evaluated on LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding (Tsinghua KEG, 2308)
#93	RETRODeepMind (Borgeaud et al.)	60	RETRO paper (arXiv:2112.04426); evaluated on LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding (Tsinghua KEG, 2308)
#94	RWKVRWKV Foundation / BlinkDL community	60	RWKV paper (arXiv:2305.13048); evaluated on LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding (Tsinghua KEG, 2308)
#95	Sana AISana Labs	60	Sana AI vendor documentation; evaluated on LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding (Tsinghua KEG, 2308)
#96	Saner AISaner.AI	60	Saner AI vendor documentation; evaluated on LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding (Tsinghua KEG, 2308)
#97	ScissorhandsRice / Stanford / Meta (Liu et al.)	60	Scissorhands paper (arXiv:2305.17118); evaluated on LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding (Tsinghua KEG, 2308)
#98	Self-RAGUniversity of Washington / Allen AI (Asai et al.)	60	Self-RAG paper (arXiv:2310.11511); evaluated on LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding (Tsinghua KEG, 2308)
#99	SelfmemTsinghua / Microsoft (Cheng et al.)	60	Selfmem paper (arXiv:2305.02437); evaluated on LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding (Tsinghua KEG, 2308)
#100	SID AISID (YC)	60	SID AI vendor documentation; evaluated on LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding (Tsinghua KEG, 2308)
#101	SingleStore VectorSingleStore Inc.	60	SingleStore Vector vendor documentation; evaluated on LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding (Tsinghua KEG, 2308)
#102	Stack AIStack AI Inc. (YC W23)	60	Stack AI vendor documentation; evaluated on LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding (Tsinghua KEG, 2308)
#103	StardogStardog Union Inc.	60	Stardog vendor documentation; evaluated on LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding (Tsinghua KEG, 2308)
#104	Supabase VectorSupabase Inc.	60	Supabase Vector vendor documentation; evaluated on LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding (Tsinghua KEG, 2308)
#105	Titanslucidrains (community) / paper by Google Research	60	Titans paper (arXiv:2501.00663); evaluated on LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding (Tsinghua KEG, 2308)
#106	TRIMEPrinceton NLP (Zhong, Lei, Chen)	60	TRIME paper (arXiv:2205.12674); evaluated on LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding (Tsinghua KEG, 2308)
#107	TrustRAGGoMate Community	60	TrustRAG (gomate-community/TrustRAG); evaluated on LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding (Tsinghua KEG, 2308)
#108	TurboPufferTurboPuffer Inc.	60	TurboPuffer vendor documentation; evaluated on LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding (Tsinghua KEG, 2308)
#109	Unstructured IOUnstructured Technologies Inc.	60	Unstructured IO vendor documentation; evaluated on LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding (Tsinghua KEG, 2308)
#110	ValdYahoo Japan	60	Vald vendor documentation; evaluated on LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding (Tsinghua KEG, 2308)
#111	VectaraVectara Inc.	60	Vectara vendor documentation; evaluated on LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding (Tsinghua KEG, 2308)
#112	vectorizeVectorize Inc.	60	vectorize vendor documentation; evaluated on LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding (Tsinghua KEG, 2308)
#113	VectorShiftVectorShift Inc. (YC S23)	60	VectorShift vendor documentation; evaluated on LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding (Tsinghua KEG, 2308)
#114	Vellum AIVellum AI Inc. (YC W23)	60	Vellum AI vendor documentation; evaluated on LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding (Tsinghua KEG, 2308)
#115	VerbaWeaviate	60	Verba (weaviate/Verba); evaluated on LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding (Tsinghua KEG, 2308)
#116	Vespa AIYahoo / Vespa.ai (independent OSS project)	60	Vespa AI vendor documentation; evaluated on LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding (Tsinghua KEG, 2308)
#117	Voyage AIVoyage AI (acquired by MongoDB, Feb 2025)	60	Voyage AI vendor documentation; evaluated on LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding (Tsinghua KEG, 2308)
#118	EM-LLMem-llm (academic consortium)	51.3	arXiv:2407.09450 Table 1 — EM-LLM (SM) on LLaMA 3.1-8B; avg of SQA 41.2 MQA 41.3 Sum 29.2 FSL 69.1 Ret 98.5 Code 64.1
#119	MemoRAGBAAI / Qhjqhj00	44.4	arXiv:2409.05591 Table 1 — Mistral-7B-v0.2-32K memory + Phi-3-mini-128K generator; avg of NarrativeQA 27.5, Qasper 43.9, MultiFieldQA 52.2, MuSiQue 33.9, 2WikiMQA 54.1, HotpotQA 54.8
#120	Activation BeaconBAAI / Renmin University (Zhang et al.)	39.8	arXiv:2401.03462 Table 3 — On Llama-2-7B-chat; avg of SQA 27.14, MQA 28.28, Sum 25.15, FSL 60.72, Code 57.83
#121	StreamingLLMMIT Han Lab / Meta AI (Xiao et al.)	24.5	arXiv:2309.17453 Table 8 — StreamingLLM 1750+1750 on Llama2-7B-chat; avg of NarrativeQA 18.2, Qasper 19.7, HotpotQA 24.9, 2WikiMQA 32.0, GovReport 26.3, MultiNews 25.9