Back to Benchmarks

MultiHop-RAG

MultiHop-RAG: Benchmarking Retrieval-Augmented Generation for Multi-Hop Queries

Benchmark Metadata

PublisherHKUST
VenueCOLM 2024
Evaluation Typeautomatic
Dimensions4
Test Prompts2,556
ScoringHigher is better
Update Frequencyannual
PaperView Paper
LeaderboardView Leaderboard

What It Measures

  • Inference, comparison, and temporal multi-hop queries
  • Retrieval recall@k for evidence chunks
  • Final answer accuracy

What It Does Not Measure

  • Personalization
  • Cross-session memory
  • Long-context window stress

All Systems Evaluated(149 systems)

5 self-reported144 estimated
RankSystemScore
#1QdrantQdrant75
#2AllegroGraphFranz Inc.75
#3AppAgentTencent / mnotgod9675
#4Athina AIAthina AI (YC W23)75
#5AutoWebGLMTHUDM75
#6Backboard IOBackboard.io75
#7BotpressBotpress Inc.75
#8BrowserGymServiceNow Research75
#9D-MemYou et al. (2025)75
#10DB-GPTeosphoros-ai75
#11DiffbotDiffbot Inc.75
#12DifyLangGenius75
#13GraphRAG-SDKFalkorDB75
#14HiMemZhu et al. (JD.com, 2026)75
#15HippoRAGOSU NLP Group (Ohio State University)75
#16HippoRAG 2OSU NLP Group75
#17HuggingGPT / JARVISMicrosoft Research75
#18KnowAgentzjunlp (Zhejiang University)75
#19MCP Memory ServerAnthropic / Model Context Protocol75
#20MemR32025 (December submission)75
#21MIRIXMIRIX AI (Wang, Chen)75
#22MultiOnMultiOn (now AGI Inc.)75
#23Nano GraphRAGgusye123475
#24Ontotext GraphDBOntotext / Graphwise (merged with Semantic Web Company, 2025)75
#25PathRAGBUPT-GAMMA75
#26RMMGoogle / UCSB (2025)75
#27StardogStardog Union Inc.75
#28SuperAGITransformerOptimus75
#29TrustRAGGoMate Community75
#30Vellum AIVellum AI Inc. (YC W23)75
#31xmemoryxmemory Inc.75
#32CradleBAAI-Agents74.7
#33HebbiaHebbia, Inc.74.6
#34R2RSciPhi-AI74.6
#35LangGraphLangChain74.5
#36Onyxonyx-dot-app74.5
#37VoiceflowVoiceflow Inc.74.4
#38AutoGen Core MemoryMicrosoft74.3
#39KAGOpenSPG / Ant Group74.3
#40LagentInternLM (Shanghai AI Lab)74.3
#41CognigyCognigy GmbH (acquired by NICE, July 2025)74.2
#42Neo4j AuraDBNeo4j Inc.74.2
#43Neo4j LLM Graph BuilderNeo4j Labs74.2
#44WebVoyagerMinorJerry et al.74.2
#45AutoGen StudioMicrosoft Research74.1
#46VectorShiftVectorShift Inc. (YC S23)74.1
#47GraphRAGMicrosoft74
#48LangSmith LangGraph CloudLangChain Inc.73.9
#49Self-RAGUniversity of Washington / Allen AI (Asai et al.)73.9
#50AutoGPT PlatformSignificant Gravitas73.7
#51Galileo AIGalileo Technologies Inc.73.6
#52RAGFlowInfiniFlow73.5
#53MemoryScopeAlibaba ModelScope73.2
#54CrewAICrewAI Inc. (Joao Moura)73.2
#55Swarmskyegomez / Swarms Corp73.2
#56Maxim AIMaxim AI Inc.73.1
#57Open InterpreterOpenInterpreter72.9
#58MilvusZilliz72.8
#59AgentVerseOpenBMB (Tsinghua)72.8
#60Dust ttDust (formerly XP1)72.7
#61Lindy AILindy AI72.7
#62ChatDBTsinghua University (Hu et al.)72.6
#63AriGraphAIRI Institute / Moscow72.5
#64Mixedbread AIMixedbread AI72.5
#65PaperQA2FutureHouse72.5
#66Bishengdataelement72.4
#67SID AISID (YC)72.4
#68SupermemorySupermemory72.4
#69MoTFudan (Li, Qiu)72.3
#70Haystack Memorydeepset72.1
#71LangflowLangflow-ai (DataStax)72
#72NemoriNemori AI (independent)72
#73Qwen-AgentQwenLM (Alibaba)72
#74SynapseNTU / Salesforce (Zheng et al.)72
#75PineconePinecone Systems71.9
#76LightRAGHKUDS (HKU Data Intelligence Lab)71.9
#77HoneyHiveHoneyHive Inc.71.7
#78FlowiseFlowiseAI71.3
#79HybridAGISynaLinks71.3
#80LarimarIBM Research71.3
#81MarkerDatalab (datalab-to)71.3
#82CAMELCAMEL-AI.org71.1
#83GleanGlean Technologies71.1
#84LangMemLangChain71
#85Cohere EmbedCohere Inc.70.9
#86Kore AIKore.ai Inc.70.8
#87WeaviateWeaviate70.7
#88ChatDev 2.0OpenBMB70.5
#89AGiXTJosh-XT70.4
#90Think-in-MemoryAnt Group / Alibaba (Liu et al.)70.4
#91Memoripycaspianmoon70.3
#92SynapseNanyang Technological University (Zheng et al.)70.3
#93LlamaIndex MemoryLlamaIndex70.2
#94MetaGPTDeepWisdom / geekan70.2
#95Stack AIStack AI Inc. (YC W23)70.2
#96GPTeam101dotxyz69.7
#97Nomic AtlasNomic AI Inc.69.7
#98Mobile-AgentAlibaba Tongyi Lab (X-PLUG)69.4
#99ChromaChroma69.2
#100FastGPTlabring69.2
#101SelfmemTsinghua / Microsoft (Cheng et al.)69
#102Voyage AIVoyage AI (acquired by MongoDB, Feb 2025)69
#103MiniRAGHKUDS68.4
#104MemoChatUniversity of Warwick / Alibaba67.9
#105MemoriGibsonAI67.6
#106GraphitiZep AI66.4
#107Mnemosyneindependent66.1
#108ParadeDBParadeDB Inc. (YC S23)65.8
#109REALMGoogle Research (Guu et al.)65.6
#110Couchbase VectorCouchbase Inc.65.5
#111txtaiNeuML65.4
#112Elasticsearch VectorElastic N.V.65.4
#113PrivateGPTZylon AI65.3
#114Mem AIMem Labs64.9
#115VectaraVectara Inc.64.9
#116ZepZep AI64.8
#117Astra DBDataStax64.6
#118Saner AISaner.AI64.6
#119MarqoMarqo Pty Ltd64.3
#120MyScaleMyScale Inc.64.1
#121ColPaliilluin-tech63.9
#122RETRODeepMind (Borgeaud et al.)63.8
#123MemoryBankHarbin Institute of Technology / SenseTime63.6
#124OpenSearch VectorOpenSearch Project (AWS-led)63.2
#125Neon VectorNeon Inc.63.1
#126Redis VectorRedis Ltd.63
#127AtlasMeta AI FAIR (Izacard et al.)62.9
#128Carbon AICarbon (acquired by Perplexity, Dec 2024)62.9
#129CognitaTrueFoundry62.9
#130LlamaCloudLlamaIndex Inc.62.9
#131Unstructured IOUnstructured Technologies Inc.62.8
#132MendableMendable (YC-backed)62.7
#133Manticore SearchManticore Software Ltd.62.3
#134VerbaWeaviate62.3
#135EpsillaEpsilla Inc. (YC S23)61.6
#136RagieRagie Inc.61.5
#137pgvector Supabase Neonpgvector OSS / Supabase Inc. / Neon Inc.61.3
#138TurboPufferTurboPuffer Inc.60.9
#139Supabase VectorSupabase Inc.60.8
#140Vespa AIYahoo / Vespa.ai (independent OSS project)60.5
#141QuivrQuivrHQ60.4
#142SingleStore VectorSingleStore Inc.60.4
#143AnythingLLMMintplex Labs60.3
#144MongoDB Atlas VectorMongoDB Inc.60.1
#145MemoroMIT Media Lab60
#146ValdYahoo Japan60
#147vectorizeVectorize Inc.59.3
#148CogneeCognee57.2
#149MemaryKingjulio823852.7