Back to Benchmarks

LongMemEval

LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memory

Benchmark Metadata

PublisherSalesforce AI Research
VenuearXiv preprint
Evaluation Typeautomatic
Dimensions5
Test Prompts500
ScoringHigher is better
Update Frequencyannual
PaperView Paper
LeaderboardView Leaderboard

What It Measures

  • Information extraction across sessions
  • Multi-session reasoning
  • Knowledge update tracking
  • Temporal reasoning
  • Abstention on missing facts

What It Does Not Measure

  • Single-turn factual recall
  • Latency
  • Token-cost efficiency
  • Open-ended generation quality

All Systems Evaluated(173 systems)

19 self-reported154 estimated
RankSystemScore
#1MemPalaceBen Sigman / Milla Jovovich (independent open-source)96.6
#2Backboard IOBackboard.io93.4
#3Lyzr CognisLyzr AI90.6
#4VoyagerNVIDIA / Caltech / UT Austin / Stanford / ASU / UW (Wang et al.)87.1
#5Pickle AISoul Computer (YC-backed)86.8
#6xmemoryxmemory Inc.86.6
#7ArcMemoUC Berkeley / Stanford (Ho et al.)85.1
#8SuperAGITransformerOptimus85.1
#9ReplikaLuka, Inc.84.9
#10Swarmskyegomez / Swarms Corp84
#11MemR32025 (December submission)83.9
#12OS-Copilot / FRIDAYShanghai AI Lab / MMLab (Wu et al.)83.3
#13A-MEMAGI Research / Rutgers83.1
#14HippoRAG 2OSU NLP Group83
#15Talkie AIMiniMax82.8
#16CognigyCognigy GmbH (acquired by NICE, July 2025)82.1
#17CrewAI EnterpriseCrewAI Inc.82.1
#18MempZhejiang University (Fang et al.)82.1
#19memUNevaMind-AI82
#20Bee ComputerBee (acquired by Amazon 2026)81.9
#21SupermemorySupermemory81.6
#22MoTFudan University (Li & Qiu)81.5
#23MIRIXMIRIX AI (Wang, Chen)81.3
#24AutoWebGLMTHUDM81.1
#25ExpeLTsinghua University (Zhao et al.)81
#26BabyAGIYohei Nakajima80.9
#27BrowserGymServiceNow Research80.7
#28Suki AISuki (formerly Robin AI)80.7
#29AgentVerseOpenBMB (Tsinghua)80.6
#30Lindy AILindy AI80.5
#31LagentInternLM (Shanghai AI Lab)80.4
#32Nabla CopilotNabla80.4
#33LangflowLangflow-ai (DataStax)80.1
#34Mobile-AgentAlibaba Tongyi Lab (X-PLUG)80.1
#35CradleBAAI-Agents80
#36HebbiaHebbia, Inc.79.9
#37JARVIS-1CraftJarvis79.8
#38VoiceflowVoiceflow Inc.79.8
#39Kore AIKore.ai Inc.79.7
#40Generative AgentsStanford University / Google Research79.6
#41Onyxonyx-dot-app79.6
#42WebVoyagerMinorJerry et al.79.5
#43AGiXTJosh-XT79.4
#44Bishengdataelement79.4
#45CAMELCAMEL-AI.org79.4
#46Nuance DAXNuance Communications (Microsoft)79.3
#47SynapseNanyang Technological University (Zheng et al.)79.3
#48ChatDev 2.0OpenBMB79.2
#49ReflexionNortheastern / MIT / Princeton (Shinn et al.)79.2
#50Tab AITab (Avi Schiffmann)79.2
#51Self-RAGUniversity of Washington / Allen AI (Asai et al.)79.1
#52AutoGPT PlatformSignificant Gravitas79
#53Stack AIStack AI Inc. (YC W23)79
#54Galileo AIGalileo Technologies Inc.78.9
#55VectorShiftVectorShift Inc. (YC S23)78.8
#56Agent Workflow MemoryCMU (Wang, Mao, Fried, Neubig)78.7
#57HiMemZhu et al. (JD.com, 2026)78.4
#58Limitless PendantLimitless AI (acquired by Meta Dec 2025)78.4
#59Athina AIAthina AI (YC W23)78.3
#60HoneyHiveHoneyHive Inc.78.3
#61SCMBeihang / NLPR (Wang et al.)78.2
#62GPTeam101dotxyz77.9
#63AutoGen Core MemoryMicrosoft77.8
#64MemOSMemTensor (Li, Zhang, et al.)77.8
#65DB-GPTeosphoros-ai77.7
#66MetaGPTDeepWisdom / geekan77.7
#67Qwen-AgentQwenLM (Alibaba)77.7
#68Character AICharacter.AI (Google investment)77.5
#69FastGPTlabring77.4
#70Vellum AIVellum AI Inc. (YC W23)77.4
#71D-MemYou et al. (2025)77.3
#72SID AISID (YC)77.1
#73Friend AIFriend77
#74DifyLangGenius76.9
#75LangGraphLangChain76.9
#76CrewAICrewAI Inc. (Joao Moura)76.8
#77Think-in-MemoryAnt Group / Alibaba (Liu et al.)76.8
#78RecallMCisco Research / independent (Kynoch & Latapie)76.7
#79Nomi AIGlimpse AI, Inc.76.6
#80GAMVectorSpaceLab (BAAI-related)76.5
#81LangSmith LangGraph CloudLangChain Inc.76.4
#82Open InterpreterOpenInterpreter76.3
#83MultiOnMultiOn (now AGI Inc.)76.1
#84Personal AIPersonal AI75.9
#85Pi InflectionInflection AI75.6
#86MemaryKingjulio823875.5
#87AutoGen StudioMicrosoft Research75.5
#88BotpressBotpress Inc.75.4
#89FlowiseFlowiseAI75.3
#90MemoryLLMUCSD / Apple (Wang et al.)75.2
#91AbridgeAbridge75.1
#92HybridAGISynaLinks75.1
#93Ontotext GraphDBOntotext / Graphwise (merged with Semantic Web Company, 2025)75.1
#94Dust ttDust (formerly XP1)75
#95Titanslucidrains (community) / paper by Google Research75
#96Maxim AIMaxim AI Inc.74.9
#97Second MeMindverse (Shang, Li, et al.)74.9
#98ParadotWithFeeling.AI74.7
#99KnowAgentzjunlp (Zhejiang University)74.6
#100NemoriNemori AI (independent)74.6
#101MoTFudan (Li, Qiu)74.4
#102Plaud NotePLAUD74.4
#103AriGraphAIRI Institute / Moscow74.2
#104Granola AIGranola74.1
#105SynapseNTU / Salesforce (Zheng et al.)74
#106Charlie MnemonicGoodAI73.8
#107LightRAGHKUDS (HKU Data Intelligence Lab)73.8
#108Nano GraphRAGgusye123473.8
#109PathRAGBUPT-GAMMA73.8
#110GleanGlean Technologies73.7
#111Memoripycaspianmoon73.6
#112HippoRAGOSU NLP Group (Ohio State University)73.5
#113AppAgentTencent / mnotgod9673.4
#114StardogStardog Union Inc.73.4
#115Neo4j LLM Graph BuilderNeo4j Labs73.3
#116GraphitiZep AI73.2
#117MCP Memory ServerAnthropic / Model Context Protocol73.1
#118MiniRAGHKUDS73.1
#119ChatDBTsinghua University (Hu et al.)72.4
#120R2RSciPhi-AI72.4
#121Neo4j AuraDBNeo4j Inc.72.3
#122GraphRAGMicrosoft71.8
#123KAGOpenSPG / Ant Group71.5
#124DiffbotDiffbot Inc.71.3
#125RAGFlowInfiniFlow71.3
#126ZepZep AI71.2
#127HuggingGPT / JARVISMicrosoft Research71.2
#128Generative AgentsStanford / Google71.1
#129MemoChatUniversity of Warwick / Alibaba71.1
#130GraphRAG-SDKFalkorDB70.9
#131AllegroGraphFranz Inc.70.7
#132Memorizing TransformerGoogle Research (Wu, Rabe, Hutchins, Szegedy)70.6
#133KindroidKindroid70.4
#134RMMGoogle / UCSB (2025)70.4
#135LarimarIBM Research68.7
#136kNN-LMStanford / Facebook AI Research (Khandelwal et al.)68.1
#137HEMAindependent (Ahn et al.)66.8
#138LongMemUCSB / Microsoft Research65.9
#139Nomic AtlasNomic AI Inc.64.6
#140EM-LLMem-llm (academic consortium)64.2
#141Claude ProjectsAnthropic64
#142MemoryScopeAlibaba ModelScope63.7
#143Haystack Memorydeepset63.2
#144WeaviateWeaviate63.1
#145Gemini MemoryGoogle62.3
#146ChatGPT MemoryOpenAI61.5
#147MnemosyneJohns Hopkins / independent (2025)61.4
#148MemoroMIT Media Lab60.6
#149QuivrQuivrHQ60.6
#150LettaLetta (formerly MemGPT)60.4
#151MemoryBankInstitute of Software, Chinese Academy of Sciences60.1
#152Mnemosyneindependent59.9
#153Copilot MemoryMicrosoft59.7
#154Sana AISana Labs59.7
#155Heyday AIHeyday (shut down 2025)59.2
#156AnythingLLMMintplex Labs59.1
#157REALMGoogle Research (Guu et al.)59.1
#158Saner AISaner.AI59.1
#159Redis VectorRedis Ltd.58.8
#160LangMemLangChain58.3
#161EpsillaEpsilla Inc. (YC S23)58.3
#162Couchbase VectorCouchbase Inc.58.2
#163RagieRagie Inc.57.7
#164MemoryBankHarbin Institute of Technology / SenseTime57.5
#165MongoDB Atlas VectorMongoDB Inc.57
#166LlamaIndex MemoryLlamaIndex56.8
#167KDB AIKX Systems56.8
#168AtlasMeta AI FAIR (Izacard et al.)56.6
#169Notion AINotion Labs55.6
#170Mem AIMem Labs55.5
#171MemoriGibsonAI54.8
#172MemGPT ClassicBerkeley / Letta52.4
#173Memory³Institute for Advanced Algorithms Research Shanghai / Peking University51.3