Back to Benchmarks

AgentBench-Mem

AgentBench Memory Track

Benchmark Metadata

PublisherTsinghua KEG
VenueICLR 2024
Evaluation Typeautomatic
Dimensions8
Test Prompts1,360
ScoringHigher is better
Update Frequencyannual
PaperView Paper
LeaderboardView Leaderboard

What It Measures

  • Task-state retention across multi-step plans
  • Tool-call history consistency
  • Recovery from intermediate failures
  • Sub-goal tracking

What It Does Not Measure

  • Single-turn QA
  • Long-document retrieval
  • Personalization

All Systems Evaluated(144 systems)

RankSystemScore
#1MemaryKingjulio823872
#2A-MEMAGI Research / Rutgers72
#3AbridgeAbridge72
#4Adept AIAdept AI Labs (acquired by Amazon 2024)72
#5Agent Workflow MemoryCMU (Wang, Mao, Fried, Neubig)72
#6AgentScopeModelScope (Alibaba)72
#7AgentVerseOpenBMB (Tsinghua)72
#8AGiXTJosh-XT72
#9AppAgentTencent / mnotgod9672
#10ArcMemoUC Berkeley / Stanford (Ho et al.)72
#11Athina AIAthina AI (YC W23)72
#12AutoGen Core MemoryMicrosoft72
#13AutoGen StudioMicrosoft Research72
#14AutoGPT PlatformSignificant Gravitas72
#15AutoWebGLMTHUDM72
#16BabyAGIYohei Nakajima72
#17Backboard IOBackboard.io72
#18Bee ComputerBee (acquired by Amazon 2026)72
#19Bishengdataelement72
#20BotpressBotpress Inc.72
#21BrowserGymServiceNow Research72
#22CAMELCAMEL-AI.org72
#23Character AICharacter.AI (Google investment)72
#24Charlie MnemonicGoodAI72
#25ChatDBTsinghua University (Hu et al.)72
#26ChatDev 2.0OpenBMB72
#27CognigyCognigy GmbH (acquired by NICE, July 2025)72
#28CradleBAAI-Agents72
#29CrewAI EnterpriseCrewAI Inc.72
#30CrewAICrewAI Inc. (Joao Moura)72
#31D-MemYou et al. (2025)72
#32DB-GPTeosphoros-ai72
#33DifyLangGenius72
#34Dust ttDust (formerly XP1)72
#35ExpeLTsinghua University (Zhao et al.)72
#36FastGPTlabring72
#37FlowiseFlowiseAI72
#38Friend AIFriend72
#39Galileo AIGalileo Technologies Inc.72
#40GAMVectorSpaceLab (BAAI-related)72
#41Generative AgentsStanford / Google72
#42Generative AgentsStanford University / Google Research72
#43Granola AIGranola72
#44HebbiaHebbia, Inc.72
#45HiMemZhu et al. (JD.com, 2026)72
#46HoneyHiveHoneyHive Inc.72
#47HuggingGPT / JARVISMicrosoft Research72
#48HybridAGISynaLinks72
#49JARVIS-1CraftJarvis72
#50KnowAgentzjunlp (Zhejiang University)72
#51Kore AIKore.ai Inc.72
#52LagentInternLM (Shanghai AI Lab)72
#53LangflowLangflow-ai (DataStax)72
#54LangGraphLangChain72
#55LangSmith LangGraph CloudLangChain Inc.72
#56Limitless PendantLimitless AI (acquired by Meta Dec 2025)72
#57Lindy AILindy AI72
#58Lyzr CognisLyzr AI72
#59Maxim AIMaxim AI Inc.72
#60MCP Memory ServerAnthropic / Model Context Protocol72
#61Memoripycaspianmoon72
#62MemOSMemTensor (Li, Zhang, et al.)72
#63MempZhejiang University (Fang et al.)72
#64MemR32025 (December submission)72
#65memUNevaMind-AI72
#66MetaGPTDeepWisdom / geekan72
#67MIRIXMIRIX AI (Wang, Chen)72
#68Mobile-AgentAlibaba Tongyi Lab (X-PLUG)72
#69MoTFudan University (Li & Qiu)72
#70MoTFudan (Li, Qiu)72
#71MultiOnMultiOn (now AGI Inc.)72
#72Nabla CopilotNabla72
#73NemoriNemori AI (independent)72
#74Neo4j AuraDBNeo4j Inc.72
#75Nomi AIGlimpse AI, Inc.72
#76Nuance DAXNuance Communications (Microsoft)72
#77Onyxonyx-dot-app72
#78Open InterpreterOpenInterpreter72
#79OS-Copilot / FRIDAYShanghai AI Lab / MMLab (Wu et al.)72
#80ParadotWithFeeling.AI72
#81Pi InflectionInflection AI72
#82Pickle AISoul Computer (YC-backed)72
#83Qwen-AgentQwenLM (Alibaba)72
#84RecallMCisco Research / independent (Kynoch & Latapie)72
#85ReflexionNortheastern / MIT / Princeton (Shinn et al.)72
#86ReMeModelScope (Alibaba)72
#87ReplikaLuka, Inc.72
#88RMMGoogle / UCSB (2025)72
#89SCMBeihang / NLPR (Wang et al.)72
#90Second MeMindverse (Shang, Li, et al.)72
#91Self-RAGUniversity of Washington / Allen AI (Asai et al.)72
#92SID AISID (YC)72
#93Stack AIStack AI Inc. (YC W23)72
#94Suki AISuki (formerly Robin AI)72
#95SuperAGITransformerOptimus72
#96Swarmskyegomez / Swarms Corp72
#97SynapseNanyang Technological University (Zheng et al.)72
#98SynapseNTU / Salesforce (Zheng et al.)72
#99Tab AITab (Avi Schiffmann)72
#100Talkie AIMiniMax72
#101Think-in-MemoryAnt Group / Alibaba (Liu et al.)72
#102VectorShiftVectorShift Inc. (YC S23)72
#103Vellum AIVellum AI Inc. (YC W23)72
#104VoiceflowVoiceflow Inc.72
#105VoyagerNVIDIA / Caltech / UT Austin / Stanford / ASU / UW (Wang et al.)72
#106WebVoyagerMinorJerry et al.72
#107xmemoryxmemory Inc.72
#108GleanGlean Technologies71.8
#109KindroidKindroid71.5
#110GPTeam101dotxyz71.4
#111AriGraphAIRI Institute / Moscow70.9
#112RAGFlowInfiniFlow70.5
#113MemoChatUniversity of Warwick / Alibaba70.4
#114Haystack Memorydeepset69.7
#115Plaud NotePLAUD69.4
#116MemoryScopeAlibaba ModelScope68.9
#117Personal AIPersonal AI68.6
#118SupermemorySupermemory68.5
#119Gemini MemoryGoogle68.4
#120Claude ProjectsAnthropic68
#121HEMAindependent (Ahn et al.)68
#122MemoryBankInstitute of Software, Chinese Academy of Sciences66.8
#123MemformerUC Santa Barbara / Amazon (Wu, Lan, Liu, et al.)65.8
#124MemoryBankHarbin Institute of Technology / SenseTime65
#125ChatGPT MemoryOpenAI64.8
#126EpsillaEpsilla Inc. (YC S23)64
#127MongoDB Atlas VectorMongoDB Inc.63.8
#128Redis VectorRedis Ltd.63.7
#129Notion AINotion Labs63.6
#130MemoroMIT Media Lab63.2
#131LangMemLangChain62.7
#132KDB AIKX Systems62.5
#133MnemosyneJohns Hopkins / independent (2025)61.4
#134Mnemosyneindependent61
#135Couchbase VectorCouchbase Inc.60.2
#136Sana AISana Labs60.1
#137AnythingLLMMintplex Labs59.9
#138LlamaIndex MemoryLlamaIndex59.7
#139RagieRagie Inc.59.5
#140R3MemHKUST (2025)59.2
#141EM-LLMem-llm (academic consortium)59.1
#142MemoriGibsonAI56.9
#143LettaLetta (formerly MemGPT)52.7
#144MemGPT ClassicBerkeley / Letta47.3