Back to Benchmarks

AgentBench-Mem

AgentBench Memory Track

Benchmark Metadata

PublisherTsinghua KEG
VenueICLR 2024
Evaluation Typeautomatic
Dimensions8
Test Prompts1,360
ScoringHigher is better
Update Frequencyannual
PaperView Paper
LeaderboardView Leaderboard

What It Measures

  • Task-state retention across multi-step plans
  • Tool-call history consistency
  • Recovery from intermediate failures
  • Sub-goal tracking

What It Does Not Measure

  • Single-turn QA
  • Long-document retrieval
  • Personalization

All Systems Evaluated(143 systems)

1 self-reported142 estimated
RankSystemScore
#1MemaryKingjulio823872
#2A-MEMAGI Research / Rutgers72
#3AbridgeAbridge72
#4Adept AIAdept AI Labs (acquired by Amazon 2024)72
#5Agent Workflow MemoryCMU (Wang, Mao, Fried, Neubig)72
#6AgentScopeModelScope (Alibaba)72
#7AgentVerseOpenBMB (Tsinghua)72
#8AGiXTJosh-XT72
#9AppAgentTencent / mnotgod9672
#10ArcMemoUC Berkeley / Stanford (Ho et al.)72
#11Athina AIAthina AI (YC W23)72
#12AutoGen Core MemoryMicrosoft72
#13AutoGen StudioMicrosoft Research72
#14AutoGPT PlatformSignificant Gravitas72
#15AutoWebGLMTHUDM72
#16BabyAGIYohei Nakajima72
#17Backboard IOBackboard.io72
#18Bee ComputerBee (acquired by Amazon 2026)72
#19Bishengdataelement72
#20BotpressBotpress Inc.72
#21BrowserGymServiceNow Research72
#22CAMELCAMEL-AI.org72
#23Character AICharacter.AI (Google investment)72
#24Charlie MnemonicGoodAI72
#25ChatDBTsinghua University (Hu et al.)72
#26ChatDev 2.0OpenBMB72
#27CognigyCognigy GmbH (acquired by NICE, July 2025)72
#28CradleBAAI-Agents72
#29CrewAI EnterpriseCrewAI Inc.72
#30CrewAICrewAI Inc. (Joao Moura)72
#31D-MemYou et al. (2025)72
#32DB-GPTeosphoros-ai72
#33DifyLangGenius72
#34Dust ttDust (formerly XP1)72
#35ExpeLTsinghua University (Zhao et al.)72
#36FastGPTlabring72
#37FlowiseFlowiseAI72
#38Friend AIFriend72
#39Galileo AIGalileo Technologies Inc.72
#40GAMVectorSpaceLab (BAAI-related)72
#41Generative AgentsStanford / Google72
#42Generative AgentsStanford University / Google Research72
#43Granola AIGranola72
#44HebbiaHebbia, Inc.72
#45HiMemZhu et al. (JD.com, 2026)72
#46HoneyHiveHoneyHive Inc.72
#47HuggingGPT / JARVISMicrosoft Research72
#48HybridAGISynaLinks72
#49JARVIS-1CraftJarvis72
#50KnowAgentzjunlp (Zhejiang University)72
#51Kore AIKore.ai Inc.72
#52LagentInternLM (Shanghai AI Lab)72
#53LangflowLangflow-ai (DataStax)72
#54LangGraphLangChain72
#55LangSmith LangGraph CloudLangChain Inc.72
#56Limitless PendantLimitless AI (acquired by Meta Dec 2025)72
#57Lindy AILindy AI72
#58Maxim AIMaxim AI Inc.72
#59MCP Memory ServerAnthropic / Model Context Protocol72
#60Memoripycaspianmoon72
#61MemOSMemTensor (Li, Zhang, et al.)72
#62MempZhejiang University (Fang et al.)72
#63MemR32025 (December submission)72
#64memUNevaMind-AI72
#65MetaGPTDeepWisdom / geekan72
#66MIRIXMIRIX AI (Wang, Chen)72
#67Mobile-AgentAlibaba Tongyi Lab (X-PLUG)72
#68MoTFudan University (Li & Qiu)72
#69MoTFudan (Li, Qiu)72
#70MultiOnMultiOn (now AGI Inc.)72
#71Nabla CopilotNabla72
#72NemoriNemori AI (independent)72
#73Neo4j AuraDBNeo4j Inc.72
#74Nomi AIGlimpse AI, Inc.72
#75Nuance DAXNuance Communications (Microsoft)72
#76Onyxonyx-dot-app72
#77Open InterpreterOpenInterpreter72
#78OS-Copilot / FRIDAYShanghai AI Lab / MMLab (Wu et al.)72
#79ParadotWithFeeling.AI72
#80Pi InflectionInflection AI72
#81Pickle AISoul Computer (YC-backed)72
#82Qwen-AgentQwenLM (Alibaba)72
#83RecallMCisco Research / independent (Kynoch & Latapie)72
#84ReflexionNortheastern / MIT / Princeton (Shinn et al.)72
#85ReMeModelScope (Alibaba)72
#86ReplikaLuka, Inc.72
#87RMMGoogle / UCSB (2025)72
#88SCMBeihang / NLPR (Wang et al.)72
#89Second MeMindverse (Shang, Li, et al.)72
#90Self-RAGUniversity of Washington / Allen AI (Asai et al.)72
#91SID AISID (YC)72
#92Stack AIStack AI Inc. (YC W23)72
#93Suki AISuki (formerly Robin AI)72
#94SuperAGITransformerOptimus72
#95Swarmskyegomez / Swarms Corp72
#96SynapseNanyang Technological University (Zheng et al.)72
#97SynapseNTU / Salesforce (Zheng et al.)72
#98Tab AITab (Avi Schiffmann)72
#99Talkie AIMiniMax72
#100Think-in-MemoryAnt Group / Alibaba (Liu et al.)72
#101VectorShiftVectorShift Inc. (YC S23)72
#102Vellum AIVellum AI Inc. (YC W23)72
#103VoiceflowVoiceflow Inc.72
#104VoyagerNVIDIA / Caltech / UT Austin / Stanford / ASU / UW (Wang et al.)72
#105WebVoyagerMinorJerry et al.72
#106xmemoryxmemory Inc.72
#107GleanGlean Technologies71.8
#108KindroidKindroid71.5
#109GPTeam101dotxyz71.4
#110AriGraphAIRI Institute / Moscow70.9
#111RAGFlowInfiniFlow70.5
#112MemoChatUniversity of Warwick / Alibaba70.4
#113Haystack Memorydeepset69.7
#114Plaud NotePLAUD69.4
#115MemoryScopeAlibaba ModelScope68.9
#116Personal AIPersonal AI68.6
#117SupermemorySupermemory68.5
#118Gemini MemoryGoogle68.4
#119Claude ProjectsAnthropic68
#120HEMAindependent (Ahn et al.)68
#121MemoryBankInstitute of Software, Chinese Academy of Sciences66.8
#122MemformerUC Santa Barbara / Amazon (Wu, Lan, Liu, et al.)65.8
#123MemoryBankHarbin Institute of Technology / SenseTime65
#124ChatGPT MemoryOpenAI64.8
#125EpsillaEpsilla Inc. (YC S23)64
#126MongoDB Atlas VectorMongoDB Inc.63.8
#127Redis VectorRedis Ltd.63.7
#128Notion AINotion Labs63.6
#129MemoroMIT Media Lab63.2
#130LangMemLangChain62.7
#131KDB AIKX Systems62.5
#132MnemosyneJohns Hopkins / independent (2025)61.4
#133Mnemosyneindependent61
#134Couchbase VectorCouchbase Inc.60.2
#135Sana AISana Labs60.1
#136AnythingLLMMintplex Labs59.9
#137LlamaIndex MemoryLlamaIndex59.7
#138RagieRagie Inc.59.5
#139R3MemHKUST (2025)59.2
#140EM-LLMem-llm (academic consortium)59.1
#141MemoriGibsonAI56.9
#142LettaLetta (formerly MemGPT)52.7
#143MemGPT ClassicBerkeley / Letta47.3