Back to Arena
Activation Beacon
by BAAI / Renmin University (Zhang et al.)
System Card
OrganizationBAAI / Renmin University (Zhang et al.)
Released2024-01
Architecturekv-cache-extension / Beacon tokens that compress activations at intervals
DetailsInserts special "beacon" tokens at intervals that compress keys and values of preceding segments at every layer. Plug-and-play module trained in ~10k steps on short sequences.
Parameters—
Domainlong-context
Open SourceYes
PaperView Paper
CodeRepository
iclr-2025beacon100x-extensionplug-and-play
Capability Profile
Benchmark Scores
6 of 14 benchmarksMulti-Turn Recall0/2
LoCoMo
no dataMemoryBank
no dataCross-Session Memory0/1
LongMemEval
no dataMulti-Hop QA1/3
Agent Task Memory0/1
AgentBench-Mem
no dataPersonalization0/1
PerLTQA
no dataFactuality / Grounding0/1
RAGAS
no dataSources:arXiv:2401.03462 Table 3 — On Llama-2-7B-chat; avg of SQA 27.14, MQA 28.28, Sum 25.15, FSL 60.72, Code 57.83Activation Beacon paper (arXiv:2401.03462); evaluated on BABILong: Testing the Limits of LLMs with Long-Context Reasoning-in-a-Haystack (AIRI, 2406)Activation Beacon paper (arXiv:2401.03462); evaluated on InfiniteBench: Extending Long Context Evaluation Beyond 100K Tokens (Tsinghua / OpenBMB, 2402)Activation Beacon paper (arXiv:2401.03462); evaluated on LooGLE: Can Long-Context Language Models Understand Long Contexts? (Peking University, 2311)Activation Beacon paper (arXiv:2401.03462); evaluated on Needle in a Haystack (Greg Kamradt, 2024)Activation Beacon paper (arXiv:2401.03462); evaluated on RULER: What's the Real Context Size of Your Long-Context Language Models (NVIDIA, 2404)