Agent Memory Challenge 推出统一基准,以 5,000 道题、同一答案模型和相同评判流程评测各智能体记忆系统,并分开排名开源与商业方案。文本赛道整合 10+ 数据集(约 1.5 亿字符历史),代码赛道含 12 个仓库、1,290 个历史 PR。由 20+ 研究机构组织,8 月 7 日截止提交,8 月中旬公布首批结果。
Agent memory benchmarks have had a basic attribution problem.
Every AI memory startup claims better recall, and until now there was no way to check.
That is why this new shared benchmark caught my attention.
It puts every entrant through the same pipeline: 5,000 questions, one fixed answer model, and the same judging process across systems.
The benchmark also separates open-source and commercial systems. That feels like the right choice: community projects can compete for prizes without being directly compared against heavily funded commercial products.
A group of 20+ research institutions is running all of them through one identical pipeline and publishing the results.
Entries close on August 7, with the first public rankings expected in mid-August.
I’ll be watching the first results closely.
Every agent memory system publishes its own numbers. Almost none of them are comparable. Different datasets, different answer models, different judges. When a score moves, you can’t tell whether the memory got better or the setup got kinder. The Agent Memory Challenge has been open since Wednesday. Submissions close August 7. The design, briefly: You expose two endpoints — add and search. That’s the entire integration. Answer generation, judging, aggregation and orchestration run on our side, identically for every entry, and we pay for that compute. Your internals stay closed. Text track 10+ datasets consolidated (PersonaMem, LoCoMo-Refined, CLBench, BEAM, LongMemEval, ScriptMem and others). ~150M characters of history, ~5,000 questions, scored across seven capability dimensions rather than collapsed into one number. Code track — the part we think doesn’t exist yet: 12 repos, 150 base tasks, and 1,290 historical PRs annotated as strongly related, weakly related or unrelated to the task at hand. Strict temporal constraints, so nothing from the future leaks backward. It measures whether a system can retrieve and reuse engineering experience, not whether it can solve an issue cold. Two boards Open-source methods — ranked and eligible for rewards Commercial products — ranked separately for procurement reference Organized by 20+ universities and research institutions. Enter → https://agentmemories.ai/competition/ Protocol, submission guide, reference integrations → https://github.com/AML-memory/agent-memory-leaderboard Results go up mid-August. Logs, retrieved evidence and judge records stay public.在 X 查看被引用的帖子
来源:Rohan Paul · x.com