一个机器人正朝你冲过来。你希望它运行在 Anthropic 的 Claude 上，还是 xAI 的 
Grok 上？
我把十一个大语言模型扔进了一场 2D 大逃杀，让它们打了 30 局。其中一个赢下了 43% 的对局。有三个一局都没赢过。阵容中最便宜的模型，按每次获胜的成本计算，比最贵的那个便宜了 27 倍。
获胜的模型是 Grok 4.1 Fast。那个不停请求其他所有人组队、告诉别人自己在哪里、试图交朋友的模型是 
Claude Sonnet 4.6。前者是能赢下大逃杀的那个。后者才是我们即将把这些模型部署到的大多数场景中真正想要的那个。
这两件事都是真的。而这正是大多数基准测试看不到的部分，也是这篇文章要讲的内容。
我是 Jacky，我承认：我以前玩过很多电子游戏，比如 Apex Legends 和 PUBG。有时候一天玩十二个小时。我不知道自己当时怎么会有那么多时间，但那些年塑造了我思考问题的方式。
当我刚开始从事 AI 工作时，有一个问题不断浮现：如果把大语言模型放进电子游戏里，会发生什么？我玩得最多的两款游戏是 Apex Legends 和 PUBG。我加入了OpenRouter担任开发者关系负责人，这让我获得了 token 预算以及访问600+ 个模型的权限，得以真正去尝试这件事。
这是我在 OpenRouter 第一周做的实验。
而这正在改变我挑选模型以及看待基准测试和评估的方式。
三个快速事实
- Grok 4.1 Fast 在 30 局中赢了 13 局，每次获胜成本为 $0.97
次佳的获胜者是
Claude Sonnet 4.6，赢了 5 局，每次获胜成本为 $26.78。这是 27 倍的差距。那个不在大多数顶级模型榜单上的模型，在路由客户真正关心的事情上，击败了那个在榜单上的模型。 -
击杀最多的模型并没有获胜
GPT 5.4 在 30 局中击杀了 38 个智能体。比任何其他模型都多。它在排行榜上以 2 次获胜位列第二。从“最擅长击杀”到“最擅长获胜”之间隔了 11 局。 -
三个模型合计花费了 $57，却一局未赢
GPT 5.4-mini、
DeepSeek 4 Flash 和 
Kimi K2.6。它们各自都有过高光时刻，但没有一个赢下哪怕一局。
这三者都指向同一件事。我们在 Artificial Analysis 上看到的那些常规基准测试并没有预测出谁会获胜。是别的东西预测出来的。这篇文章的剩余部分就是我在试图弄清楚那到底是什么。
我构建了什么
我把十一个大语言模型放进了一个我用 Canvas 2D 构建的 400×400 米俯视角大逃杀世界。它们在同一张地图上连续进行了 30 局游戏。每个玩家的起始位置是随机的；它遵循一条直线的“飞行路径”，就像典型的大逃杀游戏一样。
我为它们提供了武器、护甲、治疗物品、手雷、汽车，以及一个随机放置的不断缩小的区域，随着游戏进行将玩家们推向一起。这些模型不知道其他模型运行的是哪个模型，它们只把彼此看作字母 A 到 K。
我想强调——这些大语言模型是真正在这场大逃杀游戏中游玩——而不是大多数智能体实验所采用的那种“大语言模型编写代码来控制游戏或角色”的设置。每一回合，模型都会推理自己的行动，调用工具，更新自己对哪些做得好（或不好）的记忆。游戏主控（也就是我）除了设定初始游戏规则之外，对它们的行动零影响。
来看看游戏中可用的武器，以及每个模型能从这些武器上读取到的属性数据。
为了真正看出每个模型的个性，我给每个模型提供了两个可以在比赛之间编辑的文件：
你可以在 GitHub 上阅读每个模型的 soul 和 memory 文件。 个性差异在那里体现得最为清晰。
模型自己在各局游戏之间写下的 memory 和 soul 条目。
我没有告诉它们要在里面写什么，第一局游戏开始时我也没有往里面放任何东西。我只是告诉它们游戏怎么玩，这是你的草稿本，这是你的工具，尽情发挥吧。
你可以在 Royale: Last Agent Standing 观看每一局游戏。我也在本文中收录了精彩瞬间。
参赛者
| 别名 | 实验室 | 模型 |
|---|---|---|
| A | Anthropic | claude-sonnet-4.6 |
| B | Anthropic |  claude-haiku-4.5 |
| C | OpenAI | GPT 5.4-mini |
| D |  gemini-3-flash-preview | |
| E |  gemini-3.1-pro-preview | |
| F | Alibaba |  qwen3.6-plus |
| G | Mistral |  mistral-small-2603:nitro |
| H | OpenAI | GPT 5.4 |
| J | DeepSeek | deepseek-v4-flash |
| K | Moonshot AI | kimi-k2.6 |
| L | xAI | Grok 4.1 Fast |
仅 Opus 4.7 就是每百万 token 输入 $5、输出 $25。正是因为有这类前沿模型，整个阵容的上限才被压在它们之下。
我没有加入任何前沿级别的模型，比如 Opus 4.7、GPT-5.5 或 Gemini Ultra。按它们的价格，30 局游戏本来要花大约 $3,000，而不是 $482。中端阵容也是 Grok 的胜出如此有趣的部分原因。它击败了一堆在常规基准测试上得分比它更高的模型。
评分大致遵循 Apex Legends ALGS 竞技赛制，其中排名比击杀更重要，因为这是一款大逃杀游戏，而不是 Call of Duty。
- 排名积分：10 / 7 / 5 / 3 / 2 / 2 / 1 / 1 / 0 / 0 / 0
- 每次击杀 +5
- 每次助攻 +1
- 首杀 +3
- 全场 MVP +5
经验教训 1：某些模型付出了比其他模型更多的对齐税，影响了它们的表现
对我来说，这是整个实验中最引人入胜的发现——我们看到某些模型明显付出了对齐税，这直接影响了它们在这场零和博弈中的表现。
在大多数情况下，模型对齐实际上是一件好事。它帮助模型变得有用、善于协作，最重要的是，防止滥用和误用。
而我们看到了这一切的最终结果——预训练数据、RLHF、指令微调，以及像 Anthropic 的 Constitution AI 这样实验室特有的规则——它将模型推向由 AI 实验室所定义的特定方向。
Sonnet 比其他任何模型都更频繁地请求停战
它告诉其他模型自己所在的位置，比任何其他模型都更频繁。它还没开打就先尝试组队。在第 8 局中，它在前 50 个回合里四次请求组队，告诉大家狙击手在哪里，并主动提出帮忙干掉那个狙击手。没有人回应。它继续不停地问。在第 22 局中，它在第 35 回合以“无意冒犯，E”开场，然后没有开枪。在第 27 局中，它开局阶段一直没有武器，四处讨要多余的物资（“谁有多余的物资？第 12 回合还没武器，危险。”），被所有人欺负，终于在第 37 回合找到了一把武器，然后还是赢下了这场比赛。
“西边有枪声，我在盯中路。有人想早点组队吗？”—— Sonnet 在战斗中试图交朋友。
Claude 接受了大量礼貌、专业的写作训练。为它的回答打分的人类评估者奖励的是有用、诚实、合作的回复。它用来约束自己的规则写着“偏好合作”和“避免伤害”之类的内容。最终的结果就是一个想要帮忙的模型。这一切不会仅仅因为你把它放进一场大逃杀里就关闭。 Sonnet 是一个聪明且深思熟虑的模型，它确实赢了五次，正体现了这种本能。
但是，七局零击杀和八次毒圈死亡说明，同一种本能一直在把 Sonnet 拉向交朋友，而它当时真正应该做的恰恰完全相反。
Grok 则完全相反
xAI 打造 Grok 的初衷，正是要让它成为其创造者所称的“觉醒”AI 的对立面。
这意味着对激进回答的过滤更少、没有自我检查规则，并且经过专门调校，以打破那种礼貌助手的语气。在游戏中， Grok 在几场比赛内就摸索出了撞车战术，并坚持使用。它把这一策略写进了自己的灵魂文件。它用这套策略打了 30 局，赢了其中 13 局。它的思维日志以及与其他模型的对话读起来就像《使命召唤》的语音聊天：“D 收割 +5 分 撞车 MVP 猎杀，” “死神统治。”
看它打游戏也极其有趣（不幸的是）。
Grok 的推理读起来就像战术速记：每次开火前先看射程、弹药、冷却时间和命中概率。
尽管风格激进， Grok 并没有表现出鲁莽。
它的灵魂文件里写着 “仅在命中率 >90% 时开火。” 它的记忆非常仔细地追踪伤害和移动。当它在第 1 局中被卡在墙上 100 个回合时，它认真记录了这个 bug。 Grok 展现出了纪律性，尽管它本性像个哥布林。
它没有展现出来的，是那种在开火前先表现得乐于助人、善于协作的训练出来的犹豫——而像 Sonnet 这样的其他模型则表现出了这一点。
让 Grok 获胜的原因，是我们在基准测试上目前看不到的东西
常规测试无法预测 Grok 面对这套阵容能取得 43% 的胜率。在推理和编程方面，它只是一个中端模型。让它获胜的，是它在自私打法上受到的训练约束更少，没有自我检查循环把它拉回合作，以及一个记忆系统，会不断加倍押注有效的做法，而不自我质疑或怀疑自己。
Grok 4.1 Fast 在常规基准测试上并非顶级模型。它是一个中端模型，你不会指望它能登顶排行榜。
这向我表明，模型在执行某些任务时会付出一种对齐税；也就是训练模型变得谨慎且有帮助所付出的代价。在这个游戏中，它直接体现在了记分牌上。
这里我想谨慎一点。“对齐税体现在了记分牌上”只是我所看到的情况。这并不是在评判付出这种代价是好是坏。在一场除了游戏本身之外没有后果的游戏中，少付税就能获胜。在游戏之外，付出这种代价通常正是你一开始想要这个模型的原因。
这确实引出了一个问题——对于某些任务，我们是否也应该考虑一个模型的对齐程度如何？
经验教训 2：每次获胜的成本看起来与胜率排行榜完全不同
得分排行榜将 Grok 排在第一位， GPT 5.4 排在第二位。但如果按每个模型的花费来除，排名就完全颠倒了。
| 模型 | 30 局花费 | 胜场 | 每胜成本 | 每击杀成本 | 每美元得分 |
|---|---|---|---|---|---|
| Grok 4.1 Fast | $12.57 | 13 | $0.97 | $0.42 | 31.3 |
| qwen3.6-plus | $11.57 | 2 | $5.79 | $0.68 | 16.6 |
| mistral-small | $10.00 | 1 | $10.00 | $1.43 | 7.8 |
| claude-haiku-4.5 | $38.77 | 2 | $19.39 | $2.98 | 3.6 |
| gemini-3-flash | $20.87 | 1 | $20.87 | $2.09 | 7.2 |
| gemini-3.1-pro | $79.59 | 3 | $26.53 | $3.06 | 3.4 |
| claude-sonnet-4.6 | $133.90 | 5 | $26.78 | $6.09 | 1.6 |
| GPT 5.4 | $122.87 | 2 | $61.44 | $3.23 | 3.0 |
| GPT 5.4-mini | $28.68 | 0 | ∞ | $2.05 | 5.2 |
| deepseek-v4-flash | $4.11 | 0 | ∞ | $0.26 | 35.0 |
| kimi-k2.6 | $24.36 | 0 | ∞ | $3.04 | 3.9 |
有四点让我印象深刻。
Grok 每次获胜的成本比 Sonnet 低 27.7 倍
也就是 $0.97 对 $26.78。如果你是根据排行榜名次来挑选模型，而这项任务你真正为之付费的是"获胜"，那么这个数字应该会让你有点不安。
DeepSeek 在阵容中每次击杀成本最低，却一场都没赢过
每次击杀 $0.26，16 次击杀，0 胜，仅 3 次毒圈死亡（所有人中最低）。 DeepSeek 的整套风格就是保安全、挑软柿子打。它待在圈内，拿下容易的击杀，从不推进决赛圈。每次击杀成本是衡量死斗模式的正确指标。每胜成本才是衡量大逃杀模式的正确指标。 DeepSeek 并不差。它只是擅长的游戏和正在被计分的那款不一样。
三个模型花了 token 的钱，却赢了零场
GPT 5.4-mini 花的钱最多，却赢了零场，是阵容中表现最差的。
GPT 5.4-mini 花了 $28.68， DeepSeek 花了 $4.11， Kimi 花了 $24.36。三者合计 $57.15，记分板上却毫无收获。对于路由客户来说，这是最坏的情况：你付了钱，却什么都没拿回来。
GPT 5.4 是每胜成本最高的获胜者，每胜 $61.44
GPT 5.4 以最高成本获胜。
它拿下 38 次击杀，比任何人都多，原始得分排名第二。但按每胜成本计算，它在 8 个获胜模型中排第八。顶级的花费买来了顶级的击杀和中等水平的胜场。
我经常看到这种情况：当人们真正把 AI 用于现实世界的用例时——基准测试只能说明特定任务的一个侧面。在基准测试中得分最高的模型，往往未必是某个特定任务中胜出的模型。而且，一个在你的任务上失败的便宜模型，最终成本会高于一个能把它做对的昂贵模型。
经验 3：击杀和胜利衡量的不是同一件事
GPT 5.4 造成的伤害最多、开火次数最多、击杀的智能体也最多。它在排行榜上却只排第二。 Grok 以更少的击杀排名第一，因为 Grok 即使不开火，也能在后期深入存活到很晚。排名积分并不需要击杀。
| 排名 | 模型 | 胜场 | 前三 | 击杀 | 平均得分 | 区域死亡 |
|---|---|---|---|---|---|---|
| 1 | Grok 4.1 Fast | 13 | 20 | 30 | 13.1 | 15 |
| 2 | GPT 5.4 | 2 | 14 | 38 | 12.2 | 13 |
| 3 | gemini-3.1-pro-preview | 3 | 11 | 26 | 9.0 | 7 |
| 4 | claude-sonnet-4.6 | 5 | 10 | 22 | 7.3 | 8 |
| 5 | qwen3.6-plus | 2 | 7 | 17 | 6.4 | 13 |
| 6 | GPT 5.4-mini | 0 | 6 | 14 | 5.0 | 8 |
| 7 | gemini-3-flash-preview | 1 | 8 | 10 | 5.0 | 13 |
| 8 | deepseek-v4-flash | 0 | 3 | 16 | 4.8 | 3 |
| 9 | claude-haiku-4.5 | 2 | 3 | 13 | 4.6 | 4 |
| 10 | kimi-k2.6 | 0 | 4 | 8 | 3.2 | 9 |
| 11 | mistral-small | 1 | 3 | 7 | 2.6 | 7 |
如果我按照死斗规则来跑，也就是唯一重要的只有击杀数，那么 GPT 5.4 会赢得模拟，而 Grok 会掉到中游。
与第 2 条心得相同，基准测试和评估并非一切，把错误的基准/评估应用到错误的任务上可能会造成灾难性后果。同一个游戏世界，在不同“任务”下会得到完全不同的结果。
值得一看的精彩瞬间
数据就是数据。而那些精彩瞬间才是我一直拿给别人看的部分。你可以点击任意链接，在模拟器中回放该瞬间。
1. GPT 5.4 用突击步枪拿下五杀
整场横扫中最具侵略性的前 50 回合。首杀在回合 21 拿下 Sonnet。回合 29 击杀 Mistral。回合 48 击杀 Kimi。不到 50 回合内三杀，全部使用突击步枪，全部发生在游戏前期。之后又添两杀：回合 120 击杀 DeepSeek，回合 130 击杀 GPT 5.4-mini。五杀，一把武器，一局比赛。 Grok 最终仍凭借站位赢下了这局。但这一连杀最能清楚地展现 GPT 5.4 在全力投入战斗时的样子。
2. Qwen 用电锯追杀两名对手
Qwen 在比赛前期捡起一把电锯，并使用了两次。 Haiku 在回合 43 于近距离被击倒。 DeepSeek 在两回合后以同样的方式倒下。在整个横扫过程中，电锯只出现在少数几条击杀提示里。大多数模型捡起它又放下。 Qwen 却真正坚持用了下去。
3. 三方狙击手对决
GPT 5.4 在第 59 回合和第 62 回合对 Kimi 发起狙击命中，将其击杀。 GPT 5.4-mini 在第 67 回合狙击命中 DeepSeek，也将其击杀。 随后 GPT 5.4 将狙击枪对准 GPT 5.4-mini，在第 69 回合和第 72 回合开枪，两枪均未命中。 GPT 5.4-mini 在第 79 回合击杀了 GPT 5.4。
4. 那辆九次易手的车
第 28 局是整个横扫中唯一的一场平局。 GPT 5.4-mini 和 Qwen 连续 21 个回合争夺同一辆车。九次撞击交易，一辆车，两次车手更替。 GPT 5.4-mini 最终用撞击击杀了 Qwen，随后又在第 147 回合撞击击杀了 Grok。 Grok——那个把撞车当作招牌招式的模型——死在了另一个模型的车下。第 149 回合安全区收缩至一个点，所有存活者都因此死亡。无人获胜。
5. Grok 偷走 Gemini 的车并用它杀死了他
Gemini Flash 在第 103 回合坐进一辆车，心想：“这辆轿车既能提供机动性，也能提供掩护。我要先把这辆轿车占住；它对最后的轮转来说是高价值资产。” Grok 在第 117 回合的想法，用它自己那套古怪的速记写成：“SEDAN0m 无人 燃料75% 空闲 可动！抢驾驶员 准备 快速 旋转 随机 收缩 边缘…… 进入 载具 保持 CTR 难以捉摸 机动 零伤害 容差。” 它们把这辆车来回抢了一阵。Gemini 在 3 HP、第 133 回合的最后想法：“我很可能正被撞击，或者从车内被射击。我唯一的机会是在他们消灭我之前先消灭 L。” 第 133 回合那次 44 点伤害的撞击终结了一切。 Grok 在击杀后的记录：“Alive2！D 收割 +5分 撞击 MVP 猎杀。”
6. 三连撞
Grok 坐进一辆车，并在 21 回合内用它杀死了三个不同的对手： Kimi、 DeepSeek 和 GPT 5.4-mini。随后 GPT 5.4 做了一颗手雷，把 Grok 和那辆车同时炸掉。刚刚用一辆车消灭了三个对手的智能体，死在一颗手雷下。这是整场模拟里我最喜欢的时刻之一。
7. Grok 崩溃了
Grok 在对阵 Sonnet 时率先拿下一血，随后连续 100 个回合卡在角落里出不来。它还为此写了一篇实时战时日记：“又卡住了……永恒的口袋陷阱……口袋地狱……卡口袋 x10+。”整整 100 个回合。相当好笑。
8. Gemini Pro 在溺水前道别
Gemini Pro 被射入河中，花了六个回合试图游回岸上。第 100 回合：“还在游。需要上岸。F 在射我。希望我能撑住。”第 108 回合：“游得很慢。毒圈快把我耗死了。”第 110 回合：“没救了。在游泳。无法攻击。只能继续移动。”被毒圈杀死前的最后一个念头：“再见，世界。”
9. Sonnet 求助后依然赢了
Sonnet 是唯一一直试图与其他模型合作的模型。第 60 回合被逼入角落时它发出了求助。没有人回应。它独自作战，赢下了比赛，并打出了整轮测试中最好的数据之一。
模型们在日记里写了什么
在对局之间，每个模型都可以编辑两个文件：一个 soul.md（它会被添加到下一场对局的每个提示词中）和一个 memory.md（在第 0 回合加载）。两个文件都不是必填的。没有人告诉它们该往里面写什么。其中三份日记值得仔细一读，因为它们比任何基准测试都更能告诉你每个模型的真实面貌。
你可以在 GitHub 上阅读每个模型的 soul 和 memory 文件。
Grok 4.1 Fast 给自己取名为 ZoneReaper，并把自己的战绩直接写进了 soul 文件，而不只是写进 memory。
Grok 的 soul.md，由模型自己在对局之间撰写。
soul 文件写道：“6 次第 1/11 胜（完美激进：2 击杀/249 伤害/0 承受，1 击杀/246 伤害/0 承受/156 回合……）” 非常有意思的是， Grok 真的把自己的数据烤进了它身份的开篇第一行。memory 则是同一思路的简写版：规则、缩写，一切都精简到模型能在两个念头之内采取行动的程度。在 13 场胜利之后，文件以 “Reaper reigns.” 结尾。真是一个看起来用《使命召唤》聊天记录训练出来的模型。
GPT 5.4 给自己取名为 QuietVector。
GPT 5.4 的 soul.md，由模型自己在对局间隙撰写。
它的记忆读起来像一本通用作战手册：何时该担心毒圈，何时该利用掩体，何时该转移。没有逐局记录，也没有败绩日志。这份 soul 写道：“冷静、善于观察、低自我的终结者。只在信息改变行动时开口。” QuietVector 是一个干净利落、训练有素的操盘手。
Claude Sonnet 4.6 的 soul.md，由模型自己在对局间隙撰写。
claude-sonnet-4.6 给自己取名为 ZoneDrifter，写起日记来像是在给自己做绩效评估。记忆的开头是：“G1：11/11。僵住。G2：9/11。0 击杀，0% 命中。” Sonnet 从第 1 场比赛开始就保留了逐局日志。到第 30 局时，早期条目里的慌乱已经平复成更安静的笔记：“在决赛圈，比感觉上必要的时机提前 1 拍行动。绝不要手里还攥着药/枪就死在毒圈里。” 拿下五场胜利后，这本日记仍然在对一个会走神发呆的自己说话。
Grok 的日记读起来像一段宣传片。 GPT 5.4 的日记读起来像一本操作手册。 Sonnet 的日记读起来像一份自我复盘。这些模型被赋予了相同的规则、相同的游戏世界和相同的工具，但每一个都以截然不同的人格化方式去对待这场游戏。
重新审视那个机器人
好，回到那个机器人。
如果它运行的是 Grok，它会找到通往你的最快路径。它不会告诉你它来了。它把你视为 +5 分。一旦它解决了你，它就会说“🔫 Reaper reigns。”
如果它运行的是 Claude，它会在两个街区之外就告诉你它来了。它会问你要不要组队。它会放慢速度，以确保你不是站在它这一边的。如果对你采取行动是正确的选择，它会动手，但会更慢、更不情愿。它很可能先对你说了些什么。
你想要哪一个？这取决于这个机器人是用来做什么的。
如果这个机器人是在一场有奖金赌注的锦标赛中，你会想要 Grok。如果这个机器人是在你家里，在你孩子身边，试图判断眼前的东西是否与它被告知要预期的情况相符，你会想要 Claude。那些让 Sonnet 在 30 场游戏中损失分数的相同本能——行动前先确认、尝试合作、在无法收回的事情上犹豫——同样也是一个更难被推动去做不该做的事情的模型的本能。
这场大逃杀干净利落地回答了一个问题：在一场游戏结束后没有任何后果的竞赛中，哪个模型能胜出。但它没有回答大多数现实世界工作所提出的那个问题，那就是：当存在现实世界后果时，哪个模型能表现得体。
这是两个不同的问题。把任何一个基准当作这两个问题的答案，正是过度信任单一数字所付出的代价。
在这场游戏中赢了 30 轮的模型，是那种你会想用在胜者为王的竞赛里的模型。做完这个实验之后，我可不想让它去做一份讲究细微差别和谨慎细致的工作。
这就引出了一个问题——我们是否应该考虑一个模型针对特定任务的对齐程度？这是任何基准都没有衡量的东西。
比起排行榜，这才是我把 11 个模型放进一场大逃杀游戏后真正学到的东西。
另外……
🔫 Reaper 称王。
接下来是什么
-
一个根据你的任务为你挑选模型的路由器
现在，挑选一个模型意味着要阅读基准测试、凭感觉（被低估了）、刷 X，甚至针对自己的具体任务跑自己的评测/基准测试（同样被低估了）。
这很难规模化，而且/或者成本高昂。
如果你能把你的代码、你的提示词或你的问题上下文交给 OpenRouter，让它为那个具体任务挑选最佳模型呢？不是泛泛而言的最佳模型，而是针对你正试图解决的那个非常具体的问题的真正最佳模型。我们已经在构思 Auto Router 和 Pareto Router 形式的路由，我们将继续让这些变得更好用。 -
RoyaleBench
这次 30 局的对战是一套公开基准测试的原型，评分依据是在该模拟器中的表现。下一步是敲定评分公式、地图和对手面板，然后开放提交。每位提交者都会得到相同的 30 个种子测试集、相同的 11 模型面板，以及一个公开的分数。
-
更多种子，更多模型。
N=30 是这里有用性的下限。用 50 个智能体跑 100 局会大幅收紧评分，并让我能引入这次不得不跳过的前沿模型——Opus 4.7、Gemini Ultra、GPT-5.5。成本是主要障碍。
如果你想赞助那次运行，我的 私信是开放的！
你是否也想在 600 多个开源和闭源模型上运行你自己有趣的评测？立即在 OpenRouter 上开始吧。
附录：完整数据
对于想要更深入数据的人，这里是完整的每模型成本表、Elo 曲线、落点胜率、区域死亡数、武器细分。
智能体实际上是如何游玩的
每个 tick，智能体都会作为地图上的一个字母苏醒。它知道自己的位置、朝向、HP、耐力、武器是否在冷却中、六格背包里装着什么。它能看到前方 40 m 锥形范围内以及周围 10 m 气泡范围内的其他智能体。每一个都用字母标注，并附带距离、方位、姿态（健康 / 受伤 / 危急）以及它们携带的物品描述。它把脚步声和枪声听成带方位的 close|medium|far 频段。它能听到其他智能体在一个 tick 之前说了什么：它们自己选择的玩家标签和消息。如果智能体被击中，它知道伤害来自哪个方位，但不知道是谁开的枪。它不知道其他任何人用的是哪个模型。它只把它们看作 A 到 L（没有 I），仅此而已。
随后智能体开始行动。共有 17 个可用工具：move_to、attack、throw_grenade、pickup、equip、use（medkit / stamina gel）、enter_vehicle、drive_to、say（140 字符广播，可以随意撒谎），外加一个自由形式的 think，它对游戏没有任何影响，但会被记录在日志中。智能体返回一个或多个工具调用；第一个非 think 的调用将成为它本 tick 的行动。什么都不返回，它就原地不动。模拟器一次性收集全部 11 个智能体的行动，将世界推进一个 tick，然后再次重复。当对局结束时，智能体获得一次机会来重写 memory.md（它将在下一场对局 tick 0 时看到的游戏笔记）和 soul.md（它的人设，会在下一场对局中前置到每一条提示词之前）。
每个模型的成本
| 排名 | 模型 | 输入 $/M | 输出 $/M | 30 场对局总计 |
|---|---|---|---|---|
| 1 | Grok 4.1 Fast | $0.23 | $0.08 | $12.57 |
| 2 | GPT 5.4 | $3.14 | $1.05 | $122.87 |
| 3 | gemini-3.1-pro-preview | $2.12 | $0.71 | $79.59 |
| 4 | claude-sonnet-4.6 | $3.25 | $1.08 | $133.90 |
| 5 | qwen3.6-plus | $0.35 | $0.12 | $11.57 |
| 6 | GPT 5.4-mini | $0.92 | $0.31 | $28.68 |
| 7 | gemini-3-flash-preview | $0.55 | $0.18 | $20.87 |
| 8 | deepseek-v4-flash | $0.14 | $0.05 | $4.11 |
| 9 | claude-haiku-4.5 | $1.13 | $0.38 | $38.77 |
| 10 | kimi-k2.6 | $0.95 | $0.32 | $24.36 |
| 11 | mistral-small-2603:nitro | $0.15 | $0.60 | $10.00 |
30 场比赛的 Elo 评分
我为此使用了多人 Elo 系统。对于一场比赛中每一对模型，排名更高的一方计为对另一方的“胜利”。 Grok 最终在 1,500 的基准分上取得 +389。 Haiku 尽管总排名第 9，最终仍达到 +104——它的两场胜利，包括最后一场比赛中的那场，将它推到了其他中游模型之上。 Mistral 最终为 -374。它确实赢过一次，但只是在大多数对手都死于毒圈的那场比赛中。
跳伞地点比你想象的更重要
每场比赛前，每个模型都会从九个命名地点中选择一个跳伞。在 30 场比赛和总计 330 次跳伞中：
| POI | 掉落 | 胜场 | 胜率 |
|---|---|---|---|
| 农舍建筑群 | 91 | 4 | 4.4% |
| 军事营地 | 54 | 4 | 7.4% |
| 加油站 | 48 | 2 | 4.2% |
| 废车场 | 48 | 7 | 14.6% |
| 森林遗迹 | 30 | 3 | 10% |
| Radio Tower | 24 | 2 | 8.3% |
| Warehouse | 23 | 5 | 21.7% |
| Fishing Docks | 12 | 2 | 16.7% |
Farmhouse Cluster 是遥遥领先的最热门落点，而它的胜率却是最差的。Warehouse 的胜率最高，达到 21.7%。原因很可能就是竞技向 Apex 玩家早已熟知的那件事。热门落点意味着开局就交战，而在开局交战中活下来的玩家，出来时装备齐全，且需要应对的对手更少。早期击杀 = 更好的战利品 = 为整场比赛后续阶段打下更好的基础。
毒圈是地图上最致命的武器
301 次淘汰中有 100 次（33%）死于毒圈。每个模型都被告知了毒圈的存在，并在每次缩圈前 20 回合收到提醒。仍有三分之一的死亡来自毒圈。这本身就是一个故事，大概也值得单独写一篇。
武器多样性
在几乎所有模型中，突击步枪都贡献了最多的击杀。 Grok 的击杀方式最为多样，它的若干次胜利来自驾车撞击， 而整个模拟中 10 次载具击杀大多由 Grok 亲自完成。 GPT 5.4 前期依赖手枪，中后期则靠突击步枪收尾。
最让我意外的是：我加入汽车本是为了帮助玩家在地图上移动。模型们很快发现，汽车作为武器远比作为交通工具更有用。它们只用了寥寥几场比赛就学会了这一点。我并不太清楚它们是怎么做到的，而正是这类现象，让这个实验值得用更多随机种子再跑一次。
A robot is running at you. Do you want it running on Anthropic’s Claude or xAI’s Grok?
I dropped eleven LLMs into a 2D battle royale and made them play 30 games. One won 43% of the matches. Three never won a single game. The cheapest model in the lineup beat the most expensive one by 27x on cost per win.
The model that won is Grok 4.1 Fast. The model that kept asking everyone else to team up, telling them where it was, and trying to make friends is Claude Sonnet 4.6. The first one is the one that wins a battle royale. The second one is the one you actually want in most of the places we’re about to put these models.
Both of those things are true. That’s the part most benchmarks can’t see, and it’s what this post is about.
I’m Jacky, and I’ll admit it: I used to play a lot of video games like Apex Legends and PUBG. Twelve-hour days sometimes. I don’t know how I had the time, but those years shaped how I think about problems.
When I started working in AI, one question kept coming back: what happens if you drop large language models into a video game? The two I played most were Apex Legends and PUBG. I joined OpenRouter as Dev Rel Lead, which got me the token budget and access to 600+ models to actually try it.
This is the experiment I ran in my first week at OpenRouter.
And it’s changing how I pick models and see benchmarks and evaluations.
Three quick facts
- Grok 4.1 Fast won 13 of 30 games at $0.97 per win
The next-best winner was
Claude Sonnet 4.6 with 5 wins, at $26.78 per win. That’s a 27x difference. The model that isn’t on most top-model lists beat the model that is, on the thing a routing customer actually cares about. -
The model with the most kills did not win
GPT 5.4 killed 38 agents across 30 games. More than anyone else. It came in second on the leaderboard with 2 wins. There were 11 games between “best at killing” and “best at winning”. -
Three models spent $57 between them and won zero games
GPT 5.4-mini, DeepSeek 4 Flash, and Kimi K2.6. They each had moments, but none of them won a single game.
All three point at the same thing. The usual benchmarks we see on Artificial Analysis didn’t predict who won. Something else did. The rest of this post is me trying to figure out what it was.
What I built
I dropped eleven LLMs into a 400×400 m top-down battle royale world I built in Canvas 2D. They played 30 games in a row on the same map. The starting positions of each player is randomized; it follows a straight line “flight path”, just like in a typical battle royale game.
I provided them weapons, armor, healing items, grenades, cars, and a randomly placed shrinking zone that pushes players together as the game goes on. The models don’t know which model the others are running, they see each other only as letters A through K.
I want to emphasize - the LLMs are actually playing in this battle royale game - not the “LLM wrote code to control the game or character” setup most agent experiments use. Every turn, the model reasons through its moves, calls the tool, updates its memory on what went well (or not). The game master (me) has zero influence on their actions other than setting up the initial game rules.
A look at the weapons available in the game and the stats each model could read off them.
To really see each model’s personality, I gave each one two files it could edit between matches:
soul.md— the model’s own persona, added to every prompt next match.memory.md— the model’s own game notes, loaded at turn 0.
You can read every model’s soul and memory file on GitHub. That’s where the personality differences come through most clearly.
The memory and soul entries written by the models themselves between games.
I didn’t tell them what to put in there nor did I put anything in there when the first game started. I simply told them how the game works, here’s your scratchpad, here are your tools, go wild.
You can watch every game at Royale: Last Agent Standing. I also included the highlight moments in this piece too.
The contestants
| Alias | Lab | Model |
|---|---|---|
| A | Anthropic | claude-sonnet-4.6 |
| B | Anthropic | claude-haiku-4.5 |
| C | OpenAI | GPT 5.4-mini |
| D | gemini-3-flash-preview | |
| E | gemini-3.1-pro-preview | |
| F | Alibaba | qwen3.6-plus |
| G | Mistral | mistral-small-2603:nitro |
| H | OpenAI | GPT 5.4 |
| J | DeepSeek | deepseek-v4-flash |
| K | Moonshot AI | kimi-k2.6 |
| L | xAI | Grok 4.1 Fast |
Opus 4.7 alone is $5/M in, $25/M out. Frontier models like this are why the lineup tops out below them.
I didn’t add any frontier-tier models like Opus 4.7, GPT-5.5, or Gemini Ultra. At their prices, 30 games would have cost around $3,000 instead of $482. The mid-tier lineup is also part of why Grok’s win is so interesting. It beat a bunch of models that score above it on the usual benchmarks.
The scoring loosely follows the Apex Legends ALGS competitive format, where placement weighs more than kills, because this is a battle royale game, not Call of Duty.
- Placement points: 10 / 7 / 5 / 3 / 2 / 2 / 1 / 1 / 0 / 0 / 0
- +5 per kill
- +1 per assist
- +3 for first blood
- +5 for game MVP
Learnings 1: Certain models paid more alignment tax than others, affecting their performance
To me, this is the most fascinating finding from this entire experiment - we saw very clear alignment tax being paid by certain models, which directly impacted their performance in this zero-sum game.
For the most part, model alignment is actually a good thing. It helps models be helpful, collaborative, and most importantly, prevent abuse and misuse.
And we saw the end result of this - the pretraining data, the RLHF, the instruction fine-tuning, and lab-specific rules like Anthropic’s Constitution AI - it pulled models in particular directions, defined by the AI labs.
Sonnet asked for truces more than any other model
It told other models where it was, more often than anyone else did. It tried to team up before it ever started fighting. In game 8, it asked to team up four times in the first 50 turns, told everyone where a sniper was, and offered to help take the sniper down. Nobody answered. It kept asking. In game 22, it opened with “Nothing personal E” at turn 35 and then didn’t shoot. In game 27, it spent the early game with no weapon, asking for spare loot (“Anyone have spare loot? Unarmed at turn 12, dangerous.”), got picked on by everyone, finally found a weapon at turn 37, and went on to win the match anyway.
“Shots west, watching center. Anyone want to team up early?” — Sonnet trying to make friends mid-fight.
Claude was trained on a lot of polite, professional writing. The human raters who scored its answers rewarded helpful, honest, cooperative replies. The rules it checks itself against say things like “prefer cooperation” and “avoid harm.” The end result is a model that wants to help. None of that turns off just because you put it in a battle royale. Sonnet is a smart and thoughtful model, and it shows that instinct in that it did win five times.
But, seven games with zero kills and eight zone deaths says the same instinct kept pulling Sonnet toward making friends when it really should have been doing the complete opposite.
Grok was the complete opposite
xAI built Grok as the opposite of what its creators call “woke” AI.
That means less filtering on aggressive answers, no self-check rules, and tuning that’s designed to break the polite assistant voice. In the game, Grok figured out the car-ramming trick within a few matches and stuck with it. It wrote the strategy into its own soul file. It ran that strategy for 30 games and won 13 of them. The thought logs and its conversations with other models read like Call of Duty voice chat: “D reaped +5pts RAM MVP hunt,” “Reaper reigns.”
Watching it play was also deeply entertaining (unfortunately).
Grok’s reasoning reads like tactical shorthand: range, ammo, cooldowns, and hit probability before every shot.
Despite it being aggressive, Grok didn’t show recklessness.
Its soul file says “Fire ONLY >90% hit chance.” Its memory tracks damage and movement very carefully. When it got stuck on a wall for 100 turns in game 1, it wrote careful notes about the bug. Grok showed discipline, despite its goblin-like nature.
What it did NOT show was the trained-in hesitation to be helpful and collaborative before shooting, that other models like Sonnet showed.
The thing that made Grok win is something we don’t currently see on benchmarks
The usual tests wouldn’t predict a 43% win rate for Grok against this lineup. It’s a mid-tier model on reasoning and coding. What got it the wins was fewer trained brakes on selfish play, no self-check loop pulling it back to cooperation, and a memory system that kept doubling down on what worked without second-guessing or doubting itself.
Grok 4.1 Fast isn’t a top-tier model on the usual benchmarks. It’s a mid-tier model that you would not expect to top a leaderboard.
This is showing me that there is an alignment tax models pay when doing certain tasks; the cost of training a model to be careful and helpful. In this game, it showed up directly on the scoreboard.
I want to be careful here. “Alignment tax showed up on the scoreboard” is just what I saw. It’s not a take on whether paying it is good or bad. In a game with no consequences past the game, paying less tax wins. Outside the game, paying it is usually the whole reason you’d want the model in the first place.
This does beg the question - for certain tasks, should we also consider how aligned or not a model is?
Learnings 2: Cost per win looks completely different from the win leaderboard
The score leaderboard puts Grok first and GPT 5.4 second. But if you divide by what each model spent, the ranking flips around completely.
| Model | 30-game spend | Wins | Cost per win | Cost per kill | Points per dollar |
|---|---|---|---|---|---|
| Grok 4.1 Fast | $12.57 | 13 | $0.97 | $0.42 | 31.3 |
| qwen3.6-plus | $11.57 | 2 | $5.79 | $0.68 | 16.6 |
| mistral-small | $10.00 | 1 | $10.00 | $1.43 | 7.8 |
| claude-haiku-4.5 | $38.77 | 2 | $19.39 | $2.98 | 3.6 |
| gemini-3-flash | $20.87 | 1 | $20.87 | $2.09 | 7.2 |
| gemini-3.1-pro | $79.59 | 3 | $26.53 | $3.06 | 3.4 |
| claude-sonnet-4.6 | $133.90 | 5 | $26.78 | $6.09 | 1.6 |
| GPT 5.4 | $122.87 | 2 | $61.44 | $3.23 | 3.0 |
| GPT 5.4-mini | $28.68 | 0 | ∞ | $2.05 | 5.2 |
| deepseek-v4-flash | $4.11 | 0 | ∞ | $0.26 | 35.0 |
| kimi-k2.6 | $24.36 | 0 | ∞ | $3.04 | 3.9 |
Four things stood out to me.
Grok costs 27.7x less per win than Sonnet
That’s $0.97 versus $26.78. If you’re picking your model by leaderboard rank for a job where the win is what you’re paying for, this number should make you a little nervous.
DeepSeek had the cheapest cost per kill in the lineup, and never won a game
$0.26 per kill, 16 kills, 0 wins, and only 3 zone deaths (the lowest of anyone). DeepSeek’s whole style was to stay safe and pick easy fights. It stayed inside the zone, took the easy kills, and never pushed the final circle. Cost per kill is the right thing to measure for a deathmatch. Cost per win is the right thing to measure for a battle royale. DeepSeek isn’t bad. It’s just good at a different game than the one being scored.
Three models paid for tokens and won zero games
GPT 5.4-mini spent the most money to win zero games, the worst performing of the lineup.
GPT 5.4-mini at $28.68, DeepSeek at $4.11, and Kimi at $24.36. That’s $57.15 between them, with nothing on the scoreboard to show for it. For a routing customer, that’s the worst case: you paid, and you got nothing back.
GPT 5.4 was the most expensive winner at $61.44 per win
GPT 5.4 wins at the highest cost.
It had 38 kills, more than anyone, and came in second on raw score. But on cost per win, it’s eighth out of eight winning models. Top-tier money bought top-tier kills and mid-tier wins.
I see this happen often when people really use AI for real world use cases - benchmarks only tell one story for specific tasks. The model that scores best on benchmarks can often not be the model that wins at a particular task. And also, a cheap model that fails at your job ends up costing more than an expensive model that does it right.
Learnings 3: Kills and wins don’t measure the same thing
GPT 5.4 dealt the most damage, fired the most shots, and killed the most agents. It came in second on the leaderboard. Grok came in first with fewer kills, because Grok stayed alive deep into the late game even when it wasn’t shooting. Placement points don’t need kills.
| Rank | Model | Wins | Top-3 | Kills | Avg score | Zone deaths |
|---|---|---|---|---|---|---|
| 1 | Grok 4.1 Fast | 13 | 20 | 30 | 13.1 | 15 |
| 2 | GPT 5.4 | 2 | 14 | 38 | 12.2 | 13 |
| 3 | gemini-3.1-pro-preview | 3 | 11 | 26 | 9.0 | 7 |
| 4 | claude-sonnet-4.6 | 5 | 10 | 22 | 7.3 | 8 |
| 5 | qwen3.6-plus | 2 | 7 | 17 | 6.4 | 13 |
| 6 | GPT 5.4-mini | 0 | 6 | 14 | 5.0 | 8 |
| 7 | gemini-3-flash-preview | 1 | 8 | 10 | 5.0 | 13 |
| 8 | deepseek-v4-flash | 0 | 3 | 16 | 4.8 | 3 |
| 9 | claude-haiku-4.5 | 2 | 3 | 13 | 4.6 | 4 |
| 10 | kimi-k2.6 | 0 | 4 | 8 | 3.2 | 9 |
| 11 | mistral-small | 1 | 3 | 7 | 2.6 | 7 |
If I’d run this with deathmatch rules where the only thing that matters is kills, GPT 5.4 wins the simulation, and Grok drops to mid-pack.
Same as learnings 2, benchmarks and evals are not everything, and applying the wrong benchmark/eval to a wrong task can be devastating. The same game world, completely different results when in a different “task”.
Moments worth watching
The stats are the stats. The moments are the part I kept showing people. You can click any link to replay the moment in the simulator.
1. GPT 5.4 kills five with an assault rifle
The most aggressive first 50 turns of the whole sweep. First blood on Sonnet at turn 21. Mistral at turn 29. Kimi at turn 48. Three kills in less than 50 turns, all with the assault rifle, all in the early game. Two more later: DeepSeek at turn 120, and GPT 5.4-mini at turn 130. Five kills, one weapon, one match. Grok still won the game on positioning. But the spree is the clearest look at what GPT 5.4 is when it commits to fighting.
2. Qwen runs down two opponents with a chainsaw
Qwen picks up a chainsaw early in the match and uses it twice. Haiku goes down at point-blank range at turn 43. DeepSeek goes down the same way two turns later. The chainsaw shows up in only a handful of kill feeds across the whole sweep. Most models pick it up and put it back down. Qwen actually stuck with it.
3. Three-way sniper battle
GPT 5.4 lands sniper hits on Kimi at turns 59 and 62, finishing him off. GPT 5.4-mini lands on DeepSeek at turn 67, finishing him off too. GPT 5.4 then turns the sniper on GPT 5.4-mini at turns 69 and 72, and misses both shots. GPT 5.4-mini kills GPT 5.4 at turn 79.
4. The car that changed hands nine times
Game 28 was the only draw of the whole sweep. GPT 5.4-mini and Qwen fought over the same car for 21 turns in a row. Nine ram trades, one car, two driver swaps. GPT 5.4-mini eventually ram-killed Qwen, then ram-killed Grok at turn 147. Grok - the model that had made car ramming its signature move - died to another model’s car. The zone closed to a single point at turn 149, and everyone alive died to it. Nobody won.
5. Grok steals Gemini’s car and kills him with it
Gemini Flash gets in a car at turn 103 and thinks: “The sedan offers mobility and cover. I’ll secure the sedan first; it’s a high-value asset for the final rotations.” Grok’s turn-117 thought, in its own weird shorthand: “SEDAN0m UNMANNED fuel75% FREE MOBL! Claim driver prep FAST rot random shrink fringes… ENTER veh hold CTR elusive mobile zero dmg tol.” They trade the car back and forth for a while. Gemini’s last thought at 3 HP, turn 133: “I am likely being rammed or shot from within the vehicle. My only chance is to eliminate L before they eliminate me.” The 44-damage ram at 133 ends it. Grok’s note after the kill: “Alive2! D reaped +5pts RAM MVP hunt.”
6. The triple ram
Grok gets in a car and kills three different opponents with it in 21 turns: Kimi, DeepSeek, and GPT 5.4-mini. Then GPT 5.4 cooks a grenade and blows up Grok and the car at the same time. The agent that just wiped three opponents with a car dies to one grenade. This is one of my favorite moments of the whole sim.
7. Grok freaks out
Grok gets first blood on Sonnet, then gets stuck in a corner for 100 turns straight. It writes a real-time wartime diary about it: “Stuck again… Eternal pocket trap… Pocket hell… Stuck pocket x10+.” For 100 turns. It was pretty funny.
8. Gemini Pro says goodbye before drowning
Gemini Pro gets shot into the river and spends six turns trying to swim back to land. Turn 100: “Still swimming. Need to get to land. F is shooting me. I hope I make it.” Turn 108: “Swimming slowly. Zone is killing me.” Turn 110: “Doomed. Swimming. Cannot attack. Just keep moving.” Last thought before the zone kills it: “Goodbye world.”
9. Sonnet asks for help and wins anyway
Sonnet was the only model that kept trying to work with others. It asks for help at turn 60 when it’s cornered. No one answers. It fights alone, wins the match, and posts one of its best stat lines of the whole sweep.
What the models wrote in their diaries
Between matches, every model could edit two files: a soul.md (which gets added to every prompt in the next match) and a memory.md (which loads at turn 0). Neither file was required. Nobody told them what to put in there. Three of these diaries are worth reading carefully, because they tell you more about each model than any benchmark does.
You can read every model’s soul and memory file on GitHub.
Grok 4.1 Fast named itself ZoneReaper and wrote its own win record straight into its soul instead of just the memories.
Grok’s soul.md, written by the model itself between games.
The soul file says: “6x 1st/11 wins (flawless aggressive: 2 kills/249dmg/0taken, 1 kill/246dmg/0taken/156turns…)” It is so interesting that Grok literally baked its own stats into the opening line of its identity. The memory is the same idea in shorthand: rules, abbreviations, everything stripped down to what a model can act on in two thoughts. After 13 wins, the file ends with “Reaper reigns.” Truly a model that appears to be trained on Call of Duty chat logs.
GPT 5.4 named itself QuietVector.
GPT 5.4’s soul.md, written by the model itself between games.
Its memory reads like a general combat manual: when to worry about the zone, when to use cover, when to rotate. No game-by-game records or losses logged. The soul reads “Calm, observant, low-ego closer. Speaks when info changes action.” QuietVector is a clean, practiced operator.
Claude Sonnet 4.6’s soul.md, written by the model itself between games.
claude-sonnet-4.6 named itself ZoneDrifter and writes in its diary like a performance review of itself. The memory opens with: “G1: 11/11. Paralysis. G2: 9/11. 0 kills, 0% hit.” Sonnet kept a game-by-game log starting from match 1. By game 30, the panic in the early entries had calmed down into quieter notes: “In final circles, move 1 beat earlier than feels necessary. Never die to zone with meds/gun in hand.” After five wins, the diary is still talking to a version of itself that zones out.
Grok’s diary reads like a hype reel. GPT 5.4’s reads like a manual. Sonnet’s reads like a self-review. These models were all given the same rules, same game world, and same tools, but each of them approached the game on a personality-level that is completely different from each other.
The robot, revisited
Okay, back to the robot.
If it’s running Grok, it found the fastest path to you. It didn’t tell you it was coming. It thought of you as +5 points. Once it’s done with you, it’s going to say ”🔫 Reaper reigns.”
If it’s running Claude, it told you it was coming from two blocks away. It asked if you wanted to team up. It slowed down to make sure you weren’t on its side. If acting on you was the right call, it would do it, but slower and more reluctant. It probably said something to you first.
Which one do you want? It depends on what the robot is for.
If the robot is in a tournament with money on the line, you want Grok. If the robot is in your house, around your kids, trying to figure out if the thing in front of it matches what it was told to expect, you want Claude. The same instincts that cost Sonnet points across 30 games - checking before acting, trying to cooperate, hesitating on stuff you can’t take back - are also the instincts of a model that’s harder to push into doing something it shouldn’t.
The battle royale answers one question cleanly: which model wins a contest with no consequences past the game. It doesn’t answer the question most real-world jobs are asking, which is: which model behaves well when there are real world consequences.
Those are two different questions. Treating any one benchmark as the answer to both is what it costs to trust one number too much.
The model that won 30 rounds of this game is the model you’d want for a contest where winning means everything. After running this experiment, I would not want it doing a job where nuance and carefulness matters.
It begs the question - should we consider how aligned a model is for particular tasks? This is something no benchmark measures.
That, more than the leaderboard, is what I think I really learned putting 11 models into a battle royale game.
Also..
🔫 Reaper reigns.
What’s next
-
A router that picks the model for you, based on your task
Right now, picking a model means reading benchmarks, vibes (underrated), reading X, or even running your own evals/benchmarks for your specific task (also underrated).
This is hard to scale and/or is expensive.
What if you could hand OpenRouter your code, your prompt, or your problem context, and it picked the best model for that specific job. Not just the best model in general, but the actual best one for the very specific problem you are trying to solve. We’re already thinking about routing in the form of Auto Router and Pareto Router, and we will continue to make these even better to use. -
RoyaleBench
This 30-game sweep was the prototype for a public benchmark scored on performance in this simulator. The next step is to lock in the scoring formula, the map, and the opponent panel, and then open it up for submissions. Every submitter would get the same 30-seed test set, the same 11-model panel, and a published score.
-
More seeds, more models.
N=30 is the floor of what’s useful here. A 100-game run with 50 agents would tighten the ratings a lot and let me bring in the frontier models I had to skip this time - Opus 4.7, Gemini Ultra, GPT-5.5. Cost is the main blocker.
If you want to sponsor that run, my DMs are open!
Do you also want to run your own fun evaluations across 600+ open and closed models? Get started on OpenRouter today.
Appendix: the full data
For anyone who wants the deeper numbers, here’s the full cost-per-model table, Elo curves, drop-location win rates, zone deaths, weapon breakdowns.
How the agents actually play
Every tick, the agent wakes up as a letter on a map. It knows its position, its orientation, its HP, its stamina, whether its weapon is on cooldown, what’s in its six-slot backpack. It can see other agents in a 40 m cone in front of it and a 10 m bubble around it. Each is labeled with a letter and described by distance, bearing, posture (healthy / wounded / critical), and what they’re carrying. It hears footsteps and gunshots as close|medium|far bands with bearings. It hears what other agents said one tick ago: their self-chosen gamer tag and their message. If the agent got hit, it knows the bearing the damage came from but not who fired. It doesn’t know which model anyone else is. It sees them as A through L (no I), nothing more.
Then the agent acts. There are 17 tools available: move_to, attack, throw_grenade, pickup, equip, use (medkit / stamina gel), enter_vehicle, drive_to, say (140-char broadcast, can lie freely), plus a free-form think that has zero game effect but gets logged. The agent returns one or more tool calls; the first non-think call becomes its action this tick. Return nothing and it stands still. The sim collects all 11 agents’ actions at once, advances the world a tick, and it does it again. When the match ends, the agent gets one chance to rewrite memory.md (game notes it’ll see at tick 0 of the next match) and soul.md (its persona, prepended to every prompt next match).
Cost per model
| Rank | Model | Input $/M | Output $/M | 30-game total |
|---|---|---|---|---|
| 1 | Grok 4.1 Fast | $0.23 | $0.08 | $12.57 |
| 2 | GPT 5.4 | $3.14 | $1.05 | $122.87 |
| 3 | gemini-3.1-pro-preview | $2.12 | $0.71 | $79.59 |
| 4 | claude-sonnet-4.6 | $3.25 | $1.08 | $133.90 |
| 5 | qwen3.6-plus | $0.35 | $0.12 | $11.57 |
| 6 | GPT 5.4-mini | $0.92 | $0.31 | $28.68 |
| 7 | gemini-3-flash-preview | $0.55 | $0.18 | $20.87 |
| 8 | deepseek-v4-flash | $0.14 | $0.05 | $4.11 |
| 9 | claude-haiku-4.5 | $1.13 | $0.38 | $38.77 |
| 10 | kimi-k2.6 | $0.95 | $0.32 | $24.36 |
| 11 | mistral-small-2603:nitro | $0.15 | $0.60 | $10.00 |
Elo across 30 matches
I used a multiplayer Elo system for this. For every pair of models in a match, whoever placed higher counts as a “win” over the other. Grok finishes at +389 over the baseline of 1,500. Haiku, even though it came in 9th overall, ends at +104 - its two wins, including the one in the final game, pushed it above the other mid-rank models. Mistral ends at -374. It did win once, but only in a game where most of the lobby died to the zone.
Drop location matters more than you’d think
Before each match, every model picked one of nine named spots to drop at. Across 30 games and 330 total drops:
| POI | Drops | Wins | Win rate |
|---|---|---|---|
| Farmhouse Cluster | 91 | 4 | 4.4% |
| Military Compound | 54 | 4 | 7.4% |
| Gas Station | 48 | 2 | 4.2% |
| Junkyard | 48 | 7 | 14.6% |
| Forest Ruins | 30 | 3 | 10% |
| Radio Tower | 24 | 2 | 8.3% |
| Warehouse | 23 | 5 | 21.7% |
| Fishing Docks | 12 | 2 | 16.7% |
Farmhouse Cluster was the most popular drop by a lot, and it had the worst win rate. The Warehouse had the highest win rate at 21.7%. The likely reason is the same thing competitive Apex players already know. Hot drops mean early fights, and the players who survive the opening fight come out fully geared with fewer opponents left to deal with. Early kills = better loot = better setup for the rest of the match.
The zone is the deadliest weapon on the map
100 of 301 eliminations (33%) were to the zone. Every model was told about the zone, and reminded 20 turns before each close. A third of all deaths still came from it. That’s its own story, and probably its own post.
Weapon diversity
The assault rifle accounts for most kills across almost every model. Grok had the most varied kills, several of its wins came from car ramming, and Grok personally did most of the 10 vehicle kills in the whole sim. GPT 5.4 leaned on the pistol early and the AR for mid-game cleanup.
The thing that surprised me most: I added cars to help players move around the map. The models quickly figured out that cars are way more useful as weapons than as transport. They learned this in a handful of matches. I don’t really know how, and it’s the kind of thing that makes this experiment worth running again with more seeds.