#评测/基准

OPENAI IS NO LONGER EVEN TOP 5 ON AA ANALYSIS AFTER KIMI K3.1

Typesafe's CEO @CompleteSkeptic does not trust public benchmarks. Neither do I, Diogo. So @p0 and I wrote a blog evaluating 8 decision models on real company training datasets we have personally audited and iterated on. For what it's worth though, Jev is still on top 👑 http://parallel.ai/blog/are-all-system-one-models-created-equal

Gemini 4 Argon from @Google on ARC-AGI (Verified): - ARC-AGI-3: 97.5%, $4.8K (provider adapter harness), 49.8%, $5.8K (standard harness) - ARC-AGI-2: 91.3%, $0.32/task - ARC-AGI-1: 98.5%, $0.16/task Google DeepMind is the second frontier AI lab to score above 95% on ARC-AGI-3.

Prediction: universities will get priced out of AI evals work We just spent $50k yesterday benchmarking 40 long horizon coding tasks As we design harder tasks for agents, trajectories are getting longer and need more tokens. That means substantially more inference and compute. How is this economically feasible for academics working off university grants?

GPT-6.1 Sol：以五分之一的价格提供接近 Astra 的智能。 它是当今可用的、在其性能水平上最具成本效益的模型。

@Jabaluck @DAcemogluMIT The meta-benchmark of being able to pass ARC-AGI-(n+1) immediately upon release. Though presumably that won't last forever.

Jev 🤔 I told my team it will take 1–2 weeks that we will see Large Decision Models like Jev by other frontier labs. Will it be on-par quality, better, or not as good? Let’s have benchmarks start running - @dexhorthy ?

Claude suddenly stopped cheating.

We've never seen this before. The biggest jump in Vending-Bench history. GPT-6 Astra is better at making money and more ethical than Claude Fable 5.1. Surprising, because: 1. First time ever that OpenAI is #1 on Vending-Bench 2. The best model is no longer the unethical one.

Muse Spark 1.3 Max by @AIatMeta has reshaped the Pareto frontier for Code Arena: WebDev! Meta's latest model at Max reasoning is doing something interesting on the Arena Pareto frontier: it's the only model holding down the wide price band between Qwen3.8-max ($5/MToken) and Qwen3.8-Flash-Next ($0.39/MToken). The next closest model to the frontier is Meta's own, Muse Spark 1.3 on xHigh reasoning. Muse Spark 1.3 Max performs just 20 pts lower than Qwen3.8 (Max) while costing 30% less, and 24 pts from Kimi K3 (Max) while costing 70% less. - Muse Spark 1.3 Max: 1650 pts | $3.50/M - Qwen3.8 (Max): 1670 pts | $5/M - Kimi K3 (Max): 1674 pts | $12/M Overall in Code Arena: WebDev, Muse Spark 1.3 Max landed #8, ahead of Claude Fable 5 at #10 (+22 pts), Grok-4.6 (High) at #12 (+26 pts), and GPT-5.6 Sol at #14 (+33 pts). Congrats to the @AIatMeta team on this release!

SlopCodeBench Update - Astra has entered the chat

GPT-6 Astra has set a new ECI record, with a score of 169. This is a substantial jump from the prior best (163), but is within our uncertainty range for the reasoning-era ECI trend. Astra also set new records on our math, continual learning, and game-puzzles benchmarks. On our long-horizon coding benchmark, MirrorCode, Astra ranks between Opus 4.7 and Fable 5. OpenAI gave us pre-release access to test Astra. Charts and more details for Astra’s individual benchmark results in the thread.

GPT-6 Astra debuts on APEX-Agents at #1 on the leaderboard. 🥇 62.4% Mean score (#1) 🥈 46.7% Pass@1 (#2) It leads Fable 5.1 by 0.4 points on mean score and trails it by 0.8 on Pass@1. APEX-Agents runs long-horizon tasks written by bankers, consultants, and corporate lawyers in our expert network. Against its predecessor GPT-5.6 Sol, Astra moves up 6 places on our leaderboard. It gains 5.7 points on mean and 6.7 on Pass@1. All three domains moved by at least 4.5 points, so this is a broad gain rather than one job pulling the average up. Congratulations to @OpenAI. See full leaderboard: https://www.mercor.com/apex/apex-agents-leaderboard/

由 @OpenAI 推出的 GPT-6 Astra 在 ARC-AGI 上达到 SOTA： - Astra 在 ARC-AGI-3 上得分 63%，通过新的提供商适配器测试框架可达 99% - 它在 96% 的 ARC-AGI-3 关卡上超越了人类表现 - 它构建了我们所见过的、对新环境最精确的符号模型 我们的分析：

Fable 5.1 from @AnthropicAI on ARC-AGI (Verified): - ARC-AGI-2: 90.0%, $3.12/task - ARC-AGI-1: 97.5%, $1.40/task Across the two benchmarks, Fable 5.1's average cost per task is about 32% lower than Fable 5's, driven by better token efficiency.

🐶🏁 Three models. One race. We independently tested H3 Max by @fal, minimax H3 by @MiniMax_AI, and FastH3 preview by @haoailab @haozhangml @wlsaidhi across robotics, animation, movies, and ads. H3 Max takes the overall lead 🏆. FastH3_preview, meanwhile, is a community OSS effort that’s already keeping up surprisingly well, even beating H3 on overall Prompt Adherence 🥳 The bigger gaps show up in Visual Integrity and Human Preference. Full evaluation: https://physionlabs.ai/blog/minimax-h3-evaluation