#评测/基准

We measure the gap using the Epoch Capabilities Index, our aggregate measure of model capability. Compared to our last analysis, the gap has widened slightly — we previously estimated an average gap of three months between January 2023 and October 2025.

I gave the same Go MMRPG backend spec to two coding agents and then asked Claude to judge the performance. CC (with Sonnet 4.6) 4.6 shipped 1,409 lines of clean, readable code. CyOps with @MiniMax_AI M2.7 shipped 5,271 lines of denser, layered code.

Thrilled to welcome Step 3.7 Flash landing on ModelScope, a 198B sparse MoE VLM from @StepFun_ai 🔥🤖 https://modelscope.cn/collections/stepfun-ai/Step-37-Flash 198B total, 11B active per token, up to 400 tok/s. 256K context. Three reasoning levels (low/medium/high) to balance speed, cost, and depth. 🥇 #1 on ClawEval-1.1 (67.1), leading Kimi K2.6 by 9.3pts 🥈 #2 on SWE-bench Pro (56.3) 🥇 #1 on SimpleVQA Search (79.2) 🖼️ Multimodal: 196B language backbone + 1.8B vision encoder. Parses dense UI, charts, and financial reports natively. 📄 Apache 2.0. vLLM / SGLang / llama.cpp / Transformers supported. Hermes Agent / OpenClaw / Kilo Code compatible. Runs locally on Mac Studio or DGX Station with 128GB+ memory.

http://x.com/i/article/2053566242338795520

ANTHROPIC 🔥: CLAUDE OPUS 4.8 IS ROLLING OUT TO ALL USERS. The release also includes an updated Thinking effort selector with Low, Medium, High, Extra, and Max options available. > Switch to Opus 4.8 for your most ambitious work - and now you can set the effort level for thoroughness or speed.

Artificial Analysis and IBM Research are launching ITBench-AA, the first in a new series of benchmarks evaluating models on agentic enterprise IT tasks, starting with Site Reliability Engineering tasks where frontier models score below 50% ITBench-AA’s SRE tasks benchmark model performance on Kubernetes incident response, where models must diagnose live systems by reading logs, tracing dependencies, and identifying root-cause entities across complex infrastructure. The underlying ITBench dataset has been developed by @IBM's Software Innovation Lab, leveraging IBM’s deep expertise in enterprise IT operations Artificial Analysis has worked closely with IBM over the last 6 months to develop a implementation of the dataset for frontier AI evaluation, beginning with Site Reliability Engineering (SRE) and expanding to Financial Operations (FinOps) and Chief Information Security Officer (CISO) tasks over time ITBench-AA SRE overview: ➤ 59 SRE tasks in total: 40 public tasks and 19 brand new, held-out tasks ➤ Each task provides a Kubernetes incident snapshot containing alerts, events, traces, metrics, logs, and application topology. The model must identify the minimal set of independent root-cause Kubernetes entities responsible for the incident ➤ Faults span typical SRE failure modes including infrastructure, service, application, and chaos-injected incidents, such as resource quota exhaustion, rollout failures, connection pool exhaustion, and network partitions Methodology details: ➤ Agentic harness: each task is solved by the model running in our open-source Stirrup reference harness, with shell access to a sandboxed file system containing the relevant logs and snapshots. 100-turn cap per task, 3 repeats per task ➤ Models submit a list of root-cause entities (Kubernetes Deployments, Services, Pods, etc.) they believe caused the incident. Each submission is compared against a ground-truth set of root causes provided by IBM Research ➤ Scoring uses average precision at full recall: if a model misses any of the ground-truth root causes, it scores 0.0 for that repeat. If it identifies all of them, it is awarded a score equal to its precision - the share of its submitted entities that are actual root causes, i.e. true positives / (true positives + false positives). The headline score is the average across 59 tasks × 3 repeats. ➤ The harness (Stirrup) is held constant across all evaluated models, allowing an apples-to-apples comparison between models. Key findings: ➤ Claude Opus 4.7 (Adaptive Reasoning, Max Effort) leads at 47%, followed by GPT-5.5 (xhigh) at 46% and Qwen3.7 Max at 42% ➤ All frontier models score below 50%, making ITBench-AA SRE one of the least saturated agentic benchmarks in our suite. For context, frontier models score considerably higher on Terminal-Bench ➤ Turn counts vary nearly 3x and longer trajectories do not translate to higher accuracy. GPT-5.5 (xhigh) averages 31 turns per task at 46%, while Gemini 3.1 Pro Preview averages 83 turns at 30%. Models that over-investigate tend to surface upstream fault-injection mechanisms or co-occurring symptoms as false positives ➤ GLM-5.1 (Reasoning) leads open weights models at 40%, effectively tied with Gemini 3.5 Flash (high). DeepSeek V4 Pro (Reasoning, Max Effort) follows at 38%, with Gemma 4 31B (Reasoning) at 37%, ahead of Gemini 3.1 Pro Preview at 30%

Many developers have suspected for months that GPT-5.5 outperforms Claude Sonnet for coding. But SWE-Bench reported near-parity, and it made people question what they’d been seeing in practice. DeepSWE aligns more closely with that day-to-day experience: GPT-5.5 scores 70% versus Claude Sonnet at 32%. That difference is substantial. DeepSWE focuses on what tends to matter in real workflows: whether an agent can take a short behavioral prompt, locate the correct area of the codebase, and implement the change cleanly - without needing you to enumerate files, modules, and functions. SWE-Bench often fails to capture that, due to dataset contamination and weaker verification. https://deepswe.datacurve.ai/blog

http://x.com/i/article/2059284537503285248

Today we’re releasing DeepSWE, a new standard for agentic coding benchmarks. On public leaderboards, top models often look relatively close in capability. DeepSWE shows where they actually diverge, reflecting the realistic experience of developers in their day-to-day work.

Qwen3.7 Max (20250517) 在 Code Arena: Frontend 中首次亮相便位列第 4——是榜单上排名最高的中国实验室，超越了 GLM-5.1，如今在智能体网页开发任务上已与 Claude Opus 4.6 不相上下。 热烈祝贺 @Alibaba_Qwen 取得这一成就！

I'm 100% Codex pilled now Been using Codex and Claude Code side by side hours a day for 2 months straight No longer using them side by side. Codex has become incredible What did it for me is the self testing. Every change it makes it self tests in it's own browser I went from about 40% of my changes being buggy on first go to at most 3% maybe? So much more reliable and allows me to get in an awesome flow state Listen, Claude can literally drop an update tomorrow that changes all of this, but for now I'm really blown away by Codex Do yourself a favor and don't have loyalty to any company. Use every tool. Use whatever is the best at the moment. Switch whenever they're no longer the best. No point in tribalism But at the moment I'm REALLY enjoying my time with Codex

腾讯的 Marvis 今天手痒我真的去试了试 怎么说呢？确实是我手痒了，这手得剁 😂 整个软件，就这个 Agent 模拟办公室的 dashboard 小动画也有些意思，Marvis 这个项目经理，收到任务后，会小跑到需要调用的 Agent 面前窃窃私语一番，这个 Agent 开始干活后，Marvis 继续回到工位假装很忙的盯着进度；更有意思的是，没活儿干的 Agent 会摸鱼玩游戏 😄 腾讯的产研部门，这是按照自己部门的工作状态做的吗？永远只有几个人真的在忙、项目经理永远假装在忙和在各种催进度、大家摸鱼的功夫百花齐放，公司眼中，最关注的永远是 Token（人力）成本。。 最后说回 Agent 执行结果，算了不说了，一言难尽

Qwen 3.7-max beats Opus 4.7 and GPT-5.5 We tested three frontier models on a real agentic task: write a Tetris bot that plays the game and trains itself. Each model could read its own code, run benchmarks, and rewrite itself across 10 iterations. Then we compared the final bots head to head. Qwen 3.7-Max: training cost $1.32, bot improvement +56% Claude Opus 4.7: training cost $12.15, bot improvement +28% GPT-5.5: training cost $2.85, bot improvement +7% Qwen won on every dimension - biggest jump, 9× cheaper than Claude, 2× cheaper than GPT. Long agentic loops is where Qwen Max actually delivers.