#评测/基准

Introducing Gemini 3.5: our newest family of models combining frontier intelligence with real-world action. The first release is 3.5 Flash, our strongest model yet for agents and coding 🧵

Gemini 3.5 Flash ARC-AGI (Verified) ARC-AGI-2: - High: 72.1%, $0.85 - Minimal: 8.9%, $0.11 ARC-AGI-1: - High: 92.5%, $0.42 - Minimal: 48.8%, $0.06 Gemini 3.5 Flash is on par with GPT-5.5 (Medium) on ARC-AGI

Qwen3.7 Preview By @Alibaba_Qwen lands on Arena for Text and Vision. In Text Arena, Qwen3.7 Max Preview ranks #13 overall. Alibaba is now the #6 lab in this arena. - #7 Math - #9 Expert - #9 Software & IT - #10 Coding In Vision Arena: Qwen3.7 Plus Preview ranks #16 overall, making Alibaba the #5 lab. Congrats to the @Alibaba_Qwen team on the latest progress!

In the Vision Arena, Qwen3.7 Plus Preview makes @Alibaba_Qwen the #5 lab, ranking #16 overall.

Hermes Agent vs OpenClaw using Qwen 35B Local Model We asked agents to scrape GitHub star history for both tools, find what caused the growth spikes, build a live dashboard in the browser. MacBook Pro M5 Max 64Gb OpenClaw: 203k tokens, 12m 01s - wrote a bash script Hermes: 257k tokens, 33m 01s - wrote a SKILL.md OpenClaw hit GitHub API, got truncated responses, paginated through contributors, pulled star-history JSON, found a security incident in OpenClaw's history, fetched SVGs, fixed broken HTML from trimming, rewrote it clean. Hermes parallel tool calls across GitHub API, web search, and browser. Hit Google rate limit, auto-switched to DuckDuckGo. Fetched article contents, mapped viral moments, then built the dashboard. Both shipped a live dashboard with star growth charts and spike annotations

突发：MiMo V2.5 Pro (Thinking) 在 Design Arena 的开源权重模型中综合排名第 3。 MiMo V2.5 Pro (Thinking) 在综合排行榜上比 MiMo-V2.5 高出 8 个名次，在前端编码任务上与 Claude Sonnet 4.6 处于同一性能区间。 热烈祝贺 @XiaomiMiMo 团队取得这些进步！

突发：MiMo V2.5 Pro (Thinking) 在 Design Arena 的开源权重模型中综合排名第 3。 MiMo V2.5 Pro (Thinking) 在综合排行榜上比 MiMo-V2.5 高出 8 个名次，在前端编码任务上与 Claude Sonnet 4.6 处于同一性能区间。 热烈祝贺 @XiaomiMiMo 团队取得这些进步！

Announcing agentic performance benchmarking for Speech to Speech models on Artificial Analysis. We use 𝜏-Voice to measure tool calling and customer interaction voice agent capabilities in realistic customer service scenarios Even the strongest Speech to Speech (S2S) models today resolve only about half of realistic customer service scenarios end-to-end - a meaningful gap relative to frontier text-based agents on the same tasks. Voice channels introduce significant complexity: challenging accents, background noise, and packet loss, all while requiring fast responses, consistency across long multi-turn conversations, and reliable tool use. Performance also varies considerably by audio condition: in clean audio some models perform notably better, but realistic conditions continue to pose a challenge. Conversation duration also varies meaningfully across models, with implications for both customer experience and operational cost. About 𝜏-Voice: Our Agentic Performance benchmark is based on 𝜏-Voice (Ray, Dhandhania, Barres & Narasimhan, 2026), which extends 𝜏²-bench into the voice modality to evaluate S2S models on realistic customer service tasks. It measures multi-turn instruction following, support of a simulated customer through a complete interaction, and tool use against simulated customer service systems. The simulated user combines an LLM-driven decision model with realistic audio synthesis: diverse accents, background noise, and packet loss modelled on real network conditions. This complements our Big Bench Audio benchmark measuring intelligence and Conversational Dynamics (Full Duplex Bench subset) benchmark measuring conversational naturalness. Scores are the average of three independent pass@1 trials. We evaluate under realistic audio conditions using the 𝜏²-bench base task split across three domains: ➤ Airline (50 scenarios): e.g., changing a flight, rebooking under policy constraints ➤ Retail (114 scenarios): e.g., disputing a charge, processing a return ➤ Telecom (114 scenarios): e.g., resolving a billing issue, troubleshooting a service problem Task success is determined by deterministic checks against expected actions and final database state, consistent with the 𝜏²-bench evaluator. Key results: xAI's Grok Voice Think Fast 1.0 is the clear leader at 52.1%, averaging 5.6 minutes per conversation, the second-longest overall. OpenAI's GPT-Realtime-2 (High) (39.8%, 3.0 min) and GPT-Realtime-1.5 (38.8%, 4.8 min) follow, with Gemini 3.1 Flash Live Preview - High close behind at 37.7% (3.8 min). Speech to Speech is a fast evolving modality and we expect movement in rankings as we continue to add new models with these capabilities, and model robustness improves. Congratulations @xAI @elonmusk! See below for further detail ⬇️

🎤 Keynote announcement: @percyliang (Percy Liang), Professor of Computer Science at @Stanford, founding director of the Center for Research on Foundation Models, and co-founder of @togethercompute, is keynoting #CAIS2026. Percy's HELM framework set the standard for holistic evaluation of language models, and his Foundation Model Transparency Index (now in its third year) put every major AI lab on notice for what they do and don't disclose. His current work on Marin takes this further: an open lab where every experiment, successful or not, is public from day one. This one's going to be good. San Jose · May 26–29 https://caisconf.org

The first ProgramBench task was just solved by GPT 5.5 high/xhigh. Interestingly, high/xhigh picked two different languages for the task (C vs Python). GPT 5.5 xhigh was significantly better than Opus 4.7 xhigh in all metrics. 🧵

We are conducting an AI-assisted review of FrontierMath: Tiers 1-4. This has flagged fatal errors in about a third of problems, and we believe most of these flags to be valid. We will release updated scores on a corrected dataset after completing a thorough human review.

First update to PACT, my head-to-head LLM negotiation benchmark! 20-round buyer-seller bargaining game: each round the AIs can message, the buyer submits a bid and the seller submits an ask. If bid ≥ ask, trade clears at the midpoint. Thousands of matchups! GPT-5.5 is #1

@Miles_Brundage I am actually extremely excited about robotics, but have not been able to figure out whether the major leap that makes robots useful economically at scale is most likely 1 year, 3 years, 5 years or 10 years away.

StepFun’s new StepAudio 2.5 TTS ranks #3 on the Artificial Analysis Speech Arena Leaderboard, only behind Inworld’s Realtime TTS 1.5 Max and Google’s Gemini 3.1 Flash TTS StepAudio 2.5 TTS represents a significant step forward for StepFun from previous TTS models, with notably increased naturalness of speech samples. The model now edges out Eleven v3 on our current prompt set with an Elo score of 1,187. Key takeaways: ➤ Quality: StepAudio 2.5 TTS has an Elo of 1,187 based on 834 arena appearances, placing it 28 points behind the leading model (Inworld TTS 1.5 Max at 1,215) and 8 points ahead of Eleven v3 at 1,179 ➤ Pricing: Model is priced at $85/1M characters, a premium to leading frontier models, Inworld TTS 1.5 Max at $35/1M and Gemini 3.1 Flash TTS at $36.6/1M ➤ Speed: Model generates characters 37.6 characters per second, compared to 220.5 chars/s for Inworld TTS 1.5 Max and 30.1 chars/s for Gemini 3.1 Flash TTS ➤ Prompting: StepAudio 2.5 TTS offers two paths to control delivery of speech: 1. Global context prompt for overall style, 2. Inline contextual tags for more granular emotion and prosody See more details and listen to samples below ⬇️

Hermes Agent 现已在全球 @OpenRouter token 排行榜上排名第一。 虽然我们共同的旅程才刚刚开始，但我们想借此机会感谢我们的贡献者、支持者和用户，感谢他们为让我们走到今天所做的一切。