跳到正文
Artificial Analysis· @ArtificialAnlys · X·AI 评分39

Artificial Analysis 编码智能体指数新增安全拒答与回退模型视图

AI 导读

Artificial Analysis 的 Coding Agent Index 新增安全拒答报告,可查看拒答发生在任务提示词阶段还是任务中途,以及拒答后智能体切换到了哪个回退模型。

正文 · 原文

Safety refusal reporting in the Artificial Analysis Coding Agent Index now shows when refusals occur and which models are used as fallbacks

We've added two ways to explore the results:

  • Refusal timing: see whether a refusal occurred from the task prompt alone or later in the task, after the agent had already started working
  • Fallback model: see which model the agent switched to after a refusal, alongside attempts that were blocked

Claude Code with Sonnet 5.5 (max) currently ranks first in the Index. Its safety refusal rate is 4.5%, roughly half the 8.9% recorded for Claude Code with Opus 5.5 (max).

Around 94% of Sonnet 5.5's refusals occurred after the first turn. Following a refusal, the agent almost always switched to Opus 4.8.

Observed fallback patterns vary across model and agent configurations. With Fable 5.1, Claude Code fell back predominantly to Opus 4.8, while Opus 5 accounted for a much larger share of Devin Fusion's fallbacks in the Index.

Both views are available for the overall Index and each benchmark. Safety and fallback behavior is provider-configured and may change over time; these rates reflect behavior recorded at the time of benchmarking.

来源:Artificial Analysis · x.com

事件后续 · 1

查看事件全部