Artificial Analysis 发布 FrontierCode 1.1 编程评测：Claude Opus 5.5 以 52% 得分居首，领先模型失败率仍超 40%

Overview

FrontierCode is a coding benchmark from Cognition, built with maintainers of open source repositories. Each task asks an agent to make a change to a real open source repository, and the benchmark measures whether the change is good enough for a maintainer of that repository to merge.

FrontierCode grades more than whether tests pass. Unlike other coding evaluations, models are also assessed on behaviors like making excessive changes and large diffs, modifying files outside the relevant scope, or adding unhelpful tests. We think this approach is valuable in the current context: models are now very capable at coding tasks, but their changes can still require substantial cleanup before a maintainer would merge them.

We run FrontierCode 1.1 Main, the primary 100-task set of the benchmark's 150 tasks, with every model in the same agent harness for apples-to-apples comparisons. The tasks are private to avoid contamination.

Leading models still fail more than 40% of the time. Claude Opus 5.5 (max with fallback) leads with an average score of 52% and a pass rate of 58%. Pass rate is the share of attempts that complete all blocking criteria, while scores reflect desirable non-blockers too.

FrontierCode 1.1 launches as a standalone leaderboard and is not part of the Artificial Analysis Intelligence Index. We plan to add it to our Coding Agents leaderboard soon.

FrontierCode 1.1: Score

Mean score over three attempts per task (attempts that fail a blocking check or are flagged for looking up the solution score zero) · Independently benchmarked by Artificial Analysis

How FrontierCode works

Benchmark overview

Cognition built FrontierCode with the maintainers of 36 open source repositories, who spent more than 40 hours on each task. Cognition announced FrontierCode in June 2026 and released version 1.1 in July 2026, which added rules for fair internet use.

The FrontierCode verifier grades each attempt with deterministic tests and qualitative review: hidden tests and command checks, limits on which files and how much code a change touches, and an LLM review against the maintainers' criteria. Each check is either blocking or non-blocking. An attempt that fails any blocking check scores 0, and one that passes them all scores between 70% and 100%.

Methodology

Task set: we evaluate the 100 tasks of FrontierCode 1.1 Main with three attempts per task.
Agent harness: every model runs in Stirrup, our open source agent harness, capped at 250 turns per attempt.
Grading: an LLM judge assesses the rubric's review criteria. We use GPT-6 Luna (medium) as the judge.
Looking up the answer: an attempt that Cognition's trajectory scanner flags under the fair internet use rules scores 0.
Scoring: we average each task's three attempts, then average the task scores with equal weight.

Full details are on the methodology page.

Key findings

Leading models still fail more than 40% of the time: Claude Opus 5.5 (max with fallback) leads at 52% after passing every blocking check in 58% of its attempts, followed by GPT-6.1 Sol (max) and Claude Fable 5.1 (max with fallback) at 49%
The top of the leaderboard is tightly packed: nine models from six labs score between 43% and 52%. MiMo-V2.6-Pro is the leading open weights model at 45%, followed by DeepSeek V4.1 Flash (max) and Kimi K3 (max) at 44%
GPT-6.1 Sol (max) offers the strongest performance for its cost: at $0.81 per task it scores 49%, matching Claude Fable 5.1 (max with fallback) at less than a tenth of the cost per task of the leading Anthropic models
9 of the 100 tasks are passed by no model on any attempt, and no task is passed by every model on every attempt

Pass rate

We run every task three times. An attempt passes when it meets every blocking check. Pass rates range from 58% of attempts for Claude Opus 5.5 (max with fallback) to 16% for Muse Glimmer (high).

FrontierCode 1.1: Pass Rate

Share of attempts that pass every blocking check · Higher is better

Cost

Cost per task spans more than 100x, from $0.09 for GPT-6 Luna (max) to $12.00 for Claude Fable 5.1 (max with fallback). Four models sit on the Pareto frontier of score against cost: GPT-6 Luna (max) at 42% and $0.09 per task, MiMo-V2.6-Pro at 45% and $0.18, GPT-6.1 Sol (max) at 49% and $0.81, and Claude Opus 5.5 (max with fallback) at 52% and $10.15. The last step on the frontier, from GPT-6.1 Sol (max) to Claude Opus 5.5 (max with fallback), adds 3 points for more than 12x the cost per task.

GPT-6.1 Sol (max) scores 2 points higher than GPT-6 Astra (max) at around a fifth of the cost per task, $0.81 against $4.22. Claude Sonnet 5.5 (max with fallback) uses ~50% more output tokens per task than Claude Opus 5.5 (max with fallback), but at half the per-token price it costs less, $8.26 against $10.15.

FrontierCode 1.1: Score vs. Cost per Task

FrontierCode 1.1 score vs. average cost per task (USD)

Most attractive quadrant

Pareto line

Average cost per task in the evaluation. Costs are split by input, cache hit, cache write, reasoning, and answer token pricing where canonical token counts are available.

Token usage and time

GPT-6 models lead on token efficiency: GPT-6 Astra (max) and GPT-6.1 Sol (max) use ~27k and ~33k output tokens per task, compared with ~157k for Claude Opus 5.5 (max with fallback) and ~236k, the most of any model, for Claude Sonnet 5.5 (max with fallback). Time per task depends on both token use and live output speed, and ranges from ~3 minutes for Inkling (xhigh) to ~46 minutes for Qwen3.8 Max (0902).

FrontierCode 1.1: Output Tokens per Task

Output tokens used to run one task, broken down by reasoning and answer tokens

The average number of answer and reasoning tokens produced per benchmark task in this evaluation.