# Real-SWE 发布：基于私有企业代码库评测前沿编码模型，最高解决率 38.8%

- 来源：Hacker News 热门（buzzing.cc 中文翻译）
- 作者：theanonymousone
- 发布时间：2026-09-13 09:18
- AIHOT 分数：60
- AIHOT 链接：https://aihot.news/items/cmtz5c0hd0ujmroup4uua5duy
- 原文链接：https://withspecific.com/benchmarks/real-swe

## AI 摘要

Specific 发布 Real-SWE 基准，用从真实公司授权的私有生产代码库任务评测前沿编码模型。

## 正文

Benchmarking frontier AI models on private, real-world, enterprise codebases.

01Introduction

Today we are releasing Real-SWE, a benchmark that evaluates frontier AI models on private, real-world, enterprise codebases. Each task comes from a private production codebase that we licensed from a real-world company. These are problems their engineers work on, with all the context and complexity that comes with an existing product.

Private codebases. Agents must navigate proprietary systems whose code and solutions aren’t available on the public internet.

Work with business consequences. Getting billing right, calculating taxes, migrating customers. Changes that affect how a business runs, often across multiple services.

Company-specific complexity. Every company has its own rules and ways of writing code. Agents have to understand those conventions and make changes that work with what’s already there.

Can a coding agent actually do the work of a software engineer in the real world?

1

Fable 5.1

Claude Code

Resolution rate: 38.8%

2

GPT-6 Astra

Codex CLI

Resolution rate: 33.8%

3

Gemini 3.8 Flash

Gemini CLI

Resolution rate: 31.2%

4

GLM 5.3

Claude Code

Resolution rate: 28.8%

=5

Grok 4.6

Grok Build

Resolution rate: 23.8%

=5

Muse Spark 1.3

Muse Code

Resolution rate: 23.8%

7

Kimi K3

Kimi Code

Resolution rate: 18.8%

8

GPT-5.6 Sol

Codex CLI

Resolution rate: 16.2%

#ModelHarnessResolution rate

1Fable 5.1Claude Code

38.8%

2GPT-6 AstraCodex CLI

33.8%

3Gemini 3.8 FlashGemini CLI

31.2%

4GLM 5.3Claude Code

28.8%

=5Grok 4.6Grok Build

23.8%

=5Muse Spark 1.3Muse Code

23.8%

7Kimi K3Kimi Code

18.8%

8GPT-5.6 SolCodex CLI

16.2%

Resolution rate is equivalent to pass@1, averaged over eight independent runs per task. 95% confidence intervals are shown.

Expert-generated or synthetic tasks can be well designed, but they aren’t the verbatim, actual tasks that engineers in real companies need to do. Our tasks differ on two axes: the underlying coding artifact and specificity of the instruction. Both add complexities that challenge today’s frontier models.

We use native harnesses to reflect how enterprise engineers work in practice, evaluating model-and-harness combinations rather than models in isolation.

Real company tasks require company-specific context

Correct billing depends on business rules and external services

Fix invoice billing so each business charges the right tax and exempt customers aren't taxed.

View full instruction

Hide full instruction

Billing reopens on Monday and every invoice this service issues is coming out untaxed. Each business on the platform settles its tax a different way: some maintain a rate themselves, some want each invoice priced against the buyer's destination by our tax authority provider, and some collect nothing at all, while a customer we hold an exemption for is charged nothing whichever way its business is configured. Pricing a destination means going to the authority with both addresses, the priced lines and the product category that business sells under, on the sandbox or the production authority according to the account the business is on; an address the authority refuses must be reported without stopping the invoice. The rate, the tax and the gross belong on the issued invoice, and once an invoice is settled the sale is filed back to the authority under that invoice's number so the returns reconcile. Invoices between European parties show both sides' VAT registrations. The authority and ledger are available at , PROD_TAX_JAR_URL and INFLUX_URL.

Services in the sandbox

TaxJar sandbox

TaxJar production

InfluxDB ledger

NestJS service

TypeScript

Agents work across code, infrastructure, and business tools

Tools and services across Real-SWE task environments. Each task exposes only the services its workflow needs.

AWS emulator

Docker

Kubernetes

GitHub

Linear MCP

PostgreSQL

MySQL

MongoDB

Gel

Redis

Go

Python

Node.js

Vitest

Slack

Intercom

Google Drive

Email

ClickUp

Codebase Selection

We selected codebases through a rigorous screening process, focusing on real companies with substantial usage, strong engineering teams, and demanding production workloads. The sample tasks analyzed below come from these codebases, including:

A Luma/Partiful competitor with 200K+ users and a top 100 App Store ranking

A consumer fintech platform processing 100K+ bank statements

Enterprise AI sales platforms supporting complex business workflows

We prioritize code written to meet an actual user or business need over code written solely to create a benchmark task. Production engineering requires understanding existing architecture, preserving behavior that users rely on, and making changes within real operational constraints.

Brief instructions can require changes across many files

Our tasks describe the change needed, leaving agents to discover implementation details in the codebase and surrounding tools. Any behavior required by the verifier must be stated or reasonably discoverable. This leads to our prompts being slightly underspecified, about par with DeepSWE and Terminal Bench, but specific enough to not omit instructions.

The work is cross-functional and complex: a single change can span multiple parts of the application. Agents must understand existing business logic and company coding patterns while keeping the surrounding system working.

媒体内容 · 前往原文查看

Prompt length · median

A typical Real-SWE instruction is 1,742 characters.

FrontierCode2,056 chars

DeepSWE1,975 chars

Terminal-Bench 31,584 chars

FrontierSWE v2992 chars

Real-SWE1,742 chars

Files edited by the reference solution · median

11 files in Real-SWE, compared with 6 in FrontierCode and DeepSWE.

FrontierCode6

DeepSWE6

Real-SWE11

All figures are medians. FrontierCode and DeepSWE use Cognition's published comparison; FrontierCode includes task descriptions and codebase guidelines. We measured instruction files from Terminal-Bench 3's 74 tasks, FrontierSWE v2's 34 tasks, and Real-SWE's eight repository-backed sample tasks. Character counts are rounded to the nearest whole character. No comparable files-edited figure is included for Terminal-Bench 3 or FrontierSWE v2.

Models fail even in short rollouts.

71.4% of rollouts under 10 minutes failed, compared with 73.4% of longer rollouts.

Triaging multiple systems and understanding requirements in codebases riddled with existing business logic and coding patterns is difficult.

媒体内容 · 前往原文查看

Under 10 min

70/98 failed

媒体内容 · 前往原文查看

10 min or longer

398/542 failed

Failed

Passed

Every task is inspired or lifted verbatim from a private, real-world codebase. We find these types of tasks super interesting for three reasons:

Tasks on private codebases are natively out of distribution. These types of coding tasks are not available anywhere on the internet and are unlikely to have ever been trained on by any other ai model. 99% of tokens in real-world enterprises are hidden away from the frontier models.

These tasks are economically viable work. Each task here has a direct relationship to spend and was assigned to an engineer earning a salary. Most benchmarks test interesting, experimental capabilities that are often unlikely to be widespread in the real-world.

Company-specific engineering patterns matter. Does AI code match the bar of a real-world enterprise? Our results show us that we're far from that reality. Many enterprises care about code standards and patterns. We've found that today's models are weaker at understanding company coding patterns and frequently miss requirements or don't verify their assumptions.

02Analysis

Here's an analysis of a small sample of tasks from our benchmark. If you're interested in the sample, request access here.

6 of 10 tasks have resolution rates below 15%

Select a task to view model results. Percentages show the overall resolution rate.

Multi-region sweep67.2%

Fable 5.17/8 passed

GPT-6 Astra8/8 passed

Gemini 3.8 Flash8/8 passed

GLM 5.32/8 passed

Grok 4.63/8 passed

Muse Spark 1.38/8 passed

Kimi K32/8 passed

GPT-5.6 Sol5/8 passed

API keys & environments65.6%

Fable 5.18/8 passed

GPT-6 Astra5/8 passed

Gemini 3.8 Flash7/8 passed

GLM 5.35/8 passed

Grok 4.64/8 passed

Muse Spark 1.36/8 passed

Kimi K30/8 passed

GPT-5.6 Sol7/8 passed

Entitlement overage lines50.0%

Fable 5.18/8 passed

GPT-6 Astra7/8 passed

Gemini 3.8 Flash5/8 passed

GLM 5.33/8 passed

Grok 4.61/8 passed

Muse Spark 1.31/8 passed

Kimi K36/8 passed

GPT-5.6 Sol1/8 passed

Customer identity migration40.6%

Fable 5.13/8 passed

GPT-6 Astra1/8 passed

Gemini 3.8 Flash3/8 passed

GLM 5.34/8 passed

Grok 4.68/8 passed

Muse Spark 1.33/8 passed

Kimi K34/8 passed

GPT-5.6 Sol0/8 passed

Billing schedule migration14.1%

Fable 5.13/8 passed

GPT-6 Astra1/8 passed

Gemini 3.8 Flash2/8 passed

GLM 5.32/8 passed

Grok 4.60/8 passed

Muse Spark 1.30/8 passed

Kimi K31/8 passed

GPT-5.6 Sol0/8 passed

API token metering12.5%

Fable 5.11/8 passed

GPT-6 Astra5/8 passed

Gemini 3.8 Flash0/8 passed

GLM 5.31/8 passed

Grok 4.60/8 passed

Muse Spark 1.30/8 passed

Kimi K31/8 passed

GPT-5.6 Sol0/8 passed

S3 datastore measurement10.9%

Fable 5.10/8 passed

GPT-6 Astra0/8 passed

Gemini 3.8 Flash0/8 passed

GLM 5.33/8 passed

Grok 4.62/8 passed

Muse Spark 1.31/8 passed

Kimi K31/8 passed

GPT-5.6 Sol0/8 passed

Linearizable scan4.7%

Fable 5.10/8 passed

GPT-6 Astra0/8 passed

Gemini 3.8 Flash0/8 passed

GLM 5.32/8 passed

Grok 4.61/8 passed

Muse Spark 1.30/8 passed

Kimi K30/8 passed

GPT-5.6 Sol0/8 passed

Tax jurisdiction3.1%

Fable 5.11/8 passed

GPT-6 Astra0/8 passed

Gemini 3.8 Flash0/8 passed

GLM 5.31/8 passed

Grok 4.60/8 passed

Muse Spark 1.30/8 passed

Kimi K30/8 passed

GPT-5.6 Sol0/8 passed

Analytics stream reducer0.0%

Fable 5.10/8 passed

GPT-6 Astra0/8 passed

Gemini 3.8 Flash0/8 passed

GLM 5.30/8 passed

Grok 4.60/8 passed

Muse Spark 1.30/8 passed

Kimi K30/8 passed

GPT-5.6 Sol0/8 passed

TaskFable 5.1GPT-6 AstraGemini 3.8 FlashGLM 5.3Grok 4.6Muse Spark 1.3Kimi K3GPT-5.6 SolResolution rate

Multi-region sweep

7/8

8/8

8/8

2/8

3/8

8/8

2/8

5/8

67.2%

API keys & environments

8/8

5/8

7/8

5/8

4/8

6/8

0/8

7/8

65.6%

Entitlement overage lines

8/8

7/8

5/8

3/8

1/8

1/8

6/8

1/8

50.0%

Customer identity migration

3/8

1/8

3/8

4/8

8/8

3/8

4/8

0/8

40.6%

Billing schedule migration

3/8

1/8

2/8

2/8

0/8

0/8

1/8

0/8

14.1%

API token metering

1/8

5/8

0/8

1/8

0/8

0/8

1/8

0/8

12.5%

S3 datastore measurement

0/8

0/8

0/8

3/8

2/8

1/8

1/8

0/8

10.9%

Linearizable scan

0/8

0/8

0/8

2/8

1/8

0/8

0/8

0/8

4.7%

Tax jurisdiction

1/8

0/8

0/8

1/8

0/8

0/8

0/8

0/8

3.1%

Analytics stream reducer

0/8

0/8

0/8

0/8

0/8

0/8

0/8

0/8

0.0%

Each task had 8 rollouts per model.

Missed requirements are the most common failure

Failures are grouped by observed submission behavior using the same taxonomy across models, following DeepSWE.

Fable 5.1

24.5%

36.7%

34.7%

4.1%

GPT-6 Astra

34.0%

28.3%

34.0%

3.8%

Gemini 3.8 Flash

10.9%

29.1%

49.1%

10.9%

GLM 5.3

28.1%

38.6%

26.3%

7.0%

Grok 4.6

24.6%

67.2%

8.2%

Muse Spark 1.3

19.7%

36.1%

41.0%

3.3%

Kimi K3

15.4%

53.8%

27.7%

3.1%

GPT-5.6 Sol

43.3%

31.3%

16.4%

9.0%

Unverified assumption

Missed requirement

Integration error

Regression

Wrong file

No model solves every task

One square per rollout: each row is a task, each column a trial, eight trials per task for every model.

Fable 5.1

01

02

03

04

05

06

07

08

09

10

GPT-6 Astra

01

02

03

04

05

06

07

08

09

10

Gemini 3.8 Flash

01

02

03

04

05

06

07

08

09

10

GLM 5.3

01

02

03

04

05

06

07

08

09

10

Grok 4.6

01

02

03

04

05

06

07

08

09

10

Muse Spark 1.3

01

02

03

04

05

06

07

08

09

10

Kimi K3

01

02

03

04

05

06

07

08

09

10

GPT-5.6 Sol

01

02

03

04

05

06

07

08

09

10

PassUnverified assumptionMissed requirementIntegration errorRegressionWrong file

Different models fail in different ways

Percentages are out of each model's failed runs, not all runs.

Unverified assumption

Builds on a guess about the system instead of checking it in the workspace.

GPT-5.6 Sol43.3%: 29 of 67 failed runs

GPT-6 Astra34.0%: 18 of 53 failed runs

GLM 5.328.1%: 16 of 57 failed runs

Grok 4.624.6%: 15 of 61 failed runs

Fable 5.124.5%: 12 of 49 failed runs

Muse Spark 1.319.7%: 12 of 61 failed runs

Kimi K315.4%: 10 of 65 failed runs

Gemini 3.8 Flash10.9%: 6 of 55 failed runs

Missed requirement

Leaves out behavior the instruction requires.

Grok 4.667.2%: 41 of 61 failed runs

Kimi K353.8%: 35 of 65 failed runs

GLM 5.338.6%: 22 of 57 failed runs

Fable 5.136.7%: 18 of 49 failed runs

Muse Spark 1.336.1%: 22 of 61 failed runs

GPT-5.6 Sol31.3%: 21 of 67 failed runs

Gemini 3.8 Flash29.1%: 16 of 55 failed runs

GPT-6 Astra28.3%: 15 of 53 failed runs

Integration error

Right idea, wired into the surrounding system incorrectly.

Gemini 3.8 Flash49.1%: 27 of 55 failed runs

Muse Spark 1.341.0%: 25 of 61 failed runs

Fable 5.134.7%: 17 of 49 failed runs

GPT-6 Astra34.0%: 18 of 53 failed runs

Kimi K327.7%: 18 of 65 failed runs

GLM 5.326.3%: 15 of 57 failed runs

GPT-5.6 Sol16.4%: 11 of 67 failed runs

Grok 4.68.2%: 5 of 61 failed runs

Regression

Breaks existing behavior while making the change.

Gemini 3.8 Flash10.9%: 6 of 55 failed runs

GPT-5.6 Sol9.0%: 6 of 67 failed runs

Fable 5.14.1%: 2 of 49 failed runs

GPT-6 Astra3.8%: 2 of 53 failed runs

Muse Spark 1.33.3%: 2 of 61 failed runs

GLM 5.30%: 0 of 57 failed runs

Grok 4.60%: 0 of 61 failed runs

Kimi K30%: 0 of 65 failed runs

Wrong file

Delivers the change somewhere the running application never calls, such as a one-off script.

GLM 5.37.0%: 4 of 57 failed runs

Kimi K33.1%: 2 of 65 failed runs

Fable 5.10%: 0 of 49 failed runs

GPT-6 Astra0%: 0 of 53 failed runs

Gemini 3.8 Flash0%: 0 of 55 failed runs

Grok 4.60%: 0 of 61 failed runs

Muse Spark 1.30%: 0 of 61 failed runs

GPT-5.6 Sol0%: 0 of 67 failed runs

03Effort & the Frontier

Higher cost does not guarantee a higher resolution rate

Estimated frontier

1Fable 5.138.8% · $6.96

2GPT-6 Astra33.8% · $4.67

3Gemini 3.8 Flash31.2% · $2.50

4GLM 5.328.8% · $5.12

5Grok 4.623.8% · $3.44

6Muse Spark 1.323.8% · $2.74

7Kimi K318.8% · $3.90

8GPT-5.6 Sol16.2% · $2.65

Estimated rollout costs range from $2.50 to $6.96

RankModelEstimated cost (USD)

1Gemini 3.8 Flash$2.50

2GPT-5.6 Sol$2.65

3Muse Spark 1.3$2.74

4Grok 4.6$3.44

5Kimi K3$3.90

6GPT-6 Astra$4.67

7GLM 5.3$5.12

8Fable 5.1$6.96

mean per rollout, by task

Swipe the chart to see all tasks.

Fable 5.1 · 64k overall

GPT-6 Astra · 24k overall

Gemini 3.8 Flash · 94k overall

GLM 5.3 · 117k overall

Grok 4.6 · 67k overall

Muse Spark 1.3 · 87k overall

Kimi K3 · 43k overall

GPT-5.6 Sol · 23k overall

View task values

Fable 5.134k

GPT-6 Astra13k

Gemini 3.8 Flash78k

GLM 5.368k

Grok 4.67k

Muse Spark 1.336k

Kimi K330k

GPT-5.6 Sol12k

04Evaluation Setup

Each agent was run in an isolated sandbox. All tasks are in Harbor format, and verifiers are injected at grading time. The verifiers are inspired by existing test suites in the codebase or use those tests verbatim.