Databricks 使用内部百万行级代码库（3000+ 工程师、跨三个超大规模云）测试编码智能体，避开公开基准。结果发现开源模型 GLM 5.2 在质量上与 Claude Opus 4.8 统计持平，但任务成本仅 $1.28（Opus 为 $1.94）。自家 Pi 框架通过每轮减少约 3 倍上下文，使同一模型成本降低超 2 倍。测试区分了模型智能与智能体效率：GLM 5.2 证明能应对最高难度任务，Pi 则展示高效上下文管理的成本优势。针对 11000 名员工的 AI 成本压力，Databricks 认为该组合（模型+框架）为降低费用并保持质量提供了可行路径。
Surprising and such a good news for open source coding model, and also that there are lots of hidden chances to reduce cost while improving quality.
Databricks showed GLM-5.2 can compete with elite closed coding models inside real enterprise code.
GLM 5.2 landed in Databricks’ top capability tier and was statistically tied with Claude Opus 4.8 on quality.
- The Pareto frontier for coding tasks (i.e. best quality for a given cost) includes models from OpenAI, Anthropic, and open source.
- This means today, only a mix of tools can provide frontier performance.
- Open models, and GLM 5.2 in particular, are now able to handle even the highest level of task difficulty.
Databricks built the test because public coding benchmarks can become too familiar to models.
Instead, it used real internal PRs, real tests, and a multi-million-line codebase.
The cost result makes it more serious. GLM 5.2 at $1.28/task, compared with $1.94/task for Opus 4.8.
So in their multi-million lines of enterprise-grade code test, GLM-5.2 made open-weight routing credible, and Pi showed why a credible model must be tested inside the right harness.
Their test also separates two things people often mix up: model intelligence and agent efficiency.
A coding agent needs two parts. The model does the reasoning and code writing. The harness manages the work around it, including file search, terminal commands, test output, and context. GLM-5.2 proved the first part by landing near the top closed models on Databricks’ own private tasks.
Pi, the harness, proved the second part by showing that the same level of work can be done with far less context sent into the model.
Databricks found Pi could run the same model with the same thinking effort at over 2x lower cost because it sent about 3x less context per turn.
At 11k employees, our AI costs are going up. Which model & harness should we use to lower cost but also retain great quality? We didn't want to blindly trust public benchmarks. So we ran a comprehensive evaluation on our tasks, code base, infra. It's been produced by more than 3,000 software engineers, spans 3 hyperscalar clouds and many languages and tasks. The results are surprising. We find that for the SAME mdoel, the choice of harness can significantly save costs (~2x). We also find that GLM 5.2 performs extremely well. We run Omnigent in front of these and can easily multiplex different harnesses and models for different tasks. Check it out: https://www.databricks.com/blog/benchmarking-coding-agents-databricks-multi-million-line-codebase在 X 查看被引用的帖子
来源：Rohan Paul · x.com