截至今天，Ploy 的智能体运行在 GPT-5.6 Sol 上，这是 OpenAI 今早发布的模型家族中的旗舰层级。数月以来，没有任何模型达到我们替换 Claude Opus 的标准。GPT-5.6 Sol 做到了。经过一轮正面对比评估，我们将其设为每个 Ploy 工作区的默认模型。
Ploy 的智能体负责构建和编辑生产环境的营销网站。它会规划页面、阅读代码库、编写组件、生成图像、对自己的作品截图，并判断何时完成。我们用这一工作负载测试每一个前沿版本。在 Opus 占据默认位置的这四个月里（先是 Opus 4.7，然后是 4.8），我们测试过的任何模型都未能超越它。GPT-5.6 是第一个做到的。
第一轮评估运行暴露出了若干失败模式，我们在下文会逐一说明。它也带来了强劲的结果。完成的构建耗时不到原来的一半，成本降低 27%，得分达到或超过我们现有的模型。这些结果证明了迁移工作的合理性。
我们使用 Vercel 的 AI SDK，但从 Claude Opus 4.8 切换到 GPT-5.6 Sol 仍然暴露出我们整个技术栈中针对特定提供商的假设。各家提供商在模型如何填充工具参数、缓存提示词以及在轮次之间重放推理过程方面存在差异。
我们先修复了评估框架，随后依次处理了工具 schema、提示词缓存和推理重放。
第 0 步：在信任任何一个数字之前，先修好你的评估框架
我们的评估套件让生产环境的智能体在固定工作区上运行。它覆盖了数百个用例，从“从零构建一个主页”到“这个克隆请求是否可以安全执行”。对于构建类用例，一个视觉评判器会针对参考设计运行十项二元检查，例如“主视觉是一幅全出血的摄影场景”以及“主要 CTA 是圆角矩形，而不是胶囊形。”我们还会运行内容检查、工具轨迹检查和文件断言。我们会对照完整轨迹对每一个失败进行分诊，包括工具调用和模型文本。
第一次跨模型运行暴露了评估框架中的一个问题。
我们的工具调用预算是按照 Opus 的串行风格来设定的。GPT-5.6 会进行并行调用，在它正确解决的用例上超出了这些预算。我们的评估执行器也不支持批量文件读取，而 Opus 很少使用这一功能，GPT-5.6 却经常使用。第一次运行中大约三分之一的原始失败来自框架假设，而非模型行为，而且这些失败分布不均。如果你要用一个挑战者模型去评估一个在位模型，在信任通过率之前先对轨迹进行分诊。否则，评估实际上是在奖励新模型表现得像旧模型。
一个省略了其minScore阈值的数据集继承了 1.0 的默认值，而没有任何东西标记出这一回退。因此 GPT-5.6 在一个得分为 0.98 的主视觉上“失败”了，而 Opus 尽管通过了每一项单独检查，却仍在一个用例上“失败”了。两种设计都站得住脚，但那个隐式阈值让它们都失败了。
基准测试结果
修复测试框架后，我们重新运行了重设计测试套件，在该套件中，智能体需要参照一份参考设计重建某个品牌的首页。
| 每次完成构建的平均值 | Claude Opus 4.8（n=11） | GPT-5.6（n=10） |
|---|---|---|
| 成本 | $3.06 | $2.22 |
| 实际耗时 | 8分00秒 | 3分42秒 |
| 输入 token | 2.60M | 1.70M |
| 输出 token | 33.0K | 17.1K |
| 视觉评分 | 0.936 | 0.970 |
GPT-5.6 完成页面的速度快 2.2 倍，成本低 27%，输出 token 用量约为一半。它写的代码也更少。在一组匹配对比中，Opus 生成了一个 17,957 字符的 globals.css，包含 174 个 CSS 变量，其中大部分是未使用的颜色梯度。GPT-5.6 只用了 2,508 个字符和 45 个变量，就渲染出了效果相当、有时甚至更好的页面。
Claude Opus 4.8
GPT-5.6 Sol
设计质量与一致性
GPT-5.6 擅长干净、网格严密的布局，但在缺乏强引导的情况下，它往往会收敛到这种风格。用我们为 Opus 4.8 设计的旧版 harness 时，GPT-5.6 Sol 经常忽略现有的设计系统，产出干净但千篇一律的结果。
我们在这里所做的改动值得单独写一篇文章。我们的设计和工程团队改进了引导，直到 GPT-5.6 达到我们在生产环境中所要求的品牌一致性标准。
第 1 步：检查你的工具调用
我们智能体的 code 工具有 25 个顶层参数。其中 action 是必填的，其余都是可选的。Claude 只发送它用到的两三个参数，省略其余部分。GPT-5.6 每次调用都发送全部 25 个参数，用看似合理的值填充未使用的参数，例如 offset: 0、timeout: 120000 和 siteId: "00000000-0000-0000-0000-000000000000"。
我们在三天的生产环境 code(read) 追踪记录中看到了这一模式。
| 模型 | 调用次数 | 携带全部 25 个属性 |
|---|---|---|
| gpt-5.6 | 6,635 | 6,635（100%） |
| claude-opus-4.8 | 2,898 | 4（0.1%） |
| claude-sonnet-5 | 1,933 | 0 |
问题并不在于冗长。文件读取的实现无法区分凭空捏造的值和预期的值。它把 offset: 0 当作真实参数处理，导致 GPT-5.6 的文件读取中有 52% 到 64% 返回空结果。该工具对有效读取和空读取都返回 success: true，因此模型无法判断自己读到的其实是空文件。它只能靠更多调用来弥补，结果反而更差。
提示词工程没能解决这个问题。在工具描述中加一条“省略未使用的参数”的指令，结果仍然生成了 25 个参数中的 25 个。逐属性的“可选，未使用则省略”提示也是同样的结果。我们用 OpenAI 的 strict 模式测得了完全相同的行为，而采用它就意味着要从每个 schema 中剥离 pattern、format 以及数组边界校验。这种行为源于模型发出 函数调用的方式，所以我们改为修改 schema。
有效的修复方案是在 provider 边界处做一次 schema 转换。对于 OpenAI 系列模型，我们用 anyOf: [T, null] 把每个可选属性改写为必填但可为 null。这样模型就为未使用的参数提供了一个显式值。随后我们在共享的工具调用边界处、校验之前剥离这些 null，因此工具实现无需改动。模型看到的 schema 能够表达未使用的值，而工具收到的输入与之前相同。
// Before: 25 keys, every one carrying an invented value
{ "action": "read", "file_paths": [...], "offset": 0, "timeout": 120000, ... }
// After: 25 keys, 4 real values, 21 explicit nulls (stripped before the tool runs)
{ "action": "read", "file_paths": [...], "offset": null, "timeout": null, ... }改动之后，空文件读取从 52% 降至 0%。由于不再重复读取空白结果，智能体完成同样的工作所用的工具调用次数也减少了约 30%。
第 2 步：重建提示词缓存
两家 provider 都提供“提示词缓存”，但实现方式不同。在我们弄清这些差异之前，GPT-5.6 看起来比 Opus 贵约 50%。这个差距是我们的缓存配置造成的。模型定价本身没问题。
我们的智能体提示词以一个约 29K token 的静态前缀开头（工具 schema 加上核心系统提示词），该前缀在每次对话中都完全相同。在 Claude 上，我们用 cache_control 标记缓存断点，该前缀在整个组织范围内缓存。任何工作区中的任何对话都可以使用同一个共享条目，且不受按 key 的吞吐量预算限制。缓存命中率在 92% 到 96% 之间。
GPT-5.6 使用一种不同的 OpenAI 缓存模型。早期的 GPT 模型会缓存隐式的部分前缀匹配。GPT-5.6 取消了部分前缀匹配，因此隐式缓存现在会以最新消息为键创建整段提示词条目。一个共享我们 29K 静态前缀的新对话，其缓存命中率为 0%。每次对话都按未缓存费率重新计费整个前缀。GPT-5.6 还会对每个未缓存的提示词加收 1.25× 的缓存写入附加费，无论应用是否使用了缓存。
显式机制使用 prompt_cache_breakpoint 标记以及一个必填的 prompt_cache_key。该 key 是缓存标识的一部分，因此相同提示词搭配不同 key 不会产生缓存命中。每个 key 映射到一个缓存节点，该节点大约能承受每分钟 15 个请求，之后 OpenAI 会将流量分发到其他具有独立冷缓存的节点。
主要的设计决策是应由哪个实体来确定 key 的作用域。
- 按对话设置 key：新对话永远不会命中共享前缀。我们在此配置下测得首次调用命中率为 0%。
- 单一全局键： 每个请求都哈希到同一个缓存节点。生产流量超出了 15 rpm 的预算，请求溢出到冷节点。
- 按工作区划分的键： 客户工作区中的所有对话共享条目，而每个键的流量保持较低。
我们上线了按工作区划分的键，并将系统提示词拆分为带断点的分层，与我们此前用于 Anthropic 的结构保持一致：
request ──► hash(prompt head + prompt_cache_key) ──► cache node (~15 req/min per key)
│
┌──────────────────────────────────────────────────────┴───────────────┐
│ entries on the node, all namespaced by key ws:{workspaceId} │
│ │
│ [ tools + static prefix ]······················ A every session │
│ [ tools + static prefix + workspace context ]·· B same context │
│ [ ····················· + turn 1 + … + latest ] C this session │
└──────────────────────────────────────────────────────────────────────┘条目 A 降低了会话首次调用的成本。当工作区记忆发生变化时，请求会未命中条目 B，但仍会命中条目 A，然后写入一个新的条目 B。这只需要一次上下文大小的写入，而无需为完整的 29K 前缀重新计费。条目 C 是 OpenAI 的隐式全提示词链，它在会话内有效，因为我们的提示词严格只追加。
OpenAI 的键分区阻止了静态前缀跨工作区共享。Anthropic 可以共享前缀，因为其缓存是组织范围的，没有键分区。在 GPT-5.6 上，每个工作区在每个空闲窗口都要为一次 29K 的冷写入付费，约 $0.18。成本是有界且可预测的。
改动之后，首次调用缓存命中率从大约 0% 上升到 83.7%，未缓存输入 token 总数下降了 28%，GPT-5.6 的每套件成本降至 Opus 以下。缓存配置错误正是我们此前测得的全部成本差距所在。当一个模型从冷缓存启动时，模型之间的成本比较是没有意义的。
第 3 步：让推理重放自包含
推理重放导致生产环境对话出现间歇性失败。GPT-5.6 的 Responses API 默认会将先前轮次的推理作为服务端条目引用进行重放，我们的对话有时会因 Item 'rs_...' not found 而失败。设置 store: false 会让 SDK 请求 加密的推理内容，并重放自包含的数据块，而不是指向服务端状态的指针。我们还了解到，即使应用程序发送的字节是仅追加的，服务端的推理状态也可能改变实际生效的提示词。
在生产环境中运行 GPT-5.6 Sol
GPT-5.6 今天发布，并且已经在 Ploy 上线。你可以 在 ploy.ai 免费开始使用，给它一个网站去构建，看看不到四分钟的构建是什么样子。
Ploy 是一个 AI 层，能够规划、构建、发布并优化网站和营销活动。如果你觉得凌晨两点调试缓存节点扇出很有趣，我们正在招聘.
As of today, Ploy’s agent runs on GPT-5.6 Sol, the flagship tier of the model family OpenAI released this morning. For months, no model met our bar for replacing Claude Opus. GPT-5.6 Sol did. After a head-to-head evaluation, we made it the default model for every Ploy workspace.
Ploy’s agent builds and edits production marketing websites. It plans a page, reads the codebase, writes components, generates imagery, screenshots its own work, and decides when it is done. We test every frontier release against that workload. During the four months Opus held the default slot (first Opus 4.7, then 4.8), nothing we tested beat it. GPT-5.6 is the first model that did.
The first eval run exposed several failure modes, which we cover below. It also produced strong results. Completed builds took less than half the wall-clock time, cost 27% less, and scored at or above our incumbent. Those results justified the migration work.
We use Vercel’s AI SDK, but switching from Claude Opus 4.8 to GPT-5.6 Sol still exposed provider-specific assumptions throughout our stack. The providers differ in how models fill tool arguments, cache prompts, and replay reasoning between turns.
We fixed the eval harness first, followed by the tool schemas, prompt caching, and reasoning replay.
Step 0: Fix your harness before you trust a single number
Our eval suite runs the production agent against fixture workspaces. It covers hundreds of cases, from “build a homepage from scratch” to “is this clone request safe to execute.” For build cases, a visual judge runs ten binary checks against a reference design, such as “the hero is a full-bleed photographic scene” and “primary CTAs are rounded rectangles, not pills.” We also run content checks, tool-trajectory checks, and file assertions. We triage every failure against the full trace, including the tool calls and model text.
The first cross-model run exposed a problem in the eval harness.
Our tool-call budgets were sized for Opus’s sequential style. GPT-5.6 makes parallel calls and exceeded those budgets on cases it was solving correctly. Our eval executor also lacked support for batched file reads, which Opus rarely used and GPT-5.6 uses often. Roughly a third of the raw failures in the first run came from harness assumptions rather than model behavior, and those failures were unevenly distributed. If you are evaluating a challenger against an incumbent, triage the traces before trusting the pass rate. Otherwise, the eval rewards the new model for behaving like the old one.
A dataset that omitted its minScore threshold inherited a default of 1.0, and nothing flagged the fallback. GPT-5.6 therefore “failed” a hero that scored 0.98, while Opus “failed” a case despite passing every individual check. Both designs were defensible, but the implicit threshold failed them.
Benchmark results
After fixing the harness, we reran our redesign suite, where the agent rebuilds a brand’s homepage against a reference design.
| Mean per completed build | Claude Opus 4.8 (n=11) | GPT-5.6 (n=10) |
|---|---|---|
| Cost | $3.06 | $2.22 |
| Wall-clock time | 8m 00s | 3m 42s |
| Input tokens | 2.60M | 1.70M |
| Output tokens | 33.0K | 17.1K |
| Visual score | 0.936 | 0.970 |
GPT-5.6 finished pages 2.2× faster, cost 27% less, and used about half the output tokens. It also wrote less code. In one matched pair, Opus produced a 17,957-character globals.css with 174 CSS variables, including mostly unused color ramps. GPT-5.6 used 2,508 characters and 45 variables for a comparable, and sometimes better, rendered page.
Claude Opus 4.8
GPT-5.6 Sol
Design quality and consistency
GPT-5.6 is good at clean, tightly gridded layouts, but it tends to converge on that style without strong steering. With our older harness, which was designed for Opus 4.8, GPT-5.6 Sol often ignored the existing design system and produced clean but generic output.
The changes we made here deserve a separate post. Our design and engineering teams improved the steering until GPT-5.6 met the brand-adherence bar we require in production.
Step 1: Check your tool calls
Our agent’s code tool has 25 top-level parameters. One, action, is required; the rest are optional. Claude sends the two or three parameters it uses and omits the rest. GPT-5.6 sends all 25 on every call, filling unused parameters with plausible values such as offset: 0, timeout: 120000, and siteId: "00000000-0000-0000-0000-000000000000".
We saw the pattern across three days of production code(read) traces.
| Model | Calls | Carrying all 25 properties |
|---|---|---|
| gpt-5.6 | 6,635 | 6,635 (100%) |
| claude-opus-4.8 | 2,898 | 4 (0.1%) |
| claude-sonnet-5 | 1,933 | 0 |
Verbosity was not the problem. The file-read implementation could not distinguish an invented value from an intended one. It treated offset: 0 as a real argument, causing 52% to 64% of GPT-5.6’s file reads to return empty. The tool returned success: true for both valid and empty reads, so the model could not tell that it was reading blank files. It compensated with more calls and worse results.
Prompting did not fix this. A tool-description directive to “omit unused parameters” still produced 25 of 25 parameters. Per-property “OPTIONAL, omit if unused” hints did the same. We measured identical behavior with OpenAI’s strict mode, and adopting it would have required stripping pattern, format, and array-bound validation from every schema. This behavior comes from how the model emits function calls, so we changed the schema instead.
The working fix was a schema transform at the provider boundary. For OpenAI-family models, we rewrite every optional property as required but nullable using anyOf: [T, null]. This gives the model an explicit value for an unused parameter. We then strip the nulls before validation at the shared tool-invocation boundary, so the tool implementations do not change. The model sees a schema that represents unused values, while the tools receive the same inputs as before.
// Before: 25 keys, every one carrying an invented value
{ "action": "read", "file_paths": [...], "offset": 0, "timeout": 120000, ... }
// After: 25 keys, 4 real values, 21 explicit nulls (stripped before the tool runs)
{ "action": "read", "file_paths": [...], "offset": null, "timeout": null, ... }After the change, empty file reads fell from 52% to 0%. The agent also used roughly 30% fewer tool calls for the same work because it stopped re-reading blank results.
Step 2: Rebuild prompt caching
Both providers offer “prompt caching,” but the implementations differ. Before we accounted for those differences, GPT-5.6 appeared about 50% more expensive than Opus. Our cache configuration caused the gap. The model pricing was fine.
Our agent’s prompt opens with a static prefix of roughly 29K tokens (tool schemas plus the core system prompt) that is identical for every conversation. On Claude, we mark cache breakpoints with cache_control, and the prefix caches across the whole organization. Any conversation in any workspace can use one shared entry, without a per-key throughput budget. Cache hit rates run from 92% to 96%.
GPT-5.6 uses a different OpenAI caching model. Earlier GPT models cached implicit partial-prefix matches. GPT-5.6 dropped partial-prefix matching, so implicit caching now creates whole-prompt entries keyed on the latest message. A new conversation sharing our 29K static prefix cached 0% of it. Each conversation re-billed the full prefix at the uncached rate. GPT-5.6 also applies a 1.25× cache-write surcharge to every uncached prompt, whether or not the application uses caching.
The explicit mechanism uses prompt_cache_breakpoint markers and a mandatory prompt_cache_key. The key is part of the cache identity, so identical prompts with different keys produce no cache hits. Each key maps to a cache node that sustains roughly 15 requests per minute before OpenAI distributes traffic to other nodes with independent cold caches.
The main design decision is which entity should scope the key.
- Per-conversation key: A new conversation never hits the shared prefix. We measured a 0% first-call hit rate with this configuration.
- One global key: Every request hashes to one cache node. Production traffic exceeds the 15 rpm budget, and requests spill to cold nodes.
- Per-workspace key: All conversations in a customer workspace share entries while per-key traffic stays low.
We ship the workspace-scoped key and split the system prompt into breakpointed layers, mirroring the structure we already used for Anthropic:
request ──► hash(prompt head + prompt_cache_key) ──► cache node (~15 req/min per key)
│
┌──────────────────────────────────────────────────────┴───────────────┐
│ entries on the node, all namespaced by key ws:{workspaceId} │
│ │
│ [ tools + static prefix ]······················ A every session │
│ [ tools + static prefix + workspace context ]·· B same context │
│ [ ····················· + turn 1 + … + latest ] C this session │
└──────────────────────────────────────────────────────────────────────┘Entry A reduces the cost of a session’s first call. When workspace memory changes, the request misses Entry B but still hits Entry A, then writes a new Entry B. That requires one context-sized write instead of re-billing the full 29K prefix. Entry C is OpenAI’s implicit whole-prompt chain, which works within a session because our prompts are strictly append-only.
OpenAI’s key partitioning prevents cross-workspace sharing of the static prefix. Anthropic can share the prefix because its cache is organization-scoped without key partitioning. On GPT-5.6, every workspace pays for one 29K cold write per idle window, about $0.18. The cost is bounded and predictable.
After the change, first-call cache hits rose from roughly 0% to 83.7%, total uncached input tokens fell 28%, and GPT-5.6’s per-suite cost dropped below Opus’s. Cache misconfiguration accounted for the entire cost gap we had measured. Cost comparisons between models are not useful when one model starts with a cold cache.
Step 3: Make reasoning replay self-contained
Reasoning replay caused intermittent failures in production conversations. GPT-5.6’s Responses API replays prior-turn reasoning as server-side item references by default, and our conversations sometimes failed with Item 'rs_...' not found. Setting store: false makes the SDK request encrypted reasoning content and replay self-contained blobs instead of pointers to server state. We also learned that server-side reasoning state can change the effective prompt even when the bytes sent by the application are append-only.
Running GPT-5.6 Sol in production
GPT-5.6 launched today and is already live on Ploy. You can start free at ploy.ai, give it a website to build, and see what a sub-four-minute build looks like.
Ploy is an AI layer that plans, builds, publishes, and optimizes websites and campaigns. If debugging cache-node fan-out at 2am sounds like fun, we’re hiring.