Factory 的 Legacy-Bench 加入 Fireworks 专项智能指数，评测哪款模型最适合遗留代码

Go back

September 24, 2026 - 2 minute read

Partners

Share

Factory Team | September 24, 2026

Today, Factory’s Legacy-Bench joins Fireworks’ Specialized Intelligence Index (SII), bringing legacy software engineering to a growing collection of industry-built benchmarks comparing open, closed, and specialized AI models on real-world tasks.

Legacy-Bench measures how well frontier models can debug, implement, and migrate software written in COBOL, Java 7, BASIC, C89, Fortran, and Assembly. Its inclusion in the SII gives engineering leaders a clearer way to assess which models can work reliably on the systems behind financial settlement, payroll, insurance, telecommunications, and scientific computing.

The latest results reinforce a point we have seen throughout our research: performance on general coding benchmarks does not transfer evenly to legacy systems.

Benchmarks must reflect user workflows

Model progress has a jagged frontier. A model can perform well on modern Python repositories and struggle with COBOL. It can resolve a GitHub issue but fail to preserve a fixed-width record or packed-decimal calculation. These differences matter because legacy systems leave little room for plausible but incorrect output. A program can compile, run, and appear correct while changing a payment calculation or corrupting a downstream record by a single byte. While general coding benchmarks rarely test these conditions, Legacy-Bench was built to make them measurable.

Legacy software is a different evaluation problem

Legacy-Bench contains tasks for debugging, implementation, and migration across six legacy language families. The tasks represent workflows in financial settlement, payroll, insurance, telecom, and scientific computing. The difficulty beyond unfamiliar syntax. Models must work with:

• Exact encodings and binary layouts
• Fixed-width records and packed decimal fields
• Business rules embedded directly in code
• Output formats where one incorrect byte can break an interface
• Environments that provide limited feedback

Ten representative tasks are available on GitHub in the Harbor format. The full benchmark is used for evaluation.

Frontier models do not perform evenly when tasked with legacy software

The latest evaluation snapshot prepared for the Specialized Intelligence Index produced the following results

GPT-5.6 Sol leads GPT-6 Astra by nine percentage points on this evaluation. These results reflect the sentiment that general model progress does not transfer uniformly to legacy engineering work. The benchmark findings show how that gap can emerge. Legacy languages have less representation in model training. Silent failures give the agent little reason to reconsider a plausible answer. Exact-output tasks punish a single wrong byte as a complete failure. None of those weaknesses are captured well by comprehensive modern coding benchmarks, requiring specialized benchmarks to support.

In one Java 7 path, a failing test produced an exception that the model could use to diagnose and correct its work. In a COBOL payroll task, the program compiled and passed 24 of 29 tests, but calculated a $275 health deduction instead of $125. The output looked plausible, and the model reported success.

Legacy environments demand stronger verification precisely where they provide weaker feedback.

Specialized benchmarks are critical to push the frontier

The Specialized Intelligence Index was created to compare open, closed, and specialized models on domain-specific work designed by practitioners. The Index gives organizations evidence grounded in the workflows they need to automate, rather than a single general-purpose ranking.

By contributing Legacy-Bench results, Factory is bringing legacy software engineering into that shared evaluation layer. This collaboration helps engineering leaders assess which models are ready for their workflows.

Benchmarks should reflect the work users need to complete. For many of the world’s largest organizations, key use cases require using maturing technologies that other agentic coding benchmarks do not directly measure.

Interested in trying the benchmark yourself? Reach out to learning@factory.com.