过去几年我们看到的 AI 快速进步中，有多少来自数据，又有多少来自模型改进？这个问题的答案对前沿实验室的经济状况以及未来进步的节奏有着重大影响。
我们在相对较小的规模上、并专门针对预训练阶段，对这一问题进行了研究，时间跨度为 2019 年到 2025 年。在这期间的每一年，都有一篇新的开源模型方案发布，它汇集了当年公开已知的算法改进（例如架构、优化器、初始化、学习率调度、超参数等方面的改进）。同时，每一年也都有新的公开数据语料库出现（这些语料来自更大规模的抓取以及新的整理/提取/过滤技术）。
我们使用这些代表各年份的模型方案与数据语料库，在不同的训练计算规模（最高达 1e19 FLOPs）下训练了各种组合。
显然，我们不能用这些不同模型在固定数据集上的交叉熵损失来比较它们，因为我们改变了它们训练所用的数据集。因此，我们转而通过 OLMES 评测所衡量的最终能力来评估这些模型（该评测汇总了 10 个相对简单的基准测试，主要是多项选择问答）。遗憾的是，评估最终能力而非预训练损失会给我们的结果带来一些噪声，你会在下面的图表中看到这一点，不过我们通过运行多个随机种子来尝试获得更干净的边界。
我们发现，从 2019 年到 2025 年，在 1e19 FLOPs 的计算预算下，计算效率的提升中有 3.24 倍来自数据改进而非模型改进（数据带来 12.0 倍提升，模型带来 3.7 倍提升）。
下面这张网格图展示了：在 3.16e18 FLOPs4 的计算量下，我们训练的模型在最终测试能力上，相对于 2019 年的数据 + 架构基线，提升了多少。
我们发现，数据改进和模型改进带来的收益大体上是相互独立的，彼此之间没有交互作用（也就是说，实现某个模型改进的收益并不需要特定的训练数据堆栈，反之亦然）。OLMES 分数中 88% 的方差可以用模型改进和数据改进的加性效应来解释（使用线性模型）。
讨论
为了提供背景，我们先简要总结一下从 2019 年到 2025 年，数据和模型两方面分别发生了什么变化。
在模型方面，我们从 GPT-2 发展到了 OLMo-2，期间包含了优化器、位置编码、归一化、激活函数、初始化等方面的关键创新，以及其他更多改进5。
在数据方面，我们从 2019 年的 OpenWebText 起步，该数据集仅包含从 Reddit 链接过来的、获得足够多点赞的网页，并经过去重和过滤，因此总共只有约 9B 个 token（GPT-2 的训练数据主要就是这些）。到 2025 年，像 UltraFineWeb 这样的开源数据语料库不仅规模大得多（通过抓取整个互联网），而且采用了更复杂的过滤方法（例如，训练一个分类器来预测哪些数据能在经验上提升模型性能）。
对我们结果的一种朴素解读是，2019-2024 年（预训练时代）的大部分 AI 进展仅仅来自更好的数据工程（提取、整理等），而同期所有模型层面的工作远没有那么重要。
但这可能是思考模型改进价值的错误方式。它们的主要贡献未必是计算效率——即用更少的 FLOPs 达到同样的性能。相反，它们的贡献在于首先让更大规模的计算量变得可用。随着参数量、上下文长度、运行时长和集群规模不断扩展，各种问题都容易发生（梯度爆炸或消失、内存和带宽耗尽、训练慢到不可行）。大量模型研究正是由消除或推迟这些对扩展的限制所构成的。许多最重要的创新，如 MoE、稀疏注意力变体、稳定性创新（归一化位置、初始化等），以及 FlashAttention 这类系统/内核级优化，都属于这一范畴。
我们在这里研究的数据改进，对于更大的模型而言可能影响较小。小模型（比如我们训练的这类）能从数据质量提升中获得显著收益，因为它们本身容量有限，所以你必须非常谨慎地选择往里面塞什么内容。而大模型拥有大量冗余容量，也许你只需要尽可能多地灌入数据，哪怕其中大部分是垃圾，随机梯度下降的魔力也会自动把信号从噪声中分离出来。如果你选择激进地过滤数据，就不得不做几十个 epoch，而经验表明，这比使用平均质量较低但规模更大的数据集效果更差。事实上，考虑到前沿模型为了最小化 RL 和部署阶段的推理计算量，其过度训练程度相比 Chinchilla 最优值高达 100 倍，激进的数据筛选反而危害更大。
打个比方，这就像帆船和集装箱货轮之间的区别——货轮不一定跑得更快，但它能装载数千吨货物（相当于预训练数据中数十万亿的 token），而且不会被汹涌的海浪掀翻（相当于在数十万块 GPU 上稳定训练）。
如今我们有了容量更大、更坚固的集装箱货轮，就不必再为船上具体装什么而烦恼——我们只需把所有哪怕有一丁点潜在用途的东西都塞满船舱即可。而对于 2019 年那些又小又脆弱的帆船，你就必须极其小心，只装载最有价值的货物。
但如果说预训练进展的本质只是往这艘船里装载更多货物，那么我们是否正在耗尽货物？这是一个关于数据墙的问题，也关乎合成数据在多大程度上帮助我们跃过了这堵墙。合成数据显然在各实验室中被广泛使用，而我们完全没有研究过它能否在不损害模型性能的情况下有效扩充数据语料库。如果收益有限，那么预训练进展的主要驱动力将会停滞，因为我们不会产生更多的互联网数据，而且你能对固定数据集做的整理工作也是有限的。需要说明的是，我们目前没有积极理由认为情况会如此。但鉴于数据在推动预训练进展方面似乎如此重要，这似乎是一个值得研究的关键问题。
Ryan Greenblatt 指出，预训练数据语料库历史上的许多改进，看起来都像是自动化研究人员能够直接通过实证检验的那类进展——例如，用不同数据训练进行消融实验，观察模型表现如何。因此，这与我们的结果完全吻合：如果我们将 AI 研发自动化，那么自 2019 年以来推动预训练的数据进展可能会大幅加速。
我们想澄清一点：单就预训练而言，其进展是会加速还是放缓，其实并不是关乎整体 AI 进展最重要的问题，因为过去两年中的大量收益都来自强化学习。
未来研究方向
以下是我们认为非常值得探索、且需要回答的一些未来研究方向：
你可以在更大规模上运行这个实验，以观察数据或模型的改进是否更依赖于规模（从而在前沿领域产生远为重大的影响）。
以最终能力来衡量，新颖的高质量数据对预训练和后训练各自的边际价值是多少？
我们想大致了解合成数据的有效性如何。一个值得探究的具体问题是：如果你有一小部分高质量数据，通过合成数据生成来放大它，相对于仅仅在多个 epoch 上训练它，效果会好多少？
你可以通过实验室在数据经纪商、环境生成者等方面的支出，相对于其在算力和研究人员上的支出来推断数据的隐含价值。
我们想探究数据在推动 AI 进步中发挥了什么作用。探究这个问题还有很多其他方式，有些可能比我们的更巧妙、更有信息量。而且我们的实验也是在极小的规模上进行的。我们确实认为，我们很可能遗漏了某些东西——我们渴望听到其他人会如何研究这个问题，并且理想情况下也能看到他们的结果！
特别感谢 Charlie O'Neill 提供的许多有益讨论。
附录：方法论
我们在不同的数据语料上从头预训练这些模型配方，采用不同的计算预算，并使用多个独立种子（seed）进行实验。我们的计算预算为：1e17、3.16e17、1e18、3.16e18 和 1e19 FLOPs。计算统计的惯例是使用名义计算量 C = 6ND（N = 非嵌入参数数量，D = 数据 token 数）。
在每个计算预算下，我们改变参数数量（从而改变训练的 token 数量），以确定每个训练配方 × 语料组合的计算最优配比。我们使用语料上的留出损失（held-out loss）来确定这一计算最优点。由此，我们可以获得每个组合下游性能的计算扩展曲线，并最终从中提取出我们的计算乘数（compute multiplier）。
我们在所有运行中强制使用统一的 tokenizer 和上下文长度：GPT-2 BPE（tiktoken，50,257 词表）以及 T=2048，batch = 262,144 个 token。
我们训练运行的最终能力高度依赖于超参数。显然，不可能对所有可能的超参数组合进行穷举搜索（超参数调优确实是一门艺术！）。我们尽可能对此进行控制，并将峰值学习率视为最重要的超参数。
某些算法版本确实提供了峰值学习率应如何调整的规范（作为模型规模、数据预算、batch 大小等相关变量的函数）。这些为我们判断最优学习率提供了很好的先验依据。
我们首先在 5 个锚点上扫描学习率——涵盖 3 种不同的模型规模和 2 种不同的 D/N 比率。我们确定这些锚点的最优学习率，并拟合出一个最优学习率的参数化形式。
对于除 OLMo-2 之外的所有模型配方，我们拟合了共同的指数 a 和 b，以及各模型特有的 lr₀。对于 OLMo-2，我们按照其模型配方采用规定的最优学习率。我们对 OLMo-2 这样做的原因是，Ai2 在配方中发布了小规模模型的阶梯式配置，其中指定了我们所研究规模下的最优超参数。我们还在 3.16e18 FLOPs 的计算最优点上验证了我们的生产学习率处于或接近最优水平。
主要技术成果
解释我们图表中的一些异常现象
正如预期，我们观察到在模型维度和数据维度上，计算效率都随时间的推移总体呈上升趋势。我们注意到的一些离群点如下：
NeoX 在 1e19 处的表现不如 GPT-2（尽管它在 1e17 到 3.16e18 的范围内表现更好）。这可能源于 OLMES 评测中的噪声。我们还注意到，在 FineWeb-Edu 语料库上的保留预训练损失方面，NeoX 的表现优于 GPT-2。
The Pile 的表现似乎远差于 OpenWebText。这并不令人意外，因为 The Pile 的主要改进在于数据语料库的多样性，而非过滤。它包含一个经过精心挑选的 22 个来源的混合体，包括 PubMed 和 arXiv 论文、GitHub 代码、法律意见书、专利以及议会会议记录。对于其中许多 token 而言，向 OLMES（一个英文网页散文式多选题基准）的跨领域迁移可能微乎其微，从而导致计算效率较低。我们注意到，凭借其更大的规模，我们预期在更大的计算规模下，The Pile 最终应该会优于（非常小的）OpenWebText。
同样值得注意的是，NeoX 和 The Pile 的计算乘数是通过外推法获得的，这引入了额外的潜在误差。
计算乘数是如何计算的，以及它们的误差棒
计算缩放曲线上的每个点都是由多个独立种子的训练运行计算得出的。那里的误差棒是 OLMES 评估在这些种子上结果的标准差。
考虑我们的参考模型或数据语料库在某个计算水平下达到的某个给定的参考性能水平。
然后，我们通过找到我们候选模型或语料库的计算缩放曲线上首次达到该参考性能水平的最左侧点来计算计算乘数。参考模型所需计算量与候选模型所需计算量之比即为候选模型的计算乘数。
计算倍率上的误差条是通过对整个估计流程进行参数自助法（parametric bootstrap）得到的，为 1 个标准差区间。
我们确实想强调，我们预计模型配方（model recipes）计算倍率的实际不确定性会高于误差条所示的范围。这是因为我们仅进行了有限程度的超参数调优，由此引入了额外的不确定性，而且最终能力或保留集损失（held-out loss）可能对峰值学习率/批大小等的具体选择相当敏感。
同样重要的是要注意，我们的消融实验有很多原因可能无法完全覆盖计算效率提升的全部范围。事实上，从 2019 年到 2025 年，我们观察到模型侧的计算效率年化提升（CEG）为 1.24 倍 [1.19, 1.29]，数据侧为 1.51 倍 [1.45, 1.57]。联合衡量时，我们观察到 1.57 倍的逐年 CEG [1.49, 1.65]7。这确实远低于 Anson Ho 等人估计的 3 倍年均水平，原因如下：
许多提升可能是规模依赖的，或者可能在更长上下文下尤为重要，而我们运行的规模太小，无法实现其中的许多提升。
例如，OLMo-2 的层归一化和 QK 归一化，NeoX 中的并行注意力 + MLP 块
推理效率优化（例如 LLama-3 的 GQA，这是一种 KV cache 优化）在我们的研究中不会体现为计算倍率。我们也没有研究 tokenizer 的改进。
我们得到的算力倍增系数对每年所选用的模型配方或数据语料库相当敏感。我们选取的是我们认为具有代表性的模型配方或数据语料库，但绝不敢断言这些就是各年度最优的选择。
我们考察的是相对于 OLMES 基准（该基准综合了 10 种相对简单的任务类型）的算力倍增系数，而非达到某个困惑度指标所需的算力倍增系数。如果我们考察其他基准（例如，针对编程或问题求解的基准），得到的数字也会大不相同，因为这些基准可能更青睐截然不同的数据工程方法。
我们还想指出，我们并未研究其他数据侧的改进方法，例如从新来源收集更高质量的数据、人类专家生成的数据、合成数据生成方法等。我们研究的大多数语料库都是对同一个 Common Crawl 数据集的整理（子集），而非扩展可用数据的范围。这显然是在消耗有限的存量——这个杠杆我们能推动的空间是有限的。
模型配方与数据语料库带来的收益相互独立
以下是我们为确定模型配方与数据语料库带来的收益在多大程度上相互独立而进行的调查。我们观察了在 3.16e18 FLOPs 计算量下 OLMES 得分的网格。对 OLMES 得分 = 均值 + 模型效应 + 数据效应进行线性回归，得到的 R 平方为 0.88，这意味着 OLMES 得分中 88% 的方差可以由模型改进和数据改进的加性效应来解释，只有约 12% 的方差由交互项或高阶项以及评估噪声来解释。这暗示复杂的模型-数据交互（即利用某些模型改进依赖于特定的数据工程，反之亦然）相对次要。
Anson Ho 等人估计（预训练中的）软件效率改进为每年 3 倍（95% 置信区间：1.5 倍至 64 倍）。正如 Ho 在该博客中提到的，“大多数软件进步实际上可能归因于数据质量的改进”，以及“仅通过扩展少数几个依赖规模的算法变革即可实现”。
我们采用 C = 6ND 的名义约定来计算算力。
2019 年的模型配方是 GPT-2，2025 年的模型配方是 OLMo-2。2019 年的数据语料库是 OpenWebText，2025 年的数据语料库是 UltraFineWeb。
我们对 GPT3 的实现（在 Pile 数据集上）遇到了一些训练不稳定的情况（梯度尖峰）。
这些包括：优化器改进、预热 + 衰减调度、用 RoPE 取代学习绝对位置、RMSNorm + SwiGLU 门控 MLP、归一化层重排、QK-norm、Z-loss 正则化以及更干净的初始化。
对于计算规模扩展图，我们每种配置至少使用 3 个随机种子。在 3.16e18 算力预算下，模型配方与数据语料组合的 7x7 网格中，每种组合仅使用 1 个随机种子。
1.57 倍的同比乘数是根据从 2019 年模型与语料到 2025 年模型与语料的联合改进计算得出的，而非 1.24 倍模型侧改进与 1.51 倍数据侧改进的乘积。
How much of the rapid progress in AI that we’ve seen over the last few years1 has come from data versus model improvements? The answer has big implications for the economics of frontier labs and the pace of future progress.
We investigate this question at a relatively small scale, and for pretraining specifically, from 2019 to 2025. During each of those years, a new open model recipe was published which codified that year’s publicly known algorithmic tweaks (for example, improvements in architecture, optimizer, initializations, learning rate schedule, hyperparams, etc). And during each of those years, there was also a new public data corpus (produced by broader scrapes and new curation/extraction/filtering techniques).
We train combinations of these year-representative model recipes and data corpuses across different scales of training compute (up to 1e19 FLOPs)2.
Obviously, we can’t compare these different models by their cross-entropy loss against a fixed dataset, since we’re varying the datasets they’re trained on. So instead we evaluate these models on end capabilities as measured by the OLMES eval (which aggregates 10 different relatively easy benchmarks, mostly multiple choice QA). Unfortunately, evaluating end capability rather than pretraining loss adds some noise to our results, as you’ll see in the graphs below, though we try to get cleaner bounds by running multiple seeds.
We find that from 2019 to 2025, 3.24x more compute efficiency gains have come from data improvements rather than model improvements (12.0x for data and 3.7x for models), at the 1e19 FLOPs compute budget3.
Here is a grid which shows how much better a model we train does on the end capability we're testing it on, relative to the 2019 data + architecture baseline, at 3.16e18 FLOPs4.
We find that the gains from data and model improvements are mostly independent and don’t interact (i.e. realizing the gains from some model improvement doesn’t require a specific training datapile, or vice versa). 88% of the variance in the OLMES score can be explained by additive effects of the model and data improvements (using a linear model).
Discussion
For context, let’s briefly summarize what changed on both the data and the model side from 2019 to 2025.
On the model side, we went from GPT-2 to OLMo-2, including key innovations in optimizers, positional encodings, normalization, activation functions, initializations, and more5.
On the data side, we started with OpenWebText in 2019, which contained just web pages linked from Reddit with enough upvotes and then deduplicated and filtered, and thus amounted to only ~9B tokens (this was mostly what GPT-2 was trained on). By 2025, open source data corpuses like UltraFineWeb not only are far larger (by using scrapes of the whole Internet), but also use much more sophisticated filtering (for example, by training a classifier to predict what data will empirically improve model performance).
A naive interpretation of our result is that most of the AI progress from 2019-2024 (the era of pretraining) was just better data engineering (extraction, curation, etc.), and that all the model work during that period was much less important.
But this is probably the wrong way to think about the value of model improvements. Their main contribution was not necessarily compute efficiency - that is, achieving the same performance with fewer FLOPs. Rather, it was making larger amounts of compute usable in the first place. As the number of parameters, context lengths, run duration, and clusters scale up, all kinds of things are prone to breaking (gradients explode or vanish, memory and bandwidth run out, training becomes infeasibly slow). Much of model research has consisted of removing or pushing back these constraints to scaling. Many of the most important innovations such as MoEs, sparse attention variants, stability innovations (norm placements, initializations, etc.) and system / kernel-level optimizations like FlashAttention fall into this category.
The data improvements we investigated here might matter less for larger models. Small models (like the ones we trained) see significant gains from data quality improvements, because they don’t have that much capacity, and so you have to be really careful about what you stuff into them. Whereas big models have so much excess capacity that maybe you just want to throw in as much stuff as you can, even if it’s mostly garbage, and the magic of stochastic gradient descent will separate out the signal from the noise. If you choose to filter aggressively, you’ll have to do dozens of epochs, which empirically gives worse performance than just having a lower average quality but larger dataset. In fact, aggressive data curation is even more harmful once you take into account that frontier models are up to 100x overtrained relative to Chinchilla optimal, in order to minimize the inference compute used for RL and for deployment.
An analogy might be the difference between a sailboat and a container ship - the container ship doesn’t necessarily go faster, but it can lug thousands of tons of cargo (analogous to hundreds of trillions of tokens of pretraining data), and won’t be toppled by choppy waters (analogous to training stably across hundreds of thousands of GPUs).
Now that we have more capacious and sturdy container ships, we don’t have to fret about exactly what we load on board - we can just fill them up with everything that’s even remotely and plausibly useful. Whereas for the tiny flimsy sailboats of 2019, you’d have to be incredibly careful about only carrying the most valuable cargo.
But to the extent that the nature of pretraining progress is simply loading more cargo into this ship, are we running out of cargo? This is a question about the data wall and about how well synthetic data has helped us leap over it. Synthetic data is obviously being widely used at the labs, and we have not at all investigated whether it can effectively expand a data corpus without hurting model performance. If the gains are limited, then the main driver of pretraining progress will stall, because we’re not generating more internet, and you can only curate a fixed set of data by so much. To be clear, we have no active reason to think this. But given how important data seems to be in driving pretraining progress, this seems like a crucial question to investigate.
Ryan Greenblatt noted that many of the historical improvements in pretraining data corpuses look like the kind of progress that automated researchers would be able to just test empirically - for example, run ablations trained on different data and see how the model performs. So it’s totally compatible with our results that the data progress which has propelled pretraining since 2019 might speed up a lot if and when we automate AI R&D.
We want to clarify that whether pretraining progress in isolation will speed up or slow down is not really the most important question for overall AI progress, because so many of the gains over the last two years have come from RL.
Future research
These are some directions of future research that we think would be really cool, and important questions to answer:
You could run this experiment at larger scales to see whether the data or model improvements are more dependent on scale (and thus far more impactful at the frontier)
What is the marginal value of novel high-quality data for both pre and post-training, as measured by end capabilities?
We want to know broadly how effectively synthetic data works. One concrete question to investigate is this: if you’ve got a small corpus of high quality data, how much better is it to magnify it via synthetic data generation relative to just training on it for multiple epochs?
You could figure out the implied value of data through lab spending on data brokers, environment producers, etc., relative to their spending on compute and researchers.
We wanted to investigate what role data has played in driving AI progress. There are lots of other ways one could probe this question, and some may be more clever and informative than ours. And even our experiment was done at an extremely small scale. We definitely think it’s plausible that there is something we missed - we’re eager to hear how others would research this question, and ideally to also see their results!
Thanks especially to Charlie O’Neill for many helpful discussions.
Appendix: Methodology
We pre-train these model recipes from scratch on these different data corpuses, at varying compute budgets, with multiple independent seeds6. Our compute budgets are: 1e17, 3.16e17, 1e18, 3.16e18 and 1e19 FLOPs. The compute accounting convention is to use nominal compute C = 6ND (N = number of non-embedding parameters, D = tokens of data).
At each compute budget, we vary the number of parameters (and hence number of tokens trained on), to determine the compute-optimal mix for each training recipe x corpus combination. We use held-out loss on the corpus to determine this compute-optimal point. We can then obtain compute scaling curves of downstream performance of each combination, from which we can finally extract our compute multipliers.
We enforce a shared tokenizer and context length across every run: GPT-2 BPE (tiktoken, 50,257 vocab) and T=2048, batch = 262,144 tokens.
The end capabilities of our training runs are highly dependent on hyperparameters. Obviously, there is no way to sweep over all possible sets of hyperparams (hyperparam tuning is a fine art indeed)! We try to control for this as much as possible, and we consider peak learning rate as the main hyperparameter of significance.
Some algorithm vintages do provide specifications of what peak learning rate should be tuned to (as a function of other relevant variables such as model size, data budget, batch size, etc.). These serve as good priors for what we think the optimal learning rate is.
We first sweep learning rates at 5 anchor points - 3 different model sizes and 2 different D/N ratios. We determine the optimal learning rate of these anchor points, and fit an optimal learning rate parametric form
For all the model recipes except OLMo-2, we fit a common exponent a and b, and a model-specific lr₀. For OLMo-2, we use the prescribed optimal learning rate according to the model recipe. The reason we do this for OLMo-2 is that Ai2 published small-model ladders as part of the recipe which specified optimal hyperparameters at the scale we are investigating. We also verify, at the compute-optimal point for 3.16e18 FLOPs, that our production learning rates are at or near optimal.
Main technical results
Explaining some anomalies in our graph
We observe generally increasing compute efficiency across time for both the model and data axes as expected. Some outliers that we observed:
NeoX performs worse than GPT-2 at 1e19 (although it does better across the 1e17 to 3.16e18 range). This might arise from noise in the OLMES evaluation. We also note that on held-out pretraining loss on the FineWeb-Edu corpus, NeoX performs better than GPT-2.
The Piles seems to do much worse than OpenWebText. This is not surprising since the Pile’s main improvement was data corpus diversity over filtering. It has a curated 22-source mixture including PubMed and arXiv papers, GitHub code, legal opinions, patents, and parliamentary proceedings. The amount of cross-domain transfer to OLMES (which is English web-prose MCQ) might be minimal for many of these tokens, thus resulting in lower compute efficiency. We note that by virtue of its larger size, we expect that the Pile should eventually be better than (the really small) OpenWebText at larger scales.
It is also worth noting that the compute multipliers for NeoX and the Pile are obtained by extrapolation, which introduces further potential error.
How compute multipliers were calculated, as well as their error bars
Every point on the compute scaling curves is computed from multiple independently seeded training runs. The error bars there are the standard deviation of the OLMES eval over those seeds.
Consider some given reference level of performance at some compute level for our reference model or data corpus.
We then calculate the compute multiplier by finding the left-most point of the compute scaling curve of our candidate model or corpus that first attains that reference level of performance. The ratio of the compute required by the reference to the compute required by our candidate is the candidate’s compute multiplier
The error bars on the compute multipliers are obtained from a parametric bootstrap of the entire estimation pipeline, and are 1 standard deviation intervals
We do want to highlight that we expect the actual uncertainty in the compute multipliers of the model recipes to be higher than indicated by our error bars. This is because of additional uncertainty introduced by the limited extent of hyperparameter tuning we did, and end capabilities or held-out loss is probably quite sensitive to the exact choice of peak learning rate / batch size / etc.
It is also important to note that there are many reasons why our ablations do not necessarily capture the full scope of compute efficiency gains. Indeed, from 2019 to 2025, we observe year-over-year compute efficiency gains (CEG) of 1.24x [1.19, 1.29] on the model side and 1.51x [1.45, 1.57] on the data side. Measured jointly, we observe a 1.57x YoY CEG [1.49, 1.65]7. This is indeed much lower than Anson Ho et al.’s mean estimate of 3x YoY, for the following reasons:
Many of the gains might be scale dependent or might be especially important at longer context, and we are operating at scales too small to realize many of the gains.
For example, OLMo-2’s layer and QK norms, parallel attention + MLP block in NeoX
Inference efficiency optimizations (such as LLama-3’s GQA, which is a KV cache optimization) do not show up as compute multipliers in our study. We are also not investigating tokenizer improvements.
The compute multipliers we obtain are pretty sensitive to our choice of model recipe or data corpus for each year. We have chosen what we believe to be representative model recipes or data corpuses. But by no means do we exhaustively conclude that these are the best of each year.
We are looking at compute multipliers with respect to the OLMES benchmark (which combines 10 different relatively easy task types) rather than compute multipliers in getting to some perplexity metric. We would also have very different looking numbers if we were looking at other benchmarks (say, coding- or problem-solving-specific ones), which would probably reward very different methods of data engineering.
We also want to note that we have not investigated other data-side improvements, such as collecting more high-quality data from new sources, human expert generated data, synthetic data generation methods, etc. Most of the corpuses we have investigated are curations (subsets) of the same Common Crawl, rather than expanding the available set of data. This is clearly consumption of a finite stock - there is only so far we can push this lever.
Independence of gains from model recipe and data corpus
Here is the investigation that we did to determine how independent the gains from model recipe and data corpus are. We looked at the grid of OLMES scores at 3.16e18 FLOPs. A linear regression of OLMES score = mean + model effect + data effect gives an R squared of 0.88, which means 88% of the variance in the OLMES score can be explained by additive effects of the model and data improvements, with only ~12% of the variance accounted for by interaction or higher order terms, and eval noise. This hints that complex model-data interactions (where exploiting some model improvement is contingent on some specific data engineering, or vice versa) are relatively minor.
Anson Ho et al. estimated software efficiency improvements (in pretraining) of 3x per year (95% CI: 1.5x to 64x). As Ho mentioned in this blog, “most software progress might actually be due to data quality improvements” and “from scaling up just a small handful of scale-dependent algorithmic changes”.
We use the C = 6ND nominal convention for accounting for compute.
The 2019 model recipe was GPT-2, and 2025 model recipe was OLMo-2. The 2019 data corpus was OpenWebText, and the 2025 data corpus was UltraFineWeb.
Our implementation of GPT3 encountered some training instabilities (gradient spikes) on the Pile.
These include: Optimizer improvements, warmup + decay schedules, RoPE replacing learning absolute positions, RMSNorm + SwiGLU gated MLPs, Norm reordering, QK-norm, Z-loss regularization and cleaner inits.
For the compute-scaling plots we use at least 3 seeds each. For the 7x7 grid of combinations of model recipes and data corpus at the 3.16e18 budget, we only used 1 seed each.
The 1.57x YoY multiplier is computed using the joint improvement from 2019 model and corpus to 2025 model and corpus, and not the product of the 1.24x model side improvement and 1.51x data side improvement.